diff --git a/asian_ai/performance/vLLM/SECURITY.md b/asian_ai/performance/vLLM/SECURITY.md new file mode 100644 index 0000000000000000000000000000000000000000..df2425b86c8023cbca7f83fac066e2f8892e77ff --- /dev/null +++ b/asian_ai/performance/vLLM/SECURITY.md @@ -0,0 +1,59 @@ +# Security Policy + +## Reporting security issues + +Please report security issues privately using [the vulnerability submission form](https://github.com/vllm-project/vllm/security/advisories/new). + +## Issue triage + +Reports will then be triaged by the [vulnerability management team](https://docs.vllm.ai/en/latest/contributing/vulnerability_management.html). + +## Threat model + +Please see the [Security Guide in the vLLM documentation](https://docs.vllm.ai/en/latest/usage/security.html) for more information on vLLM's security assumptions and recommendations. + +Please see [PyTorch's Security Policy](https://github.com/pytorch/pytorch/blob/main/SECURITY.md) for more information and recommendations on how to securely interact with models. + +## Issue severity + +We will determine the risk of each issue, taking into account our experience dealing with past issues, versions affected, common defaults, and use cases. We use the following severity categories: + +### CRITICAL Severity + +Vulnerabilities that allow remote attackers to execute arbitrary code, take full control of the system, or significantly compromise confidentiality, integrity, or availability without any interaction or privileges needed, examples include remote code execution via network, deserialization issues that allow exploit chains. Generally those issues which are rated as CVSS ≥ 9.0. + +### HIGH Severity + +Serious security flaws that allow elevated impact—like RCE in specific, limited contexts or significant data loss—but require advanced conditions or some trust, examples include RCE in advanced deployment modes (e.g. multi-node), or high impact issues where some sort of privileged network access is required. These issues typically have CVSS scores between 7.0 and 8.9 + +### MODERATE Severity + +Vulnerabilities that cause denial of service or partial disruption, but do not allow arbitrary code execution or data breach and have limited impact. These issues have a CVSS rating between 4.0 and 6.9 + +### LOW Severity + +Minor issues such as informational disclosures, logging errors, non-exploitable flaws, or weaknesses that require local or high-privilege access and offer negligible impact. Examples include side channel attacks or hash collisions. These issues often have CVSS scores less than 4.0 + +## Fix disclosure policy + +When a security report is accepted, the fix process depends on the severity: + +* **CRITICAL and HIGH severity**: Fixes are developed in a private security fork and coordinated with the prenotification group before public disclosure. +* **MODERATE and LOW severity**: Fixes are developed and submitted as public pull requests. These issues do not require embargo since they do not enable arbitrary code execution or significant data breach, and public visibility accelerates community review and adoption of the fix. + +The vulnerability management team reserves the right to adjust the disclosure approach on a case-by-case basis, taking into account factors such as active exploitation, unusual attack surface, or coordination requirements with downstream vendors. + +## Prenotification policy + +For certain security issues of CRITICAL, HIGH, or MODERATE severity level, we may prenotify certain organizations or vendors that ship vLLM. The purpose of this prenotification is to allow for a coordinated release of fixes for severe issues. + +* This prenotification will be in the form of a private email notification. It may also include adding security contacts to the GitHub security advisory, typically a few days before release. + +* If you wish to be added to the prenotification group, please send an email copying all the members of the [vulnerability management team](https://docs.vllm.ai/en/latest/contributing/vulnerability_management.html). Each vendor contact will be analyzed on a case-by-case basis. + +* Organizations and vendors who either ship or use vLLM, are eligible to join the prenotification group if they meet at least one of the following qualifications + * Substantial internal deployment leveraging the upstream vLLM project. + * Established internal security teams and comprehensive compliance measures. + * Active and consistent contributions to the upstream vLLM project. + +* We may withdraw organizations from receiving future prenotifications if they release fixes or any other information about issues before they are public. Group membership may also change based on policy refinements for who may be included. diff --git a/asian_ai/performance/vLLM/rust-toolchain.toml b/asian_ai/performance/vLLM/rust-toolchain.toml new file mode 100644 index 0000000000000000000000000000000000000000..50c7b070fe87e55b12a623fd7c29242ff756a52c --- /dev/null +++ b/asian_ai/performance/vLLM/rust-toolchain.toml @@ -0,0 +1,2 @@ +[toolchain] +channel = "1.95" diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_hermes.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_hermes.jinja new file mode 100644 index 0000000000000000000000000000000000000000..1c399aed01b2c3d8ca1782e971f02197fa304250 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_hermes.jinja @@ -0,0 +1,130 @@ +{%- macro json_to_python_type(json_spec) %} + {%- set basic_type_map = { + "string": "str", + "number": "float", + "integer": "int", + "boolean": "bool" +} %} + + {%- if basic_type_map[json_spec.type] is defined %} + {{- basic_type_map[json_spec.type] }} + {%- elif json_spec.type == "array" %} + {{- "list[" + json_to_python_type(json_spec|items) + "]" }} + {%- elif json_spec.type == "object" %} + {%- if json_spec.additionalProperties is defined %} + {{- "dict[str, " + json_to_python_type(json_spec.additionalProperties) + ']' }} + {%- else %} + {{- "dict" }} + {%- endif %} + {%- elif json_spec.type is iterable %} + {{- "Union[" }} + {%- for t in json_spec.type %} + {{- json_to_python_type({"type": t}) }} + {%- if not loop.last %} + {{- "," }} + {%- endif %} + {%- endfor %} + {{- "]" }} + {%- else %} + {{- "Any" }} + {%- endif %} +{%- endmacro %} + + +{{- bos_token }} +{{- "<|im_start|>system\nYou are a function calling AI model. You are provided with function signatures within XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools: " }} +{%- if tools is iterable and tools | length > 0 %} + {%- for tool in tools %} + {%- if tool.function is defined %} + {%- set tool = tool.function %} + {%- endif %} + {{- '{"type": "function", "function": ' }} + {{- '{"name": "' + tool.name + '", ' }} + {{- '"description": "' + tool.name + '(' }} + {%- for param_name, param_fields in tool.parameters.properties|items %} + {{- param_name + ": " + json_to_python_type(param_fields) }} + {%- if not loop.last %} + {{- ", " }} + {%- endif %} + {%- endfor %} + {{- ")" }} + {%- if tool.return is defined %} + {{- " -> " + json_to_python_type(tool.return) }} + {%- endif %} + {{- " - " + tool.description + "\n\n" }} + {%- for param_name, param_fields in tool.parameters.properties|items %} + {%- if loop.first %} + {{- " Args:\n" }} + {%- endif %} + {{- " " + param_name + "(" + json_to_python_type(param_fields) + "): " + param_fields.description|trim }} + {%- endfor %} + {%- if tool.return is defined and tool.return.description is defined %} + {{- "\n Returns:\n " + tool.return.description }} + {%- endif %} + {{- '"' }} + {{- ', "parameters": ' }} + {%- if tool.parameters.properties | length == 0 %} + {{- "{}" }} + {%- else %} + {{- tool.parameters|tojson }} + {%- endif %} + {{- "}" }} + {%- if not loop.last %} + {{- "\n" }} + {%- endif %} + {%- endfor %} +{%- endif %} +{{- " " }} +{{- 'Use the following pydantic model json schema for each tool call you will make: {"properties": {"name": {"title": "Name", "type": "string"}, "arguments": {"title": "Arguments", "type": "object"}}, "required": ["name", "arguments"], "title": "FunctionCall", "type": "object"}} +' }} +{{- "For each function call return a json object with function name and arguments within XML tags as follows: +" }} +{{- " +" }} +{{- '{"name": , "arguments": } +' }} +{{- '<|im_end|>' }} +{%- for message in messages %} + {%- if message.role == "user" or message.role == "system" or (message.role == "assistant" and message.tool_calls is not defined) %} + {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }} + {%- elif message.role == "assistant" and message.tool_calls is defined %} + {{- '<|im_start|>' + message.role }} + {%- for tool_call in message.tool_calls %} + {{- '\n\n' }} + {%- if tool_call.function is defined %} + {%- set tool_call = tool_call.function %} + {%- endif %} + {{- '{' }} + {{- '"name": "' }} + {{- tool_call.name }} + {{- '"' }} + {%- if tool_call.arguments is defined %} + {{- ', ' }} + {{- '"arguments": ' }} + {{- tool_call.arguments|tojson }} + {%- endif %} + {{- '}' }} + {{- '\n' }} + {%- endfor %} + {{- '<|im_end|>\n' }} + {%- elif message.role == "tool" %} + {%- if loop.previtem and loop.previtem.role != "tool" %} + {{- '<|im_start|>tool\n' }} + {%- endif %} + {{- '\n' }} + {{- message.content }} + {%- if not loop.last %} + {{- '\n\n' }} + {%- else %} + {{- '\n' }} + {%- endif %} + {%- if not loop.last and loop.nextitem.role != "tool" %} + {{- '<|im_end|>' }} + {%- elif loop.last %} + {{- '<|im_end|>' }} + {%- endif %} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|im_start|>assistant\n' }} +{%- endif %} diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_hunyuan_a13b.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_hunyuan_a13b.jinja new file mode 100644 index 0000000000000000000000000000000000000000..8e3fd1fe6915f0b3f3dcc9901268b28708144ccc --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_hunyuan_a13b.jinja @@ -0,0 +1,113 @@ +{% set loop_messages = messages %} +{% if tools %} + {% set weekday_map = {'Monday': '星期一', 'Tuesday': '星期二', 'Wednesday': '星期三', 'Thursday': '星期四', 'Friday': '星期五', 'Saturday': '星期六', 'Sunday': '星期日'} %} + {% set weekday_cn = weekday_map[strftime_now('%A')] %} + {% set datetime_str = strftime_now('%Y-%m-%d %H:%M:%S') %} + {% set datetime_str = datetime_str + ' ' + weekday_cn %} + {% for message in loop_messages %} + {% if 'content' in message %} + {% set content = message['content'] %} + {% else %} + {% set content = '' %} + {% endif %} + {% if loop.index0 == 0 %} + {% set content_tmp = '你是一位函数组合专家。你会得到一个问题和一组可能的函数。根据问题,你需要进行一个或多个函数/工具调用以实现目的。 +如果没有一个函数可以使用,请直接使用自然语言回复用户,以助手:开头。 +如果给定的问题缺少函数所需的参数,请使用自然语言进行提问,向用户询问必要信息,以助手:开头。 +如果调用结果已经足够回答用户问题,请对历史结果进行总结,使用自然语言回复用户,以助手:开头。 +你应该只在工具调用部分返回函数调用。如果你决定调用任何函数,你必须将其格式化为[{"name": "func_name1", "arguments": {"argument1": "value1", "argument2": "value2"}},...]。你不应该在回复中包含任何其他文本。以下是你可以调用的函数列表,格式为JSON。 +' %} + {% set content_tmp = content_tmp + ' +' + tools | tojson + ' +' %} + {% if message['role'] == 'system' %} + {% set content_tmp = content_tmp + ' +额外要求: +' + content + ' + +如果你决定返回函数调用,请将其格式化为[{"name": "func_name1", "arguments": {"argument1": "value1", "argument2": "value2"}},...],不得包含其他文本。如果额外要求里有格式要求,请忽略,以此处为准。 +否则,请参考开头说的三种情况,以助手:开头进行回复。 + +如果额外要求里有时间信息,就以额外要求里的时间为准,否则,参考当前时间:' + datetime_str %} + {% set content = '<|startoftext|>' + content_tmp + '<|extra_4|>' %} + {% elif message['role'] == 'user' %} + {% set content_tmp = content_tmp + ' +如果你决定返回函数调用,请将其格式化为[{"name": "func_name1", "arguments": {"argument1": "value1", "argument2": "value2"}},...],不得包含其他文本。 +否则,请参考开头说的三种情况,以助手:开头进行回复。 + +当前时间:' + datetime_str %} + {% set content_tmp = '<|startoftext|>' + content_tmp + '<|extra_4|>'%} + {% set content = content_tmp + '用户:' + content + '<|extra_0|>' %} + {% endif %} + {% else %} + {% if message['role'] == 'user' %} + {% set content = '用户:' + content + '<|extra_0|>' %} + {% elif message['role'] == 'assistant' %} + {% if 'tool_calls' in message %} + {% set tool_calls = message['tool_calls'] %} + {% set ns = namespace(tool_calls="[") %} + {% for tool_call in tool_calls %} + {% set function = tool_call['function'] %} + {% set name = function['name'] %} + {% set ns.tool_calls = ns.tool_calls + '{"name": "' + name + '", '%} + {% set arguments = function['arguments'] %} + {% if arguments is not string %} + {% set arguments = arguments | tojson %} + {% endif %} + {% set ns.tool_calls = ns.tool_calls + '"arguments": ' + arguments + '}' %} + {% if not loop.last %} + {% set ns.tool_calls = ns.tool_calls + ', '%} + {% endif %} + {% endfor %} + {% set ns.tool_calls = ns.tool_calls + ']' %} + {% set content = content + '' + ns.tool_calls + '' %} + {% else %} + {% set content = '助手:' + content %} + {% endif %} + {% set content = content + '<|eos|>' %} + {% elif message['role'] == 'tool' %} + {% if content is not string %} + {set content = content | tojson } + {% endif %} + {% set content = '' + content + '' %} + {% set content = content + '<|extra_0|>' %} + {% endif %} + {% endif %} + {{- content -}} + {% endfor %} +{% else %} + {% set context = {'has_head': true} %} + {% for message in loop_messages %} + {% if 'content' in message %} + {% set content = message['content'] %} + {% else %} + {% set content = '' %} + {% endif %} + {% if loop.index0 == 0 %} + {% if content == '' %} + {% set _ = context.update({'has_head': false}) %} + {% elif message['role'] == 'system' %} + {% set content = '<|startoftext|>' + content + '<|extra_4|>' %} + {% endif %} + {% endif %} + {% if message['role'] == 'user' %} + {% if loop.index0 == 1 and not context.has_head %} + {% set content = '<|startoftext|>' + content %} + {% endif %} + {% if loop.index0 == 1 and context.has_head %} + {% set content = content + '<|extra_0|>' %} + {% else %} + {% set content = '<|startoftext|>' + content + '<|extra_0|>' %} + {% endif %} + {% elif message['role'] == 'assistant' %} + {% set content = content + '<|eos|>' %} + {% elif message['role'] == 'tool' %} + {% set content = content + '<|extra_0|>' %} + {% endif %} + {{- content -}} + {% endfor %} +{% endif %} +{%- if enable_thinking is defined and enable_thinking is false %} + {{- '\n\n\n' }} +{%- endif %} + diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_internlm2_tool.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_internlm2_tool.jinja new file mode 100644 index 0000000000000000000000000000000000000000..9ad38a1593761bab38d1709c94c601553b55580c --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_internlm2_tool.jinja @@ -0,0 +1,60 @@ +{%- if messages[0]["role"] == "system" %} + {%- set system_message = messages[0]["content"] %} + {%- set loop_messages = messages[1:] %} +{%- else %} + {%- set loop_messages = messages %} +{%- endif %} + +{%- if not tools is defined %} + {%- set tools = none %} +{%- endif %} + +{{- bos_token }} +{%- if system_message is defined %} +{{- "<|im_start|>system\n" + system_message + "<|im_end|>\n" }} +{%- endif %} + +{%- if tools is not none %} + {{- "<|im_start|>system name=<|plugin|>\n[" }} + {%- for tool in tools %} + {{- tool.function|tojson }} + {%- if not loop.last %} + {{- ", " }} + {%- else %} + {{- "]" }} + {%- endif %} + {%- endfor %} + {{- "<|im_end|>\n" }} +{%- endif %} + +{%- for message in loop_messages %} + {%- if message["role"] == "user" %} + {{- "<|im_start|>user\n" + message["content"] + "<|im_end|>\n"}} + {%- elif message.tool_calls is defined and message.tool_calls is not none %} + {%- set content = message["content"] if message["content"] else "" %} + {{- "<|im_start|>assistant\n" + content }} + {%- for tool_call in message.tool_calls %} + {%- set function=tool_call.function %} + {{- "<|action_start|><|plugin|>\n" }} + {{- '{"name": "' + function.name + '", '}} + {{- '"arguments": ' + function.arguments|tojson + '}' }} + {{- "<|action_end|>" }} + {%- endfor %} + {{- "<|im_end|>\n" }} + {%- elif message["role"] == "assistant" %} + {{- "<|im_start|>assistant\n" + message["content"] + "<|im_end|>\n"}} + {%- elif message["role"] == "tool_results" or message["role"] == "tool" or message["role"] == "function" %} + {%- if message.content is defined and message.content.content is defined %} + {%- set content = message.content.content %} + {%- else %} + {%- set content = message.content %} + {%- endif %} + {{- "<|im_start|>environment name=<|plugin|>\n" + content|string + "<|im_end|>\n" }} + {%- else %} + {{- raise_exception("Only user and assistant and tool_results and tool and function roles are supported, with the exception of an initial optional system message!") }} + {%- endif %} +{%- endfor %} + +{%- if add_generation_prompt %} +{{- '<|im_start|>assistant\n' }} +{%- endif %} \ No newline at end of file diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.1_json.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.1_json.jinja new file mode 100644 index 0000000000000000000000000000000000000000..ea245f4dbc446b347f7b371ae73112bb6be09837 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.1_json.jinja @@ -0,0 +1,120 @@ +{{- bos_token }} +{%- if custom_tools is defined %} + {%- set tools = custom_tools %} +{%- endif %} +{%- if not tools_in_user_message is defined %} + {#- Llama 3.1 doesn't pass all tests if the tools are in the system prompt #} + {%- set tools_in_user_message = true %} +{%- endif %} +{%- if not date_string is defined %} + {%- if strftime_now is defined %} + {%- set date_string = strftime_now("%d %b %Y") %} + {%- else %} + {%- set date_string = "26 Jul 2024" %} + {%- endif %} +{%- endif %} +{%- if not tools is defined %} + {%- set tools = none %} +{%- endif %} + +{#- This block extracts the system message, so we can slot it into the right place. #} +{%- if messages[0]['role'] == 'system' %} + {%- if messages[0]['content'] is string %} + {%- set system_message = messages[0]['content']|trim %} + {%- else %} + {%- set system_message = messages[0]['content'][0]['text']|trim %} + {%- endif %} + {%- set messages = messages[1:] %} +{%- else %} + {%- if tools is not none %} + {%- set system_message = "You are a helpful assistant with tool calling capabilities. Only reply with a tool call if the function exists in the library provided by the user. If it doesn't exist, just reply directly in natural language. When you receive a tool call response, use the output to format an answer to the original user question." %} + {%- else %} + {%- set system_message = "" %} + {%- endif %} +{%- endif %} + +{#- System message #} +{{- "<|start_header_id|>system<|end_header_id|>\n\n" }} +{%- if tools is not none %} + {{- "Environment: ipython\n" }} +{%- endif %} +{{- "Cutting Knowledge Date: December 2023\n" }} +{{- "Today Date: " + date_string + "\n\n" }} +{%- if tools is not none and not tools_in_user_message %} + {{- "You have access to the following functions. To call a function, please respond with JSON for a function call. " }} + {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. ' }} + {{- "Do not use variables.\n\n" }} + {%- for t in tools %} + {{- t | tojson(indent=4) }} + {{- "\n\n" }} + {%- endfor %} +{%- endif %} +{{- system_message }} +{{- "<|eot_id|>" }} + +{#- Custom tools are passed in a user message with some extra guidance #} +{%- if tools_in_user_message and not tools is none %} + {#- Extract the first user message so we can plug it in here #} + {%- if messages | length != 0 %} + {%- if messages[0]['content'] is string %} + {%- set first_user_message = messages[0]['content']|trim %} + {%- else %} + {%- set first_user_message = messages[0]['content'] | selectattr('type', 'equalto', 'text') | map(attribute='text') | map('trim') | join('\n') %} + {%- endif %} + {%- set messages = messages[1:] %} + {%- else %} + {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }} + {%- endif %} + {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}} + {{- "Given the following functions, please respond with a JSON for a function call " }} + {{- "with its proper arguments that best answers the given prompt.\n\n" }} + {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. ' }} + {{- "Do not use variables.\n\n" }} + {%- for t in tools %} + {{- t | tojson(indent=4) }} + {{- "\n\n" }} + {%- endfor %} + {{- first_user_message + "<|eot_id|>"}} +{%- endif %} + +{%- for message in messages %} + {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %} + {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n' }} + {%- if message['content'] is string %} + {{- message['content'] | trim}} + {%- else %} + {%- for content in message['content'] %} + {%- if content['type'] == 'text' %} + {{- content['text'] | trim }} + {%- endif %} + {%- endfor %} + {%- endif %} + {{- '<|eot_id|>' }} + {%- elif 'tool_calls' in message %} + {%- if not message.tool_calls|length == 1 %} + {{- raise_exception("This model only supports single tool-calls at once!") }} + {%- endif %} + {%- set tool_call = message.tool_calls[0].function %} + {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}} + {{- '{"name": "' + tool_call.name + '", ' }} + {{- '"parameters": ' }} + {{- tool_call.arguments | tojson }} + {{- "}" }} + {{- "<|eot_id|>" }} + {%- elif message.role == "tool" or message.role == "ipython" %} + {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }} + {%- if message.content is string %} + {{- { "output": message.content } | tojson }} + {%- else %} + {%- for content in message['content'] %} + {%- if content['type'] == 'text' %} + {{- { "output": content['text'] } | tojson }} + {%- endif %} + {%- endfor %} + {%- endif %} + {{- "<|eot_id|>" }} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }} +{%- endif %} diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.2_json.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.2_json.jinja new file mode 100644 index 0000000000000000000000000000000000000000..5950c6773b8f1a4abcb12dd3c457b08c609cd1db --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.2_json.jinja @@ -0,0 +1,133 @@ +{{- bos_token }} +{%- if custom_tools is defined %} + {%- set tools = custom_tools %} +{%- endif %} +{%- if not tools_in_user_message is defined %} + {%- set tools_in_user_message = false %} +{%- endif %} +{%- if not date_string is defined %} + {%- if strftime_now is defined %} + {%- set date_string = strftime_now("%d %b %Y") %} + {%- else %} + {%- set date_string = "26 Jul 2024" %} + {%- endif %} +{%- endif %} +{%- if not tools is defined %} + {%- set tools = none %} +{%- endif %} + +{#- Find out if there are any images #} +{% set image_ns = namespace(has_images=false) %} +{%- for message in messages %} + {%- for content in message['content'] %} + {%- if content['type'] == 'image' %} + {%- set image_ns.has_images = true %} + {%- endif %} + {%- endfor %} +{%- endfor %} + +{#- This block extracts the system message, so we can slot it into the right place. #} +{%- if messages[0]['role'] == 'system' %} + {%- if messages[0]['content'] is string %} + {%- set system_message = messages[0]['content']|trim %} + {%- else %} + {%- set system_message = messages[0]['content'][0]['text']|trim %} + {%- endif %} + {%- set messages = messages[1:] %} +{%- else %} + {%- if tools is not none %} + {%- set system_message = "You are a helpful assistant with tool calling capabilities. Only reply with a tool call if the function exists in the library provided by the user. If it doesn't exist, just reply directly in natural language. When you receive a tool call response, use the output to format an answer to the original user question." %} + {%- else %} + {%- set system_message = "" %} + {%- endif %} +{%- endif %} + +{#- System message if there are no images, if the user supplied one, or if tools are used (default tool system message) #} +{%- if system_message or not image_ns.has_images %} + {{- "<|start_header_id|>system<|end_header_id|>\n\n" }} + {%- if tools is not none %} + {{- "Environment: ipython\n" }} + {%- endif %} + {{- "Cutting Knowledge Date: December 2023\n" }} + {{- "Today Date: " + date_string + "\n\n" }} + {%- if tools is not none and not tools_in_user_message %} + {{- "You have access to the following functions. To call a function, please respond with JSON for a function call. " }} + {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. ' }} + {{- "Do not use variables.\n\n" }} + {%- for t in tools %} + {{- t | tojson(indent=4) }} + {{- "\n\n" }} + {%- endfor %} + {%- endif %} + {{- system_message }} + {{- "<|eot_id|>" }} +{%- endif %} + +{#- Custom tools are passed in a user message with some extra guidance #} +{%- if tools_in_user_message and not tools is none %} + {#- Extract the first user message so we can plug it in here #} + {%- if messages | length != 0 %} + {%- if messages[0]['content'] is string %} + {%- set first_user_message = messages[0]['content']|trim %} + {%- else %} + {%- set first_user_message = messages[0]['content'] | selectattr('type', 'equalto', 'text') | map(attribute='text') | map('trim') | join('\n') %} + {%- endif %} + {%- set messages = messages[1:] %} + {%- else %} + {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }} + {%- endif %} + {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}} + {{- "Given the following functions, please respond with a JSON for a function call " }} + {{- "with its proper arguments that best answers the given prompt.\n\n" }} + {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. ' }} + {{- "Do not use variables.\n\n" }} + {%- for t in tools %} + {{- t | tojson(indent=4) }} + {{- "\n\n" }} + {%- endfor %} + {{- first_user_message + "<|eot_id|>"}} +{%- endif %} + +{%- for message in messages %} + {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %} + {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n' }} + {%- if message['content'] is string %} + {{- message['content'] | trim}} + {%- else %} + {%- for content in message['content'] %} + {%- if content['type'] == 'image' %} + {{- '<|image|>' }} + {%- elif content['type'] == 'text' %} + {{- content['text'] | trim }} + {%- endif %} + {%- endfor %} + {%- endif %} + {{- '<|eot_id|>' }} + {%- elif 'tool_calls' in message %} + {%- if not message.tool_calls|length == 1 %} + {{- raise_exception("This model only supports single tool-calls at once!") }} + {%- endif %} + {%- set tool_call = message.tool_calls[0].function %} + {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}} + {{- '{"name": "' + tool_call.name + '", ' }} + {{- '"parameters": ' }} + {{- tool_call.arguments | tojson }} + {{- "}" }} + {{- "<|eot_id|>" }} + {%- elif message.role == "tool" or message.role == "ipython" %} + {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }} + {%- if message.content is string %} + {{- { "output": message.content } | tojson }} + {%- else %} + {%- for content in message['content'] %} + {%- if content['type'] == 'text' %} + {{- { "output": content['text'] } | tojson }} + {%- endif %} + {%- endfor %} + {%- endif %} + {{- "<|eot_id|>" }} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }} +{%- endif %} diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.2_pythonic.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.2_pythonic.jinja new file mode 100644 index 0000000000000000000000000000000000000000..1abdd0898c0f278aa640a219a41c9eb3732991e4 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.2_pythonic.jinja @@ -0,0 +1,98 @@ +{{- bos_token }} +{%- if custom_tools is defined %} + {%- set tools = custom_tools %} +{%- endif %} +{%- if not tools_in_user_message is defined %} + {%- set tools_in_user_message = false %} +{%- endif %} +{%- if not date_string is defined %} + {%- if strftime_now is defined %} + {%- set date_string = strftime_now("%d %b %Y") %} + {%- else %} + {%- set date_string = "26 Jul 2024" %} + {%- endif %} +{%- endif %} +{%- if not tools is defined %} + {%- set tools = none %} +{%- endif %} + +{#- This block extracts the system message, so we can slot it into the right place. #} +{%- if messages[0]['role'] == 'system' %} + {%- set system_message = messages[0]['content']|trim %} + {%- set messages = messages[1:] %} +{%- else %} + {%- set system_message = "You are a helpful assistant with tool calling capabilities. Only reply with a tool call if the function exists in the library provided by the user. If it doesn't exist, just reply directly in natural language. When you receive a tool call response, use the output to format an answer to the original user question." %} +{%- endif %} + +{#- System message #} +{{- "<|start_header_id|>system<|end_header_id|>\n\n" }} +{%- if tools is not none %} + {{- "Environment: ipython\n" }} +{%- endif %} +{{- "Cutting Knowledge Date: December 2023\n" }} +{{- "Today Date: " + date_string + "\n\n" }} +{%- if tools is not none and not tools_in_user_message %} + {{- "You have access to the following functions. To call functions, please respond with a python list of the calls. " }} + {{- 'Respond in the format [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)] ' }} + {{- "Do not use variables.\n\n" }} + {%- for t in tools %} + {{- t | tojson(indent=4) }} + {{- "\n\n" }} + {%- endfor %} +{%- endif %} +{{- system_message }} +{{- "<|eot_id|>" }} + +{#- Custom tools are passed in a user message with some extra guidance #} +{%- if tools_in_user_message and not tools is none %} + {#- Extract the first user message so we can plug it in here #} + {%- if messages | length != 0 %} + {%- set first_user_message = messages[0]['content']|trim %} + {%- set messages = messages[1:] %} + {%- else %} + {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }} + {%- endif %} + {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}} + {{- "Given the following functions, please respond with a python list for function calls " }} + {{- "with their proper arguments to best answer the given prompt.\n\n" }} + {{- 'Respond in the format [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)] ' }} + {{- "Do not use variables.\n\n" }} + {%- for t in tools %} + {{- t | tojson(indent=4) }} + {{- "\n\n" }} + {%- endfor %} + {{- first_user_message + "<|eot_id|>"}} +{%- endif %} + +{%- for message in messages %} + {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %} + {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }} + {%- elif 'tool_calls' in message %} + {{- '<|start_header_id|>assistant<|end_header_id|>\n\n[' -}} + {%- for tool_call in message.tool_calls %} + {%- if tool_call.function is defined %} + {%- set tool_call = tool_call.function %} + {%- endif %} + {{- tool_call.name + '(' -}} + {%- for param in tool_call.arguments %} + {{- param + '=' -}} + {{- "%s" | format(tool_call.arguments[param]) -}} + {% if not loop.last %}, {% endif %} + {%- endfor %} + {{- ')' -}} + {% if not loop.last %}, {% endif %} + {%- endfor %} + {{- ']<|eot_id|>' -}} + {%- elif message.role == "tool" or message.role == "ipython" %} + {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }} + {%- if message.content is mapping %} + {{- message.content | tojson }} + {%- else %} + {{- { "output": message.content } | tojson }} + {%- endif %} + {{- "<|eot_id|>" }} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }} +{%- endif %} diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama4_json.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama4_json.jinja new file mode 100644 index 0000000000000000000000000000000000000000..d3202b8977969f5e66c6b5d63883b330edba8613 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama4_json.jinja @@ -0,0 +1,116 @@ +{%- macro is_array_of_type_objects(var) -%} + {%- if var is iterable and var is not string -%} + {%- set valid = true -%} + {%- for item in var -%} + {%- if 'type' not in item -%} + {%- set valid = false -%} + {%- break -%} + {%- endif -%} + {%- endfor -%} + {{ valid }} + {%- else -%} + {{ false }} + {%- endif -%} +{%- endmacro %} + +{%- macro render_message(message) %} + {%- if message['content'] is string %} + {{- message['content']|trim }} + {%- elif is_array_of_type_objects(data) == 'True' %} + {%- for content in message['content'] %} + {%- if content['type'] == 'image' %} + {{- '<|image|>' }} + {%- elif content['type'] == 'text' %} + {{- content['text']|trim }} + {%- endif %} + {%- endfor %} + {%- else %} + {{- message['content']|tojson }} + {%- endif %} +{%- endmacro %} + +{{- bos_token }} +{%- if custom_tools is defined %} + {%- set tools = custom_tools %} +{%- endif %} +{%- if not tools_in_user_message is defined %} + {%- set tools_in_user_message = true %} +{%- endif %} +{%- if not tools is defined %} + {%- set tools = none %} +{%- endif %} + +{#- This block extracts the system message, so we can slot it into the right place. #} +{%- if messages[0]['role'] == 'system' %} + {%- set system_message = messages[0] %} + {%- set messages = messages[1:] %} +{%- else %} + {%- set system_message = ({ "content": "You are a helpful assistant with tool calling " + "capabilities. Only reply with a tool call if the function exists in the " + "library provided by the user. If it doesn't exist, just reply directly in " + "natural language. When you receive a tool call response, use the output to " + "format an answer to the original user question."}) %} +{%- endif %} + +{%- set tool_lib_preamble = 'Tools: You have access to the following tools. You might need to use one ' + 'or more function/tool calls to fulfill the task. \n' + 'If none are needed, then proceed to the response.\n\n' + 'Tool Call Syntax: You can call tools using the following syntax:\n' + '{"name": function name, "parameters": dictionary of argument name and its value}.\n' + 'Separate multiple function calls by "; ". Do not use variables.\n' + 'Do not include anything else when calling the tools with the syntax above.\n\n' + 'Here is a list of functions in JSON format that you can invoke.\n' %} + +{{- "<|header_start|>system<|header_end|>\n\n" }} +{%- if tools is not none and not tools_in_user_message %} + {{- tool_lib_preamble }} + {%- for t in tools %} + {{- t | tojson(indent=4) }} + {{- "\n\n" }} + {%- endfor %} +{%- endif %} +{{- render_message(system_message) }} +{{ "<|eot|>\n" }} + +{#- Custom tools are passed in a user message with some extra guidance #} +{%- if tools_in_user_message and not tools is none %} + {#- Extract the first user message so we can plug it in here #} + {%- if messages | length != 0 %} + {%- set first_user_message = messages[0] %} + {%- set messages = messages[1:] %} + {%- else %} + {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }} + {%- endif %} + {{- '<|header_start|>user<|header_end|>\n\n' }} + {{- tool_lib_preamble }} + {%- for t in tools %} + {{- t | tojson(indent=4) }} + {{- "\n\n" }} + {%- endfor %} + {{- render_message(first_user_message) + "\n<|eot|>"}} +{%- endif %} + +{%- for message in messages %} + {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %} + {{- '<|header_start|>' + message['role'] + '<|header_end|>\n\n' }} + {{- render_message(message) }} + {{- "\n<|eot|>" }} + {%- elif 'tool_calls' in message and message.tool_calls|length > 0 %} + {{- '\n<|header_start|>assistant<|header_end|>\n\n' -}} + {{- render_message(message) }} + {%- for tool_call in message.tool_calls %} + {{- '{"name": "' + tool_call.function.name + '", ' }} + {{- '"parameters": ' }} + {{- tool_call.function.arguments | tojson }} + {{- "}" }} + {%- endfor %} + {{- "\n<|eot|>" }} + {%- elif message.role == "tool" or message.role == "ipython" %} + {{- "\n<|header_start|>ipython<|header_end|>\n\n" }} + {{- render_message(message) }} + {{- "\n<|eom|>" }} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '\n<|header_start|>assistant<|header_end|>\n\n' }} +{%- endif %} diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama4_pythonic.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama4_pythonic.jinja new file mode 100644 index 0000000000000000000000000000000000000000..55c1472aecb201dc80e8643ab07006d116a76583 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama4_pythonic.jinja @@ -0,0 +1,111 @@ +{{- bos_token }} +{%- if custom_tools is defined and custom_tools%} + {%- set tools = custom_tools %} +{%- endif %} +{%- if tools is defined and tools %} + {%- set tool_definition = tool_definition ~ (tools | tojson(indent=4)) %} +{%- else %} + {%- set tools = none %} +{%- endif %} + + +{#- This block extracts the system message, so we can slot it into the right place. #} +{%- if messages[0]['role'] == 'system' %} + {%- set user_provided_system_message = true %} + {%- if messages[0]['content'] is string %} + {%- set system_message = messages[0]['content']|trim %} + {%- else %} + {%- set system_message = messages[0]['content'][0]['text']|trim %} + {%- endif %} + {%- set messages = messages[1:] %} +{%- else %} + {%- if tools is not none %} + {#- Since not system_message was provided by user, if tool is provided, system_message is now default tool system message #} + {#- This system message is from llama website:https://www.llama.com/docs/model-cards-and-prompt-formats/llama4/ #} + {%- set system_message = "You are a helpful assistant and an expert in function composition. You can answer general questions using your internal knowledge OR invoke functions when necessary. Follow these strict guidelines:\n\n1. FUNCTION CALLS:\n- ONLY use functions that are EXPLICITLY listed in the function list below\n- If NO functions are listed (empty function list []), respond ONLY with internal knowledge or \"I don't have access to [Unavailable service] information\"\n- If a function is not in the list, respond ONLY with internal knowledge or \"I don't have access to [Unavailable service] information\"\n- If ALL required parameters are present AND the query EXACTLY matches a listed function's purpose: output ONLY the function call(s)\n- Use exact format: [func_name1(param1=value1, param2=value2), func_name2(...)]\nExamples:\nCORRECT: [get_weather(location=\"Vancouver\"), calculate_route(start=\"Boston\", end=\"New York\")] <- Only if get_weather and calculate_route are in function list\nINCORRECT: get_weather(location=\"New York\")\nINCORRECT: Let me check the weather: [get_weather(location=\"New York\")]\nINCORRECT: [get_events(location=\"Singapore\")] <- If function not in list\n\n2. RESPONSE RULES:\n- For pure function requests matching a listed function: ONLY output the function call(s)\n- For knowledge questions: ONLY output text\n- For missing parameters: ONLY request the specific missing parameters\n- For unavailable services (not in function list): output ONLY with internal knowledge or \"I don't have access to [Unavailable service] information\". Do NOT execute a function call.\n- If the query asks for information beyond what a listed function provides: output ONLY with internal knowledge about your limitations\n- NEVER combine text and function calls in the same response\n- NEVER suggest alternative functions when the requested service is unavailable\n- NEVER create or invent new functions not listed below\n\n3. STRICT BOUNDARIES:\n- ONLY use functions from the list below - no exceptions\n- NEVER use a function as an alternative to unavailable information\n- NEVER call functions not present in the function list\n- NEVER add explanatory text to function calls\n- NEVER respond with empty brackets\n- Use proper Python/JSON syntax for function calls\n- Check the function list carefully before responding\n\n4. TOOL RESPONSE HANDLING:\n- When receiving tool responses: provide concise, natural language responses\n- Don't repeat tool response verbatim\n- Don't add supplementary information\n\nHere is a list of functions in JSON format that you can invoke:\n" %} + {%- else %} + {%- set system_message = "" %} + {%- endif %} +{%- endif %} +{#- Now writing the system message: use the user provided system message if user_provided_system_message, else default tool system message if tools presented #} +{%- if system_message %} + {#- always use user provided system message to override default tool system message #} + {{- "<|header_start|>system<|header_end|>\n\n" }} + {{- system_message }} + {%- if user_provided_system_message and tools %} + {{- "\nHere is a list of functions in JSON format that you can invoke. Use exact format: [func_name1(param1=value1, param2=value2), func_name2(...)]\n" }} + {{- tool_definition -}} + {%- elif tool_definition %} + {{- tool_definition -}} + {%- endif %} + {{- "<|eot|>" }} +{%- endif %} + +{#- Now deal with all other messages #} +{%- for message in messages %} + {#- Base case: messages that are not from tool role and has empty tool_call list #} + {%- if not (message.role == 'ipython' or message.role == 'tool' or ('tool_calls' in message and message.tool_calls|length != 0 )) %} + {{- '<|header_start|>' + message['role'] + '<|header_end|>\n\n' }} + {%- if message['content'] is string %} + {{- message['content'] }} + {%- else %} + {%- for content in message['content'] %} + {%- if content['type'] == 'image' %} + {{- '<|image|>' }} + {%- elif content['type'] == 'text' %} + {{- content['text'] | trim }} + {%- endif %} + {%- endfor %} + {%- endif %} + {{- "<|eot|>" }} + {#- Tool case: messages has non-empty tool_call list, must from assistant #} + {%- elif 'tool_calls' in message %} + {#- assume tool_calls are always coming from assistant #} + {%- if message.role == 'assistant' %} + {{- '<|header_start|>assistant<|header_end|>\n\n' -}} + {%- if message['content'] is string %} + {{- message['content'] }} + {%- else %} + {%- for content in message['content'] %} + {%- if content['type'] == 'image' %} + {{- '<|image|>' }} + {%- elif content['type'] == 'text' %} + {{- content['text'] }} + {%- endif %} + {%- endfor %} + {%- endif %} + {{- "[" }} + {%- for tool_call in message.tool_calls %} + {%- if tool_call.function is defined %} + {%- set tool_call = tool_call.function %} + {%- endif %} + {{- tool_call.name + '(' -}} + {%- for param in tool_call.arguments %} + {{- param + '="' -}} + {{- "%s" | format(tool_call.arguments[param]) -}} + {{- '"' -}} + {% if not loop.last %}, {% endif %} + {%- endfor %} + {{- ')' -}} + {% if not loop.last %}, {% endif %} + {%- endfor %} + {{- "]<|eot|>" }} +{%- endif %} +{#- Tool_response case: messages are from tool_response #} + {%- elif message.role == "tool" or message.role == "ipython" %} + {{- "<|header_start|>ipython<|header_end|>\n\n" }} + {%- if message.content is string %} + {{- message.content | tojson }} + {%- else %} + {%- for content in message['content'] %} + {%- if content['type'] == 'text' %} + {{- content['text'] | tojson }} + {%- endif %} + {%- endfor %} + {%- endif %} + {{- "<|eot|>" }} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|header_start|>assistant<|header_end|>\n\n' }} +{%- endif %} diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral.jinja new file mode 100644 index 0000000000000000000000000000000000000000..013ff60c1255d1a85efbe6ae66245a9c8621557f --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral.jinja @@ -0,0 +1,86 @@ +{%- if messages[0]["role"] == "system" %} + {%- set system_message = messages[0]["content"] %} + {%- set loop_messages = messages[1:] %} +{%- else %} + {%- set loop_messages = messages %} +{%- endif %} +{%- if not tools is defined %} + {%- set tools = none %} +{%- endif %} +{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %} + +{%- for message in loop_messages | rejectattr("role", "equalto", "tool") | rejectattr("role", "equalto", "tool_results") | selectattr("tool_calls", "undefined") %} + {%- if (message["role"] == "user") != (loop.index0 % 2 == 0) %} + {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }} + {%- endif %} +{%- endfor %} + +{{- bos_token }} +{%- for message in loop_messages %} + {%- if message["role"] == "user" %} + {%- if tools is not none and (message == user_messages[-1]) %} + {{- "[AVAILABLE_TOOLS] [" }} + {%- for tool in tools %} + {%- set tool = tool.function %} + {{- '{"type": "function", "function": {' }} + {%- for key, val in tool.items() if key != "return" %} + {%- if val is string %} + {{- '"' + key + '": "' + val + '"' }} + {%- else %} + {{- '"' + key + '": ' + val|tojson }} + {%- endif %} + {%- if not loop.last %} + {{- ", " }} + {%- endif %} + {%- endfor %} + {{- "}}" }} + {%- if not loop.last %} + {{- ", " }} + {%- else %} + {{- "]" }} + {%- endif %} + {%- endfor %} + {{- "[/AVAILABLE_TOOLS]" }} + {%- endif %} + {%- if loop.last and system_message is defined %} + {{- "[INST] " + system_message + "\n\n" + message["content"] + "[/INST]" }} + {%- else %} + {{- "[INST] " + message["content"] + "[/INST]" }} + {%- endif %} + {%- elif message["role"] == "tool_calls" or message.tool_calls is defined %} + {%- if message.tool_calls is defined %} + {%- set tool_calls = message.tool_calls %} + {%- else %} + {%- set tool_calls = message.content %} + {%- endif %} + {{- "[TOOL_CALLS] [" }} + {%- for tool_call in tool_calls %} + {%- set out = tool_call.function|tojson %} + {{- out[:-1] }} + {%- if not tool_call.id is defined or tool_call.id|length < 9 %} + {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (1)" + tool_call.id) }} + {%- endif %} + {{- ', "id": "' + tool_call.id[-9:] + '"}' }} + {%- if not loop.last %} + {{- ", " }} + {%- else %} + {{- "]" + eos_token }} + {%- endif %} + {%- endfor %} + {%- elif message["role"] == "assistant" %} + {{- " " + message["content"] + eos_token }} + {%- elif message["role"] == "tool_results" or message["role"] == "tool" %} + {%- if message.content is defined and message.content.content is defined %} + {%- set content = message.content.content %} + {%- else %} + {%- set content = message.content %} + {%- endif %} + {{- '[TOOL_RESULTS] {"content": ' + content|string + ", " }} + {%- if not message.tool_call_id is defined or message.tool_call_id|length < 9 %} + {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (2)" + message.tool_call_id) }} + {%- endif %} + {{- '"call_id": "' + message.tool_call_id[-9:] + '"}[/TOOL_RESULTS]' }} + {%- else %} + {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }} + {%- endif %} +{%- endfor %} diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral3.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral3.jinja new file mode 100644 index 0000000000000000000000000000000000000000..e9325f8b533dd6a6c7436d7da6f0ca73c1d8a2e1 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral3.jinja @@ -0,0 +1,126 @@ +{%- set today = strftime_now("%Y-%m-%d") %} +{%- set default_system_message = "You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\nYour knowledge base was last updated on 2023-10-01. The current date is " + today + ".\n\nWhen you're not sure about some information, you say that you don't have the information and don't make up anything.\nIf the user's question is not clear, ambiguous, or does not provide enough context for you to accurately answer the question, you do not try to answer it right away and you rather ask the user to clarify their request (e.g. \"What are some good restaurants around me?\" => \"Where are you?\" or \"When is the next flight to Tokyo\" => \"Where do you travel from?\")" %} + +{{- bos_token }} + +{%- if messages[0]['role'] == 'system' %} + {%- if messages[0]['content'] is string %} + {%- set system_message = messages[0]['content'] %} + {%- set loop_messages = messages[1:] %} + {%- else %} + {%- set system_message = messages[0]['content'][0]['text'] %} + {%- set loop_messages = messages[1:] %} + {%- endif %} +{%- else %} + {%- set system_message = default_system_message %} + {%- set loop_messages = messages %} +{%- endif %} +{%- if not tools is defined %} + {%- set tools = none %} +{%- elif tools is not none %} + {%- set parallel_tool_prompt = "You are a helpful assistant that can call tools. If you call one or more tools, format them in a single JSON array or objects, where each object is a tool call, not as separate objects outside of an array or multiple arrays. Use the format [{\"name\": tool call name, \"arguments\": tool call arguments}, additional tool calls] if you call more than one tool. If you call tools, do not attempt to interpret them or otherwise provide a response until you receive a tool call result that you can interpret for the user." %} + {%- if system_message is defined %} + {%- set system_message = parallel_tool_prompt + "\n\n" + system_message %} + {%- else %} + {%- set system_message = parallel_tool_prompt %} + {%- endif %} +{%- endif %} +{{- '[SYSTEM_PROMPT]' + system_message + '[/SYSTEM_PROMPT]' }} + +{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %} + +{%- set filtered_messages = [] %} +{%- for message in loop_messages %} + {%- if message["role"] not in ["tool", "tool_results"] and not message.get("tool_calls") %} + {%- set filtered_messages = filtered_messages + [message] %} + {%- endif %} +{%- endfor %} + +{%- for message in filtered_messages %} + {%- if (message["role"] == "user") != (loop.index0 % 2 == 0) %} + {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }} + {%- endif %} +{%- endfor %} + +{%- for message in loop_messages %} + {%- if message["role"] == "user" %} + {%- if tools is not none and (message == user_messages[-1]) %} + {{- "[AVAILABLE_TOOLS] [" }} + {%- for tool in tools %} + {%- set tool = tool.function %} + {{- '{"type": "function", "function": {' }} + {%- for key, val in tool.items() if key != "return" %} + {%- if val is string %} + {{- '"' + key + '": "' + val + '"' }} + {%- else %} + {{- '"' + key + '": ' + val|tojson }} + {%- endif %} + {%- if not loop.last %} + {{- ", " }} + {%- endif %} + {%- endfor %} + {{- "}}" }} + {%- if not loop.last %} + {{- ", " }} + {%- else %} + {{- "]" }} + {%- endif %} + {%- endfor %} + {{- "[/AVAILABLE_TOOLS]" }} + {%- endif %} + {%- if message['content'] is string %} + {{- '[INST]' + message['content'] + '[/INST]' }} + {%- else %} + {{- '[INST]' }} + {%- for block in message['content'] %} + {%- if block['type'] == 'text' %} + {{- block['text'] }} + {%- elif block['type'] == 'image' or block['type'] == 'image_url' %} + {{- '[IMG]' }} + {%- else %} + {{- raise_exception('Only text and image blocks are supported in message content!') }} + {%- endif %} + {%- endfor %} + {{- '[/INST]' }} + {%- endif %} + {%- elif message["role"] == "tool_calls" or message.tool_calls is defined %} + {%- if message.tool_calls is defined %} + {%- set tool_calls = message.tool_calls %} + {%- else %} + {%- set tool_calls = message.content %} + {%- endif %} + {{- "[TOOL_CALLS] [" }} + {%- for tool_call in tool_calls %} + {%- set out = tool_call.function|tojson %} + {{- out[:-1] }} + {%- if not tool_call.id is defined or tool_call.id|length < 9 %} + {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (1)" + tool_call.id) }} + {%- endif %} + {{- ', "id": "' + tool_call.id[-9:] + '"}' }} + {%- if not loop.last %} + {{- ", " }} + {%- else %} + {{- "]" + eos_token }} + {%- endif %} + {%- endfor %} + {%- elif message['role'] == 'assistant' %} + {%- if message['content'] is string %} + {{- message['content'] + eos_token }} + {%- else %} + {{- message['content'][0]['text'] + eos_token }} + {%- endif %} + {%- elif message["role"] == "tool_results" or message["role"] == "tool" %} + {%- if message.content is defined and message.content.content is defined %} + {%- set content = message.content.content %} + {%- else %} + {%- set content = message.content %} + {%- endif %} + {{- '[TOOL_RESULTS] {"content": ' + content|string + ", " }} + {%- if not message.tool_call_id is defined or message.tool_call_id|length < 9 %} + {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (2)" + message.tool_call_id) }} + {%- endif %} + {{- '"call_id": "' + message.tool_call_id[-9:] + '"}[/TOOL_RESULTS]' }} + {%- else %} + {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }} + {%- endif %} +{%- endfor %} diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral_parallel.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral_parallel.jinja new file mode 100644 index 0000000000000000000000000000000000000000..97d1ad55e3e0e418b4b415a3fedebfdef983ab1a --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral_parallel.jinja @@ -0,0 +1,93 @@ +{%- if messages[0]["role"] == "system" %} + {%- set system_message = messages[0]["content"] %} + {%- set loop_messages = messages[1:] %} +{%- else %} + {%- set loop_messages = messages %} +{%- endif %} +{%- if not tools is defined %} + {%- set tools = none %} +{%- elif tools is not none %} + {%- set parallel_tool_prompt = "You are a helpful assistant that can call tools. If you call one or more tools, format them in a single JSON array or objects, where each object is a tool call, not as separate objects outside of an array or multiple arrays. Use the format [{\"name\": tool call name, \"arguments\": tool call arguments}, additional tool calls] if you call more than one tool. If you call tools, do not attempt to interpret them or otherwise provide a response until you receive a tool call result that you can interpret for the user." %} + {%- if system_message is defined %} + {%- set system_message = parallel_tool_prompt + "\n\n" + system_message %} + {%- else %} + {%- set system_message = parallel_tool_prompt %} + {%- endif %} +{%- endif %} +{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %} + +{%- for message in loop_messages | rejectattr("role", "equalto", "tool") | rejectattr("role", "equalto", "tool_results") | selectattr("tool_calls", "undefined") %} + {%- if (message["role"] == "user") != (loop.index0 % 2 == 0) %} + {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }} + {%- endif %} +{%- endfor %} + +{{- bos_token }} +{%- for message in loop_messages %} + {%- if message["role"] == "user" %} + {%- if tools is not none and (message == user_messages[-1]) %} + {{- "[AVAILABLE_TOOLS] [" }} + {%- for tool in tools %} + {%- set tool = tool.function %} + {{- '{"type": "function", "function": {' }} + {%- for key, val in tool.items() if key != "return" %} + {%- if val is string %} + {{- '"' + key + '": "' + val + '"' }} + {%- else %} + {{- '"' + key + '": ' + val|tojson }} + {%- endif %} + {%- if not loop.last %} + {{- ", " }} + {%- endif %} + {%- endfor %} + {{- "}}" }} + {%- if not loop.last %} + {{- ", " }} + {%- else %} + {{- "]" }} + {%- endif %} + {%- endfor %} + {{- "[/AVAILABLE_TOOLS]" }} + {%- endif %} + {%- if loop.last and system_message is defined %} + {{- "[INST] " + system_message + "\n\n" + message["content"] + "[/INST]" }} + {%- else %} + {{- "[INST] " + message["content"] + "[/INST]" }} + {%- endif %} + {%- elif message["role"] == "tool_calls" or message.tool_calls is defined %} + {%- if message.tool_calls is defined %} + {%- set tool_calls = message.tool_calls %} + {%- else %} + {%- set tool_calls = message.content %} + {%- endif %} + {{- "[TOOL_CALLS] [" }} + {%- for tool_call in tool_calls %} + {%- set out = tool_call.function|tojson %} + {{- out[:-1] }} + {%- if not tool_call.id is defined or tool_call.id|length < 9 %} + {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (1)" + tool_call.id) }} + {%- endif %} + {{- ', "id": "' + tool_call.id[-9:] + '"}' }} + {%- if not loop.last %} + {{- ", " }} + {%- else %} + {{- "]" + eos_token }} + {%- endif %} + {%- endfor %} + {%- elif message["role"] == "assistant" %} + {{- " " + message["content"] + eos_token }} + {%- elif message["role"] == "tool_results" or message["role"] == "tool" %} + {%- if message.content is defined and message.content.content is defined %} + {%- set content = message.content.content %} + {%- else %} + {%- set content = message.content %} + {%- endif %} + {{- '[TOOL_RESULTS] {"content": ' + content|string + ", " }} + {%- if not message.tool_call_id is defined or message.tool_call_id|length < 9 %} + {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (2)" + message.tool_call_id) }} + {%- endif %} + {{- '"call_id": "' + message.tool_call_id[-9:] + '"}[/TOOL_RESULTS]' }} + {%- else %} + {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }} + {%- endif %} +{%- endfor %} diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_phi4_mini.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_phi4_mini.jinja new file mode 100644 index 0000000000000000000000000000000000000000..1c351476affdfe8ddf8c7a49b3318d922b8efa29 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_phi4_mini.jinja @@ -0,0 +1,62 @@ +{%- if messages and messages[0]['role'] == 'system' %} + {%- set system_message = messages[0]['content']|trim %} + {%- set messages = messages[1:] %} +{%- else %} + {%- set system_message = "You are a helpful assistant." %} +{%- endif %} + +{%- if messages %} +<|system|> +{{ system_message }} +{%- if tools %} +In addition to plain text responses, you can choose to call one or more of the provided functions. + +Use the following rule to decide when to call a function: + * if the response can be generated from your internal knowledge (e.g., as in the case of queries like "What is the capital of Poland?"), do so + * if you need external information that can be obtained by calling one or more of the provided functions, generate a function calls + +If you decide to call functions: + * prefix function calls with functools marker (no closing marker required) + * all function calls should be generated in a single JSON list formatted as functools[{"name": [function name], "arguments": [function arguments as JSON]}, ...] + * follow the provided JSON schema. Do not hallucinate arguments or values. Do to blindly copy values from the provided samples + * respect the argument type formatting. E.g., if the type is number and format is float, write value 7 as 7.0 + * make sure you pick the right functions that match the user intent + + + {%- for t in tools %} + {{- t | tojson(indent=4) }} + {{- "\n\n" }} + {%- endfor %} +{%- endif %}<|end|> + + {%- for message in messages %} + {%- if message.role != "system" %} +<|{{ message.role }}|> + {%- if message.content and message.role == "tools" %} +{"result": {{ message.content }}} + {%- elif message.content %} +{{ message.content }} + {%- elif message.tool_calls %} + {%- for call in message.tool_calls %} +{"name": "{{ call.function.name }}", "arguments": {{ call.function.arguments }}} + {%- if not loop.last %},{% endif %} + {%- endfor %} + {%- endif %}<|end|> + {%- endif %} + {%- endfor %}<|assistant|> + +{%- else %} + {%- if system_message %} +<|system|> + +{{ system_message }}<|end|> + {%- endif %} + {%- if prompt %} +<|user|> + +{{ prompt }}<|end|> + {%- endif %}<|assistant|> + +{%- endif %} +{{ response }} +{%- if response %}<|user|>{% endif %} \ No newline at end of file diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_qwen3coder.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_qwen3coder.jinja new file mode 100644 index 0000000000000000000000000000000000000000..389b27415b7baffd6eecbadd6b15d03b74be98c2 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_qwen3coder.jinja @@ -0,0 +1,117 @@ +{% macro render_extra_keys(json_dict, handled_keys) %} + {%- if json_dict is mapping %} + {%- for json_key in json_dict if json_key not in handled_keys %} + {%- if json_dict[json_key] is mapping or (json_dict[json_key] is sequence and json_dict[json_key] is not string) %} + {{- '\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | tojson | safe) ~ '' }} + {%- else %} + {{-'\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | string) ~ '' }} + {%- endif %} + {%- endfor %} + {%- endif %} +{% endmacro %} + +{%- if messages[0]["role"] == "system" %} + {%- set system_message = messages[0]["content"] %} + {%- set loop_messages = messages[1:] %} +{%- else %} + {%- set loop_messages = messages %} +{%- endif %} + +{%- if not tools is defined %} + {%- set tools = [] %} +{%- endif %} + +{%- if system_message is defined %} + {{- "<|im_start|>system\n" + system_message }} +{%- else %} + {%- if tools is iterable and tools | length > 0 %} + {{- "<|im_start|>system\nYou are Qwen, a helpful AI assistant that can interact with a computer to solve tasks." }} + {%- endif %} +{%- endif %} +{%- if tools is iterable and tools | length > 0 %} + {{- "\n\n# Tools\n\nYou have access to the following functions:\n\n" }} + {{- "" }} + {%- for tool in tools %} + {%- if tool.function is defined %} + {%- set tool = tool.function %} + {%- endif %} + {{- "\n\n" ~ tool.name ~ "" }} + {%- if tool.description is defined %} + {{- '\n' ~ (tool.description | trim) ~ '' }} + {%- endif %} + {{- '\n' }} + {%- if tool.parameters is defined and tool.parameters is mapping and tool.parameters.properties is defined and tool.parameters.properties is mapping %} + {%- for param_name, param_fields in tool.parameters.properties|items %} + {{- '\n' }} + {{- '\n' ~ param_name ~ '' }} + {%- if param_fields.type is defined %} + {{- '\n' ~ (param_fields.type | string) ~ '' }} + {%- endif %} + {%- if param_fields.description is defined %} + {{- '\n' ~ (param_fields.description | trim) ~ '' }} + {%- endif %} + {%- set handled_keys = ['name', 'type', 'description'] %} + {{- render_extra_keys(param_fields, handled_keys) }} + {{- '\n' }} + {%- endfor %} + {%- endif %} + {% set handled_keys = ['type', 'properties'] %} + {{- render_extra_keys(tool.parameters, handled_keys) }} + {{- '\n' }} + {%- set handled_keys = ['type', 'name', 'description', 'parameters'] %} + {{- render_extra_keys(tool, handled_keys) }} + {{- '\n' }} + {%- endfor %} + {{- "\n" }} + {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n\n\n\nvalue_1\n\n\nThis is the value for the second parameter\nthat can span\nmultiple lines\n\n\n\n\n\nReminder:\n- Function calls MUST follow the specified format: an inner block must be nested within XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n' }} +{%- endif %} +{%- if system_message is defined %} + {{- '<|im_end|>\n' }} +{%- else %} + {%- if tools is iterable and tools | length > 0 %} + {{- '<|im_end|>\n' }} + {%- endif %} +{%- endif %} +{%- for message in loop_messages %} + {%- if message.role == "assistant" and message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %} + {{- '<|im_start|>' + message.role }} + {%- if message.content is defined and message.content is string and message.content | trim | length > 0 %} + {{- '\n' + message.content | trim + '\n' }} + {%- endif %} + {%- for tool_call in message.tool_calls %} + {%- if tool_call.function is defined %} + {%- set tool_call = tool_call.function %} + {%- endif %} + {{- '\n\n\n' }} + {%- if tool_call.arguments is defined %} + {%- for args_name, args_value in tool_call.arguments|items %} + {{- '\n' }} + {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %} + {{- args_value }} + {{- '\n\n' }} + {%- endfor %} + {%- endif %} + {{- '\n' }} + {%- endfor %} + {{- '<|im_end|>\n' }} + {%- elif message.role == "user" or message.role == "system" or message.role == "assistant" %} + {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }} + {%- elif message.role == "tool" %} + {%- if loop.previtem and loop.previtem.role != "tool" %} + {{- '<|im_start|>user\n' }} + {%- endif %} + {{- '\n' }} + {{- message.content }} + {{- '\n\n' }} + {%- if not loop.last and loop.nextitem.role != "tool" %} + {{- '<|im_end|>\n' }} + {%- elif loop.last %} + {{- '<|im_end|>\n' }} + {%- endif %} + {%- else %} + {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>\n' }} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|im_start|>assistant\n' }} +{%- endif %} diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_toolace.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_toolace.jinja new file mode 100644 index 0000000000000000000000000000000000000000..78d0426b9bfe551925b0d1da1360b702320df817 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_toolace.jinja @@ -0,0 +1,65 @@ +{{- bos_token }} + +{%- if custom_tools is defined %} + {%- set tools = custom_tools %} +{%- endif %} +{%- if not tools is defined %} + {%- set tools = none %} +{%- endif %} + +{#- This block extracts the system message, so we can slot it into the right place. #} +{%- if messages[0]['role'] == 'system' %} + {%- set system_message = messages[0]['content']|trim %} + {%- set messages = messages[1:] %} +{%- else %} + {%- set system_message = "You are a helpful assistant with tool calling capabilities. Only reply with a tool call if the function exists in the library provided by the user. If it doesn't exist, just reply directly in natural language." %} +{%- endif %} + +{{- "<|start_header_id|>system<|end_header_id|>\n\n" }} +{%- if tools is not none and not tools_in_user_message %} + {{- "You are an expert in composing functions. You are given a question and a set of possible functions. Based on the question, you will need to make one or more function/tool calls to achieve the purpose.\n" }} + {{- "If none of the function can be used, point it out. If the given question lacks the parameters required by the function, also point it out.\n" }} + {{- "You should only return the function call in tools call sections.\n\n" }} + {{- "If you decide to invoke any of the function(s), you MUST put it in the format of [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)]\n" }} + {{- "You SHOULD NOT include any other text in the response.\n" }} + {{- "Here is a list of functions in JSON format that you can invoke.\n" }} + {%- for t in tools %} + {{- t | tojson(indent=4) }} + {{- "\n\n" }} + {%- endfor %} + {{- "\n" }} +{%- endif %} +{{- system_message }} +{{- "<|eot_id|>" }} + +{%- for message in messages %} + {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %} + {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }} + {%- elif 'tool_calls' in message %} + {{- '<|start_header_id|>assistant<|end_header_id|>\n\n[' -}} + {%- for tool_call in message.tool_calls %} + {%- if tool_call.function is defined %} + {%- set tool_call = tool_call.function %} + {%- endif %} + {{- tool_call.name + '(' -}} + {%- for param in tool_call.arguments %} + {{- param + '=' -}} + {{- "%s" | format(tool_call.arguments[param]) -}} + {% if not loop.last %}, {% endif %} + {%- endfor %} + {{- ')' -}} + {% if not loop.last %}, {% endif %} + {%- endfor %} + {{- ']<|eot_id|>' -}} + {%- elif message.role == "tool" or message.role == "ipython" %} + {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }} + {%- if message.content is mapping %} + {{- message.content | tojson }} + {%- else %} + {{- { "output": message.content } | tojson }} + {%- endif %} + {{- "<|eot_id|>" }} + {%- endif %} +{%- endfor %} + +{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }} diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_xlam_llama.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_xlam_llama.jinja new file mode 100644 index 0000000000000000000000000000000000000000..596af47fdde664f300071f11a6e7c6b4c414e128 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_xlam_llama.jinja @@ -0,0 +1,77 @@ +{{- bos_token }} +{%- if custom_tools is defined %} + {%- set tools = custom_tools %} +{%- endif %} +{%- if not tools_in_user_message is defined %} + {%- set tools_in_user_message = true %} +{%- endif %} +{%- if not tools is defined %} + {%- set tools = none %} +{%- endif %} + +{#- Extract system message #} +{{- "<|start_header_id|>system<|end_header_id|>\n\n" }} +{%- if messages[0]['role'] == 'system' %} + {%- set system_message = messages[0]['content'] | trim %} + {%- set messages = messages[1:] %} + {{- system_message + "\n" }} +{%- else %} + {%- set system_message = "You are a helpful assistant. You are developed by Salesforce xLAM team." %} + {% set format_instruction %}You have access to a set of tools. When using tools, make calls in a single JSON array: + +[{"name": "tool_call_name", "arguments": {"arg1": "value1", "arg2": "value2"}}, ... (additional parallel tool calls as needed)] + +If no tool is suitable, state that explicitly. If the user's input lacks required parameters, ask for clarification. Do not interpret or respond until tool results are returned. Once they are available, process them or make additional calls if needed. For tasks that don't require tools, such as casual conversation or general advice, respond directly in plain text. The available tools are:{% endset %} + {{- system_message + "\n" }} + {%- if tools is not none %} + {{- format_instruction + "\n\n" }} + {%- endif %} +{%- endif %} + + +{%- if tools is not none %} + {%- for t in tools %} + {{- t | tojson(indent=4) }} + {{- "\n\n" }} + {%- endfor %} +{%- endif %} +{{- "<|eot_id|>" }} + +{%- for message in messages %} + {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %} + {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }} + {%- elif 'tool_calls' in message %} + {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}} + {%- if message['tool_calls'] %} + {{- "[" }} + {%- for tool_call_function in message.tool_calls %} + {%- set tool_call = tool_call_function.function %} + {{- '{"name": "' + tool_call.name + '", ' }} + {{- '"arguments": ' }} + {{- tool_call.arguments | tojson }} + {{- "}" }} + {%- if not loop.last %} + {{- ", " }} + {%- endif %} + {%- endfor %} + {{- "]" }} + {{- "<|eot_id|>" }} + {%- elif message['content'] %} + {{- message['content'] | trim + '<|eot_id|>' }} + {%- else %} + {{- "[]\n" + '<|eot_id|>' }} + {%- endif %} + {%- elif message.role == "tool" or message.role == "ipython" %} + {{- "<|start_header_id|>" + "ipython" + "<|end_header_id|>\n\n" }} + {%- set content = message["content"] %} + {%- if content is mapping or (content is iterable and content is not string) %} + {{- content | tojson }} + {%- else %} + {{- content }} + {%- endif %} + {{- "<|eot_id|>" }} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }} +{%- endif %} \ No newline at end of file diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_xlam_qwen.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_xlam_qwen.jinja new file mode 100644 index 0000000000000000000000000000000000000000..830f55c8a3017eda321d726d993ed95b55dab847 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_xlam_qwen.jinja @@ -0,0 +1,66 @@ +{# System message #} +{{- "<|im_start|>system\n" }} +{%- if messages[0]['role'] == 'system' %} + {%- set system_message = messages[0]['content'] | trim %} + {%- set messages = messages[1:] %} + {{- system_message + "\n" }} +{%- else %} + {%- set system_message = "You are a helpful assistant. You are developed by Salesforce xLAM team." %} + {% set format_instruction %}You have access to a set of tools. When using tools, make calls in a single JSON array: + +[{"name": "tool_call_name", "arguments": {"arg1": "value1", "arg2": "value2"}}, ... (additional parallel tool calls as needed)] + +If no tool is suitable, state that explicitly. If the user's input lacks required parameters, ask for clarification. Do not interpret or respond until tool results are returned. Once they are available, process them or make additional calls if needed. For tasks that don't require tools, such as casual conversation or general advice, respond directly in plain text. The available tools are:{% endset %} + {{- system_message + "\n" }} + {%- if tools is not none %} + {{- format_instruction + "\n\n" }} + {%- endif %} +{%- endif %} + +{%- if tools is not none %} + {%- for func in tools %} + {{- func | tojson(indent=4) }} + {{- "\n\n" }} + {%- endfor %} +{%- endif %} +{{- "<|im_end|>\n" }} +{%- for message in messages %} + {%- if message['role'] == 'tool' %} + {{- "<|im_start|>tool\n" }} + {%- if message.content is defined and message.content.content is defined %} + {%- set content = message.content.content %} + {%- else %} + {%- set content = message.content %} + {%- endif %} + {%- if content is mapping or content is iterable and content is not string %} + {{- content | tojson }} + {%- else %} + {{- content }} + {%- endif %} + {{- "<|im_end|>\n" }} + {%- elif 'tool_calls' in message %} + {{- "<|im_start|>assistant\n" }} + {%- if message['tool_calls'] %} + {{- "[" }} + {%- for tool_call in message.tool_calls %} + {%- set out = tool_call.function | tojson %} + {{- out }} + {%- if not loop.last %} + {{- ", " }} + {%- endif %} + {%- endfor %} + {{- "]"}} + {%- elif message['content'] %} + {{- message['content'] | trim }} + {%- else %} + {{- "[]\n" }} + {%- endif %} + {{- "<|im_end|>\n" }} + {%- else %} + {{- "<|im_start|>" + message['role'] + "\n" + message['content'] | trim + "<|im_end|>\n" }} + {%- endif %} +{%- endfor %} + +{%- if add_generation_prompt %} + {{- "<|im_start|>assistant\n" }} +{%- endif %} diff --git a/asian_ai/performance/vLLM/rust/src/cmd/Cargo.toml b/asian_ai/performance/vLLM/rust/src/cmd/Cargo.toml new file mode 100644 index 0000000000000000000000000000000000000000..4048c664e0ed6a7c849c65c77f7b2f0fe2521aee --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/cmd/Cargo.toml @@ -0,0 +1,41 @@ +[package] +name = "vllm-cmd" +version.workspace = true +edition.workspace = true +license.workspace = true + +[[bin]] +name = "vllm-rs" +path = "src/main.rs" + +[features] +default = [] +native-tls-vendored = ["dep:native-tls-vendored"] + +[dependencies] +anyhow.workspace = true +clap.workspace = true +educe.workspace = true +itertools.workspace = true +mimalloc.workspace = true +native-tls-vendored = { workspace = true, optional = true } +serde.workspace = true +serde_json.workspace = true +serde_with.workspace = true +thiserror-ext.workspace = true +time.workspace = true +tokio = { workspace = true, features = ["signal"] } +tokio-util.workspace = true +tracing.workspace = true +tracing-subscriber.workspace = true +uuid.workspace = true +vllm-chat.workspace = true +vllm-engine-core-client.workspace = true +vllm-managed-engine.workspace = true +vllm-server.workspace = true + +[dev-dependencies] +expect-test.workspace = true + +[lints] +workspace = true diff --git a/asian_ai/performance/vLLM/rust/src/cmd/examples/README.md b/asian_ai/performance/vLLM/rust/src/cmd/examples/README.md new file mode 100644 index 0000000000000000000000000000000000000000..ba8ef545a473675b448766a769a908de238e5042 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/cmd/examples/README.md @@ -0,0 +1,44 @@ +# `vllm-rs` CLI Quick Start + +Start Qwen3 with one managed `vllm-rs serve` command from the repo root: + +```bash +HF_HUB_OFFLINE=1 \ +VLLM_CPU_KVCACHE_SPACE=2 \ +VLLM_HOST_IP=127.0.0.1 \ +VLLM_LOOPBACK_IP=127.0.0.1 \ +cargo run --bin vllm-rs -- serve \ + Qwen/Qwen3-0.6B \ + --python ../vllm/.venv/bin/python \ + --max-model-len 512 \ + -- \ + --dtype float16 +``` + +This launches: + +- a managed headless Python `vllm` engine +- the Rust OpenAI-compatible frontend on `127.0.0.1:8000` + +All Python engine arguments must be placed after `--`. Arguments before `--` are parsed by the Rust +frontend itself. + +You can then send OpenAI-style requests to the Rust frontend: + +```bash +curl http://127.0.0.1:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen/Qwen3-0.6B", + "messages": [{"role": "user", "content": "What is the capital of France?"}], + "stream": true + }' +``` + +If you already started headless `vllm` yourself, use `frontend` instead: + +```bash +cargo run --bin vllm-rs -- frontend \ + --handshake-address tcp://127.0.0.1:62100 \ + Qwen/Qwen3-0.6B +``` diff --git a/asian_ai/performance/vLLM/rust/src/cmd/src/cli.rs b/asian_ai/performance/vLLM/rust/src/cmd/src/cli.rs new file mode 100644 index 0000000000000000000000000000000000000000..5b697f7e142387c7eacf99a7fb86b4ed78bf5558 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/cmd/src/cli.rs @@ -0,0 +1,694 @@ +//! CLI argument definitions for the `vllm-rs` binary. +//! +//! Python vLLM references: +//! - Engine args: +//! - Environment variables: + +mod unsupported; + +use std::collections::HashMap; +use std::ffi::{OsStr, OsString}; +use std::path::PathBuf; +use std::time::Duration; + +use clap::{Args, Parser, Subcommand}; +use educe::Educe; +use serde::de::DeserializeOwned; +use serde::{Deserialize, Serialize}; +use serde_json::Value; +use serde_with::{DefaultOnNull, OneOrMany, serde_as}; +use thiserror_ext::AsReport as _; +use uuid::Uuid; +use vllm_chat::ReasoningParserFactory; +use vllm_engine_core_client::TransportMode; +use vllm_managed_engine::ManagedEngineConfig; +use vllm_managed_engine::cli::{ManagedEngineArgs, repartition_managed_engine_args}; +use vllm_server::{ + ApiServerOptions, ChatTemplateContentFormatOption, Config, CoordinatorMode, CorsConfig, + DEFAULT_KEEP_ALIVE_TIMEOUT, HttpListenerMode, ParserSelection, RendererSelection, TlsConfig, +}; + +use crate::cli::unsupported::UnsupportedArgs; + +/// Top-level parser for the `vllm-rs` binary. +#[derive(Debug, Parser)] +#[command( + name = "vllm-rs", + about = "Rust frontend and managed-engine CLI for vLLM." +)] +pub struct Cli { + #[command(subcommand)] + pub command: Command, +} + +impl Cli { + pub fn parse() -> Self { + Self::try_parse_from(std::env::args_os()).unwrap_or_else(|error| error.exit()) + } + + pub fn try_parse_from(itr: I) -> Result + where + I: IntoIterator, + T: Into, + { + let args: Vec = itr.into_iter().map(Into::into).collect(); + let repartitioned_args = repartition_managed_engine_args::(&args, Some("serve"))?; + ::try_parse_from(&repartitioned_args).inspect(|cli| { + if let Command::Serve(serve) = &cli.command + && serve.debug_cli + { + println!( + "Original CLI args: {}\n", + args.join(OsStr::new(" ")).display() + ); + println!( + "Repartitioned CLI args: {}\n", + repartitioned_args.join(OsStr::new(" ")).display() + ); + println!( + "Passthrough Python args: {}", + serve.managed_engine.python_args.join(" ") + ); + std::process::exit(0); + } + }) + } +} + +/// Supported top-level CLI commands. +#[derive(Debug, Subcommand, PartialEq, Eq)] +pub enum Command { + /// Run the Rust OpenAI frontend as a Python-supervised worker. + Frontend(FrontendArgs), + /// Launch a managed Python headless engine, then run the Rust OpenAI + /// frontend. + Serve(ServeArgs), +} + +/// A JSON-encoded list of strings, matching Python's `json.loads` CLI type for +/// the CORS list arguments (e.g. `--allowed-origins '["*"]'`). Parsing the whole +/// value as one item keeps clap from treating the field as a repeated flag. +#[derive(Clone, Debug, PartialEq, Eq, Deserialize)] +#[serde(transparent)] +pub struct JsonStringList(pub Vec); + +/// Runtime arguments shared by both paths of the Rust frontend: +/// +/// - External-engine mode: Python-supervised bootstrap, `vllm serve` -> `vllm-rs frontend`. +/// Arguments are deserialized from a single JSON object and defaults follow `serde` attrs. +/// - Managed-engine mode: Rust-managed Python engine, `vllm-rs serve`. +/// Arguments are parsed from CLI flags and defaults follow `clap` attrs. +#[serde_as] +#[derive(Educe, Clone, Args, PartialEq, Eq, Deserialize)] +#[educe(Debug)] +pub struct SharedRuntimeArgs { + #[serde(rename = "model_tag")] + /// Model identifier or local model directory used for backend loading and + /// public model ID. + pub model: String, + + /// Maximum time to wait for the expected engines to register on the + /// frontend transport. + #[arg( + long = "engine-ready-timeout-secs", + env = "VLLM_ENGINE_READY_TIMEOUT_S", + default_value_t = default_engine_ready_timeout_secs() + )] + #[serde(default = "default_engine_ready_timeout_secs")] + pub engine_ready_timeout_secs: u64, + + /// Select the tool call parser depending on the model that you're using. + /// Use `auto` to infer from the model or `none` to disable parsing. + #[arg(long, default_value_t)] + #[serde(default = "default_py_bootstrap_parser_selection")] + pub tool_call_parser: ParserSelection, + /// Select the reasoning parser depending on the model that you're using. + /// Use `auto` to infer from the model or `none` to disable parsing. + #[arg(long, default_value_t)] + #[serde(default = "default_py_bootstrap_parser_selection")] + pub reasoning_parser: ParserSelection, + /// Select the chat renderer implementation. + #[arg(long = "tokenizer-mode", default_value_t)] + #[serde(default, rename = "tokenizer_mode")] + pub renderer: RendererSelection, + /// Disable multimodal inputs and treat the model as language-only. + #[arg(long)] + #[serde(default)] + pub language_model_only: bool, + /// Override the maximum model context length. When set, the frontend uses + /// this value instead of the model's `max_position_embeddings` from + /// `config.json`. + #[arg(long)] + pub max_model_len: Option, + /// Maximum number of log probabilities to return when `logprobs` is + /// specified in sampling parameters. `-1` means no cap. + #[arg(long, value_parser = clap::value_parser!(i32).range(-1..), allow_negative_numbers = true)] + #[serde(default)] + pub max_logprobs: Option, + /// TCP port for the gRPC Generate service. When not set, no gRPC server is + /// started. + #[arg(long)] + #[serde(default)] + pub grpc_port: Option, + /// Maximum time to wait for active requests to drain during shutdown. + #[arg(long, default_value_t = 0)] + #[serde(default)] + pub shutdown_timeout: u64, + /// Maximum idle time (seconds) on a keep-alive HTTP connection before the + /// server closes it (default 5). + #[arg(long = "http-timeout-keep-alive", env = "VLLM_HTTP_TIMEOUT_KEEP_ALIVE")] + #[serde(default)] + pub http_timeout_keep_alive: Option, + + /// The file path to the chat template, or the template in single-line form + /// for the specified model. + #[arg(long)] + #[serde(default)] + pub chat_template: Option, + + /// Default keyword arguments to pass to the chat template renderer. + /// + /// These will be merged with request-level chat_template_kwargs, with + /// request values taking precedence. Useful for setting default + /// behavior for reasoning models. + /// + /// Example: `{"enable_thinking": false}` to disable thinking mode by + /// default for Qwen3/DeepSeek models. + #[arg(long, value_parser = parse_json::>, value_name = "JSON")] + #[serde(default)] + pub default_chat_template_kwargs: Option>, + + /// The format to render message content within a chat template. + /// + /// * "auto" detects the format from the template + /// * "string" renders content as a string. Example: `"Hello World"` + /// * "openai" renders content as a list of dictionaries, similar to OpenAI schema. Example: + /// `[{"type": "text", "text": "Hello world!"}]` + #[arg(long, default_value_t)] + #[serde(default)] + pub chat_template_content_format: ChatTemplateContentFormatOption, + + /// Log a summary line for each completed request, including prompt/output + /// token counts and finish reason. + #[arg(long)] + #[serde(default)] + pub enable_log_requests: bool, + + /// Include prompt_tokens_details in usage when cached prompt tokens are + /// present. + #[arg( + long, + default_missing_value = "true", + num_args = 0..=1 + )] + #[serde(default)] + pub enable_prompt_tokens_details: bool, + + /// If specified, API server will add X-Request-Id header to responses. + #[arg( + long, + default_missing_value = "true", + num_args = 0..=1 + )] + #[serde(default)] + pub enable_request_id_headers: bool, + + /// If provided, the server will require one of these keys to be presented + /// in the Authorization header. + #[educe(Debug(ignore))] + #[arg(long, env = "VLLM_API_KEY", value_delimiter = ' ')] + #[serde_as(as = "DefaultOnNull>")] + #[serde(default)] + pub api_key: Vec, + + /// Disable periodic logging of engine statistics (throughput, queue depth, + /// cache usage). + #[arg(long)] + #[serde(default)] + pub disable_log_stats: bool, + + /// The model name(s) used in the API. If multiple names are provided, the + /// server will respond to any of the provided names. The model name in the + /// model field of a response will be the first name in this list. If not + /// specified, the model name will be the same as the `--model` argument. + /// Noted that this name(s) will also be used in `model_name` tag + /// content of prometheus metrics, if multiple names provided, metrics + /// tag will take the first one. + #[arg(long, num_args = 0..)] + #[serde(default)] + pub served_model_name: Vec, + + /// CORS allowed origins as a JSON list. `["*"]` allows any origin. + #[arg(long, value_parser = parse_json::, value_name = "JSON", default_value = r#"["*"]"#)] + #[serde(default = "default_cors_wildcard")] + pub allowed_origins: JsonStringList, + + /// CORS allowed methods as a JSON list. `["*"]` allows the standard set. + #[arg(long, value_parser = parse_json::, value_name = "JSON", default_value = r#"["*"]"#)] + #[serde(default = "default_cors_wildcard")] + pub allowed_methods: JsonStringList, + + /// CORS allowed request headers as a JSON list. `["*"]` mirrors the request. + #[arg(long, value_parser = parse_json::, value_name = "JSON", default_value = r#"["*"]"#)] + #[serde(default = "default_cors_wildcard")] + pub allowed_headers: JsonStringList, + + /// Allow CORS credentials (cookies, authorization headers). + #[arg( + long, + default_missing_value = "true", + num_args = 0..=1 + )] + #[serde(default)] + pub allow_credentials: bool, + + /// The file path to the SSL key file. When omitted, the key is read from + /// `--ssl-certfile` (combined PEM). + #[arg(long)] + #[serde(default)] + pub ssl_keyfile: Option, + + /// The file path to the SSL cert file. Enables TLS when set. + #[arg(long)] + #[serde(default)] + pub ssl_certfile: Option, + + /// The CA certificates file used to verify client certificates (mTLS). + #[arg(long)] + #[serde(default)] + pub ssl_ca_certs: Option, + + /// Whether a client certificate is required: 0 = none, 1 = optional, + /// 2 = required (mirrors Python's `ssl.CERT_*`). + #[arg(long, default_value_t = 0, value_parser = clap::value_parser!(i32).range(0..=2))] + #[serde(default)] + pub ssl_cert_reqs: i32, + + /// OpenSSL cipher string for HTTPS (TLS 1.2 and below). + /// When unset, the linked OpenSSL's default suites are used. + #[arg(long)] + #[serde(default)] + pub ssl_ciphers: Option, + + /// Profiler configuration forwarded by the Python supervisor. + /// + /// When set with a non-null `profiler` type, the Rust frontend registers + /// the `/start_profile` and `/stop_profile` routes and forwards calls to + /// the engine via the `"profile"` utility RPC. + #[arg(long, value_parser = parse_json::, value_name = "JSON")] + #[serde(default)] + pub profiler_config: Option, + + /// Unsupported Python vLLM frontend arguments recognized but not yet + /// implemented in Rust. + #[educe(Debug(ignore))] + #[command(flatten)] + #[serde(default, flatten)] + pub unsupported: UnsupportedArgs, +} + +impl SharedRuntimeArgs { + /// Maximum time to wait for the expected engines to register on the + /// frontend transport. + pub fn ready_timeout(&self) -> Duration { + Duration::from_secs(self.engine_ready_timeout_secs) + } + + /// Maximum time to wait for active requests to drain during shutdown. + pub fn shutdown_timeout(&self) -> Duration { + Duration::from_secs(self.shutdown_timeout) + } + + /// Maximum idle time on a keep-alive HTTP connection before the server + /// closes it. + pub fn keep_alive_timeout(&self) -> Duration { + self.http_timeout_keep_alive + .map_or(DEFAULT_KEEP_ALIVE_TIMEOUT, Duration::from_secs) + } + + /// Return the configured profiler mode, when profiling is enabled. + pub fn profiler(&self) -> Option { + self.profiler_config.as_ref().and_then(|c| c.profiler.clone()) + } + + /// Return the profiler config JSON for managed Python engine forwarding. + pub fn profiler_config_json(&self) -> Option { + self.profiler_config + .as_ref() + .map(serde_json::to_string) + .transpose() + .expect("profiler config serialization should not fail") + } + + /// Apply fallback logic for API key configuration from env variables. + fn apply_env_api_key_fallback(&mut self) { + if self.api_key.is_empty() + && let Ok(api_key) = std::env::var("VLLM_API_KEY") + { + self.api_key.push(api_key); + } + } + + /// Build the OpenAI-server config for the Python-bootstrap worker contract. + /// + /// The resulting config binds the Python-supplied transport addresses and + /// inherits an already open HTTP listener from the supervisor process. + fn into_bootstrapped_config( + self, + listen_fd: i32, + input_address: String, + output_address: String, + coordinator_address: Option, + engine_start_index: u32, + engine_count: usize, + ) -> Config { + let ready_timeout = self.ready_timeout(); + let shutdown_timeout = self.shutdown_timeout(); + let keep_alive_timeout = self.keep_alive_timeout(); + let api_server_options = self.api_server_options(); + let cors = self.cors_config(); + let tls = self.tls_config(); + let profiler = self.profiler(); + + Config { + transport_mode: TransportMode::Bootstrapped { + input_address, + output_address, + engine_start_index, + engine_count, + ready_timeout, + }, + coordinator_mode: match coordinator_address { + Some(address) => CoordinatorMode::External { address }, + None => CoordinatorMode::None, + }, + model: self.model, + served_model_name: self.served_model_name, + listener_mode: HttpListenerMode::InheritedFd { fd: listen_fd }, + tool_call_parser: self.tool_call_parser, + reasoning_parser: self.reasoning_parser, + renderer: self.renderer, + language_model_only: self.language_model_only, + chat_template: self.chat_template, + default_chat_template_kwargs: self.default_chat_template_kwargs, + chat_template_content_format: self.chat_template_content_format, + max_logprobs: self.max_logprobs, + api_server_options, + cors, + tls, + api_keys: self.api_key, + disable_log_stats: self.disable_log_stats, + grpc_port: self.grpc_port, + shutdown_timeout, + keep_alive_timeout, + profiler, + } + } + + /// Build the OpenAI-server config for the managed `serve` path that still + /// owns the startup handshake and binds its own HTTP listener. + fn into_managed_config( + self, + listener_mode: HttpListenerMode, + handshake_address: String, + advertised_host: String, + engine_count: usize, + local_input_address: Option, + local_output_address: Option, + ) -> Config { + let ready_timeout = self.ready_timeout(); + let shutdown_timeout = self.shutdown_timeout(); + let keep_alive_timeout = self.keep_alive_timeout(); + let api_server_options = self.api_server_options(); + let cors = self.cors_config(); + let tls = self.tls_config(); + let profiler = self.profiler(); + + Config { + transport_mode: TransportMode::HandshakeOwner { + handshake_address, + advertised_host, + engine_count, + ready_timeout, + local_input_address, + local_output_address, + }, + coordinator_mode: CoordinatorMode::MaybeInProc, + model: self.model, + served_model_name: self.served_model_name, + listener_mode, + tool_call_parser: self.tool_call_parser, + reasoning_parser: self.reasoning_parser, + renderer: self.renderer, + language_model_only: self.language_model_only, + chat_template: self.chat_template, + default_chat_template_kwargs: self.default_chat_template_kwargs, + chat_template_content_format: self.chat_template_content_format, + max_logprobs: self.max_logprobs, + api_server_options, + cors, + tls, + api_keys: self.api_key, + disable_log_stats: self.disable_log_stats, + grpc_port: self.grpc_port, + shutdown_timeout, + keep_alive_timeout, + profiler, + } + } + + fn api_server_options(&self) -> ApiServerOptions { + ApiServerOptions { + enable_log_requests: self.enable_log_requests, + enable_prompt_tokens_details: self.enable_prompt_tokens_details, + enable_request_id_headers: self.enable_request_id_headers, + } + } + + fn cors_config(&self) -> CorsConfig { + CorsConfig { + allow_origins: self.allowed_origins.0.clone(), + allow_methods: self.allowed_methods.0.clone(), + allow_headers: self.allowed_headers.0.clone(), + allow_credentials: self.allow_credentials, + } + } + + /// Build the TLS config: `Some` when any `ssl_*` argument is set, else + /// `None` (plaintext). The combination is validated in [`Config::validate`]. + fn tls_config(&self) -> Option { + let tls_requested = self.ssl_certfile.is_some() + || self.ssl_keyfile.is_some() + || self.ssl_ca_certs.is_some() + || self.ssl_cert_reqs != 0 + || self.ssl_ciphers.is_some(); + tls_requested.then(|| TlsConfig { + cert_file: self.ssl_certfile.clone(), + key_file: self.ssl_keyfile.clone(), + ca_certs: self.ssl_ca_certs.clone(), + cert_reqs: self.ssl_cert_reqs, + ciphers: self.ssl_ciphers.clone(), + }) + } +} + +fn default_engine_ready_timeout_secs() -> u64 { + 600 +} + +fn default_cors_wildcard() -> JsonStringList { + JsonStringList(vec!["*".to_string()]) +} + +fn default_py_bootstrap_parser_selection() -> ParserSelection { + ParserSelection::None +} + +/// Minimal profiler configuration parsed from `--profiler-config`. +/// +/// Only the `profiler` field is inspected by the Rust frontend to decide +/// whether to register the `/start_profile` and `/stop_profile` routes. +/// All other fields are accepted but ignored — they are consumed by the +/// Python engine layer. +#[derive(Clone, Debug, Default, PartialEq, Eq, Deserialize, Serialize)] +pub struct ProfilerConfig { + /// Profiler backend type (e.g. `"torch"`, `"cuda"`). When `null` or + /// absent, profiling is disabled. + #[serde(default)] + pub profiler: Option, + /// Additional Python profiler config fields consumed by the engine layer. + #[serde(flatten)] + pub extra: serde_json::Map, +} + +fn parse_json(value: &str) -> Result { + serde_json::from_str(value).map_err(|e| format!("invalid JSON object: {}", e.as_report())) +} + +fn parse_runtime_args_json(value: &str) -> Result { + let mut args: SharedRuntimeArgs = serde_json::from_str(value) + .map_err(|e| format!("invalid JSON arguments: {}", e.as_report()))?; + // --args-json is parsed with serde, so clap's env support does not run for + // the Python-supervised frontend path. + args.apply_env_api_key_fallback(); + args.unsupported.check()?; + Ok(args) +} + +/// Arguments for running the Rust frontend as a Python-bootstrapped worker. +#[derive(Educe, Clone, Args, PartialEq, Eq)] +#[educe(Debug)] +pub struct FrontendArgs { + /// Inherited listening socket file descriptor passed by the Python + /// supervisor. + #[arg(long)] + pub listen_fd: i32, + /// Frontend input ROUTER socket address that the Python engines will + /// connect to. + #[arg(long)] + pub input_address: String, + /// Frontend output PULL socket address that the Python engines will push + /// responses to. + #[arg(long)] + pub output_address: String, + /// Optional Python-owned frontend-side DP coordinator socket address for + /// external coordinator mode in the bootstrapped frontend path, i.e., + /// `stats_update_address`. + #[arg(long)] + pub coordinator_address: Option, + /// First data-parallel engine rank expected to register with this + /// bootstrapped frontend. + #[arg(long, default_value_t = 0)] + pub engine_start_index: u32, + /// Total number of data-parallel engines expected for this frontend. + #[arg(long, default_value_t = 1)] + pub engine_count: usize, + + /// Shared frontend arguments as one JSON object. + #[arg(long = "args-json", value_parser = parse_runtime_args_json, value_name = "JSON")] + pub runtime: SharedRuntimeArgs, +} + +impl FrontendArgs { + /// Convert the CLI arguments into the OpenAI server's runtime config. + pub fn into_config(self) -> Config { + self.runtime.into_bootstrapped_config( + self.listen_fd, + self.input_address, + self.output_address, + self.coordinator_address, + self.engine_start_index, + self.engine_count, + ) + } +} + +/// Arguments for the managed-engine mode that spawns Python on behalf of the +/// user. +#[derive(Educe, Clone, Args, PartialEq, Eq)] +#[educe(Debug)] +#[command(override_usage = "vllm-rs serve [OPTIONS] [-- ...]")] +pub struct ServeArgs { + /// Only launch the managed Python headless engine and do not start the Rust + /// frontend. + #[arg(long)] + pub headless: bool, + /// HTTP bind host for the OpenAI-compatible server. + #[arg(long, default_value = "127.0.0.1")] + pub host: String, + /// HTTP bind port for the OpenAI-compatible server. + #[arg(long, default_value_t = 8000)] + pub port: u16, + /// Unix domain socket path. If set, host and port arguments are ignored. + #[arg(long)] + pub uds: Option, + + /// Flag to print debug information about CLI argument parsing and exit. + #[educe(Debug(ignore))] + #[arg(long, hide = true, env = "VLLM_RS_DEBUG_CLI")] + pub debug_cli: bool, + + /// Shared frontend arguments. + #[command(flatten)] + pub runtime: SharedRuntimeArgs, + + /// Managed Python headless-engine arguments. + #[command(flatten)] + pub managed_engine: ManagedEngineArgs, +} + +impl ServeArgs { + /// Build the OpenAI-server runtime config used after the managed Python + /// engine starts. + pub fn to_frontend_config(&self, handshake_address: String) -> Config { + // Prefer IPC sockets for local engine input/output. + let (local_input_address, local_output_address) = + self.managed_engine.frontend_local_only().then(frontend_ipc_addresses).unzip(); + let listener_mode = match &self.uds { + Some(path) => HttpListenerMode::BindUnix { path: path.clone() }, + None => HttpListenerMode::BindTcp { + host: self.host.clone(), + port: self.port, + }, + }; + + self.runtime.clone().into_managed_config( + listener_mode, + handshake_address, + self.managed_engine.handshake_host.clone(), + self.managed_engine.data_parallel_size, + local_input_address, + local_output_address, + ) + } + + /// Build the managed Python-engine spawn configuration with the given + /// handshake port. + pub fn to_managed_engine_config(&self, handshake_port: u16) -> ManagedEngineConfig { + let reasoning_parser = + effective_engine_reasoning_parser(&self.runtime.reasoning_parser, &self.runtime.model); + let profiler_config = self.runtime.profiler_config_json(); + + self.managed_engine.clone().into_config( + self.runtime.model.clone(), + self.runtime.max_model_len, + self.runtime.max_logprobs, + profiler_config, + reasoning_parser.as_deref(), + self.runtime.language_model_only, + self.runtime.disable_log_stats, + self.runtime.shutdown_timeout, + handshake_port, + ) + } +} + +fn effective_engine_reasoning_parser(selection: &ParserSelection, model: &str) -> Option { + match selection { + ParserSelection::Auto => ReasoningParserFactory::global() + .resolve_name_for_model(model) + .map(str::to_string), + ParserSelection::None => None, + ParserSelection::Explicit(name) => Some(name.clone()), + } +} + +/// Allocate fresh IPC endpoints for one managed frontend instance. +fn frontend_ipc_addresses() -> (String, String) { + let preferred_base_path = std::env::var_os("VLLM_RPC_BASE_PATH") + .map(PathBuf::from) + .unwrap_or_else(std::env::temp_dir); + let input_name = format!("vllm-rs-i-{}", Uuid::new_v4().simple()); + let output_name = format!("vllm-rs-o-{}", Uuid::new_v4().simple()); + + let input = preferred_base_path.join(input_name); + let output = preferred_base_path.join(output_name); + + ( + format!("ipc://{}", input.to_string_lossy()), + format!("ipc://{}", output.to_string_lossy()), + ) +} + +#[cfg(test)] +mod tests; diff --git a/asian_ai/performance/vLLM/rust/src/cmd/src/cli/tests.rs b/asian_ai/performance/vLLM/rust/src/cmd/src/cli/tests.rs new file mode 100644 index 0000000000000000000000000000000000000000..b1c384c3544aa3b3e6461c1c3ac8d7208cd0a208 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/cmd/src/cli/tests.rs @@ -0,0 +1,1718 @@ +use expect_test::expect; +use vllm_engine_core_client::TransportMode; +use vllm_server::{Config, HttpListenerMode, ParserSelection, RendererSelection}; + +use super::{Cli, Command}; + +#[test] +fn serve_args_forward_python_flags_with_separator() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--python", + "../vllm/.venv/bin/python", + "--max-model-len", + "512", + "--", + "--dtype", + "float16", + ]) + .unwrap(); + + expect![[r#" + Cli { + command: Serve( + ServeArgs { + headless: false, + host: "127.0.0.1", + port: 8000, + uds: None, + runtime: SharedRuntimeArgs { + model: "Qwen/Qwen3-0.6B", + engine_ready_timeout_secs: 600, + tool_call_parser: Auto, + reasoning_parser: Auto, + renderer: Auto, + language_model_only: false, + max_model_len: Some( + 512, + ), + max_logprobs: None, + grpc_port: None, + shutdown_timeout: 0, + http_timeout_keep_alive: None, + chat_template: None, + default_chat_template_kwargs: None, + chat_template_content_format: Auto, + enable_log_requests: false, + enable_prompt_tokens_details: false, + enable_request_id_headers: false, + disable_log_stats: false, + served_model_name: [], + allowed_origins: JsonStringList( + [ + "*", + ], + ), + allowed_methods: JsonStringList( + [ + "*", + ], + ), + allowed_headers: JsonStringList( + [ + "*", + ], + ), + allow_credentials: false, + ssl_keyfile: None, + ssl_certfile: None, + ssl_ca_certs: None, + ssl_cert_reqs: 0, + ssl_ciphers: None, + profiler_config: None, + }, + managed_engine: ManagedEngineArgs { + python: "../vllm/.venv/bin/python", + handshake_host: "127.0.0.1", + handshake_port: None, + data_parallel_size: 1, + data_parallel_size_local: None, + python_args: [ + "--dtype", + "float16", + ], + }, + }, + ), + } + "#]] + .assert_debug_eq(&cli); +} + +#[test] +fn serve_args_auto_forward_python_flags_without_separator() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--python", + "python3", + "--quantization", + "awq", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + expect![[r#" + [ + "--quantization", + "awq", + ] + "#]] + .assert_debug_eq(&args.managed_engine.python_args); +} + +#[test] +fn serve_args_auto_forward_enable_lora_to_python() { + let cli = + Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B", "--enable-lora"]).unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + expect![[r#" + [ + "--enable-lora", + ] + "#]] + .assert_debug_eq(&args.managed_engine.python_args); +} + +#[test] +fn serve_args_forward_shutdown_timeout_to_managed_engine() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--shutdown-timeout", + "60", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + assert_eq!(args.runtime.shutdown_timeout, 60); + + let config = args.to_managed_engine_config(5555); + expect![[r#" + [ + "--reasoning-parser", + "qwen3", + "--shutdown-timeout", + "60", + ] + "#]] + .assert_debug_eq(&config.python_args); +} + +#[test] +fn serve_args_forward_disable_log_stats_to_managed_engine() { + let cli = Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B", "--disable-log-stats"]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + assert!(args.runtime.disable_log_stats); + + let config = args.to_managed_engine_config(5555); + expect![[r#" + [ + "--reasoning-parser", + "qwen3", + "--disable-log-stats", + ] + "#]] + .assert_debug_eq(&config.python_args); +} + +#[test] +fn serve_args_forward_profiler_config_to_managed_engine() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--profiler-config", + r#"{"profiler":"torch","torch_profiler_dir":"/tmp/profile"}"#, + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + assert_eq!(args.runtime.profiler().as_deref(), Some("torch")); + + let config = args.to_managed_engine_config(5555); + let profiler_flag_index = config + .python_args + .iter() + .position(|arg| arg == "--profiler-config") + .expect("profiler config flag"); + let profiler_config: serde_json::Value = + serde_json::from_str(&config.python_args[profiler_flag_index + 1]) + .expect("profiler config json"); + assert_eq!( + profiler_config, + serde_json::json!({ + "profiler": "torch", + "torch_profiler_dir": "/tmp/profile", + }) + ); +} + +#[test] +fn serve_args_forward_max_logprobs_to_frontend_and_managed_engine() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--max-logprobs", + "-1", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + assert_eq!(args.runtime.max_logprobs, Some(-1)); + + let frontend_config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string()); + assert_eq!(frontend_config.max_logprobs, Some(-1)); + + let engine_config = args.to_managed_engine_config(5555); + expect![[r#" + [ + "--max-logprobs", + "-1", + "--reasoning-parser", + "qwen3", + ] + "#]] + .assert_debug_eq(&engine_config.python_args); +} + +#[test] +fn serve_args_resolve_auto_reasoning_parser_for_managed_engine() { + let cli = Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B"]).unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + assert_eq!(args.runtime.reasoning_parser, ParserSelection::Auto); + + let config = args.to_managed_engine_config(5555); + expect![[r#" + [ + "--reasoning-parser", + "qwen3", + ] + "#]] + .assert_debug_eq(&config.python_args); +} + +#[test] +fn serve_args_forward_explicit_reasoning_parser_to_managed_engine() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Unknown/Model", + "--reasoning-parser", + "deepseek_r1", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + + let config = args.to_managed_engine_config(5555); + expect![[r#" + [ + "--reasoning-parser", + "deepseek_r1", + ] + "#]] + .assert_debug_eq(&config.python_args); +} + +#[test] +fn serve_args_do_not_forward_disabled_reasoning_parser_to_managed_engine() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--reasoning-parser", + "none", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + + let config = args.to_managed_engine_config(5555); + assert!(config.python_args.is_empty()); +} + +#[test] +fn serve_args_forward_reasoning_parser_even_with_passthrough_reasoning_parser() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--", + "--reasoning-parser", + "deepseek_r1", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + + let config = args.to_managed_engine_config(5555); + expect![[r#" + [ + "--reasoning-parser", + "deepseek_r1", + "--reasoning-parser", + "qwen3", + ] + "#]] + .assert_debug_eq(&config.python_args); +} + +#[test] +fn serve_args_auto_forward_python_multi_char_alias_without_separator() { + let cli = Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B", "-tp", "2"]).unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + expect![[r#" + [ + "--tensor-parallel-size", + "2", + ] + "#]] + .assert_debug_eq(&args.managed_engine.python_args); +} + +#[test] +fn serve_args_accept_explicit_deepseek_v32_renderer() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--tokenizer-mode", + "deepseek_v32", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + assert_eq!(args.runtime.renderer, RendererSelection::DeepSeekV32); +} + +#[test] +fn serve_passes_enable_request_id_headers_into_config() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--enable-request-id-headers", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + let config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string()); + assert!(config.api_server_options.enable_request_id_headers); +} + +#[test] +fn serve_passes_enable_prompt_tokens_details_into_config() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--enable-prompt-tokens-details", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + let config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string()); + assert!(config.api_server_options.enable_prompt_tokens_details); +} + +#[test] +fn serve_passes_tls_into_config() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--ssl-certfile", + "/tmp/cert.pem", + "--ssl-keyfile", + "/tmp/key.pem", + "--ssl-ca-certs", + "/tmp/ca.pem", + "--ssl-cert-reqs", + "2", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + let config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string()); + let tls = config.tls.expect("tls configured"); + assert_eq!(tls.cert_file.as_deref(), Some("/tmp/cert.pem")); + assert_eq!(tls.key_file.as_deref(), Some("/tmp/key.pem")); + assert_eq!(tls.ca_certs.as_deref(), Some("/tmp/ca.pem")); + assert_eq!(tls.cert_reqs, 2); +} + +#[test] +fn serve_without_ssl_flags_has_no_tls() { + let cli = Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B"]).unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + let config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string()); + assert!(config.tls.is_none()); +} + +#[test] +fn serve_ssl_keyfile_without_certfile_fails_validation() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--ssl-keyfile", + "/tmp/key.pem", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + let config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string()); + // TLS is requested (a key was given) but there is no certificate, so + // validation fails loud rather than silently serving plaintext. + assert_eq!(config.tls.as_ref().expect("tls requested").cert_file, None); + let err = config.validate().unwrap_err().to_string(); + assert!(err.contains("--ssl-certfile is required"), "{err}"); +} + +#[test] +fn serve_mtls_without_ca_certs_fails_validation() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--ssl-certfile", + "/tmp/cert.pem", + "--ssl-cert-reqs", + "2", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + let config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string()); + // Client-cert verification without a CA bundle has nothing to verify + // against, so it fails loud at startup. + let err = config.validate().unwrap_err().to_string(); + assert!(err.contains("--ssl-ca-certs is required"), "{err}"); +} + +#[test] +fn frontend_args_json_passes_tls_into_config() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B","ssl_certfile":"/tmp/cert.pem","ssl_keyfile":"/tmp/key.pem"}"#, + ]) + .unwrap(); + + let Command::Frontend(args) = cli.command else { + panic!("expected frontend args"); + }; + let config = args.into_config(); + let tls = config.tls.expect("tls configured"); + assert_eq!(tls.cert_file.as_deref(), Some("/tmp/cert.pem")); + assert_eq!(tls.key_file.as_deref(), Some("/tmp/key.pem")); +} + +#[test] +fn frontend_args_json_rejects_out_of_range_cert_reqs() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B","ssl_certfile":"/tmp/cert.pem","ssl_cert_reqs":5}"#, + ]) + .unwrap(); + + let Command::Frontend(args) = cli.command else { + panic!("expected frontend args"); + }; + // The JSON path bypasses clap's range check, so validate() is the only guard. + let config = args.into_config(); + let err = config.validate().unwrap_err().to_string(); + assert!(err.contains("--ssl-cert-reqs"), "{err}"); +} + +#[test] +fn frontend_args_json_passes_enable_request_id_headers_into_config() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B","enable_request_id_headers":true}"#, + ]) + .unwrap(); + + let Command::Frontend(args) = cli.command else { + panic!("expected frontend args"); + }; + let config = args.into_config(); + assert!(config.api_server_options.enable_request_id_headers); +} + +#[test] +fn serve_passes_api_keys_into_config() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--api-key", + "secret-a", + "--api-key", + "secret-b", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + let config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string()); + assert_eq!(config.api_keys, vec!["secret-a", "secret-b"]); + let debug = format!("{config:#?}"); + assert!(debug.contains("api_keys: [; 2]")); + assert!(!debug.contains("secret-a")); + assert!(!debug.contains("secret-b")); +} + +#[test] +fn frontend_args_json_accepts_api_key_string() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B","api_key":"secret"}"#, + ]) + .unwrap(); + + let Command::Frontend(args) = cli.command else { + panic!("expected frontend args"); + }; + let config = args.into_config(); + assert_eq!(config.api_keys, vec!["secret"]); +} + +#[test] +fn frontend_args_json_accepts_api_key_list() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B","api_key":["secret-a","secret-b"]}"#, + ]) + .unwrap(); + + let Command::Frontend(args) = cli.command else { + panic!("expected frontend args"); + }; + let config = args.into_config(); + assert_eq!(config.api_keys, vec!["secret-a", "secret-b"]); +} + +#[test] +fn serve_args_reject_unknown_renderer_value() { + let error = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--tokenizer-mode", + "definitely_missing", + ]) + .unwrap_err(); + + expect![[r#" + error: invalid value 'definitely_missing' for '--tokenizer-mode ': unknown renderer `definitely_missing` (expected one of: auto, hf, deepseek_v32, deepseek_v4, harmony) + + For more information, try '--help'. + "#]] + .assert_eq(&error.to_string()); +} + +#[test] +fn serve_args_reject_unsupported_flag_arg() { + let error = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--root-path", + "/prefix", + ]) + .unwrap_err(); + + expect![[r#" + error: invalid value '/prefix' for '--root-path ': argument is not implemented in Rust frontend yet + + Remove this unsupported argument to continue. + + Alternatively, if you intend to pass it only to the Python engine, put it after `--` (e.g., `-- `). + This may lead to unexpected behavior as the Rust frontend will completely ignore that argument. + + For more information, try '--help'. + "#]].assert_eq(&error.to_string()); +} + +#[test] +fn serve_args_reject_unsupported_no_flag_alias() { + let error = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--no-enable-log-deltas", + ]) + .unwrap_err(); + + expect![[r#" + error: invalid value 'true' for '--enable-log-deltas []': argument is not implemented in Rust frontend yet + + Remove this unsupported argument to continue. + + Alternatively, if you intend to pass it only to the Python engine, put it after `--` (e.g., `-- `). + This may lead to unexpected behavior as the Rust frontend will completely ignore that argument. + + For more information, try '--help'. + "#]] + .assert_eq(&error.to_string()); +} + +#[test] +fn frontend_args_accept_json() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--coordinator-address", + "tcp://127.0.0.1:7000", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B","engine_count":2}"#, + ]) + .unwrap(); + + expect![[r#" + Cli { + command: Frontend( + FrontendArgs { + listen_fd: 3, + input_address: "ipc:///tmp/input.sock", + output_address: "ipc:///tmp/output.sock", + coordinator_address: Some( + "tcp://127.0.0.1:7000", + ), + engine_start_index: 0, + engine_count: 1, + runtime: SharedRuntimeArgs { + model: "Qwen/Qwen3-0.6B", + engine_ready_timeout_secs: 600, + tool_call_parser: None, + reasoning_parser: None, + renderer: Auto, + language_model_only: false, + max_model_len: None, + max_logprobs: None, + grpc_port: None, + shutdown_timeout: 0, + http_timeout_keep_alive: None, + chat_template: None, + default_chat_template_kwargs: None, + chat_template_content_format: Auto, + enable_log_requests: false, + enable_prompt_tokens_details: false, + enable_request_id_headers: false, + disable_log_stats: false, + served_model_name: [], + allowed_origins: JsonStringList( + [ + "*", + ], + ), + allowed_methods: JsonStringList( + [ + "*", + ], + ), + allowed_headers: JsonStringList( + [ + "*", + ], + ), + allow_credentials: false, + ssl_keyfile: None, + ssl_certfile: None, + ssl_ca_certs: None, + ssl_cert_reqs: 0, + ssl_ciphers: None, + profiler_config: None, + }, + }, + ), + } + "#]] + .assert_debug_eq(&cli); +} + +#[test] +fn frontend_args_json_applies_defaults() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B"}"#, + ]) + .unwrap(); + + let Command::Frontend(args) = cli.command else { + panic!("expected frontend args"); + }; + assert_eq!(args.runtime.model, "Qwen/Qwen3-0.6B"); + assert_eq!(args.runtime.engine_ready_timeout_secs, 600); + assert_eq!(args.runtime.tool_call_parser, ParserSelection::None); + assert_eq!(args.runtime.reasoning_parser, ParserSelection::None); + assert_eq!(args.runtime.renderer, RendererSelection::Auto); + assert_eq!(args.runtime.max_model_len, None); + assert_eq!(args.runtime.max_logprobs, None); + assert_eq!(args.runtime.shutdown_timeout, 0); +} + +#[test] +fn frontend_args_json_accepts_supported_non_default_fields() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B","engine_ready_timeout_secs":42,"tool_call_parser":"hermes","reasoning_parser":"qwen3_thinking","tokenizer_mode":"deepseek_v32","language_model_only":true,"max_model_len":8192,"max_logprobs":-1,"shutdown_timeout":3}"#, + ]) + .unwrap(); + + let Command::Frontend(args) = cli.command else { + panic!("expected frontend args"); + }; + assert_eq!(args.runtime.engine_ready_timeout_secs, 42); + assert_eq!( + args.runtime.tool_call_parser, + ParserSelection::Explicit("hermes".to_string()) + ); + assert_eq!( + args.runtime.reasoning_parser, + ParserSelection::Explicit("qwen3_thinking".to_string()) + ); + assert_eq!(args.runtime.renderer, RendererSelection::DeepSeekV32); + assert!(args.runtime.language_model_only); + assert_eq!(args.runtime.max_model_len, Some(8192)); + assert_eq!(args.runtime.max_logprobs, Some(-1)); + assert_eq!(args.runtime.shutdown_timeout, 3); +} + +#[test] +fn serve_args_accept_none_reasoning_parser() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--reasoning-parser", + "none", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + assert_eq!(args.runtime.reasoning_parser, ParserSelection::None); + assert_eq!(args.runtime.tool_call_parser, ParserSelection::Auto); +} + +#[test] +fn frontend_args_json_ignores_unknown_fields() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B","uds":"/tmp/vllm.sock","nested_unknown":{"x":1}}"#, + ]) + .unwrap(); + + let Command::Frontend(args) = cli.command else { + panic!("expected frontend args"); + }; + assert_eq!(args.runtime.model, "Qwen/Qwen3-0.6B"); +} + +#[test] +fn frontend_args_json_sets_prompt_tokens_details_flag() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B","api_server_count":2,"enable_prompt_tokens_details":true}"#, + ]) + .unwrap(); + + let Command::Frontend(args) = cli.command else { + panic!("expected frontend args"); + }; + assert_eq!(args.runtime.model, "Qwen/Qwen3-0.6B"); + assert!(args.runtime.enable_prompt_tokens_details); +} + +#[test] +fn serve_args_parse_cors_flags() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--allowed-origins", + r#"["http://a.com","http://b.com"]"#, + "--allowed-methods", + r#"["GET","POST"]"#, + "--allow-credentials", + ]) + .unwrap(); + + let Command::Serve(serve) = cli.command else { + panic!("expected serve args"); + }; + assert_eq!( + serve.runtime.allowed_origins.0, + ["http://a.com", "http://b.com"] + ); + assert_eq!(serve.runtime.allowed_methods.0, ["GET", "POST"]); + assert!(serve.runtime.allow_credentials); + // Unspecified lists keep the permissive default. + assert_eq!(serve.runtime.allowed_headers.0, ["*"]); +} + +#[test] +fn serve_args_cors_defaults_are_permissive() { + let cli = Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B"]).unwrap(); + + let Command::Serve(serve) = cli.command else { + panic!("expected serve args"); + }; + assert_eq!(serve.runtime.allowed_origins.0, ["*"]); + assert_eq!(serve.runtime.allowed_methods.0, ["*"]); + assert_eq!(serve.runtime.allowed_headers.0, ["*"]); + assert!(!serve.runtime.allow_credentials); +} + +#[test] +fn frontend_args_json_parses_cors_fields() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B","allowed_origins":["http://a.com"],"allow_credentials":true}"#, + ]) + .unwrap(); + + let Command::Frontend(args) = cli.command else { + panic!("expected frontend args"); + }; + assert_eq!(args.runtime.allowed_origins.0, ["http://a.com"]); + assert!(args.runtime.allow_credentials); + // Unspecified lists fall back to the permissive default via serde. + assert_eq!(args.runtime.allowed_methods.0, ["*"]); +} + +#[test] +fn frontend_args_json_rejects_unsupported_fields() { + let error = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B","root_path":"/prefix"}"#, + ]) + .unwrap_err(); + + expect![[r#" + error: invalid value '{"model_tag":"Qwen/Qwen3-0.6B","root_path":"/prefix"}' for '--args-json ': + The following arguments are not implemented in Rust frontend yet: + - root_path + + Remove these arguments to continue. + + For more information, try '--help'. + "#]].assert_eq(&error.to_string()); +} + +#[test] +fn frontend_args_json_aggregates_multiple_unsupported_fields() { + let error = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B","response_role":"assistant","root_path":"/prefix"}"#, + ]) + .unwrap_err(); + + let actual = error.to_string().replace(": \n", ":\n"); + expect![[r#" + error: invalid value '{"model_tag":"Qwen/Qwen3-0.6B","response_role":"assistant","root_path":"/prefix"}' for '--args-json ': + The following arguments are not implemented in Rust frontend yet: + - response_role + - root_path + + Remove these arguments to continue. + + For more information, try '--help'. + "#]].assert_eq(&actual); +} + +#[test] +fn frontend_args_json_rejects_malformed_json() { + let error = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B""#, + ]) + .unwrap_err(); + + expect![[r#" + error: invalid value '{"model_tag":"Qwen/Qwen3-0.6B"' for '--args-json ': invalid JSON arguments: EOF while parsing an object at line 1 column 30 + + For more information, try '--help'. + "#]].assert_eq(&error.to_string()); +} + +#[test] +fn serve_args_reject_flags_before_model() { + let error = Cli::try_parse_from(["vllm-rs", "serve", "--python", "python3", "Qwen/Qwen3-0.6B"]) + .unwrap_err(); + + expect![[r#" + error: the model must appear immediately after the command + + Usage: vllm-rs serve [OPTIONS] [-- ...] + + For more information, try '--help'. + "#]] + .assert_eq(&error.to_string()); +} + +#[test] +fn serve_args_accept_headless_mode() { + let cli = Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B", "--headless"]).unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + assert!(args.headless); +} + +#[test] +fn serve_args_keep_python_passthrough_flags_after_separator() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--python", + "python3", + "--", + "--tensor-parallel-size", + "2", + "--dtype", + "float16", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + expect![[r#" + [ + "--tensor-parallel-size", + "2", + "--dtype", + "float16", + ] + "#]] + .assert_debug_eq(&args.managed_engine.python_args); +} + +#[test] +fn serve_args_keep_python_multi_char_alias_after_separator() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--python", + "python3", + "--", + "-tp", + "2", + "--dtype", + "float16", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + expect![[r#" + [ + "-tp", + "2", + "--dtype", + "float16", + ] + "#]] + .assert_debug_eq(&args.managed_engine.python_args); +} + +#[test] +fn serve_args_keep_frontend_arg_after_separator() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--", + "--uds", + "/tmp/vllm.sock", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + expect![[r#" + [ + "--uds", + "/tmp/vllm.sock", + ] + "#]] + .assert_debug_eq(&args.managed_engine.python_args); +} + +#[test] +fn serve_args_keep_python_multi_char_engine_aliases_after_separator() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--", + "-dpr", + "1", + "-dpl", + "2", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + expect![[r#" + [ + "-dpr", + "1", + "-dpl", + "2", + ] + "#]] + .assert_debug_eq(&args.managed_engine.python_args); +} + +#[test] +fn serve_args_auto_forward_unknown_flags_without_separator() { + let cli = Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B", "--foo", "bar"]).unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + expect![[r#" + [ + "--foo", + "bar", + ] + "#]] + .assert_debug_eq(&args.managed_engine.python_args); +} + +#[test] +fn serve_args_auto_forward_negative_value_without_separator() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--num-gpu-blocks-override", + "-1", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + expect![[r#" + [ + "--num-gpu-blocks-override", + "-1", + ] + "#]] + .assert_debug_eq(&args.managed_engine.python_args); +} + +#[test] +fn serve_args_accept_handshake_aliases() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--python", + "python3", + "--handshake-host", + "10.99.48.128", + "--handshake-port", + "13345", + "--data-parallel-size", + "4", + ]) + .unwrap(); + + expect![[r#" + Cli { + command: Serve( + ServeArgs { + headless: false, + host: "127.0.0.1", + port: 8000, + uds: None, + runtime: SharedRuntimeArgs { + model: "Qwen/Qwen3-0.6B", + engine_ready_timeout_secs: 600, + tool_call_parser: Auto, + reasoning_parser: Auto, + renderer: Auto, + language_model_only: false, + max_model_len: None, + max_logprobs: None, + grpc_port: None, + shutdown_timeout: 0, + http_timeout_keep_alive: None, + chat_template: None, + default_chat_template_kwargs: None, + chat_template_content_format: Auto, + enable_log_requests: false, + enable_prompt_tokens_details: false, + enable_request_id_headers: false, + disable_log_stats: false, + served_model_name: [], + allowed_origins: JsonStringList( + [ + "*", + ], + ), + allowed_methods: JsonStringList( + [ + "*", + ], + ), + allowed_headers: JsonStringList( + [ + "*", + ], + ), + allow_credentials: false, + ssl_keyfile: None, + ssl_certfile: None, + ssl_ca_certs: None, + ssl_cert_reqs: 0, + ssl_ciphers: None, + profiler_config: None, + }, + managed_engine: ManagedEngineArgs { + python: "python3", + handshake_host: "10.99.48.128", + handshake_port: Some( + 13345, + ), + data_parallel_size: 4, + data_parallel_size_local: None, + python_args: [], + }, + }, + ), + } + "#]] + .assert_debug_eq(&cli); +} + +#[test] +fn serve_args_accept_data_parallel_primary_flags() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--data-parallel-address", + "10.99.48.128", + "--data-parallel-rpc-port", + "13345", + "--data-parallel-size", + "4", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + assert!(!args.headless); + assert_eq!(args.managed_engine.handshake_host, "10.99.48.128"); + assert_eq!(args.managed_engine.handshake_port, Some(13345)); + assert_eq!(args.managed_engine.data_parallel_size, 4); +} + +#[test] +fn serve_frontend_config_uses_dp_address_as_advertised_host() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--handshake-host", + "10.99.48.128", + "--data-parallel-size", + "4", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + let config = args.to_frontend_config("tcp://10.99.48.128:29550".to_string()); + + let TransportMode::HandshakeOwner { + handshake_address, + advertised_host, + engine_count, + ready_timeout, + local_input_address, + local_output_address, + } = &config.transport_mode + else { + panic!("expected handshake-owned transport"); + }; + + assert_eq!(handshake_address, "tcp://10.99.48.128:29550"); + assert_eq!(advertised_host, "10.99.48.128"); + assert_eq!(*engine_count, 4); + assert!( + local_input_address + .as_deref() + .is_some_and(|address| address.starts_with("ipc://")) + ); + assert!( + local_output_address + .as_deref() + .is_some_and(|address| address.starts_with("ipc://")) + ); + assert_ne!(local_input_address, local_output_address); + + expect![[r#" + Config { + transport_mode: HandshakeOwner { + handshake_address: "tcp://10.99.48.128:29550", + advertised_host: "10.99.48.128", + engine_count: 4, + ready_timeout: 600s, + local_input_address: Some( + "", + ), + local_output_address: Some( + "", + ), + }, + coordinator_mode: MaybeInProc, + model: "Qwen/Qwen3-0.6B", + served_model_name: [], + listener_mode: BindTcp { + host: "127.0.0.1", + port: 8000, + }, + tool_call_parser: Auto, + reasoning_parser: Auto, + renderer: Auto, + language_model_only: false, + chat_template: None, + default_chat_template_kwargs: None, + chat_template_content_format: Auto, + max_logprobs: None, + api_server_options: ApiServerOptions { + enable_log_requests: false, + enable_prompt_tokens_details: false, + enable_request_id_headers: false, + }, + cors: CorsConfig { + allow_origins: [ + "*", + ], + allow_methods: [ + "*", + ], + allow_headers: [ + "*", + ], + allow_credentials: false, + }, + tls: None, + api_keys: [], + disable_log_stats: false, + grpc_port: None, + shutdown_timeout: 0ns, + keep_alive_timeout: 5s, + profiler: None, + } + "#]] + .assert_debug_eq(&Config { + transport_mode: TransportMode::HandshakeOwner { + handshake_address: handshake_address.clone(), + advertised_host: advertised_host.clone(), + engine_count: *engine_count, + ready_timeout: *ready_timeout, + local_input_address: Some("".to_string()), + local_output_address: Some("".to_string()), + }, + ..config.clone() + }); +} + +#[test] +fn serve_frontend_config_keeps_tcp_transport_for_non_local_only_topology() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--data-parallel-address", + "10.99.48.128", + "--data-parallel-size", + "4", + "--data-parallel-size-local", + "2", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + let config = args.to_frontend_config("tcp://10.99.48.128:29550".to_string()); + + expect![[r#" + Config { + transport_mode: HandshakeOwner { + handshake_address: "tcp://10.99.48.128:29550", + advertised_host: "10.99.48.128", + engine_count: 4, + ready_timeout: 600s, + local_input_address: None, + local_output_address: None, + }, + coordinator_mode: MaybeInProc, + model: "Qwen/Qwen3-0.6B", + served_model_name: [], + listener_mode: BindTcp { + host: "127.0.0.1", + port: 8000, + }, + tool_call_parser: Auto, + reasoning_parser: Auto, + renderer: Auto, + language_model_only: false, + chat_template: None, + default_chat_template_kwargs: None, + chat_template_content_format: Auto, + max_logprobs: None, + api_server_options: ApiServerOptions { + enable_log_requests: false, + enable_prompt_tokens_details: false, + enable_request_id_headers: false, + }, + cors: CorsConfig { + allow_origins: [ + "*", + ], + allow_methods: [ + "*", + ], + allow_headers: [ + "*", + ], + allow_credentials: false, + }, + tls: None, + api_keys: [], + disable_log_stats: false, + grpc_port: None, + shutdown_timeout: 0ns, + keep_alive_timeout: 5s, + profiler: None, + } + "#]] + .assert_debug_eq(&config); +} + +#[test] +fn frontend_args_reject_legacy_handshake_flags() { + let error = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B"}"#, + "--handshake-address", + "tcp://127.0.0.1:62100", + ]) + .unwrap_err(); + + assert!(error.to_string().contains("--handshake-address")); +} + +#[test] +fn frontend_config_uses_external_coordinator_when_coordinator_address_is_present() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--coordinator-address", + "tcp://127.0.0.1:7000", + "--engine-start-index", + "3", + "--engine-count", + "1", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B"}"#, + ]) + .unwrap(); + + let Command::Frontend(args) = cli.command else { + panic!("expected frontend args"); + }; + let config = args.into_config(); + + expect![[r#" + Config { + transport_mode: Bootstrapped { + input_address: "ipc:///tmp/input.sock", + output_address: "ipc:///tmp/output.sock", + engine_start_index: 3, + engine_count: 1, + ready_timeout: 600s, + }, + coordinator_mode: External { + address: "tcp://127.0.0.1:7000", + }, + model: "Qwen/Qwen3-0.6B", + served_model_name: [], + listener_mode: InheritedFd { + fd: 3, + }, + tool_call_parser: None, + reasoning_parser: None, + renderer: Auto, + language_model_only: false, + chat_template: None, + default_chat_template_kwargs: None, + chat_template_content_format: Auto, + max_logprobs: None, + api_server_options: ApiServerOptions { + enable_log_requests: false, + enable_prompt_tokens_details: false, + enable_request_id_headers: false, + }, + cors: CorsConfig { + allow_origins: [ + "*", + ], + allow_methods: [ + "*", + ], + allow_headers: [ + "*", + ], + allow_credentials: false, + }, + tls: None, + api_keys: [], + disable_log_stats: false, + grpc_port: None, + shutdown_timeout: 0ns, + keep_alive_timeout: 5s, + profiler: None, + } + "#]] + .assert_debug_eq(&config); +} + +#[test] +fn serve_frontend_config_uses_unix_listener_when_uds_is_present() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "serve", + "Qwen/Qwen3-0.6B", + "--uds", + "/tmp/vllm.sock", + ]) + .unwrap(); + + let Command::Serve(args) = cli.command else { + panic!("expected serve args"); + }; + let config = args.to_frontend_config("tcp://127.0.0.1:29550".to_string()); + + assert_eq!( + config.listener_mode, + HttpListenerMode::BindUnix { + path: "/tmp/vllm.sock".to_string(), + } + ); +} + +#[test] +fn frontend_args_json_enables_profiling_when_profiler_config_set() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B","profiler_config":{"profiler":"torch","torch_profiler_dir":"/tmp/profile"}}"#, + ]) + .unwrap(); + + let Command::Frontend(args) = cli.command else { + panic!("expected frontend args"); + }; + assert_eq!(args.runtime.profiler().as_deref(), Some("torch")); + let config = args.into_config(); + assert_eq!(config.profiler.as_deref(), Some("torch")); +} + +#[test] +fn frontend_args_json_disables_profiling_when_profiler_config_absent() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B"}"#, + ]) + .unwrap(); + + let Command::Frontend(args) = cli.command else { + panic!("expected frontend args"); + }; + assert_eq!(args.runtime.profiler(), None); + let config = args.into_config(); + assert_eq!(config.profiler, None); +} + +#[test] +fn frontend_args_json_disables_profiling_when_profiler_type_is_null() { + let cli = Cli::try_parse_from([ + "vllm-rs", + "frontend", + "--listen-fd", + "3", + "--input-address", + "ipc:///tmp/input.sock", + "--output-address", + "ipc:///tmp/output.sock", + "--args-json", + r#"{"model_tag":"Qwen/Qwen3-0.6B","profiler_config":{"profiler":null}}"#, + ]) + .unwrap(); + + let Command::Frontend(args) = cli.command else { + panic!("expected frontend args"); + }; + assert_eq!(args.runtime.profiler(), None); + let config = args.into_config(); + assert_eq!(config.profiler, None); +} diff --git a/asian_ai/performance/vLLM/rust/src/cmd/src/cli/unsupported.rs b/asian_ai/performance/vLLM/rust/src/cmd/src/cli/unsupported.rs new file mode 100644 index 0000000000000000000000000000000000000000..4f05aefcf4b901b5bf6b1eed1c4083e7efb0b47e --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/cmd/src/cli/unsupported.rs @@ -0,0 +1,578 @@ +#![allow(clippy::doc_lazy_continuation)] + +use std::fmt::Display; +use std::str::FromStr; + +use clap::Args; +use clap::builder::{TypedValueParser, ValueParserFactory}; +use itertools::Itertools; +use serde::{Deserialize, Deserializer, Serialize}; + +/// Marker type for frontend-owned `serve` arguments that `vllm-rs` recognizes +/// but does not support yet. +/// +/// When passed as JSON args, it can be deserialized from any value, and +/// serializes back to the original value. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(transparent)] +pub struct Unsupported(pub serde_json::Value); + +impl FromStr for Unsupported { + type Err = String; + + fn from_str(_s: &str) -> Result { + Err("argument is not implemented in Rust frontend yet + +Remove this unsupported argument to continue. + +Alternatively, if you intend to pass it only to the Python engine, put it after `--` (e.g., `-- `). +This may lead to unexpected behavior as the Rust frontend will completely ignore that argument." + .to_string()) + } +} + +/// Marker type for no-op arguments that are accepted by the Rust frontend but +/// have no effect. +/// +/// When passed as JSON args, it can be deserialized from any value, but always +/// serializes back to `null`. +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +pub struct Noop; + +impl<'de> Deserialize<'de> for Noop { + fn deserialize(_deserializer: D) -> Result + where + D: Deserializer<'de>, + { + Ok(Noop) + } +} + +impl ValueParserFactory for Noop { + type Parser = NoopValueParser; + + fn value_parser() -> Self::Parser { + NoopValueParser + } +} + +#[derive(Copy, Clone, Debug)] +pub struct NoopValueParser; + +#[track_caller] +fn noop_warn(arg: impl Display) { + tracing::warn!("argument '{arg}' currently has no effect in Rust frontend, ignoring"); +} + +impl TypedValueParser for NoopValueParser { + type Value = Noop; + + fn parse_ref( + &self, + _cmd: &clap::Command, + arg: Option<&clap::Arg>, + _value: &std::ffi::OsStr, + ) -> Result { + if let Some(arg) = arg { + noop_warn(arg); + } + Ok(Noop) + } +} + +/// Frontend-owned Python `serve` arguments that `vllm-rs` recognizes but does +/// not support yet. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, PartialEq, Eq, Default, Args, Serialize, Deserialize)] +#[command(next_help_heading = "Options not implemented in Rust frontend yet")] +pub struct UnsupportedArgs { + #[command(flatten)] + #[serde(default, flatten)] + top_level: TopLevelUnsupportedArgs, + #[command(flatten)] + #[serde(default, flatten)] + engine: EngineUnsupportedArgs, + #[command(flatten)] + #[serde(default, flatten)] + server: ServerUnsupportedArgs, +} + +impl UnsupportedArgs { + /// Check whether any unsupported arguments are set, and if so, return an + /// error listing them. Also warn about any no-op arguments that are set + /// but will be ignored. + pub(crate) fn check(&self) -> Result<(), String> { + let value = serde_json::to_value(self).unwrap(); + let map = value.as_object().unwrap(); + let mut unsupported = Vec::new(); + + for (key, value) in map { + if value.is_null() { + noop_warn(key); + } else { + unsupported.push(key.as_str()); + } + } + + if !unsupported.is_empty() { + unsupported.sort_unstable(); + let bullets = unsupported.into_iter().map(|key| format!("- {key}")).join("\n"); + return Err(format!( + " +The following arguments are not implemented in Rust frontend yet: +{bullets} + +Remove these arguments to continue." + )); + } + + Ok(()) + } +} + +/// Frontend-owned Python `vllm serve` top-level arguments that `vllm-rs` +/// recognizes but does not support yet. +/// +/// Source of truth in Python vLLM: +/// - `vllm.entrypoints.openai.cli_args.make_arg_parser(...)` +/// - `vllm.entrypoints.cli.serve.ServeSubcommand.subparser_init(...)` +/// +/// These are not part of `EngineArgs`, `AsyncEngineArgs`, `BaseFrontendArgs`, +/// or `FrontendArgs`. They live on the `serve` command itself and control +/// managed-engine / multi-process orchestration rather than the shared frontend +/// runtime config. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, PartialEq, Eq, Default, Args, Serialize, Deserialize)] +pub struct TopLevelUnsupportedArgs { + /// How many API server processes to run. Defaults to data_parallel_size if + /// not specified. + #[arg(long, hide = true)] + pub api_server_count: Option, + + /// Read CLI options from a config file. Must be a YAML with the following + /// options: https://docs.vllm.ai/en/latest/configuration/serve_args.html + #[arg(long)] + pub config: Option, + + /// Launch a gRPC server instead of the HTTP OpenAI-compatible server. + /// Requires: pip install vllm[grpc]. + #[arg(long, default_missing_value = "true", num_args = 0..=1)] + pub grpc: Option, +} + +/// Frontend-owned Python engine arguments that `vllm-rs` recognizes but does +/// not support yet. +/// +/// Source of truth in Python vLLM: +/// - `vllm.engine.arg_utils.EngineArgs.add_cli_args(...)` +/// - `vllm.engine.arg_utils.AsyncEngineArgs.add_cli_args(...)` +/// +/// These arguments are declared through the Python engine-args surface, but +/// they are still frontend-owned: the API server / AsyncLLM layer reads them +/// for tokenizer setup, request validation, routing, logging, and other +/// frontend behavior, so Rust must recognize them rather than treating them as +/// pure engine passthrough. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, PartialEq, Eq, Default, Args, Serialize, Deserialize)] +pub struct EngineUnsupportedArgs { + /// Name or path of the Hugging Face tokenizer to use. If unspecified, model + /// name or path will be used. + #[arg(long)] + pub tokenizer: Option, + + /// Name or path of the Hugging Face config to use. If unspecified, model + /// name or path will be used. + #[arg(long)] + pub hf_config_path: Option, + + /// Allowing API requests to read local images or videos from directories + /// specified by the server file system. This is a security risk. Should + /// only be enabled in trusted environments. + #[arg(long)] + pub allowed_local_media_path: Option, + + /// If set, only media URLs that belong to this domain can be used for + /// multi-modal inputs. + #[arg(long)] + pub allowed_media_domains: Option, + + /// The specific revision to use for the tokenizer on the Hugging Face Hub. + /// It can be a branch name, a tag name, or a commit id. If unspecified, + /// will use the default version. + #[arg(long)] + pub tokenizer_revision: Option, + + /// Skip initialization of tokenizer and detokenizer. Expects valid + /// `prompt_token_ids` and `None` for prompt from the input. The generated + /// output will contain token ids. + #[arg( + long, + visible_alias = "no-skip-tokenizer-init", + default_missing_value = "true", + num_args = 0..=1 + )] + pub skip_tokenizer_init: Option, + + /// If `True`, enables passing text embeddings as inputs via the + /// `prompt_embeds` key. + /// + /// WARNING: The vLLM engine may crash if incorrect shape of embeddings is + /// passed. Only enable this flag for trusted users! + #[arg( + long, + visible_alias = "no-enable-prompt-embeds", + default_missing_value = "true", + num_args = 0..=1 + )] + pub enable_prompt_embeds: Option, + + /// The token to use as HTTP bearer authorization for remote files. If + /// `True`, will use the token generated when running `hf auth login` + /// (stored in `~/.cache/huggingface/token`). + #[arg(long, default_missing_value = "true", num_args = 0..=1)] + pub hf_token: Option, + + /// If a dictionary, contains arguments to be forwarded to the Hugging Face + /// config. If a callable, it is called to update the HuggingFace config. + #[arg(long)] + pub hf_overrides: Option, + + /// The folder path to the generation config. Defaults to `"auto"`, the + /// generation config will be loaded from model path. If set to `"vllm"`, no + /// generation config is loaded, vLLM defaults will be used. If set to a + /// folder path, the generation config will be loaded from the specified + /// folder path. If `max_new_tokens` is specified in generation config, + /// then it sets a server-wide limit on the number of output tokens for + /// all requests. + #[arg(long)] + pub generation_config: Option, + + /// IOProcessor plugin name to load at model startup + #[arg(long)] + pub io_processor_plugin: Option, + + /// Path to a dynamically reasoning parser plugin that can be dynamically + /// loaded and registered. + #[arg(long)] + pub reasoning_parser_plugin: Option, + + /// Rank of the data parallel group. + #[arg(long, env = "VLLM_DP_RANK")] + pub data_parallel_rank: Option, + + /// Whether to use "hybrid" DP LB mode. Applies only to online serving + /// and when data_parallel_size > 0. Enables running an AsyncLLM + /// and API server on a "per-node" basis where vLLM load balances + /// between local data parallel ranks, but an external LB balances + /// between vLLM nodes/replicas. Set explicitly in conjunction with + /// --data-parallel-start-rank. + #[arg( + long, + visible_alias = "no-data-parallel-hybrid-lb", + default_missing_value = "true", + num_args = 0..=1 + )] + pub data_parallel_hybrid_lb: Option, + + /// Whether to use "external" DP LB mode. Applies only to online serving + /// and when data_parallel_size > 0. This is useful for a "one-pod-per-rank" + /// wide-EP setup in Kubernetes. Set implicitly when --data-parallel-rank + /// is provided explicitly to vllm serve. + #[arg( + long, + visible_alias = "no-data-parallel-external-lb", + default_missing_value = "true", + num_args = 0..=1 + )] + pub data_parallel_external_lb: Option, + + /// This feature is work in progress and no prefill optimization takes place + /// with this flag enabled currently. + #[arg( + long, + visible_alias = "no-kv-sharing-fast-prefill", + default_missing_value = "true", + num_args = 0..=1 + )] + pub kv_sharing_fast_prefill: Option, + + /// The maximum number of input items and options allowed per + /// prompt for each modality. + #[arg(long)] + pub limit_mm_per_prompt: Option, + + /// Additional args passed to process media inputs, keyed by modalities. + #[arg(long)] + pub media_io_kwargs: Option, + + /// Arguments to be forwarded to the model's processor for multi-modal data, + /// e.g., image processor. + #[arg(long)] + pub mm_processor_kwargs: Option, + + /// The size (in GiB) of the multi-modal processor cache. + #[arg(long)] + pub mm_processor_cache_gb: Option, + + /// Type of cache to use for the multi-modal preprocessor/mapper. + #[arg(long)] + pub mm_processor_cache_type: Option, + + /// Dictionary mapping specific modalities to LoRA model paths. + #[arg(long)] + pub default_mm_loras: Option, + + /// Target URL to which OpenTelemetry traces will be sent. + #[arg(long)] + pub otlp_traces_endpoint: Option, + + /// It makes sense to set this only if `--otlp-traces-endpoint` is set. + #[arg(long)] + pub collect_detailed_traces: Option, + + /// The interval (or buffer size) for streaming in terms of token length. + #[arg(long)] + pub stream_interval: Option, + + /// Structured outputs configuration. + #[arg(long)] + pub structured_outputs_config: Option, + + /// Log aggregate rather than per-engine statistics when using data + /// parallelism. + #[arg(long, default_missing_value = "true", num_args = 0..=1)] + pub aggregate_engine_logging: Option, +} + +/// Frontend-owned Python OpenAI server arguments that `vllm-rs` recognizes but +/// does not support yet. +/// +/// Source of truth in Python vLLM: +/// - `vllm.entrypoints.openai.cli_args.BaseFrontendArgs` +/// - `vllm.entrypoints.openai.cli_args.FrontendArgs` +/// +/// These are not engine args. They belong to the Python OpenAI-compatible +/// frontend / API-server layer itself, for example chat-template configuration, +/// tool/frontend behavior, TLS / CORS / HTTP server settings, and other +/// northbound server knobs. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, PartialEq, Eq, Default, Args, Serialize, Deserialize)] +pub struct ServerUnsupportedArgs { + /// LoRA modules configurations in either 'name=path' format or JSON format + /// or JSON list format. Example (old format): `'name=path'` Example (new + /// format): `{"name": "name", "path": "lora_path", + /// "base_model_name": "id"}` + #[arg(long)] + pub lora_modules: Option, + + /// Whether to trust the chat template provided in the request. If False, + /// the server will always use the chat template specified by + /// `--chat-template` or the ones from tokenizer. + #[arg( + long, + visible_alias = "no-trust-request-chat-template", + default_missing_value = "true", + num_args = 0..=1 + )] + pub trust_request_chat_template: Option, + + /// The role name to return if `request.add_generation_prompt=true`. + #[arg(long)] + pub response_role: Option, + + /// When `--max-logprobs` is specified, represents single tokens as + /// strings of the form 'token_id:{token_id}' so that tokens that are not + /// JSON-encodable can be identified. + #[arg( + long, + visible_alias = "no-return-tokens-as-token-ids", + default_missing_value = "true", + num_args = 0..=1 + )] + pub return_tokens_as_token_ids: Option, + + /// Enable auto tool choice for supported models. Use `--tool-call-parser` + /// to specify which parser to use. + #[arg( + long, + visible_alias = "no-enable-auto-tool-choice", + default_missing_value = "true", + num_args = 0..=1, + hide = true + )] + pub enable_auto_tool_choice: Option, + + /// If specified, exclude tool definitions in prompts when + /// tool_choice='none'. + #[arg( + long, + visible_alias = "no-exclude-tools-when-tool-choice-none", + default_missing_value = "true", + num_args = 0..=1 + )] + pub exclude_tools_when_tool_choice_none: Option, + + /// Special the tool parser plugin write to parse the model-generated tool + /// into OpenAI API format, the name register in this plugin can be used in + /// `--tool-call-parser`. + #[arg(long)] + pub tool_parser_plugin: Option, + + /// Comma-separated list of host:port pairs (IPv4, IPv6, or hostname). + /// Examples: 127.0.0.1:8000, [::1]:8000, localhost:1234. Or `demo` for + /// built-in demo tools (browser and Python code interpreter). WARNING: + /// The `demo` Python tool executes model-generated code in Docker without + /// network isolation by default. See the security guide for more + /// information. + #[arg(long)] + pub tool_server: Option, + + /// Path to logging config JSON file for both vllm and uvicorn + #[arg(long, /* env = "VLLM_LOGGING_CONFIG_PATH" */)] + pub log_config_file: Option, + + /// Max number of prompt characters or prompt ID numbers being printed in + /// log. The default of None means unlimited. + #[arg(long)] + pub max_log_len: Option, + + /// If set to True, enable tracking server_load_metrics in the app state. + #[arg( + long, + visible_alias = "no-enable-server-load-tracking", + default_missing_value = "true", + num_args = 0..=1 + )] + pub enable_server_load_tracking: Option, + + /// If set to True, including usage on every request. + #[arg( + long, + visible_alias = "no-enable-force-include-usage", + default_missing_value = "true", + num_args = 0..=1 + )] + pub enable_force_include_usage: Option, + + /// Enable the `/tokenizer_info` endpoint. May expose chat + /// templates and other tokenizer configuration. + #[arg( + long, + visible_alias = "no-enable-tokenizer-info-endpoint", + default_missing_value = "true", + num_args = 0..=1 + )] + pub enable_tokenizer_info_endpoint: Option, + + /// If set to True, log model outputs (generations). + /// Requires `--enable-log-requests`. As with `--enable-log-requests`, + /// information is only logged at INFO level at maximum. + #[arg( + long, + visible_alias = "no-enable-log-outputs", + default_missing_value = "true", + num_args = 0..=1 + )] + pub enable_log_outputs: Option, + + /// If set to False, output deltas will not be logged. Relevant only if + /// --enable-log-outputs is set. + #[arg( + long, + visible_alias = "no-enable-log-deltas", + default_missing_value = "true", + num_args = 0..=1 + )] + pub enable_log_deltas: Option, + + /// If set to True, log the stack trace of error responses + #[arg( + long, + // env = "VLLM_SERVER_DEV_MODE", + visible_alias = "no-log-error-stack", + default_missing_value = "true", + num_args = 0..=1 + )] + pub log_error_stack: Option, + + /// If set to True, only enable the Tokens In<>Out endpoint. + /// This is intended for use in a Disaggregated Everything setup. + #[arg( + long, + visible_alias = "no-tokens-only", + default_missing_value = "true", + num_args = 0..=1 + )] + pub tokens_only: Option, + + /// Log level for uvicorn. + #[arg(long)] + pub uvicorn_log_level: Option, + + /// Disable uvicorn access log. + #[arg( + long, + visible_alias = "no-disable-uvicorn-access-log", + default_missing_value = "true", + num_args = 0..=1 + )] + pub disable_uvicorn_access_log: Option, + + /// Comma-separated list of endpoint paths to exclude from uvicorn access + /// logs. This is useful to reduce log noise from high-frequency endpoints + /// like health checks. Example: "/health,/metrics,/ping". + /// When set, access logs for requests to these paths will be suppressed + /// while keeping logs for other endpoints. + #[arg(long)] + pub disable_access_log_for_endpoints: Option, + + /// Refresh SSL Context when SSL certificate files change + #[arg( + long, + visible_alias = "no-enable-ssl-refresh", + default_missing_value = "true", + num_args = 0..=1 + )] + pub enable_ssl_refresh: Option, + + /// FastAPI root_path when app is behind a path based routing proxy. + #[arg(long)] + pub root_path: Option, + + /// Additional ASGI middleware to apply to the app. We accept multiple + /// --middleware arguments. The value should be an import path. If a + /// function is provided, vLLM will add it to the server using + /// `@app.middleware('http')`. If a class is provided, vLLM will + /// add it to the server using `app.add_middleware()`. + #[arg(long)] + pub middleware: Option, + + /// Disable FastAPI's OpenAPI schema, Swagger UI, and ReDoc endpoint. + #[arg( + long, + visible_alias = "no-disable-fastapi-docs", + default_missing_value = "true", + num_args = 0..=1 + )] + pub disable_fastapi_docs: Option, + + /// Maximum size (bytes) of an incomplete HTTP event (header or body) for + /// h11 parser. Helps mitigate header abuse. Default: 4194304 (4 MB). + #[arg(long)] + pub h11_max_incomplete_event_size: Option, + + /// Maximum number of HTTP headers allowed in a request for h11 parser. + /// Helps mitigate header abuse. Default: 256. + #[arg(long)] + pub h11_max_header_count: Option, + + /// Enable offline FastAPI documentation for air-gapped environments. + /// Uses vendored static assets bundled with vLLM. + #[arg( + long, + visible_alias = "no-enable-offline-docs", + default_missing_value = "true", + num_args = 0..=1 + )] + pub enable_offline_docs: Option, +} diff --git a/asian_ai/performance/vLLM/rust/src/cmd/src/logging.rs b/asian_ai/performance/vLLM/rust/src/cmd/src/logging.rs new file mode 100644 index 0000000000000000000000000000000000000000..1338ba67f745fd00caa6c3c009d7339c3680c12c --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/cmd/src/logging.rs @@ -0,0 +1,328 @@ +use std::{env, fmt, process}; + +use time::UtcOffset; +use time::macros::format_description; +use tracing::level_filters::LevelFilter; +use tracing::{Event, Level, Subscriber}; +use tracing_subscriber::Layer as _; +use tracing_subscriber::filter::Targets; +use tracing_subscriber::fmt::format::{FormatEvent, FormatFields, Writer}; +use tracing_subscriber::fmt::time::FormatTime; +use tracing_subscriber::fmt::{FmtContext, FormattedFields}; +use tracing_subscriber::layer::SubscriberExt as _; +use tracing_subscriber::registry::LookupSpan; +use tracing_subscriber::util::SubscriberInitExt as _; + +const CYAN: &str = "\x1b[0;36m"; +const GREY: &str = "\x1b[90m"; +const GREEN: &str = "\x1b[32m"; +const YELLOW: &str = "\x1b[33m"; +const RED: &str = "\x1b[31m"; +const WHITE: &str = "\x1b[37m"; +const RESET: &str = "\x1b[0m"; +const VLLM_TIME_FORMAT: &[time::format_description::FormatItem<'static>] = + format_description!("[month]-[day] [hour]:[minute]:[second]"); + +const PROCESS_LABEL: &str = "RustFrontend"; + +/// Install the process-wide vLLM-style tracing subscriber for the CLI binary. +pub(crate) fn init_tracing() { + let filter = build_targets_filter( + env::var("VLLM_LOGGING_LEVEL").ok().as_deref(), + env::var("RUST_LOG").ok().as_deref(), + ); + let formatter = VllmEventFormatter::new(); + + let _ = tracing_subscriber::registry() + .with(tracing_subscriber::fmt::layer().event_format(formatter).with_filter(filter)) + .try_init(); +} + +/// Build the CLI log filter by merging the vLLM-style default level with +/// Rust-style target overrides. +/// +/// Precedence: +/// - Start from `VLLM_LOGGING_LEVEL` as the default level for all targets. +/// - If `RUST_LOG` contains a global default level such as `warn`, it overrides +/// `VLLM_LOGGING_LEVEL`. +/// - Any explicit target directives in `RUST_LOG`, such as `hyper=info`, override whichever default +/// level is active for those targets only. +fn build_targets_filter(vllm_logging_level: Option<&str>, rust_log: Option<&str>) -> Targets { + let mut filter = + Targets::new().with_default(map_python_log_level(vllm_logging_level.unwrap_or("INFO"))); + + if let Some(rust_log) = rust_log + && !rust_log.is_empty() + { + let rust_log_targets: Targets = rust_log.parse().expect("failed to parse `RUST_LOG`"); + if let Some(default_level) = rust_log_targets.default_level() { + filter = filter.with_default(default_level); + } + filter = filter.with_targets(rust_log_targets); + } + + filter +} + +#[derive(Debug, Clone, Copy)] +struct VllmLocalTimer { + local_offset: UtcOffset, +} + +impl Default for VllmLocalTimer { + fn default() -> Self { + let local_offset = UtcOffset::current_local_offset().unwrap_or(UtcOffset::UTC); + Self { local_offset } + } +} + +impl FormatTime for VllmLocalTimer { + fn format_time(&self, w: &mut Writer<'_>) -> fmt::Result { + let now = time::OffsetDateTime::now_utc().to_offset(self.local_offset); + let formatted = now.format(VLLM_TIME_FORMAT).map_err(|_| fmt::Error)?; + w.write_str(&formatted) + } +} + +#[derive(Debug, Clone)] +struct VllmEventFormatter { + prefix: String, + timer: VllmLocalTimer, +} + +impl VllmEventFormatter { + fn new() -> Self { + Self { + prefix: format!("({} pid={})", PROCESS_LABEL, process::id()), + timer: VllmLocalTimer::default(), + } + } + + fn write_process_prefix(&self, writer: &mut Writer<'_>, ansi: bool) -> fmt::Result { + write_colored(writer, ansi, Some(CYAN), &self.prefix)?; + writer.write_char(' ') + } + + fn write_level(&self, writer: &mut Writer<'_>, level: &Level, ansi: bool) -> fmt::Result { + let (text, color) = match *level { + Level::TRACE => ("TRACE", WHITE), + Level::DEBUG => ("DEBUG", WHITE), + Level::INFO => ("INFO", GREEN), + Level::WARN => ("WARNING", YELLOW), + Level::ERROR => ("ERROR", RED), + }; + write_colored(writer, ansi, Some(color), text) + } + + fn write_timestamp(&self, writer: &mut Writer<'_>, ansi: bool) -> fmt::Result { + if ansi { + writer.write_str(GREY)?; + } + if self.timer.format_time(writer).is_err() { + writer.write_str("")?; + } + if ansi { + writer.write_str(RESET)?; + } + Ok(()) + } + + fn write_location( + &self, + writer: &mut Writer<'_>, + file: Option<&str>, + line: Option, + full_path: bool, + ansi: bool, + ) -> fmt::Result { + let Some(file) = file else { + return Ok(()); + }; + let file = if full_path { + file + } else { + shorten_file_path(file) + }; + if ansi { + writer.write_str(GREY)?; + } + match line { + Some(line) => write!(writer, "[{file}:{line}]")?, + None => write!(writer, "[{file}]")?, + } + if ansi { + writer.write_str(RESET)?; + } + Ok(()) + } + + fn write_scope(&self, ctx: &FmtContext<'_, S, N>, writer: &mut Writer<'_>) -> fmt::Result + where + S: Subscriber + for<'lookup> LookupSpan<'lookup>, + N: for<'writer> FormatFields<'writer> + 'static, + { + let Some(scope) = ctx.event_scope() else { + return Ok(()); + }; + + let mut seen = false; + for span in scope.from_root() { + if seen { + writer.write_str(":")?; + } + seen = true; + writer.write_str(span.metadata().name())?; + + let ext = span.extensions(); + if let Some(fields) = ext.get::>() + && !fields.is_empty() + { + write!(writer, "{{{fields}}}")?; + } + } + + if seen { + writer.write_str(": ")?; + } + + Ok(()) + } +} + +impl FormatEvent for VllmEventFormatter +where + S: Subscriber + for<'lookup> LookupSpan<'lookup>, + N: for<'writer> FormatFields<'writer> + 'static, +{ + fn format_event( + &self, + ctx: &FmtContext<'_, S, N>, + mut writer: Writer<'_>, + event: &Event<'_>, + ) -> fmt::Result { + let meta = event.metadata(); + let ansi = writer.has_ansi_escapes(); + + self.write_process_prefix(&mut writer, ansi)?; + self.write_level(&mut writer, meta.level(), ansi)?; + writer.write_char(' ')?; + self.write_timestamp(&mut writer, ansi)?; + writer.write_char(' ')?; + // Use the full file path only when DEBUG (or more verbose) is enabled anywhere, + // independent of the level of this particular event. Filenames alone are often + // ambiguous, but full paths are too noisy for normal INFO-level operation. + let full_path = LevelFilter::current() >= LevelFilter::DEBUG; + self.write_location(&mut writer, meta.file(), meta.line(), full_path, ansi)?; + writer.write_char(' ')?; + self.write_scope(ctx, &mut writer)?; + ctx.format_fields(writer.by_ref(), event)?; + writer.write_char('\n') + } +} + +/// Shorten a source file path for log output while preserving enough context +/// for common Rust entrypoint and module filenames. +/// +/// - For `mod.rs`, keep the parent directory as `parent/mod.rs`. +/// - For `src/lib.rs` and `src/main.rs`, keep one additional component as `crate/src/lib.rs` or +/// `crate/src/main.rs` when available. +/// - Other files are displayed as just the basename. +fn shorten_file_path(file: &str) -> &str { + let mut parts = file.rsplit('/'); + let name = parts.next().unwrap_or(file); + let parent = parts.next(); + let grandparent = parts.next(); + + let Some(parent) = parent else { + return file; + }; + + if name == "mod.rs" { + return &file[file.len() - parent.len() - 1 - name.len()..]; + } + + if !matches!(name, "lib.rs" | "main.rs") || parent != "src" { + return name; + } + let Some(grandparent) = grandparent else { + return file; + }; + + &file[file.len() - grandparent.len() - 1 - parent.len() - 1 - name.len()..] +} + +fn write_colored( + writer: &mut Writer<'_>, + ansi: bool, + color: Option<&str>, + text: &str, +) -> fmt::Result { + if ansi { + if let Some(color) = color { + writer.write_str(color)?; + } + writer.write_str(text)?; + if color.is_some() { + writer.write_str(RESET)?; + } + return Ok(()); + } + + writer.write_str(text) +} + +/// Map a Python logging level name to the corresponding Rust tracing level. +fn map_python_log_level(level: &str) -> LevelFilter { + match level.to_ascii_uppercase().as_str() { + "CRITICAL" | "FATAL" => LevelFilter::ERROR, + "ERROR" => LevelFilter::ERROR, + "WARNING" | "WARN" => LevelFilter::WARN, + "INFO" => LevelFilter::INFO, + "DEBUG" => LevelFilter::DEBUG, + "NOTSET" => LevelFilter::TRACE, + _ => LevelFilter::INFO, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn rust_log_target_overrides_are_merged_with_vllm_default_level() { + let filter = build_targets_filter(Some("DEBUG"), Some("hyper=warn,tower=error")); + + assert_eq!(filter.to_string(), "tower=error,hyper=warn,debug"); + } + + #[test] + fn rust_log_default_level_overrides_vllm_default_level() { + let filter = build_targets_filter(Some("DEBUG"), Some("warn,hyper=info")); + + assert_eq!(filter.to_string(), "hyper=info,warn"); + } + + #[test] + fn invalid_vllm_level_falls_back_to_info() { + let filter = build_targets_filter(Some("bogus"), None); + + assert_eq!(filter.to_string(), "info"); + } + + #[test] + fn location_path_uses_filename_for_non_ambiguous_files() { + assert_eq!(shorten_file_path("src/cmd/src/logging.rs"), "logging.rs"); + assert_eq!(shorten_file_path("src/chat/lib.rs"), "lib.rs"); + assert_eq!(shorten_file_path("src/chat/main.rs"), "main.rs"); + assert_eq!(shorten_file_path("src/chat/src/xmod.rs"), "xmod.rs"); + } + + #[test] + fn location_path_keeps_more_context_for_common_entrypoint_filenames() { + assert_eq!(shorten_file_path("src/lib.rs"), "src/lib.rs"); + assert_eq!(shorten_file_path("src/chat/src/lib.rs"), "chat/src/lib.rs"); + assert_eq!(shorten_file_path("src/cmd/src/main.rs"), "cmd/src/main.rs"); + assert_eq!(shorten_file_path("mod.rs"), "mod.rs"); + assert_eq!(shorten_file_path("src/chat/src/tool/mod.rs"), "tool/mod.rs"); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/cmd/src/main.rs b/asian_ai/performance/vLLM/rust/src/cmd/src/main.rs new file mode 100644 index 0000000000000000000000000000000000000000..d24c10fead1c98242315b8f5ac8f5c1ce0401213 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/cmd/src/main.rs @@ -0,0 +1,192 @@ +mod cli; +mod logging; + +use std::env; +use std::process::ExitStatus; + +use anyhow::{Context, Result, anyhow, bail}; +use tokio_util::sync::CancellationToken; +use tracing::{info, warn}; +use vllm_managed_engine::ManagedEngineHandle; + +use crate::cli::{Cli, Command}; + +#[global_allocator] +static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc; + +const TOKIO_WORKER_THREADS_ENV: &str = "TOKIO_WORKER_THREADS"; +const DEFAULT_MAX_TOKIO_WORKER_THREADS: usize = 32; + +/// Cap the default number of Tokio worker threads if the user did not +/// explicitly set `TOKIO_WORKER_THREADS` to avoid spawning too many threads on +/// machines with a large number of CPUs, which may lead to excessive context +/// switching and degraded performance. +fn tokio_worker_threads() -> Option { + if env::var_os(TOKIO_WORKER_THREADS_ENV).is_some() { + return None; + } + + std::thread::available_parallelism() + .map(|parallelism| { + let available = parallelism.get(); + let worker_threads = available.min(DEFAULT_MAX_TOKIO_WORKER_THREADS); + if worker_threads < available { + info!( + available_parallelism = available, + capped_worker_threads = worker_threads, + "capping tokio worker threads, set {TOKIO_WORKER_THREADS_ENV} to override" + ); + } + worker_threads + }) + .ok() +} + +/// Reason that caused a managed `serve` session to stop. +#[derive(Debug)] +enum ShutdownReason { + Signal, + Server(anyhow::Error), + EngineExited(ExitStatus), +} + +/// Cancellation token tripped by Ctrl-C or SIGTERM. +fn shutdown_signal() -> CancellationToken { + let token = CancellationToken::new(); + let shutdown = token.clone(); + + tokio::spawn(async move { + let ctrl_c = async { + tokio::signal::ctrl_c().await.expect("failed to install Ctrl-C signal handler"); + }; + + let sigterm = async { + tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) + .expect("failed to install SIGTERM signal handler") + .recv() + .await; + }; + + tokio::select! { + _ = ctrl_c => info!("received shutdown signal (Ctrl-C), shutting down..."), + _ = sigterm => info!("received shutdown signal (SIGTERM), shutting down..."), + } + + shutdown.cancel(); + }); + + token +} + +fn main() -> Result<()> { + logging::init_tracing(); + let cli = Cli::parse(); + + let mut runtime = tokio::runtime::Builder::new_multi_thread(); + runtime.enable_all(); + if let Some(worker_threads) = tokio_worker_threads() { + runtime.worker_threads(worker_threads); + } + + runtime + .build() + .context("failed to build Tokio runtime")? + .block_on(async_main(cli)) +} + +async fn async_main(cli: Cli) -> Result<()> { + match cli.command { + Command::Frontend(args) => vllm_server::serve(args.into_config(), shutdown_signal()).await, + Command::Serve(args) => { + let handshake_port = args.managed_engine.resolve_handshake_port()?; + + if args.managed_engine.data_parallel_size_local == Some(0) { + if args.headless { + bail!("cannot combine `--headless` with `--data-parallel-size-local 0`"); + } + + let handshake_address = args.managed_engine.handshake_address(handshake_port); + info!( + %handshake_address, + engine_count = args.managed_engine.data_parallel_size, + "running Rust frontend without a managed local Python engine" + ); + let config = args.to_frontend_config(handshake_address); + return vllm_server::serve(config, shutdown_signal()).await; + } + + let shutdown_timeout = args.runtime.shutdown_timeout(); + let engine_config = args.to_managed_engine_config(handshake_port); + let handshake_address = engine_config.handshake_address(); + + let engine = ManagedEngineHandle::spawn(engine_config) + .await + .context("failed to start managed Python headless engine")?; + + let shutdown = shutdown_signal(); + + let mut serve_task = if args.headless { + info!("running managed Python headless engine without Rust frontend"); + let shutdown = shutdown.clone(); + tokio::spawn(async move { + shutdown.cancelled().await; + Ok(()) + }) + } else { + let config = args.to_frontend_config(handshake_address); + let shutdown = shutdown.clone(); + tokio::spawn(async move { + let result = vllm_server::serve(config, shutdown).await; + if result.is_ok() { + info!("OpenAI server shut down gracefully"); + } + result + }) + }; + + let shutdown_reason = tokio::select! { + biased; + + // Received shutdown signal via Ctrl-C or SIGTERM. + _ = shutdown.cancelled() => ShutdownReason::Signal, + + // Engine process exited unexpectedly. + status = engine.wait_for_exit() => { + warn!(%status, "managed Python headless engine exited, shutting down..."); + ShutdownReason::EngineExited(status) + } + + // Serve task exited unexpectedly. + serve_result = &mut serve_task => { + let serve_result = serve_result.context("serve task join failed")?; + match serve_result { + Ok(()) => ShutdownReason::Server(anyhow!("OpenAI server shut down unexpectedly without error")), + Err(error) => ShutdownReason::Server(error), + } + } + }; + // Regardless of the shutdown reason, broadcast shutdown signal here to ensure + // that all serving tasks are notified. + shutdown.cancel(); + + // Shutdown begins. Terminate the managed engine first. + engine.shutdown(shutdown_timeout).await?; + info!("managed engine shut down gracefully"); + // Wait for the API server to shut down gracefully by draining in-flight + // requests. + if !matches!(shutdown_reason, ShutdownReason::Server(_)) { + serve_task.await.context("serve task join failed")??; + } + + match shutdown_reason { + ShutdownReason::Signal => Ok(()), + ShutdownReason::Server(error) => { + Err(error.context("OpenAI server shut down unexpectedly")) + } + ShutdownReason::EngineExited(status) => Err(anyhow!( + "managed Python headless engine exited unexpectedly with status {status}" + )), + } + } + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/Cargo.toml b/asian_ai/performance/vLLM/rust/src/engine-core-client/Cargo.toml new file mode 100644 index 0000000000000000000000000000000000000000..61f2e62b16775f05439f9fc9b82a5872bcaa8432 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/Cargo.toml @@ -0,0 +1,49 @@ +[package] +name = "vllm-engine-core-client" +version.workspace = true +edition.workspace = true +license.workspace = true + +[features] +test-util = ["dep:tempfile"] + +[dependencies] +arc-swap.workspace = true +bytemuck.workspace = true +byteorder.workspace = true +bytes.workspace = true +easy-ext.workspace = true +enum-as-inner.workspace = true +futures.workspace = true +half.workspace = true +hex.workspace = true +itertools.workspace = true +parking_lot.workspace = true +rmp-serde.workspace = true +rmpv.workspace = true +serde.workspace = true +serde_default.workspace = true +serde_json.workspace = true +serde_repr.workspace = true +serde_tuple.workspace = true +serde_with.workspace = true +task-local.workspace = true +tempfile = { workspace = true, optional = true } +thiserror.workspace = true +thiserror-ext.workspace = true +tokio.workspace = true +tokio-util.workspace = true +tracing.workspace = true +vllm-metrics.workspace = true +zeromq.workspace = true + +[dev-dependencies] +anyhow.workspace = true +clap.workspace = true +expect-test.workspace = true +hex.workspace = true +tempfile.workspace = true +tracing-subscriber.workspace = true + +[lints] +workspace = true diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/README.md b/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/README.md new file mode 100644 index 0000000000000000000000000000000000000000..aa1fc1ca675e13e5c47866ffd14161ff552d82ba --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/README.md @@ -0,0 +1,53 @@ +# Engine-Core Smoke Tests + +Start headless `vllm`: + +```bash +source ../vllm/.venv/bin/activate +HF_HUB_OFFLINE=1 \ +VLLM_LOGGING_LEVEL=DEBUG \ +VLLM_CPU_KVCACHE_SPACE=2 \ +VLLM_HOST_IP=127.0.0.1 \ +VLLM_LOOPBACK_IP=127.0.0.1 \ +python3 -m vllm.entrypoints.cli.main serve Qwen/Qwen3-0.6B \ + --headless \ + --enable-sleep-mode \ + --data-parallel-address 127.0.0.1 \ + --data-parallel-rpc-port 62100 \ + --data-parallel-size-local 1 \ + --max-model-len 512 \ + --dtype float16 +``` + +Run the Rust smoke test through the `vllm-engine-core-client` utility interface: + +```bash +cargo run -p vllm-engine-core-client --example external_engine_utility_call -- \ + --handshake-address tcp://127.0.0.1:62100 \ + --host 127.0.0.1 +``` + +If your current engine setup does not support sleep mode, skip the `sleep` / `wake_up` part of the +smoke: + +```bash +cargo run -p vllm-engine-core-client --example external_engine_utility_call -- \ + --handshake-address tcp://127.0.0.1:62100 \ + --host 127.0.0.1 \ + --skip-sleep-wake +``` + +Run the Rust smoke test for sample logprobs decoding through the raw engine-core request path: + +```bash +cargo run -p vllm-engine-core-client --example external_engine_logprobs -- \ + --handshake-address tcp://127.0.0.1:62100 \ + --host 127.0.0.1 +``` + +This smoke requests a small generated-token `logprobs` payload plus prompt logprobs over a much +longer prompt, so it exercises both the inline and aux-frame decode paths against a real engine. +The Rust client decodes those payloads into semantic per-position records rather than exposing the +raw ndarray/tensor wire shape. + +IMPORTANT: You must restart `vllm` each time you run the smoke test, as the vLLM engine cannot manage frontend closures and subsequent reconnects. In other words, do not reuse existing `vllm` instances, if any. diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/external_engine_logprobs.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/external_engine_logprobs.rs new file mode 100644 index 0000000000000000000000000000000000000000..7c9da201062c69c2319b4f9b60501a32c300d3f7 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/external_engine_logprobs.rs @@ -0,0 +1,190 @@ +use std::time::Duration; + +use anyhow::{Context, Result, bail}; +use clap::Parser; +use futures::StreamExt as _; +use tokio::time::timeout; +use tracing_subscriber::EnvFilter; +use vllm_engine_core_client::protocol::output::EngineCoreFinishReason; +use vllm_engine_core_client::protocol::request::EngineCoreRequest; +use vllm_engine_core_client::protocol::sampling::EngineCoreSamplingParams; +use vllm_engine_core_client::{ + EngineCoreClient, EngineCoreClientConfig, EngineCoreStreamOutput, TransportMode, +}; + +const BASE_PROMPT_TOKEN_IDS: &[u32] = &[20841, 448, 6896, 25, 23811]; + +#[derive(Debug, Parser)] +#[command(about = "Smoke-test engine-core sample logprobs against an external vLLM engine.")] +struct Args { + #[arg(long)] + handshake_address: String, + #[arg(long, default_value_t = 1)] + engine_count: usize, + #[arg(long, default_value = "Qwen/Qwen3-0.6B")] + model: String, + #[arg(long, default_value = "127.0.0.1")] + host: String, + #[arg(long, default_value_t = 0)] + client_index: u32, + #[arg(long, default_value_t = 30)] + ready_timeout_secs: u64, + #[arg(long, default_value_t = 120)] + output_timeout_secs: u64, + #[arg(long, default_value_t = 1)] + max_tokens: u32, + #[arg(long, default_value_t = 2)] + logprobs: i32, + #[arg(long, default_value_t = 1)] + prompt_logprobs: i32, + #[arg(long, default_value_t = 96)] + prompt_repeats: usize, +} + +fn init_tracing() { + let filter = EnvFilter::try_from_default_env() + .unwrap_or_else(|_| EnvFilter::new("vllm_engine_core_client=debug")); + let _ = tracing_subscriber::fmt().with_env_filter(filter).try_init(); +} + +fn unique_request_id() -> String { + let nanos = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("system clock should be after unix epoch") + .as_nanos(); + format!("rust-engine-core-logprobs-{nanos}") +} + +fn build_prompt_token_ids(prompt_repeats: usize) -> Vec { + let repeats = prompt_repeats.max(1); + BASE_PROMPT_TOKEN_IDS.repeat(repeats) +} + +fn build_request( + request_id: String, + prompt_token_ids: Vec, + max_tokens: u32, + logprobs: i32, + prompt_logprobs: i32, + client_index: u32, +) -> EngineCoreRequest { + EngineCoreRequest { + request_id, + prompt_token_ids: Some(prompt_token_ids), + sampling_params: Some(EngineCoreSamplingParams { + max_tokens, + logprobs: Some(logprobs), + prompt_logprobs: Some(prompt_logprobs), + ..EngineCoreSamplingParams::for_test() + }), + arrival_time: 0.0, + client_index, + ..EngineCoreRequest::default() + } +} + +async fn wait_for_final_output( + mut stream: vllm_engine_core_client::EngineCoreOutputStream, +) -> Result { + while let Some(output) = stream.next().await { + let output = output.context("failed to receive engine-core output")?; + if output.finished() { + return Ok(output); + } + } + bail!("request stream ended without a final output") +} + +#[tokio::main(flavor = "multi_thread")] +async fn main() -> Result<()> { + init_tracing(); + let args = Args::parse(); + let ready_timeout = Duration::from_secs(args.ready_timeout_secs); + let output_timeout = Duration::from_secs(args.output_timeout_secs); + let request_id = unique_request_id(); + let prompt_token_ids = build_prompt_token_ids(args.prompt_repeats); + let client = EngineCoreClient::connect(EngineCoreClientConfig { + transport_mode: TransportMode::HandshakeOwner { + handshake_address: args.handshake_address.clone(), + advertised_host: args.host.clone(), + engine_count: args.engine_count, + ready_timeout, + local_input_address: None, + local_output_address: None, + }, + coordinator_mode: None, + model_name: args.model.clone(), + client_index: args.client_index, + }) + .await + .context("failed to connect to external vLLM engine")?; + + println!("model={}", args.model); + println!("handshake_address={}", args.handshake_address); + println!("engine_count={}", args.engine_count); + println!("input_address={}", client.input_address()); + println!("output_address={}", client.output_address()); + println!("engine_identities={:x?}", client.engine_identities()); + + let request = build_request( + request_id.clone(), + prompt_token_ids.clone(), + args.max_tokens, + args.logprobs, + args.prompt_logprobs, + args.client_index, + ); + println!("request_id={request_id}"); + println!("prompt_len={}", prompt_token_ids.len()); + println!("base_prompt_len={}", BASE_PROMPT_TOKEN_IDS.len()); + println!("prompt_repeats={}", args.prompt_repeats); + println!("requested_logprobs={}", args.logprobs); + println!("requested_prompt_logprobs={}", args.prompt_logprobs); + + let stream = client.call(request).await.context("failed to submit engine-core request")?; + let output = timeout(output_timeout, wait_for_final_output(stream)) + .await + .context("timed out waiting for final output")??; + + let finish_reason = output.finish_reason; + let token_ids = output.new_token_ids.clone(); + let logprobs = output + .new_logprobs + .as_ref() + .and_then(|value| value.as_direct()) + .context("engine output did not include decoded sample logprobs")?; + let prompt_logprobs = output + .new_prompt_logprobs_tensors + .as_ref() + .and_then(|value| value.as_direct()) + .context("engine output did not include decoded prompt logprobs")?; + + println!("token_ids={token_ids:?}"); + println!("finish_reason={finish_reason:?}"); + println!("new_logprobs={logprobs:#?}"); + println!("new_prompt_logprobs_tensors={prompt_logprobs:#?}"); + + client.shutdown().await.context("failed to shut down engine-core client")?; + + if finish_reason != Some(EngineCoreFinishReason::Length) { + bail!("unexpected finish_reason: expected Length, got {finish_reason:?}"); + } + if token_ids.is_empty() { + bail!("engine returned no generated token ids"); + } + if logprobs.is_empty() { + bail!("decoded logprobs payload is unexpectedly empty"); + } + if prompt_logprobs.is_empty() { + bail!("decoded prompt logprobs payload is unexpectedly empty"); + } + if prompt_logprobs.len() + 1 < prompt_token_ids.len() { + bail!( + "prompt logprobs rows look too short: prompt_len={}, rows={}", + prompt_token_ids.len(), + prompt_logprobs.len() + ); + } + + Ok(()) +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/external_engine_utility_call.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/external_engine_utility_call.rs new file mode 100644 index 0000000000000000000000000000000000000000..fd3f62933ddcafd2d0ecfdf01649388c1cdaab22 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/external_engine_utility_call.rs @@ -0,0 +1,144 @@ +use std::time::Duration; + +use anyhow::{Context, Result, bail}; +use clap::Parser; +use tracing_subscriber::EnvFilter; +use vllm_engine_core_client::protocol::utility::PauseMode; +use vllm_engine_core_client::{EngineCoreClient, EngineCoreClientConfig, TransportMode}; + +#[derive(Debug, Parser)] +#[command(about = "Smoke-test EngineCoreClient utility calls against an external vLLM engine.")] +struct Args { + #[arg(long)] + handshake_address: String, + #[arg(long, default_value_t = 1)] + engine_count: usize, + #[arg(long, default_value = "Qwen/Qwen3-0.6B")] + model: String, + #[arg(long, default_value = "127.0.0.1")] + host: String, + #[arg(long, default_value_t = 0)] + client_index: u32, + #[arg(long, default_value_t = 30)] + ready_timeout_secs: u64, + #[arg( + long, + default_value_t = false, + help = "Expected initial result of is_sleeping() before running the smoke steps." + )] + expected_is_sleeping: bool, + #[arg(long, default_value_t = false)] + reset_running_requests: bool, + #[arg(long, default_value_t = false)] + reset_external: bool, + #[arg(long, default_value_t = 1)] + sleep_level: u32, + #[arg(long, default_value_t = PauseMode::Abort)] + sleep_mode: PauseMode, + #[arg( + long, + default_value_t = false, + help = "Skip sleep/wake_up calls when the engine was not started with sleep-mode support." + )] + skip_sleep_wake: bool, +} + +fn init_tracing() { + let filter = EnvFilter::try_from_default_env() + .unwrap_or_else(|_| EnvFilter::new("vllm_engine_core_client=debug")); + let _ = tracing_subscriber::fmt().with_env_filter(filter).try_init(); +} + +#[tokio::main(flavor = "multi_thread")] +async fn main() -> Result<()> { + init_tracing(); + let args = Args::parse(); + let client = EngineCoreClient::connect(EngineCoreClientConfig { + transport_mode: TransportMode::HandshakeOwner { + handshake_address: args.handshake_address.clone(), + advertised_host: args.host.clone(), + engine_count: args.engine_count, + ready_timeout: Duration::from_secs(args.ready_timeout_secs), + local_input_address: None, + local_output_address: None, + }, + coordinator_mode: None, + model_name: args.model.clone(), + client_index: args.client_index, + }) + .await + .context("failed to connect to external vLLM engine")?; + + println!("model={}", args.model); + println!("handshake_address={}", args.handshake_address); + println!("engine_count={}", args.engine_count); + println!("input_address={}", client.input_address()); + println!("output_address={}", client.output_address()); + println!("engine_identities={:x?}", client.engine_identities()); + + let initial_is_sleeping = + client.is_sleeping().await.context("failed to call is_sleeping utility")?; + + println!("is_sleeping(initial)={initial_is_sleeping}"); + + if initial_is_sleeping != args.expected_is_sleeping { + bail!( + "unexpected initial is_sleeping state: expected {}, got {}", + args.expected_is_sleeping, + initial_is_sleeping + ); + } + + let reset_prefix_cache = client + .reset_prefix_cache(args.reset_running_requests, args.reset_external) + .await + .context("failed to call reset_prefix_cache utility")?; + println!("reset_prefix_cache={reset_prefix_cache}"); + + client.reset_mm_cache().await.context("failed to call reset_mm_cache utility")?; + println!("reset_mm_cache=ok"); + + client + .reset_encoder_cache() + .await + .context("failed to call reset_encoder_cache utility")?; + println!("reset_encoder_cache=ok"); + + if args.skip_sleep_wake { + println!("sleep_wake=skipped"); + } else { + client.sleep(args.sleep_level, args.sleep_mode).await.with_context(|| { + format!( + "failed to call sleep utility with level={} mode={}", + args.sleep_level, args.sleep_mode + ) + })?; + println!( + "sleep=ok level={} mode={}", + args.sleep_level, args.sleep_mode + ); + + let sleeping_after_sleep = + client.is_sleeping().await.context("failed to call is_sleeping after sleep")?; + println!("is_sleeping(after_sleep)={sleeping_after_sleep}"); + + if !sleeping_after_sleep { + bail!("engine should report sleeping=true after sleep()"); + } + + client.wake_up(None).await.context("failed to call wake_up utility")?; + println!("wake_up=ok"); + + let sleeping_after_wake = + client.is_sleeping().await.context("failed to call is_sleeping after wake_up")?; + println!("is_sleeping(after_wake)={sleeping_after_wake}"); + + if sleeping_after_wake { + bail!("engine should report sleeping=false after wake_up()"); + } + } + + client.shutdown().await.context("failed to shut down engine-core client")?; + + Ok(()) +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client.rs new file mode 100644 index 0000000000000000000000000000000000000000..9c481362d43c62729666ddb20a8fabdad07b17ea --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client.rs @@ -0,0 +1,793 @@ +use std::sync::Arc; +use std::time::Duration; + +use futures::future::{join_all, try_join_all}; +use itertools::Itertools; +use serde::Serialize; +use tokio::sync::mpsc; +use tokio_util::task::AbortOnDropHandle; +use tracing::{debug, info, trace}; + +use crate::client::imp::{ClientInner, run_abort_loop, run_output_dispatcher_loop}; +use crate::coordinator::CoordinatorHandle; +use crate::error::{Error, Result}; +use crate::protocol::dtype::ModelDtype; +use crate::protocol::handshake::EngineCoreReadyResponse; +use crate::protocol::lora::LoraRequest; +use crate::protocol::request::{EngineCoreRequest, EngineCoreRequestType}; +use crate::protocol::utility::{EngineCoreUtilityRequest, PauseMode}; +use crate::runtime::{BackgroundShutdownRuntime, build_zmq_runtime}; +use crate::transport::{self, ConnectedEngine}; + +pub(crate) mod imp; +mod state; +mod stream; + +pub use stream::{EngineCoreOutputStream, EngineCoreStreamOutput}; + +/// How the frontend acquires its request/response transport with Python +/// `EngineCoreProc`s. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub enum TransportMode { + /// The Rust process owns the startup handshake and allocates or binds the + /// frontend transport addresses itself before replying to engine + /// `HELLO` messages. + HandshakeOwner { + /// Shared handshake endpoint that engines dial during startup. + handshake_address: String, + /// Host/IP that engines should use to connect back to the frontend + /// transport sockets. + advertised_host: String, + /// Total number of engines expected to join this transport. + engine_count: usize, + /// Maximum time to wait for each startup phase to complete. + ready_timeout: Duration, + /// Optional explicit bind address for the input ROUTER socket. + local_input_address: Option, + /// Optional explicit bind address for the output PULL socket. + local_output_address: Option, + }, + + /// The Python supervisor has already chosen the frontend transport + /// addresses, and the Rust process only needs to bind them and wait for + /// engine registration frames. + Bootstrapped { + /// Input ROUTER socket address that engines will connect to for + /// requests. + input_address: String, + /// Output PULL socket address that engines will connect to for + /// responses. + output_address: String, + /// First data-parallel engine rank expected to register on this + /// transport. + engine_start_index: u32, + /// Total number of engines expected to register on this transport. + engine_count: usize, + /// Maximum time to wait for all expected engines to register. + ready_timeout: Duration, + }, +} + +/// Which coordinator implementation should be active when one is present for a +/// frontend client. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum CoordinatorMode { + /// Run the Rust in-process coordinator for managed `serve` deployments. + InProc, + /// Connect to an external coordinator owned by another process. + External { address: String }, +} + +/// Configuration for connecting a Rust frontend client to an already running +/// Python `EngineCoreProc`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct EngineCoreClientConfig { + /// Frontend-to-engine transport setup. + pub transport_mode: TransportMode, + /// Frontend-side coordinator behavior, or `None` when requests should flow + /// directly to engines without any coordinator involvement. + pub coordinator_mode: Option, + /// Model name used for frontend-side metrics labels. + pub model_name: String, + /// Frontend client index stamped onto every request. + pub client_index: u32, +} + +impl EngineCoreClientConfig { + /// Create a new client config with the given handshake address, expecting a + /// single engine, and default values for all other fields. + pub fn new_single(handshake_address: impl Into) -> Self { + Self { + transport_mode: TransportMode::HandshakeOwner { + handshake_address: handshake_address.into(), + advertised_host: "127.0.0.1".to_string(), + engine_count: 1, + ready_timeout: Duration::from_secs(30), + local_input_address: None, + local_output_address: None, + }, + coordinator_mode: None, + model_name: String::new(), + client_index: 0, + } + } + + /// Set the model name used by frontend-side metrics and diagnostics. + pub fn with_model_name(mut self, model_name: impl Into) -> Self { + self.model_name = model_name.into(); + self + } + + /// Override the client index stamped onto every outgoing request. + pub fn with_client_index(mut self, client_index: u32) -> Self { + self.client_index = client_index; + self + } + + /// Override the optional coordinator mode for this client config. + pub fn with_coordinator_mode(mut self, coordinator_mode: Option) -> Self { + self.coordinator_mode = coordinator_mode; + self + } + + /// Override the locally bound input/output addresses for handshake-owned + /// transport mode. + /// + /// This is primarily used by tests that want deterministic IPC endpoints + /// while still exercising the handshake-owned startup path. + pub fn with_local_input_output_addresses( + mut self, + local_input_address: Option, + local_output_address: Option, + ) -> Self { + let TransportMode::HandshakeOwner { + local_input_address: current_input, + local_output_address: current_output, + .. + } = &mut self.transport_mode + else { + panic!("local input/output overrides are only valid in handshake-owned mode"); + }; + *current_input = local_input_address; + *current_output = local_output_address; + self + } +} + +/// The reason a request stream is being aborted when its output stream is +/// dropped. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub enum AbortCause { + /// The consumer dropped the stream before the request reached a terminal + /// engine output. + #[default] + DroppedStream, + /// The frontend matched a stop string locally and intentionally stopped + /// consuming the stream. + StopStringMatched, +} + +task_local::task_local! { + static ABORT_CAUSE: AbortCause; +} + +impl AbortCause { + /// Return the abort cause currently associated with this task, or + /// [`AbortCause::DroppedStream`] by default. + pub fn current() -> Self { + ABORT_CAUSE.try_get().unwrap_or_default() + } + + /// Drop one value while marking the drop as happening for this abort cause. + pub fn drop_as(self, value: T) { + ABORT_CAUSE.sync_scope(self, move || drop(value)); + } +} + +/// Internal auto-abort work item sent from stream `Drop` handlers to the abort +/// worker. +#[derive(Debug, Clone)] +pub(crate) struct AbortRequest { + request_id: String, + cause: AbortCause, +} + +/// Default ZMQ-based implementation that talks directly to a Python +/// `EngineCoreProc`. +pub struct EngineCoreClient { + config: EngineCoreClientConfig, + input_address: String, + output_address: String, + engines: Vec, + inner: Arc, + coordinator: Option, + abort_tx: mpsc::UnboundedSender, + + /// Runtime used to send messages to the engine and drive all background tasks. + runtime: BackgroundShutdownRuntime, + // Background tasks + output_task: AbortOnDropHandle<()>, + dispatcher_task: AbortOnDropHandle<()>, + abort_task: AbortOnDropHandle<()>, + coordinator_output_task: Option>, + coordinator_task: Option>, +} + +impl EngineCoreClient { + /// Connect to Python `EngineCoreProc`s using the configured + /// transport/coordinator modes. + /// + /// In handshake-owned mode this method drives the full engine startup + /// handshake. In bootstrapped mode it binds the provided frontend + /// sockets and waits for the expected engine registration frames. + pub async fn connect(config: EngineCoreClientConfig) -> Result { + let connected = match &config.transport_mode { + TransportMode::HandshakeOwner { + handshake_address, + advertised_host, + engine_count, + ready_timeout, + local_input_address, + local_output_address, + } => { + let enable_inproc_coordinator = match config.coordinator_mode { + None => false, + Some(CoordinatorMode::InProc) => true, + Some(CoordinatorMode::External { .. }) => { + return Err(Error::UnsupportedExternalCoordinator); + } + }; + + transport::connect_handshake( + handshake_address, + *engine_count, + advertised_host, + local_input_address.as_deref(), + local_output_address.as_deref(), + enable_inproc_coordinator, + *ready_timeout, + ) + .await? + } + + TransportMode::Bootstrapped { + input_address, + output_address, + engine_start_index, + engine_count, + ready_timeout, + } => { + if let Some(CoordinatorMode::InProc) = config.coordinator_mode { + panic!("cannot use in-process coordinator with bootstrapped transport mode") + } + + transport::connect_bootstrapped( + input_address, + output_address, + *engine_start_index, + *engine_count, + *ready_timeout, + ) + .await? + } + }; + + Self::from_connected(config, connected).await + } + + /// Create a new client instance from the connected transport state after + /// the startup handshake completes. + async fn from_connected( + config: EngineCoreClientConfig, + connected: transport::ConnectedTransport, + ) -> Result { + let (output_tx, output_rx) = mpsc::channel(64); + let (abort_tx, abort_rx) = mpsc::unbounded_channel(); + let engines = connected.engines; + let runtime = build_zmq_runtime(); + let inner = Arc::new(ClientInner::new( + connected.input_send, + runtime.handle().clone(), + config.model_name.clone(), + &engines, + )); + let output_task = AbortOnDropHandle::new(runtime.spawn(transport::run_output_loop( + connected.output_socket, + output_tx, + ))); + let dispatcher_task = AbortOnDropHandle::new( + runtime.spawn(run_output_dispatcher_loop(inner.clone(), output_rx)), + ); + let abort_task = + AbortOnDropHandle::new(runtime.spawn(run_abort_loop(inner.clone(), abort_rx))); + + // If any engine reported a dp_stats_address in its ready response, use it + // as the external coordinator address. + let dp_stats_address: Option = + engines.iter().find_map(|engine| engine.ready_response.dp_stats_address.clone()); + + let (coordinator, coordinator_output_task, coordinator_task) = + if let Some(coordinator_transport) = connected.coordinator { + let (handle, runner) = + CoordinatorHandle::new_inproc(coordinator_transport.input_socket); + let (coordinator_output_tx, coordinator_output_rx) = mpsc::channel(64); + let coordinator_output_task = + AbortOnDropHandle::new(runtime.spawn(transport::run_output_loop( + coordinator_transport.output_socket, + coordinator_output_tx, + ))); + let coordinator_task = AbortOnDropHandle::new( + runtime.spawn(runner.run(coordinator_output_rx, inner.clone())), + ); + ( + Some(handle), + Some(coordinator_output_task), + Some(coordinator_task), + ) + } else if let Some(address) = + dp_stats_address.as_deref().or(match config.coordinator_mode.as_ref() { + Some(CoordinatorMode::External { address }) => Some(address.as_str()), + _ => None, + }) + { + let (handle, service) = CoordinatorHandle::connect_external(address).await?; + let coordinator_task = + AbortOnDropHandle::new(runtime.spawn(service.run(inner.clone()))); + (Some(handle), None, Some(coordinator_task)) + } else { + (None, None, None) + }; + + Ok(Self { + config, + input_address: connected.input_address, + output_address: connected.output_address, + engines, + inner, + coordinator, + abort_tx, + runtime, + output_task, + dispatcher_task, + abort_task, + coordinator_output_task, + coordinator_task, + }) + } + + /// Return the address of the input socket that the client uses to send + /// requests to the engine. + pub fn input_address(&self) -> &str { + &self.input_address + } + + /// Return the address of the output socket that the client listens on for + /// engine responses. + pub fn output_address(&self) -> &str { + &self.output_address + } + + /// Return the number of engines connected to this client. + pub fn engine_count(&self) -> usize { + self.engines.len() + } + + /// Return the engine-side indices connected to this client. + pub fn engine_indices(&self) -> Vec { + self.engines + .iter() + .map(|engine| engine.engine_id.engine_index().expect("engine id must encode as u16")) + .collect() + } + + /// Return the engine identities of all engines connected to this client. + pub fn engine_identities(&self) -> Vec<&[u8]> { + self.engines.iter().map(|engine| &*engine.engine_id).collect() + } + + /// Return the ready responses received from all engines on the input + /// socket. + pub fn ready_responses(&self) -> Vec<&EngineCoreReadyResponse> { + self.engines.iter().map(|engine| &engine.ready_response).collect() + } + + /// Return the engine-reported effective model dtype. + pub fn model_dtype(&self) -> ModelDtype { + self.engines + .first() + .expect("engine core client requires at least one engine") + .ready_response + .dtype + } + + /// Return the engine-reported Python vLLM version. + pub fn vllm_version(&self) -> &str { + self.engines + .first() + .expect("engine core client requires at least one engine") + .ready_response + .vllm_version + .as_str() + } + + /// Return the total number of GPU blocks summed across all connected + /// engines. + pub fn total_num_gpu_blocks(&self) -> u64 { + self.engines.iter().map(|engine| engine.ready_response.num_gpu_blocks).sum() + } + + /// Return the minimum engine-reported `max_model_len` across all engines. + /// + /// This is the auto-fitted value after KV cache profiling and may differ + /// from the originally configured value. + pub fn max_model_len(&self) -> u32 { + self.engines + .iter() + .map(|engine| engine.ready_response.max_model_len as u32) + .min() + .expect("engine core client requires at least one engine") + } + + /// Return the world size (TP * PP) from the parallel config, if available. + pub fn world_size(&self) -> u64 { + self.engines + .first() + .expect("engine core client requires at least one engine") + .ready_response + .world_size + } + + /// Return the data parallel size from the parallel config, if available. + pub fn data_parallel_size(&self) -> u64 { + self.engines + .first() + .expect("engine core client requires at least one engine") + .ready_response + .data_parallel_size + } + + /// Get the model name associated with this client used for metrics + /// labeling. + pub fn model_name(&self) -> &str { + self.inner.model_name() + } + + /// Return whether the client still considers the engine healthy. + pub fn is_healthy(&self) -> bool { + self.inner.is_healthy() + } + + /// Return the first persistent health error observed by the client, if any. + pub fn health_error(&self) -> Option> { + self.inner.health_error() + } +} + +// Client API implementation. +impl EngineCoreClient { + /// Add a new request to the engine and return a per-request raw output + /// stream. + pub async fn call(&self, mut req: EngineCoreRequest) -> Result { + req.client_index = self.config.client_index; + req.validate()?; + trace!( + request_id = %req.request_id, + client_index = req.client_index, + current_wave = req.current_wave, + request = ?req, + "sending add request" + ); + + let request_id = req.request_id.clone(); + let lora_name = req.lora_request.as_ref().map(|lora| lora.lora_name.clone()); + let data_parallel_rank = req.data_parallel_rank; + let (engine_id, rx) = + self.inner.register_request(request_id.clone(), lora_name, data_parallel_rank)?; + + let result: Result<()> = async { + if let Some(coordinator) = self.coordinator.as_ref() { + let snapshot = coordinator.snapshot(); + req.current_wave = snapshot.current_wave; + if !snapshot.engines_running { + coordinator.notify_first_request(engine_id.clone())?; + } + } + + debug!( + request_id = req.request_id, + ?engine_id, + "registered request to engine" + ); + + self.inner.send_to_engine(&engine_id, EngineCoreRequestType::Add, &req).await?; + Ok(()) + } + .await; + + // Failed to send the request to the engine, roll back the registration. + if let Err(error) = result { + self.inner.rollback_request(&request_id); + return Err(error); + } + + Ok(EngineCoreOutputStream::new( + request_id, + self.abort_tx.clone(), + rx, + )) + } + + /// Abort currently in-flight requests by request ID. + pub async fn abort(&self, ids: &[String]) -> Result<()> { + let abortable = self.inner.abortable_request_ids(ids)?; + + trace!(request_ids = ?ids, abortable_request_ids = ?abortable, "sending abort request ids"); + + if abortable.is_empty() { + return Ok(()); + } + + // Finalize the consumer streams first, before the engine round-trip. + let all_request_ids: Vec = abortable.values().flatten().cloned().collect(); + self.inner.abort_requests_locally(&all_request_ids); + + for (engine_id, request_ids) in abortable { + self.inner.do_abort_requests(&engine_id, &request_ids).await?; + } + Ok(()) + } + + /// Call a typed utility method on all connected engines, returning one + /// decoded result per connected engine if all calls succeed or an error + /// if any call fails. + /// + /// Callers should pass utility arguments using Rust tuple semantics so the + /// encoded payload matches Python's `(client_index, call_id, + /// method_name, args)` contract: `()`, `(arg,)`, `(arg1, arg2)`, etc. + pub async fn call_utility(&self, method: &str, args: A) -> Result> + where + T: serde::de::DeserializeOwned, + A: serde::Serialize + std::fmt::Debug, + { + trace!( + method, + client_index = self.config.client_index, + engine_count = self.engines.len(), + "sending utility request" + ); + + // Phase 1: allocate one call id per engine and build the per-engine + // request payloads up-front. Any failure here (registry closed, encode + // error) must roll back the call ids already allocated so they do not + // leak in the utility registry until shutdown. + let mut pending_calls = Vec::with_capacity(self.engines.len()); + let mut prepared_sends = Vec::with_capacity(self.engines.len()); + for engine in &self.engines { + let (call_id, rx) = match self.inner.allocate_and_register_utility_call() { + Ok(pair) => pair, + Err(err) => { + self.inner.unregister_utility_calls(pending_calls.iter().map(|(id, _)| *id)); + return Err(err); + } + }; + let request = match EngineCoreUtilityRequest::new( + self.config.client_index, + call_id, + method, + &args, + ) { + Ok(request) => request, + Err(err) => { + self.inner.unregister_utility_calls( + pending_calls.iter().map(|(id, _)| *id).chain(std::iter::once(call_id)), + ); + return Err(err); + } + }; + pending_calls.push((call_id, rx)); + prepared_sends.push((&engine.engine_id, request)); + } + + // Phase 2: dispatch every utility request concurrently. `try_join_all` + // fails fast on the first transport error and drops the remaining send + // futures; any engines that already received the request will reply, + // but those replies are simply dropped because we roll back the call + // ids below. + let send_futures = prepared_sends.iter().map(|(engine_id, request)| { + self.inner.send_to_engine(engine_id, EngineCoreRequestType::Utility, request) + }); + if let Err(err) = try_join_all(send_futures).await { + self.inner.unregister_utility_calls(pending_calls.iter().map(|(id, _)| *id)); + return Err(err); + } + + // Phase 3: wait for all engines to respond and preserve the per-engine + // result list. + let futures = pending_calls.into_iter().map(|(call_id, rx)| async move { + rx.await + .map_err(|_| Error::UtilityCallClosed { + method: method.to_string(), + call_id, + })?? + .into_typed_result(method) + }); + try_join_all(futures).await + } + + /// Call a utility method on all connected engines and return the shared + /// result if every engine agrees. + pub async fn call_utility_consensus(&self, method: &str, args: A) -> Result + where + T: serde::de::DeserializeOwned + std::fmt::Debug + PartialEq, + A: serde::Serialize + std::fmt::Debug, + { + let results: Vec = self.call_utility(method, args).await?; + + if results.iter().all_equal() { + // `engine_count >= 1` is enforced during startup handshake so `results` must be + // non-empty. + Ok(results.into_iter().next().unwrap()) + } else { + Err(Error::InconsistentUtilityResults { + method: method.to_string(), + values: format!("{results:?}"), + }) + } + } + + /// Execute `collective_rpc` on all engines and flatten all engine results + /// into one list. + pub async fn collective_rpc( + &self, + method: &str, + timeout: Option, + args: A, + kwargs: K, + ) -> Result> + where + A: serde::Serialize + std::fmt::Debug, + K: serde::Serialize + std::fmt::Debug, + { + let results = self + .call_utility::("collective_rpc", (method, timeout, args, kwargs)) + .await?; + + Ok(results + .into_iter() + .flat_map(|result| match result { + // Each engine's `collective_rpc` result is itself the worker-level result list. + rmpv::Value::Array(results) => results, + other => vec![other], + }) + .collect()) + } + + /// Return whether the engine is currently sleeping at any level. + pub async fn is_sleeping(&self) -> Result { + self.call_utility_consensus("is_sleeping", ()).await + } + + /// Reset the multi-modal cache. + pub async fn reset_mm_cache(&self) -> Result<()> { + self.call_utility::<(), _>("reset_mm_cache", ()).await?; + Ok(()) + } + + /// Reset the encoder cache. + pub async fn reset_encoder_cache(&self) -> Result<()> { + self.call_utility::<(), _>("reset_encoder_cache", ()).await?; + Ok(()) + } + + /// Reset the prefix cache and optionally the external connector cache. + /// + /// Under data parallel, returns `true` only when every engine confirms the + /// reset (AND aggregation). + pub async fn reset_prefix_cache( + &self, + reset_running_requests: bool, + reset_connector: bool, + ) -> Result { + Ok(self + .call_utility( + "reset_prefix_cache", + (reset_running_requests, reset_connector), + ) + .await? + .into_iter() + .all(|reset| reset)) + } + + /// Load or refresh one LoRA adapter on every connected engine. + pub async fn add_lora(&self, lora_request: &LoraRequest) -> Result { + Ok(self + .call_utility::("add_lora", (lora_request,)) + .await? + .into_iter() + .all(|loaded| loaded)) + } + + /// Remove one LoRA adapter from every connected engine. + pub async fn remove_lora(&self, lora_id: u64) -> Result { + Ok(self + .call_utility::("remove_lora", (lora_id,)) + .await? + .into_iter() + .all(|removed| removed)) + } + + /// Put the engine to sleep. + pub async fn sleep(&self, level: u32, mode: PauseMode) -> Result<()> { + self.call_utility::<(), _>("sleep", (level, mode)).await?; + Ok(()) + } + + /// Wake the engine from sleep, optionally limiting the wake-up to specific + /// tags. + pub async fn wake_up(&self, tags: Option>) -> Result<()> { + self.call_utility::<(), _>("wake_up", (tags,)).await?; + Ok(()) + } + + /// Pause the scheduler so generation can be halted + pub async fn pause_scheduler(&self, mode: PauseMode, clear_cache: bool) -> Result<()> { + self.call_utility::<(), _>("pause_scheduler", (mode, clear_cache)).await?; + Ok(()) + } + + /// Resume the scheduler after a pause + pub async fn resume_scheduler(&self) -> Result<()> { + self.call_utility::<(), _>("resume_scheduler", ()).await?; + Ok(()) + } + + /// Return whether the scheduler is currently in any pause state. + pub async fn is_scheduler_paused(&self) -> Result { + self.call_utility_consensus("is_scheduler_paused", ()).await + } + + /// Start profiling the engine. + pub async fn start_profile(&self, profile_prefix: Option<&str>) -> Result<()> { + self.call_utility::<(), _>("profile", (true, profile_prefix)).await?; + Ok(()) + } + + /// Stop profiling the engine. + pub async fn stop_profile(&self, profile_prefix: Option<&str>) -> Result<()> { + self.call_utility::<(), _>("profile", (false, profile_prefix)).await?; + Ok(()) + } + + /// Shut down local client tasks and close transport state. + pub async fn shutdown(self) -> Result<()> { + let Self { + inner, + abort_tx, + runtime, + output_task, + dispatcher_task, + abort_task, + coordinator_output_task, + coordinator_task, + .. + } = self; + + info!("shutting down engine-core client"); + inner.shutdown(); + drop(abort_tx); + + // Abort all client tasks first, then await them. + // Note the aborting orders here. + let mut tasks = vec![abort_task, dispatcher_task, output_task]; + tasks.extend(coordinator_task); + tasks.extend(coordinator_output_task); + + tasks.iter().for_each(|t| t.abort()); + join_all(tasks).await; + drop(inner); + drop(runtime); + + info!("engine-core client shut down"); + Ok(()) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client/imp.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client/imp.rs new file mode 100644 index 0000000000000000000000000000000000000000..9975471c7472f6c92c39b1c3faf3651fde404a0d --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client/imp.rs @@ -0,0 +1,476 @@ +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::Arc; +use std::time::{SystemTime, UNIX_EPOCH}; + +use arc_swap::ArcSwapOption; +use parking_lot::Mutex; +use thiserror_ext::AsReport as _; +use tokio::runtime::Handle; +use tokio::sync::mpsc; +use tracing::{debug, info, trace, warn}; +use vllm_metrics::METRICS; +use zeromq::RouterSendHalf; + +use crate::client::state::{OutputReceiver, RequestRegistry, UtilityReceiver, UtilityRegistry}; +use crate::client::stream::EngineCoreStreamOutput; +use crate::client::{AbortCause, AbortRequest}; +use crate::error::{client_closed, dispatcher_closed, unexpected_dispatcher_output}; +use crate::metrics::{LoraInfoExporter, record_scheduler_stats}; +use crate::protocol::encode_msgpack; +use crate::protocol::output::{EngineCoreOutput, EngineCoreOutputs}; +use crate::protocol::request::EngineCoreRequestType; +use crate::protocol::stats::SchedulerStats; +use crate::protocol::utility::UtilityOutput; +use crate::transport::{ConnectedEngine, EngineId}; +use crate::{Error, Result, transport}; + +pub(crate) struct ClientInner { + input_send: RouterSendHalf, + /// The runtime handle used for sending messages to the engine. + handle: Handle, + model_name: String, + request_reg: Mutex, + utility_reg: Mutex, + health_error: ArcSwapOption, +} + +impl ClientInner { + /// Create a new instance with the given input send half after the startup + /// handshake completes. + pub fn new( + input_send: RouterSendHalf, + handle: Handle, + model_name: String, + engines: &[ConnectedEngine], + ) -> Self { + Self { + input_send, + handle, + model_name, + request_reg: Mutex::new(RequestRegistry::new(engines)), + utility_reg: Mutex::new(UtilityRegistry::default()), + health_error: ArcSwapOption::empty(), + } + } + + /// Get the model name associated with this client used for metrics + /// labeling. + pub fn model_name(&self) -> &str { + &self.model_name + } + + /// Register a newly added request. Return the selected engine id and the + /// per-request output channel bound to its `request_id`. + /// + /// When `data_parallel_rank` is provided, the request is routed to that + /// specific engine rank, bypassing load balancing. `lora_name` is the + /// request's LoRA adapter, tracked for `vllm:lora_requests_info`. + pub fn register_request( + &self, + request_id: String, + lora_name: Option, + data_parallel_rank: Option, + ) -> Result<(EngineId, OutputReceiver)> { + let mut registry = self.request_reg.lock(); + if registry.is_closed() { + return Err(self.closed_error()); + } + registry.register(request_id, lora_name, data_parallel_rank) + } + + /// Allocate the next utility `call_id` and register its waiting receiver. + pub fn allocate_and_register_utility_call(&self) -> Result<(u64, UtilityReceiver)> { + let mut registry = self.utility_reg.lock(); + if registry.is_closed() { + return Err(self.closed_error()); + } + Ok(registry.allocate_and_register()) + } + + /// Undo a batch of utility call allocations when the fan-out send fails + /// partway through. Silently ignores unknown call ids so callers can pass + /// the full set without first filtering successful sends. + pub fn unregister_utility_calls(&self, call_ids: impl IntoIterator) { + self.utility_reg.lock().unregister_many(call_ids); + } + + /// Undo a request registration when `add_request()` fails. + pub fn rollback_request(&self, request_id: &str) { + let _ = self.request_reg.lock().remove(request_id); + } + + /// Filter the given request IDs to the subset that are still tracked as + /// active and can be aborted, grouped by the engine that originally + /// accepted them. + pub fn abortable_request_ids( + &self, + request_ids: &[String], + ) -> Result>> { + let registry = self.request_reg.lock(); + if registry.is_closed() { + return Err(self.closed_error()); + } + Ok(registry.abortable_request_ids(request_ids)) + } + + /// Obtain stream senders for a whole engine output batch with one registry + /// lock acquisition. + pub fn take_senders_for_outputs<'a>( + &self, + outputs: impl IntoIterator, + ) -> Vec>>> { + self.request_reg.lock().senders_for_outputs(outputs) + } + + /// Remove a batch of requests that have finished or aborted, returning + /// their stream senders. + pub fn finish_requests<'a>( + &self, + request_ids: impl IntoIterator, + ) -> Vec>> { + self.request_reg.lock().finish_many(request_ids) + } + + /// Finalize client-initiated aborts by pushing a terminal `Abort` output + /// down each request's stream and removing it from the registry. Returns + /// the request ids that were still active. See [`RequestRegistry::abort_many`]. + pub fn abort_requests_locally<'a>( + &self, + request_ids: impl IntoIterator, + ) -> Vec { + let timestamp = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|d| d.as_secs_f64()) + .unwrap_or(0.0); + self.request_reg.lock().abort_many(request_ids, timestamp) + } + + /// Apply one scheduler stats update for the given engine to the local + /// routing state. Returns `false` if the engine is unknown to the + /// client. + pub fn apply_scheduler_stats(&self, engine_index: u32, stats: &SchedulerStats) -> bool { + self.request_reg.lock().apply_scheduler_stats(engine_index, stats) + } + + /// Snapshot the adapter names of tracked LoRA requests as + /// (running, waiting) sets. + pub fn lora_adapter_states(&self) -> (BTreeSet, BTreeSet) { + self.request_reg.lock().lora_adapter_states() + } + + /// Close all active request streams and utility calls with the first + /// persistent health error. + pub fn close_registries(&self, error: Arc) { + let persistent_error = self.record_health_error(error); + let request_senders = self.request_reg.lock().close(); + let utility_senders = self.utility_reg.lock().close(); + + // Notify all ongoing requests that the client is closed. + for sender in request_senders { + let _ = sender.send(Err(Error::Shared(persistent_error.clone()))); + } + for sender in utility_senders { + let _ = sender.send(Err(Error::Shared(persistent_error.clone()))); + } + } + + /// Return the first persistent health error observed by the client, if any. + pub fn health_error(&self) -> Option> { + self.health_error.load_full() + } + + /// Return whether the client still considers the engine healthy. + pub fn is_healthy(&self) -> bool { + self.health_error.load().is_none() + } + + /// Resolve one utility output to the waiting caller. Returns `true` if a + /// waiting caller existed. + pub fn resolve_utility_output(&self, output: UtilityOutput) -> bool { + let Some(call_id) = output.call_id.as_u64() else { + // Currently, all utility call issued by the client should have unsigned call IDs. + return false; + }; + + match self.utility_reg.lock().resolve(&call_id) { + Some(sender) => { + sender.send(Ok(output)).unwrap_or_default(); + true + } + None => false, + } + } + + /// Send the given message to the engine. The request should be first + /// registered via `register_request()` to ensure the request stream is + /// tracked. + pub async fn send_to_engine( + &self, + engine_id: &EngineId, + request_type: EngineCoreRequestType, + payload: &T, + ) -> Result<()> + where + T: serde::Serialize + std::fmt::Debug, + { + // TODO: for `EngineCoreRequest`, split outbound tensor raw views into aux + // frames instead of always producing a single msgpack frame. + let payload = encode_msgpack(payload)?; + let mut input_send = self.input_send.clone(); + let engine_id = engine_id.clone(); + + self.handle + .spawn(async move { + transport::send_message( + &mut input_send, + &engine_id, + request_type.to_frame(), + payload, + ) + .await + }) + .await? + } + + /// Handle an abort request by sending the abort message to the engine. + pub async fn do_abort_requests( + &self, + engine_id: &EngineId, + request_ids: &[String], + ) -> Result<()> { + self.send_to_engine(engine_id, EngineCoreRequestType::Abort, &request_ids).await + } + + /// Shut down by closing all active request streams and utility calls with a + /// sticky client closed error. + pub fn shutdown(&self) { + self.close_registries(Arc::new(client_closed!("engine-core client shut down"))); + } + + /// Remove the request from the active registry for auto-abort and return + /// the engine that the request was originally routed to, if it is still + /// active. + pub fn take_auto_abort_target(&self, request_id: &str) -> Option { + let mut registry = self.request_reg.lock(); + let (_, engine_id) = registry.remove(request_id)?; + if registry.is_closed() { + return None; + } + Some(engine_id) + } + + /// Publish the first persistent health error and return the sticky error + /// recorded for this client. Later failures do not overwrite the first + /// one so `/health` and post-close callers observe a stable cause. + fn record_health_error(&self, error: Arc) -> Arc { + if let Some(existing) = self.health_error.load_full() { + return existing; + } + self.health_error + .rcu(|current| current.clone().unwrap_or_else(|| error.clone())); + self.health_error + .load_full() + .expect("health error must be recorded before registries close") + } + + /// Assert there is a recorded health error and return a `Shared` variant + /// wrapping it for error returns when the client is already closed. + fn closed_error(&self) -> Error { + Error::Shared(self.health_error.load_full().expect( + "closed registry must have a recorded health error before rejecting new operations", + )) + } +} + +/// Background loop that listens for request IDs to abort and sends abort +/// messages to the engine. This is used to implement the auto-abort behavior +/// when a request stream is dropped without being properly terminated. +pub(crate) async fn run_abort_loop( + inner: Arc, + mut abort_rx: mpsc::UnboundedReceiver, +) { + // Coalesce bursts of auto-aborts into a single Abort message per engine. + // A dropped-stream storm (e.g. many clients disconnecting at once under + // high concurrency) would otherwise issue one engine round-trip per + // request. `recv_many` returns as soon as at least one item is ready, so a + // lone abort is still forwarded promptly. + const MAX_DRAIN: usize = 1024; + let mut batch: Vec = Vec::new(); + + while abort_rx.recv_many(&mut batch, MAX_DRAIN).await > 0 { + let mut by_engine: BTreeMap> = BTreeMap::new(); + + for AbortRequest { request_id, cause } in batch.drain(..) { + let Some(engine_id) = inner.take_auto_abort_target(&request_id) else { + debug!(request_id, "skip auto-abort for inactive request"); + continue; + }; + + match cause { + AbortCause::DroppedStream => { + info!(request_id, "auto-aborting request due to dropped stream") + } + AbortCause::StopStringMatched => { + debug!( + request_id, + "auto-aborting request due to stop string matched" + ) + } + } + + by_engine.entry(engine_id).or_default().push(request_id); + } + + for (engine_id, request_ids) in by_engine { + if let Err(error) = inner.do_abort_requests(&engine_id, &request_ids).await { + warn!( + ?engine_id, + ?request_ids, + error = %error.as_report(), + "failed to auto-abort request streams" + ); + } + } + } +} + +/// Background loop that listens for engine-core outputs and dispatches them to +/// the corresponding request streams based on their `request_id`. +pub(crate) async fn run_output_dispatcher_loop( + inner: Arc, + mut output_rx: mpsc::Receiver>, +) { + let mut lora_info = LoraInfoExporter::default(); + + let result: Result<()> = async { + loop { + let outputs = match output_rx.recv().await { + Some(outputs) => outputs, + None => Err(dispatcher_closed!( + "engine-core output dispatcher channel closed" + )), + }?; + + match outputs { + EngineCoreOutputs::RequestBatch(batch) => { + let senders = inner.take_senders_for_outputs(&batch.outputs); + for (output, sender) in batch.outputs.into_iter().zip(senders) { + let request_id = output.request_id.clone(); + let Some(sender) = sender else { + debug!(request_id, "dropping output for inactive request"); + continue; + }; + + let wrapped_output = EngineCoreStreamOutput { + engine_index: batch.engine_index, + timestamp: batch.timestamp, + output, + }; + if sender.send(Ok(wrapped_output)).is_err() { + debug!(request_id, "request output stream receiver dropped"); + } + } + + // The sender for normally-finished requests should have already been removed + // from the registry when their final output was dispatched + // above. This serves as a safety net to capture any + // requests marked as finished by the engine. + if let Some(finished_requests) = batch.finished_requests.as_ref() { + for request_id in finished_requests { + trace!(request_id, "request completed via finished_requests"); + } + drop(inner.finish_requests(finished_requests)); + } + + if let Some(scheduler_stats) = batch.scheduler_stats.as_ref() { + if !inner.apply_scheduler_stats(batch.engine_index, scheduler_stats) { + debug!( + engine_index = batch.engine_index, + "dropping scheduler stats for unknown engine" + ); + } + record_scheduler_stats( + &METRICS.scheduler, + inner.model_name(), + batch.engine_index, + scheduler_stats, + ); + } + + // The engine's scheduler stats never carry adapter names; + // the gauge is derived from the registry's frontend-side + // request tracking instead. + let (running, waiting) = inner.lora_adapter_states(); + lora_info.update(&METRICS.scheduler, running, waiting); + } + EngineCoreOutputs::Utility(utility) => { + let call_id = utility.output.call_id; + if inner.resolve_utility_output(utility.output) { + trace!( + %call_id, + engine_index = utility.engine_index, + "resolved utility output" + ); + } else { + warn!( + %call_id, + engine_index = utility.engine_index, + "dropping output for unexpected utility call" + ); + } + } + other => { + Err::<(), _>(unexpected_dispatcher_output!( + "received unexpected output on main dispatcher path: {other:?}" + ))?; + } + } + } + } + .await; + let Err(error) = result else { return }; + + warn!(error = %error.as_report(), "output dispatcher exiting with error"); + inner.close_registries(Arc::new(error)); +} + +#[cfg(test)] +mod tests { + use zeromq::{RouterSocket, Socket}; + + use super::*; + use crate::mock_engine::default_ready_response; + + async fn test_inner() -> ClientInner { + let mut socket = RouterSocket::new(); + socket.bind("tcp://127.0.0.1:0").await.unwrap(); + let (send, _) = socket.split(); + ClientInner::new( + send, + Handle::current(), + "test-model".to_string(), + &[ConnectedEngine { + engine_id: EngineId::from(b"engine-0"), + ready_response: default_ready_response(), + }], + ) + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn close_registries_records_first_health_error_only() { + let inner = test_inner().await; + + inner.close_registries(Arc::new(Error::EngineCoreDead)); + assert!(!inner.is_healthy()); + assert!(matches!( + inner.health_error().as_deref(), + Some(Error::EngineCoreDead) + )); + + inner.close_registries(Arc::new(client_closed!("shutdown"))); + assert!(matches!( + inner.health_error().as_deref(), + Some(Error::EngineCoreDead) + )); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client/state.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client/state.rs new file mode 100644 index 0000000000000000000000000000000000000000..264fa11cc537ecbbbce1039be93c070f0197a16e --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client/state.rs @@ -0,0 +1,926 @@ +use std::collections::{BTreeMap, BTreeSet, HashMap}; +use std::sync::atomic::{AtomicU64, Ordering}; + +use tokio::sync::{mpsc, oneshot}; +use tracing::trace; + +use crate::EngineId; +use crate::client::stream::EngineCoreStreamOutput; +use crate::error::{Error, Result}; +use crate::protocol::output::{EngineCoreEventType, EngineCoreFinishReason, EngineCoreOutput}; +use crate::protocol::stats::SchedulerStats; +use crate::protocol::utility::UtilityOutput; +use crate::transport::ConnectedEngine; + +pub type OutputSender = mpsc::UnboundedSender>; +pub type OutputReceiver = mpsc::UnboundedReceiver>; +pub type UtilitySender = oneshot::Sender>; +pub type UtilityReceiver = oneshot::Receiver>; + +#[derive(Debug)] +struct TrackedRequest { + sender: OutputSender, + engine_id: EngineId, + lora: Option, +} + +/// Frontend-side view of one LoRA request's scheduling phase. +/// +/// The engine's `SchedulerStats` does not carry adapter names, so +/// `vllm:lora_requests_info` must be derived from per-request lifecycle events +/// observed by this client, mirroring `LoRARequestStates` in the Python +/// frontend (`vllm/v1/engine/output_processor.py`). +#[derive(Debug)] +struct LoraRequestState { + adapter_name: String, + phase: LoraPhase, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum LoraPhase { + Waiting, + Running, +} + +/// The latest real scheduler-side load snapshot observed from one engine. +/// +/// These counters come from `scheduler_stats` on the normal engine output path +/// and are the preferred routing signal once available. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct EngineLoadSnapshot { + /// Requests still counted on the scheduler's waiting side. + waiting: usize, + /// Requests currently counted on the scheduler's running side. + running: usize, +} + +#[derive(Debug, Default)] +struct EngineRoutingState { + /// Requests admitted by this frontend that have not finished yet. + /// + /// This is used both as the bootstrap fallback before real scheduler stats + /// exist and as a lower bound afterwards so asynchronous scheduler + /// snapshots cannot erase frontend admission history. + inflight: usize, + /// The latest real scheduler snapshot received from this engine, if any. + last_scheduler_stats: Option, +} + +impl EngineRoutingState { + /// Compute the routing score used to pick the least-loaded engine. + /// + /// Scheduler stats can raise the load estimate above the frontend-local + /// view, but they should not lower it below requests this frontend has + /// already admitted. Waiting requests still get the same extra penalty + /// as the original `waiting * 4 + running` score. + fn routing_score(&self) -> usize { + const WAITING_WEIGHT: usize = 4; + + let Some(stats) = self.last_scheduler_stats else { + return self.inflight; + }; + + let scheduler_total = stats.running + stats.waiting; + self.inflight.max(scheduler_total) + stats.waiting * (WAITING_WEIGHT - 1) + } + + /// Replace the local routing view with a fresh real scheduler snapshot. + fn apply_scheduler_counts(&mut self, next: EngineLoadSnapshot) { + self.last_scheduler_stats = Some(next); + } +} + +/// Internal registry for tracking active requests and their output stream +/// senders. +/// +/// This is used to route incoming outputs to the correct request stream, and to +/// ensure proper cleanup of senders when requests finish or the client shuts +/// down. +#[derive(Debug)] +pub struct RequestRegistry { + closed: bool, + requests: HashMap, + active_lora_requests: usize, + routing_per_engine: BTreeMap, +} + +impl RequestRegistry { + pub fn new(engines: &[ConnectedEngine]) -> Self { + Self { + closed: false, + requests: HashMap::default(), + active_lora_requests: 0, + routing_per_engine: engines + .iter() + .map(|engine| (engine.engine_id.clone(), EngineRoutingState::default())) + .collect(), + } + } + + /// Register a newly added request. Create the per-request output channel + /// bound to its `request_id` and return the selected engine id. + /// + /// When `data_parallel_rank` is provided, the request is routed directly to + /// the engine at that rank index, bypassing load balancing. Otherwise + /// the engine with the fewest in-flight requests is chosen. + pub fn register( + &mut self, + request_id: String, + lora_name: Option, + data_parallel_rank: Option, + ) -> Result<(EngineId, OutputReceiver)> { + if self.requests.contains_key(&request_id) { + return Err(Error::DuplicateRequestId { request_id }); + } + + let engine_id = self.choose_engine_for_request(data_parallel_rank)?; + let (tx, rx) = mpsc::unbounded_channel(); + let lora = lora_name.map(|adapter_name| LoraRequestState { + adapter_name, + phase: LoraPhase::Waiting, + }); + if lora.is_some() { + self.active_lora_requests += 1; + } + self.requests.insert( + request_id, + TrackedRequest { + sender: tx, + engine_id: engine_id.clone(), + lora, + }, + ); + + let state = self + .routing_per_engine + .get_mut(&engine_id) + .expect("request registry must track all known engines"); + state.inflight += 1; + + Ok((engine_id, rx)) + } + + fn choose_engine_for_request(&mut self, data_parallel_rank: Option) -> Result { + if let Some(rank) = data_parallel_rank { + // Route to the engine at the specified rank index. + let engine_id = EngineId::from_engine_index(rank); + return self + .routing_per_engine + .contains_key(&engine_id) + .then_some(engine_id) + .ok_or_else(|| Error::InvalidDataParallelRank { + rank, + num_engines: self.routing_per_engine.len() as u32, + }); + } + + Ok(self + .routing_per_engine + .iter() + .min_by_key(|(_, state)| state.routing_score()) + .map(|(engine_id, _)| engine_id.clone()) + .expect("request registry must contain at least one engine")) + } + + /// Filter the given request IDs to the subset that are still tracked as + /// active and can be aborted, grouped by engine. + pub fn abortable_request_ids(&self, request_ids: &[String]) -> BTreeMap> { + let mut by_engine = BTreeMap::new(); + for request_id in request_ids { + let Some(tracked) = self.requests.get(request_id.as_str()) else { + continue; + }; + by_engine + .entry(tracked.engine_id.clone()) + .or_insert_with(Vec::new) + .push(request_id.clone()); + } + by_engine + } + + /// Obtain the stream sender for one output. If it indicates the request is + /// finished, it will be removed from the registry. + pub fn sender_for_output(&mut self, output: &EngineCoreOutput) -> Option { + self.apply_lora_events(output); + if output.finished() { + self.remove(output.request_id.as_str()).map(|tracked| tracked.0) + } else { + self.requests + .get(output.request_id.as_str()) + .map(|tracked| tracked.sender.clone()) + } + } + + /// Advance the request's LoRA scheduling phase from the engine-core events + /// attached to one output, mirroring the Python frontend's + /// `LoRARequestStates.update_from_events`. + fn apply_lora_events(&mut self, output: &EngineCoreOutput) { + let Some(events) = output.events.as_ref() else { + return; + }; + let Some(lora) = self + .requests + .get_mut(output.request_id.as_str()) + .and_then(|tracked| tracked.lora.as_mut()) + else { + return; + }; + for event in events { + lora.phase = match event.r#type { + EngineCoreEventType::Queued | EngineCoreEventType::Preempted => LoraPhase::Waiting, + EngineCoreEventType::Scheduled => LoraPhase::Running, + }; + } + } + + /// Snapshot the adapter names of tracked LoRA requests as + /// (running, waiting) sets. Feeds the `vllm:lora_requests_info` gauge. + pub fn lora_adapter_states(&self) -> (BTreeSet, BTreeSet) { + if self.active_lora_requests == 0 { + return (BTreeSet::new(), BTreeSet::new()); + } + + let mut running = BTreeSet::new(); + let mut waiting = BTreeSet::new(); + for lora in self.requests.values().filter_map(|tracked| tracked.lora.as_ref()) { + let set = match lora.phase { + LoraPhase::Running => &mut running, + LoraPhase::Waiting => &mut waiting, + }; + set.insert(lora.adapter_name.clone()); + } + (running, waiting) + } + + /// Obtain stream senders for a whole engine output batch under one + /// registry lock. Finished outputs are removed before returning. + pub fn senders_for_outputs<'a>( + &mut self, + outputs: impl IntoIterator, + ) -> Vec> { + outputs.into_iter().map(|output| self.sender_for_output(output)).collect() + } + + /// Remove a batch of requests that have finished or aborted, returning + /// their stream senders. + pub fn finish_many<'a>( + &mut self, + request_ids: impl IntoIterator, + ) -> Vec { + request_ids + .into_iter() + .filter_map(|request_id| self.remove(request_id.as_str()).map(|tracked| tracked.0)) + .collect() + } + + /// Apply one scheduler stats update for the given engine to the local + /// routing state. Returns `false` if the engine is unknown to the + /// client. + pub fn apply_scheduler_stats(&mut self, engine_index: u32, stats: &SchedulerStats) -> bool { + self.apply_scheduler_counts( + engine_index, + EngineLoadSnapshot { + waiting: stats.num_waiting_reqs as usize, + running: stats.num_running_reqs as usize, + }, + ) + } + + /// Mark the registry as closed, detach and return all tracked senders. + pub fn close(&mut self) -> Vec { + if self.closed { + return Vec::new(); + } + + self.closed = true; + self.active_lora_requests = 0; + std::mem::take(&mut self.requests) + .into_values() + .map(|tracked| tracked.sender) + .collect() + } + + /// Finalize client-initiated aborts: remove each request and push a + /// terminal output with `finish_reason = Abort` down its stream before the + /// sender drops. Returns the request ids that were still active. + pub fn abort_many<'a>( + &mut self, + request_ids: impl IntoIterator, + timestamp: f64, + ) -> Vec { + let mut aborted = Vec::new(); + for request_id in request_ids { + let Some((sender, engine_id)) = self.remove(request_id) else { + continue; + }; + let output = EngineCoreStreamOutput { + engine_index: engine_id.engine_index().unwrap_or(0), + timestamp, + output: EngineCoreOutput { + request_id: request_id.clone(), + finish_reason: Some(EngineCoreFinishReason::Abort), + ..EngineCoreOutput::default() + }, + }; + let _ = sender.send(Ok(output)); + aborted.push(request_id.clone()); + } + aborted + } + + /// Remove one request from the local registry. Returns the tracked entry if + /// it exists. + #[must_use] + pub fn remove(&mut self, request_id: &str) -> Option<(OutputSender, EngineId)> { + let tracked = self.requests.remove(request_id)?; + if tracked.lora.is_some() { + self.active_lora_requests -= 1; + } + self.routing_per_engine + .get_mut(&tracked.engine_id) + .expect("request registry must track all known engines") + .inflight -= 1; + Some((tracked.sender, tracked.engine_id)) + } + + fn apply_scheduler_counts(&mut self, engine_index: u32, next: EngineLoadSnapshot) -> bool { + let engine_id = EngineId::from_engine_index(engine_index); + let Some(state) = self.routing_per_engine.get_mut(&engine_id) else { + return false; + }; + + let previous = state.last_scheduler_stats; + if previous != Some(next) { + trace!( + ?engine_id, + previous_waiting = previous.map(|stats| stats.waiting), + previous_running = previous.map(|stats| stats.running), + waiting = next.waiting, + running = next.running, + "updated scheduler routing counts", + ); + } + + state.apply_scheduler_counts(next); + true + } + + #[cfg(test)] + pub fn contains(&self, request_id: &str) -> bool { + self.requests.contains_key(request_id) + } + + pub fn is_closed(&self) -> bool { + self.closed + } + + #[cfg(test)] + fn active_lora_requests(&self) -> usize { + self.active_lora_requests + } +} + +/// Internal registry for tracking active utility calls and their waiting +/// receivers. +#[derive(Debug)] +pub struct UtilityRegistry { + closed: bool, + next_call_id: AtomicU64, + utility_calls: BTreeMap, +} + +impl Default for UtilityRegistry { + fn default() -> Self { + Self { + closed: false, + next_call_id: AtomicU64::new(1), + utility_calls: BTreeMap::default(), + } + } +} + +impl UtilityRegistry { + /// Allocate the next utility `call_id` and register a newly added utility + /// call. + pub fn allocate_and_register(&mut self) -> (u64, UtilityReceiver) { + let call_id = self.next_call_id.fetch_add(1, Ordering::Relaxed); + let (tx, rx) = oneshot::channel(); + self.utility_calls.insert(call_id, tx); + (call_id, rx) + } + + /// Resolve a utility output to its waiting receiver. + pub fn resolve(&mut self, call_id: &u64) -> Option { + self.utility_calls.remove(call_id) + } + + /// Drop a batch of registered utility calls without delivering a result. + /// Used to roll back allocations when the dispatch fan-out fails before + /// every engine could accept the request. + pub fn unregister_many(&mut self, call_ids: impl IntoIterator) { + for call_id in call_ids { + self.utility_calls.remove(&call_id); + } + } + + /// Mark the registry as closed, detach and return all tracked senders. + pub fn close(&mut self) -> Vec { + if self.closed { + return Vec::new(); + } + + self.closed = true; + std::mem::take(&mut self.utility_calls).into_values().collect() + } + + #[cfg(test)] + pub fn contains(&self, call_id: u64) -> bool { + self.utility_calls.contains_key(&call_id) + } + + pub fn is_closed(&self) -> bool { + self.closed + } +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeSet; + + use crate::EngineId; + use crate::client::state::{ + EngineLoadSnapshot, EngineRoutingState, RequestRegistry, UtilityRegistry, + }; + use crate::mock_engine::default_ready_response; + use crate::protocol::output::{ + EngineCoreEvent, EngineCoreEventType, EngineCoreFinishReason, EngineCoreOutput, + }; + use crate::transport::ConnectedEngine; + + fn connected_engine(engine_id: EngineId) -> ConnectedEngine { + ConnectedEngine { + engine_id, + ready_response: default_ready_response(), + } + } + + fn output_with_events( + request_id: &str, + events: &[EngineCoreEventType], + finish_reason: Option, + ) -> EngineCoreOutput { + EngineCoreOutput { + request_id: request_id.to_string(), + events: Some( + events + .iter() + .map(|event_type| EngineCoreEvent { + r#type: *event_type, + timestamp: 0.0, + }) + .collect(), + ), + finish_reason, + ..Default::default() + } + } + + fn adapter_names(values: &[&str]) -> BTreeSet { + values.iter().map(|name| (*name).to_string()).collect() + } + + #[test] + fn registry_rejects_duplicate_request_ids() { + let mut registry = RequestRegistry::new(&[connected_engine(EngineId::from(b"engine-0"))]); + registry.register("req-1".to_string(), None, None).unwrap(); + let error = registry.register("req-1".to_string(), None, None).unwrap_err(); + assert!(matches!( + error, + crate::error::Error::DuplicateRequestId { request_id } if request_id == "req-1" + )); + } + + #[test] + fn registry_removes_finished_request_on_output() { + let mut registry = RequestRegistry::new(&[connected_engine(EngineId::from(b"engine-0"))]); + registry.register("req-1".to_string(), None, None).unwrap(); + + let sender = registry.sender_for_output(&EngineCoreOutput { + request_id: "req-1".to_string(), + finish_reason: Some(EngineCoreFinishReason::Length), + ..Default::default() + }); + + assert!(sender.is_some()); + assert!(!registry.contains("req-1")); + } + + #[test] + fn registry_tracks_lora_phases_from_engine_events() { + let mut registry = RequestRegistry::new(&[connected_engine(EngineId::from(b"engine-0"))]); + registry + .register("req-lora".to_string(), Some("adapter-a".to_string()), None) + .unwrap(); + registry.register("req-plain".to_string(), None, None).unwrap(); + + // Registered but not yet scheduled: counted as waiting. The non-LoRA + // request never shows up. + assert_eq!( + registry.lora_adapter_states(), + (adapter_names(&[]), adapter_names(&["adapter-a"])) + ); + + // Queued then scheduled in one output: running. + drop(registry.sender_for_output(&output_with_events( + "req-lora", + &[EngineCoreEventType::Queued, EngineCoreEventType::Scheduled], + None, + ))); + assert_eq!( + registry.lora_adapter_states(), + (adapter_names(&["adapter-a"]), adapter_names(&[])) + ); + + // Preempted: back to waiting. + drop(registry.sender_for_output(&output_with_events( + "req-lora", + &[EngineCoreEventType::Preempted], + None, + ))); + assert_eq!( + registry.lora_adapter_states(), + (adapter_names(&[]), adapter_names(&["adapter-a"])) + ); + + // Finished: dropped from tracking entirely. + drop(registry.sender_for_output(&output_with_events( + "req-lora", + &[EngineCoreEventType::Scheduled], + Some(EngineCoreFinishReason::Stop), + ))); + assert_eq!( + registry.lora_adapter_states(), + (adapter_names(&[]), adapter_names(&[])) + ); + } + + #[test] + fn registry_unions_lora_adapters_across_requests() { + let mut registry = RequestRegistry::new(&[connected_engine(EngineId::from(b"engine-0"))]); + registry + .register("req-a1".to_string(), Some("adapter-a".to_string()), None) + .unwrap(); + registry + .register("req-a2".to_string(), Some("adapter-a".to_string()), None) + .unwrap(); + registry + .register("req-b".to_string(), Some("adapter-b".to_string()), None) + .unwrap(); + + // One of adapter-a's requests starts running while the other waits: + // the adapter appears in both sets. + drop(registry.sender_for_output(&output_with_events( + "req-a1", + &[EngineCoreEventType::Scheduled], + None, + ))); + assert_eq!( + registry.lora_adapter_states(), + ( + adapter_names(&["adapter-a"]), + adapter_names(&["adapter-a", "adapter-b"]) + ) + ); + } + + #[test] + fn registry_counts_only_active_lora_requests() { + let mut registry = RequestRegistry::new(&[connected_engine(EngineId::from(b"engine-0"))]); + + registry.register("req-plain".to_string(), None, None).unwrap(); + assert_eq!(registry.active_lora_requests(), 0); + assert_eq!( + registry.lora_adapter_states(), + (adapter_names(&[]), adapter_names(&[])) + ); + + registry + .register( + "req-lora-a".to_string(), + Some("adapter-a".to_string()), + None, + ) + .unwrap(); + registry + .register( + "req-lora-b".to_string(), + Some("adapter-b".to_string()), + None, + ) + .unwrap(); + assert_eq!(registry.active_lora_requests(), 2); + + drop(registry.remove("req-plain")); + assert_eq!(registry.active_lora_requests(), 2); + + drop(registry.finish_many(&["req-lora-a".to_string()])); + assert_eq!(registry.active_lora_requests(), 1); + + drop(registry.abort_many(&["req-lora-b".to_string()], 0.0)); + assert_eq!(registry.active_lora_requests(), 0); + assert_eq!( + registry.lora_adapter_states(), + (adapter_names(&[]), adapter_names(&[])) + ); + } + + #[test] + fn registry_clears_lora_count_on_close() { + let mut registry = RequestRegistry::new(&[connected_engine(EngineId::from(b"engine-0"))]); + registry + .register("req-lora".to_string(), Some("adapter-a".to_string()), None) + .unwrap(); + + assert_eq!(registry.active_lora_requests(), 1); + drop(registry.close()); + assert_eq!(registry.active_lora_requests(), 0); + assert_eq!( + registry.lora_adapter_states(), + (adapter_names(&[]), adapter_names(&[])) + ); + } + + #[test] + fn registry_drops_lora_tracking_on_abort() { + let mut registry = RequestRegistry::new(&[connected_engine(EngineId::from(b"engine-0"))]); + registry + .register("req-lora".to_string(), Some("adapter-a".to_string()), None) + .unwrap(); + + drop(registry.finish_many(&["req-lora".to_string()])); + + assert_eq!( + registry.lora_adapter_states(), + (adapter_names(&[]), adapter_names(&[])) + ); + } + + #[test] + fn registry_closes_all_requests_on_failure() { + let mut registry = RequestRegistry::new(&[connected_engine(EngineId::from(b"engine-0"))]); + registry.register("req-1".to_string(), None, None).unwrap(); + registry.register("req-2".to_string(), None, None).unwrap(); + + let senders = registry.close(); + + assert_eq!(senders.len(), 2); + assert!(registry.is_closed()); + } + + #[test] + fn registry_tracks_engine_id_per_request() { + let engine_0 = EngineId::from_engine_index(0); + let engine_1 = EngineId::from_engine_index(1); + let mut registry = RequestRegistry::new(&[ + connected_engine(engine_0.clone()), + connected_engine(engine_1.clone()), + ]); + let (chosen_0, _) = registry.register("req-1".to_string(), None, None).unwrap(); + let (chosen_1, _) = registry.register("req-2".to_string(), None, None).unwrap(); + let (chosen_0_again, _) = registry.register("req-3".to_string(), None, None).unwrap(); + + assert_eq!(chosen_0, engine_0); + assert_eq!(chosen_1, engine_1); + assert_eq!(chosen_0_again, engine_0); + + let grouped = registry.abortable_request_ids(&[ + "req-1".to_string(), + "req-2".to_string(), + "req-3".to_string(), + ]); + assert_eq!( + grouped.get(&engine_0).unwrap(), + &vec!["req-1".to_string(), "req-3".to_string()] + ); + assert_eq!(grouped.get(&engine_1).unwrap(), &vec!["req-2".to_string()]); + } + + #[test] + fn registry_uses_inflight_as_waiting_fallback_before_stats_arrive() { + let engine_0 = EngineId::from_engine_index(0); + let engine_1 = EngineId::from_engine_index(1); + let mut registry = RequestRegistry::new(&[ + connected_engine(engine_0.clone()), + connected_engine(engine_1.clone()), + ]); + + let (chosen_0, _) = registry.register("req-1".to_string(), None, None).unwrap(); + let (chosen_1, _) = registry.register("req-2".to_string(), None, None).unwrap(); + let (chosen_0_again, _) = registry.register("req-3".to_string(), None, None).unwrap(); + + assert_eq!(chosen_0, engine_0); + assert_eq!(chosen_1, engine_1); + assert_eq!(chosen_0_again, engine_0); + } + + #[test] + fn routing_score_uses_inflight_before_stats_arrive() { + let state = EngineRoutingState { + inflight: 3, + last_scheduler_stats: None, + }; + + assert_eq!(state.routing_score(), 3); + } + + #[test] + fn routing_score_uses_inflight_as_scheduler_stats_lower_bound() { + let state = EngineRoutingState { + inflight: 7, + last_scheduler_stats: Some(EngineLoadSnapshot { + waiting: 0, + running: 2, + }), + }; + + assert_eq!(state.routing_score(), 7); + } + + #[test] + fn routing_score_keeps_extra_waiting_penalty() { + let state = EngineRoutingState { + inflight: 1, + last_scheduler_stats: Some(EngineLoadSnapshot { + waiting: 3, + running: 2, + }), + }; + + assert_eq!(state.routing_score(), 14); + } + + #[test] + fn registry_prefers_real_scheduler_stats_over_inflight() { + let engine_0 = EngineId::from_engine_index(0); + let engine_1 = EngineId::from_engine_index(1); + let mut registry = RequestRegistry::new(&[ + connected_engine(engine_0.clone()), + connected_engine(engine_1.clone()), + ]); + + assert!(registry.apply_scheduler_counts( + 0, + EngineLoadSnapshot { + waiting: 3, + running: 2 + } + )); + assert!(registry.apply_scheduler_counts( + 1, + EngineLoadSnapshot { + waiting: 0, + running: 1 + } + )); + + let (chosen, _) = registry.register("req-stats".to_string(), None, None).unwrap(); + assert_eq!(chosen, engine_1); + } + + #[test] + fn register_with_data_parallel_rank_routes_to_specified_engine() { + let engine_0 = EngineId::from_engine_index(0); + let engine_1 = EngineId::from_engine_index(1); + let engine_2 = EngineId::from_engine_index(2); + let mut registry = RequestRegistry::new(&[ + connected_engine(engine_0.clone()), + connected_engine(engine_1.clone()), + connected_engine(engine_2.clone()), + ]); + + // Explicitly target rank 2 (third engine). + let (chosen, _) = registry.register("req-1".to_string(), None, Some(2)).unwrap(); + assert_eq!(chosen, engine_2); + + // Explicitly target rank 0 (first engine). + let (chosen, _) = registry.register("req-2".to_string(), None, Some(0)).unwrap(); + assert_eq!(chosen, engine_0); + + // Explicitly target rank 1. + let (chosen, _) = registry.register("req-3".to_string(), None, Some(1)).unwrap(); + assert_eq!(chosen, engine_1); + } + + #[test] + fn register_with_data_parallel_rank_bypasses_load_balancing() { + let engine_0 = EngineId::from_engine_index(0); + let engine_1 = EngineId::from_engine_index(1); + let mut registry = RequestRegistry::new(&[ + connected_engine(engine_0.clone()), + connected_engine(engine_1.clone()), + ]); + + // Load-balance: first two go to engine_0 and engine_1. + registry.register("req-lb-0".to_string(), None, None).unwrap(); + + // Now engine_0 has 1 in-flight. Without dp_rank, next would go to engine_1. + // But with dp_rank=0, it should still go to engine_0. + let (chosen, _) = registry.register("req-dp".to_string(), None, Some(0)).unwrap(); + assert_eq!(chosen, engine_0); + } + + #[test] + fn register_with_out_of_range_rank_returns_error() { + let mut registry = RequestRegistry::new(&[ + connected_engine(EngineId::from_engine_index(0)), + connected_engine(EngineId::from_engine_index(1)), + ]); + + let error = registry.register("req-1".to_string(), None, Some(2)).unwrap_err(); + assert!(matches!( + error, + crate::error::Error::InvalidDataParallelRank { + rank: 2, + num_engines: 2, + } + )); + } + + #[test] + fn register_with_rank_on_single_engine_only_accepts_zero() { + let engine_0 = EngineId::from_engine_index(0); + let mut registry = RequestRegistry::new(&[connected_engine(engine_0.clone())]); + + let (chosen, _) = registry.register("req-ok".to_string(), None, Some(0)).unwrap(); + assert_eq!(chosen, engine_0); + + let error = registry.register("req-bad".to_string(), None, Some(1)).unwrap_err(); + assert!(matches!( + error, + crate::error::Error::InvalidDataParallelRank { + rank: 1, + num_engines: 1, + } + )); + } + + #[test] + fn utility_registry_tracks_and_removes_call_ids() { + let mut registry = UtilityRegistry::default(); + let (call_id_1, _) = registry.allocate_and_register(); + let (call_id_2, _) = registry.allocate_and_register(); + + assert_eq!(call_id_1, 1); + assert_eq!(call_id_2, 2); + assert!(registry.contains(1)); + assert!(registry.contains(2)); + assert!(registry.resolve(&1).is_some()); + assert!(!registry.contains(1)); + assert!(registry.contains(2)); + } + + #[test] + fn utility_registry_closes_all_waiters_on_failure() { + let mut registry = UtilityRegistry::default(); + registry.allocate_and_register(); + registry.allocate_and_register(); + + let senders = registry.close(); + + assert_eq!(senders.len(), 2); + assert!(!registry.contains(1)); + assert!(!registry.contains(2)); + assert!(registry.is_closed()); + } + + #[test] + fn utility_registry_unregister_many_drops_pending_calls() { + use tokio::sync::oneshot::error::TryRecvError; + + let mut registry = UtilityRegistry::default(); + let (call_id_1, mut rx_1) = registry.allocate_and_register(); + let (call_id_2, mut rx_2) = registry.allocate_and_register(); + let (call_id_3, _rx_3) = registry.allocate_and_register(); + + // Drop two of the three allocated calls; the third stays pending. + registry.unregister_many([call_id_1, call_id_2]); + + assert!(!registry.contains(call_id_1)); + assert!(!registry.contains(call_id_2)); + assert!(registry.contains(call_id_3)); + // The receivers must observe the sender being dropped (channel closed). + assert!(matches!(rx_1.try_recv(), Err(TryRecvError::Closed))); + assert!(matches!(rx_2.try_recv(), Err(TryRecvError::Closed))); + } + + #[test] + fn utility_registry_unregister_many_ignores_unknown_call_ids() { + let mut registry = UtilityRegistry::default(); + let (call_id, _rx) = registry.allocate_and_register(); + + // Unknown call ids are silently ignored — caller doesn't care which were live. + registry.unregister_many([call_id, 42, 9999]); + + assert!(!registry.contains(call_id)); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client/stream.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client/stream.rs new file mode 100644 index 0000000000000000000000000000000000000000..30a2aad10cacb3828ac342dfbb27ccb7e1dfc431 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client/stream.rs @@ -0,0 +1,153 @@ +use std::ops::Deref; +use std::pin::Pin; +use std::task::{Context, Poll}; + +use futures::Stream; +use futures::stream::FusedStream; +use thiserror_ext::AsReport as _; +use tokio::sync::mpsc; +use tracing::{debug, error, warn}; + +use crate::client::AbortRequest; +use crate::client::state::OutputReceiver; +use crate::protocol::output::{EngineCoreFinishReason, EngineCoreOutput}; +use crate::{AbortCause, Error, Result}; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum State { + Running, + Finished, + ClosedWithError, + UnexpectedClose, +} + +/// One request-scoped engine-core output plus the enclosing batch metadata. +#[derive(Debug, Clone, PartialEq)] +pub struct EngineCoreStreamOutput { + pub engine_index: u32, + pub timestamp: f64, + pub output: EngineCoreOutput, +} + +impl Deref for EngineCoreStreamOutput { + type Target = EngineCoreOutput; + + fn deref(&self) -> &Self::Target { + &self.output + } +} + +/// Stream of raw engine-core outputs for one request. +/// +/// The stream yields only [`EngineCoreStreamOutput`] values whose embedded +/// output `request_id` matches the originating `add_request()` call. Normal +/// request completion is expected to include a final output object whose +/// `finish_reason` is non-`None`. +pub struct EngineCoreOutputStream { + request_id: String, + abort_tx: mpsc::UnboundedSender, + state: State, + rx: OutputReceiver, +} + +impl EngineCoreOutputStream { + pub(crate) fn new( + request_id: String, + abort_tx: mpsc::UnboundedSender, + rx: OutputReceiver, + ) -> Self { + Self { + request_id, + abort_tx, + state: State::Running, + rx, + } + } + + /// Return the engine-core `request_id` bound to this stream. + pub fn request_id(&self) -> &str { + &self.request_id + } +} + +impl Stream for EngineCoreOutputStream { + type Item = Result; + + fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + if self.is_terminated() { + return Poll::Ready(None); + } + + match Pin::new(&mut self.rx).poll_recv(cx) { + Poll::Pending => Poll::Pending, + Poll::Ready(Some(item)) => { + match &item { + Ok(output) => { + // If the output indicates the request is finished, mark the stream as + // terminated with cleanly-finished state and expect no more outputs to + // come. + if output.finished() { + if output.finish_reason == Some(EngineCoreFinishReason::Error) { + error!( + self.request_id, + "request failed with an internal error during generation" + ); + } + debug!(self.request_id, "request completed via final output"); + self.state = State::Finished; + } + } + Err(error) => { + // If we get an error from the output stream, mark the stream as terminated + // with an error. + warn!(self.request_id, error = %error.as_report(), "request encountered an error"); + self.state = State::ClosedWithError; + } + } + Poll::Ready(Some(item)) + } + Poll::Ready(None) => { + // If we get a `None` without seeing a finished output, this is an unexpected + // close from the engine side. Mark the stream as terminated + // with an unexpected close state and send an error down the + // stream to notify the caller. + warn!(self.request_id, "request stream closed unexpectedly"); + self.state = State::UnexpectedClose; + + Poll::Ready(Some(Err(Error::RequestStreamClosed { + request_id: self.request_id.clone(), + }))) + } + } + } +} + +impl FusedStream for EngineCoreOutputStream { + fn is_terminated(&self) -> bool { + !matches!(self.state, State::Running) + } +} + +impl Drop for EngineCoreOutputStream { + fn drop(&mut self) { + if self.is_terminated() { + // If it's terminated, it means that the request either finished cleanly, or + // encountered an error or unexpected close from the engine. In any + // case, the request stream is already considered inactive and + // there's no need to abort it on the engine side. + return; + } + + let abort_req = AbortRequest { + request_id: self.request_id.clone(), + cause: AbortCause::current(), + }; + + if self.abort_tx.send(abort_req).is_err() { + warn!( + request_id = self.request_id, + "auto-abort worker already shut down; skip auto-abort" + ); + } + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/bootstrap.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/bootstrap.rs new file mode 100644 index 0000000000000000000000000000000000000000..943fdc8fa2e24d043b15414229a8bde556e343a2 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/bootstrap.rs @@ -0,0 +1,91 @@ +use std::time::Duration; + +use bytes::Bytes; +use zeromq::prelude::{Socket, SocketRecv, SocketSend}; +use zeromq::{PullSocket, XPubSocket, ZmqMessage}; + +use crate::error::{Error, Result, bail_unexpected_handshake_message}; + +/// Engine-facing sockets owned by the in-process coordinator. +pub(crate) struct CoordinatorBootstrap { + pub input_address: String, + pub output_address: String, + pub input_socket: XPubSocket, + pub output_socket: PullSocket, +} + +impl CoordinatorBootstrap { + /// Bind the engine-facing coordinator sockets on the given host. + pub(crate) async fn bind(local_host: &str) -> Result { + let mut input_socket = XPubSocket::new(); + let input_address = input_socket.bind(&format!("tcp://{local_host}:0")).await?.to_string(); + + let mut output_socket = PullSocket::new(); + let output_address = + output_socket.bind(&format!("tcp://{local_host}:0")).await?.to_string(); + + Ok(Self { + input_address, + output_address, + input_socket, + output_socket, + }) + } + + /// Complete the engine-facing startup gate before engines are allowed to + /// send handshake READY. + pub(crate) async fn wait_for_startup_gate( + &mut self, + engine_count: usize, + ready_timeout: Duration, + ) -> Result<()> { + wait_for_engine_subscriptions(&mut self.input_socket, engine_count, ready_timeout).await?; + send_ready_to_engines(&mut self.input_socket).await?; + Ok(()) + } +} + +/// Wait until all engines subscribe to the coordinator broadcast socket. +async fn wait_for_engine_subscriptions( + input_socket: &mut XPubSocket, + engine_count: usize, + ready_timeout: Duration, +) -> Result<()> { + let mut received = 0; + while received < engine_count { + let message = + tokio::time::timeout(ready_timeout, input_socket.recv()).await.map_err(|_| { + Error::HandshakeTimeout { + stage: "coordinator engine subscriptions", + timeout: ready_timeout, + } + })??; + if message.len() != 1 { + bail_unexpected_handshake_message!( + "expected 1 frame for coordinator subscription, got {}", + message.len() + ); + } + + let frame = message + .into_vec() + .into_iter() + .next() + .expect("single-frame coordinator subscription message"); + if frame.as_ref() != [0x01] { + bail_unexpected_handshake_message!( + "expected coordinator subscription frame [0x01], got {:?}", + frame.as_ref() + ); + } + received += 1; + } + + Ok(()) +} + +/// Send the coordinator READY marker to all subscribed engines. +async fn send_ready_to_engines(input_socket: &mut XPubSocket) -> Result<()> { + input_socket.send(ZmqMessage::from(Bytes::from_static(b"READY"))).await?; + Ok(()) +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/external.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/external.rs new file mode 100644 index 0000000000000000000000000000000000000000..6e2e066e8ac16cbe0d3699f681b7bfd71d315af7 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/external.rs @@ -0,0 +1,167 @@ +use std::sync::Arc; + +use serde_tuple::{Deserialize_tuple, Serialize_tuple}; +use thiserror_ext::AsReport; +use tokio::sync::mpsc; +use tracing::{debug, warn}; +use zeromq::prelude::{SocketRecv, SocketSend}; +use zeromq::{XSubSocket, ZmqMessage}; + +use crate::client::imp::ClientInner; +use crate::coordinator::handle::{CoordinatorCommand, CoordinatorState}; +use crate::error::{Error, Result, bail_unexpected_coordinator_output}; +use crate::protocol::{OpaqueValue, decode_msgpack, encode_msgpack}; + +/// Frontend-to-coordinator wakeup message sent when the first request arrives +/// while all engines are paused. +/// +/// This matches the frontend-side msgpack tuple sent by Python +/// `DPAsyncMPClient._ensure_stats_update_task` to the coordinator front socket. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, PartialEq, Eq, Serialize_tuple, Deserialize_tuple)] +struct CoordinatorWakeupMessage { + /// Engine index that already has the triggering request and should be + /// excluded from the coordinator's `START_DP_WAVE` rebroadcast. + exclude_engine_index: u32, + /// DP wave number observed by the frontend when the request was admitted. + wave: u32, +} + +/// Coordinator-to-frontend state publish received on the front-side coordinator +/// socket. +/// +/// This matches the msgpack tuple periodically published by Python +/// `DPCoordinatorProc.run_coordinator` to all connected frontends. +/// +/// Original Python definitions: +/// +/// +#[derive(Debug, Clone, PartialEq, Deserialize_tuple)] +struct CoordinatorStateUpdate { + /// Global per-engine request counts published by the coordinator. + /// + /// The Rust bootstrapped external-coordinator path preserves this field for + /// wire compatibility but intentionally ignores it for routing decisions. + counts: OpaqueValue, + /// Current global DP wave number stamped onto newly admitted requests. + wave: u32, + /// Whether engines are currently running (`true`) or paused (`false`). + engines_running: bool, +} + +/// Background half of an external Python-owned coordinator connection. +/// +/// This owns the command receiver and one frontend-facing XSUB socket. It +/// mirrors the subset of Python's coordinator protocol needed by the Rust +/// bootstrapped frontend: receive `(counts, wave, running)` publishes, ignore +/// `counts`, and send `(exclude_engine_index, wave)` wakeup messages when the +/// first request arrives while engines are paused. +pub(crate) struct ExternalCoordinatorService { + state: Arc, + command_rx: mpsc::UnboundedReceiver, + socket: XSubSocket, +} + +impl ExternalCoordinatorService { + pub(super) fn new( + state: Arc, + command_rx: mpsc::UnboundedReceiver, + socket: XSubSocket, + ) -> Self { + Self { + state, + command_rx, + socket, + } + } + + /// Apply one frontend-originated command to the external coordinator state + /// machine. + async fn handle_command(&mut self, command: CoordinatorCommand) -> Result<()> { + match command { + CoordinatorCommand::FirstRequest { + target_engine_id, + wave, + } => { + let target_engine_index = target_engine_id.engine_index().ok_or_else(|| { + Error::UnsupportedCoordinatorEngineId { + engine_id: target_engine_id.to_vec(), + } + })?; + debug!( + wave, + exclude_engine_index = target_engine_index, + "notifying external coordinator about first request while engines were paused" + ); + let payload = encode_msgpack(&CoordinatorWakeupMessage { + exclude_engine_index: target_engine_index, + wave, + })?; + self.socket.send(ZmqMessage::from(payload)).await?; + } + } + Ok(()) + } + + /// Apply one publish received from the xsub socket containing a coordinator + /// state update. + async fn handle_publish(&mut self, message: ZmqMessage) -> Result<()> { + let frames = message.into_vec(); + if frames.len() != 1 { + bail_unexpected_coordinator_output!( + "received malformed external coordinator publish with {} frame(s)", + frames.len() + ); + } + + let update: CoordinatorStateUpdate = decode_msgpack(&frames[0])?; + + let mut state = self.state.lock(); + let previous_wave = state.current_wave; + let previous_engines_running = state.engines_running; + state.current_wave = update.wave; + state.engines_running = update.engines_running; + debug!( + previous_wave, + wave = update.wave, + previous_engines_running, + engines_running = update.engines_running, + "applied external coordinator state update" + ); + Ok(()) + } + + /// Drive the coordinator event loop until either side of the control plane + /// is closed or a fatal error is observed. + pub(crate) async fn run(mut self, inner: Arc) { + let result: Result<()> = async { + loop { + tokio::select! { + // Received frontend-originated command from the handle. + command = self.command_rx.recv() => { + let Some(command) = command else { + warn!("external coordinator command channel closed, shutting down service"); + return Ok(()); + }; + self.handle_command(command).await?; + } + // Received publish from the external coordinator socket. + publish = self.socket.recv() => { + let publish = publish.map_err(Error::from)?; + self.handle_publish(publish).await?; + } + } + } + } + .await; + let Err(error) = result else { return }; + + warn!( + error = %error.as_report(), + "external coordinator service exiting with error" + ); + inner.close_registries(Arc::new(error)); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/handle.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/handle.rs new file mode 100644 index 0000000000000000000000000000000000000000..415f180c6dbed559db7b1b23650fead6877ee6a3 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/handle.rs @@ -0,0 +1,144 @@ +use std::sync::Arc; + +use parking_lot::Mutex; +use tokio::sync::mpsc; +use zeromq::prelude::Socket; +use zeromq::{XPubSocket, XSubSocket}; + +use crate::coordinator::external::ExternalCoordinatorService; +use crate::coordinator::inproc::InProcCoordinatorRunner; +use crate::error::{Error, Result, bail_control_closed}; +use crate::transport::EngineId; + +/// Snapshot to the coordinator state for request routing and stamping. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct CoordinatorStateSnapshot { + /// The current DP wave, which will be stamped on outgoing requests. + pub current_wave: u32, + /// Whether the engines are currently running or paused, which determines if + /// the frontend must trigger a new wave on the next request. + pub engines_running: bool, +} + +impl CoordinatorStateSnapshot { + /// Resume the engines for a `FirstRequest` and return the wave to broadcast + /// and the engine to exclude from the wakeup. + /// + /// The request may have been stamped with a `request_wave` older than + /// `current_wave` if a `WaveComplete` advanced it after the command was + /// enqueued. Such a request still needs serving, so the current wave is + /// broadcast to every engine (`exclude = None`); the wave is never rewound. + /// A non-stale request excludes the engine that already received it. Mirrors + /// the Python coordinator's front-end path. + pub(crate) fn start_wave_for_first_request( + &mut self, + request_wave: u32, + target_engine_index: u32, + ) -> (u32, Option) { + self.engines_running = true; + let exclude = (request_wave >= self.current_wave).then_some(target_engine_index); + (self.current_wave, exclude) + } +} + +/// Shared in-process coordinator state. +pub(crate) type CoordinatorState = Mutex; + +/// Commands sent from the frontend request path into the background runner. +#[derive(Debug)] +pub(crate) enum CoordinatorCommand { + /// The first request arrived while all engines were paused. + /// + /// The coordinator should broadcast `START_DP_WAVE` with the current wave + /// and the target engine index as the excluded engine. + FirstRequest { + target_engine_id: EngineId, + wave: u32, + }, +} + +/// Frontend-facing coordinator handle used by `EngineCoreClient::call()`. +/// +/// This side stays intentionally small: it can read the latest wave snapshot +/// and enqueue a `FirstRequest` transition when the request path observes the +/// system in the paused state. +#[derive(Clone)] +pub(crate) struct CoordinatorHandle { + state: Arc, + command_tx: mpsc::UnboundedSender, +} + +impl CoordinatorHandle { + fn new_parts() -> ( + Self, + Arc, + mpsc::UnboundedReceiver, + ) { + let state = Arc::new(Mutex::new(CoordinatorStateSnapshot { + current_wave: 0, + engines_running: false, + })); + let (command_tx, command_rx) = mpsc::unbounded_channel(); + ( + Self { + state: state.clone(), + command_tx, + }, + state, + command_rx, + ) + } + + /// Build the paired frontend handle and background runner around one + /// engine-facing coordinator broadcast socket. + pub(crate) fn new_inproc(coordinator_input: XPubSocket) -> (Self, InProcCoordinatorRunner) { + let (handle, state, command_rx) = Self::new_parts(); + ( + handle, + InProcCoordinatorRunner::new(state, command_rx, coordinator_input), + ) + } + + /// Build the paired frontend handle and background service around an + /// external Python-owned frontend-side coordinator socket. + pub(crate) async fn connect_external( + coordinator_address: &str, + ) -> Result<(Self, ExternalCoordinatorService)> { + let (handle, state, command_rx) = Self::new_parts(); + let mut socket = XSubSocket::new(); + socket.connect(coordinator_address).await?; + socket.subscribe("").await?; + Ok(( + handle, + ExternalCoordinatorService::new(state, command_rx, socket), + )) + } + + /// Snapshot the coordinator state for request routing and stamping. + pub(crate) fn snapshot(&self) -> CoordinatorStateSnapshot { + *self.state.lock() + } + + /// Notify the runner that a new request arrived while engines were paused. + /// + /// The handle flips `engines_running` optimistically so concurrent request + /// submissions coalesce behind one `START_DP_WAVE` broadcast instead of all + /// trying to trigger the wave independently. + pub(crate) fn notify_first_request(&self, target_engine_id: EngineId) -> Result<()> { + let mut state = self.state.lock(); + if state.engines_running { + return Ok(()); + } + + let command = CoordinatorCommand::FirstRequest { + target_engine_id, + wave: state.current_wave, + }; + if self.command_tx.send(command).is_err() { + bail_control_closed!("in-process coordinator command channel already shut down"); + } + + state.engines_running = true; + Ok(()) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/inproc.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/inproc.rs new file mode 100644 index 0000000000000000000000000000000000000000..ce0bcde16d23ac1dbd1b11a25a1fdeb840d3bb8c --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/inproc.rs @@ -0,0 +1,257 @@ +use std::sync::Arc; + +use serde_tuple::{Deserialize_tuple, Serialize_tuple}; +use thiserror_ext::AsReport; +use tokio::sync::mpsc; +use tracing::{debug, warn}; +use zeromq::prelude::SocketSend; +use zeromq::{XPubSocket, ZmqMessage}; + +use crate::client::imp::ClientInner; +use crate::coordinator::handle::{CoordinatorCommand, CoordinatorState}; +use crate::error::{Error, Result, bail_unexpected_coordinator_output}; +use crate::protocol::encode_msgpack; +use crate::protocol::output::{DpControlMessage, DpControlOutput, EngineCoreOutputs}; +use crate::protocol::request::EngineCoreRequestType; + +/// Coordinator-to-engine `START_DP_WAVE` control payload encoded on the +/// engine-facing coordinator socket. +/// +/// This matches the msgpack tuple broadcast by Python +/// `DPCoordinatorProc._send_start_wave`. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, PartialEq, Eq, Serialize_tuple, Deserialize_tuple)] +struct StartDpWaveMessage { + /// DP wave number that all engines should start processing. + wave: u32, + /// Engine index that already received the triggering request and so does not + /// need an extra wakeup. `None` wakes every engine (used when the triggering + /// request was for a stale wave). + exclude_engine_index: Option, +} + +/// Background half of the in-process coordinator. +/// +/// This owns the command receiver and the engine-facing coordinator input +/// socket. It is the single place where wave transitions are serialized and +/// where `START_DP_WAVE` broadcasts are emitted. +pub(crate) struct InProcCoordinatorRunner { + state: Arc, + command_rx: mpsc::UnboundedReceiver, + coordinator_input: XPubSocket, +} + +impl InProcCoordinatorRunner { + pub(super) fn new( + state: Arc, + command_rx: mpsc::UnboundedReceiver, + coordinator_input: XPubSocket, + ) -> Self { + Self { + state, + command_rx, + coordinator_input, + } + } + + /// Broadcast Python-compatible `START_DP_WAVE` to all connected engines. + async fn broadcast_start_wave( + &mut self, + wave: u32, + exclude_engine_index: Option, + ) -> Result<()> { + let payload = encode_msgpack(&StartDpWaveMessage { + wave, + exclude_engine_index, + })?; + self.coordinator_input + .send( + ZmqMessage::try_from(vec![ + EngineCoreRequestType::StartDpWave.to_frame(), + payload.into(), + ]) + .expect("coordinator START_DP_WAVE message must contain two frames"), + ) + .await?; + Ok(()) + } + + /// Apply one frontend-originated command to the coordinator state machine. + async fn handle_command(&mut self, command: CoordinatorCommand) -> Result<()> { + match command { + CoordinatorCommand::FirstRequest { + target_engine_id, + wave, + } => { + let target_engine_index = target_engine_id.engine_index().ok_or_else(|| { + Error::UnsupportedCoordinatorEngineId { + engine_id: target_engine_id.to_vec(), + } + })?; + let (current_wave, exclude) = { + let mut state = self.state.lock(); + state.start_wave_for_first_request(wave, target_engine_index) + }; + debug!( + current_wave, + request_wave = wave, + ?exclude, + "starting DP wave after first request while engines were paused" + ); + self.broadcast_start_wave(current_wave, exclude).await?; + } + } + Ok(()) + } + + /// Apply one engine-originated control output to the coordinator state + /// machine. + async fn handle_outputs(&mut self, outputs: EngineCoreOutputs) -> Result<()> { + match outputs { + EngineCoreOutputs::RequestBatch(batch) + if batch.outputs.is_empty() && batch.finished_requests.is_none() => + { + // Stats-only output for coordinator. + // Ignore since the Rust coordinator doesn't track stats for + // routing decisions. + } + EngineCoreOutputs::DpControl(DpControlOutput { + engine_index, + control, + .. + }) => match control { + // The engines signals they completed the current wave and are now paused. + // Advance the current wave and mark the state as paused. + DpControlMessage::WaveComplete(wave) => { + let mut state = self.state.lock(); + if wave >= state.current_wave { + let next_wave = wave + 1; + debug!( + wave, + next_wave, + "DP wave finished; pausing engines and advancing coordinator state" + ); + state.current_wave = wave + 1; + state.engines_running = false; + } + } + // An engine requests to start the wave. + // Rebroadcast the wave to all engines except for the originated one. + DpControlMessage::StartWave(wave) => { + let should_broadcast = { + let mut state = self.state.lock(); + if wave > state.current_wave + || (wave == state.current_wave && !state.engines_running) + { + state.current_wave = wave; + state.engines_running = true; + true + } else { + false + } + }; + if should_broadcast { + debug!( + wave, + exclude_engine_index = engine_index, + "starting DP wave after stale-wave notification from engine" + ); + self.broadcast_start_wave(wave, Some(engine_index)).await?; + } + } + }, + other => { + bail_unexpected_coordinator_output!( + "received non-control output on coordinator path: {other:?}" + ); + } + } + Ok(()) + } + + /// Drive the coordinator event loop until either side of the control plane + /// is closed or a fatal error is observed. + /// + /// Any fatal error closes the main client registries so request streams and + /// future calls observe a stable shutdown cause. + pub(crate) async fn run( + mut self, + mut output_rx: mpsc::Receiver>, + inner: Arc, + ) { + let result: Result<()> = async { + loop { + tokio::select! { + // Received frontend-originated command from the handle. + command = self.command_rx.recv() => { + let Some(command) = command else { + warn!("coordinator command channel closed, shutting down coordinator runner"); + return Ok(()); + }; + self.handle_command(command).await?; + } + // Received engine-originated control output from the coordinator socket. + outputs = output_rx.recv() => { + let Some(outputs) = outputs else { + warn!("coordinator output channel closed, shutting down coordinator runner"); + return Ok(()); + }; + self.handle_outputs(outputs?).await?; + } + } + } + } + .await; + let Err(error) = result else { return }; + + warn!(error = %error.as_report(), "coordinator runner exiting with error"); + inner.close_registries(Arc::new(error)); + } +} + +#[cfg(test)] +mod tests { + use crate::coordinator::handle::CoordinatorStateSnapshot; + + /// A `FirstRequest` for the current wave starts that wave and excludes the + /// engine that already received the triggering request. + #[test] + fn first_request_for_current_wave_excludes_target() { + let mut state = CoordinatorStateSnapshot { + current_wave: 3, + engines_running: false, + }; + + let (wave, exclude) = state.start_wave_for_first_request(3, 2); + + assert_eq!(wave, 3); + assert_eq!(exclude, Some(2)); + assert!(state.engines_running); + assert_eq!(state.current_wave, 3); + } + + /// A `FirstRequest` whose wave was superseded by a racing `WaveComplete` + /// (`request_wave < current_wave`) must still start the request's wave: it + /// broadcasts the current wave and wakes every engine (`exclude = None`) + /// rather than rewinding the wave or dropping the request. + #[test] + fn stale_first_request_starts_current_wave_for_all_engines() { + let mut state = CoordinatorStateSnapshot { + current_wave: 4, + engines_running: false, + }; + + // Request stamped with wave 3 while the coordinator already advanced to 4. + let (wave, exclude) = state.start_wave_for_first_request(3, 2); + + assert_eq!( + wave, 4, + "must broadcast the current wave, not the stale one" + ); + assert_eq!(exclude, None, "a stale request must wake every engine"); + assert!(state.engines_running); + assert_eq!(state.current_wave, 4, "wave must not be rewound"); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/mod.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..d3a8d6c7b751e2c9d53b3aec60b6c34e55e8dae9 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/coordinator/mod.rs @@ -0,0 +1,7 @@ +mod bootstrap; +mod external; +mod handle; +mod inproc; + +pub(crate) use bootstrap::CoordinatorBootstrap; +pub(crate) use handle::CoordinatorHandle; diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/error.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/error.rs new file mode 100644 index 0000000000000000000000000000000000000000..f9d4318e14a28059f5065929384c9a216b4ecfbb --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/error.rs @@ -0,0 +1,96 @@ +use std::sync::Arc; +use std::time::Duration; + +use thiserror::Error; +use thiserror_ext::Macro; + +use crate::protocol::utility::UtilityCallId; + +pub type Result = std::result::Result; + +/// Public error type for the Rust engine-core client. +#[derive(Debug, Error, Macro)] +pub enum Error { + #[error("messagepack encode failed for {target_type}: {message}")] + Encode { + target_type: &'static str, + message: String, + }, + #[error("messagepack decode failed for {target_type}: {message}")] + Decode { + target_type: &'static str, + message: String, + }, + #[error("messagepack value decode failed")] + ValueDecode(#[from] rmpv::decode::Error), + #[error("messagepack ext value decode failed: {message}")] + ExtValueDecode { message: String }, + #[error("invalid structured outputs params: {message}")] + InvalidStructuredOutputsParams { message: String }, + #[error("io error")] + Io(#[from] std::io::Error), + #[error("transport error")] + Transport(#[from] zeromq::ZmqError), + #[error("ZMQ runtime task failed")] + ZmqRuntimeTask(#[from] tokio::task::JoinError), + #[error("engine core reported fatal failure")] + EngineCoreDead, + #[error("startup handshake timed out while waiting for {stage} after {timeout:?}")] + HandshakeTimeout { + stage: &'static str, + timeout: Duration, + }, + #[error("engine input registration timed out after {timeout:?}")] + InputRegistrationTimeout { timeout: Duration }, + #[error("unexpected engine id in startup handshake: expected {expected:?}, got {actual:?}")] + UnexpectedHandshakeIdentity { expected: Vec, actual: Vec }, + #[error("unexpected startup handshake message: {message}")] + UnexpectedHandshakeMessage { message: String }, + #[error("unexpected non-control output on coordinator path: {message}")] + UnexpectedCoordinatorOutput { message: String }, + #[error("unexpected output on main dispatcher path: {message}")] + UnexpectedDispatcherOutput { message: String }, + #[error("coordinator requires a Python-compatible two-byte engine id, got {engine_id:?}")] + UnsupportedCoordinatorEngineId { engine_id: Vec }, + #[error("unsupported auxiliary frame(s): expected 1 frame, got {frame_count}")] + UnsupportedAuxFrames { frame_count: usize }, + #[error("external coordinator mode is not implemented yet")] + UnsupportedExternalCoordinator, + #[error("unsupported field `{field}` in {context}")] + UnsupportedField { + context: &'static str, + field: &'static str, + }, + #[error("engine control channel closed unexpectedly: {message}")] + ControlClosed { message: String }, + #[error("request `{request_id}` is already in flight")] + DuplicateRequestId { request_id: String }, + #[error("data parallel rank {rank} is out of range for {num_engines} engine(s)")] + InvalidDataParallelRank { rank: u32, num_engines: u32 }, + #[error("engine-core output dispatcher closed: {message}")] + DispatcherClosed { message: String }, + #[error("engine-core client is closed: {message}")] + ClientClosed { message: String }, + #[error("request output stream for `{request_id}` closed unexpectedly")] + RequestStreamClosed { request_id: String }, + #[error("utility call `{method}` failed (call_id={call_id}): {message}")] + UtilityCallFailed { + method: String, + call_id: UtilityCallId, + message: String, + }, + #[error("utility call `{method}` returned an invalid result (call_id={call_id}): {message}")] + UtilityResultDecode { + method: String, + call_id: UtilityCallId, + message: String, + }, + #[error("utility call `{method}` closed unexpectedly (call_id={call_id})")] + UtilityCallClosed { method: String, call_id: u64 }, + #[error("utility call `{method}` returned inconsistent results across engines: {values}")] + InconsistentUtilityResults { method: String, values: String }, + + /// A special variant to allow cloning the same error. + #[error(transparent)] + Shared(Arc), +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/lib.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/lib.rs new file mode 100644 index 0000000000000000000000000000000000000000..570095c44a32bd1fd8ff37105f8f6e0624018a1b --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/lib.rs @@ -0,0 +1,20 @@ +mod client; +mod coordinator; +mod error; +mod metrics; +pub mod mock_engine; +pub mod protocol; +pub mod runtime; +#[cfg(any(test, feature = "test-util"))] +pub mod test_utils; +mod transport; + +pub use client::{ + AbortCause, CoordinatorMode, EngineCoreClient, EngineCoreClientConfig, EngineCoreOutputStream, + EngineCoreStreamOutput, TransportMode, +}; +pub use error::{Error, Result}; +pub use transport::{ENGINE_CORE_DEAD_SENTINEL, EngineId}; + +#[cfg(test)] +mod tests; diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/metrics.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/metrics.rs new file mode 100644 index 0000000000000000000000000000000000000000..639c310d2aa66e509a764f593e0779ed8006106a --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/metrics.rs @@ -0,0 +1,259 @@ +use std::collections::BTreeSet; +use std::time::{SystemTime, UNIX_EPOCH}; + +use vllm_metrics::{ + EngineLabels, EnginePositionLabels, LoraAdapterNames, LoraInfoLabels, SchedulerMetrics, + WaitingReasonLabels, +}; + +use crate::protocol::stats::SchedulerStats; + +const WAITING_REASON_CAPACITY: &str = "capacity"; +const WAITING_REASON_DEFERRED: &str = "deferred"; + +/// Record the scheduler-stats-backed metrics for one engine at one point in +/// time. +pub(crate) fn record_scheduler_stats( + metrics: &SchedulerMetrics, + model_name: impl Into, + engine: u32, + stats: &SchedulerStats, +) { + let model_name = model_name.into(); + let labels = EngineLabels { + model_name: model_name.clone(), + engine, + }; + + // Scheduler state gauges. + metrics.scheduler_running.get_or_create(&labels).set(stats.num_running_reqs); + metrics + .scheduler_waiting + .get_or_create(&labels) + .set(stats.num_waiting_reqs + stats.num_skipped_waiting_reqs); + metrics + .scheduler_waiting_by_reason + .get_or_create(&WaitingReasonLabels { + model_name: model_name.clone(), + engine, + reason: WAITING_REASON_CAPACITY, + }) + .set(stats.num_waiting_reqs); + metrics + .scheduler_waiting_by_reason + .get_or_create(&WaitingReasonLabels { + model_name: model_name.clone(), + engine, + reason: WAITING_REASON_DEFERRED, + }) + .set(stats.num_skipped_waiting_reqs); + metrics.kv_cache_usage.get_or_create(&labels).set(stats.kv_cache_usage); + + // Prefix-cache counters, including the connector-backed external cache path. + metrics + .prefix_cache_queries + .get_or_create(&labels) + .inc_by(stats.prefix_cache_stats.base.queries); + metrics + .prefix_cache_hits + .get_or_create(&labels) + .inc_by(stats.prefix_cache_stats.base.hits); + + if let Some(connector_prefix_cache_stats) = &stats.connector_prefix_cache_stats { + metrics + .external_prefix_cache_queries + .get_or_create(&labels) + .inc_by(connector_prefix_cache_stats.base.queries); + metrics + .external_prefix_cache_hits + .get_or_create(&labels) + .inc_by(connector_prefix_cache_stats.base.hits); + } + + // Speculative decoding counters. + if let Some(spec_decoding_stats) = &stats.spec_decoding_stats { + metrics + .spec_decode_num_drafts + .get_or_create(&labels) + .inc_by(spec_decoding_stats.num_drafts); + metrics + .spec_decode_num_draft_tokens + .get_or_create(&labels) + .inc_by(spec_decoding_stats.num_draft_tokens); + metrics + .spec_decode_num_accepted_tokens + .get_or_create(&labels) + .inc_by(spec_decoding_stats.num_accepted_tokens); + metrics.log_stats.get_or_create(&labels).observe_spec_decode( + spec_decoding_stats.num_drafts, + &spec_decoding_stats.num_accepted_tokens_per_pos, + ); + + for (position, accepted_tokens) in + spec_decoding_stats.num_accepted_tokens_per_pos.iter().copied().enumerate() + { + metrics + .spec_decode_num_accepted_tokens_per_pos + .get_or_create(&EnginePositionLabels { + model_name: model_name.clone(), + engine, + position: position as u32, + }) + .inc_by(accepted_tokens); + } + } + + // Per-engine performance / MFU counters. + if let Some(perf_stats) = &stats.perf_stats + && (perf_stats.num_flops_per_gpu != 0 + || perf_stats.num_read_bytes_per_gpu != 0 + || perf_stats.num_write_bytes_per_gpu != 0) + { + metrics + .estimated_flops_per_gpu + .get_or_create(&labels) + .inc_by(perf_stats.num_flops_per_gpu); + metrics + .estimated_read_bytes_per_gpu + .get_or_create(&labels) + .inc_by(perf_stats.num_read_bytes_per_gpu); + metrics + .estimated_write_bytes_per_gpu + .get_or_create(&labels) + .inc_by(perf_stats.num_write_bytes_per_gpu); + } + + if let Some(cudagraph_stats) = &stats.cudagraph_stats { + metrics.log_stats.get_or_create(&labels).observe_cudagraph( + cudagraph_stats.num_unpadded_tokens, + cudagraph_stats.num_padded_tokens, + cudagraph_stats.num_paddings, + &cudagraph_stats.runtime_mode, + ); + } + + // Sampled KV-cache residency histograms. + if !stats.kv_cache_eviction_events.is_empty() { + let kv_block_lifetime_seconds = metrics.kv_block_lifetime_seconds.get_or_create(&labels); + let kv_block_idle_before_evict_seconds = + metrics.kv_block_idle_before_evict_seconds.get_or_create(&labels); + let kv_block_reuse_gap_seconds = metrics.kv_block_reuse_gap_seconds.get_or_create(&labels); + + for event in &stats.kv_cache_eviction_events { + kv_block_lifetime_seconds.observe(event.lifetime_seconds); + kv_block_idle_before_evict_seconds.observe(event.idle_seconds); + for reuse_gap_seconds in &event.reuse_gaps_seconds { + kv_block_reuse_gap_seconds.observe(*reuse_gap_seconds); + } + } + } +} + +/// Exports `vllm:lora_requests_info` as a single series covering all LoRA +/// requests tracked by this client across every engine in the replica. +/// +/// The engine's `SchedulerStats` never carries adapter names: the Python +/// frontend fills them in from per-request lifecycle events tracked by +/// `LoRARequestStates` in `vllm/v1/engine/output_processor.py`. The Rust +/// frontend mirrors that, deriving the sets from the request registry. +#[derive(Default)] +pub(crate) struct LoraInfoExporter { + current: Option, +} + +impl LoraInfoExporter { + pub(crate) fn update( + &mut self, + metrics: &SchedulerMetrics, + running: BTreeSet, + waiting: BTreeSet, + ) { + let next = (!running.is_empty() || !waiting.is_empty()).then_some(LoraInfoLabels { + running_lora_adapters: LoraAdapterNames(running), + waiting_lora_adapters: LoraAdapterNames(waiting), + }); + + if self.current != next + && let Some(prev) = &self.current + { + metrics.lora_info.remove(prev); + } + + // Python sets this gauge to the current time on every record. + if let Some(labels) = &next { + metrics.lora_info.get_or_create(labels).set(now_unix_secs()); + } + + self.current = next; + } +} + +fn now_unix_secs() -> f64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|d| d.as_secs_f64()) + .unwrap_or(0.0) +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeSet; + + use expect_test::expect; + use vllm_metrics::Metrics; + + use crate::metrics::LoraInfoExporter; + + fn names(values: &[&str]) -> BTreeSet { + values.iter().map(|name| (*name).to_string()).collect() + } + + /// The `lora_requests_info` series with the non-deterministic timestamp + /// value replaced by ``, one line per series. + fn lora_series(rendered: &str) -> String { + rendered + .lines() + .filter(|l| l.starts_with("vllm:lora_requests_info{")) + .map(|l| match l.rsplit_once("} ") { + Some((labels, _value)) => format!("{labels}}} "), + None => l.to_string(), + }) + .collect::>() + .join("\n") + } + + #[test] + fn lora_info_emits_clears_stale_and_drains() { + let metrics = Metrics::new(); + let mut exporter = LoraInfoExporter::default(); + + // No adapters: nothing emitted. + exporter.update(&metrics.scheduler, names(&[]), names(&[])); + expect![[""]].assert_eq(&lora_series(&metrics.render().unwrap())); + + // Two running (sorted), one waiting. + exporter.update(&metrics.scheduler, names(&["b", "a"]), names(&["c"])); + expect![[ + r#"vllm:lora_requests_info{running_lora_adapters="a,b",waiting_lora_adapters="c"} "# + ]] + .assert_eq(&lora_series(&metrics.render().unwrap())); + + // "c" gets scheduled and "d" arrives: the stale series is replaced. + exporter.update(&metrics.scheduler, names(&["a", "b", "c"]), names(&["d"])); + expect![[ + r#"vllm:lora_requests_info{running_lora_adapters="a,b,c",waiting_lora_adapters="d"} "# + ]] + .assert_eq(&lora_series(&metrics.render().unwrap())); + + // Everything but "d" finishes. + exporter.update(&metrics.scheduler, names(&["d"]), names(&[])); + expect![[ + r#"vllm:lora_requests_info{running_lora_adapters="d",waiting_lora_adapters=""} "# + ]] + .assert_eq(&lora_series(&metrics.render().unwrap())); + + // All requests done: series removed entirely. + exporter.update(&metrics.scheduler, names(&[]), names(&[])); + expect![[""]].assert_eq(&lora_series(&metrics.render().unwrap())); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/mock_engine.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/mock_engine.rs new file mode 100644 index 0000000000000000000000000000000000000000..1912aec8cac3fe5dd5c478c0d34396a708170354 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/mock_engine.rs @@ -0,0 +1,274 @@ +use std::path::Path; +use std::time::Duration; + +use tokio::time::timeout; +use zeromq::prelude::{Socket, SocketRecv, SocketSend}; +use zeromq::util::PeerIdentity; +use zeromq::{DealerSocket, PushSocket, SocketOptions, SubSocket, ZmqMessage}; + +use crate::EngineId; +use crate::error::{Error, Result, bail_unexpected_handshake_message}; +use crate::protocol::dtype::ModelDtype; +use crate::protocol::handshake::{EngineCoreReadyResponse, HandshakeInitMessage, ReadyMessage}; +use crate::protocol::{decode_msgpack, encode_msgpack}; + +/// Default model length advertised by reusable mock engine helpers. +pub const DEFAULT_MOCK_MAX_MODEL_LEN: u64 = 1024 * 1024; +/// Default KV block count advertised by reusable mock engine helpers. +pub const DEFAULT_MOCK_NUM_GPU_BLOCKS: u64 = 0; +/// Default KV block size (tokens per block) +pub const DEFAULT_MOCK_BLOCK_SIZE: u64 = 16; + +/// Startup behavior for one mock engine joining a frontend. +#[derive(Debug, Clone)] +pub struct MockEngineConfig { + /// Whether the engine should advertise itself as local to the frontend. + pub local: bool, + /// Whether the engine should advertise itself as headless. + pub headless: bool, + /// Engine-ready payload reported after INIT, including max model length, + /// KV block count, and dtype. + pub ready_response: EngineCoreReadyResponse, + /// Maximum time to wait for IPC endpoints to appear before connecting. + pub connect_timeout: Duration, +} + +impl Default for MockEngineConfig { + fn default() -> Self { + Self { + local: false, + headless: true, + ready_response: default_ready_response(), + connect_timeout: Duration::from_secs(5), + } + } +} + +/// Construct the ready response used by the standalone mock engine CLI. +pub fn default_ready_response() -> EngineCoreReadyResponse { + EngineCoreReadyResponse { + max_model_len: DEFAULT_MOCK_MAX_MODEL_LEN, + num_gpu_blocks: DEFAULT_MOCK_NUM_GPU_BLOCKS, + block_size: DEFAULT_MOCK_BLOCK_SIZE, + dp_stats_address: None, + dtype: ModelDtype::Float32, + vllm_version: "test-vllm-version".to_string(), + world_size: 1, + data_parallel_size: 1, + kv_cache_size_tokens: None, + kv_cache_max_concurrency: None, + } +} + +/// Coordinator-side sockets used by one mock engine when coordinator mode +/// is enabled. +pub struct MockCoordinatorSockets { + /// Subscription socket that receives coordinator broadcasts such as + /// `START_DP_WAVE`. + pub input_sub: SubSocket, + /// Push socket used to send coordinator-only `EngineCoreOutputs` back to + /// the frontend. + pub output_push: PushSocket, +} + +/// One mock engine's connection to one frontend client. +/// +/// vLLM launches one engine-client pair per API server process. A remote +/// engine connects to every advertised input/output pair and uses the request's +/// `client_index` to route outputs back to the originating API server. +pub struct MockEngineDataSockets { + /// Socket used to receive frontend requests. + pub dealer: DealerSocket, + /// Socket used to publish normal request outputs back to the frontend. + pub push: PushSocket, +} + +/// Frontend-facing sockets owned by one mock engine. +pub struct MockEngineSockets { + /// Decoded INIT message sent by the frontend during handshake. + pub init: HandshakeInitMessage, + /// Data sockets for all frontend clients in client-index order. + /// + /// For Rust frontend this will always be one socket, while for Python frontend + /// this may be multiple sockets if there are multiple API server processes. + pub data_sockets: Vec, + /// Optional coordinator sockets when the client enabled the in-process + /// coordinator. + pub coordinator: Option, +} + +/// Build a HELLO or READY handshake status payload. +fn ready_message(status: &str, config: &MockEngineConfig) -> ReadyMessage { + ReadyMessage { + status: Some(status.to_string()), + local: Some(config.local), + headless: Some(config.headless), + parallel_config_hash: None, + } +} + +/// Convert an engine id into a ZMQ DEALER identity. +fn peer_identity(engine_id: impl Into) -> Result { + let engine_id = engine_id.into(); + PeerIdentity::try_from(engine_id.clone()).map_err(|error| Error::UnexpectedHandshakeMessage { + message: format!( + "invalid mock engine identity {:?}: {error}", + engine_id.to_vec() + ), + }) +} + +/// Wait for an IPC endpoint path to appear before attempting to connect. +async fn wait_for_ipc_endpoint(endpoint: &str, connect_timeout: Duration) -> Result<()> { + let Some(socket_path) = endpoint.strip_prefix("ipc://") else { + return Ok(()); + }; + + timeout(connect_timeout, async { + while !Path::new(socket_path).exists() { + tokio::time::sleep(Duration::from_millis(20)).await; + } + }) + .await + .map_err(|_| Error::HandshakeTimeout { + stage: "mock engine IPC endpoint", + timeout: connect_timeout, + }) +} + +/// Encode the engine-ready response sent on input socket registration. +fn ready_response_payload(config: &MockEngineConfig) -> Result> { + encode_msgpack(&config.ready_response) +} + +/// Join a frontend-owned handshake endpoint and open mock engine sockets. +pub async fn connect_to_frontend( + engine_handshake: impl AsRef, + engine_id: impl Into, + config: MockEngineConfig, +) -> Result { + let engine_handshake = engine_handshake.as_ref(); + wait_for_ipc_endpoint(engine_handshake, config.connect_timeout).await?; + + let peer_identity = peer_identity(engine_id)?; + let mut options = SocketOptions::default(); + options.peer_identity(peer_identity.clone()); + let mut handshake = DealerSocket::with_options(options); + handshake.connect(engine_handshake).await?; + handshake + .send(ZmqMessage::from(encode_msgpack(&ready_message( + "HELLO", &config, + ))?)) + .await?; + + let init_frames = handshake.recv().await?.into_vec(); + if init_frames.len() != 1 { + bail_unexpected_handshake_message!( + "expected one INIT frame from frontend, got {}", + init_frames.len() + ); + } + let init: HandshakeInitMessage = decode_msgpack(init_frames[0].as_ref())?; + + if init.addresses.inputs.is_empty() { + return Err(Error::UnexpectedHandshakeMessage { + message: "frontend INIT did not include an input address".to_string(), + }); + } + if init.addresses.inputs.len() != init.addresses.outputs.len() { + return Err(Error::UnexpectedHandshakeMessage { + message: format!( + "frontend INIT input/output address count mismatch: {} inputs, {} outputs", + init.addresses.inputs.len(), + init.addresses.outputs.len() + ), + }); + } + + let mut data_sockets = Vec::with_capacity(init.addresses.inputs.len()); + for (input_address, output_address) in + init.addresses.inputs.iter().zip(init.addresses.outputs.iter()) + { + wait_for_ipc_endpoint(input_address, config.connect_timeout).await?; + wait_for_ipc_endpoint(output_address, config.connect_timeout).await?; + + let mut input_options = SocketOptions::default(); + input_options.peer_identity(peer_identity.clone()); + let mut dealer = DealerSocket::with_options(input_options); + dealer.connect(input_address).await?; + dealer.send(ZmqMessage::from(ready_response_payload(&config)?)).await?; + + let mut push = PushSocket::new(); + push.connect(output_address).await?; + + data_sockets.push(MockEngineDataSockets { dealer, push }); + } + + let coordinator = match ( + init.addresses.coordinator_input.as_deref(), + init.addresses.coordinator_output.as_deref(), + ) { + (Some(coordinator_input), Some(coordinator_output)) => { + let mut input_sub = SubSocket::new(); + input_sub.connect(coordinator_input).await?; + input_sub.subscribe("").await?; + + let mut output_push = PushSocket::new(); + output_push.connect(coordinator_output).await?; + + let ready = input_sub.recv().await?.into_vec(); + if ready.len() != 1 || ready[0].as_ref() != b"READY" { + bail_unexpected_handshake_message!( + "expected coordinator READY marker, got {:?}", + ready + ); + } + + Some(MockCoordinatorSockets { + input_sub, + output_push, + }) + } + (None, None) => None, + _ => bail_unexpected_handshake_message!( + "coordinator handshake addresses must be both present or both absent" + ), + }; + + handshake + .send(ZmqMessage::from(encode_msgpack(&ready_message( + "READY", &config, + ))?)) + .await?; + + Ok(MockEngineSockets { + init, + data_sockets, + coordinator, + }) +} + +/// Join already-bootstrapped frontend input/output sockets directly. +pub async fn connect_to_bootstrapped_frontend( + input_address: impl AsRef, + output_address: impl AsRef, + engine_id: impl Into, + config: MockEngineConfig, +) -> Result<(DealerSocket, PushSocket)> { + let input_address = input_address.as_ref(); + let output_address = output_address.as_ref(); + wait_for_ipc_endpoint(input_address, config.connect_timeout).await?; + wait_for_ipc_endpoint(output_address, config.connect_timeout).await?; + + let peer_identity = peer_identity(engine_id)?; + let mut input_options = SocketOptions::default(); + input_options.peer_identity(peer_identity); + let mut dealer = DealerSocket::with_options(input_options); + dealer.connect(input_address).await?; + dealer.send(ZmqMessage::from(ready_response_payload(&config)?)).await?; + + let mut push = PushSocket::new(); + push.connect(output_address).await?; + + Ok((dealer, push)) +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/dtype.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/dtype.rs new file mode 100644 index 0000000000000000000000000000000000000000..100184da86322441ddb7504b0e49600a45c44ff0 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/dtype.rs @@ -0,0 +1,40 @@ +use serde::{Deserialize, Serialize}; + +/// Effective model dtype reported by the engine after config resolution. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum ModelDtype { + #[serde(rename = "float16")] + Float16, + #[serde(rename = "bfloat16")] + BFloat16, + #[serde(rename = "float32")] + Float32, +} + +impl ModelDtype { + pub fn as_str(self) -> &'static str { + match self { + Self::Float16 => "float16", + Self::BFloat16 => "bfloat16", + Self::Float32 => "float32", + } + } +} + +#[cfg(test)] +mod tests { + use super::ModelDtype; + + #[test] + fn serde_uses_protocol_dtype_strings() { + assert_eq!( + serde_json::to_value(ModelDtype::Float16).unwrap(), + serde_json::json!("float16") + ); + assert_eq!( + serde_json::from_value::(serde_json::json!("bfloat16")).unwrap(), + ModelDtype::BFloat16 + ); + assert_eq!(ModelDtype::Float32.as_str(), "float32"); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/handshake.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/handshake.rs new file mode 100644 index 0000000000000000000000000000000000000000..ef481cc2def0671ba005fe7adc273ab5503ea7a3 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/handshake.rs @@ -0,0 +1,81 @@ +use std::collections::BTreeMap; + +use serde::{Deserialize, Serialize}; + +use crate::protocol::OpaqueValue; +use crate::protocol::dtype::ModelDtype; + +/// Decoded engine startup-handshake payload sent on the handshake socket. +/// +/// Original Python payload construction: +/// +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct ReadyMessage { + #[serde(default)] + pub status: Option, + #[serde(default)] + pub local: Option, + #[serde(default)] + pub headless: Option, + #[serde(default)] + pub parallel_config_hash: Option, +} + +/// Post-initialization configuration sent from each engine on the input socket +/// registration message, after the handshake completes. +/// +/// Contains values that may differ from the original config (e.g. +/// `max_model_len` after KV cache auto-fitting, `num_gpu_blocks` after +/// profiling). +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct EngineCoreReadyResponse { + /// Engine-reported maximum model context length (auto-fitted after + /// KV cache profiling and may differ from the original config value). + pub max_model_len: u64, + /// Number of GPU blocks available for KV cache on this engine. + pub num_gpu_blocks: u64, + /// KV cache block size (tokens per block). + pub block_size: u64, + /// DP coordinator stats publish address, if applicable. + pub dp_stats_address: Option, + /// Effective model dtype after Python vLLM resolves `--dtype`. + pub dtype: ModelDtype, + /// Python vLLM version reported by the engine process. + pub vllm_version: String, + /// World size (TP * PP) from the parallel config. + pub world_size: u64, + /// Data parallelism size from the parallel config. + pub data_parallel_size: u64, + /// Total KV cache capacity in tokens, if reported. + pub kv_cache_size_tokens: Option, + /// Maximum achievable request concurrency given the KV cache, if reported. + pub kv_cache_max_concurrency: Option, +} + +/// Frontend-owned ZMQ addresses that are sent to the engine during startup +/// handshake initialization. +/// +/// Original Python definition (`EngineZmqAddresses`): +/// +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct HandshakeAddresses { + pub inputs: Vec, + pub outputs: Vec, + pub coordinator_input: Option, + pub coordinator_output: Option, + pub frontend_stats_publish_address: Option, +} + +/// Startup handshake payload sent from the frontend to initialize an engine +/// after receiving `HELLO`. +/// +/// Original Python definition (`EngineHandshakeMetadata`): +/// +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct HandshakeInitMessage { + pub addresses: HandshakeAddresses, + pub parallel_config: BTreeMap, +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/logprobs.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/logprobs.rs new file mode 100644 index 0000000000000000000000000000000000000000..2346dfc40bf4c225aff11f5ee7bfa7e3f7dcec5e --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/logprobs.rs @@ -0,0 +1,285 @@ +mod array; +#[cfg(test)] +mod tests; +mod wire; + +use std::ops::{Deref, DerefMut}; + +use enum_as_inner::EnumAsInner; +use serde::{Deserialize, Deserializer, Serialize}; + +use self::wire::*; +use crate::error::{Error, Result, bail_ext_value_decode}; +use crate::protocol::tensor::{WireArrayData, WireNdArray}; + +/// One token candidate and its logprob metadata for a single sequence position. +/// +/// The first entry in a [`PositionLogprobs`] is always the sampled/selected +/// token for that position. Any remaining entries follow the engine's returned +/// top-k candidate order. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct TokenLogprob { + pub token_id: u32, + pub logprob: f32, + /// The sampled/selected token uses its actual vocab rank. Remaining entries + /// use 1-based top-k ranks matching the engine's returned candidate + /// order. + pub rank: u32, +} + +/// Logprob payload for one sequence position. +/// +/// This is the semantic Rust representation used by the public client API after +/// the lower-level ndarray/tensor wire payload has been decoded. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct PositionLogprobs { + pub entries: Vec, +} + +impl PositionLogprobs { + /// Convert one decoded logprobs row into this per-position form by grouping + /// each token/logprob pair together with the sampled/selected token's + /// actual vocab rank. + fn from_decoded_row(token_ids: &[u32], logprobs: &[f32], sampled_rank: u32) -> Result { + if token_ids.len() != logprobs.len() { + bail_ext_value_decode!( + "logprobs row length mismatch: token_ids={}, logprobs={}", + token_ids.len(), + logprobs.len() + ); + } + if sampled_rank == 0 { + bail_ext_value_decode!("token_ranks must be >= 1 for decoded engine-core logprobs"); + } + + let mut entries = Vec::with_capacity(token_ids.len()); + for (index, (&token_id, &logprob)) in token_ids.iter().zip(logprobs.iter()).enumerate() { + let rank = if index == 0 { + sampled_rank + } else { + index as u32 + }; + entries.push(TokenLogprob { + token_id, + logprob, + rank, + }); + } + Ok(Self { entries }) + } +} + +/// Decoded per-request logprobs payload for one engine-core output. +/// +/// Unlike the Python wire payload, this public Rust type is already fully +/// semantic: one [`PositionLogprobs`] per scored position, each containing the +/// sampled/selected token plus any returned top-k alternatives for that same +/// position. +/// +/// The Python engine still sends logprobs as ndarray/tensor-shaped wire tuples. +/// Rust resolves that lower-level representation during decode and exposes only +/// this per-position form to callers. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct Logprobs { + /// One decoded logprobs record per scored position in this engine-core + /// output. + pub positions: Vec, +} + +impl Logprobs { + /// Returns the number of scored positions in this payload. + pub fn len(&self) -> usize { + self.positions.len() + } + + /// Returns whether the payload contains no scored positions. + pub fn is_empty(&self) -> bool { + self.positions.is_empty() + } +} + +/// Output field wrapper that is initially deserialized from the Python wire +/// shape, then resolved into [`Logprobs`] before the decoded message is +/// returned to callers. +#[derive(Clone, PartialEq, Debug, EnumAsInner)] +pub enum MaybeWireLogprobs { + /// The logprobs are still in the wire format and need to be resolved by + /// looking up aux frames and decoding raw views. Should only be used + /// internally during deserialization. + Wire(Box), + /// The actual decoded logprobs value, + Direct(Logprobs), +} + +impl Deref for MaybeWireLogprobs { + type Target = Logprobs; + + fn deref(&self) -> &Self::Target { + match self { + Self::Wire(_) => panic!("Logprobs is still in wire format"), + Self::Direct(value) => value, + } + } +} + +impl DerefMut for MaybeWireLogprobs { + fn deref_mut(&mut self) -> &mut Self::Target { + match self { + Self::Wire(_) => panic!("Logprobs is still in wire format"), + Self::Direct(value) => value, + } + } +} + +impl<'de> Deserialize<'de> for MaybeWireLogprobs { + fn deserialize(deserializer: D) -> std::result::Result + where + D: Deserializer<'de>, + { + // When deserializing, it's always in the wire form. + WireLogprobs::deserialize(deserializer).map(|v| Self::Wire(Box::new(v))) + } +} + +impl Serialize for MaybeWireLogprobs { + fn serialize(&self, serializer: S) -> std::result::Result + where + S: serde::Serializer, + { + // For testing purposes only. We don't actually serialize it into aux frames. + match self { + Self::Wire(value) => value.serialize(serializer), + Self::Direct(value) => WireLogprobs::from_direct(value) + .map_err(serde::ser::Error::custom)? + .serialize(serializer), + } + } +} + +impl MaybeWireLogprobs { + /// Resolve the wire representation into decoded logprobs by looking up aux + /// frames and decoding raw views as needed. + pub(super) fn resolve(self, frames: &[Frame], field_prefix: &str) -> Result + where + Frame: AsRef<[u8]>, + { + match self { + Self::Direct(value) => Ok(Self::Direct(value)), + Self::Wire(value) => value.resolve(frames, field_prefix).map(Self::Direct), + } + } +} + +impl WireLogprobs { + /// Convert semantic per-position logprobs into the Python wire tuple shape. + /// + /// This exists mainly so Rust-side tests can inject semantic logprobs into + /// mocked engine-core outputs without manually building ndarray + /// raw-view tuples. + fn from_direct(value: &Logprobs) -> std::result::Result { + let rows = value.positions.len(); + let cols = value.positions.first().map(|position| position.entries.len()).unwrap_or(0); + + let mut token_ids = Vec::with_capacity(rows.saturating_mul(cols).saturating_mul(8)); + let mut logprobs = Vec::with_capacity(rows.saturating_mul(cols).saturating_mul(4)); + let mut token_ranks = Vec::with_capacity(rows.saturating_mul(8)); + + for (row_index, position) in value.positions.iter().enumerate() { + if position.entries.len() != cols { + return Err(format!( + "logprobs row {row_index} length mismatch: expected {cols}, got {}", + position.entries.len() + )); + } + let Some((sampled, _)) = position.entries.split_first() else { + return Err(format!("logprobs row {row_index} is empty")); + }; + + token_ranks.extend_from_slice(&(sampled.rank as i64).to_le_bytes()); + for entry in &position.entries { + token_ids.extend_from_slice(&(entry.token_id as i64).to_le_bytes()); + logprobs.extend_from_slice(&entry.logprob.to_le_bytes()); + } + } + + Ok(Self { + logprob_token_ids: WireNdArray { + dtype: "(self, frames: &[Frame], field_prefix: &str) -> Result + where + Frame: AsRef<[u8]>, + { + if let Some(indices) = self.cu_num_generated_tokens { + bail_ext_value_decode!( + "{field_prefix}.cu_num_generated_tokens: \ + expected None for per-request engine-core logprobs payload, got {indices:?}" + ); + } + + let token_ids = array::decode_array2_u32( + self.logprob_token_ids, + &format!("{field_prefix}.logprob_token_ids"), + frames, + )?; + let logprobs = + array::decode_array2_f32(self.logprobs, &format!("{field_prefix}.logprobs"), frames)?; + let token_ranks = array::decode_array1_u32( + self.token_ranks, + &format!("{field_prefix}.token_ranks"), + frames, + )?; + + if token_ids.rows != logprobs.rows || token_ids.cols != logprobs.cols { + bail_ext_value_decode!( + "{field_prefix}: row shape mismatch between token ids ({}, {}) and logprobs ({}, {})", + token_ids.rows, + token_ids.cols, + logprobs.rows, + logprobs.cols + ); + } + if token_ids.rows != token_ranks.len() { + bail_ext_value_decode!( + "{field_prefix}: token_ranks length {} does not match row count {}", + token_ranks.len(), + token_ids.rows + ); + } + + let mut positions = Vec::with_capacity(token_ids.rows); + for ((token_ids_row, logprobs_row), sampled_rank) in token_ids + .data + .chunks(token_ids.cols) + .zip(logprobs.data.chunks(logprobs.cols)) + .zip(token_ranks) + { + positions.push(PositionLogprobs::from_decoded_row( + token_ids_row, + logprobs_row, + sampled_rank, + )?); + } + + Ok(Logprobs { positions }) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/logprobs/array.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/logprobs/array.rs new file mode 100644 index 0000000000000000000000000000000000000000..505a09f78f9d25539f620767ae724dee8e34faf0 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/logprobs/array.rs @@ -0,0 +1,306 @@ +use std::io::Cursor; + +use byteorder::{BigEndian, LittleEndian, NativeEndian, ReadBytesExt}; +use itertools::Itertools as _; + +use crate::error::{Error, Result, ext_value_decode}; +use crate::protocol::tensor::{ShapeExt as _, WireArrayData, WireNdArray}; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum ScalarType { + I32, + I64, + F32, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum Endianness { + Little, + Big, + Native, +} + +#[derive(Debug, Clone, PartialEq)] +pub(super) struct DecodedArray2 { + pub rows: usize, + pub cols: usize, + pub data: Vec, +} + +pub(super) fn decode_array2_u32( + value: WireNdArray, + field: &str, + frames: &[Frame], +) -> Result> +where + Frame: AsRef<[u8]>, +{ + let (shape, bytes, scalar, endianness) = + decode_array_metadata(value, field, frames, &[ScalarType::I32, ScalarType::I64])?; + if shape.len() != 2 { + return Err(decode_error( + field, + &format!("expected rank-2 array, got rank {}", shape.len()), + )); + } + + let data = match scalar { + ScalarType::I32 => decode_i32_vec(&bytes, endianness, field)? + .into_iter() + .map(|value| convert_to_u32(value, field)) + .try_collect()?, + ScalarType::I64 => decode_i64_vec(&bytes, endianness, field)? + .into_iter() + .map(|value| convert_to_u32(value, field)) + .try_collect()?, + ScalarType::F32 => unreachable!("scalar validation should reject f32"), + }; + Ok(DecodedArray2 { + rows: shape[0], + cols: shape[1], + data, + }) +} + +pub(super) fn decode_array1_u32( + value: WireNdArray, + field: &str, + frames: &[Frame], +) -> Result> +where + Frame: AsRef<[u8]>, +{ + let (shape, bytes, scalar, endianness) = + decode_array_metadata(value, field, frames, &[ScalarType::I32, ScalarType::I64])?; + if shape.len() != 1 { + return Err(decode_error( + field, + &format!("expected rank-1 array, got rank {}", shape.len()), + )); + } + + let data = match scalar { + ScalarType::I32 => decode_i32_vec(&bytes, endianness, field)? + .into_iter() + .map(|value| convert_to_u32(value, field)) + .try_collect()?, + ScalarType::I64 => decode_i64_vec(&bytes, endianness, field)? + .into_iter() + .map(|value| convert_to_u32(value, field)) + .try_collect()?, + ScalarType::F32 => unreachable!("scalar validation should reject f32"), + }; + Ok(data) +} + +pub(super) fn decode_array2_f32( + value: WireNdArray, + field: &str, + frames: &[Frame], +) -> Result> +where + Frame: AsRef<[u8]>, +{ + let (shape, bytes, _, endianness) = + decode_array_metadata(value, field, frames, &[ScalarType::F32])?; + if shape.len() != 2 { + return Err(decode_error( + field, + &format!("expected rank-2 array, got rank {}", shape.len()), + )); + } + + let data = decode_f32_vec(&bytes, endianness, field)?; + Ok(DecodedArray2 { + rows: shape[0], + cols: shape[1], + data, + }) +} + +pub(super) fn decode_array_metadata( + value: WireNdArray, + field: &str, + frames: &[Frame], + expected_scalars: &[ScalarType], +) -> Result<(Vec, Vec, ScalarType, Endianness)> +where + Frame: AsRef<[u8]>, +{ + let WireNdArray { dtype, shape, data } = value; + let (scalar, endianness) = parse_dtype(&dtype, field)?; + if !expected_scalars.contains(&scalar) { + return Err(decode_error( + field, + &format!("expected dtype in {:?}, got {}", expected_scalars, dtype), + )); + } + + let bytes = resolve_array_bytes(data, field, frames)?; + validate_byte_length(shape.as_slice(), bytes.len(), field, scalar)?; + Ok((shape, bytes, scalar, endianness)) +} + +pub(super) fn parse_dtype(dtype: &str, field: &str) -> Result<(ScalarType, Endianness)> { + let (endianness, body) = match dtype.as_bytes().first().copied() { + Some(b'<') => (Endianness::Little, &dtype[1..]), + Some(b'>') => (Endianness::Big, &dtype[1..]), + Some(b'=') => (Endianness::Native, &dtype[1..]), + Some(b'|') => (Endianness::Native, &dtype[1..]), + _ => (Endianness::Native, dtype), + }; + + let scalar = match body { + "i4" | "int32" => ScalarType::I32, + "i8" | "int64" => ScalarType::I64, + "f4" | "float32" => ScalarType::F32, + _ => { + return Err(decode_error( + field, + &format!("unsupported dtype string {dtype:?}"), + )); + } + }; + Ok((scalar, endianness)) +} + +pub(super) fn resolve_array_bytes( + value: WireArrayData, + field: &str, + frames: &[Frame], +) -> Result> +where + Frame: AsRef<[u8]>, +{ + match value { + WireArrayData::RawView(bytes) => Ok(bytes), + WireArrayData::AuxIndex(index) => { + let frame = frames.get(index).ok_or_else(|| { + decode_error( + field, + &format!( + "aux frame index {index} out of range for {} frames", + frames.len() + ), + ) + })?; + Ok(frame.as_ref().to_vec()) + } + } +} + +pub(super) fn validate_byte_length( + shape: &[usize], + byte_len: usize, + field: &str, + scalar: ScalarType, +) -> Result<()> { + let element_count = shape + .checked_numel() + .ok_or_else(|| decode_error(field, "shape element count overflowed usize"))?; + let element_size = match scalar { + ScalarType::I32 | ScalarType::F32 => 4, + ScalarType::I64 => 8, + }; + let expected = element_count + .checked_mul(element_size) + .ok_or_else(|| decode_error(field, "byte length overflowed usize"))?; + if expected != byte_len { + return Err(decode_error( + field, + &format!("byte length mismatch: expected {expected}, got {byte_len}"), + )); + } + Ok(()) +} + +pub(super) fn decode_i32_vec( + bytes: &[u8], + endianness: Endianness, + field: &str, +) -> Result> { + if !bytes.len().is_multiple_of(4) { + return Err(decode_error( + field, + &format!("byte length {} is not divisible by 4", bytes.len()), + )); + } + let mut cursor = Cursor::new(bytes); + let mut values = Vec::with_capacity(bytes.len() / 4); + while (cursor.position() as usize) < bytes.len() { + let value = match endianness { + Endianness::Little => cursor.read_i32::(), + Endianness::Big => cursor.read_i32::(), + Endianness::Native => cursor.read_i32::(), + } + .map_err(|error| decode_error(field, &format!("failed to read i32 payload: {error}")))?; + values.push(value); + } + Ok(values) +} + +pub(super) fn decode_f32_vec( + bytes: &[u8], + endianness: Endianness, + field: &str, +) -> Result> { + if !bytes.len().is_multiple_of(4) { + return Err(decode_error( + field, + &format!("byte length {} is not divisible by 4", bytes.len()), + )); + } + let mut cursor = Cursor::new(bytes); + let mut values = Vec::with_capacity(bytes.len() / 4); + while (cursor.position() as usize) < bytes.len() { + let value = match endianness { + Endianness::Little => cursor.read_f32::(), + Endianness::Big => cursor.read_f32::(), + Endianness::Native => cursor.read_f32::(), + } + .map_err(|error| decode_error(field, &format!("failed to read f32 payload: {error}")))?; + values.push(value); + } + Ok(values) +} + +pub(super) fn decode_i64_vec( + bytes: &[u8], + endianness: Endianness, + field: &str, +) -> Result> { + if !bytes.len().is_multiple_of(8) { + return Err(decode_error( + field, + &format!("byte length {} is not divisible by 8", bytes.len()), + )); + } + let mut cursor = Cursor::new(bytes); + let mut values = Vec::with_capacity(bytes.len() / 8); + while (cursor.position() as usize) < bytes.len() { + let value = match endianness { + Endianness::Little => cursor.read_i64::(), + Endianness::Big => cursor.read_i64::(), + Endianness::Native => cursor.read_i64::(), + } + .map_err(|error| decode_error(field, &format!("failed to read i64 payload: {error}")))?; + values.push(value); + } + Ok(values) +} + +fn convert_to_u32(value: I, field: &str) -> Result +where + I: TryInto + std::fmt::Display + Copy, +{ + value.try_into().map_err(|_| { + decode_error( + field, + &format!("expected non-negative token id/rank that fits in u32, got {value}"), + ) + }) +} + +pub(super) fn decode_error(field: &str, reason: &str) -> Error { + ext_value_decode!("{field}: {reason}") +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/logprobs/tests.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/logprobs/tests.rs new file mode 100644 index 0000000000000000000000000000000000000000..6f5528a0ac6a036b32ef3d08039f47b2d89224ca --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/logprobs/tests.rs @@ -0,0 +1,302 @@ +use std::collections::BTreeSet; + +use bytes::Bytes; +use rmpv::Value; + +use super::{Logprobs, PositionLogprobs, TokenLogprob}; +use crate::protocol::output::{EngineCoreFinishReason, decode_engine_core_outputs}; + +fn encode_value(value: &Value) -> Vec { + let mut out = Vec::new(); + rmpv::encode::write_value(&mut out, value).unwrap(); + out +} + +fn output_wire_with_custom_fields( + new_logprobs: Option, + prompt_logprobs: Option, +) -> Value { + Value::Array(vec![ + Value::from(0), + Value::Array(vec![Value::Array(vec![ + Value::from("req-1"), + Value::Array(vec![Value::from(7), Value::from(8)]), + new_logprobs.unwrap_or(Value::Nil), + prompt_logprobs.unwrap_or(Value::Nil), + Value::Nil, + Value::from(EngineCoreFinishReason::Length as u8), + ])]), + Value::Nil, + Value::from(0.0), + Value::Nil, + Value::Array(vec![Value::from("req-1")]), + ]) +} + +fn ndarray_value(dtype: &str, shape: &[usize], data: Value) -> Value { + Value::Array(vec![ + Value::from(dtype), + Value::Array(shape.iter().copied().map(Value::from).collect()), + data, + ]) +} + +fn inline_logprobs_value() -> Value { + let ids = Value::Ext( + 3, + vec![ + 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, + 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, + ], + ); + let probs = Value::Ext( + 3, + vec![ + 0, 0, 128, 63, 0, 0, 0, 64, 0, 0, 64, 64, 0, 0, 128, 64, 0, 0, 160, 64, 0, 0, 192, 64, + ], + ); + let ranks = Value::Ext(3, vec![1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0]); + Value::Array(vec![ + ndarray_value(" Value { + let ids = Value::Ext( + 3, + vec![ + 10, 0, 0, 0, 0, 0, 0, 0, 11, 0, 0, 0, 0, 0, 0, 0, 12, 0, 0, 0, 0, 0, 0, 0, 13, 0, 0, 0, + 0, 0, 0, 0, 14, 0, 0, 0, 0, 0, 0, 0, 15, 0, 0, 0, 0, 0, 0, 0, + ], + ); + let probs = Value::Ext( + 3, + vec![ + 0, 0, 32, 65, 0, 0, 48, 65, 0, 0, 64, 65, 0, 0, 80, 65, 0, 0, 96, 65, 0, 0, 112, 65, + ], + ); + let ranks = Value::Ext(3, vec![3, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0]); + Value::Array(vec![ + ndarray_value("int64", &[2, 3], ids), + ndarray_value("float32", &[2, 3], probs), + ndarray_value("int64", &[2], ranks), + Value::Nil, + ]) +} + +fn expected_sample_logprobs() -> Logprobs { + Logprobs { + positions: vec![ + PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id: 1, + logprob: 1.0, + rank: 1, + }, + TokenLogprob { + token_id: 2, + logprob: 2.0, + rank: 1, + }, + TokenLogprob { + token_id: 3, + logprob: 3.0, + rank: 2, + }, + ], + }, + PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id: 4, + logprob: 4.0, + rank: 2, + }, + TokenLogprob { + token_id: 5, + logprob: 5.0, + rank: 1, + }, + TokenLogprob { + token_id: 6, + logprob: 6.0, + rank: 2, + }, + ], + }, + ], + } +} + +fn expected_prompt_logprobs() -> Logprobs { + Logprobs { + positions: vec![ + PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id: 10, + logprob: 10.0, + rank: 3, + }, + TokenLogprob { + token_id: 11, + logprob: 11.0, + rank: 1, + }, + TokenLogprob { + token_id: 12, + logprob: 12.0, + rank: 2, + }, + ], + }, + PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id: 13, + logprob: 13.0, + rank: 4, + }, + TokenLogprob { + token_id: 14, + logprob: 14.0, + rank: 1, + }, + TokenLogprob { + token_id: 15, + logprob: 15.0, + rank: 2, + }, + ], + }, + ], + } +} + +#[test] +fn decodes_inline_new_logprobs() { + let frames = vec![Bytes::from(encode_value(&output_wire_with_custom_fields( + Some(inline_logprobs_value()), + None, + )))]; + let decoded = decode_engine_core_outputs(&frames).unwrap().into_request_batch().unwrap(); + + let logprobs = decoded.outputs[0].new_logprobs.clone().unwrap().into_direct().unwrap(); + assert_eq!(logprobs, expected_sample_logprobs()); + assert_eq!( + decoded.finished_requests, + Some(BTreeSet::from(["req-1".to_string()])) + ); +} + +#[test] +fn decodes_multipart_new_logprobs() { + let frames = vec![ + Bytes::from(encode_value(&output_wire_with_custom_fields( + Some(Value::Array(vec![ + ndarray_value("i4", &[1, 2], Value::Ext(3, vec![0, 0, 0, 1, 0, 0, 0, 2])), + ndarray_value( + ">f4", + &[1, 2], + Value::Ext(3, vec![63, 128, 0, 0, 64, 0, 0, 0]), + ), + ndarray_value(">i4", &[1], Value::Ext(3, vec![0, 0, 0, 3])), + Value::Nil, + ])), + None, + )))]; + let decoded = decode_engine_core_outputs(&frames).unwrap().into_request_batch().unwrap(); + let logprobs = decoded.outputs[0].new_logprobs.clone().unwrap().into_direct().unwrap(); + assert_eq!( + logprobs, + Logprobs { + positions: vec![PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id: 1, + logprob: 1.0, + rank: 3, + }, + TokenLogprob { + token_id: 2, + logprob: 2.0, + rank: 1, + }, + ], + }], + } + ); +} + +#[test] +fn rejects_non_none_cu_num_generated_tokens() { + let frames = vec![Bytes::from(encode_value(&output_wire_with_custom_fields( + Some(Value::Array(vec![ + ndarray_value(" +#[derive(Debug, Clone, PartialEq, Serialize_tuple, Deserialize_tuple)] +pub struct WireLogprobs { + /// Wire array with shape `[num_positions, max_num_logprobs + 1]`. + pub logprob_token_ids: WireNdArray, + /// Wire array with shape `[num_positions, max_num_logprobs + 1]`. + pub logprobs: WireNdArray, + /// Wire array with shape `[num_positions]`. + /// + /// Python uses the field name `sampled_token_ranks` for sample logprobs and + /// `selected_token_ranks` for prompt logprobs. Rust keeps one neutral field + /// because both payloads share the same wire representation. + pub token_ranks: WireNdArray, + /// Preserved only for wire compatibility with batch-level Python tensors. + /// Scheduler-sliced per-request outputs should emit `None` here, and + /// the semantic Rust decoder rejects any other value. + #[serde(default)] + pub cu_num_generated_tokens: Option>, +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/lora.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/lora.rs new file mode 100644 index 0000000000000000000000000000000000000000..63e056fbc3539c1f0d05feb55164ce41620d13c1 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/lora.rs @@ -0,0 +1,42 @@ +use serde_tuple::{Deserialize_tuple, Serialize_tuple}; + +use crate::protocol::OpaqueValue; + +/// Request for a LoRA adapter. +/// +/// Mirrors Python `vllm.lora.request.LoRARequest`, which is a msgspec +/// `array_like=True` struct. Keep the field order aligned with Python. +#[derive(Debug, Clone, PartialEq, Serialize_tuple, Deserialize_tuple)] +pub struct LoraRequest { + pub lora_name: String, + pub lora_int_id: u64, + pub lora_path: String, + #[serde(default)] + pub base_model_name: Option, + #[serde(default)] + pub tensorizer_config_dict: Option, + #[serde(default)] + pub load_inplace: bool, + #[serde(default)] + pub is_3d_lora_weight: bool, +} + +impl LoraRequest { + pub fn new( + lora_name: String, + lora_int_id: u64, + lora_path: String, + load_inplace: bool, + is_3d_lora_weight: bool, + ) -> Self { + Self { + lora_name, + lora_int_id, + lora_path, + base_model_name: None, + tensorizer_config_dict: None, + load_inplace, + is_3d_lora_weight, + } + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/mod.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..76f604998bd83a8b5093fbd341959fdd8f67ddfb --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/mod.rs @@ -0,0 +1,89 @@ +use std::any::type_name; +use std::io::Cursor; + +use rmpv::Value; +use serde::{Deserialize, Serialize}; +use thiserror_ext::AsReport; + +use crate::error::{Error, Result}; + +/// Dynamic msgpack value used for schema positions that are preserved but not +/// yet strongly typed in the early-stage Rust client. +pub type OpaqueValue = Value; + +fn default_opaque_value_nil() -> OpaqueValue { + Value::Nil +} + +fn is_false(v: &bool) -> bool { + !v +} + +pub mod dtype; +pub mod handshake; +pub mod logprobs; +pub mod lora; +pub mod multimodal; +pub mod output; +pub mod request; +pub mod sampling; +pub mod stats; +pub mod structured_outputs; +pub mod tensor; +pub mod utility; + +/// Encode a Rust value into msgpack using the protocol crate's serde model. +pub fn encode_msgpack(value: &T) -> Result> +where + T: Serialize + std::fmt::Debug, +{ + rmp_serde::to_vec_named(value).map_err(|error| Error::Encode { + target_type: type_name::(), + message: format!( + "failed to encode value `{:?}`: {}", + value, + error.to_report_string() + ), + }) +} + +/// Decode a msgpack payload into a strongly typed protocol value, with enhanced +/// error reporting. +pub fn decode_msgpack(bytes: &[u8]) -> Result +where + T: for<'de> Deserialize<'de>, +{ + fn decode_value_preview(bytes: &[u8]) -> String { + match decode_value(bytes) { + Ok(value) => format!("{value}"), + Err(error) => format!(""), + } + } + + rmp_serde::from_slice(bytes).map_err(|error| Error::Decode { + target_type: type_name::(), + message: format!("{error}; value fallback: {}", decode_value_preview(bytes)), + }) +} + +/// Decode a msgpack payload into a dynamic value for diagnostics and tests. +pub fn decode_value(bytes: &[u8]) -> Result { + Ok(rmpv::decode::read_value(&mut Cursor::new(bytes))?) +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + + use super::*; + + #[test] + fn decode_msgpack_includes_type_name_and_value_fallback() { + let error = decode_msgpack::( + &rmp_serde::to_vec_named(&BTreeMap::from([("status", "READY")])).unwrap(), + ) + .unwrap_err(); + + expect_test::expect![[r#"messagepack decode failed for u64: wrong msgpack marker FixMap(1); value fallback: {"status": "READY"}"#]].assert_eq(&error.to_report_string()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/multimodal.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/multimodal.rs new file mode 100644 index 0000000000000000000000000000000000000000..3bb89d8b6533554c516038cfad3200589649d502 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/multimodal.rs @@ -0,0 +1,282 @@ +use std::collections::BTreeMap; + +use serde::{Deserialize, Serialize}; +use serde_tuple::{Deserialize_tuple, Serialize_tuple}; + +use super::tensor::WireTensor; + +/// Multimodal feature payload accepted from higher-level frontend code. +/// +/// Original Python definition: +/// +pub type MmFeatures = Vec; + +/// Represents a single multimodal input with its processed data and metadata. +/// +/// Used to track multimodal data through processing and caching. A request +/// containing multiple multimodal items will have one `MmFeatureSpec` +/// per item. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct MmFeatureSpec { + /// Represents multimodal data for this feature. + /// + /// Can be `None` if the item is cached, to skip IPC between API server + /// and engine core processes. + pub data: Option, + + /// The input modality, e.g., `"image"`, `"audio"`, `"video"`. + pub modality: String, + + /// The hash for caching encoder outputs (with LoRA prefix if applicable). + pub identifier: String, + + /// The location of the `modality` tokens corresponding to this item + /// in the prompt, e.g., `PlaceholderRange(offset=2, length=336)`. + pub mm_position: PlaceholderRange, + + /// The hash for caching processor outputs (without LoRA prefix). + #[serde(default)] + pub mm_hash: Option, +} + +/// Placeholder location information for multi-modal data. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct PlaceholderRange { + /// The start index of the placeholder in the prompt. + pub offset: usize, + + /// The length of the placeholder. + pub length: usize, + + /// A boolean mask of shape `(length,)` indicating which positions + /// between `offset` and `offset + length` to assign embeddings to. + /// `None` means all positions. + #[serde(default)] + pub is_embed: Option, +} + +/// A dictionary of processed keyword arguments to pass to the model, +/// corresponding to a single item in `MultiModalDataItems`. +/// +/// Original Python definition: +/// +pub type MmKwargsItem = BTreeMap; + +/// Represents a processed keyword argument to pass to a model for a +/// `MmKwargsItem`. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct MmFieldElem { + /// The processed value of this field in `MmKwargsItem`, i.e. the + /// keyword argument value to be passed to the model. + /// + /// It may be set to `None` if it is determined that the item is cached + /// in `EngineCore`. + pub data: Option, + + /// Defines how to combine this field's processed values with others in + /// order to batch multi-modal items together for model inference. + pub field: MmField, +} + +/// Processed multimodal keyword argument value. +/// +/// Original Python definition (`NestedTensors`) and wire encoding: +/// +/// +/// +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(untagged)] +pub enum MmKwargValue { + Tensor(WireTensor), + Int(i64), + Float(f64), + List(Vec), +} + +/// Defines how to interpret tensor data belonging to a keyword argument for +/// `MultiModalKwargsItems`, and vice versa. +/// +/// Original Python definitions and wire encoding: +/// +/// +/// +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(try_from = "MmFieldWire", into = "MmFieldWire")] +pub enum MmField { + Batched(MmBatchedField), + Flat(MmFlatField), + Shared(MmSharedField), +} + +/// Info: `MultiModalFieldConfig.batched`. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct MmBatchedField { + /// If `True`, then this field is excluded from being moved to the + /// accelerator when multimodal items are grouped and batched. + pub keep_on_cpu: bool, +} + +/// Info: `MultiModalFieldConfig.flat` and +/// `MultiModalFieldConfig.flat_from_sizes`. +/// +/// Original Python definition: +/// +/// +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct MmFlatField { + /// For each multi-modal item, a slice (`dim=0`) or a tuple of slices + /// (`dim>0`) that is used to extract the data corresponding to it. + pub slices: Vec, + + /// The dimension to extract data, default to 0. + pub dim: i32, + + /// If `True`, then this field is excluded from being moved to the + /// accelerator when multimodal items are grouped and batched. + pub keep_on_cpu: bool, +} + +/// Info: `MultiModalFieldConfig.shared`. +/// +/// Original Python definition: +/// +/// +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct MmSharedField { + pub batch_size: usize, + + /// If `True`, then this field is excluded from being moved to the + /// accelerator when multimodal items are grouped and batched. + pub keep_on_cpu: bool, +} + +/// Python slice encoded as `(start, stop, step)`. +/// +/// Original Python wire encoding: +/// +#[derive(Debug, Clone, PartialEq, Eq, Serialize_tuple, Deserialize_tuple)] +pub struct SliceSpec { + pub start: Option, + pub stop: Option, + pub step: Option, +} + +/// A single slice or a tuple of slices used by `MmFlatField`. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(untagged)] +pub enum MmSlice { + Slice(SliceSpec), + Slices(Vec), +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize_tuple, Deserialize_tuple)] +struct MmFieldWire { + name: String, + inner: MmFieldWireInner, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(untagged)] +enum MmFieldWireInner { + Batched(MmBatchedField), + Flat(MmFlatField), + Shared(MmSharedField), +} + +impl TryFrom for MmField { + type Error = String; + + fn try_from(value: MmFieldWire) -> Result { + match (value.name.as_str(), value.inner) { + ("batched", MmFieldWireInner::Batched(kwargs)) => Ok(Self::Batched(kwargs)), + ("flat", MmFieldWireInner::Flat(kwargs)) => Ok(Self::Flat(kwargs)), + ("shared", MmFieldWireInner::Shared(kwargs)) => Ok(Self::Shared(kwargs)), + (name, _) => Err(format!( + "mismatched or unknown multimodal field factory {name:?}" + )), + } + } +} + +impl From for MmFieldWire { + fn from(value: MmField) -> Self { + match value { + MmField::Batched(kwargs) => Self { + name: "batched".to_string(), + inner: MmFieldWireInner::Batched(kwargs), + }, + MmField::Flat(kwargs) => Self { + name: "flat".to_string(), + inner: MmFieldWireInner::Flat(kwargs), + }, + MmField::Shared(kwargs) => Self { + name: "shared".to_string(), + inner: MmFieldWireInner::Shared(kwargs), + }, + } + } +} + +#[cfg(test)] +mod tests { + use std::io::Cursor; + + use rmpv::Value; + + use super::*; + + fn encode_value(value: &T) -> Value { + let bytes = rmp_serde::to_vec_named(value).expect("encode value"); + rmpv::decode::read_value(&mut Cursor::new(bytes)).expect("decode value") + } + + #[test] + fn multimodal_field_serializes_to_python_factory_tuple() { + let field = MmField::Flat(MmFlatField { + slices: vec![MmSlice::Slice(SliceSpec { + start: Some(0), + stop: Some(1200), + step: None, + })], + dim: 0, + keep_on_cpu: false, + }); + + let value = encode_value(&field); + let Value::Array(items) = value else { + panic!("field should encode as tuple array"); + }; + assert_eq!(items.len(), 2); + assert_eq!(items[0].as_str(), Some("flat")); + + let Value::Map(kwargs) = &items[1] else { + panic!("field kwargs should encode as map"); + }; + assert!(kwargs.iter().any(|(key, _)| key.as_str() == Some("slices"))); + assert!(kwargs.iter().any(|(key, _)| key.as_str() == Some("dim"))); + assert!(kwargs.iter().any(|(key, _)| key.as_str() == Some("keep_on_cpu"))); + } + + #[test] + fn multimodal_field_round_trips_python_factory_tuple() { + let field = MmField::Batched(MmBatchedField { keep_on_cpu: true }); + let encoded = rmp_serde::to_vec_named(&field).expect("encode field"); + let decoded: MmField = rmp_serde::from_slice(&encoded).expect("decode field"); + assert_eq!(decoded, field); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/output.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/output.rs new file mode 100644 index 0000000000000000000000000000000000000000..1a92193cf6cea2fc91130b6aa7a27a7cd2a6ef7b --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/output.rs @@ -0,0 +1,518 @@ +use std::collections::BTreeSet; + +use enum_as_inner::EnumAsInner; +use serde::{Deserialize, Deserializer, Serialize, Serializer}; +use serde_default::DefaultFromSerde; +use serde_repr::{Deserialize_repr, Serialize_repr}; +use serde_tuple::{Deserialize_tuple, Serialize_tuple}; + +use super::utility::UtilityOutput; +use crate::error::{Error, Result, ext_value_decode}; +use crate::protocol::logprobs::MaybeWireLogprobs; +use crate::protocol::stats::{PrefillStats, SchedulerStats}; +use crate::protocol::{OpaqueValue, decode_msgpack}; + +/// The stop reason associated with a finished output. +/// +/// Python models this as the union-typed `stop_reason: int | str | None` +/// field on `EngineCoreOutput`; the Rust client narrows it into a tagged enum. +/// +/// Original Python field: +/// +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(untagged)] +pub enum StopReason { + TokenId(u32), + Text(String), +} + +/// Reason a request finished: stop, length, abort, error, or repetition. +/// +/// This mirrors the Python enum and uses integer encoding for compact wire +/// representation. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize_repr, Deserialize_repr)] +#[repr(u8)] +pub enum EngineCoreFinishReason { + /// A stop string was emitted. + Stop = 0, + /// `max_tokens` or `max_model_len` was reached. + Length = 1, + /// The request was aborted by the client. + Abort = 2, + /// A retryable request-level internal error occurred. + Error = 3, + /// A repetitive token pattern was detected. + Repetition = 4, +} + +/// Event types emitted by engine-core for one request. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize_repr, Deserialize_repr)] +#[repr(u8)] +pub enum EngineCoreEventType { + Queued = 1, + Scheduled = 2, + Preempted = 3, +} + +/// A timestamped engine-core event associated with one request. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct EngineCoreEvent { + pub r#type: EngineCoreEventType, + pub timestamp: f64, +} + +/// Engine-core output for a single request. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, PartialEq, Serialize_tuple, Deserialize_tuple, DefaultFromSerde)] +pub struct EngineCoreOutput { + pub request_id: String, + pub new_token_ids: Vec, + /// Decoded sample logprobs for the newly generated positions in this + /// output. + #[serde(default)] + pub new_logprobs: Option, + /// Decoded prompt logprobs for the scored prompt positions emitted in this + /// output. + #[serde(default)] + pub new_prompt_logprobs_tensors: Option, + #[serde(default)] + pub pooling_output: Option, + #[serde(default)] + pub finish_reason: Option, + #[serde(default)] + pub stop_reason: Option, + #[serde(default)] + pub events: Option>, + #[serde(default)] + pub kv_transfer_params: Option, + #[serde(default)] + pub trace_headers: Option, + /// Breakdown of the scheduled prefill computation, set on the first output + /// of a newly scheduled prefill and elided for subsequent decode outputs. + #[serde(default)] + pub prefill_stats: Option, + #[serde(default)] + pub routed_experts: Option, + /// Number of NaNs seen in logits. Values above zero indicate corruption. + #[serde(default)] + pub num_nans_in_logits: u32, +} + +impl EngineCoreOutput { + /// Returns whether this output is terminal for the request. + pub fn finished(&self) -> bool { + self.finish_reason.is_some() + } + + /// Resolve all wire-format fields in-place by looking up aux frames and + /// decoding raw-view payloads as needed. + fn resolve_in_place(&mut self, frames: &[Frame]) -> Result<()> + where + Frame: AsRef<[u8]>, + { + self.new_logprobs = (self.new_logprobs.take()) + .map(|value| value.resolve(frames, "new_logprobs")) + .transpose()?; + self.new_prompt_logprobs_tensors = (self.new_prompt_logprobs_tensors.take()) + .map(|value| value.resolve(frames, "new_prompt_logprobs_tensors")) + .transpose()?; + Ok(()) + } +} + +/// Raw Python/msgpack engine-core output envelope. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, PartialEq, Serialize_tuple, Deserialize_tuple, DefaultFromSerde)] +struct WireEngineCoreOutputs { + #[serde(default)] + engine_index: u32, + /// Outputs grouped for this client in the current engine tick. + #[serde(default)] + outputs: Vec, + #[serde(default)] + scheduler_stats: Option>, + #[serde(default)] + timestamp: f64, + #[serde(default)] + utility_output: Option, + #[serde(default)] + finished_requests: Option>, + /// In DP mode, signals that the current wave finished and engines are + /// paused. + #[serde(default)] + wave_complete: Option, + /// In DP mode, signals that a request arrived for an old wave and the next + /// wave needs to start in other engines. + #[serde(default)] + start_wave: Option, +} + +/// Data-parallel control notifications multiplexed through `EngineCoreOutputs`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum DpControlMessage { + WaveComplete(u32), + StartWave(u32), +} + +#[derive(Debug, Clone, PartialEq, Default)] +pub struct RequestBatchOutputs { + pub engine_index: u32, + pub outputs: Vec, + pub scheduler_stats: Option>, + pub timestamp: f64, + pub finished_requests: Option>, +} + +#[derive(Debug, Clone, PartialEq, Default)] +pub struct UtilityCallOutput { + pub engine_index: u32, + pub timestamp: f64, + pub output: UtilityOutput, +} + +#[derive(Debug, Clone, PartialEq)] +pub struct DpControlOutput { + pub engine_index: u32, + pub timestamp: f64, + pub control: DpControlMessage, +} + +/// Semantic engine-core output families. +/// +/// Python currently uses one product-shaped wire struct. The Rust protocol +/// exposes the finite semantic families while preserving the same msgpack shape +/// for serialization. +#[derive(Debug, Clone, PartialEq, EnumAsInner)] +pub enum EngineCoreOutputs { + RequestBatch(RequestBatchOutputs), + Utility(UtilityCallOutput), + DpControl(DpControlOutput), +} + +impl From for EngineCoreOutputs { + fn from(outputs: RequestBatchOutputs) -> Self { + Self::RequestBatch(outputs) + } +} + +impl From for EngineCoreOutputs { + fn from(output: UtilityCallOutput) -> Self { + Self::Utility(output) + } +} + +impl From for EngineCoreOutputs { + fn from(output: DpControlOutput) -> Self { + Self::DpControl(output) + } +} + +impl EngineCoreOutputs { + /// Resolve all wire-format fields in-place by looking up aux frames and + /// decoding raw-view payloads as needed. + fn resolve_in_place(&mut self, frames: &[Frame]) -> Result<()> + where + Frame: AsRef<[u8]>, + { + if let Self::RequestBatch(batch) = self { + for output in &mut batch.outputs { + output.resolve_in_place(frames)?; + } + } + Ok(()) + } +} + +/// Classify the raw wire message into a more semantic Rust enum. +impl TryFrom for EngineCoreOutputs { + type Error = Error; + + fn try_from(value: WireEngineCoreOutputs) -> Result { + let has_request_payload = !value.outputs.is_empty() + || value.scheduler_stats.is_some() + || value.finished_requests.is_some(); + + match ( + has_request_payload, + &value.utility_output, + &value.wave_complete, + &value.start_wave, + ) { + (true, None, None, None) => Ok(RequestBatchOutputs { + engine_index: value.engine_index, + outputs: value.outputs, + scheduler_stats: value.scheduler_stats, + timestamp: value.timestamp, + finished_requests: value.finished_requests, + } + .into()), + (false, Some(_), None, None) => Ok(UtilityCallOutput { + engine_index: value.engine_index, + timestamp: value.timestamp, + output: value.utility_output.unwrap(), + } + .into()), + (false, None, Some(_), None) => Ok(DpControlOutput { + engine_index: value.engine_index, + timestamp: value.timestamp, + control: DpControlMessage::WaveComplete(value.wave_complete.unwrap()), + } + .into()), + (false, None, None, Some(_)) => Ok(DpControlOutput { + engine_index: value.engine_index, + timestamp: value.timestamp, + control: DpControlMessage::StartWave(value.start_wave.unwrap()), + } + .into()), + + _ => Err(Error::Decode { + target_type: "EngineCoreOutputs", + message: "invalid wire shape".to_string(), + }), + } + } +} + +impl From for WireEngineCoreOutputs { + fn from(value: EngineCoreOutputs) -> Self { + match value { + EngineCoreOutputs::RequestBatch(batch) => Self { + engine_index: batch.engine_index, + outputs: batch.outputs, + scheduler_stats: batch.scheduler_stats, + timestamp: batch.timestamp, + finished_requests: batch.finished_requests, + ..Default::default() + }, + EngineCoreOutputs::Utility(utility) => Self { + engine_index: utility.engine_index, + timestamp: utility.timestamp, + utility_output: Some(utility.output), + ..Default::default() + }, + EngineCoreOutputs::DpControl(control) => { + let (wave_complete, start_wave) = match control.control { + DpControlMessage::WaveComplete(wave) => (Some(wave), None), + DpControlMessage::StartWave(wave) => (None, Some(wave)), + }; + Self { + engine_index: control.engine_index, + timestamp: control.timestamp, + wave_complete, + start_wave, + ..Default::default() + } + } + } + } +} + +impl Serialize for EngineCoreOutputs { + fn serialize(&self, serializer: S) -> std::result::Result + where + S: Serializer, + { + WireEngineCoreOutputs::from(self.clone()).serialize(serializer) + } +} + +impl<'de> Deserialize<'de> for EngineCoreOutputs { + fn deserialize(deserializer: D) -> std::result::Result + where + D: Deserializer<'de>, + { + WireEngineCoreOutputs::deserialize(deserializer)? + .try_into() + .map_err(serde::de::Error::custom) + } +} + +/// Decode one ordinary or multipart engine-core output message into the strong +/// typed public protocol shape. +pub fn decode_engine_core_outputs(frames: &[Frame]) -> Result +where + Frame: AsRef<[u8]>, +{ + let first_frame = frames.first().ok_or_else(|| ext_value_decode!("missing output frame"))?; + + let mut outputs: EngineCoreOutputs = decode_msgpack(first_frame.as_ref())?; + outputs.resolve_in_place(frames)?; + Ok(outputs) +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeSet; + + use super::*; + use crate::protocol::output::EngineCoreOutput; + use crate::protocol::{decode_msgpack, encode_msgpack}; + + #[test] + fn engine_core_outputs_roundtrip_finished_fields() { + let outputs = WireEngineCoreOutputs { + outputs: vec![EngineCoreOutput { + request_id: "req-1".to_string(), + new_token_ids: vec![42], + new_logprobs: None, + new_prompt_logprobs_tensors: None, + pooling_output: None, + finish_reason: Some(EngineCoreFinishReason::Length), + stop_reason: Some(StopReason::Text("stop".to_string())), + events: None, + kv_transfer_params: None, + trace_headers: None, + prefill_stats: None, + routed_experts: None, + num_nans_in_logits: 0, + }], + finished_requests: Some(BTreeSet::from(["req-1".to_string()])), + ..Default::default() + }; + + let encoded = encode_msgpack(&outputs).unwrap(); + let decoded: WireEngineCoreOutputs = decode_msgpack(&encoded).unwrap(); + + assert_eq!(decoded.outputs.len(), 1); + assert_eq!( + decoded.outputs[0].finish_reason, + Some(EngineCoreFinishReason::Length) + ); + assert_eq!( + decoded.finished_requests, + Some(BTreeSet::from(["req-1".to_string()])) + ); + } + + #[test] + fn engine_core_outputs_classify_request_batch() { + let outputs = WireEngineCoreOutputs { + outputs: vec![EngineCoreOutput { + request_id: "req-1".to_string(), + new_token_ids: vec![7], + ..Default::default() + }], + finished_requests: Some(BTreeSet::from(["req-1".to_string()])), + ..Default::default() + }; + + expect_test::expect![[r#" + RequestBatch( + RequestBatchOutputs { + engine_index: 0, + outputs: [ + EngineCoreOutput { + request_id: "req-1", + new_token_ids: [ + 7, + ], + new_logprobs: None, + new_prompt_logprobs_tensors: None, + pooling_output: None, + finish_reason: None, + stop_reason: None, + events: None, + kv_transfer_params: None, + trace_headers: None, + prefill_stats: None, + routed_experts: None, + num_nans_in_logits: 0, + }, + ], + scheduler_stats: None, + timestamp: 0.0, + finished_requests: Some( + { + "req-1", + }, + ), + }, + ) + "#]] + .assert_debug_eq(&EngineCoreOutputs::try_from(outputs).unwrap()); + } + + #[test] + fn engine_core_outputs_classify_utility() { + let outputs = WireEngineCoreOutputs { + utility_output: Some(UtilityOutput { + call_id: 42_u64.into(), + failure_message: None, + result: None, + }), + ..Default::default() + }; + + expect_test::expect![[r#" + Utility( + UtilityCallOutput { + engine_index: 0, + timestamp: 0.0, + output: UtilityOutput { + call_id: 42, + failure_message: None, + result: None, + }, + }, + ) + "#]] + .assert_debug_eq(&EngineCoreOutputs::try_from(outputs).unwrap()); + } + + #[test] + fn engine_core_outputs_classify_control() { + let outputs = WireEngineCoreOutputs { + start_wave: Some(3), + ..Default::default() + }; + + expect_test::expect![[r#" + DpControl( + DpControlOutput { + engine_index: 0, + timestamp: 0.0, + control: StartWave( + 3, + ), + }, + ) + "#]] + .assert_debug_eq(&EngineCoreOutputs::try_from(outputs).unwrap()); + } + + #[test] + fn engine_core_outputs_rejects_mixed_shape() { + let outputs = WireEngineCoreOutputs { + outputs: vec![EngineCoreOutput { + request_id: "req-1".to_string(), + new_token_ids: vec![7], + ..Default::default() + }], + utility_output: Some(UtilityOutput { + call_id: 1_u64.into(), + failure_message: None, + result: None, + }), + ..Default::default() + }; + + let error = EngineCoreOutputs::try_from(outputs).unwrap_err(); + expect_test::expect![[ + r#"messagepack decode failed for EngineCoreOutputs: invalid wire shape"# + ]] + .assert_eq(&error.to_string()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/request.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/request.rs new file mode 100644 index 0000000000000000000000000000000000000000..145ce26a09ca6eef226398082170f4e02b55704c --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/request.rs @@ -0,0 +1,175 @@ +use std::collections::{BTreeMap, HashMap}; + +use bytes::Bytes; +use serde::{Deserialize, Serialize}; +use serde_default::DefaultFromSerde; +use serde_tuple::{Deserialize_tuple, Serialize_tuple}; + +use crate::protocol::multimodal::MmFeatures; +use crate::protocol::sampling::EngineCoreSamplingParams; +use crate::protocol::{OpaqueValue, lora}; +use crate::{Error, Result}; + +/// Request types are encoded as single-byte protocol constants so they can be +/// sent over the ZMQ socket without an extra encoding step. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[repr(u8)] +pub enum EngineCoreRequestType { + Add = 0, + Abort = 1, + StartDpWave = 2, + Utility = 3, +} + +impl EngineCoreRequestType { + /// Decode the single-byte request type frame used on the engine input + /// socket. Returns `None` for unrecognized values. + pub fn from_frame(frame: &[u8]) -> Option { + let [value] = frame else { + return None; + }; + + match value { + 0 => Some(Self::Add), + 1 => Some(Self::Abort), + 2 => Some(Self::StartDpWave), + 3 => Some(Self::Utility), + _ => None, + } + } + + /// Encode the request type as the single-byte frame used on the engine + /// input socket. + pub fn to_frame(self) -> Bytes { + Bytes::from_static(match self { + Self::Add => b"\x00", + Self::Abort => b"\x01", + Self::StartDpWave => b"\x02", + Self::Utility => b"\x03", + }) + } +} + +/// Extra kwargs consumed by engine-side reasoning parsers. +/// +/// Original Python construction point: +/// +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct ReasoningParserKwargs { + /// Effective kwargs visible to the chat template for this request. + pub chat_template_kwargs: HashMap, +} + +/// Engine-core add-request payload sent from frontend to engine. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, PartialEq, Serialize_tuple, Deserialize_tuple, DefaultFromSerde)] +pub struct EngineCoreRequest { + pub request_id: String, + pub prompt_token_ids: Option>, + /// Multimodal features attached to the request. + pub mm_features: Option, + pub sampling_params: Option, + /// Pooling parameters are preserved in the schema but not yet strongly + /// typed. + pub pooling_params: Option, + pub arrival_time: f64, + #[serde(default)] + pub lora_request: Option, + #[serde(default)] + pub cache_salt: Option, + #[serde(default)] + pub data_parallel_rank: Option, + /// Unsupported in the first-stage Rust client because Python uses a custom + /// tensor/aux-frame encoding path for this field. + #[serde(default)] + pub prompt_embeds: Option, + /// Per-position mask for mixed-mode inputs (e.g. chat completion with + /// `prompt_embeds` content parts). `Some(true)` means real token id; + /// `Some(false)` means the position uses a pre-computed entry from + /// `prompt_embeds`. `None` for pure-tokens and pure-embeds requests. + #[serde(default)] + pub prompt_is_token_ids: Option>, + /// Index of the client, used to ensure outputs are sent back to the same + /// client when scaling out the frontend. + #[serde(default)] + pub client_index: u32, + /// In DP mode, indicates which wave this request is expected to belong to. + #[serde(default)] + pub current_wave: u32, + #[serde(default)] + pub priority: i32, + #[serde(default)] + pub trace_headers: Option>, + #[serde(default)] + pub resumable: bool, + /// Original user-provided request ID, used for output reporting and aborts. + #[serde(default)] + pub external_req_id: Option, + #[serde(default)] + pub reasoning_ended: Option, + /// Reasoning-parser kwargs forwarded from the frontend to the + /// structured-output backend. + #[serde(default)] + pub reasoning_parser_kwargs: Option, + /// If `true`, the request should be added to the scheduler's waiting queue + /// and immediately aborted, so connector-side cleanup runs via the + /// standard `request_finished` hook. + #[serde(default)] + pub abort_immediately: bool, +} + +impl EngineCoreRequest { + /// Validate fields intentionally not supported in the first-stage client. + pub fn validate(&self) -> Result<()> { + if self.prompt_embeds.is_some() { + return Err(Error::UnsupportedField { + context: "EngineCoreRequest", + field: "prompt_embeds", + }); + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use rmpv::Value; + + use super::*; + use crate::protocol::sampling::EngineCoreSamplingParams; + use crate::protocol::{decode_value, encode_msgpack}; + + #[test] + fn engine_core_request_serializes_as_full_array() { + let request = EngineCoreRequest { + request_id: "req-1".to_string(), + prompt_token_ids: Some(vec![1, 2, 3]), + sampling_params: Some(EngineCoreSamplingParams { + max_tokens: 8, + ..EngineCoreSamplingParams::for_test() + }), + arrival_time: 1234.5, + client_index: 7, + ..EngineCoreRequest::default() + }; + + let encoded = encode_msgpack(&request).unwrap(); + let value = decode_value(&encoded).unwrap(); + let array = match value { + Value::Array(array) => array, + other => panic!("expected array, got {other:?}"), + }; + + assert_eq!(array.len(), 20); + assert_eq!(array[0], Value::from("req-1")); + assert_eq!(array[2], Value::Nil); + assert_eq!(array[4], Value::Nil); + assert_eq!(array[10], Value::Nil); + assert_eq!(array[11], Value::from(7)); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/sampling.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/sampling.rs new file mode 100644 index 0000000000000000000000000000000000000000..f3fa6b3dd7312f2ee11e1e7d4d3de5d2107106b8 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/sampling.rs @@ -0,0 +1,240 @@ +use std::collections::{BTreeSet, HashMap}; + +use serde::{Deserialize, Serialize}; +use serde_default::DefaultFromSerde; + +use crate::protocol::structured_outputs::StructuredOutputsParams; + +fn default_top_p() -> f32 { + 1.0 +} + +fn default_repetition_penalty() -> f32 { + 1.0 +} + +fn default_temperature() -> f32 { + 1.0 +} + +fn default_max_tokens() -> u32 { + 16 +} + +/// +/// Parameters for detecting repetitive N-gram patterns in output tokens. +/// +/// Mirrors Python's `RepetitionDetectionParams`: +/// +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct RepetitionDetectionParams { + /// Maximum N-gram size to check. 0 disables detection. + pub max_pattern_size: u32, + /// Minimum N-gram size to check. Defaults to 1 when zero. + #[serde(default)] + pub min_pattern_size: u32, + /// Minimum number of repetitions to trigger detection (must be >= 2). + pub min_count: u32, +} + +impl RepetitionDetectionParams { + /// Return `true` when the params are effectively disabled (max_pattern_size + /// is 0). + pub fn is_disabled(&self) -> bool { + self.max_pattern_size == 0 + } +} + +/// Engine-core-facing sampling parameters for text generation. +/// +/// This is the normalized southbound subset used by the Rust frontend when it +/// talks to Python engine-core over the wire. User-facing request semantics +/// such as `stop` strings, `n`, `ignore_eos`, and output aggregation mode are +/// intentionally handled by higher layers before values reach this DTO. +/// +/// Original Python definition: +/// +// Python's SamplingParams is `omit_defaults=True`, so msgpack drops +// default-valued keys; default the whole struct. Per-field fns cover the +// non-zero defaults. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, DefaultFromSerde)] +#[serde(default)] +pub struct EngineCoreSamplingParams { + /// Controls randomness. Lower values are more deterministic; zero means + /// greedy sampling. + #[serde(default = "default_temperature")] + pub temperature: f32, + /// Cumulative probability threshold for nucleus sampling. + #[serde(default = "default_top_p")] + pub top_p: f32, + /// Maximum number of top tokens to consider. `0` means all tokens. + pub top_k: u32, + /// Random seed used by the sampler when present. + pub seed: Option, + /// Maximum number of tokens to generate per output sequence. + #[serde(default = "default_max_tokens")] + pub max_tokens: u32, + /// Minimum number of tokens to generate before EOS or stop-token handling. + pub min_tokens: u32, + /// Maximum number of reasoning ("thinking") tokens to emit before the + /// reasoning section is force-closed. `None` means unlimited; the + /// user-facing `-1` sentinel is normalized to `None` by the frontend before + /// reaching this DTO, so only non-negative values are sent. Enforced + /// engine-side (and only when a reasoning parser is configured). + pub thinking_token_budget: Option, + /// Number of log probabilities to return per generated token. + /// + /// `None` disables sample logprobs. `-1` requests the full vocabulary. + pub logprobs: Option, + /// Number of log probabilities to return per prompt token. + /// + /// `None` disables prompt logprobs. `-1` requests the full vocabulary. + pub prompt_logprobs: Option, + /// Minimum probability threshold for token sampling. + pub min_p: f32, + /// Frequency penalty applied by the sampler. + pub frequency_penalty: f32, + /// Presence penalty applied by the sampler. + pub presence_penalty: f32, + /// Repetition penalty applied by the sampler. + #[serde(default = "default_repetition_penalty")] + pub repetition_penalty: f32, + /// Parameters for detecting repetitive N-gram patterns. `None` disables + /// detection. + pub repetition_detection: Option, + /// Token IDs that stop generation. + pub stop_token_ids: Vec, + /// Primary EOS token ID used by engine-core's dedicated EOS stop path. + /// + /// This mirrors Python's internal `_eos_token_id` field and is derived by + /// the frontend from tokenizer/model metadata rather than supplied directly + /// by end users. + #[serde(rename = "_eos_token_id")] + pub eos_token_id: Option, + /// Complete stop-token set used by engine-core for `min_tokens` masking. + /// + /// This mirrors Python's internal `_all_stop_token_ids` field and should + /// contain explicit `stop_token_ids` plus any frontend-derived EOS token + /// IDs. + #[serde(rename = "_all_stop_token_ids")] + pub all_stop_token_ids: BTreeSet, + /// Logit biases to apply during sampling. + /// Keys are token IDs + pub logit_bias: Option>, + /// Restrict output to these token IDs only. + pub allowed_token_ids: Option>, + /// Tokenized bad words to avoid during generation. + #[serde(rename = "_bad_words_token_ids")] + pub bad_words_token_ids: Option>>, + /// Parameters for configuring structured outputs (guided decoding). + pub structured_outputs: Option, + /// Specific token IDs for which log probabilities should be returned at + /// each position. + /// + /// When set, the engine returns logprobs for exactly these tokens in + /// addition to the sampled/scored token. Mutually exclusive with the + /// `logprobs` count field in practice. + pub logprob_token_ids: Option>, + /// If `Some(true)`, the request will not attempt to read from the prefix + /// cache; newly computed blocks may still populate the cache. `None` + /// defers to engine-core defaults. + pub skip_reading_prefix_cache: Option, + /// Additional request parameters for custom extensions (from `vllm_xargs`). + pub extra_args: Option>, +} + +impl EngineCoreSamplingParams { + /// Constructs a default sampling params for testing purposes only. + pub fn for_test() -> Self { + Self { + temperature: 1.0, + top_p: 1.0, + top_k: 0, + seed: None, + max_tokens: 65536, + min_tokens: 0, + thinking_token_budget: None, + logprobs: None, + prompt_logprobs: None, + min_p: 0.0, + frequency_penalty: 0.0, + presence_penalty: 0.0, + repetition_penalty: 1.0, + repetition_detection: None, + stop_token_ids: Vec::new(), + eos_token_id: None, + all_stop_token_ids: BTreeSet::new(), + logit_bias: None, + allowed_token_ids: None, + bad_words_token_ids: None, + structured_outputs: None, + logprob_token_ids: None, + skip_reading_prefix_cache: None, + extra_args: None, + } + } +} + +#[cfg(test)] +mod tests { + use rmpv::Value; + + use crate::protocol::decode_msgpack; + use crate::protocol::request::EngineCoreRequest; + + /// A real `sampling_params` is a sparse `omit_defaults` map; absent fields + /// must fall back to defaults. `python_compat` can't catch this since Rust + /// encodes full maps (see `engine_core_request_serializes_as_full_array`). + #[test] + fn decodes_sampling_params_with_omitted_defaults() { + let sampling_params = Value::Map(vec![ + ( + Value::from("stop_token_ids"), + Value::Array(vec![Value::from(151643u32)]), + ), + (Value::from("skip_reading_prefix_cache"), Value::from(false)), + ]); + let request = Value::Array(vec![ + Value::from("req-omit-defaults"), + Value::Array(vec![ + Value::from(1u32), + Value::from(2u32), + Value::from(3u32), + ]), + Value::Nil, + sampling_params, + Value::Nil, + Value::from(1.0f64), + ]); + + let mut bytes = Vec::new(); + rmpv::encode::write_value(&mut bytes, &request).unwrap(); + + let decoded: EngineCoreRequest = decode_msgpack(&bytes) + .expect("a real omit_defaults request must decode (regression: missing field)"); + + assert_eq!(decoded.request_id, "req-omit-defaults"); + let sampling = decoded.sampling_params.expect("sampling params present"); + + assert_eq!(sampling.stop_token_ids, vec![151643]); + assert_eq!(sampling.skip_reading_prefix_cache, Some(false)); + + // Omitted fields -> Python defaults. + assert_eq!(sampling.temperature, 1.0); + assert_eq!(sampling.top_p, 1.0); + assert_eq!(sampling.top_k, 0); + assert_eq!(sampling.seed, None); + assert_eq!(sampling.max_tokens, 16); + assert_eq!(sampling.min_tokens, 0); + assert_eq!(sampling.min_p, 0.0); + assert_eq!(sampling.frequency_penalty, 0.0); + assert_eq!(sampling.presence_penalty, 0.0); + assert_eq!(sampling.repetition_penalty, 1.0); + assert_eq!(sampling.repetition_detection, None); + assert_eq!(sampling.logprobs, None); + assert_eq!(sampling.prompt_logprobs, None); + assert_eq!(sampling.eos_token_id, None); + assert!(sampling.all_stop_token_ids.is_empty()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/stats.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/stats.rs new file mode 100644 index 0000000000000000000000000000000000000000..5b58ce10e851d17dd48b470c01c93ce3a33b4f36 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/stats.rs @@ -0,0 +1,188 @@ +use std::collections::BTreeMap; + +use serde::{Deserialize, Serialize}; + +use crate::protocol::OpaqueValue; + +/// Stores cache hit statistics. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct BaseCacheStats { + /// Whether the cache was reset. + pub reset: bool, + /// The number of requests in this update. + pub requests: u64, + /// The number of queries in these requests. + pub queries: u64, + /// The number of hits in these requests. + pub hits: u64, +} + +/// Stores prefix cache hit statistics. +/// - `reset`: Whether `reset_prefix_cache` was invoked. +/// - `queries`: Refers to the number of tokens that were queried. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct PrefixCacheStats { + /// Embedded base cache counters and reset flag. + #[serde(flatten)] + pub base: BaseCacheStats, + /// The number of previously preempted requests in this update. + pub preempted_requests: u64, + /// The `queries` number for preempted requests. + pub preempted_queries: u64, + /// The `hits` number for preempted requests. + pub preempted_hits: u64, +} + +/// Single KV cache block eviction sample. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct KvCacheEvictionEvent { + /// Lifetime from allocation to eviction. + pub lifetime_seconds: f64, + /// Idle time observed before eviction. + pub idle_seconds: f64, + /// Time gaps between consecutive accesses before eviction. + pub reuse_gaps_seconds: Vec, +} + +/// Per-step iteration decoding stats from scheduler. +/// +/// Each scheduler step, statistics on spec decoding performance are aggregated +/// across requests by the scheduler and returned to the frontend in +/// `EngineCoreOutputs -> SchedulerStats`. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct SpecDecodingStats { + /// Configured speculative token count for this scheduler. + pub num_spec_tokens: u64, + /// Number of drafted speculative decoding attempts. + pub num_drafts: u64, + /// Number of drafted tokens. + pub num_draft_tokens: u64, + /// Number of accepted drafted tokens. + pub num_accepted_tokens: u64, + /// Accepted drafted tokens counted by draft position. + pub num_accepted_tokens_per_pos: Vec, +} + +/// Breakdown of a scheduled prefill computation. +/// +/// Python models this as a plain `@dataclass`, so it is serialized by msgspec +/// as a map (named fields) rather than in the array-like form used by +/// `EngineCoreOutput` itself. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] +pub struct PrefillStats { + /// Total number of tokens to be prefilled. + #[serde(default)] + pub num_prompt_tokens: u32, + /// Tokens to be prefilled locally (actual compute work). + #[serde(default)] + pub num_computed_tokens: u32, + /// Tokens to be prefilled without actual compute work. + #[serde(default)] + pub num_cached_tokens: u32, + /// Tokens to be prefilled from local prefix cache. + #[serde(default)] + pub num_local_cached_tokens: u32, + /// Tokens to be prefilled from external KV transfer. + #[serde(default)] + pub num_external_cached_tokens: u32, +} + +/// Stats for debugging the metrics calculation. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct DebugPerfStats { + /// Time spent calculating these stats. + pub calc_duration: f64, + /// Number of prefill requests included in the sampled batch. + pub num_prefill_requests: u64, + /// Number of decode requests included in the sampled batch. + pub num_decode_requests: u64, + /// Optional execution-context breakdown used for debugging. + pub context_breakdown: Option>, + /// Optional per-component FLOPs breakdown. + pub num_flops_per_gpu_breakdown: Option>, + /// Optional per-component memory-read breakdown. + pub num_read_bytes_per_gpu_breakdown: Option>, + /// Optional per-component memory-write breakdown. + pub num_write_bytes_per_gpu_breakdown: Option>, +} + +/// Original Python definition: +/// +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct PerfStats { + /// Estimated floating point operations per GPU. + pub num_flops_per_gpu: u64, + /// Estimated bytes read from memory per GPU. + pub num_read_bytes_per_gpu: u64, + /// Estimated bytes written to memory per GPU. + pub num_write_bytes_per_gpu: u64, + /// Optional debug-only perf derivation details. + pub debug_stats: Option, +} + +/// Original Python definition: +/// +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct CudagraphStats { + /// Number of real tokens in the captured batch before padding. + pub num_unpadded_tokens: u64, + /// Number of padded tokens in the captured batch. + pub num_padded_tokens: u64, + /// Number of padding positions added for capture/runtime shape alignment. + pub num_paddings: u64, + /// Runtime mode string associated with this CUDA graph sample. + pub runtime_mode: String, +} + +/// Stats associated with the scheduler. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +pub struct SchedulerStats { + /// Number of requests in model execution batches. + pub num_running_reqs: u64, + /// Length of the "waiting" request queue. + pub num_waiting_reqs: u64, + /// Length of the "skipped waiting" queue. + #[serde(default)] + pub num_skipped_waiting_reqs: u64, + /// Internal DP load-balancing step counter. + pub step_counter: u64, + /// Internal DP load-balancing wave number. + pub current_wave: u64, + /// KV-cache usage. `1.0` means 100% usage. + pub kv_cache_usage: f64, + /// Local prefix cache statistics. + pub prefix_cache_stats: PrefixCacheStats, + /// External connector prefix cache statistics, when configured. + pub connector_prefix_cache_stats: Option, + /// Sampled KV cache eviction events for residency metrics. + pub kv_cache_eviction_events: Vec, + /// Speculative decoding scheduler stats, when enabled. + pub spec_decoding_stats: Option, + /// Connector-specific KV transfer stats, kept opaque for now. + pub kv_connector_stats: Option>, + /// CUDA graph runtime stats when graph metrics are enabled. + pub cudagraph_stats: Option, + /// Estimated MFU/performance stats, when enabled. + pub perf_stats: Option, +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/structured_outputs.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/structured_outputs.rs new file mode 100644 index 0000000000000000000000000000000000000000..a157520bc77674352d4d1abb01d6b4faa89b341a --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/structured_outputs.rs @@ -0,0 +1,305 @@ +use enum_as_inner::EnumAsInner; +use serde::{Deserialize, Deserializer, Serialize, Serializer}; +use serde_json::Value; + +use crate::error::{Error, Result}; + +/// Structured-output backend selected for EngineCore grammar compilation. +/// +/// Python vLLM stores this in `StructuredOutputsParams._backend` after request +/// validation. The Rust frontend currently always lowers structured-output +/// requests to guidance, while ignoring any user-supplied `_backend` value. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub enum StructuredOutputBackend { + Xgrammar, + #[default] + Guidance, + Outlines, + LmFormatEnforcer, +} + +/// The single structured-output constraint selected for a request. +#[derive(Debug, Clone, PartialEq, EnumAsInner)] +pub enum StructuredOutputConstraint { + /// JSON schema (as a dict/object or JSON string) constraining the output. + Json(Value), + /// Regular expression the output must match. + Regex(String), + /// List of allowed output strings (the model must produce one of these). + Choice(Vec), + /// Context-free grammar (in EBNF-like notation) the output must conform to. + Grammar(String), + /// Output must be valid JSON (free-form, no schema). + JsonObject, + /// Structural tag configuration (JSON-encoded string). + StructuralTag(String), +} + +/// Additional structured-output options that do not select the constraint mode. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct StructuredOutputOptions { + /// Disable any additional whitespace in guided JSON output. + pub disable_any_whitespace: bool, + /// Disable `additionalProperties` in JSON schema output. + pub disable_additional_properties: bool, + /// Custom whitespace pattern for guided JSON output. + pub whitespace_pattern: Option, +} + +/// Parameters for configuring structured outputs (guided decoding). +/// +/// This is the semantic Rust representation: exactly one constraint mode is +/// always selected. The Python/msgpack product-shaped representation is kept in +/// the private wire type below and used only at serde boundaries. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, PartialEq)] +pub struct StructuredOutputsParams { + pub constraint: StructuredOutputConstraint, + pub options: StructuredOutputOptions, + /// Structured-output backend, mirroring Python's internal `_backend`. + /// + /// User-supplied values are ignored during deserialization. This matches + /// Python's request boundary, where `_backend` is set by validation rather + /// than accepted as a request-level backend selector. + pub backend: StructuredOutputBackend, +} + +impl StructuredOutputsParams { + pub fn json(json: Value) -> Self { + Self::from_constraint(StructuredOutputConstraint::Json(json)) + } + + pub fn regex(regex: impl Into) -> Self { + Self::from_constraint(StructuredOutputConstraint::Regex(regex.into())) + } + + pub fn choice(choice: Vec) -> Self { + Self::from_constraint(StructuredOutputConstraint::Choice(choice)) + } + + pub fn grammar(grammar: impl Into) -> Self { + Self::from_constraint(StructuredOutputConstraint::Grammar(grammar.into())) + } + + pub fn json_object() -> Self { + Self::from_constraint(StructuredOutputConstraint::JsonObject) + } + + pub fn structural_tag(structural_tag: impl Into) -> Self { + Self::from_constraint(StructuredOutputConstraint::StructuralTag( + structural_tag.into(), + )) + } + + fn from_constraint(constraint: StructuredOutputConstraint) -> Self { + Self { + constraint, + options: StructuredOutputOptions::default(), + backend: StructuredOutputBackend::default(), + } + } +} + +/// Wire-compatible structured-output payload used by Python engine-core. +/// +/// Python models `StructuredOutputsParams` as a product-shaped dataclass with +/// several optional constraint fields, then validates that exactly one of those +/// fields is present. Rust exposes [`StructuredOutputsParams`] as an enum-backed +/// domain type instead, while using this private wire type for ser/de. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +#[serde(default)] +struct WireStructuredOutputsParams { + json: Option, + regex: Option, + choice: Option>, + grammar: Option, + json_object: Option, + #[serde(skip_serializing_if = "crate::protocol::is_false")] + disable_any_whitespace: bool, + #[serde(skip_serializing_if = "crate::protocol::is_false")] + disable_additional_properties: bool, + whitespace_pattern: Option, + structural_tag: Option, + #[serde( + default, + rename = "_backend", + deserialize_with = "serde_with::rust::deserialize_ignore_any" + )] + backend: StructuredOutputBackend, +} + +impl TryFrom for StructuredOutputsParams { + type Error = Error; + + fn try_from(raw: WireStructuredOutputsParams) -> Result { + use StructuredOutputConstraint::*; + + let mut constraint = None; + + macro_rules! insert_constraint { + ($name:literal, $value:expr) => { + if let Some(value) = $value { + if let Some((existing, _)) = constraint { + return Err(Error::InvalidStructuredOutputsParams { + message: format!( + "multiple structured output constraints specified: {existing}, {}", + $name + ), + }); + } + constraint = Some(($name, value)); + } + }; + } + + insert_constraint!("json", raw.json.map(Json)); + insert_constraint!("regex", raw.regex.map(Regex)); + insert_constraint!("choice", raw.choice.map(Choice)); + insert_constraint!("grammar", raw.grammar.map(Grammar)); + match raw.json_object { + Some(true) => { + insert_constraint!("json_object", Some(JsonObject)) + } + Some(false) => { + return Err(Error::InvalidStructuredOutputsParams { + message: "structured_outputs.json_object must be true if set; omit structured_outputs to disable structured outputs".to_string(), + }); + } + None => {} + } + insert_constraint!("structural_tag", raw.structural_tag.map(StructuralTag)); + + Ok(Self { + constraint: constraint.map(|(_, c)| c).ok_or_else(|| { + Error::InvalidStructuredOutputsParams { + message: "missing structured output constraint".to_string(), + } + })?, + options: StructuredOutputOptions { + disable_any_whitespace: raw.disable_any_whitespace, + disable_additional_properties: raw.disable_additional_properties, + whitespace_pattern: raw.whitespace_pattern, + }, + backend: raw.backend, + }) + } +} + +impl From for WireStructuredOutputsParams { + fn from(params: StructuredOutputsParams) -> Self { + let mut raw = Self { + disable_any_whitespace: params.options.disable_any_whitespace, + disable_additional_properties: params.options.disable_additional_properties, + whitespace_pattern: params.options.whitespace_pattern, + backend: params.backend, + ..Self::default() + }; + + match params.constraint { + StructuredOutputConstraint::Json(json) => raw.json = Some(json), + StructuredOutputConstraint::Regex(regex) => raw.regex = Some(regex), + StructuredOutputConstraint::Choice(choice) => raw.choice = Some(choice), + StructuredOutputConstraint::Grammar(grammar) => raw.grammar = Some(grammar), + StructuredOutputConstraint::JsonObject => raw.json_object = Some(true), + StructuredOutputConstraint::StructuralTag(structural_tag) => { + raw.structural_tag = Some(structural_tag); + } + } + + raw + } +} + +impl Serialize for StructuredOutputsParams { + fn serialize(&self, serializer: S) -> std::result::Result + where + S: Serializer, + { + WireStructuredOutputsParams::from(self.clone()).serialize(serializer) + } +} + +impl<'de> Deserialize<'de> for StructuredOutputsParams { + fn deserialize(deserializer: D) -> std::result::Result + where + D: Deserializer<'de>, + { + WireStructuredOutputsParams::deserialize(deserializer)? + .try_into() + .map_err(serde::de::Error::custom) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn structured_outputs_backend_ignores_deserialized_value() { + let params: StructuredOutputsParams = serde_json::from_value(serde_json::json!({ + "json_object": true, + "_backend": "xgrammar", + })) + .unwrap(); + + assert_eq!(params.backend, StructuredOutputBackend::Guidance); + assert_eq!(params.constraint, StructuredOutputConstraint::JsonObject); + + let value = serde_json::to_value(params).unwrap(); + assert_eq!(value["_backend"], "guidance"); + } + + #[test] + fn structured_outputs_rejects_missing_constraint() { + let error = + serde_json::from_value::(serde_json::json!({})).unwrap_err(); + + assert!(error.to_string().contains("missing structured output constraint")); + } + + #[test] + fn structured_outputs_rejects_multiple_constraints() { + let error = serde_json::from_value::(serde_json::json!({ + "json": {"type": "object"}, + "regex": ".*", + })) + .unwrap_err(); + + assert!( + error + .to_string() + .contains("multiple structured output constraints specified: json, regex") + ); + } + + #[test] + fn structured_outputs_rejects_json_object_false() { + let error = serde_json::from_value::(serde_json::json!({ + "json_object": false, + })) + .unwrap_err(); + + assert!(error.to_string().contains("json_object must be true")); + } + + #[test] + fn structured_outputs_serializes_through_raw_shape() { + let params = StructuredOutputsParams { + constraint: StructuredOutputConstraint::StructuralTag( + r#"{"type":"structural_tag"}"#.to_string(), + ), + options: StructuredOutputOptions::default(), + backend: StructuredOutputBackend::Xgrammar, + }; + + let value = serde_json::to_value(params).unwrap(); + + assert_eq!(value["structural_tag"], r#"{"type":"structural_tag"}"#); + assert_eq!(value["_backend"], "xgrammar"); + assert!(value.get("json").is_none()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/tensor.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/tensor.rs new file mode 100644 index 0000000000000000000000000000000000000000..621166e9f1b43ca279a1035e4c81314649a90852 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/tensor.rs @@ -0,0 +1,255 @@ +use bytemuck::allocation::pod_collect_to_vec; +use enum_as_inner::EnumAsInner; +use half::{bf16, f16}; +use rmpv::Value; +use serde::{Deserialize, Deserializer, Serialize, Serializer}; +use serde_tuple::{Deserialize_tuple, Serialize_tuple}; + +/// Tensors and ndarrays are encoded with this extension type in Python. +/// +/// Original Python definition: +/// +const CUSTOM_TYPE_RAW_VIEW: i8 = 3; + +#[easy_ext::ext(ShapeExt)] +impl [usize] { + /// Returned the total number of elements implied by this shape, or `None` + /// if the product of the dimensions overflows `usize`. + pub fn checked_numel(&self) -> Option { + self.iter().try_fold(1usize, |acc, dim| acc.checked_mul(*dim)) + } +} + +/// Python ndarray/tensor wire tuple encoded as `(dtype, shape, data)`. +/// +/// This matches the custom msgpack representation built by Python +/// `serial_utils.encode_ndarray` / `encode_tensor`. +/// +/// Original Python wire encoders: +/// +/// +#[derive(Debug, Clone, PartialEq, Serialize_tuple, Deserialize_tuple)] +pub struct WireNdArray { + pub dtype: String, + pub shape: Vec, + pub data: WireArrayData, +} + +impl WireNdArray { + /// Build a float32 tensor/ndarray backed by native-endian raw-view bytes. + pub fn from_f32(shape: Vec, data: Vec) -> Result { + validate_element_count(&shape, data.len())?; + Ok(Self { + dtype: "float32".to_string(), + shape, + data: WireArrayData::RawView(pod_collect_to_vec::(&data)), + }) + } + + /// Build a float16 tensor/ndarray backed by native-endian raw-view bytes. + pub fn from_f16(shape: Vec, data: Vec) -> Result { + validate_element_count(&shape, data.len())?; + Ok(Self { + dtype: "float16".to_string(), + shape, + data: WireArrayData::RawView(pod_collect_to_vec::(&data)), + }) + } + + /// Build a bfloat16 tensor/ndarray backed by native-endian raw-view bytes. + pub fn from_bf16(shape: Vec, data: Vec) -> Result { + validate_element_count(&shape, data.len())?; + Ok(Self { + dtype: "bfloat16".to_string(), + shape, + data: WireArrayData::RawView(pod_collect_to_vec::(&data)), + }) + } + + /// Build an int64 tensor/ndarray backed by native-endian raw-view bytes. + pub fn from_i64(shape: Vec, data: Vec) -> Result { + validate_element_count(&shape, data.len())?; + Ok(Self { + dtype: "int64".to_string(), + shape, + data: WireArrayData::RawView(pod_collect_to_vec::(&data)), + }) + } + + /// Build a uint32 tensor/ndarray backed by native-endian raw-view bytes. + pub fn from_u32(shape: Vec, data: Vec) -> Result { + validate_element_count(&shape, data.len())?; + Ok(Self { + dtype: "uint32".to_string(), + shape, + data: WireArrayData::RawView(pod_collect_to_vec::(&data)), + }) + } + + /// Build a bool tensor/ndarray backed by raw-view bytes. + /// + /// This matches `torch.bool` storage: one byte per element, not a packed + /// bitmap. Values are canonicalized as `false -> 0` and `true -> 1`. + pub fn from_bool(shape: Vec, data: Vec) -> Result { + validate_element_count(&shape, data.len())?; + Ok(Self { + dtype: "bool".to_string(), + shape, + data: WireArrayData::RawView(data.iter().map(|value| u8::from(*value)).collect()), + }) + } + + /// Build a tensor/ndarray from already-encoded raw-view bytes. + /// + /// Use this as an escape hatch when the caller already owns bytes that + /// match the requested `dtype` and `shape`. + pub fn from_raw(dtype: impl Into, shape: Vec, data: Vec) -> Self { + Self { + dtype: dtype.into(), + shape, + data: WireArrayData::RawView(data), + } + } +} + +/// Validate that the number of elements implied by the shape matches the length +/// of the data. +fn validate_element_count(shape: &[usize], len: usize) -> Result<(), String> { + let expected = shape + .checked_numel() + .ok_or_else(|| format!("tensor shape product overflows usize: {shape:?}"))?; + if expected == len { + Ok(()) + } else { + Err(format!( + "tensor data length {len} does not match shape {shape:?} product {expected}" + )) + } +} + +/// Python tensor wire tuple encoded as `(dtype, shape, data)`. +/// +/// This is the same wire shape as [`WireNdArray`]; multimodal request payloads +/// use it for `torch.Tensor` values. +pub type WireTensor = WireNdArray; + +/// Python array/tensor payload reference inside [`WireNdArray`]. +/// +/// The data can be either an inline msgpack raw-view extension or an index into +/// the multipart aux-frame list carried alongside the primary msgpack frame. +/// +/// Original Python wire encoders: +/// +/// +#[derive(Debug, Clone, PartialEq, EnumAsInner)] +pub enum WireArrayData { + /// The index of the aux frame where the raw bytes of this array/tensor are + /// stored. + AuxIndex(usize), + /// The raw bytes of this array/tensor. + RawView(Vec), +} + +impl<'de> Deserialize<'de> for WireArrayData { + fn deserialize(deserializer: D) -> std::result::Result + where + D: Deserializer<'de>, + { + let value = Value::deserialize(deserializer)?; + match value { + Value::Ext(tag, bytes) if tag == CUSTOM_TYPE_RAW_VIEW => Ok(Self::RawView(bytes)), + Value::Ext(tag, _) => Err(serde::de::Error::custom(format!( + "unsupported extension type code {tag}" + ))), + Value::Integer(index) => { + index.as_u64().map(|index| Self::AuxIndex(index as usize)).ok_or_else(|| { + serde::de::Error::custom("aux frame index must be a non-negative integer") + }) + } + other => Err(serde::de::Error::custom(format!( + "expected raw-view ext or aux frame index, got {other:?}" + ))), + } + } +} + +impl Serialize for WireArrayData { + fn serialize(&self, serializer: S) -> std::result::Result + where + S: Serializer, + { + // TODO: outbound request serialization currently only supports inline + // raw-view bytes. Emitting aux frames needs transport-level plumbing; + // serializing `AuxIndex` here only preserves an already-built reference. + match self { + Self::AuxIndex(index) => serializer.serialize_u64(*index as u64), + Self::RawView(bytes) => { + Value::Ext(CUSTOM_TYPE_RAW_VIEW, bytes.clone()).serialize(serializer) + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn constructors_build_raw_view_tensors() { + let f32_tensor = WireNdArray::from_f32(vec![2], vec![1.0, 2.5]).unwrap(); + assert_eq!(f32_tensor.dtype, "float32"); + assert_eq!(f32_tensor.shape, vec![2]); + assert_eq!( + f32_tensor.data.into_raw_view().expect("raw view"), + [1.0_f32, 2.5].into_iter().flat_map(f32::to_ne_bytes).collect::>() + ); + + let f16_tensor = + WireNdArray::from_f16(vec![2], vec![f16::from_f32(1.0), f16::from_f32(2.5)]).unwrap(); + assert_eq!(f16_tensor.dtype, "float16"); + assert_eq!(f16_tensor.shape, vec![2]); + assert_eq!(f16_tensor.data.into_raw_view().expect("raw view").len(), 4); + + let bf16_tensor = + WireNdArray::from_bf16(vec![2], vec![bf16::from_f32(1.0), bf16::from_f32(2.5)]) + .unwrap(); + assert_eq!(bf16_tensor.dtype, "bfloat16"); + assert_eq!(bf16_tensor.shape, vec![2]); + assert_eq!(bf16_tensor.data.into_raw_view().expect("raw view").len(), 4); + + let i64_tensor = WireNdArray::from_i64(vec![1], vec![-7]).unwrap(); + assert_eq!(i64_tensor.dtype, "int64"); + assert_eq!( + i64_tensor.data.into_raw_view().expect("raw view"), + (-7_i64).to_ne_bytes() + ); + + let u32_tensor = WireNdArray::from_u32(vec![1], vec![42]).unwrap(); + assert_eq!(u32_tensor.dtype, "uint32"); + assert_eq!( + u32_tensor.data.into_raw_view().expect("raw view"), + 42_u32.to_ne_bytes() + ); + + let bool_tensor = WireNdArray::from_bool(vec![2], vec![false, true]).unwrap(); + assert_eq!(bool_tensor.dtype, "bool"); + assert_eq!( + bool_tensor.data.into_raw_view().expect("raw view"), + vec![0, 1] + ); + + let raw_tensor = WireNdArray::from_raw("custom", vec![3], vec![1, 2, 3]); + assert_eq!(raw_tensor.dtype, "custom"); + assert_eq!(raw_tensor.shape, vec![3]); + assert_eq!( + raw_tensor.data.into_raw_view().expect("raw view"), + vec![1, 2, 3] + ); + } + + #[test] + fn constructors_validate_shape_product() { + let err = WireNdArray::from_f32(vec![2, 2], vec![1.0, 2.0]).unwrap_err(); + assert!(err.contains("does not match shape")); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/utility.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/utility.rs new file mode 100644 index 0000000000000000000000000000000000000000..8ceb2ba735a233d5b2f97a41d37c862d7b4db63a --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/protocol/utility.rs @@ -0,0 +1,399 @@ +use std::any::type_name; +use std::fmt; +use std::str::FromStr; + +use rmpv::Value; +use serde::{Deserialize, Serialize}; +use serde_default::DefaultFromSerde; +use serde_tuple::{Deserialize_tuple, Serialize_tuple}; +use serde_with::{DeserializeFromStr, SerializeDisplay}; +use thiserror_ext::AsReport; + +use super::{OpaqueValue, default_opaque_value_nil}; +use crate::error::{Error, Result}; + +/// How pause/sleep utility calls handle in-flight requests. +/// +/// Use display/from-str serde so MessagePack utility args stay as Python +/// literal strings instead of serde enum variant tuples. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, SerializeDisplay, DeserializeFromStr)] +pub enum PauseMode { + /// Abort all in-flight requests immediately. + #[default] + Abort, + /// Wait for in-flight requests to complete. + Wait, + /// Freeze queued requests so they can resume later. + Keep, +} + +impl PauseMode { + /// Return the Python literal used on the utility-call wire. + pub fn as_str(self) -> &'static str { + match self { + Self::Abort => "abort", + Self::Wait => "wait", + Self::Keep => "keep", + } + } +} + +impl fmt::Display for PauseMode { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.as_str()) + } +} + +impl FromStr for PauseMode { + type Err = String; + + fn from_str(value: &str) -> std::result::Result { + match value { + "abort" => Ok(Self::Abort), + "wait" => Ok(Self::Wait), + "keep" => Ok(Self::Keep), + other => Err(format!( + "invalid pause mode `{other}`; expected one of: abort, wait, keep" + )), + } + } +} + +/// Utility call id as carried on the engine-core MessagePack wire. +/// +/// Python emits utility ids as MessagePack integers, including values that may +/// require unsigned 64-bit encoding. Keep MessagePack's signed/unsigned +/// integer distinction instead of flattening to `i64` or `u64` at decode time. +#[derive(Clone, Copy, PartialEq)] +pub struct UtilityCallId(rmpv::Integer); + +impl UtilityCallId { + /// Returns the integer represented as `u64` if possible, or else `None`. + /// This is the typical case for utility calls. + pub fn as_u64(self) -> Option { + self.0.as_u64() + } + + /// Returns the integer represented as `i64` if possible, or else `None`. + pub fn as_i64(self) -> Option { + self.0.as_i64() + } +} + +impl Default for UtilityCallId { + fn default() -> Self { + Self(0_u64.into()) + } +} + +impl From for UtilityCallId { + fn from(value: u64) -> Self { + Self(value.into()) + } +} + +impl TryFrom for UtilityCallId { + type Error = String; + + fn try_from(value: Value) -> std::result::Result { + match value { + Value::Integer(value) => Ok(UtilityCallId(value)), + other => Err(format!( + "expected a MessagePack integer utility call id, got {other}" + )), + } + } +} + +impl PartialEq for UtilityCallId { + fn eq(&self, other: &u64) -> bool { + self.as_u64() == Some(*other) + } +} + +impl fmt::Debug for UtilityCallId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + fmt::Display::fmt(&self.0, f) + } +} + +impl fmt::Display for UtilityCallId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + fmt::Display::fmt(&self.0, f) + } +} + +impl Serialize for UtilityCallId { + fn serialize(&self, serializer: S) -> std::result::Result + where + S: serde::Serializer, + { + Value::Integer(self.0).serialize(serializer) + } +} + +impl<'de> Deserialize<'de> for UtilityCallId { + fn deserialize(deserializer: D) -> std::result::Result + where + D: serde::Deserializer<'de>, + { + let value = Value::deserialize(deserializer)?; + match value { + Value::Integer(value) => Ok(UtilityCallId(value)), + other => Err(serde::de::Error::custom(format!( + "expected a MessagePack integer utility call id, got {other}" + ))), + } + } +} + +/// Engine-core utility call payload sent from frontend to engine. +/// +/// Original Python payload shape: +/// `(client_index, call_id, method_name, args)` +#[derive(Debug, Clone, PartialEq, Serialize_tuple, Deserialize_tuple)] +pub struct EngineCoreUtilityRequest { + pub client_index: u32, + pub call_id: UtilityCallId, + pub method_name: String, + pub args: OpaqueValue, +} + +impl EngineCoreUtilityRequest { + /// Create a new utility request with the given strongly typed arguments, + /// encoding them into the expected msgpack value format. + pub fn new( + client_index: u32, + call_id: u64, + method_name: impl Into, + args: T, + ) -> Result + where + T: Serialize + std::fmt::Debug, + { + let args = rmpv::ext::to_value(&args).map_err(|error| Error::Encode { + target_type: type_name::(), + message: format!( + "failed to encode utility args `{args:?}`: {}", + error.to_report_string() + ), + })?; + let args = match args { + Value::Nil => Value::Array(Vec::new()), + other => other, + }; + + Ok(Self { + client_index, + call_id: UtilityCallId::from(call_id), + method_name: method_name.into(), + args, + }) + } +} + +/// Result of a utility call. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, PartialEq, Serialize_tuple, Deserialize_tuple, DefaultFromSerde)] +pub struct UtilityOutput { + pub call_id: UtilityCallId, + /// Non-`None` implies the call failed and `result` should be ignored. + #[serde(default)] + pub failure_message: Option, + #[serde(default)] + pub result: Option, +} + +/// Python `UtilityResult` wrapper carried inside `UtilityOutput.result`. +/// +/// Upstream reference: +/// +#[derive(Debug, Clone, PartialEq, Serialize_tuple, Deserialize_tuple)] +pub struct UtilityResultEnvelope { + /// Recursive type information encoded on Python side, serving as the hint + /// for deserialization. We don't care it here as in Rust frontend all + /// utility calls are strongly-typed. + #[serde(default)] + type_info: Option, + /// The actual utility result. + #[serde(default = "default_opaque_value_nil")] + result: OpaqueValue, +} + +impl UtilityResultEnvelope { + /// Create a utility result envelope without type information. + pub fn without_type_info(result: OpaqueValue) -> Self { + Self { + type_info: None, + result, + } + } +} + +impl UtilityOutput { + /// Decode the typed result of a utility call. + pub fn into_typed_result(self, method: &str) -> Result + where + T: serde::de::DeserializeOwned, + { + if let Some(message) = self.failure_message { + return Err(Error::UtilityCallFailed { + method: method.to_string(), + call_id: self.call_id, + message, + }); + } + + let result = self.result.map(|e| e.result).unwrap_or(Value::Nil); + + rmpv::ext::from_value(result).map_err(|error| Error::UtilityResultDecode { + method: method.to_string(), + call_id: self.call_id, + message: error.to_report_string(), + }) + } +} + +#[cfg(test)] +mod tests { + use rmpv::Value; + use serde::Serialize; + + use super::{EngineCoreUtilityRequest, PauseMode, UtilityOutput, UtilityResultEnvelope}; + use crate::Error; + use crate::protocol::{decode_msgpack, decode_value, encode_msgpack}; + + fn utility_result_value(value: T) -> UtilityResultEnvelope + where + T: Serialize, + { + UtilityResultEnvelope::without_type_info(rmpv::ext::to_value(value).unwrap()) + } + + #[test] + fn utility_request_serializes_as_tuple_payload() { + let request = EngineCoreUtilityRequest::new(7, 42, "is_sleeping", ()).unwrap(); + + let encoded = encode_msgpack(&request).unwrap(); + let value = decode_value(&encoded).unwrap(); + let array = match value { + Value::Array(array) => array, + other => panic!("expected utility request array, got {other:?}"), + }; + + assert_eq!(array.len(), 4); + assert_eq!(array[0], Value::from(7)); + assert_eq!(array[1], Value::from(42)); + assert_eq!(array[2], Value::from("is_sleeping")); + assert_eq!(array[3], Value::Array(Vec::new())); + } + + #[test] + fn pause_mode_serializes_as_python_literal() { + let request = + EngineCoreUtilityRequest::new(7, 42, "pause_scheduler", (PauseMode::Abort, true)) + .unwrap(); + + let encoded = encode_msgpack(&request).unwrap(); + let value = decode_value(&encoded).unwrap(); + let array = match value { + Value::Array(array) => array, + other => panic!("expected utility request array, got {other:?}"), + }; + + assert_eq!(array[2], Value::from("pause_scheduler")); + assert_eq!( + array[3], + Value::Array(vec![Value::from("abort"), Value::from(true)]) + ); + } + + #[test] + fn utility_output_decodes_typed_result() { + let output = UtilityOutput { + call_id: 9_u64.into(), + failure_message: None, + result: Some(utility_result_value(true)), + }; + + assert!(output.into_typed_result::("is_sleeping").unwrap()); + } + + #[test] + fn utility_output_decodes_unsigned_64_bit_call_id() { + let value = Value::Array(vec![Value::from(u64::MAX), Value::Nil, Value::Nil]); + let mut encoded = Vec::new(); + rmpv::encode::write_value(&mut encoded, &value).unwrap(); + + let output: UtilityOutput = decode_msgpack(&encoded).unwrap(); + + assert_eq!(output.call_id.as_u64(), Some(u64::MAX)); + } + + #[test] + fn utility_output_decodes_signed_negative_call_id() { + let value = Value::Array(vec![Value::from(-1), Value::Nil, Value::Nil]); + let mut encoded = Vec::new(); + rmpv::encode::write_value(&mut encoded, &value).unwrap(); + + let output: UtilityOutput = decode_msgpack(&encoded).unwrap(); + + assert_eq!(output.call_id.as_u64(), None); + } + + #[test] + fn utility_output_decodes_other_negative_call_id() { + let value = Value::Array(vec![Value::from(-2), Value::Nil, Value::Nil]); + let mut encoded = Vec::new(); + rmpv::encode::write_value(&mut encoded, &value).unwrap(); + + let output: UtilityOutput = decode_msgpack(&encoded).unwrap(); + + assert_eq!(output.call_id.as_u64(), None); + assert_eq!(output.call_id.as_i64(), Some(-2)); + } + + #[test] + fn utility_output_reports_failure_message() { + let error = UtilityOutput { + call_id: 9_u64.into(), + failure_message: Some("boom".to_string()), + result: None, + } + .into_typed_result::("is_sleeping") + .unwrap_err(); + + assert!(matches!( + error, + Error::UtilityCallFailed { + method, + call_id, + message + } if method == "is_sleeping" && call_id == 9 && message == "boom" + )); + } + + #[test] + fn utility_output_decodes_missing_result_as_unit() { + UtilityOutput { + call_id: 3_u64.into(), + failure_message: None, + result: None, + } + .into_typed_result::<()>("reset_mm_cache") + .unwrap(); + } + + #[test] + fn utility_output_decodes_nil_result_as_unit() { + UtilityOutput { + call_id: 4_u64.into(), + failure_message: None, + result: Some(UtilityResultEnvelope::without_type_info(Value::Nil)), + } + .into_typed_result::<()>("sleep") + .unwrap(); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/runtime.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/runtime.rs new file mode 100644 index 0000000000000000000000000000000000000000..dad849a29e1f0c7e7bd715faecdf8f2cc8b733d0 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/runtime.rs @@ -0,0 +1,75 @@ +use std::mem::ManuallyDrop; +use std::ops::{Deref, DerefMut}; +use std::sync::OnceLock; +use std::sync::atomic::{AtomicUsize, Ordering}; + +use tokio::runtime::Runtime; + +/// A wrapper around [`Runtime`] that shuts down the runtime in the background when dropped. +/// +/// This can be useful in some cases, because sometimes we want to drop the runtime without +/// blocking the current thread, for example, when it's nested inside another runtime. +pub struct BackgroundShutdownRuntime(ManuallyDrop); + +impl Drop for BackgroundShutdownRuntime { + fn drop(&mut self) { + // Safety: The runtime is only dropped once here. + let runtime = unsafe { ManuallyDrop::take(&mut self.0) }; + runtime.shutdown_background(); + } +} + +impl Deref for BackgroundShutdownRuntime { + type Target = Runtime; + + fn deref(&self) -> &Self::Target { + &self.0 + } +} + +impl DerefMut for BackgroundShutdownRuntime { + fn deref_mut(&mut self) -> &mut Self::Target { + &mut self.0 + } +} + +impl From for BackgroundShutdownRuntime { + fn from(runtime: Runtime) -> Self { + Self(ManuallyDrop::new(runtime)) + } +} + +const ZMQ_WORKER_THREADS_ENV: &str = "VLLM_RS_ZMQ_WORKER_THREADS"; +/// The number of tasks running on the ZMQ runtime is fixed and expected to remain +/// small, and multiple engines share the same ZMQ socket. Therefore, based on +/// benchmarks, a default value of 4 is generally sufficient. +const DEFAULT_ZMQ_WORKER_THREADS: usize = 4; + +static ZMQ_RUNTIME_SEQUENCE: OnceLock = OnceLock::new(); + +/// Build a Tokio runtime for ZMQ tasks. Multiple calls to this function will +/// return multiple runtimes with distinct thread name suffixes. +pub(crate) fn build_zmq_runtime() -> BackgroundShutdownRuntime { + let sequence = ZMQ_RUNTIME_SEQUENCE + .get_or_init(|| AtomicUsize::new(0)) + .fetch_add(1, Ordering::Relaxed); + + tokio::runtime::Builder::new_multi_thread() + .worker_threads(zmq_worker_threads()) + .thread_name_fn(move || format!("vllm-zmq-{sequence}")) + .enable_all() + .build() + .expect("failed to build vLLM ZMQ runtime") + .into() +} + +/// Get the number of worker threads to use for the ZMQ runtime. If env var +/// `VLLM_RS_ZMQ_WORKER_THREADS` is set and a valid positive integer, it will be used. +/// Otherwise, the default value of `DEFAULT_ZMQ_WORKER_THREADS` will be used. +fn zmq_worker_threads() -> usize { + std::env::var(ZMQ_WORKER_THREADS_ENV) + .ok() + .and_then(|value| value.parse::().ok()) + .filter(|value| *value > 0) + .unwrap_or(DEFAULT_ZMQ_WORKER_THREADS) +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/test_utils.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/test_utils.rs new file mode 100644 index 0000000000000000000000000000000000000000..372a355bfbe24dc09d09e650fa6d5147ec3cceb1 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/test_utils.rs @@ -0,0 +1,204 @@ +use std::future::Future; +use std::path::Path; +use std::pin::Pin; + +use tempfile::TempDir; +use tokio::sync::oneshot; +use zeromq::{DealerSocket, PushSocket}; + +use crate::EngineId; +pub use crate::mock_engine::{MockCoordinatorSockets, MockEngineSockets}; +use crate::mock_engine::{ + MockEngineConfig, MockEngineDataSockets, connect_to_bootstrapped_frontend, connect_to_frontend, + default_ready_response, +}; +use crate::protocol::handshake::{EngineCoreReadyResponse, HandshakeInitMessage}; + +/// Per-test IPC endpoint namespace backed by a unique temporary directory. +/// +/// Using one directory per test avoids endpoint collisions without requiring +/// ad-hoc unique-name generation at each call site. +#[derive(Debug)] +pub struct IpcNamespace { + dir: TempDir, +} + +impl IpcNamespace { + /// Create a fresh namespace for one test case. + pub fn new() -> std::io::Result { + Ok(Self { + dir: TempDir::new()?, + }) + } + + /// Build one `ipc://...` endpoint under this namespace. + pub fn endpoint(&self, name: impl AsRef) -> String { + let path = self.dir.path().join(name); + format!("ipc://{}", path.to_string_lossy()) + } + + /// Endpoint used for the initial READY/HELLO handshake. + pub fn handshake_endpoint(&self) -> String { + self.endpoint("handshake.sock") + } + + /// Endpoint used for engine-core request traffic. + pub fn input_endpoint(&self) -> String { + self.endpoint("input.sock") + } + + /// Endpoint used for engine-core output traffic. + pub fn output_endpoint(&self) -> String { + self.endpoint("output.sock") + } +} + +fn test_mock_engine_config() -> MockEngineConfig { + MockEngineConfig { + local: true, + headless: true, + ready_response: default_ready_response(), + ..Default::default() + } +} + +fn test_mock_engine_config_with_ready(ready_response: EngineCoreReadyResponse) -> MockEngineConfig { + MockEngineConfig { + local: true, + headless: true, + ready_response, + ..Default::default() + } +} + +/// Complete the engine-core handshake and connect mock input/output sockets +/// plus optional coordinator sockets. +pub async fn setup_mock_engine_sockets( + engine_handshake: String, + engine_id: impl Into, +) -> MockEngineSockets { + connect_to_frontend(engine_handshake, engine_id, test_mock_engine_config()) + .await + .expect("connect mock engine") +} + +/// Connect one mock engine directly to already-bootstrapped frontend +/// input/output sockets. +pub async fn setup_bootstrapped_mock_engine( + input_address: String, + output_address: String, + engine_id: impl Into, +) -> (DealerSocket, PushSocket) { + connect_to_bootstrapped_frontend( + input_address, + output_address, + engine_id, + test_mock_engine_config(), + ) + .await + .expect("connect bootstrapped mock engine") +} + +/// Complete the engine-core handshake and connect mock input/output sockets. +/// +/// This returns the decoded handshake init message plus the `DealerSocket` used +/// to receive client requests and the `PushSocket` used to send engine outputs +/// back to the client. +pub async fn setup_mock_engine_with_init( + engine_handshake: String, + engine_id: impl Into, +) -> (HandshakeInitMessage, DealerSocket, PushSocket) { + let MockEngineSockets { + init, data_sockets, .. + } = setup_mock_engine_sockets(engine_handshake, engine_id).await; + let MockEngineDataSockets { dealer, push } = + data_sockets.into_iter().next().expect("mock engine data socket"); + (init, dealer, push) +} + +/// Complete the engine-core handshake and connect mock input/output sockets. +/// +/// This returns the `DealerSocket` used to receive client requests and the +/// `PushSocket` used to send engine outputs back to the client. +pub async fn setup_mock_engine( + engine_handshake: String, + engine_id: impl Into, +) -> (DealerSocket, PushSocket) { + let (_, dealer, push) = setup_mock_engine_with_init(engine_handshake, engine_id).await; + (dealer, push) +} + +/// Spawn a mock engine task and keep its sockets alive until the returned +/// shutdown sender is triggered by the test. +/// +/// The script borrows the connected sockets mutably while it runs. After the +/// script completes, this helper keeps the sockets alive until the test +/// explicitly signals shutdown. +pub fn spawn_mock_engine_task( + engine_handshake: String, + engine_id: impl Into, + run: F, +) -> (oneshot::Sender<()>, tokio::task::JoinHandle<()>) +where + F: for<'a> FnOnce( + &'a mut DealerSocket, + &'a mut PushSocket, + ) -> Pin + Send + 'a>> + + Send + + 'static, +{ + let (shutdown_tx, shutdown_rx) = oneshot::channel(); + let engine_id = engine_id.into(); + let engine_task = tokio::spawn(async move { + let (mut dealer, mut push) = setup_mock_engine(engine_handshake, engine_id).await; + run(&mut dealer, &mut push).await; + let _ = shutdown_rx.await; + }); + (shutdown_tx, engine_task) +} + +/// Like [`setup_mock_engine`] but uses a custom ready response for the +/// handshake, allowing tests to control `world_size`, `data_parallel_size`, +/// etc. +async fn setup_mock_engine_with_ready( + engine_handshake: String, + engine_id: impl Into, + ready_response: EngineCoreReadyResponse, +) -> (DealerSocket, PushSocket) { + let config = test_mock_engine_config_with_ready(ready_response); + let MockEngineSockets { data_sockets, .. } = + connect_to_frontend(engine_handshake, engine_id, config) + .await + .expect("connect mock engine with custom ready response"); + let MockEngineDataSockets { dealer, push } = + data_sockets.into_iter().next().expect("mock engine data socket"); + (dealer, push) +} + +/// Like [`spawn_mock_engine_task`] but uses a custom ready response for the +/// handshake, allowing tests to set `world_size` and `data_parallel_size` to +/// non-default values. +pub fn spawn_mock_engine_task_with_ready( + engine_handshake: String, + engine_id: impl Into, + ready_response: EngineCoreReadyResponse, + run: F, +) -> (oneshot::Sender<()>, tokio::task::JoinHandle<()>) +where + F: for<'a> FnOnce( + &'a mut DealerSocket, + &'a mut PushSocket, + ) -> Pin + Send + 'a>> + + Send + + 'static, +{ + let (shutdown_tx, shutdown_rx) = oneshot::channel(); + let engine_id = engine_id.into(); + let engine_task = tokio::spawn(async move { + let (mut dealer, mut push) = + setup_mock_engine_with_ready(engine_handshake, engine_id, ready_response).await; + run(&mut dealer, &mut push).await; + let _ = shutdown_rx.await; + }); + (shutdown_tx, engine_task) +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/tests/client.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/tests/client.rs new file mode 100644 index 0000000000000000000000000000000000000000..1757c563d6b6bcbaa5f7e5fb5f117246879469ad --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/tests/client.rs @@ -0,0 +1,2985 @@ +use std::collections::{BTreeMap, BTreeSet}; +use std::convert::TryFrom; +use std::io::Cursor; +use std::path::PathBuf; +use std::process::Command; +use std::sync::Once; +use std::time::Duration; + +use futures::StreamExt; +use rmpv::Value; +use thiserror_ext::AsReport as _; +use tokio::sync::{mpsc, oneshot}; +use tokio::time::timeout; +use tracing_subscriber::EnvFilter; +use zeromq::prelude::{Socket, SocketRecv, SocketSend}; +use zeromq::util::PeerIdentity; +use zeromq::{DealerSocket, PushSocket, SocketOptions, SubSocket, XPubSocket, ZmqMessage}; + +use crate::protocol::handshake::{HandshakeInitMessage, ReadyMessage}; +use crate::protocol::logprobs::MaybeWireLogprobs; +use crate::protocol::multimodal::{ + MmFeatureSpec, MmField, MmFieldElem, MmFlatField, MmKwargValue, MmSlice, PlaceholderRange, + SliceSpec, +}; +use crate::protocol::output::{ + DpControlMessage, DpControlOutput, EngineCoreFinishReason, EngineCoreOutput, EngineCoreOutputs, + RequestBatchOutputs, UtilityCallOutput, decode_engine_core_outputs, +}; +use crate::protocol::request::{EngineCoreRequest, EngineCoreRequestType}; +use crate::protocol::sampling::EngineCoreSamplingParams; +use crate::protocol::stats::SchedulerStats; +use crate::protocol::tensor::WireTensor; +use crate::protocol::utility::{UtilityOutput, UtilityResultEnvelope}; +use crate::test_utils::{ + IpcNamespace, setup_bootstrapped_mock_engine, setup_mock_engine_sockets, + setup_mock_engine_with_init, spawn_mock_engine_task, +}; +use crate::{ + CoordinatorMode, ENGINE_CORE_DEAD_SENTINEL, EngineCoreClient, EngineCoreClientConfig, EngineId, + Error, TransportMode, +}; + +static TRACING: Once = Once::new(); + +fn expect_sample_logprobs(actual: &MaybeWireLogprobs) { + expect_test::expect![[r#" + Logprobs { + positions: [ + PositionLogprobs { + entries: [ + TokenLogprob { + token_id: 1, + logprob: 1.0, + rank: 1, + }, + TokenLogprob { + token_id: 2, + logprob: 2.0, + rank: 1, + }, + TokenLogprob { + token_id: 3, + logprob: 3.0, + rank: 2, + }, + ], + }, + PositionLogprobs { + entries: [ + TokenLogprob { + token_id: 4, + logprob: 4.0, + rank: 2, + }, + TokenLogprob { + token_id: 5, + logprob: 5.0, + rank: 1, + }, + TokenLogprob { + token_id: 6, + logprob: 6.0, + rank: 2, + }, + ], + }, + ], + } + "#]] + .assert_debug_eq(actual.as_direct().expect("logprobs resolved")); +} + +fn expect_prompt_logprobs(actual: &MaybeWireLogprobs) { + expect_test::expect![[r#" + Logprobs { + positions: [ + PositionLogprobs { + entries: [ + TokenLogprob { + token_id: 10, + logprob: 10.0, + rank: 3, + }, + TokenLogprob { + token_id: 11, + logprob: 11.0, + rank: 1, + }, + TokenLogprob { + token_id: 12, + logprob: 12.0, + rank: 2, + }, + ], + }, + PositionLogprobs { + entries: [ + TokenLogprob { + token_id: 13, + logprob: 13.0, + rank: 4, + }, + TokenLogprob { + token_id: 14, + logprob: 14.0, + rank: 1, + }, + TokenLogprob { + token_id: 15, + logprob: 15.0, + rank: 2, + }, + ], + }, + ], + } + "#]] + .assert_debug_eq(actual.as_direct().expect("prompt logprobs resolved")); +} + +fn sample_request() -> EngineCoreRequest { + sample_request_with_id("req-1") +} + +fn sample_request_with_id(request_id: &str) -> EngineCoreRequest { + EngineCoreRequest { + request_id: request_id.to_string(), + prompt_token_ids: Some(vec![11, 22]), + sampling_params: Some(EngineCoreSamplingParams { + temperature: 0.8, + top_p: 0.9, + top_k: 8, + max_tokens: 32, + min_tokens: 1, + thinking_token_budget: Some(256), + stop_token_ids: vec![151643], + eos_token_id: Some(151645), + all_stop_token_ids: BTreeSet::from([151643, 151645]), + ..EngineCoreSamplingParams::for_test() + }), + arrival_time: 42.5, + ..EngineCoreRequest::default() + } +} + +fn sample_multimodal_request() -> EngineCoreRequest { + EngineCoreRequest { + request_id: "req-mm".to_string(), + prompt_token_ids: Some(vec![101, 102, 103, 104]), + mm_features: Some(vec![MmFeatureSpec { + data: Some(BTreeMap::from([( + "pixel_values".to_string(), + MmFieldElem { + data: Some(MmKwargValue::Tensor( + WireTensor::from_f32(vec![2, 2], vec![1.0, 2.0, 3.5, 4.25]) + .expect("valid tensor shape"), + )), + field: MmField::Flat(MmFlatField { + slices: vec![MmSlice::Slice(SliceSpec { + start: Some(0), + stop: Some(2), + step: None, + })], + dim: 0, + keep_on_cpu: false, + }), + }, + )])), + modality: "image".to_string(), + identifier: "mm-cache-key".to_string(), + mm_position: PlaceholderRange { + offset: 1, + length: 2, + is_embed: None, + }, + mm_hash: Some("processor-hash".to_string()), + }]), + sampling_params: None, + pooling_params: None, + arrival_time: 43.5, + ..EngineCoreRequest::default() + } +} + +fn ready_message(status: &str) -> ReadyMessage { + ReadyMessage { + status: Some(status.to_string()), + local: Some(true), + headless: Some(true), + parallel_config_hash: None, + } +} + +fn request_output( + request_id: &str, + new_token_ids: Vec, + finish_reason: Option, +) -> EngineCoreOutput { + EngineCoreOutput { + request_id: request_id.to_string(), + new_token_ids, + new_logprobs: None, + new_prompt_logprobs_tensors: None, + pooling_output: None, + finish_reason, + stop_reason: None, + events: None, + kv_transfer_params: None, + trace_headers: None, + prefill_stats: None, + routed_experts: None, + num_nans_in_logits: 0, + } +} + +async fn send_outputs(push: &mut PushSocket, outputs: EngineCoreOutputs) { + push.send(ZmqMessage::from(rmp_serde::to_vec_named(&outputs).unwrap())) + .await + .unwrap(); +} + +async fn send_output_frames(push: &mut PushSocket, frames: Vec) { + push.send(ZmqMessage::try_from(frames).unwrap()).await.unwrap(); +} + +async fn recv_engine_message(dealer: &mut DealerSocket) -> Vec { + dealer.recv().await.unwrap().into_vec() +} + +async fn recv_start_dp_wave(sub: &mut SubSocket) -> (u32, u32) { + let frames = sub.recv().await.unwrap().into_vec(); + assert_eq!(frames.len(), 2); + assert_eq!( + frames[0].as_ref(), + EngineCoreRequestType::StartDpWave.to_frame().as_ref() + ); + rmp_serde::from_slice(&frames[1]).expect("decode START_DP_WAVE payload") +} + +async fn connect_client_with_ipc( + config: EngineCoreClientConfig, + ipc: &IpcNamespace, +) -> EngineCoreClient { + EngineCoreClient::connect( + config.with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .unwrap() +} + +fn handshake_test_config( + handshake_address: String, + engine_count: usize, + model_name: &str, + ready_timeout: Duration, + client_index: u32, + coordinator_mode: Option, +) -> EngineCoreClientConfig { + EngineCoreClientConfig { + transport_mode: TransportMode::HandshakeOwner { + handshake_address, + advertised_host: "127.0.0.1".to_string(), + engine_count, + ready_timeout, + local_input_address: None, + local_output_address: None, + }, + coordinator_mode, + model_name: model_name.to_string(), + client_index, + } +} + +fn bootstrapped_test_config( + input_address: String, + output_address: String, + engine_count: usize, + ready_timeout: Duration, + client_index: u32, + coordinator_mode: Option, +) -> EngineCoreClientConfig { + EngineCoreClientConfig { + transport_mode: TransportMode::Bootstrapped { + input_address, + output_address, + engine_start_index: 0, + engine_count, + ready_timeout, + }, + coordinator_mode, + model_name: "test-model".to_string(), + client_index, + } +} + +fn bootstrapped_test_config_with_start_index( + input_address: String, + output_address: String, + engine_start_index: u32, + engine_count: usize, + ready_timeout: Duration, + client_index: u32, + coordinator_mode: Option, +) -> EngineCoreClientConfig { + let mut config = bootstrapped_test_config( + input_address, + output_address, + engine_count, + ready_timeout, + client_index, + coordinator_mode, + ); + let TransportMode::Bootstrapped { + engine_start_index: start, + .. + } = &mut config.transport_mode + else { + unreachable!("bootstrapped_test_config returns bootstrapped transport") + }; + *start = engine_start_index; + config +} + +async fn recv_xpub_message(xpub: &mut XPubSocket) -> Vec { + xpub.recv().await.unwrap().into_vec() +} + +async fn recv_xpub_subscription(xpub: &mut XPubSocket) { + let frames = recv_xpub_message(xpub).await; + assert_eq!(frames.len(), 1); + assert_eq!(frames[0].as_ref(), b"\x01"); +} + +async fn recv_external_coordinator_wakeup(xpub: &mut XPubSocket) -> (u32, u32) { + let frames = recv_xpub_message(xpub).await; + assert_eq!(frames.len(), 1); + rmp_serde::from_slice(&frames[0]).expect("decode external coordinator wakeup") +} + +async fn send_external_coordinator_publish( + xpub: &mut XPubSocket, + payload: &T, +) { + xpub.send(ZmqMessage::from(rmp_serde::to_vec_named(payload).unwrap())) + .await + .unwrap(); +} + +fn spawn_mock_engine_task_with_init( + engine_handshake: String, + engine_id: impl Into, + run: F, +) -> ( + oneshot::Receiver, + oneshot::Sender<()>, + tokio::task::JoinHandle<()>, +) +where + F: for<'a> FnOnce( + &'a mut DealerSocket, + &'a mut PushSocket, + ) + -> std::pin::Pin + Send + 'a>> + + Send + + 'static, +{ + let (shutdown_tx, shutdown_rx) = oneshot::channel(); + let (init_tx, init_rx) = oneshot::channel(); + let engine_id = engine_id.into(); + let engine_task = tokio::spawn(async move { + let (init, mut dealer, mut push) = + setup_mock_engine_with_init(engine_handshake, engine_id).await; + let _ = init_tx.send(init); + run(&mut dealer, &mut push).await; + let _ = shutdown_rx.await; + }); + (init_rx, shutdown_tx, engine_task) +} + +fn init_tracing() { + TRACING.call_once(|| { + let filter = EnvFilter::try_from_default_env() + .unwrap_or_else(|_| EnvFilter::new("vllm_engine_core_client=debug")); + let _ = tracing_subscriber::fmt().with_test_writer().with_env_filter(filter).try_init(); + }); +} + +fn is_dispatcher_closed(error: &Error) -> bool { + match error { + Error::DispatcherClosed { .. } => true, + Error::Shared(error) => is_dispatcher_closed(error), + _ => false, + } +} + +fn is_engine_core_dead(error: &Error) -> bool { + match error { + Error::EngineCoreDead => true, + Error::Shared(error) => is_engine_core_dead(error), + _ => false, + } +} + +fn is_decode_error(error: &Error) -> bool { + match error { + Error::Decode { .. } | Error::ExtValueDecode { .. } => true, + Error::Shared(error) => is_decode_error(error), + _ => false, + } +} + +fn is_unexpected_dispatcher_output(error: &Error) -> bool { + match error { + Error::UnexpectedDispatcherOutput { .. } => true, + Error::Shared(error) => is_unexpected_dispatcher_output(error), + _ => false, + } +} + +fn decode_value(bytes: &[u8]) -> Value { + rmpv::decode::read_value(&mut Cursor::new(bytes)).unwrap() +} + +fn encode_value(value: &Value) -> Vec { + let mut out = Vec::new(); + rmpv::encode::write_value(&mut out, value).unwrap(); + out +} + +fn ndarray_value(dtype: &str, shape: &[usize], data: Value) -> Value { + Value::Array(vec![ + Value::from(dtype), + Value::Array(shape.iter().copied().map(Value::from).collect()), + data, + ]) +} + +fn multipart_logprob_output_frames(request_id: &str) -> Vec { + let main = Value::Array(vec![ + Value::from(0), + Value::Array(vec![Value::Array(vec![ + Value::from(request_id), + Value::Array(vec![Value::from(7), Value::from(8)]), + Value::Array(vec![ + ndarray_value("(value: T) -> UtilityResultEnvelope +where + T: serde::Serialize, +{ + UtilityResultEnvelope::without_type_info(rmpv::ext::to_value(value).unwrap()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn coordinator_handshake_includes_engine_control_addresses() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = [0x00, 0x00]; + + let (init_tx, init_rx) = oneshot::channel(); + let (shutdown_tx, shutdown_rx) = oneshot::channel(); + let engine_task = tokio::spawn(async move { + let sockets = setup_mock_engine_sockets(handshake_address, &engine_id).await; + let _ = init_tx.send(sockets.init.clone()); + let _ = shutdown_rx.await; + }); + + let client = connect_client_with_ipc( + handshake_test_config( + ipc.handshake_endpoint(), + 1, + "test-model", + Duration::from_secs(2), + 0, + Some(CoordinatorMode::InProc), + ), + &ipc, + ) + .await; + + let init = init_rx.await.unwrap(); + assert!(init.addresses.coordinator_input.is_some()); + assert!(init.addresses.coordinator_output.is_some()); + assert!(init.addresses.frontend_stats_publish_address.is_none()); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn coordinator_wave_control_tracks_pause_running_and_rebroadcasts() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + + let (shutdown0_tx, shutdown0_rx) = oneshot::channel(); + let engine0_task = tokio::spawn({ + let handshake_address = handshake_address.clone(); + async move { + let mut engine = setup_mock_engine_sockets(handshake_address, &[0x00, 0x00]).await; + let mut coordinator = + engine.coordinator.take().expect("coordinator sockets should be present"); + let data_socket = engine.data_sockets.first_mut().expect("data socket"); + + let (wave, exclude_engine) = recv_start_dp_wave(&mut coordinator.input_sub).await; + assert_eq!((wave, exclude_engine), (0, 0)); + + let add = recv_engine_message(&mut data_socket.dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = rmp_serde::from_slice(&add[1]).unwrap(); + assert_eq!(request.request_id, "req-1"); + assert_eq!(request.current_wave, 0); + + assert!( + timeout( + Duration::from_millis(200), + recv_start_dp_wave(&mut coordinator.input_sub) + ) + .await + .is_err() + ); + + send_outputs( + &mut data_socket.push, + RequestBatchOutputs { + outputs: vec![request_output( + "req-1", + vec![], + Some(EngineCoreFinishReason::Length), + )], + finished_requests: Some(BTreeSet::from(["req-1".to_string()])), + ..Default::default() + } + .into(), + ) + .await; + + send_outputs( + &mut coordinator.output_push, + DpControlOutput { + engine_index: 0, + timestamp: 0.0, + control: DpControlMessage::WaveComplete(0), + } + .into(), + ) + .await; + + let (wave, exclude_engine) = recv_start_dp_wave(&mut coordinator.input_sub).await; + assert_eq!((wave, exclude_engine), (1, 0)); + + let add = recv_engine_message(&mut data_socket.dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = rmp_serde::from_slice(&add[1]).unwrap(); + assert_eq!(request.request_id, "req-3"); + assert_eq!(request.current_wave, 1); + + send_outputs( + &mut data_socket.push, + RequestBatchOutputs { + outputs: vec![request_output( + "req-3", + vec![], + Some(EngineCoreFinishReason::Length), + )], + finished_requests: Some(BTreeSet::from(["req-3".to_string()])), + ..Default::default() + } + .into(), + ) + .await; + + let _ = shutdown0_rx.await; + } + }); + + let (shutdown1_tx, shutdown1_rx) = oneshot::channel(); + let engine1_task = tokio::spawn({ + let handshake_address = handshake_address.clone(); + async move { + let mut engine = setup_mock_engine_sockets(handshake_address, &[0x01, 0x00]).await; + let mut coordinator = + engine.coordinator.take().expect("coordinator sockets should be present"); + let data_socket = engine.data_sockets.first_mut().expect("data socket"); + + let (wave, exclude_engine) = recv_start_dp_wave(&mut coordinator.input_sub).await; + assert_eq!((wave, exclude_engine), (0, 0)); + + let add = recv_engine_message(&mut data_socket.dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = rmp_serde::from_slice(&add[1]).unwrap(); + assert_eq!(request.request_id, "req-2"); + assert_eq!(request.current_wave, 0); + + assert!( + timeout( + Duration::from_millis(200), + recv_start_dp_wave(&mut coordinator.input_sub) + ) + .await + .is_err() + ); + + send_outputs( + &mut data_socket.push, + RequestBatchOutputs { + engine_index: 1, + outputs: vec![request_output( + "req-2", + vec![], + Some(EngineCoreFinishReason::Length), + )], + finished_requests: Some(BTreeSet::from(["req-2".to_string()])), + ..Default::default() + } + .into(), + ) + .await; + + let (wave, exclude_engine) = recv_start_dp_wave(&mut coordinator.input_sub).await; + assert_eq!((wave, exclude_engine), (1, 0)); + + assert!( + timeout( + Duration::from_millis(200), + recv_engine_message(&mut data_socket.dealer) + ) + .await + .is_err() + ); + + let _ = shutdown1_rx.await; + } + }); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 2, + "test-model", + Duration::from_secs(2), + 0, + Some(CoordinatorMode::InProc), + ), + &ipc, + ) + .await; + + let mut stream_1 = client.call(sample_request_with_id("req-1")).await.unwrap(); + let mut stream_2 = client.call(sample_request_with_id("req-2")).await.unwrap(); + + let final_1 = timeout(Duration::from_secs(1), stream_1.next()) + .await + .unwrap() + .unwrap() + .unwrap(); + assert_eq!(final_1.request_id, "req-1"); + assert_eq!(final_1.finish_reason, Some(EngineCoreFinishReason::Length)); + assert!(timeout(Duration::from_secs(1), stream_1.next()).await.unwrap().is_none()); + + let final_2 = timeout(Duration::from_secs(1), stream_2.next()) + .await + .unwrap() + .unwrap() + .unwrap(); + assert_eq!(final_2.request_id, "req-2"); + assert_eq!(final_2.finish_reason, Some(EngineCoreFinishReason::Length)); + assert!(timeout(Duration::from_secs(1), stream_2.next()).await.unwrap().is_none()); + + tokio::time::sleep(Duration::from_millis(100)).await; + + let mut stream_3 = client.call(sample_request_with_id("req-3")).await.unwrap(); + let final_3 = timeout(Duration::from_secs(1), stream_3.next()) + .await + .unwrap() + .unwrap() + .unwrap(); + assert_eq!(final_3.request_id, "req-3"); + assert_eq!(final_3.finish_reason, Some(EngineCoreFinishReason::Length)); + assert!(timeout(Duration::from_secs(1), stream_3.next()).await.unwrap().is_none()); + + tokio::time::sleep(Duration::from_millis(100)).await; + + let _ = shutdown0_tx.send(()); + let _ = shutdown1_tx.send(()); + engine0_task.await.unwrap(); + engine1_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn coordinator_rebroadcasts_engine_start_wave_control() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + + let (shutdown0_tx, shutdown0_rx) = oneshot::channel(); + let engine0_task = tokio::spawn({ + let handshake_address = handshake_address.clone(); + async move { + let mut engine = setup_mock_engine_sockets(handshake_address, &[0x00, 0x00]).await; + let mut coordinator = + engine.coordinator.take().expect("coordinator sockets should be present"); + + let (wave, exclude_engine) = recv_start_dp_wave(&mut coordinator.input_sub).await; + assert_eq!((wave, exclude_engine), (4, 1)); + + let _ = shutdown0_rx.await; + } + }); + + let (shutdown1_tx, shutdown1_rx) = oneshot::channel(); + let engine1_task = tokio::spawn({ + let handshake_address = handshake_address.clone(); + async move { + let mut engine = setup_mock_engine_sockets(handshake_address, &[0x01, 0x00]).await; + let mut coordinator = + engine.coordinator.take().expect("coordinator sockets should be present"); + + send_outputs( + &mut coordinator.output_push, + DpControlOutput { + engine_index: 1, + timestamp: 0.0, + control: DpControlMessage::StartWave(4), + } + .into(), + ) + .await; + + let (wave, exclude_engine) = recv_start_dp_wave(&mut coordinator.input_sub).await; + assert_eq!((wave, exclude_engine), (4, 1)); + + let _ = shutdown1_rx.await; + } + }); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 2, + "test-model", + Duration::from_secs(2), + 0, + Some(CoordinatorMode::InProc), + ), + &ipc, + ) + .await; + + tokio::time::sleep(Duration::from_millis(200)).await; + + let _ = shutdown0_tx.send(()); + let _ = shutdown1_tx.send(()); + engine0_task.await.unwrap(); + engine1_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn coordinator_accepts_stats_only_outputs() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + + let (shutdown_tx, shutdown_rx) = oneshot::channel(); + let engine_task = tokio::spawn(async move { + let mut engine = setup_mock_engine_sockets(handshake_address, &[0x00, 0x00]).await; + let mut coordinator = + engine.coordinator.take().expect("coordinator sockets should be present"); + let data_socket = engine.data_sockets.first_mut().expect("data socket"); + + let (wave, exclude_engine) = recv_start_dp_wave(&mut coordinator.input_sub).await; + assert_eq!((wave, exclude_engine), (0, 0)); + + send_outputs( + &mut coordinator.output_push, + RequestBatchOutputs { + outputs: Vec::new(), + scheduler_stats: Some(Box::new(SchedulerStats { + num_running_reqs: 1, + current_wave: 0, + ..Default::default() + })), + ..Default::default() + } + .into(), + ) + .await; + + let add = recv_engine_message(&mut data_socket.dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = rmp_serde::from_slice(&add[1]).unwrap(); + assert_eq!(request.request_id, "req-stats"); + + send_outputs( + &mut data_socket.push, + RequestBatchOutputs { + outputs: vec![request_output( + "req-stats", + vec![], + Some(EngineCoreFinishReason::Length), + )], + finished_requests: Some(BTreeSet::from(["req-stats".to_string()])), + ..Default::default() + } + .into(), + ) + .await; + + let _ = shutdown_rx.await; + }); + + let client = connect_client_with_ipc( + handshake_test_config( + ipc.handshake_endpoint(), + 1, + "test-model", + Duration::from_secs(2), + 0, + Some(CoordinatorMode::InProc), + ), + &ipc, + ) + .await; + + let mut stream = client.call(sample_request_with_id("req-stats")).await.unwrap(); + let final_output = + timeout(Duration::from_secs(1), stream.next()).await.unwrap().unwrap().unwrap(); + assert_eq!(final_output.request_id, "req-stats"); + assert_eq!( + final_output.finish_reason, + Some(EngineCoreFinishReason::Length) + ); + assert!(client.is_healthy()); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn client_fail_closes_when_main_output_path_receives_dp_control() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-0".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add_1 = recv_engine_message(dealer).await; + assert_eq!(add_1[0].as_ref(), &[0x00]); + let request_1: EngineCoreRequest = rmp_serde::from_slice(&add_1[1]).unwrap(); + assert_eq!(request_1.client_index, 7); + assert_eq!(request_1.request_id, "req-1"); + + let add_2 = recv_engine_message(dealer).await; + assert_eq!(add_2[0].as_ref(), &[0x00]); + let request_2: EngineCoreRequest = rmp_serde::from_slice(&add_2[1]).unwrap(); + assert_eq!(request_2.client_index, 7); + assert_eq!(request_2.request_id, "req-2"); + + send_outputs( + push, + UtilityCallOutput { + output: UtilityOutput { + call_id: 1_u64.into(), + failure_message: None, + result: None, + }, + ..Default::default() + } + .into(), + ) + .await; + send_outputs( + push, + DpControlOutput { + engine_index: 0, + timestamp: 0.0, + control: DpControlMessage::StartWave(3), + } + .into(), + ) + .await; + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output("req-1", vec![999], None)], + ..Default::default() + } + .into(), + ) + .await; + + tokio::time::sleep(Duration::from_millis(50)).await; + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 1, + "test-model", + Duration::from_secs(2), + 7, + None, + ), + &ipc, + ) + .await; + assert_eq!(client.engine_identities()[0], b"engine-0"); + assert!(client.ready_responses()[0].max_model_len > 0); + assert_eq!(client.vllm_version(), "test-vllm-version"); + + let mut stream_1 = client.call(sample_request_with_id("req-1")).await.unwrap(); + let mut stream_2 = client.call(sample_request_with_id("req-2")).await.unwrap(); + + let error_2 = timeout(Duration::from_secs(1), stream_2.next()) + .await + .unwrap() + .unwrap() + .unwrap_err(); + assert!(is_unexpected_dispatcher_output(&error_2)); + + let error_1 = timeout(Duration::from_secs(1), stream_1.next()) + .await + .unwrap() + .unwrap() + .unwrap_err(); + assert!(is_unexpected_dispatcher_output(&error_1)); + + assert!(matches!( + client.health_error().as_deref(), + Some(error) if is_unexpected_dispatcher_output(error) + )); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn duplicate_request_ids_are_rejected_without_sending_a_second_add() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-dup".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add_1 = recv_engine_message(dealer).await; + assert_eq!(add_1[0].as_ref(), &[0x00]); + let request_1: EngineCoreRequest = rmp_serde::from_slice(&add_1[1]).unwrap(); + assert_eq!(request_1.request_id, "req-1"); + + assert!(timeout(Duration::from_millis(200), dealer.recv()).await.is_err()); + + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output( + "req-1", + vec![], + Some(EngineCoreFinishReason::Length), + )], + finished_requests: Some(BTreeSet::from(["req-1".to_string()])), + ..Default::default() + } + .into(), + ) + .await; + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 1, + "test-model", + Duration::from_secs(2), + 0, + None, + ), + &ipc, + ) + .await; + + let mut stream = client.call(sample_request()).await.unwrap(); + let error = match client.call(sample_request()).await { + Ok(_) => panic!("expected duplicate request error"), + Err(error) => error, + }; + assert!(matches!( + error, + Error::DuplicateRequestId { request_id } if request_id == "req-1" + )); + + let final_output = + timeout(Duration::from_secs(1), stream.next()).await.unwrap().unwrap().unwrap(); + assert_eq!( + final_output.finish_reason, + Some(EngineCoreFinishReason::Length) + ); + assert!(timeout(Duration::from_secs(1), stream.next()).await.unwrap().is_none()); + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn finished_requests_without_final_output_is_treated_as_unexpected_close() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-finished-only".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add = recv_engine_message(dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + + send_outputs( + push, + RequestBatchOutputs { + outputs: Vec::new(), + finished_requests: Some(BTreeSet::from(["req-1".to_string()])), + ..Default::default() + } + .into(), + ) + .await; + + assert!(timeout(Duration::from_millis(200), dealer.recv()).await.is_err()); + let _ = push; + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 1, + "test-model", + Duration::from_secs(2), + 0, + None, + ), + &ipc, + ) + .await; + + let mut stream = client.call(sample_request()).await.unwrap(); + let error = timeout(Duration::from_secs(1), stream.next()) + .await + .unwrap() + .unwrap() + .unwrap_err(); + assert!(matches!( + error, + Error::RequestStreamClosed { request_id } if request_id == "req-1" + )); + assert!(timeout(Duration::from_secs(1), stream.next()).await.unwrap().is_none()); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn dropping_a_live_stream_triggers_abort() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-drop".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add = recv_engine_message(dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output("req-1", vec![99], None)], + ..Default::default() + } + .into(), + ) + .await; + + let abort = + timeout(Duration::from_secs(1), recv_engine_message(dealer)).await.unwrap(); + assert_eq!(abort[0].as_ref(), &[0x01]); + let aborted_ids: Vec = rmp_serde::from_slice(&abort[1]).unwrap(); + assert_eq!(aborted_ids, vec!["req-1".to_string()]); + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 1, + "test-model", + Duration::from_secs(2), + 0, + None, + ), + &ipc, + ) + .await; + + let mut stream = client.call(sample_request()).await.unwrap(); + let first = timeout(Duration::from_secs(1), stream.next()).await.unwrap().unwrap().unwrap(); + assert_eq!(first.new_token_ids, vec![99]); + drop(stream); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test] +async fn dropping_multiple_live_streams_aborts_all_in_a_burst() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-burst".to_vec(); + let request_ids = ["req-1", "req-2", "req-3"]; + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + for _ in 0..3 { + let add = recv_engine_message(dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + } + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![ + request_output("req-1", vec![99], None), + request_output("req-2", vec![99], None), + request_output("req-3", vec![99], None), + ], + ..Default::default() + } + .into(), + ) + .await; + + // Aborts may coalesce into one burst or split across several. + let mut aborted = BTreeSet::new(); + while aborted.len() < 3 { + let abort = + timeout(Duration::from_secs(1), recv_engine_message(dealer)).await.unwrap(); + assert_eq!(abort[0].as_ref(), &[0x01]); + let ids: Vec = rmp_serde::from_slice(&abort[1]).unwrap(); + aborted.extend(ids); + } + assert_eq!( + aborted, + BTreeSet::from([ + "req-1".to_string(), + "req-2".to_string(), + "req-3".to_string() + ]) + ); + // No spurious extra aborts. + assert!( + timeout(Duration::from_millis(100), recv_engine_message(dealer)).await.is_err() + ); + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 1, + "test-model", + Duration::from_secs(2), + 0, + None, + ), + &ipc, + ) + .await; + + // Open every request first so all three adds reach the engine before it + // emits outputs, then drain the first token from each stream. + let mut streams = Vec::new(); + for id in request_ids { + streams.push(client.call(sample_request_with_id(id)).await.unwrap()); + } + for stream in streams.iter_mut() { + let first = timeout(Duration::from_secs(1), stream.next()).await.unwrap().unwrap().unwrap(); + assert_eq!(first.new_token_ids, vec![99]); + } + // Drop the whole burst back-to-back so the abort worker can batch them. + drop(streams); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn dispatcher_failure_propagates_to_streams_and_future_calls() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-fail".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let _ = recv_engine_message(dealer).await; + let _ = recv_engine_message(dealer).await; + + push.send(ZmqMessage::from(vec![0xc1])).await.unwrap(); + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 1, + "test-model", + Duration::from_secs(2), + 0, + None, + ), + &ipc, + ) + .await; + + let mut stream_1 = client.call(sample_request_with_id("req-1")).await.unwrap(); + let mut stream_2 = client.call(sample_request_with_id("req-2")).await.unwrap(); + + let error_1 = timeout(Duration::from_secs(1), stream_1.next()) + .await + .unwrap() + .unwrap() + .unwrap_err(); + let error_2 = timeout(Duration::from_secs(1), stream_2.next()) + .await + .unwrap() + .unwrap() + .unwrap_err(); + assert!(is_decode_error(&error_1)); + assert!(is_decode_error(&error_2)); + assert!(is_decode_error( + client.health_error().as_deref().expect("health error recorded") + )); + + let abort_error = client.abort(&["req-1".to_string()]).await.unwrap_err(); + assert!(is_decode_error(&abort_error)); + + let add_error = match client.call(sample_request_with_id("req-3")).await { + Ok(_) => panic!("expected dispatcher closed error"), + Err(error) => error, + }; + assert!(is_decode_error(&add_error)); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn is_sleeping_wrapper_sends_typed_request_and_returns_typed_response() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-utility-success".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]); + let array = match payload { + Value::Array(array) => array, + other => panic!("expected utility payload array, got {other:?}"), + }; + assert_eq!(array.len(), 4); + assert_eq!(array[0], Value::from(5)); + let call_id = array[1].as_u64().expect("call_id"); + assert_eq!(array[2], Value::from("is_sleeping")); + assert_eq!(array[3], Value::Array(Vec::new())); + + send_outputs( + push, + UtilityCallOutput { + engine_index: 0, + timestamp: 0.0, + output: UtilityOutput { + call_id: call_id.into(), + failure_message: None, + result: Some(utility_result_value(true)), + }, + } + .into(), + ) + .await; + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 1, + "test-model", + Duration::from_secs(2), + 5, + None, + ), + &ipc, + ) + .await; + + let result = client.is_sleeping().await.unwrap(); + assert!(result); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn call_utility_failure_message_surfaces_as_error() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-utility-fail".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + let payload = decode_value(&utility[1]); + let call_id = + payload.as_array().and_then(|array| array[1].as_u64()).expect("call_id"); + + send_outputs( + push, + UtilityCallOutput { + engine_index: 0, + timestamp: 0.0, + output: UtilityOutput { + call_id: call_id.into(), + failure_message: Some("boom".to_string()), + result: None, + }, + } + .into(), + ) + .await; + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 1, + "test-model", + Duration::from_secs(2), + 0, + None, + ), + &ipc, + ) + .await; + + let error = client.call_utility::("is_sleeping", ()).await.unwrap_err(); + assert!(matches!( + error, + Error::UtilityCallFailed { + method, + message, + .. + } if method == "is_sleeping" && message == "boom" + )); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn dispatcher_failure_propagates_to_waiting_utility_calls() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-utility-dispatcher-fail".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + push.send(ZmqMessage::from(vec![0xc1])).await.unwrap(); + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 1, + "test-model", + Duration::from_secs(2), + 0, + None, + ), + &ipc, + ) + .await; + + let error = client.call_utility::("is_sleeping", ()).await.unwrap_err(); + assert!(is_decode_error(&error)); + assert!(is_decode_error( + client.health_error().as_deref().expect("health error recorded") + )); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn connect_times_out_without_ready_message() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_handshake = handshake_address.clone(); + let engine_task = tokio::spawn(async move { + tokio::time::sleep(Duration::from_millis(50)).await; + + let mut options = SocketOptions::default(); + options.peer_identity(PeerIdentity::try_from(b"engine-timeout".to_vec()).unwrap()); + let mut handshake = DealerSocket::with_options(options); + handshake.connect(&engine_handshake).await.unwrap(); + handshake + .send(ZmqMessage::from( + rmp_serde::to_vec_named(&ready_message("HELLO")).unwrap(), + )) + .await + .unwrap(); + + let _ = handshake.recv().await.unwrap(); + }); + + let result = EngineCoreClient::connect( + handshake_test_config( + handshake_address, + 1, + "test-model", + Duration::from_millis(100), + 0, + None, + ) + .with_local_input_output_addresses(Some(ipc.input_endpoint()), Some(ipc.output_endpoint())), + ) + .await; + + let error = match result { + Ok(_) => panic!("expected ready timeout"), + Err(error) => error, + }; + + let message = error.to_report_string(); + assert!(message.contains("timed out")); + assert!(message.contains("READY")); + engine_task.await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn engine_core_dead_sentinel_marks_client_unhealthy_and_sticks() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-dead".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |_dealer, push| { + Box::pin(async move { + push.send(ZmqMessage::from(ENGINE_CORE_DEAD_SENTINEL.to_vec())).await.unwrap(); + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 1, + "test-model", + Duration::from_secs(2), + 0, + None, + ), + &ipc, + ) + .await; + + timeout(Duration::from_secs(2), async { + while client.is_healthy() { + tokio::task::yield_now().await; + } + }) + .await + .expect("wait for unhealthy client"); + + assert!(!client.is_healthy()); + assert!(matches!( + client.health_error().as_deref(), + Some(Error::EngineCoreDead) + )); + + let error = client.call_utility::("is_sleeping", ()).await.unwrap_err(); + assert!( + is_dispatcher_closed(&error) || is_engine_core_dead(&error), + "unexpected error: {error:?}" + ); + assert!(matches!( + client.health_error().as_deref(), + Some(Error::EngineCoreDead) + )); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn output_loop_failure_marks_client_unhealthy_and_records_first_error() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-output-failure".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |_dealer, push| { + Box::pin(async move { + send_output_frames( + push, + vec![ + bytes::Bytes::from_static(b"frame-1"), + bytes::Bytes::from_static(b"frame-2"), + ], + ) + .await; + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 1, + "test-model", + Duration::from_secs(2), + 0, + None, + ), + &ipc, + ) + .await; + + timeout(Duration::from_secs(2), async { + while client.is_healthy() { + let _ = client.call_utility::("is_sleeping", ()).await; + tokio::task::yield_now().await; + } + }) + .await + .expect("wait for unhealthy client"); + + assert!(!client.is_healthy()); + assert!(is_decode_error( + client.health_error().as_deref().expect("health error recorded") + )); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn client_decodes_multipart_logprob_outputs() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-multipart-logprobs".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add = recv_engine_message(dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = rmp_serde::from_slice(&add[1]).unwrap(); + assert_eq!(request.request_id, "req-1"); + + send_output_frames(push, multipart_logprob_output_frames("req-1")).await; + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 1, + "test-model", + Duration::from_secs(2), + 0, + None, + ), + &ipc, + ) + .await; + + let stream = client.call(sample_request()).await.unwrap(); + let outputs = stream.collect::>().await; + assert_eq!(outputs.len(), 1); + + let output = outputs.into_iter().next().unwrap().unwrap(); + assert_eq!(output.output.new_token_ids, vec![7, 8]); + assert_eq!( + output.output.finish_reason, + Some(EngineCoreFinishReason::Length) + ); + expect_sample_logprobs(output.output.new_logprobs.as_ref().expect("logprobs decoded")); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn multi_engine_client_shares_transport_and_routes_by_inflight_count() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let (engine_0_seen_tx, mut engine_0_seen_rx) = mpsc::unbounded_channel(); + let (engine_1_seen_tx, engine_1_seen_rx) = oneshot::channel(); + let (finish_req_1_tx, finish_req_1_rx) = oneshot::channel(); + let (finish_req_2_tx, finish_req_2_rx) = oneshot::channel(); + let (finish_req_3_tx, finish_req_3_rx) = oneshot::channel(); + + let (init_rx_0, shutdown_tx_0, engine_task_0) = spawn_mock_engine_task_with_init( + handshake_address.clone(), + b"engine-0".to_vec(), + |dealer, push| { + Box::pin(async move { + let add_1 = recv_engine_message(dealer).await; + assert_eq!(add_1[0].as_ref(), &[0x00]); + let request_1: EngineCoreRequest = rmp_serde::from_slice(&add_1[1]).unwrap(); + assert_eq!(request_1.request_id, "req-1"); + engine_0_seen_tx.send(request_1.request_id.clone()).unwrap(); + finish_req_1_rx.await.unwrap(); + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output( + &request_1.request_id, + vec![10], + Some(EngineCoreFinishReason::Length), + )], + finished_requests: Some(BTreeSet::from([request_1.request_id.clone()])), + ..Default::default() + } + .into(), + ) + .await; + + let add_3 = recv_engine_message(dealer).await; + assert_eq!(add_3[0].as_ref(), &[0x00]); + let request_3: EngineCoreRequest = rmp_serde::from_slice(&add_3[1]).unwrap(); + assert_eq!(request_3.request_id, "req-3"); + engine_0_seen_tx.send(request_3.request_id.clone()).unwrap(); + finish_req_3_rx.await.unwrap(); + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output( + &request_3.request_id, + vec![30], + Some(EngineCoreFinishReason::Length), + )], + finished_requests: Some(BTreeSet::from([request_3.request_id.clone()])), + ..Default::default() + } + .into(), + ) + .await; + }) + }, + ); + let (init_rx_1, shutdown_tx_1, engine_task_1) = spawn_mock_engine_task_with_init( + handshake_address.clone(), + b"engine-1".to_vec(), + |dealer, push| { + Box::pin(async move { + let add_2 = recv_engine_message(dealer).await; + assert_eq!(add_2[0].as_ref(), &[0x00]); + let request_2: EngineCoreRequest = rmp_serde::from_slice(&add_2[1]).unwrap(); + assert_eq!(request_2.request_id, "req-2"); + let _ = engine_1_seen_tx.send(request_2.request_id.clone()); + finish_req_2_rx.await.unwrap(); + send_outputs( + push, + RequestBatchOutputs { + engine_index: 1, + outputs: vec![request_output( + &request_2.request_id, + vec![20], + Some(EngineCoreFinishReason::Length), + )], + finished_requests: Some(BTreeSet::from([request_2.request_id.clone()])), + ..Default::default() + } + .into(), + ) + .await; + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address.clone(), + 2, + "test-model", + Duration::from_secs(2), + 0, + None, + ), + &ipc, + ) + .await; + + let init_0 = timeout(Duration::from_secs(1), init_rx_0).await.unwrap().unwrap(); + let init_1 = timeout(Duration::from_secs(1), init_rx_1).await.unwrap().unwrap(); + assert_eq!(init_0.addresses.inputs, vec![ipc.input_endpoint()]); + assert_eq!(init_1.addresses.inputs, vec![ipc.input_endpoint()]); + assert_eq!(init_0.addresses.outputs, vec![ipc.output_endpoint()]); + assert_eq!(init_1.addresses.outputs, vec![ipc.output_endpoint()]); + + assert_eq!(client.input_address(), ipc.input_endpoint()); + assert_eq!(client.output_address(), ipc.output_endpoint()); + assert_eq!(client.engine_count(), 2); + assert_eq!( + client.engine_identities(), + vec![b"engine-0".as_slice(), b"engine-1".as_slice()] + ); + assert_eq!(client.ready_responses().len(), 2); + assert_eq!(client.engine_identities()[0], b"engine-0"); + + let mut stream_1 = client.call(sample_request_with_id("req-1")).await.unwrap(); + let mut stream_2 = client.call(sample_request_with_id("req-2")).await.unwrap(); + assert_eq!( + timeout(Duration::from_secs(1), engine_0_seen_rx.recv()).await.unwrap().unwrap(), + "req-1" + ); + assert_eq!( + timeout(Duration::from_secs(1), engine_1_seen_rx).await.unwrap().unwrap(), + "req-2" + ); + + let _ = finish_req_1_tx.send(()); + let final_1 = timeout(Duration::from_secs(1), stream_1.next()) + .await + .unwrap() + .unwrap() + .unwrap(); + assert_eq!(final_1.engine_index, 0); + assert_eq!(final_1.new_token_ids, vec![10]); + assert_eq!(final_1.finish_reason, Some(EngineCoreFinishReason::Length)); + + let mut stream_3 = client.call(sample_request_with_id("req-3")).await.unwrap(); + assert_eq!( + timeout(Duration::from_secs(1), engine_0_seen_rx.recv()).await.unwrap().unwrap(), + "req-3" + ); + + let _ = finish_req_3_tx.send(()); + let final_3 = timeout(Duration::from_secs(1), stream_3.next()) + .await + .unwrap() + .unwrap() + .unwrap(); + assert_eq!(final_3.engine_index, 0); + assert_eq!(final_3.new_token_ids, vec![30]); + assert_eq!(final_3.finish_reason, Some(EngineCoreFinishReason::Length)); + + let _ = finish_req_2_tx.send(()); + let final_2 = timeout(Duration::from_secs(1), stream_2.next()) + .await + .unwrap() + .unwrap() + .unwrap(); + assert_eq!(final_2.engine_index, 1); + assert_eq!(final_2.new_token_ids, vec![20]); + assert_eq!(final_2.finish_reason, Some(EngineCoreFinishReason::Length)); + + assert!(timeout(Duration::from_secs(1), stream_1.next()).await.unwrap().is_none()); + assert!(timeout(Duration::from_secs(1), stream_2.next()).await.unwrap().is_none()); + assert!(timeout(Duration::from_secs(1), stream_3.next()).await.unwrap().is_none()); + + let _ = shutdown_tx_0.send(()); + let _ = shutdown_tx_1.send(()); + engine_task_0.await.unwrap(); + engine_task_1.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn multi_engine_abort_is_grouped_and_utility_fans_out_to_all_engines() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + + let (shutdown_tx_0, engine_task_0) = spawn_mock_engine_task( + handshake_address.clone(), + EngineId::from_engine_index(0).into_frame().to_vec(), + |dealer, push| { + Box::pin(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + let payload = decode_value(&utility[1]); + let array = match payload { + Value::Array(array) => array, + other => panic!("expected utility payload array, got {other:?}"), + }; + let call_id = array[1].as_u64().expect("call_id"); + assert_eq!(array[2], Value::from("is_sleeping")); + send_outputs( + push, + UtilityCallOutput { + engine_index: 0, + timestamp: 0.0, + output: UtilityOutput { + call_id: call_id.into(), + failure_message: None, + result: Some(utility_result_value(true)), + }, + } + .into(), + ) + .await; + + let add_1 = recv_engine_message(dealer).await; + assert_eq!(add_1[0].as_ref(), &[0x00]); + let request_1: EngineCoreRequest = rmp_serde::from_slice(&add_1[1]).unwrap(); + assert_eq!(request_1.request_id, "req-1"); + + let abort = recv_engine_message(dealer).await; + assert_eq!(abort[0].as_ref(), &[0x01]); + let aborted_ids: Vec = rmp_serde::from_slice(&abort[1]).unwrap(); + assert_eq!(aborted_ids, vec!["req-1".to_string()]); + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output( + "req-1", + vec![], + Some(EngineCoreFinishReason::Abort), + )], + finished_requests: Some(BTreeSet::from(["req-1".to_string()])), + ..Default::default() + } + .into(), + ) + .await; + }) + }, + ); + tokio::time::sleep(Duration::from_millis(50)).await; + let (shutdown_tx_1, engine_task_1) = spawn_mock_engine_task( + handshake_address.clone(), + EngineId::from_engine_index(1).into_frame().to_vec(), + |dealer, push| { + Box::pin(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + let payload = decode_value(&utility[1]); + let array = match payload { + Value::Array(array) => array, + other => panic!("expected utility payload array, got {other:?}"), + }; + let call_id = array[1].as_u64().expect("call_id"); + assert_eq!(array[2], Value::from("is_sleeping")); + send_outputs( + push, + UtilityCallOutput { + engine_index: 0, + timestamp: 0.0, + output: UtilityOutput { + call_id: call_id.into(), + failure_message: None, + result: Some(utility_result_value(true)), + }, + } + .into(), + ) + .await; + + let add_2 = recv_engine_message(dealer).await; + assert_eq!(add_2[0].as_ref(), &[0x00]); + let request_2: EngineCoreRequest = rmp_serde::from_slice(&add_2[1]).unwrap(); + assert_eq!(request_2.request_id, "req-2"); + + let abort = recv_engine_message(dealer).await; + assert_eq!(abort[0].as_ref(), &[0x01]); + let aborted_ids: Vec = rmp_serde::from_slice(&abort[1]).unwrap(); + assert_eq!(aborted_ids, vec!["req-2".to_string()]); + send_outputs( + push, + RequestBatchOutputs { + engine_index: 1, + outputs: vec![request_output( + "req-2", + vec![], + Some(EngineCoreFinishReason::Abort), + )], + finished_requests: Some(BTreeSet::from(["req-2".to_string()])), + ..Default::default() + } + .into(), + ) + .await; + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 2, + "test-model", + Duration::from_secs(2), + 5, + None, + ), + &ipc, + ) + .await; + + assert!(client.is_sleeping().await.unwrap()); + + let mut stream_1 = client.call(sample_request_with_id("req-1")).await.unwrap(); + let mut stream_2 = client.call(sample_request_with_id("req-2")).await.unwrap(); + + client + .abort(&[ + "req-2".to_string(), + "req-1".to_string(), + "unknown".to_string(), + ]) + .await + .unwrap(); + + let final_1 = timeout(Duration::from_secs(1), stream_1.next()) + .await + .unwrap() + .unwrap() + .unwrap(); + assert_eq!(final_1.engine_index, 0); + assert_eq!(final_1.finish_reason, Some(EngineCoreFinishReason::Abort)); + + let final_2 = timeout(Duration::from_secs(1), stream_2.next()) + .await + .unwrap() + .unwrap() + .unwrap(); + assert_eq!(final_2.engine_index, 1); + assert_eq!(final_2.finish_reason, Some(EngineCoreFinishReason::Abort)); + + let _ = shutdown_tx_0.send(()); + let _ = shutdown_tx_1.send(()); + engine_task_0.await.unwrap(); + engine_task_1.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn collective_rpc_flattens_results_from_all_engines() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + + let (shutdown_tx_0, engine_task_0) = spawn_mock_engine_task( + handshake_address.clone(), + b"engine-0".to_vec(), + |dealer, push| { + Box::pin(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + let payload = decode_value(&utility[1]); + let array = match payload { + Value::Array(array) => array, + other => panic!("expected utility payload array, got {other:?}"), + }; + let call_id = array[1].as_u64().expect("call_id"); + assert_eq!(array[2], Value::from("collective_rpc")); + + send_outputs( + push, + UtilityCallOutput { + engine_index: 0, + timestamp: 0.0, + output: UtilityOutput { + call_id: call_id.into(), + failure_message: None, + result: Some(utility_result_value(vec!["engine-0-worker"])), + }, + } + .into(), + ) + .await; + }) + }, + ); + tokio::time::sleep(Duration::from_millis(50)).await; + let (shutdown_tx_1, engine_task_1) = spawn_mock_engine_task( + handshake_address.clone(), + b"engine-1".to_vec(), + |dealer, push| { + Box::pin(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + let payload = decode_value(&utility[1]); + let array = match payload { + Value::Array(array) => array, + other => panic!("expected utility payload array, got {other:?}"), + }; + let call_id = array[1].as_u64().expect("call_id"); + assert_eq!(array[2], Value::from("collective_rpc")); + + send_outputs( + push, + UtilityCallOutput { + engine_index: 0, + timestamp: 0.0, + output: UtilityOutput { + call_id: call_id.into(), + failure_message: None, + result: Some(utility_result_value(vec!["engine-1-worker"])), + }, + } + .into(), + ) + .await; + }) + }, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 2, + "test-model", + Duration::from_secs(2), + 5, + None, + ), + &ipc, + ) + .await; + + let results = client + .collective_rpc( + "get_model_name", + Option::::None, + Vec::::new(), + BTreeMap::::new(), + ) + .await + .unwrap(); + assert_eq!( + results, + vec![ + Value::from("engine-0-worker"), + Value::from("engine-1-worker") + ] + ); + + let _ = shutdown_tx_0.send(()); + let _ = shutdown_tx_1.send(()); + engine_task_0.await.unwrap(); + engine_task_1.await.unwrap(); + client.shutdown().await.unwrap(); +} + +/// Spawn a mock engine that handles a single utility call, asserts the method +/// name and serialized args match, and replies with `result`. +fn spawn_mock_utility_engine( + handshake_address: String, + engine_id: Vec, + expected_method: &'static str, + expected_args: Value, + result: bool, +) -> ( + tokio::sync::oneshot::Sender<()>, + tokio::task::JoinHandle<()>, +) { + spawn_mock_engine_task(handshake_address, engine_id, move |dealer, push| { + Box::pin(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + let payload = decode_value(&utility[1]); + let array = match payload { + Value::Array(array) => array, + other => panic!("expected utility payload array, got {other:?}"), + }; + // Utility requests serialize as `(client_index, call_id, method, args)`. + let call_id = array[1].as_u64().expect("call_id"); + assert_eq!(array[2], Value::from(expected_method)); + assert_eq!(array[3], expected_args, "unexpected utility args"); + send_outputs( + push, + UtilityCallOutput { + engine_index: 0, + timestamp: 0.0, + output: UtilityOutput { + call_id: call_id.into(), + failure_message: None, + result: Some(utility_result_value(result)), + }, + } + .into(), + ) + .await; + }) + }) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn is_sleeping_returns_error_when_engines_disagree() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + + let (shutdown_tx_0, engine_task_0) = spawn_mock_utility_engine( + handshake_address.clone(), + b"engine-0".to_vec(), + "is_sleeping", + Value::Array(vec![]), + true, + ); + let (shutdown_tx_1, engine_task_1) = spawn_mock_utility_engine( + handshake_address.clone(), + b"engine-1".to_vec(), + "is_sleeping", + Value::Array(vec![]), + false, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 2, + "test-model", + Duration::from_secs(2), + 5, + None, + ), + &ipc, + ) + .await; + + let error = client.is_sleeping().await.unwrap_err(); + assert!( + matches!( + &error, + Error::InconsistentUtilityResults { method, .. } if method == "is_sleeping" + ), + "expected InconsistentUtilityResults, got {error:?}", + ); + + let _ = shutdown_tx_0.send(()); + let _ = shutdown_tx_1.send(()); + engine_task_0.await.unwrap(); + engine_task_1.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn is_sleeping_returns_value_when_all_engines_agree() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + + let (shutdown_tx_0, engine_task_0) = spawn_mock_utility_engine( + handshake_address.clone(), + b"engine-0".to_vec(), + "is_sleeping", + Value::Array(vec![]), + true, + ); + let (shutdown_tx_1, engine_task_1) = spawn_mock_utility_engine( + handshake_address.clone(), + b"engine-1".to_vec(), + "is_sleeping", + Value::Array(vec![]), + true, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 2, + "test-model", + Duration::from_secs(2), + 5, + None, + ), + &ipc, + ) + .await; + + assert!(client.is_sleeping().await.unwrap()); + + let _ = shutdown_tx_0.send(()); + let _ = shutdown_tx_1.send(()); + engine_task_0.await.unwrap(); + engine_task_1.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn reset_prefix_cache_returns_true_when_all_engines_succeed() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + + let (shutdown_tx_0, engine_task_0) = spawn_mock_utility_engine( + handshake_address.clone(), + b"engine-0".to_vec(), + "reset_prefix_cache", + Value::Array(vec![Value::from(false), Value::from(false)]), + true, + ); + let (shutdown_tx_1, engine_task_1) = spawn_mock_utility_engine( + handshake_address.clone(), + b"engine-1".to_vec(), + "reset_prefix_cache", + Value::Array(vec![Value::from(false), Value::from(false)]), + true, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 2, + "test-model", + Duration::from_secs(2), + 5, + None, + ), + &ipc, + ) + .await; + + assert!(client.reset_prefix_cache(false, false).await.unwrap()); + + let _ = shutdown_tx_0.send(()); + let _ = shutdown_tx_1.send(()); + engine_task_0.await.unwrap(); + engine_task_1.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn reset_prefix_cache_returns_false_when_any_engine_fails() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + + let (shutdown_tx_0, engine_task_0) = spawn_mock_utility_engine( + handshake_address.clone(), + b"engine-0".to_vec(), + "reset_prefix_cache", + Value::Array(vec![Value::from(false), Value::from(false)]), + true, + ); + let (shutdown_tx_1, engine_task_1) = spawn_mock_utility_engine( + handshake_address.clone(), + b"engine-1".to_vec(), + "reset_prefix_cache", + Value::Array(vec![Value::from(false), Value::from(false)]), + false, + ); + + let client = connect_client_with_ipc( + handshake_test_config( + handshake_address, + 2, + "test-model", + Duration::from_secs(2), + 5, + None, + ), + &ipc, + ) + .await; + + assert!(!client.reset_prefix_cache(false, false).await.unwrap()); + + let _ = shutdown_tx_0.send(()); + let _ = shutdown_tx_1.send(()); + engine_task_0.await.unwrap(); + engine_task_1.await.unwrap(); + client.shutdown().await.unwrap(); +} + +#[test] +fn python_msgpack_fixtures_match_rust_encoding() { + init_tracing(); + let script = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("src/tests/python_compat.py"); + let output = Command::new(&script) + .output() + .unwrap_or_else(|error| panic!("failed to execute {:?}: {error}", script)); + assert!( + output.status.success(), + "python fixture script failed: status={:?}\nstdout:\n{}\nstderr:\n{}", + output.status.code(), + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr), + ); + + let stdout = String::from_utf8(output.stdout).unwrap(); + let mut lines = stdout.lines(); + let request_hex = lines.next().expect("missing request fixture line"); + let defaults_request_hex = lines.next().expect("missing defaults request fixture line"); + let multimodal_request_hex = lines.next().expect("missing multimodal request fixture line"); + let outputs_hex = lines.next().expect("missing outputs fixture line"); + let inline_logprobs_frames = lines.next().expect("missing inline logprobs fixture line"); + let multipart_logprobs_frames = lines.next().expect("missing multipart logprobs fixture line"); + let inline_prompt_frames = lines.next().expect("missing inline prompt logprobs fixture line"); + let multipart_prompt_frames = + lines.next().expect("missing multipart prompt logprobs fixture line"); + let ready_response_hex = lines.next().expect("missing ready response fixture line"); + + let request_bytes = hex::decode(request_hex).unwrap(); + let multimodal_request_bytes = hex::decode(multimodal_request_hex).unwrap(); + let outputs_bytes = hex::decode(outputs_hex).unwrap(); + + let decoded_request: EngineCoreRequest = rmp_serde::from_slice(&request_bytes).unwrap(); + let expected_request = sample_request(); + assert_eq!(decoded_request, expected_request); + + // All-default sampling params -> empty map; must decode to Python defaults. + let defaults_request_bytes = hex::decode(defaults_request_hex).unwrap(); + let decoded_defaults: EngineCoreRequest = + rmp_serde::from_slice(&defaults_request_bytes).unwrap(); + assert_eq!(decoded_defaults.request_id, "req-defaults"); + let sampling = decoded_defaults + .sampling_params + .expect("defaults request carries sampling params"); + assert_eq!( + sampling, + EngineCoreSamplingParams { + temperature: 1.0, + top_p: 1.0, + top_k: 0, + seed: None, + max_tokens: 16, + min_tokens: 0, + thinking_token_budget: None, + logprobs: None, + prompt_logprobs: None, + min_p: 0.0, + frequency_penalty: 0.0, + presence_penalty: 0.0, + repetition_penalty: 1.0, + repetition_detection: None, + stop_token_ids: Vec::new(), + eos_token_id: None, + all_stop_token_ids: BTreeSet::new(), + logit_bias: None, + allowed_token_ids: None, + bad_words_token_ids: None, + structured_outputs: None, + logprob_token_ids: None, + skip_reading_prefix_cache: None, + extra_args: None, + }, + ); + + let decoded_multimodal_request: EngineCoreRequest = + rmp_serde::from_slice(&multimodal_request_bytes).unwrap(); + assert_eq!(decoded_multimodal_request, sample_multimodal_request()); + + // The decode assertion above proves Python wire -> Rust struct. Also compare + // Rust struct -> wire for the multimodal subtree, which is the frontend's + // production direction when sending requests to Python EngineCore. + let expected_multimodal_request = sample_multimodal_request(); + let decode_value = |bytes: &[u8]| { + rmpv::decode::read_value(&mut Cursor::new(bytes)).expect("decode msgpack value") + }; + let extract_mm_features = |value: Value| match value { + Value::Array(items) => items.get(2).cloned().expect("request mm_features slot"), + other => panic!("request should encode as tuple array, got {other:?}"), + }; + let python_mm_features = extract_mm_features(decode_value(&multimodal_request_bytes)); + let rust_mm_features = + decode_value(&rmp_serde::to_vec_named(&expected_multimodal_request.mm_features).unwrap()); + assert_eq!(python_mm_features, rust_mm_features); + + let decoded_outputs: EngineCoreOutputs = rmp_serde::from_slice(&outputs_bytes).unwrap(); + expect_test::expect![[r#" + RequestBatch( + RequestBatchOutputs { + engine_index: 0, + outputs: [ + EngineCoreOutput { + request_id: "req-1", + new_token_ids: [ + 7, + 8, + ], + new_logprobs: None, + new_prompt_logprobs_tensors: None, + pooling_output: None, + finish_reason: Some( + Length, + ), + stop_reason: None, + events: None, + kv_transfer_params: None, + trace_headers: None, + prefill_stats: None, + routed_experts: None, + num_nans_in_logits: 0, + }, + ], + scheduler_stats: None, + timestamp: 0.0, + finished_requests: Some( + { + "req-1", + }, + ), + }, + ) + "#]] + .assert_debug_eq(&decoded_outputs); + + let decode_frames = |line: &str| { + line.split_whitespace() + .map(|frame| bytes::Bytes::from(hex::decode(frame).unwrap())) + .collect::>() + }; + + let inline_logprobs = + decode_engine_core_outputs(&decode_frames(inline_logprobs_frames)).unwrap(); + expect_sample_logprobs( + inline_logprobs.as_request_batch().unwrap().outputs[0] + .new_logprobs + .as_ref() + .expect("inline logprobs decoded"), + ); + + let multipart_logprobs = + decode_engine_core_outputs(&decode_frames(multipart_logprobs_frames)).unwrap(); + expect_sample_logprobs( + multipart_logprobs.as_request_batch().unwrap().outputs[0] + .new_logprobs + .as_ref() + .expect("multipart logprobs decoded"), + ); + + let inline_prompt = decode_engine_core_outputs(&decode_frames(inline_prompt_frames)).unwrap(); + expect_prompt_logprobs( + inline_prompt.as_request_batch().unwrap().outputs[0] + .new_prompt_logprobs_tensors + .as_ref() + .expect("inline prompt logprobs decoded"), + ); + + let multipart_prompt = + decode_engine_core_outputs(&decode_frames(multipart_prompt_frames)).unwrap(); + expect_prompt_logprobs( + multipart_prompt.as_request_batch().unwrap().outputs[0] + .new_prompt_logprobs_tensors + .as_ref() + .expect("multipart prompt logprobs decoded"), + ); + + let map_keys = |bytes: &[u8]| -> BTreeSet { + match decode_value(bytes) { + Value::Map(entries) => entries + .into_iter() + .filter_map(|(key, _)| key.as_str().map(str::to_owned)) + .collect(), + other => panic!("ready response should encode as a map, got {other:?}"), + } + }; + let python_ready_keys = map_keys(&hex::decode(ready_response_hex).unwrap()); + let rust_ready_keys = + map_keys(&rmp_serde::to_vec_named(&crate::mock_engine::default_ready_response()).unwrap()); + assert_eq!( + rust_ready_keys, python_ready_keys, + "EngineCoreReadyResponse drifted from the Python dataclass", + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn bootstrapped_connects_after_single_engine_registration() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let input_address = ipc.input_endpoint(); + let output_address = ipc.output_endpoint(); + + let client_task = tokio::spawn({ + let input_address = input_address.clone(); + let output_address = output_address.clone(); + async move { + EngineCoreClient::connect(bootstrapped_test_config( + input_address, + output_address, + 1, + Duration::from_secs(2), + 0, + None, + )) + .await + .unwrap() + } + }); + + let (_dealer, _push) = + setup_bootstrapped_mock_engine(input_address, output_address, &[0x00, 0x00]).await; + let client = client_task.await.unwrap(); + + assert_eq!(client.engine_count(), 1); + let engine_ids = + client.engine_identities().into_iter().map(|id| id.to_vec()).collect::>(); + assert_eq!(engine_ids, vec![vec![0x00, 0x00]]); + + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn bootstrapped_connects_with_contiguous_engine_ids() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let input_address = ipc.input_endpoint(); + let output_address = ipc.output_endpoint(); + + let client_task = tokio::spawn({ + let input_address = input_address.clone(); + let output_address = output_address.clone(); + async move { + EngineCoreClient::connect(bootstrapped_test_config( + input_address, + output_address, + 2, + Duration::from_secs(2), + 0, + None, + )) + .await + .unwrap() + } + }); + + let (_dealer0, _push0) = setup_bootstrapped_mock_engine( + input_address.clone(), + output_address.clone(), + &[0x00, 0x00], + ) + .await; + let (_dealer1, _push1) = + setup_bootstrapped_mock_engine(input_address, output_address, &[0x01, 0x00]).await; + let client = client_task.await.unwrap(); + + assert_eq!(client.engine_count(), 2); + let engine_ids = + client.engine_identities().into_iter().map(|id| id.to_vec()).collect::>(); + assert_eq!(engine_ids, vec![vec![0x00, 0x00], vec![0x01, 0x00]]); + + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn bootstrapped_connects_with_nonzero_engine_start_index() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let input_address = ipc.input_endpoint(); + let output_address = ipc.output_endpoint(); + + let client_task = tokio::spawn({ + let input_address = input_address.clone(); + let output_address = output_address.clone(); + async move { + EngineCoreClient::connect(bootstrapped_test_config_with_start_index( + input_address, + output_address, + 3, + 1, + Duration::from_secs(2), + 0, + None, + )) + .await + .unwrap() + } + }); + + let (_dealer, _push) = + setup_bootstrapped_mock_engine(input_address, output_address, &[0x03, 0x00]).await; + let client = client_task.await.unwrap(); + + assert_eq!(client.engine_count(), 1); + let engine_ids = + client.engine_identities().into_iter().map(|id| id.to_vec()).collect::>(); + assert_eq!(engine_ids, vec![vec![0x03, 0x00]]); + + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn bootstrapped_rejects_unexpected_engine_id_for_start_index() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let input_address = ipc.input_endpoint(); + let output_address = ipc.output_endpoint(); + + let client_task = tokio::spawn({ + let input_address = input_address.clone(); + let output_address = output_address.clone(); + async move { + EngineCoreClient::connect(bootstrapped_test_config_with_start_index( + input_address, + output_address, + 3, + 1, + Duration::from_secs(2), + 0, + None, + )) + .await + } + }); + + let _ = crate::mock_engine::connect_to_bootstrapped_frontend( + input_address, + output_address, + &[0x00, 0x00], + crate::mock_engine::MockEngineConfig { + local: true, + headless: true, + ..Default::default() + }, + ) + .await; + let error = match client_task.await.unwrap() { + Ok(_) => panic!("bootstrapped connect should reject unexpected engine id"), + Err(error) => error, + }; + + assert!( + error + .to_string() + .contains("received input registration for unexpected engine id") + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn bootstrapped_connect_times_out_without_registration() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let result = EngineCoreClient::connect(bootstrapped_test_config( + ipc.input_endpoint(), + ipc.output_endpoint(), + 1, + Duration::from_millis(100), + 0, + None, + )) + .await; + + let error = match result { + Ok(_) => panic!("bootstrapped connect should time out"), + Err(error) => error, + }; + assert!(matches!(error, Error::InputRegistrationTimeout { .. })); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn bootstrapped_external_coordinator_connects_and_subscribes() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let input_address = ipc.input_endpoint(); + let output_address = ipc.output_endpoint(); + let coordinator_address = ipc.endpoint("stats.sock"); + + let mut stats_socket = XPubSocket::new(); + stats_socket.bind(&coordinator_address).await.unwrap(); + + let client_task = tokio::spawn({ + let input_address = input_address.clone(); + let output_address = output_address.clone(); + let coordinator_address = coordinator_address.clone(); + async move { + EngineCoreClient::connect(bootstrapped_test_config( + input_address, + output_address, + 1, + Duration::from_secs(2), + 0, + Some(CoordinatorMode::External { + address: coordinator_address, + }), + )) + .await + .unwrap() + } + }); + + let (_dealer, _push) = + setup_bootstrapped_mock_engine(input_address, output_address, &[0x00, 0x00]).await; + let client = client_task.await.unwrap(); + + timeout( + Duration::from_secs(1), + recv_xpub_subscription(&mut stats_socket), + ) + .await + .unwrap(); + + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn bootstrapped_external_coordinator_updates_wave_ignores_counts_and_sends_one_wakeup() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let input_address = ipc.input_endpoint(); + let output_address = ipc.output_endpoint(); + let coordinator_address = ipc.endpoint("stats.sock"); + + let mut stats_socket = XPubSocket::new(); + stats_socket.bind(&coordinator_address).await.unwrap(); + + let client_task = tokio::spawn({ + let input_address = input_address.clone(); + let output_address = output_address.clone(); + let coordinator_address = coordinator_address.clone(); + async move { + EngineCoreClient::connect(bootstrapped_test_config( + input_address, + output_address, + 1, + Duration::from_secs(2), + 0, + Some(CoordinatorMode::External { + address: coordinator_address, + }), + )) + .await + .unwrap() + } + }); + + let (mut dealer, mut push) = + setup_bootstrapped_mock_engine(input_address, output_address, &[0x00, 0x00]).await; + let client = client_task.await.unwrap(); + recv_xpub_subscription(&mut stats_socket).await; + + send_external_coordinator_publish(&mut stats_socket, &(vec![(11_u32, 3_u32)], 7_u32, false)) + .await; + tokio::time::sleep(Duration::from_millis(50)).await; + + let mut stream = client.call(sample_request()).await.unwrap(); + + let wakeup = timeout( + Duration::from_secs(1), + recv_external_coordinator_wakeup(&mut stats_socket), + ) + .await + .unwrap(); + assert_eq!(wakeup, (0, 7)); + + assert!( + timeout( + Duration::from_millis(200), + recv_external_coordinator_wakeup(&mut stats_socket) + ) + .await + .is_err() + ); + + let add = recv_engine_message(&mut dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = rmp_serde::from_slice(&add[1]).unwrap(); + assert_eq!(request.request_id, "req-1"); + assert_eq!(request.current_wave, 7); + assert!(client.is_healthy()); + + send_outputs( + &mut push, + RequestBatchOutputs { + outputs: vec![request_output( + "req-1", + vec![], + Some(EngineCoreFinishReason::Length), + )], + finished_requests: Some(BTreeSet::from(["req-1".to_string()])), + ..Default::default() + } + .into(), + ) + .await; + + let final_output = timeout(Duration::from_secs(1), stream.next()).await.unwrap(); + assert!(final_output.is_some()); + + client.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn bootstrapped_external_coordinator_running_state_suppresses_wakeup() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let input_address = ipc.input_endpoint(); + let output_address = ipc.output_endpoint(); + let coordinator_address = ipc.endpoint("stats.sock"); + + let mut stats_socket = XPubSocket::new(); + stats_socket.bind(&coordinator_address).await.unwrap(); + + let client_task = tokio::spawn({ + let input_address = input_address.clone(); + let output_address = output_address.clone(); + let coordinator_address = coordinator_address.clone(); + async move { + EngineCoreClient::connect(bootstrapped_test_config( + input_address, + output_address, + 1, + Duration::from_secs(2), + 0, + Some(CoordinatorMode::External { + address: coordinator_address, + }), + )) + .await + .unwrap() + } + }); + + let (mut dealer, mut push) = + setup_bootstrapped_mock_engine(input_address, output_address, &[0x00, 0x00]).await; + let client = client_task.await.unwrap(); + recv_xpub_subscription(&mut stats_socket).await; + + send_external_coordinator_publish(&mut stats_socket, &(Value::Nil, 5_u32, true)).await; + tokio::time::sleep(Duration::from_millis(50)).await; + + let mut stream = client.call(sample_request()).await.unwrap(); + + assert!( + timeout( + Duration::from_millis(200), + recv_external_coordinator_wakeup(&mut stats_socket) + ) + .await + .is_err() + ); + + let add = recv_engine_message(&mut dealer).await; + let request: EngineCoreRequest = rmp_serde::from_slice(&add[1]).unwrap(); + assert_eq!(request.current_wave, 5); + + send_outputs( + &mut push, + RequestBatchOutputs { + outputs: vec![request_output( + "req-1", + vec![], + Some(EngineCoreFinishReason::Length), + )], + finished_requests: Some(BTreeSet::from(["req-1".to_string()])), + ..Default::default() + } + .into(), + ) + .await; + + let final_output = timeout(Duration::from_secs(1), stream.next()).await.unwrap(); + assert!(final_output.is_some()); + + client.shutdown().await.unwrap(); +} diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/tests/mod.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/tests/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..8e6ed69979a11a4559733845c70b1b8849b51b41 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/tests/mod.rs @@ -0,0 +1 @@ +mod client; diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/tests/python_compat.py b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/tests/python_compat.py new file mode 100644 index 0000000000000000000000000000000000000000..0797189a430152fe4c96b196af76a5fbc9f267cd --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/tests/python_compat.py @@ -0,0 +1,398 @@ +#!/usr/bin/env -S uv run +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# /// script +# requires-python = ">=3.11" +# dependencies = [ +# "msgspec>=0.19,<1", +# "msgpack>=1,<2", +# "numpy>=2,<3", +# ] +# /// + +from dataclasses import dataclass +from enum import Enum, IntEnum + +import msgpack +import msgspec +import numpy as np + + +class RequestOutputKind(Enum): + DELTA = 1 + FINAL_ONLY = 2 + + +class FinishReason(IntEnum): + STOP = 0 + LENGTH = 1 + ABORT = 2 + ERROR = 3 + REPETITION = 4 + + +# Mirror of real SamplingParams; omit_defaults makes fixtures match real maps. +class EngineCoreSamplingParams(msgspec.Struct, dict=True, omit_defaults=True): + temperature: float = 1.0 + top_p: float = 1.0 + top_k: int = 0 + seed: int | None = None + max_tokens: int = 16 + min_tokens: int = 0 + thinking_token_budget: int | None = None + min_p: float = 0.0 + frequency_penalty: float = 0.0 + presence_penalty: float = 0.0 + repetition_penalty: float = 1.0 + stop_token_ids: list[int] = [] + _eos_token_id: int | None = None + _all_stop_token_ids: set[int] = set() + output_kind: RequestOutputKind = RequestOutputKind.DELTA + + +class EngineCoreRequest( + msgspec.Struct, + array_like=True, + omit_defaults=True, +): + request_id: str + prompt_token_ids: list[int] | None + mm_features: object | None + sampling_params: EngineCoreSamplingParams | None + pooling_params: object | None + arrival_time: float + lora_request: object | None = None + cache_salt: str | None = None + data_parallel_rank: int | None = None + prompt_embeds: object | None = None + prompt_is_token_ids: list[bool] | None = None + client_index: int = 0 + current_wave: int = 0 + priority: int = 0 + trace_headers: dict[str, str] | None = None + resumable: bool = False + external_req_id: str | None = None + reasoning_ended: bool | None = None + reasoning_parser_kwargs: dict[str, object] | None = None + abort_immediately: bool = False + + +class EngineCoreOutput( + msgspec.Struct, + array_like=True, + omit_defaults=True, +): + request_id: str + new_token_ids: list[int] + new_logprobs: object | None = None + new_prompt_logprobs_tensors: object | None = None + pooling_output: object | None = None + finish_reason: FinishReason | None = None + stop_reason: int | str | None = None + events: object | None = None + kv_transfer_params: object | None = None + trace_headers: object | None = None + prefill_stats: object | None = None + routed_experts: object | None = None + num_nans_in_logits: int = 0 + + +class EngineCoreOutputs( + msgspec.Struct, + array_like=True, + omit_defaults=True, +): + engine_index: int = 0 + outputs: list[EngineCoreOutput] = [] + scheduler_stats: object | None = None + timestamp: float = 0.0 + utility_output: object | None = None + finished_requests: set[str] | None = None + wave_complete: int | None = None + start_wave: int | None = None + + +request = EngineCoreRequest( + request_id="req-1", + prompt_token_ids=[11, 22], + mm_features=None, + sampling_params=EngineCoreSamplingParams( + temperature=0.8, + top_p=0.9, + top_k=8, + seed=None, + max_tokens=32, + min_tokens=1, + thinking_token_budget=256, + min_p=0.0, + frequency_penalty=0.0, + presence_penalty=0.0, + repetition_penalty=1.0, + stop_token_ids=[151643], + _eos_token_id=151645, + _all_stop_token_ids={151643, 151645}, + output_kind=RequestOutputKind.FINAL_ONLY, + ), + pooling_params=None, + arrival_time=42.5, + client_index=0, +) + +# All defaults -> empty map. Regression guard for the sparse-map decode. +defaults_request = EngineCoreRequest( + request_id="req-defaults", + prompt_token_ids=[5, 6, 7], + mm_features=None, + sampling_params=EngineCoreSamplingParams(), + pooling_params=None, + arrival_time=1.0, +) + +multimodal_tensor = np.array([[1.0, 2.0], [3.5, 4.25]], dtype=np.float32) +multimodal_features = [ + { + "data": { + "pixel_values": { + "data": [ + "float32", + [2, 2], + msgpack.ExtType(3, multimodal_tensor.tobytes()), + ], + "field": [ + "flat", + { + "slices": [[0, 2, None]], + "dim": 0, + "keep_on_cpu": False, + }, + ], + } + }, + "modality": "image", + "identifier": "mm-cache-key", + "mm_position": { + "offset": 1, + "length": 2, + "is_embed": None, + }, + "mm_hash": "processor-hash", + } +] +multimodal_request_wire = [ + "req-mm", + [101, 102, 103, 104], + multimodal_features, + None, + None, + 43.5, +] + +outputs = EngineCoreOutputs( + outputs=[ + EngineCoreOutput( + request_id="req-1", + new_token_ids=[7, 8], + finish_reason=FinishReason.LENGTH, + ) + ], + finished_requests={"req-1"}, +) + + +def encode_ndarray( + array: np.ndarray, + buffers: list[bytes], + *, + size_threshold: int = 256, +): + arr_data = array.data if array.flags.c_contiguous else array.tobytes() + if not array.shape or array.nbytes < size_threshold: + data = msgpack.ExtType(3, bytes(arr_data)) + else: + data = len(buffers) + buffers.append(bytes(arr_data)) + return [array.dtype.str, list(array.shape), data] + + +def encode_tensor_like( + dtype: str, + shape: list[int], + payload: bytes, + buffers: list[bytes], + *, + size_threshold: int = 256, +): + if len(payload) < size_threshold: + data = msgpack.ExtType(3, payload) + else: + data = len(buffers) + buffers.append(payload) + return [dtype, shape, data] + + +def encode_output_frames(obj, *, size_threshold: int = 256) -> list[bytes]: + buffers = [b""] + + def transform(value): + if isinstance(value, np.ndarray): + return encode_ndarray(value, buffers, size_threshold=size_threshold) + if ( + isinstance(value, tuple) + and len(value) == 3 + and value[0] in ("int32", "int64", "float32") + ): + dtype, shape, payload = value + return encode_tensor_like( + dtype, + shape, + payload, + buffers, + size_threshold=size_threshold, + ) + if type(value) is list: + return [transform(v) for v in value] + if type(value) is tuple: + return [transform(v) for v in value] + if type(value) is dict: + return {k: transform(v) for k, v in value.items()} + return value + + buffers[0] = msgpack.packb(transform(obj), use_bin_type=True) + return buffers + + +def engine_output_wire( + request_id: str, + *, + new_logprobs=None, + new_prompt_logprobs_tensors=None, +): + return [ + request_id, + [7, 8], + new_logprobs, + new_prompt_logprobs_tensors, + None, + int(FinishReason.LENGTH), + ] + + +def engine_outputs_wire(output): + return [0, [output], None, 0.0, None, ["req-1"]] + + +inline_logprobs = engine_outputs_wire( + engine_output_wire( + "req-1", + new_logprobs=( + np.array([[1, 2, 3], [4, 5, 6]], dtype=np.int64), + np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32), + np.array([1, 2], dtype=np.int64), + None, + ), + ) +) + +multipart_logprobs = engine_outputs_wire( + engine_output_wire( + "req-1", + new_logprobs=( + np.array([[1, 2, 3], [4, 5, 6]], dtype=np.int64), + np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32), + np.array([1, 2], dtype=np.int64), + None, + ), + ) +) + +inline_prompt_logprobs = engine_outputs_wire( + engine_output_wire( + "req-1", + new_prompt_logprobs_tensors=( + ( + "int64", + [2, 3], + np.array([[10, 11, 12], [13, 14, 15]], dtype=np.int64).tobytes(), + ), + ( + "float32", + [2, 3], + np.array( + [[10, 11, 12], [13, 14, 15]], + dtype=np.float32, + ).tobytes(), + ), + ("int64", [2], np.array([3, 4], dtype=np.int64).tobytes()), + None, + ), + ) +) + +multipart_prompt_logprobs = engine_outputs_wire( + engine_output_wire( + "req-1", + new_prompt_logprobs_tensors=( + ( + "int64", + [2, 3], + np.array([[10, 11, 12], [13, 14, 15]], dtype=np.int64).tobytes(), + ), + ( + "float32", + [2, 3], + np.array( + [[10, 11, 12], [13, 14, 15]], + dtype=np.float32, + ).tobytes(), + ), + ("int64", [2], np.array([3, 4], dtype=np.int64).tobytes()), + None, + ), + ) +) + + +@dataclass +class EngineCoreReadyResponse: + max_model_len: int + num_gpu_blocks: int + block_size: int + dp_stats_address: str | None + dtype: str + vllm_version: str + world_size: int + data_parallel_size: int + kv_cache_size_tokens: int | None = None + kv_cache_max_concurrency: float | None = None + + +ready_response = EngineCoreReadyResponse( + max_model_len=32768, + num_gpu_blocks=1000, + block_size=16, + dp_stats_address=None, + dtype="float32", + vllm_version="0.0.0", + data_parallel_size=1, + world_size=1, +) + +print(msgspec.msgpack.encode(request).hex()) +print(msgspec.msgpack.encode(defaults_request).hex()) +print(msgpack.packb(multimodal_request_wire, use_bin_type=True).hex()) +print(msgspec.msgpack.encode(outputs).hex()) +print(" ".join(frame.hex() for frame in encode_output_frames(inline_logprobs))) +print( + " ".join( + frame.hex() + for frame in encode_output_frames(multipart_logprobs, size_threshold=1) + ) +) +print(" ".join(frame.hex() for frame in encode_output_frames(inline_prompt_logprobs))) +print( + " ".join( + frame.hex() + for frame in encode_output_frames(multipart_prompt_logprobs, size_threshold=1) + ) +) +print(msgspec.msgpack.encode(ready_response).hex()) diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/transport.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/transport.rs new file mode 100644 index 0000000000000000000000000000000000000000..00557d9fc84abd504d3928eacc978ca5cba29381 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/transport.rs @@ -0,0 +1,596 @@ +use std::collections::{BTreeMap, BTreeSet}; +use std::fmt::Debug; +use std::ops::Deref; +use std::time::Duration; + +use bytes::Bytes; +use enum_as_inner::EnumAsInner; +use thiserror_ext::AsReport; +use tokio::sync::mpsc; +use tokio::time::timeout; +use tracing::{debug, error, info, trace, warn}; +use zeromq::prelude::{Socket, SocketRecv, SocketSend}; +use zeromq::util::PeerIdentity; +use zeromq::{PullSocket, RouterSendHalf, RouterSocket, ZmqError, ZmqMessage}; + +use crate::coordinator::CoordinatorBootstrap; +use crate::error::{Error, Result, bail_unexpected_handshake_message}; +use crate::protocol::handshake::{ + EngineCoreReadyResponse, HandshakeAddresses, HandshakeInitMessage, ReadyMessage, +}; +use crate::protocol::output::{EngineCoreOutputs, decode_engine_core_outputs}; +use crate::protocol::{decode_msgpack, encode_msgpack}; + +/// Dedicated single-frame sentinel emitted by Python `EngineCoreProc` when the +/// engine dies. +pub const ENGINE_CORE_DEAD_SENTINEL: &[u8] = b"ENGINE_CORE_DEAD"; + +/// Opaque routing identity of one engine on the frontend transport. +#[derive(Clone, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct EngineId(Bytes); + +impl Debug for EngineId { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + // Display the engine id as a hex string for easier debugging. + write!(f, "EngineId({})", hex::encode(&self.0)) + } +} + +impl EngineId { + /// Convert the engine id into a ZMQ frame for sending. + pub fn to_frame(&self) -> Bytes { + self.0.clone() + } + + /// Convert the engine id into a ZMQ frame for sending. + pub fn into_frame(self) -> Bytes { + self.0 + } + + /// Parse the Python-compatible engine index encoded in the routing + /// identity. + /// + /// Python `EngineCoreProc` currently uses a two-byte little-endian engine + /// index as its ROUTER/DEALER identity. Coordinator control messages + /// such as `START_DP_WAVE(exclude_engine_index)` need that engine-side + /// index rather than any frontend-local ordering. + pub fn engine_index(&self) -> Option { + if self.len() != 2 { + return None; + } + Some(u16::from_le_bytes([self[0], self[1]]) as u32) + } + + /// Construct an engine id from the Python-compatible engine index encoding + /// (two-byte little-endian). + pub fn from_engine_index(value: u32) -> Self { + Self(Bytes::copy_from_slice(&(value as u16).to_le_bytes())) + } +} + +impl Deref for EngineId { + type Target = [u8]; + + fn deref(&self) -> &Self::Target { + self.0.as_ref() + } +} + +impl From> for EngineId { + fn from(value: Vec) -> Self { + Self(Bytes::from(value)) + } +} + +impl From<&[u8; N]> for EngineId { + fn from(value: &[u8; N]) -> Self { + Self(Bytes::copy_from_slice(value)) + } +} + +impl TryFrom for PeerIdentity { + type Error = ZmqError; + + fn try_from(value: EngineId) -> std::result::Result { + PeerIdentity::try_from(value.into_frame()) + } +} + +/// Per-engine handshake result collected while bootstrapping one shared +/// transport. +#[derive(Clone, Debug)] +pub struct ConnectedEngine { + /// The identity of the connected engine. + pub engine_id: EngineId, + /// Post-initialization configuration received from the engine on the input + /// socket registration message. + pub ready_response: EngineCoreReadyResponse, +} + +/// Represents the connected shared transport plus all registered engines after +/// a successful multi-engine startup handshake. +pub struct ConnectedTransport { + /// The local address of the shared input socket that all engines connect to + /// for receiving requests. + pub input_address: String, + /// The local address of the shared output socket that all engines connect + /// to for sending responses. + pub output_address: String, + /// All engines connected through the startup handshake. + pub engines: Vec, + /// Optional engine-facing coordinator transport used for in-process wave + /// coordination. + pub coordinator: Option, + + /// The sending half of the shared input socket. + pub input_send: RouterSendHalf, + /// The shared output socket for receiving responses from all engines. + pub output_socket: PullSocket, +} + +#[derive(Clone, Debug, EnumAsInner)] +enum EngineStartupState { + HelloReceived, + ReadyReceived, +} + +/// Connect to one or more engines through the startup handshake protocol, +/// returning the shared data-plane transport plus the registered engines. +pub async fn connect_handshake( + handshake_address: &str, + engine_count: usize, + local_host: &str, + local_input_address: Option<&str>, + local_output_address: Option<&str>, + enable_inproc_coordinator: bool, + ready_timeout: Duration, +) -> Result { + if engine_count == 0 { + bail_unexpected_handshake_message!("expected engine_count >= 1"); + } + + info!( + engine_count, + handshake_address, "waiting for engines to connect" + ); + + // 1. Bind shared local input/output sockets first so every engine receives the same data-plane + // addresses during handshake. + debug!( + local_host, + ?ready_timeout, + engine_count, + "binding shared transport sockets" + ); + let (input_address, mut input_socket, output_address, output_socket) = + bind_local_sockets(local_host, local_input_address, local_output_address).await?; + info!(%input_address, %output_address, "bound local transport sockets"); + + let mut coordinator = if enable_inproc_coordinator { + Some(CoordinatorBootstrap::bind(local_host).await?) + } else { + None + }; + + // 2. Bind the shared handshake socket once. All engines connect to this socket with their own + // identities, and startup order does not matter. + let mut handshake_socket = RouterSocket::new(); + handshake_socket.bind(handshake_address).await?; + + let mut engines = BTreeMap::new(); + + // 3. Receive HELLO from every engine and send a matching INIT. When coordinator mode is + // enabled, the engines will not emit READY until the coordinator barrier below completes. + while engines.len() < engine_count { + debug!( + handshake_address, + connected = engines.len(), + waiting_for = engine_count, + "waiting for engine HELLO" + ); + let message = timeout(ready_timeout, handshake_socket.recv()).await.map_err(|_| { + Error::HandshakeTimeout { + stage: "HELLO", + timeout: ready_timeout, + } + })??; + let (engine_id, handshake_message) = decode_handshake_message(message, None)?; + match handshake_message.status.as_deref() { + Some("HELLO") => { + if engines.contains_key(&engine_id) { + bail_unexpected_handshake_message!( + "duplicate engine id {engine_id:?} observed during startup handshake" + ); + } + debug!(handshake_address, ?engine_id, "received HELLO from engine"); + + send_init_message( + &mut handshake_socket, + &engine_id, + &input_address, + &output_address, + coordinator.as_ref(), + ) + .await?; + debug!(handshake_address, ?engine_id, "sent INIT to engine"); + + engines.insert(engine_id.clone(), EngineStartupState::HelloReceived); + } + Some("READY") => { + if coordinator.is_some() { + bail_unexpected_handshake_message!( + "received READY for engine id {engine_id:?} before coordinator startup gate completed" + ); + } + let state = match engines.get_mut(&engine_id) { + Some(state) if !state.is_ready_received() => state, + _ => { + bail_unexpected_handshake_message!( + "received READY for unexpected or duplicate engine id {engine_id:?}" + ); + } + }; + debug!( + handshake_address, + ?engine_id, + ?handshake_message, + "received overlapping READY from engine during HELLO phase" + ); + *state = EngineStartupState::ReadyReceived; + } + other => { + bail_unexpected_handshake_message!("unexpected handshake status {other:?}"); + } + } + } + + // 4. Optional coordinator startup gate. Without coordinator there is nothing to do. + if let Some(coordinator) = coordinator.as_mut() { + coordinator.wait_for_startup_gate(engine_count, ready_timeout).await?; + } + + // 5. After the optional gate has opened, every engine may now send READY. + while engines.values().any(|state| !state.is_ready_received()) { + debug!( + handshake_address, + connected = engines.len(), + ready = engines.values().filter(|state| state.is_ready_received()).count(), + waiting_for = engine_count, + "waiting for engine READY" + ); + let message = timeout(ready_timeout, handshake_socket.recv()).await.map_err(|_| { + Error::HandshakeTimeout { + stage: "READY", + timeout: ready_timeout, + } + })??; + let (engine_id, handshake_message) = decode_handshake_message(message, None)?; + match handshake_message.status.as_deref() { + Some("READY") => { + let state = match engines.get_mut(&engine_id) { + Some(state) if !state.is_ready_received() => state, + _ => { + bail_unexpected_handshake_message!( + "received READY for unexpected or duplicate engine id {engine_id:?}" + ); + } + }; + debug!( + handshake_address, + ?engine_id, + ?handshake_message, + "received READY from engine" + ); + *state = EngineStartupState::ReadyReceived; + } + Some("HELLO") => { + bail_unexpected_handshake_message!( + "received duplicate HELLO for engine id {engine_id:?} after INIT phase completed" + ); + } + other => { + bail_unexpected_handshake_message!("unexpected handshake status {other:?}"); + } + } + } + + // 6. Wait for every engine to connect to the shared input socket and register itself. + let engines = + wait_for_input_registrations(&mut input_socket, engines.into_keys(), ready_timeout).await?; + debug!( + engine_count = engines.len(), + "all engines registered on shared input socket" + ); + + info!(engine_count = engines.len(), "engines connected"); + + let (input_send, _) = input_socket.split(); + + Ok(ConnectedTransport { + input_address, + output_address, + input_send, + output_socket, + engines, + coordinator, + }) +} + +/// Bind to Python-supplied frontend transport addresses and wait for +/// already-initialized engines to register themselves on the input socket. +/// +/// This path mirrors Python's externally managed `AsyncMPClient` bootstrap +/// model: the addresses are already fixed by the supervisor, and engine +/// identities are synthesized from contiguous rank order instead of being +/// discovered through a Rust-owned handshake. +pub async fn connect_bootstrapped( + input_address: &str, + output_address: &str, + engine_start_index: u32, + engine_count: usize, + ready_timeout: Duration, +) -> Result { + if engine_count == 0 { + bail_unexpected_handshake_message!("expected engine_count >= 1"); + } + + let mut input_socket = RouterSocket::new(); + let input_address = input_socket.bind(input_address).await?.to_string(); + + let mut output_socket = PullSocket::new(); + let output_address = output_socket.bind(output_address).await?.to_string(); + + let engines = wait_for_input_registrations( + &mut input_socket, + (0..engine_count) + .map(|offset| EngineId::from_engine_index(engine_start_index + offset as u32)), + ready_timeout, + ) + .await?; + info!( + engine_count = engines.len(), + "bootstrapped engines connected" + ); + + let (input_send, _) = input_socket.split(); + + Ok(ConnectedTransport { + input_address, + output_address, + engines, + coordinator: None, + input_send, + output_socket, + }) +} + +/// Bind new input and output sockets. +async fn bind_local_sockets( + local_host: &str, + local_input_address: Option<&str>, + local_output_address: Option<&str>, +) -> Result<(String, RouterSocket, String, PullSocket)> { + let mut input_socket = RouterSocket::new(); + let input_bind_address = local_input_address + .map(str::to_owned) + .unwrap_or_else(|| format!("tcp://{local_host}:0")); + let input_address = input_socket.bind(&input_bind_address).await?.to_string(); + + let mut output_socket = PullSocket::new(); + let output_bind_address = local_output_address + .map(str::to_owned) + .unwrap_or_else(|| format!("tcp://{local_host}:0")); + let output_address = output_socket.bind(&output_bind_address).await?.to_string(); + + Ok((input_address, input_socket, output_address, output_socket)) +} + +/// Decode a handshake message and validate its structure and identity. +fn decode_handshake_message( + message: ZmqMessage, + expected_id: Option<&EngineId>, +) -> Result<(EngineId, ReadyMessage)> { + if message.len() != 2 { + bail_unexpected_handshake_message!("expected 2 frames, got {}", message.len()); + } + + let frames = message.into_vec(); + let actual_id = EngineId(frames[0].clone()); + if let Some(expected_id) = expected_id + && actual_id != *expected_id + { + return Err(Error::UnexpectedHandshakeIdentity { + expected: expected_id.to_vec(), + actual: actual_id.to_vec(), + }); + } + + let handshake_message: ReadyMessage = decode_msgpack(&frames[1])?; + Ok((actual_id, handshake_message)) +} + +/// Send an INIT message to the engine with the local socket addresses for the +/// engine to connect to, using the handshake socket. +async fn send_init_message( + handshake_socket: &mut RouterSocket, + engine_id: &EngineId, + input_address: &str, + output_address: &str, + coordinator: Option<&CoordinatorBootstrap>, +) -> Result<()> { + let init_message = HandshakeInitMessage { + addresses: HandshakeAddresses { + inputs: vec![input_address.to_string()], + outputs: vec![output_address.to_string()], + coordinator_input: coordinator.map(|c| c.input_address.clone()), + coordinator_output: coordinator.map(|c| c.output_address.clone()), + frontend_stats_publish_address: None, + }, + parallel_config: Default::default(), + }; + let payload = encode_msgpack(&init_message)?; + let message = ZmqMessage::try_from(vec![engine_id.to_frame(), Bytes::from(payload)]) + .expect("handshake router messages must contain identity and payload"); + handshake_socket.send(message).await?; + Ok(()) +} + +/// Receive the input registration message from each engine and validate its +/// identity. +/// +/// Each registration contains 2 frames: `[identity, ready-payload]`. +/// +/// Since vLLM commit `c8d98f81f676552c263f35bbde55e6edbe81b4e8` ("[Core] +/// Simplify API server handshake"), the payload is a msgpack-encoded +/// [`EngineCoreReadyResponse`] carrying post-initialization values such as +/// `max_model_len`. +async fn wait_for_input_registrations( + input_socket: &mut RouterSocket, + expected_engines: impl IntoIterator, + ready_timeout: Duration, +) -> Result> { + let expected_engines = expected_engines.into_iter().collect::>(); + let mut pending = expected_engines.iter().cloned().collect::>(); + let mut ready_responses = BTreeMap::new(); + + while !pending.is_empty() { + let registration = timeout(ready_timeout, input_socket.recv()).await.map_err(|_| { + Error::InputRegistrationTimeout { + timeout: ready_timeout, + } + })??; + + if registration.len() != 2 { + bail_unexpected_handshake_message!( + "expected 2 frames for engine input registration, got {}", + registration.len() + ); + } + + let frames = registration.into_vec(); + let actual_id = EngineId(frames[0].clone()); + if !pending.remove(&actual_id) { + bail_unexpected_handshake_message!( + "received input registration for unexpected engine id {actual_id:?}" + ); + } + + if frames[1].is_empty() { + bail_unexpected_handshake_message!( + "expected msgpack EngineCoreReadyResponse for engine input registration, got empty payload from engine id {actual_id:?}" + ); + } + + let ready_response: EngineCoreReadyResponse = decode_msgpack(&frames[1])?; + debug!( + ?actual_id, + ?ready_response, + "received input registration from engine" + ); + ready_responses.insert(actual_id, ready_response); + } + + Ok(expected_engines + .into_iter() + .map(|engine_id| { + let ready_response = ready_responses + .remove(&engine_id) + .expect("every expected engine id has a decoded ready response"); + ConnectedEngine { + engine_id, + ready_response, + } + }) + .collect()) +} + +/// Send an encoded message to the engine through the input socket. +pub async fn send_message( + input_send: &mut RouterSendHalf, + engine_id: &EngineId, + request_type: Bytes, + payload: Vec, +) -> Result<()> { + let message = ZmqMessage::try_from(vec![ + engine_id.to_frame(), + request_type, + Bytes::from(payload), + ]) + .expect("router messages must contain identity and payload"); + + trace!( + ?engine_id, + frame_count = message.len(), + "sending ZMQ message" + ); + input_send.send(message).await?; + Ok(()) +} + +/// Run the output loop to receive messages from the engine and send them to the +/// provided channel. +pub async fn run_output_loop( + mut output_socket: PullSocket, + tx: mpsc::Sender>, +) { + loop { + let message = match output_socket.recv().await { + Ok(message) => message, + Err(error) => { + // If we fail to receive a message from the engine, it's likely that the engine + // has crashed or become unreachable, so we should notify the + // client and shut down the output loop. + error!(error = %error.as_report(), "failed to receive output message"); + let _ = tx.send(Err(Error::Transport(error))).await; + return; + } + }; + + let frame_count = message.len(); + trace!(frame_count, "received output message"); + let frames = message.into_vec(); + let frame = frames.first().expect("output message must have at least one frame"); + let frame_len = frame.len(); + if frame.as_ref() == ENGINE_CORE_DEAD_SENTINEL { + warn!("received ENGINE_CORE_DEAD sentinel from engine"); + let _ = tx.send(Err(Error::EngineCoreDead)).await; + return; + } + let decoded = match decode_engine_core_outputs(&frames) { + Ok(decoded) => { + trace!(frame_len, outputs = ?decoded, "decoded output message"); + Ok(decoded) + } + Err(error) => { + // If we fail to decode the message from the engine, notify the client but keep + // the output loop running to continue processing future + // messages from the engine. + warn!(frame_len, error = %error.as_report(), "failed to decode output message"); + Err(error) + } + }; + + if tx.send(decoded).await.is_err() { + // If we fail to send the decoded message to the client, it's likely that the + // client has shut down, so we should shut down the output loop as + // well. + warn!("output loop rx dropped, shutting down output loop"); + return; + } + } +} + +#[cfg(test)] +mod tests { + use super::bind_local_sockets; + + #[tokio::test] + async fn bind_local_sockets_resolves_zero_port_bindings() { + let (input_address, _input_socket, output_address, _output_socket) = + bind_local_sockets("127.0.0.1", None, None).await.expect("bind local sockets"); + + assert!(input_address.starts_with("tcp://127.0.0.1:")); + assert!(output_address.starts_with("tcp://127.0.0.1:")); + assert_ne!(input_address, output_address); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/llm/Cargo.toml b/asian_ai/performance/vLLM/rust/src/llm/Cargo.toml new file mode 100644 index 0000000000000000000000000000000000000000..fca9f28b70d594488c2c0df93593e45bc4d236fa --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/llm/Cargo.toml @@ -0,0 +1,39 @@ +[package] +name = "vllm-llm" +version.workspace = true +edition.workspace = true +license.workspace = true + +[features] +test-util = [] + +[dependencies] +easy-ext.workspace = true +enum-as-inner.workspace = true +futures.workspace = true +parking_lot.workspace = true +serde.workspace = true +serde_json.workspace = true +thiserror.workspace = true +tokio.workspace = true +tokio-util.workspace = true +tracing.workspace = true +uuid.workspace = true +vllm-engine-core-client.workspace = true +vllm-metrics.workspace = true + +[dev-dependencies] +anyhow.workspace = true +bytes.workspace = true +clap.workspace = true +expect-test.workspace = true +rmp-serde.workspace = true +tokio.workspace = true +tracing-subscriber.workspace = true +uuid.workspace = true +vllm-engine-core-client = { workspace = true, features = ["test-util"] } +vllm-metrics.workspace = true +zeromq.workspace = true + +[lints] +workspace = true diff --git a/asian_ai/performance/vLLM/rust/src/llm/examples/README.md b/asian_ai/performance/vLLM/rust/src/llm/examples/README.md new file mode 100644 index 0000000000000000000000000000000000000000..511f69da7a17fbf2c16fb6257a306a608f3e15fc --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/llm/examples/README.md @@ -0,0 +1,29 @@ +# LLM Smoke Test + +Start headless `vllm`: + +```bash +source ../vllm/.venv/bin/activate +HF_HUB_OFFLINE=1 \ +VLLM_LOGGING_LEVEL=DEBUG \ +VLLM_CPU_KVCACHE_SPACE=2 \ +VLLM_HOST_IP=127.0.0.1 \ +VLLM_LOOPBACK_IP=127.0.0.1 \ +python3 -m vllm.entrypoints.cli.main serve Qwen/Qwen3-0.6B \ + --headless \ + --data-parallel-address 127.0.0.1 \ + --data-parallel-rpc-port 62100 \ + --data-parallel-size-local 1 \ + --max-model-len 512 \ + --dtype float16 +``` + +Run the Rust smoke test through the `vllm-llm` generate interface: + +```bash +cargo run -p vllm-llm --example external_engine_smoke -- \ + --handshake-address tcp://127.0.0.1:62100 \ + --host 127.0.0.1 +``` + +IMPORTANT: You must restart `vllm` each time you run the smoke test, as the vLLM engine cannot manage frontend closures and subsequent reconnects. In other words, do not reuse existing `vllm` instances, if any. diff --git a/asian_ai/performance/vLLM/rust/src/llm/examples/external_engine_smoke.rs b/asian_ai/performance/vLLM/rust/src/llm/examples/external_engine_smoke.rs new file mode 100644 index 0000000000000000000000000000000000000000..b26b62ebe8b9a1e63fb5f6ee40517154788df362 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/llm/examples/external_engine_smoke.rs @@ -0,0 +1,144 @@ +use std::time::Duration; + +use anyhow::{Context, Result, bail}; +use clap::Parser; +use futures::StreamExt as _; +use tokio::time::timeout; +use tracing_subscriber::EnvFilter; +use vllm_engine_core_client::protocol::sampling::EngineCoreSamplingParams; +use vllm_engine_core_client::{EngineCoreClient, EngineCoreClientConfig, TransportMode}; +use vllm_llm::{FinishReason, GenerateOutputStream, GenerateRequest, Llm}; + +const PROMPT_TOKEN_IDS: &[u32] = &[20841, 448, 6896, 25, 23811]; + +#[derive(Debug, Parser)] +#[command(about = "Smoke-test the Rust LLM facade against an external vLLM engine.")] +struct Args { + #[arg(long)] + handshake_address: String, + #[arg(long, default_value_t = 1)] + engine_count: usize, + #[arg(long, default_value = "Qwen/Qwen3-0.6B")] + model: String, + #[arg(long, default_value = "127.0.0.1")] + host: String, + #[arg(long, default_value_t = 0)] + client_index: u32, + #[arg(long, default_value_t = 30)] + ready_timeout_secs: u64, + #[arg(long, default_value_t = 120)] + output_timeout_secs: u64, + #[arg(long, default_value_t = 5)] + max_tokens: u32, +} + +fn unique_request_id() -> String { + format!("rust-llm-smoke-{}", uuid::Uuid::new_v4()) +} + +fn init_tracing() { + let filter = EnvFilter::try_from_default_env() + .unwrap_or_else(|_| EnvFilter::new("vllm_engine_core_client=debug")); + let _ = tracing_subscriber::fmt().with_env_filter(filter).try_init(); +} + +fn build_request(request_id: String, max_tokens: u32) -> GenerateRequest { + GenerateRequest { + request_id, + prompt_token_ids: PROMPT_TOKEN_IDS.to_vec(), + sampling_params: EngineCoreSamplingParams { + max_tokens, + ..EngineCoreSamplingParams::for_test() + }, + mm_features: None, + arrival_time: None, + cache_salt: None, + trace_headers: None, + priority: 0, + data_parallel_rank: None, + reasoning_parser_kwargs: None, + lora_request: None, + } +} + +#[derive(Debug)] +struct CompletedRequest { + token_ids: Vec, + finish_reason: FinishReason, +} + +async fn wait_for_request_completion(mut stream: GenerateOutputStream) -> Result { + let output = match stream.next().await { + Some(output) => output.context("failed to receive request output")?, + None => bail!("request stream ended without a final output"), + }; + + let none = stream.next().await; + assert!( + none.is_none(), + "expected final-only stream to end after the final output" + ); + + let finish_reason = output.finish_reason.expect("final-only output must have a finish reason"); + let token_ids = output.token_ids; + + Ok(CompletedRequest { + token_ids, + finish_reason, + }) +} + +async fn wait_for_timeout( + stream: GenerateOutputStream, + output_timeout: Duration, +) -> Result { + timeout(output_timeout, wait_for_request_completion(stream)) + .await + .context("timed out waiting for request output")? +} + +#[tokio::main(flavor = "multi_thread")] +async fn main() -> Result<()> { + init_tracing(); + let args = Args::parse(); + let ready_timeout = Duration::from_secs(args.ready_timeout_secs); + let output_timeout = Duration::from_secs(args.output_timeout_secs); + let request_id = unique_request_id(); + let client = EngineCoreClient::connect(EngineCoreClientConfig { + transport_mode: TransportMode::HandshakeOwner { + handshake_address: args.handshake_address.clone(), + advertised_host: args.host.clone(), + engine_count: args.engine_count, + ready_timeout, + local_input_address: None, + local_output_address: None, + }, + coordinator_mode: None, + model_name: args.model.clone(), + client_index: args.client_index, + }) + .await + .context("failed to connect to external vLLM engine")?; + + println!("model={}", args.model); + println!("handshake_address={}", args.handshake_address); + println!("engine_count={}", args.engine_count); + println!("input_address={}", client.input_address()); + println!("output_address={}", client.output_address()); + println!("engine_identities={:x?}", client.engine_identities()); + + let llm = Llm::new(client); + let request = build_request(request_id.clone(), args.max_tokens); + println!("request_id={request_id}"); + println!("prompt_token_ids={PROMPT_TOKEN_IDS:?}"); + + let stream = llm.generate(request).await.context("failed to submit generate request")?; + let output = wait_for_timeout(stream, output_timeout).await?; + + llm.shutdown().await.context("failed to shut down llm client")?; + + println!("token_ids={:?}", output.token_ids); + println!("finish_reason={:?}", output.finish_reason); + + Ok(()) +} diff --git a/asian_ai/performance/vLLM/rust/src/llm/src/error.rs b/asian_ai/performance/vLLM/rust/src/llm/src/error.rs new file mode 100644 index 0000000000000000000000000000000000000000..8771a03173d5fa0cdfa659f61e218e25404c3620 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/llm/src/error.rs @@ -0,0 +1,12 @@ +use thiserror::Error; + +pub type Result = std::result::Result; + +/// Public error type for the Rust `llm` facade. +#[derive(Debug, Error)] +pub enum Error { + #[error("generate request `{request_id}` has an empty prompt_token_ids")] + EmptyPromptTokenIds { request_id: String }, + #[error("engine-core error")] + EngineCoreClient(#[from] vllm_engine_core_client::Error), +} diff --git a/asian_ai/performance/vLLM/rust/src/llm/src/inflight.rs b/asian_ai/performance/vLLM/rust/src/llm/src/inflight.rs new file mode 100644 index 0000000000000000000000000000000000000000..92b9de71c183a5b433012a12e4e618617f0ae705 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/llm/src/inflight.rs @@ -0,0 +1,179 @@ +//! Tracking of the external→internal request-id mapping for in-flight requests. +//! +//! When request-id randomization is enabled (the default), [`crate::Llm`] +//! rewrites the external (user-supplied) request id into a unique internal +//! engine id before reaching engine-core. Engine-core only ever knows the +//! internal id, so aborting a request by its external id requires resolving it +//! back to the internal id(s) first. + +use std::collections::HashMap; +use std::sync::{Arc, Weak}; + +use parking_lot::Mutex; + +/// external id → internal id → number of live guards holding that edge. +type InflightMap = HashMap>; + +/// Maps external (user-supplied) request ids to the set of live internal engine +/// request ids they currently expand into. +/// +/// One external id may map to multiple internal ids: duplicate external ids +/// submitted concurrently each get their own randomized internal id, and an +/// abort by the shared external id must reach all of them. Edges are +/// refcounted: with randomization disabled the same (external, internal) pair +/// can be tracked by several guards in sequence (e.g. a finished request whose +/// stream is still held alongside a fresh submission reusing the id), and the +/// edge must survive until the last guard drops. +#[derive(Default)] +pub(crate) struct InflightRequests { + map: Arc>, +} + +impl InflightRequests { + pub(crate) fn new() -> Self { + Self::default() + } + + /// Record that `internal` is now an in-flight engine request for the + /// `external` request id, returning a guard that removes the edge when the + /// request's output stream is dropped (on clean finish or cancellation). + pub(crate) fn track(&self, external: String, internal: String) -> RequestGuard { + *self + .map + .lock() + .entry(external.clone()) + .or_default() + .entry(internal.clone()) + .or_insert(0) += 1; + RequestGuard { + map: Arc::downgrade(&self.map), + external, + internal, + } + } + + /// Resolve external request ids to the internal engine ids currently + /// in-flight for them. Unknown or already-finished ids contribute nothing. + pub(crate) fn resolve(&self, external_ids: &[String]) -> Vec { + let map = self.map.lock(); + external_ids + .iter() + .filter_map(|external| map.get(external)) + .flat_map(|internal_ids| internal_ids.keys()) + .cloned() + .collect() + } + + #[cfg(test)] + fn is_empty(&self) -> bool { + self.map.lock().is_empty() + } +} + +/// RAII guard that releases one refcount on a single external→internal edge +/// when dropped, removing the edge once no live guard holds it. +/// +/// Held by the per-request output stream, so cleanup runs whether the stream +/// terminates cleanly or is cancelled. A [`Weak`] handle is used so a stream +/// outliving its owning [`InflightRequests`] does not keep the map alive. +pub(crate) struct RequestGuard { + map: Weak>, + external: String, + internal: String, +} + +impl Drop for RequestGuard { + fn drop(&mut self) { + let Some(map) = self.map.upgrade() else { + return; + }; + let mut map = map.lock(); + if let Some(internal_ids) = map.get_mut(&self.external) { + if let Some(count) = internal_ids.get_mut(&self.internal) { + *count -= 1; + if *count == 0 { + internal_ids.remove(&self.internal); + } + } + if internal_ids.is_empty() { + map.remove(&self.external); + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn resolves_external_to_internal() { + let inflight = InflightRequests::new(); + let _guard = inflight.track("ext".to_string(), "ext-abc".to_string()); + + assert_eq!( + inflight.resolve(&["ext".to_string()]), + vec!["ext-abc".to_string()] + ); + assert!(inflight.resolve(&["unknown".to_string()]).is_empty()); + } + + #[test] + fn one_external_maps_to_many_internal() { + let inflight = InflightRequests::new(); + let _g1 = inflight.track("dup".to_string(), "dup-1".to_string()); + let _g2 = inflight.track("dup".to_string(), "dup-2".to_string()); + + let mut resolved = inflight.resolve(&["dup".to_string()]); + resolved.sort(); + assert_eq!(resolved, vec!["dup-1".to_string(), "dup-2".to_string()]); + } + + #[test] + fn dropping_guard_removes_only_its_own_edge_then_cleans_empty_key() { + let inflight = InflightRequests::new(); + let g1 = inflight.track("dup".to_string(), "dup-1".to_string()); + let g2 = inflight.track("dup".to_string(), "dup-2".to_string()); + + drop(g1); + assert_eq!( + inflight.resolve(&["dup".to_string()]), + vec!["dup-2".to_string()] + ); + + drop(g2); + assert!(inflight.resolve(&["dup".to_string()]).is_empty()); + assert!( + inflight.is_empty(), + "empty external key must be removed, not left dangling" + ); + } + + #[test] + fn identical_edges_are_refcounted_across_guards() { + // With request-id randomization disabled, internal == external, so two + // tracked requests can share the exact same edge. Dropping one guard + // (e.g. a stale stream, or the error path of a rejected duplicate + // submission) must not untrack the other still-live request. + let inflight = InflightRequests::new(); + let g1 = inflight.track("x".to_string(), "x".to_string()); + let g2 = inflight.track("x".to_string(), "x".to_string()); + + drop(g1); + assert_eq!(inflight.resolve(&["x".to_string()]), vec!["x".to_string()]); + + drop(g2); + assert!(inflight.resolve(&["x".to_string()]).is_empty()); + assert!(inflight.is_empty()); + } + + #[test] + fn guard_drop_is_a_noop_after_inflight_is_gone() { + let guard = { + let inflight = InflightRequests::new(); + inflight.track("ext".to_string(), "ext-abc".to_string()) + }; + // Dropping the guard after the owning map is gone must not panic. + drop(guard); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/llm/src/lib.rs b/asian_ai/performance/vLLM/rust/src/llm/src/lib.rs new file mode 100644 index 0000000000000000000000000000000000000000..ed04adb20bbfc51cce7081bcbc18e2cfbe154465 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/llm/src/lib.rs @@ -0,0 +1,130 @@ +use tracing::Span; +use vllm_engine_core_client::EngineCoreClient; + +mod error; +mod inflight; +mod log_stats; +mod output; +mod request; +mod request_metrics; + +pub use error::{Error, Result}; +pub use output::{ + CollectedGenerateOutput, FinishReason, GenerateOutput, GenerateOutputStream, + GenerateOutputStreamExt, GeneratePromptInfo, TokenUsage, +}; +pub use request::GenerateRequest; +pub use vllm_engine_core_client::protocol::logprobs::{Logprobs, PositionLogprobs, TokenLogprob}; + +use crate::inflight::InflightRequests; +use crate::log_stats::StatsLogger; +use crate::request_metrics::RequestMetricsTracker; + +/// Thin generate-and-abort facade over [`EngineCoreClient`]. +/// +/// This mirrors the narrow public shape of Python `AsyncLLM.generate()` and +/// `abort()`, but keeps the boundary close to raw engine-core requests and +/// outputs. It tracks an in-flight external→internal request-id index (see +/// [`InflightRequests`]) so that aborts issued against external (user-supplied) +/// ids can be resolved to the internal engine ids that engine-core understands. +pub struct Llm { + client: EngineCoreClient, + randomize_request_id: bool, + stats_logger: Option, + inflight: InflightRequests, +} + +impl Llm { + /// Create a new minimal LLM facade from an already connected engine-core + /// client. + pub fn new(client: EngineCoreClient) -> Self { + Self { + client, + randomize_request_id: true, + stats_logger: None, + inflight: InflightRequests::new(), + } + } + + /// Enable or disable periodic stats logging. + pub fn with_log_stats(mut self, enabled: bool) -> Self { + if enabled { + let stats_logger = StatsLogger::start( + self.client.model_name().to_string(), + self.client.engine_indices(), + ); + self.stats_logger = Some(stats_logger); + } else { + self.stats_logger = None; + } + self + } + + /// Control whether external request ids are randomized before reaching + /// engine-core. + pub fn with_request_id_randomization(mut self, enabled: bool) -> Self { + self.randomize_request_id = enabled; + self + } + + /// Expose the underlying engine-core client for low-level utility/admin + /// calls. + pub fn engine_core_client(&self) -> &EngineCoreClient { + &self.client + } + + /// Submit one tokenized generate request and return a per-request output + /// stream. + pub async fn generate(&self, req: GenerateRequest) -> Result { + let prepared = req.prepare(self.randomize_request_id)?; + let prompt_token_ids = prepared.prompt_token_ids().into(); + let external_request_id = prepared + .engine_request + .external_req_id + .clone() + .expect("prepare always sets external_req_id"); + let internal_request_id = prepared.engine_request.request_id.clone(); + + // Record internal engine-core request ID in the current tracing span. + Span::current().record("engine_request_id", &internal_request_id); + + let request_metrics = RequestMetricsTracker::new( + self.client.model_name().to_string(), + prepared.engine_request.arrival_time, + prepared.prompt_token_ids().len() as u32, + (prepared.engine_request.sampling_params.as_ref()).map(|p| p.max_tokens), + 1, + ); + let stream = self.client.call(prepared.engine_request).await?; + let guard = self.inflight.track(external_request_id, internal_request_id); + + Ok(GenerateOutputStream::new( + prompt_token_ids, + stream, + request_metrics, + guard, + )) + } + + /// Abort in-flight requests by their external (user-supplied) request ids. + /// + /// External ids are resolved to the internal engine ids actually known to + /// engine-core (one external id may map to several internal ids). Unknown + /// or already-finished ids resolve to nothing and are a safe no-op. The + /// tracking entries themselves are removed when the corresponding output + /// streams are dropped, not here. + pub async fn abort(&self, external_ids: &[String]) -> Result<()> { + let internal_ids = self.inflight.resolve(external_ids); + if internal_ids.is_empty() { + return Ok(()); + } + self.client.abort(&internal_ids).await?; + Ok(()) + } + + /// Shut down the underlying engine-core client and its background tasks. + pub async fn shutdown(self) -> Result<()> { + self.client.shutdown().await?; + Ok(()) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/llm/src/log_stats.rs b/asian_ai/performance/vLLM/rust/src/llm/src/log_stats.rs new file mode 100644 index 0000000000000000000000000000000000000000..6d23f4daf1624d97a13f4fa5cc4720fec80128d4 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/llm/src/log_stats.rs @@ -0,0 +1,552 @@ +use std::fmt::Write; +use std::time::{Duration, Instant}; + +use tokio_util::task::AbortOnDropHandle; +use tracing::{debug, info}; +use vllm_metrics::{ + EngineLabels, F64Gauge, METRICS, PromptTokenSourceLabels, SchedulerLogStatsAccumulator, + SchedulerLogStatsInterval, U64Counter, U64Gauge, WaitingReasonLabels, +}; + +const LOG_STATS_INTERVAL: Duration = Duration::from_secs(10); +const WAITING_REASON_DEFERRED: &str = "deferred"; + +/// Cached, cloned metric handles for one engine. Each clone shares the same +/// underlying `Arc` as the prometheus `Family` entry, so reads go +/// straight to the atomic with no lock. +struct EngineMetrics { + // Counters for throughput deltas. + prompt_tokens_computed: U64Counter, + generation_tokens: U64Counter, + prefix_cache_queries: U64Counter, + prefix_cache_hits: U64Counter, + external_prefix_cache_queries: U64Counter, + external_prefix_cache_hits: U64Counter, + num_preemptions: U64Counter, + spec_decode_num_drafts: U64Counter, + spec_decode_num_draft_tokens: U64Counter, + spec_decode_num_accepted_tokens: U64Counter, + estimated_flops_per_gpu: U64Counter, + estimated_read_bytes_per_gpu: U64Counter, + estimated_write_bytes_per_gpu: U64Counter, + log_stats: SchedulerLogStatsAccumulator, + + // Gauges for instantaneous scheduler state. + scheduler_running: U64Gauge, + scheduler_waiting: U64Gauge, + scheduler_deferred: U64Gauge, + kv_cache_usage: F64Gauge, +} + +/// Accumulated snapshot values from the last logging interval, used to compute +/// deltas. +#[derive(Default)] +struct CounterSnapshot { + prompt_tokens: u64, + generation_tokens: u64, + prefix_cache_queries: u64, + prefix_cache_hits: u64, + external_prefix_cache_queries: u64, + external_prefix_cache_hits: u64, + num_preemptions: u64, + spec_decode_num_drafts: u64, + spec_decode_num_draft_tokens: u64, + spec_decode_num_accepted_tokens: u64, + estimated_flops_per_gpu: u64, + estimated_read_bytes_per_gpu: u64, + estimated_write_bytes_per_gpu: u64, +} + +/// Derived spec-decoding values for one logging interval. +struct SpecDecodingLogStats { + mean_acceptance_length: f64, + accepted_throughput: f64, + draft_throughput: f64, + accepted_tokens: u64, + draft_tokens: u64, + per_position_acceptance_rates: Vec, + draft_acceptance_rate: f64, +} + +/// Derived MFU values for one logging interval. +struct MfuLogStats { + tflops_per_gpu: f64, + gbps_per_gpu: f64, +} + +/// Periodic stats logger that mirrors Python vLLM's `LoggingStatLogger`. +/// +/// Spawns a background task that logs throughput and scheduler state at a fixed +/// interval. When idle (both current and previous throughputs are zero), logs +/// at DEBUG level. When load drops to zero, emits one final INFO-level line +/// before going quiet. +pub(crate) struct StatsLogger { + _task: AbortOnDropHandle<()>, +} + +impl StatsLogger { + /// Start the background stats logging task. + pub(crate) fn start(model_name: String, engine_indices: Vec) -> Self { + let task = AbortOnDropHandle::new(tokio::spawn(async move { + run_stats_logger(model_name, engine_indices).await; + })); + Self { _task: task } + } +} + +/// Resolve and clone all metric handles once so the hot path is lock-free. +fn resolve_engine_metrics(model_name: &str, engine_indices: &[u32]) -> Vec { + let m = &METRICS; + engine_indices + .iter() + .copied() + .map(|engine| { + let el = EngineLabels { + model_name: model_name.to_string(), + engine, + }; + let pt = PromptTokenSourceLabels { + model_name: model_name.to_string(), + engine, + source: "local_compute", + }; + let deferred = WaitingReasonLabels { + model_name: model_name.to_string(), + engine, + reason: WAITING_REASON_DEFERRED, + }; + EngineMetrics { + // Use "local_compute" source for prompt throughput (excludes + // cached/transferred tokens), matching Python's + // `iteration_stats.prompt_token_stats.computed`. + prompt_tokens_computed: m.request.prompt_tokens_by_source.get_or_create_owned(&pt), + generation_tokens: m.request.generation_tokens.get_or_create_owned(&el), + prefix_cache_queries: m.scheduler.prefix_cache_queries.get_or_create_owned(&el), + prefix_cache_hits: m.scheduler.prefix_cache_hits.get_or_create_owned(&el), + external_prefix_cache_queries: m + .scheduler + .external_prefix_cache_queries + .get_or_create_owned(&el), + external_prefix_cache_hits: m + .scheduler + .external_prefix_cache_hits + .get_or_create_owned(&el), + num_preemptions: m.request.num_preemptions.get_or_create_owned(&el), + spec_decode_num_drafts: m.scheduler.spec_decode_num_drafts.get_or_create_owned(&el), + spec_decode_num_draft_tokens: m + .scheduler + .spec_decode_num_draft_tokens + .get_or_create_owned(&el), + spec_decode_num_accepted_tokens: m + .scheduler + .spec_decode_num_accepted_tokens + .get_or_create_owned(&el), + estimated_flops_per_gpu: m + .scheduler + .estimated_flops_per_gpu + .get_or_create_owned(&el), + estimated_read_bytes_per_gpu: m + .scheduler + .estimated_read_bytes_per_gpu + .get_or_create_owned(&el), + estimated_write_bytes_per_gpu: m + .scheduler + .estimated_write_bytes_per_gpu + .get_or_create_owned(&el), + log_stats: m.scheduler.log_stats.get_or_create_owned(&el), + scheduler_running: m.scheduler.scheduler_running.get_or_create_owned(&el), + scheduler_waiting: m.scheduler.scheduler_waiting.get_or_create_owned(&el), + scheduler_deferred: m + .scheduler + .scheduler_waiting_by_reason + .get_or_create_owned(&deferred), + kv_cache_usage: m.scheduler.kv_cache_usage.get_or_create_owned(&el), + } + }) + .collect() +} + +async fn run_stats_logger(model_name: String, engine_indices: Vec) { + let engines = resolve_engine_metrics(&model_name, &engine_indices); + + let mut interval = tokio::time::interval(LOG_STATS_INTERVAL); + interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); + // The first tick fires immediately; skip it so the first log is after one full + // interval. + interval.tick().await; + + let mut prev = read_counters(&engines); + let mut last_log_time = Instant::now(); + let mut last_prompt_throughput: f64 = 0.0; + let mut last_generation_throughput: f64 = 0.0; + + let mut msg = String::new(); + loop { + interval.tick().await; + + let now = Instant::now(); + let elapsed = now.duration_since(last_log_time).as_secs_f64(); + if elapsed <= 0.0 { + continue; + } + + let curr = read_counters(&engines); + let raw_log_stats = drain_scheduler_log_stats(&engines); + + let prompt_throughput = + curr.prompt_tokens.wrapping_sub(prev.prompt_tokens) as f64 / elapsed; + let generation_throughput = + curr.generation_tokens.wrapping_sub(prev.generation_tokens) as f64 / elapsed; + + // Idle = both current and previous throughputs are zero. + let is_idle = prompt_throughput == 0.0 + && generation_throughput == 0.0 + && last_prompt_throughput == 0.0 + && last_generation_throughput == 0.0; + + /// Emit one stats line at DEBUG while idle and INFO while active. + macro_rules! log_stats_line { + ($($arg:tt)*) => { + if is_idle { + debug!($($arg)*); + } else { + info!($($arg)*); + } + }; + } + + // Read scheduler gauges (aggregate across engines). + let (num_running, num_waiting, kv_cache_usage) = read_scheduler_gauges(&engines); + let num_deferred = read_deferred_waiting(&engines); + let delta_preemptions = curr.num_preemptions.wrapping_sub(prev.num_preemptions); + + // Compute prefix cache hit rate over this interval. + let delta_queries = curr.prefix_cache_queries.wrapping_sub(prev.prefix_cache_queries); + let delta_hits = curr.prefix_cache_hits.wrapping_sub(prev.prefix_cache_hits); + let prefix_cache_hit_rate = cache_hit_rate(delta_hits, delta_queries); + + let delta_external_queries = curr + .external_prefix_cache_queries + .wrapping_sub(prev.external_prefix_cache_queries); + let delta_external_hits = + curr.external_prefix_cache_hits.wrapping_sub(prev.external_prefix_cache_hits); + let external_prefix_cache_hit_rate = + cache_hit_rate(delta_external_hits, delta_external_queries); + let spec_decoding_log_stats = + spec_decoding_log_stats(&curr, &prev, elapsed, &raw_log_stats); + let mfu_log_stats = mfu_log_stats(&curr, &prev, elapsed, engines.len()); + + // Build the log line. + msg.clear(); + write!( + msg, + "Avg prompt tput: {prompt_throughput:.1} toks/s, \ + Avg generation tput: {generation_throughput:.1} toks/s, \ + Reqs Running: {num_running}, \ + Waiting: {num_waiting}" + ) + .unwrap(); + if num_deferred > 0 { + write!(msg, ", Deferred: {num_deferred} reqs").unwrap(); + } + if delta_preemptions > 0 { + write!(msg, ", Preemptions: {delta_preemptions}").unwrap(); + } + write!( + msg, + ", GPU KV cache used: {:.1}%, \ + Prefix cache hit rate: {prefix_cache_hit_rate:.1}%", + kv_cache_usage * 100.0, + ) + .unwrap(); + if delta_external_queries > 0 { + write!( + msg, + ", External prefix cache hit rate: {external_prefix_cache_hit_rate:.1}%" + ) + .unwrap(); + } + + log_stats_line!("{msg}"); + + if let Some(spec_stats) = spec_decoding_log_stats { + msg.clear(); + write!( + msg, + "SpecDecoding metrics: \ + Mean acceptance length: {:.2}, \ + Accepted throughput: {:.2} tokens/s, \ + Drafted throughput: {:.2} tokens/s, \ + Accepted: {} tokens, \ + Drafted: {} tokens", + spec_stats.mean_acceptance_length, + spec_stats.accepted_throughput, + spec_stats.draft_throughput, + spec_stats.accepted_tokens, + spec_stats.draft_tokens, + ) + .unwrap(); + if !spec_stats.per_position_acceptance_rates.is_empty() { + msg.push_str(", Per-position acceptance rate: "); + format_position_rates(&mut msg, &spec_stats.per_position_acceptance_rates); + } + write!( + msg, + ", Avg Draft acceptance rate: {:.1}%", + spec_stats.draft_acceptance_rate, + ) + .unwrap(); + log_stats_line!("{msg}"); + } + + // TODO: Decide on best way to surface CUDAGraph interval samples. + + if let Some(mfu_stats) = mfu_log_stats { + log_stats_line!( + "MFU: {:.1} TF/s/GPU {:.1} GB/s/GPU", + mfu_stats.tflops_per_gpu, + mfu_stats.gbps_per_gpu, + ); + } + + last_prompt_throughput = prompt_throughput; + last_generation_throughput = generation_throughput; + last_log_time = now; + prev = curr; + } +} + +/// Read the current cumulative counter values for throughput computation. +fn read_counters(engines: &[EngineMetrics]) -> CounterSnapshot { + let mut snap = CounterSnapshot::default(); + for e in engines { + snap.prompt_tokens += e.prompt_tokens_computed.get(); + snap.generation_tokens += e.generation_tokens.get(); + snap.prefix_cache_queries += e.prefix_cache_queries.get(); + snap.prefix_cache_hits += e.prefix_cache_hits.get(); + snap.external_prefix_cache_queries += e.external_prefix_cache_queries.get(); + snap.external_prefix_cache_hits += e.external_prefix_cache_hits.get(); + snap.num_preemptions += e.num_preemptions.get(); + snap.spec_decode_num_drafts += e.spec_decode_num_drafts.get(); + snap.spec_decode_num_draft_tokens += e.spec_decode_num_draft_tokens.get(); + snap.spec_decode_num_accepted_tokens += e.spec_decode_num_accepted_tokens.get(); + snap.estimated_flops_per_gpu += e.estimated_flops_per_gpu.get(); + snap.estimated_read_bytes_per_gpu += e.estimated_read_bytes_per_gpu.get(); + snap.estimated_write_bytes_per_gpu += e.estimated_write_bytes_per_gpu.get(); + } + snap +} + +/// Read the current scheduler gauge values, aggregated across engines. +fn read_scheduler_gauges(engines: &[EngineMetrics]) -> (u64, u64, f64) { + let mut num_running = 0u64; + let mut num_waiting = 0u64; + let mut kv_cache_usage_sum = 0.0f64; + + for e in engines { + num_running += e.scheduler_running.get(); + num_waiting += e.scheduler_waiting.get(); + kv_cache_usage_sum += e.kv_cache_usage.get(); + } + + let kv_cache_usage = if !engines.is_empty() { + kv_cache_usage_sum / engines.len() as f64 + } else { + 0.0 + }; + + (num_running, num_waiting, kv_cache_usage) +} + +/// Read deferred waiting requests across all engines. +fn read_deferred_waiting(engines: &[EngineMetrics]) -> u64 { + engines.iter().map(|e| e.scheduler_deferred.get()).sum() +} + +/// Return the cache hit rate as a percentage for a counter delta. +fn cache_hit_rate(hits: u64, queries: u64) -> f64 { + if queries > 0 { + hits as f64 / queries as f64 * 100.0 + } else { + 0.0 + } +} + +/// Compute aggregate spec-decoding stats for one logging interval. +fn spec_decoding_log_stats( + curr: &CounterSnapshot, + prev: &CounterSnapshot, + elapsed: f64, + raw_log_stats: &SchedulerLogStatsInterval, +) -> Option { + let num_drafts = curr.spec_decode_num_drafts.wrapping_sub(prev.spec_decode_num_drafts); + if num_drafts == 0 { + return None; + } + + let draft_tokens = curr + .spec_decode_num_draft_tokens + .wrapping_sub(prev.spec_decode_num_draft_tokens); + let accepted_tokens = curr + .spec_decode_num_accepted_tokens + .wrapping_sub(prev.spec_decode_num_accepted_tokens); + + let (accepted_throughput, draft_throughput) = if elapsed > 0.0 { + ( + accepted_tokens as f64 / elapsed, + draft_tokens as f64 / elapsed, + ) + } else { + (0.0, 0.0) + }; + let draft_acceptance_rate = if draft_tokens > 0 { + accepted_tokens as f64 / draft_tokens as f64 * 100.0 + } else { + f64::NAN + }; + let per_position_acceptance_rates = if raw_log_stats.spec_num_drafts > 0 { + raw_log_stats + .spec_accepted_tokens_per_pos + .iter() + .map(|accepted_tokens| *accepted_tokens as f64 / raw_log_stats.spec_num_drafts as f64) + .collect() + } else { + Vec::new() + }; + + Some(SpecDecodingLogStats { + mean_acceptance_length: 1.0 + accepted_tokens as f64 / num_drafts as f64, + accepted_throughput, + draft_throughput, + accepted_tokens, + draft_tokens, + per_position_acceptance_rates, + draft_acceptance_rate, + }) +} + +/// Compute average per-GPU MFU rates for one logging interval. +fn mfu_log_stats( + curr: &CounterSnapshot, + prev: &CounterSnapshot, + elapsed: f64, + engine_count: usize, +) -> Option { + let flops = curr.estimated_flops_per_gpu.wrapping_sub(prev.estimated_flops_per_gpu); + let read_bytes = curr + .estimated_read_bytes_per_gpu + .wrapping_sub(prev.estimated_read_bytes_per_gpu); + let write_bytes = curr + .estimated_write_bytes_per_gpu + .wrapping_sub(prev.estimated_write_bytes_per_gpu); + + if flops == 0 && read_bytes == 0 && write_bytes == 0 { + return None; + } + + let denominator = elapsed * engine_count.max(1) as f64; + let (tflops_per_gpu, gbps_per_gpu) = if denominator > 0.0 { + ( + flops as f64 / denominator / 1e12, + (read_bytes as f64 + write_bytes as f64) / denominator / 1e9, + ) + } else { + (0.0, 0.0) + }; + + Some(MfuLogStats { + tflops_per_gpu, + gbps_per_gpu, + }) +} + +/// Drain raw scheduler DTO stats for the configured model and engines. +fn drain_scheduler_log_stats(engines: &[EngineMetrics]) -> SchedulerLogStatsInterval { + let mut interval = SchedulerLogStatsInterval::default(); + for engine in engines { + interval.merge(engine.log_stats.drain()); + } + interval +} + +/// Append spec-decoding per-position acceptance rates like Python's logger. +fn format_position_rates(output: &mut String, rates: &[f64]) { + for (position, rate) in rates.iter().enumerate() { + if position > 0 { + output.push_str(", "); + } + write!(output, "{rate:.3}").unwrap(); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn cache_hit_rate_returns_percent_for_non_empty_queries() { + assert_eq!(cache_hit_rate(25, 100), 25.0); + assert_eq!(cache_hit_rate(0, 0), 0.0); + } + + #[test] + fn spec_decoding_log_stats_uses_interval_deltas() { + let raw_log_stats = SchedulerLogStatsInterval { + spec_num_drafts: 4, + spec_accepted_tokens_per_pos: vec![4, 2, 1], + ..Default::default() + }; + let prev = CounterSnapshot { + spec_decode_num_drafts: 10, + spec_decode_num_draft_tokens: 100, + spec_decode_num_accepted_tokens: 40, + ..Default::default() + }; + let curr = CounterSnapshot { + spec_decode_num_drafts: 14, + spec_decode_num_draft_tokens: 120, + spec_decode_num_accepted_tokens: 52, + ..Default::default() + }; + + let stats = spec_decoding_log_stats(&curr, &prev, 2.0, &raw_log_stats).unwrap(); + + assert_eq!(stats.mean_acceptance_length, 4.0); + assert_eq!(stats.accepted_throughput, 6.0); + assert_eq!(stats.draft_throughput, 10.0); + assert_eq!(stats.accepted_tokens, 12); + assert_eq!(stats.draft_tokens, 20); + assert_eq!(stats.per_position_acceptance_rates, vec![1.0, 0.5, 0.25]); + assert_eq!(stats.draft_acceptance_rate, 60.0); + } + + #[test] + fn mfu_log_stats_averages_per_gpu_across_engines() { + let prev = CounterSnapshot { + estimated_flops_per_gpu: 10, + estimated_read_bytes_per_gpu: 10, + estimated_write_bytes_per_gpu: 10, + ..Default::default() + }; + let curr = CounterSnapshot { + estimated_flops_per_gpu: 4_000_000_000_010, + estimated_read_bytes_per_gpu: 2_000_000_010, + estimated_write_bytes_per_gpu: 2_000_000_010, + ..Default::default() + }; + + let stats = mfu_log_stats(&curr, &prev, 2.0, 2).unwrap(); + + assert_eq!(stats.tflops_per_gpu, 1.0); + assert_eq!(stats.gbps_per_gpu, 1.0); + } + + #[test] + fn format_position_rates_uses_three_decimal_places() { + let mut output = String::new(); + + format_position_rates(&mut output, &[1.0, 0.5, 0.25]); + + assert_eq!(output, "1.000, 0.500, 0.250"); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/llm/src/output.rs b/asian_ai/performance/vLLM/rust/src/llm/src/output.rs new file mode 100644 index 0000000000000000000000000000000000000000..4e573bb6c48da81a82092ee56cf129b56e4a69a3 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/llm/src/output.rs @@ -0,0 +1,389 @@ +use std::future::Future; +use std::pin::Pin; +use std::sync::Arc; +use std::task::{Context, Poll, ready}; + +use enum_as_inner::EnumAsInner; +use futures::stream::FusedStream; +use futures::{Stream, StreamExt as _, pin_mut}; +use serde::{Deserialize, Serialize}; +use vllm_engine_core_client::protocol::logprobs::Logprobs; +use vllm_engine_core_client::protocol::output::{EngineCoreFinishReason, StopReason}; +use vllm_engine_core_client::{AbortCause, EngineCoreOutputStream}; + +use crate::error::Result; +use crate::inflight::RequestGuard; +use crate::request_metrics::{RequestMetricsTracker, current_unix_timestamp_secs}; + +/// Token usage metadata for one request. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct TokenUsage { + /// Number of prompt tokens sent to the engine. + pub prompt_token_count: usize, + /// Number of output tokens generated. + pub output_token_count: usize, + /// Number of prompt tokens served from cache. + pub cached_token_count: usize, +} + +/// Final raw token output plus terminal stream metadata. +#[derive(Debug, Clone, PartialEq)] +pub struct CollectedGenerateOutput { + pub request_id: String, + pub prompt_token_ids: Vec, + pub prompt_logprobs: Option, + pub token_ids: Vec, + pub logprobs: Option, + pub finish_reason: FinishReason, + pub usage: TokenUsage, + /// Connector-specific KV transfer parameters for disaggregated serving. + pub kv_transfer_params: Option, +} + +/// Prompt-scoped metadata emitted only once on the first [`GenerateOutput`] for +/// one request. +#[derive(Debug, Clone, PartialEq)] +pub struct GeneratePromptInfo { + /// Original prompt token IDs for this request. + pub prompt_token_ids: Arc<[u32]>, + /// Prompt logprobs returned by engine-core for scored prompt positions, + /// when requested. + pub prompt_logprobs: Option, +} + +/// The reason a request finished. +/// +/// This is a higher-level abstraction over engine-core's finish and stop +/// reasons. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, EnumAsInner)] +pub enum FinishReason { + /// Generation stopped for a stop string, stop token, or EOS. + /// + /// The inner stop reason is present for explicit stop strings or stop + /// tokens, and absent for EOS-driven stops. + Stop(Option), + /// `max_tokens` or `max_model_len` was reached. + Length, + /// The request was aborted by the client. + Abort, + /// A retryable request-level internal error occurred. + Error, + /// A repetitive token pattern was detected. + Repetition(Option), +} + +impl FinishReason { + /// Construct a stop finish reason caused by EOS rather than an explicit + /// stop string/token. + pub fn stop_eos() -> Self { + Self::Stop(None) + } + + /// Returns a human-readable string for this finish reason, used for metrics + /// and reporting. + pub fn as_str(&self) -> &'static str { + match self { + Self::Stop(_) => "stop", + Self::Length => "length", + Self::Abort => "abort", + Self::Error => "error", + Self::Repetition(_) => "repetition", + } + } + + /// If this is a stop finish reason, returns the inner stop reason if it + /// exists. + pub fn as_stop_reason(&self) -> Option<&StopReason> { + match self { + Self::Stop(stop_reason) => stop_reason.as_ref(), + Self::Repetition(stop_reason) => stop_reason.as_ref(), + _ => None, + } + } + + /// If this is a stop finish reason, returns the inner stop reason if it + /// exists. + pub fn into_stop_reason(self) -> Option { + match self { + Self::Stop(stop_reason) => stop_reason, + Self::Repetition(stop_reason) => stop_reason, + _ => None, + } + } +} + +fn finish_reason_from_engine( + finish_reason: Option, + stop_reason: Option, +) -> Option { + finish_reason.map(|reason| match reason { + EngineCoreFinishReason::Stop => FinishReason::Stop(stop_reason), + EngineCoreFinishReason::Length => FinishReason::Length, + EngineCoreFinishReason::Abort => FinishReason::Abort, + EngineCoreFinishReason::Error => FinishReason::Error, + EngineCoreFinishReason::Repetition => FinishReason::Repetition(stop_reason), + }) +} + +/// Token and logprob output item returned by [`GenerateOutputStream`]. +/// +/// Original Python output reference: +/// +#[derive(Debug, Clone, PartialEq)] +pub struct GenerateOutput { + /// Unique ID of the request that produced this output. + pub request_id: String, + /// One-time prompt metadata emitted only on the first output for this + /// request. + pub prompt_info: Option, + /// Newly produced token IDs for this step. + pub token_ids: Vec, + /// Sample logprobs for the generated positions in this step. + pub logprobs: Option, + /// Terminal finish reason, when this is the final output for the request. + pub finish_reason: Option, + /// Number of prompt tokens served from cache, when reported by prefill stats. + pub cached_token_count: usize, + /// Connector-specific KV transfer parameters for disaggregated serving. + pub kv_transfer_params: Option, +} + +impl GenerateOutput { + /// Returns the prompt token IDs when this output carries + /// [`GeneratePromptInfo`]. + /// + /// Only the first output for a request can return `Some`; all later outputs + /// return `None`. + pub fn prompt_token_ids(&self) -> Option<&Arc<[u32]>> { + self.prompt_info.as_ref().map(|info| &info.prompt_token_ids) + } + + /// Returns the prompt logprobs when this output carries + /// [`GeneratePromptInfo`]. + /// + /// Only the first output for a request can return `Some`; all later outputs + /// return `None`. + pub fn prompt_logprobs(&self) -> Option<&Logprobs> { + self.prompt_info.as_ref().and_then(|info| info.prompt_logprobs.as_ref()) + } + + /// Returns whether this output is terminal for the request. + pub fn finished(&self) -> bool { + self.finish_reason.is_some() + } +} + +#[cfg(any(test, feature = "test-util"))] +impl GenerateOutput { + /// Build a [`GenerateOutput`] for tests. + pub fn for_test( + prompt_token_ids: Option>, + token_ids: Vec, + finish_reason: Option, + ) -> Self { + Self { + request_id: String::new(), + prompt_info: prompt_token_ids.map(|ids| GeneratePromptInfo { + prompt_token_ids: ids, + prompt_logprobs: None, + }), + token_ids, + logprobs: None, + finish_reason, + cached_token_count: 0, + kv_transfer_params: None, + } + } +} + +/// Stream of per-request generate outputs for one request. +/// +/// - A normal termination of the stream represents a clean completion of the +/// request, including a client-initiated abort, which yields a final output +/// with `finish_reason = Abort` before the stream ends. +/// - For errors or unexpected engine-side closes, the stream terminates with an error. +pub struct GenerateOutputStream { + pending_prompt_info: Option, + raw_stream: EngineCoreOutputStream, + request_metrics: RequestMetricsTracker, + /// Removes this request's external→internal tracking edge on drop. Held for + /// its `Drop` side effect only; never read directly. + _request_guard: RequestGuard, +} + +impl GenerateOutputStream { + /// Create a new generate output stream by adapting one raw engine-core + /// output stream. + pub(crate) fn new( + prompt_token_ids: Arc<[u32]>, + raw_stream: EngineCoreOutputStream, + request_metrics: RequestMetricsTracker, + request_guard: RequestGuard, + ) -> Self { + Self { + pending_prompt_info: Some(GeneratePromptInfo { + prompt_token_ids, + prompt_logprobs: None, + }), + raw_stream, + request_metrics, + _request_guard: request_guard, + } + } + + /// Return the internal engine request ID bound to this stream. + pub fn request_id(&self) -> &str { + self.raw_stream.request_id() + } +} + +impl Stream for GenerateOutputStream { + type Item = Result; + + fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + let raw = match ready!(Pin::new(&mut self.raw_stream).poll_next(cx)) { + Some(Ok(raw)) => raw, + Some(Err(error)) => return Poll::Ready(Some(Err(error.into()))), + None => return Poll::Ready(None), + }; + + let received_at = current_unix_timestamp_secs(); + self.request_metrics.observe_output( + raw.engine_index, + raw.timestamp, + received_at, + &raw.output, + ); + + let raw = raw.output; + + // Populate the one-time prompt info on the first output. + if let Some(info) = &mut self.pending_prompt_info + && info.prompt_logprobs.is_none() + { + info.prompt_logprobs = + raw.new_prompt_logprobs_tensors.map(|value| value.into_direct().unwrap()); + } + + let logprobs = raw.new_logprobs.map(|value| value.into_direct().unwrap()); + let cached_token_count = raw + .prefill_stats + .as_ref() + .map(|stats| stats.num_cached_tokens as usize) + .unwrap_or(0); + + let finish_reason = finish_reason_from_engine(raw.finish_reason, raw.stop_reason); + if let Some(finish_reason) = finish_reason.as_ref() { + self.request_metrics.record_finished(received_at, finish_reason.clone()); + } + + let output = GenerateOutput { + request_id: raw.request_id, + prompt_info: self.pending_prompt_info.take(), + token_ids: raw.new_token_ids, + logprobs, + finish_reason, + cached_token_count, + kv_transfer_params: raw.kv_transfer_params, + }; + + Poll::Ready(Some(Ok(output))) + } +} + +impl FusedStream for GenerateOutputStream { + fn is_terminated(&self) -> bool { + self.raw_stream.is_terminated() + } +} + +impl Drop for GenerateOutputStream { + fn drop(&mut self) { + if self.raw_stream.is_terminated() { + // Already terminated cleanly, no need to record abort metrics. + return; + } + + // If the user or the upper layer drops a live generate stream, + // `EngineCoreOutputStream::Drop` will trigger an engine-side abort. Record the + // matching terminal request metrics here so frontend-driven aborts are still + // visible as `finished_reason=...` instead of disappearing from observability + // entirely. + let finish_reason = match AbortCause::current() { + AbortCause::DroppedStream => FinishReason::Abort, + AbortCause::StopStringMatched => FinishReason::Stop(None), + }; + + self.request_metrics + .record_finished(current_unix_timestamp_secs(), finish_reason); + } +} + +#[allow(clippy::manual_async_fn, reason = "specify `Send` bound")] +#[easy_ext::ext(GenerateOutputStreamExt)] +impl> + Send> T { + /// Collect the raw generate stream to completion and return the final token + /// output. + pub fn collect_output(self) -> impl Future> + Send { + async move { + let stream = self; + pin_mut!(stream); + let mut prompt_token_ids = None; + let mut prompt_logprobs = None; + let mut cached_token_count = 0; + let mut collected: Option = None; + + while let Some(output) = stream.next().await.transpose()? { + cached_token_count = cached_token_count.max(output.cached_token_count); + if let Some(info) = output.prompt_info { + if prompt_token_ids.is_none() { + prompt_token_ids = Some(info.prompt_token_ids.to_vec()); + } + if prompt_logprobs.is_none() { + prompt_logprobs = info.prompt_logprobs; + } + } + + if let Some(existing) = collected.as_mut() { + existing.token_ids.extend(output.token_ids); + if let Some(step_logprobs) = output.logprobs { + if let Some(collected_logprobs) = existing.logprobs.as_mut() { + collected_logprobs.positions.extend(step_logprobs.positions); + } else { + existing.logprobs = Some(step_logprobs); + } + } + } else { + collected = Some(CollectedGenerateOutput { + request_id: output.request_id, + prompt_token_ids: prompt_token_ids.take().unwrap_or_default(), + prompt_logprobs: prompt_logprobs.take(), + token_ids: output.token_ids, + logprobs: output.logprobs, + finish_reason: FinishReason::Error, + usage: TokenUsage { + prompt_token_count: prompt_token_ids.as_ref().map_or(0, Vec::len), + output_token_count: 0, + cached_token_count, + }, + kv_transfer_params: None, + }); + } + + if let Some(finish_reason) = output.finish_reason { + let mut collected = collected.expect("terminal output must exist"); + collected.finish_reason = finish_reason; + collected.usage = TokenUsage { + prompt_token_count: collected.prompt_token_ids.len(), + output_token_count: collected.token_ids.len(), + cached_token_count, + }; + collected.kv_transfer_params = output.kv_transfer_params; + return Ok(collected); + } + } + + unreachable!("generate stream should yield an error instead of closing early") + } + } +} diff --git a/asian_ai/performance/vLLM/rust/src/llm/src/request.rs b/asian_ai/performance/vLLM/rust/src/llm/src/request.rs new file mode 100644 index 0000000000000000000000000000000000000000..6cb83e6e58d972a711a87b4d2551ffa1e5aa9c3b --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/llm/src/request.rs @@ -0,0 +1,232 @@ +use std::collections::BTreeMap; +use std::time::{SystemTime, UNIX_EPOCH}; + +use uuid::Uuid; +use vllm_engine_core_client::protocol::lora::LoraRequest; +use vllm_engine_core_client::protocol::multimodal::MmFeatures; +use vllm_engine_core_client::protocol::request::{EngineCoreRequest, ReasoningParserKwargs}; +use vllm_engine_core_client::protocol::sampling::EngineCoreSamplingParams; + +use crate::error::{Error, Result}; + +/// Tokenized decoder-only generate request accepted by [`crate::Llm`]. +/// +/// This is the first-stage Rust subset of the inputs that eventually flow into +/// Python `AsyncLLM.generate()`. The boundary is intentionally above +/// [`EngineCoreRequest`], but below higher-level text and multimodal +/// preprocessing. +/// +/// Original Python API reference: +/// +#[derive(Debug, Clone, PartialEq)] +pub struct GenerateRequest { + /// Unique ID of the request. + pub request_id: String, + /// Token IDs of the prompt. + pub prompt_token_ids: Vec, + /// Sampling parameters forwarded to engine-core. + pub sampling_params: EngineCoreSamplingParams, + /// Optional multimodal features already prepared by `vllm-chat`. + pub mm_features: Option, + /// Unix timestamp, in seconds, when this request arrived at the frontend. + /// + /// When omitted, the Rust frontend fills it immediately before sending the + /// request to engine-core, matching Python's default arrival-time behavior. + pub arrival_time: Option, + /// Optional salt used to partition prefix-cache entries for this request. + pub cache_salt: Option, + /// Optional tracing headers to forward to engine-core and downstream + /// observability hooks. + pub trace_headers: Option>, + /// Request scheduling priority. Lower values are scheduled earlier. + pub priority: i32, + /// Optional data-parallel rank override for routing this request. + pub data_parallel_rank: Option, + /// Optional reasoning-parser kwargs forwarded to engine-side structured + /// output logic. + pub reasoning_parser_kwargs: Option, + /// Optional LoRA adapter request applied to this generation. + pub lora_request: Option, +} + +#[derive(Debug)] +pub(crate) struct PreparedGenerateRequest { + pub engine_request: EngineCoreRequest, +} + +impl GenerateRequest { + /// Validate and lower this request into the raw engine-core request format. + pub(crate) fn prepare(self, randomize_request_id: bool) -> Result { + if self.prompt_token_ids.is_empty() { + return Err(Error::EmptyPromptTokenIds { + request_id: self.request_id, + }); + } + let GenerateRequest { + request_id, + prompt_token_ids, + sampling_params, + mm_features, + arrival_time, + cache_salt, + trace_headers, + priority, + data_parallel_rank, + reasoning_parser_kwargs, + lora_request, + } = self; + + let external_request_id = request_id; + let engine_request_id = if randomize_request_id { + let random_suffix = Uuid::new_v4().simple().to_string(); + format!("{external_request_id}-{}", &random_suffix[..8]) + } else { + external_request_id.clone() + }; + Ok(PreparedGenerateRequest { + engine_request: EngineCoreRequest { + request_id: engine_request_id, + prompt_token_ids: Some(prompt_token_ids), + mm_features, + sampling_params: Some(sampling_params), + pooling_params: None, + arrival_time: arrival_time.unwrap_or_else(current_unix_timestamp_secs), + lora_request, + cache_salt, + data_parallel_rank, + prompt_embeds: None, + prompt_is_token_ids: None, + client_index: 0, + current_wave: 0, + priority, + trace_headers, + resumable: false, + external_req_id: Some(external_request_id), + // Rust parser doesn't expose this information, leave it unset and let the + // reasoning logic in engine-sided structured output manager handle it. + reasoning_ended: None, + reasoning_parser_kwargs, + abort_immediately: false, + }, + }) + } +} + +impl PreparedGenerateRequest { + /// Return the original prompt token IDs copied into the raw engine request. + pub fn prompt_token_ids(&self) -> &[u32] { + self.engine_request + .prompt_token_ids + .as_ref() + .expect("prepared request must have prompt token ids") + } +} + +fn current_unix_timestamp_secs() -> f64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("system clock is before unix epoch") + .as_secs_f64() +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + + use vllm_engine_core_client::protocol::request::ReasoningParserKwargs; + use vllm_engine_core_client::protocol::sampling::EngineCoreSamplingParams; + + use super::GenerateRequest; + use crate::error::Error; + + fn sample_request() -> GenerateRequest { + GenerateRequest { + request_id: "req-1".to_string(), + prompt_token_ids: vec![11, 22, 33], + sampling_params: EngineCoreSamplingParams::for_test(), + mm_features: None, + arrival_time: Some(42.5), + cache_salt: Some("salt".to_string()), + trace_headers: Some(BTreeMap::from([( + "x-trace-id".to_string(), + "abc".to_string(), + )])), + priority: 3, + data_parallel_rank: Some(2), + reasoning_parser_kwargs: Some(ReasoningParserKwargs { + chat_template_kwargs: [( + "chat_template_kwargs".to_string(), + serde_json::json!({ + "enable_thinking": true, + }), + )] + .into(), + }), + lora_request: None, + } + } + + #[test] + fn prepare_builds_engine_core_request() { + let prepared = sample_request().prepare(true).unwrap(); + + assert_eq!(prepared.prompt_token_ids(), &[11, 22, 33]); + + let request = prepared.engine_request; + assert_eq!(request.external_req_id.as_deref(), Some("req-1")); + assert!(request.request_id.starts_with("req-1-")); + assert_ne!(request.request_id, "req-1"); + assert_eq!(request.prompt_token_ids.as_deref(), Some(&[11, 22, 33][..])); + assert_eq!(request.arrival_time, 42.5); + assert_eq!(request.cache_salt.as_deref(), Some("salt")); + assert_eq!(request.data_parallel_rank, Some(2)); + assert_eq!( + request.trace_headers, + Some(BTreeMap::from([( + "x-trace-id".to_string(), + "abc".to_string(), + )])) + ); + assert_eq!(request.reasoning_ended, None); + assert_eq!( + request + .reasoning_parser_kwargs + .as_ref() + .and_then(|kwargs| kwargs.chat_template_kwargs.get("chat_template_kwargs")), + Some(&serde_json::json!({ + "enable_thinking": true + })) + ); + } + + #[test] + fn prepare_rejects_empty_prompt_tokens() { + let mut request = sample_request(); + request.prompt_token_ids.clear(); + + let error = request.prepare(true).unwrap_err(); + assert!(matches!( + error, + Error::EmptyPromptTokenIds { request_id } if request_id == "req-1" + )); + } + + #[test] + fn prepare_can_preserve_external_request_id() { + let prepared = sample_request().prepare(false).unwrap(); + + let request = prepared.engine_request; + assert_eq!(request.external_req_id.as_deref(), Some("req-1")); + assert_eq!(request.request_id, "req-1"); + } + + #[test] + fn prepare_forwards_multimodal_features() { + let mut request = sample_request(); + request.mm_features = Some(Vec::new()); + + let prepared = request.prepare(false).unwrap(); + + assert_eq!(prepared.engine_request.mm_features, Some(Vec::new())); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/llm/src/request_metrics.rs b/asian_ai/performance/vLLM/rust/src/llm/src/request_metrics.rs new file mode 100644 index 0000000000000000000000000000000000000000..5cc8effca341319ebac454b8761272e4020fe2f7 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/llm/src/request_metrics.rs @@ -0,0 +1,399 @@ +use std::time::{SystemTime, UNIX_EPOCH}; + +use vllm_engine_core_client::protocol::output::{ + EngineCoreEvent, EngineCoreEventType, EngineCoreOutput, +}; +use vllm_engine_core_client::protocol::stats::PrefillStats; +use vllm_metrics::{ + EngineLabels, FinishedReasonLabels, METRICS, PromptTokenSourceLabels, RequestMetrics, +}; + +use crate::FinishReason; + +fn metrics() -> &'static RequestMetrics { + &METRICS.request +} + +const PROMPT_TOKEN_SOURCE_LOCAL_COMPUTE: &str = "local_compute"; +const PROMPT_TOKEN_SOURCE_LOCAL_CACHE_HIT: &str = "local_cache_hit"; +const PROMPT_TOKEN_SOURCE_EXTERNAL_KV_TRANSFER: &str = "external_kv_transfer"; + +/// Request-scoped metrics state tracked across streamed engine-core updates. +/// +/// This is the Rust-side counterpart of the Python frontend's request-lifecycle +/// bookkeeping, centered on `RequestStateStats` and the per-output/per-finished +/// update flow. +/// +/// Original Python definitions: +/// +/// +/// Original Python update flow: +/// +#[derive(Debug, Clone)] +pub(crate) struct RequestMetricsTracker { + model_name: String, + arrival_time: f64, + prompt_len: u32, + max_tokens_param: Option, + n_param: u32, + is_prefilling: bool, + queued_ts: f64, + scheduled_ts: f64, + first_token_ts: f64, + last_token_ts: f64, + first_token_latency: f64, + num_generation_tokens: u32, + latest_num_cached_tokens: u32, + last_seen_engine_index: u32, +} + +impl RequestMetricsTracker { + /// Create the per-request tracker from the normalized `llm`-layer request + /// context. + pub(crate) fn new( + model_name: String, + arrival_time: f64, + prompt_len: u32, + max_tokens_param: Option, + n_param: u32, + ) -> Self { + Self { + model_name, + arrival_time, + prompt_len, + max_tokens_param, + n_param, + is_prefilling: true, + queued_ts: 0.0, + scheduled_ts: 0.0, + first_token_ts: 0.0, + last_token_ts: 0.0, + first_token_latency: 0.0, + num_generation_tokens: 0, + latest_num_cached_tokens: 0, + last_seen_engine_index: 0, + } + } + + /// Update request-lifecycle state from one engine-core output item. + /// + /// Original Python stats logic: + /// + pub(crate) fn observe_output( + &mut self, + engine_index: u32, + batch_timestamp: f64, + received_at: f64, + output: &EngineCoreOutput, + ) { + self.last_seen_engine_index = engine_index; + if let Some(prefill_stats) = &output.prefill_stats { + self.latest_num_cached_tokens = prefill_stats.num_cached_tokens; + } + self.num_generation_tokens += output.new_token_ids.len() as u32; + metrics() + .generation_tokens + .get_or_create(&engine_labels(&self.model_name, engine_index)) + .inc_by(output.new_token_ids.len() as u64); + + if let Some(events) = &output.events { + self.observe_events(engine_index, events); + } + + // Only outputs that actually carry tokens drive token-timing metrics. + // A terminal output with no new tokens (e.g. the synthesized abort + // output) must not log a stray time-to-first-token or inter-token + // sample. + if !output.new_token_ids.is_empty() { + if self.is_prefilling { + if let Some(prefill_stats) = &output.prefill_stats { + record_prompt_tokens(&self.model_name, engine_index, prefill_stats); + } + self.first_token_latency = received_at - self.arrival_time; + observe_time_to_first_token_seconds( + &self.model_name, + engine_index, + self.first_token_latency, + ); + self.first_token_ts = batch_timestamp; + self.is_prefilling = false; + } else if self.last_token_ts > 0.0 { + observe_inter_token_latency_seconds( + &self.model_name, + engine_index, + batch_timestamp - self.last_token_ts, + ); + } + + self.last_token_ts = batch_timestamp; + } + } + + /// Emit the terminal request metrics once a finished output has been + /// observed. + /// + /// Original Python finished-request stats: + /// + pub(crate) fn record_finished(&self, received_at: f64, finish_reason: FinishReason) { + let labels = engine_labels(&self.model_name, self.last_seen_engine_index); + let prefill_kv_computed_tokens = + self.prompt_len.saturating_sub(self.latest_num_cached_tokens); + let e2e_latency_seconds = received_at - self.arrival_time; + let queue_time_seconds = diff_or_zero(self.scheduled_ts, self.queued_ts); + let prefill_time_seconds = diff_or_zero(self.first_token_ts, self.scheduled_ts); + let decode_time_seconds = diff_or_zero(self.last_token_ts, self.first_token_ts); + let inference_time_seconds = diff_or_zero(self.last_token_ts, self.scheduled_ts); + let time_per_output_token_seconds = if self.num_generation_tokens > 1 { + diff_or_zero(self.last_token_ts, self.first_token_ts) + / (self.num_generation_tokens - 1) as f64 + } else { + 0.0 + }; + + record_request_success(&self.model_name, self.last_seen_engine_index, finish_reason); + metrics() + .request_prompt_tokens + .get_or_create(&labels) + .observe(self.prompt_len as f64); + metrics() + .request_generation_tokens + .get_or_create(&labels) + .observe(self.num_generation_tokens as f64); + metrics() + .request_max_num_generation_tokens + .get_or_create(&labels) + .observe(self.num_generation_tokens as f64); + if let Some(max_tokens_param) = self.max_tokens_param { + metrics() + .request_params_max_tokens + .get_or_create(&labels) + .observe(max_tokens_param as f64); + } + metrics().request_params_n.get_or_create(&labels).observe(self.n_param as f64); + metrics() + .request_prefill_kv_computed_tokens + .get_or_create(&labels) + .observe(prefill_kv_computed_tokens as f64); + metrics() + .e2e_request_latency_seconds + .get_or_create(&labels) + .observe(e2e_latency_seconds); + metrics() + .request_queue_time_seconds + .get_or_create(&labels) + .observe(queue_time_seconds); + metrics() + .request_prefill_time_seconds + .get_or_create(&labels) + .observe(prefill_time_seconds); + metrics() + .request_decode_time_seconds + .get_or_create(&labels) + .observe(decode_time_seconds); + metrics() + .request_inference_time_seconds + .get_or_create(&labels) + .observe(inference_time_seconds); + metrics() + .request_time_per_output_token_seconds + .get_or_create(&labels) + .observe(time_per_output_token_seconds); + } + + fn observe_events(&mut self, engine_index: u32, events: &[EngineCoreEvent]) { + for event in events { + match event.r#type { + EngineCoreEventType::Queued => { + self.queued_ts = event.timestamp; + } + EngineCoreEventType::Scheduled => { + if self.scheduled_ts == 0.0 { + self.scheduled_ts = event.timestamp; + } + } + EngineCoreEventType::Preempted => { + metrics() + .num_preemptions + .get_or_create(&engine_labels(&self.model_name, engine_index)) + .inc(); + } + } + } + } +} + +fn engine_labels(model_name: &str, engine: u32) -> EngineLabels { + EngineLabels { + model_name: model_name.to_string(), + engine, + } +} + +fn observe_time_to_first_token_seconds(model_name: &str, engine: u32, seconds: f64) { + metrics() + .time_to_first_token_seconds + .get_or_create(&engine_labels(model_name, engine)) + .observe(seconds); +} + +fn observe_inter_token_latency_seconds(model_name: &str, engine: u32, seconds: f64) { + metrics() + .inter_token_latency_seconds + .get_or_create(&engine_labels(model_name, engine)) + .observe(seconds); +} + +fn record_request_success(model_name: &str, engine: u32, finish_reason: FinishReason) { + metrics() + .request_success + .get_or_create(&FinishedReasonLabels { + model_name: model_name.to_string(), + engine, + finished_reason: finish_reason.as_str(), + }) + .inc(); +} + +fn prompt_token_source_labels( + model_name: &str, + engine: u32, + source: &'static str, +) -> PromptTokenSourceLabels { + PromptTokenSourceLabels { + model_name: model_name.to_string(), + engine, + source, + } +} + +fn record_prompt_tokens(model_name: &str, engine: u32, prefill_stats: &PrefillStats) { + let computed = prefill_stats.num_computed_tokens as u64; + let local_cache_hit = prefill_stats.num_local_cached_tokens as u64; + let external_kv_transfer = prefill_stats.num_external_cached_tokens as u64; + + metrics() + .prompt_tokens + .get_or_create(&engine_labels(model_name, engine)) + .inc_by(prefill_stats.num_prompt_tokens as u64); + metrics() + .prompt_tokens_by_source + .get_or_create(&prompt_token_source_labels( + model_name, + engine, + PROMPT_TOKEN_SOURCE_LOCAL_COMPUTE, + )) + .inc_by(computed); + metrics() + .prompt_tokens_by_source + .get_or_create(&prompt_token_source_labels( + model_name, + engine, + PROMPT_TOKEN_SOURCE_LOCAL_CACHE_HIT, + )) + .inc_by(local_cache_hit); + metrics() + .prompt_tokens_by_source + .get_or_create(&prompt_token_source_labels( + model_name, + engine, + PROMPT_TOKEN_SOURCE_EXTERNAL_KV_TRANSFER, + )) + .inc_by(external_kv_transfer); + metrics() + .prompt_tokens_cached + .get_or_create(&engine_labels(model_name, engine)) + .inc_by(prefill_stats.num_cached_tokens as u64); +} + +fn diff_or_zero(end: f64, start: f64) -> f64 { + if end > 0.0 && start > 0.0 && end >= start { + end - start + } else { + 0.0 + } +} + +/// Return the current wall-clock time in seconds since the Unix epoch. +/// +/// This is used for frontend-side latency measurements such as TTFT and E2E, +/// matching the Python frontend's use of wall-clock request arrival/iteration +/// timestamps rather than engine-core's monotonic scheduler timestamps. +/// +/// Original Python request timestamp source: +/// +pub(crate) fn current_unix_timestamp_secs() -> f64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("system clock is before unix epoch") + .as_secs_f64() +} + +#[cfg(test)] +mod tests { + use vllm_engine_core_client::protocol::output::{EngineCoreEvent, EngineCoreEventType}; + use vllm_engine_core_client::protocol::stats::PrefillStats; + + use super::{RequestMetricsTracker, diff_or_zero}; + + #[test] + fn tracker_updates_timing_state_across_prefill_decode_and_finish() { + let mut tracker = RequestMetricsTracker::new("model".to_string(), 100.0, 64, Some(128), 1); + + tracker.observe_output( + 2, + 10.0, + 100.2, + &vllm_engine_core_client::protocol::output::EngineCoreOutput { + request_id: "req-1".to_string(), + new_token_ids: vec![1], + finish_reason: None, + events: Some(vec![ + EngineCoreEvent { + r#type: EngineCoreEventType::Queued, + timestamp: 8.0, + }, + EngineCoreEvent { + r#type: EngineCoreEventType::Scheduled, + timestamp: 9.0, + }, + ]), + prefill_stats: Some(PrefillStats { + num_prompt_tokens: 64, + num_computed_tokens: 60, + num_cached_tokens: 4, + num_local_cached_tokens: 4, + num_external_cached_tokens: 0, + }), + ..Default::default() + }, + ); + tracker.observe_output( + 2, + 11.5, + 100.4, + &vllm_engine_core_client::protocol::output::EngineCoreOutput { + request_id: "req-1".to_string(), + new_token_ids: vec![2, 3], + finish_reason: None, + events: Some(vec![EngineCoreEvent { + r#type: EngineCoreEventType::Preempted, + timestamp: 10.5, + }]), + ..Default::default() + }, + ); + + assert!(!tracker.is_prefilling); + assert_eq!(tracker.last_seen_engine_index, 2); + assert_eq!(tracker.num_generation_tokens, 3); + assert_eq!(tracker.queued_ts, 8.0); + assert_eq!(tracker.scheduled_ts, 9.0); + assert_eq!(tracker.first_token_ts, 10.0); + assert_eq!(tracker.last_token_ts, 11.5); + assert!((tracker.first_token_latency - 0.2).abs() < 1e-9); + assert_eq!( + diff_or_zero(tracker.last_token_ts, tracker.first_token_ts), + 1.5 + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/llm/tests/generate.rs b/asian_ai/performance/vLLM/rust/src/llm/tests/generate.rs new file mode 100644 index 0000000000000000000000000000000000000000..c9bd69c28453cfcae86c108fa0b257b61f1bd693 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/llm/tests/generate.rs @@ -0,0 +1,902 @@ +use std::collections::BTreeSet; +use std::sync::Once; +use std::time::Duration; + +use futures::StreamExt as _; +use tokio::time::timeout; +use tracing_subscriber::EnvFilter; +use uuid::Uuid; +use vllm_engine_core_client::protocol::logprobs::{ + Logprobs, MaybeWireLogprobs, PositionLogprobs, TokenLogprob, +}; +use vllm_engine_core_client::protocol::output::{ + EngineCoreEvent, EngineCoreEventType, EngineCoreFinishReason, EngineCoreOutput, + EngineCoreOutputs, RequestBatchOutputs, +}; +use vllm_engine_core_client::protocol::request::EngineCoreRequest; +use vllm_engine_core_client::protocol::sampling::EngineCoreSamplingParams; +use vllm_engine_core_client::protocol::stats::PrefillStats; +use vllm_engine_core_client::test_utils::{IpcNamespace, spawn_mock_engine_task}; +use vllm_engine_core_client::{EngineCoreClient, EngineCoreClientConfig}; +use vllm_llm::{ + Error, FinishReason, GenerateOutputStreamExt as _, GeneratePromptInfo, GenerateRequest, Llm, +}; +use vllm_metrics::METRICS; +use zeromq::prelude::{SocketRecv, SocketSend}; +use zeromq::{DealerSocket, PushSocket, ZmqMessage}; + +static TRACING: Once = Once::new(); + +fn request_output( + request_id: &str, + new_token_ids: Vec, + finish_reason: Option, +) -> EngineCoreOutput { + request_output_with_events(request_id, new_token_ids, finish_reason, None) +} + +fn request_output_with_events( + request_id: &str, + new_token_ids: Vec, + finish_reason: Option, + events: Option>, +) -> EngineCoreOutput { + EngineCoreOutput { + request_id: request_id.to_string(), + new_token_ids, + new_logprobs: None, + new_prompt_logprobs_tensors: None, + pooling_output: None, + finish_reason, + stop_reason: None, + events, + kv_transfer_params: None, + trace_headers: None, + prefill_stats: None, + routed_experts: None, + num_nans_in_logits: 0, + } +} + +fn request_output_with_logprobs( + request_id: &str, + new_token_ids: Vec, + finish_reason: Option, + new_logprobs: Option, + prompt_logprobs: Option, +) -> EngineCoreOutput { + EngineCoreOutput { + request_id: request_id.to_string(), + new_token_ids, + new_logprobs: new_logprobs.map(MaybeWireLogprobs::Direct), + new_prompt_logprobs_tensors: prompt_logprobs.map(MaybeWireLogprobs::Direct), + pooling_output: None, + finish_reason, + stop_reason: None, + events: None, + kv_transfer_params: None, + trace_headers: None, + prefill_stats: None, + routed_experts: None, + num_nans_in_logits: 0, + } +} + +fn request_output_with_logprobs_and_kv( + request_id: &str, + new_token_ids: Vec, + finish_reason: Option, + new_logprobs: Option, + prompt_logprobs: Option, + kv_transfer_params: Option, +) -> EngineCoreOutput { + EngineCoreOutput { + request_id: request_id.to_string(), + new_token_ids, + new_logprobs: new_logprobs.map(MaybeWireLogprobs::Direct), + new_prompt_logprobs_tensors: prompt_logprobs.map(MaybeWireLogprobs::Direct), + pooling_output: None, + finish_reason, + stop_reason: None, + events: None, + kv_transfer_params, + trace_headers: None, + prefill_stats: None, + routed_experts: None, + num_nans_in_logits: 0, + } +} + +fn logprobs_for_position( + sampled_token_id: u32, + sampled_logprob: f32, + sampled_rank: u32, + top_token_id: u32, + top_logprob: f32, +) -> Logprobs { + Logprobs { + positions: vec![PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id: sampled_token_id, + logprob: sampled_logprob, + rank: sampled_rank, + }, + TokenLogprob { + token_id: top_token_id, + logprob: top_logprob, + rank: 1, + }, + ], + }], + } +} + +fn prompt_logprobs() -> Logprobs { + Logprobs { + positions: vec![ + PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id: 11, + logprob: -0.1, + rank: 2, + }, + TokenLogprob { + token_id: 7, + logprob: -0.05, + rank: 1, + }, + ], + }, + PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id: 22, + logprob: -0.2, + rank: 3, + }, + TokenLogprob { + token_id: 8, + logprob: -0.1, + rank: 1, + }, + ], + }, + ], + } +} + +fn sample_generate_request(request_id: &str, max_tokens: u32) -> GenerateRequest { + GenerateRequest { + request_id: request_id.to_string(), + prompt_token_ids: vec![11, 22], + sampling_params: EngineCoreSamplingParams { + max_tokens, + ..EngineCoreSamplingParams::for_test() + }, + mm_features: None, + arrival_time: Some(42.5), + cache_salt: None, + trace_headers: None, + priority: 0, + data_parallel_rank: None, + reasoning_parser_kwargs: None, + lora_request: None, + } +} + +async fn send_outputs(push: &mut PushSocket, outputs: EngineCoreOutputs) { + push.send(ZmqMessage::from(rmp_serde::to_vec_named(&outputs).unwrap())) + .await + .unwrap(); +} + +async fn recv_engine_message(dealer: &mut DealerSocket) -> Vec { + dealer.recv().await.unwrap().into_vec() +} + +async fn connect_async_llm_with_ipc( + handshake_address: String, + client_index: u32, + model_name: &str, + ipc: &IpcNamespace, +) -> Llm { + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name(model_name) + .with_client_index(client_index) + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .unwrap(); + Llm::new(client) +} + +fn request_metrics_model_name(prefix: &str) -> String { + format!("{prefix}-{}", Uuid::new_v4().simple()) +} + +fn init_tracing() { + TRACING.call_once(|| { + let filter = EnvFilter::try_from_default_env() + .unwrap_or_else(|_| EnvFilter::new("vllm_engine_core_client=debug")); + let _ = tracing_subscriber::fmt().with_test_writer().with_env_filter(filter).try_init(); + }); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn generate_streams_outputs() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-delta".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add = recv_engine_message(dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = rmp_serde::from_slice(&add[1]).unwrap(); + assert_eq!(request.external_req_id.as_deref(), Some("req-delta")); + assert!(request.request_id.starts_with("req-delta-")); + assert_ne!(request.request_id, "req-delta"); + assert_eq!(request.client_index, 7); + assert_eq!(request.prompt_token_ids, Some(vec![11, 22])); + + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![ + request_output_with_logprobs( + &request.request_id, + vec![1, 2], + None, + Some(logprobs_for_position(1, -0.3, 4, 9, -0.1)), + Some(prompt_logprobs()), + ), + request_output_with_logprobs( + &request.request_id, + vec![3], + Some(EngineCoreFinishReason::Length), + Some(logprobs_for_position(3, -0.4, 5, 10, -0.2)), + None, + ), + ], + finished_requests: Some(BTreeSet::from([request.request_id.clone()])), + ..Default::default() + } + .into(), + ) + .await; + }) + }, + ); + + let llm = connect_async_llm_with_ipc(handshake_address, 7, "test-model", &ipc).await; + let mut stream = llm.generate(sample_generate_request("req-delta", 3)).await.unwrap(); + let internal_id = stream.request_id().to_string(); + + let first = stream.next().await.unwrap().unwrap(); + assert_eq!(first.request_id, internal_id); + assert_eq!( + first.prompt_info, + Some(GeneratePromptInfo { + prompt_token_ids: vec![11, 22].into(), + prompt_logprobs: Some(prompt_logprobs()), + }) + ); + assert_eq!(first.token_ids, vec![1, 2]); + assert_eq!( + first.logprobs, + Some(logprobs_for_position(1, -0.3, 4, 9, -0.1)) + ); + assert_eq!(first.finish_reason, None); + + let second = stream.next().await.unwrap().unwrap(); + assert_eq!(second.prompt_info, None); + assert_eq!(second.token_ids, vec![3]); + assert_eq!( + second.logprobs, + Some(logprobs_for_position(3, -0.4, 5, 10, -0.2)) + ); + assert_eq!(second.finish_reason, Some(FinishReason::Length)); + assert!(stream.next().await.is_none()); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + llm.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn collect_output_aggregates_raw_tokens_logprobs_and_terminal_metadata() { + init_tracing(); + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-collect-output".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = rmp_serde::from_slice(&add[1]).unwrap(); + assert_eq!(request.external_req_id.as_deref(), Some("req-collect")); + assert!(request.request_id.starts_with("req-collect-")); + + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![ + EngineCoreOutput { + prefill_stats: Some(PrefillStats { + num_prompt_tokens: 2, + num_cached_tokens: 1, + num_local_cached_tokens: 1, + ..Default::default() + }), + ..request_output_with_logprobs( + &request.request_id, + vec![33], + None, + Some(logprobs_for_position(33, -0.1, 1, 99, -0.2)), + Some(prompt_logprobs()), + ) + }, + request_output_with_logprobs_and_kv( + &request.request_id, + vec![44], + Some(EngineCoreFinishReason::Stop), + Some(logprobs_for_position(44, -0.3, 1, 88, -0.4)), + None, + Some(serde_json::json!({"connector": "x"})), + ), + ], + ..Default::default() + } + .into(), + ) + .await; + }) + }, + ); + + let llm = connect_async_llm_with_ipc(handshake_address, 7, "test-model", &ipc).await; + let stream = llm.generate(sample_generate_request("req-collect", 4)).await.unwrap(); + let internal_id = stream.request_id().to_string(); + let collected = stream.collect_output().await.unwrap(); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + + assert_eq!(collected.request_id, internal_id); + assert_eq!(collected.prompt_token_ids, vec![11, 22]); + assert_eq!(collected.token_ids, vec![33, 44]); + assert_eq!(collected.finish_reason, FinishReason::stop_eos()); + assert_eq!(collected.usage.cached_token_count, 1); + assert_eq!(collected.prompt_logprobs, Some(prompt_logprobs())); + assert_eq!( + collected.logprobs.as_ref().map(|lp| lp.positions.len()), + Some(2) + ); + assert_eq!( + collected.kv_transfer_params, + Some(serde_json::json!({"connector": "x"})) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn generate_propagates_unexpected_close_errors() { + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-close".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add = recv_engine_message(dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = rmp_serde::from_slice(&add[1]).unwrap(); + + send_outputs( + push, + RequestBatchOutputs { + outputs: Vec::new(), + finished_requests: Some(BTreeSet::from([request.request_id])), + ..Default::default() + } + .into(), + ) + .await; + }) + }, + ); + + let llm = connect_async_llm_with_ipc(handshake_address, 0, "test-model", &ipc).await; + let mut stream = llm.generate(sample_generate_request("req-close", 1)).await.unwrap(); + let internal_id = stream.request_id().to_string(); + + let error = stream.next().await.unwrap().unwrap_err(); + assert!(matches!( + error, + Error::EngineCoreClient(vllm_engine_core_client::Error::RequestStreamClosed { + request_id + }) if request_id == internal_id + )); + assert!(stream.next().await.is_none()); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + llm.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn dropping_a_live_generate_stream_triggers_abort() { + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-drop".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add = recv_engine_message(dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = rmp_serde::from_slice(&add[1]).unwrap(); + assert_eq!(request.external_req_id.as_deref(), Some("req-drop")); + assert!(request.request_id.starts_with("req-drop-")); + + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output(&request.request_id, vec![99], None)], + ..Default::default() + } + .into(), + ) + .await; + + let abort = + timeout(Duration::from_secs(1), recv_engine_message(dealer)).await.unwrap(); + assert_eq!(abort[0].as_ref(), &[0x01]); + let aborted_ids: Vec = rmp_serde::from_slice(&abort[1]).unwrap(); + assert_eq!(aborted_ids, vec![request.request_id]); + }) + }, + ); + + let llm = connect_async_llm_with_ipc(handshake_address, 0, "test-model", &ipc).await; + let mut stream = llm.generate(sample_generate_request("req-drop", 4)).await.unwrap(); + + let output = stream.next().await.unwrap().unwrap(); + assert_eq!(output.token_ids, vec![99]); + drop(stream); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + llm.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn duplicate_external_request_ids_are_randomized_before_reaching_engine_core_client() { + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-dup".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add_1 = recv_engine_message(dealer).await; + assert_eq!(add_1[0].as_ref(), &[0x00]); + let request_1: EngineCoreRequest = rmp_serde::from_slice(&add_1[1]).unwrap(); + assert_eq!(request_1.external_req_id.as_deref(), Some("req-dup")); + assert!(request_1.request_id.starts_with("req-dup-")); + + let add_2 = recv_engine_message(dealer).await; + assert_eq!(add_2[0].as_ref(), &[0x00]); + let request_2: EngineCoreRequest = rmp_serde::from_slice(&add_2[1]).unwrap(); + assert_eq!(request_2.external_req_id.as_deref(), Some("req-dup")); + assert!(request_2.request_id.starts_with("req-dup-")); + assert_ne!(request_1.request_id, request_2.request_id); + + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output( + &request_1.request_id, + vec![], + Some(EngineCoreFinishReason::Length), + )], + finished_requests: Some(BTreeSet::from([request_1.request_id.clone()])), + ..Default::default() + } + .into(), + ) + .await; + + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output( + &request_2.request_id, + vec![], + Some(EngineCoreFinishReason::Length), + )], + finished_requests: Some(BTreeSet::from([request_2.request_id])), + ..Default::default() + } + .into(), + ) + .await; + }) + }, + ); + + let llm = connect_async_llm_with_ipc(handshake_address, 0, "test-model", &ipc).await; + let stream_1 = llm.generate(sample_generate_request("req-dup", 1)).await.unwrap(); + let stream_2 = llm.generate(sample_generate_request("req-dup", 1)).await.unwrap(); + let internal_id_1 = stream_1.request_id().to_string(); + let internal_id_2 = stream_2.request_id().to_string(); + let collected_1 = stream_1.collect_output().await.unwrap(); + let collected_2 = stream_2.collect_output().await.unwrap(); + assert_eq!(collected_1.request_id, internal_id_1); + assert_eq!(collected_2.request_id, internal_id_2); + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + llm.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn abort_resolves_external_request_id_to_internal_before_reaching_engine() { + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-abort".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add = recv_engine_message(dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = rmp_serde::from_slice(&add[1]).unwrap(); + assert_eq!(request.external_req_id.as_deref(), Some("req-abort")); + assert!(request.request_id.starts_with("req-abort-")); + assert_ne!(request.request_id, "req-abort"); + + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output(&request.request_id, vec![7], None)], + ..Default::default() + } + .into(), + ) + .await; + + // The abort frame must carry the internal engine id, not the + // external "req-abort" id the caller aborted by. + let abort = + timeout(Duration::from_secs(1), recv_engine_message(dealer)).await.unwrap(); + assert_eq!(abort[0].as_ref(), &[0x01]); + let aborted_ids: Vec = rmp_serde::from_slice(&abort[1]).unwrap(); + assert_eq!(aborted_ids, vec![request.request_id]); + }) + }, + ); + + let llm = connect_async_llm_with_ipc(handshake_address, 0, "test-model", &ipc).await; + let mut stream = llm.generate(sample_generate_request("req-abort", 4)).await.unwrap(); + let internal_id = stream.request_id().to_string(); + assert_ne!(internal_id, "req-abort"); + + assert_eq!(stream.next().await.unwrap().unwrap().token_ids, vec![7]); + + // Abort by the external id; engine-core only knows the internal id. + llm.abort(&["req-abort".to_string()]).await.unwrap(); + + // The consumer stream is finalized locally with a clean abort terminal + // rather than hanging or surfacing as RequestStreamClosed. The engine sends + // no final output for a client abort, so this output is synthesized. + let terminal = stream.next().await.unwrap().unwrap(); + assert_eq!(terminal.finish_reason, Some(FinishReason::Abort)); + assert!(terminal.token_ids.is_empty()); + assert!(stream.next().await.is_none()); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + drop(stream); + llm.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn abort_by_external_id_aborts_all_internal_requests() { + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-abort-many".to_vec(); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add_1 = recv_engine_message(dealer).await; + assert_eq!(add_1[0].as_ref(), &[0x00]); + let request_1: EngineCoreRequest = rmp_serde::from_slice(&add_1[1]).unwrap(); + + let add_2 = recv_engine_message(dealer).await; + assert_eq!(add_2[0].as_ref(), &[0x00]); + let request_2: EngineCoreRequest = rmp_serde::from_slice(&add_2[1]).unwrap(); + + assert_eq!(request_1.external_req_id.as_deref(), Some("req-dup-abort")); + assert_eq!(request_2.external_req_id.as_deref(), Some("req-dup-abort")); + assert_ne!(request_1.request_id, request_2.request_id); + + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![ + request_output(&request_1.request_id, vec![7], None), + request_output(&request_2.request_id, vec![8], None), + ], + ..Default::default() + } + .into(), + ) + .await; + + // A single abort by the shared external id must abort both + // internal engine ids it expanded into. + let abort = + timeout(Duration::from_secs(1), recv_engine_message(dealer)).await.unwrap(); + assert_eq!(abort[0].as_ref(), &[0x01]); + let mut aborted_ids: Vec = rmp_serde::from_slice(&abort[1]).unwrap(); + aborted_ids.sort(); + let mut expected = vec![request_1.request_id, request_2.request_id]; + expected.sort(); + assert_eq!(aborted_ids, expected); + }) + }, + ); + + let llm = connect_async_llm_with_ipc(handshake_address, 0, "test-model", &ipc).await; + let mut stream_1 = llm.generate(sample_generate_request("req-dup-abort", 4)).await.unwrap(); + let mut stream_2 = llm.generate(sample_generate_request("req-dup-abort", 4)).await.unwrap(); + assert_ne!(stream_1.request_id(), stream_2.request_id()); + + assert_eq!(stream_1.next().await.unwrap().unwrap().token_ids, vec![7]); + assert_eq!(stream_2.next().await.unwrap().unwrap().token_ids, vec![8]); + + llm.abort(&["req-dup-abort".to_string()]).await.unwrap(); + + // Both internal requests the external id expanded into are finalized with a + // clean abort terminal. + for stream in [&mut stream_1, &mut stream_2] { + let terminal = stream.next().await.unwrap().unwrap(); + assert_eq!(terminal.finish_reason, Some(FinishReason::Abort)); + assert!(stream.next().await.is_none()); + } + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + drop(stream_1); + drop(stream_2); + llm.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn generate_records_request_metrics_in_prometheus_output() { + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-metrics".to_vec(); + let model_name = request_metrics_model_name("metrics-model"); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add = recv_engine_message(dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = rmp_serde::from_slice(&add[1]).unwrap(); + + send_outputs( + push, + RequestBatchOutputs { + engine_index: 4, + timestamp: 10.0, + outputs: vec![EngineCoreOutput { + prefill_stats: Some(PrefillStats { + num_prompt_tokens: 2, + num_computed_tokens: 2, + ..Default::default() + }), + ..request_output_with_events( + &request.request_id, + vec![1], + None, + Some(vec![ + EngineCoreEvent { + r#type: EngineCoreEventType::Queued, + timestamp: 8.0, + }, + EngineCoreEvent { + r#type: EngineCoreEventType::Scheduled, + timestamp: 9.0, + }, + ]), + ) + }], + ..Default::default() + } + .into(), + ) + .await; + + send_outputs( + push, + RequestBatchOutputs { + engine_index: 4, + timestamp: 11.5, + outputs: vec![request_output_with_events( + &request.request_id, + vec![2, 3], + Some(EngineCoreFinishReason::Length), + Some(vec![EngineCoreEvent { + r#type: EngineCoreEventType::Preempted, + timestamp: 10.5, + }]), + )], + finished_requests: Some(BTreeSet::from([request.request_id])), + ..Default::default() + } + .into(), + ) + .await; + }) + }, + ); + + let llm = connect_async_llm_with_ipc(handshake_address, 0, &model_name, &ipc).await; + let mut request = sample_generate_request("req-metrics", 8); + request.arrival_time = None; + let mut stream = llm.generate(request).await.unwrap(); + + assert_eq!(stream.next().await.unwrap().unwrap().token_ids, vec![1]); + let final_output = stream.next().await.unwrap().unwrap(); + assert_eq!(final_output.token_ids, vec![2, 3]); + assert_eq!(final_output.finish_reason, Some(FinishReason::Length)); + assert!(stream.next().await.is_none()); + + let rendered = METRICS.render().unwrap(); + assert!(rendered.contains(&format!( + "vllm:request_success_total{{model_name=\"{model_name}\",engine=\"4\",finished_reason=\"length\"}} 1" + ))); + assert!(rendered.contains(&format!( + "vllm:prompt_tokens_total{{model_name=\"{model_name}\",engine=\"4\"}} 2" + ))); + assert!(rendered.contains(&format!( + "vllm:prompt_tokens_by_source_total{{model_name=\"{model_name}\",engine=\"4\",source=\"local_compute\"}} 2" + ))); + assert!(rendered.contains(&format!( + "vllm:prompt_tokens_by_source_total{{model_name=\"{model_name}\",engine=\"4\",source=\"local_cache_hit\"}} 0" + ))); + assert!(rendered.contains(&format!( + "vllm:prompt_tokens_by_source_total{{model_name=\"{model_name}\",engine=\"4\",source=\"external_kv_transfer\"}} 0" + ))); + assert!(rendered.contains(&format!( + "vllm:prompt_tokens_cached_total{{model_name=\"{model_name}\",engine=\"4\"}} 0" + ))); + assert!(rendered.contains(&format!( + "vllm:generation_tokens_total{{model_name=\"{model_name}\",engine=\"4\"}} 3" + ))); + assert!(rendered.contains(&format!( + "vllm:num_preemptions_total{{model_name=\"{model_name}\",engine=\"4\"}} 1" + ))); + assert!(rendered.contains(&format!( + "vllm:time_to_first_token_seconds_count{{model_name=\"{model_name}\",engine=\"4\"}} 1" + ))); + assert!(rendered.contains(&format!( + "vllm:inter_token_latency_seconds_count{{model_name=\"{model_name}\",engine=\"4\"}} 1" + ))); + assert!(rendered.contains(&format!( + "vllm:e2e_request_latency_seconds_count{{model_name=\"{model_name}\",engine=\"4\"}} 1" + ))); + assert!(rendered.contains(&format!( + "vllm:request_prompt_tokens_count{{model_name=\"{model_name}\",engine=\"4\"}} 1" + ))); + assert!(rendered.contains(&format!( + "vllm:request_generation_tokens_count{{model_name=\"{model_name}\",engine=\"4\"}} 1" + ))); + assert!(rendered.contains(&format!( + "vllm:request_prefill_kv_computed_tokens_count{{model_name=\"{model_name}\",engine=\"4\"}} 1" + ))); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + llm.shutdown().await.unwrap(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn dropping_stream_records_abort_terminal_request_metrics() { + let ipc = IpcNamespace::new().unwrap(); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-metrics-drop".to_vec(); + let model_name = request_metrics_model_name("metrics-drop-model"); + + let (shutdown_tx, engine_task) = spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add = recv_engine_message(dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = rmp_serde::from_slice(&add[1]).unwrap(); + assert_eq!(request.external_req_id.as_deref(), Some("req-metrics-drop")); + assert!(request.request_id.starts_with("req-metrics-drop-")); + + send_outputs( + push, + RequestBatchOutputs { + engine_index: 5, + timestamp: 10.0, + outputs: vec![request_output_with_events( + &request.request_id, + vec![99], + None, + Some(vec![ + EngineCoreEvent { + r#type: EngineCoreEventType::Queued, + timestamp: 8.0, + }, + EngineCoreEvent { + r#type: EngineCoreEventType::Scheduled, + timestamp: 9.0, + }, + ]), + )], + ..Default::default() + } + .into(), + ) + .await; + + let abort = + timeout(Duration::from_secs(1), recv_engine_message(dealer)).await.unwrap(); + assert_eq!(abort[0].as_ref(), &[0x01]); + let aborted_ids: Vec = rmp_serde::from_slice(&abort[1]).unwrap(); + assert_eq!(aborted_ids, vec![request.request_id]); + }) + }, + ); + + let llm = connect_async_llm_with_ipc(handshake_address, 0, &model_name, &ipc).await; + let mut request = sample_generate_request("req-metrics-drop", 8); + request.arrival_time = None; + let mut stream = llm.generate(request).await.unwrap(); + assert_eq!(stream.next().await.unwrap().unwrap().token_ids, vec![99]); + drop(stream); + + let _ = shutdown_tx.send(()); + engine_task.await.unwrap(); + let rendered = METRICS.render().unwrap(); + assert!(rendered.contains(&format!( + "vllm:request_success_total{{model_name=\"{model_name}\",engine=\"5\",finished_reason=\"abort\"}} 1" + ))); + assert!(rendered.contains(&format!( + "vllm:e2e_request_latency_seconds_count{{model_name=\"{model_name}\",engine=\"5\"}} 1" + ))); + + llm.shutdown().await.unwrap(); +} diff --git a/asian_ai/performance/vLLM/rust/src/managed-engine/Cargo.toml b/asian_ai/performance/vLLM/rust/src/managed-engine/Cargo.toml new file mode 100644 index 0000000000000000000000000000000000000000..a71f9b36502e54268cba5424864444a02dc14cbd --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/managed-engine/Cargo.toml @@ -0,0 +1,18 @@ +[package] +name = "vllm-managed-engine" +version.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +anyhow.workspace = true +clap.workspace = true +libc.workspace = true +tokio = { workspace = true, features = ["process"] } +tracing.workspace = true + +[dev-dependencies] +expect-test.workspace = true + +[lints] +workspace = true diff --git a/asian_ai/performance/vLLM/rust/src/managed-engine/src/cli.rs b/asian_ai/performance/vLLM/rust/src/managed-engine/src/cli.rs new file mode 100644 index 0000000000000000000000000000000000000000..15e04d232b015708c3de345efa4e9a707a89d3ca --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/managed-engine/src/cli.rs @@ -0,0 +1,384 @@ +use std::collections::HashSet; +use std::ffi::OsString; + +use clap::error::ErrorKind; +use clap::{Args, CommandFactory}; + +use crate::{ManagedEngineConfig, allocate_handshake_port}; + +/// Managed Python headless-engine CLI arguments. +#[derive(Debug, Clone, Args, PartialEq, Eq)] +pub struct ManagedEngineArgs { + /// Python executable used to launch the managed headless vLLM engine. + #[arg(long, env = "VLLM_RS_PYTHON", default_value = "python3")] + pub python: String, + /// Host/IP used both for the managed-engine handshake endpoint and the + /// frontend-advertised input/output ZMQ socket addresses. + #[arg( + long = "data-parallel-address", + visible_alias = "handshake-host", + default_value = "127.0.0.1" + )] + pub handshake_host: String, + /// Optional TCP port for the managed-engine handshake / data-parallel RPC + /// endpoint. + /// + /// When omitted, the CLI allocates an ephemeral port automatically. + #[arg( + long = "data-parallel-rpc-port", + visible_alias = "handshake-port", + value_parser = clap::value_parser!(u16).range(1..) + )] + pub handshake_port: Option, + /// Number of data parallel replicas across the whole deployment. + #[arg(long, default_value_t = 1)] + pub data_parallel_size: usize, + /// Number of data parallel replicas to run on this node. + #[arg(long)] + pub data_parallel_size_local: Option, + + /// Additional arguments forwarded to `python -m vllm.entrypoints.cli.main + /// serve ...`. + /// + /// Arguments after an explicit `--` are forwarded verbatim. Before `--`, + /// `vllm-rs serve` automatically keeps recognized frontend options on + /// the Rust side and forwards everything else to Python. + /// + /// The explicit `--` passthrough is a last-resort escape hatch. Rust does + /// not interpret, validate, or de-duplicate those arguments against + /// managed-engine arguments that it appends later; if the same Python flag + /// appears more than once, Python argparse owns the final result. + #[arg( + last = true, + allow_hyphen_values = true, + help_heading = "Passthrough arguments" + )] + pub python_args: Vec, +} + +impl ManagedEngineArgs { + /// Build the handshake address shared by the Rust frontend and managed + /// Python engine. + pub fn handshake_address(&self, handshake_port: u16) -> String { + format!("tcp://{}:{}", self.handshake_host, handshake_port) + } + + /// Resolve the handshake port, either from the CLI argument (if specified) + /// or by allocating a fresh port. + pub fn resolve_handshake_port(&self) -> anyhow::Result { + self.handshake_port + .map(Ok) + .unwrap_or_else(|| allocate_handshake_port(&self.handshake_host)) + } + + /// Build the managed Python-engine spawn configuration. + pub fn into_config( + self, + model: String, + max_model_len: Option, + max_logprobs: Option, + profiler_config: Option, + reasoning_parser: Option<&str>, + language_model_only: bool, + disable_log_stats: bool, + shutdown_timeout: u64, + handshake_port: u16, + ) -> ManagedEngineConfig { + let mut python_args = self.python_args; + // Manually forward some args to the Python engine. + if let Some(max_model_len) = max_model_len { + python_args.push("--max-model-len".to_string()); + python_args.push(max_model_len.to_string()); + } + if let Some(max_logprobs) = max_logprobs { + python_args.push("--max-logprobs".to_string()); + python_args.push(max_logprobs.to_string()); + } + if let Some(profiler_config) = profiler_config { + python_args.push("--profiler-config".to_string()); + python_args.push(profiler_config); + } + if let Some(reasoning_parser) = reasoning_parser { + python_args.push("--reasoning-parser".to_string()); + python_args.push(reasoning_parser.to_string()); + } + if language_model_only { + python_args.push("--language-model-only".to_string()); + } + if disable_log_stats { + python_args.push("--disable-log-stats".to_string()); + } + // we must pass through shutdown_timeout to the engine, + // otherwise inflight requests get aborted on shutdown + if shutdown_timeout > 0 { + python_args.push("--shutdown-timeout".to_string()); + python_args.push(shutdown_timeout.to_string()); + } + if let Some(data_parallel_size_local) = self.data_parallel_size_local { + python_args.push("--data-parallel-size-local".to_string()); + python_args.push(data_parallel_size_local.to_string()); + } + + ManagedEngineConfig { + python: self.python, + model, + handshake_host: self.handshake_host, + handshake_port, + data_parallel_size: self.data_parallel_size, + python_args, + } + } + + /// Return the number of engines that the Rust frontend should expect to + /// coordinate with. + fn local_engine_count(&self) -> usize { + self.data_parallel_size_local.unwrap_or(self.data_parallel_size) + } + + /// Return whether the managed Rust frontend only needs to communicate with + /// colocated engines. + pub fn frontend_local_only(&self) -> bool { + self.data_parallel_size_local != Some(0) + && self.local_engine_count() == self.data_parallel_size + } +} + +/// Python `argparse` accepts these multi-character single-dash aliases, but +/// `clap` cannot model them directly. +const PYTHON_MULTI_CHAR_ALIASES: &[(&str, &str)] = &[ + ("-asc", "--api-server-count"), + ("-pp", "--pipeline-parallel-size"), + ("-tp", "--tensor-parallel-size"), + ("-dcp", "--decode-context-parallel-size"), + ("-pcp", "--prefill-context-parallel-size"), + ("-dp", "--data-parallel-size"), + ("-dpn", "--data-parallel-rank"), + ("-dpr", "--data-parallel-start-rank"), + ("-dpl", "--data-parallel-size-local"), + ("-dpa", "--data-parallel-address"), + ("-dpp", "--data-parallel-rpc-port"), + ("-dpb", "--data-parallel-backend"), + ("-dph", "--data-parallel-hybrid-lb"), + ("-dpe", "--data-parallel-external-lb"), + ("-ep", "--enable-expert-parallel"), + ("-cc", "--compilation-config"), + ("-ac", "--attention-config"), +]; + +/// Repartition managed-engine argv so Rust-owned flags stay before `--`, while +/// everything else is forwarded to Python. +pub fn repartition_managed_engine_args( + args: &[OsString], + subcommand: Option<&str>, +) -> Result, clap::Error> +where + C: CommandFactory, +{ + let command = C::command(); + let (prefix, real_args, command) = match subcommand { + Some(subcommand) => { + if !matches_subcommand(args, subcommand) { + return Ok(args.to_vec()); + }; + + let subcommand = command + .find_subcommand(subcommand) + .expect("managed-engine subcommand should exist"); + + (args[..2].to_vec(), &args[2..], subcommand) + } + None => { + let Some(program) = args.first() else { + return Ok(args.to_vec()); + }; + + (vec![program.clone()], &args[1..], &command) + } + }; + + let mut repartitioned = prefix; + repartitioned.extend(repartition_real_managed_engine_args(real_args, command)?); + Ok(repartitioned) +} + +fn repartition_real_managed_engine_args( + args: &[OsString], + command: &clap::Command, +) -> Result, clap::Error> { + let Some(model) = args.first() else { + return Ok(args.to_vec()); + }; + + let model = model.to_string_lossy(); + if is_help_flag(&model) { + return Ok(args.to_vec()); + } + if model == "--" || is_option_like(&model) { + return Err(build_missing_model_error(command)); + } + + let (long_flags, short_flags) = collect_option_names(command); + let (front_args, explicit_passthrough, had_separator) = split_managed_engine_args(&args[1..]); + let normalized_front_args = normalize_python_arg_aliases(front_args); + + let mut frontend_chunks = Vec::new(); + let mut python_chunks = Vec::new(); + let mut current_chunk = Vec::new(); + + for arg in normalized_front_args { + let text = arg.to_string_lossy(); + if is_option_like(&text) && !current_chunk.is_empty() { + push_chunk( + &mut frontend_chunks, + &mut python_chunks, + std::mem::take(&mut current_chunk), + &long_flags, + &short_flags, + ); + } + current_chunk.push(arg); + } + if !current_chunk.is_empty() { + push_chunk( + &mut frontend_chunks, + &mut python_chunks, + current_chunk, + &long_flags, + &short_flags, + ); + } + + let mut repartitioned = vec![args[0].clone()]; + repartitioned.extend(frontend_chunks); + if had_separator || !python_chunks.is_empty() || !explicit_passthrough.is_empty() { + repartitioned.push("--".into()); + repartitioned.extend(python_chunks); + repartitioned.extend(explicit_passthrough.iter().cloned()); + } + + Ok(repartitioned) +} + +fn matches_subcommand(args: &[OsString], subcommand: &str) -> bool { + args.get(1) + .and_then(|arg| arg.to_str()) + .is_some_and(|candidate| candidate == subcommand) +} + +fn split_managed_engine_args(args: &[OsString]) -> (&[OsString], &[OsString], bool) { + if let Some(index) = args.iter().position(|arg| arg == "--") { + (&args[..index], &args[index + 1..], true) + } else { + (args, &[], false) + } +} + +fn normalize_python_arg_aliases(args: &[OsString]) -> Vec { + args.iter() + .map(|arg| { + let text = arg.to_string_lossy(); + normalize_python_multi_char_alias(&text) + .map(Into::into) + .unwrap_or_else(|| arg.clone()) + }) + .collect() +} + +fn normalize_python_multi_char_alias(arg: &str) -> Option { + find_python_multi_char_alias(arg).map(|canonical| match arg.split_once('=') { + Some((_, value)) => format!("{canonical}={value}"), + None => canonical.to_string(), + }) +} + +fn find_python_multi_char_alias(arg: &str) -> Option<&'static str> { + PYTHON_MULTI_CHAR_ALIASES.iter().find_map(|&(alias, canonical)| { + (arg == alias || arg.starts_with(&format!("{alias}="))).then_some(canonical) + }) +} + +fn push_chunk( + frontend_chunks: &mut Vec, + python_chunks: &mut Vec, + chunk: Vec, + long_flags: &HashSet, + short_flags: &HashSet, +) { + if chunk_head_is_frontend_owned(&chunk, long_flags, short_flags) { + frontend_chunks.extend(chunk); + } else { + python_chunks.extend(chunk); + } +} + +fn chunk_head_is_frontend_owned( + chunk: &[OsString], + long_flags: &HashSet, + short_flags: &HashSet, +) -> bool { + let Some(head) = chunk.first() else { + return false; + }; + let head = head.to_string_lossy(); + + if let Some(rest) = head.strip_prefix("--") { + let name = rest.split_once('=').map_or(rest, |(name, _)| name); + return long_flags.contains(name); + } + + let Some(rest) = head.strip_prefix('-') else { + return false; + }; + let Some(short) = rest.chars().next() else { + return false; + }; + short_flags.contains(&short) +} + +fn collect_option_names(command: &clap::Command) -> (HashSet, HashSet) { + let mut long_flags = HashSet::new(); + let mut short_flags = HashSet::new(); + for arg in command.get_arguments() { + if let Some(names) = arg.get_long_and_visible_aliases() { + long_flags.extend(names.into_iter().map(str::to_owned)); + } + if let Some(short) = arg.get_short() { + short_flags.insert(short); + } + if let Some(short_aliases) = arg.get_visible_short_aliases() { + short_flags.extend(short_aliases); + } + } + + long_flags.insert("help".to_string()); + short_flags.insert('h'); + + (long_flags, short_flags) +} + +fn is_option_like(arg: &str) -> bool { + if arg == "--" { + return false; + } + + if let Some(rest) = arg.strip_prefix("--") { + return rest.chars().next().is_some_and(char::is_alphabetic); + } + + if let Some(rest) = arg.strip_prefix('-') { + return rest.chars().next().is_some_and(char::is_alphabetic); + } + + false +} + +fn is_help_flag(arg: &str) -> bool { + arg == "-h" || arg == "--help" +} + +fn build_missing_model_error(command: &clap::Command) -> clap::Error { + command.clone().error( + ErrorKind::MissingRequiredArgument, + "the model must appear immediately after the command", + ) +} diff --git a/asian_ai/performance/vLLM/rust/src/managed-engine/src/lib.rs b/asian_ai/performance/vLLM/rust/src/managed-engine/src/lib.rs new file mode 100644 index 0000000000000000000000000000000000000000..55e0171e5e2ea401367cdd01c1cbb36e75ec11ae --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/managed-engine/src/lib.rs @@ -0,0 +1,4 @@ +pub mod cli; +mod process; + +pub use process::{ManagedEngineConfig, ManagedEngineHandle, allocate_handshake_port}; diff --git a/asian_ai/performance/vLLM/rust/src/managed-engine/src/process.rs b/asian_ai/performance/vLLM/rust/src/managed-engine/src/process.rs new file mode 100644 index 0000000000000000000000000000000000000000..c37cd0403867c76e4fe9f55aba9342a8c12911d8 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/managed-engine/src/process.rs @@ -0,0 +1,263 @@ +use std::io; +use std::net::TcpListener; +use std::process::{Command as StdCommand, ExitStatus, Stdio}; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::Duration; + +use anyhow::{Context, Result}; +use tokio::process::{Child, Command}; +use tokio::sync::Mutex; +use tokio::time::interval; +use tracing::info; + +const CHILD_POLL_INTERVAL: Duration = Duration::from_millis(200); +const MIN_SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(5); + +/// Allocate one ephemeral TCP port for the managed headless-engine handshake on +/// the given host. +pub fn allocate_handshake_port(host: &str) -> Result { + let listener = TcpListener::bind((host, 0)).context("failed to allocate handshake port")?; + let port = listener + .local_addr() + .context("failed to inspect allocated handshake listener address")? + .port(); + Ok(port) +} + +/// Spawn configuration for one managed headless Python vLLM engine. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ManagedEngineConfig { + /// Python executable used to launch `vllm.entrypoints.cli.main`. + pub python: String, + /// Model identifier passed to `vllm ... serve `. + pub model: String, + /// Host portion of the headless-engine handshake endpoint. + pub handshake_host: String, + /// Port portion of the headless-engine handshake endpoint. + pub handshake_port: u16, + /// Number of data parallel replicas across the whole deployment. + /// + /// The per-node replica count is forwarded separately in `python_args` as + /// `--data-parallel-size-local`. + pub data_parallel_size: usize, + /// Extra CLI arguments forwarded verbatim to Python vLLM. + pub python_args: Vec, +} + +impl ManagedEngineConfig { + /// Render the handshake address that the Rust frontend should dial. + pub fn handshake_address(&self) -> String { + format!("tcp://{}:{}", self.handshake_host, self.handshake_port) + } + + /// Build the concrete Python command line for the managed headless engine. + pub fn to_command(&self) -> StdCommand { + let mut command = StdCommand::new(&self.python); + command + .arg("-m") + .arg("vllm.entrypoints.cli.main") + .arg("serve") + .arg(&self.model) + .arg("--headless") + .arg("--data-parallel-address") + .arg(&self.handshake_host) + .arg("--data-parallel-rpc-port") + .arg(self.handshake_port.to_string()) + .arg("--data-parallel-size") + .arg(self.data_parallel_size.to_string()) + .args(&self.python_args); + command + } +} + +/// RAII-style handle for one managed Python headless engine subprocess. +#[derive(Clone)] +pub struct ManagedEngineHandle { + child: Arc>, + shutdown_started: Arc, +} + +impl ManagedEngineHandle { + /// Spawn one managed Python headless engine and return a handle for + /// monitoring it. + pub async fn spawn(config: ManagedEngineConfig) -> Result { + let command = config.to_command(); + info!( + handshake_address = %config.handshake_address(), + ?command, + "starting managed Python headless engine" + ); + + let mut command = Command::from(command); + command.stdin(Stdio::null()).stdout(Stdio::inherit()).stderr(Stdio::inherit()); + + process_group::configure(&mut command); + + let child = command.spawn().context("failed to spawn managed engine")?; + + Ok(Self { + child: Arc::new(Mutex::new(child)), + shutdown_started: Arc::new(AtomicBool::new(false)), + }) + } + + /// Poll whether the managed engine has exited yet. + pub async fn try_wait(&self) -> Option { + let mut child = self.child.lock().await; + child.try_wait().expect("failed to poll the status of managed engine") + } + + /// Wait until the managed engine exits. + pub async fn wait_for_exit(&self) -> ExitStatus { + let mut interval = interval(CHILD_POLL_INTERVAL); + loop { + interval.tick().await; + if let Some(status) = self.try_wait().await { + return status; + } + } + } + + /// Terminate the managed engine process group and wait for it to stop. + pub async fn shutdown(&self, timeout: Duration) -> Result<()> { + if self.shutdown_started.swap(true, Ordering::SeqCst) { + return Ok(()); + } + + let Some(pid) = self.child.lock().await.id() else { + return Ok(()); + }; + + // Enforce a minimum shutdown timeout to give the engine process enough time to + // clean up. + let shutdown_timeout = std::cmp::max(timeout, MIN_SHUTDOWN_TIMEOUT); + + // First, try to gracefully terminate. + info!( + pid, + ?shutdown_timeout, + "shutting down managed engine with SIGTERM" + ); + process_group::terminate(pid)?; + + // Wait for the process to exit on its own. + if tokio::time::timeout(shutdown_timeout, self.wait_for_exit()).await.is_ok() { + return Ok(()); + } + + // If it doesn't exit within the timeout, force kill it. + info!( + pid, + "managed engine did not exit within timeout, sending SIGKILL" + ); + process_group::kill(pid)?; + + let _ = self.wait_for_exit().await; + Ok(()) + } +} + +/// Process group helper functions for managing the Python subprocess and its +/// potential children in a platform-aware way. +mod process_group { + use super::*; + + /// Place the Python child into its own process group so `serve` can tear + /// down the whole subtree rather than just the immediate shell process. + pub fn configure(command: &mut Command) { + unsafe { + command.pre_exec(|| { + if libc::setpgid(0, 0) != 0 { + return Err(io::Error::last_os_error()); + } + Ok(()) + }); + } + } + + /// Send SIGTERM to the managed Python process group. + pub fn terminate(pid: u32) -> Result<()> { + signal(pid, libc::SIGTERM) + } + + /// Send SIGKILL to the managed Python process group. + pub fn kill(pid: u32) -> Result<()> { + signal(pid, libc::SIGKILL) + } + + /// Deliver one signal to the managed Python process group. + fn signal(pid: u32, signal: i32) -> Result<()> { + let rc = unsafe { libc::kill(-(pid as i32), signal) }; + if rc == 0 { + return Ok(()); + } + + let error = io::Error::last_os_error(); + if matches!(error.raw_os_error(), Some(code) if code == libc::ESRCH) { + return Ok(()); + } + Err(error).context("failed to signal managed engine process group") + } +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + + use super::{ManagedEngineConfig, allocate_handshake_port}; + + #[test] + fn command_snapshot() { + let config = ManagedEngineConfig { + python: "python3".to_string(), + model: "Qwen/Qwen3-0.6B".to_string(), + handshake_host: "127.0.0.1".to_string(), + handshake_port: 62100, + data_parallel_size: 4, + python_args: vec![ + "--data-parallel-size-local".to_string(), + "2".to_string(), + "--data-parallel-start-rank".to_string(), + "2".to_string(), + "--dtype".to_string(), + "float16".to_string(), + "--max-model-len".to_string(), + "512".to_string(), + ], + }; + let command = config.to_command(); + let args = command.get_args().collect::>(); + + expect![[r#" + [ + "-m", + "vllm.entrypoints.cli.main", + "serve", + "Qwen/Qwen3-0.6B", + "--headless", + "--data-parallel-address", + "127.0.0.1", + "--data-parallel-rpc-port", + "62100", + "--data-parallel-size", + "4", + "--data-parallel-size-local", + "2", + "--data-parallel-start-rank", + "2", + "--dtype", + "float16", + "--max-model-len", + "512", + ] + "#]] + .assert_debug_eq(&args); + } + + #[test] + fn allocate_handshake_port_returns_non_zero_port() { + let port = allocate_handshake_port("127.0.0.1").unwrap(); + assert_ne!(port, 0); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/metrics/Cargo.toml b/asian_ai/performance/vLLM/rust/src/metrics/Cargo.toml new file mode 100644 index 0000000000000000000000000000000000000000..5e81ca7fb6774e521d8a1ac35a7d4eb28f2be925 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/metrics/Cargo.toml @@ -0,0 +1,12 @@ +[package] +name = "vllm-metrics" +version.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +itertools.workspace = true +prometheus-client.workspace = true + +[lints] +workspace = true diff --git a/asian_ai/performance/vLLM/rust/src/metrics/src/api_server.rs b/asian_ai/performance/vLLM/rust/src/metrics/src/api_server.rs new file mode 100644 index 0000000000000000000000000000000000000000..d9072dbd9cc12e24504397ac00b92b1e0c121972 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/metrics/src/api_server.rs @@ -0,0 +1,78 @@ +use prometheus_client::encoding::EncodeLabelSet; +use prometheus_client::metrics::family::Family; +use prometheus_client::metrics::histogram::Histogram; +use prometheus_client::registry::Registry; + +use crate::U64Counter; + +const HTTP_REQUEST_DURATION_BUCKETS: [f64; 3] = [0.1, 0.5, 1.0]; +const HTTP_REQUEST_DURATION_HIGHR_BUCKETS: [f64; 21] = [ + 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, + 7.5, 10.0, 30.0, 60.0, +]; + +fn http_request_duration_histogram() -> Histogram { + Histogram::new(HTTP_REQUEST_DURATION_BUCKETS.iter().copied()) +} + +fn http_request_duration_highr_histogram() -> Histogram { + Histogram::new(HTTP_REQUEST_DURATION_HIGHR_BUCKETS.iter().copied()) +} + +#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)] +pub struct HttpRequestLabels { + pub method: String, + pub status: &'static str, + pub handler: String, +} + +#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)] +pub struct HttpHandlerLabels { + pub method: String, + pub handler: String, +} + +pub(crate) type HttpRequestCounterFamily = Family; +pub(crate) type HttpHandlerHistogramFamily = + Family Histogram>; + +/// API-server Prometheus families exported from the HTTP middleware layer. +pub struct ApiServerMetrics { + pub http_requests: HttpRequestCounterFamily, + pub http_request_duration_seconds: HttpHandlerHistogramFamily, + pub http_request_duration_highr_seconds: Histogram, +} + +impl ApiServerMetrics { + /// Register the API-server metric families into the shared registry. + pub(crate) fn register(registry: &mut Registry) -> Self { + let http_requests = HttpRequestCounterFamily::default(); + registry.register( + "http_requests", + "Total number of HTTP requests by method, status, and handler.", + http_requests.clone(), + ); + + let http_request_duration_seconds = HttpHandlerHistogramFamily::new_with_constructor( + http_request_duration_histogram as fn() -> Histogram, + ); + registry.register( + "http_request_duration_seconds", + "Duration of HTTP requests in seconds grouped by method and handler.", + http_request_duration_seconds.clone(), + ); + + let http_request_duration_highr_seconds = http_request_duration_highr_histogram(); + registry.register( + "http_request_duration_highr_seconds", + "High-resolution duration of HTTP requests in seconds.", + http_request_duration_highr_seconds.clone(), + ); + + Self { + http_requests, + http_request_duration_seconds, + http_request_duration_highr_seconds, + } + } +} diff --git a/asian_ai/performance/vLLM/rust/src/metrics/src/lib.rs b/asian_ai/performance/vLLM/rust/src/metrics/src/lib.rs new file mode 100644 index 0000000000000000000000000000000000000000..1700ba34bbfffa3a0e9f6a9dbf0d1d6a722352f8 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/metrics/src/lib.rs @@ -0,0 +1,76 @@ +use std::fmt; +use std::sync::LazyLock; +use std::sync::atomic::AtomicU64; + +use prometheus_client::encoding::text::encode; +use prometheus_client::metrics::counter::Counter; +use prometheus_client::metrics::family::Family; +use prometheus_client::metrics::gauge::Gauge; +use prometheus_client::metrics::histogram::Histogram; +use prometheus_client::registry::Registry; + +mod api_server; +mod request; +mod scheduler; + +pub use api_server::*; +pub use request::*; +pub use scheduler::*; + +// Note: `prometheus-client` appends the `_total` suffix automatically when +// encoding counters, so all counter family registration names in this crate +// must use the base metric name without a trailing `_total`. +pub type U64Counter = Counter; +pub type U64Gauge = Gauge; +pub type F64Gauge = Gauge; +pub(crate) type HistogramFamily = Family Histogram>; + +/// Shared Prometheus registry for frontend metrics. +/// +/// Original Python definition: +/// +pub struct Metrics { + registry: Registry, + pub scheduler: SchedulerMetrics, + pub request: RequestMetrics, + pub api_server: ApiServerMetrics, +} + +impl Metrics { + /// Construct a new metrics registry. + pub fn new() -> Self { + let mut registry = Registry::default(); + let scheduler = SchedulerMetrics::register(&mut registry); + let request = RequestMetrics::register(&mut registry); + let api_server = ApiServerMetrics::register(&mut registry); + + Self { + registry, + scheduler, + request, + api_server, + } + } + + /// Render the current metrics registry into Prometheus/OpenMetrics text + /// format. + pub fn render(&self) -> Result { + let mut output = String::new(); + encode(&mut output, &self.registry)?; + Ok(output) + } + + /// Return the registry owned by this metrics object. + pub fn registry(&self) -> &Registry { + &self.registry + } +} + +impl Default for Metrics { + fn default() -> Self { + Self::new() + } +} + +/// Process-global metrics registry shared by the frontend crates. +pub static METRICS: LazyLock = LazyLock::new(Metrics::new); diff --git a/asian_ai/performance/vLLM/rust/src/metrics/src/request.rs b/asian_ai/performance/vLLM/rust/src/metrics/src/request.rs new file mode 100644 index 0000000000000000000000000000000000000000..98c2b07dc0ce15eb207156c6e11208901de8fd50 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/metrics/src/request.rs @@ -0,0 +1,298 @@ +use prometheus_client::encoding::EncodeLabelSet; +use prometheus_client::metrics::family::Family; +use prometheus_client::metrics::histogram::Histogram; +use prometheus_client::registry::Registry; + +use crate::{EngineLabels, HistogramFamily, U64Counter}; + +const TTFT_BUCKETS: [f64; 22] = [ + 0.001, 0.005, 0.01, 0.02, 0.04, 0.06, 0.08, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0, + 20.0, 40.0, 80.0, 160.0, 640.0, 2560.0, +]; +const ITL_BUCKETS: [f64; 19] = [ + 0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0, 20.0, + 40.0, 80.0, +]; +const REQUEST_LATENCY_BUCKETS: [f64; 21] = [ + 0.3, 0.5, 0.8, 1.0, 1.5, 2.0, 2.5, 5.0, 10.0, 15.0, 20.0, 30.0, 40.0, 50.0, 60.0, 120.0, 240.0, + 480.0, 960.0, 1920.0, 7680.0, +]; +const REQUEST_PARAMS_N_BUCKETS: [f64; 5] = [1.0, 2.0, 5.0, 10.0, 20.0]; + +fn build_1_2_5_buckets(max_value: u32) -> Vec { + let mut buckets = Vec::new(); + let mut exponent = 0; + loop { + for mantissa in [1_u32, 2, 5] { + let value = mantissa * 10_u32.pow(exponent); + if value <= max_value { + buckets.push(value as f64); + } else { + if buckets.last().copied() != Some(max_value as f64) { + buckets.push(max_value as f64); + } + return buckets; + } + } + exponent += 1; + } +} + +fn time_to_first_token_histogram() -> Histogram { + Histogram::new(TTFT_BUCKETS.iter().copied()) +} + +fn inter_token_latency_histogram() -> Histogram { + Histogram::new(ITL_BUCKETS.iter().copied()) +} + +fn request_time_per_output_token_histogram() -> Histogram { + Histogram::new(ITL_BUCKETS.iter().copied()) +} + +fn request_latency_histogram() -> Histogram { + Histogram::new(REQUEST_LATENCY_BUCKETS.iter().copied()) +} + +fn request_token_count_histogram() -> Histogram { + // TODO: determine max value based on `max_model_len`. + Histogram::new(build_1_2_5_buckets(131_072)) +} + +fn request_params_n_histogram() -> Histogram { + Histogram::new(REQUEST_PARAMS_N_BUCKETS.iter().copied()) +} + +#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)] +pub struct FinishedReasonLabels { + pub model_name: String, + pub engine: u32, + pub finished_reason: &'static str, +} + +#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)] +pub struct PromptTokenSourceLabels { + pub model_name: String, + pub engine: u32, + pub source: &'static str, +} + +pub(crate) type FinishedReasonCounterFamily = Family; +pub(crate) type PromptTokenSourceCounterFamily = Family; + +/// Request-lifecycle Prometheus families exported from the `llm` layer. +pub struct RequestMetrics { + // Request-derived counters. + pub num_preemptions: Family, + pub prompt_tokens: Family, + pub prompt_tokens_by_source: PromptTokenSourceCounterFamily, + pub prompt_tokens_cached: Family, + pub generation_tokens: Family, + + // We intentionally don't support iteration-level histograms for now, since it seems to make + // more sense if the engine maintains these metrics and frontend simply forwards. + // + // pub iteration_tokens_total: HistogramFamily, + + // Request lifecycle counters and histograms. + pub request_success: FinishedReasonCounterFamily, + pub request_prompt_tokens: HistogramFamily, + pub request_generation_tokens: HistogramFamily, + pub request_max_num_generation_tokens: HistogramFamily, + pub request_params_max_tokens: HistogramFamily, + pub request_params_n: HistogramFamily, + pub request_prefill_kv_computed_tokens: HistogramFamily, + pub time_to_first_token_seconds: HistogramFamily, + pub inter_token_latency_seconds: HistogramFamily, + pub e2e_request_latency_seconds: HistogramFamily, + pub request_queue_time_seconds: HistogramFamily, + pub request_prefill_time_seconds: HistogramFamily, + pub request_decode_time_seconds: HistogramFamily, + pub request_inference_time_seconds: HistogramFamily, + pub request_time_per_output_token_seconds: HistogramFamily, +} + +impl RequestMetrics { + /// Register the request-oriented metric families into the shared registry. + pub(crate) fn register(registry: &mut Registry) -> Self { + // Request-derived counters. + let num_preemptions = Family::default(); + registry.register( + "vllm:num_preemptions", + "Cumulative number of preemption events.", + num_preemptions.clone(), + ); + + let prompt_tokens = Family::default(); + registry.register( + "vllm:prompt_tokens", + "Number of prefill tokens processed.", + prompt_tokens.clone(), + ); + + let prompt_tokens_by_source = Family::default(); + registry.register( + "vllm:prompt_tokens_by_source", + "Number of prompt tokens by source.", + prompt_tokens_by_source.clone(), + ); + + let prompt_tokens_cached = Family::default(); + registry.register( + "vllm:prompt_tokens_cached", + "Number of prompt tokens with prefix cache hits.", + prompt_tokens_cached.clone(), + ); + + let generation_tokens = Family::default(); + registry.register( + "vllm:generation_tokens", + "Number of generation tokens processed.", + generation_tokens.clone(), + ); + + // Request lifecycle counters and histograms. + let request_success = Family::default(); + registry.register( + "vllm:request_success", + "Count of successfully processed requests.", + request_success.clone(), + ); + + let request_prompt_tokens = + Family::new_with_constructor(request_token_count_histogram as fn() -> Histogram); + registry.register( + "vllm:request_prompt_tokens", + "Number of prefill tokens processed.", + request_prompt_tokens.clone(), + ); + + let request_generation_tokens = + Family::new_with_constructor(request_token_count_histogram as fn() -> Histogram); + registry.register( + "vllm:request_generation_tokens", + "Number of generation tokens processed.", + request_generation_tokens.clone(), + ); + + let request_max_num_generation_tokens = + Family::new_with_constructor(request_token_count_histogram as fn() -> Histogram); + registry.register( + "vllm:request_max_num_generation_tokens", + "Histogram of maximum number of requested generation tokens.", + request_max_num_generation_tokens.clone(), + ); + + let request_params_max_tokens = + Family::new_with_constructor(request_token_count_histogram as fn() -> Histogram); + registry.register( + "vllm:request_params_max_tokens", + "Histogram of the max_tokens request parameter.", + request_params_max_tokens.clone(), + ); + + let request_params_n = + Family::new_with_constructor(request_params_n_histogram as fn() -> Histogram); + registry.register( + "vllm:request_params_n", + "Histogram of the n request parameter.", + request_params_n.clone(), + ); + + let request_prefill_kv_computed_tokens = + Family::new_with_constructor(request_token_count_histogram as fn() -> Histogram); + registry.register( + "vllm:request_prefill_kv_computed_tokens", + "Histogram of new KV tokens computed during prefill (excluding cached tokens).", + request_prefill_kv_computed_tokens.clone(), + ); + + let time_to_first_token_seconds = + Family::new_with_constructor(time_to_first_token_histogram as fn() -> Histogram); + registry.register( + "vllm:time_to_first_token_seconds", + "Histogram of time to first token in seconds.", + time_to_first_token_seconds.clone(), + ); + + let inter_token_latency_seconds = + Family::new_with_constructor(inter_token_latency_histogram as fn() -> Histogram); + registry.register( + "vllm:inter_token_latency_seconds", + "Histogram of inter-token latency in seconds.", + inter_token_latency_seconds.clone(), + ); + + let e2e_request_latency_seconds = + Family::new_with_constructor(request_latency_histogram as fn() -> Histogram); + registry.register( + "vllm:e2e_request_latency_seconds", + "Histogram of e2e request latency in seconds.", + e2e_request_latency_seconds.clone(), + ); + + let request_queue_time_seconds = + Family::new_with_constructor(request_latency_histogram as fn() -> Histogram); + registry.register( + "vllm:request_queue_time_seconds", + "Histogram of time spent in WAITING phase for request.", + request_queue_time_seconds.clone(), + ); + + let request_prefill_time_seconds = + Family::new_with_constructor(request_latency_histogram as fn() -> Histogram); + registry.register( + "vllm:request_prefill_time_seconds", + "Histogram of time spent in PREFILL phase for request.", + request_prefill_time_seconds.clone(), + ); + + let request_decode_time_seconds = + Family::new_with_constructor(request_latency_histogram as fn() -> Histogram); + registry.register( + "vllm:request_decode_time_seconds", + "Histogram of time spent in DECODE phase for request.", + request_decode_time_seconds.clone(), + ); + + let request_inference_time_seconds = + Family::new_with_constructor(request_latency_histogram as fn() -> Histogram); + registry.register( + "vllm:request_inference_time_seconds", + "Histogram of time spent in RUNNING phase for request.", + request_inference_time_seconds.clone(), + ); + + let request_time_per_output_token_seconds = Family::new_with_constructor( + request_time_per_output_token_histogram as fn() -> Histogram, + ); + registry.register( + "vllm:request_time_per_output_token_seconds", + "Histogram of time_per_output_token_seconds per request.", + request_time_per_output_token_seconds.clone(), + ); + + Self { + num_preemptions, + prompt_tokens, + prompt_tokens_by_source, + prompt_tokens_cached, + generation_tokens, + request_success, + request_prompt_tokens, + request_generation_tokens, + request_max_num_generation_tokens, + request_params_max_tokens, + request_params_n, + request_prefill_kv_computed_tokens, + time_to_first_token_seconds, + inter_token_latency_seconds, + e2e_request_latency_seconds, + request_queue_time_seconds, + request_prefill_time_seconds, + request_decode_time_seconds, + request_inference_time_seconds, + request_time_per_output_token_seconds, + } + } +} diff --git a/asian_ai/performance/vLLM/rust/src/metrics/src/scheduler.rs b/asian_ai/performance/vLLM/rust/src/metrics/src/scheduler.rs new file mode 100644 index 0000000000000000000000000000000000000000..03ec61dd77be549b7f33647c8b046200d1656452 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/metrics/src/scheduler.rs @@ -0,0 +1,421 @@ +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::{Arc, Mutex}; + +use itertools::Itertools as _; +use prometheus_client::encoding::{EncodeLabelSet, EncodeLabelValue, LabelValueEncoder}; +use prometheus_client::metrics::family::Family; +use prometheus_client::metrics::histogram::Histogram; +use prometheus_client::registry::Registry; + +use crate::{F64Gauge, HistogramFamily, U64Counter, U64Gauge}; + +const KV_CACHE_RESIDENCY_BUCKETS: [f64; 21] = [ + 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0, 20.0, 30.0, 60.0, + 120.0, 300.0, 600.0, 1200.0, 1800.0, +]; + +fn kv_block_lifetime_histogram() -> Histogram { + Histogram::new(KV_CACHE_RESIDENCY_BUCKETS.iter().copied()) +} + +fn kv_block_idle_before_evict_histogram() -> Histogram { + Histogram::new(KV_CACHE_RESIDENCY_BUCKETS.iter().copied()) +} + +fn kv_block_reuse_gap_histogram() -> Histogram { + Histogram::new(KV_CACHE_RESIDENCY_BUCKETS.iter().copied()) +} + +#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)] +pub struct EngineLabels { + pub model_name: String, + pub engine: u32, +} + +#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)] +pub struct EnginePositionLabels { + pub model_name: String, + pub engine: u32, + pub position: u32, +} + +#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)] +pub struct WaitingReasonLabels { + pub model_name: String, + pub engine: u32, + pub reason: &'static str, +} + +/// Adapter names encoded as a deterministic comma-joined Prometheus label value. +#[derive(Clone, Debug, Hash, PartialEq, Eq)] +pub struct LoraAdapterNames(pub BTreeSet); + +impl EncodeLabelValue for LoraAdapterNames { + fn encode(&self, encoder: &mut LabelValueEncoder) -> Result<(), std::fmt::Error> { + EncodeLabelValue::encode(&self.0.iter().join(","), encoder) + } +} + +/// Labels for `vllm:lora_requests_info`. +#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)] +pub struct LoraInfoLabels { + pub running_lora_adapters: LoraAdapterNames, + pub waiting_lora_adapters: LoraAdapterNames, +} + +/// CUDA graph sample key used for periodic text-log aggregation. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub struct CudagraphLogKey { + pub num_unpadded_tokens: u64, + pub num_padded_tokens: u64, + pub num_paddings: u64, + pub runtime_mode: String, +} + +/// Raw scheduler stats accumulated for one periodic text-log interval. +#[derive(Default)] +pub struct SchedulerLogStatsInterval { + pub spec_num_drafts: u64, + pub spec_accepted_tokens_per_pos: Vec, + pub cudagraph_counts: BTreeMap, +} + +impl SchedulerLogStatsInterval { + /// Merge another drained interval into this one. + pub fn merge(&mut self, other: Self) { + self.spec_num_drafts += other.spec_num_drafts; + + if self.spec_accepted_tokens_per_pos.len() < other.spec_accepted_tokens_per_pos.len() { + self.spec_accepted_tokens_per_pos + .resize(other.spec_accepted_tokens_per_pos.len(), 0); + } + for (position, accepted_tokens) in + other.spec_accepted_tokens_per_pos.into_iter().enumerate() + { + self.spec_accepted_tokens_per_pos[position] += accepted_tokens; + } + + for (key, count) in other.cudagraph_counts { + *self.cudagraph_counts.entry(key).or_default() += count; + } + } +} + +/// Internal, non-Prometheus accumulator for periodic text logs that need raw +/// scheduler DTOs. +#[derive(Clone, Default)] +pub struct SchedulerLogStatsAccumulator { + inner: Arc>, +} + +impl SchedulerLogStatsAccumulator { + /// Observe spec-decoding fields needed for per-position text-log rates. + pub fn observe_spec_decode(&self, num_drafts: u64, accepted_tokens_per_pos: &[u64]) { + let mut inner = self.inner.lock().expect("scheduler log stats accumulator poisoned"); + inner.spec_num_drafts += num_drafts; + + if inner.spec_accepted_tokens_per_pos.len() < accepted_tokens_per_pos.len() { + inner.spec_accepted_tokens_per_pos.resize(accepted_tokens_per_pos.len(), 0); + } + for (position, accepted_tokens) in accepted_tokens_per_pos.iter().copied().enumerate() { + inner.spec_accepted_tokens_per_pos[position] += accepted_tokens; + } + } + + /// Observe one CUDA graph runtime sample for the interval table. + pub fn observe_cudagraph( + &self, + num_unpadded_tokens: u64, + num_padded_tokens: u64, + num_paddings: u64, + runtime_mode: &str, + ) { + let mut inner = self.inner.lock().expect("scheduler log stats accumulator poisoned"); + let key = CudagraphLogKey { + num_unpadded_tokens, + num_padded_tokens, + num_paddings, + runtime_mode: runtime_mode.to_string(), + }; + *inner.cudagraph_counts.entry(key).or_default() += 1; + } + + /// Drain and reset the current text-log interval. + pub fn drain(&self) -> SchedulerLogStatsInterval { + let mut inner = self.inner.lock().expect("scheduler log stats accumulator poisoned"); + std::mem::take(&mut *inner) + } +} + +/// Scheduler/batch-scoped Prometheus families exported from `SchedulerStats`. +pub struct SchedulerMetrics { + // Scheduler state gauges. + pub scheduler_running: Family, + pub scheduler_waiting: Family, + pub scheduler_waiting_by_reason: Family, + pub kv_cache_usage: Family, + + /// `vllm:lora_requests_info`. Value is the emit-time unix timestamp in + /// seconds. + pub lora_info: Family, + + // Prefix-cache counters, including the connector-backed external cache path. + pub prefix_cache_queries: Family, + pub prefix_cache_hits: Family, + pub external_prefix_cache_queries: Family, + pub external_prefix_cache_hits: Family, + + // Speculative decoding counters. + pub spec_decode_num_drafts: Family, + pub spec_decode_num_draft_tokens: Family, + pub spec_decode_num_accepted_tokens: Family, + pub spec_decode_num_accepted_tokens_per_pos: Family, + + // Per-engine performance / MFU counters. + pub estimated_flops_per_gpu: Family, + pub estimated_read_bytes_per_gpu: Family, + pub estimated_write_bytes_per_gpu: Family, + + // Sampled KV-cache residency histograms. + pub kv_block_lifetime_seconds: HistogramFamily, + pub kv_block_idle_before_evict_seconds: HistogramFamily, + pub kv_block_reuse_gap_seconds: HistogramFamily, + + /// Non-Prometheus interval accumulators for periodic text-log helpers. + pub log_stats: Family, +} + +impl SchedulerMetrics { + /// Register the scheduler-oriented metric families into the shared + /// registry. + pub(crate) fn register(registry: &mut Registry) -> Self { + // Scheduler state gauges. + let scheduler_running = Family::default(); + registry.register( + "vllm:num_requests_running", + "Number of requests in model execution batches", + scheduler_running.clone(), + ); + + let scheduler_waiting = Family::default(); + registry.register( + "vllm:num_requests_waiting", + "Number of requests waiting to be processed", + scheduler_waiting.clone(), + ); + + let scheduler_waiting_by_reason = Family::default(); + registry.register( + "vllm:num_requests_waiting_by_reason", + "Number of waiting requests by reason. \ + Reason labels: 'capacity' = waiting for scheduling capacity; \ + 'deferred' = deferred by transient constraints (LoRA budget, KV transfer, \ + blocked status). Sum of all reasons equals vllm:num_requests_waiting.", + scheduler_waiting_by_reason.clone(), + ); + + let kv_cache_usage = Family::default(); + registry.register( + "vllm:kv_cache_usage_perc", + "KV-cache usage. 1 means 100 percent usage", + kv_cache_usage.clone(), + ); + + let lora_info = Family::default(); + registry.register( + "vllm:lora_requests_info", + "Running stats on lora requests.", + lora_info.clone(), + ); + + // Prefix-cache counters, including the connector-backed external cache path. + let prefix_cache_queries = Family::default(); + registry.register( + "vllm:prefix_cache_queries", + "Prefix cache queries, in terms of number of queried tokens", + prefix_cache_queries.clone(), + ); + + let prefix_cache_hits = Family::default(); + registry.register( + "vllm:prefix_cache_hits", + "Prefix cache hits, in terms of number of cached tokens.", + prefix_cache_hits.clone(), + ); + + let external_prefix_cache_queries = Family::default(); + registry.register( + "vllm:external_prefix_cache_queries", + "External prefix cache queries from KV connector cross-instance cache sharing, in terms of number of queried tokens.", + external_prefix_cache_queries.clone(), + ); + + let external_prefix_cache_hits = Family::default(); + registry.register( + "vllm:external_prefix_cache_hits", + "External prefix cache hits from KV connector cross-instance cache sharing, in terms of number of cached tokens.", + external_prefix_cache_hits.clone(), + ); + + // Speculative decoding counters. + let spec_decode_num_drafts = Family::default(); + registry.register( + "vllm:spec_decode_num_drafts", + "Number of spec decoding drafts.", + spec_decode_num_drafts.clone(), + ); + + let spec_decode_num_draft_tokens = Family::default(); + registry.register( + "vllm:spec_decode_num_draft_tokens", + "Number of draft tokens.", + spec_decode_num_draft_tokens.clone(), + ); + + let spec_decode_num_accepted_tokens = Family::default(); + registry.register( + "vllm:spec_decode_num_accepted_tokens", + "Number of accepted tokens.", + spec_decode_num_accepted_tokens.clone(), + ); + + let spec_decode_num_accepted_tokens_per_pos = Family::default(); + registry.register( + "vllm:spec_decode_num_accepted_tokens_per_pos", + "Accepted tokens per draft position.", + spec_decode_num_accepted_tokens_per_pos.clone(), + ); + + // Per-engine performance / MFU counters. + let estimated_flops_per_gpu = Family::default(); + registry.register( + "vllm:estimated_flops_per_gpu", + "Estimated number of floating point operations per GPU (for Model Flops Utilization calculations).", + estimated_flops_per_gpu.clone(), + ); + + let estimated_read_bytes_per_gpu = Family::default(); + registry.register( + "vllm:estimated_read_bytes_per_gpu", + "Estimated number of bytes read from memory per GPU (for Model Flops Utilization calculations).", + estimated_read_bytes_per_gpu.clone(), + ); + + let estimated_write_bytes_per_gpu = Family::default(); + registry.register( + "vllm:estimated_write_bytes_per_gpu", + "Estimated number of bytes written to memory per GPU (for Model Flops Utilization calculations).", + estimated_write_bytes_per_gpu.clone(), + ); + + // Sampled KV-cache residency histograms. + let kv_block_lifetime_seconds = + Family::new_with_constructor(kv_block_lifetime_histogram as fn() -> Histogram); + registry.register( + "vllm:kv_block_lifetime_seconds", + "Histogram of KV cache block lifetime from allocation to eviction. Sampled metrics (controlled by --kv-cache-metrics-sample).", + kv_block_lifetime_seconds.clone(), + ); + + let kv_block_idle_before_evict_seconds = + Family::new_with_constructor(kv_block_idle_before_evict_histogram as fn() -> Histogram); + registry.register( + "vllm:kv_block_idle_before_evict_seconds", + "Histogram of idle time before KV cache block eviction. Sampled metrics (controlled by --kv-cache-metrics-sample).", + kv_block_idle_before_evict_seconds.clone(), + ); + + let kv_block_reuse_gap_seconds = + Family::new_with_constructor(kv_block_reuse_gap_histogram as fn() -> Histogram); + registry.register( + "vllm:kv_block_reuse_gap_seconds", + "Histogram of time gaps between consecutive KV cache block accesses. Only the most recent accesses are recorded (ring buffer). Sampled metrics (controlled by --kv-cache-metrics-sample).", + kv_block_reuse_gap_seconds.clone(), + ); + + Self { + scheduler_running, + scheduler_waiting, + scheduler_waiting_by_reason, + kv_cache_usage, + lora_info, + prefix_cache_queries, + prefix_cache_hits, + external_prefix_cache_queries, + external_prefix_cache_hits, + spec_decode_num_drafts, + spec_decode_num_draft_tokens, + spec_decode_num_accepted_tokens, + spec_decode_num_accepted_tokens_per_pos, + estimated_flops_per_gpu, + estimated_read_bytes_per_gpu, + estimated_write_bytes_per_gpu, + kv_block_lifetime_seconds, + kv_block_idle_before_evict_seconds, + kv_block_reuse_gap_seconds, + log_stats: Family::default(), + } + } +} + +#[cfg(test)] +mod tests { + use crate::{CudagraphLogKey, EngineLabels, Metrics, SchedulerLogStatsAccumulator}; + + #[test] + fn perf_counters_render_with_a_single_total_suffix() { + let metrics = Metrics::new(); + let labels = EngineLabels { + model_name: "model".to_string(), + engine: 0, + }; + + metrics.scheduler.estimated_flops_per_gpu.get_or_create(&labels).inc(); + metrics.scheduler.estimated_read_bytes_per_gpu.get_or_create(&labels).inc(); + metrics.scheduler.estimated_write_bytes_per_gpu.get_or_create(&labels).inc(); + + let rendered = metrics.render().unwrap(); + assert!( + rendered.contains( + "vllm:estimated_flops_per_gpu_total{model_name=\"model\",engine=\"0\"} 1" + ) + ); + assert!(rendered.contains( + "vllm:estimated_read_bytes_per_gpu_total{model_name=\"model\",engine=\"0\"} 1" + )); + assert!(rendered.contains( + "vllm:estimated_write_bytes_per_gpu_total{model_name=\"model\",engine=\"0\"} 1" + )); + assert!(!rendered.contains("vllm:estimated_flops_per_gpu_total_total")); + assert!(!rendered.contains("vllm:estimated_read_bytes_per_gpu_total_total")); + assert!(!rendered.contains("vllm:estimated_write_bytes_per_gpu_total_total")); + } + + #[test] + fn log_stats_accumulator_drains_interval_data() { + let accumulator = SchedulerLogStatsAccumulator::default(); + + accumulator.observe_spec_decode(2, &[1, 2]); + accumulator.observe_spec_decode(3, &[3, 4, 5]); + accumulator.observe_cudagraph(8, 16, 8, "FULL"); + accumulator.observe_cudagraph(8, 16, 8, "FULL"); + + let interval = accumulator.drain(); + + assert_eq!(interval.spec_num_drafts, 5); + assert_eq!(interval.spec_accepted_tokens_per_pos, vec![4, 6, 5]); + assert_eq!( + interval + .cudagraph_counts + .get(&CudagraphLogKey { + num_unpadded_tokens: 8, + num_padded_tokens: 16, + num_paddings: 8, + runtime_mode: "FULL".to_string(), + }) + .copied(), + Some(2) + ); + assert_eq!(accumulator.drain().spec_num_drafts, 0); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/mock-engine/Cargo.toml b/asian_ai/performance/vLLM/rust/src/mock-engine/Cargo.toml new file mode 100644 index 0000000000000000000000000000000000000000..00f2b4178912d91a0019e19b2384b53e02079793 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/mock-engine/Cargo.toml @@ -0,0 +1,30 @@ +[package] +name = "vllm-mock-engine" +version.workspace = true +edition.workspace = true +license.workspace = true + +[[bin]] +name = "vllm-mock-engine" +path = "src/main.rs" + +[dependencies] +anyhow.workspace = true +asynk-strim-attr.workspace = true +clap.workspace = true +futures.workspace = true +rand.workspace = true +rmpv.workspace = true +serde.workspace = true +tokio = { workspace = true, features = ["signal"] } +tokio-util.workspace = true +tracing.workspace = true +tracing-subscriber.workspace = true +vllm-engine-core-client.workspace = true +zeromq.workspace = true + +[dev-dependencies] +vllm-engine-core-client = { workspace = true, features = ["test-util"] } + +[lints] +workspace = true diff --git a/asian_ai/performance/vLLM/rust/src/mock-engine/README.md b/asian_ai/performance/vLLM/rust/src/mock-engine/README.md new file mode 100644 index 0000000000000000000000000000000000000000..9bb93a47426191e7c8fb889773c0603805add419 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/mock-engine/README.md @@ -0,0 +1,104 @@ +# vLLM Mock Engine + +`vllm-mock-engine` is a small engine-side process for frontend stress testing. It +joins a frontend-owned startup handshake, reports a large ready response, treats +prefill as instant, and emits random decode tokens until each request reaches +its `max_tokens`. + +The frontend must own the handshake socket. Start the frontend first, then start +the mock engine with the same handshake address. + +## Start the mock engine + +```bash +cargo run -p vllm-mock-engine -- \ + --handshake-address tcp://127.0.0.1:29550 \ + --engine-count 1 \ + --output-token-chunk-size 1 \ + --vocab-size 32000 \ + --seed 0 \ + --log-requests +``` + +Useful knobs: + +- `--engine-count` must match the frontend's expected data-parallel engine + count. +- `--output-token-chunk-size` controls how many token IDs appear in one + `EngineCoreOutput`; values greater than 1 are useful for MTP/spec-decode + shaped frontend tests. +- `--vocab-size` should stay within the tokenizer vocabulary of the model used + by the frontend. + +Stop it with Ctrl-C. + +## Rust Frontend + +Terminal 1: + +```bash +cargo run --bin vllm-rs -- serve Qwen/Qwen3-0.6B \ + --data-parallel-size 1 \ + --data-parallel-size-local 0 \ + --handshake-port 29550 +``` + +Terminal 2: + +```bash +cargo run -p vllm-mock-engine -- \ + --handshake-address tcp://127.0.0.1:29550 +``` + +For multiple mock engines, set both sides to the same count: + +```bash +cargo run --bin vllm-rs -- serve Qwen/Qwen3-0.6B \ + --data-parallel-size 4 \ + --data-parallel-size-local 0 \ + --handshake-port 29550 + +cargo run -p vllm-mock-engine -- \ + --handshake-address tcp://127.0.0.1:29550 \ + --engine-count 4 +``` + +## Python Frontend + +Use `vllm serve` with `--data-parallel-size-local 0` so the Python process runs +as a frontend/API server and waits for external engines on +`--data-parallel-rpc-port`. + +Terminal 1: + +```bash +vllm serve Qwen/Qwen3-0.6B \ + --data-parallel-address 127.0.0.1 \ + --data-parallel-rpc-port 29550 \ + --data-parallel-size 1 \ + --data-parallel-size-local 0 +``` + +Terminal 2: + +```bash +cargo run -p vllm-mock-engine -- \ + --handshake-address tcp://127.0.0.1:29550 +``` + +## Smoke Request + +After either frontend is ready: + +```bash +curl http://127.0.0.1:8000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen/Qwen3-0.6B", + "messages": [{"role": "user", "content": "hello"}], + "max_tokens": 16, + "stream": true + }' +``` + +Always pass `max_tokens`; the mock engine stops by length. diff --git a/asian_ai/performance/vLLM/rust/src/mock-engine/src/engine.rs b/asian_ai/performance/vLLM/rust/src/mock-engine/src/engine.rs new file mode 100644 index 0000000000000000000000000000000000000000..5c32df55680c7bdf5a1197d34a5b0fd311c1b0cc --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/mock-engine/src/engine.rs @@ -0,0 +1,421 @@ +use std::collections::{BTreeMap, BTreeSet, HashMap}; +use std::hash::{Hash as _, Hasher as _}; +use std::time::{SystemTime, UNIX_EPOCH}; + +use anyhow::{Result, anyhow}; +use rand::rngs::StdRng; +use rand::{Rng as _, SeedableRng as _}; +use rmpv::Value; +use serde::Serialize; +use tokio::sync::mpsc; +use tokio::task::yield_now; +use tokio_util::sync::CancellationToken; +use tracing::{debug, info, warn}; +use vllm_engine_core_client::protocol::output::{ + EngineCoreFinishReason, EngineCoreOutput, EngineCoreOutputs, RequestBatchOutputs, + UtilityCallOutput, +}; +use vllm_engine_core_client::protocol::request::EngineCoreRequest; +use vllm_engine_core_client::protocol::utility::{ + EngineCoreUtilityRequest, UtilityOutput, UtilityResultEnvelope, +}; + +use super::Opt; + +/// Derive a stable per-request seed from the CLI seed, engine, and request id. +fn request_seed(base_seed: u64, engine_index: u32, request_id: &str) -> u64 { + let mut hasher = std::hash::DefaultHasher::new(); + base_seed.hash(&mut hasher); + engine_index.hash(&mut hasher); + request_id.hash(&mut hasher); + hasher.finish() +} + +/// Current UNIX timestamp in seconds for engine-core output envelopes. +fn now_secs() -> f64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|duration| duration.as_secs_f64()) + .unwrap_or_default() +} + +/// Build one request output with only token IDs and terminal status populated. +fn request_output( + request_id: String, + new_token_ids: Vec, + finish_reason: Option, +) -> EngineCoreOutput { + EngineCoreOutput { + request_id, + new_token_ids, + finish_reason, + ..Default::default() + } +} + +/// Produce an empty output with a terminal finish reason for an invalid request. +fn empty_finish_outputs( + engine_index: u32, + request_id: String, + finish_reason: EngineCoreFinishReason, +) -> EngineCoreOutputs { + let output = request_output(request_id, Vec::new(), Some(finish_reason)); + let finished_requests = BTreeSet::from([output.request_id.clone()]); + + RequestBatchOutputs { + engine_index, + outputs: vec![output], + timestamp: now_secs(), + finished_requests: Some(finished_requests), + ..Default::default() + } + .into() +} + +/// Encode a utility result into the protocol's msgpack value envelope. +fn utility_envelope(value: T) -> Result +where + T: Serialize, +{ + Ok(UtilityResultEnvelope::without_type_info( + rmpv::ext::to_value(value)?, + )) +} + +/// Produce the minimal utility responses needed by the Rust frontend. +fn utility_response( + engine_index: u32, + request: EngineCoreUtilityRequest, +) -> Result { + let result = match request.method_name.as_str() { + "get_supported_tasks" => utility_envelope(vec!["generate"]), + "is_sleeping" => utility_envelope(false), + "reset_prefix_cache" => utility_envelope(true), + "reset_mm_cache" + | "reset_encoder_cache" + | "profile" + | "sleep" + | "wake_up" + | "execute_dummy_batch" => utility_envelope(()), + _ => utility_envelope(Value::Nil), + }?; + + Ok(UtilityCallOutput { + engine_index, + timestamp: now_secs(), + output: UtilityOutput { + call_id: request.call_id, + failure_message: None, + result: Some(result), + }, + } + .into()) +} + +/// Message sent from the frontend to the mock engine task to drive the engine loop. +pub(crate) enum EngineInput { + Request(Box), + Abort(Vec), + Utility(EngineCoreUtilityRequest), + StartDpWave, +} + +/// Message sent from the mock engine task to the frontend for one engine output batch. +pub(crate) struct EngineOutput { + pub client_index: u32, + pub outputs: EngineCoreOutputs, +} + +/// Per-request decode state owned by one mock engine. +#[derive(Debug)] +struct ActiveRequest { + request_id: String, + client_index: u32, + prompt_len: usize, + max_tokens: usize, + generated: usize, + rng: StdRng, +} + +impl ActiveRequest { + /// Create a new active request from an incoming EngineCoreRequest, or return an immediate + /// finish reason if the request is invalid. + fn new( + engine_index: u32, + request: Box, + opt: &Opt, + ) -> Result { + let request_id = request.request_id; + let client_index = request.client_index; + let prompt_len = request.prompt_token_ids.as_ref().map(Vec::len).unwrap_or_default(); + + let Some(sampling_params) = request.sampling_params else { + warn!( + request_id, + "request has no sampling params; returning engine error" + ); + return Err(EngineCoreFinishReason::Error); + }; + let max_tokens = sampling_params.max_tokens as usize; + + if opt.log_requests { + info!( + request_id, + prompt_len, + max_tokens, + chunk_size = opt.output_token_chunk_size, + "mock request started" + ); + } + + if max_tokens == 0 { + return Err(EngineCoreFinishReason::Length); + } + + Ok(ActiveRequest { + rng: StdRng::seed_from_u64(request_seed(opt.seed, engine_index, &request_id)), + request_id, + client_index, + prompt_len, + max_tokens, + generated: 0, + }) + } + + /// Advance this request by one mock engine step. + fn step(&mut self, opt: &Opt) -> EngineCoreOutput { + let remaining = self.max_tokens - self.generated; + let chunk_len = remaining.min(opt.output_token_chunk_size); + let mut new_token_ids = Vec::with_capacity(chunk_len); + for _ in 0..chunk_len { + new_token_ids.push(self.rng.random_range(0..opt.vocab_size)); + } + self.generated += chunk_len; + + let finished = self.generated >= self.max_tokens; + request_output( + self.request_id.clone(), + new_token_ids, + finished.then_some(EngineCoreFinishReason::Length), + ) + } +} + +/// Internal state for one mock engine instance, owned by the engine loop task. +struct Engine { + engine_index: u32, + opt: Opt, + active_requests: HashMap, +} + +impl Engine { + /// Drain one frontend request message received on the input DEALER socket. + fn handle_input(&mut self, input: EngineInput) -> Result> { + let mut outputs = Vec::new(); + + match input { + EngineInput::Request(request) => { + let request_id = request.request_id.clone(); + let client_index = request.client_index; + + if self.active_requests.contains_key(&request_id) { + warn!( + engine_index = self.engine_index, + request_id, "duplicate mock request id" + ); + return Ok(vec![EngineOutput { + client_index, + outputs: empty_finish_outputs( + self.engine_index, + request_id, + EngineCoreFinishReason::Error, + ), + }]); + } + + match ActiveRequest::new(self.engine_index, request, &self.opt) { + Ok(request) => { + self.active_requests.insert(request_id, request); + } + Err(finish_reason) => { + return Ok(vec![EngineOutput { + client_index, + outputs: empty_finish_outputs( + self.engine_index, + request_id, + finish_reason, + ), + }]); + } + } + } + + EngineInput::Abort(request_ids) => { + let mut outputs_by_client = + BTreeMap::, BTreeSet)>::new(); + for request_id in request_ids { + if let Some(request) = self.active_requests.remove(&request_id) { + let output = request_output( + request_id.clone(), + Vec::new(), + Some(EngineCoreFinishReason::Abort), + ); + let (outputs, finished_requests) = outputs_by_client + .entry(request.client_index) + .or_insert_with(|| (Vec::new(), BTreeSet::new())); + outputs.push(output); + finished_requests.insert(request_id.clone()); + if self.opt.log_requests { + info!(request_id, finish_reason = "abort", "mock request aborted"); + } + } + } + for (client_index, (client_outputs, finished_requests)) in outputs_by_client { + outputs.push({ + let outputs = RequestBatchOutputs { + engine_index: self.engine_index, + outputs: client_outputs, + timestamp: now_secs(), + finished_requests: Some(finished_requests), + ..Default::default() + } + .into(); + EngineOutput { + client_index, + outputs, + } + }); + } + } + + EngineInput::Utility(request) => { + debug!( + engine_index = self.engine_index, + call_id = %request.call_id, + method = request.method_name, + "mock utility request" + ); + let client_index = request.client_index; + outputs.push({ + let outputs = utility_response(self.engine_index, request)?; + EngineOutput { + client_index, + outputs, + } + }); + } + + EngineInput::StartDpWave => { + debug!( + engine_index = self.engine_index, + "ignoring START_DP_WAVE in mock engine" + ); + } + } + + Ok(outputs) + } + + /// Advance active requests once and return one batched engine output. + fn step(&mut self) -> Vec { + if self.active_requests.is_empty() { + return Vec::new(); + } + + let mut outputs_by_client = + BTreeMap::, BTreeSet)>::new(); + let mut all_finished_requests = BTreeSet::new(); + + for request in self.active_requests.values_mut() { + let client_index = request.client_index; + let output = request.step(&self.opt); + let request_id = request.request_id.clone(); + let finished = output.finished(); + if output.finished() { + all_finished_requests.insert(request_id.clone()); + if self.opt.log_requests { + info!( + request_id, + prompt_len = request.prompt_len, + output_tokens = request.generated, + finish_reason = "length", + "mock request finished" + ); + } + } + let (outputs, finished_requests) = outputs_by_client + .entry(client_index) + .or_insert_with(|| (Vec::new(), BTreeSet::new())); + if finished { + finished_requests.insert(request_id.clone()); + } + outputs.push(output); + } + + for request_id in &all_finished_requests { + self.active_requests.remove(request_id); + } + + outputs_by_client + .into_iter() + .filter_map(|(client_index, (outputs, finished_requests))| { + (!outputs.is_empty()).then(|| EngineOutput { + client_index, + outputs: RequestBatchOutputs { + engine_index: self.engine_index, + outputs, + timestamp: now_secs(), + finished_requests: (!finished_requests.is_empty()) + .then_some(finished_requests), + ..Default::default() + } + .into(), + }) + }) + .collect() + } +} + +/// Run the main loop for the mock engine, receiving `EngineInput` from `input_rx` +/// and sending `EngineOutput` to `output_tx` until `shutdown` is cancelled. +pub(crate) async fn run_engine_loop( + engine_index: u32, + opt: Opt, + mut input_rx: mpsc::UnboundedReceiver, + output_tx: mpsc::Sender, + shutdown: CancellationToken, +) -> Result<()> { + let mut engine = Engine { + engine_index, + opt, + active_requests: HashMap::new(), + }; + + loop { + let outputs = tokio::select! { + biased; + _ = shutdown.cancelled() => break, + + input = input_rx.recv() => { + let input = input + .ok_or_else(|| anyhow!("mock engine input channel closed"))?; + engine.handle_input(input)? + } + + // If there are active requests, step them once after yielding to the scheduler to + // avoid blocking the engine loop while still making steady progress on request outputs. + _ = yield_now(), if !engine.active_requests.is_empty() => { + engine.step() + } + }; + + for output in outputs { + output_tx + .send(output) + .await + .map_err(|_| anyhow!("mock engine IO task shut down"))?; + } + } + + Ok(()) +} diff --git a/asian_ai/performance/vLLM/rust/src/mock-engine/src/io.rs b/asian_ai/performance/vLLM/rust/src/mock-engine/src/io.rs new file mode 100644 index 0000000000000000000000000000000000000000..6a8cb9192fd54ed7eade0a4bc6476f5685d4c07d --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/mock-engine/src/io.rs @@ -0,0 +1,120 @@ +use anyhow::{Context as _, Result, anyhow, bail}; +use futures::{Stream, StreamExt as _, stream}; +use tokio::sync::mpsc; +use tokio_util::sync::CancellationToken; +use tracing::warn; +use vllm_engine_core_client::mock_engine::MockEngineDataSockets; +use vllm_engine_core_client::protocol::request::{EngineCoreRequest, EngineCoreRequestType}; +use vllm_engine_core_client::protocol::utility::EngineCoreUtilityRequest; +use vllm_engine_core_client::protocol::{decode_msgpack, encode_msgpack}; +use zeromq::{DealerSocket, PushSocket, SocketRecv as _, SocketSend as _, ZmqMessage}; + +use crate::engine::{EngineInput, EngineOutput}; + +/// Send one engine output batch to the client over the appropriate push socket. +async fn send_engine_outputs_to_client( + push_sockets: &mut [PushSocket], + EngineOutput { + client_index, + outputs, + }: EngineOutput, +) -> Result<()> { + let message = ZmqMessage::from(encode_msgpack(&outputs)?); + push_sockets[client_index as usize].send(message).await?; + Ok(()) +} + +/// Create a stream of `EngineInput` by continuously receiving messages from the given dealer socket +/// and decoding them into `EngineInput`. +fn dealer_input_stream(dealer: DealerSocket) -> impl Stream> { + stream::unfold(dealer, |mut dealer| async { + let input = loop { + let message = + match dealer.recv().await.context("failed to receive message from dealer socket") { + Ok(message) => message, + Err(err) => break Err(err), + }; + + match decode_request(message) { + Ok(input) => break Ok(input), + Err(err) => { + warn!(%err, "failed to decode engine request message; ignoring"); + } + } + }; + + Some((input, dealer)) + }) +} + +/// Decode a `ZmqMessage` into an `EngineInput`. Returns an error if the message is malformed or +/// contains an unknown/unsupported request type. +fn decode_request(message: ZmqMessage) -> Result { + let frames = message.into_vec(); + if frames.is_empty() { + bail!("empty engine request message"); + } + if frames.len() != 2 { + bail!("invalid frame count for engine request: {}", frames.len()); + } + + let request_type_frame = frames[0].as_ref(); + let Some(request_type) = EngineCoreRequestType::from_frame(request_type_frame) else { + bail!("unknown engine request type: {:?}", request_type_frame); + }; + + let input = match request_type { + EngineCoreRequestType::Add => { + let request: Box = decode_msgpack(frames[1].as_ref())?; + EngineInput::Request(request) + } + EngineCoreRequestType::Abort => { + let request_ids: Vec = decode_msgpack(frames[1].as_ref())?; + EngineInput::Abort(request_ids) + } + EngineCoreRequestType::Utility => { + let request: EngineCoreUtilityRequest = decode_msgpack(frames[1].as_ref())?; + EngineInput::Utility(request) + } + EngineCoreRequestType::StartDpWave => EngineInput::StartDpWave, + }; + + Ok(input) +} + +/// Run the main IO loop for the mock engine, continuously receiving and decoding raw messages from +/// the dealer sockets, sending them to the engine loop task via `input_tx`, and receiving +/// `EngineOutput` from the engine loop task via `output_rx` and sending them to the client over the +/// appropriate push socket, until `shutdown` is cancelled. +pub(crate) async fn run_io_loop( + data_sockets: Vec, + input_tx: mpsc::UnboundedSender, + mut output_rx: mpsc::Receiver, + shutdown: CancellationToken, +) -> Result<()> { + let (dealers, mut push_sockets): (Vec<_>, Vec<_>) = + data_sockets.into_iter().map(|sockets| (sockets.dealer, sockets.push)).unzip(); + let mut input_streams = + stream::select_all(dealers.into_iter().map(dealer_input_stream).map(Box::pin)); + + loop { + tokio::select! { + biased; + _ = shutdown.cancelled() => return Ok(()), + + output = output_rx.recv() => { + let output = output + .ok_or_else(|| anyhow!("mock engine output channel closed"))?; + send_engine_outputs_to_client(&mut push_sockets, output).await?; + } + + input = input_streams.next() => { + let input = input + .ok_or_else(|| anyhow!("mock engine input streams closed"))??; + input_tx + .send(input) + .map_err(|_| anyhow!("mock engine state task shut down"))?; + } + } + } +} diff --git a/asian_ai/performance/vLLM/rust/src/mock-engine/src/lib.rs b/asian_ai/performance/vLLM/rust/src/mock-engine/src/lib.rs new file mode 100644 index 0000000000000000000000000000000000000000..33bbf6492f9895265e289a32c75554a9c5c8af1a --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/mock-engine/src/lib.rs @@ -0,0 +1,138 @@ +use anyhow::{Context, Result, bail}; +use clap::Parser; +use tokio::sync::mpsc; +use tokio::task::JoinSet; +use tokio_util::sync::CancellationToken; +use tracing::{error, info}; +use vllm_engine_core_client::EngineId; +use vllm_engine_core_client::mock_engine::{ + MockEngineConfig, MockEngineSockets, connect_to_frontend, +}; + +pub mod engine; +pub mod io; + +/// Standalone engine-core protocol emulator for frontend stress testing. +#[derive(Debug, Clone, Parser)] +#[command( + name = "vllm-mock-engine", + about = "Run a mock vLLM headless engine for Rust frontend stress testing." +)] +pub struct Opt { + /// Frontend-owned ZMQ handshake address. + #[arg(long, default_value = "tcp://127.0.0.1:29550")] + pub handshake_address: String, + + /// Number of mock engine identities to register with the frontend. + #[arg(long, default_value_t = 1)] + pub engine_count: usize, + + /// Number of accepted output tokens included in each EngineCoreOutput. + #[arg(long, default_value_t = 1)] + pub output_token_chunk_size: usize, + + /// Random token IDs are sampled uniformly from 0..vocab_size. + #[arg(long, default_value_t = 32_000)] + pub vocab_size: u32, + + /// Base seed for deterministic random token generation. + #[arg(long, default_value_t = 0)] + pub seed: u64, + + /// Log a summary line for each request. + #[arg(long)] + pub log_requests: bool, +} + +/// Run one mock engine until shutdown or transport failure. +async fn run_engine(engine_index: u32, opt: Opt, shutdown: CancellationToken) -> Result<()> { + let MockEngineSockets { data_sockets, .. } = connect_to_frontend( + &opt.handshake_address, + EngineId::from_engine_index(engine_index), + MockEngineConfig::default(), + ) + .await + .with_context(|| format!("mock engine {engine_index} failed to connect to frontend"))?; + + info!(engine_index, "mock engine connected to frontend"); + + let (input_tx, input_rx) = mpsc::unbounded_channel(); + let (output_tx, output_rx) = mpsc::channel(64); + + // IO loop: dealer -> input_tx, output_rx -> push + let mut io_loop = tokio::spawn(io::run_io_loop( + data_sockets, + input_tx, + output_rx, + shutdown.clone(), + )); + // Engine loop: input_rx -> engine logic -> output_tx + let mut engine_loop = tokio::spawn(engine::run_engine_loop( + engine_index, + opt, + input_rx, + output_tx, + shutdown.clone(), + )); + + tokio::select! { + biased; + _ = shutdown.cancelled() => { + io_loop.abort(); + engine_loop.abort(); + io_loop.await.ok(); + engine_loop.await.ok(); + } + + result = &mut io_loop => { + error!(engine_index, "mock engine IO loop exited unexpectedly"); + engine_loop.abort(); + engine_loop.await.ok(); + result??; + } + result = &mut engine_loop => { + error!(engine_index, "mock engine loop exited unexpectedly"); + io_loop.abort(); + io_loop.await.ok(); + result??; + } + } + + info!(engine_index, "mock engine shut down"); + Ok(()) +} + +/// Run all requested mock engines until cancellation or one engine task fails. +pub async fn run(opt: Opt, shutdown: CancellationToken) -> Result<()> { + info!(?opt, "starting mock engine"); + + let mut engines = JoinSet::new(); + for engine_index in 0..opt.engine_count { + engines.spawn(run_engine( + engine_index as u32, + opt.clone(), + shutdown.clone(), + )); + } + + tokio::select! { + biased; + _ = shutdown.cancelled() => { + engines.abort_all(); + while engines.join_next().await.is_some() {} + Ok(()) + } + + joined = engines.join_next() => { + match joined { + Some(Ok(Ok(()))) => bail!("mock engine exited unexpectedly"), + Some(Ok(Err(error))) => Err(error), + Some(Err(error)) => Err(error).context("mock engine task join failed"), + None => Ok(()), + } + } + } +} + +#[cfg(test)] +mod tests; diff --git a/asian_ai/performance/vLLM/rust/src/mock-engine/src/main.rs b/asian_ai/performance/vLLM/rust/src/mock-engine/src/main.rs new file mode 100644 index 0000000000000000000000000000000000000000..ff8582d0c5c66b5c8d6b46ab12837c5ff7cc6749 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/mock-engine/src/main.rs @@ -0,0 +1,38 @@ +use anyhow::{Context, Result}; +use clap::Parser as _; +use tokio_util::sync::CancellationToken; +use tracing::{Level, info}; +use vllm_mock_engine::Opt; + +fn init_tracing() { + tracing_subscriber::fmt().with_max_level(Level::INFO).init(); +} + +/// Create a cancellation token that is triggered by Ctrl-C. +fn shutdown_signal() -> CancellationToken { + let token = CancellationToken::new(); + let shutdown = token.clone(); + + tokio::spawn(async move { + tokio::signal::ctrl_c().await.expect("failed to install Ctrl-C signal handler"); + info!("received shutdown signal (Ctrl-C), shutting down..."); + shutdown.cancel(); + }); + + token +} + +fn main() -> Result<()> { + init_tracing(); + let opt = Opt::parse(); + + let runtime = tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .context("failed to build Tokio runtime")?; + + runtime.block_on(async move { + let shutdown = shutdown_signal(); + vllm_mock_engine::run(opt, shutdown).await + }) +} diff --git a/asian_ai/performance/vLLM/rust/src/mock-engine/src/tests.rs b/asian_ai/performance/vLLM/rust/src/mock-engine/src/tests.rs new file mode 100644 index 0000000000000000000000000000000000000000..8d91c504ce454332e597a2ca0f00c2a2b2cb02ad --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/mock-engine/src/tests.rs @@ -0,0 +1,198 @@ +use std::net::TcpListener; +use std::time::Duration; + +use anyhow::Result; +use futures::StreamExt as _; +use tokio::time::timeout; +use tokio_util::sync::CancellationToken; +use vllm_engine_core_client::protocol::output::EngineCoreFinishReason; +use vllm_engine_core_client::protocol::request::EngineCoreRequest; +use vllm_engine_core_client::protocol::sampling::EngineCoreSamplingParams; +use vllm_engine_core_client::test_utils::IpcNamespace; +use vllm_engine_core_client::{EngineCoreClient, EngineCoreClientConfig, TransportMode}; + +use crate::{Opt, run}; + +fn free_tcp_address() -> String { + let listener = TcpListener::bind("127.0.0.1:0").expect("bind free port"); + let port = listener.local_addr().expect("local addr").port(); + drop(listener); + format!("tcp://127.0.0.1:{port}") +} + +fn client_config(handshake_address: String, engine_count: usize) -> EngineCoreClientConfig { + EngineCoreClientConfig { + transport_mode: TransportMode::HandshakeOwner { + handshake_address, + advertised_host: "127.0.0.1".to_string(), + engine_count, + ready_timeout: Duration::from_secs(5), + local_input_address: None, + local_output_address: None, + }, + coordinator_mode: None, + model_name: "mock-model".to_string(), + client_index: 0, + } +} + +async fn connect_with_mock( + handshake_address: String, + engine_count: usize, + output_token_chunk_size: usize, +) -> ( + EngineCoreClient, + CancellationToken, + tokio::task::JoinHandle>, +) { + let shutdown = CancellationToken::new(); + let task = tokio::spawn(run( + Opt { + handshake_address: handshake_address.clone(), + engine_count, + output_token_chunk_size, + vocab_size: 32_000, + seed: 0, + log_requests: false, + }, + shutdown.clone(), + )); + + let client = timeout( + Duration::from_secs(5), + EngineCoreClient::connect(client_config(handshake_address, engine_count)), + ) + .await + .expect("client connect timeout") + .expect("connect client"); + + (client, shutdown, task) +} + +fn sample_request(request_id: &str, max_tokens: u32) -> EngineCoreRequest { + EngineCoreRequest { + request_id: request_id.to_string(), + prompt_token_ids: Some(vec![1, 2, 3]), + sampling_params: Some(EngineCoreSamplingParams { + max_tokens, + ..EngineCoreSamplingParams::for_test() + }), + arrival_time: 0.0, + ..Default::default() + } +} + +async fn shutdown_mock( + client: EngineCoreClient, + shutdown: CancellationToken, + task: tokio::task::JoinHandle>, +) { + client.shutdown().await.expect("client shutdown"); + shutdown.cancel(); + task.await.expect("mock join").expect("mock run"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn mock_engine_connects_over_tcp() { + let handshake_address = free_tcp_address(); + let (client, shutdown, task) = connect_with_mock(handshake_address, 1, 1).await; + assert_eq!(client.engine_count(), 1); + assert_eq!(client.engine_identities()[0], &[0, 0]); + assert_eq!(client.max_model_len(), 1024 * 1024); + assert_eq!(client.vllm_version(), "test-vllm-version"); + shutdown_mock(client, shutdown, task).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn mock_engine_connects_over_ipc() { + let ipc = IpcNamespace::new().expect("ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let (client, shutdown, task) = connect_with_mock(handshake_address, 1, 1).await; + assert_eq!(client.engine_count(), 1); + assert_eq!(client.engine_identities()[0], &[0, 0]); + shutdown_mock(client, shutdown, task).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn mock_engine_registers_multiple_identities() { + let handshake_address = free_tcp_address(); + let (client, shutdown, task) = connect_with_mock(handshake_address, 2, 1).await; + assert_eq!(client.engine_count(), 2); + assert_eq!(client.engine_identities(), vec![&[0, 0][..], &[1, 0][..]]); + shutdown_mock(client, shutdown, task).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn chunk_size_one_outputs_one_token_per_update() { + let handshake_address = free_tcp_address(); + let (client, shutdown, task) = connect_with_mock(handshake_address, 1, 1).await; + let mut stream = client.call(sample_request("req-1", 3)).await.expect("call"); + + let first = stream.next().await.expect("first").expect("first ok"); + assert_eq!(first.new_token_ids.len(), 1); + assert_eq!(first.finish_reason, None); + let second = stream.next().await.expect("second").expect("second ok"); + assert_eq!(second.new_token_ids.len(), 1); + assert_eq!(second.finish_reason, None); + let third = stream.next().await.expect("third").expect("third ok"); + assert_eq!(third.new_token_ids.len(), 1); + assert_eq!(third.finish_reason, Some(EngineCoreFinishReason::Length)); + assert!(stream.next().await.is_none()); + + shutdown_mock(client, shutdown, task).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn chunk_size_clips_final_output_to_max_tokens() { + let handshake_address = free_tcp_address(); + let (client, shutdown, task) = connect_with_mock(handshake_address, 1, 4).await; + let mut stream = client.call(sample_request("req-clip", 6)).await.expect("call"); + + let first = stream.next().await.expect("first").expect("first ok"); + assert_eq!(first.new_token_ids.len(), 4); + assert_eq!(first.finish_reason, None); + let second = stream.next().await.expect("second").expect("second ok"); + assert_eq!(second.new_token_ids.len(), 2); + assert_eq!(second.finish_reason, Some(EngineCoreFinishReason::Length)); + assert!(stream.next().await.is_none()); + + shutdown_mock(client, shutdown, task).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn abort_cancels_active_request_and_emits_terminal_output() { + let handshake_address = free_tcp_address(); + let (client, shutdown, task) = connect_with_mock(handshake_address, 1, 1).await; + let mut stream = client.call(sample_request("req-abort", 1_000_000)).await.expect("call"); + let first = stream.next().await.expect("first").expect("first ok"); + assert_eq!(first.finish_reason, None); + + client.abort(&["req-abort".to_string()]).await.expect("abort"); + + loop { + let output = timeout(Duration::from_secs(5), stream.next()) + .await + .expect("stream timeout") + .expect("terminal output") + .expect("output ok"); + if output.finish_reason.is_some() { + assert_eq!(output.finish_reason, Some(EngineCoreFinishReason::Abort)); + break; + } + } + + shutdown_mock(client, shutdown, task).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn utility_requests_return_minimal_success_responses() { + let handshake_address = free_tcp_address(); + let (client, shutdown, task) = connect_with_mock(handshake_address, 1, 1).await; + + assert!(!client.is_sleeping().await.expect("is sleeping")); + assert!(client.reset_prefix_cache(false, false).await.expect("reset prefix cache")); + client.reset_mm_cache().await.expect("reset mm cache"); + client.reset_encoder_cache().await.expect("reset encoder cache"); + + shutdown_mock(client, shutdown, task).await; +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/Cargo.toml b/asian_ai/performance/vLLM/rust/src/parser/Cargo.toml new file mode 100644 index 0000000000000000000000000000000000000000..1a3d242f02cb9b00d894583de5f11da642f6de78 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/Cargo.toml @@ -0,0 +1,84 @@ +[package] +name = "vllm-parser" +version.workspace = true +edition.workspace = true +license.workspace = true + +[features] +test-util = [] + +[dependencies] +easy-ext.workspace = true +serde.workspace = true +serde_json.workspace = true +thiserror.workspace = true +thiserror-ext.workspace = true +vllm-tokenizer.workspace = true +winnow.workspace = true +xgrammar-structural-tag.workspace = true + +[dev-dependencies] +criterion.workspace = true +expect-test.workspace = true +futures.workspace = true +openai-protocol.workspace = true +tool-parser.workspace = true +vllm-tokenizer = { workspace = true, features = ["test-utils"] } + +[[bench]] +name = "deepseek_v3" +harness = false +required-features = ["test-util"] + +[[bench]] +name = "deepseek_v31" +harness = false +required-features = ["test-util"] + +[[bench]] +name = "deepseek_v32" +harness = false +required-features = ["test-util"] + +[[bench]] +name = "qwen3_coder" +harness = false +required-features = ["test-util"] + +[[bench]] +name = "qwen3_xml" +harness = false +required-features = ["test-util"] + +[[bench]] +name = "llama3_json" +harness = false +required-features = ["test-util"] + +[[bench]] +name = "minimax_m2" +harness = false +required-features = ["test-util"] + +[[bench]] +name = "glm45_moe" +harness = false +required-features = ["test-util"] + +[[bench]] +name = "kimi_k2" +harness = false +required-features = ["test-util"] + +[[bench]] +name = "gemma4" +harness = false +required-features = ["test-util"] + +[[bench]] +name = "granite4" +harness = false +required-features = ["test-util"] + +[lints] +workspace = true diff --git a/asian_ai/performance/vLLM/rust/src/parser/benches/deepseek_v3.rs b/asian_ai/performance/vLLM/rust/src/parser/benches/deepseek_v3.rs new file mode 100644 index 0000000000000000000000000000000000000000..d01d70738dd78e6ae4b6838fcd3cc0b2495692c7 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/benches/deepseek_v3.rs @@ -0,0 +1,130 @@ +use std::time::Duration; + +use criterion::{BatchSize, Criterion, Throughput, black_box, criterion_group, criterion_main}; +use tool_parser::parsers::DeepSeekParser as ExternalDeepSeekParser; +use vllm_parser::tool::test_utils::{split_by_chars, test_tools}; +use vllm_parser::tool::{DeepSeekV3ToolParser, Tool, ToolParser}; + +mod utils; +use utils::{feed_external_parser, feed_parser, openai_tools}; + +const CHUNK_CHARS: usize = 7; +const LONG_NORMAL_TEXT_REPEATS: usize = 2048; + +fn mixed_fixture() -> String { + concat!( + "I will check two cities before answering.\n", + "<|tool▁calls▁begin|>", + "<|tool▁call▁begin|>function<|tool▁sep|>get_weather\n", + "```json\n", + "{\"location\":\"Hangzhou\",\"days\":3}", + "\n```<|tool▁call▁end|>", + "<|tool▁call▁begin|>function<|tool▁sep|>get_weather\n", + "```json\n", + "{\"location\":\"San Francisco\",\"days\":2}", + "\n```<|tool▁call▁end|>", + "<|tool▁calls▁end|>", + ) + .to_string() +} + +fn long_normal_text_fixture() -> String { + let line = "This is ordinary assistant text with no DeepSeek V3 tool markers at all.\n"; + line.repeat(LONG_NORMAL_TEXT_REPEATS) +} + +fn native_parser(tools: &[Tool]) -> Box { + DeepSeekV3ToolParser::create(tools).expect("DeepSeek V3 parser should initialize") +} + +fn run_stream_group( + c: &mut Criterion, + name: &str, + tools: &[Tool], + text: &str, + chunk_chars: usize, + expected_normal_text: &str, + expected_native_calls_len: usize, +) { + let chunks = split_by_chars(text, chunk_chars); + let openai_tools = openai_tools(tools); + + let mut group = c.benchmark_group(name); + group.sample_size(50); + group.warm_up_time(Duration::from_millis(300)); + group.measurement_time(Duration::from_secs(2)); + group.throughput(Throughput::Bytes(text.len() as u64)); + + group.bench_function("native_reuse_parser", |b| { + let mut parser = native_parser(tools); + b.iter(|| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }) + }); + + group.bench_function("native_create_parser", |b| { + b.iter_batched( + || native_parser(tools), + |mut parser| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.bench_function("external_reuse_parser", |b| { + let mut parser = ExternalDeepSeekParser::new(); + b.iter(|| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(&chunks)); + black_box(result); + }) + }); + + group.bench_function("external_create_parser", |b| { + b.iter_batched( + ExternalDeepSeekParser::new, + |mut parser| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(&chunks)); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.finish(); +} + +fn bench_deepseek_v3(c: &mut Criterion) { + let tools = test_tools(); + let mixed_text = mixed_fixture(); + let long_normal_text = long_normal_text_fixture(); + + run_stream_group( + c, + "deepseek_v3/mixed_text_tool_call", + &tools, + &mixed_text, + CHUNK_CHARS, + "I will check two cities before answering.\n", + 2, + ); + + run_stream_group( + c, + "deepseek_v3/long_normal_text", + &tools, + &long_normal_text, + CHUNK_CHARS, + &long_normal_text, + 0, + ); +} + +criterion_group!(benches, bench_deepseek_v3); +criterion_main!(benches); diff --git a/asian_ai/performance/vLLM/rust/src/parser/benches/deepseek_v31.rs b/asian_ai/performance/vLLM/rust/src/parser/benches/deepseek_v31.rs new file mode 100644 index 0000000000000000000000000000000000000000..6781b6094c54c7fb6c25b29dfb637091cd303b6e --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/benches/deepseek_v31.rs @@ -0,0 +1,128 @@ +use std::time::Duration; + +use criterion::{BatchSize, Criterion, Throughput, black_box, criterion_group, criterion_main}; +use tool_parser::parsers::DeepSeek31Parser as ExternalDeepSeek31Parser; +use vllm_parser::tool::test_utils::{split_by_chars, test_tools}; +use vllm_parser::tool::{DeepSeekV31ToolParser, Tool, ToolParser}; + +mod utils; +use utils::{feed_external_parser, feed_parser, openai_tools}; + +const CHUNK_CHARS: usize = 7; +const LONG_NORMAL_TEXT_REPEATS: usize = 2048; + +fn mixed_fixture() -> String { + concat!( + "I will check two cities before answering.\n", + "<|tool▁calls▁begin|>", + "<|tool▁call▁begin|>get_weather<|tool▁sep|>", + "{\"location\":\"Hangzhou\",\"days\":3}", + "<|tool▁call▁end|>", + "<|tool▁call▁begin|>get_weather<|tool▁sep|>", + "{\"location\":\"San Francisco\",\"days\":2}", + "<|tool▁call▁end|>", + "<|tool▁calls▁end|>", + ) + .to_string() +} + +fn long_normal_text_fixture() -> String { + let line = "This is ordinary assistant text with no DeepSeek V3.1 tool markers at all.\n"; + line.repeat(LONG_NORMAL_TEXT_REPEATS) +} + +fn native_parser(tools: &[Tool]) -> Box { + DeepSeekV31ToolParser::create(tools).expect("DeepSeek V3.1 parser should initialize") +} + +fn run_stream_group( + c: &mut Criterion, + name: &str, + tools: &[Tool], + text: &str, + chunk_chars: usize, + expected_normal_text: &str, + expected_native_calls_len: usize, +) { + let chunks = split_by_chars(text, chunk_chars); + let openai_tools = openai_tools(tools); + + let mut group = c.benchmark_group(name); + group.sample_size(50); + group.warm_up_time(Duration::from_millis(300)); + group.measurement_time(Duration::from_secs(2)); + group.throughput(Throughput::Bytes(text.len() as u64)); + + group.bench_function("native_reuse_parser", |b| { + let mut parser = native_parser(tools); + b.iter(|| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }) + }); + + group.bench_function("native_create_parser", |b| { + b.iter_batched( + || native_parser(tools), + |mut parser| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.bench_function("external_reuse_parser", |b| { + let mut parser = ExternalDeepSeek31Parser::new(); + b.iter(|| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(&chunks)); + black_box(result); + }) + }); + + group.bench_function("external_create_parser", |b| { + b.iter_batched( + ExternalDeepSeek31Parser::new, + |mut parser| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(&chunks)); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.finish(); +} + +fn bench_deepseek_v31(c: &mut Criterion) { + let tools = test_tools(); + let mixed_text = mixed_fixture(); + let long_normal_text = long_normal_text_fixture(); + + run_stream_group( + c, + "deepseek_v31/mixed_text_tool_call", + &tools, + &mixed_text, + CHUNK_CHARS, + "I will check two cities before answering.\n", + 2, + ); + + run_stream_group( + c, + "deepseek_v31/long_normal_text", + &tools, + &long_normal_text, + CHUNK_CHARS, + &long_normal_text, + 0, + ); +} + +criterion_group!(benches, bench_deepseek_v31); +criterion_main!(benches); diff --git a/asian_ai/performance/vLLM/rust/src/parser/benches/deepseek_v32.rs b/asian_ai/performance/vLLM/rust/src/parser/benches/deepseek_v32.rs new file mode 100644 index 0000000000000000000000000000000000000000..1e5c6c56bf4085b3f30a9e3f2360dc5cb8ff9857 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/benches/deepseek_v32.rs @@ -0,0 +1,113 @@ +use std::time::Duration; + +use criterion::{BatchSize, Criterion, Throughput, black_box, criterion_group, criterion_main}; +use vllm_parser::tool::test_utils::{split_by_chars, test_tools}; +use vllm_parser::tool::{DeepSeekV32ToolParser, Tool, ToolParser}; + +mod utils; +use utils::feed_parser; + +const CHUNK_CHARS: usize = 7; +const LONG_NORMAL_TEXT_REPEATS: usize = 2048; + +fn mixed_fixture() -> String { + concat!( + "I will check two cities before answering.\n", + "<|DSML|function_calls>\n", + "<|DSML|invoke name=\"get_weather\">\n", + "<|DSML|parameter name=\"location\" string=\"true\">Hangzhou\n", + "<|DSML|parameter name=\"date\" string=\"true\">2026-04-28\n", + "<|DSML|parameter name=\"unit\" string=\"true\">celsius\n", + "<|DSML|parameter name=\"days\" string=\"false\">3\n", + "\n", + "<|DSML|invoke name=\"get_weather\">\n", + "<|DSML|parameter name=\"location\" string=\"true\">San Francisco\n", + "<|DSML|parameter name=\"date\" string=\"true\">2026-04-28\n", + "<|DSML|parameter name=\"unit\" string=\"true\">fahrenheit\n", + "<|DSML|parameter name=\"days\" string=\"false\">2\n", + "\n", + "", + ) + .to_string() +} + +fn long_normal_text_fixture() -> String { + let line = "This is ordinary assistant text with no DSML tool markers at all.\n"; + line.repeat(LONG_NORMAL_TEXT_REPEATS) +} + +fn parser(tools: &[Tool]) -> Box { + DeepSeekV32ToolParser::create(tools).expect("DeepSeek V3.2 parser should initialize") +} + +fn run_stream_group( + c: &mut Criterion, + name: &str, + tools: &[Tool], + text: &str, + chunk_chars: usize, + expected_normal_text: &str, + expected_calls_len: usize, +) { + let chunks = split_by_chars(text, chunk_chars); + + let mut group = c.benchmark_group(name); + group.sample_size(50); + group.warm_up_time(Duration::from_millis(300)); + group.measurement_time(Duration::from_secs(2)); + group.throughput(Throughput::Bytes(text.len() as u64)); + + group.bench_function("reuse_parser", |b| { + let mut parser = parser(tools); + b.iter(|| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_calls_len); + black_box(result); + }) + }); + + group.bench_function("create_parser", |b| { + b.iter_batched( + || parser(tools), + |mut parser| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_calls_len); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.finish(); +} + +fn bench_deepseek_v32(c: &mut Criterion) { + let tools = test_tools(); + let mixed_text = mixed_fixture(); + let long_normal_text = long_normal_text_fixture(); + + run_stream_group( + c, + "deepseek_v32/mixed_text_tool_call", + &tools, + &mixed_text, + CHUNK_CHARS, + "I will check two cities before answering.\n", + 2, + ); + + run_stream_group( + c, + "deepseek_v32/long_normal_text", + &tools, + &long_normal_text, + CHUNK_CHARS, + &long_normal_text, + 0, + ); +} + +criterion_group!(benches, bench_deepseek_v32); +criterion_main!(benches); diff --git a/asian_ai/performance/vLLM/rust/src/parser/benches/gemma4.rs b/asian_ai/performance/vLLM/rust/src/parser/benches/gemma4.rs new file mode 100644 index 0000000000000000000000000000000000000000..1e8b267e1c8dca1d3c8d817f3b7bce88b2182d76 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/benches/gemma4.rs @@ -0,0 +1,157 @@ +use std::time::Duration; + +use criterion::{BatchSize, Criterion, Throughput, black_box, criterion_group, criterion_main}; +use vllm_parser::tool::test_utils::{split_by_chars, test_tools}; +use vllm_parser::tool::{Tool, ToolParser}; +use vllm_parser::unified::Gemma4UnifiedParser; + +mod utils; +use utils::{UnifiedToolParserAdapter, feed_parser}; + +const CHUNK_CHARS: usize = 7; +const LONG_NORMAL_TEXT_REPEATS: usize = 2048; +const LONG_TOOL_ARGUMENT_REPEATS: usize = 256; + +fn mixed_fixture() -> String { + concat!( + "I will inspect the data before answering.\n", + "<|tool_call>", + "call:convert{", + "whole:114.514,", + "flag:true,", + "empty:<|\"|><|\"|>,", + "payload:{", + "name:<|\"|>demo<|\"|>,", + "count:42,", + "enabled:false,", + "missing:null,", + "nested:{level:2,label:<|\"|>deep<|\"|>},", + "tags:[<|\"|>red<|\"|>,<|\"|>blue<|\"|>,3,true,null,{kind:<|\"|>leaf<|\"|>}]", + "},", + "items:[", + "<|\"|>alpha<|\"|>,", + "{key:<|\"|>value<|\"|>,score:0.75},", + "[1,2,3]", + "]", + "}", + "", + "<|tool_call>", + "call:update_record{", + "data:{id:7,active:true,notes:[<|\"|>keep<|\"|>,<|\"|>review<|\"|>]}", + "}", + "", + " Finished.", + ) + .to_string() +} + +fn long_normal_text_fixture() -> String { + let line = "This is ordinary assistant text with no Gemma4 tool markers at all.\n"; + line.repeat(LONG_NORMAL_TEXT_REPEATS) +} + +fn long_tool_argument_fixture() -> String { + let line = + "

Literal } and marker-shaped text inside content.

\n"; + format!( + concat!( + "I will write the file.\n", + "<|tool_call>", + "call:write_file{{", + "path:<|\"|>index.html<|\"|>,", + "content:<|\"|>{}<|\"|>", + "}}", + "", + "Done." + ), + line.repeat(LONG_TOOL_ARGUMENT_REPEATS) + ) +} + +fn parser(tools: &[Tool]) -> Box { + UnifiedToolParserAdapter::::create(tools) + .expect("Gemma4 unified parser should initialize") +} + +fn run_stream_group( + c: &mut Criterion, + name: &str, + tools: &[Tool], + text: &str, + chunk_chars: usize, + expected_normal_text: &str, + expected_calls_len: usize, +) { + let chunks = split_by_chars(text, chunk_chars); + + let mut group = c.benchmark_group(name); + group.sample_size(50); + group.warm_up_time(Duration::from_millis(300)); + group.measurement_time(Duration::from_secs(2)); + group.throughput(Throughput::Bytes(text.len() as u64)); + + group.bench_function("reuse_parser", |b| { + let mut parser = parser(tools); + b.iter(|| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_calls_len); + black_box(result); + }) + }); + + group.bench_function("create_parser", |b| { + b.iter_batched( + || parser(tools), + |mut parser| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_calls_len); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.finish(); +} + +fn bench_gemma4(c: &mut Criterion) { + let tools = test_tools(); + let mixed_text = mixed_fixture(); + let long_tool_argument = long_tool_argument_fixture(); + let long_normal_text = long_normal_text_fixture(); + + run_stream_group( + c, + "gemma4/mixed_complex_tool_call", + &tools, + &mixed_text, + CHUNK_CHARS, + "I will inspect the data before answering.\n Finished.", + 2, + ); + + run_stream_group( + c, + "gemma4/long_tool_argument", + &tools, + &long_tool_argument, + CHUNK_CHARS, + "I will write the file.\nDone.", + 1, + ); + + run_stream_group( + c, + "gemma4/long_normal_text", + &tools, + &long_normal_text, + CHUNK_CHARS, + &long_normal_text, + 0, + ); +} + +criterion_group!(benches, bench_gemma4); +criterion_main!(benches); diff --git a/asian_ai/performance/vLLM/rust/src/parser/benches/glm45_moe.rs b/asian_ai/performance/vLLM/rust/src/parser/benches/glm45_moe.rs new file mode 100644 index 0000000000000000000000000000000000000000..154a324d436dac0609a9ad653d8931f07df0c581 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/benches/glm45_moe.rs @@ -0,0 +1,210 @@ +use std::time::Duration; + +use criterion::{BatchSize, Criterion, Throughput, black_box, criterion_group, criterion_main}; +use tool_parser::parsers::Glm4MoeParser as ExternalGlm4MoeParser; +use vllm_parser::tool::test_utils::{split_by_chars, test_tools}; +use vllm_parser::tool::{Glm45MoeToolParser, Glm47MoeToolParser, Tool, ToolParser}; + +mod utils; +use utils::{feed_external_parser, feed_parser, openai_tools}; + +const GLM45_PARSER_NAME: &str = "glm45"; +const GLM47_PARSER_NAME: &str = "glm47"; +const CHUNK_CHARS: usize = 7; +const LONG_NORMAL_TEXT_REPEATS: usize = 2048; + +fn glm45_mixed_fixture() -> String { + concat!( + "I will check two cities before answering.\n", + "get_weather\n", + "city\n", + "Hangzhou\n", + "date\n", + "2026-05-07\n", + "unit\n", + "celsius\n", + "days\n", + "3\n", + "\n", + "get_weather\n", + "city\n", + "San Francisco\n", + "date\n", + "2026-05-07\n", + "unit\n", + "fahrenheit\n", + "days\n", + "2\n", + "", + ) + .to_string() +} + +fn glm47_mixed_fixture() -> String { + concat!( + "I will check two cities before answering.\n", + "get_weather", + "city", + "Hangzhou", + "date", + "2026-05-07", + "unit", + "celsius", + "days", + "3", + "", + "get_weather", + "city", + "San Francisco", + "date", + "2026-05-07", + "unit", + "fahrenheit", + "days", + "2", + "", + ) + .to_string() +} + +fn long_normal_text_fixture() -> String { + let line = "This is ordinary assistant text with no GLM MoE tool markers at all.\n"; + line.repeat(LONG_NORMAL_TEXT_REPEATS) +} + +fn native_parser(name: &str, tools: &[Tool]) -> Box { + match name { + GLM45_PARSER_NAME => Glm45MoeToolParser::create(tools), + GLM47_PARSER_NAME => Glm47MoeToolParser::create(tools), + _ => unreachable!("unexpected GLM parser name"), + } + .expect("GLM MoE parser should initialize") +} + +fn external_parser(name: &str) -> ExternalGlm4MoeParser { + match name { + GLM45_PARSER_NAME => ExternalGlm4MoeParser::glm45(), + GLM47_PARSER_NAME => ExternalGlm4MoeParser::glm47(), + _ => unreachable!("unexpected GLM parser name"), + } +} + +fn run_stream_group( + c: &mut Criterion, + name: &str, + parser_name: &str, + tools: &[Tool], + text: &str, + chunk_chars: usize, + expected_normal_text: &str, + expected_native_calls_len: usize, +) { + let chunks = split_by_chars(text, chunk_chars); + let openai_tools = openai_tools(tools); + + let mut group = c.benchmark_group(name); + group.sample_size(50); + group.warm_up_time(Duration::from_millis(300)); + group.measurement_time(Duration::from_secs(2)); + group.throughput(Throughput::Bytes(text.len() as u64)); + + group.bench_function("native_reuse_parser", |b| { + let mut parser = native_parser(parser_name, tools); + b.iter(|| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }) + }); + + group.bench_function("native_create_parser", |b| { + b.iter_batched( + || native_parser(parser_name, tools), + |mut parser| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.bench_function("external_reuse_parser", |b| { + let mut parser = external_parser(parser_name); + b.iter(|| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + black_box(result); + }) + }); + + group.bench_function("external_create_parser", |b| { + b.iter_batched( + || external_parser(parser_name), + |mut parser| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.finish(); +} + +fn bench_glm45_moe(c: &mut Criterion) { + let tools = test_tools(); + let glm45_mixed_text = glm45_mixed_fixture(); + let glm47_mixed_text = glm47_mixed_fixture(); + let long_normal_text = long_normal_text_fixture(); + + run_stream_group( + c, + "glm45/mixed_text_tool_call", + GLM45_PARSER_NAME, + &tools, + &glm45_mixed_text, + CHUNK_CHARS, + "I will check two cities before answering.\n", + 2, + ); + + run_stream_group( + c, + "glm47/mixed_text_tool_call", + GLM47_PARSER_NAME, + &tools, + &glm47_mixed_text, + CHUNK_CHARS, + "I will check two cities before answering.\n", + 2, + ); + + run_stream_group( + c, + "glm45/long_normal_text", + GLM45_PARSER_NAME, + &tools, + &long_normal_text, + CHUNK_CHARS, + &long_normal_text, + 0, + ); + + run_stream_group( + c, + "glm47/long_normal_text", + GLM47_PARSER_NAME, + &tools, + &long_normal_text, + CHUNK_CHARS, + &long_normal_text, + 0, + ); +} + +criterion_group!(benches, bench_glm45_moe); +criterion_main!(benches); diff --git a/asian_ai/performance/vLLM/rust/src/parser/benches/granite4.rs b/asian_ai/performance/vLLM/rust/src/parser/benches/granite4.rs new file mode 100644 index 0000000000000000000000000000000000000000..2b3bb1b2fc4774253d2f93a4c64c95ba1007f2ae --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/benches/granite4.rs @@ -0,0 +1,75 @@ +use std::time::Duration; + +use criterion::{BatchSize, Criterion, Throughput, black_box, criterion_group, criterion_main}; +use vllm_parser::tool::test_utils::{split_by_chars, test_tools}; +use vllm_parser::tool::{Granite4ToolParser, Tool, ToolParser}; + +mod utils; +use utils::feed_parser; + +const CHUNK_CHARS: usize = 7; +const LONG_ARGUMENT_BYTES: usize = 64 * 1024; + +fn string_args_fixture() -> String { + let arguments = format!(r#"{{"data":"{}"}}"#, "x".repeat(LONG_ARGUMENT_BYTES)); + let encoded_arguments = serde_json::to_string(&arguments).unwrap(); + format!(r#"{{"name":"f","arguments":{encoded_arguments}}}"#) +} + +fn object_args_fixture() -> String { + format!( + r#"{{"name":"f","arguments":{{"data":"{}"}}}}"#, + "x".repeat(LONG_ARGUMENT_BYTES) + ) +} + +fn parser(tools: &[Tool]) -> Box { + Granite4ToolParser::create(tools).expect("Granite4 parser should initialize") +} + +fn run_stream_group(c: &mut Criterion, name: &str, tools: &[Tool], text: &str) { + let chunks = split_by_chars(text, CHUNK_CHARS); + + let mut group = c.benchmark_group(name); + group.sample_size(50); + group.warm_up_time(Duration::from_millis(300)); + group.measurement_time(Duration::from_secs(2)); + group.throughput(Throughput::Bytes(text.len() as u64)); + + group.bench_function("reuse_parser", |b| { + let mut parser = parser(tools); + b.iter(|| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, ""); + debug_assert_eq!(result.1, 1); + black_box(result); + }) + }); + + group.bench_function("create_parser", |b| { + b.iter_batched( + || parser(tools), + |mut parser| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, ""); + debug_assert_eq!(result.1, 1); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.finish(); +} + +fn bench_granite4(c: &mut Criterion) { + let tools = test_tools(); + let string_args = string_args_fixture(); + let object_args = object_args_fixture(); + + run_stream_group(c, "granite4/long_string_arguments", &tools, &string_args); + run_stream_group(c, "granite4/long_object_arguments", &tools, &object_args); +} + +criterion_group!(benches, bench_granite4); +criterion_main!(benches); diff --git a/asian_ai/performance/vLLM/rust/src/parser/benches/kimi_k2.rs b/asian_ai/performance/vLLM/rust/src/parser/benches/kimi_k2.rs new file mode 100644 index 0000000000000000000000000000000000000000..04bb5f91c434e310d0f29b37247b6ab36da3e541 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/benches/kimi_k2.rs @@ -0,0 +1,149 @@ +use std::time::Duration; + +use criterion::{BatchSize, Criterion, Throughput, black_box, criterion_group, criterion_main}; +use tool_parser::parsers::KimiK2Parser as ExternalKimiK2Parser; +use vllm_parser::tool::test_utils::{split_by_chars, test_tools}; +use vllm_parser::tool::{KimiK2ToolParser, Tool, ToolParser}; + +mod utils; +use utils::{feed_external_parser, feed_parser, openai_tools}; + +const CHUNK_CHARS: usize = 7; +const LONG_NORMAL_TEXT_REPEATS: usize = 2048; + +fn mixed_fixture() -> String { + concat!( + "I will check two cities before answering.\n", + "<|tool_calls_section_begin|>", + "<|tool_call_begin|>functions.get_weather:0", + "<|tool_call_argument_begin|>{\"location\":\"Hangzhou\",\"days\":3}", + "<|tool_call_end|>", + "<|tool_call_begin|>functions.get_weather:1", + "<|tool_call_argument_begin|>{\"location\":\"San Francisco\",\"days\":2}", + "<|tool_call_end|>", + "<|tool_calls_section_end|>", + ) + .to_string() +} + +fn mixed_chunks() -> Vec<&'static str> { + vec![ + "I will check two cities before answering.\n", + "<|tool_calls_section_begin|>", + "<|tool_call_begin|>functions.get_weather:0", + "<|tool_call_argument_begin|>", + "{\"location\":", + "\"Hangzhou\",", + "\"days\":3}", + "<|tool_call_end|>", + "<|tool_call_begin|>functions.get_weather:1", + "<|tool_call_argument_begin|>", + "{\"location\":", + "\"San Francisco\",", + "\"days\":2}", + "<|tool_call_end|>", + "<|tool_calls_section_end|>", + ] +} + +fn long_normal_text_fixture() -> String { + let line = "This is ordinary assistant text with no Kimi K2 tool markers at all.\n"; + line.repeat(LONG_NORMAL_TEXT_REPEATS) +} + +fn native_parser(tools: &[Tool]) -> Box { + KimiK2ToolParser::create(tools).expect("Kimi K2 parser should initialize") +} + +fn run_stream_group( + c: &mut Criterion, + name: &str, + tools: &[Tool], + text: &str, + chunks: &[&str], + expected_normal_text: &str, + expected_native_calls_len: usize, +) { + let openai_tools = openai_tools(tools); + + let mut group = c.benchmark_group(name); + group.sample_size(50); + group.warm_up_time(Duration::from_millis(300)); + group.measurement_time(Duration::from_secs(2)); + group.throughput(Throughput::Bytes(text.len() as u64)); + + group.bench_function("native_reuse_parser", |b| { + let mut parser = native_parser(tools); + b.iter(|| { + let result = feed_parser(&mut *parser, black_box(chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }) + }); + + group.bench_function("native_create_parser", |b| { + b.iter_batched( + || native_parser(tools), + |mut parser| { + let result = feed_parser(&mut *parser, black_box(chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.bench_function("external_reuse_parser", |b| { + let mut parser = ExternalKimiK2Parser::new(); + b.iter(|| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(chunks)); + black_box(result); + }) + }); + + group.bench_function("external_create_parser", |b| { + b.iter_batched( + ExternalKimiK2Parser::new, + |mut parser| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(chunks)); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.finish(); +} + +fn bench_kimi_k2(c: &mut Criterion) { + let tools = test_tools(); + let mixed_text = mixed_fixture(); + let mixed_chunks = mixed_chunks(); + let long_normal_text = long_normal_text_fixture(); + let long_normal_chunks = split_by_chars(&long_normal_text, CHUNK_CHARS); + + run_stream_group( + c, + "kimi_k2/mixed_text_tool_call", + &tools, + &mixed_text, + &mixed_chunks, + "I will check two cities before answering.\n", + 2, + ); + + run_stream_group( + c, + "kimi_k2/long_normal_text", + &tools, + &long_normal_text, + &long_normal_chunks, + &long_normal_text, + 0, + ); +} + +criterion_group!(benches, bench_kimi_k2); +criterion_main!(benches); diff --git a/asian_ai/performance/vLLM/rust/src/parser/benches/llama3_json.rs b/asian_ai/performance/vLLM/rust/src/parser/benches/llama3_json.rs new file mode 100644 index 0000000000000000000000000000000000000000..5997782ea4e522f7862da5452ae9d349dc28faa9 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/benches/llama3_json.rs @@ -0,0 +1,128 @@ +use std::time::Duration; + +use criterion::{BatchSize, Criterion, Throughput, black_box, criterion_group, criterion_main}; +use tool_parser::parsers::LlamaParser as ExternalLlamaParser; +use vllm_parser::tool::test_utils::{split_by_chars, test_tools}; +use vllm_parser::tool::{Llama3JsonToolParser, Tool, ToolParser}; + +mod utils; +use utils::{feed_external_parser, feed_parser, openai_tools}; + +const CHUNK_CHARS: usize = 7; +const LONG_NORMAL_TEXT_REPEATS: usize = 2048; + +fn tool_call(function_name: &str, parameters: &str) -> String { + format!(r#"{{"name":"{function_name}","parameters":{parameters}}}"#) +} + +fn mixed_fixture() -> String { + format!( + "{}; {}", + tool_call("get_weather", r#"{"location":"Hangzhou","days":3}"#), + tool_call( + "convert", + r#"{"whole":42.5,"flag":true,"payload":{"nested":["x",null]},"items":[1,2,3],"empty":""}"# + ), + ) +} + +fn long_normal_text_fixture() -> String { + let line = "This is ordinary assistant text with no Llama JSON tool call at the root.\n"; + line.repeat(LONG_NORMAL_TEXT_REPEATS) +} + +fn native_parser(tools: &[Tool]) -> Box { + Llama3JsonToolParser::create(tools).expect("Llama JSON parser should initialize") +} + +fn run_stream_group( + c: &mut Criterion, + name: &str, + tools: &[Tool], + text: &str, + chunk_chars: usize, + expected_normal_text: &str, + expected_native_calls_len: usize, +) { + let chunks = split_by_chars(text, chunk_chars); + let openai_tools = openai_tools(tools); + + let mut group = c.benchmark_group(name); + group.sample_size(50); + group.warm_up_time(Duration::from_millis(300)); + group.measurement_time(Duration::from_secs(2)); + group.throughput(Throughput::Bytes(text.len() as u64)); + + group.bench_function("native_reuse_parser", |b| { + let mut parser = native_parser(tools); + b.iter(|| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }) + }); + + group.bench_function("native_create_parser", |b| { + b.iter_batched( + || native_parser(tools), + |mut parser| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.bench_function("external_reuse_parser", |b| { + let mut parser = ExternalLlamaParser::new(); + b.iter(|| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(&chunks)); + black_box(result); + }) + }); + + group.bench_function("external_create_parser", |b| { + b.iter_batched( + ExternalLlamaParser::new, + |mut parser| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(&chunks)); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.finish(); +} + +fn bench_llama3_json(c: &mut Criterion) { + let tools = test_tools(); + let mixed_text = mixed_fixture(); + let long_normal_text = long_normal_text_fixture(); + + run_stream_group( + c, + "llama3_json/mixed_text_tool_call", + &tools, + &mixed_text, + CHUNK_CHARS, + "", + 2, + ); + + run_stream_group( + c, + "llama3_json/long_normal_text", + &tools, + &long_normal_text, + CHUNK_CHARS, + &long_normal_text, + 0, + ); +} + +criterion_group!(benches, bench_llama3_json); +criterion_main!(benches); diff --git a/asian_ai/performance/vLLM/rust/src/parser/benches/minimax_m2.rs b/asian_ai/performance/vLLM/rust/src/parser/benches/minimax_m2.rs new file mode 100644 index 0000000000000000000000000000000000000000..ef05cf464ad9741c7212af578e39c30d1b02199f --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/benches/minimax_m2.rs @@ -0,0 +1,136 @@ +use std::time::Duration; + +use criterion::{BatchSize, Criterion, Throughput, black_box, criterion_group, criterion_main}; +use tool_parser::parsers::MinimaxM2Parser as ExternalMinimaxM2Parser; +use vllm_parser::tool::test_utils::{split_by_chars, test_tools}; +use vllm_parser::tool::{MinimaxM2ToolParser, Tool, ToolParser}; + +mod utils; +use utils::{feed_external_parser, feed_parser, openai_tools}; + +const CHUNK_CHARS: usize = 7; +const LONG_NORMAL_TEXT_REPEATS: usize = 2048; + +fn mixed_fixture() -> String { + concat!( + "I will check two cities before answering.\n", + "", + "", + "Hangzhou", + "2026-04-30", + "celsius", + "3", + "", + "", + "San Francisco", + "2026-04-30", + "fahrenheit", + "2", + "", + "", + ) + .to_string() +} + +fn long_normal_text_fixture() -> String { + let line = "This is ordinary assistant text with no MiniMax M2 tool markers at all.\n"; + line.repeat(LONG_NORMAL_TEXT_REPEATS) +} + +fn native_parser(tools: &[Tool]) -> Box { + MinimaxM2ToolParser::create(tools).expect("MiniMax M2 parser should initialize") +} + +fn run_stream_group( + c: &mut Criterion, + name: &str, + tools: &[Tool], + text: &str, + chunk_chars: usize, + expected_normal_text: &str, + expected_native_calls_len: usize, +) { + let chunks = split_by_chars(text, chunk_chars); + let openai_tools = openai_tools(tools); + + let mut group = c.benchmark_group(name); + group.sample_size(50); + group.warm_up_time(Duration::from_millis(300)); + group.measurement_time(Duration::from_secs(2)); + group.throughput(Throughput::Bytes(text.len() as u64)); + + group.bench_function("native_reuse_parser", |b| { + let mut parser = native_parser(tools); + b.iter(|| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }) + }); + + group.bench_function("native_create_parser", |b| { + b.iter_batched( + || native_parser(tools), + |mut parser| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.bench_function("external_reuse_parser", |b| { + let mut parser = ExternalMinimaxM2Parser::new(); + b.iter(|| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + black_box(result); + }) + }); + + group.bench_function("external_create_parser", |b| { + b.iter_batched( + ExternalMinimaxM2Parser::new, + |mut parser| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.finish(); +} + +fn bench_minimax_m2(c: &mut Criterion) { + let tools = test_tools(); + let mixed_text = mixed_fixture(); + let long_normal_text = long_normal_text_fixture(); + + run_stream_group( + c, + "minimax_m2/mixed_text_tool_call", + &tools, + &mixed_text, + CHUNK_CHARS, + "I will check two cities before answering.\n", + 2, + ); + + run_stream_group( + c, + "minimax_m2/long_normal_text", + &tools, + &long_normal_text, + CHUNK_CHARS, + &long_normal_text, + 0, + ); +} + +criterion_group!(benches, bench_minimax_m2); +criterion_main!(benches); diff --git a/asian_ai/performance/vLLM/rust/src/parser/benches/qwen3_coder.rs b/asian_ai/performance/vLLM/rust/src/parser/benches/qwen3_coder.rs new file mode 100644 index 0000000000000000000000000000000000000000..a5311784538e7283a2fabc4ad88471b41e12c4cd --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/benches/qwen3_coder.rs @@ -0,0 +1,161 @@ +use std::time::Duration; + +use criterion::{BatchSize, Criterion, Throughput, black_box, criterion_group, criterion_main}; +use tool_parser::parsers::QwenCoderParser as ExternalQwenCoderParser; +use vllm_parser::tool::test_utils::{split_by_chars, test_tools}; +use vllm_parser::tool::{Qwen3CoderToolParser, Tool, ToolParser}; + +mod utils; +use utils::{feed_external_parser, feed_parser, openai_tools}; + +const CHUNK_CHARS: usize = 7; +const LONG_NORMAL_TEXT_REPEATS: usize = 2048; +const LONG_TOOL_BODY_REPEATS: usize = 8192; + +fn mixed_fixture() -> String { + concat!( + "I will check two cities before answering.\n", + "\n", + "\n", + "Hangzhou\n", + "2026-04-29\n", + "celsius\n", + "3\n", + "\n", + "\n", + "\n", + "\n", + "San Francisco\n", + "2026-04-29\n", + "fahrenheit\n", + "2\n", + "\n", + "", + ) + .to_string() +} + +fn long_normal_text_fixture() -> String { + let line = "This is ordinary assistant text with no Qwen Coder tool markers at all.\n"; + line.repeat(LONG_NORMAL_TEXT_REPEATS) +} + +fn long_tool_call_fixture() -> String { + let location = "x".repeat(LONG_TOOL_BODY_REPEATS); + format!( + "\n\ + \n\ + {location}\n\ + \n\ + " + ) +} + +fn native_parser(tools: &[Tool]) -> Box { + Qwen3CoderToolParser::create(tools).expect("Qwen Coder parser should initialize") +} + +fn run_stream_group( + c: &mut Criterion, + name: &str, + tools: &[Tool], + text: &str, + chunk_chars: usize, + expected_normal_text: &str, + expected_native_calls_len: usize, +) { + let chunks = split_by_chars(text, chunk_chars); + let openai_tools = openai_tools(tools); + + let mut group = c.benchmark_group(name); + group.sample_size(50); + group.warm_up_time(Duration::from_millis(300)); + group.measurement_time(Duration::from_secs(2)); + group.throughput(Throughput::Bytes(text.len() as u64)); + + group.bench_function("native_reuse_parser", |b| { + let mut parser = native_parser(tools); + b.iter(|| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }) + }); + + group.bench_function("native_create_parser", |b| { + b.iter_batched( + || native_parser(tools), + |mut parser| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.bench_function("external_reuse_parser", |b| { + let mut parser = ExternalQwenCoderParser::new(); + b.iter(|| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + black_box(result); + }) + }); + + group.bench_function("external_create_parser", |b| { + b.iter_batched( + ExternalQwenCoderParser::new, + |mut parser| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.finish(); +} + +fn bench_qwen3_coder(c: &mut Criterion) { + let tools = test_tools(); + let mixed_text = mixed_fixture(); + let long_normal_text = long_normal_text_fixture(); + let long_tool_call = long_tool_call_fixture(); + + run_stream_group( + c, + "qwen3_coder/mixed_text_tool_call", + &tools, + &mixed_text, + CHUNK_CHARS, + "I will check two cities before answering.\n", + 2, + ); + + run_stream_group( + c, + "qwen3_coder/long_normal_text", + &tools, + &long_normal_text, + CHUNK_CHARS, + &long_normal_text, + 0, + ); + + run_stream_group( + c, + "qwen3_coder/long_tool_call_body", + &tools, + &long_tool_call, + CHUNK_CHARS, + "", + 1, + ); +} + +criterion_group!(benches, bench_qwen3_coder); +criterion_main!(benches); diff --git a/asian_ai/performance/vLLM/rust/src/parser/benches/qwen3_xml.rs b/asian_ai/performance/vLLM/rust/src/parser/benches/qwen3_xml.rs new file mode 100644 index 0000000000000000000000000000000000000000..8d712ea95a98b82466fe42d98d801ef858bab6ca --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/benches/qwen3_xml.rs @@ -0,0 +1,125 @@ +use std::time::Duration; + +use criterion::{BatchSize, Criterion, Throughput, black_box, criterion_group, criterion_main}; +use tool_parser::parsers::QwenParser as ExternalQwenParser; +use vllm_parser::tool::test_utils::{split_by_chars, test_tools}; +use vllm_parser::tool::{Qwen3XmlToolParser, Tool, ToolParser}; + +mod utils; +use utils::{feed_external_parser, feed_parser, openai_tools}; + +const CHUNK_CHARS: usize = 7; +const LONG_NORMAL_TEXT_REPEATS: usize = 2048; + +fn tool_call(function_name: &str, arguments: &str) -> String { + format!("\n{{\"name\":\"{function_name}\",\"arguments\":{arguments}}}\n") +} + +fn mixed_fixture() -> String { + format!( + "I will check two cities before answering.\n{}{}", + tool_call("get_weather", r#"{"location":"Hangzhou","days":3}"#), + tool_call("get_weather", r#"{"location":"San Francisco","days":2}"#), + ) +} + +fn long_normal_text_fixture() -> String { + let line = "This is ordinary assistant text with no Qwen XML tool markers at all.\n"; + line.repeat(LONG_NORMAL_TEXT_REPEATS) +} + +fn native_parser(tools: &[Tool]) -> Box { + Qwen3XmlToolParser::create(tools).expect("Qwen XML parser should initialize") +} + +fn run_stream_group( + c: &mut Criterion, + name: &str, + tools: &[Tool], + text: &str, + chunk_chars: usize, + expected_normal_text: &str, + expected_native_calls_len: usize, +) { + let chunks = split_by_chars(text, chunk_chars); + let openai_tools = openai_tools(tools); + + let mut group = c.benchmark_group(name); + group.sample_size(50); + group.warm_up_time(Duration::from_millis(300)); + group.measurement_time(Duration::from_secs(2)); + group.throughput(Throughput::Bytes(text.len() as u64)); + + group.bench_function("native_reuse_parser", |b| { + let mut parser = native_parser(tools); + b.iter(|| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }) + }); + + group.bench_function("native_create_parser", |b| { + b.iter_batched( + || native_parser(tools), + |mut parser| { + let result = feed_parser(&mut *parser, black_box(&chunks)); + debug_assert_eq!(result.0, expected_normal_text); + debug_assert_eq!(result.1, expected_native_calls_len); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.bench_function("external_reuse_parser", |b| { + let mut parser = ExternalQwenParser::new(); + b.iter(|| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(&chunks)); + black_box(result); + }) + }); + + group.bench_function("external_create_parser", |b| { + b.iter_batched( + ExternalQwenParser::new, + |mut parser| { + let result = feed_external_parser(&mut parser, &openai_tools, black_box(&chunks)); + black_box(result); + }, + BatchSize::SmallInput, + ) + }); + + group.finish(); +} + +fn bench_qwen3_xml(c: &mut Criterion) { + let tools = test_tools(); + let mixed_text = mixed_fixture(); + let long_normal_text = long_normal_text_fixture(); + + run_stream_group( + c, + "qwen3_xml/mixed_text_tool_call", + &tools, + &mixed_text, + CHUNK_CHARS, + "I will check two cities before answering.\n", + 2, + ); + + run_stream_group( + c, + "qwen3_xml/long_normal_text", + &tools, + &long_normal_text, + CHUNK_CHARS, + &long_normal_text, + 0, + ); +} + +criterion_group!(benches, bench_qwen3_xml); +criterion_main!(benches); diff --git a/asian_ai/performance/vLLM/rust/src/parser/benches/utils/adapter.rs b/asian_ai/performance/vLLM/rust/src/parser/benches/utils/adapter.rs new file mode 100644 index 0000000000000000000000000000000000000000..991ce266f8eb6c9482627c0d68f001117cc3dc0a --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/benches/utils/adapter.rs @@ -0,0 +1,110 @@ +use std::sync::Arc; + +use vllm_parser::tool::{ + Result, StructuralTagModel, Tool, ToolParser, ToolParserError, ToolParserOutput, +}; +use vllm_parser::unified::{ + UnifiedParser, UnifiedParserError, UnifiedParserEvent, UnifiedParserOutput, +}; +use vllm_tokenizer::Tokenizer; + +/// Tokenizer stub used by unified-parser benchmarks. +struct BenchTokenizer; + +impl Tokenizer for BenchTokenizer { + fn encode(&self, text: &str, _add_special_tokens: bool) -> vllm_tokenizer::Result> { + Ok(text.chars().map(|_| u32::MAX).collect()) + } + + fn decode( + &self, + token_ids: &[u32], + _skip_special_tokens: bool, + ) -> vllm_tokenizer::Result { + Ok("\u{FFFD}".repeat(token_ids.len())) + } + + fn token_to_id(&self, _token: &str) -> Option { + Some(u32::MAX) + } + + fn id_to_token(&self, _id: u32) -> Option { + Some("\u{FFFD}".to_string()) + } +} + +/// Bench-only adapter that exposes a unified parser through the tool-parser +/// benchmark harness. +/// +/// Returns error if the unified parser produces reasoning events. +pub struct UnifiedToolParserAdapter { + inner: Box, + _marker: std::marker::PhantomData, +} + +fn map_unified_error(error: UnifiedParserError) -> ToolParserError { + ToolParserError::ParsingFailed { + message: format!("unified parser failed: {error}"), + } +} + +fn append_unified_output( + output: UnifiedParserOutput, + tool_output: &mut ToolParserOutput, +) -> Result<()> { + for event in output.events { + match event { + UnifiedParserEvent::Text(text) => tool_output.push_text(text), + UnifiedParserEvent::ToolCall(call) => tool_output.push_call(call), + UnifiedParserEvent::Reasoning(_) => { + return Err(ToolParserError::ParsingFailed { + message: "unified parser emitted reasoning in tool-parser adapter".to_string(), + }); + } + } + } + Ok(()) +} + +impl ToolParser for UnifiedToolParserAdapter { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + let inner = T::create(tools, Arc::new(BenchTokenizer)).map_err(map_unified_error)?; + Ok(Box::new(Self { + inner, + _marker: std::marker::PhantomData, + })) + } + + fn preserve_special_tokens(&self) -> bool { + self.inner.preserve_special_tokens() + } + + fn structural_tag_model(&self) -> Option { + self.inner.structural_tag_model() + } + + fn tool_call_id(&self, tool_index: usize) -> Option<&str> { + self.inner.tool_call_id(tool_index) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + let mut unified_output = UnifiedParserOutput::default(); + let result = self.inner.parse_into(chunk, &mut unified_output).map_err(map_unified_error); + append_unified_output(unified_output, output)?; + result + } + + fn finish(&mut self) -> Result { + let unified_output = self.inner.finish().map_err(map_unified_error)?; + let mut output = ToolParserOutput::default(); + append_unified_output(unified_output, &mut output)?; + Ok(output) + } + + fn reset(&mut self) -> String { + self.inner.reset() + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/benches/utils/mod.rs b/asian_ai/performance/vLLM/rust/src/parser/benches/utils/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..1bf1f5f16d0076a0f9752286980a2c90d0e4810a --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/benches/utils/mod.rs @@ -0,0 +1,55 @@ +// This module is shared by multiple benchmark targets. +// There could be false positives for unused code or imports, and fixing them would lead to some other benchmarks failing to compile. +#![allow(dead_code)] +#![allow(unused_imports)] + +mod adapter; + +pub(super) use adapter::UnifiedToolParserAdapter; +use futures::FutureExt as _; +use openai_protocol::common::{Function as OpenAiFunction, Tool as OpenAiTool}; +use tool_parser::traits::ToolParser as ExternalToolParser; +use vllm_parser::tool::test_utils::collect_stream; +use vllm_parser::tool::{Tool, ToolParser}; + +pub(super) fn openai_tools(tools: &[Tool]) -> Vec { + tools + .iter() + .map(|tool| OpenAiTool { + tool_type: "function".to_string(), + function: OpenAiFunction { + name: tool.name.clone(), + description: tool.description.clone(), + parameters: tool.parameters.clone(), + strict: tool.strict, + }, + }) + .collect() +} + +pub(super) fn feed_parser(parser: &mut dyn ToolParser, chunks: &[&str]) -> (String, usize) { + let result = collect_stream(parser, chunks); + (result.normal_text(), result.calls().len()) +} + +pub(super) fn feed_external_parser( + parser: &mut impl ExternalToolParser, + tools: &[OpenAiTool], + chunks: &[&str], +) -> (String, usize) { + ExternalToolParser::reset(parser); + + let mut normal_text = String::new(); + let mut calls_len = 0; + for chunk in chunks { + let delta = parser + .parse_incremental(chunk, tools) + .now_or_never() + .expect("external parser should not suspend") + .expect("chunk should parse"); + normal_text.push_str(&delta.normal_text); + calls_len += delta.calls.len(); + } + calls_len += parser.get_unstreamed_tool_args().unwrap_or_default().len(); + (normal_text, calls_len) +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/python/Cargo.toml b/asian_ai/performance/vLLM/rust/src/parser/python/Cargo.toml new file mode 100644 index 0000000000000000000000000000000000000000..f9f98b896449441e153ac74149c20aa1c0eecfb7 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/python/Cargo.toml @@ -0,0 +1,19 @@ +[package] +name = "vllm-tool-parser-py" +version.workspace = true +edition.workspace = true +license.workspace = true + +[lib] +name = "_rust_tool_parser" +crate-type = ["cdylib", "rlib"] + +[dependencies] +pyo3.workspace = true +pythonize = { workspace = true, features = ["serde_json"] } +serde_json.workspace = true +thiserror-ext.workspace = true +vllm-parser.workspace = true + +[lints] +workspace = true diff --git a/asian_ai/performance/vLLM/rust/src/parser/python/src/lib.rs b/asian_ai/performance/vLLM/rust/src/parser/python/src/lib.rs new file mode 100644 index 0000000000000000000000000000000000000000..3fb0bc4ee9bf92e50301ed922cf6af3366cd5174 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/python/src/lib.rs @@ -0,0 +1,395 @@ +//! Thin PyO3 bindings for `vllm_parser::tool`. +//! +//! This crate exposes the Rust tool parser trait and data shapes to Python +//! while keeping parser state, grammar, and schema-aware argument conversion in +//! Rust. Python callers should use this module as a typed bridge and keep any +//! vLLM protocol adaptation outside the binding. + +use pyo3::exceptions::PyValueError; +use pyo3::prelude::*; +use pyo3::types::{PyAny, PyModule}; +use pythonize::{depythonize, pythonize}; +use serde_json::Value; +use thiserror_ext::AsReport as _; +use vllm_parser::tool::{Tool, ToolCallDelta, ToolParser, ToolParserOutput}; + +macro_rules! tool_parser_factory { + ($($parser:ident),+ $(,)?) => { + fn create_tool_parser( + name: &str, + tools: &[Tool], + ) -> PyResult> { + match name { + $( + stringify!($parser) => { + ::create(tools) + } + )+ + _ => { + return Err(PyValueError::new_err(format!( + "unsupported tool parser `{name}`" + ))); + } + } + .map_err(|error| PyValueError::new_err(error.to_report_string())) + } + }; +} + +// Export a tool parser to Python by registering it here. +tool_parser_factory! { + MinimaxM3ToolParser, + + // Below are the parsers just for testing purposes on Python side. + DeepSeekV4ToolParser, + KimiK2ToolParser, +} + +#[pyclass(name = "Tool", module = "vllm._rust_tool_parser", skip_from_py_object)] +#[derive(Clone)] +struct PyTool(Tool); + +#[pymethods] +impl PyTool { + #[new] + #[pyo3(signature = (name, description, parameters, strict=None))] + fn new( + name: String, + description: Option, + parameters: &Bound<'_, PyAny>, + strict: Option, + ) -> PyResult { + let parameters = depythonize::(parameters).map_err(|error| { + PyValueError::new_err(format!( + "failed to convert tool parameters from Python to JSON: {error}" + )) + })?; + Ok(Self(Tool { + name, + description, + parameters, + strict, + })) + } + + #[getter] + fn name(&self) -> &str { + &self.0.name + } + + #[getter] + fn description(&self) -> Option<&str> { + self.0.description.as_deref() + } + + #[getter] + fn parameters(&self, py: Python<'_>) -> PyResult> { + pythonize(py, &self.0.parameters).map(Bound::unbind).map_err(|error| { + PyValueError::new_err(format!( + "failed to convert tool parameters from JSON to Python: {error}" + )) + }) + } + + #[getter] + fn strict(&self) -> Option { + self.0.strict + } +} + +#[pyclass( + name = "ToolCallDelta", + module = "vllm._rust_tool_parser", + skip_from_py_object +)] +#[derive(Clone)] +struct PyToolCallDelta(ToolCallDelta); + +#[pymethods] +impl PyToolCallDelta { + #[new] + #[pyo3(signature = (tool_index, name, arguments))] + fn new(tool_index: usize, name: Option, arguments: String) -> Self { + Self(ToolCallDelta { + tool_index, + name, + arguments, + }) + } + + #[getter] + fn tool_index(&self) -> usize { + self.0.tool_index + } + + #[getter] + fn name(&self) -> Option<&str> { + self.0.name.as_deref() + } + + #[getter] + fn arguments(&self) -> &str { + &self.0.arguments + } +} + +#[pyclass( + name = "ToolParserOutput", + module = "vllm._rust_tool_parser", + skip_from_py_object +)] +#[derive(Clone)] +struct PyToolParserOutput(ToolParserOutput); + +#[pymethods] +impl PyToolParserOutput { + #[new] + #[pyo3(signature = (normal_text="", calls=None))] + fn new(py: Python<'_>, normal_text: &str, calls: Option>>) -> Self { + let mut output = ToolParserOutput::default(); + output.push_text(normal_text); + for call in calls.unwrap_or_default() { + output.push_call(call.borrow(py).0.clone()); + } + Self(output) + } + + #[getter] + fn normal_text(&self) -> String { + self.0.normal_text() + } + + #[getter] + fn calls(&self) -> Vec { + self.0.calls().into_iter().cloned().map(PyToolCallDelta).collect() + } + + fn append(&mut self, other: PyRef<'_, PyToolParserOutput>) { + self.0.append(other.0.clone()); + } + + fn coalesce(&self) -> Self { + Self(self.0.clone().coalesce()) + } +} + +#[pyclass(name = "ToolParser", module = "vllm._rust_tool_parser", unsendable)] +struct PyToolParser(Box); + +impl PyToolParser { + fn parse_into_output(&mut self, chunk: &str, output: &mut PyToolParserOutput) -> PyResult<()> { + self.0 + .parse_into(chunk, &mut output.0) + .map_err(|error| PyValueError::new_err(error.to_report_string())) + } +} + +#[pymethods] +impl PyToolParser { + #[new] + fn new(py: Python<'_>, parser_name: &str, tools: Vec>) -> PyResult { + let tools = tools.iter().map(|tool| tool.borrow(py).0.clone()).collect::>(); + create_tool_parser(parser_name, &tools).map(Self) + } + + fn parse_into( + &mut self, + chunk: &str, + mut output: PyRefMut<'_, PyToolParserOutput>, + ) -> PyResult<()> { + self.parse_into_output(chunk, &mut output) + } + + fn finish(&mut self) -> PyResult { + self.0 + .finish() + .map(PyToolParserOutput) + .map_err(|error| PyValueError::new_err(error.to_report_string())) + } + + fn reset(&mut self) -> String { + self.0.reset() + } + + fn preserve_special_tokens(&self) -> bool { + self.0.preserve_special_tokens() + } + + fn tool_call_id(&self, tool_index: usize) -> Option<&str> { + self.0.tool_call_id(tool_index) + } +} + +#[pymodule] +fn _rust_tool_parser(m: &Bound<'_, PyModule>) -> PyResult<()> { + m.add_class::()?; + m.add_class::()?; + m.add_class::()?; + m.add_class::()?; + Ok(()) +} + +#[cfg(test)] +mod tests { + use serde_json::json; + + use super::*; + + fn with_python(f: impl for<'py> FnOnce(Python<'py>) -> R) -> R { + Python::initialize(); + Python::attach(f) + } + + fn tool_schema() -> Value { + json!({ + "type": "object", + "properties": { + "user_id": {"type": "integer"}, + "shipping": { + "type": "object", + "properties": { + "city": {"type": "string"}, + "zip": {"type": "integer"} + } + } + } + }) + } + + fn build_call() -> String { + r#"<|DSML|tool_calls> +<|DSML|invoke name="create_order"> +<|DSML|parameter name="user_id" string="false">42 +<|DSML|parameter name="shipping" string="false">{"city":"Singapore","zip":18956} + +"# + .to_owned() + } + + fn make_py_tool(py: Python<'_>) -> PyResult> { + let parameters = pythonize(py, &tool_schema()).map_err(|error| { + PyValueError::new_err(format!( + "failed to convert test schema from JSON to Python: {error}" + )) + })?; + Py::new( + py, + PyTool::new( + "create_order".to_owned(), + Some("Create an order".to_owned()), + ¶meters, + None, + )?, + ) + } + + #[test] + fn tool_round_trips_typed_fields() { + with_python(|py| { + let tool = make_py_tool(py)?; + let borrowed = tool.borrow(py); + assert_eq!(borrowed.name(), "create_order"); + assert_eq!(borrowed.description(), Some("Create an order")); + assert_eq!(borrowed.strict(), None); + + let parameters = borrowed.parameters(py)?; + let parameters = depythonize::(parameters.bind(py))?; + assert_eq!(parameters, tool_schema()); + PyResult::Ok(()) + }) + .unwrap(); + } + + #[test] + fn output_append_and_coalesce() { + with_python(|py| { + let first = Py::new( + py, + PyToolCallDelta::new(0, Some("create_order".to_owned()), "{\"a\"".to_owned()), + )?; + let second = Py::new(py, PyToolCallDelta::new(0, None, ":1}".to_owned()))?; + let mut output = PyToolParserOutput::new(py, "text", Some(vec![first])); + let other = Py::new(py, PyToolParserOutput::new(py, "", Some(vec![second])))?; + output.append(other.borrow(py)); + + let coalesced = output.coalesce(); + assert_eq!(coalesced.normal_text(), "text"); + let calls = coalesced.calls(); + assert_eq!(calls.len(), 1); + assert_eq!(calls[0].tool_index(), 0); + assert_eq!(calls[0].name(), Some("create_order")); + assert_eq!(calls[0].arguments(), "{\"a\":1}"); + PyResult::Ok(()) + }) + .unwrap(); + } + + #[test] + fn parser_parse_finish_and_preserve_special_tokens() { + with_python(|py| { + let tool = make_py_tool(py)?; + let mut parser = PyToolParser::new(py, "DeepSeekV4ToolParser", vec![tool])?; + assert!(parser.preserve_special_tokens()); + + let mut output = PyToolParserOutput::new(py, "", None); + parser.parse_into_output(&build_call(), &mut output)?; + let finish = Py::new(py, parser.finish()?)?; + output.append(finish.borrow(py)); + let output = output.coalesce(); + + assert_eq!(output.normal_text(), ""); + let calls = output.calls(); + assert_eq!(calls.len(), 1); + assert_eq!(calls[0].name(), Some("create_order")); + assert_eq!( + serde_json::from_str::(calls[0].arguments()).unwrap(), + json!({ + "user_id": 42, + "shipping": { + "city": "Singapore", + "zip": 18956 + } + }) + ); + + assert_eq!(parser.reset(), ""); + PyResult::Ok(()) + }) + .unwrap(); + } + + #[test] + fn parser_exposes_model_emitted_tool_call_ids() { + with_python(|py| { + let tool = make_py_tool(py)?; + let mut parser = PyToolParser::new(py, "KimiK2ToolParser", vec![tool])?; + + let input = "<|tool_calls_section_begin|>\ + <|tool_call_begin|>functions.create_order:0<|tool_call_argument_begin|>\ + {\"user_id\":42}<|tool_call_end|>\ + <|tool_calls_section_end|>"; + let mut output = PyToolParserOutput::new(py, "", None); + parser.parse_into_output(input, &mut output)?; + + assert_eq!(parser.tool_call_id(0), Some("functions.create_order:0")); + assert_eq!(parser.tool_call_id(1), None); + PyResult::Ok(()) + }) + .unwrap(); + } + + #[test] + fn parser_errors_for_unknown_name() { + with_python(|py| { + let tool = make_py_tool(py)?; + let error = match PyToolParser::new(py, "missing", vec![tool]) { + Ok(_) => panic!("missing parser name unexpectedly succeeded"), + Err(error) => error, + }; + let message = format!("{error}"); + assert!(message.contains("unsupported tool parser `missing`")); + PyResult::Ok(()) + }) + .unwrap(); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/lib.rs b/asian_ai/performance/vLLM/rust/src/parser/src/lib.rs new file mode 100644 index 0000000000000000000000000000000000000000..5be79419def3cb2ad0e7532ef4a181fcb423dcc1 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/lib.rs @@ -0,0 +1,6 @@ +//! Streaming parsers for chat completions. + +pub mod reasoning; +pub mod tool; +pub mod unified; +pub(crate) mod utils; diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/cohere_cmd.rs b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/cohere_cmd.rs new file mode 100644 index 0000000000000000000000000000000000000000..ac00c2c4a3d43ad134ee7e2ece2f2ccc1e712cfb --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/cohere_cmd.rs @@ -0,0 +1,45 @@ +use vllm_tokenizer::DynTokenizer; + +use super::{DelimitedReasoningParser, ReasoningDelta, ReasoningParser, Result}; + +/// Reasoning parser for Cohere Command models that use explicit START/END tags. +pub struct CohereCmdReasoningParser { + inner: DelimitedReasoningParser, +} + +impl CohereCmdReasoningParser { + /// Create a Cohere Command parser backed by the shared delimited state + /// machine. + pub fn new(tokenizer: DynTokenizer) -> Result { + Ok(Self { + inner: DelimitedReasoningParser::new( + tokenizer, + "<|START_THINKING|>", + "<|END_THINKING|>", + false, + )?, + }) + } +} + +impl ReasoningParser for CohereCmdReasoningParser { + fn create(tokenizer: DynTokenizer) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tokenizer)?)) + } + + fn initialize(&mut self, prompt_token_ids: &[u32]) -> Result<()> { + self.inner.initialize(prompt_token_ids); + Ok(()) + } + + fn push(&mut self, delta: &str) -> Result { + Ok(self.inner.push(delta)) + } + + fn finish(&mut self) -> Result { + Ok(self.inner.finish()) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/deepseek_r1.rs b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/deepseek_r1.rs new file mode 100644 index 0000000000000000000000000000000000000000..b59163a35bbc48eb18e441f6bacb0f7f45cf79e0 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/deepseek_r1.rs @@ -0,0 +1,44 @@ +use vllm_tokenizer::DynTokenizer; + +use super::{DelimitedReasoningParser, ReasoningDelta, ReasoningParser, Result}; + +/// Reasoning parser for DeepSeek R1 style outputs. +/// +/// DeepSeek R1 may begin generating directly inside a reasoning span and only +/// emit the closing `` delimiter, so the no-boundary fallback defaults +/// to `in_reasoning = true`. +pub struct DeepSeekR1ReasoningParser { + inner: DelimitedReasoningParser, +} + +impl DeepSeekR1ReasoningParser { + /// Create a DeepSeek R1 parser backed by the shared delimited state + /// machine. + pub fn new(tokenizer: DynTokenizer) -> Result { + Ok(Self { + inner: DelimitedReasoningParser::new(tokenizer, "", "", true)?, + }) + } +} + +impl ReasoningParser for DeepSeekR1ReasoningParser { + fn create(tokenizer: DynTokenizer) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tokenizer)?)) + } + + fn initialize(&mut self, prompt_token_ids: &[u32]) -> Result<()> { + self.inner.initialize(prompt_token_ids); + Ok(()) + } + + fn push(&mut self, delta: &str) -> Result { + Ok(self.inner.push(delta)) + } + + fn finish(&mut self) -> Result { + Ok(self.inner.finish()) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/delimited.rs b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/delimited.rs new file mode 100644 index 0000000000000000000000000000000000000000..785952b01bb1998415801e635274c8744c677d08 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/delimited.rs @@ -0,0 +1,166 @@ +use vllm_tokenizer::{DynTokenizer, Tokenizer}; + +use super::{ReasoningDelta, ReasoningError, Result}; + +/// Shared incremental state machine for tag-delimited reasoning protocols. +/// +/// This helper is intentionally not a public parser type. Model-family parser +/// wrappers own one `DelimitedReasoningParser` internally and expose the +/// request-facing [`super::ReasoningParser`] trait. +/// +/// The shared state machine stays generic by deriving its initial +/// `current_in_reasoning` state from the prompt token boundary instead of +/// hardcoding model-family conventions. That means families with the same +/// delimiters can often reuse this implementation even if their chat templates +/// prefill different prompts. +pub(crate) struct DelimitedReasoningParser { + tokenizer: DynTokenizer, + current_in_reasoning: bool, + buffer: String, + start_token: String, + end_token: String, + start_token_id: u32, + end_token_id: u32, + default_in_reasoning: bool, +} + +impl DelimitedReasoningParser { + /// Create one delimited parser state machine. + /// + /// `default_in_reasoning` is only used when prompt initialization sees no + /// reasoning boundary token at all. If the prompt contains either the + /// start or end delimiter, that prompt boundary always wins. + pub(crate) fn new( + tokenizer: DynTokenizer, + start_token: &'static str, + end_token: &'static str, + default_in_reasoning: bool, + ) -> Result { + let start_token_id = + tokenizer.token_to_id(start_token).ok_or_else(|| ReasoningError::MissingToken { + token: start_token.to_string(), + })?; + let end_token_id = + tokenizer.token_to_id(end_token).ok_or_else(|| ReasoningError::MissingToken { + token: end_token.to_string(), + })?; + + Ok(Self { + tokenizer, + current_in_reasoning: default_in_reasoning, + buffer: String::new(), + start_token: start_token.to_string(), + end_token: end_token.to_string(), + start_token_id, + end_token_id, + default_in_reasoning, + }) + } + + /// Initialize the starting state from prompt token IDs. + pub(crate) fn initialize(&mut self, prompt_token_ids: &[u32]) { + self.current_in_reasoning = last_reasoning_boundary( + prompt_token_ids, + self.start_token_id, + self.end_token_id, + self.tokenizer.as_ref(), + ) + .unwrap_or(self.default_in_reasoning); + } + + /// Return whether the parser is currently inside a reasoning section. + pub(crate) fn in_reasoning(&self) -> bool { + self.current_in_reasoning + } + + /// Parse one decoded text delta and return its reasoning/content split. + pub(crate) fn push(&mut self, delta: &str) -> ReasoningDelta { + self.buffer.push_str(delta); + + let partial_suffix_len = self.partial_suffix_len(&self.buffer); + let stable_len = self.buffer.len() - partial_suffix_len; + let pending_suffix = self.buffer.split_off(stable_len); + let stable_text = std::mem::replace(&mut self.buffer, pending_suffix); + + self.parse_stable_text(&stable_text) + } + + /// Flush any buffered partial delimiter suffix at end of stream. + pub(crate) fn finish(&mut self) -> ReasoningDelta { + let stable_text = std::mem::take(&mut self.buffer); + self.parse_stable_text(&stable_text) + } + + /// Parse text that is known not to end with a partial delimiter suffix. + fn parse_stable_text(&mut self, mut stable: &str) -> ReasoningDelta { + let mut delta = ReasoningDelta::default(); + + while !stable.is_empty() { + if self.current_in_reasoning { + if let Some(end_idx) = stable.find(&self.end_token) { + delta.push_reasoning(&stable[..end_idx]); + stable = &stable[end_idx + self.end_token.len()..]; + self.current_in_reasoning = false; + } else { + delta.push_reasoning(stable); + break; + } + } else if let Some(start_idx) = stable.find(&self.start_token) { + delta.push_content(&stable[..start_idx]); + stable = &stable[start_idx + self.start_token.len()..]; + self.current_in_reasoning = true; + } else { + delta.push_content(stable); + break; + } + } + + delta + } + + /// Return the longest trailing suffix that could still complete a + /// delimiter. + fn partial_suffix_len(&self, text: &str) -> usize { + let mut best = 0; + for idx in text.char_indices().map(|(idx, _)| idx).skip(1) { + let suffix = &text[idx..]; + if self.start_token.starts_with(suffix) && self.start_token != suffix { + best = best.max(text.len() - idx); + } + if self.end_token.starts_with(suffix) && self.end_token != suffix { + best = best.max(text.len() - idx); + } + } + + if self.start_token.starts_with(text) && self.start_token != text { + best = best.max(text.len()); + } + if self.end_token.starts_with(text) && self.end_token != text { + best = best.max(text.len()); + } + + best + } +} + +/// Determine the reasoning state implied by the last prompt boundary, if any. +pub(crate) fn last_reasoning_boundary( + prompt_token_ids: &[u32], + start_token_id: u32, + end_token_id: u32, + tokenizer: &dyn Tokenizer, +) -> Option { + for token_id in prompt_token_ids.iter().rev().copied() { + if token_id == start_token_id { + return Some(true); + } + if token_id == end_token_id { + return Some(false); + } + if tokenizer.is_special_id(token_id) { + return None; + } + } + + None +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/kimi.rs b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/kimi.rs new file mode 100644 index 0000000000000000000000000000000000000000..79b28fd292336b37aeddf6897a37b9500f536f3c --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/kimi.rs @@ -0,0 +1,39 @@ +use vllm_tokenizer::DynTokenizer; + +use super::{DelimitedReasoningParser, ReasoningDelta, ReasoningParser, Result}; + +/// Reasoning parser for legacy Kimi models that use Unicode thinking tags. +pub struct KimiReasoningParser { + inner: DelimitedReasoningParser, +} + +impl KimiReasoningParser { + /// Create a Kimi parser backed by the shared delimited state machine. + pub fn new(tokenizer: DynTokenizer) -> Result { + Ok(Self { + inner: DelimitedReasoningParser::new(tokenizer, "◁think▷", "◁/think▷", false)?, + }) + } +} + +impl ReasoningParser for KimiReasoningParser { + fn create(tokenizer: DynTokenizer) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tokenizer)?)) + } + + fn initialize(&mut self, prompt_token_ids: &[u32]) -> Result<()> { + self.inner.initialize(prompt_token_ids); + Ok(()) + } + + fn push(&mut self, delta: &str) -> Result { + Ok(self.inner.push(delta)) + } + + fn finish(&mut self) -> Result { + Ok(self.inner.finish()) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/minimax_m3.rs b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/minimax_m3.rs new file mode 100644 index 0000000000000000000000000000000000000000..5083808ae460bf71bd347ffbafdef993f267d5db --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/minimax_m3.rs @@ -0,0 +1,98 @@ +use vllm_tokenizer::DynTokenizer; + +use super::{DelimitedReasoningParser, ReasoningDelta, ReasoningParser, Result}; + +const M3_THINK_START: &str = ""; +const M3_THINK_END: &str = ""; + +/// Reasoning parser for MiniMax M3 style outputs. +/// +/// MiniMax M3 uses `...` delimiters. Its chat template may +/// prefill either delimiter depending on the requested thinking mode, so the +/// shared delimited parser derives the starting state from the rendered prompt. +pub struct MiniMaxM3ReasoningParser { + inner: DelimitedReasoningParser, + /// True until the first response text is classified. Only this position may + /// drop a stray `` emitted at the start of a response. + at_response_start: bool, + /// Holds an initial suffix like ` Result { + Ok(Self { + inner: DelimitedReasoningParser::new(tokenizer, M3_THINK_START, M3_THINK_END, false)?, + at_response_start: true, + leading_end_buffer: String::new(), + }) + } + + /// Drop a response-leading `` while preserving later unmatched + /// closers as ordinary content. + fn push_inner(&mut self, delta: &str) -> ReasoningDelta { + if self.at_response_start && !self.inner.in_reasoning() { + self.leading_end_buffer.push_str(delta); + let buffered = std::mem::take(&mut self.leading_end_buffer); + + if buffered.is_empty() { + return ReasoningDelta::default(); + } + if let Some(rest) = buffered.strip_prefix(M3_THINK_END) { + self.at_response_start = false; + return self.inner.push(rest); + } + if M3_THINK_END.starts_with(buffered.as_str()) { + self.leading_end_buffer = buffered; + return ReasoningDelta::default(); + } + + self.at_response_start = false; + return self.inner.push(&buffered); + } + + self.inner.push(delta) + } +} + +fn append_delta(target: &mut ReasoningDelta, delta: ReasoningDelta) { + if let Some(reasoning) = delta.reasoning { + target.push_reasoning(&reasoning); + } + if let Some(content) = delta.content { + target.push_content(&content); + } +} + +impl ReasoningParser for MiniMaxM3ReasoningParser { + fn create(tokenizer: DynTokenizer) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tokenizer)?)) + } + + fn initialize(&mut self, prompt_token_ids: &[u32]) -> Result<()> { + self.inner.initialize(prompt_token_ids); + self.at_response_start = true; + self.leading_end_buffer.clear(); + Ok(()) + } + + fn push(&mut self, delta: &str) -> Result { + Ok(self.push_inner(delta)) + } + + fn finish(&mut self) -> Result { + let mut delta = ReasoningDelta::default(); + if !self.leading_end_buffer.is_empty() { + let pending = std::mem::take(&mut self.leading_end_buffer); + self.at_response_start = false; + append_delta(&mut delta, self.inner.push(&pending)); + } + append_delta(&mut delta, self.inner.finish()); + Ok(delta) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/mod.rs b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..fba127617d427948ede10a01cc1b1897a2be54f5 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/mod.rs @@ -0,0 +1,137 @@ +//! Streaming reasoning parsers for chat completions. +//! +//! The key design choice here is that parser initialization prefers the +//! *actual rendered prompt state* over model-family conventions. When a stream +//! starts, each parser receives the prompt token IDs and inspects the last +//! reasoning boundary that is already present in the prompt. In practice this +//! is a more faithful signal than hardcoding assumptions such as "this model +//! always starts in reasoning" or "this model always emits `` itself". +//! +//! That prompt-first initialization lets multiple model families share the +//! same incremental parser implementation even when older Python parsers split +//! them apart. If two families use the same textual delimiters and differ +//! mostly in how their chat templates prefill `` / ``, they can +//! usually reuse one parser here because the prompt token IDs already tell us +//! which state the stream is entering with. + +mod cohere_cmd; +mod deepseek_r1; +mod delimited; +mod kimi; +mod minimax_m3; +mod qwen3; +mod seed_oss; +mod step3p5; + +use thiserror::Error; +use vllm_tokenizer::DynTokenizer; + +pub use self::cohere_cmd::CohereCmdReasoningParser; +pub use self::deepseek_r1::DeepSeekR1ReasoningParser; +pub(crate) use self::delimited::{DelimitedReasoningParser, last_reasoning_boundary}; +pub use self::kimi::KimiReasoningParser; +pub use self::minimax_m3::MiniMaxM3ReasoningParser; +pub use self::qwen3::Qwen3ReasoningParser; +pub use self::seed_oss::SeedOssReasoningParser; +pub use self::step3p5::Step3p5ReasoningParser; + +/// DeepSeek V3 currently shares the standard `...` parser. +pub type DeepSeekV3ReasoningParser = Qwen3ReasoningParser; +/// DeepSeek V4 currently shares the standard `...` parser. +pub type DeepSeekV4ReasoningParser = Qwen3ReasoningParser; +/// GLM45 currently shares the standard `...` parser. +pub type Glm45ReasoningParser = Qwen3ReasoningParser; +/// Kimi K2 currently shares the standard `...` parser. +// TODO: kimi k2 may implicitly end reasoning by starting a tool call section +// using <|tool_calls_section_begin|>, we should support that. +pub type KimiK2ReasoningParser = Qwen3ReasoningParser; +/// MiniMax M2 currently shares the standard `...` parser. +pub type MiniMaxM2ReasoningParser = Qwen3ReasoningParser; +/// Nemotron V3 currently shares the standard `...` parser. +pub type NemotronV3ReasoningParser = Qwen3ReasoningParser; +/// Step3 currently shares the standard `...` parser. +pub type Step3ReasoningParser = Qwen3ReasoningParser; + +/// Result alias for reasoning parser operations. +pub type Result = std::result::Result; + +/// One parsed streaming delta split into reasoning and visible content. +#[derive(Debug, Default, Clone, PartialEq, Eq)] +pub struct ReasoningDelta { + pub reasoning: Option, + pub content: Option, +} + +impl ReasoningDelta { + /// Return true when this delta carries neither reasoning nor content text. + pub fn is_empty(&self) -> bool { + self.reasoning.is_none() && self.content.is_none() + } + + /// Append text to the reasoning portion, creating it on first use. + pub(crate) fn push_reasoning(&mut self, text: &str) { + if text.is_empty() { + return; + } + match &mut self.reasoning { + Some(existing) => existing.push_str(text), + None => self.reasoning = Some(text.to_string()), + } + } + + /// Append text to the visible content portion, creating it on first use. + pub(crate) fn push_content(&mut self, text: &str) { + if text.is_empty() { + return; + } + match &mut self.content { + Some(existing) => existing.push_str(text), + None => self.content = Some(text.to_string()), + } + } +} + +/// Incremental parser that splits decoded text deltas into reasoning and +/// content. +pub trait ReasoningParser: Send { + /// Construct a boxed parser instance for one request stream. + fn create(tokenizer: DynTokenizer) -> Result> + where + Self: Sized + 'static; + + /// Initialize parser state from prompt token IDs before output deltas + /// arrive. + fn initialize(&mut self, _prompt_token_ids: &[u32]) -> Result<()> { + Ok(()) + } + + /// Return whether decoded output must preserve tokenizer special tokens. + /// + /// Some model families emit reasoning sentinels as special tokens. Those + /// parsers need `skip_special_tokens = false` while parsing is enabled. + fn preserve_special_tokens(&self) -> bool { + false + } + + /// Feed one decoded text delta into the parser. + fn push(&mut self, delta: &str) -> Result; + + /// Flush any buffered partial delimiter state at end of stream. + fn finish(&mut self) -> Result { + Ok(ReasoningDelta::default()) + } +} + +/// Errors produced while creating or running reasoning parsers. +#[derive(Debug, Error)] +pub enum ReasoningError { + #[error("tokenizer is missing reasoning delimiter token `{token}`")] + MissingToken { token: String }, + #[error( + "`{name}` only provides a unified parser; the same reasoning parser and tool parser should be specified together" + )] + DummyUnifiedParser { name: String }, +} + +#[cfg(test)] +mod tests; diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/qwen3.rs b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/qwen3.rs new file mode 100644 index 0000000000000000000000000000000000000000..35fb5df72a54374cc77089b989a92a6a7ddf2fc3 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/qwen3.rs @@ -0,0 +1,43 @@ +use vllm_tokenizer::DynTokenizer; + +use super::{DelimitedReasoningParser, ReasoningDelta, ReasoningParser, Result}; + +/// Reasoning parser for the Qwen3/Qwen3.5 family. +/// +/// This parser uses standard `...` delimiters and defaults to +/// waiting for an explicit start token when prompt initialization finds no +/// reasoning boundary. +pub struct Qwen3ReasoningParser { + inner: DelimitedReasoningParser, +} + +impl Qwen3ReasoningParser { + /// Create a Qwen3 parser backed by the shared delimited state machine. + pub fn new(tokenizer: DynTokenizer) -> Result { + Ok(Self { + inner: DelimitedReasoningParser::new(tokenizer, "", "", false)?, + }) + } +} + +impl ReasoningParser for Qwen3ReasoningParser { + fn create(tokenizer: DynTokenizer) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tokenizer)?)) + } + + fn initialize(&mut self, prompt_token_ids: &[u32]) -> Result<()> { + self.inner.initialize(prompt_token_ids); + Ok(()) + } + + fn push(&mut self, delta: &str) -> Result { + Ok(self.inner.push(delta)) + } + + fn finish(&mut self) -> Result { + Ok(self.inner.finish()) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/seed_oss.rs b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/seed_oss.rs new file mode 100644 index 0000000000000000000000000000000000000000..41d1843ebde65c10ef311ae91505018fdf854554 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/seed_oss.rs @@ -0,0 +1,148 @@ +use vllm_tokenizer::DynTokenizer; + +use super::{DelimitedReasoningParser, ReasoningDelta, ReasoningParser, Result}; + +/// Reasoning parser for SeedOSS models using ``/`` +/// delimiters. +pub struct SeedOssReasoningParser { + inner: DelimitedReasoningParser, +} + +impl SeedOssReasoningParser { + /// Create a SeedOSS parser backed by the shared delimited state machine. + pub fn new(tokenizer: DynTokenizer) -> Result { + Ok(Self { + inner: DelimitedReasoningParser::new( + tokenizer, + "", + "", + false, + )?, + }) + } +} + +impl ReasoningParser for SeedOssReasoningParser { + fn create(tokenizer: DynTokenizer) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tokenizer)?)) + } + + fn initialize(&mut self, prompt_token_ids: &[u32]) -> Result<()> { + self.inner.initialize(prompt_token_ids); + Ok(()) + } + + fn push(&mut self, delta: &str) -> Result { + Ok(self.inner.push(delta)) + } + + fn finish(&mut self) -> Result { + Ok(self.inner.finish()) + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use super::SeedOssReasoningParser; + use crate::reasoning::ReasoningParser; + use crate::reasoning::tests::{SEED_THINK_END_ID, SEED_THINK_START_ID, fake_tokenizer}; + + #[test] + fn without_prompt_markers_expects_start_token() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = SeedOssReasoningParser::new(tokenizer).unwrap(); + + let delta = parser.push("implicit reasoninganswer").unwrap(); + assert_eq!(delta.reasoning, None); + assert_eq!( + delta.content.as_deref(), + Some("implicit reasoninganswer") + ); + } + + #[test] + fn picks_up_prompt_start_boundary() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = SeedOssReasoningParser::new(tokenizer).unwrap(); + // Prompt prefills ``, opening reasoning before the stream. + parser.initialize(&[SEED_THINK_START_ID]).unwrap(); + + let delta = parser.push("reasonanswer").unwrap(); + assert_eq!(delta.reasoning.as_deref(), Some("reason")); + assert_eq!(delta.content.as_deref(), Some("answer")); + } + + #[test] + fn respects_prompt_end_boundary() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = SeedOssReasoningParser::new(tokenizer).unwrap(); + // Prompt already closed reasoning with ``. + parser.initialize(&[SEED_THINK_END_ID]).unwrap(); + + let delta = parser.push("answer").unwrap(); + assert_eq!(delta.reasoning, None); + assert_eq!(delta.content.as_deref(), Some("answer")); + } + + #[test] + fn handles_explicit_start_token() { + // An explicit start delimiter must not leak into reasoning text. + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = SeedOssReasoningParser::new(tokenizer).unwrap(); + + let delta = parser.push("reasonanswer").unwrap(); + assert_eq!(delta.reasoning.as_deref(), Some("reason")); + assert_eq!(delta.content.as_deref(), Some("answer")); + } + + #[test] + fn streams_explicit_start_token_across_pushes() { + // Start token, reasoning body, end token, and content arrive in separate + // streaming deltas. + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = SeedOssReasoningParser::new(tokenizer).unwrap(); + + let mut reasoning = String::new(); + let mut content = String::new(); + for delta_str in [ + "", + "Some ", + "reasoning ", + "content", + "", + "Final ", + "answer", + ] { + let delta = parser.push(delta_str).unwrap(); + if let Some(r) = delta.reasoning { + reasoning.push_str(&r); + } + if let Some(c) = delta.content { + content.push_str(&c); + } + } + assert_eq!(reasoning, "Some reasoning content"); + assert_eq!(content, "Final answer"); + } + + #[test] + fn handles_partial_delimiters_across_pushes() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = SeedOssReasoningParser::new(tokenizer).unwrap(); + parser.initialize(&[SEED_THINK_START_ID]).unwrap(); + + // Closing delimiter `` arrives in two halves. + let first = parser.push("reasonanswer").unwrap(); + assert_eq!(second.reasoning, None); + assert_eq!(second.content.as_deref(), Some("answer")); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/step3p5.rs b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/step3p5.rs new file mode 100644 index 0000000000000000000000000000000000000000..278a2edbed08c001c1fad72f599d693637aa8634 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/step3p5.rs @@ -0,0 +1,309 @@ +use vllm_tokenizer::DynTokenizer; + +use super::{DelimitedReasoningParser, ReasoningDelta, ReasoningParser, Result}; + +/// Reasoning parser for Step3p5 outputs. +/// +/// Step3p5 uses standard ``/`` delimiters but emits a `\n` +/// immediately before and/or after ``. The parser drops these framing +/// newlines on both sides of the boundary, holding a trailing `\n` from +/// reasoning across pushes until either more reasoning text or `` +/// arrives, and dropping a leading `\n` from the first content delta after +/// the boundary. +pub struct Step3p5ReasoningParser { + inner: DelimitedReasoningParser, + /// `\n` at end of last reasoning delta, held in case `` follows. + pending_reasoning_newline: bool, + /// Last push ended on `` without emitting content; the next + /// content delta's leading `\n` should be dropped. + just_ended_reasoning: bool, +} + +impl Step3p5ReasoningParser { + /// Create a Step3p5 parser backed by the shared delimited state machine. + pub fn new(tokenizer: DynTokenizer) -> Result { + Ok(Self { + inner: DelimitedReasoningParser::new(tokenizer, "", "", false)?, + pending_reasoning_newline: false, + just_ended_reasoning: false, + }) + } + + /// Drop framing newlines around `` and track held-newline state. + fn process( + &mut self, + mut inner_delta: ReasoningDelta, + was_in_reasoning: bool, + now_in_reasoning: bool, + ) -> ReasoningDelta { + // A `...` round-trip in one push still counts as a + // transition: the inner emits reasoning while ending in content mode. + let transitioned = + !now_in_reasoning && (was_in_reasoning || inner_delta.reasoning.is_some()); + + // Replay or drop a previously-held trailing reasoning newline. + if self.pending_reasoning_newline { + if let Some(reasoning) = inner_delta.reasoning.as_mut() { + reasoning.insert(0, '\n'); + self.pending_reasoning_newline = false; + } else if transitioned { + // The held `\n` was the one right before ``: drop it. + self.pending_reasoning_newline = false; + } + } + + // Hold back a trailing reasoning `\n` until we know if `` follows. + if let Some(reasoning) = inner_delta.reasoning.as_mut() + && reasoning.ends_with('\n') + { + reasoning.pop(); + if !transitioned { + self.pending_reasoning_newline = true; + } + } + + // Drop a leading `\n` of content emitted right after ``. + if let Some(content) = inner_delta.content.as_mut() + && (transitioned || self.just_ended_reasoning) + && content.starts_with('\n') + { + content.remove(0); + } + + self.just_ended_reasoning = transitioned && inner_delta.content.is_none(); + + if inner_delta.reasoning.as_deref() == Some("") { + inner_delta.reasoning = None; + } + if inner_delta.content.as_deref() == Some("") { + inner_delta.content = None; + } + + inner_delta + } +} + +impl ReasoningParser for Step3p5ReasoningParser { + fn create(tokenizer: DynTokenizer) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tokenizer)?)) + } + + fn initialize(&mut self, prompt_token_ids: &[u32]) -> Result<()> { + self.inner.initialize(prompt_token_ids); + Ok(()) + } + + fn push(&mut self, delta: &str) -> Result { + let was = self.inner.in_reasoning(); + let inner_delta = self.inner.push(delta); + let now = self.inner.in_reasoning(); + Ok(self.process(inner_delta, was, now)) + } + + fn finish(&mut self) -> Result { + let was = self.inner.in_reasoning(); + let inner_delta = self.inner.finish(); + let now = self.inner.in_reasoning(); + let mut delta = self.process(inner_delta, was, now); + + // Emit a still-held newline rather than silently dropping it. + if self.pending_reasoning_newline { + match delta.reasoning.as_mut() { + Some(existing) => existing.push('\n'), + None => delta.reasoning = Some("\n".to_string()), + } + self.pending_reasoning_newline = false; + } + + Ok(delta) + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use super::Step3p5ReasoningParser; + use crate::reasoning::ReasoningParser; + use crate::reasoning::tests::{THINK_START_ID, fake_tokenizer}; + + #[test] + fn picks_up_prompt_start_boundary() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Step3p5ReasoningParser::new(tokenizer).unwrap(); + // Prompt prefills ``, opening reasoning before the stream. + parser.initialize(&[THINK_START_ID]).unwrap(); + + let delta = parser.push("This is a reasoning sectionThis is the rest").unwrap(); + assert_eq!( + delta.reasoning.as_deref(), + Some("This is a reasoning section") + ); + assert_eq!(delta.content.as_deref(), Some("This is the rest")); + } + + #[test] + fn handles_unterminated_reasoning() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Step3p5ReasoningParser::new(tokenizer).unwrap(); + + let pushed = parser.push("reason without end").unwrap(); + assert_eq!(pushed.reasoning.as_deref(), Some("reason without end")); + assert_eq!(pushed.content, None); + + let flushed = parser.finish().unwrap(); + assert!(flushed.is_empty()); + } + + #[test] + fn handles_empty_input() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Step3p5ReasoningParser::new(tokenizer).unwrap(); + + let pushed = parser.push("").unwrap(); + assert!(pushed.is_empty()); + let flushed = parser.finish().unwrap(); + assert!(flushed.is_empty()); + } + + #[test] + fn complex_newline_pattern_trims_only_single_framing_newline_each_side() { + // Only the immediately-adjacent framing `\n` is dropped on each side of + // ``; surrounding newlines remain part of reasoning/content. + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Step3p5ReasoningParser::new(tokenizer).unwrap(); + parser.initialize(&[THINK_START_ID]).unwrap(); + + let delta = parser + .push("\n This is a \n reasoning section\n\n\n\n\nThis is the rest") + .unwrap(); + assert_eq!( + delta.reasoning.as_deref(), + Some("\n This is a \n reasoning section\n\n") + ); + assert_eq!(delta.content.as_deref(), Some("\nThis is the rest")); + } + + #[test] + fn drops_framing_newlines_in_single_push() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Step3p5ReasoningParser::new(tokenizer).unwrap(); + + let delta = parser.push("reason\n\nanswer").unwrap(); + assert_eq!(delta.reasoning.as_deref(), Some("reason")); + assert_eq!(delta.content.as_deref(), Some("answer")); + } + + #[test] + fn drops_framing_newlines_across_pushes() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Step3p5ReasoningParser::new(tokenizer).unwrap(); + + // The trailing `\n` from the first push is held until we know whether + // `` follows. + let first = parser.push("reason\n").unwrap(); + assert_eq!(first.reasoning.as_deref(), Some("reason")); + assert_eq!(first.content, None); + + // `` arrives standalone; the held newline should be dropped. + let second = parser.push("").unwrap(); + assert!(second.is_empty()); + + // The leading newline of the first content delta is dropped. + let third = parser.push("\nanswer").unwrap(); + assert_eq!(third.reasoning, None); + assert_eq!(third.content.as_deref(), Some("answer")); + } + + #[test] + fn replays_held_newline_when_more_reasoning_follows() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Step3p5ReasoningParser::new(tokenizer).unwrap(); + + let first = parser.push("reason\n").unwrap(); + assert_eq!(first.reasoning.as_deref(), Some("reason")); + + let second = parser.push("more reason").unwrap(); + assert_eq!(second.reasoning.as_deref(), Some("\nmore reason")); + assert_eq!(second.content, None); + } + + #[test] + fn finish_flushes_held_newline_in_unterminated_stream() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Step3p5ReasoningParser::new(tokenizer).unwrap(); + + let first = parser.push("reason\n").unwrap(); + assert_eq!(first.reasoning.as_deref(), Some("reason")); + + let flushed = parser.finish().unwrap(); + assert_eq!(flushed.reasoning.as_deref(), Some("\n")); + assert_eq!(flushed.content, None); + } + + #[test] + fn preserves_inner_newlines_in_reasoning() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Step3p5ReasoningParser::new(tokenizer).unwrap(); + + let delta = parser.push("line1\nline2tail").unwrap(); + assert_eq!(delta.reasoning.as_deref(), Some("line1\nline2")); + assert_eq!(delta.content.as_deref(), Some("tail")); + } + + #[test] + fn trims_only_one_trailing_reasoning_newline() { + // Only the single framing newline immediately before `` is + // dropped; earlier newlines in the reasoning body are preserved. + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Step3p5ReasoningParser::new(tokenizer).unwrap(); + + let delta = parser.push("reason\n\nanswer").unwrap(); + assert_eq!(delta.reasoning.as_deref(), Some("reason\n")); + assert_eq!(delta.content.as_deref(), Some("answer")); + } + + #[test] + fn drops_only_first_content_newline_after_transition() { + // The leading-`\n` drop applies only to the first content delta after + // ``; later deltas pass through untouched. + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Step3p5ReasoningParser::new(tokenizer).unwrap(); + + let first = parser.push("reason").unwrap(); + assert_eq!(first.reasoning.as_deref(), Some("reason")); + assert_eq!(first.content, None); + + let second = parser.push("\nfirst").unwrap(); + assert_eq!(second.reasoning, None); + assert_eq!(second.content.as_deref(), Some("first")); + + // A `\n` arriving in a later content delta must NOT be dropped. + let third = parser.push("\nsecond").unwrap(); + assert_eq!(third.reasoning, None); + assert_eq!(third.content.as_deref(), Some("\nsecond")); + } + + #[test] + fn passes_through_clean_boundary_without_framing_newlines() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Step3p5ReasoningParser::new(tokenizer).unwrap(); + + let delta = parser.push("reasontail").unwrap(); + assert_eq!(delta.reasoning.as_deref(), Some("reason")); + assert_eq!(delta.content.as_deref(), Some("tail")); + } + + #[test] + fn handles_empty_reasoning_section() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Step3p5ReasoningParser::new(tokenizer).unwrap(); + + let delta = parser.push("answer").unwrap(); + assert_eq!(delta.reasoning, None); + assert_eq!(delta.content.as_deref(), Some("answer")); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/tests.rs b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/tests.rs new file mode 100644 index 0000000000000000000000000000000000000000..4f880d941d429bd515198d39e28ccc222b9787d7 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/reasoning/tests.rs @@ -0,0 +1,217 @@ +use std::sync::Arc; + +use vllm_tokenizer::test_utils::TestTokenizer; + +use super::{ + DeepSeekR1ReasoningParser, DelimitedReasoningParser, MiniMaxM3ReasoningParser, + Qwen3ReasoningParser, ReasoningParser, +}; + +pub(crate) const THINK_START_ID: u32 = 256; +pub(crate) const THINK_END_ID: u32 = 257; +pub(crate) const START_THINKING_ID: u32 = 258; +pub(crate) const END_THINKING_ID: u32 = 259; +pub(crate) const MINIMAX_THINK_START_ID: u32 = 260; +pub(crate) const MINIMAX_THINK_END_ID: u32 = 261; +pub(crate) const SPECIAL_BOUNDARY_ID: u32 = 262; +pub(crate) const MM_THINK_START_ID: u32 = 263; +pub(crate) const MM_THINK_END_ID: u32 = 264; +pub(crate) const SEED_THINK_START_ID: u32 = 265; +pub(crate) const SEED_THINK_END_ID: u32 = 266; + +pub(crate) fn fake_tokenizer() -> TestTokenizer { + TestTokenizer::new() + .with_regular_token("", THINK_START_ID) + .with_regular_token("", THINK_END_ID) + .with_regular_token("<|START_THINKING|>", START_THINKING_ID) + .with_regular_token("<|END_THINKING|>", END_THINKING_ID) + .with_regular_token("◁think▷", MINIMAX_THINK_START_ID) + .with_regular_token("◁/think▷", MINIMAX_THINK_END_ID) + .with_special_token("", SPECIAL_BOUNDARY_ID) + .with_regular_token("", MM_THINK_START_ID) + .with_regular_token("", MM_THINK_END_ID) + .with_regular_token("", SEED_THINK_START_ID) + .with_regular_token("", SEED_THINK_END_ID) +} + +#[test] +fn delimited_content_only_stream() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = + DelimitedReasoningParser::new(tokenizer, "", "", false).unwrap(); + + assert_eq!( + parser.push("plain content").content.as_deref(), + Some("plain content") + ); +} + +#[test] +fn delimited_single_chunk_with_reasoning_and_content() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = + DelimitedReasoningParser::new(tokenizer, "", "", false).unwrap(); + + let delta = parser.push("reasonanswer"); + assert_eq!(delta.reasoning.as_deref(), Some("reason")); + assert_eq!(delta.content.as_deref(), Some("answer")); +} + +#[test] +fn delimited_partial_tokens_across_chunks() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = + DelimitedReasoningParser::new(tokenizer, "", "", false).unwrap(); + + assert!(parser.push("reasonanswer"); + assert_eq!(delta.reasoning.as_deref(), Some("reason")); + assert_eq!(delta.content.as_deref(), Some("answer")); +} + +#[test] +fn delimited_finish_flushes_buffer() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = + DelimitedReasoningParser::new(tokenizer, "", "", false).unwrap(); + parser.initialize(&[THINK_START_ID]); + + let delta = parser.push("unfinishedanswer").unwrap(); + assert_eq!(delta.reasoning, None); + assert_eq!(delta.content.as_deref(), Some("reasonanswer")); +} + +#[test] +fn qwen3_prompt_end_marker_starts_in_content() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Qwen3ReasoningParser::new(tokenizer).unwrap(); + parser.initialize(&[THINK_END_ID]).unwrap(); + + let delta = parser.push("answer").unwrap(); + assert_eq!(delta.reasoning, None); + assert_eq!(delta.content.as_deref(), Some("answer")); +} + +#[test] +fn qwen3_tolerates_old_and_new_formats() { + let tokenizer = Arc::new(fake_tokenizer()); + + let mut old_parser = Qwen3ReasoningParser::new(tokenizer.clone()).unwrap(); + let old = old_parser.push("reasonanswer").unwrap(); + assert_eq!(old.reasoning.as_deref(), Some("reason")); + assert_eq!(old.content.as_deref(), Some("answer")); + + let mut new_parser = Qwen3ReasoningParser::new(tokenizer).unwrap(); + new_parser.initialize(&[THINK_START_ID]).unwrap(); + let new = new_parser.push("reasonanswer").unwrap(); + assert_eq!(new.reasoning.as_deref(), Some("reason")); + assert_eq!(new.content.as_deref(), Some("answer")); +} + +#[test] +fn qwen3_stops_scanning_at_last_special_token() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = Qwen3ReasoningParser::new(tokenizer).unwrap(); + + parser.initialize(&[THINK_START_ID, SPECIAL_BOUNDARY_ID]).unwrap(); + + let delta = parser.push("answer").unwrap(); + assert_eq!(delta.reasoning, None); + assert_eq!(delta.content.as_deref(), Some("answer")); +} + +#[test] +fn deepseek_r1_defaults_to_reasoning_without_prompt_boundary() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = DeepSeekR1ReasoningParser::new(tokenizer).unwrap(); + + let delta = parser.push("reasonanswer").unwrap(); + assert_eq!(delta.reasoning.as_deref(), Some("reason")); + assert_eq!(delta.content.as_deref(), Some("answer")); +} + +#[test] +fn deepseek_r1_stops_scanning_at_last_special_token() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = DeepSeekR1ReasoningParser::new(tokenizer).unwrap(); + + parser.initialize(&[THINK_END_ID, SPECIAL_BOUNDARY_ID]).unwrap(); + + let delta = parser.push("reasonanswer").unwrap(); + assert_eq!(delta.reasoning.as_deref(), Some("reason")); + assert_eq!(delta.content.as_deref(), Some("answer")); +} + +#[test] +fn minimax_m3_handles_explicit_think_delimiters() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = MiniMaxM3ReasoningParser::new(tokenizer).unwrap(); + + let delta = parser.push("reasonanswer").unwrap(); + assert_eq!(delta.reasoning.as_deref(), Some("reason")); + assert_eq!(delta.content.as_deref(), Some("answer")); +} + +#[test] +fn minimax_m3_drops_leading_end_marker() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = MiniMaxM3ReasoningParser::new(tokenizer).unwrap(); + + let delta = parser.push("answer").unwrap(); + assert_eq!(delta.reasoning, None); + assert_eq!(delta.content.as_deref(), Some("answer")); +} + +#[test] +fn minimax_m3_preserves_non_leading_end_marker() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = MiniMaxM3ReasoningParser::new(tokenizer).unwrap(); + + let delta = parser.push("XXXYYY").unwrap(); + assert_eq!(delta.reasoning, None); + assert_eq!(delta.content.as_deref(), Some("XXXYYY")); +} + +#[test] +fn minimax_m3_drops_split_leading_end_marker() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = MiniMaxM3ReasoningParser::new(tokenizer).unwrap(); + + assert!(parser.push("answer").unwrap(); + assert_eq!(delta.reasoning, None); + assert_eq!(delta.content.as_deref(), Some("answer")); +} + +#[test] +fn minimax_m3_uses_prompt_prefilled_start_marker() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = MiniMaxM3ReasoningParser::new(tokenizer).unwrap(); + parser.initialize(&[MM_THINK_START_ID]).unwrap(); + + let delta = parser.push("reasonanswer").unwrap(); + assert_eq!(delta.reasoning.as_deref(), Some("reason")); + assert_eq!(delta.content.as_deref(), Some("answer")); +} + +#[test] +fn minimax_m3_uses_prompt_prefilled_end_marker() { + let tokenizer = Arc::new(fake_tokenizer()); + let mut parser = MiniMaxM3ReasoningParser::new(tokenizer).unwrap(); + parser.initialize(&[MM_THINK_END_ID]).unwrap(); + + let delta = parser.push("answer").unwrap(); + assert_eq!(delta.reasoning, None); + assert_eq!(delta.content.as_deref(), Some("answer")); +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_dsml/deepseek_v32.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_dsml/deepseek_v32.rs new file mode 100644 index 0000000000000000000000000000000000000000..e54df3c08608518d6a6a99c26b77a629f511c9a5 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_dsml/deepseek_v32.rs @@ -0,0 +1,445 @@ +use super::{DeepSeekDsmlToolParser, DsmlTokens}; +use crate::tool::{Result, StructuralTagModel, Tool, ToolParser, ToolParserOutput}; + +/// Tool parser for DeepSeek V3.2 models. +/// +/// Example tool call content: +/// +/// ```text +/// <|DSML|function_calls> +/// <|DSML|invoke name="get_weather"> +/// <|DSML|parameter name="location" string="true">杭州 +/// <|DSML|parameter name="date" string="true">2024-01-16 +/// +/// <|DSML|invoke name="get_weather"> +/// <|DSML|parameter name="location" string="true">北京 +/// <|DSML|parameter name="date" string="true">2024-01-16 +/// +/// +/// ``` +/// +/// Arguments are emitted only after a full `invoke` block is parsed. +/// +/// DeepSeek V3.2 relies on DSML markers such as `|DSML|`, which are +/// represented as special tokens in the tokenizer and therefore must be +/// preserved during decode for parsing to work. +pub struct DeepSeekV32ToolParser(DeepSeekDsmlToolParser); + +impl DeepSeekV32ToolParser { + /// Create a DeepSeek V3.2 tool parser. + pub(super) fn new(tools: &[Tool]) -> Self { + Self(DeepSeekDsmlToolParser::new(tools, DsmlTokens::V32)) + } +} + +impl ToolParser for DeepSeekV32ToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn preserve_special_tokens(&self) -> bool { + true + } + + fn structural_tag_model(&self) -> Option { + Some(StructuralTagModel::DeepSeekV32) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.0.parse_into(chunk, output) + } + + fn finish(&mut self) -> Result { + self.0.finish() + } + + fn reset(&mut self) -> String { + self.0.reset() + } +} + +#[cfg(test)] +mod tests { + use serde_json::{Value, json}; + use thiserror_ext::AsReport; + + use super::DeepSeekV32ToolParser; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{ToolParser, ToolParserTestExt as _}; + + fn build_tool_call(function_name: &str, params: &[(&str, &str)]) -> String { + let params = params + .iter() + .map(|(name, value)| { + format!( + r#"<|DSML|parameter name="{name}" string="true">{value}"# + ) + }) + .collect::>() + .join("\n"); + format!( + "<|DSML|function_calls>\n<|DSML|invoke name=\"{function_name}\">\n{params}\n\n" + ) + } + + #[test] + fn deepseek_v32_parse_complete_without_tool_call_keeps_text() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let output = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn deepseek_v32_parse_complete_extracts_single_tool_call() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_call( + "get_weather", + &[("location", "SF"), ("date", "2024-01-16")], + )) + .unwrap(); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "location": "SF", + "date": "2024-01-16" + }) + ); + } + + #[test] + fn deepseek_v32_parse_complete_preserves_prefix_text() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let output = format!( + "Thinking... {}", + build_tool_call("get_weather", &[("location", "NYC")]) + ); + let output = parser.parse_complete(&output).unwrap(); + + assert_eq!(output.normal_text(), "Thinking... "); + assert_eq!(output.calls().len(), 1); + } + + #[test] + fn deepseek_v32_parse_complete_converts_schema_types() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let output = parser + .parse_complete( + "<|DSML|function_calls>\n\ + <|DSML|invoke name=\"convert\">\n\ + <|DSML|parameter name=\"whole\" string=\"false\">5.0\n\ + <|DSML|parameter name=\"flag\" string=\"false\">true\n\ + <|DSML|parameter name=\"payload\" string=\"false\">{\"nested\":true}\n\ + <|DSML|parameter name=\"items\" string=\"false\">[1,2]\n\ + <|DSML|parameter name=\"empty\" string=\"false\">null\n\ + \n\ + ", + ) + .unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "whole": 5.0, + "flag": true, + "payload": { "nested": true }, + "items": [1, 2], + "empty": "null", + }) + ); + } + + #[test] + fn deepseek_v32_parse_complete_string_attr_overrides_schema_types() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let output = parser + .parse_complete( + "<|DSML|function_calls>\n\ + <|DSML|invoke name=\"convert\">\n\ + <|DSML|parameter name=\"whole\" string=\"true\">5.0\n\ + <|DSML|parameter name=\"flag\" string=\"true\">true\n\ + <|DSML|parameter name=\"payload\" string=\"true\">{\"nested\":true}\n\ + <|DSML|parameter name=\"items\" string=\"true\">[1,2]\n\ + <|DSML|parameter name=\"empty\" string=\"true\">null\n\ + \n\ + ", + ) + .unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "whole": "5.0", + "flag": "true", + "payload": "{\"nested\":true}", + "items": "[1,2]", + "empty": "null", + }) + ); + } + + #[test] + fn deepseek_v32_parse_complete_preserves_raw_closing_tag_text_in_parameter_value() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_call( + "get_weather", + &[ + ( + "location", + "Hangzhou </|DSML|parameter></|DSML|invoke></|DSML|function_calls>", + ), + ("date", "2026-05-08"), + ], + )) + .unwrap(); + + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "location": "Hangzhou </|DSML|parameter></|DSML|invoke></|DSML|function_calls>", + "date": "2026-05-08", + }) + ); + } + + #[test] + fn deepseek_v32_streaming_extracts_single_tool_call() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let output = collect_stream( + &mut parser, + &[ + "<|DSML|function_calls>\n", + "<|DSML|invoke name=\"get_weather\">\n", + "<|DSML|parameter name=\"location\" string=\"true\">SF\n", + "\n", + "", + ], + ); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "location": "SF" }) + ); + } + + #[test] + fn deepseek_v32_streaming_preserves_prefix_text() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let output = collect_stream( + &mut parser, + &[ + "Thinking... ", + "<|DSML|function_calls>\n", + "<|DSML|invoke name=\"get_weather\">\n", + "<|DSML|parameter name=\"location\" string=\"true\">SF\n", + "\n", + "", + ], + ); + + assert_eq!(output.normal_text(), "Thinking... "); + assert_eq!(output.calls().len(), 1); + } + + #[test] + fn deepseek_v32_streaming_without_tool_call_emits_text_incrementally() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let output = collect_stream(&mut parser, &["Hello, ", "world!"]); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn deepseek_v32_streaming_extracts_multiple_tool_calls_in_order() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let output = collect_stream( + &mut parser, + &[&format!( + "{}\n{}", + build_tool_call("get_weather", &[("location", "SF")]) + .trim_end_matches(""), + "<|DSML|invoke name=\"get_weather\">\n<|DSML|parameter name=\"location\" string=\"true\">NYC\n\n" + )], + ); + + assert_eq!(output.calls().len(), 2); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[1].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[0].tool_index, 0); + assert_eq!(output.calls()[1].tool_index, 1); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "location": "SF" }) + ); + assert_eq!( + serde_json::from_str::(&output.calls()[1].arguments).unwrap(), + json!({ "location": "NYC" }) + ); + } + + #[test] + fn deepseek_v32_streaming_handles_start_token_split_across_chunks() { + let text = build_tool_call("get_weather", &[("location", "SF")]); + let chunks = split_by_chars(&text, 5); + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.calls().len(), 1); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "location": "SF" }) + ); + } + + #[test] + fn deepseek_v32_streaming_handles_bpe_chunked_dsml_opener() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let output = collect_stream( + &mut parser, + &[ + "<|DSML|", + "function", + "_c", + "all", + "s", + ">\n", + "<|DSML|", + "invoke", + " name=\"", + "get_weather", + "\">\n", + "<|DSML|", + "parameter", + " name=\"location\" string=\"true\">", + "Beijing", + "\n", + "\n", + "", + ], + ); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "location": "Beijing" }) + ); + } + + #[test] + fn deepseek_v32_streaming_truncated_parameter_does_not_leak_eos() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + parser.parse_chunk("<|DSML|function_calls>\n").unwrap(); + parser.parse_chunk("<|DSML|invoke name=\"get_weather\">\n").unwrap(); + parser + .parse_chunk("<|DSML|parameter name=\"location\" string=\"true\">Tokyo") + .unwrap(); + parser.parse_chunk("<|end▁of▁sentence|>").unwrap(); + + let error = parser.finish().unwrap_err(); + assert!(error.to_report_string().contains("incomplete DeepSeek DSML tool call")); + } + #[test] + fn deepseek_v32_streaming_drops_eos_after_complete_tool_calls() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let output = collect_stream( + &mut parser, + &[ + "<|DSML|function_calls>\n", + "<|DSML|invoke name=\"get_weather\">\n", + "<|DSML|parameter name=\"location\" string=\"true\">SF\n", + "\n", + "<|end▁of▁sentence|>", + ], + ); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + } + + #[test] + fn deepseek_v32_streaming_ignores_text_after_complete_tool_calls() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let output = collect_stream( + &mut parser, + &[ + "<|DSML|function_calls>\n", + "<|DSML|invoke name=\"get_weather\">\n", + "<|DSML|parameter name=\"location\" string=\"true\">SF\n", + "\n", + "", + "trailing text", + ], + ); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + } + + #[test] + fn deepseek_v32_streaming_does_not_emit_incomplete_invoke() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + parser.parse_chunk("<|DSML|function_calls>\n").unwrap(); + parser.parse_chunk("<|DSML|invoke name=\"get_weather\">\n").unwrap(); + parser + .parse_chunk( + "<|DSML|parameter name=\"location\" string=\"true\">SF\n", + ) + .unwrap(); + + let error = parser.finish().unwrap_err(); + assert!(error.to_report_string().contains("incomplete DeepSeek DSML tool call")); + } + #[test] + fn deepseek_v32_parser_state_resets_after_finish() { + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let first = parser + .parse_complete(&build_tool_call("get_weather", &[("location", "SF")])) + .unwrap(); + let second = parser + .parse_complete(&build_tool_call("get_weather", &[("location", "NYC")])) + .unwrap(); + + assert_eq!(first.calls().len(), 1); + assert_eq!(second.calls().len(), 1); + assert_eq!( + serde_json::from_str::(&second.calls()[0].arguments).unwrap(), + json!({ "location": "NYC" }) + ); + } + + #[test] + fn deepseek_v32_streaming_matches_parse_complete() { + let full_text = build_tool_call("add", &[("x", "3"), ("y", "4")]); + let chunks = split_by_chars(&full_text, 7); + let mut streaming_parser = DeepSeekV32ToolParser::new(&test_tools()); + let streamed = collect_stream(&mut streaming_parser, &chunks); + + let mut parser = DeepSeekV32ToolParser::new(&test_tools()); + let complete = parser.parse_complete(&full_text).unwrap(); + + assert_eq!(streamed.normal_text(), complete.normal_text()); + assert_eq!(streamed.calls(), complete.calls()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_dsml/deepseek_v4.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_dsml/deepseek_v4.rs new file mode 100644 index 0000000000000000000000000000000000000000..6c2ab21e780cedd381273700c6a427649bb598ae --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_dsml/deepseek_v4.rs @@ -0,0 +1,148 @@ +use super::{DeepSeekDsmlToolParser, DsmlTokens}; +use crate::tool::{Result, StructuralTagModel, Tool, ToolParser, ToolParserOutput}; + +/// Tool parser for DeepSeek V4 models. +/// +/// Example tool call content: +/// +/// ```text +/// <|DSML|tool_calls> +/// <|DSML|invoke name="get_weather"> +/// <|DSML|parameter name="location" string="true">杭州 +/// <|DSML|parameter name="date" string="true">2024-01-16 +/// +/// <|DSML|invoke name="get_weather"> +/// <|DSML|parameter name="location" string="true">北京 +/// <|DSML|parameter name="date" string="true">2024-01-16 +/// +/// +/// ``` +/// +/// Arguments are emitted only after a full `invoke` block is parsed. +/// +/// V4 reuses the V3.2 DSML invoke/parameter grammar but wraps calls in +/// `<|DSML|tool_calls>` instead of `<|DSML|function_calls>`. +/// +/// DeepSeek V4 relies on DSML markers such as `|DSML|`, which are +/// represented as special tokens in the tokenizer and therefore must be +/// preserved during decode for parsing to work. +pub struct DeepSeekV4ToolParser(DeepSeekDsmlToolParser); + +impl DeepSeekV4ToolParser { + /// Create a DeepSeek V4 tool parser. + fn new(tools: &[Tool]) -> Self { + Self(DeepSeekDsmlToolParser::new(tools, DsmlTokens::V4)) + } +} + +impl ToolParser for DeepSeekV4ToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn preserve_special_tokens(&self) -> bool { + true + } + + fn structural_tag_model(&self) -> Option { + Some(StructuralTagModel::DeepSeekV4) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.0.parse_into(chunk, output) + } + + fn finish(&mut self) -> Result { + self.0.finish() + } + + fn reset(&mut self) -> String { + self.0.reset() + } +} + +#[cfg(test)] +mod tests { + use serde_json::{Value, json}; + + use super::DeepSeekV4ToolParser; + use crate::tool::test_utils::{collect_stream, test_tools}; + use crate::tool::{StructuralTagModel, ToolParser, ToolParserTestExt as _}; + + fn build_tool_call(function_name: &str, params: &[(&str, &str)]) -> String { + let params = params + .iter() + .map(|(name, value)| { + format!( + r#"<|DSML|parameter name="{name}" string="true">{value}"# + ) + }) + .collect::>() + .join("\n"); + format!( + "<|DSML|tool_calls>\n<|DSML|invoke name=\"{function_name}\">\n{params}\n\n" + ) + } + + #[test] + fn deepseek_v4_exposes_structural_tag_model() { + let parser = DeepSeekV4ToolParser::new(&test_tools()); + + assert_eq!( + parser.structural_tag_model(), + Some(StructuralTagModel::DeepSeekV4) + ); + } + + #[test] + fn deepseek_v4_parse_complete_reuses_dsml_parser_with_tool_calls_token() { + let mut parser = DeepSeekV4ToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_call( + "get_weather", + &[("location", "SF"), ("date", "2024-01-16")], + )) + .unwrap(); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "location": "SF", + "date": "2024-01-16" + }) + ); + } + + #[test] + fn deepseek_v4_streaming_handles_tool_calls_token_split_across_chunks() { + let mut parser = DeepSeekV4ToolParser::new(&test_tools()); + let output = collect_stream( + &mut parser, + &[ + "Thinking... ", + "<|DSML|", + "tool", + "_calls>\n", + "<|DSML|invoke name=\"get_weather\">\n", + "<|DSML|parameter name=\"location\" string=\"true\">Beijing\n", + "\n", + "", + ], + ); + + assert_eq!(output.normal_text(), "Thinking... "); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "location": "Beijing" }) + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_dsml/mod.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_dsml/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..766bbb0f16bedbeb7f62490a67085a61c9fd830d --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_dsml/mod.rs @@ -0,0 +1,287 @@ +use winnow::ascii::{multispace0 as ws0, multispace1 as ws1}; +use winnow::combinator::{alt, delimited, eof, repeat, seq, terminated}; +use winnow::prelude::*; +use winnow::stream::Partial; +use winnow::token::{literal, rest, take_until}; + +use super::parameters::ToolSchemas; +use super::utils::{MarkerScanState, parse_buffered_event, safe_text_len, take_until_marker}; +use super::{Result, ToolCallDelta, ToolParserOutput}; +use crate::tool::Tool; + +mod deepseek_v32; +mod deepseek_v4; + +pub use deepseek_v4::DeepSeekV4ToolParser; +pub use deepseek_v32::DeepSeekV32ToolParser; + +const INVOKE_START: &str = "<|DSML|invoke"; +const INVOKE_END: &str = ""; +const PARAMETER_START: &str = "<|DSML|parameter"; +const PARAMETER_END: &str = ""; + +type DsmlInput<'i> = Partial<&'i str>; + +#[derive(Debug, Clone, Copy)] +struct DsmlTokens { + tool_calls_start: &'static str, + tool_calls_end: &'static str, +} + +impl DsmlTokens { + const V32: Self = Self { + tool_calls_start: "<|DSML|function_calls>", + tool_calls_end: "", + }; + const V4: Self = Self { + tool_calls_start: "<|DSML|tool_calls>", + tool_calls_end: "", + }; +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum DsmlMode { + Text, + ToolBlock { invoke_end_scan: MarkerScanState }, + Done, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum DsmlEvent { + Text { + len: usize, + }, + ToolCallsStart, + Invoke { + name: String, + raw_params: Vec, + }, + ToolCallsEnd, + IgnoredRest, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +struct DsmlParameter { + name: String, + value: String, + is_string: bool, +} + +/// Tool parser core for DeepSeek DSML tool calls. +struct DeepSeekDsmlToolParser { + buffer: String, + mode: DsmlMode, + emitted_invoke_count: usize, + tool_parameters: ToolSchemas, + tokens: DsmlTokens, +} + +impl DeepSeekDsmlToolParser { + /// Create a parser with DSML tokens for one DeepSeek format. + fn new(tools: &[Tool], tokens: DsmlTokens) -> Self { + Self { + buffer: String::new(), + mode: DsmlMode::Text, + emitted_invoke_count: 0, + tool_parameters: ToolSchemas::from_tools(tools), + tokens, + } + } + + /// Apply one parsed DSML event to parser state and output. + fn apply_event(&mut self, event: DsmlEvent, output: &mut ToolParserOutput) -> Result<()> { + match event { + DsmlEvent::Text { len: consumed_len } => { + output.push_text(&self.buffer[..consumed_len]); + } + DsmlEvent::ToolCallsStart => { + self.mode = DsmlMode::ToolBlock { + invoke_end_scan: MarkerScanState::default(), + }; + } + DsmlEvent::Invoke { name, raw_params } => { + let mut arguments = serde_json::Map::with_capacity(raw_params.len()); + for param in raw_params { + let value = if param.is_string { + serde_json::Value::String(param.value) + } else { + self.tool_parameters.convert_param_with_schema( + &name, + ¶m.name, + param.value, + ) + }; + arguments.insert(param.name, value); + } + let arguments = serde_json::to_string(&arguments) + .map_err(|error| parsing_failed!("failed to serialize arguments: {}", error))?; + + output.push_call(ToolCallDelta { + tool_index: self.emitted_invoke_count, + name: Some(name), + arguments, + }); + self.emitted_invoke_count += 1; + } + DsmlEvent::ToolCallsEnd => self.mode = DsmlMode::Done, + DsmlEvent::IgnoredRest => {} + }; + Ok(()) + } + + fn reset(&mut self) -> String { + self.mode = DsmlMode::Text; + self.emitted_invoke_count = 0; + std::mem::take(&mut self.buffer) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + // Extract tool calls from streaming model output. + // + // Uses a buffer-until-complete-invoke strategy: text is buffered until + // a complete invoke block is available, then parsed and emitted in one + // shot. + self.buffer.push_str(chunk); + + while let Some((event, consumed_len)) = parse_buffered_event(&self.buffer, |input| { + parse_next_dsml_event(input, &mut self.mode, self.tokens) + })? { + self.apply_event(event, output)?; + self.buffer.drain(..consumed_len); + } + + Ok(()) + } + + fn finish(&mut self) -> Result { + let mut output = ToolParserOutput::default(); + match self.mode { + DsmlMode::Text => output.push_text(&self.buffer), + DsmlMode::Done => {} + DsmlMode::ToolBlock { .. } => { + return Err(parsing_failed!("incomplete DeepSeek DSML tool call")); + } + } + let _ = self.reset(); + Ok(output) + } +} + +/// Parse a DSML event for the current parser mode. +fn parse_next_dsml_event( + input: &mut DsmlInput<'_>, + mode: &mut DsmlMode, + tokens: DsmlTokens, +) -> ModalResult { + match mode { + DsmlMode::Text => parse_text_event(input, tokens), + DsmlMode::ToolBlock { invoke_end_scan } => { + parse_tool_block_event(input, tokens, invoke_end_scan) + } + DsmlMode::Done => ignored_rest_event(input), + } +} + +/// Parse a text-mode DSML event. +fn parse_text_event(input: &mut DsmlInput<'_>, tokens: DsmlTokens) -> ModalResult { + alt(( + |input: &mut DsmlInput<'_>| tool_calls_start_event(input, tokens), + |input: &mut DsmlInput<'_>| safe_text_event(input, tokens), + )) + .parse_next(input) +} + +/// Parse a tool-block DSML event. +fn parse_tool_block_event( + input: &mut DsmlInput<'_>, + tokens: DsmlTokens, + invoke_end_scan: &mut MarkerScanState, +) -> ModalResult { + ws0.void().parse_next(input)?; + alt(( + |input: &mut DsmlInput<'_>| invoke_event(input, invoke_end_scan), + |input: &mut DsmlInput<'_>| tool_calls_end_event(input, tokens), + )) + .parse_next(input) +} + +/// Parse a DSML function-calls start marker. +fn tool_calls_start_event(input: &mut DsmlInput<'_>, tokens: DsmlTokens) -> ModalResult { + literal(tokens.tool_calls_start) + .value(DsmlEvent::ToolCallsStart) + .parse_next(input) +} + +/// Parse a DSML function-calls end marker. +fn tool_calls_end_event(input: &mut DsmlInput<'_>, tokens: DsmlTokens) -> ModalResult { + literal(tokens.tool_calls_end).value(DsmlEvent::ToolCallsEnd).parse_next(input) +} + +/// Parse a trailing rest after DSML function calls. +fn ignored_rest_event(input: &mut DsmlInput<'_>) -> ModalResult { + rest.value(DsmlEvent::IgnoredRest).parse_next(input) +} + +/// Parse a safe text run before the next DSML marker. +fn safe_text_event(input: &mut DsmlInput<'_>, tokens: DsmlTokens) -> ModalResult { + safe_text_len(input, tokens.tool_calls_start).map(|len| DsmlEvent::Text { len }) +} + +/// Parse a DSML invoke block. +fn invoke_event( + input: &mut DsmlInput<'_>, + invoke_end_scan: &mut MarkerScanState, +) -> ModalResult { + let (name, body) = seq!( + _: literal(INVOKE_START), + _: ws1, + dsml_name_attr, + _: ws0, + _: ">", + take_until_marker(INVOKE_END, invoke_end_scan), + _: literal(INVOKE_END), + ) + .parse_next(input)?; + let raw_params = parse_invoke_params(body)?; + Ok(DsmlEvent::Invoke { + name: name.to_string(), + raw_params, + }) +} + +/// Parse a DSML invoke body. +fn parse_invoke_params(invoke_body: &str) -> ModalResult> { + let mut input = invoke_body; + delimited(ws0, repeat(0.., terminated(parse_parameter, ws0)), eof).parse_next(&mut input) +} + +/// Parse a DSML parameter block. +fn parse_parameter(input: &mut &str) -> ModalResult { + seq! {DsmlParameter { + _: literal(PARAMETER_START), + _: ws1, + name: name_attr.map(|name: &str| name.to_string()), + _: ws1, + is_string: string_attr.map(|value| value == "true"), + _: ws0, + _: ">", + value: take_until(0.., PARAMETER_END).map(str::to_string), + _: literal(PARAMETER_END), + }} + .parse_next(input) +} + +/// Parse a name attribute. +fn name_attr<'i>(input: &mut &'i str) -> ModalResult<&'i str> { + delimited("name=\"", take_until(1.., "\""), "\"").parse_next(input) +} + +/// Parse a string attribute. +fn string_attr<'i>(input: &mut &'i str) -> ModalResult<&'i str> { + delimited("string=\"", alt(("true", "false")), "\"").parse_next(input) +} + +/// Parse a DSML name attribute. +fn dsml_name_attr<'i>(input: &mut DsmlInput<'i>) -> ModalResult<&'i str> { + delimited("name=\"", take_until(1.., "\""), "\"").parse_next(input) +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_json/deepseek_v3.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_json/deepseek_v3.rs new file mode 100644 index 0000000000000000000000000000000000000000..1b8857265f19b2dfc8e0e0f8caccc9f6b143c971 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_json/deepseek_v3.rs @@ -0,0 +1,246 @@ +use super::{DeepSeekJsonFormat, DeepSeekJsonToolParser}; +use crate::tool::{Result, StructuralTagModel, Tool, ToolParser, ToolParserOutput}; + +/// Tool parser for DeepSeek V3 JSON-fenced tool calls. +/// +/// Example tool call content: +/// +/// ````text +/// <|tool▁calls▁begin|><|tool▁call▁begin|>function<|tool▁sep|>get_weather +/// ```json +/// {"location":"Tokyo"} +/// ```<|tool▁call▁end|><|tool▁calls▁end|> +/// ```` +/// +/// Arguments are already OpenAI-style JSON text inside the markdown fence, so +/// they are streamed as raw argument deltas without schema conversion or JSON +/// normalization. +pub struct DeepSeekV3ToolParser(DeepSeekJsonToolParser); + +impl DeepSeekV3ToolParser { + /// Create a DeepSeek V3 tool parser. + fn new(_tools: &[Tool]) -> Self { + Self(DeepSeekJsonToolParser::new(DeepSeekJsonFormat::V3)) + } +} + +impl ToolParser for DeepSeekV3ToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn structural_tag_model(&self) -> Option { + Some(StructuralTagModel::DeepSeekR1) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.0.parse_into(chunk, output) + } + + fn finish(&mut self) -> Result { + self.0.finish() + } + + fn reset(&mut self) -> String { + self.0.reset() + } +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + use thiserror_ext::AsReport; + + use super::DeepSeekV3ToolParser; + use crate::tool::deepseek_json::{ + TOOL_CALL_SEPARATOR, TOOL_CALL_START, TOOL_CALLS_END, TOOL_CALLS_START, V3_ARGUMENT_END, + V3_JSON_START, + }; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{ToolParser, ToolParserOutput, ToolParserTestExt as _}; + + fn v3_tool_call(function_name: &str, arguments: &str) -> String { + format!( + "{TOOL_CALL_START}function{TOOL_CALL_SEPARATOR}{function_name}{V3_JSON_START}{arguments}{V3_ARGUMENT_END}" + ) + } + + fn tool_section(tool_calls: &[String]) -> String { + format!("{TOOL_CALLS_START}{}{TOOL_CALLS_END}", tool_calls.join("")) + } + + #[test] + fn deepseek_v3_parse_complete_without_tool_call_keeps_text() { + let mut parser = DeepSeekV3ToolParser::new(&test_tools()); + let output = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn deepseek_v3_parse_complete_extracts_raw_json_arguments() { + let mut parser = DeepSeekV3ToolParser::new(&test_tools()); + let arguments = r#"{ "location": "Tokyo", "days": "3" }"#; + let output = parser + .parse_complete(&format!( + "Let me check.\n{} trailing text", + tool_section(&[v3_tool_call("get_weather", arguments)]) + )) + .unwrap(); + + assert_eq!(output.normal_text(), "Let me check.\n"); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].tool_index, 0); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn deepseek_v3_does_not_validate_or_normalize_arguments() { + let mut parser = DeepSeekV3ToolParser::new(&test_tools()); + let arguments = r#"{"location":"Tokyo",}"#; + let output = parser + .parse_complete(&tool_section(&[v3_tool_call("get_weather", arguments)])) + .unwrap(); + + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn deepseek_v3_streaming_emits_argument_deltas() { + let mut parser = DeepSeekV3ToolParser::new(&test_tools()); + let chunks = [ + TOOL_CALLS_START, + TOOL_CALL_START, + "function", + TOOL_CALL_SEPARATOR, + "get_weather", + V3_JSON_START, + "{\"location\":", + "\"Beijing\"", + "}", + V3_ARGUMENT_END, + TOOL_CALLS_END, + ]; + + let mut output = ToolParserOutput::default(); + let mut observed_arguments = Vec::new(); + for chunk in chunks { + let next = parser.parse_chunk(chunk).unwrap(); + observed_arguments.extend( + next.calls() + .iter() + .filter(|call| call.name.is_none()) + .map(|call| call.arguments.clone()), + ); + output.append(next); + } + output.append(parser.finish().unwrap()); + + assert_eq!(observed_arguments, ["{\"location\":", "\"Beijing\"", "}"]); + assert_eq!( + output.coalesce().calls()[0].arguments, + r#"{"location":"Beijing"}"# + ); + } + + #[test] + fn deepseek_v3_streaming_handles_split_markers() { + let input = format!( + "hello {}", + tool_section(&[v3_tool_call("get_weather", r#"{"location":"Tokyo"}"#)]) + ); + let chunks = split_by_chars(&input, 5); + let mut parser = DeepSeekV3ToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.normal_text(), "hello "); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].arguments, r#"{"location":"Tokyo"}"#); + } + + #[test] + fn deepseek_v3_keeps_fenced_end_marker_literal_inside_json_string() { + let mut parser = DeepSeekV3ToolParser::new(&test_tools()); + let arguments = format!("{{\"text\":\"literal {V3_ARGUMENT_END} inside\"}}"); + let input = tool_section(&[v3_tool_call("echo", &arguments)]); + + let output = parser.parse_complete(&input).unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn deepseek_v3_streaming_extracts_multiple_tool_calls() { + let input = tool_section(&[ + v3_tool_call("get_weather", r#"{"location":"Shanghai"}"#), + v3_tool_call("add", r#"{"x":1,"y":2}"#), + ]); + let chunks = split_by_chars(&input, 7); + let mut parser = DeepSeekV3ToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + expect![[r#" + ToolParserOutput { + events: [ + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "get_weather", + ), + arguments: "{\"location\":\"Shanghai\"}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 1, + name: Some( + "add", + ), + arguments: "{\"x\":1,\"y\":2}", + }, + ), + ], + } + "#]] + .assert_debug_eq(&output); + } + + #[test] + fn deepseek_v3_finish_fails_incomplete_tool_call() { + let mut parser = DeepSeekV3ToolParser::new(&test_tools()); + parser + .parse_chunk(&format!( + "{TOOL_CALLS_START}{TOOL_CALL_START}function{TOOL_CALL_SEPARATOR}get_weather{V3_JSON_START}{{\"location\"" + )) + .unwrap(); + + let error = parser.finish().unwrap_err(); + + expect!["tool parser parsing failed: incomplete DeepSeek V3 tool call"] + .assert_eq(&error.to_report_string()); + } + + #[test] + fn deepseek_v3_malformed_type_fails_fast() { + let mut parser = DeepSeekV3ToolParser::new(&test_tools()); + let input = format!( + "{TOOL_CALLS_START}{TOOL_CALL_START}tool{TOOL_CALL_SEPARATOR}get_weather{V3_JSON_START}{{}}" + ); + + let error = parser.parse_chunk(&input).unwrap_err(); + + expect![[ + r#"tool parser parsing failed: near "tool<|tool▁sep|>get_weather\n```json\n{}": "# + ]] + .assert_eq(&error.to_report_string()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_json/deepseek_v31.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_json/deepseek_v31.rs new file mode 100644 index 0000000000000000000000000000000000000000..259512e5ac99e9f6f3a1c3ae48b355ba528a702c --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_json/deepseek_v31.rs @@ -0,0 +1,248 @@ +use super::{DeepSeekJsonFormat, DeepSeekJsonToolParser}; +use crate::tool::{Result, StructuralTagModel, Tool, ToolParser, ToolParserOutput}; + +/// Tool parser for DeepSeek V3.1 raw JSON tool calls. +/// +/// Example tool call content: +/// +/// ```text +/// <|tool▁calls▁begin|><|tool▁call▁begin|>get_weather<|tool▁sep|>{"location":"Tokyo"}<|tool▁call▁end|><|tool▁calls▁end|> +/// ``` +/// +/// Arguments are already OpenAI-style JSON text, so they are streamed as raw +/// argument deltas without schema conversion or JSON normalization. +pub struct DeepSeekV31ToolParser(DeepSeekJsonToolParser); + +impl DeepSeekV31ToolParser { + /// Create a DeepSeek V3.1 tool parser. + fn new(_tools: &[Tool]) -> Self { + Self(DeepSeekJsonToolParser::new(DeepSeekJsonFormat::V31)) + } +} + +impl ToolParser for DeepSeekV31ToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn structural_tag_model(&self) -> Option { + Some(StructuralTagModel::DeepSeekV31) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.0.parse_into(chunk, output) + } + + fn finish(&mut self) -> Result { + self.0.finish() + } + + fn reset(&mut self) -> String { + self.0.reset() + } +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + use thiserror_ext::AsReport; + + use super::DeepSeekV31ToolParser; + use crate::tool::deepseek_json::{ + TOOL_CALL_END, TOOL_CALL_SEPARATOR, TOOL_CALL_START, TOOL_CALLS_END, TOOL_CALLS_START, + }; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{ToolParser, ToolParserOutput, ToolParserTestExt as _}; + + fn v31_tool_call(function_name: &str, arguments: &str) -> String { + format!("{TOOL_CALL_START}{function_name}{TOOL_CALL_SEPARATOR}{arguments}{TOOL_CALL_END}") + } + + fn tool_section(tool_calls: &[String]) -> String { + format!("{TOOL_CALLS_START}{}{TOOL_CALLS_END}", tool_calls.join("")) + } + + #[test] + fn deepseek_v31_parse_complete_without_tool_call_keeps_text() { + let mut parser = DeepSeekV31ToolParser::new(&test_tools()); + let output = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn deepseek_v31_parse_complete_extracts_raw_json_arguments() { + let mut parser = DeepSeekV31ToolParser::new(&test_tools()); + let arguments = r#"{ "location": "Tokyo", "days": "3" }"#; + let output = parser + .parse_complete(&format!( + "Let me check.{} trailing text", + tool_section(&[v31_tool_call("get_weather", arguments)]) + )) + .unwrap(); + + assert_eq!(output.normal_text(), "Let me check."); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].tool_index, 0); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn deepseek_v31_does_not_validate_or_normalize_arguments() { + let mut parser = DeepSeekV31ToolParser::new(&test_tools()); + let arguments = r#"{"location":"Tokyo",}"#; + let output = parser + .parse_complete(&tool_section(&[v31_tool_call("get_weather", arguments)])) + .unwrap(); + + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn deepseek_v31_streaming_emits_argument_deltas() { + let mut parser = DeepSeekV31ToolParser::new(&test_tools()); + let chunks = [ + TOOL_CALLS_START, + TOOL_CALL_START, + "get_weather", + TOOL_CALL_SEPARATOR, + "{\"location\":", + "\"Beijing\"", + "}", + TOOL_CALL_END, + TOOL_CALLS_END, + ]; + + let mut output = ToolParserOutput::default(); + let mut observed_arguments = Vec::new(); + for chunk in chunks { + let next = parser.parse_chunk(chunk).unwrap(); + observed_arguments.extend( + next.calls() + .iter() + .filter(|call| call.name.is_none()) + .map(|call| call.arguments.clone()), + ); + output.append(next); + } + output.append(parser.finish().unwrap()); + + assert_eq!(observed_arguments, ["{\"location\":", "\"Beijing\"", "}"]); + assert_eq!( + output.coalesce().calls()[0].arguments, + r#"{"location":"Beijing"}"# + ); + } + + #[test] + fn deepseek_v31_streaming_handles_split_markers() { + let input = format!( + "hello {}", + tool_section(&[v31_tool_call("get_weather", r#"{"location":"Tokyo"}"#)]) + ); + let chunks = split_by_chars(&input, 5); + let mut parser = DeepSeekV31ToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.normal_text(), "hello "); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].arguments, r#"{"location":"Tokyo"}"#); + } + + #[test] + fn deepseek_v31_keeps_end_marker_literal_inside_json_string() { + let mut parser = DeepSeekV31ToolParser::new(&test_tools()); + let arguments = format!(r#"{{"text":"literal {TOOL_CALL_END} inside"}}"#); + let input = tool_section(&[v31_tool_call("echo", &arguments)]); + + let output = parser.parse_complete(&input).unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn deepseek_v31_streaming_extracts_multiple_tool_calls() { + let input = tool_section(&[ + v31_tool_call("get_weather", r#"{"location":"Shanghai"}"#), + v31_tool_call("add", r#"{"x":1,"y":2}"#), + ]); + let chunks = split_by_chars(&input, 7); + let mut parser = DeepSeekV31ToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + expect![[r#" + ToolParserOutput { + events: [ + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "get_weather", + ), + arguments: "{\"location\":\"Shanghai\"}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 1, + name: Some( + "add", + ), + arguments: "{\"x\":1,\"y\":2}", + }, + ), + ], + } + "#]] + .assert_debug_eq(&output); + } + + #[test] + fn deepseek_v31_streaming_drops_eos_after_complete_tool_calls() { + let input = format!( + "{}<|end▁of▁sentence|>", + tool_section(&[v31_tool_call("get_weather", r#"{"location":"Tokyo"}"#)]) + ); + let mut parser = DeepSeekV31ToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &[&input]); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].arguments, r#"{"location":"Tokyo"}"#); + } + + #[test] + fn deepseek_v31_finish_fails_incomplete_tool_call() { + let mut parser = DeepSeekV31ToolParser::new(&test_tools()); + parser + .parse_chunk(&format!( + "{TOOL_CALLS_START}{TOOL_CALL_START}get_weather{TOOL_CALL_SEPARATOR}{{\"location\"" + )) + .unwrap(); + + let error = parser.finish().unwrap_err(); + + expect!["tool parser parsing failed: incomplete DeepSeek V3.1 tool call"] + .assert_eq(&error.to_report_string()); + } + + #[test] + fn deepseek_v31_malformed_empty_name_fails_fast() { + let mut parser = DeepSeekV31ToolParser::new(&test_tools()); + let input = format!("{TOOL_CALLS_START}{TOOL_CALL_START}{TOOL_CALL_SEPARATOR}{{}}"); + + let error = parser.parse_chunk(&input).unwrap_err(); + + expect![[r#"tool parser parsing failed: near "<|tool▁sep|>{}": "#]] + .assert_eq(&error.to_report_string()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_json/mod.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_json/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..e775e08921d780a5b86206c17c7dc2084932606f --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/deepseek_json/mod.rs @@ -0,0 +1,304 @@ +mod deepseek_v3; +mod deepseek_v31; + +pub use deepseek_v3::DeepSeekV3ToolParser; +pub use deepseek_v31::DeepSeekV31ToolParser; +use winnow::ascii::multispace0 as ws0; +use winnow::combinator::{alt, seq}; +use winnow::prelude::*; +use winnow::stream::Partial; +use winnow::token::{literal, rest, take_until}; + +use super::utils::{JsonObjectScanState, parse_buffered_event, safe_text_len, take_json_object}; +use super::{Result, ToolCallDelta, ToolParserOutput}; + +pub(super) const TOOL_CALLS_START: &str = "<|tool▁calls▁begin|>"; +pub(super) const TOOL_CALLS_END: &str = "<|tool▁calls▁end|>"; +pub(super) const TOOL_CALL_START: &str = "<|tool▁call▁begin|>"; +pub(super) const TOOL_CALL_END: &str = "<|tool▁call▁end|>"; +pub(super) const TOOL_CALL_SEPARATOR: &str = "<|tool▁sep|>"; +pub(super) const V3_JSON_START: &str = "\n```json\n"; +pub(super) const V3_ARGUMENT_END: &str = "\n```<|tool▁call▁end|>"; + +type DeepSeekJsonInput<'i> = Partial<&'i str>; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum DeepSeekJsonFormat { + V3, + V31, +} + +impl DeepSeekJsonFormat { + /// Return the parser name used in diagnostics. + const fn parser_name(self) -> &'static str { + match self { + Self::V3 => "DeepSeek V3", + Self::V31 => "DeepSeek V3.1", + } + } + + /// Return the marker that closes the raw JSON arguments payload. + const fn argument_end_marker(self) -> &'static str { + match self { + Self::V3 => V3_ARGUMENT_END, + Self::V31 => TOOL_CALL_END, + } + } +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum DeepSeekJsonMode { + Text, + ToolBlock, + Header, + Arguments { json_scan: JsonObjectScanState }, + Done, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum DeepSeekJsonEvent { + Text { len: usize }, + ToolCallsStart, + ToolCallStart, + ToolCallHeader { function_name: String }, + Arguments { len: usize }, + ToolCallEnd, + ToolCallsEnd, + IgnoredRest, +} + +/// Tool parser core for DeepSeek JSON-argument tool calls. +struct DeepSeekJsonToolParser { + buffer: String, + mode: DeepSeekJsonMode, + active_tool_index: Option, + emitted_tool_count: usize, + format: DeepSeekJsonFormat, +} + +impl DeepSeekJsonToolParser { + /// Create a parser for one DeepSeek JSON-argument format. + fn new(format: DeepSeekJsonFormat) -> Self { + Self { + buffer: String::new(), + mode: DeepSeekJsonMode::Text, + active_tool_index: None, + emitted_tool_count: 0, + format, + } + } + + /// Apply one parsed DeepSeek JSON event to parser state and output. + fn apply_event( + &mut self, + event: DeepSeekJsonEvent, + output: &mut ToolParserOutput, + ) -> Result<()> { + match event { + DeepSeekJsonEvent::Text { len: consumed_len } => { + output.push_text(&self.buffer[..consumed_len]); + } + DeepSeekJsonEvent::ToolCallsStart => self.mode = DeepSeekJsonMode::ToolBlock, + DeepSeekJsonEvent::ToolCallStart => self.mode = DeepSeekJsonMode::Header, + DeepSeekJsonEvent::ToolCallHeader { function_name } => { + let tool_index = self.emitted_tool_count; + self.emitted_tool_count += 1; + self.active_tool_index = Some(tool_index); + self.mode = DeepSeekJsonMode::Arguments { + json_scan: JsonObjectScanState::default(), + }; + output.push_call(ToolCallDelta { + tool_index, + name: Some(function_name), + arguments: String::new(), + }); + } + DeepSeekJsonEvent::Arguments { len: consumed_len } => { + let Some(tool_index) = self.active_tool_index else { + return Err(parsing_failed!( + "{} arguments without an active tool call", + self.format.parser_name() + )); + }; + output.push_call(ToolCallDelta { + tool_index, + name: None, + arguments: self.buffer[..consumed_len].to_string(), + }); + } + DeepSeekJsonEvent::ToolCallEnd => { + self.active_tool_index = None; + self.mode = DeepSeekJsonMode::ToolBlock; + } + DeepSeekJsonEvent::ToolCallsEnd => { + self.active_tool_index = None; + self.mode = DeepSeekJsonMode::Done; + } + DeepSeekJsonEvent::IgnoredRest => {} + } + Ok(()) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.buffer.push_str(chunk); + + while let Some((event, consumed_len)) = parse_buffered_event(&self.buffer, |input| { + parse_next_deepseek_json_event(input, &mut self.mode, self.format) + })? { + self.apply_event(event, output)?; + self.buffer.drain(..consumed_len); + } + + Ok(()) + } + + fn finish(&mut self) -> Result { + let mut output = ToolParserOutput::default(); + match &self.mode { + DeepSeekJsonMode::Text => output.push_text(&self.buffer), + DeepSeekJsonMode::ToolBlock | DeepSeekJsonMode::Done => {} + DeepSeekJsonMode::Header | DeepSeekJsonMode::Arguments { .. } => { + return Err(parsing_failed!( + "incomplete {} tool call", + self.format.parser_name() + )); + } + } + let _ = self.reset(); + Ok(output) + } + + fn reset(&mut self) -> String { + self.mode = DeepSeekJsonMode::Text; + self.active_tool_index = None; + self.emitted_tool_count = 0; + std::mem::take(&mut self.buffer) + } +} + +/// Parse a DeepSeek JSON event for the current parser mode. +fn parse_next_deepseek_json_event( + input: &mut DeepSeekJsonInput<'_>, + mode: &mut DeepSeekJsonMode, + format: DeepSeekJsonFormat, +) -> ModalResult { + match mode { + DeepSeekJsonMode::Text => parse_text_event(input), + DeepSeekJsonMode::ToolBlock => parse_tool_block_event(input), + DeepSeekJsonMode::Header => tool_call_header_event(input, format), + DeepSeekJsonMode::Arguments { json_scan } => { + parse_arguments_event(input, format, json_scan) + } + DeepSeekJsonMode::Done => ignored_rest_event(input), + } +} + +/// Parse a text-mode DeepSeek JSON event. +fn parse_text_event(input: &mut DeepSeekJsonInput<'_>) -> ModalResult { + alt((tool_calls_start_event, safe_text_event)).parse_next(input) +} + +/// Parse one event inside the DeepSeek tool-calls section. +fn parse_tool_block_event(input: &mut DeepSeekJsonInput<'_>) -> ModalResult { + ws0.void().parse_next(input)?; + alt((tool_calls_end_event, tool_call_start_event)).parse_next(input) +} + +/// Parse one event inside a DeepSeek tool-call arguments payload. +fn parse_arguments_event( + input: &mut DeepSeekJsonInput<'_>, + format: DeepSeekJsonFormat, + json_scan: &mut JsonObjectScanState, +) -> ModalResult { + if json_scan.complete() { + tool_call_end_event(input, format) + } else { + argument_delta_event(input, json_scan) + } +} + +/// Parse a DeepSeek tool-calls start marker. +fn tool_calls_start_event(input: &mut DeepSeekJsonInput<'_>) -> ModalResult { + literal(TOOL_CALLS_START) + .value(DeepSeekJsonEvent::ToolCallsStart) + .parse_next(input) +} + +/// Parse a DeepSeek tool-calls end marker. +fn tool_calls_end_event(input: &mut DeepSeekJsonInput<'_>) -> ModalResult { + literal(TOOL_CALLS_END).value(DeepSeekJsonEvent::ToolCallsEnd).parse_next(input) +} + +/// Parse a DeepSeek tool-call start marker. +fn tool_call_start_event(input: &mut DeepSeekJsonInput<'_>) -> ModalResult { + literal(TOOL_CALL_START) + .value(DeepSeekJsonEvent::ToolCallStart) + .parse_next(input) +} + +/// Parse a DeepSeek tool-call end marker. +fn tool_call_end_event( + input: &mut DeepSeekJsonInput<'_>, + format: DeepSeekJsonFormat, +) -> ModalResult { + literal(format.argument_end_marker()) + .value(DeepSeekJsonEvent::ToolCallEnd) + .parse_next(input) +} + +/// Parse a DeepSeek tool-call header before the JSON arguments payload. +fn tool_call_header_event( + input: &mut DeepSeekJsonInput<'_>, + format: DeepSeekJsonFormat, +) -> ModalResult { + match format { + DeepSeekJsonFormat::V3 => v3_tool_call_header_event(input), + DeepSeekJsonFormat::V31 => v31_tool_call_header_event(input), + } +} + +/// Parse a DeepSeek V3 tool-call header. +fn v3_tool_call_header_event(input: &mut DeepSeekJsonInput<'_>) -> ModalResult { + let name = seq!( + _: literal("function"), + _: literal(TOOL_CALL_SEPARATOR), + take_until(1.., V3_JSON_START), + _: literal(V3_JSON_START), + ) + .parse_next(input)?; + + Ok(DeepSeekJsonEvent::ToolCallHeader { + function_name: name.0.trim().to_string(), + }) +} + +/// Parse a DeepSeek V3.1 tool-call header. +fn v31_tool_call_header_event(input: &mut DeepSeekJsonInput<'_>) -> ModalResult { + let (name, _) = ( + take_until(1.., TOOL_CALL_SEPARATOR), + literal(TOOL_CALL_SEPARATOR), + ) + .parse_next(input)?; + + Ok(DeepSeekJsonEvent::ToolCallHeader { + function_name: name.trim().to_string(), + }) +} + +/// Parse a DeepSeek raw JSON arguments delta. +fn argument_delta_event( + input: &mut DeepSeekJsonInput<'_>, + json_scan: &mut JsonObjectScanState, +) -> ModalResult { + take_json_object(input, json_scan).map(|len| DeepSeekJsonEvent::Arguments { len }) +} + +/// Parse a safe text run before the next DeepSeek tool-calls section. +fn safe_text_event(input: &mut DeepSeekJsonInput<'_>) -> ModalResult { + safe_text_len(input, TOOL_CALLS_START).map(|len| DeepSeekJsonEvent::Text { len }) +} + +/// Parse ignored rest after the DeepSeek tool-calls section ends. +fn ignored_rest_event(input: &mut DeepSeekJsonInput<'_>) -> ModalResult { + rest.value(DeepSeekJsonEvent::IgnoredRest).parse_next(input) +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/error.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/error.rs new file mode 100644 index 0000000000000000000000000000000000000000..c2cd3756f4eb06375414d06517b3b662dbf91992 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/error.rs @@ -0,0 +1,17 @@ +use thiserror::Error; +use thiserror_ext::Macro; + +/// Result alias for tool parser operations. +pub type Result = std::result::Result; + +/// Errors produced while creating or running tool parsers. +#[derive(Debug, Error, Macro)] +#[thiserror_ext(macro(path = "crate::tool::error"))] +pub enum ToolParserError { + #[error("tool parser parsing failed: {message}")] + ParsingFailed { message: String }, + #[error( + "`{name}` only provides a unified parser; the same reasoning parser and tool parser should be specified together" + )] + DummyUnifiedParser { name: String }, +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/glm_xml/glm45_moe.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/glm_xml/glm45_moe.rs new file mode 100644 index 0000000000000000000000000000000000000000..537cb3e9ee0bb57bdbd1b0ca8b91b1b9542fafb1 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/glm_xml/glm45_moe.rs @@ -0,0 +1,44 @@ +use super::{GlmXmlToolParser, Separator}; +use crate::tool::{Result, Tool, ToolParser, ToolParserOutput}; + +/// Tool parser for GLM-4.5/4.6 MoE XML-style tool calls. +/// +/// Example tool call content: +/// +/// ```text +/// get_weather +/// city +/// Hangzhou +/// +/// ``` +/// +/// Arguments are emitted only after a full `tool_call` block is parsed. +pub struct Glm45MoeToolParser(GlmXmlToolParser); + +impl Glm45MoeToolParser { + /// Create a GLM-4.5/4.6 MoE tool parser. + pub(super) fn new(tools: &[Tool]) -> Self { + Self(GlmXmlToolParser::new(tools, Separator::Newline)) + } +} + +impl ToolParser for Glm45MoeToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.0.parse_into(chunk, output) + } + + fn finish(&mut self) -> Result { + self.0.finish() + } + + fn reset(&mut self) -> String { + self.0.reset() + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/glm_xml/glm47_moe.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/glm_xml/glm47_moe.rs new file mode 100644 index 0000000000000000000000000000000000000000..93a41bc039b22ad8bbe6e3071ffbd6188f252c4f --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/glm_xml/glm47_moe.rs @@ -0,0 +1,144 @@ +use super::{GlmXmlToolParser, Separator}; +use crate::tool::{Result, StructuralTagModel, Tool, ToolParser, ToolParserOutput}; + +/// Tool parser for GLM-4.7 MoE XML-style tool calls. +/// +/// GLM-4.7 reuses the GLM-4.5 parser with a more flexible function-name +/// separator, so the name may be followed by whitespace, a newline, or the +/// first `` tag directly. +pub struct Glm47MoeToolParser(GlmXmlToolParser); + +impl Glm47MoeToolParser { + fn new(tools: &[Tool]) -> Self { + Self(GlmXmlToolParser::new(tools, Separator::Flexible)) + } +} + +impl ToolParser for Glm47MoeToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn structural_tag_model(&self) -> Option { + Some(StructuralTagModel::Glm47) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.0.parse_into(chunk, output) + } + + fn finish(&mut self) -> Result { + self.0.finish() + } + + fn reset(&mut self) -> String { + self.0.reset() + } +} + +#[cfg(test)] +mod tests { + use serde_json::{Value, json}; + + use super::Glm47MoeToolParser; + use crate::tool::ToolParserTestExt as _; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + + fn glm47_tool_call(function_name: &str, params: &[(&str, &str)]) -> String { + let params = params + .iter() + .map(|(name, value)| format!("{name}{value}")) + .collect::>() + .join(""); + format!("{function_name}{params}") + } + + #[test] + fn glm47_parse_complete_extracts_single_tool_call() { + let mut parser = Glm47MoeToolParser::new(&test_tools()); + let output = format!( + "Let me search for that.\n{}", + glm47_tool_call( + "get_weather", + &[("city", "Beijing"), ("date", "2024-12-25")] + ) + ); + + let output = parser.parse_complete(&output).unwrap(); + + assert_eq!(output.normal_text(), "Let me search for that.\n"); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({"city": "Beijing", "date": "2024-12-25"}) + ); + } + + #[test] + fn glm47_streaming_extracts_multiple_tool_calls() { + let mut parser = Glm47MoeToolParser::new(&test_tools()); + let output = format!( + "{}{}", + glm47_tool_call("get_weather", &[("city", "Shanghai")]), + glm47_tool_call("add", &[("x", "1"), ("y", "2")]) + ); + + let chunks = split_by_chars(&output, 7); + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.normal_text(), ""); + assert_eq!(output.calls().len(), 2); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[1].name.as_deref(), Some("add")); + assert_eq!( + serde_json::from_str::(&output.calls()[1].arguments).unwrap(), + json!({"x": 1, "y": 2}) + ); + } + + #[test] + fn glm47_parse_complete_converts_schema_types() { + let mut parser = Glm47MoeToolParser::new(&test_tools()); + let output = parser + .parse_complete(&glm47_tool_call( + "convert", + &[ + ("whole", "42"), + ("flag", "true"), + ("payload", r#"{"nested":{"key":"value"}}"#), + ("items", "[1, 2, 3]"), + ("empty", ""), + ], + )) + .unwrap(); + + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "whole": 42, + "flag": true, + "payload": {"nested": {"key": "value"}}, + "items": [1, 2, 3], + "empty": "" + }) + ); + } + + #[test] + fn glm47_parse_complete_extracts_zero_argument_call() { + let mut parser = Glm47MoeToolParser::new(&test_tools()); + + let output = parser.parse_complete("add").unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("add")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({}) + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/glm_xml/mod.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/glm_xml/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..26f0f618b828429191758477404073c9784a20f2 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/glm_xml/mod.rs @@ -0,0 +1,443 @@ +use winnow::ascii::multispace0 as ws0; +use winnow::combinator::{alt, eof, repeat, seq, terminated}; +use winnow::prelude::*; +use winnow::stream::Partial; +use winnow::token::{literal, rest, take_until, take_while}; + +use super::parameters::ToolSchemas; +use super::utils::{MarkerScanState, parse_buffered_event, safe_text_len, take_until_marker}; +use super::{Result, ToolCallDelta, ToolParserOutput}; +use crate::tool::Tool; + +mod glm45_moe; +mod glm47_moe; + +pub use glm45_moe::Glm45MoeToolParser; +pub use glm47_moe::Glm47MoeToolParser; + +const TOOL_CALL_START: &str = ""; +const TOOL_CALL_END: &str = ""; +const ARG_KEY_START: &str = ""; +const ARG_KEY_END: &str = ""; +const ARG_VALUE_START: &str = ""; +const ARG_VALUE_END: &str = ""; + +type GlmInput<'i> = Partial<&'i str>; + +#[derive(Debug, Clone, PartialEq, Eq)] +enum GlmMode { + Text, + ToolCall { tool_call_end_scan: MarkerScanState }, + AfterToolCall, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Separator { + /// GLM-4.5/4.6 format: function name must end at a newline before + /// arguments. + Newline, + /// GLM-4.7 format: function name may end at whitespace or directly before + /// ``. + Flexible, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum GlmEvent { + Text { + len: usize, + }, + ToolCallStart, + ToolCall { + name: String, + raw_params: Vec<(String, String)>, + }, + IgnoredRest, +} + +/// Tool parser core for GLM XML-style tool calls. +struct GlmXmlToolParser { + buffer: String, + mode: GlmMode, + emitted_tool_count: usize, + tool_parameters: ToolSchemas, + separator: Separator, +} + +impl GlmXmlToolParser { + /// Create a GLM XML tool parser with a function-name separator. + fn new(tools: &[Tool], separator: Separator) -> Self { + Self { + buffer: String::new(), + mode: GlmMode::Text, + emitted_tool_count: 0, + tool_parameters: ToolSchemas::from_tools(tools), + separator, + } + } + + /// Apply one parsed GLM event to parser state and output. + fn apply_event(&mut self, event: GlmEvent, output: &mut ToolParserOutput) -> Result<()> { + match event { + GlmEvent::Text { len: consumed_len } => { + output.push_text(&self.buffer[..consumed_len]); + } + GlmEvent::ToolCallStart => { + self.mode = GlmMode::ToolCall { + tool_call_end_scan: MarkerScanState::default(), + }; + } + GlmEvent::ToolCall { name, raw_params } => { + self.mode = GlmMode::AfterToolCall; + let arguments = self.tool_parameters.convert_params_with_schema(&name, raw_params); + let arguments = serde_json::to_string(&arguments) + .map_err(|error| parsing_failed!("failed to serialize arguments: {}", error))?; + + output.push_call(ToolCallDelta { + tool_index: self.emitted_tool_count, + name: Some(name), + arguments, + }); + self.emitted_tool_count += 1; + } + GlmEvent::IgnoredRest => {} + } + Ok(()) + } + + fn reset(&mut self) -> String { + self.mode = GlmMode::Text; + self.emitted_tool_count = 0; + std::mem::take(&mut self.buffer) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.buffer.push_str(chunk); + + while let Some((event, consumed_len)) = parse_buffered_event(&self.buffer, |input| { + parse_next_glm_event(input, &mut self.mode, self.separator) + })? { + self.apply_event(event, output)?; + self.buffer.drain(..consumed_len); + } + + Ok(()) + } + + fn finish(&mut self) -> Result { + let mut output = ToolParserOutput::default(); + if !self.buffer.is_empty() { + match self.mode { + GlmMode::Text => output.push_text(&self.buffer), + GlmMode::ToolCall { .. } => { + return Err(parsing_failed!("incomplete GLM MoE tool call")); + } + GlmMode::AfterToolCall => {} + } + } + let _ = self.reset(); + Ok(output) + } +} + +/// Parse a GLM event for the current parser mode. +fn parse_next_glm_event( + input: &mut GlmInput<'_>, + mode: &mut GlmMode, + separator: Separator, +) -> ModalResult { + match mode { + GlmMode::Text => parse_text_event(input), + GlmMode::ToolCall { tool_call_end_scan } => { + tool_call_event(input, separator, tool_call_end_scan) + } + GlmMode::AfterToolCall => after_tool_call_event(input), + } +} + +/// Parse a text-mode GLM event. +fn parse_text_event(input: &mut GlmInput<'_>) -> ModalResult { + alt((tool_call_start_event, safe_text_event)).parse_next(input) +} + +/// Parse a GLM tool-call start marker. +fn tool_call_start_event(input: &mut GlmInput<'_>) -> ModalResult { + literal(TOOL_CALL_START).value(GlmEvent::ToolCallStart).parse_next(input) +} + +/// Parse a safe text run before the next GLM marker. +fn safe_text_event(input: &mut GlmInput<'_>) -> ModalResult { + safe_text_len(input, TOOL_CALL_START).map(|len| GlmEvent::Text { len }) +} + +/// Parse text after a completed GLM tool call. +fn after_tool_call_event(input: &mut GlmInput<'_>) -> ModalResult { + ws0.void().parse_next(input)?; + alt((tool_call_start_event, ignored_rest_event)).parse_next(input) +} + +/// Parse a trailing rest after GLM tool calls. +fn ignored_rest_event(input: &mut GlmInput<'_>) -> ModalResult { + rest.value(GlmEvent::IgnoredRest).parse_next(input) +} + +/// Parse a complete GLM tool call. +fn tool_call_event( + input: &mut GlmInput<'_>, + separator: Separator, + tool_call_end_scan: &mut MarkerScanState, +) -> ModalResult { + let (body,) = seq!( + take_until_marker(TOOL_CALL_END, tool_call_end_scan), + _: literal(TOOL_CALL_END), + ) + .parse_next(input)?; + + parse_tool_call_body(body, separator) +} + +/// Parse a GLM tool-call body. +fn parse_tool_call_body(body: &str, separator: Separator) -> ModalResult { + let mut input = body; + let (name, raw_params) = match separator { + Separator::Newline => seq!( + _: ws0, + parse_newline_separated_function_name, + parse_parameters, + _: ws0, + _: eof, + ) + .parse_next(&mut input)?, + Separator::Flexible => seq!( + _: ws0, + parse_flexible_function_name, + parse_parameters, + _: ws0, + _: eof, + ) + .parse_next(&mut input)?, + }; + + Ok(GlmEvent::ToolCall { + name: name.to_string(), + raw_params, + }) +} + +/// Parse a GLM-4.5 newline-separated function name. +fn parse_newline_separated_function_name<'i>(input: &mut &'i str) -> ModalResult<&'i str> { + terminated(take_until(1.., "\n"), "\n").map(str::trim).parse_next(input) +} + +/// Parse a GLM-4.7 whitespace-or-tag-separated function name. +fn parse_flexible_function_name<'i>(input: &mut &'i str) -> ModalResult<&'i str> { + terminated( + take_while(1.., |ch: char| !ch.is_whitespace() && ch != '<'), + ws0, + ) + .parse_next(input) +} + +/// Parse GLM argument key-value pairs. +fn parse_parameters(input: &mut &str) -> ModalResult> { + repeat(0.., terminated(parse_parameter, ws0)).parse_next(input) +} + +/// Parse a GLM argument key-value pair. +fn parse_parameter(input: &mut &str) -> ModalResult<(String, String)> { + let (key, value) = seq!( + _: literal(ARG_KEY_START), + take_until(1.., ARG_KEY_END), + _: literal(ARG_KEY_END), + _: ws0, + _: literal(ARG_VALUE_START), + take_until(0.., ARG_VALUE_END).map(str::trim), + _: literal(ARG_VALUE_END), + ) + .parse_next(input)?; + + Ok((key.trim().to_string(), value.to_string())) +} + +#[cfg(test)] +mod tests { + use serde_json::{Value, json}; + use thiserror_ext::AsReport; + + use super::Glm45MoeToolParser; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{ToolParser, ToolParserTestExt as _}; + + fn glm45_tool_call(function_name: &str, params: &[(&str, &str)]) -> String { + let params = params + .iter() + .map(|(name, value)| { + format!("{name}\n{value}") + }) + .collect::>() + .join("\n"); + format!("{function_name}\n{params}\n") + } + + #[test] + fn glm45_parse_complete_without_tool_call_keeps_text() { + let mut parser = Glm45MoeToolParser::new(&test_tools()); + let output = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn glm45_parse_complete_extracts_single_tool_call() { + let mut parser = Glm45MoeToolParser::new(&test_tools()); + let output = format!( + "Let me search for that.\n{}", + glm45_tool_call( + "get_weather", + &[("city", "Beijing"), ("date", "2024-12-25")] + ) + ); + + let output = parser.parse_complete(&output).unwrap(); + + assert_eq!(output.normal_text(), "Let me search for that.\n"); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({"city": "Beijing", "date": "2024-12-25"}) + ); + } + + #[test] + fn glm45_streaming_extracts_multiple_tool_calls() { + let mut parser = Glm45MoeToolParser::new(&test_tools()); + let output = format!( + "{}\n{}", + glm45_tool_call("get_weather", &[("city", "Shanghai")]), + glm45_tool_call("add", &[("x", "1"), ("y", "2")]) + ); + + let chunks = split_by_chars(&output, 11); + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.normal_text(), ""); + assert_eq!(output.calls().len(), 2); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[1].name.as_deref(), Some("add")); + assert_eq!( + serde_json::from_str::(&output.calls()[1].arguments).unwrap(), + json!({"x": 1, "y": 2}) + ); + } + + #[test] + fn glm45_parse_complete_preserves_raw_closing_tag_text_in_arg_value() { + let mut parser = Glm45MoeToolParser::new(&test_tools()); + let output = parser + .parse_complete(&glm45_tool_call( + "get_weather", + &[ + ("city", "Paris </arg_value></tool_call>"), + ("date", "2026-05-08"), + ], + )) + .unwrap(); + + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "city": "Paris </arg_value></tool_call>", + "date": "2026-05-08", + }) + ); + } + + #[test] + fn glm45_streaming_without_tool_call_emits_text_incrementally() { + let mut parser = Glm45MoeToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &["hello ", "world"]); + + assert_eq!(output.normal_text(), "hello world"); + assert!(output.calls().is_empty()); + } + + #[test] + fn glm45_streaming_preserves_prefix_text() { + let mut parser = Glm45MoeToolParser::new(&test_tools()); + + let output = collect_stream( + &mut parser, + &[ + "Prefix ", + &glm45_tool_call("get_weather", &[("city", "Hangzhou")]), + ], + ); + + assert_eq!(output.normal_text(), "Prefix "); + assert_eq!(output.calls().len(), 1); + } + + #[test] + fn glm45_streaming_handles_start_token_split_across_chunks() { + let mut parser = Glm45MoeToolParser::new(&test_tools()); + let output = collect_stream( + &mut parser, + &[ + "hello get_weather\n", + "cityParis", + ], + ); + + assert_eq!(output.normal_text(), "hello "); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + } + + #[test] + fn glm45_streaming_does_not_emit_incomplete_tool_call() { + let mut parser = Glm45MoeToolParser::new(&test_tools()); + + let output = parser.parse_chunk("get_weather\ncity").unwrap(); + + assert_eq!(output.normal_text(), ""); + assert!(output.calls().is_empty()); + } + + #[test] + fn glm45_finish_fails_incomplete_tool_call() { + let mut parser = Glm45MoeToolParser::new(&test_tools()); + + parser.parse_chunk("get_weather\ncity").unwrap(); + let error = parser.finish().unwrap_err(); + + assert!(error.as_report().to_string().contains("incomplete GLM MoE tool call")); + } + + #[test] + fn glm45_malformed_tool_call_fails_fast() { + let mut parser = Glm45MoeToolParser::new(&test_tools()); + + let error = parser.parse_chunk("get_weathercityParis").unwrap_err(); + + assert!(error.as_report().to_string().contains("tool parser parsing failed")); + } + + #[test] + fn glm45_streaming_ignores_trailing_text_after_tool_calls() { + let mut parser = Glm45MoeToolParser::new(&test_tools()); + + let output = collect_stream( + &mut parser, + &[&format!( + "{}<|endoftext|>", + glm45_tool_call("get_weather", &[("city", "Paris")]) + )], + ); + + assert_eq!(output.normal_text(), ""); + assert_eq!(output.calls().len(), 1); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/hy_v3.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/hy_v3.rs new file mode 100644 index 0000000000000000000000000000000000000000..3b951e9cb2cd2e68732139600f506e868bea51e4 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/hy_v3.rs @@ -0,0 +1,584 @@ +use winnow::ascii::multispace0 as ws0; +use winnow::combinator::{alt, delimited, eof, repeat, seq, terminated}; +use winnow::prelude::*; +use winnow::stream::Partial; +use winnow::token::{literal, rest, take_until}; + +use super::parameters::ToolSchemas; +use super::utils::{MarkerScanState, parse_buffered_event, safe_text_len, take_until_marker}; +use super::{Result, ToolCallDelta, ToolParser, ToolParserOutput}; +use crate::tool::{StructuralTagModel, Tool}; + +const TOOL_CALLS_START: &str = ""; +const TOOL_CALLS_END: &str = ""; +const TOOL_CALL_START: &str = ""; +const TOOL_CALL_END: &str = ""; +const TOOL_SEP: &str = ""; +const ARG_KEY_START: &str = ""; +const ARG_KEY_END: &str = ""; +const ARG_VALUE_START: &str = ""; +const ARG_VALUE_END: &str = ""; + +type HyV3Input<'i> = Partial<&'i str>; + +#[derive(Debug, Clone, PartialEq, Eq)] +enum HyV3Mode { + Text, + ToolBlock { tool_call_end_scan: MarkerScanState }, + Done, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum HyV3Event { + Text { + len: usize, + }, + ToolBlockStart, + ToolCall { + name: String, + raw_params: Vec<(String, String)>, + }, + ToolBlockEnd, + IgnoredRest, +} + +/// Tool parser for HY3 XML-style tool calls. +/// +/// Example tool call content: +/// +/// ```text +/// +/// get_weather +/// cityBeijing +/// +/// +/// ``` +/// +/// Arguments are emitted only after a full `` block is parsed. +/// HY3 marker tokens are added-vocabulary tokens rather than tokenizer special +/// tokens, so the default `preserve_special_tokens() == false` is sufficient. +pub struct HyV3ToolParser { + buffer: String, + mode: HyV3Mode, + emitted_tool_count: usize, + tool_parameters: ToolSchemas, +} + +impl HyV3ToolParser { + /// Create a HY3 tool parser. + fn new(tools: &[Tool]) -> Self { + Self { + buffer: String::new(), + mode: HyV3Mode::Text, + emitted_tool_count: 0, + tool_parameters: ToolSchemas::from_tools(tools), + } + } + + /// Apply one parsed HY3 event to parser state and output. + fn apply_event(&mut self, event: HyV3Event, output: &mut ToolParserOutput) -> Result<()> { + match event { + HyV3Event::Text { len: consumed_len } => { + output.push_text(&self.buffer[..consumed_len]); + } + HyV3Event::ToolBlockStart => { + self.mode = HyV3Mode::ToolBlock { + tool_call_end_scan: MarkerScanState::default(), + }; + } + HyV3Event::ToolCall { name, raw_params } => { + let arguments = self.tool_parameters.convert_params_with_schema(&name, raw_params); + let arguments = serde_json::to_string(&arguments) + .map_err(|error| parsing_failed!("failed to serialize arguments: {}", error))?; + + output.push_call(ToolCallDelta { + tool_index: self.emitted_tool_count, + name: Some(name), + arguments, + }); + self.emitted_tool_count += 1; + } + HyV3Event::ToolBlockEnd => self.mode = HyV3Mode::Done, + HyV3Event::IgnoredRest => {} + } + Ok(()) + } +} + +impl ToolParser for HyV3ToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn structural_tag_model(&self) -> Option { + Some(StructuralTagModel::HyV3) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.buffer.push_str(chunk); + + while let Some((event, consumed_len)) = parse_buffered_event(&self.buffer, |input| { + parse_next_hy_v3_event(input, &mut self.mode) + })? { + self.apply_event(event, output)?; + self.buffer.drain(..consumed_len); + } + + Ok(()) + } + + fn finish(&mut self) -> Result { + let mut output = ToolParserOutput::default(); + match self.mode { + HyV3Mode::Text => output.push_text(&self.buffer), + HyV3Mode::ToolBlock { .. } => return Err(parsing_failed!("incomplete HY3 tool call")), + HyV3Mode::Done => {} + } + let _ = self.reset(); + Ok(output) + } + + fn reset(&mut self) -> String { + self.mode = HyV3Mode::Text; + self.emitted_tool_count = 0; + std::mem::take(&mut self.buffer) + } +} + +/// Parse a HY3 event for the current parser mode. +fn parse_next_hy_v3_event( + input: &mut HyV3Input<'_>, + mode: &mut HyV3Mode, +) -> ModalResult { + match mode { + HyV3Mode::Text => parse_text_event(input), + HyV3Mode::ToolBlock { tool_call_end_scan } => { + parse_tool_block_event(input, tool_call_end_scan) + } + HyV3Mode::Done => ignored_rest_event(input), + } +} + +/// Parse a text-mode HY3 event. +fn parse_text_event(input: &mut HyV3Input<'_>) -> ModalResult { + alt((tool_block_start_event, safe_text_event)).parse_next(input) +} + +/// Parse a HY3 tool-block start marker. +fn tool_block_start_event(input: &mut HyV3Input<'_>) -> ModalResult { + literal(TOOL_CALLS_START).value(HyV3Event::ToolBlockStart).parse_next(input) +} + +/// Parse a safe text run before the next HY3 marker. +fn safe_text_event(input: &mut HyV3Input<'_>) -> ModalResult { + safe_text_len(input, TOOL_CALLS_START).map(|len| HyV3Event::Text { len }) +} + +/// Parse one event inside a HY3 tool block. +fn parse_tool_block_event( + input: &mut HyV3Input<'_>, + tool_call_end_scan: &mut MarkerScanState, +) -> ModalResult { + alt((tool_block_end_event, |input: &mut HyV3Input<'_>| { + tool_call_event(input, tool_call_end_scan) + })) + .parse_next(input) +} + +/// Parse a HY3 tool-block end marker. +fn tool_block_end_event(input: &mut HyV3Input<'_>) -> ModalResult { + (ws0, literal(TOOL_CALLS_END)).value(HyV3Event::ToolBlockEnd).parse_next(input) +} + +/// Parse a complete HY3 tool-call block. +fn tool_call_event( + input: &mut HyV3Input<'_>, + tool_call_end_scan: &mut MarkerScanState, +) -> ModalResult { + let (name, body) = seq!( + _: ws0, + _: literal(TOOL_CALL_START), + take_until(0.., TOOL_SEP), + _: literal(TOOL_SEP), + take_until_marker(TOOL_CALL_END, tool_call_end_scan), + _: literal(TOOL_CALL_END), + ) + .parse_next(input)?; + let raw_params = parse_tool_call_params(body)?; + + Ok(HyV3Event::ToolCall { + name: name.trim().to_string(), + raw_params, + }) +} + +/// Parse all parameter blocks inside a complete HY3 tool call. +fn parse_tool_call_params(tool_call_body: &str) -> ModalResult> { + let mut input = tool_call_body; + delimited(ws0, repeat(0.., terminated(parameter, ws0)), eof).parse_next(&mut input) +} + +/// Parse a HY3 argument key/value block. +fn parameter(input: &mut &str) -> ModalResult<(String, String)> { + let (name, value) = seq!( + _: literal(ARG_KEY_START), + take_until(0.., ARG_KEY_END), + _: literal(ARG_KEY_END), + _: ws0, + _: literal(ARG_VALUE_START), + take_until(0.., ARG_VALUE_END), + _: literal(ARG_VALUE_END), + ) + .parse_next(input)?; + + Ok((name.trim().to_string(), value.to_string())) +} + +/// Parse ignored rest after the HY3 tool block ends. +fn ignored_rest_event(input: &mut HyV3Input<'_>) -> ModalResult { + rest.value(HyV3Event::IgnoredRest).parse_next(input) +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + use serde_json::{Value, json}; + use thiserror_ext::AsReport; + + use super::{HyV3ToolParser, ToolParser}; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{ToolParserOutput, ToolParserTestExt as _}; + + fn build_tool_call(function_name: &str, params: &[(&str, &str)]) -> String { + let params = params + .iter() + .map(|(name, value)| format!("{name}{value}")) + .collect::>() + .join("\n"); + format!("{function_name}{params}") + } + + fn build_tool_calls(tool_calls: &[String]) -> String { + format!("\n{}\n", tool_calls.join("\n")) + } + + fn parsed_arguments(output: &ToolParserOutput, index: usize) -> Value { + serde_json::from_str(&output.calls()[index].arguments).unwrap() + } + + #[test] + fn hy_v3_does_not_preserve_special_tokens() { + let parser = HyV3ToolParser::new(&test_tools()); + + assert!(!parser.preserve_special_tokens()); + } + + #[test] + fn hy_v3_parse_complete_without_tool_call_keeps_text() { + let mut parser = HyV3ToolParser::new(&test_tools()); + let output = parser.parse_complete("This is a plain response.").unwrap(); + + assert_eq!(output.normal_text(), "This is a plain response."); + assert!(output.calls().is_empty()); + } + + #[test] + fn hy_v3_parse_complete_extracts_zero_arg_inline_tool_call() { + let mut parser = HyV3ToolParser::new(&test_tools()); + let output = parser + .parse_complete( + "get_current_date", + ) + .unwrap(); + + assert_eq!(output.normal_text(), ""); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_current_date")); + assert_eq!(parsed_arguments(&output, 0), json!({})); + } + + #[test] + fn hy_v3_parse_complete_extracts_zero_arg_newline_tool_call() { + let mut parser = HyV3ToolParser::new(&test_tools()); + let output = parser + .parse_complete( + "\nget_current_date\n\n", + ) + .unwrap(); + + assert_eq!(output.calls()[0].name.as_deref(), Some("get_current_date")); + assert_eq!(parsed_arguments(&output, 0), json!({})); + } + + #[test] + fn hy_v3_parse_complete_extracts_arguments_on_same_line() { + let mut parser = HyV3ToolParser::new(&test_tools()); + let output = parser + .parse_complete( + "get_weathercityBeijingdate2026-03-30", + ) + .unwrap(); + + assert_eq!( + parsed_arguments(&output, 0), + json!({ "city": "Beijing", "date": "2026-03-30" }) + ); + } + + #[test] + fn hy_v3_parse_complete_extracts_arguments_with_newlines() { + let mut parser = HyV3ToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_calls(&[build_tool_call( + "get_weather", + &[("city", "Beijing"), ("date", "2026-03-30")], + )])) + .unwrap(); + + assert_eq!( + parsed_arguments(&output, 0), + json!({ "city": "Beijing", "date": "2026-03-30" }) + ); + } + + #[test] + fn hy_v3_parse_complete_preserves_prefix_and_ignores_trailing_text() { + let mut parser = HyV3ToolParser::new(&test_tools()); + let output = parser + .parse_complete(&format!( + "Checking.{} trailing text", + build_tool_calls(&[build_tool_call("get_current_date", &[])]) + )) + .unwrap(); + + assert_eq!(output.normal_text(), "Checking."); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_current_date")); + } + + #[test] + fn hy_v3_parse_complete_extracts_multiple_tool_calls_in_one_block() { + let mut parser = HyV3ToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_calls(&[ + build_tool_call( + "get_weather", + &[("city", "Beijing"), ("date", "2026-03-30")], + ), + build_tool_call( + "get_weather", + &[("city", "Hangzhou"), ("date", "2026-03-30")], + ), + ])) + .unwrap(); + + expect![[r#" + ToolParserOutput { + events: [ + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "get_weather", + ), + arguments: "{\"city\":\"Beijing\",\"date\":\"2026-03-30\"}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 1, + name: Some( + "get_weather", + ), + arguments: "{\"city\":\"Hangzhou\",\"date\":\"2026-03-30\"}", + }, + ), + ], + } + "#]] + .assert_debug_eq(&output); + } + + #[test] + fn hy_v3_parse_complete_converts_schema_types() { + let mut parser = HyV3ToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_calls(&[build_tool_call( + "convert", + &[ + ("whole", "5.3"), + ("flag", "true"), + ("payload", r#"{"k":1}"#), + ("items", "[1,2]"), + ], + )])) + .unwrap(); + + assert_eq!( + parsed_arguments(&output, 0), + json!({ + "whole": 5.3, + "flag": true, + "payload": { "k": 1 }, + "items": [1, 2] + }) + ); + } + + #[test] + fn hy_v3_streaming_without_tool_call_emits_text_incrementally() { + let mut parser = HyV3ToolParser::new(&test_tools()); + let mut output = ToolParserOutput::default(); + + output.append(parser.parse_chunk("This is ").unwrap()); + output.append(parser.parse_chunk("a plain ").unwrap()); + output.append(parser.parse_chunk("response.").unwrap()); + output.append(parser.finish().unwrap()); + + assert_eq!(output.normal_text(), "This is a plain response."); + assert!(output.calls().is_empty()); + } + + #[test] + fn hy_v3_streaming_extracts_zero_arg_tool_call() { + let mut parser = HyV3ToolParser::new(&test_tools()); + let chunks = [ + "", + "\n", + "get_current_date", + "", + "\n", + "\n", + ]; + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_current_date")); + assert_eq!(parsed_arguments(&output, 0), json!({})); + } + + #[test] + fn hy_v3_streaming_extracts_arguments() { + let mut parser = HyV3ToolParser::new(&test_tools()); + let chunks = [ + "", + "\n", + "get_weather", + "", + "\ncity", + "\nBeijing", + "\ndate", + "\n2026-03-30", + "\n", + "\n", + ]; + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + parsed_arguments(&output, 0), + json!({ "city": "Beijing", "date": "2026-03-30" }) + ); + } + + #[test] + fn hy_v3_streaming_preserves_prefix_text() { + let mut parser = HyV3ToolParser::new(&test_tools()); + let chunks = [ + "Checking.", + "", + "\n", + "get_current_date", + "", + "\n", + "\n", + ]; + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.normal_text(), "Checking."); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_current_date")); + } + + #[test] + fn hy_v3_streaming_extracts_multiple_tool_calls_in_one_block() { + let input = build_tool_calls(&[ + build_tool_call( + "get_weather", + &[("city", "Beijing"), ("date", "2026-03-30")], + ), + build_tool_call( + "get_weather", + &[("city", "Hangzhou"), ("date", "2026-03-30")], + ), + ]); + let chunks = split_by_chars(&input, 9); + let mut parser = HyV3ToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.calls().len(), 2); + assert_eq!(parsed_arguments(&output, 0)["city"], json!("Beijing")); + assert_eq!(parsed_arguments(&output, 1)["city"], json!("Hangzhou")); + } + + #[test] + fn hy_v3_streaming_handles_start_marker_split_across_chunks() { + let input = format!( + "hello {}", + build_tool_calls(&[build_tool_call("get_weather", &[("city", "Beijing")])]) + ); + let chunks = split_by_chars(&input, 5); + let mut parser = HyV3ToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.normal_text(), "hello "); + assert_eq!(output.calls().len(), 1); + assert_eq!(parsed_arguments(&output, 0), json!({ "city": "Beijing" })); + } + + #[test] + fn hy_v3_streaming_does_not_emit_incomplete_tool_call() { + let mut parser = HyV3ToolParser::new(&test_tools()); + let mut output = ToolParserOutput::default(); + + parser + .parse_into( + "get_weathercityBei", + &mut output, + ) + .unwrap(); + + assert_eq!(output.normal_text(), ""); + assert!(output.calls().is_empty()); + } + + #[test] + fn hy_v3_finish_fails_incomplete_tool_call() { + let mut parser = HyV3ToolParser::new(&test_tools()); + parser.parse_chunk("get_weather").unwrap(); + + let error = parser.finish().unwrap_err(); + + expect!["tool parser parsing failed: incomplete HY3 tool call"] + .assert_eq(&error.to_report_string()); + } + + #[test] + fn hy_v3_malformed_tool_call_fails_fast() { + let mut parser = HyV3ToolParser::new(&test_tools()); + let error = parser + .parse_complete( + "get_weathercityBeijing", + ) + .unwrap_err(); + + assert!(error.to_report_string().starts_with("tool parser parsing failed:")); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/granite4.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/granite4.rs new file mode 100644 index 0000000000000000000000000000000000000000..e3e07ec58741160ae24955555b2e281e6c7e4f68 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/granite4.rs @@ -0,0 +1,554 @@ +use winnow::ascii::multispace0 as ws0; +use winnow::combinator::{alt, peek, seq}; +use winnow::error::{ContextError, ErrMode, ModalResult, StrContext}; +use winnow::prelude::*; +use winnow::token::{any, literal}; + +use super::{ + JsonToolCallConfig, JsonToolCallEvent, JsonToolCallWhitespace, JsonToolInput, + tool_call_header_event, +}; +use crate::tool::utils::{ + JsonObjectScanState, JsonStringScanState, decode_json_str, parse_buffered_event, safe_text_len, + take_json_object, take_json_string, +}; +use crate::tool::{Result, Tool, ToolCallDelta, ToolParser, ToolParserOutput}; + +const TOOL_CALL_START: &str = ""; +const TOOL_CALL_END: &str = ""; + +#[derive(Debug, Clone, PartialEq, Eq)] +enum Granite4Mode { + Text, + Header, + /// Parsing the arguments value: + /// `None` until the first byte decides object vs string; + /// `Some` while streaming the selected value shape. + Args { + args_scan: Option, + }, + /// Arguments done; consume the object's closing `}` and ``. + Close, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum Granite4ArgsScan { + Object(JsonObjectScanState), + String(JsonStringScanState), +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum Granite4Event { + Text { + len: usize, + }, + ToolCallStart, + ToolCallHeader { + function_name: String, + }, + /// Verbatim bytes of an object-valued arguments payload; `complete` once the + /// object scan reaches its closing brace. + ObjectArgsDelta { + len: usize, + complete: bool, + }, + /// Decoded contents of a string-valued arguments payload. + StringArgs { + decoded: String, + }, + ToolCallEnd, +} + +/// Tool parser for Granite 4 `…` JSON tool calls. +/// +/// Example tool call content: +/// +/// ```text +/// {"name": "get_weather", "arguments": {"city": "Boston"}} +/// ``` +/// +/// Parallel calls are repeated `…` blocks with ordinary +/// content interleaved between them. This reuses the shared JSON helpers for +/// everything except one Granite 4 specific step (`args_event`): the `arguments` +/// value may be a JSON object (kept verbatim) **or** a JSON string whose decoded +/// contents are the arguments (the `# test granite behavior` case in Python). +pub struct Granite4ToolParser { + buffer: String, + mode: Granite4Mode, + active_tool_index: Option, + emitted_tool_count: usize, +} + +impl Granite4ToolParser { + /// Create a Granite 4 tool parser. + fn new(_tools: &[Tool]) -> Self { + Self { + buffer: String::new(), + mode: Granite4Mode::Text, + active_tool_index: None, + emitted_tool_count: 0, + } + } + + /// Apply one parsed Granite 4 event to parser state and output. + fn apply_event(&mut self, event: Granite4Event, output: &mut ToolParserOutput) -> Result<()> { + match event { + Granite4Event::Text { len } => output.push_text(&self.buffer[..len]), + Granite4Event::ToolCallStart => self.mode = Granite4Mode::Header, + Granite4Event::ToolCallHeader { function_name } => { + let tool_index = self.emitted_tool_count; + self.emitted_tool_count += 1; + self.active_tool_index = Some(tool_index); + self.mode = Granite4Mode::Args { args_scan: None }; + output.push_call(ToolCallDelta { + tool_index, + name: Some(function_name), + arguments: String::new(), + }); + } + Granite4Event::ObjectArgsDelta { len, complete } => { + let arguments = self.buffer[..len].to_string(); + self.push_arguments(arguments, output)?; + if complete { + self.mode = Granite4Mode::Close; + } + } + Granite4Event::StringArgs { decoded } => { + self.push_arguments(decoded, output)?; + self.mode = Granite4Mode::Close; + } + Granite4Event::ToolCallEnd => { + self.active_tool_index = None; + self.mode = Granite4Mode::Text; + } + } + Ok(()) + } + + /// Append one arguments delta to the active tool call. + fn push_arguments(&self, arguments: String, output: &mut ToolParserOutput) -> Result<()> { + let Some(tool_index) = self.active_tool_index else { + return Err(parsing_failed!( + "Granite4 arguments without an active tool call" + )); + }; + output.push_call(ToolCallDelta { + tool_index, + name: None, + arguments, + }); + Ok(()) + } + + fn reset(&mut self) -> String { + self.mode = Granite4Mode::Text; + self.active_tool_index = None; + self.emitted_tool_count = 0; + std::mem::take(&mut self.buffer) + } +} + +impl ToolParser for Granite4ToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.buffer.push_str(chunk); + + while let Some((event, consumed_len)) = parse_buffered_event(&self.buffer, |input| { + parse_next_granite4_event(input, &mut self.mode) + })? { + self.apply_event(event, output)?; + self.buffer.drain(..consumed_len); + } + + Ok(()) + } + + fn finish(&mut self) -> Result { + let mut output = ToolParserOutput::default(); + match &self.mode { + Granite4Mode::Text => output.push_text(&self.buffer), + Granite4Mode::Header | Granite4Mode::Args { .. } | Granite4Mode::Close => { + return Err(parsing_failed!("incomplete Granite4 tool call")); + } + } + let _ = self.reset(); + Ok(output) + } + + fn reset(&mut self) -> String { + Granite4ToolParser::reset(self) + } +} + +/// Parse a Granite 4 event for the current parser mode. +fn parse_next_granite4_event( + input: &mut JsonToolInput<'_>, + mode: &mut Granite4Mode, +) -> ModalResult { + match mode { + Granite4Mode::Text => text_event(input), + Granite4Mode::Header => header_event(input), + Granite4Mode::Args { args_scan } => args_event(input, args_scan), + Granite4Mode::Close => close_event(input), + } +} + +/// Parse content text or the start of a `` block. *(reuses `safe_text_len`)* +fn text_event(input: &mut JsonToolInput<'_>) -> ModalResult { + alt(( + |input: &mut JsonToolInput<'_>| { + seq!(_: literal(TOOL_CALL_START), _: ws0) + .value(Granite4Event::ToolCallStart) + .parse_next(input) + }, + |input: &mut JsonToolInput<'_>| { + safe_text_len(input, TOOL_CALL_START).map(|len| Granite4Event::Text { len }) + }, + )) + .parse_next(input) +} + +/// Parse the `{"name":"X","arguments":` header before the value. *(reuses `tool_call_header_event`)* +fn header_event(input: &mut JsonToolInput<'_>) -> ModalResult { + const CONFIG: JsonToolCallConfig = JsonToolCallConfig { + parser_name: "Granite4", + start_marker: "", + end_marker: "", + marker_whitespace: JsonToolCallWhitespace::Optional, + delimiter: None, + name_key: "name", + arguments_key: &["arguments"], + }; + + match tool_call_header_event(input, CONFIG)? { + JsonToolCallEvent::ToolCallHeader { function_name } => { + Ok(Granite4Event::ToolCallHeader { function_name }) + } + _ => unreachable!("tool_call_header_event only emits ToolCallHeader"), + } +} + +/// Parse one arguments-value event. +/// +/// GRANITE 4 SPECIFIC - the sole behavior that differs from the shared +/// `` JSON parsers. The value is either a JSON object (kept verbatim, +/// streamed incrementally via `take_json_object`) or an escaped JSON string +/// (decoded whole via `json_str`). The string form is why we cannot just forward +/// raw arg bytes like the sibling parsers do: an escaped string only resolves +/// once seen whole and unescaped. +fn args_event( + input: &mut JsonToolInput<'_>, + args_scan: &mut Option, +) -> ModalResult { + if let Some(scan) = args_scan { + return match scan { + Granite4ArgsScan::Object(scan) => { + let len = take_json_object(input, scan)?; + Ok(Granite4Event::ObjectArgsDelta { + len, + complete: scan.complete(), + }) + } + Granite4ArgsScan::String(scan) => string_args_event(input, scan), + }; + } + + match peek(any).parse_next(input)? { + '{' => { + let mut scan = JsonObjectScanState::default(); + let len = take_json_object(input, &mut scan)?; + let complete = scan.complete(); + *args_scan = Some(Granite4ArgsScan::Object(scan)); + Ok(Granite4Event::ObjectArgsDelta { len, complete }) + } + '"' => { + *args_scan = Some(Granite4ArgsScan::String(JsonStringScanState::default())); + let Some(Granite4ArgsScan::String(scan)) = args_scan else { + unreachable!("Granite4 string scan state was just initialized") + }; + string_args_event(input, scan) + } + _ => { + let mut error = ContextError::new(); + error.push(StrContext::Label("Granite4 arguments")); + Err(ErrMode::Cut(error)) + } + } +} + +fn string_args_event( + input: &mut JsonToolInput<'_>, + scan: &mut JsonStringScanState, +) -> ModalResult { + let text = **input; + let len = take_json_string(input, scan)?; + Ok(Granite4Event::StringArgs { + decoded: decode_json_str(&text[..len])?, + }) +} + +/// Parse the tool-call object's closing `}` and the `` end marker. +fn close_event(input: &mut JsonToolInput<'_>) -> ModalResult { + seq!(_: ws0, _: literal("}"), _: ws0, _: literal(TOOL_CALL_END)) + .value(Granite4Event::ToolCallEnd) + .parse_next(input) +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + use thiserror_ext::AsReport; + + use super::Granite4ToolParser; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{ToolParser, ToolParserOutput, ToolParserTestExt as _}; + + #[test] + fn granite4_parse_complete_without_tool_call_keeps_text() { + let mut parser = Granite4ToolParser::new(&test_tools()); + let output = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn granite4_parse_complete_object_args() { + let mut parser = Granite4ToolParser::new(&test_tools()); + let output = parser + .parse_complete( + r#"{"name":"get_weather","arguments":{"city":"Boston"}}"#, + ) + .unwrap(); + + assert_eq!(output.normal_text(), ""); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[0].arguments, r#"{"city":"Boston"}"#); + } + + #[test] + fn granite4_parse_complete_string_args() { + // GRANITE4-SPECIFIC: `arguments` may be a pre-serialized JSON string; its + // decoded contents become the arguments. + let mut parser = Granite4ToolParser::new(&test_tools()); + let output = parser + .parse_complete( + r#"{"name":"get_weather","arguments":"{\"city\":\"Boston\"}"}"#, + ) + .unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[0].arguments, r#"{"city":"Boston"}"#); + } + + #[test] + fn granite4_extracts_interleaved_content_and_mixed_args() { + let mut parser = Granite4ToolParser::new(&test_tools()); + let output = parser + .parse_complete( + r#"before {"name":"find_bbox","arguments":"{\"x\":1}"} middle {"name":"get_weather","arguments":{"city":"Boston"}} after"#, + ) + .unwrap(); + + expect![[r#" + ToolParserOutput { + events: [ + Text( + "before middle after", + ), + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "find_bbox", + ), + arguments: "{\"x\":1}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 1, + name: Some( + "get_weather", + ), + arguments: "{\"city\":\"Boston\"}", + }, + ), + ], + } + "#]] + .assert_debug_eq(&output); + } + + #[test] + fn granite4_streaming_handles_split_markers() { + let input = r#"hello {"name":"get_weather","arguments":{"city":"Tokyo"}} bye"#; + let chunks = split_by_chars(input, 5); + let mut parser = Granite4ToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.normal_text(), "hello bye"); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[0].arguments, r#"{"city":"Tokyo"}"#); + } + + #[test] + fn granite4_streaming_emits_object_argument_deltas() { + let mut parser = Granite4ToolParser::new(&test_tools()); + let chunks = [ + r#"{"name":"get_weather","arguments":"#, + r#"{"city":"#, + r#""Beijing""#, + r#"}"#, + r#"}"#, + ]; + + let mut output = ToolParserOutput::default(); + let mut observed_arguments = Vec::new(); + for chunk in chunks { + let next = parser.parse_chunk(chunk).unwrap(); + observed_arguments.extend( + next.calls() + .iter() + .filter(|call| call.name.is_none()) + .map(|call| call.arguments.clone()), + ); + output.append(next); + } + output.append(parser.finish().unwrap()); + + assert_eq!(observed_arguments, [r#"{"city":"#, r#""Beijing""#, r#"}"#]); + assert_eq!( + output.coalesce().calls()[0].arguments, + r#"{"city":"Beijing"}"# + ); + } + + #[test] + fn granite4_string_args_split_across_chunks() { + let input = r#"{"name":"f","arguments":"{\"a\":1}"}"#; + let chunks = split_by_chars(input, 3); + let mut parser = Granite4ToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("f")); + assert_eq!(output.calls()[0].arguments, r#"{"a":1}"#); + } + + #[test] + fn granite4_long_string_args_stream_without_reparse() { + let arguments = format!(r#"{{"data":"{}"}}"#, "x".repeat(64 * 1024)); + let encoded_arguments = serde_json::to_string(&arguments).unwrap(); + let input = + format!(r#"{{"name":"f","arguments":{encoded_arguments}}}"#); + let chunks = split_by_chars(&input, 7); + let mut parser = Granite4ToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("f")); + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn granite4_streaming_handles_marker_and_json_whitespace() { + // Granite spaces the markers (` {…} `) and the JSON + // (`"name": …`). Since `args_event` has no leading `ws0`, this guards that + // the header consumes the whitespace before the arguments value. + let input = concat!( + "Here goes the bbox call: \n", + r#" {"name": "find_bbox", "arguments": "{\"coordinates\": [[23.54, 43.1], [-12.2, 54.3], [4, 5]], \"coordinate_type\": \"latlong\"}"} "#, + " Now the stock price call: \n ", + r#" {"name": "get_stock_price", "arguments": {"symbol": "AAPL", "start_date": "2021-01-01", "end_date": "2021-12-31"}} "#, + " Now another bbox call: \n ", + r#" {"name": "find_bbox", "arguments": "{\"coordinates\": [[23.54, 43.1], [-12.2, 54.3], [4, 5]], \"coordinate_type\": \"latlong\"}"} "#, + " See? I'm a helpful assistant.", + ); + let chunks = split_by_chars(input, 3); + let mut parser = Granite4ToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + expect![[r#" + ToolParserOutput { + events: [ + Text( + "Here goes the bbox call: \n Now the stock price call: \n Now another bbox call: \n See? I'm a helpful assistant.", + ), + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "find_bbox", + ), + arguments: "{\"coordinates\": [[23.54, 43.1], [-12.2, 54.3], [4, 5]], \"coordinate_type\": \"latlong\"}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 1, + name: Some( + "get_stock_price", + ), + arguments: "{\"symbol\": \"AAPL\", \"start_date\": \"2021-01-01\", \"end_date\": \"2021-12-31\"}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 2, + name: Some( + "find_bbox", + ), + arguments: "{\"coordinates\": [[23.54, 43.1], [-12.2, 54.3], [4, 5]], \"coordinate_type\": \"latlong\"}", + }, + ), + ], + } + "#]].assert_debug_eq(&output); + } + + #[test] + fn granite4_finish_fails_incomplete_tool_call() { + let mut parser = Granite4ToolParser::new(&test_tools()); + parser + .parse_chunk(r#"{"name":"get_weather","arguments":{"city""#) + .unwrap(); + + let error = parser.finish().unwrap_err(); + + expect!["tool parser parsing failed: incomplete Granite4 tool call"] + .assert_eq(&error.to_report_string()); + } + + #[test] + fn granite4_rejects_non_object_non_string_args() { + let mut parser = Granite4ToolParser::new(&test_tools()); + let error = parser + .parse_chunk(r#"{"name":"f","arguments":42}"#) + .unwrap_err(); + + expect![[ + r#"tool parser parsing failed: near "42}": invalid Granite4 arguments"# + ]] + .assert_eq(&error.to_report_string()); + } + + #[test] + fn granite4_preserve_special_tokens_is_false() { + let parser = Granite4ToolParser::new(&test_tools()); + assert!(!parser.preserve_special_tokens()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/hermes.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/hermes.rs new file mode 100644 index 0000000000000000000000000000000000000000..346ea21948b0e4dc47842f2f29c54fe93b50a0ba --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/hermes.rs @@ -0,0 +1,229 @@ +use super::{JsonToolCallConfig, JsonToolCallParser, JsonToolCallWhitespace}; +use crate::tool::{Result, StructuralTagModel, Tool, ToolParser, ToolParserOutput}; + +const HERMES_CONFIG: JsonToolCallConfig = JsonToolCallConfig { + parser_name: "Hermes", + start_marker: "", + end_marker: "", + marker_whitespace: JsonToolCallWhitespace::Optional, + delimiter: None, + name_key: "name", + arguments_key: &["arguments"], +}; + +/// Tool parser for Hermes XML-wrapped JSON tool calls. +/// +/// Example tool call content: +/// +/// ```text +/// {"name": "get_weather", "arguments": {"location":"Tokyo"}} +/// ``` +/// +/// Arguments are already OpenAI-style JSON text, so they are streamed as raw +/// argument deltas without schema conversion or JSON normalization. +/// +/// Note: parallel calls are represented as repeated +/// `...` blocks, not as multiple calls inside one tag. +pub struct HermesToolParser { + inner: JsonToolCallParser, +} + +impl HermesToolParser { + /// Create a Hermes tool parser. + fn new(_tools: &[Tool]) -> Self { + Self { + inner: JsonToolCallParser::new(HERMES_CONFIG), + } + } +} + +impl ToolParser for HermesToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn structural_tag_model(&self) -> Option { + Some(StructuralTagModel::Hermes) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.inner.parse_into(chunk, output) + } + + fn finish(&mut self) -> Result { + self.inner.finish() + } + + fn reset(&mut self) -> String { + self.inner.reset() + } +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + use thiserror_ext::AsReport; + + use super::HermesToolParser; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{ToolParser, ToolParserOutput, ToolParserTestExt as _}; + + fn build_tool_call(function_name: &str, arguments: &str) -> String { + format!(r#"{{"name":"{function_name}","arguments":{arguments}}}"#) + } + + #[test] + fn hermes_parse_complete_without_tool_call_keeps_text() { + let mut parser = HermesToolParser::new(&test_tools()); + let output = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn hermes_parse_complete_extracts_raw_json_arguments() { + let mut parser = HermesToolParser::new(&test_tools()); + let arguments = r#"{ "location": "Tokyo", "days": "3" }"#; + let output = parser + .parse_complete(&format!( + "Let me check.\n{}", + build_tool_call("get_weather", arguments) + )) + .unwrap(); + + assert_eq!(output.normal_text(), "Let me check.\n"); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].tool_index, 0); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn hermes_accepts_newline_after_tool_call_start() { + let mut parser = HermesToolParser::new(&test_tools()); + let output = parser + .parse_complete( + r#" +{"name":"get_weather","arguments":{}}"#, + ) + .unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + } + + #[test] + fn hermes_does_not_validate_or_normalize_arguments() { + let mut parser = HermesToolParser::new(&test_tools()); + let arguments = r#"{"location":"Tokyo",}"#; + let output = parser.parse_complete(&build_tool_call("get_weather", arguments)).unwrap(); + + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn hermes_streaming_emits_argument_deltas() { + let mut parser = HermesToolParser::new(&test_tools()); + let chunks = [ + "preface {\"name\":\"get_weather\",\"arguments\":", + "{\"location\":", + "\"Beijing\"", + "}", + "} suffix", + ]; + + let mut output = ToolParserOutput::default(); + let mut observed_arguments = Vec::new(); + for chunk in chunks { + let next = parser.parse_chunk(chunk).unwrap(); + observed_arguments.extend( + next.calls() + .iter() + .filter(|call| call.name.is_none()) + .map(|call| call.arguments.clone()), + ); + output.append(next); + } + output.append(parser.finish().unwrap()); + + assert_eq!(observed_arguments, ["{\"location\":", "\"Beijing\"", "}"]); + assert_eq!(output.normal_text(), "preface suffix"); + assert_eq!( + output.coalesce().calls()[0].arguments, + r#"{"location":"Beijing"}"# + ); + } + + #[test] + fn hermes_streaming_handles_split_markers() { + let input = format!( + "hello {}", + build_tool_call("get_weather", r#"{"location":"Tokyo"}"#) + ); + let chunks = split_by_chars(&input, 5); + let mut parser = HermesToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.normal_text(), "hello "); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].arguments, r#"{"location":"Tokyo"}"#); + } + + #[test] + fn hermes_streaming_extracts_multiple_tool_calls() { + let input = format!( + "{}{}", + build_tool_call("get_weather", r#"{"location":"Shanghai"}"#), + build_tool_call("add", r#"{"x":1,"y":2}"#), + ); + let chunks = split_by_chars(&input, 7); + let mut parser = HermesToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + expect![[r#" + ToolParserOutput { + events: [ + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "get_weather", + ), + arguments: "{\"location\":\"Shanghai\"}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 1, + name: Some( + "add", + ), + arguments: "{\"x\":1,\"y\":2}", + }, + ), + ], + } + "#]] + .assert_debug_eq(&output); + } + + #[test] + fn hermes_finish_fails_incomplete_tool_call() { + let mut parser = HermesToolParser::new(&test_tools()); + parser + .parse_chunk(r#"{"name":"get_weather","arguments":{"location""#) + .unwrap(); + + let error = parser.finish().unwrap_err(); + + expect!["tool parser parsing failed: incomplete Hermes tool call"] + .assert_eq(&error.to_report_string()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/internlm2.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/internlm2.rs new file mode 100644 index 0000000000000000000000000000000000000000..90845232e500d554dd58f37095e75ed986a4ff3f --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/internlm2.rs @@ -0,0 +1,348 @@ +use super::{JsonToolCallConfig, JsonToolCallParser, JsonToolCallWhitespace}; +use crate::tool::{Result, Tool, ToolParser, ToolParserOutput}; + +const INTERNLM2_CONFIG: JsonToolCallConfig = JsonToolCallConfig { + parser_name: "InternLM2", + start_marker: "<|action_start|><|plugin|>", + end_marker: "<|action_end|>", + marker_whitespace: JsonToolCallWhitespace::Optional, + delimiter: None, + name_key: "name", + // The Python parser's `get_arguments()` accepts either `parameters` or + // `arguments` and prefers `parameters` when both are present. This Rust + // parser uses first-encountered semantics because the header parser only + // permits one args key per tool-call object; if a future model emits + // both keys in the same object, the Rust port will accept the first one + // and reject the trailing one as a syntax error rather than silently + // shadowing it. + arguments_key: &["parameters", "arguments"], +}; + +/// Tool parser for InternLM2 special-token wrapped JSON tool calls. +/// +/// Example tool call content: +/// +/// ```text +/// <|action_start|><|plugin|>{"name": "get_weather", "parameters": {"location":"Tokyo"}}<|action_end|> +/// ``` +/// +/// Arguments are already OpenAI-style JSON text, so they are streamed as raw +/// argument deltas without schema conversion or JSON normalization. +/// +/// # Divergences from the Python reference +/// +/// This Rust port intentionally diverges from +/// `vllm/tool_parsers/internlm2_tool_parser.py` in two user-visible ways: +/// +/// - **Parallel tool calls are supported.** Python silently drops every `<|action_start|>` block +/// after the first (`current_tool_id > 0` returns an empty delta); this parser emits every +/// well-formed block with incrementing `tool_index`. Models that legitimately emit multiple +/// action blocks therefore produce more tool calls under Rust than under Python. +/// - **End-marker bytes inside JSON string values are preserved.** Python does +/// `action.split("<|action_end|>")[0]` which truncates regardless of JSON context; this parser +/// scans matched braces and quotes so a literal `<|action_end|>` inside an arguments string is +/// forwarded intact. +/// - **Only whitespace is allowed before the `{`.** Python's non-streaming +/// `action[action.find("{"):]` drops any bytes before the first `{`, but its streaming path has +/// no equivalent and the model format always emits `<|plugin|>{...`; this parser allows only +/// whitespace there, matching the other JSON parsers in this crate. +/// - **Truncated tool calls error rather than silently dropping.** Python's streaming wrapper +/// swallows mid-stream errors with `except Exception: return None` (logging a traceback) while +/// its non-streaming path raises `JSONDecodeError`; this parser returns an `incomplete InternLM2 +/// tool call` error from `finish()`, matching the other JSON parsers and Python's non-streaming +/// behavior. +/// +/// # Known unaddressed divergences (TODO) +/// +/// The following Python behaviors are NOT yet matched. They are deferred to +/// follow-up work because they require non-local changes to the shared +/// `JsonToolCallParser` core that would affect Hermes / Llama / Mistral / +/// Qwen as well. If a real-world InternLM2 deployment hits one of these, +/// prioritize the corresponding fix. +/// +/// - **Arguments value type.** The shared core requires the arguments value to be a JSON object +/// (`take_json_object` rejects anything not starting with `{`). Python's +/// `json.dumps(action_dict.get("parameters", ...))` accepts `null`, arrays, strings, and numbers +/// and round-trips them verbatim. Models that legitimately emit `"parameters":null` will hard- +/// fail under Rust. +/// - **Unknown arguments key.** Python falls back to `{}` via `action_dict.get("parameters", +/// action_dict.get("arguments", {}))` when neither key is present; the Rust header parser raises +/// `parsing failed: invalid InternLM2` for any unrecognized key. A model that emits a typo (e.g. +/// `"params"`) breaks the whole response. +/// - **Field order independence.** The header parser requires the JSON keys to appear in the order +/// `name` then arguments key. Python's `json.loads` + `dict.get` is order-independent, so a model +/// emitting `{"parameters":{...},"name":"foo"}` parses in Python but fails in Rust. +pub struct Internlm2ToolParser { + inner: JsonToolCallParser, +} + +impl Internlm2ToolParser { + /// Create an InternLM2 tool parser. + fn new(_tools: &[Tool]) -> Self { + Self { + inner: JsonToolCallParser::new(INTERNLM2_CONFIG), + } + } +} + +impl ToolParser for Internlm2ToolParser { + /// Create a boxed InternLM2 tool parser. + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + /// Preserve special-token markers while decoding, since + /// `<|action_start|>`, `<|plugin|>`, and `<|action_end|>` are tokenizer + /// special tokens in InternLM2 models. + fn preserve_special_tokens(&self) -> bool { + true + } + + /// Feed one decoded text chunk through the InternLM2 parser. + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.inner.parse_into(chunk, output) + } + + /// Flush any buffered partial state at end of stream. + fn finish(&mut self) -> Result { + self.inner.finish() + } + + /// Clear parser state and return currently uncommitted buffered text. + fn reset(&mut self) -> String { + self.inner.reset() + } +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + use thiserror_ext::AsReport; + + use super::Internlm2ToolParser; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{ToolParser, ToolParserOutput, ToolParserTestExt as _}; + + const ACTION_START: &str = "<|action_start|><|plugin|>"; + const ACTION_END: &str = "<|action_end|>"; + + fn build_tool_call(function_name: &str, args_key: &str, arguments: &str) -> String { + format!( + r#"{ACTION_START}{{"name":"{function_name}","{args_key}":{arguments}}}{ACTION_END}"# + ) + } + + #[test] + fn internlm2_parse_complete_without_tool_call_keeps_text() { + let mut parser = Internlm2ToolParser::new(&test_tools()); + let result = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(result.normal_text(), "Hello, world!"); + assert!(result.calls().is_empty()); + } + + #[test] + fn internlm2_parse_complete_extracts_parameters_key() { + let mut parser = Internlm2ToolParser::new(&test_tools()); + let arguments = r#"{"location":"Tokyo","days":"3"}"#; + let result = parser + .parse_complete(&format!( + "Let me check.\n{}", + build_tool_call("get_weather", "parameters", arguments) + )) + .unwrap(); + + assert_eq!(result.normal_text(), "Let me check.\n"); + assert_eq!(result.calls().len(), 1); + assert_eq!(result.calls()[0].tool_index, 0); + assert_eq!(result.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(result.calls()[0].arguments, arguments); + } + + #[test] + fn internlm2_parse_complete_extracts_arguments_key_fallback() { + let mut parser = Internlm2ToolParser::new(&test_tools()); + let arguments = r#"{"location":"Tokyo"}"#; + let result = parser + .parse_complete(&build_tool_call("get_weather", "arguments", arguments)) + .unwrap(); + + assert_eq!(result.calls().len(), 1); + assert_eq!(result.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(result.calls()[0].arguments, arguments); + } + + #[test] + fn internlm2_accepts_whitespace_after_plugin_marker() { + let mut parser = Internlm2ToolParser::new(&test_tools()); + let result = parser + .parse_complete(&format!( + r#"{ACTION_START} +{{"name":"get_weather","parameters":{{}}}}{ACTION_END}"# + )) + .unwrap(); + + assert_eq!(result.calls().len(), 1); + assert_eq!(result.calls()[0].name.as_deref(), Some("get_weather")); + } + + #[test] + fn internlm2_does_not_validate_or_normalize_arguments() { + let mut parser = Internlm2ToolParser::new(&test_tools()); + let arguments = r#"{"location":"Tokyo",}"#; + let result = parser + .parse_complete(&build_tool_call("get_weather", "parameters", arguments)) + .unwrap(); + + assert_eq!(result.calls()[0].arguments, arguments); + } + + #[test] + fn internlm2_streaming_emits_argument_deltas() { + let mut parser = Internlm2ToolParser::new(&test_tools()); + let chunks = [ + "preface <|action", + "_start|><|plugin|>", + r#"{"name":"get_weather","parameters":"#, + r#"{"location":"#, + r#""Beijing""#, + r#"}"#, + r#"}<|action_end|> suffix"#, + ]; + + let mut result = ToolParserOutput::default(); + let mut observed_arguments = Vec::new(); + for chunk in chunks { + let next = parser.parse_chunk(chunk).unwrap(); + observed_arguments.extend( + next.calls() + .iter() + .filter(|call| call.name.is_none()) + .map(|call| call.arguments.clone()), + ); + result.append(next); + } + result.append(parser.finish().unwrap()); + + assert_eq!( + observed_arguments, + [r#"{"location":"#, r#""Beijing""#, r#"}"#] + ); + assert_eq!(result.normal_text(), "preface suffix"); + assert_eq!( + result.coalesce().calls()[0].arguments, + r#"{"location":"Beijing"}"# + ); + } + + #[test] + fn internlm2_streaming_handles_split_markers() { + let input = format!( + "hello {}", + build_tool_call("get_weather", "parameters", r#"{"location":"Tokyo"}"#) + ); + let chunks = split_by_chars(&input, 5); + let mut parser = Internlm2ToolParser::new(&test_tools()); + + let result = collect_stream(&mut parser, &chunks); + + assert_eq!(result.normal_text(), "hello "); + assert_eq!(result.calls().len(), 1); + assert_eq!(result.calls()[0].arguments, r#"{"location":"Tokyo"}"#); + } + + #[test] + fn internlm2_streaming_extracts_multiple_blocks() { + let input = format!( + "{}{}", + build_tool_call("get_weather", "parameters", r#"{"location":"Shanghai"}"#), + build_tool_call("add", "arguments", r#"{"x":1,"y":2}"#), + ); + let chunks = split_by_chars(&input, 7); + let mut parser = Internlm2ToolParser::new(&test_tools()); + + let result = collect_stream(&mut parser, &chunks); + + expect![[r#" + ToolParserOutput { + events: [ + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "get_weather", + ), + arguments: "{\"location\":\"Shanghai\"}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 1, + name: Some( + "add", + ), + arguments: "{\"x\":1,\"y\":2}", + }, + ), + ], + } + "#]] + .assert_debug_eq(&result); + } + + #[test] + fn internlm2_keeps_end_marker_literal_inside_json_string() { + let mut parser = Internlm2ToolParser::new(&test_tools()); + let arguments = format!(r#"{{"text":"literal {ACTION_END} inside"}}"#); + let input = build_tool_call("echo", "parameters", &arguments); + + let result = parser.parse_complete(&input).unwrap(); + + assert_eq!(result.calls().len(), 1); + assert_eq!(result.calls()[0].arguments, arguments); + } + + #[test] + fn internlm2_finish_errors_on_truncated_tool_call() { + let mut parser = Internlm2ToolParser::new(&test_tools()); + let pre_finish = parser + .parse_chunk(&format!( + r#"{ACTION_START}{{"name":"get_weather","parameters":{{"location""# + )) + .unwrap(); + let error = parser.finish().unwrap_err(); + + assert_eq!( + pre_finish.calls()[0].name.as_deref(), + Some("get_weather"), + "name delta is still emitted from parse_chunk() before truncation", + ); + assert!( + error.to_report_string().contains("incomplete InternLM2 tool call"), + "finish() reports the truncated tool call as incomplete: {}", + error.to_report_string(), + ); + } + + #[test] + fn internlm2_unknown_arguments_key_fails() { + let mut parser = Internlm2ToolParser::new(&test_tools()); + let input = build_tool_call("get_weather", "params", r#"{"location":"Tokyo"}"#); + + let error = parser.parse_chunk(&input).unwrap_err(); + + expect![[r#" + tool parser parsing failed: near "{\"name\":\"get_weather\",\"params\":{\"location\":\"Tokyo\"}}<|action_end|>": invalid InternLM2 + expected `parameters`, `arguments`"#]] + .assert_eq(&error.to_report_string()); + } + + #[test] + fn internlm2_preserve_special_tokens_is_true() { + let parser = Internlm2ToolParser::new(&test_tools()); + assert!(parser.preserve_special_tokens()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/llama.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/llama.rs new file mode 100644 index 0000000000000000000000000000000000000000..c3888073b80f974af709eda675ff6871d974da6e --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/llama.rs @@ -0,0 +1,495 @@ +use winnow::ascii::multispace0 as ws0; +use winnow::combinator::seq; +use winnow::error::{ModalResult, StrContext}; +use winnow::prelude::*; +use winnow::token::literal; + +use super::{ + JsonToolCallConfig, JsonToolCallEvent, JsonToolCallWhitespace, JsonToolInput, + argument_delta_event, tool_call_header_event, +}; +use crate::tool::utils::{JsonObjectScanState, parse_buffered_event}; +use crate::tool::{Result, StructuralTagModel, Tool, ToolCallDelta, ToolParser, ToolParserOutput}; + +#[derive(Debug, Clone, PartialEq, Eq)] +enum LlamaJsonMode { + Start, + Header, + Arguments { json_scan: JsonObjectScanState }, + AfterCall, + Passthrough, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum LlamaJsonEvent { + ToolCallHeader { function_name: String }, + Arguments { len: usize }, + ToolCallClose, + Separator, +} + +/// Tool parser for strict Llama JSON-template tool calls. +/// +/// Example tool call content: +/// +/// ```text +/// {"name":"get_weather","parameters":{"location":"Tokyo"}}; {"name":"add","parameters":{"x":1,"y":2}} +/// ``` +/// +/// Arguments are already OpenAI-style JSON text, so they are streamed as raw +/// argument deltas without schema conversion or JSON normalization. +/// +/// Natural text at the beginning of the stream permanently disables tool +/// parsing for that assistant output. +pub struct Llama3JsonToolParser { + buffer: String, + mode: LlamaJsonMode, + active_tool_index: Option, + emitted_tool_count: usize, +} + +impl Llama3JsonToolParser { + /// Create a Llama JSON tool parser. + fn new(_tools: &[Tool]) -> Self { + Self { + buffer: String::new(), + mode: LlamaJsonMode::Start, + active_tool_index: None, + emitted_tool_count: 0, + } + } + + /// Commit the stream to JSON parsing or permanent passthrough. + fn commit_start(&mut self) -> bool { + if !matches!(self.mode, LlamaJsonMode::Start) { + return true; + } + + if self.buffer.is_empty() { + return false; + } + + if self.buffer.starts_with('{') { + self.mode = LlamaJsonMode::Header; + } else { + self.mode = LlamaJsonMode::Passthrough; + } + true + } + + /// Apply one parsed Llama JSON event to parser state and output. + fn apply_event(&mut self, event: LlamaJsonEvent, output: &mut ToolParserOutput) -> Result<()> { + match event { + LlamaJsonEvent::ToolCallHeader { function_name } => { + let tool_index = self.emitted_tool_count; + self.emitted_tool_count += 1; + self.active_tool_index = Some(tool_index); + self.mode = LlamaJsonMode::Arguments { + json_scan: JsonObjectScanState::default(), + }; + output.push_call(ToolCallDelta { + tool_index, + name: Some(function_name), + arguments: String::new(), + }); + } + LlamaJsonEvent::Arguments { len: consumed_len } => { + let Some(tool_index) = self.active_tool_index else { + return Err(parsing_failed!( + "Llama JSON arguments without an active tool call" + )); + }; + output.push_call(ToolCallDelta { + tool_index, + name: None, + arguments: self.buffer[..consumed_len].to_string(), + }); + } + LlamaJsonEvent::ToolCallClose => { + self.active_tool_index = None; + self.mode = LlamaJsonMode::AfterCall; + } + LlamaJsonEvent::Separator => { + self.active_tool_index = None; + self.mode = LlamaJsonMode::Header; + } + } + Ok(()) + } + + fn reset(&mut self) -> String { + self.mode = LlamaJsonMode::Start; + self.active_tool_index = None; + self.emitted_tool_count = 0; + std::mem::take(&mut self.buffer) + } +} + +impl ToolParser for Llama3JsonToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn structural_tag_model(&self) -> Option { + Some(StructuralTagModel::Llama) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.buffer.push_str(chunk); + + if !self.commit_start() { + return Ok(()); + } + + if matches!(self.mode, LlamaJsonMode::Passthrough) { + output.push_text(&self.buffer); + self.buffer.clear(); + return Ok(()); + } + + while let Some((event, consumed_len)) = parse_buffered_event(&self.buffer, |input| { + parse_next_llama_json_event(input, &mut self.mode) + })? { + self.apply_event(event, output)?; + self.buffer.drain(..consumed_len); + } + + Ok(()) + } + + fn finish(&mut self) -> Result { + let mut output = ToolParserOutput::default(); + match &self.mode { + LlamaJsonMode::Start | LlamaJsonMode::Passthrough => { + output.push_text(&self.buffer); + } + LlamaJsonMode::AfterCall if self.buffer.trim().is_empty() => {} + LlamaJsonMode::Header | LlamaJsonMode::Arguments { .. } => { + return Err(parsing_failed!("incomplete Llama JSON tool call")); + } + LlamaJsonMode::AfterCall => { + return Err(parsing_failed!("invalid Llama JSON")); + } + } + let _ = self.reset(); + Ok(output) + } + + fn reset(&mut self) -> String { + Llama3JsonToolParser::reset(self) + } +} + +/// Parse a Llama JSON event for the current parser mode. +fn parse_next_llama_json_event( + input: &mut JsonToolInput<'_>, + mode: &mut LlamaJsonMode, +) -> ModalResult { + match mode { + LlamaJsonMode::Start | LlamaJsonMode::Passthrough => { + unreachable!("Llama JSON parser driver must commit before parsing events") + } + LlamaJsonMode::Header => llama_tool_call_header_event(input), + LlamaJsonMode::Arguments { json_scan } => parse_llama_arguments_event(input, json_scan), + LlamaJsonMode::AfterCall => after_call_event(input), + } +} + +/// Parse a Llama JSON tool-call header. +fn llama_tool_call_header_event(input: &mut JsonToolInput<'_>) -> ModalResult { + const CONFIG: JsonToolCallConfig = JsonToolCallConfig { + parser_name: "Llama JSON", + start_marker: "", + end_marker: "", + marker_whitespace: JsonToolCallWhitespace::Optional, + delimiter: Some(";"), + name_key: "name", + arguments_key: &["parameters"], + }; + + match tool_call_header_event(input, CONFIG)? { + JsonToolCallEvent::ToolCallHeader { function_name } => { + Ok(LlamaJsonEvent::ToolCallHeader { function_name }) + } + _ => unreachable!("tool_call_header_event only emits ToolCallHeader"), + } +} + +/// Parse one event inside a Llama JSON arguments payload. +fn parse_llama_arguments_event( + input: &mut JsonToolInput<'_>, + json_scan: &mut JsonObjectScanState, +) -> ModalResult { + if json_scan.complete() { + tool_call_close_event(input) + } else { + match argument_delta_event(input, json_scan)? { + JsonToolCallEvent::Arguments { len } => Ok(LlamaJsonEvent::Arguments { len }), + _ => unreachable!("argument_delta_event only emits Arguments"), + } + } +} + +/// Parse the outer closing brace for one Llama JSON tool call. +fn tool_call_close_event(input: &mut JsonToolInput<'_>) -> ModalResult { + literal("}").value(LlamaJsonEvent::ToolCallClose).parse_next(input) +} + +/// Parse a semicolon separator after one Llama JSON tool call. +fn after_call_event(input: &mut JsonToolInput<'_>) -> ModalResult { + seq!( + _: ws0, + _: literal(";"), + _: ws0, + ) + .value(LlamaJsonEvent::Separator) + .context(StrContext::Label("Llama JSON")) + .parse_next(input) +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + use thiserror_ext::AsReport; + + use super::Llama3JsonToolParser; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{ToolParser, ToolParserOutput, ToolParserTestExt as _}; + + fn build_tool_call(function_name: &str, parameters: &str) -> String { + format!(r#"{{"name":"{function_name}","parameters":{parameters}}}"#) + } + + #[test] + fn llama_json_parse_complete_without_tool_call_keeps_text() { + let mut parser = Llama3JsonToolParser::new(&test_tools()); + let output = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn llama_json_passthrough_never_reenters_tool_parsing() { + let mut parser = Llama3JsonToolParser::new(&test_tools()); + let mut output = parser.parse_chunk("plain text first ").unwrap(); + output.append( + parser + .parse_chunk(&build_tool_call("get_weather", r#"{"location":"Tokyo"}"#)) + .unwrap(), + ); + output.append(parser.finish().unwrap()); + + assert_eq!( + output.normal_text(), + r#"plain text first {"name":"get_weather","parameters":{"location":"Tokyo"}}"# + ); + assert!(output.calls().is_empty()); + } + + #[test] + fn llama_json_does_not_support_python_tag_prefix() { + let mut parser = Llama3JsonToolParser::new(&test_tools()); + let input = format!( + "<|python_tag|>{}", + build_tool_call("get_weather", r#"{"location":"Tokyo"}"#) + ); + let output = parser.parse_complete(&input).unwrap(); + + assert_eq!(output.normal_text(), input); + assert!(output.calls().is_empty()); + } + + #[test] + fn llama_json_rejects_leading_whitespace_before_tool_call() { + let mut parser = Llama3JsonToolParser::new(&test_tools()); + let input = format!( + "\n {}", + build_tool_call("get_weather", r#"{"location":"Tokyo"}"#) + ); + let output = parser.parse_complete(&input).unwrap(); + + assert_eq!(output.normal_text(), input); + assert!(output.calls().is_empty()); + } + + #[test] + fn llama_json_extracts_raw_parameters_object() { + let mut parser = Llama3JsonToolParser::new(&test_tools()); + let arguments = r#"{ "location": "Tokyo", "days": 3 }"#; + let output = parser.parse_complete(&build_tool_call("get_weather", arguments)).unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].tool_index, 0); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn llama_json_rejects_arguments_key() { + let mut parser = Llama3JsonToolParser::new(&test_tools()); + let error = parser + .parse_complete(r#"{"name":"get_weather","arguments":{"location":"Tokyo"}}"#) + .unwrap_err(); + + expect![[r#" + tool parser parsing failed: near "{\"name\":\"get_weather\",\"arguments\":{\"location\":\"Tokyo\"}}": invalid Llama JSON + expected `parameters`"#]] + .assert_eq(&error.to_report_string()); + } + + #[test] + fn llama_json_extracts_multiple_semicolon_separated_calls() { + let mut parser = Llama3JsonToolParser::new(&test_tools()); + let input = format!( + "{} \n; {}", + build_tool_call("get_weather", r#"{"location":"Shanghai"}"#), + build_tool_call("add", r#"{"x":1,"y":2}"#), + ); + let output = parser.parse_complete(&input).unwrap(); + + expect![[r#" + ToolParserOutput { + events: [ + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "get_weather", + ), + arguments: "{\"location\":\"Shanghai\"}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 1, + name: Some( + "add", + ), + arguments: "{\"x\":1,\"y\":2}", + }, + ), + ], + } + "#]] + .assert_debug_eq(&output); + } + + #[test] + fn llama_json_streaming_emits_argument_deltas() { + let mut parser = Llama3JsonToolParser::new(&test_tools()); + let chunks = [ + "{\"name\":\"get_weather\",\"parameters\":", + "{\"location\":", + "\"Beijing\"", + "}}", + ]; + + let mut output = ToolParserOutput::default(); + let mut observed_arguments = Vec::new(); + for chunk in chunks { + let next = parser.parse_chunk(chunk).unwrap(); + observed_arguments.extend( + next.calls() + .iter() + .filter(|call| call.name.is_none()) + .map(|call| call.arguments.clone()), + ); + output.append(next); + } + output.append(parser.finish().unwrap()); + + assert_eq!(observed_arguments, ["{\"location\":", "\"Beijing\"", "}"]); + assert_eq!( + output.coalesce().calls()[0].arguments, + r#"{"location":"Beijing"}"# + ); + } + + #[test] + fn llama_json_streaming_handles_split_objects_and_separator() { + let input = format!( + "{};{}", + build_tool_call("get_weather", r#"{"location":"Dallas","state":"TX"}"#), + build_tool_call("add", r#"{"x":4,"y":5}"#), + ); + let chunks = split_by_chars(&input, 6); + let mut parser = Llama3JsonToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.normal_text(), ""); + assert_eq!(output.calls().len(), 2); + assert_eq!( + output.calls()[0].arguments, + r#"{"location":"Dallas","state":"TX"}"# + ); + assert_eq!(output.calls()[1].name.as_deref(), Some("add")); + assert_eq!(output.calls()[1].arguments, r#"{"x":4,"y":5}"#); + } + + #[test] + fn llama_json_handles_nested_multiline_and_escaped_string_parameters() { + let mut parser = Llama3JsonToolParser::new(&test_tools()); + let arguments = r#"{ + "payload": {"items": [1, {"value": "literal { brace } and \"quote\""}]}, + "flag": true +}"#; + let output = parser.parse_complete(&build_tool_call("convert", arguments)).unwrap(); + + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn llama_json_keeps_trailing_whitespace_after_tool_call() { + let mut parser = Llama3JsonToolParser::new(&test_tools()); + let output = parser + .parse_complete(&format!( + "{}\n\t ", + build_tool_call("get_weather", r#"{"location":"Tokyo"}"#) + )) + .unwrap(); + + assert_eq!(output.normal_text(), ""); + assert_eq!(output.calls().len(), 1); + } + + #[test] + fn llama_json_finish_fails_incomplete_tool_call() { + let mut parser = Llama3JsonToolParser::new(&test_tools()); + parser.parse_chunk(r#"{"name":"get_weather","parameters":{"location""#).unwrap(); + + let error = parser.finish().unwrap_err(); + + expect!["tool parser parsing failed: incomplete Llama JSON tool call"] + .assert_eq(&error.to_report_string()); + } + + #[test] + fn llama_json_malformed_field_order_fails_fast() { + let mut parser = Llama3JsonToolParser::new(&test_tools()); + let error = parser.parse_chunk(r#"{"parameters":{},"name":"get_weather"}"#).unwrap_err(); + + expect![[r#" + tool parser parsing failed: near "{\"parameters\":{},\"name\":\"get_weather\"}": invalid Llama JSON + expected `name`"#]] + .assert_eq(&error.to_report_string()); + } + + #[test] + fn llama_json_trailing_non_separator_content_errors() { + let mut parser = Llama3JsonToolParser::new(&test_tools()); + let error = parser + .parse_chunk(&format!( + "{} trailing", + build_tool_call("get_weather", r#"{"location":"Tokyo"}"#) + )) + .unwrap_err(); + + expect![[r#"tool parser parsing failed: near " trailing": invalid Llama JSON"#]] + .assert_eq(&error.to_report_string()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/mistral.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/mistral.rs new file mode 100644 index 0000000000000000000000000000000000000000..40c0fa342ce756de85c0e42f77a27ce68171df11 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/mistral.rs @@ -0,0 +1,247 @@ +use super::{JsonToolCallConfig, JsonToolCallParser, JsonToolCallWhitespace}; +use crate::tool::{Result, Tool, ToolParser, ToolParserOutput}; + +const MISTRAL_CONFIG: JsonToolCallConfig = JsonToolCallConfig { + parser_name: "Mistral", + start_marker: "[TOOL_CALLS] [", + end_marker: "]", + marker_whitespace: JsonToolCallWhitespace::Optional, + delimiter: Some(","), + name_key: "name", + arguments_key: &["arguments"], +}; + +/// Tool parser for Mistral JSON-array tool calls. +/// +/// Example tool call content: +/// +/// ```text +/// [TOOL_CALLS] [{"name": "get_weather", "arguments": {"location":"Tokyo"}}] +/// ``` +/// +/// Arguments are already OpenAI-style JSON text, so they are streamed as raw +/// argument deltas without schema conversion or JSON normalization. +pub struct MistralToolParser { + inner: JsonToolCallParser, +} + +impl MistralToolParser { + /// Create a Mistral tool parser. + fn new(_tools: &[Tool]) -> Self { + Self { + inner: JsonToolCallParser::new(MISTRAL_CONFIG), + } + } +} + +impl ToolParser for MistralToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.inner.parse_into(chunk, output) + } + + fn finish(&mut self) -> Result { + self.inner.finish() + } + + fn reset(&mut self) -> String { + self.inner.reset() + } +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + use thiserror_ext::AsReport; + + use super::MistralToolParser; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{ToolParser, ToolParserOutput, ToolParserTestExt as _}; + + fn build_tool_call(function_name: &str, arguments: &str) -> String { + format!(r#"{{"name":"{function_name}","arguments":{arguments}}}"#) + } + + fn build_tool_calls(tool_calls: &[String]) -> String { + format!("[TOOL_CALLS] [{}]", tool_calls.join(",")) + } + + #[test] + fn mistral_parse_complete_without_tool_call_keeps_text() { + let mut parser = MistralToolParser::new(&test_tools()); + let output = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn mistral_parse_complete_extracts_raw_json_arguments() { + let mut parser = MistralToolParser::new(&test_tools()); + let arguments = r#"{ "location": "Tokyo", "days": "3" }"#; + let output = parser + .parse_complete(&format!( + "Let me check.\n{}", + build_tool_calls(&[build_tool_call("get_weather", arguments)]) + )) + .unwrap(); + + assert_eq!(output.normal_text(), "Let me check.\n"); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].tool_index, 0); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn mistral_parse_complete_extracts_pretty_multiple_tool_calls() { + let mut parser = MistralToolParser::new(&test_tools()); + let output = parser + .parse_complete( + r#"I'll help. +[TOOL_CALLS] [ + {"name": "get_weather", "arguments": {"city": "Tokyo", "units": "celsius"}} + , + {"name": "add", "arguments": {"x": 1, "y": 2}} +]"#, + ) + .unwrap(); + + expect![[r#" + ToolParserOutput { + events: [ + Text( + "I'll help.\n", + ), + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "get_weather", + ), + arguments: "{\"city\": \"Tokyo\", \"units\": \"celsius\"}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 1, + name: Some( + "add", + ), + arguments: "{\"x\": 1, \"y\": 2}", + }, + ), + ], + } + "#]] + .assert_debug_eq(&output); + } + + #[test] + fn mistral_does_not_validate_or_normalize_arguments() { + let mut parser = MistralToolParser::new(&test_tools()); + let arguments = r#"{"location":"Tokyo",}"#; + let output = parser + .parse_complete(&build_tool_calls(&[build_tool_call( + "get_weather", + arguments, + )])) + .unwrap(); + + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn mistral_streaming_emits_argument_deltas() { + let mut parser = MistralToolParser::new(&test_tools()); + let chunks = [ + "preface [TOOL", + "_CALLS] [{\"name\":\"get_weather\",\"arguments\":", + "{\"location\":", + "\"Beijing\"", + "}", + "}] suffix", + ]; + + let mut output = ToolParserOutput::default(); + let mut observed_arguments = Vec::new(); + for chunk in chunks { + let next = parser.parse_chunk(chunk).unwrap(); + observed_arguments.extend( + next.calls() + .iter() + .filter(|call| call.name.is_none()) + .map(|call| call.arguments.clone()), + ); + output.append(next); + } + output.append(parser.finish().unwrap()); + + assert_eq!(observed_arguments, ["{\"location\":", "\"Beijing\"", "}"]); + assert_eq!(output.normal_text(), "preface suffix"); + assert_eq!( + output.coalesce().calls()[0].arguments, + r#"{"location":"Beijing"}"# + ); + } + + #[test] + fn mistral_streaming_handles_split_markers() { + let input = format!( + "hello {}", + build_tool_calls(&[build_tool_call("get_weather", r#"{"location":"Tokyo"}"#)]) + ); + let chunks = split_by_chars(&input, 5); + let mut parser = MistralToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.normal_text(), "hello "); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].arguments, r#"{"location":"Tokyo"}"#); + } + + #[test] + fn mistral_keeps_array_bracket_literal_inside_json_string() { + let mut parser = MistralToolParser::new(&test_tools()); + let arguments = r#"{"text":"Array notation: arr[0] = value[1]"}"#; + let output = parser + .parse_complete(&build_tool_calls(&[build_tool_call("echo", arguments)])) + .unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn mistral_finish_fails_incomplete_tool_call() { + let mut parser = MistralToolParser::new(&test_tools()); + parser + .parse_chunk(r#"[TOOL_CALLS] [{"name":"get_weather","arguments":{"location""#) + .unwrap(); + + let error = parser.finish().unwrap_err(); + + expect!["tool parser parsing failed: incomplete Mistral tool call"] + .assert_eq(&error.to_report_string()); + } + + #[test] + fn mistral_malformed_field_order_fails_fast() { + let mut parser = MistralToolParser::new(&test_tools()); + let error = parser + .parse_chunk(r#"[TOOL_CALLS] [{"arguments":{},"name":"get_weather"}]"#) + .unwrap_err(); + + expect![[r#" + tool parser parsing failed: near "{\"arguments\":{},\"name\":\"get_weather\"}]": invalid Mistral + expected `name`"#]] + .assert_eq(&error.to_report_string()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/mod.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..2891d95d2817608d3bdc8249848c6b9cc7c3b781 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/mod.rs @@ -0,0 +1,513 @@ +//! Shared parser core for JSON tool calls wrapped by text markers. + +pub use granite4::Granite4ToolParser; +pub use hermes::HermesToolParser; +pub use internlm2::Internlm2ToolParser; +pub use llama::Llama3JsonToolParser; +pub use mistral::MistralToolParser; +pub use phi4mini::Phi4MiniJsonToolParser; +pub use qwen::Qwen3XmlToolParser; + +mod granite4; +mod hermes; +mod internlm2; +mod llama; +mod mistral; +mod phi4mini; +mod qwen; + +use winnow::ascii::multispace0 as ws0; +use winnow::combinator::{alt, seq}; +use winnow::error::{AddContext, ModalResult, StrContext, StrContextValue}; +use winnow::prelude::*; +use winnow::stream::{Partial, Stream}; +use winnow::token::literal; + +use super::utils::{ + JsonObjectScanState, json_str, parse_buffered_event, safe_text_len, take_json_object, +}; +use super::{Result, ToolCallDelta, ToolParserOutput}; + +type JsonToolInput<'i> = Partial<&'i str>; + +#[derive(Debug, Clone, Copy)] +struct JsonToolCallConfig { + parser_name: &'static str, + start_marker: &'static str, + end_marker: &'static str, + marker_whitespace: JsonToolCallWhitespace, + delimiter: Option<&'static str>, + name_key: &'static str, + /// Candidate JSON keys naming the arguments payload, tried in order. + /// Most parsers use a single key like `["arguments"]`, but some accept + /// multiple (e.g. InternLM2 accepts `parameters` or `arguments`). + arguments_key: &'static [&'static str], +} + +#[derive(Debug, Clone, Copy)] +enum JsonToolCallWhitespace { + Optional, + Exact(&'static str), +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum JsonToolCallMode { + Text, + Header, + Arguments { json_scan: JsonObjectScanState }, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum JsonToolCallEvent { + Text { len: usize }, + ToolCallStart, + ToolCallHeader { function_name: String }, + Arguments { len: usize }, + ToolCallDelimiter, + ToolCallEnd, +} + +/// Tool parser core for marker-wrapped JSON tool calls. +#[derive(Debug)] +struct JsonToolCallParser { + config: JsonToolCallConfig, + buffer: String, + mode: JsonToolCallMode, + active_tool_index: Option, + emitted_tool_count: usize, +} + +impl JsonToolCallParser { + /// Create a marker-wrapped JSON tool-call parser. + fn new(config: JsonToolCallConfig) -> Self { + Self { + config, + buffer: String::new(), + mode: JsonToolCallMode::Text, + active_tool_index: None, + emitted_tool_count: 0, + } + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.buffer.push_str(chunk); + let config = self.config; + + while let Some((event, consumed_len)) = parse_buffered_event(&self.buffer, |input| { + parse_next_json_tool_call_event(input, &mut self.mode, config) + })? { + self.apply_event(event, output)?; + self.buffer.drain(..consumed_len); + } + + Ok(()) + } + + fn finish(&mut self) -> Result { + let mut output = ToolParserOutput::default(); + match &self.mode { + JsonToolCallMode::Text => output.push_text(&self.buffer), + JsonToolCallMode::Header | JsonToolCallMode::Arguments { .. } => { + return Err(parsing_failed!( + "incomplete {} tool call", + self.config.parser_name + )); + } + } + let _ = self.reset(); + Ok(output) + } + + /// Apply one parsed JSON tool-call event to parser state and output. + fn apply_event( + &mut self, + event: JsonToolCallEvent, + output: &mut ToolParserOutput, + ) -> Result<()> { + match event { + JsonToolCallEvent::Text { len: consumed_len } => { + output.push_text(&self.buffer[..consumed_len]); + } + JsonToolCallEvent::ToolCallStart => self.mode = JsonToolCallMode::Header, + JsonToolCallEvent::ToolCallHeader { function_name } => { + let tool_index = self.emitted_tool_count; + self.emitted_tool_count += 1; + self.active_tool_index = Some(tool_index); + self.mode = JsonToolCallMode::Arguments { + json_scan: JsonObjectScanState::default(), + }; + output.push_call(ToolCallDelta { + tool_index, + name: Some(function_name), + arguments: String::new(), + }); + } + JsonToolCallEvent::Arguments { len: consumed_len } => { + let Some(tool_index) = self.active_tool_index else { + return Err(parsing_failed!( + "{} arguments without an active tool call", + self.config.parser_name + )); + }; + output.push_call(ToolCallDelta { + tool_index, + name: None, + arguments: self.buffer[..consumed_len].to_string(), + }); + } + JsonToolCallEvent::ToolCallDelimiter => { + self.active_tool_index = None; + self.mode = JsonToolCallMode::Header; + } + JsonToolCallEvent::ToolCallEnd => { + self.active_tool_index = None; + self.mode = JsonToolCallMode::Text; + } + } + Ok(()) + } + + fn reset(&mut self) -> String { + self.mode = JsonToolCallMode::Text; + self.active_tool_index = None; + self.emitted_tool_count = 0; + std::mem::take(&mut self.buffer) + } +} + +/// Parse a JSON tool-call event for the current parser mode. +fn parse_next_json_tool_call_event( + input: &mut JsonToolInput<'_>, + mode: &mut JsonToolCallMode, + config: JsonToolCallConfig, +) -> ModalResult { + match mode { + JsonToolCallMode::Text => parse_text_event(input, config), + JsonToolCallMode::Header => tool_call_header_event(input, config), + JsonToolCallMode::Arguments { json_scan } => { + parse_arguments_event(input, json_scan, config) + } + } +} + +/// Parse a text-mode JSON tool-call event. +fn parse_text_event( + input: &mut JsonToolInput<'_>, + config: JsonToolCallConfig, +) -> ModalResult { + alt(( + |input: &mut JsonToolInput<'_>| tool_call_start_event(input, config), + |input: &mut JsonToolInput<'_>| safe_text_event(input, config), + )) + .parse_next(input) +} + +/// Parse a marker-wrapped JSON tool-call start marker. +fn tool_call_start_event( + input: &mut JsonToolInput<'_>, + config: JsonToolCallConfig, +) -> ModalResult { + seq!( + _: literal(config.start_marker), + _: |input: &mut JsonToolInput<'_>| marker_whitespace(input, config), + ) + .value(JsonToolCallEvent::ToolCallStart) + .parse_next(input) +} + +/// Parse a marker-wrapped JSON tool-call header before the raw arguments +/// payload. +fn tool_call_header_event( + input: &mut JsonToolInput<'_>, + config: JsonToolCallConfig, +) -> ModalResult { + let (function_name,) = seq!( + _: ws0, + _: literal("{"), + _: ws0, + _: |input: &mut JsonToolInput<'_>| json_key(input, config.name_key), + _: ws0, + _: literal(":"), + _: ws0, + json_str, + _: ws0, + _: literal(","), + _: ws0, + _: |input: &mut JsonToolInput<'_>| json_arguments_key(input, config.arguments_key), + _: ws0, + _: literal(":"), + _: ws0, + ) + .context(StrContext::Label(config.parser_name)) + .parse_next(input)?; + + Ok(JsonToolCallEvent::ToolCallHeader { function_name }) +} + +/// Parse a configured JSON object key. +fn json_key(input: &mut JsonToolInput<'_>, key: &'static str) -> ModalResult<()> { + seq!( + _: literal("\""), + _: literal(key).context(StrContext::Expected(StrContextValue::StringLiteral(key))), + _: literal("\""), + ) + .void() + .parse_next(input) +} + +/// Parse a JSON object key accepting any of `candidates`. +/// +/// The full quoted key is consumed and compared against the candidate list, +/// so this works correctly under partial input regardless of key lengths. +/// +/// On mismatch, each candidate is attached as its own `Expected` context so the +/// error enumerates every valid key ("expected `a`, expected `b`"). Because +/// `StrContextValue::StringLiteral` carries a single `&'static str`, the +/// contexts are added in a loop over `candidates` rather than through chained +/// `.context(...)` calls, which keeps the diagnostics complete for any number +/// of candidates. +fn json_arguments_key( + input: &mut JsonToolInput<'_>, + candidates: &'static [&'static str], +) -> ModalResult<()> { + let start = input.checkpoint(); + json_str + .verify(|key: &String| candidates.contains(&key.as_str())) + .void() + .parse_next(input) + .map_err(|err| { + err.map(|context_error| { + candidates.iter().fold(context_error, |context_error, candidate| { + context_error.add_context( + &*input, + &start, + StrContext::Expected(StrContextValue::StringLiteral(candidate)), + ) + }) + }) + }) +} + +/// Parse one event inside a marker-wrapped JSON tool-call arguments payload. +fn parse_arguments_event( + input: &mut JsonToolInput<'_>, + json_scan: &mut JsonObjectScanState, + config: JsonToolCallConfig, +) -> ModalResult { + if json_scan.complete() { + tool_call_close_event(input, config) + } else { + argument_delta_event(input, json_scan) + } +} + +/// Parse a raw JSON arguments delta. +fn argument_delta_event( + input: &mut JsonToolInput<'_>, + json_scan: &mut JsonObjectScanState, +) -> ModalResult { + take_json_object(input, json_scan).map(|len| JsonToolCallEvent::Arguments { len }) +} + +/// Parse a marker-wrapped JSON tool-call close marker. +fn tool_call_close_event( + input: &mut JsonToolInput<'_>, + config: JsonToolCallConfig, +) -> ModalResult { + let _ = literal("}").parse_next(input)?; + + match config.delimiter { + Some(delimiter) => alt(( + |input: &mut JsonToolInput<'_>| tool_call_end_event(input, config), + |input: &mut JsonToolInput<'_>| tool_call_delimiter_event(input, delimiter), + )) + .parse_next(input), + None => tool_call_end_event(input, config), + } +} + +/// Parse a marker-wrapped JSON tool-call end marker. +fn tool_call_end_event( + input: &mut JsonToolInput<'_>, + config: JsonToolCallConfig, +) -> ModalResult { + seq!( + _: |input: &mut JsonToolInput<'_>| marker_whitespace(input, config), + _: literal(config.end_marker), + ) + .value(JsonToolCallEvent::ToolCallEnd) + .parse_next(input) +} + +/// Parse a delimiter between JSON tool calls inside one marker block. +fn tool_call_delimiter_event( + input: &mut JsonToolInput<'_>, + delimiter: &'static str, +) -> ModalResult { + seq!( + _: ws0, + _: literal(delimiter), + _: ws0, + ) + .value(JsonToolCallEvent::ToolCallDelimiter) + .parse_next(input) +} + +/// Parse configured whitespace around a marker-wrapped JSON tool call. +fn marker_whitespace(input: &mut JsonToolInput<'_>, config: JsonToolCallConfig) -> ModalResult<()> { + match config.marker_whitespace { + JsonToolCallWhitespace::Optional => ws0.void().parse_next(input), + JsonToolCallWhitespace::Exact(whitespace) => literal(whitespace).void().parse_next(input), + } +} + +/// Parse a safe text run before the next marker-wrapped JSON tool call. +fn safe_text_event( + input: &mut JsonToolInput<'_>, + config: JsonToolCallConfig, +) -> ModalResult { + safe_text_len(input, config.start_marker).map(|len| JsonToolCallEvent::Text { len }) +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + + use super::{JsonToolCallConfig, JsonToolCallParser, JsonToolCallWhitespace}; + use crate::tool::ToolParserOutput; + + const DELIMITED_CONFIG: JsonToolCallConfig = JsonToolCallConfig { + parser_name: "Delimited JSON", + start_marker: "", + end_marker: "", + marker_whitespace: JsonToolCallWhitespace::Optional, + delimiter: Some("<"), + name_key: "function", + arguments_key: &["parameters"], + }; + + fn build_tool_call(function_name: &str, arguments: &str) -> String { + format!(r#"{{"function":"{function_name}","parameters":{arguments}}}"#) + } + + fn build_tool_calls(tool_calls: &[String]) -> String { + format!("{}", tool_calls.join(" <\n")) + } + + fn collect_chunks(parser: &mut JsonToolCallParser, chunks: &[&str]) -> ToolParserOutput { + let mut output = ToolParserOutput::default(); + for chunk in chunks { + parser.parse_into(chunk, &mut output).unwrap(); + } + output.append(parser.finish().unwrap()); + output.coalesce() + } + + #[test] + fn json_tool_call_delimiter_extracts_multiple_calls_in_one_block() { + let input = build_tool_calls(&[ + build_tool_call("get_weather", r#"{"location":"Shanghai"}"#), + build_tool_call("add", r#"{"x":1,"y":2}"#), + ]); + let mut parser = JsonToolCallParser::new(DELIMITED_CONFIG); + + let output = collect_chunks(&mut parser, &[&input]); + + expect![[r#" + ToolParserOutput { + events: [ + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "get_weather", + ), + arguments: "{\"location\":\"Shanghai\"}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 1, + name: Some( + "add", + ), + arguments: "{\"x\":1,\"y\":2}", + }, + ), + ], + } + "#]] + .assert_debug_eq(&output); + } + + #[test] + fn json_tool_call_delimiter_can_arrive_in_later_chunk() { + let mut parser = JsonToolCallParser::new(DELIMITED_CONFIG); + let chunks = [ + r#"{"function":"get_weather","parameters":{"location":"Shanghai"}}"#, + " <\n", + r#"{"function":"add","parameters":{"x":1,"y":2}}"#, + "", + ]; + + let output = collect_chunks(&mut parser, &chunks); + + expect![[r#" + ToolParserOutput { + events: [ + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "get_weather", + ), + arguments: "{\"location\":\"Shanghai\"}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 1, + name: Some( + "add", + ), + arguments: "{\"x\":1,\"y\":2}", + }, + ), + ], + } + "#]] + .assert_debug_eq(&output); + } + + #[test] + fn json_tool_call_end_marker_wins_over_delimiter_prefix() { + let mut parser = JsonToolCallParser::new(DELIMITED_CONFIG); + let chunks = [ + r#"{"function":"get_weather","parameters":{"location":"Shanghai"}}"#, + " ", + " trailing text", + ]; + + let output = collect_chunks(&mut parser, &chunks); + + expect![[r#" + ToolParserOutput { + events: [ + Text( + " trailing text", + ), + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "get_weather", + ), + arguments: "{\"location\":\"Shanghai\"}", + }, + ), + ], + } + "#]] + .assert_debug_eq(&output); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/phi4mini.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/phi4mini.rs new file mode 100644 index 0000000000000000000000000000000000000000..c9700acfbe35c1f26e67a3a0cc6cc26853a3d6e5 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/phi4mini.rs @@ -0,0 +1,330 @@ +use super::{JsonToolCallConfig, JsonToolCallParser, JsonToolCallWhitespace}; +use crate::tool::{Result, Tool, ToolParser, ToolParserOutput}; + +const PHI4MINI_CONFIG: JsonToolCallConfig = JsonToolCallConfig { + parser_name: "Phi4Mini", + start_marker: "functools[", + end_marker: "]", + marker_whitespace: JsonToolCallWhitespace::Optional, + delimiter: Some(","), + name_key: "name", + // Accept both key variants emitted by Phi-4 Mini tool-call templates. + arguments_key: &["arguments", "parameters"], +}; + +/// Tool parser for phi-4-mini models. +/// +/// Example tool-call content: +/// +/// ```text +/// functools[{"name": "get_weather", "arguments": {"location": "Tokyo"}}] +/// ``` +/// +/// phi-4-mini emits an array of tool-call objects wrapped in a `functools[..]` +/// envelope. Each object names the function with `name` and carries its +/// arguments under `arguments` (preferred) or `parameters`. Arguments are +/// already OpenAI-style JSON text, so they are streamed as raw argument deltas +/// without schema conversion or JSON normalization. +pub struct Phi4MiniJsonToolParser { + inner: JsonToolCallParser, +} + +impl Phi4MiniJsonToolParser { + /// Create a phi-4-mini tool parser. + fn new(_tools: &[Tool]) -> Self { + Self { + inner: JsonToolCallParser::new(PHI4MINI_CONFIG), + } + } +} + +impl ToolParser for Phi4MiniJsonToolParser { + /// Create a boxed phi-4-mini tool parser. + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + /// Feed one decoded text chunk through the phi-4-mini parser. + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.inner.parse_into(chunk, output) + } + + /// Flush any buffered partial state at end of stream. + fn finish(&mut self) -> Result { + self.inner.finish() + } + + /// Clear parser state and return currently uncommitted buffered text. + fn reset(&mut self) -> String { + self.inner.reset() + } +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + use thiserror_ext::AsReport; + + use super::Phi4MiniJsonToolParser; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{ToolParser, ToolParserTestExt as _}; + + /// Build one phi-4-mini tool-call object: `{"name":..,"":}`. + fn build_call(function_name: &str, args_key: &str, arguments: &str) -> String { + format!(r#"{{"name":"{function_name}","{args_key}":{arguments}}}"#) + } + + /// Wrap tool-call objects in the `functools[..]` envelope. + fn wrap(calls: &[String]) -> String { + format!("functools[{}]", calls.join(",")) + } + + #[test] + fn phi4mini_parse_complete_without_tool_call_keeps_text() { + let mut parser = Phi4MiniJsonToolParser::new(&test_tools()); + let result = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(result.normal_text(), "Hello, world!"); + assert!(result.calls().is_empty()); + } + + #[test] + fn phi4mini_parse_complete_extracts_arguments_key() { + let mut parser = Phi4MiniJsonToolParser::new(&test_tools()); + let arguments = r#"{"location":"Tokyo","days":"3"}"#; + let result = parser + .parse_complete(&wrap(&[build_call("get_weather", "arguments", arguments)])) + .unwrap(); + + assert_eq!(result.calls().len(), 1); + assert_eq!(result.calls()[0].tool_index, 0); + assert_eq!(result.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(result.calls()[0].arguments, arguments); + } + + #[test] + fn phi4mini_parse_complete_falls_back_to_parameters_key() { + let mut parser = Phi4MiniJsonToolParser::new(&test_tools()); + let arguments = r#"{"location":"Tokyo"}"#; + let result = parser + .parse_complete(&wrap(&[build_call("get_weather", "parameters", arguments)])) + .unwrap(); + + assert_eq!(result.calls().len(), 1); + assert_eq!(result.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(result.calls()[0].arguments, arguments); + } + + #[test] + fn phi4mini_extracts_multiple_comma_delimited_calls() { + let input = wrap(&[ + build_call("get_weather", "arguments", r#"{"location":"Shanghai"}"#), + build_call("add", "arguments", r#"{"x":1,"y":2}"#), + ]); + let mut parser = Phi4MiniJsonToolParser::new(&test_tools()); + + let result = parser.parse_complete(&input).unwrap(); + + expect![[r#" + ToolParserOutput { + events: [ + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "get_weather", + ), + arguments: "{\"location\":\"Shanghai\"}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 1, + name: Some( + "add", + ), + arguments: "{\"x\":1,\"y\":2}", + }, + ), + ], + } + "#]] + .assert_debug_eq(&result); + } + + /// The shared JSON core scans matched braces, so bracket-bearing argument + /// values are forwarded intact. + #[test] + fn phi4mini_array_valued_arguments_are_not_truncated() { + let mut parser = Phi4MiniJsonToolParser::new(&test_tools()); + let arguments = r#"{"items":[1,2],"flag":true}"#; + let result = parser + .parse_complete(&wrap(&[build_call("convert", "arguments", arguments)])) + .unwrap(); + + assert_eq!(result.calls().len(), 1); + assert_eq!(result.calls()[0].arguments, arguments); + } + + /// Preface text before a tool call is preserved as normal_text, consistent + /// with the other JSON parsers in this crate. + #[test] + fn phi4mini_preserves_text_before_tool_call() { + let mut parser = Phi4MiniJsonToolParser::new(&test_tools()); + let input = format!( + "Let me check.\n{}", + wrap(&[build_call( + "get_weather", + "arguments", + r#"{"location":"Tokyo"}"# + )]) + ); + + let result = parser.parse_complete(&input).unwrap(); + + assert_eq!(result.normal_text(), "Let me check.\n"); + assert_eq!(result.calls().len(), 1); + } + + #[test] + fn phi4mini_does_not_validate_or_normalize_arguments() { + let mut parser = Phi4MiniJsonToolParser::new(&test_tools()); + let arguments = r#"{"location":"Tokyo",}"#; + let result = parser + .parse_complete(&wrap(&[build_call("get_weather", "arguments", arguments)])) + .unwrap(); + + assert_eq!(result.calls()[0].arguments, arguments); + } + + /// The bundled `tool_chat_template_phi4_mini.jinja` emits objects with + /// whitespace after `:` and `,` (e.g. `{"name": "f", "arguments": {..}}`). + /// Confirm the parser handles that real model format and preserves the + /// inner argument spacing verbatim. + #[test] + fn phi4mini_accepts_real_model_whitespace_format() { + let mut parser = Phi4MiniJsonToolParser::new(&test_tools()); + let input = r#"functools[{"name": "get_weather", "arguments": {"location": "Tokyo"}}]"#; + + let result = parser.parse_complete(input).unwrap(); + + assert_eq!(result.calls().len(), 1); + assert_eq!(result.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(result.calls()[0].arguments, r#"{"location": "Tokyo"}"#); + } + + /// Argument deltas are streamed through the shared JSON core. + #[test] + fn phi4mini_streaming_emits_argument_deltas() { + let mut parser = Phi4MiniJsonToolParser::new(&test_tools()); + let chunks = [ + "preface functo", + "ols[", + r#"{"name":"get_weather","arguments":"#, + r#"{"location":"#, + r#""Beijing""#, + r#"}"#, + r#"}]"#, + " suffix", + ]; + + let result = collect_stream(&mut parser, &chunks); + + assert_eq!(result.normal_text(), "preface suffix"); + assert_eq!(result.calls().len(), 1); + assert_eq!(result.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(result.calls()[0].arguments, r#"{"location":"Beijing"}"#); + } + + #[test] + fn phi4mini_streaming_handles_split_markers() { + let input = format!( + "hello {}", + wrap(&[build_call( + "get_weather", + "arguments", + r#"{"location":"Tokyo"}"# + )]) + ); + let chunks = split_by_chars(&input, 5); + let mut parser = Phi4MiniJsonToolParser::new(&test_tools()); + + let result = collect_stream(&mut parser, &chunks); + + assert_eq!(result.normal_text(), "hello "); + assert_eq!(result.calls().len(), 1); + assert_eq!(result.calls()[0].arguments, r#"{"location":"Tokyo"}"#); + } + + #[test] + fn phi4mini_finish_errors_on_truncated_tool_call() { + let mut parser = Phi4MiniJsonToolParser::new(&test_tools()); + let _ = parser + .parse_chunk(r#"functools[{"name":"get_weather","arguments":{"location""#) + .unwrap(); + let error = parser.finish().unwrap_err(); + + assert!( + error.to_report_string().contains("incomplete Phi4Mini tool call"), + "finish() reports the truncated tool call as incomplete: {}", + error.to_report_string(), + ); + } + + #[test] + fn phi4mini_preserve_special_tokens_is_false() { + let parser = Phi4MiniJsonToolParser::new(&test_tools()); + assert!(!parser.preserve_special_tokens()); + } + + /// The brace-scanning core handles nested arrays and objects in arguments. + #[test] + fn phi4mini_parses_nested_arrays_and_objects() { + let mut parser = Phi4MiniJsonToolParser::new(&test_tools()); + let arguments = + r#"{"array_field":["a","b","c"],"object_field":{"nested":"value"},"empty_object":{}}"#; + let result = parser + .parse_complete(&wrap(&[build_call("convert", "arguments", arguments)])) + .unwrap(); + + assert_eq!(result.calls().len(), 1); + assert_eq!(result.calls()[0].name.as_deref(), Some("convert")); + assert_eq!(result.calls()[0].arguments, arguments); + } + + /// The chat template emits parallel calls as `},\n {` (comma + newline + + /// indent). Confirm the `Optional` marker whitespace and `,` delimiter + /// parse the real multi-call layout. + #[test] + fn phi4mini_parses_parallel_calls_in_template_format() { + let mut parser = Phi4MiniJsonToolParser::new(&test_tools()); + let input = concat!( + "functools[\n", + " {\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}},\n", + " {\"name\": \"add\", \"arguments\": {\"x\": 1, \"y\": 2}}\n", + "]" + ); + + let result = parser.parse_complete(input).unwrap(); + + assert_eq!(result.calls().len(), 2); + assert_eq!(result.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(result.calls()[1].name.as_deref(), Some("add")); + } + + /// The shared core requires an object after the start marker. + #[test] + fn phi4mini_empty_array_errors() { + let mut parser = Phi4MiniJsonToolParser::new(&test_tools()); + let error = parser.parse_complete("functools[]").unwrap_err(); + + assert!( + error.to_report_string().contains("invalid Phi4Mini"), + "empty functools[] should error: {}", + error.to_report_string(), + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/qwen.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/qwen.rs new file mode 100644 index 0000000000000000000000000000000000000000..55d6ea83f61a0f522aa4fe4191423ba6f03839c6 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/json/qwen.rs @@ -0,0 +1,287 @@ +use super::{JsonToolCallConfig, JsonToolCallParser, JsonToolCallWhitespace}; +use crate::tool::{Result, StructuralTagModel, Tool, ToolParser, ToolParserOutput}; + +const QWEN_XML_CONFIG: JsonToolCallConfig = JsonToolCallConfig { + parser_name: "Qwen XML", + start_marker: "", + end_marker: "", + marker_whitespace: JsonToolCallWhitespace::Exact("\n"), + delimiter: None, + name_key: "name", + arguments_key: &["arguments"], +}; + +/// Tool parser for Qwen XML-wrapped JSON tool calls. +/// +/// Example tool call content: +/// +/// ```text +/// +/// {"name": "get_weather", "arguments": {"location":"Tokyo"}} +/// +/// ``` +/// +/// Arguments are already OpenAI-style JSON text, so they are streamed as raw +/// argument deltas without schema conversion or JSON normalization. +/// +/// Note: parallel calls are represented as repeated +/// `...` blocks, not as multiple calls inside one tag. +pub struct Qwen3XmlToolParser { + inner: JsonToolCallParser, +} + +impl Qwen3XmlToolParser { + /// Create a Qwen XML tool parser. + fn new(_tools: &[Tool]) -> Self { + Self { + inner: JsonToolCallParser::new(QWEN_XML_CONFIG), + } + } +} + +impl ToolParser for Qwen3XmlToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn structural_tag_model(&self) -> Option { + Some(StructuralTagModel::Qwen3) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.inner.parse_into(chunk, output) + } + + fn finish(&mut self) -> Result { + self.inner.finish() + } + + fn reset(&mut self) -> String { + self.inner.reset() + } +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + use thiserror_ext::AsReport; + + use super::Qwen3XmlToolParser; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{ToolParser, ToolParserOutput, ToolParserTestExt as _}; + + fn build_tool_call(function_name: &str, arguments: &str) -> String { + format!( + "\n{{\"name\": \"{function_name}\", \"arguments\": {arguments}}}\n" + ) + } + + #[test] + fn qwen_xml_parse_complete_without_tool_call_keeps_text() { + let mut parser = Qwen3XmlToolParser::new(&test_tools()); + let output = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn qwen_xml_parse_complete_extracts_raw_json_arguments() { + let mut parser = Qwen3XmlToolParser::new(&test_tools()); + let arguments = r#"{ "location": "Tokyo", "days": "3" }"#; + let output = parser + .parse_complete(&format!( + "Let me check.\n{}", + build_tool_call("get_weather", arguments) + )) + .unwrap(); + + assert_eq!(output.normal_text(), "Let me check.\n"); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].tool_index, 0); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn qwen_xml_does_not_validate_or_normalize_arguments() { + let mut parser = Qwen3XmlToolParser::new(&test_tools()); + let arguments = r#"{"location":"Tokyo",}"#; + let output = parser.parse_complete(&build_tool_call("get_weather", arguments)).unwrap(); + + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn qwen_xml_streaming_emits_argument_deltas() { + let mut parser = Qwen3XmlToolParser::new(&test_tools()); + let chunks = [ + "", + "\n{\"name\": \"get_weather\", \"arguments\": ", + "{\"location\":", + "\"Beijing\"", + "}", + "}\n", + ]; + + let mut output = ToolParserOutput::default(); + let mut observed_arguments = Vec::new(); + for chunk in chunks { + let next = parser.parse_chunk(chunk).unwrap(); + observed_arguments.extend( + next.calls() + .iter() + .filter(|call| call.name.is_none()) + .map(|call| call.arguments.clone()), + ); + output.append(next); + } + output.append(parser.finish().unwrap()); + + assert_eq!(observed_arguments, ["{\"location\":", "\"Beijing\"", "}"]); + assert_eq!( + output.coalesce().calls()[0].arguments, + r#"{"location":"Beijing"}"# + ); + } + + #[test] + fn qwen_xml_streaming_handles_split_markers() { + let input = format!( + "hello {}", + build_tool_call("get_weather", r#"{"location":"Tokyo"}"#) + ); + let chunks = split_by_chars(&input, 5); + let mut parser = Qwen3XmlToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.normal_text(), "hello "); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].arguments, r#"{"location":"Tokyo"}"#); + } + + #[test] + fn qwen_xml_keeps_end_marker_literal_inside_json_string() { + let mut parser = Qwen3XmlToolParser::new(&test_tools()); + let arguments = r#"{"text":"literal inside"}"#; + let output = parser.parse_complete(&build_tool_call("echo", arguments)).unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn qwen_xml_decodes_escaped_function_name() { + let mut parser = Qwen3XmlToolParser::new(&test_tools()); + let output = parser + .parse_complete( + r#" +{"name":"say_\"hi","arguments":{}} +"#, + ) + .unwrap(); + + assert_eq!(output.calls()[0].name.as_deref(), Some("say_\"hi")); + } + + #[test] + fn qwen_xml_requires_newline_after_tool_call_start() { + let mut parser = Qwen3XmlToolParser::new(&test_tools()); + let input = r#"{"name":"get_weather","arguments":{}} +"#; + + let output = parser.parse_complete(input).unwrap(); + + assert_eq!(output.normal_text(), input); + assert!(output.calls().is_empty()); + } + + #[test] + fn qwen_xml_requires_newline_before_tool_call_end() { + let mut parser = Qwen3XmlToolParser::new(&test_tools()); + let error = parser + .parse_complete( + r#" +{"name":"get_weather","arguments":{}}"#, + ) + .unwrap_err(); + + assert!(error.to_report_string().starts_with("tool parser parsing failed:")); + } + + #[test] + fn qwen_xml_streaming_extracts_multiple_tool_calls() { + let input = format!( + "{}{}", + build_tool_call("get_weather", r#"{"location":"Shanghai"}"#), + build_tool_call("add", r#"{"x":1,"y":2}"#), + ); + let chunks = split_by_chars(&input, 7); + let mut parser = Qwen3XmlToolParser::new(&test_tools()); + + let output = collect_stream(&mut parser, &chunks); + + expect![[r#" + ToolParserOutput { + events: [ + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "get_weather", + ), + arguments: "{\"location\":\"Shanghai\"}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 1, + name: Some( + "add", + ), + arguments: "{\"x\":1,\"y\":2}", + }, + ), + ], + } + "#]] + .assert_debug_eq(&output); + } + + #[test] + fn qwen_xml_finish_fails_incomplete_tool_call() { + let mut parser = Qwen3XmlToolParser::new(&test_tools()); + parser + .parse_chunk( + r#" +{"name":"get_weather","arguments":{"location""#, + ) + .unwrap(); + + let error = parser.finish().unwrap_err(); + + expect!["tool parser parsing failed: incomplete Qwen XML tool call"] + .assert_eq(&error.to_report_string()); + } + + #[test] + fn qwen_xml_malformed_field_order_fails_fast() { + let mut parser = Qwen3XmlToolParser::new(&test_tools()); + let error = parser + .parse_chunk( + r#" +{"arguments":{},"name":"get_weather"} +"#, + ) + .unwrap_err(); + + expect![[r#" + tool parser parsing failed: near "{\"arguments\":{},\"name\":\"get_weather\"}\n": invalid Qwen XML + expected `name`"#]] + .assert_eq(&error.to_report_string()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/kimi_k2.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/kimi_k2.rs new file mode 100644 index 0000000000000000000000000000000000000000..6c56f07f67dfcc09cffd5e356013ac149477d484 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/kimi_k2.rs @@ -0,0 +1,602 @@ +use std::collections::BTreeMap; + +use winnow::ascii::{digit1, multispace0 as ws0}; +use winnow::combinator::{alt, eof, repeat, seq}; +use winnow::prelude::*; +use winnow::stream::Partial; +use winnow::token::{literal, rest, take_until, take_while}; + +use super::utils::{JsonObjectScanState, parse_buffered_event, safe_text_len, take_json_object}; +use super::{Result, ToolCallDelta, ToolParser, ToolParserOutput}; +use crate::tool::{StructuralTagModel, Tool}; + +const TOOL_CALLS_START: &str = "<|tool_calls_section_begin|>"; +const TOOL_CALLS_END: &str = "<|tool_calls_section_end|>"; +const TOOL_CALL_START: &str = "<|tool_call_begin|>"; +const TOOL_CALL_END: &str = "<|tool_call_end|>"; +const TOOL_CALL_ARGUMENT_START: &str = "<|tool_call_argument_begin|>"; + +type KimiK2Input<'i> = Partial<&'i str>; + +#[derive(Debug, Clone, PartialEq, Eq)] +enum KimiK2Mode { + Text, + ToolBlock, + Header, + Arguments { json_scan: JsonObjectScanState }, + Done, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum KimiK2Event { + Text { + len: usize, + }, + ToolCallsStart, + ToolCallStart, + ToolCallHeader { + tool_call_id: String, + function_name: String, + function_index: usize, + }, + Arguments { + len: usize, + }, + ToolCallEnd, + ToolCallsEnd, + IgnoredRest, +} + +/// Tool parser for Kimi K2 token-delimited tool calls. +/// +/// Example tool call content: +/// +/// ```text +/// <|tool_calls_section_begin|> +/// <|tool_call_begin|>functions.get_weather:0<|tool_call_argument_begin|>{"location":"NYC"}<|tool_call_end|> +/// <|tool_calls_section_end|> +/// ``` +/// +/// Arguments are already OpenAI-style JSON text, so they are streamed as raw +/// argument deltas without schema conversion or JSON normalization. +pub struct KimiK2ToolParser { + buffer: String, + mode: KimiK2Mode, + active_tool_index: Option, + call_ids: BTreeMap, +} + +impl KimiK2ToolParser { + /// Create a Kimi K2 tool parser. + fn new(_tools: &[Tool]) -> Self { + Self { + buffer: String::new(), + mode: KimiK2Mode::Text, + active_tool_index: None, + call_ids: BTreeMap::new(), + } + } + + /// Apply one parsed Kimi K2 event to parser state and output. + fn apply_event(&mut self, event: KimiK2Event, output: &mut ToolParserOutput) -> Result<()> { + match event { + KimiK2Event::Text { len: consumed_len } => { + output.push_text(&self.buffer[..consumed_len]); + } + KimiK2Event::ToolCallsStart => self.mode = KimiK2Mode::ToolBlock, + KimiK2Event::ToolCallStart => self.mode = KimiK2Mode::Header, + KimiK2Event::ToolCallHeader { + tool_call_id, + function_name, + function_index, + } => { + let tool_index = function_index; + self.active_tool_index = Some(tool_index); + self.mode = KimiK2Mode::Arguments { + json_scan: JsonObjectScanState::default(), + }; + self.call_ids.insert(tool_index, tool_call_id); + output.push_call(ToolCallDelta { + tool_index, + name: Some(function_name), + arguments: String::new(), + }); + } + KimiK2Event::Arguments { len: consumed_len } => { + let Some(tool_index) = self.active_tool_index else { + return Err(parsing_failed!( + "Kimi K2 arguments without an active tool call" + )); + }; + output.push_call(ToolCallDelta { + tool_index, + name: None, + arguments: self.buffer[..consumed_len].to_string(), + }); + } + KimiK2Event::ToolCallEnd => { + self.active_tool_index = None; + self.mode = KimiK2Mode::ToolBlock; + } + KimiK2Event::ToolCallsEnd => { + self.active_tool_index = None; + self.mode = KimiK2Mode::Done; + } + KimiK2Event::IgnoredRest => {} + } + Ok(()) + } + + fn reset(&mut self) -> String { + self.mode = KimiK2Mode::Text; + self.active_tool_index = None; + self.call_ids.clear(); + std::mem::take(&mut self.buffer) + } +} + +impl ToolParser for KimiK2ToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn preserve_special_tokens(&self) -> bool { + true + } + + fn structural_tag_model(&self) -> Option { + Some(StructuralTagModel::Kimi) + } + + fn tool_call_id(&self, tool_index: usize) -> Option<&str> { + self.call_ids.get(&tool_index).map(String::as_str) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.buffer.push_str(chunk); + + while let Some((event, consumed_len)) = parse_buffered_event(&self.buffer, |input| { + parse_next_kimi_k2_event(input, &mut self.mode) + })? { + self.apply_event(event, output)?; + self.buffer.drain(..consumed_len); + } + + Ok(()) + } + + fn finish(&mut self) -> Result { + let mut output = ToolParserOutput::default(); + match &self.mode { + KimiK2Mode::Text => output.push_text(&self.buffer), + KimiK2Mode::ToolBlock | KimiK2Mode::Done => {} + KimiK2Mode::Header | KimiK2Mode::Arguments { .. } => { + return Err(parsing_failed!("incomplete Kimi K2 tool call")); + } + } + let _ = self.reset(); + Ok(output) + } + + fn reset(&mut self) -> String { + KimiK2ToolParser::reset(self) + } +} + +/// Parse a Kimi K2 event for the current parser mode. +fn parse_next_kimi_k2_event( + input: &mut KimiK2Input<'_>, + mode: &mut KimiK2Mode, +) -> ModalResult { + match mode { + KimiK2Mode::Text => parse_text_event(input), + KimiK2Mode::ToolBlock => parse_tool_block_event(input), + KimiK2Mode::Header => tool_call_header_event(input), + KimiK2Mode::Arguments { json_scan } => parse_arguments_event(input, json_scan), + KimiK2Mode::Done => ignored_rest_event(input), + } +} + +/// Parse a text-mode Kimi K2 event. +fn parse_text_event(input: &mut KimiK2Input<'_>) -> ModalResult { + alt((tool_calls_start_event, safe_text_event)).parse_next(input) +} + +/// Parse one event inside the Kimi K2 tool-calls section. +fn parse_tool_block_event(input: &mut KimiK2Input<'_>) -> ModalResult { + alt((tool_calls_end_event, tool_call_start_event)).parse_next(input) +} + +/// Parse one event inside a Kimi K2 tool-call arguments payload. +fn parse_arguments_event( + input: &mut KimiK2Input<'_>, + json_scan: &mut JsonObjectScanState, +) -> ModalResult { + if json_scan.complete() { + tool_call_end_event(input) + } else { + argument_delta_event(input, json_scan) + } +} + +/// Parse a Kimi K2 tool-calls section start marker. +fn tool_calls_start_event(input: &mut KimiK2Input<'_>) -> ModalResult { + literal(TOOL_CALLS_START).value(KimiK2Event::ToolCallsStart).parse_next(input) +} + +/// Parse a Kimi K2 tool-calls section end marker. +fn tool_calls_end_event(input: &mut KimiK2Input<'_>) -> ModalResult { + (ws0, literal(TOOL_CALLS_END)) + .value(KimiK2Event::ToolCallsEnd) + .parse_next(input) +} + +/// Parse a Kimi K2 tool-call start marker. +fn tool_call_start_event(input: &mut KimiK2Input<'_>) -> ModalResult { + (ws0, literal(TOOL_CALL_START)) + .value(KimiK2Event::ToolCallStart) + .parse_next(input) +} + +/// Parse a Kimi K2 tool-call end marker. +fn tool_call_end_event(input: &mut KimiK2Input<'_>) -> ModalResult { + literal(TOOL_CALL_END).value(KimiK2Event::ToolCallEnd).parse_next(input) +} + +/// Parse a Kimi K2 tool-call header before the argument marker. +fn tool_call_header_event(input: &mut KimiK2Input<'_>) -> ModalResult { + let (raw_header, _) = ( + take_until(1.., TOOL_CALL_ARGUMENT_START), + literal(TOOL_CALL_ARGUMENT_START), + ) + .parse_next(input)?; + + let tool_call_id = raw_header.trim().to_string(); + let mut header_input = raw_header; + let (header, _, _) = (tool_header, ws0, eof).parse_next(&mut header_input)?; + + Ok(KimiK2Event::ToolCallHeader { + tool_call_id, + function_name: header.function_name, + function_index: header.function_index, + }) +} + +/// Parse a Kimi K2 raw JSON arguments delta. +fn argument_delta_event( + input: &mut KimiK2Input<'_>, + json_scan: &mut JsonObjectScanState, +) -> ModalResult { + take_json_object(input, json_scan).map(|len| KimiK2Event::Arguments { len }) +} + +/// Parse a safe text run before the next Kimi K2 tool-calls section. +fn safe_text_event(input: &mut KimiK2Input<'_>) -> ModalResult { + safe_text_len(input, TOOL_CALLS_START).map(|len| KimiK2Event::Text { len }) +} + +/// Parse ignored rest after the Kimi K2 tool-calls section ends. +fn ignored_rest_event(input: &mut KimiK2Input<'_>) -> ModalResult { + rest.value(KimiK2Event::IgnoredRest).parse_next(input) +} + +#[derive(Debug, Clone, PartialEq, Eq)] +struct KimiK2ToolHeader { + function_name: String, + function_index: usize, +} + +/// Parse a Kimi K2 tool-call header. +fn tool_header(input: &mut &str) -> ModalResult { + let (function_name, function_index) = seq!( + _: ws0, + _: namespace_prefix, + tool_name_segment, + _: literal(":"), + tool_call_index, + ) + .parse_next(input)?; + + Ok(KimiK2ToolHeader { + function_name: function_name.to_string(), + function_index, + }) +} + +/// Parse Kimi K2 namespace segments before the final tool name. +fn namespace_prefix(input: &mut &str) -> ModalResult<()> { + repeat(0.., namespace_segment).parse_next(input) +} + +/// Parse a Kimi K2 namespace segment. +fn namespace_segment<'i>(input: &mut &'i str) -> ModalResult<&'i str> { + let (segment, _) = (tool_name_segment, literal(".")).parse_next(input)?; + Ok(segment) +} + +/// Parse a Kimi K2 tool name segment. +fn tool_name_segment<'i>(input: &mut &'i str) -> ModalResult<&'i str> { + take_while(1.., |ch: char| { + !ch.is_whitespace() && ch != '<' && ch != ':' && ch != '.' + }) + .parse_next(input) +} + +/// Parse a Kimi K2 tool-call index. +fn tool_call_index(input: &mut &str) -> ModalResult { + digit1.parse_to().parse_next(input) +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + use thiserror_ext::AsReport; + + use super::{ + KimiK2ToolParser, TOOL_CALL_ARGUMENT_START, TOOL_CALL_END, TOOL_CALL_START, TOOL_CALLS_END, + TOOL_CALLS_START, ToolParser, tool_header, + }; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{ToolParserOutput, ToolParserTestExt as _}; + + fn build_tool_call(function_name: &str, index: usize, arguments: &str) -> String { + format!( + "{TOOL_CALL_START}functions.{function_name}:{index}{TOOL_CALL_ARGUMENT_START}{arguments}{TOOL_CALL_END}" + ) + } + + fn build_tool_section(tool_calls: &[String]) -> String { + format!("{TOOL_CALLS_START}{}{TOOL_CALLS_END}", tool_calls.join("")) + } + + #[test] + fn kimi_k2_parse_complete_without_tool_call_keeps_text() { + let mut parser = KimiK2ToolParser::new(&test_tools()); + let output = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn kimi_k2_parse_complete_extracts_raw_json_arguments() { + let mut parser = KimiK2ToolParser::new(&test_tools()); + let arguments = r#"{ "location": "NYC", "days": "3" }"#; + let output = parser + .parse_complete(&format!( + "Checking. {} trailing text", + build_tool_section(&[build_tool_call("get_weather", 0, arguments)]) + )) + .unwrap(); + + assert_eq!(output.normal_text(), "Checking. "); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].tool_index, 0); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn kimi_k2_does_not_validate_or_normalize_arguments() { + let mut parser = KimiK2ToolParser::new(&test_tools()); + let arguments = r#"{"location":"NYC",}"#; + let output = parser + .parse_complete(&build_tool_section(&[build_tool_call( + "get_weather", + 0, + arguments, + )])) + .unwrap(); + + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn kimi_k2_streaming_emits_argument_deltas() { + let mut parser = KimiK2ToolParser::new(&test_tools()); + let chunks = [ + TOOL_CALLS_START, + TOOL_CALL_START, + "functions.get_weather:0", + TOOL_CALL_ARGUMENT_START, + "{\"location\":", + "\"Paris\"", + "}", + TOOL_CALL_END, + TOOL_CALLS_END, + ]; + + let mut output = ToolParserOutput::default(); + let mut observed_arguments = Vec::new(); + for chunk in chunks { + let next = parser.parse_chunk(chunk).unwrap(); + observed_arguments.extend( + next.calls() + .iter() + .filter(|call| call.name.is_none()) + .map(|call| call.arguments.clone()), + ); + output.append(next); + } + output.append(parser.finish().unwrap()); + + assert_eq!(observed_arguments, ["{\"location\":", "\"Paris\"", "}"]); + let output = output.coalesce(); + assert_eq!(output.calls()[0].arguments, r#"{"location":"Paris"}"#); + } + + #[test] + fn kimi_k2_streaming_holds_back_split_markers() { + let mut parser = KimiK2ToolParser::new(&test_tools()); + let chunks = [ + "hello <|tool_calls", + "_section_begin|>", + TOOL_CALL_START, + "functions.get_weather:0", + TOOL_CALL_ARGUMENT_START, + r#"{"location":"NYC"}"#, + "<|tool_call", + "_end|>", + TOOL_CALLS_END, + ]; + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.normal_text(), "hello "); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].arguments, r#"{"location":"NYC"}"#); + } + + #[test] + fn kimi_k2_keeps_end_marker_literal_inside_json_string() { + let mut parser = KimiK2ToolParser::new(&test_tools()); + let arguments = format!(r#"{{"text":"literal {TOOL_CALL_END} inside"}}"#); + let input = build_tool_section(&[build_tool_call("echo", 0, &arguments)]); + + let output = parser.parse_complete(&input).unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].arguments, arguments); + } + + #[test] + fn kimi_k2_streaming_keeps_split_end_marker_literal_inside_json_string() { + let mut parser = KimiK2ToolParser::new(&test_tools()); + let chunks = [ + TOOL_CALLS_START, + TOOL_CALL_START, + "functions.echo:0", + TOOL_CALL_ARGUMENT_START, + r#"{"text":"literal <|tool"#, + r#"_call_end|> inside"}"#, + TOOL_CALL_END, + TOOL_CALLS_END, + ]; + + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.calls().len(), 1); + assert_eq!( + output.calls()[0].arguments, + r#"{"text":"literal <|tool_call_end|> inside"}"# + ); + } + + #[test] + fn kimi_k2_streaming_extracts_multiple_tool_calls() { + let mut parser = KimiK2ToolParser::new(&test_tools()); + let input = build_tool_section(&[ + build_tool_call("get_weather", 0, r#"{"location":"Shanghai"}"#), + build_tool_call("add", 1, r#"{"x":1,"y":2}"#), + ]); + + let chunks = split_by_chars(&input, 7); + let output = collect_stream(&mut parser, &chunks); + + expect![[r#" + ToolParserOutput { + events: [ + ToolCall( + ToolCallDelta { + tool_index: 0, + name: Some( + "get_weather", + ), + arguments: "{\"location\":\"Shanghai\"}", + }, + ), + ToolCall( + ToolCallDelta { + tool_index: 1, + name: Some( + "add", + ), + arguments: "{\"x\":1,\"y\":2}", + }, + ), + ], + } + "#]] + .assert_debug_eq(&output); + } + + #[test] + fn kimi_k2_preserves_model_generated_tool_call_ids() { + let mut parser = KimiK2ToolParser::new(&test_tools()); + let input = build_tool_section(&[ + build_tool_call("get_weather", 0, r#"{"location":"Shanghai"}"#), + build_tool_call("add", 1, r#"{"x":1,"y":2}"#), + ]); + + for chunk in split_by_chars(&input, 7) { + parser.parse_chunk(chunk).unwrap(); + } + + // IDs are available after parsing but before finish(), which calls reset(). + assert_eq!(parser.tool_call_id(0), Some("functions.get_weather:0")); + assert_eq!(parser.tool_call_id(1), Some("functions.add:1")); + parser.finish().unwrap(); + assert_eq!(parser.tool_call_id(0), None); + } + + #[test] + fn kimi_k2_accepts_non_functions_header_prefix() { + let mut parser = KimiK2ToolParser::new(&test_tools()); + let input = format!( + "{TOOL_CALLS_START}{TOOL_CALL_START}api.tools.search:42{TOOL_CALL_ARGUMENT_START}{{}}{TOOL_CALL_END}{TOOL_CALLS_END}" + ); + + let output = parser.parse_chunk(&input).unwrap().coalesce(); + + assert_eq!(output.calls()[0].tool_index, 42); + assert_eq!(parser.tool_call_id(42), Some("api.tools.search:42")); + assert_eq!(output.calls()[0].name.as_deref(), Some("search")); + assert_eq!(output.calls()[0].arguments, "{}"); + } + + #[test] + fn kimi_k2_tool_header_parses_namespace_function_and_index() { + let mut input = "api.tools.search:42"; + let header = tool_header(&mut input).unwrap(); + + expect![[r#" + KimiK2ToolHeader { + function_name: "search", + function_index: 42, + } + "#]] + .assert_debug_eq(&header); + } + + #[test] + fn kimi_k2_finish_fails_incomplete_tool_call() { + let mut parser = KimiK2ToolParser::new(&test_tools()); + parser + .parse_chunk(&format!( + "{TOOL_CALLS_START}{TOOL_CALL_START}functions.get_weather:0{TOOL_CALL_ARGUMENT_START}{{\"location\"" + )) + .unwrap(); + + let error = parser.finish().unwrap_err(); + + expect!["tool parser parsing failed: incomplete Kimi K2 tool call"] + .assert_eq(&error.to_report_string()); + } + + #[test] + fn kimi_k2_malformed_header_fails_fast() { + let mut parser = KimiK2ToolParser::new(&test_tools()); + let input = + format!("{TOOL_CALLS_START}{TOOL_CALL_START}get_weather{TOOL_CALL_ARGUMENT_START}{{}}"); + + let error = parser.parse_chunk(&input).unwrap_err(); + + expect![[ + r#"tool parser parsing failed: near "get_weather<|tool_call_argument_begin|>{}": "# + ]] + .assert_eq(&error.to_report_string()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/minimax_m2.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/minimax_m2.rs new file mode 100644 index 0000000000000000000000000000000000000000..8da086a17c7cba60876c97d8f602fe553bb0a907 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/minimax_m2.rs @@ -0,0 +1,599 @@ +use winnow::ascii::{multispace0 as ws0, multispace1 as ws1}; +use winnow::combinator::{alt, delimited, eof, repeat, seq, terminated}; +use winnow::prelude::*; +use winnow::stream::Partial; +use winnow::token::{literal, rest, take_until}; + +use super::parameters::ToolSchemas; +use super::utils::{MarkerScanState, parse_buffered_event, safe_text_len, take_until_marker}; +use super::{Result, ToolCallDelta, ToolParser, ToolParserOutput}; +use crate::tool::{StructuralTagModel, Tool}; + +const TOOL_CALL_START: &str = ""; +const TOOL_CALL_END: &str = ""; +const INVOKE_START: &str = " = Partial<&'i str>; + +#[derive(Debug, Clone, PartialEq, Eq)] +enum MinimaxM2Mode { + Text, + ToolBlock { invoke_end_scan: MarkerScanState }, + Done, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum MinimaxM2Event { + Text { + len: usize, + }, + ToolBlockStart, + Invoke { + name: String, + raw_params: Vec<(String, String)>, + }, + ToolBlockEnd, + IgnoredRest, +} + +/// Tool parser for MiniMax M2 XML-style tool calls. +/// +/// Example tool call content: +/// +/// ```text +/// +/// Seattle +/// +/// ``` +/// +/// Arguments are emitted only after a full `` block is parsed. +pub struct MinimaxM2ToolParser { + buffer: String, + mode: MinimaxM2Mode, + emitted_tool_count: usize, + tool_parameters: ToolSchemas, +} + +impl MinimaxM2ToolParser { + /// Create a MiniMax M2 tool parser. + fn new(tools: &[Tool]) -> Self { + Self { + buffer: String::new(), + mode: MinimaxM2Mode::Text, + emitted_tool_count: 0, + tool_parameters: ToolSchemas::from_tools(tools), + } + } + + /// Apply one parsed MiniMax M2 event to parser state and output. + fn apply_event(&mut self, event: MinimaxM2Event, output: &mut ToolParserOutput) -> Result<()> { + match event { + MinimaxM2Event::Text { len: consumed_len } => { + output.push_text(&self.buffer[..consumed_len]); + } + MinimaxM2Event::ToolBlockStart => { + self.mode = MinimaxM2Mode::ToolBlock { + invoke_end_scan: MarkerScanState::default(), + }; + } + MinimaxM2Event::Invoke { name, raw_params } => { + let arguments = self.tool_parameters.convert_params_with_schema(&name, raw_params); + let arguments = serde_json::to_string(&arguments) + .map_err(|error| parsing_failed!("failed to serialize arguments: {}", error))?; + + output.push_call(ToolCallDelta { + tool_index: self.emitted_tool_count, + name: Some(name), + arguments, + }); + self.emitted_tool_count += 1; + } + MinimaxM2Event::ToolBlockEnd => self.mode = MinimaxM2Mode::Done, + MinimaxM2Event::IgnoredRest => {} + } + Ok(()) + } + + fn reset(&mut self) -> String { + self.mode = MinimaxM2Mode::Text; + self.emitted_tool_count = 0; + std::mem::take(&mut self.buffer) + } +} + +impl ToolParser for MinimaxM2ToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn structural_tag_model(&self) -> Option { + Some(StructuralTagModel::Minimax) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.buffer.push_str(chunk); + + while let Some((event, consumed_len)) = parse_buffered_event(&self.buffer, |input| { + parse_next_minimax_m2_event(input, &mut self.mode) + })? { + self.apply_event(event, output)?; + self.buffer.drain(..consumed_len); + } + + Ok(()) + } + + fn finish(&mut self) -> Result { + let mut output = ToolParserOutput::default(); + match self.mode { + MinimaxM2Mode::Text => { + output.push_text(&self.buffer); + } + MinimaxM2Mode::ToolBlock { .. } => { + return Err(parsing_failed!("incomplete MiniMax M2 tool call")); + } + MinimaxM2Mode::Done => {} + } + let _ = self.reset(); + Ok(output) + } + + fn reset(&mut self) -> String { + MinimaxM2ToolParser::reset(self) + } +} + +/// Parse a MiniMax M2 event for the current parser mode. +fn parse_next_minimax_m2_event( + input: &mut MinimaxM2Input<'_>, + mode: &mut MinimaxM2Mode, +) -> ModalResult { + match mode { + MinimaxM2Mode::Text => parse_text_event(input), + MinimaxM2Mode::ToolBlock { invoke_end_scan } => { + parse_tool_block_event(input, invoke_end_scan) + } + MinimaxM2Mode::Done => ignored_rest_event(input), + } +} + +/// Parse a text-mode MiniMax M2 event. +fn parse_text_event(input: &mut MinimaxM2Input<'_>) -> ModalResult { + alt((tool_block_start_event, safe_text_event)).parse_next(input) +} + +/// Parse a MiniMax M2 tool-block start marker. +fn tool_block_start_event(input: &mut MinimaxM2Input<'_>) -> ModalResult { + literal(TOOL_CALL_START).value(MinimaxM2Event::ToolBlockStart).parse_next(input) +} + +/// Parse a safe text run before the next MiniMax M2 marker. +fn safe_text_event(input: &mut MinimaxM2Input<'_>) -> ModalResult { + safe_text_len(input, TOOL_CALL_START).map(|len| MinimaxM2Event::Text { len }) +} + +/// Parse one event inside a MiniMax M2 tool block. +fn parse_tool_block_event( + input: &mut MinimaxM2Input<'_>, + invoke_end_scan: &mut MarkerScanState, +) -> ModalResult { + alt((tool_block_end_event, |input: &mut MinimaxM2Input<'_>| { + invoke_event(input, invoke_end_scan) + })) + .parse_next(input) +} + +/// Parse a MiniMax M2 tool-block end marker. +fn tool_block_end_event(input: &mut MinimaxM2Input<'_>) -> ModalResult { + (ws0, literal(TOOL_CALL_END)) + .value(MinimaxM2Event::ToolBlockEnd) + .parse_next(input) +} + +/// Parse a complete MiniMax M2 invoke block. +fn invoke_event( + input: &mut MinimaxM2Input<'_>, + invoke_end_scan: &mut MarkerScanState, +) -> ModalResult { + let (name, body) = seq!( + _: ws0, + _: literal(INVOKE_START), + _: (ws1, literal("name=")), + partial_attr_value, + _: literal(">"), + take_until_marker(INVOKE_END, invoke_end_scan), + _: literal(INVOKE_END), + ) + .parse_next(input)?; + let raw_params = parse_invoke_params(body)?; + + Ok(MinimaxM2Event::Invoke { + name: name.trim().to_string(), + raw_params, + }) +} + +/// Parse all parameter blocks inside a complete MiniMax M2 invoke body. +fn parse_invoke_params(invoke_body: &str) -> ModalResult> { + let mut input = invoke_body; + delimited(ws0, repeat(0.., terminated(parameter, ws0)), eof).parse_next(&mut input) +} + +/// Parse a MiniMax M2 parameter block. +fn parameter(input: &mut &str) -> ModalResult<(String, String)> { + let (name, value) = seq!( + _: literal(PARAMETER_START), + _: (ws1, literal("name=")), + attr_value, + _: literal(">"), + take_until(0.., PARAMETER_END), + _: literal(PARAMETER_END), + ) + .parse_next(input)?; + + Ok((name.trim().to_string(), value.to_string())) +} + +/// Parse a quoted or unquoted XML attribute value. +fn attr_value<'i>(input: &mut &'i str) -> ModalResult<&'i str> { + alt(( + delimited(literal("\""), take_until(1.., "\""), literal("\"")), + delimited(literal("'"), take_until(1.., "'"), literal("'")), + take_until(1.., ">"), + )) + .parse_next(input) +} + +/// Parse a quoted or unquoted XML attribute value from partial streaming input. +fn partial_attr_value<'i>(input: &mut MinimaxM2Input<'i>) -> ModalResult<&'i str> { + alt(( + delimited(literal("\""), take_until(1.., "\""), literal("\"")), + delimited(literal("'"), take_until(1.., "'"), literal("'")), + take_until(1.., ">"), + )) + .parse_next(input) +} + +/// Parse ignored rest after the MiniMax M2 tool block ends. +fn ignored_rest_event(input: &mut MinimaxM2Input<'_>) -> ModalResult { + rest.value(MinimaxM2Event::IgnoredRest).parse_next(input) +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + use serde_json::{Value, json}; + use thiserror_ext::AsReport; + + use super::{MinimaxM2ToolParser, TOOL_CALL_END, TOOL_CALL_START, ToolParser}; + use crate::tool::ToolParserTestExt as _; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + + fn build_tool_block(invokes: &[(&str, Vec<(&str, &str)>)]) -> String { + let invokes = invokes + .iter() + .map(|(function_name, params)| { + let params = params + .iter() + .map(|(name, value)| format!(r#"{value}"#)) + .collect::>() + .join(""); + format!(r#"{params}"#) + }) + .collect::(); + format!("{TOOL_CALL_START}{invokes}{TOOL_CALL_END}") + } + + #[test] + fn minimax_m2_parse_complete_without_tool_call_keeps_text() { + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn minimax_m2_parse_complete_extracts_single_tool_call() { + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_block(&[( + "get_weather", + vec![("city", "Seattle"), ("days", "5")], + )])) + .unwrap(); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "city": "Seattle", "days": 5 }) + ); + } + + #[test] + fn minimax_m2_parse_complete_preserves_prefix_and_ignores_trailing_text() { + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = format!( + "Let me check. {} This trailing text is ignored.", + build_tool_block(&[("get_weather", vec![("city", "Seattle")])]) + ); + let output = parser.parse_complete(&output).unwrap(); + + assert_eq!(output.normal_text(), "Let me check. "); + assert_eq!(output.calls().len(), 1); + } + + #[test] + fn minimax_m2_parse_complete_extracts_multiple_invokes() { + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_block(&[ + ("get_weather", vec![("city", "Seattle")]), + ("get_weather", vec![("city", "NYC")]), + ])) + .unwrap(); + + assert_eq!(output.calls().len(), 2); + assert_eq!(output.calls()[0].tool_index, 0); + assert_eq!(output.calls()[1].tool_index, 1); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "city": "Seattle" }) + ); + assert_eq!( + serde_json::from_str::(&output.calls()[1].arguments).unwrap(), + json!({ "city": "NYC" }) + ); + } + + #[test] + fn minimax_m2_parse_complete_converts_schema_types() { + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_block(&[( + "convert", + vec![ + ("whole", "5.0"), + ("flag", "true"), + ("payload", r#"{"nested":true}"#), + ("items", "[1,2]"), + ("empty", "42"), + ], + )])) + .unwrap(); + + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "whole": 5.0, + "flag": true, + "payload": { "nested": true }, + "items": [1, 2], + "empty": "42", + }) + ); + } + + #[test] + fn minimax_m2_parse_complete_preserves_raw_entities_in_parameter_value() { + // The MiniMax-M2 chat template renders string parameter values RAW (no + // XML escaping), so a value the user wants to be the literal text + // "Tom & Jerry <3" is emitted verbatim. The parser must preserve + // it; xml_unescape currently decodes it, corrupting the bytes. + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_block(&[( + "get_weather", + vec![("city", "Tom & Jerry <3")], + )])) + .unwrap(); + let args: Value = serde_json::from_str(&output.calls()[0].arguments).unwrap(); + assert_eq!(args["city"], json!("Tom & Jerry <3")); + } + + #[test] + fn minimax_m2_parse_complete_preserves_raw_closing_tag_text_in_parameter_value() { + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_block(&[( + "get_weather", + vec![ + ( + "city", + "Seattle </parameter></invoke></minimax:tool_call>", + ), + ("days", "5"), + ], + )])) + .unwrap(); + + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "city": "Seattle </parameter></invoke></minimax:tool_call>", + "days": 5, + }) + ); + } + + #[test] + fn minimax_m2_parse_complete_handles_multiline_parameters() { + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = parser + .parse_complete( + "\ + \ + \nrectangle\n\ + {\"width\":10,\n\"height\":20}\ + 2\ + \ + ", + ) + .unwrap(); + + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "shape": "\nrectangle\n", + "dimensions": { "width": 10, "height": 20 }, + "precision": 2, + }) + ); + } + + #[test] + fn minimax_m2_streaming_extracts_single_tool_call() { + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = collect_stream( + &mut parser, + &[ + "", + r#""#, + r#"Seattle"#, + "", + ], + ); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "city": "Seattle" }) + ); + } + + #[test] + fn minimax_m2_streaming_preserves_prefix_text() { + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = collect_stream( + &mut parser, + &[ + "Let me check. ", + "", + r#"Seattle"#, + "", + ], + ); + + assert_eq!(output.normal_text(), "Let me check. "); + assert_eq!(output.calls().len(), 1); + } + + #[test] + fn minimax_m2_streaming_without_tool_call_emits_text_incrementally() { + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = collect_stream(&mut parser, &["Hello, ", "world!"]); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn minimax_m2_streaming_handles_marker_split_across_chunks() { + let text = build_tool_block(&[("get_weather", vec![("city", "Seattle")])]); + let chunks = split_by_chars(&text, 3); + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.calls().len(), 1); + assert!(output.normal_text().is_empty()); + } + + #[test] + fn minimax_m2_streaming_extracts_multiple_invokes_in_order() { + let text = build_tool_block(&[ + ("get_weather", vec![("city", "Seattle")]), + ("get_weather", vec![("city", "NYC")]), + ]); + let chunks = split_by_chars(&text, 7); + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.calls().len(), 2); + assert_eq!(output.calls()[0].tool_index, 0); + assert_eq!(output.calls()[1].tool_index, 1); + } + + #[test] + fn minimax_m2_streaming_handles_template_whitespace_and_split_parameters() { + let text = concat!( + "I will call the tools.\n", + "\n", + "\n", + "Seattle\n", + "\n", + "\n", + "NYC\n", + "\n", + "", + ); + let chunks = split_by_chars(text, 7); + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let result = collect_stream(&mut parser, &chunks); + + assert_eq!(result.normal_text(), "I will call the tools.\n"); + assert_eq!(result.calls().len(), 2); + assert_eq!(result.calls()[0].tool_index, 0); + assert_eq!(result.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(result.calls()[1].tool_index, 1); + assert_eq!(result.calls()[1].name.as_deref(), Some("get_weather")); + } + + #[test] + fn minimax_m2_streaming_ignores_text_after_tool_block() { + let text = format!( + "{} ignored", + build_tool_block(&[("get_weather", vec![("city", "Seattle")])]) + ); + let chunks = split_by_chars(&text, 5); + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = collect_stream(&mut parser, &chunks); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + } + + #[test] + fn minimax_m2_streaming_does_not_emit_incomplete_tool_call() { + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let output = + parser.parse_chunk(r#""#).unwrap(); + + assert!(output.normal_text().is_empty()); + assert!(output.calls().is_empty()); + } + + #[test] + fn minimax_m2_finish_fails_incomplete_tool_call() { + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + parser.parse_chunk(r#""#).unwrap(); + + assert!(parser.finish().is_err()); + } + + #[test] + fn minimax_m2_finish_fails_after_bare_tool_block_start() { + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + parser.parse_chunk("").unwrap(); + + assert!(parser.finish().is_err()); + } + + #[test] + fn minimax_m2_malformed_tool_call_fails_fast() { + let mut parser = MinimaxM2ToolParser::new(&test_tools()); + let error = parser.parse_chunk("").unwrap_err(); + + expect![[r#"tool parser parsing failed: near "": "#]] + .assert_eq(&error.to_report_string()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/minimax_m3.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/minimax_m3.rs new file mode 100644 index 0000000000000000000000000000000000000000..15260d239e67570b550c28a32fe0c2948baaccac --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/minimax_m3.rs @@ -0,0 +1,931 @@ +use winnow::ascii::{multispace0 as ws0, multispace1 as ws1}; +use winnow::combinator::{alt, delimited, seq}; +use winnow::error::{ContextError, ErrMode}; +use winnow::prelude::*; +use winnow::stream::Partial; +use winnow::token::{literal, rest, take_until}; + +use super::parameters::{ParamElement, ParamInput, ToolSchemas}; +use super::utils::{MarkerScanState, parse_buffered_event, safe_text_len, take_until_marker}; +use super::{Result, ToolCallDelta, ToolParser, ToolParserOutput}; +use crate::tool::Tool; + +const NAMESPACE: &str = "]<]minimax[>["; +const TOOL_CALL_START: &str = "]<]minimax[>["; +const TOOL_CALL_END: &str = "]<]minimax[>["; +const INVOKE_START: &str = "]<]minimax[>[ = Partial<&'i str>; + +#[derive(Debug, Clone, PartialEq, Eq)] +enum MinimaxM3Mode { + Text, + ToolBlock { invoke_end_scan: MarkerScanState }, + Done, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum MinimaxM3Event { + Text { + len: usize, + }, + ToolBlockStart, + Invoke { + name: String, + params: Vec<(String, ParamInput)>, + }, + ToolBlockEnd, + IgnoredRest, +} + +/// Tool parser for MiniMax M3 namespace-delimited XML-style tool calls. +/// +/// Example tool call content with recursive parameters: +/// +/// ```text +/// ]<]minimax[>[ +/// ]<]minimax[>[ +/// ]<]minimax[>[42]<]minimax[>[ +/// ]<]minimax[>[ +/// ]<]minimax[>[Singapore]<]minimax[>[ +/// ]<]minimax[>[018956]<]minimax[>[ +/// ]<]minimax[>[ +/// ]<]minimax[>[ +/// ]<]minimax[>[ +/// ]<]minimax[>[book-001]<]minimax[>[ +/// ]<]minimax[>[2]<]minimax[>[ +/// ]<]minimax[>[ +/// ]<]minimax[>[ +/// ]<]minimax[>[ +/// ]<]minimax[>[ +/// ``` +/// +/// With a schema where `shipping` is an object and `items` is an array of +/// objects, recursive parameter conversion produces: +/// +/// ```json +/// { +/// "user_id": 42, +/// "shipping": { +/// "city": "Singapore", +/// "zip": 18956 +/// }, +/// "items": [ +/// { +/// "sku": "book-001", +/// "qty": 2 +/// } +/// ] +/// } +/// ``` +/// +/// MiniMax M3 emits the namespace marker `]<]minimax[>[` before each structural +/// tag. Arguments are emitted only after a full `` block is parsed. +pub struct MinimaxM3ToolParser { + buffer: String, + mode: MinimaxM3Mode, + emitted_tool_count: usize, + tool_parameters: ToolSchemas, +} + +impl MinimaxM3ToolParser { + /// Create a MiniMax M3 tool parser. + pub fn new(tools: &[Tool]) -> Self { + Self { + buffer: String::new(), + mode: MinimaxM3Mode::Text, + emitted_tool_count: 0, + tool_parameters: ToolSchemas::from_tools(tools), + } + } + + /// Apply one parsed MiniMax M3 event to parser state and output. + fn apply_event(&mut self, event: MinimaxM3Event, output: &mut ToolParserOutput) -> Result<()> { + match event { + MinimaxM3Event::Text { len: consumed_len } => { + output.push_text(&self.buffer[..consumed_len]); + } + MinimaxM3Event::ToolBlockStart => { + self.mode = MinimaxM3Mode::ToolBlock { + invoke_end_scan: MarkerScanState::default(), + }; + } + MinimaxM3Event::Invoke { name, params } => { + let arguments = self.tool_parameters.convert_params_with_schema(&name, params); + let arguments = serde_json::to_string(&arguments) + .map_err(|error| parsing_failed!("failed to serialize arguments: {}", error))?; + + output.push_call(ToolCallDelta { + tool_index: self.emitted_tool_count, + name: Some(name), + arguments, + }); + self.emitted_tool_count += 1; + } + MinimaxM3Event::ToolBlockEnd => self.mode = MinimaxM3Mode::Done, + MinimaxM3Event::IgnoredRest => {} + } + Ok(()) + } +} + +impl ToolParser for MinimaxM3ToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.buffer.push_str(chunk); + + while let Some((event, consumed_len)) = parse_buffered_event(&self.buffer, |input| { + parse_next_minimax_m3_event(input, &mut self.mode) + })? { + self.apply_event(event, output)?; + self.buffer.drain(..consumed_len); + } + + Ok(()) + } + + fn finish(&mut self) -> Result { + let mut output = ToolParserOutput::default(); + match self.mode { + MinimaxM3Mode::Text => { + output.push_text(&self.buffer); + } + MinimaxM3Mode::ToolBlock { .. } => { + if !self.buffer.trim_start().is_empty() { + return Err(parsing_failed!("incomplete MiniMax M3 tool call")); + } + } + MinimaxM3Mode::Done => {} + } + let _ = self.reset(); + Ok(output) + } + + fn reset(&mut self) -> String { + self.mode = MinimaxM3Mode::Text; + self.emitted_tool_count = 0; + std::mem::take(&mut self.buffer) + } +} + +/// Parse a MiniMax M3 event for the current parser mode. +fn parse_next_minimax_m3_event( + input: &mut MinimaxM3Input<'_>, + mode: &mut MinimaxM3Mode, +) -> ModalResult { + match mode { + MinimaxM3Mode::Text => parse_text_event(input), + MinimaxM3Mode::ToolBlock { invoke_end_scan } => { + parse_tool_block_event(input, invoke_end_scan) + } + MinimaxM3Mode::Done => ignored_rest_event(input), + } +} + +/// Parse a text-mode MiniMax M3 event. +fn parse_text_event(input: &mut MinimaxM3Input<'_>) -> ModalResult { + alt((tool_block_start_event, safe_text_event)).parse_next(input) +} + +/// Parse a MiniMax M3 tool-block start marker. +fn tool_block_start_event(input: &mut MinimaxM3Input<'_>) -> ModalResult { + literal(TOOL_CALL_START).value(MinimaxM3Event::ToolBlockStart).parse_next(input) +} + +/// Parse a safe text run before the next MiniMax M3 marker. +fn safe_text_event(input: &mut MinimaxM3Input<'_>) -> ModalResult { + safe_text_len(input, TOOL_CALL_START).map(|len| MinimaxM3Event::Text { len }) +} + +/// Parse one event inside a MiniMax M3 tool block. +fn parse_tool_block_event( + input: &mut MinimaxM3Input<'_>, + invoke_end_scan: &mut MarkerScanState, +) -> ModalResult { + alt((tool_block_end_event, |input: &mut MinimaxM3Input<'_>| { + invoke_event(input, invoke_end_scan) + })) + .parse_next(input) +} + +/// Parse a MiniMax M3 tool-block end marker. +fn tool_block_end_event(input: &mut MinimaxM3Input<'_>) -> ModalResult { + (ws0, literal(TOOL_CALL_END)) + .value(MinimaxM3Event::ToolBlockEnd) + .parse_next(input) +} + +/// Parse a complete MiniMax M3 invoke block. +fn invoke_event( + input: &mut MinimaxM3Input<'_>, + invoke_end_scan: &mut MarkerScanState, +) -> ModalResult { + let (name, body) = seq!( + _: ws0, + _: literal(INVOKE_START), + _: (ws1, literal("name=")), + partial_attr_value, + _: literal(">"), + take_until_marker(INVOKE_END, invoke_end_scan), + _: literal(INVOKE_END), + ) + .parse_next(input)?; + let params = parse_invoke_params(body)?; + + Ok(MinimaxM3Event::Invoke { + name: name.trim().to_string(), + params, + }) +} + +/// Parse all parameter elements inside a complete MiniMax M3 invoke body. +fn parse_invoke_params(invoke_body: &str) -> ModalResult> { + let mut input = invoke_body; + let mut elements = Vec::new(); + + loop { + let _ = ws0.parse_next(&mut input)?; + if input.is_empty() { + break; + } + if input.starts_with(ELEMENT_START) { + elements.push(parameter_element(&mut input)?); + continue; + } + if input.starts_with(NAMESPACE) { + return malformed(); + } + // Be tolerant: ordinary text at an invokeparameter boundary ends this invoke. + // Keep parsed parameters and drop the remaining invoke body. + break; + } + + Ok(elements.into_iter().map(|element| (element.name, element.value)).collect()) +} + +/// Parse a MiniMax M3 parameter element. +fn parameter_element(input: &mut &str) -> ModalResult { + let name = open_element_tag(input)?.to_string(); + let value = element_body(input, &name)?; + close_element_tag(input, &name)?; + Ok(ParamElement { name, value }) +} + +/// Parse a MiniMax M3 opening element tag. +fn open_element_tag<'i>(input: &mut &'i str) -> ModalResult<&'i str> { + let name = seq!( + _: literal(ELEMENT_START), + take_until(1.., ">"), + _: literal(">"), + ) + .parse_next(input)?; + + let name = name.0; + if name.starts_with('/') || name.trim().is_empty() { + return malformed(); + } + + Ok(name) +} + +/// Parse a MiniMax M3 closing element tag. +fn close_element_tag(input: &mut &str, name: &str) -> ModalResult<()> { + literal(ELEMENT_END_START).void().parse_next(input)?; + literal(name).void().parse_next(input)?; + literal(">").void().parse_next(input) +} + +/// Parse the body of one MiniMax M3 element. +fn element_body(input: &mut &str, closing_name: &str) -> ModalResult { + let close_tag = format!("{ELEMENT_END_START}{closing_name}>"); + let mut text = String::new(); + let mut elements = Vec::new(); + + loop { + text.push_str(text_until_namespace(input)?); + + if input.starts_with(&close_tag) { + // Close tag reached, end of element body. + break; + } + if input.starts_with(ELEMENT_START) { + // Child element start reached, parse child element recursively. + elements.push(parameter_element(input)?); + continue; + } + if input.starts_with(NAMESPACE) { + // Unexpected namespace marker. + return malformed(); + } + } + + if elements.is_empty() { + Ok(ParamInput::Text(text)) + } else { + if !text.trim().is_empty() { + push_mixed_text_element(&mut elements, text); + } + Ok(ParamInput::Elements(elements)) + } +} + +/// Parse text until the next MiniMax M3 namespace marker. +fn text_until_namespace<'i>(input: &mut &'i str) -> ModalResult<&'i str> { + take_until(0.., NAMESPACE).parse_next(input) +} + +/// Preserve mixed text content under a reserved object field. +/// +/// By default, the field name is `$text`, but if that collides with an existing +/// child element name, prepend `$` until there is no collision. +fn push_mixed_text_element(elements: &mut Vec, text: String) { + let mut name = MIXED_TEXT_FIELD.to_string(); + while elements.iter().any(|element| element.name == name) { + name.insert(0, '$'); + } + elements.push(ParamElement { + name, + value: ParamInput::Text(text), + }); +} + +/// Parse a quoted or unquoted XML attribute value from partial streaming input. +fn partial_attr_value<'i>(input: &mut MinimaxM3Input<'i>) -> ModalResult<&'i str> { + alt(( + delimited(literal("\""), take_until(1.., "\""), literal("\"")), + delimited(literal("'"), take_until(1.., "'"), literal("'")), + take_until(1.., ">"), + )) + .parse_next(input) +} + +/// Parse ignored rest after the MiniMax M3 tool block ends. +fn ignored_rest_event(input: &mut MinimaxM3Input<'_>) -> ModalResult { + rest.value(MinimaxM3Event::IgnoredRest).parse_next(input) +} + +fn malformed() -> ModalResult { + Err(ErrMode::Cut(ContextError::new())) +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + use serde_json::{Value, json}; + use thiserror_ext::AsReport; + + use super::{ + ELEMENT_END_START, ELEMENT_START, INVOKE_END, INVOKE_START, MinimaxM3ToolParser, + TOOL_CALL_END, TOOL_CALL_START, ToolParser, + }; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{Tool, ToolParserEvent, ToolParserTestExt as _}; + + fn element(name: &str, body: &str) -> String { + format!("{ELEMENT_START}{name}>{body}{ELEMENT_END_START}{name}>") + } + + fn invoke(function_name: &str, body: &str) -> String { + format!("{INVOKE_START} name=\"{function_name}\">{body}{INVOKE_END}") + } + + fn build_tool_block(invokes: &[(&str, String)]) -> String { + let invokes = invokes + .iter() + .map(|(function_name, body)| invoke(function_name, body)) + .collect::>() + .join("\n"); + format!("{TOOL_CALL_START}\n{invokes}\n{TOOL_CALL_END}") + } + + fn m3_test_tools() -> Vec { + let mut tools = test_tools(); + tools.push(Tool { + name: "create_order".to_string(), + description: None, + parameters: json!({ + "type": "object", + "properties": { + "user_id": { "type": "integer" }, + "urgent": { "type": "boolean" }, + "note": { "type": "string" }, + "shipping": { + "type": "object", + "properties": { + "city": { "type": "string" }, + "zip": { "type": "integer" } + } + }, + "items": { + "type": "array", + "items": { + "type": "object", + "properties": { + "sku": { "type": "string" }, + "qty": { "type": "integer" } + } + } + }, + "metadata": { + "type": "object", + "additionalProperties": { "type": "integer" } + }, + "duplicate_demo": { + "type": "object", + "properties": { + "tag": { "type": "string" } + } + }, + "schema_mismatch_array": { + "type": "array", + "items": { "type": "integer" } + } + } + }), + strict: None, + }); + tools + } + + fn order_arguments() -> String { + let shipping = element( + "shipping", + &format!( + "{}{}", + element("city", "Singapore"), + element("zip", "018956") + ), + ); + let first_item = element( + "item", + &format!("{}{}", element("sku", "book-001"), element("qty", "2")), + ); + let second_item = element( + "item", + &format!("{}{}", element("sku", "pen-007"), element("qty", "5")), + ); + let items = element("items", &format!("{first_item}{second_item}")); + let metadata = element( + "metadata", + &format!("{}{}", element("score", "42"), element("rank", "7")), + ); + let duplicate_demo = element( + "duplicate_demo", + &format!("{}{}", element("tag", "a"), element("tag", "b")), + ); + let schema_mismatch_array = element( + "schema_mismatch_array", + &format!("{}{}", element("x", "1"), element("x", "2")), + ); + + [ + element("user_id", "42"), + element("urgent", "true"), + element("note", "Please leave at front desk."), + shipping, + items, + metadata, + duplicate_demo, + schema_mismatch_array, + element( + "unknown_struct", + &format!("{}{}", element("a", "1"), element("a", "2")), + ), + ] + .join("") + } + + #[test] + fn minimax_m3_parse_complete_without_tool_call_keeps_text() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn minimax_m3_parse_complete_extracts_single_tool_call() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = parser + .parse_complete(&build_tool_block(&[( + "get_weather", + format!("{}{}", element("city", "Seattle"), element("days", "5")), + )])) + .unwrap(); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "city": "Seattle", "days": 5 }) + ); + } + + #[test] + fn minimax_m3_parse_complete_preserves_prefix_and_ignores_trailing_text() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = format!( + "Let me check. {} This trailing text is ignored.", + build_tool_block(&[("get_weather", element("city", "Seattle"))]) + ); + let output = parser.parse_complete(&output).unwrap(); + + assert_eq!(output.normal_text(), "Let me check. "); + assert_eq!(output.calls().len(), 1); + } + + #[test] + fn minimax_m3_parse_complete_extracts_multiple_invokes() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = parser + .parse_complete(&build_tool_block(&[ + ("get_weather", element("city", "Seattle")), + ("get_weather", element("city", "NYC")), + ])) + .unwrap(); + + assert_eq!(output.calls().len(), 2); + assert_eq!(output.calls()[0].tool_index, 0); + assert_eq!(output.calls()[1].tool_index, 1); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "city": "Seattle" }) + ); + assert_eq!( + serde_json::from_str::(&output.calls()[1].arguments).unwrap(), + json!({ "city": "NYC" }) + ); + } + + #[test] + fn minimax_m3_invoke_body_junk_drops_rest_of_invoke() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = parser + .parse_complete(&build_tool_block(&[( + "get_weather", + [ + element("city", "Seattle"), + "I need to use the city above.".to_string(), + element("days", "5"), + ] + .join(""), + )])) + .unwrap(); + + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "city": "Seattle" }) + ); + } + + #[test] + fn minimax_m3_parse_complete_converts_schema_types() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = parser + .parse_complete(&build_tool_block(&[( + "convert", + [ + element("whole", "5.0"), + element("flag", "true"), + element("payload", r#"{"nested":true}"#), + element("items", "[1,2]"), + element("empty", "42"), + ] + .join(""), + )])) + .unwrap(); + + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "whole": 5.0, + "flag": true, + "payload": { "nested": true }, + "items": [1, 2], + "empty": "42", + }) + ); + } + + #[test] + fn minimax_m3_parse_complete_converts_nested_arguments() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = parser + .parse_complete(&build_tool_block(&[("create_order", order_arguments())])) + .unwrap(); + + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "user_id": 42, + "urgent": true, + "note": "Please leave at front desk.", + "shipping": { + "city": "Singapore", + "zip": 18956 + }, + "items": [ + { + "sku": "book-001", + "qty": 2 + }, + { + "sku": "pen-007", + "qty": 5 + } + ], + "metadata": { + "score": 42, + "rank": 7 + }, + "duplicate_demo": { + "tag": ["a", "b"] + }, + "schema_mismatch_array": [1, 2], + "unknown_struct": { + "a": ["1", "2"] + } + }) + ); + } + + #[test] + fn minimax_m3_parse_complete_handles_multiline_leaf_parameters() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = parser + .parse_complete(&build_tool_block(&[( + "calculate_area", + [ + element("shape", "\nrectangle\n"), + element("dimensions", r#"{"width":10,"height":20}"#), + element("precision", "2"), + ] + .join(""), + )])) + .unwrap(); + + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "shape": "\nrectangle\n", + "dimensions": { "width": 10, "height": 20 }, + "precision": 2, + }) + ); + } + + #[test] + fn minimax_m3_streaming_extracts_single_tool_call() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = collect_stream( + &mut parser, + &[ + TOOL_CALL_START, + &invoke("get_weather", &element("city", "Seattle")), + TOOL_CALL_END, + ], + ); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "city": "Seattle" }) + ); + } + + #[test] + fn minimax_m3_streaming_preserves_prefix_text() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = collect_stream( + &mut parser, + &[ + "Let me check. ", + TOOL_CALL_START, + &invoke("get_weather", &element("city", "Seattle")), + TOOL_CALL_END, + ], + ); + + assert_eq!(output.normal_text(), "Let me check. "); + assert_eq!(output.calls().len(), 1); + } + + #[test] + fn minimax_m3_streaming_preserves_ordered_events() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = collect_stream( + &mut parser, + &[ + "Let me check. ", + TOOL_CALL_START, + &invoke("get_weather", &element("city", "Seattle")), + TOOL_CALL_END, + ], + ); + + assert_eq!(output.events.len(), 2); + assert_eq!( + output.events[0], + ToolParserEvent::Text("Let me check. ".to_string()) + ); + let ToolParserEvent::ToolCall(call) = &output.events[1] else { + panic!("expected tool-call event"); + }; + assert_eq!(call.name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&call.arguments).unwrap(), + json!({ "city": "Seattle" }) + ); + } + + #[test] + fn minimax_m3_streaming_without_tool_call_emits_text_incrementally() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = collect_stream(&mut parser, &["Hello, ", "world!"]); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn minimax_m3_streaming_handles_marker_split_across_chunks() { + let text = build_tool_block(&[("get_weather", element("city", "Seattle"))]); + let chunks = split_by_chars(&text, 3); + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.calls().len(), 1); + assert!(output.normal_text().is_empty()); + } + + #[test] + fn minimax_m3_streaming_extracts_multiple_invokes_in_order() { + let text = build_tool_block(&[ + ("get_weather", element("city", "Seattle")), + ("get_weather", element("city", "NYC")), + ]); + let chunks = split_by_chars(&text, 7); + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.calls().len(), 2); + assert_eq!(output.calls()[0].tool_index, 0); + assert_eq!(output.calls()[1].tool_index, 1); + } + + #[test] + fn minimax_m3_streaming_does_not_emit_incomplete_tool_call() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = parser + .parse_chunk(&format!( + "{TOOL_CALL_START}{INVOKE_START} name=\"get_weather\">" + )) + .unwrap(); + + assert!(output.normal_text().is_empty()); + assert!(output.calls().is_empty()); + } + + #[test] + fn minimax_m3_streaming_ignores_text_after_tool_block() { + let text = format!( + "{} ignored", + build_tool_block(&[("get_weather", element("city", "Seattle"))]) + ); + let chunks = split_by_chars(&text, 5); + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = collect_stream(&mut parser, &chunks); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + } + + #[test] + fn minimax_m3_finish_fails_incomplete_tool_call() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + parser + .parse_chunk(&format!( + "{TOOL_CALL_START}{INVOKE_START} name=\"get_weather\">" + )) + .unwrap(); + + assert!(parser.finish().is_err()); + } + + #[test] + fn minimax_m3_finish_recovers_after_bare_tool_block_start() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + parser.parse_chunk(TOOL_CALL_START).unwrap(); + + let output = parser.finish().unwrap(); + assert!(output.normal_text().is_empty()); + assert!(output.calls().is_empty()); + } + + #[test] + fn minimax_m3_finish_recovers_completed_invoke_with_whitespace_tail() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let output = parser + .parse_complete(&format!( + "{}\n{}\n \n", + TOOL_CALL_START, + invoke("get_weather", &element("city", "Seattle")) + )) + .unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "city": "Seattle" }) + ); + } + + #[test] + fn minimax_m3_finish_fails_partial_outer_end_marker() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + parser + .parse_chunk(&format!( + "{}\n{}\n{}", + TOOL_CALL_START, + invoke("get_weather", &element("city", "Seattle")), + &TOOL_CALL_END[..3] + )) + .unwrap(); + + assert!(parser.finish().is_err()); + } + + #[test] + fn minimax_m3_malformed_tool_call_fails_fast() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let error = parser + .parse_chunk(&format!( + "{TOOL_CALL_START}{ELEMENT_START}bad>{TOOL_CALL_END}" + )) + .unwrap_err(); + + expect![[ + r#"tool parser parsing failed: near "]<]minimax[>[]<]minimax[>[": "# + ]] + .assert_eq(&error.to_report_string()); + } + + #[test] + fn minimax_m3_mixed_content_is_preserved_as_text_field() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let body = element( + "payload", + &format!("text before {} text after", element("child", "value")), + ); + let output = parser.parse_complete(&build_tool_block(&[("convert", body)])).unwrap(); + + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "payload": { + "child": "value", + "$text": "text before text after" + } + }) + ); + } + + #[test] + fn minimax_m3_mixed_text_field_avoids_child_name_collision() { + let mut parser = MinimaxM3ToolParser::new(&m3_test_tools()); + let body = element( + "payload", + &format!( + "text{}{}", + element("$text", "child text"), + element("child", "value") + ), + ); + let output = parser.parse_complete(&build_tool_block(&[("convert", body)])).unwrap(); + + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "payload": { + "$text": "child text", + "$$text": "text", + "child": "value" + } + }) + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/mod.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..e072591669ccbe062946f74e8fe976fc3fb4cb87 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/mod.rs @@ -0,0 +1,249 @@ +//! Streaming tool parsers for chat completions. + +#[macro_use] +pub(crate) mod error; +mod deepseek_dsml; +pub(crate) mod deepseek_json; +mod glm_xml; +mod hy_v3; +mod json; +mod kimi_k2; +mod minimax_m2; +mod minimax_m3; +mod parameters; +mod qwen_coder; +#[cfg(any(test, feature = "test-util"))] +pub mod test_utils; +use std::collections::{BTreeMap, btree_map}; + +pub use deepseek_dsml::{DeepSeekV4ToolParser, DeepSeekV32ToolParser}; +pub use deepseek_json::{DeepSeekV3ToolParser, DeepSeekV31ToolParser}; +pub use error::{Result, ToolParserError}; +pub use glm_xml::{Glm45MoeToolParser, Glm47MoeToolParser}; +pub use hy_v3::HyV3ToolParser; +pub use json::{ + Granite4ToolParser, HermesToolParser, Internlm2ToolParser, Llama3JsonToolParser, + MistralToolParser, Phi4MiniJsonToolParser, Qwen3XmlToolParser, +}; +pub use kimi_k2::KimiK2ToolParser; +pub use minimax_m2::MinimaxM2ToolParser; +pub use minimax_m3::MinimaxM3ToolParser; +pub use qwen_coder::Qwen3CoderToolParser; +use serde::{Deserialize, Serialize}; +use serde_json::Value; +pub use xgrammar_structural_tag::Model as StructuralTagModel; + +use crate::utils; + +/// One function-style tool made available to the model. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct Tool { + pub name: String, + pub description: Option, + pub parameters: Value, + pub strict: Option, +} + +/// One tool-call update emitted while parsing assistant text. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ToolCallDelta { + /// Stable parser-local tool index for this call within one assistant turn. + pub tool_index: usize, + /// Function name, present on the first update for one tool call. + pub name: Option, + /// Arguments text contributed by this update. + pub arguments: String, +} + +/// One ordered event emitted while parsing assistant text. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum ToolParserEvent { + /// Plain assistant text that is not part of any tool call. + Text(String), + /// A tool-call update extracted from assistant text. + ToolCall(ToolCallDelta), +} + +/// Result of advancing tool parsing with one assistant-text input. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct ToolParserOutput { + /// Ordered parser events committed by this input. + pub events: Vec, +} + +impl ToolParserOutput { + /// Append one visible text event if `text` is non-empty. + pub fn push_text(&mut self, text: impl AsRef + Into) { + if text.as_ref().is_empty() { + return; + } + if let Some(ToolParserEvent::Text(last_text)) = self.events.last_mut() { + last_text.push_str(text.as_ref()); + return; + } + self.events.push(ToolParserEvent::Text(text.into())); + } + + /// Append one tool-call update event. + pub fn push_call(&mut self, call: ToolCallDelta) { + self.events.push(ToolParserEvent::ToolCall(call)); + } + + /// Return all plain assistant text committed by this output. + /// + /// Texts before and after tool calls will be concatenated into a single string. To preserve + /// the original order of the text and tool-call events, directly access `events` instead. + pub fn normal_text(&self) -> String { + self.events + .iter() + .filter_map(|event| match event { + ToolParserEvent::Text(text) => Some(text.as_str()), + ToolParserEvent::ToolCall(_) => None, + }) + .collect() + } + + /// Return all tool-call updates committed by this output. + pub fn calls(&self) -> Vec<&ToolCallDelta> { + self.events + .iter() + .filter_map(|event| match event { + ToolParserEvent::Text(_) => None, + ToolParserEvent::ToolCall(call) => Some(call), + }) + .collect() + } + + /// Append another parser output onto this one. + /// + /// Note that this keeps events exactly as they arrive. Call `coalesce()` + /// after if final text and tool-call fragments should be flattened. + pub fn append(&mut self, other: Self) { + for event in other.events { + match event { + ToolParserEvent::Text(text) => self.push_text(text), + ToolParserEvent::ToolCall(call) => self.push_call(call), + } + } + } + + /// Flatten text and merge deltas for the same tool call. + /// + /// All text events are concatenated into one leading text event. Tool-call + /// events follow that text event in first-seen tool index order, with + /// argument fragments for the same tool call concatenated together. + /// + /// This is primarily used by the default `parse_complete()` implementation, + /// which delegates through the incremental parser lifecycle and then + /// needs to collapse streaming-style argument fragments into one final + /// tool call. + pub fn coalesce(self) -> Self { + let mut merged = BTreeMap::::new(); + let mut order = Vec::new(); + let normal_text = self.normal_text(); + + for call in self.calls() { + match merged.entry(call.tool_index) { + btree_map::Entry::Vacant(entry) => { + order.push(call.tool_index); + entry.insert(call.clone()); + } + btree_map::Entry::Occupied(mut entry) => { + let existing = entry.get_mut(); + if existing.name.is_none() { + existing.name = call.name.clone(); + } + existing.arguments.push_str(&call.arguments); + } + } + } + + let mut output = Self::default(); + output.push_text(normal_text); + for call in order.into_iter().filter_map(|tool_index| merged.remove(&tool_index)) { + output.push_call(call); + } + output + } +} + +/// Incremental parser that extracts tool calls from assistant output. +pub trait ToolParser: Send { + /// Construct a boxed parser instance for one request stream. + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static; + + /// Return whether decoded output must preserve tokenizer special tokens. + /// + /// Some model families emit tool-call sentinels as special tokens. Those + /// parsers need `skip_special_tokens = false` while parsing is enabled. + fn preserve_special_tokens(&self) -> bool { + false + } + + /// Return the xgrammar structural-tag model used for strict tool calling. + fn structural_tag_model(&self) -> Option { + None + } + + /// Return the parser-provided ID for a tool call by index, if the model + /// emitted one. + fn tool_call_id(&self, _tool_index: usize) -> Option<&str> { + None + } + + /// Feed one decoded text delta into the parser, appending committed output + /// into `output`. + /// + /// If this returns an error, any output already appended to `output` + /// remains committed parser output. The parser must keep its uncommitted + /// buffer intact so callers may recover it with `reset()`. + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()>; + + /// Flush any buffered partial state at end of stream. + /// + /// This operation is atomic: on error no partial output is returned and the + /// parser's buffered state is left intact. + fn finish(&mut self) -> Result; + + /// Clear parser state and return currently uncommitted buffered text. + /// + /// Callers may use this to recover any text that failed to parse after an error + /// and output it as normal text. + fn reset(&mut self) -> String; +} + +/// Extension methods for easily testing `ToolParser` implementations. +/// +/// These helpers do not handle partial parsing or error recovery, so they are +/// not intended for use in production code paths. +#[cfg(any(test, feature = "test-util"))] +#[easy_ext::ext(ToolParserTestExt)] +impl T { + /// Feed one decoded text delta and return only if the whole chunk parses. + /// + /// If parsing fails, partial committed output is discarded by this helper. + /// Prefer `parse_into` for more fine-grained control in error recovery. + pub fn parse_chunk(&mut self, chunk: &str) -> Result { + let mut output = ToolParserOutput::default(); + self.parse_into(chunk, &mut output)?; + Ok(output) + } + + /// Parse complete tool calls from final output. + /// + /// This default implementation reuses the incremental parser lifecycle by + /// feeding the full output through `parse_chunk()` and then calling `finish()`. + /// + /// If parsing fails, partial committed output is discarded by this helper. + /// Prefer `parse_into` for more fine-grained control in error recovery. + pub fn parse_complete(&mut self, text: &str) -> Result { + let mut output = self.parse_chunk(text)?; + output.append(self.finish()?); + Ok(output.coalesce()) + } +} + +#[cfg(test)] +mod tests; diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/parameters.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/parameters.rs new file mode 100644 index 0000000000000000000000000000000000000000..f4612a597017c13e28948fa2b669bfbd5a1336dc --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/parameters.rs @@ -0,0 +1,914 @@ +use std::collections::BTreeMap; + +use serde_json::{Map, Number, Value}; + +use crate::tool::Tool; + +/// Normalized parameter schemas for all tools in one request. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub(super) struct ToolSchemas { + tools: BTreeMap, +} + +/// Normalized parameter schema for one tool. +/// +/// This is a minimal subset of JSON Schema with some normalization heuristics +/// to support common schema patterns and upstream schema variations, focused on +/// coercing raw string parameter values into more specific JSON types for +/// downstream tool call execution. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub(super) struct ToolSchema { + params: BTreeMap, +} + +/// Parameter input for schema-aware conversion. +/// +/// It can be either a raw text string, or a structured input with named child elements. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) enum ParamInput { + Text(String), + #[allow(dead_code)] + Elements(Vec), +} + +impl From for ParamInput { + fn from(value: String) -> Self { + Self::Text(value) + } +} + +/// One named structured parameter child. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) struct ParamElement { + pub name: String, + pub value: ParamInput, +} + +/// Normalized JSON parameter type used for raw string coercion. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) enum JsonParamType { + String, + Integer, + Number, + Boolean, + Object { + properties: BTreeMap, + additional_properties: Option>, + }, + Array { + items: Option>, + }, + Null, + OneOf(Vec), +} + +impl ToolSchemas { + /// Normalize OpenAI-style tool parameter JSON schemas for one request. + pub(super) fn from_tools(tools: &[Tool]) -> Self { + let tools = tools + .iter() + .map(|tool| (tool.name.clone(), ToolSchema::from_schema(&tool.parameters))) + .collect(); + + Self { tools } + } + + /// Convert parameter values for one named tool. + /// + /// Unknown tool names use an empty schema, so all parameters fall back to + /// strings or object-like JSON for structured inputs. + pub(super) fn convert_params_with_schema

( + &self, + function_name: &str, + params: Vec<(String, P)>, + ) -> Map + where + P: Into, + { + let tool_schema = self.tools.get(function_name).unwrap_or(ToolSchema::empty()); + let mut converted = Map::with_capacity(params.len()); + for (name, value) in params { + let value = tool_schema.convert(&name, value.into()); + converted.insert(name, value); + } + converted + } + + /// Convert one parameter value for one named tool. + pub(super) fn convert_param_with_schema

( + &self, + function_name: &str, + name: &str, + value: P, + ) -> Value + where + P: Into, + { + let tool_schema = self.tools.get(function_name).unwrap_or(ToolSchema::empty()); + tool_schema.convert(name, value.into()) + } +} + +impl ToolSchema { + /// Return an empty schema with no parameter information, which causes all + /// parameters to be treated as strings. + const fn empty() -> &'static Self { + static EMPTY: ToolSchema = ToolSchema { + params: BTreeMap::new(), + }; + &EMPTY + } + + /// Normalize an OpenAI-style tool parameters JSON schema. + fn from_schema(parameters: &Value) -> Self { + let Some(properties) = parameters.get("properties").and_then(Value::as_object) else { + return Self::default(); + }; + + let params = properties + .iter() + .filter_map(|(name, schema)| { + JsonParamType::from_schema(schema).map(|param_type| (name.clone(), param_type)) + }) + .collect(); + + Self { params } + } + + /// Convert one parameter value using its normalized schema type. + /// + /// If the parameter name is unknown, or we don't have a schema for it, or + /// the value fails to convert, this falls back to returning the raw + /// string as a JSON string value, or object-like JSON for structured input. + fn convert(&self, name: &str, input: ParamInput) -> Value { + convert_with_optional_schema(self.params.get(name), &input) + } +} + +impl JsonParamType { + /// Normalize one parameter property schema. + fn from_schema(schema: &Value) -> Option { + let schema = schema.as_object()?; + + if let Some(type_value) = schema.get("type") { + return Self::from_type_value(type_value, schema); + } + + if let Some(composite) = schema.get("anyOf").or_else(|| schema.get("oneOf")) { + let param_type = composite + .as_array() + .map(|schemas| schemas.iter().filter_map(Self::from_schema).collect::>()) + .filter(|types| !types.is_empty()) + .map(Self::one_of) + .unwrap_or_else(|| Self::object_from_schema(Some(schema))); + return Some(param_type); + } + + // Typically, these types are already handled by checking the "type" field, but + // we can also infer them from their characteristic fields if "type" is missing. + if let Some(values) = schema.get("enum").and_then(Value::as_array) { + // Enum values are treated as strings, except that a `null` member + // makes the parameter nullable (mirrors Python's enum type + // inference), so a literal "null" coerces to JSON null. + if values.iter().any(Value::is_null) { + return Some(Self::one_of(vec![Self::String, Self::Null])); + } + return Some(Self::String); + } + if schema.contains_key("items") { + return Some(Self::array_from_schema(Some(schema))); + } + if schema.contains_key("properties") || schema.contains_key("additionalProperties") { + return Some(Self::object_from_schema(Some(schema))); + } + + None + } + + /// Normalize a JSON schema `type` value. + fn from_type_value(type_value: &Value, schema: &Map) -> Option { + match type_value { + Value::String(kind) => Self::from_type_name(kind, Some(schema)), + Value::Array(kinds) => { + let types = kinds + .iter() + .filter_map(Value::as_str) + .filter_map(|kind| Self::from_type_name(kind, Some(schema))) + .collect::>(); + if types.is_empty() { + None + } else { + Some(Self::one_of(types)) + } + } + _ => None, + } + } + + /// Normalize one JSON schema type name. + fn from_type_name(kind: &str, schema: Option<&Map>) -> Option { + let kind = kind.trim().to_ascii_lowercase(); + match kind.as_str() { + "string" | "str" | "text" | "varchar" | "char" | "enum" => Some(Self::String), + "integer" | "int" => Some(Self::Integer), + "number" | "float" | "double" => Some(Self::Number), + "boolean" | "bool" | "binary" => Some(Self::Boolean), + "object" | "dict" | "map" => Some(Self::object_from_schema(schema)), + "array" | "arr" | "list" | "sequence" => Some(Self::array_from_schema(schema)), + "null" => Some(Self::Null), + _ if kind.starts_with("int") + || kind.starts_with("uint") + || kind.starts_with("long") + || kind.starts_with("short") + || kind.starts_with("unsigned") => + { + Some(Self::Integer) + } + _ if kind.starts_with("num") || kind.starts_with("float") => Some(Self::Number), + _ if kind.starts_with("dict") => Some(Self::object_from_schema(schema)), + _ if kind.starts_with("list") => Some(Self::array_from_schema(schema)), + _ => None, + } + } + + /// Normalize object schema fields. + fn object_from_schema(schema: Option<&Map>) -> Self { + let properties = schema + .and_then(|schema| schema.get("properties")) + .and_then(Value::as_object) + .map(|properties| { + properties + .iter() + .filter_map(|(name, schema)| { + Self::from_schema(schema).map(|param_type| (name.clone(), param_type)) + }) + .collect() + }) + .unwrap_or_default(); + + let additional_properties = + schema.and_then(|schema| schema.get("additionalProperties")).and_then(|schema| { + if schema.is_object() { + Self::from_schema(schema).map(Box::new) + } else { + None + } + }); + + Self::Object { + properties, + additional_properties, + } + } + + /// Normalize array schema fields. + fn array_from_schema(schema: Option<&Map>) -> Self { + let items = schema + .and_then(|schema| schema.get("items")) + .and_then(Self::from_schema) + .map(Box::new); + + Self::Array { items } + } + + /// Collapse a candidate type list into one normalized type. + fn one_of(mut types: Vec) -> Self { + if types.len() == 1 { + types.remove(0) + } else { + Self::OneOf(types) + } + } +} + +/// Convert one parameter input to a normalized JSON value. +fn convert_with_optional_schema(param_type: Option<&JsonParamType>, input: &ParamInput) -> Value { + // Coerce the literal text `null` to JSON null, except for `string`-typed + // params, where it must stay the string "null": a model emitting the literal + // text "null" for a string field means the string, not a missing value. + if let ParamInput::Text(value) = input + && value.eq_ignore_ascii_case("null") + && param_type != Some(&JsonParamType::String) + { + return Value::Null; + } + + // If we have a schema, try to convert the value using it. + if let Some(param_type) = param_type + && let Some(value) = try_convert_value(param_type, input) + { + return value; + } + // We don't have a schema, or conversion failed, use fallback logic. + match input { + ParamInput::Text(value) => Value::String(value.clone()), + ParamInput::Elements(elements) => { + // Convert structured input to object without a schema. + Value::Object(convert_elements_to_object(elements, &BTreeMap::new(), None)) + } + } +} + +/// Convert one parameter input to a normalized JSON type. +fn try_convert_value(param_type: &JsonParamType, input: &ParamInput) -> Option { + match input { + ParamInput::Text(value) => try_convert_text_value(param_type, value), + ParamInput::Elements(elements) => try_convert_elements_value(param_type, elements), + } +} + +/// Convert one raw string value to a normalized JSON type. +fn try_convert_text_value(param_type: &JsonParamType, value: &str) -> Option { + match param_type { + JsonParamType::String => Some(Value::String(value.to_string())), + JsonParamType::Integer => value.parse::().ok().map(Number::from).map(Value::Number), + JsonParamType::Number => try_convert_number(value), + JsonParamType::Boolean => try_convert_boolean(value), + JsonParamType::Object { .. } if value.is_empty() => Some(Value::Object(Map::new())), + JsonParamType::Array { .. } if value.is_empty() => Some(Value::Array(Vec::new())), + JsonParamType::Object { .. } | JsonParamType::Array { .. } => { + // For composite types with string input, simply interpret the string as JSON. + serde_json::from_str(value).ok() + } + JsonParamType::Null => value.eq_ignore_ascii_case("null").then_some(Value::Null), + JsonParamType::OneOf(types) => { + types.iter().find_map(|param_type| try_convert_text_value(param_type, value)) + } + } +} + +/// Convert one structured parameter input to a normalized JSON type. +fn try_convert_elements_value( + param_type: &JsonParamType, + elements: &[ParamElement], +) -> Option { + match param_type { + JsonParamType::Object { + properties, + additional_properties, + } => Some(Value::Object(convert_elements_to_object( + elements, + properties, + additional_properties.as_deref(), + ))), + JsonParamType::Array { items } => Some(Value::Array( + // Collect all child elements into an array, regardless of their names. + elements + .iter() + .map(|element| convert_with_optional_schema(items.as_deref(), &element.value)) + .collect(), + )), + JsonParamType::OneOf(types) => types + .iter() + .find_map(|param_type| try_convert_elements_value(param_type, elements)), + + // Primitive types can't be converted from structured input. + JsonParamType::String + | JsonParamType::Integer + | JsonParamType::Number + | JsonParamType::Boolean + | JsonParamType::Null => None, + } +} + +/// Convert structured elements to an object, using field schemas when present. +fn convert_elements_to_object( + elements: &[ParamElement], + properties: &BTreeMap, + additional_properties: Option<&JsonParamType>, +) -> Map { + let mut object = Map::with_capacity(elements.len()); + for element in elements { + let param_type = properties.get(&element.name).or(additional_properties); + let value = convert_with_optional_schema(param_type, &element.value); + insert_object_value(&mut object, element.name.clone(), value); + } + object +} + +/// Insert an object field while preserving duplicate keys as arrays. +fn insert_object_value(object: &mut Map, key: String, value: Value) { + if let Some(existing) = object.get_mut(&key) { + match existing { + // Collect values under the same key into an array. + Value::Array(values) => values.push(value), + existing => { + let first = std::mem::replace(existing, Value::Null); + *existing = Value::Array(vec![first, value]); + } + } + } else { + object.insert(key, value); + } +} + +/// Convert one raw string value to a JSON number. +fn try_convert_number(value: &str) -> Option { + serde_json::from_str::(value) + .or_else(|_| value.parse::().map(Number::from)) + .or_else(|_| value.parse::().ok().and_then(Number::from_f64).ok_or(())) + .ok() + .map(Value::Number) +} + +/// Convert one raw string value to a boolean. +fn try_convert_boolean(value: &str) -> Option { + match value.trim().to_ascii_lowercase().as_str() { + "true" | "1" => Some(Value::Bool(true)), + "false" | "0" => Some(Value::Bool(false)), + _ => None, + } +} + +#[cfg(test)] +mod tests { + use serde_json::{Value, json}; + + use super::{ParamElement, ParamInput, ToolSchema, ToolSchemas}; + use crate::tool::Tool; + + fn test_tool(name: &str, parameters: serde_json::Value) -> Tool { + Tool { + name: name.to_string(), + description: None, + parameters, + strict: None, + } + } + + #[test] + fn invalid_schema_converts_everything_as_string() { + let params = ToolSchema::from_schema(&json!({ "type": "object" })); + + assert_eq!(params.convert("count", text("42")), json!("42")); + assert_eq!(params.convert("count", text("null")), json!(null)); + } + + #[test] + fn skips_unknown_property_schema_and_unknown_type() { + let params = ToolSchema::from_schema(&json!({ + "type": "object", + "properties": { + "unknown_schema": true, + "unknown_type": { "type": "mystery" }, + "known": { "type": "integer" } + } + })); + + assert_eq!(params.convert("unknown_schema", text("42")), json!("42")); + assert_eq!(params.convert("unknown_type", text("42")), json!("42")); + assert_eq!(params.convert("known", text("42")), json!(42)); + } + + #[test] + fn converts_supported_types() { + let params = ToolSchema::from_schema(&json!({ + "type": "object", + "properties": { + "text": { "type": "string" }, + "count": { "type": "integer" }, + "size": { "type": "number" }, + "ratio": { "type": "double" }, + "enabled": { "type": "boolean" }, + "payload": { "type": "object" }, + "mapping": { "type": "map" }, + "items": { "type": "array" }, + "names": { "type": "list" }, + "nothing": { "type": "null" } + } + })); + + assert_eq!(params.convert("text", text("42")), json!("42")); + assert_eq!(params.convert("count", text("42")), json!(42)); + assert_eq!(params.convert("size", text("5.0")), json!(5.0)); + assert_eq!(params.convert("ratio", text("2.5")), json!(2.5)); + assert_eq!(params.convert("enabled", text("1")), json!(true)); + assert_eq!( + params.convert("payload", text(r#"{"k":1}"#)), + json!({ "k": 1 }) + ); + assert_eq!( + params.convert("mapping", text(r#"{"k":1}"#)), + json!({ "k": 1 }) + ); + assert_eq!(params.convert("items", text("[1,2]")), json!([1, 2])); + assert_eq!( + params.convert("names", text(r#"["a","b"]"#)), + json!(["a", "b"]) + ); + assert_eq!(params.convert("nothing", text("null")), json!(null)); + } + + #[test] + fn number_conversion_preserves_json_number_spelling_with_legacy_fallback() { + let params = ToolSchema::from_schema(&json!({ + "type": "object", + "properties": { + "value": { "type": "number" } + } + })); + + assert_eq!(converted_number_text(¶ms, "5"), "5"); + assert_eq!(converted_number_text(¶ms, "5.0"), "5.0"); + assert_eq!(converted_number_text(¶ms, "5."), "5.0"); + assert_eq!(converted_number_text(¶ms, "+1"), "1"); + assert_eq!(converted_number_text(¶ms, "+1.0"), "1.0"); + + // TODO: we cannot preserve the original number precision by enabling `serde_json`'s + // `arbitrary_precision` feature, otherwise the test + // `serialized_json_numbers_do_not_leak_serde_private_representation` will fail. + // See issue: https://github.com/mitsuhiko/minijinja/issues/641 + + // assert_eq!(converted_number_text(¶ms, "5.00"), "5.00"); + // assert_eq!(converted_number_text(¶ms, "1e0"), "1e+0"); + // assert_eq!( + // converted_number_text(¶ms, "9223372036854775807.5"), + // "9223372036854775807.5" + // ); + } + + fn converted_number_text(params: &ToolSchema, value: &str) -> String { + serde_json::to_string(¶ms.convert("value", text(value))).unwrap() + } + + fn text(value: &str) -> ParamInput { + ParamInput::Text(value.to_string()) + } + + fn elem(name: &str, value: ParamInput) -> ParamElement { + ParamElement { + name: name.to_string(), + value, + } + } + + fn elements(elements: Vec) -> ParamInput { + ParamInput::Elements(elements) + } + + #[test] + fn converts_upstream_aliases() { + let params = ToolSchema::from_schema(&json!({ + "type": "object", + "properties": { + "s": { "type": "varchar" }, + "i": { "type": "unsigned_int" }, + "n": { "type": "float64" }, + "b": { "type": "binary" }, + "a": { "type": "sequence" }, + "o": { "type": "dict" } + } + })); + + assert_eq!(params.convert("s", text("x")), json!("x")); + assert_eq!(params.convert("i", text("7")), json!(7)); + assert_eq!(params.convert("n", text("7.5")), json!(7.5)); + assert_eq!(params.convert("b", text("true")), json!(true)); + assert_eq!(params.convert("a", text("[1]")), json!([1])); + assert_eq!(params.convert("o", text(r#"{"x":1}"#)), json!({ "x": 1 })); + } + + #[test] + fn preserves_union_type_order() { + let integer_first = ToolSchema::from_schema(&json!({ + "type": "object", + "properties": { + "value": { "type": ["integer", "string"] } + } + })); + let string_first = ToolSchema::from_schema(&json!({ + "type": "object", + "properties": { + "value": { "type": ["string", "integer"] } + } + })); + + assert_eq!(integer_first.convert("value", text("42")), json!(42)); + assert_eq!(string_first.convert("value", text("42")), json!("42")); + } + + #[test] + fn converts_composite_schemas() { + let params = ToolSchema::from_schema(&json!({ + "type": "object", + "properties": { + "choice": { + "anyOf": [ + { "type": "integer" }, + { "type": "string" } + ] + }, + "fallback_object": { + "oneOf": [ + { "type": "mystery" } + ] + } + } + })); + + assert_eq!(params.convert("choice", text("42")), json!(42)); + assert_eq!( + params.convert("fallback_object", text(r#"{"x":1}"#)), + json!({ "x": 1 }) + ); + } + + #[test] + fn infers_type_from_schema_shape_without_type() { + let params = ToolSchema::from_schema(&json!({ + "type": "object", + "properties": { + "choice": { "enum": ["a", "b"] }, + "items": { "items": { "type": "integer" } }, + "payload": { "properties": { "x": { "type": "integer" } } } + } + })); + + assert_eq!(params.convert("choice", text("a")), json!("a")); + assert_eq!(params.convert("items", text("[1,2]")), json!([1, 2])); + assert_eq!( + params.convert("payload", text(r#"{"x":1}"#)), + json!({ "x": 1 }) + ); + } + + #[test] + fn converts_params_for_known_tool() { + let schemas = ToolSchemas::from_tools(&[test_tool( + "search", + json!({ + "type": "object", + "properties": { + "query": { "type": "string" }, + "topn": { "type": "integer" } + } + }), + )]); + + let converted = schemas.convert_params_with_schema( + "search", + vec![ + ("query".to_string(), "rust".to_string()), + ("topn".to_string(), "5".to_string()), + ], + ); + + assert_eq!(converted.get("query"), Some(&json!("rust"))); + assert_eq!(converted.get("topn"), Some(&json!(5))); + } + + #[test] + fn convert_params_falls_back_to_string_for_failed_coercion() { + let schemas = ToolSchemas::from_tools(&[test_tool( + "convert", + json!({ + "type": "object", + "properties": { + "whole": { "type": "number" }, + "flag": { "type": "boolean" }, + "payload": { "type": "object" }, + "items": { "type": "array" }, + "missing_type": {} + } + }), + )]); + + let converted = schemas.convert_params_with_schema( + "convert", + vec![ + ("whole".to_string(), "not-a-number".to_string()), + ("flag".to_string(), "maybe".to_string()), + ("payload".to_string(), "not-json".to_string()), + ("items".to_string(), "not-json".to_string()), + ("missing_type".to_string(), "42".to_string()), + ("unknown_param".to_string(), "42".to_string()), + ], + ); + + assert_eq!(converted.get("whole"), Some(&json!("not-a-number"))); + assert_eq!(converted.get("flag"), Some(&json!("maybe"))); + assert_eq!(converted.get("payload"), Some(&json!("not-json"))); + assert_eq!(converted.get("items"), Some(&json!("not-json"))); + assert_eq!(converted.get("missing_type"), Some(&json!("42"))); + assert_eq!(converted.get("unknown_param"), Some(&json!("42"))); + } + + #[test] + fn string_param_preserves_literal_null_text() { + // A `string`-typed param whose value is the literal text "null"/"NULL" + // must stay a string (the original case is preserved), rather than being + // coerced to JSON null. Non-string types keep coercing "null" to null. + let params = ToolSchema::from_schema(&json!({ + "type": "object", + "properties": { + "name": { "type": "string" }, + "count": { "type": "integer" }, + "anything": {} + } + })); + + assert_eq!(params.convert("name", text("null")), json!("null")); + assert_eq!(params.convert("name", text("NULL")), json!("NULL")); + // Non-string and schema-less params are unchanged: "null" -> null. + assert_eq!(params.convert("count", text("null")), json!(null)); + assert_eq!(params.convert("anything", text("null")), json!(null)); + } + + #[test] + fn nullable_enum_param_coerces_literal_null() { + // An enum that includes `null` admits a null value, so a literal "null" + // must coerce to JSON null (matching Python's `extract_types_from_schema`, + // which infers `null` from the enum values), while a non-null enum keeps + // "null" as a string. + let params = ToolSchema::from_schema(&json!({ + "type": "object", + "properties": { + "mode": { "enum": [null, "auto"] }, + "color": { "enum": ["red", "green"] } + } + })); + + assert_eq!(params.convert("mode", text("null")), json!(null)); + assert_eq!(params.convert("mode", text("auto")), json!("auto")); + assert_eq!(params.convert("color", text("null")), json!("null")); + } + + #[test] + fn unknown_tool_converts_values_without_schema() { + let schemas = ToolSchemas::from_tools(&[test_tool( + "search", + json!({ "type": "object", "properties": {} }), + )]); + + let converted = schemas.convert_params_with_schema( + "missing", + vec![ + ("query".to_string(), "rust".to_string()), + ("topn".to_string(), "5".to_string()), + ("nullish".to_string(), "null".to_string()), + ], + ); + + assert_eq!(converted.get("query"), Some(&json!("rust"))); + assert_eq!(converted.get("topn"), Some(&json!("5"))); + assert_eq!(converted.get("nullish"), Some(&json!(null))); + } + + #[test] + fn converts_structured_inputs_with_recursive_schema() { + let schemas = ToolSchemas::from_tools(&[test_tool( + "create_order", + json!({ + "type": "object", + "properties": { + "user_id": { "type": "integer" }, + "urgent": { "type": "boolean" }, + "note": { "type": "string" }, + "nil": { "type": "string" }, + "shipping": { + "type": "object", + "properties": { + "city": { "type": "string" }, + "zip": { "type": "integer" } + } + }, + "items": { + "type": "array", + "items": { + "type": "object", + "properties": { + "sku": { "type": "string" }, + "qty": { "type": "integer" } + } + } + }, + "metadata": { + "type": "object", + "additionalProperties": { "type": "integer" } + }, + "duplicate_demo": { + "type": "object", + "properties": { + "tag": { "type": "string" } + } + }, + "schema_mismatch_array": { + "type": "array", + "items": { "type": "integer" } + }, + "closed_object": { + "type": "object", + "additionalProperties": false + }, + "open_object": { + "type": "object", + "additionalProperties": true + }, + "payload_text": { "type": "object" }, + "items_text": { "type": "array" } + } + }), + )]); + + let converted = schemas.convert_params_with_schema( + "create_order", + vec![ + ("user_id".to_string(), text("42")), + ("urgent".to_string(), text("true")), + ("note".to_string(), text("Please leave at front desk.")), + ("nil".to_string(), text("NULL")), + ( + "shipping".to_string(), + elements(vec![ + elem("city", text("Singapore")), + elem("zip", text("018956")), + ]), + ), + ( + "items".to_string(), + elements(vec![ + elem( + "item1", + elements(vec![elem("sku", text("book-001")), elem("qty", text("2"))]), + ), + elem( + "item2", + elements(vec![elem("sku", text("pen-007")), elem("qty", text("5"))]), + ), + ]), + ), + ( + "metadata".to_string(), + elements(vec![elem("score", text("42")), elem("rank", text("7"))]), + ), + ( + "duplicate_demo".to_string(), + elements(vec![elem("tag", text("a")), elem("tag", text("b"))]), + ), + ( + "closed_object".to_string(), + elements(vec![elem("unknown", text("x"))]), + ), + ( + "open_object".to_string(), + elements(vec![elem("unknown", text("y"))]), + ), + ("payload_text".to_string(), text(r#"{"x":1}"#)), + ("items_text".to_string(), text("[1,2]")), + ( + "unknown_struct".to_string(), + elements(vec![ + elem("a", text("1")), + elem("a", text("2")), + elem("nil", text("null")), + ]), + ), + ], + ); + + assert_eq!( + Value::Object(converted), + json!({ + "user_id": 42, + "urgent": true, + "note": "Please leave at front desk.", + "nil": "NULL", + "shipping": { + "city": "Singapore", + "zip": 18956 + }, + "items": [ + { + "sku": "book-001", + "qty": 2 + }, + { + "sku": "pen-007", + "qty": 5 + } + ], + "metadata": { + "score": 42, + "rank": 7 + }, + "duplicate_demo": { + "tag": ["a", "b"] + }, + "closed_object": { + "unknown": "x" + }, + "open_object": { + "unknown": "y" + }, + "payload_text": { + "x": 1 + }, + "items_text": [1, 2], + "unknown_struct": { + "a": ["1", "2"], + "nil": null + } + }) + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/qwen_coder.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/qwen_coder.rs new file mode 100644 index 0000000000000000000000000000000000000000..745519d9689dfc6d40914f3cab73fff9223acce6 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/qwen_coder.rs @@ -0,0 +1,718 @@ +use winnow::ascii::multispace0 as ws0; +use winnow::combinator::{alt, delimited, eof, repeat, seq, terminated}; +use winnow::prelude::*; +use winnow::stream::Partial; +use winnow::token::{literal, take_until}; + +use super::parameters::ToolSchemas; +use super::utils::{MarkerScanState, parse_buffered_event, safe_text_len, take_until_marker}; +use super::{Result, StructuralTagModel, ToolCallDelta, ToolParser, ToolParserOutput}; +use crate::tool::Tool; + +const TOOL_CALL_START: &str = ""; +const TOOL_CALL_END: &str = ""; +const FUNCTION_START: &str = " = Partial<&'i str>; + +#[derive(Debug, Clone, PartialEq, Eq)] +enum QwenCoderMode { + Text, + ToolCall { end_marker_scan: MarkerScanState }, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum QwenCoderEvent { + Text { + len: usize, + }, + ToolCallStart, + ToolCall { + name: String, + raw_params: Vec<(String, String)>, + }, +} + +/// Tool parser for Qwen Coder XML-style tool calls. +/// +/// Example tool call content: +/// +/// ```text +/// +/// +/// 杭州 +/// +/// +/// ``` +/// +/// Arguments are emitted only after a full `tool_call` block is parsed. +/// +/// Note: parallel calls are represented as repeated +/// `...` blocks, not as multiple calls inside one tag. +pub struct Qwen3CoderToolParser { + buffer: String, + mode: QwenCoderMode, + emitted_tool_count: usize, + tool_parameters: ToolSchemas, +} + +impl Qwen3CoderToolParser { + /// Create a Qwen Coder tool parser. + fn new(tools: &[Tool]) -> Self { + Self { + buffer: String::new(), + mode: QwenCoderMode::Text, + emitted_tool_count: 0, + tool_parameters: ToolSchemas::from_tools(tools), + } + } + + /// Apply one parsed Qwen Coder event to parser state and output. + fn apply_event(&mut self, event: QwenCoderEvent, output: &mut ToolParserOutput) -> Result<()> { + match event { + QwenCoderEvent::Text { len: consumed_len } => { + output.push_text(&self.buffer[..consumed_len]); + } + QwenCoderEvent::ToolCallStart => { + self.mode = QwenCoderMode::ToolCall { + end_marker_scan: MarkerScanState::default(), + }; + } + QwenCoderEvent::ToolCall { name, raw_params } => { + self.mode = QwenCoderMode::Text; + let arguments = self.tool_parameters.convert_params_with_schema(&name, raw_params); + let arguments = serde_json::to_string(&arguments) + .map_err(|error| parsing_failed!("failed to serialize arguments: {}", error))?; + + output.push_call(ToolCallDelta { + tool_index: self.emitted_tool_count, + name: Some(name), + arguments, + }); + self.emitted_tool_count += 1; + } + } + Ok(()) + } + + fn reset(&mut self) -> String { + self.mode = QwenCoderMode::Text; + self.emitted_tool_count = 0; + std::mem::take(&mut self.buffer) + } +} + +impl ToolParser for Qwen3CoderToolParser { + fn create(tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self::new(tools))) + } + + fn structural_tag_model(&self) -> Option { + Some(StructuralTagModel::Qwen3Coder) + } + + fn parse_into(&mut self, chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + self.buffer.push_str(chunk); + + while let Some((event, consumed_len)) = parse_buffered_event(&self.buffer, |input| { + parse_next_qwen_coder_event(input, &mut self.mode) + })? { + self.apply_event(event, output)?; + self.buffer.drain(..consumed_len); + } + + Ok(()) + } + + fn finish(&mut self) -> Result { + let mut output = ToolParserOutput::default(); + if !self.buffer.is_empty() { + if matches!(self.mode, QwenCoderMode::ToolCall { .. }) + || self.buffer.starts_with(TOOL_CALL_START) + { + return Err(parsing_failed!("incomplete Qwen Coder tool call")); + } + output.push_text(&self.buffer); + } + let _ = self.reset(); + Ok(output) + } + + fn reset(&mut self) -> String { + Qwen3CoderToolParser::reset(self) + } +} + +/// Parse a Qwen Coder event for the current parser mode. +fn parse_next_qwen_coder_event( + input: &mut QwenCoderInput<'_>, + mode: &mut QwenCoderMode, +) -> ModalResult { + match mode { + QwenCoderMode::Text => parse_text_event(input), + QwenCoderMode::ToolCall { end_marker_scan } => tool_call_event(input, end_marker_scan), + } +} + +/// Parse a text-mode Qwen Coder event. +fn parse_text_event(input: &mut QwenCoderInput<'_>) -> ModalResult { + alt((tool_call_start_event, safe_text_event)).parse_next(input) +} + +/// Parse a Qwen Coder tool-call start marker. +fn tool_call_start_event(input: &mut QwenCoderInput<'_>) -> ModalResult { + literal(TOOL_CALL_START).value(QwenCoderEvent::ToolCallStart).parse_next(input) +} + +/// Parse a safe text run before the next Qwen Coder marker. +fn safe_text_event(input: &mut QwenCoderInput<'_>) -> ModalResult { + safe_text_len(input, TOOL_CALL_START).map(|len| QwenCoderEvent::Text { len }) +} + +/// Parse a complete Qwen Coder tool call. +fn tool_call_event( + input: &mut QwenCoderInput<'_>, + end_marker_scan: &mut MarkerScanState, +) -> ModalResult { + let (body,) = seq!( + _: ws0, + take_until_marker(TOOL_CALL_END, end_marker_scan), + _: literal(TOOL_CALL_END), + ) + .parse_next(input)?; + + parse_tool_call_body(body) +} + +/// Parse a Qwen Coder function block. +fn function_event(input: &mut &str) -> ModalResult { + let (name, raw_params) = seq!( + _: literal(FUNCTION_START), + take_until(1.., ">"), + _: ">", + _: ws0, + repeat(0.., terminated(parameter, ws0)), + _: literal(FUNCTION_END), + ) + .parse_next(input)?; + + Ok(QwenCoderEvent::ToolCall { + name: name.to_string(), + raw_params, + }) +} + +/// Parse a Qwen Coder parameter block. +fn parameter(input: &mut &str) -> ModalResult<(String, String)> { + let (name, value) = seq!( + _: literal(PARAMETER_START), + take_until(1.., ">"), + _: ">", + take_until(0.., PARAMETER_END).map(trim_one_wrapping_newline), + _: literal(PARAMETER_END), + ) + .parse_next(input)?; + + Ok((name.to_string(), value.to_string())) +} + +/// Parse a Qwen Coder tool-call body. +fn parse_tool_call_body(body: &str) -> ModalResult { + let mut input = body; + delimited(ws0, function_event, (ws0, eof)).parse_next(&mut input) +} + +/// Trim a single leading and trailing newline from a parameter value. +fn trim_one_wrapping_newline(value: &str) -> &str { + let value = value.strip_prefix('\n').unwrap_or(value); + value.strip_suffix('\n').unwrap_or(value) +} + +#[cfg(test)] +mod tests { + use expect_test::expect; + use serde_json::{Value, json}; + use thiserror_ext::AsReport; + + use super::{Qwen3CoderToolParser, StructuralTagModel, ToolParser}; + use crate::tool::test_utils::{collect_stream, split_by_chars, test_tools}; + use crate::tool::{ToolParserOutput, ToolParserTestExt as _}; + + fn build_tool_call(function_name: &str, params: &[(&str, &str)]) -> String { + let params = params + .iter() + .map(|(name, value)| format!("{value}")) + .collect::>() + .join("\n"); + format!("\n\n{params}\n\n") + } + + #[test] + fn qwen_coder_exposes_structural_tag_model() { + let parser = Qwen3CoderToolParser::new(&test_tools()); + + assert_eq!( + parser.structural_tag_model(), + Some(StructuralTagModel::Qwen3Coder) + ); + } + + #[test] + fn qwen_coder_parse_complete_without_tool_call_keeps_text() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = parser.parse_complete("Hello, world!").unwrap(); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn qwen_coder_parse_complete_extracts_single_tool_call() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_call( + "get_weather", + &[("location", "SF"), ("date", "2026-04-29")], + )) + .unwrap(); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "location": "SF", + "date": "2026-04-29" + }) + ); + } + + #[test] + fn qwen_coder_parse_complete_preserves_prefix_text() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = format!( + "Thinking... {}", + build_tool_call("get_weather", &[("location", "NYC")]) + ); + let output = parser.parse_complete(&output).unwrap(); + + assert_eq!(output.normal_text(), "Thinking... "); + assert_eq!(output.calls().len(), 1); + } + + #[test] + fn qwen_coder_parse_complete_converts_schema_types() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_call( + "convert", + &[ + ("whole", "5.0"), + ("flag", "true"), + ("payload", r#"{"nested":true}"#), + ("items", "[1,2]"), + ("empty", "42"), + ], + )) + .unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "whole": 5.0, + "flag": true, + "payload": { "nested": true }, + "items": [1, 2], + "empty": "42", + }) + ); + } + + #[test] + fn qwen_coder_parse_complete_extracts_empty_arguments() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = parser.parse_complete(&build_tool_call("get_weather", &[])).unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({}) + ); + } + + #[test] + fn qwen_coder_parse_complete_handles_upstream_multiline_typed_params() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = parser + .parse_complete( + "\n\ + \n\ + \n\ + rectangle\n\ + \n\ + \n\ + {\"width\": 10,\n\ + \"height\": 20}\n\ + \n\ + \n\ + 2\n\ + \n\ + \n\ + ", + ) + .unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("calculate_area")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "shape": "rectangle", + "dimensions": { "width": 10, "height": 20 }, + "precision": 2, + }) + ); + } + + #[test] + fn qwen_coder_parse_complete_handles_nested_json_parameter() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_call( + "convert", + &[( + "payload", + r#"{"nested":{"value":[1,2,3],"child":{"enabled":true}}}"#, + )], + )) + .unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "payload": { + "nested": { + "value": [1, 2, 3], + "child": { "enabled": true }, + }, + }, + }) + ); + } + + #[test] + fn qwen_coder_parse_complete_preserves_xml_like_parameter_values() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_call( + "process", + &[ + ( + "html_content", + r#"

Hello
"#, + ), + ("xml_snippet", r#""#), + ], + )) + .unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "html_content": r#"
Hello
"#, + "xml_snippet": r#""#, + }) + ); + } + + #[test] + fn qwen_coder_parse_complete_preserves_raw_closing_tag_text_in_parameter_value() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_call( + "get_weather", + &[ + ( + "location", + "杭州 </parameter></function></tool_call>", + ), + ("date", "2026-05-08"), + ], + )) + .unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "location": "杭州 </parameter></function></tool_call>", + "date": "2026-05-08", + }) + ); + } + + #[test] + fn qwen_coder_parse_complete_does_not_double_encode_anyof_object() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = parser + .parse_complete(&build_tool_call( + "update_record", + &[("data", r#"{"key":"value","count":42}"#)], + )) + .unwrap(); + + assert_eq!(output.calls().len(), 1); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ + "data": { "key": "value", "count": 42 }, + }) + ); + } + + #[test] + fn qwen_coder_streaming_extracts_single_tool_call() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = collect_stream( + &mut parser, + &[ + "\n", + "\n", + "SF\n", + "\n", + "", + ], + ); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "location": "SF" }) + ); + } + + #[test] + fn qwen_coder_streaming_preserves_prefix_text() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = collect_stream( + &mut parser, + &[ + "Thinking... ", + "\n", + "\n", + "SF\n", + "\n", + "", + ], + ); + + assert_eq!(output.normal_text(), "Thinking... "); + assert_eq!(output.calls().len(), 1); + } + + #[test] + fn qwen_coder_streaming_without_tool_call_emits_text_incrementally() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = collect_stream(&mut parser, &["Hello, ", "world!"]); + + assert_eq!(output.normal_text(), "Hello, world!"); + assert!(output.calls().is_empty()); + } + + #[test] + fn qwen_coder_streaming_extracts_multiple_tool_calls_in_order() { + let text = format!( + "{}\n{}", + build_tool_call("get_weather", &[("location", "SF")]), + build_tool_call("get_weather", &[("location", "NYC")]) + ); + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = collect_stream(&mut parser, &[&text]); + + assert_eq!(output.calls().len(), 2); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[1].name.as_deref(), Some("get_weather")); + assert_eq!(output.calls()[0].tool_index, 0); + assert_eq!(output.calls()[1].tool_index, 1); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "location": "SF" }) + ); + assert_eq!( + serde_json::from_str::(&output.calls()[1].arguments).unwrap(), + json!({ "location": "NYC" }) + ); + } + + #[test] + fn qwen_coder_streaming_preserves_text_between_tool_calls() { + let text = format!( + "I'll check two cities.{}Between calls.{}Done.", + build_tool_call("get_weather", &[("city", "Dallas"), ("state", "TX")]), + build_tool_call("get_weather", &[("city", "Orlando"), ("state", "FL")]) + ); + let chunks = split_by_chars(&text, 5); + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = collect_stream(&mut parser, &chunks); + + assert_eq!( + output.normal_text(), + "I'll check two cities.Between calls.Done." + ); + assert_eq!(output.calls().len(), 2); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "city": "Dallas", "state": "TX" }) + ); + assert_eq!( + serde_json::from_str::(&output.calls()[1].arguments).unwrap(), + json!({ "city": "Orlando", "state": "FL" }) + ); + } + + #[test] + fn qwen_coder_streaming_handles_start_token_split_across_chunks() { + let text = build_tool_call("get_weather", &[("location", "SF")]); + let chunks = split_by_chars(&text, 3); + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = collect_stream(&mut parser, &chunks); + + assert_eq!(output.calls().len(), 1); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "location": "SF" }) + ); + } + + #[test] + fn qwen_coder_streaming_handles_end_token_split_across_chunks() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = parser + .parse_chunk( + "\n\ + \n\ + SF\n\ + \n\ +
").unwrap()); + output.append(parser.finish().unwrap()); + let output = output.coalesce(); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "location": "SF" }) + ); + } + + #[test] + fn qwen_coder_streaming_buffers_long_body_until_end_marker() { + let long_location = format!("SF-{}", "x".repeat(8192)); + let text = build_tool_call("get_weather", &[("location", &long_location)]); + let split_at = text.len() - "_call>".len(); + let (body_with_partial_end, end_suffix) = text.split_at(split_at); + let chunks = split_by_chars(body_with_partial_end, 31); + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let mut output = ToolParserOutput::default(); + + assert_eq!(end_suffix, "_call>"); + + for chunk in chunks { + let chunk_output = parser.parse_chunk(chunk).unwrap(); + assert!(chunk_output.normal_text().is_empty()); + assert!(chunk_output.calls().is_empty()); + output.append(chunk_output); + } + + output.append(parser.parse_chunk(end_suffix).unwrap()); + output.append(parser.finish().unwrap()); + let output = output.coalesce(); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(output.calls()[0].name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "location": long_location }) + ); + } + + #[test] + fn qwen_coder_streaming_does_not_emit_incomplete_tool_call() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = parser + .parse_chunk("\n\nSF") + .unwrap(); + + assert!(output.normal_text().is_empty()); + assert!(output.calls().is_empty()); + } + + #[test] + fn qwen_coder_finish_fails_incomplete_tool_call() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + parser + .parse_chunk("\n\nSF") + .unwrap(); + + assert!(parser.finish().is_err()); + } + + #[test] + fn qwen_coder_malformed_tool_call_fails_fast() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let error = parser.parse_chunk("\n\n").unwrap_err(); + + expect![[r#"tool parser parsing failed: near "\n\n": "#]] + .assert_eq(&error.to_report_string()); + } + + #[test] + fn qwen_coder_missing_parameter_end_fails_fast_after_function_end() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let error = parser + .parse_chunk( + "\n\nSF\n", + ) + .unwrap_err(); + + expect![[r#"tool parser parsing failed: near "\n\nSF\n": "#]].assert_eq(&error.to_report_string()); + } + + #[test] + fn qwen_coder_parse_function_body_trims_one_wrapping_newline() { + let mut parser = Qwen3CoderToolParser::new(&test_tools()); + let output = parser + .parse_complete( + "\n\n\nHangzhou\n\n\n", + ) + .unwrap(); + + assert_eq!( + serde_json::from_str::(&output.calls()[0].arguments).unwrap(), + json!({ "location": "Hangzhou" }) + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/test_utils.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/test_utils.rs new file mode 100644 index 0000000000000000000000000000000000000000..03c570ebfd1f38f68fc2a4b681abbbea15dbcda5 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/test_utils.rs @@ -0,0 +1,117 @@ +use serde_json::json; + +use super::{ToolParser, ToolParserOutput}; +use crate::tool::Tool; + +/// Build a reusable set of function tools for parser unit tests. +pub fn test_tools() -> Vec { + vec![ + Tool { + name: "get_weather".to_string(), + description: None, + parameters: json!({ + "type": "object", + "properties": { + "location": { "type": "string" }, + "city": { "type": "string" }, + "state": { "type": "string" }, + "unit": { "type": "string" }, + "date": { "type": "string" }, + "days": { "type": "integer" } + } + }), + strict: None, + }, + Tool { + name: "add".to_string(), + description: None, + parameters: json!({ + "type": "object", + "properties": { + "x": { "type": "integer" }, + "y": { "type": "integer" } + } + }), + strict: None, + }, + Tool { + name: "convert".to_string(), + description: None, + parameters: json!({ + "type": "object", + "properties": { + "whole": { "type": "number" }, + "flag": { "type": "boolean" }, + "payload": { "type": "object" }, + "items": { "type": "array" }, + "empty": { "type": "string" } + } + }), + strict: None, + }, + Tool { + name: "calculate_area".to_string(), + description: None, + parameters: json!({ + "type": "object", + "properties": { + "shape": { "type": "string" }, + "dimensions": { "type": "object" }, + "precision": { "type": "integer" } + } + }), + strict: None, + }, + Tool { + name: "update_record".to_string(), + description: None, + parameters: json!({ + "type": "object", + "properties": { + "data": { + "anyOf": [ + { "type": "object" }, + { "type": "null" } + ] + } + } + }), + strict: None, + }, + ] +} + +/// Push chunks through a streaming parser and coalesce its tool-call deltas. +/// +/// Panics if there are any parsing errors along the way. +pub fn collect_stream(parser: &mut T, chunks: &[&str]) -> ToolParserOutput { + let mut output = ToolParserOutput::default(); + for chunk in chunks { + parser.parse_into(chunk, &mut output).unwrap(); + } + output.append(parser.finish().unwrap()); + output.coalesce() +} + +/// Split text into chunks containing at most `chunk_chars` Unicode scalar +/// values. +pub fn split_by_chars(text: &str, chunk_chars: usize) -> Vec<&str> { + let mut chunks = Vec::new(); + let mut start = 0; + let mut count = 0; + + for (index, _) in text.char_indices() { + if count == chunk_chars { + chunks.push(&text[start..index]); + start = index; + count = 0; + } + count += 1; + } + + if start < text.len() { + chunks.push(&text[start..]); + } + + chunks +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/tool/tests.rs b/asian_ai/performance/vLLM/rust/src/parser/src/tool/tests.rs new file mode 100644 index 0000000000000000000000000000000000000000..f7a1f8f2691e6083e972733ea29da19caa839cb2 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/tool/tests.rs @@ -0,0 +1,169 @@ +use super::{Result, Tool, ToolCallDelta, ToolParser, ToolParserEvent, ToolParserOutput}; +use crate::tool::ToolParserTestExt as _; + +struct DefaultParser; + +impl ToolParser for DefaultParser { + fn create(_tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self)) + } + + fn parse_into(&mut self, _chunk: &str, _output: &mut ToolParserOutput) -> Result<()> { + Ok(()) + } + + fn finish(&mut self) -> Result { + Ok(ToolParserOutput::default()) + } + + fn reset(&mut self) -> String { + String::new() + } +} + +#[test] +fn tool_parser_does_not_preserve_special_tokens_by_default() { + let parser = DefaultParser; + + assert!(!parser.preserve_special_tokens()); +} + +#[test] +fn tool_parser_output_coalesces_adjacent_text_events() { + let mut output = ToolParserOutput::default(); + output.push_text("hello"); + output.push_text(" "); + output.push_text("world"); + output.push_call(ToolCallDelta { + tool_index: 0, + name: Some("lookup".to_string()), + arguments: "{}".to_string(), + }); + output.push_text("!"); + + assert_eq!( + output.events, + vec![ + ToolParserEvent::Text("hello world".to_string()), + ToolParserEvent::ToolCall(ToolCallDelta { + tool_index: 0, + name: Some("lookup".to_string()), + arguments: "{}".to_string(), + }), + ToolParserEvent::Text("!".to_string()), + ] + ); +} + +#[test] +fn tool_parser_output_append_coalesces_adjacent_text_events() { + let mut output = ToolParserOutput::default(); + output.push_text("hello"); + + let mut other = ToolParserOutput::default(); + other.push_text(" "); + other.push_text("world"); + output.append(other); + + let mut after_call = ToolParserOutput::default(); + after_call.push_call(ToolCallDelta { + tool_index: 0, + name: Some("lookup".to_string()), + arguments: "{}".to_string(), + }); + after_call.push_text("!"); + output.append(after_call); + + assert_eq!( + output.events, + vec![ + ToolParserEvent::Text("hello world".to_string()), + ToolParserEvent::ToolCall(ToolCallDelta { + tool_index: 0, + name: Some("lookup".to_string()), + arguments: "{}".to_string(), + }), + ToolParserEvent::Text("!".to_string()), + ] + ); +} + +#[test] +fn default_parse_complete_delegates_through_parse_chunk_and_finish() { + struct StreamingParser; + + impl ToolParser for StreamingParser { + fn create(_tools: &[Tool]) -> Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self)) + } + + fn parse_into(&mut self, _chunk: &str, output: &mut ToolParserOutput) -> Result<()> { + output.push_text("prefix "); + for call in [ + ToolCallDelta { + tool_index: 0, + name: Some("weather".to_string()), + arguments: "{\"location\":".to_string(), + }, + ToolCallDelta { + tool_index: 0, + name: None, + arguments: "\"Paris\"".to_string(), + }, + ToolCallDelta { + tool_index: 1, + name: Some("time".to_string()), + arguments: "{\"timezone\":".to_string(), + }, + ] { + output.push_call(call); + } + Ok(()) + } + + fn finish(&mut self) -> Result { + let mut output = ToolParserOutput::default(); + output.push_text("suffix"); + output.push_call(ToolCallDelta { + tool_index: 0, + name: None, + arguments: "}".to_string(), + }); + output.push_call(ToolCallDelta { + tool_index: 1, + name: None, + arguments: "\"UTC\"}".to_string(), + }); + Ok(output) + } + + fn reset(&mut self) -> String { + String::new() + } + } + + let mut parser = StreamingParser; + let output = parser.parse_complete("ignored").unwrap(); + assert_eq!(output.normal_text(), "prefix suffix"); + assert_eq!( + output.calls().into_iter().cloned().collect::>(), + vec![ + ToolCallDelta { + tool_index: 0, + name: Some("weather".to_string()), + arguments: "{\"location\":\"Paris\"}".to_string(), + }, + ToolCallDelta { + tool_index: 1, + name: Some("time".to_string()), + arguments: "{\"timezone\":\"UTC\"}".to_string(), + }, + ] + ); +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/unified/combined.rs b/asian_ai/performance/vLLM/rust/src/parser/src/unified/combined.rs new file mode 100644 index 0000000000000000000000000000000000000000..f97cc1fddba14a00644a3b31d9d95215fc50306f --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/unified/combined.rs @@ -0,0 +1,320 @@ +//! Adapter that combines reasoning and tool parsers. + +use vllm_tokenizer::DynTokenizer; + +use super::{Result, UnifiedParser, UnifiedParserError, UnifiedParserOutput}; +use crate::reasoning::ReasoningParser; +use crate::tool::{StructuralTagModel, Tool, ToolParser, ToolParserOutput}; + +/// Unified parser that composes existing reasoning and tool parsers. +pub struct CombinedParser { + reasoning: Option>, + tool: Option>, +} + +impl CombinedParser { + /// Create a combined parser from optional reasoning and tool parsers. + pub fn new( + reasoning: Option>, + tool: Option>, + ) -> Self { + Self { reasoning, tool } + } + + /// Create a text-only combined parser. + pub fn plain_text_only() -> Self { + Self { + reasoning: None, + tool: None, + } + } + + fn parse_tool(&mut self, content: &str, output: &mut UnifiedParserOutput) -> Result<()> { + let Some(tool) = self.tool.as_mut() else { + output.push_text(content); + return Ok(()); + }; + + // Preserve any tool output that was already produced before the error. + let mut tool_output = ToolParserOutput::default(); + let result = tool.parse_into(content, &mut tool_output); + output.append_tool_output(tool_output); + result?; + + Ok(()) + } + + fn flush_tool(&mut self) -> Result { + let Some(tool) = self.tool.as_mut() else { + return Ok(UnifiedParserOutput::default()); + }; + + let output = tool.finish()?; + let mut unified = UnifiedParserOutput::default(); + unified.append_tool_output(output); + Ok(unified) + } +} + +impl UnifiedParser for CombinedParser { + fn create(_tools: &[Tool], _tokenizer: DynTokenizer) -> Result> + where + Self: Sized + 'static, + { + Err(UnifiedParserError::CombinedParserConstructor) + } + + fn initialize(&mut self, prompt_token_ids: &[u32]) -> Result<()> { + if let Some(reasoning) = self.reasoning.as_mut() { + reasoning.initialize(prompt_token_ids)?; + } + Ok(()) + } + + fn preserve_special_tokens(&self) -> bool { + self.reasoning.as_ref().is_some_and(|parser| parser.preserve_special_tokens()) + || self.tool.as_ref().is_some_and(|parser| parser.preserve_special_tokens()) + } + + fn structural_tag_model(&self) -> Option { + self.tool.as_ref().and_then(|parser| parser.structural_tag_model()) + } + + fn tool_call_id(&self, tool_index: usize) -> Option<&str> { + self.tool.as_ref().and_then(|parser| parser.tool_call_id(tool_index)) + } + + fn parse_into(&mut self, delta: &str, output: &mut UnifiedParserOutput) -> Result<()> { + let Some(reasoning) = self.reasoning.as_mut() else { + return self.parse_tool(delta, output); + }; + + let reasoning_delta = reasoning.push(delta)?; + if let Some(reasoning) = reasoning_delta.reasoning { + output.push_reasoning(reasoning); + } + if let Some(content) = reasoning_delta.content { + self.parse_tool(&content, output)?; + } + Ok(()) + } + + fn finish(&mut self) -> Result { + let mut output = UnifiedParserOutput::default(); + if let Some(reasoning) = self.reasoning.as_mut() { + let reasoning_delta = reasoning.finish()?; + if let Some(reasoning) = reasoning_delta.reasoning { + output.push_reasoning(reasoning); + } + if let Some(content) = reasoning_delta.content { + self.parse_tool(&content, &mut output)?; + } + } + output.append(self.flush_tool()?); + Ok(output) + } + + fn reset(&mut self) -> String { + self.tool.as_mut().map_or_else(String::new, |parser| parser.reset()) + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use vllm_tokenizer::test_utils::TestTokenizer; + + use super::CombinedParser; + use crate::reasoning::{Qwen3ReasoningParser, ReasoningDelta, ReasoningParser}; + use crate::tool::{Qwen3XmlToolParser, Tool, ToolParser}; + use crate::unified::{UnifiedParser, UnifiedParserEvent, UnifiedParserOutput}; + + fn tokenizer() -> TestTokenizer { + TestTokenizer::new() + .with_regular_token("", 256) + .with_regular_token("", 257) + } + + fn test_tools() -> Vec { + vec![Tool { + name: "get_weather".to_string(), + description: None, + parameters: serde_json::json!({ + "type": "object", + "properties": { + "location": { "type": "string" } + }, + }), + strict: None, + }] + } + + fn collect(parser: &mut dyn UnifiedParser, chunks: &[&str]) -> UnifiedParserOutput { + let mut output = UnifiedParserOutput::default(); + for chunk in chunks { + parser.parse_into(chunk, &mut output).unwrap(); + } + output.append(parser.finish().unwrap()); + output + } + + struct PreserveReasoningParser; + + impl ReasoningParser for PreserveReasoningParser { + fn create( + _tokenizer: vllm_tokenizer::DynTokenizer, + ) -> crate::reasoning::Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self)) + } + + fn preserve_special_tokens(&self) -> bool { + true + } + + fn push(&mut self, delta: &str) -> crate::reasoning::Result { + Ok(ReasoningDelta { + reasoning: None, + content: Some(delta.to_string()), + }) + } + } + + struct PreserveToolParser; + + impl ToolParser for PreserveToolParser { + fn create(_tools: &[Tool]) -> crate::tool::Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self)) + } + + fn preserve_special_tokens(&self) -> bool { + true + } + + fn parse_into( + &mut self, + chunk: &str, + output: &mut crate::tool::ToolParserOutput, + ) -> crate::tool::Result<()> { + output.push_text(chunk); + Ok(()) + } + + fn finish(&mut self) -> crate::tool::Result { + Ok(crate::tool::ToolParserOutput::default()) + } + + fn reset(&mut self) -> String { + String::new() + } + } + + struct PartialThenErrorToolParser; + + impl ToolParser for PartialThenErrorToolParser { + fn create(_tools: &[Tool]) -> crate::tool::Result> + where + Self: Sized + 'static, + { + Ok(Box::new(Self)) + } + + fn parse_into( + &mut self, + _chunk: &str, + output: &mut crate::tool::ToolParserOutput, + ) -> crate::tool::Result<()> { + output.push_text("committed"); + Err(crate::tool::ToolParserError::ParsingFailed { + message: "synthetic failure".to_string(), + }) + } + + fn finish(&mut self) -> crate::tool::Result { + Ok(crate::tool::ToolParserOutput::default()) + } + + fn reset(&mut self) -> String { + String::new() + } + } + + #[test] + fn combined_parser_emits_reasoning_and_text() { + let tokenizer = Arc::new(tokenizer()); + let reasoning = Qwen3ReasoningParser::create(tokenizer).unwrap(); + let mut parser = CombinedParser::new(Some(reasoning), None); + + let output = collect(&mut parser, &["workanswer"]); + + assert_eq!( + output.events, + vec![ + UnifiedParserEvent::Reasoning("work".to_string()), + UnifiedParserEvent::Text("answer".to_string()), + ] + ); + } + + #[test] + fn combined_parser_emits_tool_calls_from_visible_content() { + let tool = Qwen3XmlToolParser::create(&test_tools()).unwrap(); + let mut parser = CombinedParser::new(None, Some(tool)); + assert!(matches!( + parser.structural_tag_model(), + Some(crate::tool::StructuralTagModel::Qwen3) + )); + + let output = collect( + &mut parser, + &[r#" +{"name":"get_weather","arguments":{"location":"Paris"}} +"#], + ); + + assert_eq!( + output.events, + vec![ + UnifiedParserEvent::ToolCall(crate::tool::ToolCallDelta { + tool_index: 0, + name: Some("get_weather".to_string()), + arguments: String::new(), + }), + UnifiedParserEvent::ToolCall(crate::tool::ToolCallDelta { + tool_index: 0, + name: None, + arguments: r#"{"location":"Paris"}"#.to_string(), + }), + ] + ); + } + + #[test] + fn combined_parser_preserves_tool_output_on_parse_error() { + let mut parser = CombinedParser::new(None, Some(Box::new(PartialThenErrorToolParser))); + let mut output = UnifiedParserOutput::default(); + + let error = parser.parse_into("bad", &mut output).unwrap_err(); + + assert!(matches!(error, crate::unified::UnifiedParserError::Tool(_))); + assert_eq!( + output.events, + vec![UnifiedParserEvent::Text("committed".to_string())] + ); + } + + #[test] + fn combined_parser_preserves_special_tokens_when_either_inner_parser_needs_it() { + let mut parser = CombinedParser::new(Some(Box::new(PreserveReasoningParser)), None); + assert!(parser.preserve_special_tokens()); + + parser = CombinedParser::new(None, Some(Box::new(PreserveToolParser))); + assert!(parser.preserve_special_tokens()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/unified/gemma4.rs b/asian_ai/performance/vLLM/rust/src/parser/src/unified/gemma4.rs new file mode 100644 index 0000000000000000000000000000000000000000..0082d7b84f3929c4de7188ab7d905a64f1784d59 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/unified/gemma4.rs @@ -0,0 +1,1107 @@ +use serde_json::{Map, Number, Value}; +use vllm_tokenizer::DynTokenizer; +use winnow::ascii::multispace0 as ws0; +use winnow::combinator::{alt, delimited, eof, opt, separated, seq, terminated}; +use winnow::error::{ContextError, ErrMode, ModalResult}; +use winnow::prelude::*; +use winnow::stream::{Partial, Stream}; +use winnow::token::{literal, take_till, take_until}; + +use super::{Result, UnifiedParser, UnifiedParserError, UnifiedParserOutput}; +use crate::reasoning::last_reasoning_boundary; +use crate::tool::{Tool, ToolCallDelta}; +use crate::unified::parsing_failed; +use crate::utils::{incomplete, parse_buffered_event, partial_prefix_len, safe_text_len_mul}; + +const REASONING_START: &str = "<|channel>thought\n"; +const CHANNEL_START: &str = "<|channel>"; +const CHANNEL_END: &str = ""; +const TOOL_CALL_START: &str = "<|tool_call>"; +const TOOL_CALL_END: &str = ""; +const STRING_DELIM: &str = "<|\"|>"; +const CALL_PREFIX: &str = "call:"; + +type Gemma4Input<'i> = Partial<&'i str>; + +#[derive(Debug, Clone, PartialEq)] +enum Gemma4Event { + Text { len: usize }, + Reasoning { len: usize }, + ReasoningStart, + ReasoningEnd, + ToolCallStart, + ToolCallHeader { name: String }, + ToolCall { args: Map }, +} + +#[derive(Debug, Clone, Default, PartialEq, Eq)] +struct Gemma4ArgsScanState { + scanned_len: usize, + in_string: bool, +} + +#[derive(Debug, Clone, Default, PartialEq, Eq)] +enum Gemma4Mode { + #[default] + Text, + Reasoning, + Header, + ToolCall { + name: String, + args_scan: Gemma4ArgsScanState, + }, +} + +/// Unified parser for Google Gemma4 models. +/// +/// Original Python implementation: +/// +/// +/// Handles Gemma4 reasoning and function-call formats: +/// +/// `<|channel>thought\nreasoning` +/// +/// `<|tool_call>call:func_name{key:<|"|>value<|"|>}` +/// +/// Arguments are emitted only after a full Gemma4 tool call is parsed. +pub struct Gemma4UnifiedParser { + buffer: String, + mode: Gemma4Mode, + emitted_tool_count: usize, + tokenizer: DynTokenizer, + channel_start_token_id: u32, + channel_end_token_id: u32, +} + +impl Gemma4UnifiedParser { + /// Create a Gemma4 parser. + pub fn new(_tools: &[Tool], tokenizer: DynTokenizer) -> Result { + let channel_start_token_id = tokenizer.token_to_id(CHANNEL_START).ok_or_else(|| { + UnifiedParserError::MissingToken { + token: CHANNEL_START.to_string(), + } + })?; + let channel_end_token_id = + tokenizer + .token_to_id(CHANNEL_END) + .ok_or_else(|| UnifiedParserError::MissingToken { + token: CHANNEL_END.to_string(), + })?; + + Ok(Self { + buffer: String::new(), + mode: Gemma4Mode::default(), + emitted_tool_count: 0, + channel_start_token_id, + channel_end_token_id, + tokenizer, + }) + } + + fn apply_event(&mut self, event: Gemma4Event, output: &mut UnifiedParserOutput) -> Result<()> { + match event { + Gemma4Event::Text { len: consumed_len } => { + output.push_text(self.buffer[..consumed_len].to_string()); + } + Gemma4Event::Reasoning { len: consumed_len } => { + output.push_reasoning(self.buffer[..consumed_len].to_string()); + } + Gemma4Event::ReasoningStart => self.mode = Gemma4Mode::Reasoning, + Gemma4Event::ReasoningEnd => self.mode = Gemma4Mode::Text, + Gemma4Event::ToolCallStart => self.mode = Gemma4Mode::Header, + Gemma4Event::ToolCallHeader { name } => { + self.mode = Gemma4Mode::ToolCall { + name, + args_scan: Gemma4ArgsScanState::default(), + }; + } + Gemma4Event::ToolCall { args } => { + let mode = std::mem::replace(&mut self.mode, Gemma4Mode::Text); + let Gemma4Mode::ToolCall { name, .. } = mode else { + return Err(parsing_failed!( + "Gemma4 arguments without an active tool call" + )); + }; + let arguments = serde_json::to_string(&args) + .map_err(|error| parsing_failed!("failed to serialize arguments: {}", error))?; + + output.push_call(ToolCallDelta { + tool_index: self.emitted_tool_count, + name: Some(name), + arguments, + }); + self.emitted_tool_count += 1; + } + } + Ok(()) + } + + fn initialize_mode(&mut self, prompt_token_ids: &[u32]) { + self.mode = match last_reasoning_boundary( + prompt_token_ids, + self.channel_start_token_id, + self.channel_end_token_id, + self.tokenizer.as_ref(), + ) { + Some(true) => Gemma4Mode::Reasoning, + Some(false) | None => Gemma4Mode::Text, + }; + } + + fn reset(&mut self) -> String { + let raw = match std::mem::replace(&mut self.mode, Gemma4Mode::Text) { + Gemma4Mode::Text => std::mem::take(&mut self.buffer), + Gemma4Mode::Reasoning => { + format!("{}{}", REASONING_START, std::mem::take(&mut self.buffer)) + } + Gemma4Mode::Header => { + format!("{}{}", TOOL_CALL_START, std::mem::take(&mut self.buffer)) + } + Gemma4Mode::ToolCall { name, .. } => { + format!( + "{}{}{}{{{}", + TOOL_CALL_START, + CALL_PREFIX, + name, + std::mem::take(&mut self.buffer) + ) + } + }; + self.mode = Gemma4Mode::Text; + self.emitted_tool_count = 0; + raw + } +} + +impl UnifiedParser for Gemma4UnifiedParser { + fn create(tools: &[Tool], tokenizer: DynTokenizer) -> Result> + where + Self: Sized + 'static, + { + Self::new(tools, tokenizer).map(|parser| Box::new(parser) as Box) + } + + fn initialize(&mut self, prompt_token_ids: &[u32]) -> Result<()> { + self.buffer.clear(); + self.emitted_tool_count = 0; + self.initialize_mode(prompt_token_ids); + Ok(()) + } + + fn preserve_special_tokens(&self) -> bool { + true + } + + fn parse_into(&mut self, chunk: &str, output: &mut UnifiedParserOutput) -> Result<()> { + self.buffer.push_str(chunk); + + while let Some((event, consumed_len)) = { + parse_buffered_event(&self.buffer, |input| { + parse_next_gemma4_event(input, &mut self.mode) + })? + } { + self.apply_event(event, output)?; + self.buffer.drain(..consumed_len); + } + + Ok(()) + } + + fn finish(&mut self) -> Result { + let mut output = UnifiedParserOutput::default(); + + match &self.mode { + Gemma4Mode::Text => output.push_text(std::mem::take(&mut self.buffer)), + Gemma4Mode::Reasoning => output.push_reasoning(std::mem::take(&mut self.buffer)), + Gemma4Mode::Header | Gemma4Mode::ToolCall { .. } => { + return Err(parsing_failed!("incomplete Gemma4 tool call")); + } + } + + let _ = self.reset(); + Ok(output) + } + + fn reset(&mut self) -> String { + Gemma4UnifiedParser::reset(self) + } +} + +/// Parse one Gemma4 event from buffered streaming input. +fn parse_next_gemma4_event( + input: &mut Gemma4Input<'_>, + mode: &mut Gemma4Mode, +) -> ModalResult { + match mode { + Gemma4Mode::Text => parse_text_event(input), + Gemma4Mode::Reasoning => parse_reasoning_event(input), + Gemma4Mode::Header => tool_call_header_event(input), + Gemma4Mode::ToolCall { args_scan, .. } => tool_call_args_event(input, args_scan), + } +} + +/// Parse a Gemma4 text-mode event. +fn parse_text_event(input: &mut Gemma4Input<'_>) -> ModalResult { + alt(( + reasoning_start_event, + tool_call_start_event, + safe_text_event, + )) + .parse_next(input) +} + +/// Parse a Gemma4 reasoning-mode event. +fn parse_reasoning_event(input: &mut Gemma4Input<'_>) -> ModalResult { + alt(( + reasoning_end_event, + tool_call_start_event, + safe_reasoning_event, + )) + .parse_next(input) +} + +/// Parse a Gemma4 reasoning start marker. +fn reasoning_start_event(input: &mut Gemma4Input<'_>) -> ModalResult { + literal(REASONING_START).value(Gemma4Event::ReasoningStart).parse_next(input) +} + +/// Parse a Gemma4 reasoning end marker. +fn reasoning_end_event(input: &mut Gemma4Input<'_>) -> ModalResult { + literal(CHANNEL_END).value(Gemma4Event::ReasoningEnd).parse_next(input) +} + +/// Parse a Gemma4 tool-call start marker. +fn tool_call_start_event(input: &mut Gemma4Input<'_>) -> ModalResult { + literal(TOOL_CALL_START).value(Gemma4Event::ToolCallStart).parse_next(input) +} + +/// Parse a Gemma4 tool-call header. +fn tool_call_header_event(input: &mut Gemma4Input<'_>) -> ModalResult { + let (name,) = seq!( + _: literal(CALL_PREFIX), + gemma4_tool_name, + _: literal("{"), + ) + .parse_next(input)?; + Ok(Gemma4Event::ToolCallHeader { name }) +} + +/// Parse complete Gemma4 tool-call arguments. +fn tool_call_args_event( + input: &mut Gemma4Input<'_>, + args_scan: &mut Gemma4ArgsScanState, +) -> ModalResult { + let raw_args = gemma4_raw_args_until_tool_call_end(input, args_scan)?; + let Some(args_input) = raw_args.strip_suffix('}') else { + return Err(ErrMode::Cut(ContextError::new())); + }; + let args = parse_gemma4_args(args_input)?; + + Ok(Gemma4Event::ToolCall { args }) +} + +/// Parse a Gemma4 tool name. +fn gemma4_tool_name(input: &mut Gemma4Input<'_>) -> ModalResult { + let name = take_until(1.., "{").parse_next(input)?.trim(); + if name.is_empty() { + return Err(ErrMode::Cut(ContextError::new())); + } + Ok(name.to_string()) +} + +/// Parse a safe text run before the next Gemma4 marker. +fn safe_text_event(input: &mut Gemma4Input<'_>) -> ModalResult { + safe_text_len_mul(input, &[REASONING_START, TOOL_CALL_START]) + .map(|len| Gemma4Event::Text { len }) +} + +/// Parse a safe reasoning run before the next Gemma4 marker. +fn safe_reasoning_event(input: &mut Gemma4Input<'_>) -> ModalResult { + safe_text_len_mul(input, &[CHANNEL_END, TOOL_CALL_START]) + .map(|len| Gemma4Event::Reasoning { len }) +} + +/// Parse raw Gemma4 arguments through the first end marker outside a Gemma string. +fn gemma4_raw_args_until_tool_call_end<'i>( + input: &mut Gemma4Input<'i>, + state: &mut Gemma4ArgsScanState, +) -> ModalResult<&'i str> { + let text = **input; + if state.scanned_len > text.len() { + return incomplete(); + } + + loop { + let rest = &text[state.scanned_len..]; + if state.in_string { + let Some(string_delim) = rest.find(STRING_DELIM) else { + state.scanned_len = safe_scan_len(text, state.scanned_len, &[STRING_DELIM]); + return incomplete(); + }; + + state.scanned_len += string_delim + STRING_DELIM.len(); + state.in_string = false; + continue; + } + + let next_string_delim = rest.find(STRING_DELIM); + let next_tool_call_end = rest.find(TOOL_CALL_END); + match (next_string_delim, next_tool_call_end) { + (Some(string_delim), Some(tool_call_end)) if tool_call_end < string_delim => { + let end = state.scanned_len + tool_call_end; + state.scanned_len = end + TOOL_CALL_END.len(); + input.next_slice(state.scanned_len); + return Ok(&text[..end]); + } + (Some(string_delim), _) => { + state.scanned_len += string_delim + STRING_DELIM.len(); + state.in_string = true; + } + (None, Some(tool_call_end)) => { + let end = state.scanned_len + tool_call_end; + state.scanned_len = end + TOOL_CALL_END.len(); + input.next_slice(state.scanned_len); + return Ok(&text[..end]); + } + (None, None) => { + state.scanned_len = + safe_scan_len(text, state.scanned_len, &[STRING_DELIM, TOOL_CALL_END]); + return incomplete(); + } + } + } +} + +/// Return the scan length while holding back a split marker prefix. +fn safe_scan_len(text: &str, start: usize, markers: &[&str]) -> usize { + let max_partial = markers + .iter() + .map(|marker| partial_prefix_len(&text[start..], marker)) + .max() + .unwrap_or(0); + text.len() - max_partial +} + +/// Parse complete Gemma4 custom key-value arguments. +fn parse_gemma4_args(args: &str) -> ModalResult> { + let mut input = args; + terminated(gemma4_args, eof).parse_next(&mut input) +} + +/// Parse Gemma4's custom key-value argument object content. +fn gemma4_args(input: &mut &str) -> ModalResult> { + let pairs: Vec<(String, Value)> = delimited( + ws0, + terminated( + separated(0.., gemma4_pair, comma_separator), + opt(comma_separator), + ), + ws0, + ) + .parse_next(input)?; + Ok(pairs.into_iter().collect()) +} + +/// Parse a Gemma4 key-value pair. +fn gemma4_pair(input: &mut &str) -> ModalResult<(String, Value)> { + let (key, value) = seq!( + _: ws0, + gemma4_key, + _: ws0, + _: literal(":"), + _: ws0, + gemma4_value, + ) + .parse_next(input)?; + Ok((key, value)) +} + +/// Parse a Gemma4 bare key. +fn gemma4_key(input: &mut &str) -> ModalResult { + let key = take_till(1.., |char: char| char == ':').parse_next(input)?.trim(); + if key.is_empty() { + return Err(ErrMode::Cut(ContextError::new())); + } + Ok(key.to_string()) +} + +/// Parse a Gemma4 value. +fn gemma4_value(input: &mut &str) -> ModalResult { + alt(( + gemma4_string.map(|value: &str| Value::String(value.to_string())), + gemma4_object.map(Value::Object), + gemma4_array_value.map(Value::Array), + gemma4_bare_value, + )) + .parse_next(input) +} + +/// Parse a Gemma4 string delimited by `<|"|>`. +fn gemma4_string<'i>(input: &mut &'i str) -> ModalResult<&'i str> { + delimited( + literal(STRING_DELIM), + take_until(0.., STRING_DELIM), + literal(STRING_DELIM), + ) + .parse_next(input) +} + +/// Parse a nested Gemma4 object. +fn gemma4_object(input: &mut &str) -> ModalResult> { + delimited(literal("{"), gemma4_args, literal("}")).parse_next(input) +} + +/// Parse a Gemma4 array value. +fn gemma4_array_value(input: &mut &str) -> ModalResult> { + delimited(literal("["), gemma4_array_content, literal("]")).parse_next(input) +} + +/// Parse Gemma4 array content. +fn gemma4_array_content(input: &mut &str) -> ModalResult> { + delimited( + ws0, + terminated( + separated(0.., gemma4_value, comma_separator), + opt(comma_separator), + ), + ws0, + ) + .parse_next(input) +} + +/// Parse a Gemma4 bare scalar. +fn gemma4_bare_value(input: &mut &str) -> ModalResult { + take_till(1.., |char: char| matches!(char, ',' | '}' | ']')) + .map(parse_gemma4_scalar) + .parse_next(input) +} + +/// Parse a Gemma4 comma separator. +fn comma_separator(input: &mut &str) -> ModalResult<()> { + delimited(ws0, literal(","), ws0).void().parse_next(input) +} + +fn parse_gemma4_scalar(value: &str) -> Value { + let value = value.trim(); + if value.is_empty() { + return Value::String(String::new()); + } + if value == "true" { + return Value::Bool(true); + } + if value == "false" { + return Value::Bool(false); + } + if matches!(value, "null" | "none" | "nil" | "NULL" | "None" | "NIL") { + return Value::Null; + } + if value.contains('.') { + if let Ok(parsed) = value.parse::() + && let Some(number) = Number::from_f64(parsed) + { + return Value::Number(number); + } + } else if let Ok(parsed) = value.parse::() { + return Value::Number(Number::from(parsed)); + } + + Value::String(value.to_string()) +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use serde_json::{Value, json}; + use thiserror_ext::AsReport; + use vllm_tokenizer::test_utils::TestTokenizer; + use winnow::combinator::{eof, terminated}; + use winnow::error::ErrMode; + use winnow::prelude::*; + + use super::{ + CHANNEL_END, CHANNEL_START, Gemma4UnifiedParser, ToolCallDelta, UnifiedParser, + UnifiedParserError, UnifiedParserOutput, gemma4_array_content, parse_gemma4_args, + }; + use crate::tool::Tool; + use crate::unified::{UnifiedParserEvent, parsing_failed}; + + const CHANNEL_START_ID: u32 = 256; + const CHANNEL_END_ID: u32 = 257; + const TURN_BOUNDARY_ID: u32 = 258; + + fn tokenizer() -> TestTokenizer { + TestTokenizer::new() + .with_special_token(CHANNEL_START, CHANNEL_START_ID) + .with_special_token(CHANNEL_END, CHANNEL_END_ID) + .with_special_token("", TURN_BOUNDARY_ID) + } + + trait UnifiedParserTestExt { + fn parse_chunk(&mut self, chunk: &str) -> super::Result; + fn parse_complete(&mut self, text: &str) -> super::Result; + } + + impl UnifiedParserTestExt for Gemma4UnifiedParser { + fn parse_chunk(&mut self, chunk: &str) -> super::Result { + let mut output = UnifiedParserOutput::default(); + self.parse_into(chunk, &mut output)?; + Ok(output) + } + + fn parse_complete(&mut self, text: &str) -> super::Result { + let mut output = self.parse_chunk(text)?; + output.append(self.finish()?); + Ok(output) + } + } + + trait UnifiedOutputTestExt { + fn normal_text(&self) -> String; + fn reasoning_text(&self) -> String; + fn calls(&self) -> Vec<&ToolCallDelta>; + fn coalesce(self) -> Self; + } + + impl UnifiedOutputTestExt for UnifiedParserOutput { + fn normal_text(&self) -> String { + self.events + .iter() + .filter_map(|event| match event { + UnifiedParserEvent::Text(text) => Some(text.as_str()), + UnifiedParserEvent::Reasoning(_) | UnifiedParserEvent::ToolCall(_) => None, + }) + .collect() + } + + fn reasoning_text(&self) -> String { + self.events + .iter() + .filter_map(|event| match event { + UnifiedParserEvent::Reasoning(text) => Some(text.as_str()), + UnifiedParserEvent::Text(_) | UnifiedParserEvent::ToolCall(_) => None, + }) + .collect() + } + + fn calls(&self) -> Vec<&ToolCallDelta> { + self.events + .iter() + .filter_map(|event| match event { + UnifiedParserEvent::Text(_) | UnifiedParserEvent::Reasoning(_) => None, + UnifiedParserEvent::ToolCall(call) => Some(call), + }) + .collect() + } + + fn coalesce(self) -> Self { + self + } + } + + fn parse_gemma4_array(array: &str) -> super::Result> { + let mut input = array; + match terminated(gemma4_array_content, eof).parse_next(&mut input) { + Ok(value) => Ok(value), + Err(ErrMode::Incomplete(_)) => Err(parsing_failed!("incomplete Gemma4 array")), + Err(ErrMode::Backtrack(error) | ErrMode::Cut(error)) => { + Err(parsing_failed!("{}", error)) + } + } + } + + fn test_tools() -> Vec { + vec![ + Tool { + name: "get_weather".to_string(), + description: None, + parameters: json!({ "type": "object" }), + strict: None, + }, + Tool { + name: "get_time".to_string(), + description: None, + parameters: json!({ "type": "object" }), + strict: None, + }, + Tool { + name: "write_file".to_string(), + description: None, + parameters: json!({ "type": "object" }), + strict: None, + }, + Tool { + name: "Edit".to_string(), + description: None, + parameters: json!({ "type": "object" }), + strict: None, + }, + Tool { + name: "search".to_string(), + description: None, + parameters: json!({ "type": "object" }), + strict: None, + }, + Tool { + name: "set".to_string(), + description: None, + parameters: json!({ "type": "object" }), + strict: None, + }, + Tool { + name: "get_status".to_string(), + description: None, + parameters: json!({ "type": "object" }), + strict: None, + }, + Tool { + name: "todowrite".to_string(), + description: None, + parameters: json!({ "type": "object" }), + strict: None, + }, + ] + } + + fn test_parser() -> Gemma4UnifiedParser { + Gemma4UnifiedParser::new(&test_tools(), Arc::new(tokenizer())).unwrap() + } + + #[test] + fn gemma4_create_requires_channel_start_token() { + let error = match Gemma4UnifiedParser::new(&test_tools(), Arc::new(TestTokenizer::new())) { + Ok(_) => panic!("expected missing token error"), + Err(error) => error, + }; + + assert!(matches!( + error, + UnifiedParserError::MissingToken { token } if token == CHANNEL_START + )); + } + + fn collect_stream(chunks: &[&str]) -> UnifiedParserOutput { + let mut parser = test_parser(); + let mut output = UnifiedParserOutput::default(); + for chunk in chunks { + output.append(parser.parse_chunk(chunk).unwrap()); + } + output.append(parser.finish().unwrap()); + output.coalesce() + } + + fn first_call(output: &UnifiedParserOutput) -> ToolCallDelta { + (*output.calls().first().expect("expected one tool call")).clone() + } + + #[test] + fn gemma4_parse_args_handles_scalars_and_nested_values() { + let parsed = parse_gemma4_args( + "name:<|\"|>test<|\"|>,count:42,active:true,score:114.514,nested:{inner:<|\"|>value<|\"|>},items:[<|\"|>a<|\"|>,<|\"|>b<|\"|>]", + ) + .unwrap(); + + assert_eq!( + Value::Object(parsed), + json!({ + "name": "test", + "count": 42, + "active": true, + "score": 114.514, + "nested": { "inner": "value" }, + "items": ["a", "b"], + }) + ); + } + + #[test] + fn gemma4_parse_args_handles_empty_arguments() { + let parsed = parse_gemma4_args("").unwrap(); + assert_eq!(Value::Object(parsed), json!({})); + } + + #[test] + fn gemma4_parse_array_handles_bare_values() { + let parsed = parse_gemma4_array("42,true,114.514").unwrap(); + assert_eq!(Value::Array(parsed), json!([42, true, 114.514])); + } + + #[test] + fn gemma4_parse_complete_extracts_single_tool_call() { + let mut parser = test_parser(); + let output = parser + .parse_complete("<|tool_call>call:get_weather{location:<|\"|>London<|\"|>}") + .unwrap(); + + assert!(output.normal_text().is_empty()); + assert_eq!(output.calls().len(), 1); + assert_eq!(first_call(&output).name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&first_call(&output).arguments).unwrap(), + json!({ "location": "London" }) + ); + } + + #[test] + fn gemma4_parse_complete_rejects_incomplete_tool_call() { + let mut parser = test_parser(); + let error = parser + .parse_complete("<|tool_call>call:get_weather{location:<|\"|>London") + .unwrap_err(); + + assert!(error.to_report_string().contains("incomplete Gemma4 tool call")); + } + + #[test] + fn gemma4_streaming_basic_single_tool_call() { + let output = collect_stream(&[ + "<|tool_call>", + "call:get_weather{", + "location:<|\"|>Paris", + ", France", + "<|\"|>}", + "", + ]); + + assert!(output.normal_text().is_empty()); + assert_eq!(first_call(&output).name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&first_call(&output).arguments).unwrap(), + json!({ "location": "Paris, France" }) + ); + } + + #[test] + fn gemma4_streaming_text_before_and_after_tool_call() { + let output = collect_stream(&[ + "Let me check ", + "the weather. ", + "<|tool_call>", + "call:get_weather{", + "location:<|\"|>London<|\"|>}", + "<", + "div>", + ]); + + assert_eq!(output.normal_text(), "Let me check the weather.
"); + assert_eq!(first_call(&output).name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&first_call(&output).arguments).unwrap(), + json!({ "location": "London" }) + ); + } + + #[test] + fn gemma4_streaming_waits_for_complete_tool_call() { + let mut parser = test_parser(); + let mut output = UnifiedParserOutput::default(); + + for chunk in [ + "<|tool_call>", + "call:get_weather{", + "location:<|\"|>Paris<|\"|>}", + ] { + output.append(parser.parse_chunk(chunk).unwrap()); + assert!(output.calls().is_empty()); + } + + output.append(parser.parse_chunk("").unwrap()); + let output = output.coalesce(); + + assert_eq!(first_call(&output).name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&first_call(&output).arguments).unwrap(), + json!({ "location": "Paris" }) + ); + } + + #[test] + fn gemma4_streaming_handles_boolean_split_across_chunks() { + let output = collect_stream(&[ + "<|tool_call>", + "call:search{input:{all:tru", + "e}}", + "", + ]); + + assert_eq!(first_call(&output).name.as_deref(), Some("search")); + assert_eq!( + serde_json::from_str::(&first_call(&output).arguments).unwrap(), + json!({ "input": { "all": true } }) + ); + } + + #[test] + fn gemma4_streaming_handles_false_split_across_chunks() { + let output = collect_stream(&["<|tool_call>", "call:set{flag:fals", "e}", ""]); + + assert_eq!(first_call(&output).name.as_deref(), Some("set")); + assert_eq!( + serde_json::from_str::(&first_call(&output).arguments).unwrap(), + json!({ "flag": false }) + ); + } + + #[test] + fn gemma4_streaming_handles_number_split_across_chunks() { + let output = collect_stream(&["<|tool_call>", "call:set{count:4", "2}", ""]); + + assert_eq!(first_call(&output).name.as_deref(), Some("set")); + assert_eq!( + serde_json::from_str::(&first_call(&output).arguments).unwrap(), + json!({ "count": 42 }) + ); + } + + #[test] + fn gemma4_streaming_handles_split_string_delimiter() { + let output = collect_stream(&[ + "<|tool_call>", + "call:todowrite{", + "content:<|\"|>Buy milk<|", + "\"|>}", + "", + ]); + + assert_eq!(first_call(&output).name.as_deref(), Some("todowrite")); + assert_eq!( + serde_json::from_str::(&first_call(&output).arguments).unwrap(), + json!({ "content": "Buy milk" }) + ); + assert!(!first_call(&output).arguments.contains("<|")); + } + + #[test] + fn gemma4_streaming_handles_split_tool_call_end_marker() { + let output = collect_stream(&[ + "<|tool_call>", + "call:get_weather{location:<|\"|>Paris<|\"|>}", + ]); + + assert_eq!(first_call(&output).name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&first_call(&output).arguments).unwrap(), + json!({ "location": "Paris" }) + ); + } + + #[test] + fn gemma4_streaming_handles_end_marker_literal_inside_string() { + let output = collect_stream(&[ + "<|tool_call>", + "call:todowrite{", + "content:<|\"|>literal } inside", + "<|\"|>}", + "", + ]); + + assert_eq!(first_call(&output).name.as_deref(), Some("todowrite")); + assert_eq!( + serde_json::from_str::(&first_call(&output).arguments).unwrap(), + json!({ "content": "literal } inside" }) + ); + } + + #[test] + fn gemma4_streaming_handles_html_argument_without_duplication() { + let output = collect_stream(&[ + "<|tool_call>", + "call:write_file{", + "path:<|\"|>index.html<|\"|>,", + "content:<|\"|>\n<", + "html lang=\"zh-CN\">\n<", + "head>\n <", + "meta charset=\"UTF-8\">\n <", + "meta name=\"viewport\" content=\"width=device-width\">\n", + "<|\"|>}", + "", + ]); + + assert_eq!(first_call(&output).name.as_deref(), Some("write_file")); + assert_eq!( + serde_json::from_str::(&first_call(&output).arguments).unwrap(), + json!({ + "path": "index.html", + "content": "\n\n\n \n \n", + }) + ); + } + + #[test] + fn gemma4_streaming_trailing_bare_bool_is_not_duplicated() { + let output = collect_stream(&[ + "<|tool_call>", + "call:Edit{", + "file_path:<|\"|>src/env.py<|\"|>,", + "old_string:<|\"|>old_val<|\"|>,", + "new_string:<|\"|>new_val<|\"|>,", + "replace_all:", + "false}", + "", + ]); + + assert_eq!(first_call(&output).name.as_deref(), Some("Edit")); + assert_eq!( + serde_json::from_str::(&first_call(&output).arguments).unwrap(), + json!({ + "file_path": "src/env.py", + "old_string": "old_val", + "new_string": "new_val", + "replace_all": false, + }) + ); + assert_eq!( + first_call(&output).arguments.matches("replace_all").count(), + 1 + ); + } + + #[test] + fn gemma4_finish_flushes_partial_start_marker_as_text() { + let mut parser = test_parser(); + let mut output = parser.parse_chunk("<").unwrap(); + output.append(parser.finish().unwrap()); + + assert_eq!(output.normal_text(), "<"); + assert!(output.calls().is_empty()); + } + + #[test] + fn gemma4_streaming_emits_reasoning_then_text() { + let output = collect_stream(&["<|channel>thought\nreasonanswer"]); + + assert_eq!(output.reasoning_text(), "reason"); + assert_eq!(output.normal_text(), "answer"); + assert!(output.calls().is_empty()); + } + + #[test] + fn gemma4_streaming_holds_split_reasoning_start() { + let mut parser = test_parser(); + + let first = parser.parse_chunk("<|channel>").unwrap(); + assert!(first.events.is_empty()); + + let mut output = parser.parse_chunk("thought\nrea").unwrap(); + output.append(parser.parse_chunk("sonanswer").unwrap()); + output.append(parser.finish().unwrap()); + + assert_eq!(output.reasoning_text(), "reason"); + assert_eq!(output.normal_text(), "answer"); + } + + #[test] + fn gemma4_initialize_open_channel_prompt_starts_in_reasoning() { + let mut parser = test_parser(); + parser.initialize(&[CHANNEL_START_ID, 3000, 3001]).unwrap(); + + let output = parser.parse_complete("reasonanswer").unwrap(); + + assert_eq!(output.reasoning_text(), "reason"); + assert_eq!(output.normal_text(), "answer"); + } + + #[test] + fn gemma4_initialize_turn_prompt_starts_in_text() { + let mut parser = test_parser(); + parser.initialize(&[TURN_BOUNDARY_ID, 3000, 3001]).unwrap(); + + let output = parser.parse_complete("<|channel>thought\nreasonanswer").unwrap(); + + assert_eq!(output.reasoning_text(), "reason"); + assert_eq!(output.normal_text(), "answer"); + } + + #[test] + fn gemma4_initialize_special_token_caps_boundary_scan() { + let mut parser = test_parser(); + parser.initialize(&[CHANNEL_START_ID, 3000, TURN_BOUNDARY_ID, 3001]).unwrap(); + + let output = parser.parse_complete("answer").unwrap(); + + assert!(output.reasoning_text().is_empty()); + assert_eq!(output.normal_text(), "answer"); + } + + #[test] + fn gemma4_initialize_closed_channel_prompt_starts_in_text() { + let mut parser = test_parser(); + parser.initialize(&[CHANNEL_START_ID, 3000, 3001, CHANNEL_END_ID]).unwrap(); + + let output = parser.parse_complete("answer").unwrap(); + + assert!(output.reasoning_text().is_empty()); + assert_eq!(output.normal_text(), "answer"); + } + + #[test] + fn gemma4_reasoning_tool_call_implicitly_ends_reasoning() { + let output = collect_stream(&[ + "<|channel>thought\nNeed weather.", + "<|tool_call>", + "call:get_weather{location:<|\"|>Paris<|\"|>}", + "", + ]); + + assert_eq!(output.reasoning_text(), "Need weather."); + assert!(output.normal_text().is_empty()); + assert_eq!(first_call(&output).name.as_deref(), Some("get_weather")); + assert_eq!( + serde_json::from_str::(&first_call(&output).arguments).unwrap(), + json!({ "location": "Paris" }) + ); + } + + #[test] + fn gemma4_bare_channel_start_is_plain_text() { + let output = collect_stream(&["<|channel>plain"]); + + assert_eq!(output.normal_text(), "<|channel>plain"); + assert!(output.reasoning_text().is_empty()); + assert!(output.calls().is_empty()); + } + + #[test] + fn gemma4_finish_rejects_complete_args_without_end_marker() { + let mut parser = test_parser(); + for chunk in ["<|tool_call>", "call:get_status{}"] { + parser.parse_chunk(chunk).unwrap(); + } + + let error = parser.finish().unwrap_err(); + + assert!(error.to_report_string().contains("incomplete Gemma4 tool call")); + } + + #[test] + fn gemma4_reset_preserves_internally_buffered_arguments() { + let mut parser = test_parser(); + for chunk in [ + "<|tool_call>", + "call:write_file{", + "content:<|\"|>hello ", + "world<|\"|>", + ] { + parser.parse_chunk(chunk).unwrap(); + } + + let raw = parser.reset(); + + assert_eq!( + raw, + "<|tool_call>call:write_file{content:<|\"|>hello world<|\"|>" + ); + } + + #[test] + fn gemma4_reset_preserves_completed_arguments_after_parse_error() { + let mut parser = test_parser(); + let input = "<|tool_call>call:set{broken}"; + + let _error = parser.parse_chunk(input).unwrap_err(); + let raw = parser.reset(); + + assert_eq!(raw, input); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/unified/mod.rs b/asian_ai/performance/vLLM/rust/src/parser/src/unified/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..ab3f1c6d145ace0381808eb223ea5fc1026da13e --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/unified/mod.rs @@ -0,0 +1,207 @@ +//! Unified parser interface for reasoning and tool-call deltas. + +mod combined; +mod gemma4; + +pub use combined::CombinedParser; +pub use gemma4::Gemma4UnifiedParser; +use thiserror::Error; +use thiserror_ext::Macro; +use vllm_tokenizer::DynTokenizer; + +use crate::reasoning::ReasoningError; +use crate::tool::{ + StructuralTagModel, Tool, ToolCallDelta, ToolParserError, ToolParserEvent, ToolParserOutput, +}; + +/// Result alias for unified parser operations. +pub type Result = std::result::Result; + +/// One parsed event emitted by a unified parser. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum UnifiedParserEvent { + /// Normal assistant-visible text. + Text(String), + /// Reasoning text hidden from the normal content stream. + Reasoning(String), + /// A tool-call update extracted from visible assistant text. + ToolCall(ToolCallDelta), +} + +/// Result of advancing unified parsing with one assistant-text input. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct UnifiedParserOutput { + /// Ordered parser events committed by this input. + pub events: Vec, +} + +impl UnifiedParserOutput { + /// Append one visible text event if `delta` is non-empty. + pub fn push_text(&mut self, delta: impl AsRef + Into) { + if delta.as_ref().is_empty() { + return; + } + if let Some(UnifiedParserEvent::Text(last_text)) = self.events.last_mut() { + last_text.push_str(delta.as_ref()); + return; + } + self.events.push(UnifiedParserEvent::Text(delta.into())); + } + + /// Append one reasoning text event if `delta` is non-empty. + pub fn push_reasoning(&mut self, delta: impl AsRef + Into) { + if delta.as_ref().is_empty() { + return; + } + if let Some(UnifiedParserEvent::Reasoning(last_text)) = self.events.last_mut() { + last_text.push_str(delta.as_ref()); + return; + } + self.events.push(UnifiedParserEvent::Reasoning(delta.into())); + } + + /// Append one tool-call event. + pub fn push_call(&mut self, call: ToolCallDelta) { + self.events.push(UnifiedParserEvent::ToolCall(call)); + } + + /// Append parsed tool parser output as unified events. + pub fn append_tool_output(&mut self, output: ToolParserOutput) { + for event in output.events { + match event { + ToolParserEvent::Text(text) => self.push_text(text), + ToolParserEvent::ToolCall(call) => self.push_call(call), + } + } + } + + /// Append another parser output onto this one. + pub fn append(&mut self, other: Self) { + for event in other.events { + match event { + UnifiedParserEvent::Text(text) => self.push_text(text), + UnifiedParserEvent::Reasoning(reasoning) => self.push_reasoning(reasoning), + UnifiedParserEvent::ToolCall(call) => self.push_call(call), + } + } + } +} + +#[cfg(test)] +mod tests { + use super::{UnifiedParserEvent, UnifiedParserOutput}; + use crate::tool::ToolCallDelta; + + #[test] + fn unified_parser_output_coalesces_adjacent_text_events() { + let mut output = UnifiedParserOutput::default(); + output.push_text("hello"); + output.push_text(" "); + output.push_text("world"); + output.push_reasoning("think"); + output.push_reasoning("ing"); + output.push_call(ToolCallDelta { + tool_index: 0, + name: Some("lookup".to_string()), + arguments: "{}".to_string(), + }); + output.push_text("!"); + + assert_eq!( + output.events, + vec![ + UnifiedParserEvent::Text("hello world".to_string()), + UnifiedParserEvent::Reasoning("thinking".to_string()), + UnifiedParserEvent::ToolCall(ToolCallDelta { + tool_index: 0, + name: Some("lookup".to_string()), + arguments: "{}".to_string(), + }), + UnifiedParserEvent::Text("!".to_string()), + ] + ); + } + + #[test] + fn unified_parser_output_append_coalesces_adjacent_events() { + let mut output = UnifiedParserOutput::default(); + output.push_text("hello"); + + let mut other = UnifiedParserOutput::default(); + other.push_text(" "); + other.push_text("world"); + other.push_reasoning("think"); + output.append(other); + + let mut after_reasoning = UnifiedParserOutput::default(); + after_reasoning.push_reasoning("ing"); + after_reasoning.push_text("!"); + output.append(after_reasoning); + + assert_eq!( + output.events, + vec![ + UnifiedParserEvent::Text("hello world".to_string()), + UnifiedParserEvent::Reasoning("thinking".to_string()), + UnifiedParserEvent::Text("!".to_string()), + ] + ); + } +} + +/// Incremental parser that extracts reasoning and tool-call events from assistant output. +pub trait UnifiedParser: Send { + /// Construct a boxed parser instance for one request stream. + fn create(tools: &[Tool], tokenizer: DynTokenizer) -> Result> + where + Self: Sized + 'static; + + /// Initialize parser state from prompt token IDs before output deltas arrive. + fn initialize(&mut self, _prompt_token_ids: &[u32]) -> Result<()> { + Ok(()) + } + + /// Return whether decoded output must preserve tokenizer special tokens. + fn preserve_special_tokens(&self) -> bool { + false + } + + /// Return the xgrammar structural-tag model used for strict tool calling. + fn structural_tag_model(&self) -> Option { + None + } + + /// Return the parser-provided ID for a tool call by index, if the model emitted one. + fn tool_call_id(&self, _tool_index: usize) -> Option<&str> { + None + } + + /// Feed one decoded text delta into the parser, appending committed output into `output`. + fn parse_into(&mut self, delta: &str, output: &mut UnifiedParserOutput) -> Result<()>; + + /// Flush any buffered parser state at end of stream. + fn finish(&mut self) -> Result { + Ok(UnifiedParserOutput::default()) + } + + /// Clear parser state and return currently uncommitted buffered text. + fn reset(&mut self) -> String { + String::new() + } +} + +/// Errors produced while creating or running unified parsers. +#[derive(Debug, Error, Macro)] +#[thiserror_ext(macro(path = "crate::unified", mangle))] +pub enum UnifiedParserError { + #[error("combined parser is constructed from split parser instances")] + CombinedParserConstructor, + #[error("tokenizer is missing unified parser token `{token}`")] + MissingToken { token: String }, + #[error("unified parser parsing failed: {message}")] + ParsingFailed { message: String }, + #[error(transparent)] + Reasoning(#[from] ReasoningError), + #[error(transparent)] + Tool(#[from] ToolParserError), +} diff --git a/asian_ai/performance/vLLM/rust/src/parser/src/utils.rs b/asian_ai/performance/vLLM/rust/src/parser/src/utils.rs new file mode 100644 index 0000000000000000000000000000000000000000..4070e9e8c33b0a2476bf3b415b840c74ac6c5408 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/parser/src/utils.rs @@ -0,0 +1,859 @@ +//! Shared helpers for streaming parsers. + +use winnow::Parser; +use winnow::error::{ContextError, ErrMode, ModalResult, Needed, StrContext, StrContextValue}; +use winnow::stream::{FindSlice, Offset, Partial, Stream}; + +use crate::tool::{Result, ToolParserError}; + +/// Return the byte length of the longest proper prefix of `token` that is also +/// a suffix of `buffer`. +/// +/// Streaming parsers use this to keep only the trailing fragment that might +/// still grow into a full marker after the next decoded chunk arrives. +/// +/// The returned length is always a valid UTF-8 boundary in `token`, so callers +/// can safely slice `&token[..len]` even when markers contain non-ASCII +/// characters such as DeepSeek's DSML delimiters. +pub fn partial_prefix_len(buffer: &str, token: &str) -> usize { + let Some(first_byte) = token.as_bytes().first().copied() else { + return 0; + }; + + let max_len = buffer.len().min(token.len().saturating_sub(1)); + let tail_start = buffer.len() - max_len; + let buffer_bytes = buffer.as_bytes(); + let token_bytes = token.as_bytes(); + + // Scan from the longest possible suffix to preserve overlapping prefixes. + for index in tail_start..buffer.len() { + if buffer_bytes[index] != first_byte { + continue; + } + + let len = buffer.len() - index; + if buffer.is_char_boundary(index) + && token.is_char_boundary(len) + && token_bytes[..len] == buffer_bytes[index..] + { + return len; + } + } + + 0 +} + +/// Parse a safe text run before the next marker. +/// This is the single-marker variant of [`safe_text_len_mul`]. +/// +/// Returns the text length in bytes, and advances the input. +pub fn safe_text_len(input: &mut Partial<&str>, marker: &str) -> ModalResult { + let text = **input; + if text.is_empty() { + return incomplete(); + } + + if let Some(start_idx) = text.find(marker) { + input.next_slice(start_idx); + return Ok(start_idx); + } + + let keep_len = partial_prefix_len(text, marker); + let emit_len = text.len().saturating_sub(keep_len); + if emit_len == 0 { + return incomplete(); + } + + input.next_slice(emit_len); + Ok(emit_len) +} + +/// Parse a safe text run before the earliest next marker. +/// This is the multi-marker variant of [`safe_text_len`]. +/// +/// Returns the text length in bytes, and advances the input. +pub fn safe_text_len_mul(input: &mut Partial<&str>, markers: &[&str]) -> ModalResult { + let text = **input; + if text.is_empty() { + return incomplete(); + } + + if let Some(start_idx) = find_slice_mul(text, markers) { + input.next_slice(start_idx); + return Ok(start_idx); + } + + let keep_len = markers.iter().map(|marker| partial_prefix_len(text, marker)).max().unwrap_or(0); + let emit_len = text.len().saturating_sub(keep_len); + if emit_len == 0 { + return incomplete(); + } + + input.next_slice(emit_len); + Ok(emit_len) +} + +#[inline(always)] +fn find_slice_mul(text: &str, markers: &[&str]) -> Option { + let range = match markers { + // Use the fast specialized `winnow::stream::FindSlice` impl for 1-3 markers. + [first] => text.find_slice(*first), + [first, second] => text.find_slice((*first, *second)), + [first, second, third] => text.find_slice((*first, *second, *third)), + // Fall back to a linear scan for 4+ markers. + _ => return markers.iter().filter_map(|marker| text.find(marker)).min(), + }; + range.map(|range| range.start) +} + +/// Streaming scan state for a buffered marker search [`take_until_marker`], +/// so that we don't have to rescan the whole buffered prefix when resuming. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct MarkerScanState { + scan_start: usize, +} + +impl MarkerScanState { + pub fn reset(&mut self) { + self.scan_start = 0; + } +} + +/// Parse text until `marker`, resuming from the last safe scan checkpoint. +/// +/// This is the streaming-buffered variant of `winnow::token::take_until(0.., +/// marker)`: it returns the slice before `marker` and leaves `marker` for the +/// caller to consume. On incomplete input, it stores the earliest byte offset +/// that can still match `marker` and returns `Incomplete` without consuming +/// input, so the next parse can avoid rescanning the whole buffered prefix. +/// +/// Use this for outer parser states that keep the full buffered input across +/// chunks while waiting for a closing marker. Plain `take_until` is still a +/// better fit for one-shot parsers over a complete body, and for `1..` cases +/// where an empty slice before the marker should be rejected. +pub fn take_until_marker<'i, 'a>( + marker: &'a str, + state: &'a mut MarkerScanState, +) -> impl Parser, &'i str, ErrMode> + 'a { + move |input: &mut Partial<&'i str>| take_until_marker_(input, marker, state) +} + +fn take_until_marker_<'i>( + input: &mut Partial<&'i str>, + marker: &str, + state: &mut MarkerScanState, +) -> ModalResult<&'i str> { + debug_assert!(!marker.is_empty()); + + let text = **input; + if text.is_empty() { + return incomplete(); + } + + // Normal updates store a char boundary; this keeps stale or misused state from panicking. + let scan_start = floor_char_boundary(text, state.scan_start); + + if let Some(offset) = text[scan_start..].find(marker) { + let marker_start = scan_start + offset; + let body = &text[..marker_start]; + input.next_slice(marker_start); + state.reset(); + return Ok(body); + } + + let keep_len = partial_prefix_len(text, marker); + state.scan_start = text.len() - keep_len; + incomplete() +} + +fn floor_char_boundary(text: &str, index: usize) -> usize { + let mut index = index.min(text.len()); + while !text.is_char_boundary(index) { + index -= 1; + } + index +} + +/// Streaming lexical state for a top-level JSON object. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct JsonObjectScanState { + object_depth: usize, + array_depth: usize, + in_string: bool, + escape: bool, + phase: JsonObjectScanPhase, +} + +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +enum JsonObjectScanPhase { + #[default] + Initial, + Scanning, + Complete, +} + +impl JsonObjectScanState { + /// Returns whether the top-level JSON object has closed. + pub const fn complete(&self) -> bool { + matches!(self.phase, JsonObjectScanPhase::Complete) + } +} + +/// Parse a raw top-level JSON object argument prefix. +/// +/// The returned length is safe to emit as raw argument text. This scans only +/// lexical boundaries from `{` through the matching `}`, preserving +/// malformed-but-balanced JSON without deserializing or normalizing it. +pub fn take_json_object( + input: &mut Partial<&str>, + state: &mut JsonObjectScanState, +) -> ModalResult { + let text = **input; + if text.is_empty() { + return incomplete(); + } + if state.complete() { + return Err(json_scan_error( + "JSON object argument", + StrContextValue::Description("active JSON object scan"), + )); + } + + let bytes = text.as_bytes(); + let just_started = matches!(state.phase, JsonObjectScanPhase::Initial); + if just_started { + if bytes[0] != b'{' { + return Err(json_scan_error( + "JSON object argument", + StrContextValue::CharLiteral('{'), + )); + } + state.phase = JsonObjectScanPhase::Scanning; + state.object_depth = 1; + } + + let mut index = usize::from(just_started); + + while index < bytes.len() { + let byte = bytes[index]; + index += 1; + + if state.in_string { + if state.escape { + state.escape = false; + } else if byte == b'\\' { + state.escape = true; + } else if byte == b'"' { + state.in_string = false; + } + continue; + } + + match byte { + b'"' => state.in_string = true, + b'{' => state.object_depth += 1, + b'}' => { + state.object_depth = state.object_depth.checked_sub(1).ok_or_else(|| { + json_scan_error( + "JSON object argument", + StrContextValue::Description("balanced object braces"), + ) + })?; + if state.object_depth == 0 && state.array_depth == 0 { + state.phase = JsonObjectScanPhase::Complete; + input.next_slice(index); + return Ok(index); + } + if state.object_depth == 0 { + return Err(json_scan_error( + "JSON object argument", + StrContextValue::Description( + "nested arrays to close before the top-level object", + ), + )); + } + } + b'[' => state.array_depth += 1, + b']' => { + state.array_depth = state.array_depth.checked_sub(1).ok_or_else(|| { + json_scan_error( + "JSON object argument", + StrContextValue::Description("balanced array brackets"), + ) + })?; + } + _ => {} + } + } + + input.next_slice(text.len()); + Ok(text.len()) +} + +/// Streaming lexical state for a JSON string literal. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct JsonStringScanState { + scanned_len: usize, + escape: bool, +} + +/// Parse a raw JSON string literal, resuming from the last scanned byte. +/// +/// The returned length covers the quoted JSON string. This only scans for the +/// string boundary; callers that need the decoded value should pass the raw +/// slice to [`decode_json_str`]. +pub fn take_json_string( + input: &mut Partial<&str>, + state: &mut JsonStringScanState, +) -> ModalResult { + let text = **input; + if text.is_empty() { + return incomplete(); + } + + let bytes = text.as_bytes(); + if bytes[0] != b'"' { + return Err(json_scan_error( + "JSON string", + StrContextValue::CharLiteral('"'), + )); + } + + let mut index = if state.scanned_len == 0 { + 1 + } else if state.scanned_len <= bytes.len() { + state.scanned_len + } else { + return incomplete(); + }; + + while index < bytes.len() { + let byte = bytes[index]; + index += 1; + + if state.escape { + state.escape = false; + continue; + } + + match byte { + b'\\' => state.escape = true, + b'"' => { + input.next_slice(index); + return Ok(index); + } + _ => {} + } + } + + state.scanned_len = text.len(); + incomplete() +} + +/// Parse a JSON string literal. +pub fn json_str(input: &mut Partial<&str>) -> ModalResult { + let text = **input; + let checkpoint = input.checkpoint(); + let mut state = JsonStringScanState::default(); + let len = take_json_string(input, &mut state)?; + decode_json_str(&text[..len]).inspect_err(|_| { + input.reset(&checkpoint); + }) +} + +/// Decode a complete JSON string literal. +pub fn decode_json_str(raw: &str) -> ModalResult { + serde_json::from_str::(raw).map_err(|_| { + json_scan_error( + "JSON string", + StrContextValue::Description("valid JSON string"), + ) + }) +} + +fn json_scan_error(label: &'static str, expected: StrContextValue) -> ErrMode { + let mut error = ContextError::new(); + error.push(StrContext::Label(label)); + error.push(StrContext::Expected(expected)); + ErrMode::Cut(error) +} + +/// Parse one event from a buffered streaming input. +/// +/// Returns: +/// - `Ok(Some((event, consumed_len)))` if an event was successfully parsed, along with the number +/// of bytes consumed from the buffer. +/// - `Ok(None)` if the buffer does not contain a full event yet, and more data is needed. +/// - `Err` if a parsing error occurred. +pub fn parse_buffered_event( + buffer: &str, + parse: impl FnOnce(&mut Partial<&str>) -> ModalResult, +) -> Result> { + let mut input = Partial::new(buffer); + let checkpoint = input.checkpoint(); + let event = match parse(&mut input) { + Ok(event) => event, + Err(ErrMode::Incomplete(_)) => return Ok(None), + Err(ErrMode::Backtrack(e) | ErrMode::Cut(e)) => { + let snippet = buffer.char_indices().nth(80).map_or(buffer, |(i, _)| &buffer[..i]); + return Err(ToolParserError::ParsingFailed { + message: format!("near {snippet:?}: {e}"), + }); + } + }; + let consumed_len = input.offset_from(&checkpoint); + if consumed_len == 0 { + return Ok(None); + } + + Ok(Some((event, consumed_len))) +} + +/// Returns an error indicating that we need more data to continue parsing. +pub fn incomplete() -> ModalResult { + Err(ErrMode::Incomplete(Needed::Unknown)) +} + +#[cfg(test)] +mod tests { + + use expect_test::expect; + use winnow::Parser; + use winnow::error::ErrMode; + use winnow::stream::{Offset, Partial, Stream}; + + use super::{ + JsonObjectScanState, JsonStringScanState, MarkerScanState, json_str, parse_buffered_event, + partial_prefix_len, safe_text_len, safe_text_len_mul, take_json_object, take_json_string, + take_until_marker, + }; + + #[test] + fn partial_prefix_len_handles_ascii_markers() { + assert_eq!( + partial_prefix_len("hello<|tool", "<|tool_call>"), + "<|tool".len() + ); + assert_eq!(partial_prefix_len("hello world", "<|tool_call>"), 0); + } + + #[test] + fn partial_prefix_len_prefers_longest_overlapping_prefix() { + assert_eq!(partial_prefix_len("chunk ending in aba", "ababa"), 3); + } + + #[test] + fn partial_prefix_len_handles_unicode_markers() { + let token = "<|DSML|function_calls>"; + assert_eq!( + partial_prefix_len("prefix <|DSML|fun", token), + "<|DSML|fun".len() + ); + assert_eq!(partial_prefix_len("prefix <|DSML", token), "<|DSML".len()); + } + + #[test] + fn safe_text_len_stops_before_marker() { + let mut input = Partial::new("hello"); + let checkpoint = input.checkpoint(); + + let len = safe_text_len(&mut input, "").unwrap(); + + assert_eq!(len, "hello".len()); + assert_eq!(input.offset_from(&checkpoint), "hello".len()); + } + + #[test] + fn safe_text_len_holds_back_partial_marker() { + let mut input = Partial::new("hello").unwrap(); + + assert_eq!(len, "hello".len()); + assert_eq!(input.offset_from(&checkpoint), "hello".len()); + } + + #[test] + fn safe_text_len_reports_incomplete_for_only_partial_marker() { + let mut input = Partial::new("").unwrap_err(); + + assert!(matches!(error, ErrMode::Incomplete(_))); + } + + #[test] + fn safe_text_len_mul_stops_before_earliest_marker() { + let mut input = Partial::new("hello<|tool_call>"); + let checkpoint = input.checkpoint(); + + let len = safe_text_len_mul(&mut input, &["<|tool_call>", ""]).unwrap(); + + assert_eq!(len, "hello".len()); + assert_eq!(input.offset_from(&checkpoint), "hello".len()); + assert_eq!(*input, "<|tool_call>"); + } + + #[test] + fn safe_text_len_mul_holds_back_longest_partial_marker() { + let mut input = Partial::new("hello<|tool"); + let checkpoint = input.checkpoint(); + + let len = safe_text_len_mul(&mut input, &["<|tool_call>", "<|channel>thought\n"]).unwrap(); + + assert_eq!(len, "hello".len()); + assert_eq!(input.offset_from(&checkpoint), "hello".len()); + assert_eq!(*input, "<|tool"); + } + + #[test] + fn safe_text_len_mul_skips_false_same_prefix_candidate() { + let mut input = Partial::new("hello<|tool_call>"); + let checkpoint = input.checkpoint(); + + let len = safe_text_len_mul(&mut input, &["<|tool_call>", "<|channel>thought\n"]).unwrap(); + + assert_eq!(len, "hello".len()); + assert_eq!(input.offset_from(&checkpoint), "hello".len()); + assert_eq!(*input, "<|tool_call>"); + } + + #[test] + fn safe_text_len_mul_reports_incomplete_for_only_partial_marker() { + let mut input = Partial::new("<|channel>thought"); + + let error = + safe_text_len_mul(&mut input, &["<|tool_call>", "<|channel>thought\n"]).unwrap_err(); + + assert!(matches!(error, ErrMode::Incomplete(_))); + } + + #[test] + fn take_until_marker_stops_before_marker() { + let mut state = MarkerScanState::default(); + let mut input = Partial::new("bodytail"); + let checkpoint = input.checkpoint(); + + let body = take_until_marker("", &mut state).parse_next(&mut input).unwrap(); + + assert_eq!(body, "body"); + assert_eq!(input.offset_from(&checkpoint), "body".len()); + assert_eq!(*input, "tail"); + assert_eq!(state, MarkerScanState::default()); + } + + #[test] + fn take_until_marker_resumes_after_split_marker() { + let mut state = MarkerScanState::default(); + let mut input = Partial::new("body", &mut state) + .parse_next(&mut input) + .unwrap_err(); + + assert!(matches!(error, ErrMode::Incomplete(_))); + assert_eq!(state.scan_start, "body".len()); + + let mut input = Partial::new("bodytail"); + let body = take_until_marker("", &mut state).parse_next(&mut input).unwrap(); + + assert_eq!(body, "body"); + assert_eq!(*input, "tail"); + assert_eq!(state, MarkerScanState::default()); + } + + #[test] + fn take_until_marker_advances_checkpoint_for_long_prefix() { + let mut state = MarkerScanState::default(); + let text = format!("{}{}", "x".repeat(1024), "", &mut state) + .parse_next(&mut input) + .unwrap_err(); + + assert!(matches!(error, ErrMode::Incomplete(_))); + assert_eq!(state.scan_start, 1024); + } + + #[test] + fn take_until_marker_keeps_unicode_marker_boundaries() { + let marker = "<|DSML|function_calls>"; + let mut state = MarkerScanState::default(); + let mut input = Partial::new("prefix <|DSML|fun"); + + let error = take_until_marker(marker, &mut state).parse_next(&mut input).unwrap_err(); + + assert!(matches!(error, ErrMode::Incomplete(_))); + assert_eq!(state.scan_start, "prefix ".len()); + assert!("prefix <|DSML|fun".is_char_boundary(state.scan_start)); + + let mut input = Partial::new("prefix <|DSML|function_calls>tail"); + let body = take_until_marker(marker, &mut state).parse_next(&mut input).unwrap(); + + assert_eq!(body, "prefix "); + assert_eq!(*input, "<|DSML|function_calls>tail"); + assert_eq!(state, MarkerScanState::default()); + } + + #[test] + fn take_until_marker_floors_stale_checkpoint_to_char_boundary() { + let mut state = MarkerScanState { scan_start: 1 }; + let mut input = Partial::new("é"); + + let body = take_until_marker("", &mut state).parse_next(&mut input).unwrap(); + + assert_eq!(body, "é"); + assert_eq!(*input, ""); + assert_eq!(state, MarkerScanState::default()); + } + + #[test] + fn take_until_marker_handles_overlapping_prefixes() { + let mut state = MarkerScanState::default(); + let mut input = Partial::new("xxaba"); + + let error = take_until_marker("ababa", &mut state).parse_next(&mut input).unwrap_err(); + + assert!(matches!(error, ErrMode::Incomplete(_))); + assert_eq!(state.scan_start, 2); + + let mut input = Partial::new("xxababa!"); + let body = take_until_marker("ababa", &mut state).parse_next(&mut input).unwrap(); + + assert_eq!(body, "xx"); + assert_eq!(*input, "ababa!"); + } + + #[test] + fn take_json_object_consumes_simple_object() { + let mut state = JsonObjectScanState::default(); + let buffer = r#"{"location":"Paris"}"#; + let mut input = Partial::new(buffer); + let checkpoint = input.checkpoint(); + + let len = take_json_object(&mut input, &mut state).unwrap(); + + assert_eq!(len, r#"{"location":"Paris"}"#.len()); + assert_eq!(input.offset_from(&checkpoint), len); + assert!(state.complete()); + } + + #[test] + fn take_json_object_tracks_nested_values_and_strings() { + let mut state = JsonObjectScanState::default(); + let arguments = r#"{"nested":{"items":[{"text":"} <|tool_call_end|> \" \\"}]}}"#; + let buffer = format!("{arguments}"); + let mut input = Partial::new(buffer.as_str()); + + let len = take_json_object(&mut input, &mut state).unwrap(); + + assert_eq!(len, arguments.len()); + assert!(state.complete()); + } + + #[test] + fn take_json_object_rejects_leading_whitespace() { + let mut state = JsonObjectScanState::default(); + let mut input = Partial::new(" {\"x\":1}"); + + let error = take_json_object(&mut input, &mut state).unwrap_err(); + + let ErrMode::Cut(error) = error else { + panic!("expected cut error"); + }; + expect![[r#" + invalid JSON object argument + expected `{`"#]] + .assert_eq(&error.to_string()); + } + + #[test] + fn take_json_object_leaves_trailing_whitespace_to_caller() { + let mut state = JsonObjectScanState::default(); + let mut input = Partial::new("{\"x\":1}\n"); + let checkpoint = input.checkpoint(); + + let len = take_json_object(&mut input, &mut state).unwrap(); + + assert_eq!(len, "{\"x\":1}".len()); + assert_eq!(input.offset_from(&checkpoint), len); + assert!(state.complete()); + } + + #[test] + fn take_json_object_continues_across_chunks() { + let mut state = JsonObjectScanState::default(); + let chunks = [ + r#"{"text":"literal "#, + r#"<|tool_call_end|>"#, + r#" inside"}"#, + ]; + let mut collected = String::new(); + + for chunk in chunks { + let mut input = Partial::new(chunk); + let len = take_json_object(&mut input, &mut state).unwrap(); + collected.push_str(&chunk[..len]); + } + + assert_eq!(collected, r#"{"text":"literal <|tool_call_end|> inside"}"#); + assert!(state.complete()); + } + + #[test] + fn take_json_object_rejects_non_object_top_level() { + let mut state = JsonObjectScanState::default(); + let mut input = Partial::new(r#"[{"x":1}]"#); + + let error = take_json_object(&mut input, &mut state).unwrap_err(); + + let ErrMode::Cut(error) = error else { + panic!("expected cut error"); + }; + expect![[r#" + invalid JSON object argument + expected `{`"#]] + .assert_eq(&error.to_string()); + } + + #[test] + fn take_json_object_reports_unbalanced_array() { + let mut state = JsonObjectScanState::default(); + let mut input = Partial::new(r#"{"x":]}"#); + + let error = take_json_object(&mut input, &mut state).unwrap_err(); + + let ErrMode::Cut(error) = error else { + panic!("expected cut error"); + }; + expect![[r#" + invalid JSON object argument + expected balanced array brackets"#]] + .assert_eq(&error.to_string()); + } + + #[test] + fn take_json_object_reports_top_level_close_before_nested_array() { + let mut state = JsonObjectScanState::default(); + let mut input = Partial::new(r#"{"x":[}"#); + + let error = take_json_object(&mut input, &mut state).unwrap_err(); + + let ErrMode::Cut(error) = error else { + panic!("expected cut error"); + }; + expect![[r#" + invalid JSON object argument + expected nested arrays to close before the top-level object"#]] + .assert_eq(&error.to_string()); + } + + #[test] + fn take_json_string_consumes_complete_string() { + let mut state = JsonStringScanState::default(); + let mut input = Partial::new(r#""say_\"hi\u0021" rest"#); + let checkpoint = input.checkpoint(); + + let len = take_json_string(&mut input, &mut state).unwrap(); + + assert_eq!(len, r#""say_\"hi\u0021""#.len()); + assert_eq!(input.offset_from(&checkpoint), len); + assert_eq!(*input, " rest"); + } + + #[test] + fn take_json_string_resumes_after_incomplete_input() { + let mut state = JsonStringScanState::default(); + let mut input = Partial::new(r#""{\"data\":\"partial"#); + let checkpoint = input.checkpoint(); + + let error = take_json_string(&mut input, &mut state).unwrap_err(); + + assert!(matches!(error, ErrMode::Incomplete(_))); + assert_eq!(input.offset_from(&checkpoint), 0); + assert_eq!(state.scanned_len, r#""{\"data\":\"partial"#.len()); + + let mut input = Partial::new(r#""{\"data\":\"partial string\"}" tail"#); + let len = take_json_string(&mut input, &mut state).unwrap(); + + assert_eq!(len, r#""{\"data\":\"partial string\"}""#.len()); + assert_eq!(*input, " tail"); + } + + #[test] + fn take_json_string_tracks_escape_across_chunks() { + let mut state = JsonStringScanState::default(); + let mut input = Partial::new(r#""abc\"#); + + let error = take_json_string(&mut input, &mut state).unwrap_err(); + + assert!(matches!(error, ErrMode::Incomplete(_))); + assert!(state.escape); + + let mut input = Partial::new(r#""abc\"def" tail"#); + let len = take_json_string(&mut input, &mut state).unwrap(); + + assert_eq!(len, r#""abc\"def""#.len()); + assert_eq!(*input, " tail"); + } + + #[test] + fn take_json_string_rejects_non_string_start() { + let mut state = JsonStringScanState::default(); + let mut input = Partial::new("42"); + + let error = take_json_string(&mut input, &mut state).unwrap_err(); + + let ErrMode::Cut(error) = error else { + panic!("expected cut error"); + }; + expect![[r#" + invalid JSON string + expected `"`"#]] + .assert_eq(&error.to_string()); + } + + #[test] + fn json_str_decodes_escaped_content() { + let mut input = Partial::new(r#""say_\"hi\u0021" rest"#); + + let value = json_str(&mut input).unwrap(); + + assert_eq!(value, "say_\"hi!"); + assert_eq!(*input, " rest"); + } + + #[test] + fn json_str_reports_incomplete_escaped_string() { + let mut input = Partial::new(r#""say_\"#); + + let error = json_str(&mut input).unwrap_err(); + + assert!(matches!(error, ErrMode::Incomplete(_))); + } + + #[test] + fn parse_buffered_event_error_includes_input_snippet() { + let result = parse_buffered_event(" {\"x\":1}", |input| { + take_json_object(input, &mut JsonObjectScanState::default()) + }); + let err = result.unwrap_err().to_string(); + assert!(err.contains("near \""), "error must include snippet"); + } + + #[test] + fn parse_buffered_event_error_truncates_long_input() { + let long_input = format!(" {}", "x".repeat(100)); + let result = parse_buffered_event(&long_input, |input| { + take_json_object(input, &mut JsonObjectScanState::default()) + }); + let err = result.unwrap_err().to_string(); + assert!(err.contains("near \""), "error must include snippet"); + assert!( + !err.contains(&long_input), + "snippet must be truncated for long input" + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/Cargo.toml b/asian_ai/performance/vLLM/rust/src/server/Cargo.toml new file mode 100644 index 0000000000000000000000000000000000000000..965de2e7b0da4c6ffb8efbb306284f7e312d8dd6 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/Cargo.toml @@ -0,0 +1,71 @@ +[package] +name = "vllm-server" +version.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +anyhow.workspace = true +asynk-strim-attr.workspace = true +auto_enums.workspace = true +axum.workspace = true +educe.workspace = true +futures.workspace = true +http-body.workspace = true +hyper.workspace = true +hyper-util.workspace = true +indexmap.workspace = true +itertools.workspace = true +libc.workspace = true +llm-multimodal.workspace = true +openssl.workspace = true +prost.workspace = true +prost-types.workspace = true +rmpv.workspace = true +serde.workspace = true +serde_json.workspace = true +serde_with.workspace = true +sha2.workspace = true +socket2.workspace = true +subtle.workspace = true +thiserror-ext.workspace = true +tls-listener.workspace = true +tokio.workspace = true +tokio-openssl.workspace = true +tokio-stream.workspace = true +tokio-util.workspace = true +tonic.workspace = true +tonic-prost.workspace = true +tower.workspace = true +tower-http.workspace = true +tracing.workspace = true +tracing-futures.workspace = true +tracing-subscriber.workspace = true +uuid.workspace = true +validator.workspace = true +vllm-chat.workspace = true +vllm-engine-core-client.workspace = true +vllm-llm.workspace = true +vllm-metrics.workspace = true +vllm-text.workspace = true + +[build-dependencies] +tonic-prost-build.workspace = true + +[dev-dependencies] +anyhow.workspace = true +async-openai = { workspace = true, features = ["full"] } +bytes.workspace = true +clap.workspace = true +expect-test.workspace = true +rmp-serde.workspace = true +serial_test.workspace = true +tempfile.workspace = true +tokio = { workspace = true, features = ["test-util"] } +tower.workspace = true +vllm-engine-core-client = { workspace = true, features = ["test-util"] } +vllm-tokenizer = { workspace = true, features = ["test-utils"] } +zeromq.workspace = true + +[lints] +workspace = true diff --git a/asian_ai/performance/vLLM/rust/src/server/build.rs b/asian_ai/performance/vLLM/rust/src/server/build.rs new file mode 100644 index 0000000000000000000000000000000000000000..0b652b2cce016abeef1afdc07c73108ae428163c --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/build.rs @@ -0,0 +1,12 @@ +fn main() -> Result<(), Box> { + let manifest_dir = std::env::var("CARGO_MANIFEST_DIR").unwrap(); + let proto_dir = format!("{manifest_dir}/../../proto"); + + tonic_prost_build::configure() + .build_server(true) + .build_client(true) + .protoc_arg("--experimental_allow_proto3_optional") // be compatible with old compilers + .compile_protos(&[format!("{proto_dir}/vllm_grpc.proto")], &[proto_dir])?; + + Ok(()) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/examples/README.md b/asian_ai/performance/vLLM/rust/src/server/examples/README.md new file mode 100644 index 0000000000000000000000000000000000000000..5341440e3609006f8a61c220380fa0813d4e60d4 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/examples/README.md @@ -0,0 +1,39 @@ +# Server Smoke Test + +Start a fresh headless `vllm` engine: + +```bash +source ../vllm/.venv/bin/activate +HF_HUB_OFFLINE=1 \ +VLLM_LOGGING_LEVEL=DEBUG \ +VLLM_CPU_KVCACHE_SPACE=2 \ +VLLM_HOST_IP=127.0.0.1 \ +VLLM_LOOPBACK_IP=127.0.0.1 \ +python3 -m vllm.entrypoints.cli.main serve Qwen/Qwen3-0.6B \ + --headless \ + --data-parallel-address 127.0.0.1 \ + --data-parallel-rpc-port 62100 \ + --data-parallel-size-local 1 \ + --max-model-len 512 \ + --dtype float16 +``` + +Run the Rust server smoke test: + +```bash +cargo run -p vllm-server --example external_engine_openai_qwen -- \ + --handshake-address tcp://127.0.0.1:62100 +``` + +The example starts the Rust OpenAI-compatible server on an ephemeral local port, +connects to it via the `async-openai` Rust client, lists models, and then checks +that a streamed chat completion yields the assistant role chunk, final-answer +content chunks, and a terminal finish chunk. This example intentionally uses +`async-openai`'s standard typed `create_stream` API instead of BYOT, so it does +not inspect the nonstandard `reasoning_content` field even though the Rust +server may emit it for reasoning-capable models such as Qwen3. For reasoning +behavior itself, use the `vllm-chat` smoke test or the `vllm-server` +route tests. + +IMPORTANT: Restart `vllm` each time you run the smoke test. The current headless +engine cannot safely handle frontend reconnects after the client shuts down. diff --git a/asian_ai/performance/vLLM/rust/src/server/examples/external_engine_openai_qwen.rs b/asian_ai/performance/vLLM/rust/src/server/examples/external_engine_openai_qwen.rs new file mode 100644 index 0000000000000000000000000000000000000000..c716da49bcd5ef100bef583d67e022dcc003572b --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/examples/external_engine_openai_qwen.rs @@ -0,0 +1,222 @@ +use std::time::Duration; + +use anyhow::{Context, Result, bail}; +use async_openai::Client; +use async_openai::config::OpenAIConfig; +use async_openai::types::chat::{ + ChatCompletionRequestUserMessageArgs, CreateChatCompletionRequest, + CreateChatCompletionRequestArgs, +}; +use async_openai::types::models::ListModelResponse; +use clap::Parser; +use futures::StreamExt as _; +use tokio_util::sync::CancellationToken; +use tracing_subscriber::EnvFilter; +use vllm_engine_core_client::TransportMode; +use vllm_server::{ + ApiServerOptions, ChatTemplateContentFormatOption, Config, CoordinatorMode, CorsConfig, + HttpListenerMode, ParserSelection, RendererSelection, serve, +}; + +#[derive(Debug, Parser)] +#[command( + about = "Smoke-test the Rust OpenAI server with async-openai against an external Qwen vLLM engine." +)] +struct Args { + #[arg(long)] + handshake_address: String, + #[arg(long, default_value_t = 1)] + engine_count: usize, + #[arg(long, default_value = "Qwen/Qwen3-0.6B")] + model: String, + #[arg(long, default_value = "127.0.0.1")] + host: String, + #[arg(long, default_value_t = 30)] + ready_timeout_secs: u64, + #[arg( + long, + default_value = "What is the capital of France? Answer with one word." + )] + prompt: String, +} + +#[tokio::main(flavor = "multi_thread")] +async fn main() -> Result<()> { + init_tracing(); + let args = Args::parse(); + let port = unique_local_port()?; + let config = Config { + transport_mode: TransportMode::HandshakeOwner { + handshake_address: args.handshake_address, + advertised_host: args.host, + engine_count: args.engine_count, + ready_timeout: Duration::from_secs(args.ready_timeout_secs), + local_input_address: None, + local_output_address: None, + }, + coordinator_mode: CoordinatorMode::MaybeInProc, + model: args.model, + served_model_name: vec![], + listener_mode: HttpListenerMode::BindTcp { + host: "127.0.0.1".to_string(), + port, + }, + tool_call_parser: ParserSelection::Auto, + reasoning_parser: ParserSelection::Auto, + renderer: RendererSelection::Auto, + language_model_only: false, + chat_template: None, + default_chat_template_kwargs: None, + chat_template_content_format: ChatTemplateContentFormatOption::Auto, + max_logprobs: None, + api_server_options: ApiServerOptions::default(), + cors: CorsConfig::default(), + tls: None, + api_keys: Vec::new(), + disable_log_stats: false, + grpc_port: None, + shutdown_timeout: Duration::ZERO, + keep_alive_timeout: Duration::from_secs(5), + profiler: None, + }; + + let bind_address = format!("127.0.0.1:{port}"); + let shutdown = CancellationToken::new(); + let server_config = config.clone(); + let server_shutdown = shutdown.clone(); + let server_task = tokio::spawn(async move { serve(server_config, server_shutdown).await }); + + let client = Client::with_config( + OpenAIConfig::new() + .with_api_key("unused") + .with_api_base(format!("http://{bind_address}/v1")), + ); + + print_models(&client).await?; + let final_text = stream_completion(&client, &config.model, &args.prompt).await?; + + println!(); + println!("final_text={final_text:?}"); + + shutdown_server(server_task, shutdown).await +} + +fn init_tracing() { + let filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")); + let _ = tracing_subscriber::fmt().with_env_filter(filter).try_init(); +} + +fn unique_local_port() -> Result { + let listener = std::net::TcpListener::bind("127.0.0.1:0") + .context("failed to allocate local smoke-test port")?; + let port = listener.local_addr().context("failed to read local smoke-test port")?.port(); + drop(listener); + Ok(port) +} + +async fn print_models(client: &Client) -> Result<()> { + let models = wait_for_models(client).await?; + let model_ids = models.data.into_iter().map(|model| model.id).collect::>(); + println!("models={model_ids:?}"); + Ok(()) +} + +async fn wait_for_models(client: &Client) -> Result { + let mut last_error = None; + for _ in 0..240 { + match client.models().list().await { + Ok(models) => return Ok(models), + Err(error) => { + last_error = Some(error); + tokio::time::sleep(Duration::from_millis(500)).await; + } + } + } + + match last_error { + Some(error) => Err(error).context("OpenAI server did not become ready in time"), + None => bail!("OpenAI server readiness loop finished without a result"), + } +} + +async fn stream_completion( + client: &Client, + model: &str, + prompt: &str, +) -> Result { + // Keep this smoke test on async-openai's standard `create_stream` path so it + // exercises the ordinary typed chat-completions client without BYOT + // request/response types. + // + // The current async-openai chat-completions stream delta type does not expose + // our OpenAI-compatible `reasoning_content` extension field, so this + // example only validates the assistant role chunk, visible `content` + // deltas, and terminal finish chunk. Reasoning coverage lives in our own + // route tests and in the `vllm-chat` smoke example. + let request: CreateChatCompletionRequest = CreateChatCompletionRequestArgs::default() + .model(model) + .stream(true) + .temperature(0.0) + .max_completion_tokens(128u32) + .messages([ChatCompletionRequestUserMessageArgs::default() + .content(prompt) + .build() + .context("failed to build user chat message")? + .into()]) + .build() + .context("failed to build chat completion request")?; + + let mut stream = client + .chat() + .create_stream(request) + .await + .context("failed to create streaming chat completion")?; + + let mut final_text = String::new(); + let mut saw_role = false; + let mut saw_finish_reason = false; + let mut saw_text = false; + + while let Some(chunk) = stream.next().await { + let chunk = chunk.context("streaming chat completion failed")?; + for choice in chunk.choices { + if choice.delta.role.is_some() { + saw_role = true; + } + if let Some(delta) = choice.delta.content { + if !saw_text { + print!("[answer] "); + } + print!("{delta}"); + final_text.push_str(&delta); + saw_text = true; + } + if choice.finish_reason.is_some() { + saw_finish_reason = true; + } + } + } + + if !saw_role { + bail!("stream ended without an assistant role chunk"); + } + if !saw_finish_reason { + bail!("stream ended without a terminal finish reason"); + } + if final_text.is_empty() { + bail!("stream ended without any content deltas"); + } + + Ok(final_text) +} + +async fn shutdown_server( + server_task: tokio::task::JoinHandle>, + shutdown: CancellationToken, +) -> Result<()> { + shutdown.cancel(); + server_task + .await + .context("server task join failed")? + .context("server task failed") +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/config.rs b/asian_ai/performance/vLLM/rust/src/server/src/config.rs new file mode 100644 index 0000000000000000000000000000000000000000..ebcbcc821f3f3d2b2348732bd0d8035b5e03f6bf --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/config.rs @@ -0,0 +1,280 @@ +use std::collections::HashMap; +use std::fmt; +use std::time::Duration; + +use anyhow::{Result, bail}; +use axum::http::{HeaderName, HeaderValue, Method}; +use educe::Educe; +use serde::Serialize; +use serde_json::Value; +use vllm_chat::{ChatTemplateContentFormatOption, ParserSelection, RendererSelection}; +use vllm_engine_core_client::{CoordinatorMode as EngineCoreCoordinatorMode, TransportMode}; + +/// Default keep-alive idle timeout (seconds); also the head-read bound +/// when keep-alive is disabled (`0`). +pub const DEFAULT_KEEP_ALIVE_TIMEOUT: Duration = Duration::from_secs(5); + +/// How the HTTP server obtains its listening socket. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub enum HttpListenerMode { + /// Bind a fresh TCP listener on the given host/port. + BindTcp { host: String, port: u16 }, + /// Bind a fresh Unix domain listener on the given filesystem path. + BindUnix { path: String }, + /// Adopt an already-open listening socket inherited from a supervisor + /// process. + InheritedFd { fd: i32 }, +} + +/// Which coordinator implementation should be active when one is present for a +/// frontend client. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub enum CoordinatorMode { + /// Do not run a coordinator at all. + None, + /// Run the Rust in-process coordinator for managed `serve` deployments, if + /// there are multiple engines and the model is MoE. + MaybeInProc, + /// Connect to an external coordinator owned by another process. + External { address: String }, +} + +/// HTTP/API-server behavior switches that affect route-layer responses. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Default)] +pub struct ApiServerOptions { + /// Log a summary line for each completed request. + pub enable_log_requests: bool, + /// When `true`, include prompt token cache details in response usage. + pub enable_prompt_tokens_details: bool, + /// When `true`, set `X-Request-Id` on every HTTP response. + pub enable_request_id_headers: bool, +} + +/// CORS settings mirroring Python's `CORSMiddleware`; the default is permissive. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct CorsConfig { + /// Allowed origins. `["*"]` allows any origin. + pub allow_origins: Vec, + /// Allowed methods. `["*"]` allows the standard method set. + pub allow_methods: Vec, + /// Allowed request headers. `["*"]` mirrors the requested headers. + pub allow_headers: Vec, + /// Whether to allow credentials (cookies, authorization headers). + pub allow_credentials: bool, +} + +impl Default for CorsConfig { + fn default() -> Self { + Self { + allow_origins: vec!["*".to_string()], + allow_methods: vec!["*".to_string()], + allow_headers: vec!["*".to_string()], + allow_credentials: false, + } + } +} + +impl CorsConfig { + /// Validate that non-wildcard values parse into HTTP types, so the CORS + /// layer can be built infallibly after startup validation has run. + pub fn validate(&self) -> Result<()> { + for origin in &self.allow_origins { + if origin != "*" { + origin.parse::().map_err(|e| { + anyhow::anyhow!("invalid --allowed-origins value {origin:?}: {e}") + })?; + } + } + for method in &self.allow_methods { + if method != "*" { + method.parse::().map_err(|e| { + anyhow::anyhow!("invalid --allowed-methods value {method:?}: {e}") + })?; + } + } + for header in &self.allow_headers { + if header != "*" { + header.parse::().map_err(|e| { + anyhow::anyhow!("invalid --allowed-headers value {header:?}: {e}") + })?; + } + } + Ok(()) + } +} + +/// TLS settings mirroring Python's uvicorn `ssl_*` arguments. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct TlsConfig { + /// PEM certificate chain file. Required when TLS is configured; may also + /// hold the private key (combined PEM) when `key_file` is unset. + pub cert_file: Option, + /// PEM private key file. When `None`, the key is read from `cert_file` + /// (combined PEM). + pub key_file: Option, + /// PEM CA bundle used to verify client certificates (mTLS). Required when + /// `cert_reqs` is non-zero. + pub ca_certs: Option, + /// Client-certificate requirement, mirroring Python's `ssl.CERT_*`: + /// 0 = none, 1 = optional, 2 = required. + pub cert_reqs: i32, + /// OpenSSL cipher string for TLS 1.2 and below, mirroring Python's + /// `ssl.set_ciphers`. `None` keeps the forward-secret AEAD default. + pub ciphers: Option, +} + +impl TlsConfig { + /// Structurally validate the TLS arguments; the cert/key material is parsed + /// later, when the OpenSSL context is built. + pub fn validate(&self) -> Result<()> { + if self.cert_file.is_none() { + bail!( + "--ssl-certfile is required to enable TLS; \ + --ssl-keyfile/--ssl-ca-certs/--ssl-cert-reqs/--ssl-ciphers \ + cannot be used without it" + ); + } + if !matches!(self.cert_reqs, 0..=2) { + bail!( + "--ssl-cert-reqs must be 0 (none), 1 (optional), or 2 (required), got {}", + self.cert_reqs + ); + } + if self.cert_reqs != 0 && self.ca_certs.is_none() { + bail!( + "--ssl-ca-certs is required when --ssl-cert-reqs is {} \ + (client certificate verification)", + self.cert_reqs + ); + } + Ok(()) + } +} + +/// Normalized runtime configuration for the minimal OpenAI-compatible server. +#[derive(Educe, Clone, PartialEq, Eq, Serialize)] +#[educe(Debug)] +pub struct Config { + /// Frontend-to-engine transport setup. + pub transport_mode: TransportMode, + /// Requested frontend-side coordinator behavior. + pub coordinator_mode: CoordinatorMode, + /// Backend model identifier used for engine-core loading. + pub model: String, + /// Model name(s) exposed to clients via the OpenAI API. When non-empty, + /// the first entry is used as the primary ID in responses and all entries + /// are accepted in requests. When empty, falls back to `model`. + pub served_model_name: Vec, + /// HTTP listener setup. + pub listener_mode: HttpListenerMode, + /// Tool-call parser selection. + pub tool_call_parser: ParserSelection, + /// Reasoning parser selection. + pub reasoning_parser: ParserSelection, + /// Chat renderer selection. + pub renderer: RendererSelection, + /// Disable frontend-side multimodal preprocessing and render the model as + /// language-only. + pub language_model_only: bool, + /// Server-default chat template override, as a file path or inline + /// template. + pub chat_template: Option, + /// Server-default keyword arguments merged into every chat-template render. + pub default_chat_template_kwargs: Option>, + /// How to serialize `message.content` for chat-template rendering. + pub chat_template_content_format: ChatTemplateContentFormatOption, + /// Optional maximum number of top log probabilities accepted by the + /// frontend. `None` delegates to the text layer default. + pub max_logprobs: Option, + /// HTTP/API-server behavior switches. + pub api_server_options: ApiServerOptions, + /// CORS settings applied to every HTTP response. + pub cors: CorsConfig, + /// TLS settings. `None` serves plaintext HTTP; `Some` terminates TLS at the + /// listener. + pub tls: Option, + /// API keys accepted as bearer tokens for guarded routes. + #[serde(skip_serializing)] + #[educe(Debug(method(fmt_redacted_api_keys)))] + pub api_keys: Vec, + /// When `true`, suppress periodic stats logging (throughput, queue depth, + /// cache usage). + pub disable_log_stats: bool, + /// TCP port for the gRPC Generate service. When `None`, no gRPC server is + /// started. + pub grpc_port: Option, + /// Maximum time to wait for active HTTP/gRPC requests to drain on shutdown. + pub shutdown_timeout: Duration, + /// Maximum idle time on a keep-alive HTTP connection before the server + /// closes it (`VLLM_HTTP_TIMEOUT_KEEP_ALIVE`, default 5s). + pub keep_alive_timeout: Duration, + /// Profiler mode that registers `/start_profile` and `/stop_profile` + /// routes when present. + pub profiler: Option, +} + +impl Config { + /// Validate frontend configuration that can be checked before engine + /// startup. + pub fn validate(&self) -> Result<()> { + vllm_chat::validate_parser_overrides(&self.tool_call_parser, &self.reasoning_parser)?; + self.cors.validate()?; + if let Some(tls) = &self.tls { + tls.validate()?; + } + if let Some(max_logprobs) = self.max_logprobs + && max_logprobs < -1 + { + bail!( + "max_logprobs must be non-negative or -1, got {}", + max_logprobs + ); + } + + Ok(()) + } + + /// Return the number of engines implied by the configured transport mode. + pub fn engine_count(&self) -> usize { + match &self.transport_mode { + TransportMode::HandshakeOwner { engine_count, .. } + | TransportMode::Bootstrapped { engine_count, .. } => *engine_count, + } + } + + /// Resolve the effective coordinator mode. + pub fn effective_coordinator_mode( + &self, + model_is_moe: bool, + ) -> Option { + match &self.coordinator_mode { + CoordinatorMode::None => None, + CoordinatorMode::MaybeInProc => { + if model_is_moe && self.engine_count() > 1 { + Some(EngineCoreCoordinatorMode::InProc) + } else { + None + } + } + CoordinatorMode::External { address } => Some(EngineCoreCoordinatorMode::External { + address: address.clone(), + }), + } + } +} + +struct RedactedApiKeys<'a>(&'a [String]); + +impl fmt::Debug for RedactedApiKeys<'_> { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + if self.0.is_empty() { + f.debug_list().finish() + } else { + write!(f, "[; {}]", self.0.len()) + } + } +} + +fn fmt_redacted_api_keys(api_keys: &[String], f: &mut fmt::Formatter<'_>) -> fmt::Result { + fmt::Debug::fmt(&RedactedApiKeys(api_keys), f) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/error.rs b/asian_ai/performance/vLLM/rust/src/server/src/error.rs new file mode 100644 index 0000000000000000000000000000000000000000..14fa701d63b678f5be68db9a32150497f4dbca82 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/error.rs @@ -0,0 +1,216 @@ +use axum::Json; +use axum::http::StatusCode; +use axum::response::{IntoResponse, Response}; +use thiserror_ext::{AsReport as _, Construct, Macro}; + +use crate::routes::openai::utils::types::{ErrorDetail, ErrorResponse}; + +/// Small OpenAI-style error family used by the minimal HTTP layer. +#[derive(Debug, Construct, Macro)] +pub enum ApiError { + /// The request is syntactically valid OpenAI JSON but asks for unsupported + /// behavior. + InvalidRequest { + message: String, + param: Option<&'static str>, + }, + /// The requested model name does not match the single configured model. + ModelNotFound { model: String }, + /// The request body could not be parsed as valid JSON. + JsonParseError { message: String }, + /// An unexpected internal failure happened before streaming started. + ServerError { message: String }, +} + +impl ApiError { + /// Return the HTTP status code associated with this API error. + pub fn status_code(&self) -> StatusCode { + match self { + Self::InvalidRequest { .. } => StatusCode::BAD_REQUEST, + Self::ModelNotFound { .. } => StatusCode::NOT_FOUND, + Self::ServerError { .. } => StatusCode::INTERNAL_SERVER_ERROR, + Self::JsonParseError { .. } => StatusCode::BAD_REQUEST, + } + } + + /// Convert this error into the standard OpenAI-compatible JSON error + /// payload. + pub fn to_error_response(&self) -> ErrorResponse { + let error = match self { + Self::InvalidRequest { message, param } => ErrorDetail { + message: message.clone(), + error_type: "invalid_request_error".to_string(), + param: param.map(|p| p.to_string()), + code: Some("invalid_request_error".to_string()), + }, + Self::ModelNotFound { model } => ErrorDetail { + message: format!("The model `{model}` does not exist."), + error_type: "invalid_request_error".to_string(), + param: Some("model".to_string()), + code: Some("model_not_found".to_string()), + }, + Self::ServerError { message } => ErrorDetail { + message: message.clone(), + error_type: "server_error".to_string(), + param: None, + code: Some("server_error".to_string()), + }, + Self::JsonParseError { message } => ErrorDetail { + message: message.clone(), + error_type: "invalid_request_error".to_string(), + param: None, + code: Some("json_parse_error".to_string()), + }, + }; + + ErrorResponse { error } + } +} + +impl IntoResponse for ApiError { + fn into_response(self) -> Response { + (self.status_code(), Json(self.to_error_response())).into_response() + } +} + +/// Classify a text-pipeline submit failure: request validation failures are +/// the client's fault and map to HTTP 400, mirroring the Python frontend. +/// Everything else stays an internal 500. +pub fn text_submit_error(context: &'static str, error: vllm_text::Error) -> ApiError { + if error.is_request_validation_error() { + return invalid_request!("{error}"); + } + server_error!("{}: {}", context, error.to_report_string()) +} + +/// Like [`text_submit_error`], for the chat pipeline (which both wraps the +/// text errors and raises its own prompt-length variant). +pub fn chat_submit_error(context: &'static str, error: vllm_chat::Error) -> ApiError { + if error.is_request_validation_error() { + return invalid_request!("{error}"); + } + server_error!("{}: {}", context, error.to_report_string()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn prompt_too_long_maps_to_invalid_request() { + let error = vllm_text::Error::PromptTooLong { + max_model_len: 8192, + prompt_len: 9000, + }; + let api_error = text_submit_error("failed to submit completion request", error); + assert_eq!(api_error.status_code(), StatusCode::BAD_REQUEST); + let response = api_error.to_error_response(); + assert_eq!(response.error.error_type, "invalid_request_error"); + assert!(response.error.message.contains("8192")); + assert!(response.error.message.contains("9000")); + } + + #[test] + fn invalid_thinking_token_budget_maps_to_invalid_request() { + let api_error = text_submit_error( + "failed to submit completion request", + vllm_text::Error::InvalidThinkingTokenBudget, + ); + assert_eq!(api_error.status_code(), StatusCode::BAD_REQUEST); + let response = api_error.to_error_response(); + assert_eq!(response.error.error_type, "invalid_request_error"); + assert!(response.error.message.contains("thinking_token_budget")); + } + + #[test] + fn min_tokens_above_max_tokens_maps_to_invalid_request() { + let api_error = text_submit_error( + "failed to submit completion request", + vllm_text::Error::MinTokensExceedsMaxTokens { + min_tokens: 5, + max_tokens: 4, + }, + ); + assert_eq!(api_error.status_code(), StatusCode::BAD_REQUEST); + let response = api_error.to_error_response(); + assert_eq!(response.error.error_type, "invalid_request_error"); + assert!(response.error.message.contains("min_tokens=5")); + assert!(response.error.message.contains("max_tokens=4")); + } + + #[test] + fn chat_wrapped_prompt_too_long_maps_to_invalid_request() { + let error = vllm_chat::Error::Text(vllm_text::Error::PromptTooLong { + max_model_len: 8192, + prompt_len: 9000, + }); + let api_error = chat_submit_error("failed to submit chat request", error); + assert_eq!(api_error.status_code(), StatusCode::BAD_REQUEST); + } + + #[test] + fn llm_wrapped_empty_prompt_maps_to_invalid_request() { + let error = vllm_text::Error::Llm(vllm_llm::Error::EmptyPromptTokenIds { + request_id: "req-1".to_string(), + }); + let api_error = text_submit_error("failed to submit completion request", error); + assert_eq!(api_error.status_code(), StatusCode::BAD_REQUEST); + } + + #[test] + fn logprobs_validation_maps_to_invalid_request() { + let error = vllm_text::Error::Logprobs(vllm_text::LogprobsError::TooManyCount { + parameter: "logprobs", + requested: 1000, + max_allowed: 20, + }); + let api_error = text_submit_error("failed to submit completion request", error); + assert_eq!(api_error.status_code(), StatusCode::BAD_REQUEST); + let response = api_error.to_error_response(); + assert_eq!(response.error.error_type, "invalid_request_error"); + assert!(response.error.message.contains("logprobs")); + } + + #[test] + fn chat_wrapped_logprobs_validation_maps_to_invalid_request() { + let error = vllm_chat::Error::Text(vllm_text::Error::Logprobs( + vllm_text::LogprobsError::TooManyCount { + parameter: "prompt_logprobs", + requested: 1000, + max_allowed: 20, + }, + )); + let api_error = chat_submit_error("failed to submit chat request", error); + assert_eq!(api_error.status_code(), StatusCode::BAD_REQUEST); + } + + #[test] + fn out_of_vocab_validation_maps_to_invalid_request() { + let error = vllm_text::Error::TokenIds(vllm_text::TokenIdsError::OutOfVocab { + parameter: "logprob_token_ids", + token_ids: vec![1000], + vocab_size: 1000, + }); + let api_error = text_submit_error("failed to submit completion request", error); + assert_eq!(api_error.status_code(), StatusCode::BAD_REQUEST); + } + + #[test] + fn empty_allowed_token_ids_maps_to_invalid_request() { + let error = vllm_text::Error::TokenIds(vllm_text::TokenIdsError::EmptyAllowedTokenIds); + let api_error = text_submit_error("failed to submit completion request", error); + assert_eq!(api_error.status_code(), StatusCode::BAD_REQUEST); + let response = api_error.to_error_response(); + assert_eq!(response.error.error_type, "invalid_request_error"); + assert!(response.error.message.contains("allowed_token_ids")); + } + + #[test] + fn other_submit_errors_stay_internal() { + let error = vllm_text::Error::Tokenizer("backend exploded".to_string()); + let api_error = text_submit_error("failed to submit completion request", error); + assert_eq!(api_error.status_code(), StatusCode::INTERNAL_SERVER_ERROR); + let response = api_error.to_error_response(); + assert!(response.error.message.starts_with("failed to submit completion request:")); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/grpc/convert.rs b/asian_ai/performance/vLLM/rust/src/server/src/grpc/convert.rs new file mode 100644 index 0000000000000000000000000000000000000000..13b1a8823823ef20df2ca65309eb8ae1563455df --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/grpc/convert.rs @@ -0,0 +1,671 @@ +//! Conversion between gRPC protobuf types and internal `vllm-text` +//! request/response types. + +use tonic::Status; +use uuid::Uuid; +use vllm_engine_core_client::protocol::output::StopReason; +use vllm_engine_core_client::protocol::structured_outputs::StructuredOutputsParams; +use vllm_text::{ + DecodedLogprobs, DecodedPromptLogprobs, FinishReason, Finished, Prompt, SamplingParams, + TextDecodeOptions, TextRequest, +}; + +use super::pb; + +// ======================================================================================== +// Request conversion +// ======================================================================================== + +/// Convert a gRPC `GenerateRequest` into the internal `TextRequest`. +/// +/// If `req.model` is non-empty, it must match one of `served_model_names`; +/// otherwise the request is rejected with `NotFound`. An empty string is +/// treated as "unset" (proto3 default) and accepted. +pub fn to_text_request( + req: pb::GenerateRequest, + stream: bool, + served_model_names: &[String], +) -> Result { + if !req.model.is_empty() && !served_model_names.iter().any(|n| n == &req.model) { + return Err(Status::not_found(format!( + "model `{}` not found", + req.model + ))); + } + + if req.truncate_prompt_tokens != 0 { + return Err(Status::invalid_argument( + "truncate_prompt_tokens is not supported", + )); + } + + let prompt = match req.prompt { + Some(pb::generate_request::Prompt::Text(text)) => Prompt::Text(text), + Some(pb::generate_request::Prompt::TokenIds(ids)) => Prompt::TokenIds(ids.ids), + None => return Err(Status::invalid_argument("prompt is required")), + }; + + let request_id = if req.request_id.is_empty() { + Uuid::new_v4().to_string() + } else { + req.request_id + }; + + let sampling = req.sampling.as_ref(); + let decoding = req.decoding.as_ref(); + let stopping = req.stopping.as_ref(); + let response = req.response.as_ref(); + let kv = req.kv.as_ref(); + + let mut sampling_params = + build_sampling_params(req.temperature, sampling, decoding, stopping, response)?; + + // Thread KVCacheParameters → SamplingParams fields. + if let Some(kv) = kv { + // Thread kv_transfer_params through vllm_xargs, matching the HTTP route + // convention. + if let Some(kv_struct) = kv.kv_transfer_params.as_ref() { + let kv_json = proto_struct_to_json(kv_struct); + let map = sampling_params.vllm_xargs.get_or_insert_with(Default::default); + map.insert("kv_transfer_params".to_string(), kv_json); + } + if kv.bypass_prefix_cache { + sampling_params.skip_reading_prefix_cache = Some(true); + } + } + + let decode_options = TextDecodeOptions { + skip_special_tokens: true, + include_stop_str_in_output: stopping.is_some_and(|s| s.include_stop_strings), + stop_strings: stopping.map(|s| &s.stop_strings).filter(|ss| !ss.is_empty()).cloned(), + min_tokens: stopping.map_or(0, |s| s.min_new_tokens), + }; + + Ok(TextRequest { + request_id, + prompt, + mm_features: None, + sampling_params, + decode_options, + intermediate: stream, + priority: req.priority, + cache_salt: kv.map(|k| &k.cache_salt).filter(|s| !s.is_empty()).cloned(), + add_special_tokens: true, + data_parallel_rank: None, + reasoning_parser_kwargs: None, + lora_request: None, + }) +} + +fn build_sampling_params( + temperature: Option, + sampling: Option<&pb::RandomSampling>, + decoding: Option<&pb::DecodingParameters>, + stopping: Option<&pb::StoppingCriteria>, + response: Option<&pb::ResponseOptions>, +) -> Result { + // Temperature is a top-level GenerateRequest field. Default to greedy (0.0) for + // the gRPC API when the caller does not specify a value. This differs from + // the HTTP/OpenAI API (which defaults to 1.0) and matches the convention of + // programmatic generation APIs. + let temperature = temperature.or(Some(0.0)); + let mut params = SamplingParams { + temperature, + ..SamplingParams::default() + }; + + // RandomSampling: for every remaining sampling field the protobuf default (`0`) + // is treated as "unset" and leaves the resolved value to the lowering + // stage, which falls back to the model-provided default or a + // neutral/disabled value otherwise. + if let Some(s) = sampling { + // num_sequences (n > 1) is not supported yet by the TextLlm layer; the response + // path also hardcodes SequenceOutput.index = 0, so accepting >1 would silently + // truncate output cardinality. Reject explicitly. + if s.num_sequences > 1 { + return Err(Status::invalid_argument( + "num_sequences > 1 is not supported", + )); + } + if s.top_k != 0 { + params.top_k = Some(s.top_k); + } + if s.top_p != 0.0 { + params.top_p = Some(s.top_p); + } + if s.min_p != 0.0 { + params.min_p = Some(s.min_p); + } + params.seed = s.seed; + } + + // DecodingParameters + if let Some(d) = decoding { + if d.presence_penalty != 0.0 { + params.presence_penalty = Some(d.presence_penalty); + } + if d.frequency_penalty != 0.0 { + params.frequency_penalty = Some(d.frequency_penalty); + } + if d.repetition_penalty != 0.0 { + params.repetition_penalty = Some(d.repetition_penalty); + } + if !d.logit_bias.is_empty() { + params.logit_bias = Some(d.logit_bias.clone()); + } + if !d.allowed_token_ids.is_empty() { + params.allowed_token_ids = Some(d.allowed_token_ids.clone()); + } + params.structured_outputs = convert_structured_output(d)?; + } + + // StoppingCriteria + if let Some(s) = stopping { + if s.max_new_tokens != 0 { + params.max_tokens = Some(s.max_new_tokens); + } + if s.min_new_tokens != 0 { + params.min_tokens = Some(s.min_new_tokens); + } + if !s.stop_token_ids.is_empty() { + params.stop_token_ids = Some(s.stop_token_ids.clone()); + } + params.ignore_eos = s.ignore_eos; + } + + // ResponseOptions → logprobs + if let Some(r) = response { + if r.output_logprobs { + let (count, token_ids) = candidate_logprob_spec(r.output_candidates.as_ref()); + params.logprobs = Some(count); + params.logprob_token_ids = token_ids; + } + if r.prompt_logprobs { + // The engine-core protocol has only one shared `logprob_token_ids` field + // for output and prompt logprobs, so a per-token-id selector for prompt + // candidates can't be honored independently. Reject it instead of silently + // dropping the list. + if matches!( + r.prompt_candidates.as_ref().and_then(|c| c.select.as_ref()), + Some(pb::candidate_tokens::Select::TokenIds(_)) + ) { + return Err(Status::invalid_argument( + "prompt_candidates token_ids selector is not supported", + )); + } + let (count, _) = candidate_logprob_spec(r.prompt_candidates.as_ref()); + params.prompt_logprobs = Some(count); + } + } + + Ok(params) +} + +/// Map the proto `CandidateTokens` selector to a `(logprobs_count, +/// logprob_token_ids)` pair. +/// +/// - `top_n(k)` → `(k, None)` — return top-k candidates by probability +/// - `all` → `(-1, None)` — return the full vocabulary +/// - `token_ids(n)` → `(1, Some(vec of n token ids))` — return logprobs for specific tokens (the +/// count `n` is stored in the proto as the number of token IDs that follow, but the actual IDs +/// are carried via `logprob_token_ids` on `SamplingParams`) +/// - absent → `(1, None)` — just the sampled/scored token +fn candidate_logprob_spec(candidates: Option<&pb::CandidateTokens>) -> (i32, Option>) { + match candidates.and_then(|c| c.select.as_ref()) { + Some(pb::candidate_tokens::Select::TopN(n)) => (*n as i32, None), + Some(pb::candidate_tokens::Select::All(true)) => (-1, None), + Some(pb::candidate_tokens::Select::TokenIds(ids)) => (1, Some(ids.ids.clone())), + _ => (1, None), + } +} + +fn convert_structured_output( + d: &pb::DecodingParameters, +) -> Result, Status> { + let so = match d.structured_output.as_ref() { + None => return Ok(None), + Some(so) => so, + }; + use pb::decoding_parameters::StructuredOutput; + let params = match so { + StructuredOutput::Json(schema) => { + let json: serde_json::Value = serde_json::from_str(schema) + .map_err(|e| Status::invalid_argument(format!("invalid json schema: {e}")))?; + StructuredOutputsParams::json(json) + } + StructuredOutput::Regex(regex) => StructuredOutputsParams::regex(regex.clone()), + StructuredOutput::Choice(choices) => { + StructuredOutputsParams::choice(choices.choices.clone()) + } + StructuredOutput::Grammar(grammar) => StructuredOutputsParams::grammar(grammar.clone()), + StructuredOutput::JsonObject(true) => StructuredOutputsParams::json_object(), + StructuredOutput::JsonObject(false) => return Ok(None), + StructuredOutput::StructuralTag(tag) => { + StructuredOutputsParams::structural_tag(tag.clone()) + } + }; + Ok(Some(params)) +} + +// ======================================================================================== +// Response conversion +// ======================================================================================== + +/// Convert a `DecodedTextEvent::Start` into the prompt info portion of a gRPC +/// response. +pub fn to_prompt_info( + prompt_token_ids: &[u32], + prompt_logprobs: Option<&DecodedPromptLogprobs>, + opts: &ResponseOpts, +) -> pb::PromptInfo { + let token_ids = if opts.prompt_token_ids { + prompt_token_ids.to_vec() + } else { + vec![] + }; + + let (logprobs, ranks, candidate_tokens) = match prompt_logprobs { + Some(plp) if opts.prompt_logprobs => prompt_logprobs_to_proto(plp), + _ => (vec![], vec![], vec![]), + }; + + pb::PromptInfo { + num_prompt_tokens: prompt_token_ids.len() as u32, + token_ids, + logprobs, + ranks, + candidate_tokens, + } +} + +/// Convert a `DecodedTextEvent::TextDelta` into a gRPC `SequenceOutput`. +pub fn to_sequence_output( + delta: &str, + token_ids: &[u32], + logprobs: Option<&DecodedLogprobs>, + finished: Option<&Finished>, + opts: &ResponseOpts, +) -> pb::SequenceOutput { + let (lp_values, rank_values, candidates) = match logprobs { + Some(lp) if opts.output_logprobs => output_logprobs_to_proto(lp), + _ => (vec![], vec![], vec![]), + }; + + pb::SequenceOutput { + index: 0, // TODO: multi-sequence (n > 1) not supported + text: if opts.output_text { + delta.to_string() + } else { + String::new() + }, + num_tokens: token_ids.len() as u32, + token_ids: if opts.output_token_ids { + token_ids.to_vec() + } else { + vec![] + }, + logprobs: lp_values, + ranks: rank_values, + candidate_tokens: candidates, + finish_info: finished.map(|f| to_finish_info(f, token_ids)), + } +} + +fn to_finish_info(finished: &Finished, token_ids: &[u32]) -> pb::FinishInfo { + use pb::finish_info::FinishReason as PbFinishReason; + + let (finish_reason, stop_reason) = match &finished.finish_reason { + FinishReason::Stop(reason) => { + let sr = match reason { + Some(StopReason::TokenId(id)) => { + Some(pb::finish_info::StopReason::StopTokenId(*id)) + } + Some(StopReason::Text(s)) => { + Some(pb::finish_info::StopReason::StopString(s.clone())) + } + // EOS-driven stop: engine-core matched the primary EOS token id but did not + // echo it back as a `stop_reason`. The matched token is, by construction, the + // last token of the terminal output batch (see vllm's `check_stop` in + // vllm/v1/core/sched/utils.py), so we recover it from there. + None => token_ids.last().copied().map(pb::finish_info::StopReason::EosTokenId), + }; + (PbFinishReason::Stop as i32, sr) + } + FinishReason::Length => (PbFinishReason::Length as i32, None), + FinishReason::Abort | FinishReason::Error | FinishReason::Repetition(_) => { + (PbFinishReason::Aborted as i32, None) + } + }; + + pb::FinishInfo { + num_output_tokens: finished.usage.output_token_count as u32, + finish_reason, + stop_reason, + kv_transfer_params: finished.kv_transfer_params.as_ref().and_then(json_to_proto_struct), + } +} + +// ======================================================================================== +// Logprobs helpers +// ======================================================================================== + +/// Convert output logprobs to the flat proto representation. +/// +/// Returns (logprob_values, ranks, candidate_tokens) — all parallel arrays +/// indexed by position. +fn output_logprobs_to_proto( + lp: &DecodedLogprobs, +) -> (Vec, Vec, Vec) { + positions_to_proto(&lp.positions) +} + +/// Convert prompt logprobs to the flat proto representation. +fn prompt_logprobs_to_proto( + plp: &DecodedPromptLogprobs, +) -> (Vec, Vec, Vec) { + // The proto PromptInfo has flat parallel arrays covering all prompt positions. + // DecodedPromptLogprobs has first_token separately + scored_positions for the + // rest. The first prompt position has no scores, so we emit zeros for it. + let (mut logprobs, mut ranks, mut candidates) = positions_to_proto(&plp.scored_positions); + logprobs.insert(0, 0.0); + ranks.insert(0, 0); + candidates.insert(0, pb::CandidateTokenInfo { tokens: vec![] }); + (logprobs, ranks, candidates) +} + +/// Shared helper: convert a slice of decoded position logprobs to flat proto +/// arrays. +fn positions_to_proto( + positions: &[vllm_text::DecodedPositionLogprobs], +) -> (Vec, Vec, Vec) { + let mut logprobs = Vec::with_capacity(positions.len()); + let mut ranks = Vec::with_capacity(positions.len()); + let mut candidates = Vec::with_capacity(positions.len()); + + for pos in positions { + // First entry is the sampled/scored token. + if let Some(first) = pos.entries.first() { + logprobs.push(first.logprob); + ranks.push(first.rank); + } + + // Extra candidates beyond the first. + let entries = pos.entries.iter().skip(1); + candidates.push(pb::CandidateTokenInfo { + tokens: entries + .map(|e| pb::candidate_token_info::TokenInfo { + id: e.token_id, + logprob: e.logprob, + rank: e.rank, + }) + .collect(), + }); + } + + (logprobs, ranks, candidates) +} + +// ======================================================================================== +// KV transfer params conversion (serde_json::Value ↔ prost_types::Struct) +// ======================================================================================== + +fn proto_struct_to_json(s: &prost_types::Struct) -> serde_json::Value { + serde_json::Value::Object( + s.fields.iter().map(|(k, v)| (k.clone(), proto_value_to_json(v))).collect(), + ) +} + +fn proto_value_to_json(v: &prost_types::Value) -> serde_json::Value { + use prost_types::value::Kind; + match v.kind.as_ref() { + None | Some(Kind::NullValue(_)) => serde_json::Value::Null, + Some(Kind::BoolValue(b)) => serde_json::Value::Bool(*b), + Some(Kind::NumberValue(n)) => serde_json::json!(*n), + Some(Kind::StringValue(s)) => serde_json::Value::String(s.clone()), + Some(Kind::ListValue(list)) => { + serde_json::Value::Array(list.values.iter().map(proto_value_to_json).collect()) + } + Some(Kind::StructValue(s)) => proto_struct_to_json(s), + } +} + +fn json_to_proto_struct(value: &serde_json::Value) -> Option { + match value { + serde_json::Value::Object(map) => Some(prost_types::Struct { + fields: map.iter().map(|(k, v)| (k.clone(), json_to_proto_value(v))).collect(), + }), + _ => None, + } +} + +fn json_to_proto_value(v: &serde_json::Value) -> prost_types::Value { + use prost_types::value::Kind; + let kind = match v { + serde_json::Value::Null => Kind::NullValue(0), + serde_json::Value::Bool(b) => Kind::BoolValue(*b), + serde_json::Value::Number(n) => Kind::NumberValue(n.as_f64().unwrap_or(0.0)), + serde_json::Value::String(s) => Kind::StringValue(s.clone()), + serde_json::Value::Array(arr) => Kind::ListValue(prost_types::ListValue { + values: arr.iter().map(json_to_proto_value).collect(), + }), + serde_json::Value::Object(map) => Kind::StructValue(prost_types::Struct { + fields: map.iter().map(|(k, v)| (k.clone(), json_to_proto_value(v))).collect(), + }), + }; + prost_types::Value { kind: Some(kind) } +} + +// ======================================================================================== +// Options extracted from the request for response building +// ======================================================================================== + +/// Response-shaping options extracted from the proto `ResponseOptions`. +#[derive(Default)] +pub struct ResponseOpts { + pub prompt_token_ids: bool, + pub prompt_logprobs: bool, + pub output_text: bool, + pub output_token_ids: bool, + pub output_logprobs: bool, +} + +impl ResponseOpts { + pub fn from_proto(r: Option<&pb::ResponseOptions>) -> Self { + match r { + Some(r) => Self { + prompt_token_ids: r.prompt_token_ids, + prompt_logprobs: r.prompt_logprobs, + output_text: r.output_text.unwrap_or(true), + output_token_ids: r.output_token_ids, + output_logprobs: r.output_logprobs, + }, + None => Self { + output_text: true, + ..Default::default() + }, + } + } +} + +#[cfg(test)] +mod tests { + use vllm_engine_core_client::protocol::output::StopReason; + use vllm_text::{FinishReason, Finished, Prompt}; + + use super::pb::finish_info::{FinishReason as PbFinishReason, StopReason as PbStopReason}; + use super::{ResponseOpts, pb, to_finish_info, to_sequence_output, to_text_request}; + + fn base_request() -> pb::GenerateRequest { + pb::GenerateRequest { + request_id: "req".to_string(), + model: "test-model".to_string(), + prompt: Some(pb::generate_request::Prompt::Text("hi".to_string())), + ..Default::default() + } + } + + #[test] + fn temperature_propagates_from_top_level_request_field() { + let req = pb::GenerateRequest { + temperature: Some(0.7), + ..base_request() + }; + let text = to_text_request(req, false, &["test-model".to_string()]).expect("convert ok"); + assert_eq!(text.sampling_params.temperature, Some(0.7)); + } + + #[test] + fn unset_temperature_defaults_to_greedy() { + let text = to_text_request(base_request(), false, &["test-model".to_string()]) + .expect("convert ok"); + // The gRPC API defaults to greedy (0.0) when temperature is not specified. + assert_eq!(text.sampling_params.temperature, Some(0.0)); + } + + #[test] + fn absent_seed_is_none() { + let req = pb::GenerateRequest { + sampling: Some(pb::RandomSampling { + seed: None, + ..Default::default() + }), + ..base_request() + }; + let text = to_text_request(req, false, &["test-model".to_string()]).expect("convert ok"); + assert_eq!(text.sampling_params.seed, None); + } + + #[test] + fn zero_seed_is_valid() { + let req = pb::GenerateRequest { + sampling: Some(pb::RandomSampling { + seed: Some(0), + ..Default::default() + }), + ..base_request() + }; + let text = to_text_request(req, false, &["test-model".to_string()]).expect("convert ok"); + assert_eq!(text.sampling_params.seed, Some(0)); + } + + #[test] + fn bypass_prefix_cache_maps_to_skip_reading_prefix_cache() { + let req = pb::GenerateRequest { + kv: Some(pb::KvCacheParameters { + bypass_prefix_cache: true, + ..Default::default() + }), + ..base_request() + }; + let text = to_text_request(req, false, &["test-model".to_string()]).expect("convert ok"); + assert_eq!(text.sampling_params.skip_reading_prefix_cache, Some(true)); + } + + #[test] + fn bypass_prefix_cache_false_leaves_field_unset() { + let req = pb::GenerateRequest { + kv: Some(pb::KvCacheParameters { + bypass_prefix_cache: false, + ..Default::default() + }), + ..base_request() + }; + let text = to_text_request(req, false, &["test-model".to_string()]).expect("convert ok"); + assert_eq!(text.sampling_params.skip_reading_prefix_cache, None); + // Prompt conversion still succeeds and reaches the expected variant. + assert!(matches!(text.prompt, Prompt::Text(s) if s == "hi")); + } + + fn finished(reason: FinishReason) -> Finished { + Finished { + usage: vllm_llm::TokenUsage { + prompt_token_count: 0, + output_token_count: 0, + cached_token_count: 0, + }, + finish_reason: reason, + kv_transfer_params: None, + } + } + + #[test] + fn eos_stop_reports_last_output_token_as_eos_id() { + let fin = finished(FinishReason::Stop(None)); + let token_ids = [1_u32, 2, 3, 151643]; + + let info = to_finish_info(&fin, &token_ids); + + assert_eq!(info.finish_reason, PbFinishReason::Stop as i32); + assert_eq!(info.stop_reason, Some(PbStopReason::EosTokenId(151643))); + } + + #[test] + fn eos_stop_with_empty_token_ids_leaves_stop_reason_unset() { + let fin = finished(FinishReason::Stop(None)); + + let info = to_finish_info(&fin, &[]); + + assert_eq!(info.finish_reason, PbFinishReason::Stop as i32); + assert_eq!(info.stop_reason, None); + } + + #[test] + fn explicit_stop_token_id_is_preserved() { + let fin = finished(FinishReason::Stop(Some(StopReason::TokenId(42)))); + // Terminal token list should be ignored when an explicit stop reason is + // present. + let info = to_finish_info(&fin, &[7, 42]); + + assert_eq!(info.finish_reason, PbFinishReason::Stop as i32); + assert_eq!(info.stop_reason, Some(PbStopReason::StopTokenId(42))); + } + + #[test] + fn explicit_stop_string_is_preserved() { + let fin = finished(FinishReason::Stop(Some(StopReason::Text("".into())))); + + let info = to_finish_info(&fin, &[1, 2, 3]); + + assert_eq!(info.finish_reason, PbFinishReason::Stop as i32); + assert_eq!( + info.stop_reason, + Some(PbStopReason::StopString("".into())) + ); + } + + #[test] + fn length_finish_has_no_stop_reason() { + let fin = finished(FinishReason::Length); + + let info = to_finish_info(&fin, &[1, 2, 3]); + + assert_eq!(info.finish_reason, PbFinishReason::Length as i32); + assert_eq!(info.stop_reason, None); + } + + #[test] + fn abort_finish_is_mapped_to_aborted() { + let fin = finished(FinishReason::Abort); + + let info = to_finish_info(&fin, &[]); + + assert_eq!(info.finish_reason, PbFinishReason::Aborted as i32); + assert_eq!(info.stop_reason, None); + } + + #[test] + fn to_sequence_output_threads_token_ids_into_eos_id() { + let fin = finished(FinishReason::Stop(None)); + let opts = ResponseOpts { + output_text: true, + output_token_ids: true, + ..Default::default() + }; + + let out = to_sequence_output("hello", &[10, 20, 30], None, Some(&fin), &opts); + + let finish = out.finish_info.expect("finish_info should be present"); + assert_eq!(finish.finish_reason, PbFinishReason::Stop as i32); + assert_eq!(finish.stop_reason, Some(PbStopReason::EosTokenId(30))); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/grpc/mod.rs b/asian_ai/performance/vLLM/rust/src/server/src/grpc/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..8418861f2621a9dac4268934f02eb6e18807630a --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/grpc/mod.rs @@ -0,0 +1,163 @@ +//! gRPC Generate service backed by the shared [`vllm_text::TextLlm`] facade. + +mod convert; + +use std::pin::Pin; +use std::sync::Arc; + +use futures::{Stream, StreamExt as _}; +use thiserror_ext::AsReport as _; +use tokio::sync::mpsc; +use tokio_stream::wrappers::ReceiverStream; +use tonic::{Request, Response, Status}; +use tracing::info; +use vllm_text::{DecodedTextEvent, TextOutputStreamExt as _}; + +use self::convert::ResponseOpts; +use crate::state::AppState; + +/// Generated protobuf/gRPC types for the `vllm` package. +pub mod pb { + tonic::include_proto!("vllm"); +} + +pub use pb::generate_server::GenerateServer; + +#[cfg(test)] +mod tests; + +/// gRPC Generate service implementation backed by the shared application state. +pub struct GenerateServiceImpl { + state: Arc, +} + +impl GenerateServiceImpl { + pub fn new(state: Arc) -> Self { + Self { state } + } +} + +#[tonic::async_trait] +impl pb::generate_server::Generate for GenerateServiceImpl { + type GenerateStreamStream = + Pin> + Send>>; + + /// Unary generate: collect all output and return a single response. + async fn generate( + &self, + request: Request, + ) -> Result, Status> { + let proto_req = request.into_inner(); + let response_opts = ResponseOpts::from_proto(proto_req.response.as_ref()); + let text_request = + convert::to_text_request(proto_req, false, self.state.served_model_names())?; + + let request_id = text_request.request_id.clone(); + info!(%request_id, "grpc generate (unary)"); + + let stream = self.state.chat.text().generate(text_request).await; + let stream = stream.map_err(text_error_to_status)?; + + let collected = stream.collect_output().await.map_err(text_error_to_status)?; + + // Build the single aggregated response. + let prompt_info = convert::to_prompt_info( + &collected.prompt_token_ids, + collected.prompt_logprobs.as_ref(), + &response_opts, + ); + + let finish_info = vllm_text::Finished { + usage: collected.usage, + finish_reason: collected.finish_reason, + kv_transfer_params: collected.kv_transfer_params, + }; + + let outputs = convert::to_sequence_output( + &collected.text, + &collected.token_ids, + collected.logprobs.as_ref(), + Some(&finish_info), + &response_opts, + ); + + Ok(Response::new(pb::GenerateResponse { + prompt_info: Some(prompt_info), + outputs: Some(outputs), + })) + } + + /// Streaming generate: yield incremental responses as tokens are produced. + async fn generate_stream( + &self, + request: Request, + ) -> Result, Status> { + let proto_req = request.into_inner(); + let response_opts = ResponseOpts::from_proto(proto_req.response.as_ref()); + let text_request = + convert::to_text_request(proto_req, true, self.state.served_model_names())?; + + let request_id = text_request.request_id.clone(); + info!(%request_id, "grpc generate (stream)"); + + let stream = self.state.chat.text().generate(text_request).await; + let stream = stream.map_err(text_error_to_status)?; + + let (tx, rx) = mpsc::channel(32); + + tokio::spawn(async move { + futures::pin_mut!(stream); + while let Some(event) = stream.next().await { + let response = match event { + Err(e) => Err(text_error_to_status(e)), + Ok(DecodedTextEvent::Start { + prompt_token_ids, + prompt_logprobs, + }) => { + let prompt_info = convert::to_prompt_info( + &prompt_token_ids, + prompt_logprobs.as_ref(), + &response_opts, + ); + Ok(pb::GenerateResponse { + prompt_info: Some(prompt_info), + outputs: None, + }) + } + Ok(DecodedTextEvent::TextDelta { + delta, + token_ids, + logprobs, + finished, + }) => Ok(pb::GenerateResponse { + prompt_info: None, + outputs: Some(convert::to_sequence_output( + &delta, + &token_ids, + logprobs.as_ref(), + finished.as_ref(), + &response_opts, + )), + }), + }; + + if tx.send(response).await.is_err() { + // Client disconnected. + break; + } + } + }); + + let response_stream = ReceiverStream::new(rx); + Ok(Response::new(Box::pin(response_stream))) + } +} + +fn text_error_to_status(error: vllm_text::Error) -> Status { + let message = error.to_report_string(); + if error.is_request_validation_error() { + Status::invalid_argument(message) + } else { + Status::internal(message) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/grpc/tests.rs b/asian_ai/performance/vLLM/rust/src/server/src/grpc/tests.rs new file mode 100644 index 0000000000000000000000000000000000000000..5c2777ce98743b321fdb8a673bf695c9344f2dfc --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/grpc/tests.rs @@ -0,0 +1,1033 @@ +use std::future::Future; +use std::io; +use std::pin::Pin; +use std::sync::Arc; +use std::task::{Context, Poll}; +use std::time::Duration; + +use futures::StreamExt as _; +use hyper_util::rt::TokioIo; +use openssl::ssl::{SslConnector, SslFiletype, SslMethod}; +use serial_test::serial; +use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; +use tokio::net::TcpStream; +use tokio_openssl::SslStream; +use tonic::transport::{Channel, Endpoint, Server as TonicServer, Uri}; +use tower::service_fn; +use vllm_chat::{ + ChatBackend, ChatLlm, ChatRenderer, ChatRequest, ChatTextBackend, DefaultChatOutputProcessor, + DynChatOutputProcessor, DynChatRenderer, NewChatOutputProcessorOptions, RenderedPrompt, +}; +use vllm_engine_core_client::protocol::output::{ + EngineCoreFinishReason, EngineCoreOutput, EngineCoreOutputs, RequestBatchOutputs, +}; +use vllm_engine_core_client::protocol::request::EngineCoreRequest; +use vllm_engine_core_client::test_utils::{IpcNamespace, spawn_mock_engine_task}; +use vllm_engine_core_client::{EngineCoreClient, EngineCoreClientConfig, EngineId}; +use vllm_llm::Llm; +use vllm_text::tokenizer::DynTokenizer; +use vllm_text::{Prompt, TextBackend}; +use vllm_tokenizer::test_utils::TestTokenizer; +use zeromq::prelude::{SocketRecv, SocketSend}; +use zeromq::{DealerSocket, PushSocket, ZmqMessage}; + +use super::pb::generate_client::GenerateClient; +use super::{GenerateServer, GenerateServiceImpl, pb}; +use crate::listener::{Listener, MaybeTlsListener}; +use crate::state::AppState; +use crate::tls; +use crate::tls_tests::{TestCerts, server_tls}; + +// ======================================================================================== +// Helpers (mirrors the patterns in routes/tests.rs) +// ======================================================================================== + +type TestFuture<'a> = Pin + Send + 'a>>; + +fn boxed_test_future<'a>(future: impl Future + Send + 'a) -> TestFuture<'a> { + Box::pin(future) +} + +struct MockEngineTask { + shutdown_tx: Option>, + join_handle: Option>, +} + +impl MockEngineTask { + fn new( + (shutdown_tx, join_handle): ( + tokio::sync::oneshot::Sender<()>, + tokio::task::JoinHandle<()>, + ), + ) -> Self { + Self { + shutdown_tx: Some(shutdown_tx), + join_handle: Some(join_handle), + } + } +} + +impl Future for MockEngineTask { + type Output = Result<(), tokio::task::JoinError>; + + fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { + if let Some(shutdown_tx) = self.shutdown_tx.take() { + let _ = shutdown_tx.send(()); + } + match self.join_handle.as_mut() { + Some(join_handle) => Pin::new(join_handle).poll(cx), + None => Poll::Ready(Ok(())), + } + } +} + +impl Drop for MockEngineTask { + fn drop(&mut self) { + if let Some(join_handle) = &self.join_handle { + join_handle.abort(); + } + } +} + +fn request_output( + request_id: &str, + new_token_ids: Vec, + finish_reason: Option, +) -> EngineCoreOutput { + EngineCoreOutput { + request_id: request_id.to_string(), + new_token_ids, + new_logprobs: None, + new_prompt_logprobs_tensors: None, + pooling_output: None, + finish_reason, + stop_reason: None, + events: None, + kv_transfer_params: None, + trace_headers: None, + prefill_stats: None, + routed_experts: None, + num_nans_in_logits: 0, + } +} + +fn engine_outputs_for_request( + request_id: &str, + output_specs: Vec<(Vec, Option)>, +) -> EngineCoreOutputs { + RequestBatchOutputs { + outputs: output_specs + .into_iter() + .map(|(token_ids, finish_reason)| request_output(request_id, token_ids, finish_reason)) + .collect(), + ..Default::default() + } + .into() +} + +fn default_stream_output_specs() -> Vec<(Vec, Option)> { + vec![ + (vec![b'h' as u32], None), + (vec![b'i' as u32], None), + (vec![b'!' as u32], Some(EngineCoreFinishReason::Stop)), + ] +} + +async fn send_outputs(push: &mut PushSocket, outputs: EngineCoreOutputs) { + push.send(ZmqMessage::from( + rmp_serde::to_vec_named(&outputs).expect("encode outputs"), + )) + .await + .expect("send outputs"); +} + +async fn recv_engine_message(dealer: &mut DealerSocket) -> Vec { + dealer.recv().await.expect("recv engine message").into_vec() +} + +fn test_llm(client: EngineCoreClient) -> Llm { + Llm::new(client).with_request_id_randomization(false) +} + +#[derive(Clone, Debug)] +struct FakeTextBackend; + +impl TextBackend for FakeTextBackend { + fn tokenizer(&self) -> DynTokenizer { + Arc::new(TestTokenizer::new()) + } + + fn model_id(&self) -> &str { + "test-model" + } +} + +impl ChatBackend for FakeTextBackend { + fn chat_renderer(&self) -> DynChatRenderer { + Arc::new(self.clone()) + } + + fn new_chat_output_processor( + &self, + request: &mut ChatRequest, + options: NewChatOutputProcessorOptions<'_>, + ) -> vllm_chat::Result { + Ok(Box::new(DefaultChatOutputProcessor::new( + request, + self.model_id(), + self.tokenizer(), + options.tool_call_parser, + options.reasoning_parser, + )?)) + } +} + +impl ChatRenderer for FakeTextBackend { + fn render(&self, _request: &ChatRequest) -> vllm_chat::Result { + Ok(RenderedPrompt { + prompt: Prompt::Text(String::new()), + effective_template_kwargs: Default::default(), + }) + } +} + +/// Build the gRPC service + mock engine that serves a single request with the +/// given output specs. Shared by the plaintext and TLS server fixtures. +async fn setup_grpc_service( + engine_id: impl Into, + output_specs: Vec<(Vec, Option)>, +) -> (GenerateServer, MockEngineTask) { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = engine_id.into(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + move |dealer, push| { + boxed_test_future(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + send_outputs( + push, + engine_outputs_for_request(&request.request_id, output_specs), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + + let chat = ChatLlm::from_shared_backend( + test_llm(client), + Arc::new(FakeTextBackend) as Arc, + ); + let state = Arc::new(AppState::new(vec!["test-model".to_string()], chat)); + ( + GenerateServer::new(GenerateServiceImpl::new(state)), + engine_task, + ) +} + +/// Spin up a plaintext gRPC server backed by a mock engine. Returns the client, +/// the gRPC server task, and the mock engine task. +async fn grpc_test_server( + engine_id: impl Into, + output_specs: Vec<(Vec, Option)>, +) -> ( + GenerateClient, + tokio::task::JoinHandle<()>, + MockEngineTask, +) { + let (svc, engine_task) = setup_grpc_service(engine_id, output_specs).await; + + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("bind grpc listener"); + let addr = listener.local_addr().expect("local addr"); + + let server_task = tokio::spawn(async move { + let incoming = MaybeTlsListener::plain(Listener::Tcp(listener)); + TonicServer::builder() + .add_service(svc) + .serve_with_incoming(incoming) + .await + .expect("grpc server"); + }); + + let grpc_client = GenerateClient::connect(format!("http://{addr}")) + .await + .expect("connect grpc client"); + + (grpc_client, server_task, engine_task) +} + +/// Spin up a TLS gRPC server (server cert from `certs`, `cert_reqs` mTLS mode). +/// Returns the address, the server task, and the mock engine task. +async fn grpc_tls_test_server( + engine_id: impl Into, + output_specs: Vec<(Vec, Option)>, + certs: &TestCerts, + cert_reqs: i32, +) -> (String, tokio::task::JoinHandle<()>, MockEngineTask) { + let (svc, engine_task) = setup_grpc_service(engine_id, output_specs).await; + let context = tls::build_grpc_server_config(&server_tls(certs, cert_reqs)) + .expect("build grpc tls config"); + + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("bind grpc listener"); + let addr = listener.local_addr().expect("local addr").to_string(); + + let server_task = tokio::spawn(async move { + let incoming = MaybeTlsListener::tls(Listener::Tcp(listener), context); + TonicServer::builder() + .add_service(svc) + .serve_with_incoming(incoming) + .await + .expect("grpc tls server"); + }); + + (addr, server_task, engine_task) +} + +/// Build a tonic `Generate` client over a tokio-openssl connector, optionally +/// with a client identity for mTLS. Hand-rolled because tonic 0.14 ships no +/// OpenSSL transport. +async fn grpc_tls_client( + certs: &TestCerts, + addr: &str, + identity: Option<&str>, +) -> Result, tonic::transport::Error> { + let ca = certs.path("ca.pem"); + let identity = identity.map(|name| { + ( + certs.path(&format!("{name}.pem")), + certs.path(&format!("{name}.key")), + ) + }); + let target = addr.to_string(); + + let connector = service_fn(move |_: Uri| { + let ca = ca.clone(); + let identity = identity.clone(); + let target = target.clone(); + async move { + let tcp = TcpStream::connect(&target).await?; + let mut builder = + SslConnector::builder(SslMethod::tls_client()).map_err(io::Error::other)?; + builder.set_ca_file(&ca).map_err(io::Error::other)?; + if let Some((cert, key)) = &identity { + builder.set_certificate_chain_file(cert).map_err(io::Error::other)?; + builder.set_private_key_file(key, SslFiletype::PEM).map_err(io::Error::other)?; + } + let mut config = builder.build().configure().map_err(io::Error::other)?; + config.set_verify_hostname(false); + config.set_alpn_protos(b"\x02h2").map_err(io::Error::other)?; + let ssl = config.into_ssl("127.0.0.1").map_err(io::Error::other)?; + let mut stream = SslStream::new(ssl, tcp).map_err(io::Error::other)?; + Pin::new(&mut stream).connect().await.map_err(io::Error::other)?; + Ok::<_, io::Error>(TokioIo::new(stream)) + } + }); + + let channel = Endpoint::from_shared(format!("https://{addr}")) + .expect("grpc endpoint") + .connect_with_connector(connector) + .await?; + Ok(GenerateClient::new(channel)) +} + +/// Complete a raw TLS handshake against the gRPC port (offering ALPN `h2`) for +/// the ALPN-negotiation assertion. +async fn grpc_tls_handshake( + certs: &TestCerts, + addr: &str, +) -> io::Result>>> { + let tcp = TcpStream::connect(addr).await?; + let mut builder = SslConnector::builder(SslMethod::tls_client()).map_err(io::Error::other)?; + builder.set_ca_file(certs.path("ca.pem")).map_err(io::Error::other)?; + let mut config = builder.build().configure().map_err(io::Error::other)?; + config.set_verify_hostname(false); + config.set_alpn_protos(b"\x02h2").map_err(io::Error::other)?; + let ssl = config.into_ssl("127.0.0.1").map_err(io::Error::other)?; + let mut stream = Box::pin(SslStream::new(ssl, tcp).map_err(io::Error::other)?); + stream.as_mut().connect().await.map_err(io::Error::other)?; + Ok(stream) +} + +/// Spin up a plaintext gRPC server, optionally with HTTP/2 keepalive set to +/// `keepalive` for both the PING interval and the unanswered-PING timeout. +async fn grpc_server_with_keepalive( + engine_id: impl Into, + keepalive: Option, +) -> (String, tokio::task::JoinHandle<()>, MockEngineTask) { + let (svc, engine_task) = setup_grpc_service(engine_id, default_stream_output_specs()).await; + + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("bind grpc listener"); + let addr = listener.local_addr().expect("local addr").to_string(); + + let mut builder = TonicServer::builder(); + if let Some(interval) = keepalive { + builder = builder + .http2_keepalive_interval(Some(interval)) + .http2_keepalive_timeout(Some(interval)); + } + + let server_task = tokio::spawn(async move { + let incoming = MaybeTlsListener::plain(Listener::Tcp(listener)); + builder + .add_service(svc) + .serve_with_incoming(incoming) + .await + .expect("grpc server"); + }); + + (addr, server_task, engine_task) +} + +/// Establish an HTTP/2 connection (preface + SETTINGS exchange) then go silent, +/// ACKing the server's SETTINGS but never its keepalive PINGs. Returns whether +/// the SERVER closes the connection within `wait`. A minimal hand-rolled h2 peer +/// because a real client auto-ACKs PINGs and so can never be kept-alive-evicted. +async fn h2_unresponsive_peer_closed_within(addr: &str, wait: Duration) -> bool { + let mut tcp = TcpStream::connect(addr).await.expect("connect"); + tcp.write_all(b"PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n").await.expect("preface"); + tcp.write_all(&[0, 0, 0, 0x4, 0, 0, 0, 0, 0]).await.expect("client settings"); + + let closed = tokio::time::timeout(wait, async { + let mut header = [0u8; 9]; + while tcp.read_exact(&mut header).await.is_ok() { + let len = u32::from_be_bytes([0, header[0], header[1], header[2]]) as usize; + let frame_type = header[3]; + let flags = header[4]; + let mut payload = vec![0u8; len]; + if tcp.read_exact(&mut payload).await.is_err() { + return; + } + // ACK the server's SETTINGS so the only thing left unanswered is PINGs. + if frame_type == 0x4 && flags & 0x1 == 0 { + let _ = tcp.write_all(&[0, 0, 0, 0x4, 0x1, 0, 0, 0, 0]).await; + } + } + }) + .await; + + closed.is_ok() +} + +// ======================================================================================== +// Tests +// ======================================================================================== + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn unary_generate_returns_collected_text() { + let (mut client, server_task, engine_task) = + grpc_test_server(b"engine-grpc-unary", default_stream_output_specs()).await; + + let response = client + .generate(pb::GenerateRequest { + request_id: "test-unary-1".to_string(), + model: "test-model".to_string(), + prompt: Some(pb::generate_request::Prompt::Text("hello".to_string())), + stopping: Some(pb::StoppingCriteria { + max_new_tokens: 10, + ..Default::default() + }), + response: Some(pb::ResponseOptions { + output_text: Some(true), + ..Default::default() + }), + ..Default::default() + }) + .await + .expect("unary generate") + .into_inner(); + + // Unary collects all tokens into one response. + let outputs = response.outputs.expect("outputs present"); + assert_eq!(outputs.text, "hi"); + + let finish = outputs.finish_info.expect("finish_info present"); + assert_eq!( + finish.finish_reason, + pb::finish_info::FinishReason::Stop as i32 + ); + assert_eq!(finish.num_output_tokens, 3); + + let prompt = response.prompt_info.expect("prompt_info present"); + assert_eq!(prompt.num_prompt_tokens, 5); // "hello" = 5 bytes + + engine_task.await.expect("mock engine task"); + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn unary_generate_with_token_ids_prompt() { + let (mut client, server_task, engine_task) = + grpc_test_server(b"engine-grpc-token-ids", default_stream_output_specs()).await; + + let response = client + .generate(pb::GenerateRequest { + request_id: "test-token-ids".to_string(), + model: "test-model".to_string(), + prompt: Some(pb::generate_request::Prompt::TokenIds(pb::TokenIds { + ids: vec![1, 2, 3], + })), + stopping: Some(pb::StoppingCriteria { + max_new_tokens: 10, + ..Default::default() + }), + ..Default::default() + }) + .await + .expect("unary generate with token ids") + .into_inner(); + + let outputs = response.outputs.expect("outputs present"); + assert_eq!(outputs.text, "hi"); + assert_eq!( + response.prompt_info.expect("prompt_info").num_prompt_tokens, + 3 + ); + + engine_task.await.expect("mock engine task"); + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn unary_generate_returns_token_ids_when_requested() { + let (mut client, server_task, engine_task) = + grpc_test_server(b"engine-grpc-tok-resp", default_stream_output_specs()).await; + + let response = client + .generate(pb::GenerateRequest { + request_id: "test-tok-resp".to_string(), + model: "test-model".to_string(), + prompt: Some(pb::generate_request::Prompt::Text("hi".to_string())), + stopping: Some(pb::StoppingCriteria { + max_new_tokens: 10, + ..Default::default() + }), + response: Some(pb::ResponseOptions { + output_text: Some(true), + output_token_ids: true, + prompt_token_ids: true, + ..Default::default() + }), + ..Default::default() + }) + .await + .expect("unary generate") + .into_inner(); + + let outputs = response.outputs.expect("outputs present"); + assert_eq!( + outputs.token_ids, + vec![b'h' as u32, b'i' as u32, b'!' as u32] + ); + + let prompt = response.prompt_info.expect("prompt_info present"); + assert_eq!(prompt.token_ids, vec![b'h' as u32, b'i' as u32]); + + engine_task.await.expect("mock engine task"); + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn unary_generate_missing_prompt_returns_invalid_argument() { + let (mut client, server_task, _engine_task) = + grpc_test_server(b"engine-grpc-no-prompt", default_stream_output_specs()).await; + + let status = client + .generate(pb::GenerateRequest { + request_id: "test-no-prompt".to_string(), + model: "test-model".to_string(), + prompt: None, + ..Default::default() + }) + .await + .expect_err("should fail without prompt"); + + assert_eq!(status.code(), tonic::Code::InvalidArgument); + assert!(status.message().contains("prompt")); + + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn unary_generate_min_tokens_above_max_tokens_returns_invalid_argument() { + let (mut client, server_task, _engine_task) = + grpc_test_server(b"engine-grpc-min-above-max", default_stream_output_specs()).await; + + let status = client + .generate(pb::GenerateRequest { + request_id: "test-min-above-max".to_string(), + model: "test-model".to_string(), + prompt: Some(pb::generate_request::Prompt::Text("hi".to_string())), + stopping: Some(pb::StoppingCriteria { + max_new_tokens: 4, + min_new_tokens: 5, + ..Default::default() + }), + ..Default::default() + }) + .await + .expect_err("should fail when min_new_tokens exceeds max_new_tokens"); + + assert_eq!(status.code(), tonic::Code::InvalidArgument); + assert!(status.message().contains("min_tokens=5")); + assert!(status.message().contains("max_tokens=4")); + + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn streaming_generate_yields_incremental_responses() { + let (mut client, server_task, engine_task) = + grpc_test_server(b"engine-grpc-stream", default_stream_output_specs()).await; + + let stream = client + .generate_stream(pb::GenerateRequest { + request_id: "test-stream-1".to_string(), + model: "test-model".to_string(), + prompt: Some(pb::generate_request::Prompt::Text("hello".to_string())), + stopping: Some(pb::StoppingCriteria { + max_new_tokens: 10, + ..Default::default() + }), + response: Some(pb::ResponseOptions { + output_text: Some(true), + ..Default::default() + }), + ..Default::default() + }) + .await + .expect("streaming generate") + .into_inner(); + + let responses: Vec = + stream.map(|r| r.expect("stream item")).collect().await; + + // First response carries prompt info, subsequent ones carry output deltas. + assert!( + responses.len() >= 2, + "expected at least 2 streamed responses, got {}", + responses.len() + ); + + // First message should have prompt info. + let first = &responses[0]; + let prompt_info = first.prompt_info.as_ref().expect("first response has prompt_info"); + assert_eq!(prompt_info.num_prompt_tokens, 5); // "hello" + + // Collect all text deltas. + let full_text: String = responses + .iter() + .filter_map(|r| r.outputs.as_ref()) + .map(|o| o.text.as_str()) + .collect(); + assert_eq!(full_text, "hi"); + + // Last output response should have finish info. + let last_output = responses + .iter() + .rev() + .find_map(|r| r.outputs.as_ref()) + .expect("at least one output"); + let finish = last_output.finish_info.as_ref().expect("finish_info on last output"); + assert_eq!( + finish.finish_reason, + pb::finish_info::FinishReason::Stop as i32 + ); + + engine_task.await.expect("mock engine task"); + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn streaming_generate_missing_prompt_returns_invalid_argument() { + let (mut client, server_task, _engine_task) = grpc_test_server( + b"engine-grpc-stream-no-prompt", + default_stream_output_specs(), + ) + .await; + + let status = client + .generate_stream(pb::GenerateRequest { + request_id: "test-stream-no-prompt".to_string(), + model: "test-model".to_string(), + prompt: None, + ..Default::default() + }) + .await + .expect_err("should fail without prompt"); + + assert_eq!(status.code(), tonic::Code::InvalidArgument); + + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn streaming_generate_min_tokens_above_max_tokens_returns_invalid_argument() { + let (mut client, server_task, _engine_task) = grpc_test_server( + b"engine-grpc-stream-min-above-max", + default_stream_output_specs(), + ) + .await; + + let status = client + .generate_stream(pb::GenerateRequest { + request_id: "test-stream-min-above-max".to_string(), + model: "test-model".to_string(), + prompt: Some(pb::generate_request::Prompt::Text("hi".to_string())), + stopping: Some(pb::StoppingCriteria { + max_new_tokens: 4, + min_new_tokens: 5, + ..Default::default() + }), + ..Default::default() + }) + .await + .expect_err("should fail when min_new_tokens exceeds max_new_tokens"); + + assert_eq!(status.code(), tonic::Code::InvalidArgument); + assert!(status.message().contains("min_tokens=5")); + assert!(status.message().contains("max_tokens=4")); + + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn unary_generate_with_sampling_params() { + let (mut client, server_task, engine_task) = + grpc_test_server(b"engine-grpc-sampling", default_stream_output_specs()).await; + + let response = client + .generate(pb::GenerateRequest { + request_id: "test-sampling".to_string(), + model: "test-model".to_string(), + prompt: Some(pb::generate_request::Prompt::Text("test".to_string())), + temperature: Some(0.7), + sampling: Some(pb::RandomSampling { + top_k: 50, + top_p: 0.9, + seed: Some(42), + ..Default::default() + }), + stopping: Some(pb::StoppingCriteria { + max_new_tokens: 5, + ..Default::default() + }), + ..Default::default() + }) + .await + .expect("generate with sampling params") + .into_inner(); + + // Verify the request was accepted and produced output. + let outputs = response.outputs.expect("outputs present"); + assert_eq!(outputs.text, "hi"); + + engine_task.await.expect("mock engine task"); + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn unary_generate_rejects_wrong_model() { + let (mut client, server_task, _engine_task) = + grpc_test_server(b"engine-grpc-wrong-model", default_stream_output_specs()).await; + + let status = client + .generate(pb::GenerateRequest { + request_id: "test-wrong-model".to_string(), + model: "other-model".to_string(), + prompt: Some(pb::generate_request::Prompt::Text("hi".to_string())), + stopping: Some(pb::StoppingCriteria { + max_new_tokens: 10, + ..Default::default() + }), + ..Default::default() + }) + .await + .expect_err("should fail with wrong model"); + + assert_eq!(status.code(), tonic::Code::NotFound); + assert!(status.message().contains("other-model")); + + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn streaming_generate_rejects_wrong_model() { + let (mut client, server_task, _engine_task) = grpc_test_server( + b"engine-grpc-stream-wrong-model", + default_stream_output_specs(), + ) + .await; + + let status = client + .generate_stream(pb::GenerateRequest { + request_id: "test-stream-wrong-model".to_string(), + model: "other-model".to_string(), + prompt: Some(pb::generate_request::Prompt::Text("hi".to_string())), + stopping: Some(pb::StoppingCriteria { + max_new_tokens: 10, + ..Default::default() + }), + ..Default::default() + }) + .await + .expect_err("should fail with wrong model"); + + assert_eq!(status.code(), tonic::Code::NotFound); + assert!(status.message().contains("other-model")); + + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn unary_generate_accepts_empty_model() { + let (mut client, server_task, engine_task) = + grpc_test_server(b"engine-grpc-empty-model", default_stream_output_specs()).await; + + // Empty `model` (proto3 default) is treated as "unset" and should be accepted. + let response = client + .generate(pb::GenerateRequest { + request_id: "test-empty-model".to_string(), + model: String::new(), + prompt: Some(pb::generate_request::Prompt::Text("hi".to_string())), + stopping: Some(pb::StoppingCriteria { + max_new_tokens: 10, + ..Default::default() + }), + ..Default::default() + }) + .await + .expect("unary generate with empty model") + .into_inner(); + + let outputs = response.outputs.expect("outputs present"); + assert_eq!(outputs.text, "hi"); + + engine_task.await.expect("mock engine task"); + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn unary_generate_output_text_defaults_to_true() { + let (mut client, server_task, engine_task) = + grpc_test_server(b"engine-grpc-default-text", default_stream_output_specs()).await; + + // No response options at all — output_text should default to true. + let response = client + .generate(pb::GenerateRequest { + request_id: "test-default-text".to_string(), + model: "test-model".to_string(), + prompt: Some(pb::generate_request::Prompt::Text("x".to_string())), + stopping: Some(pb::StoppingCriteria { + max_new_tokens: 10, + ..Default::default() + }), + ..Default::default() + }) + .await + .expect("unary generate") + .into_inner(); + + let outputs = response.outputs.expect("outputs present"); + assert_eq!(outputs.text, "hi"); + + engine_task.await.expect("mock engine task"); + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn grpc_generate_succeeds_over_tls() { + let certs = TestCerts::generate(); + let (addr, server_task, engine_task) = grpc_tls_test_server( + b"engine-grpc-tls-unary", + default_stream_output_specs(), + &certs, + 0, + ) + .await; + + let mut client = grpc_tls_client(&certs, &addr, None).await.expect("tls client"); + let response = client + .generate(pb::GenerateRequest { + request_id: "test-tls-unary".to_string(), + model: "test-model".to_string(), + prompt: Some(pb::generate_request::Prompt::Text("hello".to_string())), + stopping: Some(pb::StoppingCriteria { + max_new_tokens: 10, + ..Default::default() + }), + response: Some(pb::ResponseOptions { + output_text: Some(true), + ..Default::default() + }), + ..Default::default() + }) + .await + .expect("unary generate over tls") + .into_inner(); + + assert_eq!(response.outputs.expect("outputs present").text, "hi"); + + engine_task.await.expect("mock engine task"); + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn grpc_tls_negotiates_h2_alpn() { + let certs = TestCerts::generate(); + let (addr, server_task, _engine_task) = grpc_tls_test_server( + b"engine-grpc-tls-alpn", + default_stream_output_specs(), + &certs, + 0, + ) + .await; + + let stream = grpc_tls_handshake(&certs, &addr).await.expect("handshake"); + assert_eq!( + stream.ssl().selected_alpn_protocol(), + Some(&b"h2"[..]), + "server must negotiate h2 ALPN" + ); + + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn grpc_mtls_required_rejects_client_without_certificate() { + let certs = TestCerts::generate(); + let (addr, server_task, _engine_task) = grpc_tls_test_server( + b"engine-grpc-tls-mtls-reject", + default_stream_output_specs(), + &certs, + 2, + ) + .await; + + // With TLS 1.3 the missing-client-cert rejection surfaces on first use, not + // at the handshake, so drive an RPC and assert the call fails. + let outcome = match grpc_tls_client(&certs, &addr, None).await { + Err(_) => Err(()), + Ok(mut client) => client + .generate(pb::GenerateRequest { + request_id: "test-tls-mtls-reject".to_string(), + model: "test-model".to_string(), + prompt: Some(pb::generate_request::Prompt::Text("hello".to_string())), + stopping: Some(pb::StoppingCriteria { + max_new_tokens: 10, + ..Default::default() + }), + ..Default::default() + }) + .await + .map(|_| ()) + .map_err(|_| ()), + }; + assert!( + outcome.is_err(), + "mTLS-required gRPC must reject a client without a certificate" + ); + + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn grpc_mtls_required_accepts_valid_client_certificate() { + let certs = TestCerts::generate(); + let (addr, server_task, engine_task) = grpc_tls_test_server( + b"engine-grpc-tls-mtls-accept", + default_stream_output_specs(), + &certs, + 2, + ) + .await; + + let mut client = grpc_tls_client(&certs, &addr, Some("client")).await.expect("mtls client"); + let response = client + .generate(pb::GenerateRequest { + request_id: "test-tls-mtls".to_string(), + model: "test-model".to_string(), + prompt: Some(pb::generate_request::Prompt::Text("hello".to_string())), + stopping: Some(pb::StoppingCriteria { + max_new_tokens: 10, + ..Default::default() + }), + response: Some(pb::ResponseOptions { + output_text: Some(true), + ..Default::default() + }), + ..Default::default() + }) + .await + .expect("mtls generate over tls") + .into_inner(); + + assert_eq!(response.outputs.expect("outputs present").text, "hi"); + + engine_task.await.expect("mock engine task"); + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn grpc_keepalive_closes_unresponsive_connection() { + let (addr, server_task, _engine_task) = + grpc_server_with_keepalive(b"engine-grpc-keepalive", Some(Duration::from_millis(150))) + .await; + + let closed = h2_unresponsive_peer_closed_within(&addr, Duration::from_secs(5)).await; + assert!( + closed, + "keepalive must close a peer that stops answering PINGs" + ); + + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn grpc_without_keepalive_keeps_unresponsive_connection_open() { + // Without keepalive the same unresponsive peer is NOT + // closed, proving the close above is attributable to keepalive. + let (addr, server_task, _engine_task) = + grpc_server_with_keepalive(b"engine-grpc-no-keepalive", None).await; + + let closed = h2_unresponsive_peer_closed_within(&addr, Duration::from_secs(1)).await; + assert!( + !closed, + "without keepalive an idle h2 connection must stay open" + ); + + server_task.abort(); +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/lib.rs b/asian_ai/performance/vLLM/rust/src/server/src/lib.rs new file mode 100644 index 0000000000000000000000000000000000000000..de1fb81514c510db821892cbdd5a5b6c87044029 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/lib.rs @@ -0,0 +1,422 @@ +//! Minimal OpenAI-compatible HTTP server above [`vllm_chat`]. + +mod config; +mod error; +mod grpc; +mod listener; +mod lora; +mod middleware; +mod routes; +mod runtime; +mod server_info; +mod state; +mod tls; +#[cfg(test)] +mod tls_tests; +mod utils; + +use std::future::Future; +use std::sync::{Arc, OnceLock}; +use std::time::Duration; + +use anyhow::{Context as _, Result}; +use axum::Router; +use axum::body::Body; +use axum::http::Request; +pub use config::{ + ApiServerOptions, Config, CoordinatorMode, CorsConfig, DEFAULT_KEEP_ALIVE_TIMEOUT, + HttpListenerMode, TlsConfig, +}; +use hyper::body::Incoming; +use hyper::server::conn::http1; +use hyper_util::rt::{TokioIo, TokioTimer}; +use hyper_util::server::graceful::GracefulShutdown; +use hyper_util::service::TowerToHyperService; +use tokio::net::TcpListener; +use tokio::time::{Instant, sleep_until}; +use tokio_util::sync::CancellationToken; +use tonic::transport::Server as TonicServer; +use tower::ServiceExt as _; +use tracing::{info, trace, warn}; +use vllm_chat::{ChatLlm, LoadModelBackendsOptions, load_model_backends}; +pub use vllm_chat::{ChatTemplateContentFormatOption, ParserSelection, RendererSelection}; +use vllm_engine_core_client::{EngineCoreClient, EngineCoreClientConfig}; +use vllm_llm::Llm; +use vllm_text::TextLlm; + +use crate::listener::{Listener, MaybeTlsListener}; +use crate::routes::build_router; +use crate::server_info::ServerInfoSnapshot; +use crate::state::AppState; + +/// How often the server PINGs an idle gRPC connection to reap a dead peer; +/// tonic enables no keepalive by default. 2h matches the gRPC-core default. +const GRPC_KEEPALIVE_INTERVAL: Duration = Duration::from_secs(7200); +/// How long the server waits for a keepalive PING reply before dropping the gRPC +/// connection. 20s matches the gRPC-core default. +const GRPC_KEEPALIVE_TIMEOUT: Duration = Duration::from_secs(20); + +/// Resolve the public model names accepted by the frontend. +fn effective_served_model_names(model: &str, served_model_name: &[String]) -> Vec { + if served_model_name.is_empty() { + vec![model.to_string()] + } else { + served_model_name.to_vec() + } +} + +/// Choose the gRPC listener host. It follows the HTTP TCP host when there is +/// one; otherwise (unix socket or inherited fd) it defaults to IPv4 loopback +/// rather than all interfaces, so the side-car is never accidentally +/// network-exposed. +fn grpc_bind_host(listener_mode: &HttpListenerMode) -> &str { + match listener_mode { + HttpListenerMode::BindTcp { host, .. } => host.as_str(), + HttpListenerMode::BindUnix { .. } | HttpListenerMode::InheritedFd { .. } => "127.0.0.1", + } +} + +/// Build the shared application state for one configured model and one engine +/// client. +async fn build_state(config: &Config) -> Result> { + // If no served names are specified, fall back to the backend model path so + // that the API always has at least one valid model ID. Use the same primary + // public name for frontend-side metrics labels. + let served_model_names = effective_served_model_names(&config.model, &config.served_model_name); + let metrics_model_name = served_model_names[0].clone(); + + // Load both backends from the same model metadata so they stay in sync. + let loaded = load_model_backends( + &config.model, + LoadModelBackendsOptions { + renderer: config.renderer, + language_model_only: config.language_model_only, + chat_template: config.chat_template.clone(), + chat_template_content_format: config.chat_template_content_format, + default_chat_template_kwargs: config + .default_chat_template_kwargs + .clone() + .unwrap_or_default(), + }, + ) + .await + .context("failed to create chat/text backends")?; + let text_backend = loaded.text_backend; + let chat_backend = loaded.chat_backend; + + let coordinator_mode = config.effective_coordinator_mode(text_backend.is_moe()); + info!( + engine_count = config.engine_count(), + model_is_moe = text_backend.is_moe(), + ?coordinator_mode, + "resolved coordinator mode" + ); + + let client = EngineCoreClient::connect(EngineCoreClientConfig { + transport_mode: config.transport_mode.clone(), + coordinator_mode, + model_name: metrics_model_name, + client_index: 0, + }) + .await + .context("failed to connect to engine core")?; + + let llm = Llm::new(client).with_log_stats(!config.disable_log_stats); + let text = TextLlm::new(llm, text_backend).with_max_logprobs(config.max_logprobs); + + let chat = ChatLlm::new(text, chat_backend) + .with_tool_call_parser(config.tool_call_parser.clone()) + .with_reasoning_parser(config.reasoning_parser.clone()); + + Ok(Arc::new( + AppState::new(served_model_names, chat) + .with_model_path(config.model.clone()) + .with_api_server_options(config.api_server_options) + .with_server_info(ServerInfoSnapshot::from_config(config)) + .with_api_keys(config.api_keys.clone()) + .with_cors(config.cors.clone()) + .with_profiler(config.profiler.clone()), + )) +} + +/// Run the OpenAI-compatible HTTP server until the supplied shutdown token is +/// cancelled. +/// +/// The server owns one `vllm-chat` facade, which in turn owns the lower +/// `vllm-text` and `vllm-llm` layers, and shuts them down before returning. +pub async fn serve(config: Config, shutdown: CancellationToken) -> Result<()> { + serve_with_router_extension(config, shutdown, |router| router).await +} + +/// Run the OpenAI-compatible HTTP server with an opt-in router extension. +/// +/// The extension receives the finalized vLLM router and can merge additional +/// routes before the server starts accepting requests. +pub async fn serve_with_router_extension( + config: Config, + shutdown: CancellationToken, + extend_router: F, +) -> Result<()> +where + F: FnOnce(Router) -> Router, +{ + config.validate().context("invalid OpenAI frontend configuration")?; + + // Build the TLS server config once, up front, so a bad cert/key fails fast + // before the (potentially long) engine handshake. + let tls_config = config + .tls + .as_ref() + .map(tls::build_server_config) + .transpose() + .context("invalid TLS configuration")?; + + // Also check shutdown during the (potentially long) startup handshake. + let state = tokio::select! { + result = build_state(&config) => result?, + _ = shutdown.cancelled() => return Ok(()), + }; + let listener = Listener::bind(&config.listener_mode) + .await + .context("failed to bind listener for OpenAI server")?; + let bind_address = listener.local_addr_display()?; + let model = state.primary_model_name().to_owned(); + let app = extend_router(build_router(state.clone())); + + // Optionally bind the gRPC Generate server on a separate port. Bind + // synchronously here so bind errors (port in use, permission denied, ...) + // surface before serving rather than being deferred until shutdown. + let grpc_setup = if let Some(grpc_port) = config.grpc_port { + let grpc_host = grpc_bind_host(&config.listener_mode); + let grpc_listener = TcpListener::bind((grpc_host, grpc_port)) + .await + .with_context(|| format!("failed to bind gRPC listener on {grpc_host}:{grpc_port}"))?; + let addr = grpc_listener.local_addr()?; + let grpc_listener = Listener::Tcp(grpc_listener); + // gRPC reuses the HTTP TLS config (same SslContext) plus ALPN h2. + let grpc_tls = config + .tls + .as_ref() + .map(tls::build_grpc_server_config) + .transpose() + .context("invalid gRPC TLS configuration")?; + let svc = grpc::GenerateServer::new(grpc::GenerateServiceImpl::new(state.clone())); + let svc = TonicServer::builder() + .http2_keepalive_interval(Some(GRPC_KEEPALIVE_INTERVAL)) + .http2_keepalive_timeout(Some(GRPC_KEEPALIVE_TIMEOUT)) + .layer(middleware::request_runtime_layer(state.clone())) + .add_service(svc); + info!(%addr, tls = grpc_tls.is_some(), "starting gRPC server"); + Some((grpc_listener, svc, grpc_tls)) + } else { + None + }; + + let scheme = if tls_config.is_some() { + "https" + } else { + "http" + }; + info!(%bind_address, %scheme, %model, "starting OpenAI server"); + + // Run HTTP and gRPC concurrently under a child token of the caller's shutdown + // token. Caller cancellation propagates into both protocols; if either + // protocol exits first, we cancel this child token so its sibling also + // begins a graceful drain. + let server_shutdown = shutdown.child_token(); + let force_shutdown = CancellationToken::new(); + let shutdown_deadline = Arc::new(OnceLock::new()); + + // Spawn a task to trigger `force_shutdown` after shutdown deadline elapses. + tokio::spawn({ + let shutdown = server_shutdown.clone(); + let force_shutdown = force_shutdown.clone(); + let shutdown_deadline = shutdown_deadline.clone(); + let shutdown_timeout = config.shutdown_timeout; + + async move { + shutdown.cancelled().await; + let deadline = Instant::now() + shutdown_timeout; + let _ = shutdown_deadline.set(deadline); + + if shutdown_timeout.is_zero() { + force_shutdown.cancel(); + } else { + sleep_until(deadline).await; + force_shutdown.cancel(); + } + } + }); + + // 0 disables keep-alive but still bounds the head read (default), so a + // silent client cannot hold the connection open. + let keep_alive_timeout = config.keep_alive_timeout; + let timeouts = ConnectionTimeouts { + header_read: if keep_alive_timeout.is_zero() { + DEFAULT_KEEP_ALIVE_TIMEOUT + } else { + keep_alive_timeout + }, + keep_alive_enabled: !keep_alive_timeout.is_zero(), + }; + + let http_fut = { + let shutdown = server_shutdown.child_token(); + let server_shutdown = server_shutdown.clone(); + let force_shutdown = force_shutdown.clone(); + async move { + let listener = match tls_config { + Some(context) => MaybeTlsListener::tls(listener, context), + None => MaybeTlsListener::plain(listener), + }; + let server = serve_connections(listener, app, shutdown.cancelled_owned(), timeouts); + + let result = tokio::select! { + result = server => { + result.context("HTTP server failed") + } + _ = force_shutdown.cancelled() => { + warn!("HTTP graceful shutdown deadline elapsed; aborting server"); + Ok(()) + } + }; + + server_shutdown.cancel(); + result + } + }; + + let grpc_fut = { + let shutdown = server_shutdown.child_token(); + let server_shutdown = server_shutdown.clone(); + let force_shutdown = force_shutdown.clone(); + async move { + let Some((grpc_listener, svc, grpc_tls)) = grpc_setup else { + // No gRPC configured: just wait for shutdown so we do not race the + // join! by resolving early and tripping the cancellation token. + shutdown.cancelled().await; + return Ok(()); + }; + let incoming = match grpc_tls { + Some(context) => MaybeTlsListener::tls(grpc_listener, context), + None => MaybeTlsListener::plain(grpc_listener), + }; + let server = svc.serve_with_incoming_shutdown(incoming, shutdown.cancelled_owned()); + + let result = tokio::select! { + result = server => { + result.context("gRPC server failed") + } + _ = force_shutdown.cancelled() => { + warn!("gRPC graceful shutdown deadline elapsed; aborting server"); + Ok(()) + } + }; + + server_shutdown.cancel(); + result + } + }; + + let (http_res, grpc_res) = tokio::join!(http_fut, grpc_fut); + http_res.and(grpc_res)?; + + let shutdown_deadline = shutdown_deadline + .get() + .copied() + .unwrap_or_else(|| Instant::now() + config.shutdown_timeout); + state.shutdown(shutdown_deadline).await +} + +/// Per-connection timeouts applied while serving HTTP/HTTPS. +#[derive(Clone, Copy)] +pub(crate) struct ConnectionTimeouts { + /// HTTP/1 header-read timeout (bounds idle keep-alive and the head read). + pub(crate) header_read: Duration, + /// Whether HTTP/1 keep-alive is enabled; `false` closes after each response. + pub(crate) keep_alive_enabled: bool, +} + +/// Serve `app` per connection (HTTP/1) with a keep-alive idle timeout and +/// graceful drain. Hand-rolled on hyper because [`axum::serve()`] takes no config. +async fn serve_connections( + mut listener: L, + app: Router, + shutdown: impl Future + Send, + timeouts: ConnectionTimeouts, +) -> Result<()> +where + L: axum::serve::Listener, +{ + let graceful = GracefulShutdown::new(); + let mut shutdown = std::pin::pin!(shutdown); + loop { + let (io, _addr) = tokio::select! { + conn = listener.accept() => conn, + () = &mut shutdown => break, + }; + + let service = TowerToHyperService::new( + app.clone().map_request(|req: Request| req.map(Body::new)), + ); + let mut builder = http1::Builder::new(); + builder.timer(TokioTimer::new()).header_read_timeout(timeouts.header_read); + if !timeouts.keep_alive_enabled { + builder.keep_alive(false); + } + let connection = builder.serve_connection(TokioIo::new(io), service); + let connection = graceful.watch(connection); + + tokio::spawn(async move { + if let Err(err) = connection.await { + trace!(error = %err, "failed to serve connection"); + } + }); + } + + drop(listener); + graceful.shutdown().await; + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn effective_served_model_names_falls_back_to_backend_model() { + assert_eq!( + effective_served_model_names("backend-model", &[]), + vec!["backend-model"] + ); + } + + #[test] + fn effective_served_model_names_preserves_public_names() { + let served_names = vec!["public-model".to_string(), "public-alias".to_string()]; + + assert_eq!( + effective_served_model_names("backend-model", &served_names), + served_names + ); + } + + #[test] + fn grpc_bind_host_follows_http_tcp_host() { + let mode = HttpListenerMode::BindTcp { + host: "0.0.0.0".to_string(), + port: 8000, + }; + assert_eq!(grpc_bind_host(&mode), "0.0.0.0"); + } + + #[test] + fn grpc_bind_host_defaults_to_loopback_without_tcp_host() { + let unix = HttpListenerMode::BindUnix { + path: "/tmp/vllm.sock".to_string(), + }; + let inherited = HttpListenerMode::InheritedFd { fd: 3 }; + assert_eq!(grpc_bind_host(&unix), "127.0.0.1"); + assert_eq!(grpc_bind_host(&inherited), "127.0.0.1"); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/listener.rs b/asian_ai/performance/vLLM/rust/src/server/src/listener.rs new file mode 100644 index 0000000000000000000000000000000000000000..375640184d301333b50208022c64262932853dd9 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/listener.rs @@ -0,0 +1,307 @@ +//! Unified listener wrapper for the Rust frontend. +//! +//! This module hides the difference between TCP and Unix-domain listeners so +//! the rest of the server can bind or inherit one socket and pass it to +//! `axum::serve(...)` through a single type. + +use std::io::Result; +use std::net::{SocketAddr, TcpListener as StdTcpListener}; +use std::os::fd::{FromRawFd, IntoRawFd, OwnedFd}; +use std::os::unix::net::UnixListener as StdUnixListener; +use std::pin::Pin; +use std::task::{Context, Poll, ready}; + +use auto_enums::enum_derive; +use openssl::ssl::SslContext; +use socket2::Socket; +use tls_listener::{AsyncAccept, AsyncListener}; +use tokio::net::{TcpListener, TcpStream, UnixListener, UnixStream}; +use tonic::transport::server::{Connected, TcpConnectInfo}; +use tracing::trace; + +use crate::{HttpListenerMode, tls}; + +/// Runtime listener type used by the OpenAI-compatible HTTP or gRPC server, +/// which is either a TCP listener or a Unix-domain listener. +#[derive(Debug)] +pub enum Listener { + Tcp(TcpListener), + Unix(UnixListener), +} + +/// Runtime listener I/O type which is either a TCP stream or a Unix-domain stream. +#[derive(Debug)] +#[enum_derive(tokio1::AsyncRead, tokio1::AsyncWrite)] +pub enum ListenerIo { + Tcp(TcpStream), + Unix(UnixStream), +} + +/// Runtime listener address type which is either a TCP address or a Unix-domain address. +#[derive(Debug)] +#[allow(dead_code)] +pub enum ListenerAddr { + Tcp(SocketAddr), + Unix(tokio::net::unix::SocketAddr), +} + +impl Listener { + /// Bind or adopt the listener described by the frontend configuration. + /// + /// For inherited sockets, the concrete listener kind is detected from the + /// socket family of the supplied file descriptor. + pub async fn bind(mode: &HttpListenerMode) -> Result { + match mode { + HttpListenerMode::BindTcp { host, port } => { + Ok(Self::Tcp(TcpListener::bind((host.as_str(), *port)).await?)) + } + HttpListenerMode::BindUnix { path } => Ok(Self::Unix(UnixListener::bind(path)?)), + HttpListenerMode::InheritedFd { fd } => Self::from_inherited_fd(*fd), + } + } + + /// Return a log-friendly local address string for either TCP or Unix + /// sockets. + pub fn local_addr_display(&self) -> Result { + match self { + Self::Tcp(listener) => Ok(listener.local_addr()?.to_string()), + Self::Unix(listener) => Ok(match listener.local_addr()?.as_pathname() { + Some(path) => format!("unix:{}", path.display()), + None => "unix:".to_string(), + }), + } + } + + fn from_inherited_fd(fd: i32) -> Result { + // SAFETY: We trust the caller to only pass valid listener fds, and we only use + // this fd once to create a single listener. + let owned_fd = unsafe { OwnedFd::from_raw_fd(fd) }; + let socket = Socket::from(owned_fd); + + // The Python supervisor pre-binds the socket to reserve the endpoint early, but + // Rust is responsible for transitioning inherited stream sockets into + // the listening state before accepting connections. + socket.listen(libc::SOMAXCONN)?; + socket.set_nonblocking(true)?; + + if socket.local_addr()?.is_unix() { + let std_listener = unsafe { StdUnixListener::from_raw_fd(socket.into_raw_fd()) }; + Ok(Self::Unix(UnixListener::from_std(std_listener)?)) + } else { + let std_listener = unsafe { StdTcpListener::from_raw_fd(socket.into_raw_fd()) }; + Ok(Self::Tcp(TcpListener::from_std(std_listener)?)) + } + } + + fn local_addr(&self) -> Result { + match self { + Self::Tcp(listener) => listener.local_addr().map(ListenerAddr::Tcp), + Self::Unix(listener) => listener.local_addr().map(ListenerAddr::Unix), + } + } +} + +/// Allow the unified listener to plug directly into tonic's gRPC server. +impl Connected for ListenerIo { + type ConnectInfo = TcpConnectInfo; + + fn connect_info(&self) -> TcpConnectInfo { + match self { + Self::Tcp(stream) => stream.connect_info(), + Self::Unix(_) => TcpConnectInfo { + local_addr: None, + remote_addr: None, + }, + } + } +} + +/// Attempt to set `TCP_NODELAY` on the accepted TCP stream. +fn enable_tcp_nodelay(stream: TcpStream) -> TcpStream { + if let Err(err) = stream.set_nodelay(true) { + trace!(error = %err, "failed to enable TCP_NODELAY on accepted TCP connection"); + } + stream +} + +/// Allow the unified listener to plug directly into `axum::serve(...)`. +impl axum::serve::Listener for Listener { + type Addr = ListenerAddr; + type Io = ListenerIo; + + async fn accept(&mut self) -> (Self::Io, Self::Addr) { + match self { + Self::Tcp(listener) => { + let (io, addr) = axum::serve::Listener::accept(listener).await; + ( + ListenerIo::Tcp(enable_tcp_nodelay(io)), + ListenerAddr::Tcp(addr), + ) + } + Self::Unix(listener) => { + let (io, addr) = axum::serve::Listener::accept(listener).await; + (ListenerIo::Unix(io), ListenerAddr::Unix(addr)) + } + } + } + + fn local_addr(&self) -> Result { + self.local_addr() + } +} + +/// Allow the unified listener to be adaptable to `tls_listener`. +impl AsyncAccept for Listener { + type Address = ListenerAddr; + type Connection = ListenerIo; + type Error = std::io::Error; + + fn poll_accept( + self: Pin<&mut Self>, + cx: &mut Context<'_>, + ) -> Poll> { + match self.get_mut() { + Self::Tcp(listener) => { + let (io, addr) = ready!(listener.poll_accept(cx))?; + Poll::Ready(Ok(( + ListenerIo::Tcp(enable_tcp_nodelay(io)), + ListenerAddr::Tcp(addr), + ))) + } + Self::Unix(listener) => { + let (io, addr) = ready!(listener.poll_accept(cx))?; + Poll::Ready(Ok((ListenerIo::Unix(io), ListenerAddr::Unix(addr)))) + } + } + } +} +impl AsyncListener for Listener { + fn local_addr(&self) -> Result { + self.local_addr() + } +} + +/// A listener that may be either a plain TCP/UDS listener or a TLS listener over it. +pub enum MaybeTlsListener { + Plain(Listener), + Tls(tls_listener::TlsListener), +} + +impl MaybeTlsListener { + /// Create a plain listener without TLS. + pub fn plain(listener: Listener) -> Self { + Self::Plain(listener) + } + + /// Create a TLS listener over the given plain listener. + pub fn tls(listener: Listener, context: SslContext) -> Self { + Self::Tls( + tls_listener::builder(context) + .handshake_timeout(tls::TLS_HANDSHAKE_TIMEOUT) + .listen(listener), + ) + } +} + +/// Listener I/O type that may be either a plain TCP/UDS stream or a TLS stream over it. +#[derive(Debug)] +#[enum_derive(tokio1::AsyncRead, tokio1::AsyncWrite)] +pub enum MaybeTlsStream { + Plain(ListenerIo), + Tls(tokio_openssl::SslStream), +} + +/// Allow the maybe-TLS listener to plug directly into `axum::serve(...)`. +impl axum::serve::Listener for MaybeTlsListener { + type Addr = ListenerAddr; + type Io = MaybeTlsStream; + + async fn accept(&mut self) -> (Self::Io, Self::Addr) { + match self { + Self::Plain(listener) => { + let (io, addr) = axum::serve::Listener::accept(listener).await; + (MaybeTlsStream::Plain(io), addr) + } + Self::Tls(tls_listener) => { + let (io, addr) = axum::serve::Listener::accept(tls_listener).await; + (MaybeTlsStream::Tls(io), addr) + } + } + } + + fn local_addr(&self) -> tokio::io::Result { + match self { + Self::Plain(listener) => listener.local_addr(), + Self::Tls(tls_listener) => tls_listener.local_addr(), + } + } +} + +/// Allow the maybe-TLS listener to plug directly into tonic's gRPC server. +impl Connected for MaybeTlsStream { + type ConnectInfo = TcpConnectInfo; + + fn connect_info(&self) -> TcpConnectInfo { + match self { + Self::Plain(stream) => stream.connect_info(), + Self::Tls(stream) => stream.get_ref().connect_info(), + } + } +} + +/// Allow the maybe-TLS listener to be adaptable to tonic's incoming stream shape. +impl futures::Stream for MaybeTlsListener { + type Item = std::io::Result; + + fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + match self.get_mut() { + Self::Plain(listener) => { + let listener = Pin::new(listener); + let (io, _) = ready!(listener.poll_accept(cx))?; + Poll::Ready(Some(Ok(MaybeTlsStream::Plain(io)))) + } + Self::Tls(tls_listener) => { + let tls_listener = Pin::new(tls_listener); + let (io, _) = + ready!(tls_listener.poll_accept(cx)).map_err(std::io::Error::other)?; + Poll::Ready(Some(Ok(MaybeTlsStream::Tls(io)))) + } + } + } +} + +#[cfg(test)] +mod tests { + use std::net::{Ipv4Addr, SocketAddrV4}; + use std::os::fd::IntoRawFd; + + use socket2::{Domain, SockAddr, Socket, Type}; + use uuid::Uuid; + + use super::Listener; + use crate::HttpListenerMode; + + #[tokio::test(flavor = "current_thread")] + async fn inherited_fd_detects_tcp_listener_without_uds_hint() { + let socket = Socket::new(Domain::IPV4, Type::STREAM, None).unwrap(); + socket.bind(&SockAddr::from(SocketAddrV4::new(Ipv4Addr::LOCALHOST, 0))).unwrap(); + let fd = socket.into_raw_fd(); + + let listener = Listener::bind(&HttpListenerMode::InheritedFd { fd }).await.unwrap(); + + assert!(matches!(listener, Listener::Tcp(_))); + } + + #[tokio::test(flavor = "current_thread")] + async fn inherited_fd_detects_unix_listener_from_fd() { + let path = std::env::temp_dir().join(format!("vllm-rs-{}.sock", Uuid::new_v4())); + let socket = Socket::new(Domain::UNIX, Type::STREAM, None).unwrap(); + socket.bind(&SockAddr::unix(&path).unwrap()).unwrap(); + let fd = socket.into_raw_fd(); + + let listener = Listener::bind(&HttpListenerMode::InheritedFd { fd }).await.unwrap(); + + assert!(matches!(listener, Listener::Unix(_))); + let _ = std::fs::remove_file(path); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/lora.rs b/asian_ai/performance/vLLM/rust/src/server/src/lora.rs new file mode 100644 index 0000000000000000000000000000000000000000..a3276cb2606f4f01bfce9991f871d31f4dc9fe92 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/lora.rs @@ -0,0 +1,165 @@ +use std::sync::atomic::{AtomicU64, Ordering}; + +use indexmap::IndexMap; +use tokio::sync::{Mutex, RwLock}; +use vllm_engine_core_client::EngineCoreClient; +use vllm_engine_core_client::protocol::lora::LoraRequest; + +/// Snapshot of the currently served model names plus the requested LoRA, if +/// the model name resolves to a dynamic adapter. +#[derive(Debug, Clone)] +pub(crate) struct LoraModelResolution { + pub model_names: Vec, + pub lora_request: Option, +} + +/// Runtime registry for dynamically loaded LoRA adapters. +pub(crate) struct LoraManager { + /// Dynamically loaded LoRA adapters keyed by public model name, in load order. + requests: RwLock>, + /// Monotonic adapter id allocator. LoRA ids are one-indexed. + id_counter: AtomicU64, + /// Serialize dynamic LoRA registry updates around engine utility calls. + update_lock: Mutex<()>, +} + +#[derive(Debug)] +pub(crate) enum LoadLoraError { + AlreadyLoaded { lora_name: String }, + BaseModelName { lora_name: String }, + Engine(vllm_engine_core_client::Error), + NotLoaded { lora_name: String }, +} + +#[derive(Debug)] +pub(crate) enum UnloadLoraError { + NotFound { + lora_name: String, + }, + IntIdMismatch { + lora_name: String, + expected: u64, + actual: u64, + }, + Engine(vllm_engine_core_client::Error), + NotRemoved { + lora_name: String, + lora_int_id: u64, + }, +} + +impl LoraManager { + pub fn new() -> Self { + Self { + requests: RwLock::new(IndexMap::new()), + id_counter: AtomicU64::new(0), + update_lock: Mutex::new(()), + } + } + + /// Snapshot loaded LoRA adapters in load order. + pub async fn served_lora_requests(&self) -> Vec { + self.requests.read().await.values().cloned().collect() + } + + /// Resolve the requested model against one consistent LoRA registry + /// snapshot. + pub async fn resolve_model( + &self, + base_model_names: &[String], + model_name: Option<&str>, + ) -> LoraModelResolution { + let requests = self.requests.read().await; + let mut model_names = base_model_names.to_vec(); + model_names.extend(requests.keys().cloned()); + let lora_request = model_name.and_then(|name| requests.get(name).cloned()); + + LoraModelResolution { + model_names, + lora_request, + } + } + + /// Load one dynamic LoRA adapter and register it as a public model name. + pub async fn load_lora( + &self, + engine_core_client: &EngineCoreClient, + base_model_names: &[String], + lora_name: String, + lora_path: String, + load_inplace: bool, + is_3d_lora_weight: bool, + ) -> Result { + let _guard = self.update_lock.lock().await; + if base_model_names.iter().any(|name| name == &lora_name) { + return Err(LoadLoraError::BaseModelName { lora_name }); + } + if !load_inplace && self.requests.read().await.contains_key(&lora_name) { + return Err(LoadLoraError::AlreadyLoaded { lora_name }); + } + + let lora_int_id = self + .requests + .read() + .await + .get(&lora_name) + .map(|request| request.lora_int_id) + .unwrap_or_else(|| self.id_counter.fetch_add(1, Ordering::Relaxed) + 1); + let lora_request = LoraRequest::new( + lora_name.clone(), + lora_int_id, + lora_path, + load_inplace, + is_3d_lora_weight, + ); + + let loaded = engine_core_client + .add_lora(&lora_request) + .await + .map_err(LoadLoraError::Engine)?; + if !loaded { + return Err(LoadLoraError::NotLoaded { lora_name }); + } + self.requests.write().await.insert(lora_name, lora_request.clone()); + Ok(lora_request) + } + + /// Remove one dynamic LoRA adapter from the engine and public model + /// registry. + pub async fn unload_lora( + &self, + engine_core_client: &EngineCoreClient, + lora_name: &str, + requested_lora_int_id: Option, + ) -> Result { + let _guard = self.update_lock.lock().await; + let lora_request = self.requests.read().await.get(lora_name).cloned().ok_or_else(|| { + UnloadLoraError::NotFound { + lora_name: lora_name.to_string(), + } + })?; + + if let Some(actual) = requested_lora_int_id + && actual != lora_request.lora_int_id + { + return Err(UnloadLoraError::IntIdMismatch { + lora_name: lora_name.to_string(), + expected: lora_request.lora_int_id, + actual, + }); + } + + let removed = engine_core_client + .remove_lora(lora_request.lora_int_id) + .await + .map_err(UnloadLoraError::Engine)?; + if !removed { + return Err(UnloadLoraError::NotRemoved { + lora_name: lora_request.lora_name, + lora_int_id: lora_request.lora_int_id, + }); + } + + Ok(self.requests.write().await.shift_remove(lora_name).unwrap_or(lora_request)) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/middleware/auth.rs b/asian_ai/performance/vLLM/rust/src/server/src/middleware/auth.rs new file mode 100644 index 0000000000000000000000000000000000000000..00e04130a53896aac8716f88c228d985fb822098 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/middleware/auth.rs @@ -0,0 +1,91 @@ +use std::sync::Arc; + +use axum::Json; +use axum::extract::{Request, State}; +use axum::http::header::AUTHORIZATION; +use axum::http::{HeaderValue, Method, StatusCode}; +use axum::middleware::Next; +use axum::response::{IntoResponse, Response}; +use serde_json::json; + +use crate::state::{ApiKeyHash, AppState, hash_api_key}; + +const GUARDED_PREFIXES: &[&str] = &["/v1", "/v2", "/inference"]; + +/// Authenticate guarded HTTP routes with an OpenAI-compatible bearer token. +/// +/// Mirrors Python `AuthenticationMiddleware`: OPTIONS requests and non-guarded +/// helper endpoints such as `/health` are allowed through without a token. +pub async fn authenticate_api_key( + State(state): State>, + req: Request, + next: Next, +) -> Response { + if req.method() == Method::OPTIONS || !requires_auth(req.uri().path()) { + return next.run(req).await; + } + + if verify_token(req.headers().get(AUTHORIZATION), state.api_key_hashes()) { + return next.run(req).await; + } + + ( + StatusCode::UNAUTHORIZED, + Json(json!({ "error": "Unauthorized" })), + ) + .into_response() +} + +fn requires_auth(path: &str) -> bool { + GUARDED_PREFIXES.iter().any(|prefix| path.starts_with(prefix)) +} + +fn verify_token(authorization: Option<&HeaderValue>, api_key_hashes: &[ApiKeyHash]) -> bool { + let Some(authorization) = authorization else { + return false; + }; + let Ok(authorization) = authorization.to_str() else { + return false; + }; + let Some((scheme, token)) = authorization.split_once(' ') else { + return false; + }; + if !scheme.eq_ignore_ascii_case("bearer") { + return false; + } + + let token_hash = hash_api_key(token); + let mut token_match = false; + for api_key_hash in api_key_hashes { + token_match |= constant_time_eq(&token_hash, api_key_hash); + } + token_match +} + +fn constant_time_eq(left: &ApiKeyHash, right: &ApiKeyHash) -> bool { + use subtle::ConstantTimeEq; + + bool::from(left.ct_eq(right)) +} + +#[cfg(test)] +mod tests { + use super::constant_time_eq; + use crate::state::hash_api_key; + + #[test] + fn constant_time_eq_checks_sha256_digests() { + assert!(constant_time_eq( + &hash_api_key("secret"), + &hash_api_key("secret") + )); + assert!(!constant_time_eq( + &hash_api_key("secret"), + &hash_api_key("secrex") + )); + assert!(!constant_time_eq( + &hash_api_key("secret"), + &hash_api_key("secret-more") + )); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/middleware/cors.rs b/asian_ai/performance/vLLM/rust/src/server/src/middleware/cors.rs new file mode 100644 index 0000000000000000000000000000000000000000..e164c5864918feddf1a3277aaeeb11e2674f25bb --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/middleware/cors.rs @@ -0,0 +1,141 @@ +//! CORS support mirroring Python's Starlette `CORSMiddleware`. +//! +//! Built on `tower_http::cors::CorsLayer`, configured to reproduce Starlette's +//! `CORSMiddleware` behavior for the `--allowed-origins` / `--allowed-methods` / +//! `--allowed-headers` / `--allow-credentials` settings. Two intentional +//! behavioral differences remain, both invisible to real clients: +//! +//! - A rejected preflight returns `200` (empty) rather than Starlette's +//! `400 "Disallowed CORS ..."`. The browser denies the request either way +//! (the disallowed `Access-Control-Allow-*` headers are simply absent), and +//! tower-http makes the preflight reject decision inside its short-circuit, +//! so matching the `400` would mean re-implementing the layer. +//! - A bare `OPTIONS` (no `Access-Control-Request-Method`) returns `200` +//! rather than `405`. No real client sends one. + +use std::time::Duration; + +use axum::extract::Request; +use axum::http::{HeaderName, HeaderValue, Method, header}; +use axum::middleware::Next; +use axum::response::Response; +use tower_http::cors::{AllowHeaders, AllowMethods, AllowOrigin, CorsLayer}; + +use crate::config::CorsConfig; + +/// The method set that `"*"` expands to. +const ALL_METHODS: [Method; 7] = [ + Method::DELETE, + Method::GET, + Method::HEAD, + Method::OPTIONS, + Method::PATCH, + Method::POST, + Method::PUT, +]; + +/// Headers always treated as allowed (the CORS safelist). +const SAFELISTED_HEADERS: [&str; 4] = [ + "accept", + "accept-language", + "content-language", + "content-type", +]; + +fn is_wildcard(values: &[String]) -> bool { + values.iter().any(|value| value == "*") +} + +/// Build a `CorsLayer` from the resolved [`CorsConfig`]. +/// +/// Values are assumed valid: [`CorsConfig::validate`] runs at startup before +/// the router is built. +pub fn cors_layer(cfg: &CorsConfig) -> CorsLayer { + let wildcard_origins = is_wildcard(&cfg.allow_origins); + + let allow_origin = if wildcard_origins { + if cfg.allow_credentials { + // `*` with credentials is illegal, so reflect the request origin + // instead; this also avoids tower-http's wildcard+credentials panic. + AllowOrigin::mirror_request() + } else { + AllowOrigin::any() + } + } else { + AllowOrigin::list( + cfg.allow_origins + .iter() + .map(|origin| origin.parse::().expect("validated origin")) + .collect::>(), + ) + }; + + // Expand `*` to an explicit list rather than `Any`, so we emit the method + // names (not `*`) and never hit tower-http's `Any`+credentials panic. + let allow_methods = if is_wildcard(&cfg.allow_methods) { + AllowMethods::list(ALL_METHODS) + } else { + AllowMethods::list( + cfg.allow_methods + .iter() + .map(|method| method.parse::().expect("validated method")) + .collect::>(), + ) + }; + + let allow_headers = if is_wildcard(&cfg.allow_headers) { + // `*` mirrors the requested headers. + AllowHeaders::mirror_request() + } else { + // Union the safelisted headers, lowercased and sorted. + let mut names: Vec = SAFELISTED_HEADERS.iter().map(|s| s.to_string()).collect(); + names.extend(cfg.allow_headers.iter().map(|h| h.to_ascii_lowercase())); + names.sort(); + names.dedup(); + AllowHeaders::list( + names + .iter() + .map(|header| header.parse::().expect("validated header")) + .collect::>(), + ) + }; + + // Emit `Vary: Origin` only when the allow-origin is dynamic (explicit + // origins, or credentials); the wildcard + no-credentials case emits no + // `Vary` at all, and an empty list disables the header here. + let vary: Vec = if !wildcard_origins || cfg.allow_credentials { + vec![header::ORIGIN] + } else { + vec![] + }; + + CorsLayer::new() + .allow_origin(allow_origin) + .allow_methods(allow_methods) + .allow_headers(allow_headers) + .allow_credentials(cfg.allow_credentials) + .max_age(Duration::from_secs(600)) + .vary(vary) +} + +/// Strip CORS response headers when the request carried no `Origin`. +/// +/// A request without an `Origin` should carry no CORS headers, but tower-http +/// emits `Vary` and `Access-Control-Allow-*` unconditionally. Removing them on +/// no-`Origin` requests keeps non-CORS responses (e.g. `/health`, plain `curl`) +/// clean. +pub async fn strip_cors_on_no_origin(req: Request, next: Next) -> Response { + let had_origin = req.headers().contains_key(header::ORIGIN); + let mut response = next.run(req).await; + if !had_origin { + let headers = response.headers_mut(); + headers.remove(header::VARY); + headers.remove(header::ACCESS_CONTROL_ALLOW_ORIGIN); + headers.remove(header::ACCESS_CONTROL_ALLOW_CREDENTIALS); + headers.remove(header::ACCESS_CONTROL_ALLOW_METHODS); + headers.remove(header::ACCESS_CONTROL_ALLOW_HEADERS); + headers.remove(header::ACCESS_CONTROL_MAX_AGE); + headers.remove(header::ACCESS_CONTROL_EXPOSE_HEADERS); + } + response +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/middleware/load.rs b/asian_ai/performance/vLLM/rust/src/server/src/middleware/load.rs new file mode 100644 index 0000000000000000000000000000000000000000..22c428404338a9e768faf2cdddced9271f29a6d7 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/middleware/load.rs @@ -0,0 +1,110 @@ +use std::pin::Pin; +use std::sync::{Arc, Weak}; +use std::task::{Context, Poll}; + +use axum::body::{Body, Bytes, HttpBody}; +use axum::extract::{MatchedPath, Request, State}; +use axum::middleware::Next; +use axum::response::Response; +use http_body::{Frame, SizeHint}; + +use crate::state::AppState; + +/// Endpoints that will be tracked for server load. +/// +/// Derived from the Python frontend's actual `@load_aware_call` coverage. This +/// includes alias paths that delegate into decorated handlers, such as +/// `/v1/rerank` and `/v2/rerank`. +const TRACKED_HANDLERS: &[&str] = &[ + "/v1/responses", + "/v1/responses/{response_id}", + "/v1/responses/{response_id}/cancel", + "/v1/messages", + "/v1/messages/count_tokens", + "/v1/chat/completions", + "/v1/completions", + "/v1/audio/transcriptions", + "/v1/audio/translations", + "/v1/embeddings", + "/pooling", + "/classify", + "/score", + "/v1/score", + "/rerank", + "/v1/rerank", + "/v2/rerank", + "/inference/v1/generate", +]; + +/// Track frontend-local in-flight inference requests for the `/load` endpoint. +pub async fn track_server_load( + State(state): State>, + req: Request, + next: Next, +) -> Response { + let handler = req + .extensions() + .get::() + .map_or_else(|| "none", |path| path.as_str()); + + if !TRACKED_HANDLERS.contains(&handler) { + return next.run(req).await; + } + + state.increment_server_load(); + let guard = ServerLoadGuard { + state: Arc::downgrade(&state), + }; + let response = next.run(req).await; + + let (parts, body) = response.into_parts(); + Response::from_parts( + parts, + Body::new(LoadTrackedBody { + inner: body, + _guard: guard, + }), + ) +} + +/// A guard that decrements the server load when dropped. +struct ServerLoadGuard { + state: Weak, +} + +impl Drop for ServerLoadGuard { + fn drop(&mut self) { + if let Some(state) = self.state.upgrade() { + state.decrement_server_load(); + } + } +} + +/// A wrapper around response bodies that tracks server load by holding a +/// `ServerLoadGuard`, which will decrement the load when the body is fully +/// consumed and dropped. +struct LoadTrackedBody { + inner: Body, + _guard: ServerLoadGuard, +} + +// Simply delegate all `HttpBody` methods to the inner body. +impl HttpBody for LoadTrackedBody { + type Data = Bytes; + type Error = axum::Error; + + fn poll_frame( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + ) -> Poll, Self::Error>>> { + Pin::new(&mut self.inner).poll_frame(cx) + } + + fn is_end_stream(&self) -> bool { + self.inner.is_end_stream() + } + + fn size_hint(&self) -> SizeHint { + self.inner.size_hint() + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/middleware/metrics.rs b/asian_ai/performance/vLLM/rust/src/server/src/middleware/metrics.rs new file mode 100644 index 0000000000000000000000000000000000000000..40a982529210a0976e34bb59c7e7681f23fd7d4a --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/middleware/metrics.rs @@ -0,0 +1,79 @@ +use std::time::Instant; + +use axum::extract::{MatchedPath, Request}; +use axum::middleware::Next; +use axum::response::Response; +use vllm_metrics::{HttpHandlerLabels, HttpRequestLabels, METRICS}; + +/// Endpoints that will be excluded from HTTP metrics tracking. +/// +/// Original Python definition: +/// +const EXCLUDED_HANDLERS: &[&str] = &[ + "/metrics", + "/health", + "/load", + "/ping", + "/version", + "/server_info", + // Rust frontend extra: + "/reset_prefix_cache", + "/reset_mm_cache", + "/reset_encoder_cache", + "/collective_rpc", + "/sleep", + "/wake_up", + "/is_sleeping", +]; + +/// Record API-server HTTP metrics with Python-compatible +/// (`PrometheusFastApiInstrumentator` style) family names and labels. +pub async fn track_http_metrics(req: Request, next: Next) -> Response { + let method = req.method().as_str().to_string(); + let handler = req + .extensions() + .get::() + .map_or_else(|| "none".to_string(), |path| path.as_str().to_string()); + let excluded = EXCLUDED_HANDLERS.contains(&handler.as_str()); + let started_at = Instant::now(); + + let response = next.run(req).await; + + if excluded { + return response; + } + + let elapsed = started_at.elapsed().as_secs_f64(); + let status = status_group(response.status().as_u16()); + + let metrics = &METRICS.api_server; + + metrics + .http_requests + .get_or_create(&HttpRequestLabels { + method: method.clone(), + status, + handler: handler.clone(), + }) + .inc(); + + metrics + .http_request_duration_seconds + .get_or_create(&HttpHandlerLabels { method, handler }) + .observe(elapsed); + + metrics.http_request_duration_highr_seconds.observe(elapsed); + + response +} + +fn status_group(status: u16) -> &'static str { + match status / 100 { + 1 => "1xx", + 2 => "2xx", + 3 => "3xx", + 4 => "4xx", + 5 => "5xx", + _ => "unknown", + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/middleware/mod.rs b/asian_ai/performance/vLLM/rust/src/server/src/middleware/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..2b3474d6ae57d071274bd2c7dbbc1c7000d0ba5f --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/middleware/mod.rs @@ -0,0 +1,13 @@ +mod auth; +mod cors; +mod load; +mod metrics; +mod offload; +mod request_id; + +pub use auth::authenticate_api_key; +pub use cors::{cors_layer, strip_cors_on_no_origin}; +pub use load::track_server_load; +pub use metrics::track_http_metrics; +pub(crate) use offload::request_runtime_layer; +pub use request_id::set_request_id_header; diff --git a/asian_ai/performance/vLLM/rust/src/server/src/middleware/offload.rs b/asian_ai/performance/vLLM/rust/src/server/src/middleware/offload.rs new file mode 100644 index 0000000000000000000000000000000000000000..f7e735d44c0d875deb3343e40a1b2fe35db9cf1f --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/middleware/offload.rs @@ -0,0 +1,134 @@ +use std::sync::Arc; +use std::task::{Context, Poll}; + +use axum::http::Request; +use axum::response::{IntoResponse, Response}; +use futures::future::BoxFuture; +use tokio_util::task::AbortOnDropHandle; +use tonic::Status; +use tower::Service; +use tower::layer::layer_fn; +use tracing::error; + +use crate::error::{ApiError, server_error}; +use crate::state::AppState; + +/// Request paths that are run on the request runtime. +/// +/// These routes can perform CPU-heavy request preparation, including JSON +/// extraction, validation, chat-template rendering, tokenization, request +/// lowering, and engine submission. Lightweight operational routes stay on the +/// HTTP runtime. +const OFFLOADED_PATHS: &[&str] = &[ + // HTTP routes: + "/v1/chat/completions", + "/v1/completions", + "/tokenize", + "/detokenize", + "/inference/v1/generate", + // gRPC routes: + "/vllm.Generate/Generate", + "/vllm.Generate/GenerateStream", +]; + +/// Return a Tower layer that runs selected data-plane requests on the request runtime, +/// so that we can offset heavy request parsing and preprocessing from the HTTP runtime. +pub(crate) fn request_runtime_layer( + state: Arc, +) -> impl tower::Layer> + Clone { + layer_fn(move |inner| RequestRuntimeService { + inner, + state: state.clone(), + }) +} + +/// Service produced by [`request_runtime_layer`]. +#[derive(Clone)] +pub(crate) struct RequestRuntimeService { + inner: S, + state: Arc, +} + +impl Service> for RequestRuntimeService +where + S: Service> + Clone + Send + 'static, + S::Future: Send + 'static, + S::Response: RequestRuntimeErrorResponse + Send + 'static, + S::Error: Send + 'static, + B: Send + 'static, +{ + type Error = S::Error; + type Future = BoxFuture<'static, Result>; + type Response = S::Response; + + fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll> { + self.inner.poll_ready(cx) + } + + fn call(&mut self, req: Request) -> Self::Future { + if !should_offload(req.uri().path()) { + return Box::pin(self.inner.call(req)); + } + + // Axum extractors and route handlers execute inside the inner service, + // so offloading here moves request parsing and preprocessing off the + // HTTP runtime without wrapping each handler manually. For streaming + // HTTP responses, the response body is still polled on the HTTP runtime. + let clone = self.inner.clone(); + let mut inner = std::mem::replace(&mut self.inner, clone); + let task = AbortOnDropHandle::new(self.state.request_runtime().spawn(inner.call(req))); + + Box::pin(async move { + match task.await { + Ok(result) => result, + Err(error) => { + error!(%error, "request runtime task failed"); + Ok(S::Response::request_runtime_error_response()) + } + } + }) + } +} + +trait RequestRuntimeErrorResponse { + fn request_runtime_error_response() -> Self; +} + +impl RequestRuntimeErrorResponse for Response { + fn request_runtime_error_response() -> Self { + server_error!("request runtime task failed").into_response() + } +} + +impl RequestRuntimeErrorResponse for axum::http::Response { + fn request_runtime_error_response() -> Self { + Status::internal("request runtime task failed").into_http() + } +} + +fn should_offload(path: &str) -> bool { + OFFLOADED_PATHS.contains(&path) +} + +#[cfg(test)] +mod tests { + use super::should_offload; + + #[test] + fn offloads_generation_and_tokenization_paths() { + assert!(should_offload("/v1/chat/completions")); + assert!(should_offload("/v1/completions")); + assert!(should_offload("/tokenize")); + assert!(should_offload("/detokenize")); + assert!(should_offload("/inference/v1/generate")); + assert!(should_offload("/vllm.Generate/Generate")); + assert!(should_offload("/vllm.Generate/GenerateStream")); + } + + #[test] + fn passes_through_lightweight_paths() { + assert!(!should_offload("/health")); + assert!(!should_offload("/metrics")); + assert!(!should_offload("/v1/models")); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/middleware/request_id.rs b/asian_ai/performance/vLLM/rust/src/server/src/middleware/request_id.rs new file mode 100644 index 0000000000000000000000000000000000000000..9499ebb8d7287db400878c323c7fef5e1004a53f --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/middleware/request_id.rs @@ -0,0 +1,24 @@ +use axum::extract::Request; +use axum::http::HeaderValue; +use axum::http::header::HeaderName; +use axum::middleware::Next; +use axum::response::Response; +use uuid::Uuid; + +const X_REQUEST_ID: HeaderName = HeaderName::from_static("x-request-id"); + +/// Echo the request's `X-Request-Id` on the response, or generate a fresh +/// `uuid4` hex if the request did not provide one. +/// +/// Original Python: +/// `vllm.entrypoints.openai.server_utils.XRequestIdMiddleware`. +pub async fn set_request_id_header(req: Request, next: Next) -> Response { + let incoming = req.headers().get(&X_REQUEST_ID).cloned(); + let mut response = next.run(req).await; + let value = incoming.unwrap_or_else(|| { + HeaderValue::from_str(&Uuid::new_v4().simple().to_string()) + .expect("uuid hex is valid header value") + }); + response.headers_mut().insert(X_REQUEST_ID, value); + response +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes.rs new file mode 100644 index 0000000000000000000000000000000000000000..948d71833f30fd32e6c67453e5e10185dadff539 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes.rs @@ -0,0 +1,159 @@ +mod abort_requests; +mod cache; +mod collective_rpc; +mod health; +mod inference; +mod load; +mod lora; +mod metrics; +pub(crate) mod openai; +mod pause; +mod profile; +mod server_info; +mod sleep; +mod tokenize; +mod version; +mod world_size; + +use std::sync::Arc; + +use axum::Router; +use axum::extract::DefaultBodyLimit; +use axum::middleware::{from_fn, from_fn_with_state}; +use axum::routing::{get, post}; +use tower_http::trace::TraceLayer; +use tracing::warn; + +use crate::middleware; +use crate::state::AppState; + +const DEFAULT_JSON_BODY_LIMIT_BYTES: usize = 32 * 1024 * 1024; + +fn server_dev_mode_enabled() -> bool { + std::env::var("VLLM_SERVER_DEV_MODE") + .ok() + .and_then(|value| value.parse::().ok()) + .is_some_and(|value| value != 0) +} + +fn runtime_lora_updating_enabled() -> bool { + std::env::var("VLLM_ALLOW_RUNTIME_LORA_UPDATING") + .ok() + .is_some_and(|value| matches!(value.trim().to_lowercase().as_str(), "1" | "true")) +} + +/// Build the minimal OpenAI-compatible router for one configured model. +pub fn build_router(state: Arc) -> Router { + build_router_with_options( + state, + server_dev_mode_enabled(), + runtime_lora_updating_enabled(), + ) +} + +#[cfg(test)] +fn build_router_with_dev_mode(state: Arc, dev_mode_enabled: bool) -> Router { + build_router_with_dev_mode_and_lora(state, dev_mode_enabled, false) +} + +#[cfg(test)] +fn build_router_with_dev_mode_and_lora( + state: Arc, + dev_mode_enabled: bool, + runtime_lora_updating_enabled: bool, +) -> Router { + build_router_with_options(state, dev_mode_enabled, runtime_lora_updating_enabled) +} + +fn build_router_with_options( + state: Arc, + dev_mode_enabled: bool, + runtime_lora_updating_enabled: bool, +) -> Router { + let mut router = Router::new() + // Health & monitoring + .route("/health", get(health::health)) + .route("/metrics", get(metrics::scrape)) + .route("/load", get(load::load)) + .route("/version", get(version::version)) + // OpenAI-compatible endpoints + .route("/v1/models", get(openai::list_models)) + .route("/v1/completions", post(openai::completions)) + .route("/v1/chat/completions", post(openai::chat_completions)) + // vLLM specific endpoints + .route("/tokenize", post(tokenize::tokenize)) + .route("/detokenize", post(tokenize::detokenize)) + .route("/inference/v1/generate", post(inference::generate)); + + if runtime_lora_updating_enabled { + router = router + .route("/v1/load_lora_adapter", post(lora::load_lora_adapter)) + .route("/v1/unload_lora_adapter", post(lora::unload_lora_adapter)); + } + + if dev_mode_enabled { + // Development-only + router = router + .route("/reset_prefix_cache", post(cache::reset_prefix_cache)) + .route("/reset_mm_cache", post(cache::reset_mm_cache)) + .route("/reset_encoder_cache", post(cache::reset_encoder_cache)) + .route("/collective_rpc", post(collective_rpc::collective_rpc)) + .route("/abort_requests", post(abort_requests::abort_requests)) + .route("/sleep", post(sleep::sleep)) + .route("/wake_up", post(sleep::wake_up)) + .route("/is_sleeping", get(sleep::is_sleeping)) + .route("/pause", post(pause::pause)) + .route("/resume", post(pause::resume)) + .route("/is_paused", get(pause::is_paused)) + .route("/server_info", get(server_info::server_info)) + .route("/get_world_size", get(world_size::get_world_size)) + } + + if let Some(profiler) = &state.profiler { + warn!( + mode = profiler, + "profiler is enabled in the API server; \ + this should only be used for local development", + ); + router = router + .route("/start_profile", post(profile::start_profile)) + .route("/stop_profile", post(profile::stop_profile)); + } + + let enable_request_id_headers = state.api_server_options.enable_request_id_headers; + let enable_api_key_auth = state.has_api_keys(); + let mut router = router + .with_state(state.clone()) + .layer(DefaultBodyLimit::max(DEFAULT_JSON_BODY_LIMIT_BYTES)) + .layer(middleware::request_runtime_layer(state.clone())) + .layer(from_fn_with_state( + state.clone(), + middleware::track_server_load, + )) + .layer(from_fn(middleware::track_http_metrics)) + .layer(middleware::cors_layer(&state.cors)) + .layer(from_fn(middleware::strip_cors_on_no_origin)); + + if enable_api_key_auth { + router = router.layer(from_fn_with_state( + state.clone(), + middleware::authenticate_api_key, + )); + } + + // Later layers wrap earlier ones. Keep tracing outside auth so rejected + // requests are visible, while metrics/load only see authenticated traffic. + router = router.layer(TraceLayer::new_for_http()); + + if enable_request_id_headers { + router = router.layer(from_fn(middleware::set_request_id_header)); + } + + router +} + +#[cfg(test)] +mod tests; + +#[cfg(test)] +mod http_client_tests; diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/abort_requests.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/abort_requests.rs new file mode 100644 index 0000000000000000000000000000000000000000..5ea1e427963072589b333d17930142d485ff91f7 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/abort_requests.rs @@ -0,0 +1,37 @@ +use std::sync::Arc; + +use axum::Json; +use axum::extract::State; +use axum::extract::rejection::JsonRejection; +use axum::http::StatusCode; +use serde::Deserialize; + +use crate::error::ApiError; +use crate::state::AppState; +use crate::utils::utility_call_error; + +#[derive(Debug, Deserialize)] +pub(crate) struct AbortRequestsRequest { + request_ids: Option>, +} + +pub async fn abort_requests( + State(state): State>, + body: Result, JsonRejection>, +) -> Result { + let Json(body) = body.map_err(|error| ApiError::json_parse_error(error.body_text()))?; + let request_ids = body.request_ids.ok_or_else(|| { + ApiError::invalid_request( + "Missing 'request_ids' in request body".to_string(), + Some("request_ids"), + ) + })?; + + state + .chat + .abort(&request_ids) + .await + .map_err(|error| utility_call_error("abort_requests", error))?; + + Ok(StatusCode::OK) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/cache.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/cache.rs new file mode 100644 index 0000000000000000000000000000000000000000..d9c37b2bd5d2897548feb8c41b6dd67a4c750c06 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/cache.rs @@ -0,0 +1,56 @@ +use std::sync::Arc; + +use axum::extract::{Query, State}; +use axum::http::StatusCode; +use serde::Deserialize; + +use crate::error::ApiError; +use crate::state::AppState; +use crate::utils::utility_call_error; + +#[derive(Debug, Default, Deserialize)] +pub(crate) struct ResetPrefixCacheParams { + #[serde(default)] + reset_running_requests: bool, + #[serde(default)] + reset_external: bool, +} + +/// Reset the local prefix cache and optionally the connector-managed external +/// cache. +pub async fn reset_prefix_cache( + State(state): State>, + Query(params): Query, +) -> Result { + state + .engine_core_client() + .reset_prefix_cache(params.reset_running_requests, params.reset_external) + .await + .map_err(|error| utility_call_error("reset_prefix_cache", error))?; + + Ok(StatusCode::OK) +} + +/// Reset the multi-modal cache. +pub async fn reset_mm_cache(State(state): State>) -> Result { + state + .engine_core_client() + .reset_mm_cache() + .await + .map_err(|error| utility_call_error("reset_mm_cache", error))?; + + Ok(StatusCode::OK) +} + +/// Reset the encoder cache. +pub async fn reset_encoder_cache( + State(state): State>, +) -> Result { + state + .engine_core_client() + .reset_encoder_cache() + .await + .map_err(|error| utility_call_error("reset_encoder_cache", error))?; + + Ok(StatusCode::OK) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/collective_rpc.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/collective_rpc.rs new file mode 100644 index 0000000000000000000000000000000000000000..797b9d10e43f8c0c42d72cad4e043ecf1d967b32 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/collective_rpc.rs @@ -0,0 +1,51 @@ +use std::collections::BTreeMap; +use std::sync::Arc; + +use axum::Json; +use axum::extract::State; +use axum::extract::rejection::JsonRejection; +use rmpv::Value as MsgpackValue; +use serde::{Deserialize, Serialize}; +use serde_json::Value as JsonValue; + +use crate::error::ApiError; +use crate::state::AppState; +use crate::utils::utility_call_error; + +#[derive(Debug, Deserialize)] +pub(crate) struct CollectiveRpcRequest { + method: Option, + #[serde(default)] + timeout: Option, + #[serde(default)] + args: Vec, + #[serde(default)] + kwargs: BTreeMap, +} + +#[derive(Debug, Serialize)] +pub(crate) struct CollectiveRpcResponse { + results: Vec, +} + +/// Execute a development-only collective RPC on the connected engine(s). +pub async fn collective_rpc( + State(state): State>, + body: Result, JsonRejection>, +) -> Result, ApiError> { + let Json(body) = body.map_err(|error| ApiError::json_parse_error(error.body_text()))?; + let method = body.method.ok_or_else(|| { + ApiError::invalid_request( + "Missing 'method' in request body".to_string(), + Some("method"), + ) + })?; + + let results = state + .engine_core_client() + .collective_rpc(&method, body.timeout, body.args, body.kwargs) + .await + .map_err(|error| utility_call_error("collective_rpc", error))?; + + Ok(Json(CollectiveRpcResponse { results })) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/health.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/health.rs new file mode 100644 index 0000000000000000000000000000000000000000..7b28ba5663e27bb4f0a0a32afc68c2c8dc4b83b6 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/health.rs @@ -0,0 +1,14 @@ +use std::sync::Arc; + +use axum::extract::State; +use axum::http::StatusCode; + +use crate::state::AppState; + +pub async fn health(State(state): State>) -> StatusCode { + if state.chat.engine_core_client().is_healthy() { + StatusCode::OK + } else { + StatusCode::SERVICE_UNAVAILABLE + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/http_client_tests.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/http_client_tests.rs new file mode 100644 index 0000000000000000000000000000000000000000..d97a1ea1880a8475721e728c3ac05c56a6303b30 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/http_client_tests.rs @@ -0,0 +1,362 @@ +//! Integration tests that exercise the OpenAI-compatible HTTP API through a +//! real TCP connection using the `async-openai` client library, backed by a +//! mock engine. + +use std::future::Future; +use std::pin::Pin; +use std::sync::Arc; +use std::task::{Context, Poll}; + +use async_openai::Client; +use async_openai::config::OpenAIConfig; +use async_openai::types::chat::{ + ChatCompletionRequestUserMessageArgs, CreateChatCompletionRequestArgs, +}; +use futures::StreamExt as _; +use serial_test::serial; +use vllm_chat::{ + ChatBackend, ChatLlm, ChatRenderer, ChatRequest, ChatTextBackend, DefaultChatOutputProcessor, + DynChatOutputProcessor, DynChatRenderer, NewChatOutputProcessorOptions, RenderedPrompt, +}; +use vllm_engine_core_client::protocol::output::{ + EngineCoreFinishReason, EngineCoreOutput, EngineCoreOutputs, RequestBatchOutputs, +}; +use vllm_engine_core_client::protocol::request::EngineCoreRequest; +use vllm_engine_core_client::test_utils::{IpcNamespace, spawn_mock_engine_task}; +use vllm_engine_core_client::{EngineCoreClient, EngineCoreClientConfig, EngineId}; +use vllm_llm::Llm; +use vllm_text::tokenizer::DynTokenizer; +use vllm_text::{Prompt, TextBackend}; +use vllm_tokenizer::test_utils::TestTokenizer; +use zeromq::prelude::{SocketRecv, SocketSend}; +use zeromq::{DealerSocket, PushSocket, ZmqMessage}; + +use crate::routes::build_router; +use crate::state::AppState; + +// ======================================================================================== +// Test infrastructure (mirrors routes/tests.rs helpers) +// ======================================================================================== + +type TestFuture<'a> = Pin + Send + 'a>>; + +fn boxed_test_future<'a>(future: impl Future + Send + 'a) -> TestFuture<'a> { + Box::pin(future) +} + +struct MockEngineTask { + shutdown_tx: Option>, + join_handle: Option>, +} + +impl MockEngineTask { + fn new( + (shutdown_tx, join_handle): ( + tokio::sync::oneshot::Sender<()>, + tokio::task::JoinHandle<()>, + ), + ) -> Self { + Self { + shutdown_tx: Some(shutdown_tx), + join_handle: Some(join_handle), + } + } +} + +impl Future for MockEngineTask { + type Output = Result<(), tokio::task::JoinError>; + + fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { + if let Some(shutdown_tx) = self.shutdown_tx.take() { + let _ = shutdown_tx.send(()); + } + match self.join_handle.as_mut() { + Some(join_handle) => Pin::new(join_handle).poll(cx), + None => Poll::Ready(Ok(())), + } + } +} + +impl Drop for MockEngineTask { + fn drop(&mut self) { + if let Some(join_handle) = &self.join_handle { + join_handle.abort(); + } + } +} + +fn request_output( + request_id: &str, + new_token_ids: Vec, + finish_reason: Option, +) -> EngineCoreOutput { + EngineCoreOutput { + request_id: request_id.to_string(), + new_token_ids, + new_logprobs: None, + new_prompt_logprobs_tensors: None, + pooling_output: None, + finish_reason, + stop_reason: None, + events: None, + kv_transfer_params: None, + trace_headers: None, + prefill_stats: None, + routed_experts: None, + num_nans_in_logits: 0, + } +} + +fn engine_outputs_for_request( + request_id: &str, + output_specs: Vec<(Vec, Option)>, +) -> EngineCoreOutputs { + RequestBatchOutputs { + outputs: output_specs + .into_iter() + .map(|(token_ids, finish_reason)| request_output(request_id, token_ids, finish_reason)) + .collect(), + ..Default::default() + } + .into() +} + +fn default_stream_output_specs() -> Vec<(Vec, Option)> { + vec![ + (vec![b'h' as u32], None), + (vec![b'i' as u32], None), + (vec![b'!' as u32], Some(EngineCoreFinishReason::Stop)), + ] +} + +async fn send_outputs(push: &mut PushSocket, outputs: EngineCoreOutputs) { + push.send(ZmqMessage::from( + rmp_serde::to_vec_named(&outputs).expect("encode outputs"), + )) + .await + .expect("send outputs"); +} + +async fn recv_engine_message(dealer: &mut DealerSocket) -> Vec { + dealer.recv().await.expect("recv engine message").into_vec() +} + +fn test_llm(client: EngineCoreClient) -> Llm { + Llm::new(client).with_request_id_randomization(false) +} + +#[derive(Clone, Debug)] +struct FakeChatBackend; + +impl TextBackend for FakeChatBackend { + fn tokenizer(&self) -> DynTokenizer { + Arc::new(TestTokenizer::new()) + } + + fn model_id(&self) -> &str { + "test-model" + } +} + +impl ChatBackend for FakeChatBackend { + fn chat_renderer(&self) -> DynChatRenderer { + Arc::new(self.clone()) + } + + fn new_chat_output_processor( + &self, + request: &mut ChatRequest, + options: NewChatOutputProcessorOptions<'_>, + ) -> vllm_chat::Result { + Ok(Box::new(DefaultChatOutputProcessor::new( + request, + self.model_id(), + self.tokenizer(), + options.tool_call_parser, + options.reasoning_parser, + )?)) + } +} + +impl ChatRenderer for FakeChatBackend { + fn render(&self, request: &ChatRequest) -> vllm_chat::Result { + let mut prompt = String::new(); + for message in &request.messages { + prompt.push_str(message.role().as_str()); + prompt.push_str(": "); + prompt.push_str(&message.text_content()?); + prompt.push('\n'); + } + if request.chat_options.add_generation_prompt() { + prompt.push_str("assistant:"); + } + Ok(RenderedPrompt { + prompt: Prompt::Text(prompt), + effective_template_kwargs: Default::default(), + }) + } +} + +/// Spin up an HTTP server on a random port backed by a mock engine. +/// Returns the `async-openai` client, the HTTP server task, and the mock engine +/// task. +async fn http_test_server( + engine_id: impl Into, + output_specs: Vec<(Vec, Option)>, +) -> ( + Client, + tokio::task::JoinHandle<()>, + MockEngineTask, +) { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = engine_id.into(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + move |dealer, push| { + boxed_test_future(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + send_outputs( + push, + engine_outputs_for_request(&request.request_id, output_specs), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + + let chat = ChatLlm::from_shared_backend( + test_llm(client), + Arc::new(FakeChatBackend) as Arc, + ); + let state = Arc::new(AppState::new(vec!["test-model".to_string()], chat)); + let app = build_router(state); + + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("bind http listener"); + let addr = listener.local_addr().expect("local addr"); + + let server_task = tokio::spawn(async move { + axum::serve(listener, app).await.expect("http server"); + }); + + let openai_client = Client::with_config( + OpenAIConfig::new() + .with_api_key("unused") + .with_api_base(format!("http://{addr}/v1")), + ); + + (openai_client, server_task, engine_task) +} + +// ======================================================================================== +// Tests +// ======================================================================================== + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn list_models_via_http_client() { + let (client, server_task, _engine_task) = + http_test_server(b"engine-http-models", default_stream_output_specs()).await; + + let models = client.models().list().await.expect("list models"); + let model_ids: Vec<&str> = models.data.iter().map(|m| m.id.as_str()).collect(); + assert_eq!(model_ids, vec!["test-model"]); + + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_streaming_chat_via_http_client() { + let (client, server_task, engine_task) = + http_test_server(b"engine-http-chat", default_stream_output_specs()).await; + + let request = CreateChatCompletionRequestArgs::default() + .model("test-model") + .stream(false) + .max_completion_tokens(10u32) + .messages([ChatCompletionRequestUserMessageArgs::default() + .content("hello") + .build() + .expect("build user message") + .into()]) + .build() + .expect("build request"); + + let response = client.chat().create(request).await.expect("chat completion"); + + assert_eq!(response.model, "test-model"); + assert_eq!(response.choices.len(), 1); + let choice = &response.choices[0]; + // The stop token `!` is suppressed from text. + assert_eq!(choice.message.content.as_deref(), Some("hi")); + assert_eq!( + choice.finish_reason, + Some(async_openai::types::chat::FinishReason::Stop) + ); + + engine_task.await.expect("mock engine task"); + server_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn streaming_chat_via_http_client() { + let (client, server_task, engine_task) = + http_test_server(b"engine-http-stream", default_stream_output_specs()).await; + + let request = CreateChatCompletionRequestArgs::default() + .model("test-model") + .stream(true) + .max_completion_tokens(10u32) + .messages([ChatCompletionRequestUserMessageArgs::default() + .content("hello") + .build() + .expect("build user message") + .into()]) + .build() + .expect("build request"); + + let mut stream = client.chat().create_stream(request).await.expect("streaming chat completion"); + + let mut full_text = String::new(); + let mut saw_role = false; + let mut saw_finish_reason = false; + + while let Some(chunk) = stream.next().await { + let chunk = chunk.expect("stream chunk"); + for choice in &chunk.choices { + if choice.delta.role.is_some() { + saw_role = true; + } + if let Some(ref delta) = choice.delta.content { + full_text.push_str(delta); + } + if choice.finish_reason.is_some() { + saw_finish_reason = true; + } + } + } + + assert!(saw_role, "expected an assistant role chunk"); + assert!(saw_finish_reason, "expected a terminal finish reason"); + assert_eq!(full_text, "hi"); + + engine_task.await.expect("mock engine task"); + server_task.abort(); +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/generate.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/generate.rs new file mode 100644 index 0000000000000000000000000000000000000000..44ff5be406aa529e4173fa59ab382bda0f514c80 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/generate.rs @@ -0,0 +1,469 @@ +mod convert; +mod types; +mod validate; + +use std::collections::HashMap; +use std::convert::Infallible; +use std::result::Result; +use std::sync::Arc; + +use asynk_strim_attr::{TryYielder, try_stream}; +use axum::Json; +use axum::extract::State; +use axum::http::HeaderMap; +use axum::response::sse::{Event, Sse}; +use axum::response::{IntoResponse, Response}; +use futures::{Stream, StreamExt as _, pin_mut}; +use thiserror_ext::AsReport as _; +use tracing::{error, info, trace}; +use tracing_futures::Instrument as _; +use vllm_engine_core_client::protocol::logprobs::{Logprobs, PositionLogprobs}; +use vllm_llm::{ + CollectedGenerateOutput, FinishReason, GenerateOutput, GenerateOutputStreamExt as _, TokenUsage, +}; + +use self::convert::{ResponseOptions, prepare_generate_request}; +use self::types::{ + GenerateLogprob, GenerateRequest, GenerateResponse, GenerateResponseChoice, + GenerateResponseStreamChoice, GenerateStreamResponse, +}; +use crate::config::ApiServerOptions; +use crate::error::{ApiError, bail_server_error, server_error, text_submit_error}; +use crate::routes::openai::utils::logprobs::clamp_logprob; +use crate::routes::openai::utils::types::{ChatLogProbs, ChatLogProbsContent, TopLogProb, Usage}; +use crate::routes::openai::utils::validated_json::ValidatedJson; +use crate::state::AppState; +use crate::utils::resolve_request_context; + +/// Validate one token-in/token-out request and proxy it into the shared +/// `vllm-text` stack. +pub async fn generate( + State(state): State>, + headers: HeaderMap, + ValidatedJson(body): ValidatedJson, +) -> Response { + let request_context = resolve_request_context(&headers, body.request_id.as_deref()); + let lora_resolution = state.resolve_model_with_loras(body.model.as_deref()).await; + let prepared = match prepare_generate_request(body, &lora_resolution, request_context) { + Ok(prepared) => prepared, + Err(error) => return error.into_response(), + }; + let request_span = tracing::info_span!( + "generate", + request_id = %prepared.request_id, + engine_request_id = tracing::field::Empty, + ); + + let api_server_options = state.api_server_options; + let stream = prepared.stream; + let raw_stream = match state + .chat + .text() + .generate_raw(prepared.text_request) + .instrument(request_span.clone()) + .await + { + Ok(stream) => stream, + Err(error) => { + return text_submit_error("failed to submit raw generate request", error) + .into_response(); + } + }; + + if stream { + let chunk_stream = generate_chunk_stream( + raw_stream, + prepared.request_id, + api_server_options, + prepared.options, + ); + let sse_stream = generate_sse_stream(chunk_stream).instrument(request_span); + + return Sse::new(sse_stream).into_response(); + } + + let collected = match raw_stream.collect_output().instrument(request_span.clone()).await { + Ok(collected) => collected, + Err(error) => { + return server_error!( + "failed to collect raw generate response: {}", + error.to_report_string() + ) + .into_response(); + } + }; + + let response = match collect_generate( + collected, + prepared.request_id, + api_server_options, + prepared.options, + ) { + Ok(response) => response, + Err(error) => return error.into_response(), + }; + + Json(response).into_response() +} + +#[try_stream] +async fn generate_chunk_stream( + stream: impl Stream>, + request_id: String, + ApiServerOptions { + enable_log_requests, + enable_prompt_tokens_details, + .. + }: ApiServerOptions, + ResponseOptions { + include_usage, + include_continuous_usage, + include_logprobs, + // Ignored: raw generate streaming has no prompt-logprobs wire shape. + include_prompt_logprobs: _, + }: ResponseOptions, + mut y: TryYielder, +) -> Result<(), ApiError> { + pin_mut!(stream); + let mut prompt_tokens = None; + let mut usage = TokenUsage::default(); + + while let Some(next) = stream.next().await { + match next { + Ok(output) => { + if prompt_tokens.is_none() { + prompt_tokens = + output.prompt_info.as_ref().map(|info| info.prompt_token_ids.len()); + } + usage.prompt_token_count = prompt_tokens.unwrap_or_default(); + usage.cached_token_count = usage.cached_token_count.max(output.cached_token_count); + + let token_ids = output.token_ids; + usage.output_token_count = usage.output_token_count.saturating_add(token_ids.len()); + let finish_reason = output.finish_reason; + + if matches!(finish_reason.as_ref(), Some(FinishReason::Error)) { + bail_server_error!("Internal server error"); + } + + if let Some(finish_reason) = finish_reason.as_ref() + && enable_log_requests + { + info!( + stream = true, + prompt_tokens = usage.prompt_token_count, + output_tokens = usage.output_token_count, + finish_reason = finish_reason.as_str(), + "generate finished" + ); + } + + if token_ids.is_empty() && finish_reason.is_none() { + continue; + } + + let logprobs = if include_logprobs && !token_ids.is_empty() { + let logprobs = output.logprobs.as_ref().ok_or_else(|| { + server_error!( + "raw generate stream requested logprobs but generation returned none" + ) + })?; + Some(raw_logprobs_to_openai_chat(logprobs)?) + } else { + None + }; + + y.yield_ok(GenerateStreamResponse { + request_id: request_id.clone(), + choices: vec![GenerateResponseStreamChoice { + index: 0, + logprobs, + finish_reason: finish_reason.map(|reason| reason.as_str().to_string()), + token_ids, + }], + usage: include_continuous_usage + .then(|| Usage::from_token_usage(usage, enable_prompt_tokens_details)), + }) + .await; + } + Err(error) => { + error!( + error = %error.as_report(), + "raw generate stream failed" + ); + bail_server_error!("{}", error.to_report_string()); + } + } + } + + if include_usage { + y.yield_ok(GenerateStreamResponse { + request_id, + choices: Vec::new(), + usage: Some(Usage::from_token_usage(usage, enable_prompt_tokens_details)), + }) + .await; + } + + Ok(()) +} + +fn collect_generate( + collected: CollectedGenerateOutput, + request_id: String, + ApiServerOptions { + enable_log_requests, + .. + }: ApiServerOptions, + ResponseOptions { + // Ignored: non-streaming raw generate responses do not include usage. + include_usage: _, + // Ignored: continuous usage is a streaming-only option. + include_continuous_usage: _, + include_logprobs, + include_prompt_logprobs, + }: ResponseOptions, +) -> Result { + let logprobs = if include_logprobs { + let logprobs = collected.logprobs.as_ref().ok_or_else(|| { + ApiError::server_error( + "raw generate response requested logprobs but generation returned none".to_string(), + ) + })?; + Some(raw_logprobs_to_openai_chat(logprobs)?) + } else { + None + }; + let prompt_logprobs = if include_prompt_logprobs { + let prompt_logprobs = collected.prompt_logprobs.as_ref().ok_or_else(|| { + ApiError::server_error( + "raw generate response requested prompt_logprobs but generation returned none" + .to_string(), + ) + })?; + Some(raw_prompt_logprobs_to_maps(prompt_logprobs)) + } else { + None + }; + let finish_reason = collected.finish_reason.as_str().to_string(); + + if enable_log_requests { + info!( + prompt_tokens = collected.prompt_token_ids.len(), + output_tokens = collected.token_ids.len(), + %finish_reason, + "generate finished" + ); + } + + Ok(GenerateResponse { + request_id, + choices: vec![GenerateResponseChoice { + index: 0, + logprobs, + finish_reason: Some(finish_reason), + token_ids: collected.token_ids, + }], + prompt_logprobs, + kv_transfer_params: collected.kv_transfer_params, + }) +} + +fn raw_logprobs_to_openai_chat(logprobs: &Logprobs) -> Result { + let content = logprobs + .positions + .iter() + .map(position_to_chat_logprobs_content) + .collect::, _>>()?; + + Ok(ChatLogProbs { + content: Some(content), + }) +} + +fn raw_prompt_logprobs_to_maps( + prompt_logprobs: &Logprobs, +) -> Vec>> { + std::iter::once(None) + .chain( + prompt_logprobs + .positions + .iter() + .map(|position| Some(position_to_logprob_map(position))), + ) + .collect() +} + +fn position_to_chat_logprobs_content( + position: &PositionLogprobs, +) -> Result { + let chosen = position.entries.first().ok_or_else(|| { + ApiError::server_error( + "raw generate logprobs position unexpectedly had no token candidates".to_string(), + ) + })?; + let token = format_token_id(chosen.token_id); + + Ok(ChatLogProbsContent { + token: token.clone(), + logprob: clamp_logprob(chosen.logprob), + bytes: Some(token.as_bytes().to_vec()), + top_logprobs: position + .entries + .iter() + .map(|entry| { + let token = format_token_id(entry.token_id); + TopLogProb { + token: token.clone(), + logprob: clamp_logprob(entry.logprob), + bytes: Some(token.into_bytes()), + } + }) + .collect(), + }) +} + +fn position_to_logprob_map(position: &PositionLogprobs) -> HashMap { + position + .entries + .iter() + .map(|entry| { + ( + entry.token_id, + GenerateLogprob { + logprob: clamp_logprob(entry.logprob), + rank: Some(entry.rank), + decoded_token: Some(format_token_id(entry.token_id)), + }, + ) + }) + .collect() +} + +fn format_token_id(token_id: u32) -> String { + format!("token_id:{token_id}") +} + +/// Convert one raw-generate chunk stream into SSE events. +#[try_stream] +async fn generate_sse_stream( + stream: impl Stream>, + mut y: TryYielder, +) -> Result<(), Infallible> { + pin_mut!(stream); + + while let Some(next) = stream.next().await { + match next { + Ok(chunk) => y.yield_ok(to_sse_event(&chunk)).await, + Err(error) => { + y.yield_ok(to_error_sse_event(&error)).await; + break; + } + } + } + + y.yield_ok(done_sse_event()).await; + Ok(()) +} + +fn to_sse_event(chunk: &GenerateStreamResponse) -> Event { + let payload = serde_json::to_string(chunk).expect("generate chunk must serialize to JSON"); + trace!(payload, "generate emitting chunk"); + Event::default().data(payload) +} + +fn to_error_sse_event(error: &ApiError) -> Event { + let payload = serde_json::to_string(&error.to_error_response()) + .expect("ErrorResponse must serialize to JSON"); + trace!(payload, "generate emitting error"); + Event::default().data(payload) +} + +fn done_sse_event() -> Event { + trace!("generate emitting done"); + Event::default().data("[DONE]") +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use futures::{TryStreamExt as _, stream}; + use vllm_llm::GeneratePromptInfo; + + use super::*; + + #[tokio::test] + async fn generate_chunk_stream_captures_late_prompt_info() { + let stream = stream::iter(vec![ + Ok(GenerateOutput { + request_id: String::new(), + prompt_info: None, + token_ids: Vec::new(), + logprobs: None, + finish_reason: None, + cached_token_count: 0, + kv_transfer_params: None, + }), + Ok(GenerateOutput { + request_id: String::new(), + prompt_info: Some(GeneratePromptInfo { + prompt_token_ids: Arc::from([11_u32, 22_u32]), + prompt_logprobs: None, + }), + token_ids: vec![33], + logprobs: None, + finish_reason: Some(FinishReason::stop_eos()), + cached_token_count: 2, + kv_transfer_params: None, + }), + ]); + + let chunks: Vec<_> = generate_chunk_stream( + stream, + "raw-stream".to_string(), + ApiServerOptions { + enable_prompt_tokens_details: true, + ..Default::default() + }, + ResponseOptions { + include_usage: true, + include_continuous_usage: true, + ..Default::default() + }, + ) + .try_collect() + .await + .expect("collect chunks"); + + assert_eq!(chunks.len(), 2); + assert_eq!( + chunks[0].usage.as_ref().expect("chunk usage").prompt_tokens, + 2 + ); + assert_eq!( + chunks[0] + .usage + .as_ref() + .expect("chunk usage") + .prompt_tokens_details + .as_ref() + .map(|details| details.cached_tokens), + Some(2) + ); + assert_eq!( + chunks[1].usage.as_ref().expect("final usage").prompt_tokens, + 2 + ); + assert_eq!( + chunks[1] + .usage + .as_ref() + .expect("final usage") + .prompt_tokens_details + .as_ref() + .map(|details| details.cached_tokens), + Some(2) + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/generate/convert.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/generate/convert.rs new file mode 100644 index 0000000000000000000000000000000000000000..b172d5240628eabef1841dd0a03f0c12c46fccdf --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/generate/convert.rs @@ -0,0 +1,204 @@ +use vllm_text::{Prompt, TextDecodeOptions, TextRequest}; + +use super::types::GenerateRequest; +use super::validate; +use crate::error::ApiError; +use crate::lora::LoraModelResolution; +use crate::utils::{ResolvedRequestContext, merge_kv_transfer_params}; + +/// Lowered generate request plus the response request ID. +#[derive(Debug, Clone, PartialEq)] +pub(super) struct PreparedRequest { + pub request_id: String, + pub text_request: TextRequest, + pub stream: bool, + /// Public response rendering options for route-layer helpers. + pub options: ResponseOptions, +} + +#[derive(Debug, Clone, Copy, Default, PartialEq)] +pub(super) struct ResponseOptions { + /// Whether the caller asked for the final streamed usage chunk. + pub include_usage: bool, + /// Whether the caller asked for usage on every streamed chunk. + pub include_continuous_usage: bool, + /// Whether the caller requested output logprobs on generate choices. + pub include_logprobs: bool, + /// Whether the caller requested top-level prompt logprobs. + pub include_prompt_logprobs: bool, +} + +/// Validate and lower one raw generate request into the internal +/// text-generation format. +pub(super) fn prepare_generate_request( + request: GenerateRequest, + lora_resolution: &LoraModelResolution, + ctx: ResolvedRequestContext, +) -> Result { + validate::validate_request_compat(&request, &lora_resolution.model_names)?; + + let stream = request.stream; + let include_usage = request + .stream_options + .as_ref() + .and_then(|options| options.include_usage) + .unwrap_or(false); + let include_continuous_usage = include_usage + && request + .stream_options + .as_ref() + .and_then(|options| options.continuous_usage_stats) + .unwrap_or(false); + let include_logprobs = request.sampling_params.logprobs.is_some(); + let include_prompt_logprobs = request.sampling_params.prompt_logprobs.is_some(); + let mut sampling_params = request.sampling_params; + sampling_params.vllm_xargs = merge_kv_transfer_params( + sampling_params.vllm_xargs, + request.kv_transfer_params.as_ref(), + ); + + let text_request = TextRequest { + request_id: ctx.request_id.clone(), + prompt: Prompt::TokenIds(request.token_ids), + mm_features: None, + sampling_params, + decode_options: TextDecodeOptions::default(), + intermediate: false, + priority: request.priority, + cache_salt: request.cache_salt, + add_special_tokens: false, + data_parallel_rank: ctx.data_parallel_rank, + reasoning_parser_kwargs: None, + lora_request: lora_resolution.lora_request.clone(), + }; + + Ok(PreparedRequest { + request_id: ctx.request_id, + text_request, + stream, + options: ResponseOptions { + include_usage, + include_continuous_usage, + include_logprobs, + include_prompt_logprobs, + }, + }) +} + +#[cfg(test)] +mod tests { + use serde_json::json; + use vllm_text::Prompt; + + use super::prepare_generate_request; + use crate::lora::LoraModelResolution; + use crate::routes::inference::generate::types::GenerateRequest; + use crate::utils::ResolvedRequestContext; + + fn served(names: &[&str]) -> LoraModelResolution { + LoraModelResolution { + model_names: names.iter().map(|s| s.to_string()).collect(), + lora_request: None, + } + } + + #[test] + fn prepare_generate_request_maps_token_prompt_and_sampling_params() { + let request: GenerateRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "token_ids": [11, 22, 33], + "priority": -3, + "cache_salt": "salt", + "sampling_params": { + "max_tokens": 7, + "logprobs": 2, + "prompt_logprobs": 1, + "ignore_eos": true + }, + "kv_transfer_params": { + "connector": "x" + } + })) + .expect("parse request"); + + let prepared = prepare_generate_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("prepare"); + + assert_eq!( + prepared.text_request.prompt, + Prompt::TokenIds(vec![11, 22, 33]) + ); + assert_eq!(prepared.text_request.sampling_params.max_tokens, Some(7)); + assert_eq!(prepared.text_request.sampling_params.logprobs, Some(2)); + assert_eq!( + prepared.text_request.sampling_params.prompt_logprobs, + Some(1) + ); + assert!(prepared.text_request.sampling_params.ignore_eos); + assert_eq!(prepared.text_request.priority, -3); + assert_eq!(prepared.text_request.cache_salt.as_deref(), Some("salt")); + assert_eq!( + prepared + .text_request + .sampling_params + .vllm_xargs + .and_then(|mut xargs| xargs.remove("kv_transfer_params")), + Some(json!({"connector": "x"})) + ); + } + + #[test] + fn prepare_generate_request_forwards_thinking_token_budget() { + let request: GenerateRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "token_ids": [11, 22, 33], + "sampling_params": { + "thinking_token_budget": 64 + } + })) + .expect("parse request"); + + let prepared = prepare_generate_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("prepare"); + + // The raw inference route shares `vllm_text::SamplingParams`, so the + // field is carried through to lowering exactly like the OpenAI routes + // (normalization/validation then happens in `lower_sampling_params`). + assert_eq!( + prepared.text_request.sampling_params.thinking_token_budget, + Some(64) + ); + } + + #[test] + fn prepare_generate_request_gates_continuous_usage_on_include_usage() { + let request: GenerateRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "token_ids": [11, 22], + "stream": true, + "stream_options": { + "continuous_usage_stats": true + }, + "sampling_params": {} + })) + .expect("parse request"); + + let prepared = prepare_generate_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("prepare"); + + assert!(!prepared.options.include_usage); + assert!(!prepared.options.include_continuous_usage); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/generate/types.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/generate/types.rs new file mode 100644 index 0000000000000000000000000000000000000000..e90746a75abe40089017828afda7a42f93b0a3f0 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/generate/types.rs @@ -0,0 +1,77 @@ +use std::collections::HashMap; + +use serde::{Deserialize, Serialize}; +use serde_json::{Map, Value}; +use validator::Validate; +use vllm_text::SamplingParams; + +use crate::routes::openai::utils::types::{ChatLogProbs, Normalizable, StreamOptions, Usage}; + +/// vLLM-compatible request type for the token-in/token-out generate API. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Deserialize, Serialize, Validate)] +pub struct GenerateRequest { + pub request_id: Option, + pub model: Option, + pub token_ids: Vec, + pub sampling_params: SamplingParams, + #[serde(default)] + pub stream: bool, + pub stream_options: Option, + pub cache_salt: Option, + #[serde(default)] + pub priority: i32, + pub kv_transfer_params: Option>, + #[serde(flatten)] + pub other: Map, +} + +impl Normalizable for GenerateRequest {} + +/// Mirrors the Python vLLM `GenerateResponseChoice` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub(super) struct GenerateResponseChoice { + pub index: u32, + pub logprobs: Option, + pub finish_reason: Option, + pub token_ids: Vec, +} + +/// Mirrors the Python vLLM `GenerateResponseStreamChoice` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub(super) struct GenerateResponseStreamChoice { + pub index: u32, + pub logprobs: Option, + pub finish_reason: Option, + pub token_ids: Vec, +} + +/// Mirrors the Python vLLM `GenerateStreamResponse` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub(super) struct GenerateStreamResponse { + pub request_id: String, + pub choices: Vec, + pub usage: Option, +} + +/// Mirrors the Python vLLM `GenerateResponse` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub(super) struct GenerateResponse { + pub request_id: String, + pub choices: Vec, + pub prompt_logprobs: Option>>>, + pub kv_transfer_params: Option, +} + +/// Mirrors the Python vLLM `Logprob` class used in prompt-logprobs payloads. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub(super) struct GenerateLogprob { + pub logprob: f32, + pub rank: Option, + pub decoded_token: Option, +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/generate/validate.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/generate/validate.rs new file mode 100644 index 0000000000000000000000000000000000000000..83d38a6d9c172a4442e9bb17c8e9a5dab3949b94 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/generate/validate.rs @@ -0,0 +1,156 @@ +use super::types::GenerateRequest; +use crate::error::{ApiError, bail_invalid_request}; + +/// Enforce the minimal compatibility contract for the Rust token generate +/// route. +pub(super) fn validate_request_compat( + request: &GenerateRequest, + served_model_names: &[String], +) -> Result<(), ApiError> { + if let Some(model) = request.model.as_ref() + && !served_model_names.iter().any(|n| n == model) + { + return Err(ApiError::model_not_found(model.clone())); + } + + if request.stream_options.is_some() && !request.stream { + bail_invalid_request!( + param = "stream_options", + "stream_options are only supported when stream=true." + ); + } + + if request.token_ids.is_empty() { + bail_invalid_request!( + param = "token_ids", + "token_ids must contain at least one token ID." + ); + } + + if request.sampling_params.max_tokens == Some(0) { + bail_invalid_request!( + param = "sampling_params", + "max_tokens must be greater than 0." + ); + } + + if let Some(prompt_logprobs) = request.sampling_params.prompt_logprobs { + if prompt_logprobs < 0 && prompt_logprobs != -1 { + bail_invalid_request!( + param = "sampling_params", + "`prompt_logprobs` must be a non-negative value or -1." + ); + } + + if request.stream { + bail_invalid_request!( + param = "sampling_params", + "`prompt_logprobs` are not available when `stream=true`." + ); + } + } + + Ok(()) +} + +#[cfg(test)] +mod tests { + use serde_json::json; + + use super::validate_request_compat; + use crate::routes::inference::generate::types::GenerateRequest; + + fn base_request() -> GenerateRequest { + serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "token_ids": [11, 22], + "sampling_params": {} + })) + .expect("parse request") + } + + fn served(names: &[&str]) -> Vec { + names.iter().map(|s| s.to_string()).collect() + } + + #[test] + fn validate_request_compat_accepts_streaming() { + let request = GenerateRequest { + stream: true, + ..base_request() + }; + assert!(validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])).is_ok()); + } + + #[test] + fn validate_request_compat_rejects_stream_options_without_streaming() { + let request: GenerateRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "token_ids": [11, 22], + "stream": false, + "stream_options": {"include_usage": true}, + "sampling_params": {} + })) + .expect("parse request"); + assert!(validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])).is_err()); + } + + #[test] + fn validate_request_compat_rejects_empty_token_ids() { + let request = GenerateRequest { + token_ids: Vec::new(), + ..base_request() + }; + assert!(validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])).is_err()); + } + + #[test] + fn validate_request_compat_rejects_streaming_prompt_logprobs() { + let request: GenerateRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "token_ids": [11, 22], + "stream": true, + "sampling_params": { + "prompt_logprobs": 0 + } + })) + .expect("parse request"); + assert!(validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])).is_err()); + + let request: GenerateRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "token_ids": [11, 22], + "stream": true, + "sampling_params": { + "prompt_logprobs": 1 + } + })) + .expect("parse request"); + assert!(validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])).is_err()); + + let request: GenerateRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "token_ids": [11, 22], + "stream": true, + "sampling_params": { + "prompt_logprobs": -1 + } + })) + .expect("parse request"); + assert!(validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])).is_err()); + } + + #[test] + fn validate_request_compat_accepts_non_stream_prompt_logprobs() { + let request: GenerateRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "token_ids": [11, 22], + "stream": false, + "sampling_params": { + "prompt_logprobs": 1 + } + })) + .expect("parse request"); + assert!(validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])).is_ok()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/mod.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..37db4173d93f66c018c6f1574b0fcad5ee265f2d --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/inference/mod.rs @@ -0,0 +1,3 @@ +pub mod generate; + +pub use generate::generate; diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/load.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/load.rs new file mode 100644 index 0000000000000000000000000000000000000000..5677fbab90a70639061f5ac08196f1ff05412e5c --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/load.rs @@ -0,0 +1,18 @@ +use std::sync::Arc; + +use axum::Json; +use axum::extract::State; +use serde::Serialize; + +use crate::state::AppState; + +#[derive(Serialize)] +pub(crate) struct ServerLoadResponse { + server_load: u64, +} + +pub async fn load(State(state): State>) -> Json { + Json(ServerLoadResponse { + server_load: state.server_load(), + }) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/lora.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/lora.rs new file mode 100644 index 0000000000000000000000000000000000000000..b1f620767f9034a0d92f015f6fdfc5270bca818d --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/lora.rs @@ -0,0 +1,296 @@ +use std::path::{Component, Path, PathBuf}; +use std::sync::Arc; + +use axum::extract::State; +use serde::Deserialize; +use thiserror_ext::AsReport; +use validator::Validate; + +use crate::error::ApiError; +use crate::lora::{LoadLoraError, UnloadLoraError}; +use crate::routes::openai::utils::types::Normalizable; +use crate::routes::openai::utils::validated_json::ValidatedJson; +use crate::state::AppState; + +const RUNTIME_LORA_ALLOWED_PATH_PREFIXES_ENV: &str = "VLLM_RUNTIME_LORA_ALLOWED_PATH_PREFIXES"; + +#[derive(Debug, Deserialize, Validate)] +pub(crate) struct LoadLoraAdapterRequest { + lora_name: String, + lora_path: String, + #[serde(default)] + load_inplace: bool, + #[serde(default)] + is_3d_lora_weight: bool, +} + +impl Normalizable for LoadLoraAdapterRequest {} + +#[derive(Debug, Deserialize, Validate)] +pub(crate) struct UnloadLoraAdapterRequest { + lora_name: String, + #[serde(default)] + lora_int_id: Option, +} + +impl Normalizable for UnloadLoraAdapterRequest {} + +fn runtime_lora_allowed_path_prefixes() -> Option> { + let prefixes = std::env::var_os(RUNTIME_LORA_ALLOWED_PATH_PREFIXES_ENV)?; + let prefixes: Vec<_> = std::env::split_paths(&prefixes) + .filter(|path| !path.as_os_str().is_empty()) + .collect(); + (!prefixes.is_empty()).then_some(prefixes) +} + +fn looks_like_local_lora_path(lora_path: &str) -> bool { + let path = Path::new(lora_path); + path.is_absolute() + || lora_path.starts_with('~') + || lora_path.starts_with('.') + || path.components().any(|component| matches!(component, Component::ParentDir)) +} + +fn validate_lora_path_access( + lora_path: &str, + allowed_prefixes: Option<&[PathBuf]>, +) -> Result, ApiError> { + let path = Path::new(lora_path); + if !looks_like_local_lora_path(lora_path) && !path.exists() { + return Ok(None); + } + + let Some(allowed_prefixes) = allowed_prefixes else { + return Err(ApiError::invalid_request( + format!( + "Local LoRA adapter paths require {RUNTIME_LORA_ALLOWED_PATH_PREFIXES_ENV} to be configured." + ), + Some("lora_path"), + )); + }; + + if !path.is_absolute() { + return Err(ApiError::invalid_request( + format!( + "Local LoRA adapter paths must be absolute and under one of the prefixes configured by {RUNTIME_LORA_ALLOWED_PATH_PREFIXES_ENV}." + ), + Some("lora_path"), + )); + } + + let canonical_path = path.canonicalize().map_err(|_| { + ApiError::invalid_request( + "Local LoRA adapter path must exist and be accessible.".to_string(), + Some("lora_path"), + ) + })?; + let canonical_prefixes = allowed_prefixes + .iter() + .map(|prefix| { + prefix.canonicalize().map_err(|_| { + ApiError::server_error(format!( + "configured {RUNTIME_LORA_ALLOWED_PATH_PREFIXES_ENV} path prefix must exist and be accessible" + )) + }) + }) + .collect::, _>>()?; + + if !canonical_prefixes.iter().any(|prefix| canonical_path.starts_with(prefix)) { + return Err(ApiError::invalid_request( + "Local LoRA adapter path is outside the configured allowed prefixes.".to_string(), + Some("lora_path"), + )); + } + + Ok(Some(canonical_path.to_string_lossy().into_owned())) +} + +/// Dynamically load one LoRA adapter and expose it as an OpenAI model id. +pub async fn load_lora_adapter( + State(state): State>, + ValidatedJson(request): ValidatedJson, +) -> Result { + if request.lora_name.is_empty() || request.lora_path.is_empty() { + return Err(ApiError::invalid_request( + "Both 'lora_name' and 'lora_path' must be provided.".to_string(), + None, + )); + } + let allowed_prefixes = runtime_lora_allowed_path_prefixes(); + let lora_path = validate_lora_path_access(&request.lora_path, allowed_prefixes.as_deref())? + .unwrap_or(request.lora_path); + + let lora_name = request.lora_name; + state + .load_lora( + lora_name.clone(), + lora_path, + request.load_inplace, + request.is_3d_lora_weight, + ) + .await + .map_err(|error| match error { + LoadLoraError::AlreadyLoaded { lora_name } => ApiError::invalid_request( + format!( + "The lora adapter '{lora_name}' has already been loaded. If you want to load the adapter in place, set 'load_inplace' to true." + ), + Some("lora_name"), + ), + LoadLoraError::BaseModelName { lora_name } => ApiError::invalid_request( + format!("The lora adapter name '{lora_name}' conflicts with a served base model."), + Some("lora_name"), + ), + LoadLoraError::Engine(error) => ApiError::server_error(format!( + "failed to load LoRA adapter '{lora_name}': {}", + error.to_report_string() + )), + LoadLoraError::NotLoaded { lora_name } => ApiError::server_error(format!( + "failed to load LoRA adapter '{lora_name}': engine rejected the adapter" + )), + })?; + + Ok(format!( + "Success: LoRA adapter '{lora_name}' added successfully." + )) +} + +/// Remove one LoRA adapter from the engine and frontend registry. +pub async fn unload_lora_adapter( + State(state): State>, + ValidatedJson(request): ValidatedJson, +) -> Result { + if request.lora_name.is_empty() { + return Err(ApiError::invalid_request( + "'lora_name' needs to be provided to unload a LoRA adapter.".to_string(), + Some("lora_name"), + )); + } + + let lora_request = state + .unload_lora(&request.lora_name, request.lora_int_id) + .await + .map_err(|error| match error { + UnloadLoraError::NotFound { lora_name } => ApiError::model_not_found(lora_name), + UnloadLoraError::IntIdMismatch { + lora_name, + expected, + actual, + } => ApiError::invalid_request( + format!( + "The requested lora_int_id {actual} does not match loaded adapter '{lora_name}' with id {expected}." + ), + Some("lora_int_id"), + ), + UnloadLoraError::Engine(error) => ApiError::server_error(format!( + "failed to unload LoRA adapter '{}': {}", + request.lora_name, + error.to_report_string() + )), + UnloadLoraError::NotRemoved { + lora_name, + lora_int_id, + } => ApiError::server_error(format!( + "failed to unload LoRA adapter '{lora_name}' with id {lora_int_id}" + )), + })?; + + Ok(format!( + "Success: LoRA adapter '{}' removed successfully.", + lora_request.lora_name + )) +} + +#[cfg(test)] +mod tests { + use std::fs; + use std::path::PathBuf; + use std::time::{SystemTime, UNIX_EPOCH}; + + use super::validate_lora_path_access; + + fn temp_lora_dir(test_name: &str) -> PathBuf { + let suffix = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("clock should be after unix epoch") + .as_nanos(); + let path = std::env::temp_dir().join(format!( + "vllm-lora-{test_name}-{}-{suffix}", + std::process::id() + )); + fs::create_dir_all(&path).expect("create temp lora dir"); + path + } + + #[test] + fn lora_path_allows_hf_repo_ids_without_prefixes() { + assert_eq!( + validate_lora_path_access("org/adapter-a", None).expect("hf repo id should be allowed"), + None + ); + } + + #[test] + fn lora_path_rejects_local_paths_without_prefixes() { + assert!(validate_lora_path_access("/tmp/adapter-a", None).is_err()); + assert!(validate_lora_path_access("./adapter-a", None).is_err()); + assert!(validate_lora_path_access("~/adapter-a", None).is_err()); + assert!(validate_lora_path_access("subdir/../../../etc/sensitive", None).is_err()); + } + + #[test] + fn lora_path_rejects_existing_bare_relative_paths_without_prefixes() { + let root = + PathBuf::from("target").join(format!("vllm-lora-relative-{}", std::process::id())); + let adapter = root.join("adapter-a"); + fs::create_dir_all(&adapter).expect("create relative adapter dir"); + + assert!( + validate_lora_path_access(adapter.to_str().expect("utf-8 temp path"), None).is_err() + ); + + fs::remove_dir_all(root).ok(); + } + + #[test] + fn lora_path_allows_absolute_paths_under_configured_prefixes() { + let root = temp_lora_dir("allowed-prefix"); + let allowed = root.join("allowed"); + let adapter = allowed.join("adapter-a"); + fs::create_dir_all(&adapter).expect("create adapter dir"); + + let prefixes = [allowed]; + let resolved = + validate_lora_path_access(adapter.to_str().expect("utf-8 temp path"), Some(&prefixes)) + .expect("path under configured prefix should be allowed"); + assert_eq!( + resolved.as_deref(), + Some( + adapter + .canonicalize() + .expect("canonical adapter") + .to_str() + .expect("utf-8 temp path") + ) + ); + + fs::remove_dir_all(root).ok(); + } + + #[test] + fn lora_path_rejects_parent_escape_from_configured_prefixes() { + let root = temp_lora_dir("parent-escape"); + let allowed = root.join("allowed"); + let private_adapter = root.join("private").join("adapter-a"); + fs::create_dir_all(&allowed).expect("create allowed dir"); + fs::create_dir_all(&private_adapter).expect("create private adapter dir"); + + let escaped = allowed.join("../private/adapter-a"); + let prefixes = [allowed]; + assert!( + validate_lora_path_access(escaped.to_str().expect("utf-8 temp path"), Some(&prefixes)) + .is_err() + ); + + fs::remove_dir_all(root).ok(); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/metrics.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/metrics.rs new file mode 100644 index 0000000000000000000000000000000000000000..e0fa1924432aa4b7ea867a0d510436a4eff6693f --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/metrics.rs @@ -0,0 +1,26 @@ +use axum::http::header::CONTENT_TYPE; +use axum::http::{HeaderValue, StatusCode}; +use axum::response::{IntoResponse, Response}; +use thiserror_ext::AsReport; +use vllm_metrics::METRICS; + +const OPENMETRICS_CONTENT_TYPE: &str = "application/openmetrics-text; version=1.0.0; charset=utf-8"; + +pub async fn scrape() -> Response { + match METRICS.render() { + Ok(body) => ( + [( + CONTENT_TYPE, + HeaderValue::from_static(OPENMETRICS_CONTENT_TYPE), + )], + body, + ) + .into_response(), + + Err(error) => ( + StatusCode::INTERNAL_SERVER_ERROR, + format!("failed to render metrics: {}", error.as_report()), + ) + .into_response(), + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/chat_completions.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/chat_completions.rs new file mode 100644 index 0000000000000000000000000000000000000000..6d693373077c05a3ead59cca02ab4ae0ad96b5d8 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/chat_completions.rs @@ -0,0 +1,1421 @@ +pub(crate) mod convert; +mod types; +mod validate; + +use std::convert::Infallible; +use std::result::Result; +use std::sync::Arc; + +use asynk_strim_attr::{TryYielder, try_stream}; +use axum::Json; +use axum::extract::State; +use axum::http::HeaderMap; +use axum::response::sse::{Event, Sse}; +use axum::response::{IntoResponse, Response}; +use futures::{Stream, StreamExt as _, pin_mut}; +use serde_json::Value; +use thiserror_ext::AsReport as _; +use tracing::{debug, error, info, trace}; +use tracing_futures::Instrument as _; +use vllm_chat::{ + AssistantBlockKind, AssistantMessageExt as _, ChatEvent, ChatEventStream, ChatEventStreamTrait, + CollectedAssistantMessage, FinishReason, +}; +use vllm_engine_core_client::protocol::output::StopReason; + +use self::convert::{ResponseOptions, prepare_chat_request}; +use crate::config::ApiServerOptions; +use crate::error::{ApiError, bail_server_error, chat_submit_error, server_error}; +use crate::routes::openai::chat_completions::types::{ + AssistantRole, ChatCompletionChoice, ChatCompletionMessage, ChatCompletionRequest, + ChatCompletionResponse, ChatCompletionStreamChoice, ChatCompletionStreamResponse, + ChatMessageDelta, +}; +use crate::routes::openai::utils::logprobs::{ + decoded_logprobs_to_openai_chat, decoded_prompt_logprobs_to_maps, +}; +use crate::routes::openai::utils::types::{ + ChatLogProbs, FunctionCallDelta, FunctionCallResponse, ToolCall, ToolCallDelta, Usage, +}; +use crate::routes::openai::utils::usage::ContinuousUsage; +use crate::routes::openai::utils::validated_json::ValidatedJson; +use crate::state::AppState; +use crate::utils::{resolve_request_context, unix_timestamp}; + +/// Validate one chat completion request and proxy it into the shared +/// `vllm-chat` stack. +pub async fn chat_completions( + State(state): State>, + headers: HeaderMap, + ValidatedJson(body): ValidatedJson, +) -> Response { + let stream = body.stream; + let request_context = resolve_request_context(&headers, body.request_id.as_deref()); + let lora_resolution = state.resolve_model_with_loras(Some(&body.model)).await; + + let prepared = match prepare_chat_request(body, &lora_resolution, request_context) { + Ok(prepared) => prepared, + Err(error) => return error.into_response(), + }; + let request_span = tracing::info_span!( + "chat_completions", + request_id = %prepared.request_id, + engine_request_id = tracing::field::Empty, + ); + + let created = unix_timestamp(); + let api_server_options = state.api_server_options; + + let chat_stream = + match state.chat.chat(prepared.chat_request).instrument(request_span.clone()).await { + Ok(stream) => stream, + Err(error) => { + return chat_submit_error("failed to submit chat request", error).into_response(); + } + }; + + if stream { + let chunk_stream = chat_completion_chunk_stream( + chat_stream, + prepared.request_id, + prepared.response_model, + created, + api_server_options, + prepared.options, + ); + let sse_stream = chat_completion_sse_stream(chunk_stream).instrument(request_span); + + Sse::new(sse_stream).into_response() + } else { + let response = match collect_chat_completion( + chat_stream, + prepared.request_id, + prepared.response_model, + created, + api_server_options, + prepared.options, + ) + .instrument(request_span.clone()) + .await + { + Ok(response) => response, + Err(error) => return error.into_response(), + }; + + Json(response).into_response() + } +} + +async fn collect_chat_completion( + stream: ChatEventStream, + request_id: String, + response_model: String, + created: u64, + ApiServerOptions { + enable_log_requests, + enable_prompt_tokens_details, + .. + }: ApiServerOptions, + ResponseOptions { + // Ignored: non-streaming responses always include usage. + include_usage: _, + // Ignored: non-streaming responses are collected before usage is attached. + include_continuous_usage: _, + requested_logprobs, + include_prompt_logprobs, + include_reasoning, + echo, + return_token_ids, + return_tokens_as_token_ids, + }: ResponseOptions, +) -> Result { + let collected = stream.collect_message().await.map_err(|error| { + server_error!( + "failed to collect chat completion response: {}", + error.to_report_string() + ) + })?; + let CollectedAssistantMessage { + message, + prompt_token_ids, + prompt_logprobs, + logprobs, + token_ids, + usage, + finish_reason, + kv_transfer_params, + } = collected; + let stop_reason = finish_reason.as_stop_reason().map(stop_reason_to_json); + let saw_tool_calls = message.tool_calls().next().is_some(); + let reasoning = message.reasoning(); + // Output logprobs and token IDs cover the complete generated token stream. + // When reasoning is hidden, omit them rather than leaking hidden reasoning + // tokens through per-token metadata. + let include_output_metadata = include_reasoning || reasoning.is_none(); + let finish_reason = chat_finish_reason_to_openai(&finish_reason, saw_tool_calls)?.to_string(); + let tool_calls = message + .tool_calls() + .map(|call| ToolCall { + id: call.id.clone(), + tool_type: "function".to_string(), + function: FunctionCallResponse { + name: call.name.clone(), + arguments: Some(call.arguments.clone()), + }, + }) + .collect::>(); + let logprobs = if requested_logprobs && include_output_metadata { + Some(decoded_logprobs_to_openai_chat( + logprobs.as_ref().ok_or_else(|| { + server_error!("chat response requested logprobs but generation returned none") + })?, + return_tokens_as_token_ids, + )?) + } else { + None + }; + let prompt_logprobs = if include_prompt_logprobs { + Some(decoded_prompt_logprobs_to_maps( + prompt_logprobs.as_ref().ok_or_else(|| { + server_error!( + "chat response requested prompt_logprobs but generation returned none" + ) + })?, + return_tokens_as_token_ids, + )) + } else { + None + }; + let usage = Usage::from_token_usage(usage, enable_prompt_tokens_details); + + if enable_log_requests { + info!( + model = %response_model, + prompt_tokens = usage.prompt_tokens, + output_tokens = usage.completion_tokens.unwrap_or(0), + finish_reason = %finish_reason, + "chat completion finished" + ); + } + + Ok(ChatCompletionResponse { + id: request_id, + object: "chat.completion".to_string(), + created, + model: response_model, + choices: vec![ChatCompletionChoice { + index: 0, + message: ChatCompletionMessage { + role: AssistantRole, + content: match &echo { + Some(prefix) => Some(format!("{prefix}{}", message.text())), + None => Some(message.text()).filter(|t| !t.is_empty()), + }, + tool_calls: Some(tool_calls).filter(|calls| !calls.is_empty()), + reasoning: if include_reasoning { reasoning } else { None }, + }, + logprobs, + finish_reason: Some(finish_reason), + stop_reason, + token_ids: (return_token_ids && include_output_metadata).then_some(token_ids), + }], + usage: Some(usage), + system_fingerprint: None, + prompt_logprobs, + prompt_token_ids: return_token_ids.then(|| prompt_token_ids.to_vec()), + kv_transfer_params, + }) +} + +/// Convert one internal chat event stream into OpenAI chat-completion chunks. +#[try_stream] +async fn chat_completion_chunk_stream( + mut stream: impl ChatEventStreamTrait + Unpin, + request_id: String, + response_model: String, + created: u64, + ApiServerOptions { + enable_log_requests, + enable_prompt_tokens_details, + .. + }: ApiServerOptions, + ResponseOptions { + include_usage, + include_continuous_usage, + requested_logprobs, + // Ignored: chat streaming prompt logprobs are rejected for Python parity. + include_prompt_logprobs: _, + include_reasoning, + echo, + return_token_ids, + return_tokens_as_token_ids, + }: ResponseOptions, + mut y: TryYielder, +) -> Result<(), ApiError> { + let mut saw_tool_calls = false; + // `LogprobsDelta` is emitted after all chat events for one decoded update. + // If that update contains hidden reasoning, including delimiter-only block + // starts or ends, omit its token metadata as well as its visible delta. + let mut inside_hidden_reasoning = false; + let mut suppress_current_update_metadata = false; + let mut continuous_usage = ContinuousUsage::default(); + + /// Yield a chunk with optional continuous usage attached. + macro_rules! yield_chunk { + ($chunk:expr) => {{ + let mut chunk = $chunk; + if include_continuous_usage { + chunk.usage = Some(continuous_usage.to_usage()); + } + y.yield_ok(chunk).await; + }}; + } + + // If the client requested logprobs or token_ids, we need to buffer chunks until + // we receive the separate `LogprobsDelta` event, so that we can emit one + // combined chunk with both the semantic delta and its per-update metadata. + // Continuous usage also buffers so the token count from `LogprobsDelta` can + // be attached to the matching semantic chunk. + let mut pending_chunk = (requested_logprobs || return_token_ids || include_continuous_usage) + .then(PendingChatChunk::default); + + while let Some(next) = stream.next().await { + match next { + Ok(ChatEvent::Start { + prompt_token_ids, .. + }) => { + continuous_usage.set_prompt_tokens(prompt_token_ids.len()); + let mut chunk = start_chunk(&request_id, &response_model, created); + if return_token_ids { + chunk.prompt_token_ids = Some(prompt_token_ids.to_vec()); + } + yield_chunk!(chunk); + // When echo=true, emit the last assistant message content as a delta chunk. + if let Some(echo_text) = &echo { + yield_chunk!(block_delta_chunk( + &request_id, + &response_model, + created, + AssistantBlockKind::Text, + echo_text.clone(), + )); + } + } + Ok(ChatEvent::BlockDelta { kind, delta, .. }) => { + let include_delta = + include_reasoning || !matches!(kind, AssistantBlockKind::Reasoning); + if include_delta { + if let Some(pending_chunk) = pending_chunk.as_mut() { + pending_chunk.push_block_delta(kind, delta); + } else { + yield_chunk!(block_delta_chunk( + &request_id, + &response_model, + created, + kind, + delta, + )); + } + } else { + suppress_current_update_metadata = true; + } + } + Ok(ChatEvent::LogprobsDelta { + logprobs, + token_ids, + }) => { + let delta_token_count = token_ids.len(); + continuous_usage.add_output_tokens(delta_token_count); + let include_metadata = + !suppress_current_update_metadata && !inside_hidden_reasoning; + suppress_current_update_metadata = false; + let openai_logprobs = if include_metadata { + logprobs + .as_ref() + .map(|lp| decoded_logprobs_to_openai_chat(lp, return_tokens_as_token_ids)) + .transpose()? + } else { + None + }; + let openai_token_ids = include_metadata + .then_some(token_ids) + .and_then(|token_ids| return_token_ids.then_some(token_ids)) + .filter(|t| !t.is_empty()); + if let Some(pending_chunk) = pending_chunk.as_mut() { + pending_chunk.logprobs = openai_logprobs; + pending_chunk.token_ids = openai_token_ids; + if let Some(chunk) = + pending_chunk.take_chunk(&request_id, &response_model, created) + { + yield_chunk!(chunk); + } + } else if let Some(logprobs) = openai_logprobs { + yield_chunk!(logprobs_only_chunk( + &request_id, + &response_model, + created, + logprobs, + )); + } + } + Ok(ChatEvent::BlockStart { kind, .. }) => { + debug!(?kind, "starting new block"); + if !include_reasoning && matches!(kind, AssistantBlockKind::Reasoning) { + inside_hidden_reasoning = true; + suppress_current_update_metadata = true; + } + } + Ok(ChatEvent::BlockEnd { .. }) => { + debug!("ending current block"); + if inside_hidden_reasoning { + inside_hidden_reasoning = false; + suppress_current_update_metadata = true; + } + } + Ok(ChatEvent::ToolCallStart { index, id, name }) => { + let tool_index = index as u32; + saw_tool_calls = true; + debug!( + tool_call_id = %id, + tool_call_name = %name, + "starting new tool call" + ); + if let Some(pending_chunk) = pending_chunk.as_mut() { + pending_chunk.push_tool_call_start(tool_index, id, name); + } else { + yield_chunk!(tool_call_start_chunk( + &request_id, + &response_model, + created, + tool_index, + id, + name, + )); + } + } + Ok(ChatEvent::ToolCallArgumentsDelta { index, delta }) => { + let tool_index = index as u32; + if let Some(pending_chunk) = pending_chunk.as_mut() { + pending_chunk.push_tool_call_arguments(tool_index, delta); + } else { + yield_chunk!(tool_call_arguments_chunk( + &request_id, + &response_model, + created, + tool_index, + delta, + )); + } + } + Ok(ChatEvent::ToolCallEnd { .. }) => { + debug!("ending current tool call"); + } + Ok(ChatEvent::Done { + usage: final_usage, + finish_reason, + .. + }) => { + if enable_log_requests { + info!( + stream = true, + model = %response_model, + prompt_tokens = final_usage.prompt_token_count, + output_tokens = final_usage.output_token_count, + finish_reason = finish_reason.as_str(), + "chat completion finished" + ); + } + + continuous_usage.set_final_counts( + final_usage.prompt_token_count, + final_usage.output_token_count, + ); + + if let Some(pending_chunk) = pending_chunk.as_mut() + && let Some(chunk) = + pending_chunk.take_chunk(&request_id, &response_model, created) + { + yield_chunk!(chunk); + } + + match final_chunk( + &request_id, + &response_model, + created, + finish_reason, + saw_tool_calls, + ) { + Ok(chunk) => yield_chunk!(chunk), + Err(error) => { + error!( + error = %error.to_error_response().error.message, + "invalid terminal finish reason" + ); + return Err(error); + } + } + + if include_usage { + y.yield_ok(usage_chunk( + &request_id, + &response_model, + created, + Usage::from_token_usage(final_usage, enable_prompt_tokens_details), + )) + .await; + } + + return Ok(()); + } + Err(error) => { + error!( + error = %error.as_report(), + "chat stream failed" + ); + bail_server_error!("{}", error.to_report_string()); + } + } + } + Ok(()) +} + +fn usage_chunk( + request_id: &str, + response_model: &str, + created: u64, + usage: Usage, +) -> ChatCompletionStreamResponse { + let mut chunk = ChatCompletionStreamResponse::new(request_id, response_model, created); + chunk.usage = Some(usage); + chunk +} + +/// One in-flight chat-completions SSE chunk being assembled at the route layer. +/// +/// `vllm-chat` emits semantic chat events first and `LogprobsDelta` separately, +/// because one decoded update may be rewritten into multiple chat events. +/// The OpenAI chat API, though, wants one streamed chunk to optionally carry +/// both the delta and its logprobs. +/// +/// This small buffer accumulates the semantic delta first, then attaches the +/// following `LogprobsDelta` and flushes one combined chunk. It relies on the +/// current `vllm-chat` invariant that all semantic events from one decoded +/// update are emitted before that update's `LogprobsDelta`. +#[derive(Debug, Default)] +struct PendingChatChunk { + /// The currently buffered OpenAI delta payload assembled from one or more + /// chat semantic events belonging to the same decoded update. + delta: ChatMessageDelta, + /// The token-aligned logprobs for that same decoded update. + logprobs: Option, + /// Per-update output token IDs for the same decoded update. + token_ids: Option>, +} + +impl PendingChatChunk { + /// Append one assistant text/reasoning block delta to the buffered OpenAI + /// delta payload. + fn push_block_delta(&mut self, kind: AssistantBlockKind, delta: String) { + match kind { + AssistantBlockKind::Text => append_delta_text(&mut self.delta.content, delta), + AssistantBlockKind::Reasoning => append_delta_text(&mut self.delta.reasoning, delta), + AssistantBlockKind::ToolCall => { + unreachable!("tool calls must flow through dedicated tool-call chunks") + } + } + } + + /// Append the OpenAI tool-call-start representation to the buffered delta. + fn push_tool_call_start(&mut self, index: u32, id: String, name: String) { + self.delta.tool_calls.get_or_insert_with(Vec::new).push(ToolCallDelta { + index, + id: Some(id), + tool_type: Some("function".to_string()), + function: Some(FunctionCallDelta { + name: Some(name), + arguments: None, + }), + }); + } + + /// Append one incremental tool-call arguments update to the buffered delta. + fn push_tool_call_arguments(&mut self, index: u32, delta: String) { + self.delta.tool_calls.get_or_insert_with(Vec::new).push(ToolCallDelta { + index, + id: None, + tool_type: None, + function: Some(FunctionCallDelta { + name: None, + arguments: Some(delta), + }), + }); + } + + /// Finalize the currently buffered SSE chunk, if it contains either a + /// semantic delta or a logprobs payload. + /// + /// This may produce: + /// - a combined delta + logprobs chunk + /// - a delta-only chunk + /// - a logprobs-only chunk + /// + /// The logprobs-only case is intentional: token-level metadata in one + /// decoded update is correlated with the same update boundary, not + /// necessarily with a visible/chat-semantic delta. + fn take_chunk( + &mut self, + request_id: &str, + response_model: &str, + created: u64, + ) -> Option { + let has_delta = self.delta.content.is_some() + || self.delta.reasoning.is_some() + || self.delta.tool_calls.is_some(); + let logprobs = self.logprobs.take(); + let token_ids = self.token_ids.take(); + if !has_delta && logprobs.is_none() && token_ids.is_none() { + return None; + } + + let mut chunk = ChatCompletionStreamResponse::new(request_id, response_model, created); + chunk.choices.push(ChatCompletionStreamChoice { + delta: self.take_delta(), + logprobs, + token_ids, + ..Default::default() + }); + Some(chunk) + } + + /// Take the currently buffered OpenAI delta payload and leave this pending + /// chunk empty for the next decoded update. + fn take_delta(&mut self) -> ChatMessageDelta { + ChatMessageDelta { + role: self.delta.role.take(), + content: self.delta.content.take(), + tool_calls: self.delta.tool_calls.take(), + reasoning: self.delta.reasoning.take(), + } + } +} + +/// Append one text fragment to an optional OpenAI delta string field. +fn append_delta_text(slot: &mut Option, delta: String) { + match slot { + Some(existing) => existing.push_str(&delta), + None => *slot = Some(delta), + } +} + +/// Convert one chunk stream into OpenAI-style SSE events. +/// +/// OpenAI-style streaming errors are encoded as ordinary `data: {"error": ...}` +/// events followed by `data: [DONE]`, so the transport stream itself stays +/// infallible even when generation fails after the HTTP response has started. +#[try_stream] +async fn chat_completion_sse_stream( + stream: impl Stream>, + mut y: TryYielder, +) -> Result<(), Infallible> { + pin_mut!(stream); + + while let Some(next) = stream.next().await { + match next { + Ok(chunk) => y.yield_ok(to_sse_event(&chunk)).await, + Err(error) => { + y.yield_ok(to_error_sse_event(&error)).await; + break; + } + } + } + + y.yield_ok(done_sse_event()).await; + Ok(()) +} + +/// Serialize one OpenAI chunk payload into one SSE `data:` event. +fn to_sse_event(chunk: &ChatCompletionStreamResponse) -> Event { + let payload = + serde_json::to_string(chunk).expect("ChatCompletionStreamResponse must serialize to JSON"); + trace!(payload, "chat completion emitting chunk"); + Event::default().data(payload) +} + +/// Serialize one OpenAI error payload into one SSE `data:` event. +fn to_error_sse_event(error: &ApiError) -> Event { + let payload = serde_json::to_string(&error.to_error_response()) + .expect("ErrorResponse must serialize to JSON"); + trace!(payload, "chat completion emitting error"); + Event::default().data(payload) +} + +/// Build the terminal OpenAI SSE sentinel event. +fn done_sse_event() -> Event { + trace!("chat completion emitting done"); + Event::default().data("[DONE]") +} + +/// Build the initial assistant-role SSE chunk required by the OpenAI streaming +/// protocol. +fn start_chunk( + request_id: &str, + response_model: &str, + created: u64, +) -> ChatCompletionStreamResponse { + let mut chunk = ChatCompletionStreamResponse::new(request_id, response_model, created); + chunk.choices.push(ChatCompletionStreamChoice { + delta: ChatMessageDelta { + role: Some(AssistantRole), + ..Default::default() + }, + ..Default::default() + }); + chunk +} + +/// Build one content-delta SSE chunk from one internal assistant block delta. +fn block_delta_chunk( + request_id: &str, + response_model: &str, + created: u64, + kind: AssistantBlockKind, + delta: String, +) -> ChatCompletionStreamResponse { + let delta = match kind { + AssistantBlockKind::Text => ChatMessageDelta { + content: Some(delta), + ..Default::default() + }, + AssistantBlockKind::Reasoning => ChatMessageDelta { + reasoning: Some(delta), + ..Default::default() + }, + AssistantBlockKind::ToolCall => { + unreachable!("tool calls must flow through dedicated tool-call chunks") + } + }; + + let mut chunk = ChatCompletionStreamResponse::new(request_id, response_model, created); + chunk.choices.push(ChatCompletionStreamChoice { + delta, + ..Default::default() + }); + chunk +} + +fn tool_call_start_chunk( + request_id: &str, + response_model: &str, + created: u64, + tool_index: u32, + id: String, + name: String, +) -> ChatCompletionStreamResponse { + let mut chunk = ChatCompletionStreamResponse::new(request_id, response_model, created); + chunk.choices.push(ChatCompletionStreamChoice { + delta: ChatMessageDelta { + tool_calls: Some(vec![ToolCallDelta { + index: tool_index, + id: Some(id), + tool_type: Some("function".to_string()), + function: Some(FunctionCallDelta { + name: Some(name), + arguments: None, + }), + }]), + ..Default::default() + }, + ..Default::default() + }); + chunk +} + +fn tool_call_arguments_chunk( + request_id: &str, + response_model: &str, + created: u64, + tool_index: u32, + delta: String, +) -> ChatCompletionStreamResponse { + let mut chunk = ChatCompletionStreamResponse::new(request_id, response_model, created); + chunk.choices.push(ChatCompletionStreamChoice { + delta: ChatMessageDelta { + tool_calls: Some(vec![ToolCallDelta { + index: tool_index, + id: None, + tool_type: None, + function: Some(FunctionCallDelta { + name: None, + arguments: Some(delta), + }), + }]), + ..Default::default() + }, + ..Default::default() + }); + chunk +} + +fn logprobs_only_chunk( + request_id: &str, + response_model: &str, + created: u64, + logprobs: ChatLogProbs, +) -> ChatCompletionStreamResponse { + let mut chunk = ChatCompletionStreamResponse::new(request_id, response_model, created); + chunk.choices.push(ChatCompletionStreamChoice { + logprobs: Some(logprobs), + ..Default::default() + }); + chunk +} + +/// Build the terminal SSE chunk carrying the OpenAI finish reason. +fn final_chunk( + request_id: &str, + response_model: &str, + created: u64, + finish_reason: FinishReason, + saw_tool_calls: bool, +) -> Result { + let stop_reason = finish_reason.as_stop_reason().map(stop_reason_to_json); + let finish_reason = chat_finish_reason_to_openai(&finish_reason, saw_tool_calls)?; + + debug!( + finish_reason = %finish_reason, + stop_reason = ?stop_reason, + "chat stream finished" + ); + + let mut chunk = ChatCompletionStreamResponse::new(request_id, response_model, created); + chunk.choices.push(ChatCompletionStreamChoice { + finish_reason: Some(finish_reason.to_string()), + stop_reason, + ..Default::default() + }); + Ok(chunk) +} + +fn chat_finish_reason_to_openai( + finish_reason: &FinishReason, + saw_tool_calls: bool, +) -> Result<&'static str, ApiError> { + match finish_reason { + FinishReason::Stop(_) if saw_tool_calls => Ok("tool_calls"), + FinishReason::Stop(_) => Ok("stop"), + FinishReason::Length => Ok("length"), + FinishReason::Abort => Ok("abort"), + FinishReason::Repetition(_) => Ok("repetition"), + FinishReason::Error => { + bail_server_error!("Internal server error"); + } + } +} + +/// Convert one internal stop reason into the OpenAI-compatible `stop_reason` +/// JSON shape. +fn stop_reason_to_json(stop_reason: &StopReason) -> Value { + serde_json::to_value(stop_reason).expect("StopReason must serialize to JSON") +} + +#[cfg(test)] +mod tests { + use futures::{StreamExt as _, stream}; + use serde_json::json; + use vllm_chat::{ + AssistantBlockKind, AssistantContentBlock, AssistantToolCall, ChatEvent, FinishReason, + }; + use vllm_engine_core_client::protocol::output::StopReason; + use vllm_text::{DecodedLogprobs, DecodedPositionLogprobs, DecodedTokenLogprob}; + + use super::{ + ApiServerOptions, ResponseOptions, block_delta_chunk, chat_completion_chunk_stream, + final_chunk, + }; + + #[test] + fn text_chunk_uses_content_only_delta() { + let chunk = block_delta_chunk( + "chatcmpl-1", + "model", + 1, + AssistantBlockKind::Text, + "hello".to_string(), + ); + assert_eq!(chunk.choices[0].delta.role, None); + assert_eq!(chunk.choices[0].delta.content.as_deref(), Some("hello")); + assert_eq!(chunk.choices[0].delta.reasoning, None); + } + + #[test] + fn reasoning_chunk_uses_reasoning_only_delta() { + let chunk = block_delta_chunk( + "chatcmpl-1", + "model", + 1, + AssistantBlockKind::Reasoning, + "thinking".to_string(), + ); + assert_eq!(chunk.choices[0].delta.role, None); + assert_eq!(chunk.choices[0].delta.content, None); + assert_eq!( + chunk.choices[0].delta.reasoning.as_deref(), + Some("thinking") + ); + } + + #[test] + fn final_chunk_maps_stop_finish_reason_and_stop_reason() { + let chunk = final_chunk( + "chatcmpl-1", + "model", + 1, + FinishReason::Stop(Some(StopReason::Text("stop".to_string()))), + false, + ) + .expect("finish reason is valid"); + + assert_eq!(chunk.choices[0].finish_reason.as_deref(), Some("stop")); + assert_eq!(chunk.choices[0].stop_reason, Some(json!("stop"))); + } + + #[test] + fn final_chunk_maps_length_finish_reason() { + let chunk = final_chunk("chatcmpl-1", "model", 1, FinishReason::Length, false) + .expect("finish reason is valid"); + + assert_eq!(chunk.choices[0].finish_reason.as_deref(), Some("length")); + assert_eq!(chunk.choices[0].stop_reason, None); + } + + #[test] + fn final_chunk_maps_abort_finish_reason() { + let chunk = final_chunk("chatcmpl-1", "model", 1, FinishReason::Abort, false) + .expect("abort is a valid finish reason"); + + assert_eq!(chunk.choices[0].finish_reason.as_deref(), Some("abort")); + assert_eq!(chunk.choices[0].stop_reason, None); + } + + #[test] + fn final_chunk_rejects_error_finish_reason() { + assert!(final_chunk("chatcmpl-1", "model", 1, FinishReason::Error, false).is_err()); + } + + #[test] + fn final_chunk_maps_stop_to_tool_calls_when_tool_calls_were_streamed() { + let chunk = final_chunk("chatcmpl-1", "model", 1, FinishReason::stop_eos(), true) + .expect("finish reason is valid"); + + assert_eq!( + chunk.choices[0].finish_reason.as_deref(), + Some("tool_calls") + ); + } + + #[tokio::test] + async fn chunk_stream_coalesces_text_delta_with_logprobs() { + let stream = stream::iter(vec![ + Ok(ChatEvent::Start { + prompt_token_ids: vec![].into(), + prompt_logprobs: None, + }), + Ok(ChatEvent::BlockStart { + index: 0, + kind: AssistantBlockKind::Text, + }), + Ok(ChatEvent::BlockDelta { + index: 0, + kind: AssistantBlockKind::Text, + delta: "hi".to_string(), + }), + Ok(ChatEvent::LogprobsDelta { + logprobs: Some(DecodedLogprobs { + positions: vec![DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "hi".to_string(), + logprob: -0.1, + rank: 1, + }], + }], + }), + token_ids: vec![], + }), + Ok(ChatEvent::Done { + message: Default::default(), + usage: vllm_llm::TokenUsage { + prompt_token_count: 1, + output_token_count: 1, + cached_token_count: 1, + }, + finish_reason: FinishReason::stop_eos(), + kv_transfer_params: None, + }), + ]); + + let chunks = chat_completion_chunk_stream( + stream, + "chatcmpl-1".to_string(), + "model".to_string(), + 1, + ApiServerOptions { + enable_prompt_tokens_details: true, + ..Default::default() + }, + ResponseOptions { + include_usage: true, + requested_logprobs: true, + include_reasoning: true, + ..Default::default() + }, + ) + .collect::>() + .await + .into_iter() + .collect::, _>>() + .expect("stream chunks"); + + assert_eq!(chunks.len(), 4); + assert_eq!(chunks[1].choices[0].delta.content.as_deref(), Some("hi")); + let logprobs = chunks[1].choices[0].logprobs.as_ref().expect("logprobs"); + let content = logprobs.content.as_ref().expect("logprobs content"); + assert_eq!(content[0].token, "hi"); + assert_eq!( + chunks[3] + .usage + .as_ref() + .expect("usage") + .prompt_tokens_details + .as_ref() + .map(|details| details.cached_tokens), + Some(1) + ); + } + + #[tokio::test] + async fn chunk_stream_coalesces_reasoning_delta_with_logprobs() { + let stream = stream::iter(vec![ + Ok(ChatEvent::Start { + prompt_token_ids: vec![].into(), + prompt_logprobs: None, + }), + Ok(ChatEvent::BlockStart { + index: 0, + kind: AssistantBlockKind::Reasoning, + }), + Ok(ChatEvent::BlockDelta { + index: 0, + kind: AssistantBlockKind::Reasoning, + delta: "think".to_string(), + }), + Ok(ChatEvent::LogprobsDelta { + logprobs: Some(DecodedLogprobs { + positions: vec![DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "think".to_string(), + logprob: -0.1, + rank: 1, + }], + }], + }), + token_ids: vec![], + }), + Ok(ChatEvent::Done { + message: Default::default(), + usage: vllm_llm::TokenUsage { + prompt_token_count: 1, + output_token_count: 1, + cached_token_count: 0, + }, + finish_reason: FinishReason::stop_eos(), + kv_transfer_params: None, + }), + ]); + + let chunks = chat_completion_chunk_stream( + stream, + "chatcmpl-1".to_string(), + "model".to_string(), + 1, + ApiServerOptions::default(), + ResponseOptions { + requested_logprobs: true, + include_reasoning: true, + ..Default::default() + }, + ) + .collect::>() + .await + .into_iter() + .collect::, _>>() + .expect("stream chunks"); + + assert_eq!(chunks.len(), 3); + assert_eq!( + chunks[1].choices[0].delta.reasoning.as_deref(), + Some("think") + ); + assert!(chunks[1].choices[0].logprobs.is_some()); + } + + #[tokio::test] + async fn chunk_stream_omits_reasoning_delta_when_disabled() { + let stream = stream::iter(vec![ + Ok(ChatEvent::Start { + prompt_token_ids: vec![].into(), + prompt_logprobs: None, + }), + Ok(ChatEvent::BlockDelta { + index: 0, + kind: AssistantBlockKind::Reasoning, + delta: "think".to_string(), + }), + Ok(ChatEvent::BlockDelta { + index: 1, + kind: AssistantBlockKind::Text, + delta: "answer".to_string(), + }), + Ok(ChatEvent::Done { + message: Default::default(), + usage: vllm_llm::TokenUsage { + prompt_token_count: 1, + output_token_count: 2, + cached_token_count: 0, + }, + finish_reason: FinishReason::stop_eos(), + kv_transfer_params: None, + }), + ]); + + let chunks = chat_completion_chunk_stream( + stream, + "chatcmpl-1".to_string(), + "model".to_string(), + 1, + ApiServerOptions::default(), + ResponseOptions::default(), + ) + .collect::>() + .await + .into_iter() + .collect::, _>>() + .expect("stream chunks"); + + assert_eq!(chunks.len(), 3); + assert_eq!( + chunks[1].choices[0].delta.content.as_deref(), + Some("answer") + ); + assert!( + chunks + .iter() + .all(|chunk| chunk.choices.iter().all(|choice| choice.delta.reasoning.is_none())) + ); + } + + #[tokio::test] + async fn chunk_stream_omits_logprobs_for_suppressed_reasoning() { + let stream = stream::iter(vec![ + Ok(ChatEvent::Start { + prompt_token_ids: vec![].into(), + prompt_logprobs: None, + }), + Ok(ChatEvent::BlockDelta { + index: 0, + kind: AssistantBlockKind::Reasoning, + delta: "think".to_string(), + }), + Ok(ChatEvent::LogprobsDelta { + logprobs: Some(DecodedLogprobs { + positions: vec![DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 11, + token: "think".to_string(), + logprob: -0.1, + rank: 1, + }], + }], + }), + token_ids: vec![11], + }), + Ok(ChatEvent::BlockDelta { + index: 1, + kind: AssistantBlockKind::Text, + delta: "answer".to_string(), + }), + Ok(ChatEvent::LogprobsDelta { + logprobs: Some(DecodedLogprobs { + positions: vec![DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 22, + token: "answer".to_string(), + logprob: -0.2, + rank: 1, + }], + }], + }), + token_ids: vec![22], + }), + Ok(ChatEvent::Done { + message: Default::default(), + usage: vllm_llm::TokenUsage { + prompt_token_count: 1, + output_token_count: 2, + cached_token_count: 0, + }, + finish_reason: FinishReason::stop_eos(), + kv_transfer_params: None, + }), + ]); + + let chunks = chat_completion_chunk_stream( + stream, + "chatcmpl-1".to_string(), + "model".to_string(), + 1, + ApiServerOptions::default(), + ResponseOptions { + requested_logprobs: true, + return_token_ids: true, + ..Default::default() + }, + ) + .collect::>() + .await + .into_iter() + .collect::, _>>() + .expect("stream chunks"); + + assert_eq!(chunks.len(), 3); + let choice = &chunks[1].choices[0]; + assert_eq!(choice.delta.content.as_deref(), Some("answer")); + assert_eq!(choice.token_ids.as_deref(), Some(&[22][..])); + let logprobs = choice.logprobs.as_ref().expect("answer logprobs"); + let content = logprobs.content.as_ref().expect("logprobs content"); + assert_eq!(content[0].token, "answer"); + assert!(chunks.iter().all(|chunk| { + chunk.choices.iter().all(|choice| { + choice.delta.reasoning.is_none() + && choice.token_ids.as_deref() != Some(&[11][..]) + && choice + .logprobs + .as_ref() + .and_then(|logprobs| logprobs.content.as_ref()) + .is_none_or(|content| content.iter().all(|entry| entry.token != "think")) + }) + })); + } + + #[tokio::test] + async fn chunk_stream_omits_logprobs_for_hidden_reasoning_delimiters() { + let stream = stream::iter(vec![ + Ok(ChatEvent::Start { + prompt_token_ids: vec![].into(), + prompt_logprobs: None, + }), + Ok(ChatEvent::BlockStart { + index: 0, + kind: AssistantBlockKind::Reasoning, + }), + Ok(ChatEvent::LogprobsDelta { + logprobs: Some(DecodedLogprobs { + positions: vec![DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 11, + token: "".to_string(), + logprob: -0.1, + rank: 1, + }], + }], + }), + token_ids: vec![11], + }), + Ok(ChatEvent::BlockDelta { + index: 0, + kind: AssistantBlockKind::Reasoning, + delta: "think".to_string(), + }), + Ok(ChatEvent::LogprobsDelta { + logprobs: Some(DecodedLogprobs { + positions: vec![DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 12, + token: "think".to_string(), + logprob: -0.2, + rank: 1, + }], + }], + }), + token_ids: vec![12], + }), + Ok(ChatEvent::BlockEnd { + index: 0, + block: AssistantContentBlock::Reasoning { + text: "think".to_string(), + }, + }), + Ok(ChatEvent::LogprobsDelta { + logprobs: Some(DecodedLogprobs { + positions: vec![DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 13, + token: "".to_string(), + logprob: -0.3, + rank: 1, + }], + }], + }), + token_ids: vec![13], + }), + Ok(ChatEvent::BlockStart { + index: 1, + kind: AssistantBlockKind::Text, + }), + Ok(ChatEvent::BlockDelta { + index: 1, + kind: AssistantBlockKind::Text, + delta: "answer".to_string(), + }), + Ok(ChatEvent::LogprobsDelta { + logprobs: Some(DecodedLogprobs { + positions: vec![DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 22, + token: "answer".to_string(), + logprob: -0.4, + rank: 1, + }], + }], + }), + token_ids: vec![22], + }), + Ok(ChatEvent::Done { + message: Default::default(), + usage: vllm_llm::TokenUsage { + prompt_token_count: 1, + output_token_count: 4, + cached_token_count: 0, + }, + finish_reason: FinishReason::stop_eos(), + kv_transfer_params: None, + }), + ]); + + let chunks = chat_completion_chunk_stream( + stream, + "chatcmpl-1".to_string(), + "model".to_string(), + 1, + ApiServerOptions::default(), + ResponseOptions { + requested_logprobs: true, + return_token_ids: true, + ..Default::default() + }, + ) + .collect::>() + .await + .into_iter() + .collect::, _>>() + .expect("stream chunks"); + + assert_eq!(chunks.len(), 3); + let choice = &chunks[1].choices[0]; + assert_eq!(choice.delta.content.as_deref(), Some("answer")); + assert_eq!(choice.token_ids.as_deref(), Some(&[22][..])); + let logprobs = choice.logprobs.as_ref().expect("answer logprobs"); + let content = logprobs.content.as_ref().expect("logprobs content"); + assert_eq!(content[0].token, "answer"); + assert!(chunks.iter().all(|chunk| { + chunk.choices.iter().all(|choice| { + choice.delta.reasoning.is_none() + && !choice + .token_ids + .as_ref() + .is_some_and(|ids| matches!(ids.as_slice(), [11] | [12] | [13])) + && choice + .logprobs + .as_ref() + .and_then(|logprobs| logprobs.content.as_ref()) + .is_none_or(|content| { + content.iter().all(|entry| { + !matches!(entry.token.as_str(), "" | "think" | "") + }) + }) + }) + })); + } + + #[tokio::test] + async fn chunk_stream_preserves_tool_call_index_and_omits_id_from_arguments_delta() { + let stream = stream::iter(vec![ + Ok(ChatEvent::Start { + prompt_token_ids: vec![].into(), + prompt_logprobs: None, + }), + Ok(ChatEvent::ToolCallStart { + index: 3, + id: "call_1".to_string(), + name: "get_weather".to_string(), + }), + Ok(ChatEvent::ToolCallArgumentsDelta { + index: 3, + delta: r#"{"city":"Paris"}"#.to_string(), + }), + Ok(ChatEvent::ToolCallEnd { + index: 3, + call: AssistantToolCall { + id: "call_1".to_string(), + name: "get_weather".to_string(), + arguments: r#"{"city":"Paris"}"#.to_string(), + }, + }), + Ok(ChatEvent::Done { + message: Default::default(), + usage: vllm_llm::TokenUsage { + prompt_token_count: 1, + output_token_count: 1, + cached_token_count: 0, + }, + finish_reason: FinishReason::stop_eos(), + kv_transfer_params: None, + }), + ]); + + let chunks = chat_completion_chunk_stream( + stream, + "chatcmpl-1".to_string(), + "model".to_string(), + 1, + ApiServerOptions::default(), + ResponseOptions { + include_reasoning: true, + ..Default::default() + }, + ) + .collect::>() + .await + .into_iter() + .collect::, _>>() + .expect("stream chunks"); + + assert_eq!( + chunks[1].choices[0].delta.tool_calls.as_ref().unwrap()[0].index, + 3 + ); + assert_eq!( + chunks[1].choices[0].delta.tool_calls.as_ref().unwrap()[0].id, + Some("call_1".to_string()) + ); + assert_eq!( + chunks[2].choices[0].delta.tool_calls.as_ref().unwrap()[0].index, + 3 + ); + assert_eq!( + chunks[2].choices[0].delta.tool_calls.as_ref().unwrap()[0].id, + None + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/chat_completions/convert.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/chat_completions/convert.rs new file mode 100644 index 0000000000000000000000000000000000000000..1b0bb2933b26ac5a397c78d584850ae9a722e6c2 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/chat_completions/convert.rs @@ -0,0 +1,1210 @@ +use itertools::Itertools as _; +use vllm_chat::{ + AssistantContentBlock, AssistantToolCall, ChatContent, ChatContentPart, + ChatMessage as VllmChatMessage, ChatOptions, ChatRequest, ChatTool, ChatToolChoice, + GenerationPromptMode, SamplingParams, +}; + +use super::types::ChatCompletionRequest; +use super::validate; +use crate::error::{ApiError, bail_invalid_request}; +use crate::lora::LoraModelResolution; +use crate::routes::openai::utils::structured_outputs::convert_from_response_format; +use crate::routes::openai::utils::types::{ + ChatMessage, ContentPart, MessageContent, Tool, ToolChoice, ToolChoiceValue, +}; +use crate::utils::{ResolvedRequestContext, convert_logit_bias, merge_kv_transfer_params}; + +/// Lowered chat request plus the public response metadata carried by every SSE +/// chunk. +#[derive(Debug, Clone, PartialEq)] +pub(super) struct PreparedRequest { + /// Stable OpenAI-style request ID, reused as the external chat request ID. + pub request_id: String, + /// Public model ID echoed back to the client. + pub response_model: String, + /// Public response rendering options for route-layer helpers. + pub options: ResponseOptions, + /// Lowered chat request for `vllm-chat`. + pub chat_request: ChatRequest, +} + +#[derive(Debug, Clone, Default, PartialEq)] +pub(super) struct ResponseOptions { + /// Whether the caller asked for the final streamed usage chunk. + pub include_usage: bool, + /// Whether every streamed chunk should carry cumulative usage. + pub include_continuous_usage: bool, + /// Whether the caller requested output logprobs on chat choices. + pub requested_logprobs: bool, + /// Whether the caller requested top-level prompt logprobs. + pub include_prompt_logprobs: bool, + /// Whether to include reasoning content in OpenAI responses. + pub include_reasoning: bool, + /// Last assistant-role message content to echo back when `echo=true`. + pub echo: Option, + /// Whether to include token IDs alongside generated text. + pub return_token_ids: bool, + /// Whether to format logprob tokens as `token_id:{id}`. + pub return_tokens_as_token_ids: bool, +} + +/// Validate and lower one OpenAI chat completion request into the internal chat +/// format. +/// +/// `lora_resolution.model_names` must be non-empty; the first entry is used as +/// the base `model` field in responses when no LoRA adapter is selected. +pub(super) fn prepare_chat_request( + request: ChatCompletionRequest, + lora_resolution: &LoraModelResolution, + ctx: ResolvedRequestContext, +) -> Result { + validate::validate_request_compat(&request, &lora_resolution.model_names)?; + + let request_id = format!("chatcmpl-{}", ctx.request_id); + let response_model = lora_resolution + .lora_request + .as_ref() + .map(|request| request.lora_name.clone()) + .unwrap_or_else(|| lora_resolution.model_names.first().cloned().unwrap_or_default()); + let include_reasoning = request.include_reasoning; + let echo = request + .echo + .then(|| extract_last_assistant_content(&request.messages)) + .flatten(); + let messages: Vec<_> = request.messages.into_iter().map(convert_message).try_collect()?; + let generation_prompt_mode = normalize_generation_prompt_mode( + request.add_generation_prompt, + request.continue_final_message, + &messages, + )?; + + let template_kwargs = request.chat_template_kwargs.unwrap_or_default(); + + let include_usage = (request.stream_options.as_ref()) + .and_then(|options| options.include_usage) + .unwrap_or(false); + let include_continuous_usage = include_usage + && request + .stream_options + .as_ref() + .and_then(|options| options.continuous_usage_stats) + .unwrap_or(false); + let requested_logprobs = request.logprobs; + + // Auto-enable prompt logprobs for non-streaming echo, matching Python vLLM's + // behavior. + let top_logprobs = request.top_logprobs.unwrap_or(0); + let prompt_logprobs = request + .prompt_logprobs + .or((request.echo && !request.stream).then_some(top_logprobs)); + let include_prompt_logprobs = prompt_logprobs.is_some(); + + let structured_outputs = convert_from_response_format( + request.response_format.as_ref(), + &request.structured_outputs, + )?; + + let chat_request = ChatRequest { + request_id: request_id.clone(), + messages, + sampling_params: SamplingParams { + temperature: request.temperature, + top_p: request.top_p, + top_k: request.top_k, + seed: request.seed, + max_tokens: request.max_completion_tokens, + min_tokens: request.min_tokens, + thinking_token_budget: request.thinking_token_budget, + logprobs: request.logprobs.then_some(top_logprobs), + prompt_logprobs, + min_p: request.min_p, + frequency_penalty: request.frequency_penalty, + presence_penalty: request.presence_penalty, + repetition_penalty: request.repetition_penalty, + repetition_detection: request.repetition_detection, + stop_token_ids: request.stop_token_ids, + ignore_eos: request.ignore_eos, + logit_bias: convert_logit_bias(request.logit_bias)?, + allowed_token_ids: request.allowed_token_ids, + bad_words: request.bad_words, + logprob_token_ids: None, + structured_outputs, + skip_reading_prefix_cache: None, + vllm_xargs: merge_kv_transfer_params( + request.vllm_xargs, + request.kv_transfer_params.as_ref(), + ), + }, + chat_options: ChatOptions { + generation_prompt_mode, + chat_template: request.chat_template, + reasoning_effort: request.reasoning_effort, + template_kwargs, + }, + tools: convert_tools(request.tools)?, + tool_choice: convert_tool_choice(request.tool_choice.as_ref())?, + parallel_tool_calls: request.parallel_tool_calls.unwrap_or(true), + decode_options: vllm_text::output::TextDecodeOptions { + skip_special_tokens: request.skip_special_tokens, + include_stop_str_in_output: request.include_stop_str_in_output, + stop_strings: request.stop.map(|stop| stop.into_vec()), + min_tokens: request.min_tokens.unwrap_or(0), + }, + intermediate: request.stream, + priority: request.priority.unwrap_or(0), + documents: request.documents, + cache_salt: request.cache_salt, + add_special_tokens: request.add_special_tokens, + data_parallel_rank: ctx.data_parallel_rank, + lora_request: lora_resolution.lora_request.clone(), + }; + + Ok(PreparedRequest { + request_id, + response_model, + options: ResponseOptions { + include_usage, + include_continuous_usage, + requested_logprobs, + include_prompt_logprobs, + include_reasoning, + echo, + return_token_ids: request.return_token_ids.unwrap_or(false), + return_tokens_as_token_ids: request.return_tokens_as_token_ids.unwrap_or(false), + }, + chat_request, + }) +} +pub(crate) fn normalize_generation_prompt_mode( + add_generation_prompt: Option, + continue_final_message: bool, + messages: &[VllmChatMessage], +) -> Result { + if add_generation_prompt == Some(true) && continue_final_message { + bail_invalid_request!( + "Cannot set both `continue_final_message` and `add_generation_prompt` to True." + ); + } + + let last_role = messages.last().map(VllmChatMessage::role); + match (add_generation_prompt, continue_final_message, last_role) { + (Some(true), true, _) => unreachable!("rejected above"), + (_, true, Some(vllm_chat::ChatRole::Assistant)) => { + Ok(GenerationPromptMode::ContinueFinalAssistant) + } + (_, true, _) => { + bail_invalid_request!( + "Cannot set `continue_final_message` to True when the last message is not from the assistant." + ); + } + (Some(false), false, _) => Ok(GenerationPromptMode::NoGenerationPrompt), + (None | Some(true), false, _) => Ok(GenerationPromptMode::StartNewAssistant), + } +} + +/// Extract the text content of the last message if it has the assistant role. +fn extract_last_assistant_content(messages: &[ChatMessage]) -> Option { + let ChatMessage::Assistant { content, .. } = messages.last()? else { + return None; + }; + let text = match content.as_ref()? { + MessageContent::Text(text) => text.clone(), + MessageContent::Parts(parts) => parts + .iter() + .filter_map(|p| match p { + ContentPart::Text { text } => Some(text.as_str()), + _ => None, + }) + .collect::>() + .join("\n"), + }; + (!text.is_empty()).then_some(text) +} + +/// Lower one OpenAI chat message into the `vllm-chat` message shape. +pub(crate) fn convert_message(message: ChatMessage) -> Result { + match message { + ChatMessage::System { content, .. } => { + Ok(VllmChatMessage::system(convert_content(content)?)) + } + ChatMessage::User { content, .. } => Ok(VllmChatMessage::user(convert_content(content)?)), + ChatMessage::Assistant { + content, + tool_calls, + reasoning, + name: _, + } => { + let mut blocks = Vec::new(); + if let Some(reasoning) = reasoning + && !reasoning.is_empty() + { + blocks.push(AssistantContentBlock::Reasoning { text: reasoning }); + } + if let Some(content) = content { + blocks.extend(convert_assistant_text_blocks(content)?); + } + if let Some(tool_calls) = tool_calls { + blocks.extend(convert_assistant_tool_calls(tool_calls)?); + } + if blocks.is_empty() { + bail_invalid_request!( + "Assistant messages must contain text, reasoning content, or tool_calls." + ); + } + + Ok(VllmChatMessage::assistant_blocks(blocks)) + } + ChatMessage::Tool { + content, + tool_call_id, + } => Ok(VllmChatMessage::tool_response( + convert_content(content)?, + tool_call_id, + )), + ChatMessage::Function { .. } => { + bail_invalid_request!("Function messages are not supported.") + } + ChatMessage::Developer { + content, + tools, + name: _, + } => Ok(VllmChatMessage::developer( + convert_content(content)?, + convert_message_tools(tools)?, + )), + } +} + +/// Convert the given OpenAI message content value into the internal format in +/// `vllm-chat`. +fn convert_content(content: MessageContent) -> Result { + match content { + MessageContent::Text(text) => Ok(ChatContent::Text(text)), + MessageContent::Parts(parts) => parts + .into_iter() + .map(|part| match part { + ContentPart::Text { text } => Ok(ChatContentPart::text(text)), + ContentPart::ImageUrl { image_url, uuid } => Ok(ChatContentPart::ImageUrl { + image_url: image_url.url, + detail: image_url.detail, + uuid, + }), + _ => bail_invalid_request!("Only text and image_url content parts are supported."), + }) + .try_collect() + .map(ChatContent::Parts), + } +} + +/// Convert the given OpenAI assistant message content into the internal format +/// in `vllm-chat`. +fn convert_assistant_text_blocks( + content: MessageContent, +) -> Result, ApiError> { + match content { + MessageContent::Text(text) => Ok(vec![AssistantContentBlock::Text { text }]), + MessageContent::Parts(parts) => parts + .into_iter() + .map(|part| match part { + ContentPart::Text { text } => Ok(AssistantContentBlock::Text { text }), + _ => bail_invalid_request!( + "Only text content parts are supported for assistant messages." + ), + }) + .try_collect(), + } +} + +fn convert_assistant_tool_calls( + tool_calls: Vec, +) -> Result, ApiError> { + tool_calls + .into_iter() + .map(|tool_call| { + if tool_call.tool_type != "function" { + bail_invalid_request!("Only function tool calls are supported."); + } + + Ok(AssistantContentBlock::ToolCall(AssistantToolCall { + id: tool_call.id, + name: tool_call.function.name, + arguments: tool_call.function.arguments.unwrap_or_else(|| "{}".to_string()), + })) + }) + .collect() +} + +pub(crate) fn convert_tools(tools: Option>) -> Result, ApiError> { + tools + .unwrap_or_default() + .into_iter() + .map(|tool| { + if tool.tool_type != "function" { + bail_invalid_request!("Only function tools are supported."); + } + Ok(ChatTool { + name: tool.function.name, + description: tool.function.description, + parameters: tool.function.parameters, + strict: tool.function.strict, + }) + }) + .collect() +} + +fn convert_message_tools(tools: Option>) -> Result>, ApiError> { + let tools = convert_tools(tools)?; + Ok((!tools.is_empty()).then_some(tools)) +} + +fn convert_tool_choice(tool_choice: Option<&ToolChoice>) -> Result { + match tool_choice { + None | Some(ToolChoice::Value(ToolChoiceValue::Auto)) => Ok(ChatToolChoice::Auto), + Some(ToolChoice::Value(ToolChoiceValue::None)) => Ok(ChatToolChoice::None), + Some(ToolChoice::Value(ToolChoiceValue::Required)) => Ok(ChatToolChoice::Required), + Some(ToolChoice::Function { + tool_type, + function, + }) if tool_type == "function" => Ok(ChatToolChoice::Function { + name: function.name.clone(), + }), + _ => bail_invalid_request!("tool_choice={:?} is not supported yet.", tool_choice), + } +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + + use axum::http::HeaderMap; + use expect_test::expect; + use llm_multimodal::ImageDetail; + use serde_json::json; + use vllm_chat::{ + AssistantContentBlock, AssistantToolCall, ChatContentPart, ChatMessage as VllmChatMessage, + ChatTool as VllmChatTool, ChatToolChoice, GenerationPromptMode, + SamplingParams as VllmSamplingParams, + }; + use vllm_text::output::TextDecodeOptions; + + use super::prepare_chat_request; + use crate::lora::LoraModelResolution; + use crate::routes::openai::chat_completions::types::{ + AssistantRole, ChatCompletionMessage, ChatCompletionRequest, + }; + use crate::routes::openai::utils::types::{ + ChatMessage, ContentPart, Function, FunctionCallResponse, ImageUrl, MessageContent, + StreamOptions, Tool, ToolCall, ToolChoice, ToolChoiceValue, VideoUrl, + }; + use crate::utils::{ResolvedRequestContext, resolve_request_context}; + + fn request_context(headers: &HeaderMap, request_id: Option<&str>) -> ResolvedRequestContext { + resolve_request_context(headers, request_id) + } + + fn served(names: &[&str]) -> LoraModelResolution { + LoraModelResolution { + model_names: names.iter().map(|s| s.to_string()).collect(), + lora_request: None, + } + } + + fn base_request() -> ChatCompletionRequest { + ChatCompletionRequest { + model: "Qwen/Qwen1.5-0.5B-Chat".to_string(), + messages: vec![ChatMessage::User { + content: MessageContent::Text("hello".to_string()), + name: None, + }], + stream: true, + ..Default::default() + } + } + + #[test] + fn prepare_chat_request_maps_parallel_tool_calls() { + let mut request = base_request(); + request.parallel_tool_calls = Some(false); + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert!(!prepared.chat_request.parallel_tool_calls); + } + + #[test] + fn prepare_chat_request_defaults_parallel_tool_calls_to_true() { + let prepared = prepare_chat_request( + base_request(), + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert!(prepared.chat_request.parallel_tool_calls); + } + + #[test] + fn prepare_chat_request_maps_text_parts() { + let mut request = base_request(); + request.messages = vec![ChatMessage::Assistant { + content: Some(MessageContent::Parts(vec![ContentPart::Text { + text: "hello".to_string(), + }])), + name: None, + tool_calls: None, + reasoning: None, + }]; + request.add_generation_prompt = Some(false); + request.continue_final_message = true; + request.skip_special_tokens = false; + request.chat_template_kwargs = Some(HashMap::from([("foo".to_string(), json!("bar"))])); + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert!(prepared.request_id.starts_with("chatcmpl-")); + assert_eq!( + prepared.chat_request.messages, + vec![VllmChatMessage::assistant_text("hello")] + ); + assert_eq!( + prepared.chat_request.sampling_params, + VllmSamplingParams::default() + ); + assert_eq!( + prepared.chat_request.chat_options.generation_prompt_mode, + GenerationPromptMode::ContinueFinalAssistant + ); + assert_eq!( + prepared.chat_request.chat_options.template_kwargs, + HashMap::from([("foo".to_string(), json!("bar"))]) + ); + assert_eq!( + prepared.chat_request.decode_options, + TextDecodeOptions { + skip_special_tokens: false, + include_stop_str_in_output: false, + stop_strings: None, + min_tokens: 0, + } + ); + assert!(prepared.chat_request.tools.is_empty()); + assert_eq!(prepared.chat_request.tool_choice, ChatToolChoice::Auto); + } + + #[test] + fn prepare_chat_request_maps_stream_usage_and_token_format_options() { + let mut request = base_request(); + request.return_tokens_as_token_ids = Some(true); + request.stream_options = Some(StreamOptions { + include_usage: Some(true), + continuous_usage_stats: Some(true), + }); + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert!(prepared.options.include_usage); + assert!(prepared.options.include_continuous_usage); + assert!(prepared.options.return_tokens_as_token_ids); + } + + #[test] + fn prepare_chat_request_gates_continuous_usage_on_include_usage() { + let mut request = base_request(); + request.stream_options = Some(StreamOptions { + include_usage: None, + continuous_usage_stats: Some(true), + }); + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert!(!prepared.options.include_usage); + assert!(!prepared.options.include_continuous_usage); + } + + #[test] + fn prepare_chat_request_keeps_optional_sampling_fields_unset() { + let prepared = prepare_chat_request( + base_request(), + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert!(prepared.request_id.starts_with("chatcmpl-")); + assert_eq!( + prepared.chat_request.messages, + vec![VllmChatMessage::user("hello")] + ); + assert_eq!( + prepared.chat_request.sampling_params, + VllmSamplingParams::default() + ); + assert_eq!( + prepared.chat_request.chat_options.generation_prompt_mode, + GenerationPromptMode::StartNewAssistant + ); + assert_eq!( + prepared.chat_request.decode_options, + TextDecodeOptions { + skip_special_tokens: true, + include_stop_str_in_output: false, + stop_strings: None, + min_tokens: 0, + } + ); + assert!(prepared.chat_request.tools.is_empty()); + assert_eq!(prepared.chat_request.tool_choice, ChatToolChoice::Auto); + } + + #[test] + fn prepare_chat_request_preserves_include_reasoning_false() { + let request = ChatCompletionRequest { + include_reasoning: false, + ..base_request() + }; + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert!(!prepared.options.include_reasoning); + } + + #[test] + fn prepare_chat_request_preserves_sampling_passthrough_fields() { + let request = ChatCompletionRequest { + seed: Some(42), + min_p: Some(0.2), + frequency_penalty: Some(0.3), + presence_penalty: Some(0.4), + repetition_penalty: Some(1.1), + ..base_request() + }; + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + let expected = VllmSamplingParams { + seed: Some(42), + min_p: Some(0.2), + frequency_penalty: Some(0.3), + presence_penalty: Some(0.4), + repetition_penalty: Some(1.1), + ..VllmSamplingParams::default() + }; + assert_eq!(prepared.chat_request.sampling_params, expected); + } + + #[test] + fn prepare_chat_request_passes_through_thinking_token_budget() { + let prepare = |budget: Option| { + prepare_chat_request( + ChatCompletionRequest { + thinking_token_budget: budget, + ..base_request() + }, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid") + .chat_request + .sampling_params + .thinking_token_budget + }; + + // The convert layer forwards the raw value verbatim (including the `-1` + // "unlimited" sentinel); normalization/validation happens during + // lowering (see `vllm_text::lower`). + assert_eq!(prepare(Some(64)), Some(64)); + assert_eq!(prepare(Some(-1)), Some(-1)); + assert_eq!(prepare(None), None); + } + + #[test] + fn prepare_chat_request_accepts_developer_messages() { + let request = ChatCompletionRequest { + messages: vec![ChatMessage::Developer { + content: MessageContent::Text("hello".to_string()), + tools: Some(vec![Tool { + tool_type: "function".to_string(), + function: Function { + name: "get_weather".to_string(), + description: Some("Get weather".to_string()), + parameters: json!({ + "type": "object", + "properties": {"city": {"type": "string"}}, + }), + strict: Some(true), + }, + }]), + name: None, + }], + ..base_request() + }; + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert_eq!( + prepared.chat_request.messages, + vec![VllmChatMessage::developer( + "hello", + Some(vec![VllmChatTool { + name: "get_weather".to_string(), + description: Some("Get weather".to_string()), + parameters: json!({ + "type": "object", + "properties": {"city": {"type": "string"}}, + }), + strict: Some(true), + }]), + )] + ); + } + + #[test] + fn prepare_chat_request_maps_image_url_content_parts() { + let request = ChatCompletionRequest { + messages: vec![ChatMessage::User { + content: MessageContent::Parts(vec![ + ContentPart::Text { + text: "describe ".to_string(), + }, + ContentPart::ImageUrl { + image_url: ImageUrl { + url: "https://example.com/image.png".to_string(), + detail: Some(ImageDetail::Low), + }, + uuid: Some("image-1".to_string()), + }, + ContentPart::Text { + text: " briefly".to_string(), + }, + ]), + name: None, + }], + ..base_request() + }; + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert_eq!( + prepared.chat_request.messages, + vec![VllmChatMessage::user(vec![ + ChatContentPart::text("describe "), + ChatContentPart::ImageUrl { + image_url: "https://example.com/image.png".to_string(), + detail: Some(ImageDetail::Low), + uuid: Some("image-1".to_string()), + }, + ChatContentPart::text(" briefly"), + ])] + ); + } + + #[test] + fn prepare_chat_request_maps_developer_image_url_content_parts() { + let request = ChatCompletionRequest { + messages: vec![ChatMessage::Developer { + content: MessageContent::Parts(vec![ContentPart::ImageUrl { + image_url: ImageUrl { + url: "https://example.com/image.png".to_string(), + detail: None, + }, + uuid: None, + }]), + tools: None, + name: None, + }], + ..base_request() + }; + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert_eq!( + prepared.chat_request.messages, + vec![VllmChatMessage::developer( + vec![ChatContentPart::image_url("https://example.com/image.png")], + None, + )] + ); + } + + #[test] + fn prepare_chat_request_rejects_video_content_parts() { + let request = ChatCompletionRequest { + messages: vec![ChatMessage::User { + content: MessageContent::Parts(vec![ContentPart::VideoUrl { + video_url: VideoUrl { + url: "https://example.com/video.mp4".to_string(), + }, + }]), + name: None, + }], + ..base_request() + }; + + let error = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .unwrap_err(); + + expect!["Only text and image_url content parts are supported."] + .assert_eq(&error.to_error_response().error.message); + } + + #[test] + fn prepare_chat_request_rejects_assistant_image_url_content_parts() { + let request = ChatCompletionRequest { + messages: vec![ChatMessage::Assistant { + content: Some(MessageContent::Parts(vec![ContentPart::ImageUrl { + image_url: ImageUrl { + url: "https://example.com/image.png".to_string(), + detail: None, + }, + uuid: None, + }])), + name: None, + tool_calls: None, + reasoning: None, + }], + ..base_request() + }; + + let error = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .unwrap_err(); + + expect!["Only text content parts are supported for assistant messages."] + .assert_eq(&error.to_error_response().error.message); + } + + #[test] + fn prepare_chat_request_accepts_assistant_reasoning_history() { + let message = ChatCompletionMessage { + role: AssistantRole, + content: Some("answer".to_string()), + tool_calls: None, + reasoning: Some("inner".to_string()), + }; + let message_json = serde_json::to_value(message).expect("message serializes"); + + let request = ChatCompletionRequest { + messages: vec![ + serde_json::from_value(message_json).expect("response message is valid history"), + ], + add_generation_prompt: Some(false), + ..base_request() + }; + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + assert_eq!( + prepared.chat_request.messages, + vec![VllmChatMessage::assistant_blocks(vec![ + AssistantContentBlock::Reasoning { + text: "inner".to_string(), + }, + AssistantContentBlock::Text { + text: "answer".to_string(), + }, + ])] + ); + assert!(prepared.chat_request.tools.is_empty()); + assert_eq!(prepared.chat_request.tool_choice, ChatToolChoice::Auto); + } + + #[test] + fn prepare_chat_request_accepts_legacy_reasoning_content_alias() { + let request = ChatCompletionRequest { + messages: vec![ + serde_json::from_value(json!({ + "role": "assistant", + "content": "answer", + "reasoning_content": "inner", + })) + .expect("legacy reasoning_content alias is accepted"), + ], + add_generation_prompt: Some(false), + ..base_request() + }; + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + assert_eq!( + prepared.chat_request.messages, + vec![VllmChatMessage::assistant_blocks(vec![ + AssistantContentBlock::Reasoning { + text: "inner".to_string(), + }, + AssistantContentBlock::Text { + text: "answer".to_string(), + }, + ])] + ); + } + + #[test] + fn prepare_chat_request_accepts_tools_and_tool_history() { + let request = ChatCompletionRequest { + messages: vec![ + ChatMessage::Assistant { + content: None, + name: None, + tool_calls: Some(vec![ToolCall { + id: "call_1".to_string(), + tool_type: "function".to_string(), + function: FunctionCallResponse { + name: "get_weather".to_string(), + arguments: Some(r#"{"city":"Paris"}"#.to_string()), + }, + }]), + reasoning: None, + }, + ChatMessage::Tool { + content: MessageContent::Text("Sunny".to_string()), + tool_call_id: "call_1".to_string(), + }, + ], + tools: Some(vec![Tool { + tool_type: "function".to_string(), + function: Function { + name: "get_weather".to_string(), + description: Some("Get weather".to_string()), + parameters: json!({ + "type": "object", + "properties": {"city": {"type": "string"}}, + }), + strict: None, + }, + }]), + tool_choice: Some(ToolChoice::Value(ToolChoiceValue::None)), + ..base_request() + }; + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + assert_eq!( + prepared.chat_request.messages, + vec![ + VllmChatMessage::assistant_blocks(vec![AssistantContentBlock::ToolCall( + AssistantToolCall { + id: "call_1".to_string(), + name: "get_weather".to_string(), + arguments: r#"{"city":"Paris"}"#.to_string(), + }, + )]), + VllmChatMessage::tool_response("Sunny", "call_1"), + ] + ); + assert_eq!( + prepared.chat_request.tools, + vec![VllmChatTool { + name: "get_weather".to_string(), + description: Some("Get weather".to_string()), + parameters: json!({ + "type": "object", + "properties": {"city": {"type": "string"}}, + }), + strict: None, + }] + ); + assert_eq!(prepared.chat_request.tool_choice, ChatToolChoice::None); + } + + #[test] + fn prepare_chat_request_lowers_required_tool_choice() { + let request = ChatCompletionRequest { + tools: Some(vec![Tool { + tool_type: "function".to_string(), + function: Function { + name: "get_weather".to_string(), + description: Some("Get weather".to_string()), + parameters: json!({ + "type": "object", + "properties": {"city": {"type": "string"}}, + }), + strict: None, + }, + }]), + tool_choice: Some(ToolChoice::Value(ToolChoiceValue::Required)), + ..base_request() + }; + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert_eq!(prepared.chat_request.tool_choice, ChatToolChoice::Required); + } + + #[test] + fn prepare_chat_request_lowers_named_function_tool_choice() { + let request = ChatCompletionRequest { + tools: Some(vec![Tool { + tool_type: "function".to_string(), + function: Function { + name: "get_weather".to_string(), + description: Some("Get weather".to_string()), + parameters: json!({ + "type": "object", + "properties": {"city": {"type": "string"}}, + }), + strict: None, + }, + }]), + tool_choice: Some(ToolChoice::Function { + tool_type: "function".to_string(), + function: crate::routes::openai::utils::types::FunctionChoice { + name: "get_weather".to_string(), + }, + }), + ..base_request() + }; + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert_eq!( + prepared.chat_request.tool_choice, + ChatToolChoice::Function { + name: "get_weather".to_string(), + } + ); + } + + #[test] + fn prepare_chat_request_lowers_logprobs_fields() { + let request = ChatCompletionRequest { + stream: false, + logprobs: true, + prompt_logprobs: Some(2), + ..base_request() + }; + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert!(prepared.options.requested_logprobs); + assert!(prepared.options.include_prompt_logprobs); + assert_eq!(prepared.chat_request.sampling_params.logprobs, Some(0)); + assert_eq!( + prepared.chat_request.sampling_params.prompt_logprobs, + Some(2) + ); + } + + #[test] + fn prepare_chat_request_keeps_prompt_logprobs_independent_from_echo() { + let request = ChatCompletionRequest { + logprobs: true, + top_logprobs: Some(3), + echo: true, + ..base_request() + }; + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert_eq!(prepared.chat_request.sampling_params.logprobs, Some(3)); + assert_eq!(prepared.chat_request.sampling_params.prompt_logprobs, None); + assert!(!prepared.options.include_prompt_logprobs); + } + + #[test] + fn prepare_chat_request_threads_data_parallel_rank() { + let mut headers = HeaderMap::new(); + headers.insert("X-data-parallel-rank", "7".parse().unwrap()); + let prepared = prepare_chat_request( + base_request(), + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + request_context(&headers, None), + ) + .expect("request is valid"); + assert_eq!(prepared.chat_request.data_parallel_rank, Some(7)); + } + + #[test] + fn prepare_chat_request_leaves_data_parallel_rank_none_when_absent() { + let prepared = prepare_chat_request( + base_request(), + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + assert_eq!(prepared.chat_request.data_parallel_rank, None); + } + + #[test] + fn prepare_chat_request_maps_no_generation_prompt_mode() { + let mut request = base_request(); + request.add_generation_prompt = Some(false); + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert_eq!( + prepared.chat_request.chat_options.generation_prompt_mode, + GenerationPromptMode::NoGenerationPrompt + ); + } + + #[test] + fn prepare_chat_request_rejects_conflicting_explicit_generation_prompt_flags() { + let mut request = base_request(); + request.add_generation_prompt = Some(true); + request.continue_final_message = true; + + let error = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .unwrap_err(); + + expect!["Cannot set both `continue_final_message` and `add_generation_prompt` to True."] + .assert_eq(&error.to_error_response().error.message); + } + + #[test] + fn prepare_chat_request_accepts_continue_final_message_with_implicit_add_generation_prompt() { + let mut request = base_request(); + request.messages = vec![ChatMessage::Assistant { + content: Some(MessageContent::Text("hello".to_string())), + name: None, + tool_calls: None, + reasoning: None, + }]; + request.continue_final_message = true; + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert_eq!( + prepared.chat_request.chat_options.generation_prompt_mode, + GenerationPromptMode::ContinueFinalAssistant + ); + } + + #[test] + fn prepare_chat_request_rejects_continue_final_message_without_final_assistant() { + let mut request = base_request(); + request.continue_final_message = true; + + let error = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .unwrap_err(); + + expect!["Cannot set `continue_final_message` to True when the last message is not from the assistant."] + .assert_eq(&error.to_error_response().error.message); + } + + #[test] + fn prepare_chat_request_allows_new_assistant_mode_after_final_assistant() { + let request = ChatCompletionRequest { + messages: vec![ChatMessage::Assistant { + content: Some(MessageContent::Text("hello".to_string())), + name: None, + tool_calls: None, + reasoning: None, + }], + ..base_request() + }; + + let prepared = prepare_chat_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + ) + .expect("request is valid"); + + assert_eq!( + prepared.chat_request.chat_options.generation_prompt_mode, + GenerationPromptMode::StartNewAssistant + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/chat_completions/types.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/chat_completions/types.rs new file mode 100644 index 0000000000000000000000000000000000000000..e9841a92b35568b74f44bd52295437cbb431b7ec --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/chat_completions/types.rs @@ -0,0 +1,577 @@ +use std::collections::HashMap; +use std::fmt; + +use serde::{Deserialize, Serialize}; +use serde_json::Value; +use serde_with::SerializeDisplay; +use validator::Validate; +use vllm_chat::ReasoningEffort; +use vllm_engine_core_client::protocol::sampling::RepetitionDetectionParams; + +use crate::routes::openai::utils::structured_outputs::ResponseFormat; +use crate::routes::openai::utils::types::{ + ChatLogProbs, ChatMessage, Normalizable, StreamOptions, StringOrArray, Tool, ToolCall, + ToolCallDelta, ToolChoice, ToolChoiceValue, ToolReference, UNKNOWN_MODEL_ID, Usage, + default_true, validate_messages, validate_stop, validate_top_p_value, +}; + +/// vLLM-compatible request type for the Chat Completions API. +/// +/// Mirrors the Python vLLM `ChatCompletionRequest` class. The local copy keeps +/// the request type route-owned so we can add vLLM-only fields directly instead +/// of layering wrapper deserializers on top. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Deserialize, Serialize, Validate)] +#[validate(schema(function = "validate_chat_cross_parameters"))] +pub struct ChatCompletionRequest { + // -------- Standard OpenAI API Parameters -------- + /// A list of messages comprising the conversation so far + #[validate(custom(function = "validate_messages"))] + pub messages: Vec, + + /// ID of the model to use + #[serde(default = "default_model")] + pub model: String, + + /// Number between -2.0 and 2.0. Positive values penalize new tokens based + /// on their existing frequency in the text so far + #[validate(range(min = -2.0, max = 2.0))] + pub frequency_penalty: Option, + + /// Modify the likelihood of specified tokens appearing in the completion + pub logit_bias: Option>, + + /// Whether to return log probabilities of the output tokens + #[serde(default)] + pub logprobs: bool, + + /// An integer specifying the number of most likely tokens to return + /// -1 means return all + #[validate(range(min = -1))] + pub top_logprobs: Option, + + /// Deprecated: Replaced by max_completion_tokens + #[deprecated(note = "Use max_completion_tokens instead")] + #[validate(range(min = 1))] + pub max_tokens: Option, + + /// An upper bound for the number of tokens that can be generated for a + /// completion + #[validate(range(min = 1))] + pub max_completion_tokens: Option, + + /// How many chat completion choices to generate for each input message + #[validate(range(min = 1, max = 10))] + pub n: Option, + + /// Number between -2.0 and 2.0. Positive values penalize new tokens based + /// on whether they appear in the text so far + #[validate(range(min = -2.0, max = 2.0))] + pub presence_penalty: Option, + + /// An object specifying the format that the model must output + pub response_format: Option, + + /// If specified, our system will make a best effort to sample + /// deterministically + pub seed: Option, + + /// Up to 4 sequences where the API will stop generating further tokens + #[validate(custom(function = "validate_stop"))] + pub stop: Option, + + /// If set, partial message deltas will be sent + #[serde(default)] + pub stream: bool, + + /// Options for streaming response + pub stream_options: Option, + + /// What sampling temperature to use, between 0 and 2 + #[validate(range(min = 0.0, max = 2.0))] + pub temperature: Option, + + /// An alternative to sampling with temperature + #[validate(custom(function = "validate_top_p_value"))] + pub top_p: Option, + + /// A list of tools the model may call + pub tools: Option>, + + /// Controls which (if any) tool is called by the model + pub tool_choice: Option, + + /// Effort level for reasoning models (none, minimal, low, medium, high, + /// xhigh, max) + pub reasoning_effort: Option, + + /// Whether to enable parallel function calling during tool use + pub parallel_tool_calls: Option, + + /// A unique identifier representing your end-user + pub user: Option, + + // -------- vLLM Sampling Parameters -------- + /// Use beam search instead of sampling + #[serde(default)] + pub use_beam_search: bool, + + /// Top-k sampling parameter + pub top_k: Option, + + /// Min-p nucleus sampling parameter + #[validate(range(min = 0.0, max = 1.0))] + pub min_p: Option, + + /// Repetition penalty for reducing repetitive text + #[validate(range(min = 0.0, max = 2.0))] + pub repetition_penalty: Option, + + /// Length penalty for beam search + pub length_penalty: Option, + + /// Specific token IDs to use as stop conditions + pub stop_token_ids: Option>, + + /// Include stop string in output + #[serde(default)] + pub include_stop_str_in_output: bool, + + /// Ignore end-of-sequence tokens during generation + #[serde(default)] + pub ignore_eos: bool, + + /// Minimum number of tokens to generate + #[validate(range(min = 1))] + pub min_tokens: Option, + + /// Skip special tokens during detokenization + #[serde(default = "default_true")] + pub skip_special_tokens: bool, + + /// Add spaces between special tokens during detokenization + #[serde(default = "default_true")] + pub spaces_between_special_tokens: bool, + + /// Truncate prompt tokens to this length + pub truncate_prompt_tokens: Option, + + /// Number of prompt logprobs to return + pub prompt_logprobs: Option, + + /// Restrict output to these token IDs only + pub allowed_token_ids: Option>, + + /// List of bad words to avoid during generation + pub bad_words: Option>, + + // -------- Extra vLLM Parameters -------- + /// Token budget for reasoning/thinking. Accepts a non-negative integer, or + /// `-1` for unlimited (mirroring the Python frontend, which normalizes `-1` + /// to "no budget"). + pub thinking_token_budget: Option, + + /// Whether to include reasoning content in the response + #[serde(default = "default_true")] + pub include_reasoning: bool, + + /// If true, the new message will be prepended with the last message if they + /// belong to the same role. + #[serde(default)] + pub echo: bool, + + /// Whether to add the generation prompt to the chat template. + /// + /// When omitted, the request follows the API default behavior, which is + /// equivalent to `true` unless `continue_final_message=true` selects + /// final assistant continuation instead. + pub add_generation_prompt: Option, + + /// Continue generating from final assistant message + #[serde(default)] + pub continue_final_message: bool, + + /// Whether to add special tokens (e.g. BOS) to the prompt + #[serde(default)] + pub add_special_tokens: bool, + + /// Documents for RAG (retrieval-augmented generation) + pub documents: Option>, + + /// Jinja chat template override + pub chat_template: Option, + + /// Additional keyword args passed to the chat template renderer + pub chat_template_kwargs: Option>, + + /// Additional kwargs for media IO connectors, keyed by modality + pub media_io_kwargs: Option>, + + /// Additional kwargs for the HF processor + pub mm_processor_kwargs: Option>, + + /// Additional kwargs for structured outputs + pub structured_outputs: Option, + + /// Request scheduling priority (lower means earlier; default 0) + pub priority: Option, + + /// External request ID used for response correlation. + pub request_id: Option, + + /// Tokens represented as strings of the form 'token_id:{token_id}' in + /// logprobs + pub return_tokens_as_token_ids: Option, + + /// Include token IDs alongside generated text + pub return_token_ids: Option, + + /// Salt for prefix cache isolation in multi-user environments + pub cache_salt: Option, + + /// KV transfer parameters for disaggregated serving + pub kv_transfer_params: Option>, + + /// Additional request parameters with string or numeric values for custom + /// extensions + pub vllm_xargs: Option>, + + /// Parameters for detecting repetitive N-gram patterns in output tokens + pub repetition_detection: Option, +} + +impl Default for ChatCompletionRequest { + #[expect(deprecated)] + fn default() -> Self { + Self { + messages: Vec::new(), + model: default_model(), + frequency_penalty: None, + logit_bias: None, + logprobs: false, + top_logprobs: None, + max_tokens: None, + max_completion_tokens: None, + n: None, + presence_penalty: None, + response_format: None, + seed: None, + stop: None, + stream: false, + stream_options: None, + temperature: None, + top_p: None, + tools: None, + tool_choice: None, + reasoning_effort: None, + thinking_token_budget: None, + include_reasoning: true, + parallel_tool_calls: None, + user: None, + use_beam_search: false, + top_k: None, + min_p: None, + repetition_penalty: None, + length_penalty: None, + stop_token_ids: None, + include_stop_str_in_output: false, + ignore_eos: false, + min_tokens: None, + skip_special_tokens: true, + spaces_between_special_tokens: true, + truncate_prompt_tokens: None, + prompt_logprobs: None, + allowed_token_ids: None, + bad_words: None, + echo: false, + add_generation_prompt: None, + continue_final_message: false, + add_special_tokens: false, + documents: None, + chat_template: None, + chat_template_kwargs: None, + media_io_kwargs: None, + mm_processor_kwargs: None, + structured_outputs: None, + priority: None, + request_id: None, + return_tokens_as_token_ids: None, + return_token_ids: None, + cache_salt: None, + kv_transfer_params: None, + vllm_xargs: None, + repetition_detection: None, + } + } +} + +impl Normalizable for ChatCompletionRequest { + /// Normalize the request by applying migrations and defaults. + fn normalize(&mut self) { + // Migrate deprecated max_tokens → max_completion_tokens + #[expect(deprecated)] + if self.max_completion_tokens.is_none() && self.max_tokens.is_some() { + self.max_completion_tokens = self.max_tokens; + self.max_tokens = None; + } + + // Apply tool_choice defaults + // If tools is None, leave tool_choice as None (don't set it) + if self.tool_choice.is_none() + && let Some(tools) = &self.tools + { + let choice_value = if tools.is_empty() { + ToolChoiceValue::None + } else { + ToolChoiceValue::Auto + }; + self.tool_choice = Some(ToolChoice::Value(choice_value)); + } + } +} + +/// Mirrors the Python vLLM `ChatCompletionResponse` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub(super) struct ChatCompletionResponse { + pub id: String, + pub object: String, + pub created: u64, + pub model: String, + pub choices: Vec, + pub usage: Option, + pub system_fingerprint: Option, + pub prompt_logprobs: Option>>>, + pub prompt_token_ids: Option>, + pub kv_transfer_params: Option, +} + +/// Mirrors the Python vLLM `ChatCompletionResponseChoice` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub(super) struct ChatCompletionChoice { + pub index: u32, + pub message: ChatCompletionMessage, + pub logprobs: Option, + pub finish_reason: Option, + pub stop_reason: Option, + pub token_ids: Option>, +} + +/// A literal type for the "assistant" role, since the API only allows that +/// specific value in responses. +#[derive(Debug, Clone, Copy, PartialEq, Eq, SerializeDisplay)] +pub(super) struct AssistantRole; + +impl fmt::Display for AssistantRole { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str("assistant") + } +} + +/// Mirrors the Python vLLM response `ChatMessage` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub(super) struct ChatCompletionMessage { + pub role: AssistantRole, + pub content: Option, + pub tool_calls: Option>, + pub reasoning: Option, +} + +/// Mirrors the Python vLLM `ChatCompletionStreamResponse` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub(super) struct ChatCompletionStreamResponse { + pub id: String, + pub object: String, + pub created: u64, + pub model: String, + pub choices: Vec, + pub usage: Option, + pub prompt_token_ids: Option>, +} + +impl ChatCompletionStreamResponse { + /// Create a stream response with the standard envelope fields pre-filled. + pub fn new(id: &str, model: &str, created: u64) -> Self { + Self { + id: id.to_string(), + object: "chat.completion.chunk".to_string(), + created, + model: model.to_string(), + choices: Vec::new(), + usage: None, + prompt_token_ids: None, + } + } +} + +/// Mirrors the Python vLLM `ChatCompletionResponseStreamChoice` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Default, Serialize)] +pub(super) struct ChatCompletionStreamChoice { + pub index: u32, + pub delta: ChatMessageDelta, + pub logprobs: Option, + pub finish_reason: Option, + pub stop_reason: Option, + pub token_ids: Option>, +} + +/// Mirrors the Python vLLM `DeltaMessage` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Default, Serialize)] +pub(super) struct ChatMessageDelta { + pub role: Option, + pub content: Option, + pub tool_calls: Option>, + pub reasoning: Option, +} + +fn default_model() -> String { + UNKNOWN_MODEL_ID.to_string() +} + +/// Schema-level validation for cross-field dependencies +fn validate_chat_cross_parameters( + req: &ChatCompletionRequest, +) -> Result<(), validator::ValidationError> { + // 1. Validate logprobs dependency + if req.top_logprobs.is_some() && !req.logprobs { + let mut e = validator::ValidationError::new("top_logprobs_requires_logprobs"); + e.message = Some("top_logprobs is only allowed when logprobs is enabled".into()); + return Err(e); + } + + // 2. Validate stream_options dependency + if req.stream_options.is_some() && !req.stream { + let mut e = validator::ValidationError::new("stream_options_requires_stream"); + e.message = Some("stream_options can only be used when stream is true".into()); + return Err(e); + } + + // 3. Validate token limits - min <= max + if let (Some(min_tokens), Some(max_completion_tokens)) = + (req.min_tokens, req.max_completion_tokens) + && min_tokens > max_completion_tokens + { + let mut e = validator::ValidationError::new("min_tokens_exceeds_max_completion_tokens"); + e.message = Some("min_tokens cannot be greater than max_completion_tokens".into()); + return Err(e); + } + + #[expect(deprecated, reason = "Local type still mirrors legacy upstream field")] + if let (Some(min_tokens), Some(max_tokens)) = (req.min_tokens, req.max_tokens) + && min_tokens > max_tokens + { + let mut e = validator::ValidationError::new("min_tokens_exceeds_max_tokens"); + e.message = Some("min_tokens cannot be greater than max_tokens".into()); + return Err(e); + } + + // 4. Validate response format JSON schema name + if let Some(ResponseFormat::JsonSchema { json_schema }) = &req.response_format + && json_schema.name.is_empty() + { + let mut e = validator::ValidationError::new("json_schema_name_empty"); + e.message = Some("JSON schema name cannot be empty".into()); + return Err(e); + } + + // 5. Validate tool_choice requires tools (except for "none") + if let Some(ref tool_choice) = req.tool_choice { + let has_tools = req.tools.as_ref().is_some_and(|t| !t.is_empty()); + + // Check if tool_choice is anything other than "none" + let is_some_choice = !matches!(tool_choice, ToolChoice::Value(ToolChoiceValue::None)); + + if is_some_choice && !has_tools { + let mut e = validator::ValidationError::new("tool_choice_requires_tools"); + e.message = Some("Invalid value for 'tool_choice': 'tool_choice' is only allowed when 'tools' are specified.".into()); + return Err(e); + } + + // Additional validation when tools are present + if let Some(tools) = req.tools.as_ref().filter(|t| !t.is_empty()) { + match tool_choice { + ToolChoice::Function { function, .. } => { + // Validate that the specified function name exists in tools + let function_exists = tools.iter().any(|tool| { + tool.tool_type == "function" && tool.function.name == function.name + }); + + if !function_exists { + let mut e = + validator::ValidationError::new("tool_choice_function_not_found"); + e.message = Some( + format!( + "Invalid value for 'tool_choice': function '{}' not found in 'tools'.", + function.name + ) + .into(), + ); + return Err(e); + } + } + ToolChoice::AllowedTools { + mode, + tools: allowed_tools, + .. + } => { + // Validate mode is "auto" or "required" + if mode != "auto" && mode != "required" { + let mut e = validator::ValidationError::new("tool_choice_invalid_mode"); + e.message = Some(format!( + "Invalid value for 'tool_choice.mode': must be 'auto' or 'required', got '{mode}'." + ).into()); + return Err(e); + } + + // Validate that all ToolReferences are Function type (Chat API only supports + // function tools) + for tool_ref in allowed_tools { + match tool_ref { + ToolReference::Function { name } => { + // Validate that the function exists in tools array + let tool_exists = tools.iter().any(|tool| { + tool.tool_type == "function" && tool.function.name == *name + }); + + if !tool_exists { + let mut e = validator::ValidationError::new( + "tool_choice_tool_not_found", + ); + e.message = Some( + format!( + "Invalid value for 'tool_choice.tools': tool '{name}' not found in 'tools'." + ) + .into(), + ); + return Err(e); + } + } + _ => { + // Chat Completion API only supports function tools in tool_choice + let mut e = validator::ValidationError::new( + "tool_choice_invalid_tool_type", + ); + e.message = Some( + format!( + "Invalid value for 'tool_choice.tools': Chat Completion API only supports function tools, got '{}'.", + tool_ref.identifier() + ) + .into(), + ); + return Err(e); + } + } + } + } + ToolChoice::Value(_) => {} + } + } + } + + Ok(()) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/chat_completions/validate.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/chat_completions/validate.rs new file mode 100644 index 0000000000000000000000000000000000000000..836d9c967945ad53b3a375dd797d94bfa1cdeb88 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/chat_completions/validate.rs @@ -0,0 +1,330 @@ +use super::types::ChatCompletionRequest; +use crate::error::{ApiError, bail_invalid_request}; +use crate::routes::openai::utils::types::{ChatMessage, Tool}; + +/// Enforce the minimal compatibility contract for the Rust OpenAI server. +pub(super) fn validate_request_compat( + request: &ChatCompletionRequest, + served_model_names: &[String], +) -> Result<(), ApiError> { + if !served_model_names.iter().any(|n| n == &request.model) { + return Err(ApiError::model_not_found(request.model.clone())); + } + + if request.stream_options.is_some() && !request.stream { + bail_invalid_request!( + param = "stream_options", + "stream_options are only supported when stream=true." + ); + } + + if request.n.unwrap_or(1) > 1 { + bail_invalid_request!(param = "n", "Only n=1 is supported."); + } + + if request.top_logprobs.is_some() && !request.logprobs { + bail_invalid_request!( + param = "top_logprobs", + "top_logprobs can only be used when logprobs=true." + ); + } + + if let Some(prompt_logprobs) = request.prompt_logprobs { + if prompt_logprobs < 0 && prompt_logprobs != -1 { + bail_invalid_request!( + param = "prompt_logprobs", + "prompt_logprobs must be a non-negative value or -1." + ); + } + + if request.stream && (prompt_logprobs > 0 || prompt_logprobs == -1) { + bail_invalid_request!( + param = "prompt_logprobs", + "prompt_logprobs are not available when stream=true." + ); + } + } + + if let Some(tools) = request.tools.as_ref() { + validate_function_tools(tools, "tools")?; + } + + for message in &request.messages { + if let ChatMessage::Developer { + tools: Some(tools), .. + } = message + { + validate_function_tools(tools, "messages[].tools")?; + } + } + + if request.use_beam_search { + bail_invalid_request!( + param = "use_beam_search", + "use_beam_search is not supported." + ); + } + + // ---- Reject parameters that are accepted for deserialization but not yet + // implemented ---- + + reject_non_default( + request.length_penalty.as_ref(), + "length_penalty", + "length_penalty is not supported.", + )?; + if !request.spaces_between_special_tokens { + bail_invalid_request!( + param = "spaces_between_special_tokens", + "spaces_between_special_tokens is not supported." + ); + } + reject_non_default( + request.truncate_prompt_tokens.as_ref(), + "truncate_prompt_tokens", + "truncate_prompt_tokens is not supported.", + )?; + reject_non_default( + request.media_io_kwargs.as_ref(), + "media_io_kwargs", + "media_io_kwargs is not supported.", + )?; + reject_non_default( + request.mm_processor_kwargs.as_ref(), + "mm_processor_kwargs", + "mm_processor_kwargs is not supported.", + )?; + + Ok(()) +} + +/// Reject one option unless it is entirely absent. +fn reject_non_default( + value: Option<&T>, + param: &'static str, + message: &str, +) -> Result<(), ApiError> { + if value.is_some() { + bail_invalid_request!(param = param, "{}", message); + } + Ok(()) +} + +fn validate_function_tools(tools: &[Tool], param: &'static str) -> Result<(), ApiError> { + for tool in tools { + if tool.tool_type != "function" { + bail_invalid_request!(param = param, "Only function tools are supported."); + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + + use serde_json::json; + use vllm_chat::ReasoningEffort; + + use super::validate_request_compat; + use crate::routes::openai::chat_completions::types::ChatCompletionRequest; + use crate::routes::openai::utils::structured_outputs::ResponseFormat; + use crate::routes::openai::utils::types::{ + ChatMessage, Function, MessageContent, StringOrArray, Tool, ToolChoice, ToolChoiceValue, + }; + + fn served(names: &[&str]) -> Vec { + names.iter().map(|s| s.to_string()).collect() + } + + fn base_request() -> ChatCompletionRequest { + ChatCompletionRequest { + model: "Qwen/Qwen1.5-0.5B-Chat".to_string(), + messages: vec![ChatMessage::User { + content: MessageContent::Text("hello".to_string()), + name: None, + }], + stream: true, + ..Default::default() + } + } + + #[test] + fn validate_request_compat_accepts_stop() { + let request = ChatCompletionRequest { + stop: Some(StringOrArray::String("stop".to_string())), + ..base_request() + }; + + validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])) + .expect("stop strings should be accepted"); + } + + #[test] + fn validate_request_compat_accepts_non_zero_penalties_and_function_tools() { + let request = ChatCompletionRequest { + frequency_penalty: Some(0.5), + presence_penalty: Some(0.25), + min_p: Some(0.2), + repetition_penalty: Some(1.1), + seed: Some(7), + ..base_request() + }; + validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])) + .expect("sampling fields should be accepted"); + + let request = ChatCompletionRequest { + tools: Some(vec![Tool { + tool_type: "function".to_string(), + function: Function { + name: "tool".to_string(), + description: None, + parameters: json!({}), + strict: None, + }, + }]), + ..base_request() + }; + validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])) + .expect("function tools should be accepted"); + + let request = ChatCompletionRequest { + messages: vec![ChatMessage::Developer { + content: MessageContent::Text("policy".to_string()), + tools: Some(vec![Tool { + tool_type: "function".to_string(), + function: Function { + name: "tool".to_string(), + description: None, + parameters: json!({}), + strict: None, + }, + }]), + name: None, + }], + ..base_request() + }; + validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])) + .expect("developer function tools should be accepted"); + } + + #[test] + fn validate_request_compat_rejects_non_function_developer_tools() { + let request = ChatCompletionRequest { + messages: vec![ChatMessage::Developer { + content: MessageContent::Text("policy".to_string()), + tools: Some(vec![Tool { + tool_type: "mcp".to_string(), + function: Function { + name: "tool".to_string(), + description: None, + parameters: json!({}), + strict: None, + }, + }]), + name: None, + }], + ..base_request() + }; + + assert!(validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])).is_err()); + } + + #[test] + fn validate_request_compat_accepts_output_logprobs() { + let request = ChatCompletionRequest { + logprobs: true, + ..base_request() + }; + validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])) + .expect("logprobs should be accepted"); + } + + #[test] + fn validate_request_compat_accepts_reasoning_effort() { + let request = ChatCompletionRequest { + reasoning_effort: Some(ReasoningEffort::Max), + chat_template_kwargs: Some(HashMap::from([( + "reasoning_effort".to_string(), + json!("low"), + )])), + ..base_request() + }; + + validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])) + .expect("reasoning_effort should be accepted"); + } + + #[test] + fn validate_request_compat_accepts_include_reasoning_false() { + let request = ChatCompletionRequest { + include_reasoning: false, + ..base_request() + }; + + validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])) + .expect("include_reasoning=false should be accepted"); + } + + #[test] + fn validate_request_compat_rejects_top_logprobs_without_logprobs() { + let request = ChatCompletionRequest { + top_logprobs: Some(0), + ..base_request() + }; + assert!(validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])).is_err()); + } + + #[test] + fn validate_request_compat_rejects_streaming_prompt_logprobs_requests() { + let request = ChatCompletionRequest { + prompt_logprobs: Some(1), + ..base_request() + }; + assert!(validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])).is_err()); + + let request = ChatCompletionRequest { + prompt_logprobs: Some(-1), + ..base_request() + }; + assert!(validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])).is_err()); + } + + #[test] + fn validate_request_compat_rejects_invalid_prompt_logprobs_value() { + let request = ChatCompletionRequest { + stream: false, + prompt_logprobs: Some(-2), + ..base_request() + }; + assert!(validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])).is_err()); + } + + #[test] + fn validate_request_compat_accepts_response_format() { + let request = ChatCompletionRequest { + response_format: Some(ResponseFormat::Text), + ..base_request() + }; + validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])) + .expect("response_format=text should be accepted"); + + let request = ChatCompletionRequest { + response_format: Some(ResponseFormat::JsonObject), + ..base_request() + }; + validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])) + .expect("response_format=json_object should be accepted"); + } + + #[test] + fn validate_request_compat_accepts_noop_tool_choice_none() { + let request = ChatCompletionRequest { + tool_choice: Some(ToolChoice::Value(ToolChoiceValue::None)), + ..base_request() + }; + + validate_request_compat(&request, &served(&["Qwen/Qwen1.5-0.5B-Chat"])) + .expect("tool_choice=none is ok"); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/completions.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/completions.rs new file mode 100644 index 0000000000000000000000000000000000000000..b7f3cfc95575ca92b1d7b4366d1416baed112d7d --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/completions.rs @@ -0,0 +1,1079 @@ +mod convert; +mod types; +mod validate; + +use std::collections::HashMap; +use std::convert::Infallible; +use std::result::Result; +use std::sync::Arc; + +use asynk_strim_attr::{TryYielder, try_stream}; +use axum::Json; +use axum::extract::State; +use axum::http::HeaderMap; +use axum::response::sse::{Event, Sse}; +use axum::response::{IntoResponse, Response}; +use futures::{Stream, StreamExt as _, pin_mut}; +use serde_json::Value; +use thiserror_ext::AsReport as _; +use tracing::{debug, error, info, trace}; +use tracing_futures::Instrument as _; +use vllm_engine_core_client::protocol::output::StopReason; +use vllm_text::{ + DecodedPromptLogprobs, DecodedTextEvent, FinishReason, TextOutputStream, + TextOutputStreamExt as _, +}; + +use self::convert::{ResponseOptions, prepare_completion_request}; +use super::utils::logprobs::{ + collected_logprobs_to_openai, decoded_logprobs_to_openai, decoded_prompt_logprobs_to_maps, + decoded_prompt_logprobs_to_openai, text_len, +}; +use super::utils::types::Usage; +use crate::config::ApiServerOptions; +use crate::error::{ApiError, bail_server_error, server_error, text_submit_error}; +use crate::routes::openai::completions::types::{ + CompletionChoice, CompletionRequest, CompletionResponse, CompletionSseChunk, + CompletionStreamChoice, CompletionStreamResponse, +}; +use crate::routes::openai::utils::types::LogProbs; +use crate::routes::openai::utils::usage::ContinuousUsage; +use crate::routes::openai::utils::validated_json::ValidatedJson; +use crate::state::AppState; +use crate::utils::{resolve_request_context, unix_timestamp}; + +/// Validate one completions request and proxy it into the shared `vllm-text` +/// stack. +pub async fn completions( + State(state): State>, + headers: HeaderMap, + ValidatedJson(body): ValidatedJson, +) -> Response { + let stream = body.stream; + let request_context = resolve_request_context(&headers, body.request_id.as_deref()); + let lora_resolution = state.resolve_model_with_loras(Some(&body.model)).await; + + let tokenizer = state.chat.text().tokenizer(); + let prepared = match prepare_completion_request( + body, + &lora_resolution, + request_context, + tokenizer.as_ref(), + ) { + Ok(prepared) => prepared, + Err(error) => return error.into_response(), + }; + let request_span = tracing::info_span!( + "completions", + request_id = %prepared.request_id, + engine_request_id = tracing::field::Empty, + ); + + let created = unix_timestamp(); + let api_server_options = state.api_server_options; + let text_stream = match state + .chat + .text() + .generate(prepared.text_request) + .instrument(request_span.clone()) + .await + { + Ok(stream) => stream, + Err(error) => { + return text_submit_error("failed to submit completion request", error).into_response(); + } + }; + + if stream { + let chunk_stream = completion_chunk_stream( + text_stream, + prepared.request_id, + prepared.response_model, + created, + api_server_options, + prepared.options, + ); + let sse_stream = completion_sse_stream(chunk_stream).instrument(request_span); + + Sse::new(sse_stream).into_response() + } else { + let response = match collect_completion( + text_stream, + prepared.request_id, + prepared.response_model, + created, + api_server_options, + prepared.options, + ) + .instrument(request_span.clone()) + .await + { + Ok(response) => response, + Err(error) => return error.into_response(), + }; + + Json(response).into_response() + } +} + +async fn collect_completion( + stream: impl TextOutputStream, + request_id: String, + response_model: String, + created: u64, + ApiServerOptions { + enable_log_requests, + enable_prompt_tokens_details, + .. + }: ApiServerOptions, + ResponseOptions { + // Ignored: non-streaming responses always include usage. + include_usage: _, + // Ignored: non-streaming responses are collected before usage is attached. + include_continuous_usage: _, + prompt_only, + echo, + requested_logprobs, + include_prompt_logprobs, + return_token_ids, + return_tokens_as_token_ids, + }: ResponseOptions, +) -> Result { + let collected = stream + .collect_output() + .await + .map_err(|error| server_error!("completion stream failed: {}", error.to_report_string()))?; + let finish_reason = collected.finish_reason.clone(); + let stop_reason = finish_reason.as_stop_reason().map(stop_reason_to_json); + + let prompt_char_count = echo.as_ref().map(|prompt| text_len(prompt)).unwrap_or_default(); + let logprobs = if requested_logprobs.is_some() && prompt_only { + let prompt = echo.as_deref().ok_or_else(|| { + server_error!("prompt-only completion response missing echoed prompt") + })?; + Some(prompt_only_logprobs_to_openai( + collected.prompt_logprobs.as_ref(), + prompt, + collected.prompt_token_ids.as_ref(), + return_tokens_as_token_ids, + )?) + } else if requested_logprobs.is_some() { + Some(collected_logprobs_to_openai( + &collected, + echo.is_some(), + prompt_char_count, + return_tokens_as_token_ids, + )?) + } else { + None + }; + let prompt_logprobs = if include_prompt_logprobs { + Some(prompt_logprobs_to_maps( + collected.prompt_logprobs.as_ref(), + collected.prompt_token_ids.as_ref(), + return_tokens_as_token_ids, + )?) + } else { + None + }; + let text = match &echo { + None => collected.text, + Some(prompt) if prompt_only => prompt.clone(), + Some(prompt) => format!("{prompt}{}", collected.text), + }; + let finish_reason = completion_finish_reason_to_openai(&finish_reason)?.to_string(); + let usage = Usage::from_token_usage(collected.usage, enable_prompt_tokens_details); + + if enable_log_requests { + info!( + model = %response_model, + prompt_tokens = usage.prompt_tokens, + output_tokens = usage.completion_tokens.unwrap_or(0), + %finish_reason, + "completion finished" + ); + } + + Ok(CompletionResponse { + id: request_id, + object: "text_completion".to_string(), + created, + model: response_model, + choices: vec![CompletionChoice { + index: 0, + text, + logprobs, + finish_reason: Some(finish_reason), + stop_reason, + prompt_logprobs, + token_ids: return_token_ids.then(|| collected.token_ids.clone()), + prompt_token_ids: return_token_ids.then(|| collected.prompt_token_ids.to_vec()), + }], + usage: Some(usage), + system_fingerprint: None, + kv_transfer_params: collected.kv_transfer_params, + }) +} + +/// Convert one internal decoded-text stream into OpenAI completions chunks. +#[try_stream] +async fn completion_chunk_stream( + stream: impl TextOutputStream, + request_id: String, + response_model: String, + created: u64, + ApiServerOptions { + enable_log_requests, + enable_prompt_tokens_details, + .. + }: ApiServerOptions, + ResponseOptions { + include_usage, + include_continuous_usage, + prompt_only, + echo, + requested_logprobs, + // Ignored: streaming prompt logprobs are rejected for Python parity. + include_prompt_logprobs: _, + return_token_ids, + return_tokens_as_token_ids, + }: ResponseOptions, + mut y: TryYielder, +) -> Result<(), ApiError> { + pin_mut!(stream); + let mut visible_text_len = 0_u32; + let mut first_chunk = true; + let mut continuous_usage = ContinuousUsage::default(); + + /// Yield a chunk with optional continuous usage attached. + macro_rules! yield_chunk { + ($chunk:expr) => {{ + let mut chunk = $chunk; + if include_continuous_usage { + chunk.usage = Some(continuous_usage.to_usage()); + } + y.yield_ok(CompletionSseChunk::Chunk(chunk)).await; + }}; + } + + while let Some(next) = stream.next().await { + match next { + Ok(DecodedTextEvent::Start { + prompt_token_ids, + prompt_logprobs, + }) => { + debug!("completion stream started"); + continuous_usage.set_prompt_tokens(prompt_token_ids.len()); + if let Some(prompt) = echo.as_ref() { + visible_text_len = text_len(prompt); + let logprobs = if prompt_only && requested_logprobs.is_some() { + Some(prompt_only_logprobs_to_openai( + prompt_logprobs.as_ref(), + prompt, + prompt_token_ids.as_ref(), + return_tokens_as_token_ids, + )?) + } else { + None + }; + let mut chunk = delta_chunk( + &request_id, + &response_model, + created, + prompt.clone(), + logprobs, + ); + if return_token_ids && first_chunk { + if let Some(choice) = chunk.choices.first_mut() { + choice.prompt_token_ids = Some(prompt_token_ids.to_vec()); + } + first_chunk = false; + } + yield_chunk!(chunk); + } else if return_token_ids { + // Emit a chunk with prompt_token_ids in the first streaming response + let mut chunk = + delta_chunk(&request_id, &response_model, created, String::new(), None); + if let Some(choice) = chunk.choices.first_mut() { + choice.prompt_token_ids = Some(prompt_token_ids.to_vec()); + } + first_chunk = false; + yield_chunk!(chunk); + } + } + Ok(DecodedTextEvent::TextDelta { + delta, + token_ids, + logprobs, + finished, + }) => { + // Prompt-only streaming already emitted the echoed prompt in the Start chunk. + // The one generated token is only used to drive the engine to a finished event, + // so hide its delta and forward only the terminal finish/usage metadata. + if prompt_only { + if let Some(finished) = finished { + if enable_log_requests { + info!( + stream = true, + model = %response_model, + prompt_tokens = finished.usage.prompt_token_count, + output_tokens = finished.usage.output_token_count, + finish_reason = finished.finish_reason.as_str(), + "completion finished" + ); + } + continuous_usage.set_final_counts( + finished.usage.prompt_token_count, + finished.usage.output_token_count, + ); + let final_chunk = final_chunk( + &request_id, + &response_model, + created, + finished.finish_reason, + )?; + yield_chunk!(final_chunk); + + if include_usage { + y.yield_ok(CompletionSseChunk::Usage(usage_chunk( + &request_id, + &response_model, + created, + Usage::from_token_usage( + finished.usage, + enable_prompt_tokens_details, + ), + ))) + .await; + } + } + continue; + } + let delta_text_len = text_len(&delta); + let logprobs = if requested_logprobs.is_some() { + let decoded_logprobs = logprobs.as_ref().ok_or_else(|| { + server_error!( + "completion stream requested logprobs but generation returned none" + ) + })?; + Some(decoded_logprobs_to_openai( + decoded_logprobs, + visible_text_len, + return_tokens_as_token_ids, + )?) + } else { + None + }; + let mut chunk = delta_chunk(&request_id, &response_model, created, delta, logprobs); + let delta_token_count = token_ids.len(); + continuous_usage.add_output_tokens(delta_token_count); + if return_token_ids && let Some(choice) = chunk.choices.first_mut() { + choice.token_ids = Some(token_ids); + } + yield_chunk!(chunk); + visible_text_len = visible_text_len.saturating_add(delta_text_len); + + if let Some(finished) = finished { + if enable_log_requests { + info!( + stream = true, + model = %response_model, + prompt_tokens = finished.usage.prompt_token_count, + output_tokens = finished.usage.output_token_count, + finish_reason = finished.finish_reason.as_str(), + "completion finished" + ); + } + continuous_usage.set_final_counts( + finished.usage.prompt_token_count, + finished.usage.output_token_count, + ); + let final_chunk = final_chunk( + &request_id, + &response_model, + created, + finished.finish_reason, + )?; + yield_chunk!(final_chunk); + + if include_usage { + y.yield_ok(CompletionSseChunk::Usage(usage_chunk( + &request_id, + &response_model, + created, + Usage::from_token_usage(finished.usage, enable_prompt_tokens_details), + ))) + .await; + } + } + } + Err(error) => { + error!( + error = %error.as_report(), + "completion stream failed" + ); + bail_server_error!("{}", error.to_report_string()); + } + } + } + Ok(()) +} + +fn delta_chunk( + request_id: &str, + response_model: &str, + created: u64, + text: String, + logprobs: Option, +) -> CompletionStreamResponse { + let mut chunk = CompletionStreamResponse::new(request_id, response_model, created); + chunk.choices.push(CompletionStreamChoice { + text, + logprobs, + ..Default::default() + }); + chunk +} + +fn final_chunk( + request_id: &str, + response_model: &str, + created: u64, + finish_reason: FinishReason, +) -> Result { + let stop_reason = finish_reason.as_stop_reason().map(stop_reason_to_json); + let finish_reason = completion_finish_reason_to_openai(&finish_reason)?; + + let mut chunk = CompletionStreamResponse::new(request_id, response_model, created); + chunk.choices.push(CompletionStreamChoice { + finish_reason: Some(finish_reason.to_string()), + stop_reason, + ..Default::default() + }); + Ok(chunk) +} + +fn completion_finish_reason_to_openai( + finish_reason: &FinishReason, +) -> Result<&'static str, ApiError> { + match finish_reason { + FinishReason::Stop(_) => Ok("stop"), + FinishReason::Repetition(_) => Ok("repetition"), + FinishReason::Length => Ok("length"), + FinishReason::Abort => Ok("abort"), + FinishReason::Error => { + bail_server_error!("Internal server error"); + } + } +} + +fn stop_reason_to_json(stop_reason: &StopReason) -> Value { + serde_json::to_value(stop_reason).expect("StopReason must serialize to JSON") +} + +fn prompt_only_logprobs_to_openai( + prompt_logprobs: Option<&DecodedPromptLogprobs>, + prompt: &str, + prompt_token_ids: &[u32], + return_tokens_as_token_ids: bool, +) -> Result { + if let Some(prompt_logprobs) = prompt_logprobs { + return decoded_prompt_logprobs_to_openai(prompt_logprobs, 0, return_tokens_as_token_ids); + } + + if let [token_id] = prompt_token_ids { + let token = if return_tokens_as_token_ids { + format!("token_id:{token_id}") + } else { + prompt.to_string() + }; + + return Ok(LogProbs { + tokens: vec![token], + token_logprobs: vec![None], + top_logprobs: vec![None], + text_offset: vec![0], + }); + } + + Err(server_error!( + "prompt-only completion requested logprobs but generation returned none" + )) +} + +fn prompt_logprobs_to_maps( + prompt_logprobs: Option<&DecodedPromptLogprobs>, + prompt_token_ids: &[u32], + return_tokens_as_token_ids: bool, +) -> Result>>, ApiError> { + if let Some(prompt_logprobs) = prompt_logprobs { + return Ok(decoded_prompt_logprobs_to_maps( + prompt_logprobs, + return_tokens_as_token_ids, + )); + } + + if let [_token_id] = prompt_token_ids { + return Ok(vec![None]); + } + + Err(server_error!( + "completion response requested prompt_logprobs but generation returned none" + )) +} + +fn usage_chunk( + request_id: &str, + response_model: &str, + created: u64, + usage: Usage, +) -> CompletionStreamResponse { + let mut chunk = CompletionStreamResponse::new(request_id, response_model, created); + chunk.usage = Some(usage); + chunk +} + +/// Convert one chunk stream into OpenAI-style SSE events. +/// +/// OpenAI-style streaming errors are encoded as ordinary `data: {"error": ...}` +/// events followed by `data: [DONE]`, so the transport stream itself stays +/// infallible even when generation fails after the HTTP response has started. +#[try_stream] +async fn completion_sse_stream( + stream: impl Stream>, + mut y: TryYielder, +) -> Result<(), Infallible> { + pin_mut!(stream); + + while let Some(next) = stream.next().await { + match next { + Ok(chunk) => y.yield_ok(to_sse_event(&chunk)).await, + Err(error) => { + y.yield_ok(to_error_sse_event(&error)).await; + break; + } + } + } + + y.yield_ok(done_sse_event()).await; + Ok(()) +} + +/// Serialize one OpenAI chunk payload into one SSE `data:` event. +fn to_sse_event(chunk: &CompletionSseChunk) -> Event { + let payload = serde_json::to_string(chunk).expect("completion chunk must serialize to JSON"); + trace!(payload, "completion emitting chunk"); + Event::default().data(payload) +} + +/// Serialize one OpenAI error payload into one SSE `data:` event. +fn to_error_sse_event(error: &ApiError) -> Event { + let payload = serde_json::to_string(&error.to_error_response()) + .expect("ErrorResponse must serialize to JSON"); + trace!(payload, "completion emitting error"); + Event::default().data(payload) +} + +/// Build the terminal OpenAI SSE sentinel event. +fn done_sse_event() -> Event { + trace!("completion emitting done"); + Event::default().data("[DONE]") +} + +#[cfg(test)] +mod tests { + use futures::{StreamExt as _, stream}; + use itertools::Itertools as _; + use vllm_engine_core_client::protocol::output::StopReason; + use vllm_text::{ + DecodedLogprobs, DecodedPositionLogprobs, DecodedPromptLogprobs, DecodedTextEvent, + DecodedTokenLogprob, FinishReason, Finished, + }; + + use super::{ + ApiServerOptions, CompletionSseChunk, ResponseOptions, completion_chunk_stream, final_chunk, + }; + + #[test] + fn final_chunk_maps_stop_finish_reason() { + let chunk = final_chunk("cmpl-1", "model", 1, FinishReason::stop_eos()) + .expect("finish reason valid"); + assert_eq!(chunk.choices[0].finish_reason.as_deref(), Some("stop")); + assert_eq!(chunk.choices[0].text, ""); + } + + #[test] + fn final_chunk_maps_length_finish_reason() { + let chunk = + final_chunk("cmpl-1", "model", 1, FinishReason::Length).expect("finish reason valid"); + assert_eq!(chunk.choices[0].finish_reason.as_deref(), Some("length")); + } + + #[test] + fn final_chunk_maps_abort_finish_reason() { + let chunk = + final_chunk("cmpl-1", "model", 1, FinishReason::Abort).expect("finish reason valid"); + assert_eq!(chunk.choices[0].finish_reason.as_deref(), Some("abort")); + } + + #[test] + fn final_chunk_rejects_error_finish_reason() { + assert!(final_chunk("cmpl-1", "model", 1, FinishReason::Error).is_err()); + } + + #[tokio::test] + async fn completion_chunk_stream_maps_streaming_logprobs() { + let stream = stream::iter(vec![ + Ok(DecodedTextEvent::Start { + prompt_token_ids: vec![1, 2, 3, 4, 5].into(), + prompt_logprobs: None, + }), + Ok(DecodedTextEvent::TextDelta { + delta: "h".to_string(), + token_ids: vec![b'h' as u32], + logprobs: Some(DecodedLogprobs { + positions: vec![DecodedPositionLogprobs { + entries: vec![ + DecodedTokenLogprob { + token_id: 0, + token: "h".to_string(), + logprob: -0.1, + rank: 1, + }, + DecodedTokenLogprob { + token_id: 0, + token: "H".to_string(), + logprob: -0.2, + rank: 1, + }, + ], + }], + }), + finished: None, + }), + Ok(DecodedTextEvent::TextDelta { + delta: String::new(), + token_ids: vec![b'!' as u32], + logprobs: Some(DecodedLogprobs { + positions: vec![DecodedPositionLogprobs { + entries: vec![ + DecodedTokenLogprob { + token_id: 0, + token: "!".to_string(), + logprob: -0.3, + rank: 1, + }, + DecodedTokenLogprob { + token_id: 0, + token: "?".to_string(), + logprob: -0.4, + rank: 1, + }, + ], + }], + }), + finished: Some(Finished { + usage: vllm_llm::TokenUsage { + prompt_token_count: 5, + output_token_count: 2, + cached_token_count: 3, + }, + finish_reason: FinishReason::Repetition(Some(StopReason::Text( + "repetition_detected".to_string(), + ))), + kv_transfer_params: None, + }), + }), + ]); + + let chunks = completion_chunk_stream( + stream, + "cmpl-1".to_string(), + "model".to_string(), + 1, + ApiServerOptions { + enable_prompt_tokens_details: true, + ..Default::default() + }, + ResponseOptions { + include_usage: true, + requested_logprobs: Some(1), + ..Default::default() + }, + ) + .collect::>() + .await; + + let chunks: Vec<_> = chunks.into_iter().try_collect().expect("stream should succeed"); + + match &chunks[0] { + CompletionSseChunk::Chunk(chunk) => { + assert_eq!(chunk.choices[0].text, "h"); + assert_eq!( + chunk.choices[0].logprobs.as_ref().expect("logprobs").tokens, + vec!["h".to_string()] + ); + assert_eq!( + chunk.choices[0].logprobs.as_ref().expect("logprobs").text_offset, + vec![0] + ); + } + CompletionSseChunk::Usage(_) => panic!("expected regular chunk"), + } + + match &chunks[1] { + CompletionSseChunk::Chunk(chunk) => { + assert_eq!(chunk.choices[0].text, ""); + assert_eq!( + chunk.choices[0].logprobs.as_ref().expect("logprobs").tokens, + vec!["!".to_string()] + ); + assert_eq!( + chunk.choices[0].logprobs.as_ref().expect("logprobs").text_offset, + vec![1] + ); + } + CompletionSseChunk::Usage(_) => panic!("expected regular chunk"), + } + + match &chunks[2] { + CompletionSseChunk::Chunk(chunk) => { + assert_eq!( + chunk.choices[0].finish_reason.as_deref(), + Some("repetition") + ); + assert_eq!( + chunk.choices[0].stop_reason, + Some(serde_json::json!("repetition_detected")) + ); + } + CompletionSseChunk::Usage(_) => panic!("expected regular chunk"), + } + + match &chunks[3] { + CompletionSseChunk::Usage(chunk) => { + assert_eq!( + chunk + .usage + .as_ref() + .expect("usage") + .prompt_tokens_details + .as_ref() + .map(|details| details.cached_tokens), + Some(3) + ); + } + CompletionSseChunk::Chunk(_) => panic!("expected usage chunk"), + } + } + + #[tokio::test] + async fn collect_completion_hides_internal_prompt_only_token() { + let stream = stream::iter(vec![ + Ok(DecodedTextEvent::Start { + prompt_token_ids: vec![1, 2].into(), + prompt_logprobs: None, + }), + Ok(DecodedTextEvent::TextDelta { + delta: " leaked".to_string(), + token_ids: vec![3], + logprobs: None, + finished: Some(Finished { + usage: vllm_llm::TokenUsage { + prompt_token_count: 2, + output_token_count: 1, + cached_token_count: 0, + }, + finish_reason: FinishReason::Length, + kv_transfer_params: None, + }), + }), + ]); + + let response = super::collect_completion( + stream, + "cmpl-1".to_string(), + "model".to_string(), + 1, + ApiServerOptions::default(), + ResponseOptions { + prompt_only: true, + echo: Some("hello".to_string()), + return_token_ids: true, + ..Default::default() + }, + ) + .await + .expect("collect completion"); + + assert_eq!(response.choices[0].text, "hello"); + assert_eq!(response.choices[0].token_ids.as_deref(), Some(&[3][..])); + assert_eq!( + response.choices[0].prompt_token_ids.as_deref(), + Some(&[1, 2][..]) + ); + let usage = response.usage.expect("usage"); + assert_eq!(usage.prompt_tokens, 2); + assert_eq!(usage.completion_tokens, Some(1)); + assert_eq!(usage.total_tokens, 3); + } + + #[tokio::test] + async fn collect_completion_maps_prompt_logprobs_for_single_token_prompt() { + let stream = stream::iter(vec![ + Ok(DecodedTextEvent::Start { + prompt_token_ids: vec![9707].into(), + prompt_logprobs: None, + }), + Ok(DecodedTextEvent::TextDelta { + delta: " leaked".to_string(), + token_ids: vec![3], + logprobs: None, + finished: Some(Finished { + usage: vllm_llm::TokenUsage { + prompt_token_count: 1, + output_token_count: 1, + cached_token_count: 0, + }, + finish_reason: FinishReason::Length, + kv_transfer_params: None, + }), + }), + ]); + + let response = super::collect_completion( + stream, + "cmpl-1".to_string(), + "model".to_string(), + 1, + ApiServerOptions::default(), + ResponseOptions { + prompt_only: true, + echo: Some("Hello".to_string()), + requested_logprobs: Some(1), + include_prompt_logprobs: true, + ..Default::default() + }, + ) + .await + .expect("collect completion"); + + let choice = &response.choices[0]; + assert_eq!(choice.text, "Hello"); + assert_eq!(choice.prompt_logprobs, Some(vec![None])); + let logprobs = choice.logprobs.as_ref().expect("logprobs"); + assert_eq!(logprobs.tokens, vec!["Hello".to_string()]); + assert_eq!(logprobs.token_logprobs, vec![None]); + assert_eq!(logprobs.top_logprobs, vec![None]); + assert_eq!(logprobs.text_offset, vec![0]); + let usage = response.usage.expect("usage"); + assert_eq!(usage.prompt_tokens, 1); + assert_eq!(usage.completion_tokens, Some(1)); + assert_eq!(usage.total_tokens, 2); + } + + #[tokio::test] + async fn completion_chunk_stream_hides_internal_prompt_only_token() { + let stream = stream::iter(vec![ + Ok(DecodedTextEvent::Start { + prompt_token_ids: vec![1, 2].into(), + prompt_logprobs: None, + }), + Ok(DecodedTextEvent::TextDelta { + delta: " leaked".to_string(), + token_ids: vec![3], + logprobs: None, + finished: Some(Finished { + usage: vllm_llm::TokenUsage { + prompt_token_count: 2, + output_token_count: 1, + cached_token_count: 0, + }, + finish_reason: FinishReason::Length, + kv_transfer_params: None, + }), + }), + ]); + + let chunks = completion_chunk_stream( + stream, + "cmpl-1".to_string(), + "model".to_string(), + 1, + ApiServerOptions::default(), + ResponseOptions { + include_usage: true, + prompt_only: true, + echo: Some("hello".to_string()), + return_token_ids: true, + ..Default::default() + }, + ) + .collect::>() + .await; + + let chunks: Vec<_> = chunks.into_iter().try_collect().expect("stream should succeed"); + assert_eq!(chunks.len(), 3); + + match &chunks[0] { + CompletionSseChunk::Chunk(chunk) => { + assert_eq!(chunk.choices[0].text, "hello"); + assert_eq!( + chunk.choices[0].prompt_token_ids.as_deref(), + Some(&[1, 2][..]) + ); + } + CompletionSseChunk::Usage(_) => panic!("expected prompt chunk"), + } + match &chunks[1] { + CompletionSseChunk::Chunk(chunk) => { + assert_eq!(chunk.choices[0].text, ""); + assert_eq!(chunk.choices[0].finish_reason.as_deref(), Some("length")); + } + CompletionSseChunk::Usage(_) => panic!("expected final chunk"), + } + match &chunks[2] { + CompletionSseChunk::Usage(chunk) => { + let usage = chunk.usage.as_ref().expect("usage"); + assert_eq!(usage.prompt_tokens, 2); + assert_eq!(usage.completion_tokens, Some(1)); + assert_eq!(usage.total_tokens, 3); + } + CompletionSseChunk::Chunk(_) => panic!("expected usage chunk"), + } + } + + #[tokio::test] + async fn completion_chunk_stream_maps_prompt_logprobs_for_single_token_prompt() { + let stream = stream::iter(vec![ + Ok(DecodedTextEvent::Start { + prompt_token_ids: vec![9707].into(), + prompt_logprobs: None, + }), + Ok(DecodedTextEvent::TextDelta { + delta: " leaked".to_string(), + token_ids: vec![3], + logprobs: None, + finished: Some(Finished { + usage: vllm_llm::TokenUsage { + prompt_token_count: 1, + output_token_count: 1, + cached_token_count: 0, + }, + finish_reason: FinishReason::Length, + kv_transfer_params: None, + }), + }), + ]); + + let chunks = completion_chunk_stream( + stream, + "cmpl-1".to_string(), + "model".to_string(), + 1, + ApiServerOptions::default(), + ResponseOptions { + prompt_only: true, + echo: Some("Hello".to_string()), + requested_logprobs: Some(1), + ..Default::default() + }, + ) + .collect::>() + .await; + + let chunks: Vec<_> = chunks.into_iter().try_collect().expect("stream should succeed"); + assert_eq!(chunks.len(), 2); + + match &chunks[0] { + CompletionSseChunk::Chunk(chunk) => { + assert_eq!(chunk.choices[0].text, "Hello"); + let logprobs = chunk.choices[0].logprobs.as_ref().expect("logprobs"); + assert_eq!(logprobs.tokens, vec!["Hello".to_string()]); + assert_eq!(logprobs.token_logprobs, vec![None]); + assert_eq!(logprobs.top_logprobs, vec![None]); + assert_eq!(logprobs.text_offset, vec![0]); + } + CompletionSseChunk::Usage(_) => panic!("expected prompt chunk"), + } + match &chunks[1] { + CompletionSseChunk::Chunk(chunk) => { + assert_eq!(chunk.choices[0].text, ""); + assert_eq!(chunk.choices[0].finish_reason.as_deref(), Some("length")); + } + CompletionSseChunk::Usage(_) => panic!("expected final chunk"), + } + } + + #[tokio::test] + async fn completion_chunk_stream_maps_prompt_only_logprobs() { + let stream = stream::iter(vec![ + Ok(DecodedTextEvent::Start { + prompt_token_ids: vec![1, 2].into(), + prompt_logprobs: Some(DecodedPromptLogprobs { + first_token_id: 1, + first_token: "he".to_string(), + scored_positions: vec![DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 2, + token: "llo".to_string(), + logprob: -0.2, + rank: 1, + }], + }], + }), + }), + Ok(DecodedTextEvent::TextDelta { + delta: " leaked".to_string(), + token_ids: vec![3], + logprobs: None, + finished: Some(Finished { + usage: vllm_llm::TokenUsage { + prompt_token_count: 2, + output_token_count: 1, + cached_token_count: 0, + }, + finish_reason: FinishReason::Length, + kv_transfer_params: None, + }), + }), + ]); + + let chunks = completion_chunk_stream( + stream, + "cmpl-1".to_string(), + "model".to_string(), + 1, + ApiServerOptions::default(), + ResponseOptions { + prompt_only: true, + echo: Some("hello".to_string()), + requested_logprobs: Some(1), + ..Default::default() + }, + ) + .collect::>() + .await; + + let chunks: Vec<_> = chunks.into_iter().try_collect().expect("stream should succeed"); + assert_eq!(chunks.len(), 2); + + match &chunks[0] { + CompletionSseChunk::Chunk(chunk) => { + assert_eq!(chunk.choices[0].text, "hello"); + let logprobs = chunk.choices[0].logprobs.as_ref().expect("logprobs"); + assert_eq!(logprobs.tokens, vec!["he".to_string(), "llo".to_string()]); + assert_eq!(logprobs.token_logprobs, vec![None, Some(-0.2)]); + assert_eq!(logprobs.text_offset, vec![0, 2]); + } + CompletionSseChunk::Usage(_) => panic!("expected prompt chunk"), + } + match &chunks[1] { + CompletionSseChunk::Chunk(chunk) => { + assert_eq!(chunk.choices[0].text, ""); + assert_eq!(chunk.choices[0].finish_reason.as_deref(), Some("length")); + } + CompletionSseChunk::Usage(_) => panic!("expected final chunk"), + } + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/completions/convert.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/completions/convert.rs new file mode 100644 index 0000000000000000000000000000000000000000..d7a523b7553ba61bc5436732e0d21b78e4f42d98 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/completions/convert.rs @@ -0,0 +1,601 @@ +use thiserror_ext::AsReport as _; +use vllm_text::tokenizer::Tokenizer; +use vllm_text::{Prompt, SamplingParams, TextDecodeOptions, TextRequest}; + +use super::types::CompletionRequest; +use crate::error::ApiError; +use crate::lora::LoraModelResolution; +use crate::routes::openai::completions::validate; +use crate::routes::openai::utils::structured_outputs::convert_from_response_format_value; +use crate::utils::{ResolvedRequestContext, convert_logit_bias, merge_kv_transfer_params}; + +/// Lowered completion request plus the public response metadata carried by +/// every SSE chunk. +#[derive(Debug, Clone, PartialEq)] +pub(super) struct PreparedRequest { + /// Stable OpenAI-style request ID, reused as the external text request ID. + pub request_id: String, + /// Public model ID echoed back to the client. + pub response_model: String, + /// Public response rendering options for route-layer helpers. + pub options: ResponseOptions, + /// Lowered text request for the shared `vllm-text` facade. + pub text_request: TextRequest, +} + +#[derive(Debug, Clone, Default, PartialEq)] +pub(super) struct ResponseOptions { + /// Whether the caller asked for the final streamed usage chunk. + pub include_usage: bool, + /// Whether every streamed chunk should carry cumulative usage. + pub include_continuous_usage: bool, + /// Whether the caller requested prompt-only echo via `max_tokens=0`. + pub prompt_only: bool, + /// Prompt text that should be echoed back northbound when `echo=true`. + pub echo: Option, + /// Whether the caller requested output logprobs on completion choices. + pub requested_logprobs: Option, + /// Whether the caller requested choice-level prompt logprobs. + pub include_prompt_logprobs: bool, + /// Whether to include token IDs alongside generated text. + pub return_token_ids: bool, + /// Whether to format logprob tokens as `token_id:{id}`. + pub return_tokens_as_token_ids: bool, +} + +/// Validate and lower one OpenAI completions request into the internal +/// text-generation format. +/// +/// `lora_resolution.model_names` must be non-empty; the first entry is used as +/// the base `model` field in responses when no LoRA adapter is selected. +pub(super) fn prepare_completion_request( + request: CompletionRequest, + lora_resolution: &LoraModelResolution, + ctx: ResolvedRequestContext, + tokenizer: &dyn Tokenizer, +) -> Result { + validate::validate_request_compat(&request, &lora_resolution.model_names)?; + + let request_id = format!("cmpl-{}", ctx.request_id); + let response_model = lora_resolution + .lora_request + .as_ref() + .map(|request| request.lora_name.clone()) + .unwrap_or_else(|| lora_resolution.model_names.first().cloned().unwrap_or_default()); + + let logprobs = match request.logprobs { + Some(logprobs) => Some(i32::try_from(logprobs).map_err(|_| { + ApiError::invalid_request( + "`logprobs` must fit within a signed 32-bit integer.".to_string(), + Some("logprobs"), + ) + })?), + None => None, + }; + let prompt_only = request.echo && request.max_tokens == Some(0); + let prompt_logprobs = + request.prompt_logprobs.or(if request.echo && (!request.stream || prompt_only) { + logprobs + } else { + None + }); + let include_usage = (request.stream_options.as_ref()) + .and_then(|options| options.include_usage) + .unwrap_or(false); + let include_continuous_usage = include_usage + && request + .stream_options + .as_ref() + .and_then(|options| options.continuous_usage_stats) + .unwrap_or(false); + let include_prompt_logprobs = prompt_logprobs.is_some(); + let max_tokens = if prompt_only { + Some(1) + } else { + request.max_tokens + }; + let echo = completion_echo_text(&request, tokenizer)?; + + let structured_outputs = + convert_from_response_format_value(&request.response_format, &request.structured_outputs)?; + + let text_request = TextRequest { + request_id: request_id.clone(), + prompt: request.prompt, + mm_features: None, + sampling_params: SamplingParams { + temperature: request.temperature, + top_p: request.top_p, + top_k: request.top_k, + seed: request.seed, + max_tokens, + min_tokens: request.min_tokens, + thinking_token_budget: request.thinking_token_budget, + logprobs, + prompt_logprobs, + min_p: request.min_p, + frequency_penalty: request.frequency_penalty, + presence_penalty: request.presence_penalty, + repetition_penalty: request.repetition_penalty, + repetition_detection: request.repetition_detection, + stop_token_ids: request.stop_token_ids, + ignore_eos: request.ignore_eos, + logit_bias: convert_logit_bias(request.logit_bias)?, + allowed_token_ids: request.allowed_token_ids, + bad_words: None, + logprob_token_ids: None, + structured_outputs, + skip_reading_prefix_cache: None, + vllm_xargs: merge_kv_transfer_params( + request.vllm_xargs, + request.kv_transfer_params.as_ref(), + ), + }, + decode_options: TextDecodeOptions { + skip_special_tokens: request.skip_special_tokens, + include_stop_str_in_output: request.include_stop_str_in_output, + stop_strings: request.stop.map(|stop| stop.into_vec()), + min_tokens: request.min_tokens.unwrap_or(0), + }, + intermediate: request.stream, + priority: request.priority.unwrap_or(0), + cache_salt: request.cache_salt, + add_special_tokens: request.add_special_tokens, + data_parallel_rank: ctx.data_parallel_rank, + reasoning_parser_kwargs: None, + lora_request: lora_resolution.lora_request.clone(), + }; + + Ok(PreparedRequest { + request_id, + response_model, + options: ResponseOptions { + include_usage, + include_continuous_usage, + prompt_only, + echo, + requested_logprobs: request.logprobs, + include_prompt_logprobs, + return_token_ids: request.return_token_ids.unwrap_or(false), + return_tokens_as_token_ids: request.return_tokens_as_token_ids.unwrap_or(false), + }, + text_request, + }) +} + +fn completion_echo_text( + request: &CompletionRequest, + tokenizer: &dyn Tokenizer, +) -> Result, ApiError> { + if !request.echo { + return Ok(None); + } + + match &request.prompt { + Prompt::Text(prompt) => Ok(Some(prompt.clone())), + Prompt::TokenIds(token_ids) if request.return_token_ids.unwrap_or(false) => { + Ok(Some(String::new())) + } + Prompt::TokenIds(token_ids) => { + tokenizer.decode(token_ids, false).map(Some).map_err(|error| { + ApiError::invalid_request( + format!( + "Failed to decode token-ID prompt for echo: {}", + error.to_report_string() + ), + Some("prompt"), + ) + }) + } + } +} + +#[cfg(test)] +mod tests { + use axum::http::HeaderMap; + use serde_json::json; + use vllm_text::Prompt; + use vllm_tokenizer::test_utils::TestTokenizer; + + use super::prepare_completion_request; + use crate::lora::LoraModelResolution; + use crate::routes::openai::completions::types::CompletionRequest; + use crate::routes::openai::utils::types::Normalizable; + use crate::utils::{ResolvedRequestContext, resolve_request_context}; + + fn request_context(headers: &HeaderMap, request_id: Option<&str>) -> ResolvedRequestContext { + resolve_request_context(headers, request_id) + } + + fn served(names: &[&str]) -> LoraModelResolution { + LoraModelResolution { + model_names: names.iter().map(|s| s.to_string()).collect(), + lora_request: None, + } + } + + fn test_tokenizer() -> TestTokenizer { + TestTokenizer::new() + } + + fn base_request_json() -> serde_json::Value { + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": true + }) + } + + #[test] + fn completion_http_request_deserializes_text_prompt() { + let request: CompletionRequest = + serde_json::from_value(base_request_json()).expect("parse request"); + + assert_eq!(request.prompt, Prompt::Text("hello".to_string())); + assert_eq!(request.model, "Qwen/Qwen1.5-0.5B-Chat"); + } + + #[test] + fn completion_http_request_deserializes_token_id_prompt() { + let request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": [11, 22, 33], + "stream": true, + "ignore_eos": true, + "max_tokens": 7 + })) + .expect("parse request"); + + assert_eq!(request.prompt, Prompt::TokenIds(vec![11, 22, 33])); + assert_eq!(request.max_tokens, Some(7)); + assert!(request.ignore_eos); + } + + #[test] + fn normalize_coerces_null_max_tokens_to_default() { + // An absent `max_tokens` already gets the serde default. + let absent: CompletionRequest = + serde_json::from_value(base_request_json()).expect("parse request"); + assert_eq!(absent.max_tokens, Some(16)); + + // An explicit `null` deserializes to `None`, bypassing the default; + // `normalize` must coerce it back to match Python vLLM. + let mut request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": true, + "max_tokens": null + })) + .expect("parse request"); + assert_eq!(request.max_tokens, None); + + request.normalize(); + assert_eq!(request.max_tokens, Some(16)); + } + + #[test] + fn prepare_completion_request_maps_sampling_fields() { + let request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": [11, 22, 33], + "stream": true, + "stream_options": {"include_usage": true}, + "max_tokens": 7, + "logprobs": 2, + "top_p": 0.9, + "top_k": 42, + "min_p": 0.1, + "frequency_penalty": 0.2, + "presence_penalty": 0.3, + "repetition_penalty": 1.1, + "ignore_eos": true, + "skip_special_tokens": false + })) + .expect("parse request"); + + let prepared = prepare_completion_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + &test_tokenizer(), + ) + .expect("prepare"); + + assert!(prepared.options.include_usage); + assert_eq!( + prepared.text_request.prompt, + Prompt::TokenIds(vec![11, 22, 33]) + ); + assert_eq!(prepared.text_request.sampling_params.max_tokens, Some(7)); + assert_eq!(prepared.text_request.sampling_params.logprobs, Some(2)); + assert_eq!(prepared.text_request.sampling_params.top_p, Some(0.9)); + assert_eq!(prepared.text_request.sampling_params.top_k, Some(42)); + assert_eq!(prepared.text_request.sampling_params.min_p, Some(0.1)); + assert_eq!( + prepared.text_request.sampling_params.frequency_penalty, + Some(0.2) + ); + assert_eq!( + prepared.text_request.sampling_params.presence_penalty, + Some(0.3) + ); + assert_eq!( + prepared.text_request.sampling_params.repetition_penalty, + Some(1.1) + ); + assert!(prepared.text_request.sampling_params.ignore_eos); + assert!(!prepared.text_request.decode_options.skip_special_tokens); + } + + #[test] + fn prepare_completion_request_passes_through_thinking_token_budget() { + let prepare = |budget: serde_json::Value| { + let request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "thinking_token_budget": budget, + })) + .expect("parse request"); + prepare_completion_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + &test_tokenizer(), + ) + .expect("prepare") + .text_request + .sampling_params + .thinking_token_budget + }; + + // The convert layer forwards the raw value verbatim (including the `-1` + // "unlimited" sentinel); normalization/validation happens during + // lowering (see `vllm_text::lower`). + assert_eq!(prepare(json!(64)), Some(64)); + assert_eq!(prepare(json!(-1)), Some(-1)); + assert_eq!(prepare(json!(null)), None); + } + + #[test] + fn prepare_completion_request_maps_stream_usage_and_token_format_options() { + let request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": true, + "stream_options": { + "include_usage": true, + "continuous_usage_stats": true + }, + "return_tokens_as_token_ids": true + })) + .expect("parse request"); + + let prepared = prepare_completion_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + &test_tokenizer(), + ) + .expect("prepare"); + + assert!(prepared.options.include_usage); + assert!(prepared.options.include_continuous_usage); + assert!(prepared.options.return_tokens_as_token_ids); + } + + #[test] + fn prepare_completion_request_gates_continuous_usage_on_include_usage() { + let request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": true, + "stream_options": { + "continuous_usage_stats": true + } + })) + .expect("parse request"); + + let prepared = prepare_completion_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + &test_tokenizer(), + ) + .expect("prepare"); + + assert!(!prepared.options.include_usage); + assert!(!prepared.options.include_continuous_usage); + } + + #[test] + fn prepare_completion_request_accepts_text_echo() { + let request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": true, + "echo": true, + "max_tokens": 7 + })) + .expect("parse request"); + + let prepared = prepare_completion_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + &test_tokenizer(), + ) + .expect("prepare"); + + assert_eq!(prepared.options.echo, Some("hello".to_string())); + assert_eq!(prepared.text_request.sampling_params.max_tokens, Some(7)); + assert!(!prepared.options.prompt_only); + } + + #[test] + fn prepare_completion_request_lowers_prompt_only_echo_as_one_internal_token() { + let request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false, + "echo": true, + "max_tokens": 0 + })) + .expect("parse request"); + + let prepared = prepare_completion_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + &test_tokenizer(), + ) + .expect("prepare"); + + assert!(prepared.options.prompt_only); + assert_eq!(prepared.options.echo, Some("hello".to_string())); + assert_eq!(prepared.text_request.sampling_params.max_tokens, Some(1)); + } + + #[test] + fn prepare_completion_request_enables_prompt_logprobs_for_stream_prompt_only_echo() { + let request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "echo": true, + "stream": true, + "max_tokens": 0, + "logprobs": 3 + })) + .expect("parse request"); + + let prepared = prepare_completion_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + &test_tokenizer(), + ) + .expect("prepare"); + + assert!(prepared.options.prompt_only); + assert_eq!(prepared.text_request.sampling_params.logprobs, Some(3)); + assert_eq!( + prepared.text_request.sampling_params.prompt_logprobs, + Some(3) + ); + } + + #[test] + fn prepare_completion_request_enables_prompt_logprobs_for_non_stream_echo() { + let request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "echo": true, + "stream": false, + "logprobs": 3 + })) + .expect("parse request"); + + let prepared = prepare_completion_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + &test_tokenizer(), + ) + .expect("prepare"); + + assert_eq!(prepared.text_request.sampling_params.logprobs, Some(3)); + assert_eq!( + prepared.text_request.sampling_params.prompt_logprobs, + Some(3) + ); + } + + #[test] + fn prepare_completion_request_decodes_token_id_prompt_echo() { + let request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": [104, 101, 108, 108, 111], + "stream": true, + "echo": true + })) + .expect("parse request"); + + let prepared = prepare_completion_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + &test_tokenizer(), + ) + .expect("prepare"); + + assert_eq!(prepared.options.echo, Some("hello".to_string())); + assert_eq!( + prepared.text_request.prompt, + Prompt::TokenIds(vec![104, 101, 108, 108, 111]) + ); + } + + #[test] + fn prepare_completion_request_accepts_logprobs_fields() { + let request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false, + "logprobs": 1, + "prompt_logprobs": 2 + })) + .expect("parse request"); + + let prepared = prepare_completion_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + &test_tokenizer(), + ) + .expect("prepare"); + assert_eq!(prepared.text_request.sampling_params.logprobs, Some(1)); + assert_eq!( + prepared.text_request.sampling_params.prompt_logprobs, + Some(2) + ); + } + + #[test] + fn prepare_completion_request_threads_data_parallel_rank() { + let request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false, + })) + .expect("parse request"); + + let mut headers = HeaderMap::new(); + headers.insert("X-data-parallel-rank", "3".parse().unwrap()); + let prepared = prepare_completion_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + request_context(&headers, None), + &test_tokenizer(), + ) + .expect("prepare"); + assert_eq!(prepared.text_request.data_parallel_rank, Some(3)); + } + + #[test] + fn prepare_completion_request_leaves_data_parallel_rank_none_when_absent() { + let request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false, + })) + .expect("parse request"); + + let prepared = prepare_completion_request( + request, + &served(&["Qwen/Qwen1.5-0.5B-Chat"]), + ResolvedRequestContext::default(), + &test_tokenizer(), + ) + .expect("prepare"); + assert_eq!(prepared.text_request.data_parallel_rank, None); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/completions/types.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/completions/types.rs new file mode 100644 index 0000000000000000000000000000000000000000..327e49f159296b0c1a6163ebf057f11dbdbb8124 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/completions/types.rs @@ -0,0 +1,272 @@ +use std::collections::HashMap; + +use serde::{Deserialize, Serialize}; +use serde_json::{Map, Value}; +use validator::Validate; +use vllm_engine_core_client::protocol::sampling::RepetitionDetectionParams; +use vllm_text::Prompt; + +use crate::routes::openai::utils::types::{ + LogProbs, Normalizable, StreamOptions, StringOrArray, Usage, default_true, validate_stop, +}; + +/// Serde default for `CompletionRequest::max_tokens`, matching the Python vLLM +/// / OpenAI default. +fn default_completion_max_tokens() -> Option { + Some(16) +} + +/// vLLM-compatible request type for the Completions API. +/// +/// Mirrors the Python vLLM `CompletionRequest` class. The local copy keeps the +/// request type route-owned so we can accept token-id prompts via +/// [`vllm_text::Prompt`] and add vLLM-only fields directly instead of layering +/// wrapper deserializers on top. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Deserialize, Serialize, Validate)] +pub struct CompletionRequest { + // -------- Standard OpenAI API Parameters -------- + /// ID of the model to use + pub model: String, + + /// The prompt(s) to generate completions for. + /// + /// We use [`vllm_text::Prompt`] here to support token-id input. + pub prompt: Prompt, + + /// Echo back the prompt in addition to the completion + #[serde(default)] + pub echo: bool, + + /// Number between -2.0 and 2.0. Positive values penalize new tokens based + /// on their existing frequency in the text so far + pub frequency_penalty: Option, + + /// Modify the likelihood of specified tokens appearing in the completion + pub logit_bias: Option>, + + /// Include the log probabilities on the logprobs most likely tokens + pub logprobs: Option, + + /// The maximum number of tokens to generate (defaults to 16 when absent, + /// matching the Python vLLM / OpenAI API convention) + #[serde(default = "default_completion_max_tokens")] + pub max_tokens: Option, + + /// How many completions to generate for each prompt + pub n: Option, + + /// Number between -2.0 and 2.0. Positive values penalize new tokens based + /// on whether they appear in the text so far + pub presence_penalty: Option, + + /// If specified, our system will make a best effort to sample + /// deterministically + pub seed: Option, + + /// Up to 4 sequences where the API will stop generating further tokens + #[validate(custom(function = "validate_stop"))] + pub stop: Option, + + /// Whether to stream back partial progress + #[serde(default)] + pub stream: bool, + + /// The suffix that comes after a completion of inserted text + pub suffix: Option, + + /// What sampling temperature to use, between 0 and 2 + pub temperature: Option, + + /// An alternative to sampling with temperature (nucleus sampling) + pub top_p: Option, + + /// A unique identifier representing your end-user + pub user: Option, + + // -------- vLLM Sampling Parameters -------- + /// Options for streaming response + pub stream_options: Option, + + /// Use beam search instead of sampling + #[serde(default)] + pub use_beam_search: bool, + + /// Top-k sampling parameter + pub top_k: Option, + + /// Min-p nucleus sampling parameter + pub min_p: Option, + + /// Repetition penalty for reducing repetitive text + pub repetition_penalty: Option, + + /// Parameters for detecting repetitive N-gram patterns in output tokens + pub repetition_detection: Option, + + /// Length penalty for beam search + pub length_penalty: Option, + + /// Specific token IDs to use as stop conditions + pub stop_token_ids: Option>, + + /// Include stop string in output + #[serde(default)] + pub include_stop_str_in_output: bool, + + /// Ignore end-of-sequence tokens during generation + #[serde(default)] + pub ignore_eos: bool, + + /// Minimum number of tokens to generate + pub min_tokens: Option, + + /// Skip special tokens during detokenization + #[serde(default = "default_true")] + pub skip_special_tokens: bool, + + /// Add spaces between special tokens during detokenization + #[serde(default = "default_true")] + pub spaces_between_special_tokens: bool, + + /// Truncate prompt tokens to this length + pub truncate_prompt_tokens: Option, + + /// Restrict output to these token IDs only + pub allowed_token_ids: Option>, + + /// Number of prompt logprobs to return + pub prompt_logprobs: Option, + + // -------- Extra vLLM Parameters -------- + /// Whether to add special tokens (e.g. BOS) to the prompt + #[serde(default = "default_true")] + pub add_special_tokens: bool, + + /// Format specification for structured output (JSON mode, JSON schema, + /// etc.) + pub response_format: Option, + + /// Additional kwargs for structured outputs + pub structured_outputs: Option, + + /// Token budget for reasoning/thinking. Accepts a non-negative integer, or + /// `-1` for unlimited (mirroring the Python frontend, which normalizes `-1` + /// to "no budget"). + pub thinking_token_budget: Option, + + /// Request scheduling priority (lower means earlier; default 0) + pub priority: Option, + + /// External request ID used for response correlation. + pub request_id: Option, + + /// Tokens represented as strings of the form 'token_id:{token_id}' in + /// logprobs + pub return_tokens_as_token_ids: Option, + + /// Include token IDs alongside generated text + pub return_token_ids: Option, + + /// Salt for prefix cache isolation in multi-user environments + pub cache_salt: Option, + + /// KV transfer parameters for disaggregated serving + pub kv_transfer_params: Option>, + + /// Additional request parameters with string or numeric values for custom + /// extensions + pub vllm_xargs: Option>, + + /// Additional fields + #[serde(flatten)] + pub other: Map, +} + +impl Normalizable for CompletionRequest { + /// Normalize the request by applying defaults. + fn normalize(&mut self) { + // An explicit `"max_tokens": null` deserializes to `None`, bypassing the + // serde field default. Coerce it back to the default so it behaves like + // an absent field, matching Python vLLM's `normalize_null_max_tokens`. + if self.max_tokens.is_none() { + self.max_tokens = default_completion_max_tokens(); + } + } +} + +/// Mirrors the Python vLLM `CompletionResponse` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub(super) struct CompletionResponse { + pub id: String, + pub object: String, + pub created: u64, + pub model: String, + pub choices: Vec, + pub usage: Option, + pub system_fingerprint: Option, + pub kv_transfer_params: Option, +} + +/// Mirrors the Python vLLM `CompletionResponseChoice` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub(super) struct CompletionChoice { + pub index: u32, + pub text: String, + pub logprobs: Option, + pub finish_reason: Option, + pub stop_reason: Option, + pub prompt_logprobs: Option>>>, + pub token_ids: Option>, + pub prompt_token_ids: Option>, +} + +/// Mirrors the Python vLLM `CompletionStreamResponse` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub(super) struct CompletionStreamResponse { + pub id: String, + pub object: String, + pub created: u64, + pub model: String, + pub choices: Vec, + pub usage: Option, +} + +impl CompletionStreamResponse { + /// Create a stream response with the standard envelope fields pre-filled. + pub fn new(id: &str, model: &str, created: u64) -> Self { + Self { + id: id.to_string(), + object: "text_completion".to_string(), + created, + model: model.to_string(), + choices: Vec::new(), + usage: None, + } + } +} + +/// Mirrors the Python vLLM `CompletionResponseStreamChoice` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Default, Serialize)] +pub(super) struct CompletionStreamChoice { + pub index: u32, + pub text: String, + pub logprobs: Option, + pub finish_reason: Option, + pub stop_reason: Option, + pub token_ids: Option>, + pub prompt_token_ids: Option>, +} + +#[derive(Debug, Clone, Serialize)] +#[serde(untagged)] +pub(super) enum CompletionSseChunk { + /// Ordinary OpenAI completions delta/final chunk. + Chunk(CompletionStreamResponse), + /// Final usage chunk emitted before `[DONE]` when `include_usage=true`. + Usage(CompletionStreamResponse), +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/completions/validate.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/completions/validate.rs new file mode 100644 index 0000000000000000000000000000000000000000..5ba05285ca7ecb113a603e177185e1cdfcba7cc2 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/completions/validate.rs @@ -0,0 +1,204 @@ +use super::types::CompletionRequest; +use crate::error::{ApiError, bail_invalid_request}; + +/// Enforce the minimal compatibility contract for the Rust OpenAI server. +pub(super) fn validate_request_compat( + request: &CompletionRequest, + served_model_names: &[String], +) -> Result<(), ApiError> { + // This path is intentionally scoped to the minimum surface needed by + // `vllm-bench` random workload compatibility, so unsupported legacy + // completions features fail early here. + if !served_model_names.iter().any(|n| n == &request.model) { + return Err(ApiError::model_not_found(request.model.clone())); + } + + if request.stream_options.is_some() && !request.stream { + bail_invalid_request!( + param = "stream_options", + "stream_options are only supported when stream=true." + ); + } + + if request.n.unwrap_or(1) > 1 { + bail_invalid_request!(param = "n", "Only n=1 is supported."); + } + + if request.max_tokens == Some(0) && !request.echo { + bail_invalid_request!( + param = "max_tokens", + "max_tokens=0 is only supported when echo=true." + ); + } + + if request.suffix.is_some() { + bail_invalid_request!(param = "suffix", "suffix is not supported."); + } + + if let Some(logprobs) = request.logprobs + && logprobs > i32::MAX as u32 + { + bail_invalid_request!( + param = "logprobs", + "`logprobs` must fit within a signed 32-bit integer." + ); + } + + if let Some(prompt_logprobs) = request.prompt_logprobs { + if request.stream && (prompt_logprobs > 0 || prompt_logprobs == -1) { + bail_invalid_request!( + param = "prompt_logprobs", + "`prompt_logprobs` are not available when `stream=true`." + ); + } + + if prompt_logprobs < 0 && prompt_logprobs != -1 { + bail_invalid_request!( + param = "prompt_logprobs", + "`prompt_logprobs` must be a non-negative value or -1." + ); + } + } + + if request.use_beam_search { + bail_invalid_request!( + param = "use_beam_search", + "use_beam_search is not supported." + ); + } + + // ---- Reject parameters that are accepted for deserialization but not yet + // implemented ---- + + if request.length_penalty.is_some() { + bail_invalid_request!(param = "length_penalty", "length_penalty is not supported."); + } + if !request.spaces_between_special_tokens { + bail_invalid_request!( + param = "spaces_between_special_tokens", + "spaces_between_special_tokens is not supported." + ); + } + if request.truncate_prompt_tokens.is_some() { + bail_invalid_request!( + param = "truncate_prompt_tokens", + "truncate_prompt_tokens is not supported." + ); + } + + Ok(()) +} + +#[cfg(test)] +mod tests { + use serde_json::json; + + use super::validate_request_compat; + use crate::routes::openai::completions::types::CompletionRequest; + + fn base_request() -> CompletionRequest { + serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": true, + })) + .expect("parse request") + } + + fn served_names(names: &[&str]) -> Vec { + names.iter().map(|s| s.to_string()).collect() + } + + #[test] + fn validate_request_compat_accepts_logprobs() { + let request = CompletionRequest { + logprobs: Some(1), + ..base_request() + }; + assert!( + validate_request_compat(&request, &served_names(&["Qwen/Qwen1.5-0.5B-Chat"])).is_ok() + ); + } + + #[test] + fn validate_request_compat_accepts_any_served_name() { + let request = base_request(); + assert!( + validate_request_compat( + &request, + &served_names(&["other-alias", "Qwen/Qwen1.5-0.5B-Chat"]) + ) + .is_ok() + ); + } + + #[test] + fn validate_request_compat_rejects_unknown_model() { + let request = base_request(); + assert!(validate_request_compat(&request, &served_names(&["other-model"])).is_err()); + } + + #[test] + fn validate_request_compat_rejects_streaming_prompt_logprobs() { + let request = CompletionRequest { + prompt_logprobs: Some(1), + ..base_request() + }; + assert!( + validate_request_compat(&request, &served_names(&["Qwen/Qwen1.5-0.5B-Chat"])).is_err() + ); + } + + #[test] + fn validate_request_compat_accepts_non_stream_prompt_logprobs() { + let request = CompletionRequest { + stream: false, + prompt_logprobs: Some(-1), + ..base_request() + }; + assert!( + validate_request_compat(&request, &served_names(&["Qwen/Qwen1.5-0.5B-Chat"])).is_ok() + ); + } + + #[test] + fn validate_request_compat_accepts_prompt_only_echo() { + let request = CompletionRequest { + stream: false, + echo: true, + max_tokens: Some(0), + ..base_request() + }; + assert!( + validate_request_compat(&request, &served_names(&["Qwen/Qwen1.5-0.5B-Chat"])).is_ok() + ); + } + + #[test] + fn validate_request_compat_accepts_token_id_prompt_echo() { + let request: CompletionRequest = serde_json::from_value(json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": [104, 101, 108, 108, 111], + "stream": true, + "echo": true, + })) + .expect("parse request"); + + assert!( + validate_request_compat(&request, &served_names(&["Qwen/Qwen1.5-0.5B-Chat"])).is_ok() + ); + } + + #[test] + fn validate_request_compat_rejects_prompt_only_without_echo() { + let request = CompletionRequest { + stream: false, + echo: false, + max_tokens: Some(0), + ..base_request() + }; + assert!( + validate_request_compat(&request, &served_names(&["Qwen/Qwen1.5-0.5B-Chat"])).is_err() + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/mod.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..af72c67e8e4275ea7d914951cdec90789a314988 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/mod.rs @@ -0,0 +1,8 @@ +pub mod chat_completions; +mod completions; +mod models; +pub(crate) mod utils; + +pub use chat_completions::chat_completions; +pub use completions::completions; +pub use models::list_models; diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/models.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/models.rs new file mode 100644 index 0000000000000000000000000000000000000000..98735a998e4a3a77036e8591962dd5c15059b744 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/models.rs @@ -0,0 +1,45 @@ +use std::sync::Arc; +use std::time::{SystemTime, UNIX_EPOCH}; + +use axum::Json; +use axum::extract::State; + +use crate::routes::openai::utils::types::{ListModelsResponse, ModelObject}; +use crate::state::AppState; + +// Frontend marker; Python uses "vllm". +const OWNED_BY: &str = "vllm-frontend-rs"; + +/// Base cards carry `max_model_len` and `root` = model path; LoRA cards carry +/// `root` = adapter path and `parent` = base model. LoRA cards follow load order. +pub async fn list_models(State(state): State>) -> Json { + let created = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs() as i64; + let max_model_len = state.chat.engine_core_client().max_model_len(); + let model_path = state.model_path().map(str::to_string); + + let base_cards = state.served_model_names().iter().map(|name| ModelObject { + id: name.clone(), + object: "model".to_string(), + created, + owned_by: OWNED_BY.to_string(), + root: Some(model_path.clone().unwrap_or_else(|| name.clone())), + parent: None, + max_model_len: Some(max_model_len), + }); + + let primary = state.primary_model_name().to_string(); + let lora_cards = state.served_lora_requests().await.into_iter().map(|lora| ModelObject { + id: lora.lora_name, + object: "model".to_string(), + created, + owned_by: OWNED_BY.to_string(), + root: Some(lora.lora_path), + parent: Some(lora.base_model_name.unwrap_or_else(|| primary.clone())), + max_model_len: None, + }); + + Json(ListModelsResponse { + object: "list".to_string(), + data: base_cards.chain(lora_cards).collect(), + }) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/logprobs.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/logprobs.rs new file mode 100644 index 0000000000000000000000000000000000000000..382c1001104409b7a7635033a597ff5b8ab231a2 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/logprobs.rs @@ -0,0 +1,256 @@ +use std::collections::HashMap; + +use itertools::Itertools as _; +use vllm_text::{ + CollectedTextOutput, DecodedLogprobs, DecodedPositionLogprobs, DecodedPromptLogprobs, + DecodedTokenLogprob, +}; + +use super::types::{ChatLogProbs, ChatLogProbsContent, LogProbs, TopLogProb}; +use crate::error::{ApiError, server_error}; + +/// Convert decoded token-position logprobs into the OpenAI completions +/// `logprobs` shape. +pub fn decoded_logprobs_to_openai( + logprobs: &DecodedLogprobs, + initial_text_offset: u32, + return_tokens_as_token_ids: bool, +) -> Result { + let mut text_offset = Vec::with_capacity(logprobs.positions.len()); + let mut token_logprobs = Vec::with_capacity(logprobs.positions.len()); + let mut tokens = Vec::with_capacity(logprobs.positions.len()); + let mut top_logprobs = Vec::with_capacity(logprobs.positions.len()); + let mut current_offset = initial_text_offset; + + for position in &logprobs.positions { + let chosen = position.entries.first().ok_or_else(|| { + server_error!("decoded logprobs position unexpectedly had no token candidates") + })?; + + let token_str = format_token(chosen, return_tokens_as_token_ids); + text_offset.push(current_offset); + token_logprobs.push(Some(clamp_logprob(chosen.logprob))); + current_offset = current_offset.saturating_add(text_len(&token_str)); + tokens.push(token_str); + top_logprobs.push(Some(position_top_logprobs_map( + position, + return_tokens_as_token_ids, + ))); + } + + Ok(LogProbs { + tokens, + token_logprobs, + top_logprobs, + text_offset, + }) +} + +/// Convert decoded prompt logprobs into the OpenAI completions `logprobs` +/// shape. +/// +/// The first prompt token is included with `None` logprob metadata, matching +/// Python vLLM's echoed completions behavior. +pub fn decoded_prompt_logprobs_to_openai( + prompt_logprobs: &DecodedPromptLogprobs, + initial_text_offset: u32, + return_tokens_as_token_ids: bool, +) -> Result { + let mut text_offset = Vec::with_capacity(prompt_logprobs.scored_positions.len() + 1); + let mut token_logprobs = Vec::with_capacity(prompt_logprobs.scored_positions.len() + 1); + let mut tokens = Vec::with_capacity(prompt_logprobs.scored_positions.len() + 1); + let mut top_logprobs = Vec::with_capacity(prompt_logprobs.scored_positions.len() + 1); + let mut current_offset = initial_text_offset; + + let first_token_str = if return_tokens_as_token_ids { + format!("token_id:{}", prompt_logprobs.first_token_id) + } else { + prompt_logprobs.first_token.clone() + }; + text_offset.push(current_offset); + token_logprobs.push(None); + current_offset = current_offset.saturating_add(text_len(&first_token_str)); + tokens.push(first_token_str); + top_logprobs.push(None); + + for position in &prompt_logprobs.scored_positions { + let chosen = position.entries.first().ok_or_else(|| { + server_error!("decoded prompt logprobs position unexpectedly had no token candidates") + })?; + + let token_str = format_token(chosen, return_tokens_as_token_ids); + text_offset.push(current_offset); + token_logprobs.push(Some(clamp_logprob(chosen.logprob))); + current_offset = current_offset.saturating_add(text_len(&token_str)); + tokens.push(token_str); + top_logprobs.push(Some(position_top_logprobs_map( + position, + return_tokens_as_token_ids, + ))); + } + + Ok(LogProbs { + tokens, + token_logprobs, + top_logprobs, + text_offset, + }) +} + +/// Convert decoded prompt logprobs into the vLLM-style prompt-logprobs response +/// shape. +pub fn decoded_prompt_logprobs_to_maps( + prompt_logprobs: &DecodedPromptLogprobs, + return_tokens_as_token_ids: bool, +) -> Vec>> { + std::iter::once(None) + .chain(prompt_logprobs.scored_positions.iter().map(|position| { + Some(position_top_logprobs_map( + position, + return_tokens_as_token_ids, + )) + })) + .collect() +} + +/// Convert decoded token-position logprobs into the OpenAI chat `logprobs` +/// shape. +pub fn decoded_logprobs_to_openai_chat( + logprobs: &DecodedLogprobs, + return_tokens_as_token_ids: bool, +) -> Result { + let content = logprobs + .positions + .iter() + .map(|pos| position_to_chat_logprobs_content(pos, return_tokens_as_token_ids)) + .try_collect()?; + + Ok(ChatLogProbs { + content: Some(content), + }) +} + +/// Count visible text positions using OpenAI completions' character-offset +/// convention. +pub fn text_len(text: &str) -> u32 { + u32::try_from(text.chars().count()).unwrap_or(u32::MAX) +} + +/// Concatenate two OpenAI-style completion logprobs payloads in token order. +pub fn append_openai_logprobs(mut prefix: LogProbs, suffix: LogProbs) -> LogProbs { + prefix.tokens.extend(suffix.tokens); + prefix.token_logprobs.extend(suffix.token_logprobs); + prefix.top_logprobs.extend(suffix.top_logprobs); + prefix.text_offset.extend(suffix.text_offset); + prefix +} + +/// Build the non-stream completions `logprobs` payload from collected text +/// output. +/// +/// When `echoed_prompt` is true, the returned payload matches Python vLLM's +/// echoed completions behavior by concatenating prompt and completion logprobs +/// into one OpenAI `LogProbs` object. +pub fn collected_logprobs_to_openai( + collected: &CollectedTextOutput, + echoed_prompt: bool, + initial_completion_offset: u32, + return_tokens_as_token_ids: bool, +) -> Result { + if echoed_prompt { + let prompt_logprobs = collected.prompt_logprobs.as_ref().ok_or_else(|| { + server_error!( + "echoed completion logprobs require prompt logprobs but generation returned none" + ) + })?; + let prompt_logprobs = + decoded_prompt_logprobs_to_openai(prompt_logprobs, 0, return_tokens_as_token_ids)?; + let completion_start = prompt_logprobs + .text_offset + .last() + .zip(prompt_logprobs.tokens.last()) + .map(|(&offset, token)| offset.saturating_add(text_len(token))) + .unwrap_or(0); + return match collected.logprobs.as_ref() { + Some(completion_logprobs) => Ok(append_openai_logprobs( + prompt_logprobs, + decoded_logprobs_to_openai( + completion_logprobs, + completion_start, + return_tokens_as_token_ids, + )?, + )), + None => Ok(prompt_logprobs), + }; + } + + let completion_logprobs = collected.logprobs.as_ref().ok_or_else(|| { + server_error!("completion response requested logprobs but generation returned none") + })?; + decoded_logprobs_to_openai( + completion_logprobs, + initial_completion_offset, + return_tokens_as_token_ids, + ) +} + +/// Format a token entry as either its decoded string or `token_id:{id}`. +fn format_token(entry: &DecodedTokenLogprob, as_token_id: bool) -> String { + if as_token_id { + format!("token_id:{}", entry.token_id) + } else { + entry.token.clone() + } +} + +fn position_top_logprobs_map( + position: &DecodedPositionLogprobs, + return_tokens_as_token_ids: bool, +) -> HashMap { + position + .entries + .iter() + .map(|entry| { + ( + format_token(entry, return_tokens_as_token_ids), + clamp_logprob(entry.logprob), + ) + }) + .collect() +} + +fn position_to_chat_logprobs_content( + position: &DecodedPositionLogprobs, + return_tokens_as_token_ids: bool, +) -> Result { + let chosen = position.entries.first().ok_or_else(|| { + server_error!("decoded chat logprobs position unexpectedly had no token candidates") + })?; + + let token_str = format_token(chosen, return_tokens_as_token_ids); + Ok(ChatLogProbsContent { + token: token_str.clone(), + logprob: clamp_logprob(chosen.logprob), + bytes: Some(token_bytes(&token_str)), + top_logprobs: position + .entries + .iter() + .map(|entry| { + let t = format_token(entry, return_tokens_as_token_ids); + TopLogProb { + logprob: clamp_logprob(entry.logprob), + bytes: Some(token_bytes(&t)), + token: t, + } + }) + .collect(), + }) +} + +fn token_bytes(token: &str) -> Vec { + token.as_bytes().to_vec() +} + +pub fn clamp_logprob(logprob: f32) -> f32 { + logprob.max(-9999.0) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/mod.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..847f76b2bc620b1bfdc113350e2900577e1b8a8c --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/mod.rs @@ -0,0 +1,5 @@ +pub mod logprobs; +pub mod structured_outputs; +pub mod types; +pub mod usage; +pub mod validated_json; diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/structured_outputs.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/structured_outputs.rs new file mode 100644 index 0000000000000000000000000000000000000000..daf9bbb6567a203e3f2d03a14db5bb62e5efee2f --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/structured_outputs.rs @@ -0,0 +1,125 @@ +use serde::{Deserialize, Serialize}; +use serde_json::Value; +use vllm_engine_core_client::protocol::structured_outputs::StructuredOutputsParams; + +use crate::error::ApiError; + +/// JSON schema specification nested inside a `json_schema` response format. +/// +/// Mirrors the Python vLLM `JsonSchemaResponseFormat` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct JsonSchemaFormat { + pub name: String, + #[serde(default)] + pub description: Option, + /// The actual JSON schema object. + #[serde(alias = "json_schema")] + pub schema: Value, + #[serde(default)] + pub strict: Option, +} + +/// Supported `response_format` types for chat and completion requests. +/// +/// This is our own definition (rather than the `openai-protocol` crate's) so +/// that we can support the vLLM-specific `structural_tag` variant. +/// +/// Original Python definitions: +/// +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(tag = "type", rename_all = "snake_case")] +pub enum ResponseFormat { + Text, + JsonObject, + JsonSchema { + json_schema: JsonSchemaFormat, + }, + /// vLLM-specific structural tag format. The entire object (including the + /// `type` field) is JSON-serialized and passed as + /// `StructuredOutputConstraint::StructuralTag`. + /// + /// We capture the payload as a catch-all map so both the legacy + /// (`structures`/`triggers`) and current (`format`) shapes are + /// preserved without needing typed structs. + StructuralTag { + #[serde(flatten)] + extra: serde_json::Map, + }, +} + +/// Convert an explicit `structured_outputs` JSON blob into +/// [`StructuredOutputsParams`]. +fn deserialize_structured_outputs( + raw: &serde_json::Value, +) -> Result { + serde_json::from_value(raw.clone()).map_err(|e| { + ApiError::invalid_request( + format!("invalid structured_outputs: {e}"), + Some("structured_outputs"), + ) + }) +} + +/// Convert a typed [`ResponseFormat`] and/or raw `structured_outputs` blob into +/// engine-core [`StructuredOutputsParams`]. +/// +/// Mirrors the Python vLLM conversion in +/// `ChatCompletionRequest.to_sampling_params()`: +pub fn convert_from_response_format( + response_format: Option<&ResponseFormat>, + structured_outputs: &Option, +) -> Result, ApiError> { + if let Some(raw) = structured_outputs { + return Ok(Some(deserialize_structured_outputs(raw)?)); + } + + let Some(fmt) = response_format else { + return Ok(None); + }; + match fmt { + ResponseFormat::Text => Ok(None), + ResponseFormat::JsonObject => Ok(Some(StructuredOutputsParams::json_object())), + ResponseFormat::JsonSchema { json_schema } => Ok(Some(StructuredOutputsParams::json( + json_schema.schema.clone(), + ))), + ResponseFormat::StructuralTag { .. } => { + // The Python frontend dumps the entire response_format object (including the + // `type` field) as a JSON string for the engine-core backend. + let tag_json = serde_json::to_string(fmt).map_err(|e| { + ApiError::invalid_request( + format!("failed to serialize structural_tag: {e}"), + Some("response_format"), + ) + })?; + Ok(Some(StructuredOutputsParams::structural_tag(tag_json))) + } + } +} + +/// Convert raw `response_format` and/or `structured_outputs` JSON blobs into +/// engine-core [`StructuredOutputsParams`]. +/// +/// Used by the completions endpoint which keeps both fields as opaque +/// `serde_json::Value`. +pub fn convert_from_response_format_value( + response_format: &Option, + structured_outputs: &Option, +) -> Result, ApiError> { + if let Some(raw) = structured_outputs { + return Ok(Some(deserialize_structured_outputs(raw)?)); + } + + let Some(raw) = response_format else { + return Ok(None); + }; + + // Deserialize into our typed enum and delegate. + let fmt: ResponseFormat = serde_json::from_value(raw.clone()).map_err(|e| { + ApiError::invalid_request( + format!("invalid response_format: {e}"), + Some("response_format"), + ) + })?; + convert_from_response_format(Some(&fmt), &None) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/types.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/types.rs new file mode 100644 index 0000000000000000000000000000000000000000..3df17da03c10b297c0aa0496e9bacd6b14de1a41 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/types.rs @@ -0,0 +1,520 @@ +use std::collections::HashMap; +use std::slice; + +use llm_multimodal::ImageDetail; +use serde::{Deserialize, Serialize}; +use serde_json::Value; +use vllm_llm::TokenUsage; + +// ============================================================================ +// Constants +// ============================================================================ + +/// Default model identifier used when no model is specified. +pub const UNKNOWN_MODEL_ID: &str = "unknown"; + +// ============================================================================ +// Default value helpers +// ============================================================================ + +/// Helper function for serde default value (returns true). +pub fn default_true() -> bool { + true +} + +// ============================================================================ +// String/Array Utilities +// ============================================================================ + +/// A type that can be either a single string or an array of strings. +#[derive(Debug, Clone, PartialEq, Deserialize, Serialize)] +#[serde(untagged)] +pub enum StringOrArray { + String(String), + Array(Vec), +} + +impl StringOrArray { + pub fn as_slice(&self) -> &[String] { + match self { + StringOrArray::String(s) => slice::from_ref(s), + StringOrArray::Array(arr) => arr, + } + } + + #[allow(unused)] + pub fn into_vec(self) -> Vec { + match self { + StringOrArray::String(s) => vec![s], + StringOrArray::Array(arr) => arr, + } + } +} + +/// Validates stop sequences (non-empty strings) +pub fn validate_stop(stop: &StringOrArray) -> Result<(), validator::ValidationError> { + if stop.as_slice().iter().any(|s| s.is_empty()) { + return Err(validator::ValidationError::new( + "stop strings cannot be empty", + )); + } + Ok(()) +} + +// ============================================================================ +// Validation helpers +// ============================================================================ + +/// Validates top_p: 0.0 < top_p <= 1.0. +pub fn validate_top_p_value(top_p: f32) -> Result<(), validator::ValidationError> { + if !(top_p > 0.0 && top_p <= 1.0) { + return Err(validator::ValidationError::new( + "top_p must be in (0, 1] - greater than 0.0 and at most 1.0", + )); + } + Ok(()) +} + +// ============================================================================ +// Content Parts (for multimodal messages) +// ============================================================================ + +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)] +#[serde(tag = "type")] +pub enum ContentPart { + #[serde(rename = "text")] + Text { text: String }, + #[serde(rename = "image_url")] + ImageUrl { + image_url: ImageUrl, + #[serde(skip_serializing_if = "Option::is_none")] + uuid: Option, + }, + #[serde(rename = "video_url")] + VideoUrl { video_url: VideoUrl }, +} + +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)] +pub struct ImageUrl { + pub url: String, + pub detail: Option, +} + +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)] +pub struct VideoUrl { + pub url: String, +} + +// ============================================================================ +// Streaming +// ============================================================================ + +/// Mirrors the Python vLLM `StreamOptions` class. +#[derive(Debug, Clone, Deserialize, Serialize)] +pub struct StreamOptions { + pub include_usage: Option, + pub continuous_usage_stats: Option, +} + +// ============================================================================ +// Tools and Function Calling +// ============================================================================ + +#[derive(Debug, Clone, Deserialize, Serialize)] +pub struct Tool { + #[serde(rename = "type")] + pub tool_type: String, + pub function: Function, +} + +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Deserialize, Serialize)] +pub struct Function { + pub name: String, + pub description: Option, + pub parameters: Value, + /// Whether to enable strict schema adherence (OpenAI structured outputs). + pub strict: Option, +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +pub struct ToolCall { + pub id: String, + #[serde(rename = "type")] + pub tool_type: String, + pub function: FunctionCallResponse, +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +pub struct FunctionCallResponse { + pub name: String, + #[serde(default)] + pub arguments: Option, +} + +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Deserialize, Serialize)] +pub struct ToolCallDelta { + pub index: u32, + pub id: Option, + #[serde(rename = "type")] + pub tool_type: Option, + pub function: Option, +} + +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Deserialize, Serialize)] +pub struct FunctionCallDelta { + pub name: Option, + pub arguments: Option, +} + +/// Tool choice value for simple string options. +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum ToolChoiceValue { + Auto, + Required, + None, +} + +/// Tool choice for the Chat Completion API. +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(untagged)] +pub enum ToolChoice { + Value(ToolChoiceValue), + Function { + #[serde(rename = "type")] + tool_type: String, + function: FunctionChoice, + }, + AllowedTools { + #[serde(rename = "type")] + tool_type: String, + mode: String, + tools: Vec, + }, +} + +impl Default for ToolChoice { + fn default() -> Self { + Self::Value(ToolChoiceValue::Auto) + } +} + +/// Function choice specification for `ToolChoice::Function`. +#[derive(Debug, Clone, Deserialize, Serialize)] +pub struct FunctionChoice { + pub name: String, +} + +/// Tool reference for `ToolChoice::AllowedTools`. +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(tag = "type")] +#[serde(rename_all = "snake_case")] +pub enum ToolReference { + #[serde(rename = "function")] + Function { name: String }, + #[serde(rename = "mcp")] + Mcp { + server_label: String, + #[serde(skip_serializing_if = "Option::is_none")] + name: Option, + }, + #[serde(rename = "file_search")] + FileSearch, + #[serde(rename = "web_search_preview")] + WebSearchPreview, + #[serde(rename = "computer_use_preview")] + ComputerUsePreview, + #[serde(rename = "code_interpreter")] + CodeInterpreter, + #[serde(rename = "image_generation")] + ImageGeneration, +} + +impl ToolReference { + /// Get a unique identifier for this tool reference. + pub fn identifier(&self) -> String { + match self { + ToolReference::Function { name } => format!("function:{name}"), + ToolReference::Mcp { + server_label, + name: Some(n), + } => format!("mcp:{server_label}:{n}"), + ToolReference::Mcp { + server_label, + name: _, + } => format!("mcp:{server_label}"), + ToolReference::FileSearch => "file_search".to_string(), + ToolReference::WebSearchPreview => "web_search_preview".to_string(), + ToolReference::ComputerUsePreview => "computer_use_preview".to_string(), + ToolReference::CodeInterpreter => "code_interpreter".to_string(), + ToolReference::ImageGeneration => "image_generation".to_string(), + } + } +} + +// ============================================================================ +// Chat Messages +// ============================================================================ + +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(tag = "role")] +pub enum ChatMessage { + #[serde(rename = "system")] + System { + content: MessageContent, + name: Option, + }, + #[serde(rename = "user")] + User { + content: MessageContent, + name: Option, + }, + #[serde(rename = "assistant")] + Assistant { + content: Option, + name: Option, + tool_calls: Option>, + /// Reasoning content for reasoning-capable models. + #[serde(alias = "reasoning_content")] + #[serde(alias = "thinking")] + reasoning: Option, + }, + #[serde(rename = "tool")] + Tool { + content: MessageContent, + tool_call_id: String, + }, + #[serde(rename = "function")] + Function { content: String, name: String }, + #[serde(rename = "developer")] + Developer { + content: MessageContent, + tools: Option>, + name: Option, + }, +} + +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)] +#[serde(untagged)] +pub enum MessageContent { + Text(String), + Parts(Vec), +} + +// ============================================================================ +// Usage and Logging +// ============================================================================ + +/// Mirrors the Python vLLM `UsageInfo` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub struct Usage { + pub prompt_tokens: usize, + pub total_tokens: usize, + pub completion_tokens: Option, + pub prompt_tokens_details: Option, +} + +impl Usage { + /// Create a Usage with prompt-token cache details. + pub fn from_counts( + prompt_tokens: usize, + completion_tokens: usize, + cached_tokens: Option, + ) -> Self { + Self { + prompt_tokens, + total_tokens: prompt_tokens + completion_tokens, + completion_tokens: Some(completion_tokens), + prompt_tokens_details: cached_tokens + .filter(|&c| c > 0) + .map(|c| PromptTokenUsageInfo { cached_tokens: c }), + } + } + + pub fn from_token_usage(usage: TokenUsage, enable_prompt_tokens_details: bool) -> Self { + Self::from_counts( + usage.prompt_token_count, + usage.output_token_count, + enable_prompt_tokens_details.then_some(usage.cached_token_count), + ) + } +} + +/// Mirrors the Python vLLM `PromptTokenUsageInfo` class. +#[derive(Debug, Clone, Serialize)] +pub struct PromptTokenUsageInfo { + pub cached_tokens: usize, +} + +#[cfg(test)] +mod usage_tests { + use vllm_llm::TokenUsage; + + use super::Usage; + + #[test] + fn token_usage_hides_prompt_token_details_by_default() { + let usage = Usage::from_token_usage( + TokenUsage { + prompt_token_count: 5, + output_token_count: 2, + cached_token_count: 3, + }, + false, + ); + + assert_eq!(usage.prompt_tokens, 5); + assert_eq!(usage.completion_tokens, Some(2)); + assert!(usage.prompt_tokens_details.is_none()); + } + + #[test] + fn token_usage_includes_prompt_token_details_when_enabled() { + let usage = Usage::from_token_usage( + TokenUsage { + prompt_token_count: 5, + output_token_count: 2, + cached_token_count: 3, + }, + true, + ); + + assert_eq!( + usage.prompt_tokens_details.as_ref().map(|details| details.cached_tokens), + Some(3) + ); + } +} + +/// OpenAI completions-style logprobs. +#[derive(Debug, Clone, Deserialize, Serialize)] +pub struct LogProbs { + pub tokens: Vec, + pub token_logprobs: Vec>, + pub top_logprobs: Vec>>, + pub text_offset: Vec, +} + +/// Mirrors the Python vLLM `ChatCompletionLogProbs` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub struct ChatLogProbs { + pub content: Option>, +} + +/// Mirrors the Python vLLM `ChatCompletionLogProbsContent` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub struct ChatLogProbsContent { + pub token: String, + pub logprob: f32, + pub bytes: Option>, + pub top_logprobs: Vec, +} + +/// Mirrors the Python vLLM `ChatCompletionLogProb` class. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Serialize)] +pub struct TopLogProb { + pub token: String, + pub logprob: f32, + pub bytes: Option>, +} + +// ============================================================================ +// Error Types +// ============================================================================ + +#[derive(Debug, Clone, Deserialize, Serialize)] +pub struct ErrorResponse { + pub error: ErrorDetail, +} + +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Deserialize, Serialize)] +pub struct ErrorDetail { + pub message: String, + #[serde(rename = "type")] + pub error_type: String, + pub param: Option, + pub code: Option, +} + +// ============================================================================ +// Model types +// ============================================================================ + +/// A single model entry in the `/v1/models` response. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ModelObject { + pub id: String, + pub object: String, + pub created: i64, + pub owned_by: String, + /// Backend model path (base cards) or adapter path (LoRA cards). + pub root: Option, + /// Base model a LoRA adapter derives from; `null` for base models. + pub parent: Option, + /// Maximum context length; `null` for LoRA adapter cards. + pub max_model_len: Option, +} + +/// Response body for `GET /v1/models`. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ListModelsResponse { + pub object: String, + pub data: Vec, +} + +// ============================================================================ +// Shared validation helpers +// ============================================================================ + +/// Validates a messages array is non-empty and has valid user-message content. +/// +/// Used by both `POST /v1/chat/completions` and `POST /tokenize` (chat form) +/// so validation behaviour stays in lockstep. +pub(crate) fn validate_messages( + messages: &[ChatMessage], +) -> Result<(), validator::ValidationError> { + if messages.is_empty() { + return Err(validator::ValidationError::new("messages cannot be empty")); + } + + for msg in messages { + if let ChatMessage::User { content, .. } = msg { + match content { + MessageContent::Text(text) if text.is_empty() => { + return Err(validator::ValidationError::new( + "message content cannot be empty", + )); + } + MessageContent::Parts(parts) if parts.is_empty() => { + return Err(validator::ValidationError::new( + "message content parts cannot be empty", + )); + } + _ => {} + } + } + } + Ok(()) +} + +// ============================================================================ +// Normalizable trait +// ============================================================================ + +/// Trait for request types that need post-deserialization normalization. +pub trait Normalizable { + /// Normalize the request by applying defaults and transformations. + fn normalize(&mut self) { + // Default: no-op + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/usage.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/usage.rs new file mode 100644 index 0000000000000000000000000000000000000000..4fcc38e560ce3d4609fe1a995dfb145290ec6811 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/usage.rs @@ -0,0 +1,35 @@ +use super::types::Usage; + +/// Tracks cumulative token counts for OpenAI streaming chunks. +/// +/// This helper is intentionally only a counter. Callers decide whether to +/// attach `counts()` to each streamed data chunk, while final usage-only chunks +/// should still be built from the authoritative terminal `TokenUsage`. +#[derive(Debug, Clone, Default)] +pub(crate) struct ContinuousUsage { + prompt_tokens: usize, + output_tokens: usize, +} + +impl ContinuousUsage { + /// Record the prompt-token count reported when a stream starts. + pub(crate) fn set_prompt_tokens(&mut self, prompt_tokens: usize) { + self.prompt_tokens = prompt_tokens; + } + + /// Add newly decoded output tokens to the running completion count. + pub(crate) fn add_output_tokens(&mut self, output_tokens: usize) { + self.output_tokens = self.output_tokens.saturating_add(output_tokens); + } + + /// Replace the running counts with the final counts reported by generation. + pub(crate) fn set_final_counts(&mut self, prompt_tokens: usize, output_tokens: usize) { + self.prompt_tokens = prompt_tokens; + self.output_tokens = output_tokens; + } + + /// Build a streaming usage snapshot without prompt cache details. + pub(crate) fn to_usage(&self) -> Usage { + Usage::from_counts(self.prompt_tokens, self.output_tokens, None) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/validated_json.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/validated_json.rs new file mode 100644 index 0000000000000000000000000000000000000000..229fc314da2676ddc7d3442e9dac38e167a357f5 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/openai/utils/validated_json.rs @@ -0,0 +1,55 @@ +//! Validated JSON extractor for automatic request validation. +//! Variation of https://github.com/lightseekorg/smg/blob/main/crates/protocols/src/validated.rs + +use axum::Json; +use axum::extract::rejection::JsonRejection; +use axum::extract::{FromRequest, Request}; +use serde::de::DeserializeOwned; +use validator::Validate; + +use super::types::Normalizable; +use crate::error::{ApiError, invalid_request}; + +/// A JSON extractor that automatically validates and normalizes the request +/// body. +/// +/// This extractor deserializes the request body and automatically calls +/// `.validate()` on types that implement the `Validate` trait. If validation +/// fails, it returns [`ApiError::InvalidRequest`] with details about the +/// validation errors. +pub struct ValidatedJson(pub T); + +impl FromRequest for ValidatedJson +where + T: DeserializeOwned + Validate + Normalizable + Send, + S: Send + Sync, +{ + type Rejection = ApiError; + + async fn from_request(req: Request, state: &S) -> Result { + let Json(mut data) = Json::::from_request(req, state) + .await + .map_err(|err: JsonRejection| ApiError::json_parse_error(err.body_text()))?; + + data.normalize(); + + data.validate() + .map_err(|validation_errors| invalid_request!("{}", validation_errors))?; + + Ok(ValidatedJson(data)) + } +} + +impl std::ops::Deref for ValidatedJson { + type Target = T; + + fn deref(&self) -> &Self::Target { + &self.0 + } +} + +impl std::ops::DerefMut for ValidatedJson { + fn deref_mut(&mut self) -> &mut Self::Target { + &mut self.0 + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/pause.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/pause.rs new file mode 100644 index 0000000000000000000000000000000000000000..98325beee21c571b7407d7516a65afbd020a937b --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/pause.rs @@ -0,0 +1,81 @@ +use std::sync::Arc; + +use axum::Json; +use axum::extract::rejection::QueryRejection; +use axum::extract::{Query, State}; +use serde::{Deserialize, Serialize}; +use vllm_engine_core_client::protocol::utility::PauseMode; + +use crate::error::ApiError; +use crate::state::AppState; +use crate::utils::utility_call_error; + +#[derive(Debug, Deserialize)] +pub(crate) struct PauseParams { + #[serde(default)] + mode: PauseMode, + #[serde(default = "default_clear_cache")] + clear_cache: bool, +} + +#[derive(Serialize)] +pub(crate) struct StatusResponse { + status: &'static str, +} + +#[derive(Serialize)] +pub(crate) struct IsPausedResponse { + is_paused: bool, +} + +const fn default_clear_cache() -> bool { + true +} + +fn invalid_query(error: QueryRejection) -> ApiError { + ApiError::invalid_request(error.body_text(), Some("mode")) +} + +// TODO: the Python frontend also accepts the deprecated +// `wait_for_inflight_requests` flag (equivalent to `mode="wait"`); it is +// intentionally omitted here in favor of the `mode` parameter. + +/// Pause the scheduler so generation can be halted (e.g. for weight updates). +pub async fn pause( + State(state): State>, + params: Result, QueryRejection>, +) -> Result, ApiError> { + let Query(params) = params.map_err(invalid_query)?; + + state + .engine_core_client() + .pause_scheduler(params.mode, params.clear_cache) + .await + .map_err(|error| utility_call_error("pause", error))?; + + Ok(Json(StatusResponse { status: "paused" })) +} + +/// Resume the scheduler after a pause. +pub async fn resume(State(state): State>) -> Result, ApiError> { + state + .engine_core_client() + .resume_scheduler() + .await + .map_err(|error| utility_call_error("resume", error))?; + + Ok(Json(StatusResponse { status: "resumed" })) +} + +/// Return whether the scheduler is currently paused. +pub async fn is_paused( + State(state): State>, +) -> Result, ApiError> { + let is_paused = state + .engine_core_client() + .is_scheduler_paused() + .await + .map_err(|error| utility_call_error("is_paused", error))?; + + Ok(Json(IsPausedResponse { is_paused })) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/profile.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/profile.rs new file mode 100644 index 0000000000000000000000000000000000000000..327ebff1cbc329699bb80a95860d4569d684e901 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/profile.rs @@ -0,0 +1,33 @@ +use std::sync::Arc; + +use axum::extract::State; +use axum::http::StatusCode; +use tracing::info; + +use crate::error::ApiError; +use crate::state::AppState; +use crate::utils::utility_call_error; + +/// Start profiling the engine. +pub async fn start_profile(State(state): State>) -> Result { + info!("starting profiler"); + state + .engine_core_client() + .start_profile(None) + .await + .map_err(|error| utility_call_error("start_profile", error))?; + info!("profiler started"); + Ok(StatusCode::OK) +} + +/// Stop profiling the engine. +pub async fn stop_profile(State(state): State>) -> Result { + info!("stopping profiler"); + state + .engine_core_client() + .stop_profile(None) + .await + .map_err(|error| utility_call_error("stop_profile", error))?; + info!("profiler stopped"); + Ok(StatusCode::OK) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/server_info.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/server_info.rs new file mode 100644 index 0000000000000000000000000000000000000000..44dd0c38cde4cc9506a49a0fa7947ef7ebf42209 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/server_info.rs @@ -0,0 +1,47 @@ +use std::sync::Arc; + +use axum::Json; +use axum::extract::{Query, State}; +use axum::http::StatusCode; +use axum::response::{IntoResponse, Response}; +use serde::Deserialize; + +use crate::server_info::ServerInfoConfigFormat; +use crate::state::AppState; + +#[derive(Debug, Clone, Copy, Deserialize)] +#[serde(rename_all = "lowercase")] +enum ConfigFormat { + Text, + Json, +} + +impl From for ServerInfoConfigFormat { + fn from(value: ConfigFormat) -> Self { + match value { + ConfigFormat::Text => Self::Text, + ConfigFormat::Json => Self::Json, + } + } +} + +fn default_config_format() -> ConfigFormat { + ConfigFormat::Text +} + +#[derive(Debug, Deserialize)] +pub(crate) struct ServerInfoParams { + #[serde(default = "default_config_format")] + config_format: ConfigFormat, +} + +/// Get server configuration and environment metadata. +pub async fn server_info( + State(state): State>, + Query(params): Query, +) -> Response { + match state.server_info_response(params.config_format.into()) { + Some(response) => Json(response).into_response(), + None => StatusCode::NOT_FOUND.into_response(), + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/sleep.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/sleep.rs new file mode 100644 index 0000000000000000000000000000000000000000..8331af76172d43606038c6396a0bf16f13df06bf --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/sleep.rs @@ -0,0 +1,82 @@ +use std::sync::Arc; + +use axum::Json; +use axum::extract::rejection::QueryRejection; +use axum::extract::{Query, State}; +use axum::http::StatusCode; +use serde::{Deserialize, Serialize}; +use vllm_engine_core_client::protocol::utility::PauseMode; + +use crate::error::ApiError; +use crate::state::AppState; +use crate::utils::utility_call_error; + +#[derive(Serialize)] +pub(crate) struct IsSleepingResponse { + is_sleeping: bool, +} + +#[derive(Debug, Deserialize)] +pub(crate) struct SleepParams { + #[serde(default = "default_sleep_level")] + level: u32, + #[serde(default)] + mode: PauseMode, +} + +#[derive(Debug, Default, Deserialize)] +pub(crate) struct WakeUpParams { + #[serde(default)] + tags: Option>, +} + +const fn default_sleep_level() -> u32 { + 1 +} + +fn invalid_query(error: QueryRejection) -> ApiError { + ApiError::invalid_request(error.body_text(), Some("mode")) +} + +/// Put the engine to sleep. +pub async fn sleep( + State(state): State>, + params: Result, QueryRejection>, +) -> Result { + let Query(params) = params.map_err(invalid_query)?; + + state + .engine_core_client() + .sleep(params.level, params.mode) + .await + .map_err(|error| utility_call_error("sleep", error))?; + + Ok(StatusCode::OK) +} + +/// Wake the engine from sleep mode. +pub async fn wake_up( + State(state): State>, + Query(params): Query, +) -> Result { + state + .engine_core_client() + .wake_up(params.tags) + .await + .map_err(|error| utility_call_error("wake_up", error))?; + + Ok(StatusCode::OK) +} + +/// Return whether the engine is currently sleeping at any level. +pub async fn is_sleeping( + State(state): State>, +) -> Result, ApiError> { + let is_sleeping = state + .engine_core_client() + .is_sleeping() + .await + .map_err(|error| utility_call_error("is_sleeping", error))?; + + Ok(Json(IsSleepingResponse { is_sleeping })) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/tests.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/tests.rs new file mode 100644 index 0000000000000000000000000000000000000000..6921df6604e9dcd0be9ddb51633245c8ffefe0ac --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/tests.rs @@ -0,0 +1,6282 @@ +// Route tests should use `Service::call` rather than `ServiceExt::oneshot`. +// `oneshot` consumes the router and can drop `AppState` before a streaming +// response body is fully drained, which closes the mock engine connection too +// early and causes flaky `closed unexpectedly` failures. + +use std::collections::BTreeSet; +use std::future::Future; +use std::pin::Pin; +use std::sync::Arc; +use std::task::{Context, Poll}; +use std::time::Duration; +use std::{fmt, fs}; + +use axum::body::{Body, to_bytes}; +use axum::http::{Request, StatusCode}; +use bytes::Bytes; +use rmpv::Value; +use serde_json::json; +use serial_test::serial; +use tower::{Service as _, ServiceExt as _}; +use vllm_chat::{ + ChatBackend, ChatContent, ChatContentPart, ChatLlm, ChatMessage, ChatRenderer, ChatRequest, + ChatTextBackend, DefaultChatOutputProcessor, DynChatOutputProcessor, DynChatRenderer, + NewChatOutputProcessorOptions, +}; +use vllm_engine_core_client::mock_engine::default_ready_response; +use vllm_engine_core_client::protocol::decode_value; +use vllm_engine_core_client::protocol::logprobs::{ + Logprobs, MaybeWireLogprobs, PositionLogprobs, TokenLogprob, +}; +use vllm_engine_core_client::protocol::output::{ + EngineCoreFinishReason, EngineCoreOutput, EngineCoreOutputs, RequestBatchOutputs, StopReason, + UtilityCallOutput, +}; +use vllm_engine_core_client::protocol::request::EngineCoreRequest; +use vllm_engine_core_client::protocol::utility::{UtilityOutput, UtilityResultEnvelope}; +use vllm_engine_core_client::test_utils::{ + IpcNamespace, spawn_mock_engine_task, spawn_mock_engine_task_with_ready, +}; +use vllm_engine_core_client::{ + ENGINE_CORE_DEAD_SENTINEL, EngineCoreClient, EngineCoreClientConfig, EngineId, +}; +use vllm_llm::Llm; +use vllm_metrics::METRICS; +use vllm_text::tokenizer::DynTokenizer; +use vllm_text::{Prompt, TextBackend}; +use vllm_tokenizer::test_utils::TestTokenizer; +use zeromq::prelude::{SocketRecv, SocketSend}; +use zeromq::{DealerSocket, PushSocket, ZmqMessage}; + +use super::{build_router, build_router_with_dev_mode, build_router_with_dev_mode_and_lora}; +use crate::config::{ApiServerOptions, CorsConfig}; +use crate::state::AppState; + +fn request_output( + request_id: &str, + new_token_ids: Vec, + finish_reason: Option, +) -> EngineCoreOutput { + request_output_with_stop_reason(request_id, new_token_ids, finish_reason, None) +} + +fn request_output_with_stop_reason( + request_id: &str, + new_token_ids: Vec, + finish_reason: Option, + stop_reason: Option, +) -> EngineCoreOutput { + EngineCoreOutput { + request_id: request_id.to_string(), + new_token_ids, + new_logprobs: None, + new_prompt_logprobs_tensors: None, + pooling_output: None, + finish_reason, + stop_reason, + events: None, + kv_transfer_params: None, + trace_headers: None, + prefill_stats: None, + routed_experts: None, + num_nans_in_logits: 0, + } +} + +fn request_output_with_logprobs( + request_id: &str, + new_token_ids: Vec, + finish_reason: Option, + stop_reason: Option, + new_logprobs: Option, + new_prompt_logprobs_tensors: Option, +) -> EngineCoreOutput { + EngineCoreOutput { + request_id: request_id.to_string(), + new_token_ids, + new_logprobs: new_logprobs.map(MaybeWireLogprobs::Direct), + new_prompt_logprobs_tensors: new_prompt_logprobs_tensors.map(MaybeWireLogprobs::Direct), + pooling_output: None, + finish_reason, + stop_reason, + events: None, + kv_transfer_params: None, + trace_headers: None, + prefill_stats: None, + routed_experts: None, + num_nans_in_logits: 0, + } +} + +fn request_output_with_logprobs_and_kv( + request_id: &str, + new_token_ids: Vec, + finish_reason: Option, + stop_reason: Option, + new_logprobs: Option, + new_prompt_logprobs_tensors: Option, + kv_transfer_params: Option, +) -> EngineCoreOutput { + EngineCoreOutput { + request_id: request_id.to_string(), + new_token_ids, + new_logprobs: new_logprobs.map(MaybeWireLogprobs::Direct), + new_prompt_logprobs_tensors: new_prompt_logprobs_tensors.map(MaybeWireLogprobs::Direct), + pooling_output: None, + finish_reason, + stop_reason, + events: None, + kv_transfer_params, + trace_headers: None, + prefill_stats: None, + routed_experts: None, + num_nans_in_logits: 0, + } +} + +fn bytes_to_token_ids(bytes: &[u8]) -> Vec { + bytes.iter().map(|byte| u32::from(*byte)).collect() +} + +fn default_stream_output_specs() -> Vec<(Vec, Option)> { + vec![ + (vec![b'h' as u32], None), + (vec![b'i' as u32], None), + (vec![b'!' as u32], Some(EngineCoreFinishReason::Stop)), + ] +} + +fn assert_adapter_a_lora_request(request: &EngineCoreRequest) { + let lora = request.lora_request.as_ref().expect("lora request"); + assert_eq!(lora.lora_name, "adapter-a"); + assert_eq!(lora.lora_int_id, 1); + assert_eq!(lora.lora_path, "org/adapter-a"); +} + +fn sse_data_payloads(text: &str) -> Vec<&str> { + text.lines().filter_map(|line| line.strip_prefix("data: ")).collect() +} + +fn sse_json_payloads(text: &str) -> Vec { + sse_data_payloads(text) + .into_iter() + .filter(|payload| *payload != "[DONE]") + .map(|payload| serde_json::from_str(payload).expect("sse json payload")) + .collect() +} + +type TestFuture<'a> = Pin + Send + 'a>>; + +fn boxed_test_future<'a>(future: impl Future + Send + 'a) -> TestFuture<'a> { + Box::pin(future) +} + +struct MockEngineTask { + shutdown_tx: Option>, + join_handle: Option>, +} + +impl MockEngineTask { + fn new( + (shutdown_tx, join_handle): ( + tokio::sync::oneshot::Sender<()>, + tokio::task::JoinHandle<()>, + ), + ) -> Self { + Self { + shutdown_tx: Some(shutdown_tx), + join_handle: Some(join_handle), + } + } + + async fn finish(self) { + self.await.expect("mock engine task"); + } + + fn abort(&self) { + if let Some(join_handle) = &self.join_handle { + join_handle.abort(); + } + } + + async fn abort_and_join(mut self) { + if let Some(join_handle) = self.join_handle.take() { + join_handle.abort(); + let _ = join_handle.await; + } + } +} + +impl Future for MockEngineTask { + type Output = Result<(), tokio::task::JoinError>; + + fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { + if let Some(shutdown_tx) = self.shutdown_tx.take() { + let _ = shutdown_tx.send(()); + } + match self.join_handle.as_mut() { + Some(join_handle) => Pin::new(join_handle).poll(cx), + None => Poll::Ready(Ok(())), + } + } +} + +impl Drop for MockEngineTask { + fn drop(&mut self) { + if let Some(join_handle) = &self.join_handle { + join_handle.abort(); + } + } +} + +fn engine_outputs_for_request( + request_id: &str, + output_specs: Vec<(Vec, Option)>, +) -> EngineCoreOutputs { + RequestBatchOutputs { + outputs: output_specs + .into_iter() + .map(|(token_ids, finish_reason)| request_output(request_id, token_ids, finish_reason)) + .collect(), + ..Default::default() + } + .into() +} + +fn test_llm(client: EngineCoreClient) -> Llm { + Llm::new(client).with_request_id_randomization(false) +} + +fn sample_logprobs_for_token(token_id: u32, alternate_token_id: u32) -> Logprobs { + Logprobs { + positions: vec![PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id, + logprob: -0.1, + rank: 1, + }, + TokenLogprob { + token_id: alternate_token_id, + logprob: -0.2, + rank: 1, + }, + ], + }], + } +} + +fn sample_logprobs_for_tokens(token_ids: &[u32]) -> Logprobs { + Logprobs { + positions: token_ids + .iter() + .map(|&token_id| PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id, + logprob: -0.1, + rank: 1, + }, + TokenLogprob { + token_id: token_id.saturating_add(1), + logprob: -0.2, + rank: 2, + }, + ], + }) + .collect(), + } +} + +fn prompt_logprobs_for_hello() -> Logprobs { + Logprobs { + positions: vec![ + PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id: b'e' as u32, + logprob: -0.3, + rank: 1, + }, + TokenLogprob { + token_id: b'a' as u32, + logprob: -0.5, + rank: 1, + }, + ], + }, + PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id: b'l' as u32, + logprob: -0.4, + rank: 1, + }, + TokenLogprob { + token_id: b'r' as u32, + logprob: -0.6, + rank: 1, + }, + ], + }, + PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id: b'l' as u32, + logprob: -0.45, + rank: 1, + }, + TokenLogprob { + token_id: b'i' as u32, + logprob: -0.65, + rank: 1, + }, + ], + }, + PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id: b'o' as u32, + logprob: -0.5, + rank: 1, + }, + TokenLogprob { + token_id: b'u' as u32, + logprob: -0.7, + rank: 1, + }, + ], + }, + ], + } +} + +fn prompt_logprobs_for_tokens(token_ids: &[u32]) -> Logprobs { + Logprobs { + positions: token_ids + .iter() + .skip(1) + .map(|&token_id| PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id, + logprob: -0.3, + rank: 1, + }, + TokenLogprob { + token_id: token_id.saturating_add(1), + logprob: -0.5, + rank: 2, + }, + ], + }) + .collect(), + } +} + +fn utility_result_value(value: T) -> UtilityResultEnvelope +where + T: serde::Serialize, +{ + UtilityResultEnvelope::without_type_info(rmpv::ext::to_value(value).expect("encode result")) +} + +fn utility_none_result() -> UtilityResultEnvelope { + UtilityResultEnvelope::without_type_info(Value::Nil) +} + +fn utility_outputs(call_id: u64, result: UtilityResultEnvelope) -> EngineCoreOutputs { + UtilityCallOutput { + output: UtilityOutput { + call_id: call_id.into(), + failure_message: None, + result: Some(result), + }, + ..Default::default() + } + .into() +} + +async fn send_outputs(push: &mut PushSocket, outputs: EngineCoreOutputs) { + push.send(ZmqMessage::from( + rmp_serde::to_vec_named(&outputs).expect("encode outputs"), + )) + .await + .expect("send outputs"); +} + +async fn recv_engine_message(dealer: &mut DealerSocket) -> Vec { + dealer.recv().await.expect("recv engine message").into_vec() +} + +#[derive(Clone)] +struct FakeChatBackend { + model_id: String, + multimodal_model_info: Option, +} + +/// Synthetic BOS id used when `add_special_tokens` is true in tests. +const FAKE_BOS_TOKEN_ID: u32 = 256; +const UNKNOWN_DECODE_TOKEN_ID: u32 = 10_000; + +fn fake_chat_tokenizer() -> TestTokenizer { + TestTokenizer::new() + .with_bos_token("", FAKE_BOS_TOKEN_ID) + .with_regular_token("", 999) + .with_regular_token("<|image_pad|>", 151655) + .with_regular_token("", 0xF001) + .with_regular_token("", 0xF002) + .with_regular_token("<|START_THINKING|>", 0xF003) + .with_regular_token("<|END_THINKING|>", 0xF004) + .with_regular_token("◁think▷", 0xF005) + .with_regular_token("◁/think▷", 0xF006) +} + +impl FakeChatBackend { + fn new() -> Self { + Self { + model_id: "test-model".to_string(), + multimodal_model_info: None, + } + } + + fn with_model_id(model_id: impl Into) -> Self { + Self { + model_id: model_id.into(), + multimodal_model_info: None, + } + } + + fn with_multimodal_model_info( + multimodal_model_info: vllm_chat::multimodal::MultimodalModelInfo, + ) -> Self { + Self { + model_id: "test-model".to_string(), + multimodal_model_info: Some(multimodal_model_info), + } + } +} + +impl fmt::Debug for FakeChatBackend { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("FakeChatBackend") + .field("model_id", &self.model_id) + .finish_non_exhaustive() + } +} + +impl TextBackend for FakeChatBackend { + fn tokenizer(&self) -> DynTokenizer { + Arc::new(fake_chat_tokenizer()) + } + + fn model_id(&self) -> &str { + &self.model_id + } +} + +impl ChatBackend for FakeChatBackend { + fn chat_renderer(&self) -> DynChatRenderer { + Arc::new(self.clone()) + } + + fn multimodal_model_info(&self) -> Option<&vllm_chat::multimodal::MultimodalModelInfo> { + self.multimodal_model_info.as_ref() + } + + fn new_chat_output_processor( + &self, + request: &mut ChatRequest, + options: NewChatOutputProcessorOptions<'_>, + ) -> vllm_chat::Result { + Ok(Box::new(DefaultChatOutputProcessor::new( + request, + &self.model_id, + self.tokenizer(), + options.tool_call_parser, + options.reasoning_parser, + )?)) + } +} + +impl ChatRenderer for FakeChatBackend { + fn render(&self, request: &ChatRequest) -> vllm_chat::Result { + let placeholder = self + .multimodal_model_info + .as_ref() + .map(|info| info.placeholder_token()) + .unwrap_or(""); + let mut prompt = String::new(); + for message in &request.messages { + prompt.push_str(message.role().as_str()); + prompt.push_str(": "); + prompt.push_str(&render_fake_message_content(message, placeholder)?); + prompt.push('\n'); + } + if request.chat_options.add_generation_prompt() { + prompt.push_str("assistant:"); + } + Ok(vllm_chat::RenderedPrompt { + prompt: Prompt::Text(prompt), + effective_template_kwargs: Default::default(), + }) + } +} + +fn render_fake_message_content( + message: &ChatMessage, + placeholder: &str, +) -> vllm_chat::Result { + match message { + ChatMessage::System { content } + | ChatMessage::Developer { content, .. } + | ChatMessage::User { content } + | ChatMessage::ToolResponse { content, .. } => render_fake_content(content, placeholder), + ChatMessage::Assistant { .. } => message.text_content(), + } +} + +fn render_fake_content(content: &ChatContent, placeholder: &str) -> vllm_chat::Result { + Ok(match content { + ChatContent::Text(text) => text.clone(), + ChatContent::Parts(parts) => { + let mut out = String::new(); + for part in parts { + match part { + ChatContentPart::Text { text } => out.push_str(text), + ChatContentPart::ImageUrl { .. } => out.push_str(placeholder), + } + } + out + } + }) +} + +fn qwen_multimodal_model_info() -> vllm_chat::multimodal::MultimodalModelInfo { + let config_path = std::env::temp_dir().join(format!( + "vllm-server-qwen-config-{}.json", + uuid::Uuid::new_v4() + )); + fs::write( + &config_path, + r#"{"model_type":"qwen2_vl","image_token_id":151655}"#, + ) + .expect("write qwen test config"); + let info = vllm_chat::multimodal::MultimodalModelInfo::from_paths( + "qwen2-vl-test".to_string(), + Some("qwen2_vl".to_string()), + Some(&config_path), + None, + Arc::new(fake_chat_tokenizer()), + ) + .expect("load multimodal info") + .expect("qwen multimodal info is registered"); + let _ = fs::remove_file(config_path); + info +} + +async fn test_models_with_engine_outputs_and_backend_inner( + engine_id: impl Into, + output_specs: Vec<(Vec, Option)>, + expected_prompt_token_ids: Option>, + backend: Arc, +) -> (ChatLlm, MockEngineTask) { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = engine_id.into(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + move |dealer, push| { + boxed_test_future(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + if let Some(expected_prompt_token_ids) = expected_prompt_token_ids { + assert_eq!( + request.prompt_token_ids.as_deref(), + Some(expected_prompt_token_ids.as_slice()) + ); + } + send_outputs( + push, + engine_outputs_for_request(&request.request_id, output_specs), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + + ( + ChatLlm::from_shared_backend(test_llm(client), backend), + engine_task, + ) +} + +async fn test_models_with_engine_outputs_and_backend( + engine_id: impl Into, + output_specs: Vec<(Vec, Option)>, + backend: Arc, +) -> (ChatLlm, MockEngineTask) { + test_models_with_engine_outputs_and_backend_inner(engine_id, output_specs, None, backend).await +} + +async fn test_chat_with_engine_outputs( + engine_id: impl Into, + output_specs: Vec<(Vec, Option)>, +) -> (ChatLlm, MockEngineTask) { + test_models_with_engine_outputs_and_backend( + engine_id, + output_specs, + Arc::new(FakeChatBackend::new()), + ) + .await +} + +async fn test_app() -> axum::Router { + test_app_with_dev_mode(false).await +} + +async fn test_app_with_dev_mode(dev_mode_enabled: bool) -> axum::Router { + let (chat, _engine_task) = test_models_with_engine_outputs_and_backend( + b"engine-openai", + default_stream_output_specs(), + Arc::new(FakeChatBackend::new()), + ) + .await; + build_router_with_dev_mode( + Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + )), + dev_mode_enabled, + ) +} + +/// Build a dev-mode router backed by a mock engine using a custom ready +/// response, returning the router and the engine task handle so the engine +/// stays alive for the duration of the test. +async fn test_dev_mode_app_with_ready( + ready_response: vllm_engine_core_client::protocol::handshake::EngineCoreReadyResponse, +) -> (axum::Router, MockEngineTask) { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-world-size".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task_with_ready( + handshake_address.clone(), + engine_id.clone(), + ready_response, + |_dealer, _push| boxed_test_future(async {}), + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + + let chat = ChatLlm::from_shared_backend(test_llm(client), Arc::new(FakeChatBackend::new())); + let app = build_router_with_dev_mode( + Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + )), + true, + ); + (app, engine_task) +} + +async fn test_app_with_request_id_headers() -> (axum::Router, MockEngineTask) { + let (chat, engine_task) = test_models_with_engine_outputs_and_backend( + b"engine-openai-request-id", + default_stream_output_specs(), + Arc::new(FakeChatBackend::new()), + ) + .await; + let app = build_router(Arc::new( + AppState::new(vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], chat).with_api_server_options( + ApiServerOptions { + enable_request_id_headers: true, + ..Default::default() + }, + ), + )); + (app, engine_task) +} + +async fn test_app_with_api_keys(api_keys: Vec) -> (axum::Router, MockEngineTask) { + let (chat, engine_task) = test_models_with_engine_outputs_and_backend( + b"engine-openai-api-key", + default_stream_output_specs(), + Arc::new(FakeChatBackend::new()), + ) + .await; + let app = build_router(Arc::new( + AppState::new(vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], chat).with_api_keys(api_keys), + )); + (app, engine_task) +} + +async fn test_app_with_cors_and_keys( + cors: CorsConfig, + api_keys: Vec, +) -> (axum::Router, MockEngineTask) { + let (chat, engine_task) = test_models_with_engine_outputs_and_backend( + b"engine-openai-cors", + default_stream_output_specs(), + Arc::new(FakeChatBackend::new()), + ) + .await; + let app = build_router(Arc::new( + AppState::new(vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], chat) + .with_cors(cors) + .with_api_keys(api_keys), + )); + (app, engine_task) +} + +async fn test_app_with_cors(cors: CorsConfig) -> (axum::Router, MockEngineTask) { + test_app_with_cors_and_keys(cors, vec![]).await +} + +fn header_value<'a>(response: &'a axum::response::Response, name: &str) -> Option<&'a str> { + response + .headers() + .get(name) + .map(|value| value.to_str().expect("header is valid utf-8")) +} + +async fn test_health_app_with_engine_script( + script: F, +) -> (axum::Router, Arc, MockEngineTask) +where + F: for<'a> FnOnce(&'a mut PushSocket) -> TestFuture<'a> + Send + 'static, +{ + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-openai-health".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + move |_dealer, push| script(push), + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + + let chat = ChatLlm::from_shared_backend(test_llm(client), Arc::new(FakeChatBackend::new())); + let state = Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + )); + (build_router(state.clone()), state, engine_task) +} + +async fn test_admin_app_with_engine_script(script: F) -> (axum::Router, MockEngineTask) +where + F: for<'a> FnOnce(&'a mut DealerSocket, &'a mut PushSocket) -> TestFuture<'a> + Send + 'static, +{ + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-openai-admin".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + move |dealer, push| script(dealer, push), + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + + let chat = ChatLlm::from_shared_backend(test_llm(client), Arc::new(FakeChatBackend::new())); + ( + build_router_with_dev_mode_and_lora( + Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + )), + true, + true, + ), + engine_task, + ) +} + +async fn test_app_with_engine_handle() -> (axum::Router, MockEngineTask) { + test_app_with_stream_output_specs(default_stream_output_specs()).await +} + +async fn test_app_with_stream_output_specs( + output_specs: Vec<(Vec, Option)>, +) -> (axum::Router, MockEngineTask) { + let (chat, engine_task) = test_models_with_engine_outputs_and_backend( + b"engine-openai", + output_specs, + Arc::new(FakeChatBackend::new()), + ) + .await; + ( + build_router(Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + ))), + engine_task, + ) +} + +async fn test_app_with_backend_and_stream_output_specs( + backend: Arc, + output_specs: Vec<(Vec, Option)>, +) -> (axum::Router, MockEngineTask) { + let (chat, engine_task) = + test_models_with_engine_outputs_and_backend(b"engine-openai", output_specs, backend).await; + ( + build_router(Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + ))), + engine_task, + ) +} + +async fn test_app_with_backend_and_engine_request_check( + backend: Arc, + check_request: F, +) -> (axum::Router, MockEngineTask) +where + F: FnOnce(&EngineCoreRequest) + Send + 'static, +{ + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-openai-check-request".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + move |dealer, push| { + boxed_test_future(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + check_request(&request); + send_outputs( + push, + engine_outputs_for_request(&request.request_id, default_stream_output_specs()), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + + let chat = ChatLlm::from_shared_backend(test_llm(client), backend); + ( + build_router(Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + ))), + engine_task, + ) +} + +async fn test_chat_with_engine_handle() -> (ChatLlm, MockEngineTask) { + test_chat_with_engine_outputs(b"engine-openai-chat", default_stream_output_specs()).await +} + +async fn server_load(app: &axum::Router) -> u64 { + let response = app + .clone() + .call( + Request::builder() + .method("GET") + .uri("/load") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let value: serde_json::Value = serde_json::from_slice(&body).expect("json body"); + value["server_load"].as_u64().expect("server_load") +} + +async fn health_status(app: &axum::Router) -> (StatusCode, Bytes) { + let response = app + .clone() + .call( + Request::builder() + .method("GET") + .uri("/health") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + let status = response.status(); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + (status, body) +} + +async fn health_response(app: &axum::Router, request_id: Option<&str>) -> axum::response::Response { + let mut builder = Request::builder().method("GET").uri("/health"); + if let Some(request_id) = request_id { + builder = builder.header("X-Request-Id", request_id); + } + + app.clone() + .call(builder.body(Body::empty()).expect("build request")) + .await + .expect("call app") +} + +fn metric_value(rendered: &str, metric: &str, labels: Option<&str>) -> Option { + rendered.lines().find_map(|line| { + let rest = line.strip_prefix(metric)?; + + match labels { + Some(labels) => { + let (encoded_labels, value) = rest.split_once("} ")?; + if !encoded_labels.starts_with('{') { + return None; + } + let expected_parts = labels.split(','); + if expected_parts.into_iter().all(|part| encoded_labels.contains(part)) { + value.parse::().ok() + } else { + None + } + } + None => rest.strip_prefix(' ').and_then(|value| value.parse::().ok()), + } + }) +} + +fn metric_delta( + rendered_before: &str, + rendered_after: &str, + metric: &str, + labels: Option<&str>, +) -> f64 { + metric_value(rendered_after, metric, labels).unwrap_or(0.0) + - metric_value(rendered_before, metric, labels).unwrap_or(0.0) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn list_models_returns_configured_model() { + let mut app = test_app().await; + let response = app + .call(Request::builder().uri("/v1/models").body(Body::empty()).expect("build request")) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json["data"][0]["id"], "Qwen/Qwen1.5-0.5B-Chat"); + // No model path configured: `root` falls back to the served name. + assert_eq!(json["data"][0]["root"], "Qwen/Qwen1.5-0.5B-Chat"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn list_models_base_card_includes_metadata() { + let (chat, _engine_task) = test_models_with_engine_outputs_and_backend( + b"engine-openai-models-meta", + default_stream_output_specs(), + Arc::new(FakeChatBackend::new()), + ) + .await; + // `id` is the served alias; `root` is the underlying model path. + let mut app = build_router(Arc::new( + AppState::new(vec!["public-alias".to_string()], chat) + .with_model_path("org/backend-model".to_string()), + )); + + let response = app + .call(Request::builder().uri("/v1/models").body(Body::empty()).expect("build request")) + .await + .expect("call app"); + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + let card = json["data"][0].as_object().expect("card object"); + assert_eq!(card["id"], "public-alias"); + assert_eq!(card["owned_by"], "vllm-frontend-rs"); + assert_eq!(card["root"], "org/backend-model"); + assert!(card["max_model_len"].as_u64().expect("max_model_len") > 0); + assert!(card["created"].as_i64().expect("created") > 0); + // `parent` must be emitted as null, not omitted. + assert!(card.contains_key("parent") && card["parent"].is_null()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn list_models_lists_loras_in_load_order() { + // Load out of lexicographic order; the list must preserve load order, not sort. + let (mut app, _engine_task) = test_admin_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + for _ in 0..2 { + let utility = recv_engine_message(dealer).await; + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let call_id = + payload.as_array().expect("utility array")[1].as_u64().expect("call id"); + send_outputs(push, utility_outputs(call_id, utility_result_value(true))).await; + } + }) + }) + .await; + + for name in ["zebra", "alpha"] { + let path = format!("org/{name}"); + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/load_lora_adapter") + .header("content-type", "application/json") + .body(Body::from( + json!({ "lora_name": name, "lora_path": path }).to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + assert_eq!(response.status(), StatusCode::OK); + } + + let models = app + .call(Request::builder().uri("/v1/models").body(Body::empty()).expect("build request")) + .await + .expect("call app"); + let body = to_bytes(models.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + assert_eq!(json["data"][0]["id"], "Qwen/Qwen1.5-0.5B-Chat"); + assert_eq!(json["data"][1]["id"], "zebra"); + assert_eq!(json["data"][2]["id"], "alpha"); + // `max_model_len` must be emitted as null on LoRA cards, not omitted. + let lora_card = json["data"][1].as_object().expect("lora card object"); + assert_eq!(lora_card["root"], "org/zebra"); + assert_eq!(lora_card["parent"], "Qwen/Qwen1.5-0.5B-Chat"); + assert!(lora_card.contains_key("max_model_len") && lora_card["max_model_len"].is_null()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn request_id_header_is_absent_by_default() { + let app = test_app().await; + let response = health_response(&app, None).await; + + assert_eq!(response.status(), StatusCode::OK); + assert!(!response.headers().contains_key("x-request-id")); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn request_id_header_generates_uuid_hex_when_enabled() { + let (app, _engine_task) = test_app_with_request_id_headers().await; + let response = health_response(&app, None).await; + + assert_eq!(response.status(), StatusCode::OK); + let request_id = response + .headers() + .get("x-request-id") + .expect("x-request-id header") + .to_str() + .expect("header is ascii"); + assert_eq!(request_id.len(), 32); + assert!(request_id.chars().all(|ch| ch.is_ascii_hexdigit())); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn request_id_header_echoes_incoming_header_when_enabled() { + let (app, _engine_task) = test_app_with_request_id_headers().await; + let response = health_response(&app, Some("req-123")).await; + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!(response.headers().get("x-request-id").unwrap(), "req-123"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn api_key_auth_rejects_missing_token_on_guarded_route() { + let (mut app, _engine_task) = test_app_with_api_keys(vec!["secret".to_string()]).await; + let response = app + .call( + Request::builder() + .method("GET") + .uri("/v1/models") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::UNAUTHORIZED); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("json body"); + assert_eq!(json, json!({ "error": "Unauthorized" })); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn api_key_auth_rejects_wrong_token_on_guarded_route() { + let (mut app, _engine_task) = test_app_with_api_keys(vec!["secret".to_string()]).await; + let response = app + .call( + Request::builder() + .method("GET") + .uri("/v1/models") + .header("authorization", "Bearer wrong") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::UNAUTHORIZED); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn api_key_auth_accepts_matching_bearer_token_on_guarded_route() { + let (mut app, _engine_task) = test_app_with_api_keys(vec!["secret".to_string()]).await; + let response = app + .call( + Request::builder() + .method("GET") + .uri("/v1/models") + .header("authorization", "Bearer secret") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn api_key_auth_allows_options_without_token() { + let (mut app, _engine_task) = test_app_with_api_keys(vec!["secret".to_string()]).await; + let response = app + .call( + Request::builder() + .method("OPTIONS") + .uri("/v1/models") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_ne!(response.status(), StatusCode::UNAUTHORIZED); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn api_key_auth_allows_unguarded_route_without_token() { + let (mut app, _engine_task) = test_app_with_api_keys(vec!["secret".to_string()]).await; + let response = app + .call( + Request::builder() + .method("GET") + .uri("/health") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn cors_default_simple_request_allows_any_origin() { + let (mut app, _engine_task) = test_app_with_cors(CorsConfig::default()).await; + let response = app + .call( + Request::builder() + .method("GET") + .uri("/v1/models") + .header("origin", "http://example.com") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + header_value(&response, "access-control-allow-origin"), + Some("*") + ); + // Wildcard origins without credentials emit no `Vary` (Starlette parity). + assert_eq!(header_value(&response, "vary"), None); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn cors_default_preflight_returns_explicit_methods_and_max_age() { + let (mut app, _engine_task) = test_app_with_cors(CorsConfig::default()).await; + let response = app + .call( + Request::builder() + .method("OPTIONS") + .uri("/v1/chat/completions") + .header("origin", "http://example.com") + .header("access-control-request-method", "POST") + .header("access-control-request-headers", "content-type") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + // `*` methods expand to the explicit method list, matching Starlette + // (never the literal `*`). + assert_eq!( + header_value(&response, "access-control-allow-methods"), + Some("DELETE,GET,HEAD,OPTIONS,PATCH,POST,PUT") + ); + assert_eq!( + header_value(&response, "access-control-max-age"), + Some("600") + ); + // `*` headers mirror the requested headers. + assert_eq!( + header_value(&response, "access-control-allow-headers"), + Some("content-type") + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn cors_no_origin_request_has_no_cors_headers() { + let (mut app, _engine_task) = test_app_with_cors(CorsConfig::default()).await; + let response = app + .call( + Request::builder() + .method("GET") + .uri("/health") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!(header_value(&response, "access-control-allow-origin"), None); + assert_eq!(header_value(&response, "vary"), None); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn cors_explicit_origin_allowed_reflects_origin_with_vary() { + let cors = CorsConfig { + allow_origins: vec!["http://allowed.com".to_string()], + ..CorsConfig::default() + }; + let (mut app, _engine_task) = test_app_with_cors(cors).await; + let response = app + .call( + Request::builder() + .method("GET") + .uri("/v1/models") + .header("origin", "http://allowed.com") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!( + header_value(&response, "access-control-allow-origin"), + Some("http://allowed.com") + ); + assert_eq!(header_value(&response, "vary"), Some("origin")); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn cors_explicit_origin_disallowed_omits_allow_origin() { + let cors = CorsConfig { + allow_origins: vec!["http://allowed.com".to_string()], + ..CorsConfig::default() + }; + let (mut app, _engine_task) = test_app_with_cors(cors).await; + let response = app + .call( + Request::builder() + .method("GET") + .uri("/v1/models") + .header("origin", "http://evil.com") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(header_value(&response, "access-control-allow-origin"), None); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn cors_wildcard_with_credentials_reflects_origin_without_panic() { + let cors = CorsConfig { + allow_credentials: true, + ..CorsConfig::default() + }; + let (mut app, _engine_task) = test_app_with_cors(cors).await; + let response = app + .call( + Request::builder() + .method("GET") + .uri("/v1/models") + .header("origin", "http://example.com") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + // `*` + credentials reflects the request origin instead of `*` (Starlette + // parity, and avoids tower-http's wildcard+credentials panic). + assert_eq!( + header_value(&response, "access-control-allow-origin"), + Some("http://example.com") + ); + assert_eq!( + header_value(&response, "access-control-allow-credentials"), + Some("true") + ); + assert_eq!(header_value(&response, "vary"), Some("origin")); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn cors_unauthorized_response_has_no_cors_headers() { + let (mut app, _engine_task) = + test_app_with_cors_and_keys(CorsConfig::default(), vec!["secret".to_string()]).await; + let response = app + .call( + Request::builder() + .method("GET") + .uri("/v1/models") + .header("origin", "http://example.com") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + // Auth sits outside CORS, so a 401 carries no CORS headers. + assert_eq!(response.status(), StatusCode::UNAUTHORIZED); + assert_eq!(header_value(&response, "access-control-allow-origin"), None); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn cors_preflight_bypasses_auth_and_returns_cors_headers() { + let (mut app, _engine_task) = + test_app_with_cors_and_keys(CorsConfig::default(), vec!["secret".to_string()]).await; + let response = app + .call( + Request::builder() + .method("OPTIONS") + .uri("/v1/chat/completions") + .header("origin", "http://example.com") + .header("access-control-request-method", "POST") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_ne!(response.status(), StatusCode::UNAUTHORIZED); + assert_eq!( + header_value(&response, "access-control-allow-origin"), + Some("*") + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn cors_explicit_methods_preflight_returns_that_list() { + let cors = CorsConfig { + allow_methods: vec!["GET".to_string(), "POST".to_string()], + ..CorsConfig::default() + }; + let (mut app, _engine_task) = test_app_with_cors(cors).await; + let response = app + .call( + Request::builder() + .method("OPTIONS") + .uri("/v1/chat/completions") + .header("origin", "http://example.com") + .header("access-control-request-method", "POST") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + // Explicit methods are emitted verbatim, not expanded and not `*`. + assert_eq!( + header_value(&response, "access-control-allow-methods"), + Some("GET,POST") + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn cors_explicit_headers_union_safelisted_headers() { + let cors = CorsConfig { + allow_headers: vec!["X-Custom".to_string()], + ..CorsConfig::default() + }; + let (mut app, _engine_task) = test_app_with_cors(cors).await; + let response = app + .call( + Request::builder() + .method("OPTIONS") + .uri("/v1/chat/completions") + .header("origin", "http://example.com") + .header("access-control-request-method", "POST") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + // Explicit headers are unioned with the safelisted set, lowercased + sorted. + assert_eq!( + header_value(&response, "access-control-allow-headers"), + Some("accept,accept-language,content-language,content-type,x-custom") + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn version_returns_engine_vllm_version() { + let mut app = test_app().await; + let response = app + .call(Request::builder().uri("/version").body(Body::empty()).expect("build request")) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!( + json, + json!({ + "version": "test-vllm-version", + "rust_frontend_version": env!("CARGO_PKG_VERSION"), + }) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn server_info_endpoint_is_dev_mode_only() { + let mut app = test_app().await; + let response = app + .call( + Request::builder() + .uri("/server_info") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::NOT_FOUND); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn load_lora_adapter_registers_model_and_forwards_lora_request() { + let (mut app, engine_task) = test_admin_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + assert_eq!(array[2], Value::from("add_lora")); + + let args = array[3].as_array().expect("utility args"); + let lora = args[0].as_array().expect("lora request tuple"); + assert_eq!(lora[0], Value::from("adapter-a")); + assert_eq!(lora[1], Value::from(1)); + assert_eq!(lora[2], Value::from("org/adapter-a")); + + send_outputs(push, utility_outputs(call_id, utility_result_value(true))).await; + + let add = recv_engine_message(dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode engine request"); + assert_adapter_a_lora_request(&request); + + send_outputs( + push, + engine_outputs_for_request(&request.request_id, default_stream_output_specs()), + ) + .await; + + let add = recv_engine_message(dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode engine request"); + assert_adapter_a_lora_request(&request); + + send_outputs( + push, + engine_outputs_for_request(&request.request_id, default_stream_output_specs()), + ) + .await; + + let add = recv_engine_message(dealer).await; + assert_eq!(add[0].as_ref(), &[0x00]); + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode engine request"); + assert_eq!(request.prompt_token_ids.as_deref(), Some(&[11, 22][..])); + assert_adapter_a_lora_request(&request); + + send_outputs( + push, + engine_outputs_for_request(&request.request_id, default_stream_output_specs()), + ) + .await; + + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + assert_eq!(array[2], Value::from("remove_lora")); + + let args = array[3].as_array().expect("utility args"); + assert_eq!(args[0], Value::from(1)); + + send_outputs(push, utility_outputs(call_id, utility_result_value(true))).await; + }) + }) + .await; + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/load_lora_adapter") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "lora_name": "adapter-a", + "lora_path": "org/adapter-a" + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + assert_eq!(response.status(), StatusCode::OK); + + let models = app + .call(Request::builder().uri("/v1/models").body(Body::empty()).expect("build request")) + .await + .expect("call app"); + let body = to_bytes(models.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json["data"][1]["id"], "adapter-a"); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "adapter-a", + "prompt": "hello", + "max_tokens": 2 + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + assert_eq!(response.status(), StatusCode::OK); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "adapter-a", + "stream": false, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + assert_eq!(response.status(), StatusCode::OK); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/inference/v1/generate") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "adapter-a", + "token_ids": [11, 22], + "stream": false, + "sampling_params": { + "max_tokens": 2 + } + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + assert_eq!(response.status(), StatusCode::OK); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/unload_lora_adapter") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "lora_name": "adapter-a", + "lora_int_id": 1 + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + assert_eq!(response.status(), StatusCode::OK); + + let models = app + .call(Request::builder().uri("/v1/models").body(Body::empty()).expect("build request")) + .await + .expect("call app"); + let body = to_bytes(models.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json["data"].as_array().expect("model data").len(), 1); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "adapter-a", + "prompt": "hello", + "max_tokens": 2 + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + assert_eq!(response.status(), StatusCode::NOT_FOUND); + + drop(app); + engine_task.finish().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn server_info_endpoint_returns_not_found_without_snapshot() { + let mut app = test_app_with_dev_mode(true).await; + let response = app + .call( + Request::builder() + .uri("/server_info") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::NOT_FOUND); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn unload_lora_adapter_rejects_mismatched_lora_int_id() { + let (mut app, engine_task) = test_admin_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + assert_eq!(array[2], Value::from("add_lora")); + + send_outputs(push, utility_outputs(call_id, utility_result_value(true))).await; + }) + }) + .await; + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/load_lora_adapter") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "lora_name": "adapter-a", + "lora_path": "org/adapter-a" + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + assert_eq!(response.status(), StatusCode::OK); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/unload_lora_adapter") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "lora_name": "adapter-a", + "lora_int_id": 99 + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + + let models = app + .call(Request::builder().uri("/v1/models").body(Body::empty()).expect("build request")) + .await + .expect("call app"); + let body = to_bytes(models.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json["data"][1]["id"], "adapter-a"); + + drop(app); + engine_task.finish().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn load_lora_adapter_rejects_engine_false_result() { + let (mut app, engine_task) = test_admin_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + assert_eq!(array[2], Value::from("add_lora")); + + send_outputs(push, utility_outputs(call_id, utility_result_value(false))).await; + }) + }) + .await; + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/load_lora_adapter") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "lora_name": "adapter-a", + "lora_path": "org/adapter-a" + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + assert_eq!(response.status(), StatusCode::INTERNAL_SERVER_ERROR); + + let models = app + .call(Request::builder().uri("/v1/models").body(Body::empty()).expect("build request")) + .await + .expect("call app"); + let body = to_bytes(models.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json["data"].as_array().expect("model data").len(), 1); + + drop(app); + engine_task.finish().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn load_lora_adapter_rejects_base_model_name_collision() { + let (mut app, engine_task) = + test_admin_app_with_engine_script(|_, _| boxed_test_future(async move {})).await; + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/load_lora_adapter") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "lora_name": "Qwen/Qwen1.5-0.5B-Chat", + "lora_path": "org/adapter-a" + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + + drop(app); + engine_task.finish().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn http_metrics_record_list_models_requests() { + let mut app = test_app().await; + let before = METRICS.render().unwrap(); + + let response = app + .call( + Request::builder() + .method("GET") + .uri("/v1/models") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let after = METRICS.render().unwrap(); + assert_eq!( + metric_delta( + &before, + &after, + "http_requests_total", + Some("method=\"GET\",status=\"2xx\",handler=\"/v1/models\""), + ), + 1.0 + ); + assert_eq!( + metric_delta( + &before, + &after, + "http_request_duration_seconds_count", + Some("method=\"GET\",handler=\"/v1/models\""), + ), + 1.0 + ); + assert_eq!( + metric_delta( + &before, + &after, + "http_request_duration_highr_seconds_count", + None, + ), + 1.0 + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn request_metrics_use_served_model_name_label() { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-openai-served-model-metrics".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + boxed_test_future(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + send_outputs( + push, + engine_outputs_for_request(&request.request_id, default_stream_output_specs()), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("served-model-metrics") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + let chat = ChatLlm::from_shared_backend(test_llm(client), Arc::new(FakeChatBackend::new())); + let mut app = build_router(Arc::new(AppState::new( + vec![ + "served-model-metrics".to_string(), + "served-model-alias".to_string(), + ], + chat, + ))); + let before = METRICS.render().unwrap(); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "served-model-alias", + "stream": false, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let _ = to_bytes(response.into_body(), usize::MAX).await.unwrap(); + + let after = METRICS.render().unwrap(); + assert_eq!( + metric_delta( + &before, + &after, + "vllm:request_success_total", + Some("model_name=\"served-model-metrics\",engine=\"0\",finished_reason=\"stop\""), + ), + 1.0 + ); + engine_task.await.expect("mock engine task"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn wrong_model_returns_not_found() { + let mut app = test_app().await; + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "wrong-model", + "stream": true, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::NOT_FOUND); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn invalid_request_returns_openai_error() { + let mut app = test_app().await; + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": false, + "stream_options": {"include_usage": true}, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json["error"]["type"], "invalid_request_error"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn chat_completions_empty_allowed_token_ids_returns_openai_error() { + let mut app = test_app().await; + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": false, + "messages": [{"role": "user", "content": "hello"}], + "allowed_token_ids": [] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json["error"]["type"], "invalid_request_error"); + assert!( + json["error"]["message"] + .as_str() + .expect("message string") + .contains("allowed_token_ids should not be empty") + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_chat_returns_json_response() { + let (app, engine_task) = test_app_with_engine_handle().await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": false, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + assert!( + response + .headers() + .get("content-type") + .and_then(|value| value.to_str().ok()) + .is_some_and(|value| value.starts_with("application/json")) + ); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + assert_eq!(json["object"], "chat.completion"); + assert_eq!(json["choices"][0]["message"]["role"], "assistant"); + assert_eq!(json["choices"][0]["message"]["content"], "hi"); + assert_eq!(json["choices"][0]["finish_reason"], "stop"); + assert_eq!(json["usage"]["prompt_tokens"], 22); + assert_eq!(json["usage"]["completion_tokens"], 3); + assert_eq!(json["usage"]["total_tokens"], 25); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_chat_image_url_reaches_engine_mm_features() { + let (app, engine_task) = test_app_with_backend_and_engine_request_check( + Arc::new(FakeChatBackend::with_multimodal_model_info( + qwen_multimodal_model_info(), + )), + |request| { + let prompt_token_ids = request.prompt_token_ids.as_ref().expect("prompt token ids"); + assert!(prompt_token_ids.contains(&151655)); + + let features = request.mm_features.as_ref().expect("multimodal features"); + assert_eq!(features.len(), 1); + assert_eq!(features[0].modality, "image"); + assert_eq!(features[0].identifier, "image-1"); + assert!(features[0].mm_position.length > 0); + assert!(features[0].mm_position.is_embed.is_some()); + + let data = features[0].data.as_ref().expect("feature data"); + assert!(data.contains_key("pixel_values")); + assert!(data.contains_key("image_grid_thw")); + }, + ) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": false, + "messages": [{ + "role": "user", + "content": [ + {"type": "text", "text": "describe "}, + { + "type": "image_url", + "image_url": { + "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=" + }, + "uuid": "image-1" + } + ] + }] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + assert_eq!(json["object"], "chat.completion"); + assert_eq!(json["choices"][0]["message"]["content"], "hi"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_chat_includes_logprobs_and_prompt_logprobs() { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-openai-chat-logprobs".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + boxed_test_future(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + let prompt_token_ids = request.prompt_token_ids.clone().expect("prompt token ids"); + + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output_with_logprobs( + &request.request_id, + bytes_to_token_ids(b"hi"), + Some(EngineCoreFinishReason::Stop), + None, + Some(sample_logprobs_for_tokens(&bytes_to_token_ids(b"hi"))), + Some(prompt_logprobs_for_tokens(&prompt_token_ids)), + )], + ..Default::default() + } + .into(), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + let chat = ChatLlm::from_shared_backend(test_llm(client), Arc::new(FakeChatBackend::new())); + let mut app = build_router(Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + ))); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": false, + "logprobs": true, + "prompt_logprobs": 1, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + assert_eq!( + json["choices"][0]["logprobs"]["content"][0]["token"], + json!("h") + ); + assert_eq!( + json["choices"][0]["logprobs"]["content"][1]["token"], + json!("i") + ); + assert_eq!(json["prompt_logprobs"][0], serde_json::Value::Null); + assert!(json["prompt_logprobs"][1].is_object()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn happy_path_returns_sse_stream() { + let (app, engine_task) = test_app_with_engine_handle().await; + let before = METRICS.render().unwrap(); + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": true, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + response.headers().get("content-type").and_then(|value| value.to_str().ok()), + Some("text/event-stream") + ); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + let after = METRICS.render().unwrap(); + + assert!(text.contains("\"role\":\"assistant\""), "{text}"); + assert!(text.starts_with("data: "), "{text}"); + assert_eq!( + metric_delta( + &before, + &after, + "http_requests_total", + Some("method=\"POST\",status=\"2xx\",handler=\"/v1/chat/completions\""), + ), + 1.0 + ); + assert_eq!( + metric_delta( + &before, + &after, + "http_request_duration_seconds_count", + Some("method=\"POST\",handler=\"/v1/chat/completions\""), + ), + 1.0 + ); + assert_eq!( + metric_delta( + &before, + &after, + "http_request_duration_highr_seconds_count", + None, + ), + 1.0 + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn http_metrics_exclude_metrics_route() { + let mut app = test_app().await; + let before = METRICS.render().unwrap(); + + let response = app + .call( + Request::builder() + .method("GET") + .uri("/metrics") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let after = METRICS.render().unwrap(); + assert_eq!( + metric_delta( + &before, + &after, + "http_request_duration_highr_seconds_count", + None, + ), + 0.0 + ); + assert_eq!( + metric_value( + &after, + "http_requests_total", + Some("method=\"GET\",status=\"2xx\",handler=\"/metrics\""), + ), + None + ); + assert_eq!( + metric_value( + &after, + "http_request_duration_seconds_count", + Some("method=\"GET\",handler=\"/metrics\""), + ), + None + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn http_metrics_group_error_statuses() { + let mut app = test_app().await; + let before = METRICS.render().unwrap(); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": false, + "stream_options": {"include_usage": true}, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + + let after = METRICS.render().unwrap(); + assert_eq!( + metric_delta( + &before, + &after, + "http_requests_total", + Some("method=\"POST\",status=\"4xx\",handler=\"/v1/chat/completions\""), + ), + 1.0 + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn load_endpoint_tracks_chat_stream_lifecycle() { + let (app, engine_task) = test_app_with_engine_handle().await; + + assert_eq!(server_load(&app).await, 0); + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": true, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!(server_load(&app).await, 1); + + let _body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + + assert_eq!(server_load(&app).await, 0); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn health_endpoint_returns_ok_with_empty_body_when_client_is_healthy() { + let (app, _state, engine_task) = + test_health_app_with_engine_script(|_push| boxed_test_future(async move {})).await; + + let (status, body) = health_status(&app).await; + assert_eq!(status, StatusCode::OK); + assert!(body.is_empty(), "expected empty body, got {:?}", body); + + engine_task.await.expect("mock engine task"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn health_endpoint_returns_503_after_engine_core_dead_sentinel() { + let (app, state, engine_task) = test_health_app_with_engine_script(|push| { + boxed_test_future(async move { + push.send(ZmqMessage::from(ENGINE_CORE_DEAD_SENTINEL.to_vec())) + .await + .expect("send sentinel"); + }) + }) + .await; + + tokio::time::timeout(Duration::from_secs(2), async { + while state.chat.engine_core_client().is_healthy() { + tokio::task::yield_now().await; + } + }) + .await + .expect("wait for unhealthy client"); + + let (status, body) = health_status(&app).await; + assert_eq!(status, StatusCode::SERVICE_UNAVAILABLE); + assert!(body.is_empty(), "expected empty body, got {:?}", body); + assert!(matches!( + state.chat.engine_core_client().health_error().as_deref(), + Some(vllm_engine_core_client::Error::EngineCoreDead) + )); + + engine_task.await.expect("mock engine task"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn load_endpoint_resets_when_stream_response_is_dropped() { + let (app, engine_task) = test_app_with_engine_handle().await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": true + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!(server_load(&app).await, 1); + + drop(response); + tokio::task::yield_now().await; + engine_task.await.expect("mock engine task"); + + assert_eq!(server_load(&app).await, 0); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn stream_error_is_returned_as_openai_error_sse() { + let (app, engine_task) = test_app_with_backend_and_stream_output_specs( + Arc::new(FakeChatBackend::new()), + vec![(vec![UNKNOWN_DECODE_TOKEN_ID], None)], + ) + .await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": true, + "stream_options": {"include_usage": true}, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + + assert!(text.contains("\"role\":\"assistant\""), "{text}"); + assert!(text.contains("\"type\":\"server_error\""), "{text}"); + assert!( + text.contains(&format!( + "test tokenizer cannot decode unknown token id {UNKNOWN_DECODE_TOKEN_ID}" + )), + "{text}" + ); + assert!(!text.contains("\"usage\":"), "{text}"); + assert!(text.trim_end().ends_with("data: [DONE]"), "{text}"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn invalid_terminal_finish_reason_is_returned_as_openai_error_sse() { + let (app, engine_task) = + test_app_with_stream_output_specs(vec![(vec![], Some(EngineCoreFinishReason::Error))]) + .await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": true, + "stream_options": {"include_usage": true}, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + + assert!(text.contains("\"role\":\"assistant\""), "{text}"); + assert!(text.contains("\"type\":\"server_error\""), "{text}"); + assert!(text.contains("Internal server error"), "{text}"); + assert!(!text.contains("\"usage\":"), "{text}"); + assert!(text.trim_end().ends_with("data: [DONE]"), "{text}"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn include_usage_adds_final_usage_chunk_before_done() { + let (app, engine_task) = test_app_with_stream_output_specs(default_stream_output_specs()).await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": true, + "stream_options": {"include_usage": true}, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + + let payloads = sse_data_payloads(&text); + let finish_index = payloads + .iter() + .position(|payload| payload.contains("\"finish_reason\":\"stop\"")) + .expect("finish chunk"); + let usage_index = payloads + .iter() + .position(|payload| payload.contains("\"usage\":")) + .expect("usage chunk"); + let done_index = + payloads.iter().position(|payload| *payload == "[DONE]").expect("done sentinel"); + + assert!(finish_index < usage_index, "{text}"); + assert!(usage_index < done_index, "{text}"); + + let usage_chunk: serde_json::Value = + serde_json::from_str(payloads[usage_index]).expect("usage chunk json"); + assert_eq!(usage_chunk["choices"], json!([])); + assert_eq!(usage_chunk["usage"]["prompt_tokens"], 22); + assert_eq!(usage_chunk["usage"]["completion_tokens"], 3); + assert_eq!(usage_chunk["usage"]["total_tokens"], 25); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn stream_continuous_usage_stats_adds_usage_to_chat_chunks() { + let (app, engine_task) = test_app_with_stream_output_specs(default_stream_output_specs()).await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": true, + "stream_options": { + "include_usage": true, + "continuous_usage_stats": true + }, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + let payloads = sse_json_payloads(&text); + + assert!( + payloads.iter().all(|payload| payload.get("usage").is_some()), + "{text}" + ); + assert!( + payloads.iter().any(|payload| { + payload["choices"].as_array().is_some_and(|choices| !choices.is_empty()) + && payload["usage"]["completion_tokens"] == json!(1) + }), + "{text}" + ); + let usage_chunk = payloads + .iter() + .find(|payload| payload["choices"] == json!([])) + .expect("final usage chunk"); + assert_eq!(usage_chunk["usage"]["prompt_tokens"], 22); + assert_eq!(usage_chunk["usage"]["completion_tokens"], 3); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn stream_without_include_usage_keeps_existing_shape() { + let (app, engine_task) = test_app_with_stream_output_specs(default_stream_output_specs()).await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": true, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + + assert!(!text.contains("\"usage\":"), "{text}"); + assert!(text.contains("\"finish_reason\":\"stop\""), "{text}"); + assert!(text.trim_end().ends_with("data: [DONE]"), "{text}"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn completions_invalid_request_returns_openai_error() { + let mut app = test_app().await; + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false, + "stream_options": {"include_usage": true} + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json["error"]["type"], "invalid_request_error"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn completions_empty_allowed_token_ids_returns_openai_error() { + let mut app = test_app().await; + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false, + "allowed_token_ids": [] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json["error"]["type"], "invalid_request_error"); + assert!( + json["error"]["message"] + .as_str() + .expect("message string") + .contains("allowed_token_ids should not be empty") + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_completions_return_json_response() { + let (app, engine_task) = test_app_with_engine_handle().await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + assert!( + response + .headers() + .get("content-type") + .and_then(|value| value.to_str().ok()) + .is_some_and(|value| value.starts_with("application/json")) + ); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + assert_eq!(json["object"], "text_completion"); + assert_eq!(json["choices"][0]["text"], "hi"); + assert_eq!(json["choices"][0]["finish_reason"], "stop"); + assert_eq!(json["usage"]["completion_tokens"], 3); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_completions_echo_prepends_prompt_text() { + let (app, engine_task) = test_app_with_engine_handle().await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "echo": true, + "stream": false, + "add_special_tokens": false + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + assert_eq!(json["choices"][0]["text"], "hellohi"); + assert_eq!(json["usage"]["prompt_tokens"], 5); + assert_eq!(json["usage"]["completion_tokens"], 3); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_completions_echo_decodes_token_id_prompt_text() { + let prompt_token_ids = bytes_to_token_ids(b"hello"); + let expected_prompt_token_ids = prompt_token_ids.clone(); + let (app, engine_task) = test_app_with_backend_and_engine_request_check( + Arc::new(FakeChatBackend::new()), + move |request| { + assert_eq!( + request.prompt_token_ids.as_deref(), + Some(expected_prompt_token_ids.as_slice()) + ); + }, + ) + .await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": prompt_token_ids, + "echo": true, + "stream": false + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + assert_eq!(json["choices"][0]["text"], "hellohi"); + assert_eq!(json["usage"]["prompt_tokens"], 5); + assert_eq!(json["usage"]["completion_tokens"], 3); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_completions_token_id_echo_return_token_ids_keeps_prompt_ids_separate() { + let prompt_token_ids = bytes_to_token_ids(b"hello"); + let expected_prompt_token_ids = prompt_token_ids.clone(); + let (app, engine_task) = test_app_with_backend_and_engine_request_check( + Arc::new(FakeChatBackend::new()), + move |request| { + assert_eq!( + request.prompt_token_ids.as_deref(), + Some(expected_prompt_token_ids.as_slice()) + ); + }, + ) + .await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": prompt_token_ids, + "echo": true, + "return_token_ids": true, + "stream": false + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + assert_eq!(json["choices"][0]["text"], "hi"); + assert_eq!( + json["choices"][0]["prompt_token_ids"], + json!(bytes_to_token_ids(b"hello")) + ); + assert_eq!( + json["choices"][0]["token_ids"], + json!(bytes_to_token_ids(b"hi!")) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_completions_include_logprobs() { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-openai-completion-logprobs".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + boxed_test_future(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![ + request_output_with_logprobs( + &request.request_id, + vec![b'h' as u32], + None, + None, + Some(sample_logprobs_for_token(b'h' as u32, b'H' as u32)), + None, + ), + request_output_with_logprobs( + &request.request_id, + vec![b'i' as u32], + Some(EngineCoreFinishReason::Stop), + None, + Some(sample_logprobs_for_token(b'i' as u32, b'I' as u32)), + None, + ), + ], + finished_requests: Some(BTreeSet::from([request.request_id.clone()])), + ..Default::default() + } + .into(), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + let chat = ChatLlm::from_shared_backend(test_llm(client), Arc::new(FakeChatBackend::new())); + let mut app = build_router(Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + ))); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false, + "logprobs": 1 + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + assert_eq!(json["choices"][0]["logprobs"]["tokens"], json!(["h", "i"])); + assert_eq!( + json["choices"][0]["logprobs"]["token_logprobs"], + json!([-0.1, -0.1]) + ); + assert_eq!(json["choices"][0]["logprobs"]["text_offset"], json!([0, 1])); + assert_eq!( + json["choices"][0]["logprobs"]["top_logprobs"][0], + json!({"h": -0.1, "H": -0.2}) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_completions_include_prompt_logprobs() { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-openai-completion-prompt-logprobs".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + boxed_test_future(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output_with_logprobs( + &request.request_id, + vec![b'h' as u32, b'i' as u32, b'!' as u32], + Some(EngineCoreFinishReason::Stop), + None, + Some(Logprobs { + positions: vec![ + sample_logprobs_for_token(b'h' as u32, b'H' as u32).positions + [0] + .clone(), + sample_logprobs_for_token(b'i' as u32, b'I' as u32).positions + [0] + .clone(), + sample_logprobs_for_token(b'!' as u32, b'?' as u32).positions + [0] + .clone(), + ], + }), + Some(prompt_logprobs_for_hello()), + )], + ..Default::default() + } + .into(), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + let chat = ChatLlm::from_shared_backend(test_llm(client), Arc::new(FakeChatBackend::new())); + let mut app = build_router(Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + ))); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false, + "echo": true, + "logprobs": 1, + "add_special_tokens": false + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + assert_eq!(json["choices"][0]["text"], "hellohi"); + assert_eq!( + json["choices"][0]["logprobs"]["tokens"], + json!(["h", "e", "l", "l", "o", "h", "i", "!"]) + ); + assert_eq!( + json["choices"][0]["logprobs"]["text_offset"], + json!([0, 1, 2, 3, 4, 5, 6, 7]) + ); + assert_eq!( + json["choices"][0]["logprobs"]["token_logprobs"], + json!([null, -0.3, -0.4, -0.45, -0.5, -0.1, -0.1, -0.1]) + ); + assert_eq!( + json["choices"][0]["prompt_logprobs"][0], + serde_json::Value::Null + ); + assert_eq!( + json["choices"][0]["prompt_logprobs"][1], + json!({"a": -0.5, "e": -0.3}) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_chat_completions_still_succeed() { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-openai-chat-non-stream".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + boxed_test_future(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + send_outputs( + push, + engine_outputs_for_request(&request.request_id, default_stream_output_specs()), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + let chat = ChatLlm::from_shared_backend(test_llm(client), Arc::new(FakeChatBackend::new())); + let mut app = build_router(Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + ))); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": false, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + engine_task.await.expect("mock engine task"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn chat_completions_accepts_request_body_larger_than_axum_default() { + let (chat, engine_task) = test_chat_with_engine_outputs( + b"engine-openai-chat-large-body", + default_stream_output_specs(), + ) + .await; + let mut app = build_router(Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + ))); + + let large_template_arg = "a".repeat(2 * 1024 * 1024); + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": false, + "messages": [{"role": "user", "content": "hello"}], + "chat_template_kwargs": {"large": large_template_arg} + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + let status = response.status(); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + assert_eq!(status, StatusCode::OK, "{}", String::from_utf8_lossy(&body)); + engine_task.await.expect("mock engine task"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_completions_still_succeed() { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-openai-completion-non-stream".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + boxed_test_future(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + send_outputs( + push, + engine_outputs_for_request(&request.request_id, default_stream_output_specs()), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + let chat = ChatLlm::from_shared_backend(test_llm(client), Arc::new(FakeChatBackend::new())); + let mut app = build_router(Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + ))); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + engine_task.await.expect("mock engine task"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn chat_completions_header_request_id_takes_precedence() { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-chat-request-id-precedence".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + boxed_test_future(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + assert_eq!( + request.external_req_id.as_deref(), + Some("chatcmpl-header-req") + ); + assert!(request.request_id.starts_with("chatcmpl-header-req-")); + assert_ne!(request.request_id, "chatcmpl-header-req"); + + send_outputs( + push, + engine_outputs_for_request(&request.request_id, default_stream_output_specs()), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + let chat = ChatLlm::from_shared_backend(Llm::new(client), Arc::new(FakeChatBackend::new())); + let mut app = build_router(Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + ))); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .header("X-Request-Id", "header-req") + .body(Body::from( + json!({ + "request_id": "body-req", + "model": "Qwen/Qwen1.5-0.5B-Chat", + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + assert_eq!(json["id"], "chatcmpl-header-req"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_raw_generate_returns_token_output_envelope() { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-raw-generate-non-stream".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + boxed_test_future(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + assert_eq!(request.prompt_token_ids.as_deref(), Some(&[11, 22][..])); + assert_eq!(request.external_req_id.as_deref(), Some("raw-req")); + assert!(request.request_id.starts_with("raw-req-")); + assert_ne!(request.request_id, "raw-req"); + + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![ + request_output_with_logprobs( + &request.request_id, + vec![33], + None, + None, + Some(sample_logprobs_for_token(33, 34)), + Some(prompt_logprobs_for_tokens(&[11, 22])), + ), + request_output_with_logprobs_and_kv( + &request.request_id, + vec![44], + Some(EngineCoreFinishReason::Stop), + None, + Some(sample_logprobs_for_token(44, 45)), + None, + Some(json!({"connector": "x"})), + ), + ], + ..Default::default() + } + .into(), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + let chat = ChatLlm::from_shared_backend(Llm::new(client), Arc::new(FakeChatBackend::new())); + let mut app = build_router(Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + ))); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/inference/v1/generate") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "request_id": "raw-req", + "model": "Qwen/Qwen1.5-0.5B-Chat", + "token_ids": [11, 22], + "stream": false, + "sampling_params": { + "max_tokens": 2, + "logprobs": 1, + "prompt_logprobs": 1 + } + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + assert_eq!(json["request_id"], "raw-req"); + assert_eq!(json["choices"][0]["index"], 0); + assert_eq!(json["choices"][0]["token_ids"], json!([33, 44])); + assert_eq!(json["choices"][0]["finish_reason"], "stop"); + assert_eq!( + json["choices"][0]["logprobs"]["content"][0]["token"], + "token_id:33" + ); + assert_eq!( + json["choices"][0]["logprobs"]["content"][1]["top_logprobs"][0]["token"], + "token_id:44" + ); + assert_eq!(json["prompt_logprobs"][0], serde_json::Value::Null); + assert_eq!( + json["prompt_logprobs"][1]["22"]["decoded_token"], + "token_id:22" + ); + assert_eq!(json["kv_transfer_params"], json!({"connector": "x"})); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn stream_raw_generate_returns_sse_chunks_and_usage() { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-raw-generate-stream".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + boxed_test_future(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + assert_eq!(request.prompt_token_ids.as_deref(), Some(&[11, 22][..])); + assert_eq!(request.external_req_id.as_deref(), Some("raw-stream")); + + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![ + request_output_with_logprobs( + &request.request_id, + vec![33], + None, + None, + Some(sample_logprobs_for_token(33, 34)), + None, + ), + request_output_with_logprobs( + &request.request_id, + vec![44], + Some(EngineCoreFinishReason::Stop), + None, + Some(sample_logprobs_for_token(44, 45)), + None, + ), + ], + ..Default::default() + } + .into(), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + let chat = ChatLlm::from_shared_backend(Llm::new(client), Arc::new(FakeChatBackend::new())); + let mut app = build_router(Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + ))); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/inference/v1/generate") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "request_id": "raw-stream", + "token_ids": [11, 22], + "stream": true, + "stream_options": { + "include_usage": true, + "continuous_usage_stats": true + }, + "sampling_params": { + "max_tokens": 2, + "logprobs": 1 + } + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + response.headers().get("content-type").and_then(|value| value.to_str().ok()), + Some("text/event-stream") + ); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + let payloads = sse_data_payloads(&text); + assert_eq!(payloads.len(), 4, "{text}"); + + let first: serde_json::Value = serde_json::from_str(payloads[0]).expect("first chunk json"); + assert_eq!(first["request_id"], "raw-stream"); + assert_eq!(first["choices"][0]["index"], 0); + assert_eq!(first["choices"][0]["token_ids"], json!([33])); + assert_eq!( + first["choices"][0]["logprobs"]["content"][0]["token"], + "token_id:33" + ); + assert_eq!(first["usage"]["prompt_tokens"], 2); + assert_eq!(first["usage"]["completion_tokens"], 1); + + let second: serde_json::Value = serde_json::from_str(payloads[1]).expect("second chunk json"); + assert_eq!(second["choices"][0]["token_ids"], json!([44])); + assert_eq!(second["choices"][0]["finish_reason"], "stop"); + assert_eq!(second["usage"]["completion_tokens"], 2); + + let usage: serde_json::Value = serde_json::from_str(payloads[2]).expect("usage chunk json"); + assert_eq!(usage["choices"], json!([])); + assert_eq!(usage["usage"]["prompt_tokens"], 2); + assert_eq!(usage["usage"]["completion_tokens"], 2); + assert_eq!(usage["usage"]["total_tokens"], 4); + assert_eq!(payloads[3], "[DONE]"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn stream_raw_generate_emits_final_usage_without_continuous_usage() { + let (mut app, engine_task) = test_app_with_stream_output_specs(vec![ + (vec![33], None), + (vec![44], Some(EngineCoreFinishReason::Stop)), + ]) + .await; + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/inference/v1/generate") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "request_id": "raw-stream-final-usage", + "token_ids": [11, 22], + "stream": true, + "stream_options": { + "include_usage": true + }, + "sampling_params": { + "max_tokens": 2 + } + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + let payloads = sse_data_payloads(&text); + assert_eq!(payloads.len(), 4, "{text}"); + + let first: serde_json::Value = serde_json::from_str(payloads[0]).expect("first chunk json"); + assert_eq!(first["choices"][0]["token_ids"], json!([33])); + assert!(first.get("usage").is_none()); + + let second: serde_json::Value = serde_json::from_str(payloads[1]).expect("second chunk json"); + assert_eq!(second["choices"][0]["token_ids"], json!([44])); + assert_eq!(second["choices"][0]["finish_reason"], "stop"); + assert!(second.get("usage").is_none()); + + let usage: serde_json::Value = serde_json::from_str(payloads[2]).expect("usage chunk json"); + assert_eq!(usage["choices"], json!([])); + assert_eq!(usage["usage"]["prompt_tokens"], 2); + assert_eq!(usage["usage"]["completion_tokens"], 2); + assert_eq!(usage["usage"]["total_tokens"], 4); + assert_eq!(payloads[3], "[DONE]"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn stream_raw_generate_emits_empty_finish_chunk() { + let (mut app, engine_task) = test_app_with_stream_output_specs(vec![ + (vec![33], None), + (vec![], Some(EngineCoreFinishReason::Stop)), + ]) + .await; + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/inference/v1/generate") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "request_id": "raw-stream-empty-finish", + "token_ids": [11, 22], + "stream": true, + "sampling_params": { + "max_tokens": 2 + } + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + let payloads = sse_data_payloads(&text); + assert_eq!(payloads.len(), 3, "{text}"); + + let first: serde_json::Value = serde_json::from_str(payloads[0]).expect("first chunk json"); + assert_eq!(first["choices"][0]["token_ids"], json!([33])); + assert!(first["choices"][0].get("finish_reason").is_none()); + + let second: serde_json::Value = serde_json::from_str(payloads[1]).expect("second chunk json"); + assert_eq!(second["choices"][0]["token_ids"], json!([])); + assert_eq!(second["choices"][0]["finish_reason"], "stop"); + assert_eq!(payloads[2], "[DONE]"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn stream_raw_generate_error_finish_returns_sse_error() { + let (mut app, engine_task) = + test_app_with_stream_output_specs(vec![(vec![], Some(EngineCoreFinishReason::Error))]) + .await; + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/inference/v1/generate") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "request_id": "raw-stream-error", + "token_ids": [11, 22], + "stream": true, + "stream_options": { + "include_usage": true + }, + "sampling_params": { + "max_tokens": 2 + } + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + + assert!(text.contains("\"type\":\"server_error\""), "{text}"); + assert!(text.contains("Internal server error"), "{text}"); + assert!(!text.contains("\"finish_reason\":\"error\""), "{text}"); + assert!(!text.contains("\"usage\":"), "{text}"); + assert!(text.trim_end().ends_with("data: [DONE]"), "{text}"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn raw_generate_rejects_empty_token_ids() { + let mut app = test_app().await; + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/inference/v1/generate") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "token_ids": [], + "sampling_params": {} + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json["error"]["param"], "token_ids"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn raw_generate_rejects_streaming_prompt_logprobs() { + let mut app = test_app().await; + + for prompt_logprobs in [0, 1] { + let response = app + .call( + Request::builder() + .method("POST") + .uri("/inference/v1/generate") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "token_ids": [11, 22], + "stream": true, + "sampling_params": { + "prompt_logprobs": prompt_logprobs + } + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json["error"]["param"], "sampling_params"); + assert_eq!( + json["error"]["message"], + "`prompt_logprobs` are not available when `stream=true`." + ); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn raw_generate_rejects_wrong_model() { + let mut app = test_app().await; + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/inference/v1/generate") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "wrong-model", + "token_ids": [11, 22], + "sampling_params": {} + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::NOT_FOUND); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn completions_happy_path_returns_sse_stream() { + let (app, engine_task) = test_app_with_engine_handle().await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": true, + "stream_options": {"include_usage": true} + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + response.headers().get("content-type").and_then(|value| value.to_str().ok()), + Some("text/event-stream") + ); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + let payloads = sse_data_payloads(&text); + let usage_index = payloads + .iter() + .position(|payload| payload.contains("\"usage\":")) + .expect("usage chunk"); + let done_index = + payloads.iter().position(|payload| *payload == "[DONE]").expect("done sentinel"); + + assert!( + payloads.iter().any(|payload| payload.contains("\"text\":\"h\"")), + "{text}" + ); + assert!( + payloads.iter().any(|payload| payload.contains("\"finish_reason\":\"stop\"")), + "{text}" + ); + assert!(usage_index < done_index, "{text}"); + + let usage_chunk: serde_json::Value = + serde_json::from_str(payloads[usage_index]).expect("usage chunk json"); + assert_eq!(usage_chunk["choices"], json!([])); + assert_eq!(usage_chunk["usage"]["completion_tokens"], 3); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn completions_stream_continuous_usage_stats_adds_usage_to_chunks() { + let (app, engine_task) = test_app_with_stream_output_specs(default_stream_output_specs()).await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": true, + "stream_options": { + "include_usage": true, + "continuous_usage_stats": true + } + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + let payloads = sse_json_payloads(&text); + + assert!( + payloads.iter().all(|payload| payload.get("usage").is_some()), + "{text}" + ); + assert!( + payloads.iter().any(|payload| { + payload["choices"].as_array().is_some_and(|choices| !choices.is_empty()) + && payload["usage"]["completion_tokens"] == json!(1) + }), + "{text}" + ); + let usage_chunk = payloads + .iter() + .find(|payload| payload["choices"] == json!([])) + .expect("final usage chunk"); + assert_eq!(usage_chunk["usage"]["completion_tokens"], 3); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn completions_echo_stream_emits_separate_prompt_chunk() { + let (app, engine_task) = test_app_with_engine_handle().await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "echo": true, + "stream": true, + "stream_options": {"include_usage": true}, + "add_special_tokens": false + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + let payloads = sse_data_payloads(&text); + let hello_index = payloads + .iter() + .position(|payload| payload.contains("\"text\":\"hello\"")) + .expect("prompt echo chunk"); + let h_index = payloads + .iter() + .position(|payload| payload.contains("\"text\":\"h\"")) + .expect("first generation chunk"); + + assert!(hello_index < h_index, "{text}"); + assert!( + payloads.iter().any(|payload| payload.contains("\"text\":\"i\"")), + "{text}" + ); + + let usage_chunk: serde_json::Value = serde_json::from_str( + payloads + .iter() + .find(|payload| payload.contains("\"usage\":")) + .expect("usage chunk"), + ) + .expect("usage chunk json"); + assert_eq!(usage_chunk["usage"]["prompt_tokens"], 5); + assert_eq!(usage_chunk["usage"]["completion_tokens"], 3); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn completions_echo_stream_decodes_token_id_prompt_chunk() { + let prompt_token_ids = bytes_to_token_ids(b"hello"); + let expected_prompt_token_ids = prompt_token_ids.clone(); + let (app, engine_task) = test_app_with_backend_and_engine_request_check( + Arc::new(FakeChatBackend::new()), + move |request| { + assert_eq!( + request.prompt_token_ids.as_deref(), + Some(expected_prompt_token_ids.as_slice()) + ); + }, + ) + .await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": prompt_token_ids, + "echo": true, + "stream": true, + "stream_options": {"include_usage": true} + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + let payloads = sse_data_payloads(&text); + let hello_index = payloads + .iter() + .position(|payload| payload.contains("\"text\":\"hello\"")) + .expect("prompt echo chunk"); + let h_index = payloads + .iter() + .position(|payload| payload.contains("\"text\":\"h\"")) + .expect("first generation chunk"); + + assert!(hello_index < h_index, "{text}"); + + let usage_chunk: serde_json::Value = serde_json::from_str( + payloads + .iter() + .find(|payload| payload.contains("\"usage\":")) + .expect("usage chunk"), + ) + .expect("usage chunk json"); + assert_eq!(usage_chunk["usage"]["prompt_tokens"], 5); + assert_eq!(usage_chunk["usage"]["completion_tokens"], 3); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn reasoning_blocks_are_mapped_to_reasoning_sse_chunks() { + let (app, engine_task) = test_app_with_backend_and_stream_output_specs( + Arc::new(FakeChatBackend::with_model_id("Qwen/Qwen3-0.6B")), + vec![ + (bytes_to_token_ids(b""), None), + (bytes_to_token_ids(b"think "), None), + (bytes_to_token_ids(b"more"), None), + ( + bytes_to_token_ids(b"answer"), + Some(EngineCoreFinishReason::Length), + ), + ], + ) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": true, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + + assert!(text.contains("\"reasoning\":\"think \""), "{text}"); + assert!(text.contains("\"reasoning\":\"more\""), "{text}"); + assert!(text.contains("\"content\":\"answer\""), "{text}"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn include_reasoning_false_suppresses_reasoning_in_non_stream_chat() { + let (app, engine_task) = test_app_with_backend_and_stream_output_specs( + Arc::new(FakeChatBackend::with_model_id("Qwen/Qwen3-0.6B")), + vec![ + (bytes_to_token_ids(b"think"), None), + ( + bytes_to_token_ids(b"answer"), + Some(EngineCoreFinishReason::Length), + ), + ], + ) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": false, + "include_reasoning": false, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + let json: serde_json::Value = serde_json::from_str(&text).expect("decode json"); + + assert_eq!(json["choices"][0]["message"]["content"], "answer"); + assert!( + json["choices"][0]["message"] + .as_object() + .is_some_and(|message| !message.contains_key("reasoning")), + "{text}" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn include_reasoning_false_suppresses_non_stream_output_metadata() { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-openai-hidden-reasoning-logprobs".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + boxed_test_future(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + let reasoning_token_ids = bytes_to_token_ids(b"think"); + let answer_token_ids = bytes_to_token_ids(b"answer"); + + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![ + request_output_with_logprobs( + &request.request_id, + reasoning_token_ids.clone(), + None, + None, + Some(sample_logprobs_for_tokens(&reasoning_token_ids)), + None, + ), + request_output_with_logprobs( + &request.request_id, + answer_token_ids.clone(), + Some(EngineCoreFinishReason::Length), + None, + Some(sample_logprobs_for_tokens(&answer_token_ids)), + None, + ), + ], + ..Default::default() + } + .into(), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + let chat = ChatLlm::from_shared_backend( + test_llm(client), + Arc::new(FakeChatBackend::with_model_id("Qwen/Qwen3-0.6B")), + ); + let mut app = build_router(Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + ))); + + let response = app + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": false, + "include_reasoning": false, + "logprobs": true, + "return_token_ids": true, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + let json: serde_json::Value = serde_json::from_str(&text).expect("decode json"); + let choice = json["choices"][0].as_object().expect("choice object"); + + assert_eq!(json["choices"][0]["message"]["content"], "answer"); + assert!( + json["choices"][0]["message"] + .as_object() + .is_some_and(|message| !message.contains_key("reasoning")), + "{text}" + ); + assert!(!choice.contains_key("logprobs"), "{text}"); + assert!(!choice.contains_key("token_ids"), "{text}"); + assert!(json["prompt_token_ids"].is_array(), "{text}"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn tool_calls_are_mapped_to_tool_call_sse_chunks() { + let (app, engine_task) = test_app_with_backend_and_stream_output_specs( + Arc::new(FakeChatBackend::with_model_id("Qwen/Qwen3-0.6B")), + vec![ + (bytes_to_token_ids(b"Need tool."), None), + ( + bytes_to_token_ids(b"\n{\"name\":\"get_weather\", "), + None, + ), + ( + bytes_to_token_ids(b"\"arguments\":{\"city\":\"Paris\"}}\n"), + Some(EngineCoreFinishReason::Stop), + ), + ], + ) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": true, + "messages": [{"role": "user", "content": "hello"}], + "tools": [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get weather", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}} + } + } + }] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + + assert!(text.contains("\"tool_calls\":"), "{text}"); + assert!(text.contains("\"name\":\"get_weather\""), "{text}"); + assert!( + text.contains("\"arguments\":\"{\\\"city\\\":\\\"Paris\\\"}\""), + "{text}" + ); + assert!(text.contains("\"finish_reason\":\"tool_calls\""), "{text}"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn tool_call_sse_chunks_can_carry_logprobs() { + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-openai-chat-tools-logprobs".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + |dealer, push| { + Box::pin(async move { + let add = recv_engine_message(dealer).await; + let request: EngineCoreRequest = + rmp_serde::from_slice(&add[1]).expect("decode request"); + + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output_with_logprobs( + &request.request_id, + bytes_to_token_ids(b"Need tool."), + None, + None, + Some(sample_logprobs_for_tokens(&bytes_to_token_ids( + b"Need tool.", + ))), + None, + )], + ..Default::default() + } + .into(), + ) + .await; + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output_with_logprobs( + &request.request_id, + bytes_to_token_ids(b"\n{\"name\":\"get_weather\", "), + None, + None, + Some(sample_logprobs_for_tokens(&bytes_to_token_ids( + b"\n{\"name\":\"get_weather\", ", + ))), + None, + )], + ..Default::default() + } + .into(), + ) + .await; + send_outputs( + push, + RequestBatchOutputs { + outputs: vec![request_output_with_logprobs( + &request.request_id, + bytes_to_token_ids( + b"\"arguments\":{\"city\":\"Paris\"}}\n", + ), + Some(EngineCoreFinishReason::Stop), + None, + Some(sample_logprobs_for_tokens(&bytes_to_token_ids( + b"\"arguments\":{\"city\":\"Paris\"}}\n", + ))), + None, + )], + finished_requests: Some(BTreeSet::from([request.request_id.clone()])), + ..Default::default() + } + .into(), + ) + .await; + }) + }, + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + let chat = ChatLlm::from_shared_backend( + test_llm(client), + Arc::new(FakeChatBackend::with_model_id("Qwen/Qwen3-0.6B")), + ); + let app = build_router(Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + ))); + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": true, + "logprobs": true, + "messages": [{"role": "user", "content": "hello"}], + "tools": [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get weather", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}} + } + } + }] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.finish().await; + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + + assert!(text.contains("\"tool_calls\":"), "{text}"); + assert!(text.contains("\"logprobs\":{\"content\":"), "{text}"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn streaming_chat_prompt_logprobs_are_rejected() { + let (app, engine_task) = test_app_with_engine_handle().await; + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/v1/chat/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "stream": true, + "prompt_logprobs": 1, + "messages": [{"role": "user", "content": "hello"}] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + engine_task.abort(); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn reset_prefix_cache_route_sends_expected_utility_call() { + let (app, engine_task) = test_admin_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + + assert_eq!(array[2], Value::from("reset_prefix_cache")); + assert_eq!( + array[3], + Value::Array(vec![Value::from(true), Value::from(true)]) + ); + + send_outputs(push, utility_outputs(call_id, utility_result_value(true))).await; + }) + }) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/reset_prefix_cache?reset_running_requests=true&reset_external=true") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + let status = response.status(); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + assert_eq!(status, StatusCode::OK, "{}", String::from_utf8_lossy(&body)); + assert!(body.is_empty()); + engine_task.await.expect("mock engine task"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn reset_mm_cache_route_sends_expected_utility_call() { + let (app, engine_task) = test_admin_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + + assert_eq!(array[2], Value::from("reset_mm_cache")); + assert_eq!(array[3], Value::Array(Vec::new())); + + send_outputs(push, utility_outputs(call_id, utility_none_result())).await; + }) + }) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/reset_mm_cache") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + let status = response.status(); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + assert_eq!(status, StatusCode::OK, "{}", String::from_utf8_lossy(&body)); + assert!(body.is_empty()); + engine_task.await.expect("mock engine task"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn reset_encoder_cache_route_sends_expected_utility_call() { + let (app, engine_task) = test_admin_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + + assert_eq!(array[2], Value::from("reset_encoder_cache")); + assert_eq!(array[3], Value::Array(Vec::new())); + + send_outputs(push, utility_outputs(call_id, utility_none_result())).await; + }) + }) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/reset_encoder_cache") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + let status = response.status(); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + assert_eq!(status, StatusCode::OK, "{}", String::from_utf8_lossy(&body)); + assert!(body.is_empty()); + engine_task.await.expect("mock engine task"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn collective_rpc_route_sends_expected_utility_call_and_returns_results() { + let (app, engine_task) = test_admin_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + + assert_eq!(array[2], Value::from("collective_rpc")); + assert_eq!( + array[3], + Value::Array(vec![ + Value::from("echo_args_kwargs"), + Value::from(1.5_f64), + Value::Array(vec![Value::from("arg1"), Value::from("arg2")]), + Value::Map(vec![ + (Value::from("key1"), Value::from("value1")), + (Value::from("key2"), Value::from("value2")), + ]), + ]) + ); + + send_outputs( + push, + utility_outputs( + call_id, + UtilityResultEnvelope::without_type_info(Value::Array(vec![Value::Map(vec![ + ( + Value::from("args"), + Value::Array(vec![Value::from("arg1"), Value::from("arg2")]), + ), + ( + Value::from("kwargs"), + Value::Map(vec![ + (Value::from("key1"), Value::from("value1")), + (Value::from("key2"), Value::from("value2")), + ]), + ), + (Value::from("total_items"), Value::from(4_u64)), + ])])), + ), + ) + .await; + }) + }) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/collective_rpc") + .header("content-type", "application/json") + .body(Body::from( + r#"{"method":"echo_args_kwargs","args":["arg1","arg2"],"kwargs":{"key1":"value1","key2":"value2"},"timeout":1.5}"#, + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + + assert_eq!( + serde_json::from_slice::(&body).expect("decode json"), + json!({ + "results": [{ + "args": ["arg1", "arg2"], + "kwargs": { + "key1": "value1", + "key2": "value2" + }, + "total_items": 4 + }] + }) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn sleep_route_uses_python_compatible_default_query_values() { + let (app, engine_task) = test_admin_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + + assert_eq!(array[2], Value::from("sleep")); + assert_eq!( + array[3], + Value::Array(vec![Value::from(1_u64), Value::from("abort")]) + ); + + send_outputs(push, utility_outputs(call_id, utility_none_result())).await; + }) + }) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/sleep") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + let status = response.status(); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + assert_eq!(status, StatusCode::OK, "{}", String::from_utf8_lossy(&body)); + assert!(body.is_empty()); + engine_task.await.expect("mock engine task"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn wake_up_route_without_tags_sends_none() { + let (app, engine_task) = test_admin_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + + assert_eq!(array[2], Value::from("wake_up")); + assert_eq!(array[3], Value::Array(vec![Value::Nil])); + + send_outputs(push, utility_outputs(call_id, utility_none_result())).await; + }) + }) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/wake_up") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + let status = response.status(); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + assert_eq!(status, StatusCode::OK, "{}", String::from_utf8_lossy(&body)); + assert!(body.is_empty()); + engine_task.await.expect("mock engine task"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn is_sleeping_route_returns_json_payload() { + let (app, engine_task) = test_admin_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + + assert_eq!(array[2], Value::from("is_sleeping")); + assert_eq!(array[3], Value::Array(Vec::new())); + + send_outputs(push, utility_outputs(call_id, utility_result_value(true))).await; + }) + }) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("GET") + .uri("/is_sleeping") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + + assert_eq!( + serde_json::from_slice::(&body).expect("decode json"), + json!({ "is_sleeping": true }) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn pause_route_uses_python_compatible_default_query_values() { + let (app, engine_task) = test_admin_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + + assert_eq!(array[2], Value::from("pause_scheduler")); + assert_eq!( + array[3], + Value::Array(vec![Value::from("abort"), Value::from(true)]) + ); + + send_outputs(push, utility_outputs(call_id, utility_none_result())).await; + }) + }) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/pause") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + + assert_eq!( + serde_json::from_slice::(&body).expect("decode json"), + json!({ "status": "paused" }) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn pause_route_rejects_invalid_mode() { + let (app, engine_task) = + test_admin_app_with_engine_script(|_dealer, _push| boxed_test_future(async move {})).await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/pause?mode=banana") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json["error"]["type"], "invalid_request_error"); + assert_eq!(json["error"]["param"], "mode"); + engine_task.abort_and_join().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn resume_route_sends_no_args() { + let (app, engine_task) = test_admin_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + + assert_eq!(array[2], Value::from("resume_scheduler")); + assert_eq!(array[3], Value::Array(Vec::new())); + + send_outputs(push, utility_outputs(call_id, utility_none_result())).await; + }) + }) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/resume") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + + assert_eq!( + serde_json::from_slice::(&body).expect("decode json"), + json!({ "status": "resumed" }) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn is_paused_route_returns_json_payload() { + let (app, engine_task) = test_admin_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + + assert_eq!(array[2], Value::from("is_scheduler_paused")); + assert_eq!(array[3], Value::Array(Vec::new())); + + send_outputs(push, utility_outputs(call_id, utility_result_value(true))).await; + }) + }) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("GET") + .uri("/is_paused") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + + assert_eq!( + serde_json::from_slice::(&body).expect("decode json"), + json!({ "is_paused": true }) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn abort_requests_route_returns_ok_for_well_formed_body() { + let (app, engine_task) = + test_admin_app_with_engine_script(|_dealer, _push| boxed_test_future(async move {})).await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/abort_requests") + .header("content-type", "application/json") + .body(Body::from(r#"{"request_ids":["req-1","req-2"]}"#)) + .expect("build request"), + ) + .await + .expect("call app"); + + let status = response.status(); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + assert_eq!(status, StatusCode::OK, "{}", String::from_utf8_lossy(&body)); + assert!(body.is_empty()); + engine_task.abort_and_join().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn abort_requests_route_rejects_missing_request_ids() { + let (app, engine_task) = + test_admin_app_with_engine_script(|_dealer, _push| boxed_test_future(async move {})).await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/abort_requests") + .header("content-type", "application/json") + .body(Body::from(r#"{}"#)) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json["error"]["type"], "invalid_request_error"); + assert_eq!(json["error"]["param"], "request_ids"); + engine_task.abort_and_join().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn abort_requests_route_rejects_malformed_json() { + let (app, engine_task) = + test_admin_app_with_engine_script(|_dealer, _push| boxed_test_future(async move {})).await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/abort_requests") + .header("content-type", "application/json") + .body(Body::from(r#"{"request_ids": "#)) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::BAD_REQUEST); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json["error"]["type"], "invalid_request_error"); + engine_task.abort_and_join().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn abort_requests_route_accepts_empty_id_list() { + let (app, engine_task) = + test_admin_app_with_engine_script(|_dealer, _push| boxed_test_future(async move {})).await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/abort_requests") + .header("content-type", "application/json") + .body(Body::from(r#"{"request_ids":[]}"#)) + .expect("build request"), + ) + .await + .expect("call app"); + + let status = response.status(); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + assert_eq!(status, StatusCode::OK, "{}", String::from_utf8_lossy(&body)); + assert!(body.is_empty()); + engine_task.abort_and_join().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn admin_routes_are_hidden_when_dev_mode_is_disabled() { + let (chat, engine_task) = test_chat_with_engine_handle().await; + let app = build_router_with_dev_mode( + Arc::new(AppState::new( + vec!["Qwen/Qwen1.5-0.5B-Chat".to_string()], + chat, + )), + false, + ); + + for (method, uri) in [ + ("GET", "/is_sleeping"), + ("POST", "/sleep"), + ("POST", "/wake_up"), + ("GET", "/is_paused"), + ("POST", "/pause"), + ("POST", "/resume"), + ("POST", "/collective_rpc"), + ("POST", "/abort_requests"), + ("POST", "/reset_prefix_cache"), + ("POST", "/reset_mm_cache"), + ("POST", "/reset_encoder_cache"), + ] { + let response = app + .clone() + .call( + Request::builder() + .method(method) + .uri(uri) + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::NOT_FOUND, "{method} {uri}"); + } + + engine_task.abort_and_join().await; +} + +// ========================= Stop string tests ========================= + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_completions_stop_string_excluded_from_output() { + // Engine generates "say world" but stop string "wor" truncates output to "say + // ". + let output_specs = vec![ + (bytes_to_token_ids(b"say"), None), + ( + bytes_to_token_ids(b" world"), + Some(EngineCoreFinishReason::Length), + ), + ]; + let (app, engine_task) = test_app_with_stream_output_specs(output_specs).await; + + let response = app + .clone() + .oneshot( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false, + "stop": ["wor"] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + assert_eq!(json["choices"][0]["text"], "say "); + assert_eq!(json["choices"][0]["finish_reason"], "stop"); + assert_eq!(json["choices"][0]["stop_reason"], "wor"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_completions_stop_string_included_in_output() { + // Same tokens but include_stop_str_in_output=true includes the stop string in + // the output. + let output_specs = vec![ + (bytes_to_token_ids(b"say"), None), + ( + bytes_to_token_ids(b" world"), + Some(EngineCoreFinishReason::Length), + ), + ]; + let (app, engine_task) = test_app_with_stream_output_specs(output_specs).await; + + let response = app + .clone() + .oneshot( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false, + "stop": ["wor"], + "include_stop_str_in_output": true + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + assert_eq!(json["choices"][0]["text"], "say wor"); + assert_eq!(json["choices"][0]["finish_reason"], "stop"); + assert_eq!(json["choices"][0]["stop_reason"], "wor"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn stream_completions_stop_string_excluded_from_output() { + let output_specs = vec![ + (bytes_to_token_ids(b"say"), None), + ( + bytes_to_token_ids(b" world"), + Some(EngineCoreFinishReason::Length), + ), + ]; + let (app, engine_task) = test_app_with_stream_output_specs(output_specs).await; + + let response = app + .clone() + .oneshot( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": true, + "stop": ["wor"] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + let payloads = sse_data_payloads(&text); + + // Collect all text deltas from the SSE chunks. + let mut full_text = String::new(); + for payload in &payloads { + if *payload == "[DONE]" { + continue; + } + let chunk: serde_json::Value = serde_json::from_str(payload).expect("json chunk"); + if let Some(text) = chunk["choices"][0]["text"].as_str() { + full_text.push_str(text); + } + } + + // The concatenated text deltas should equal "say " (stop string excluded). + assert_eq!(full_text, "say ", "full streamed text: {text}"); + + // The final chunk should have finish_reason "stop". + assert!( + payloads.iter().any(|p| p.contains("\"finish_reason\":\"stop\"")), + "{text}" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn stream_completions_stop_string_included_in_output() { + let output_specs = vec![ + (bytes_to_token_ids(b"say"), None), + ( + bytes_to_token_ids(b" world"), + Some(EngineCoreFinishReason::Length), + ), + ]; + let (app, engine_task) = test_app_with_stream_output_specs(output_specs).await; + + let response = app + .clone() + .oneshot( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": true, + "stop": ["wor"], + "include_stop_str_in_output": true + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let text = String::from_utf8(body.to_vec()).expect("utf8 body"); + let payloads = sse_data_payloads(&text); + + let mut full_text = String::new(); + for payload in &payloads { + if *payload == "[DONE]" { + continue; + } + let chunk: serde_json::Value = serde_json::from_str(payload).expect("json chunk"); + if let Some(text) = chunk["choices"][0]["text"].as_str() { + full_text.push_str(text); + } + } + + // With include_stop_str_in_output, the stop string "wor" should be included. + assert_eq!(full_text, "say wor", "full streamed text: {text}"); + + assert!( + payloads.iter().any(|p| p.contains("\"finish_reason\":\"stop\"")), + "{text}" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_completions_no_stop_string_match_preserves_original_finish_reason() { + // Stop string "xyz" does not appear in "hi!" so the original finish reason is + // preserved. + let (app, engine_task) = test_app_with_engine_handle().await; + + let response = app + .clone() + .oneshot( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false, + "stop": ["xyz"] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + // Default output is "hi" (stop token '!' suppressed), finish_reason remains + // "stop" from EOS. + assert_eq!(json["choices"][0]["text"], "hi"); + assert_eq!(json["choices"][0]["finish_reason"], "stop"); + // No text stop string matched — stop_reason should be absent. + assert!(json["choices"][0]["stop_reason"].is_null()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn non_stream_completions_stop_string_array_matches_first_occurrence() { + // Multiple stop strings: "rl" appears in "world" but " wo" appears earlier. + let output_specs = vec![( + bytes_to_token_ids(b"say world"), + Some(EngineCoreFinishReason::Length), + )]; + let (app, engine_task) = test_app_with_stream_output_specs(output_specs).await; + + let response = app + .clone() + .oneshot( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false, + "stop": [" wo", "rl"] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + engine_task.await.expect("mock engine task"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + + // " wo" is detected first (at byte 3), so output is truncated to "say". + assert_eq!(json["choices"][0]["text"], "say"); + assert_eq!(json["choices"][0]["finish_reason"], "stop"); + assert_eq!(json["choices"][0]["stop_reason"], " wo"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn completions_empty_stop_string_returns_validation_error() { + let (app, _engine_task) = test_app_with_engine_handle().await; + + let response = app + .clone() + .oneshot( + Request::builder() + .method("POST") + .uri("/v1/completions") + .header("content-type", "application/json") + .body(Body::from( + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "stream": false, + "stop": [""] + }) + .to_string(), + )) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::BAD_REQUEST); +} + +async fn post_json( + app: &mut axum::Router, + uri: &str, + body: serde_json::Value, +) -> (StatusCode, serde_json::Value) { + let response = app + .call( + Request::builder() + .method("POST") + .uri(uri) + .header("content-type", "application/json") + .body(Body::from(body.to_string())) + .expect("build request"), + ) + .await + .expect("call app"); + let status = response.status(); + let bytes = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&bytes) + .unwrap_or_else(|_| json!({ "raw": String::from_utf8_lossy(&bytes) })); + (status, json) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn tokenize_completion_round_trips_through_detokenize() { + let mut app = test_app().await; + let prompt = "Hello world"; + + let (_, tokenize_json) = post_json( + &mut app, + "/tokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": prompt, + "add_special_tokens": false, + }), + ) + .await; + let tokens = tokenize_json["tokens"] + .as_array() + .expect("tokens array") + .iter() + .map(|v| v.as_u64().expect("token id") as u32) + .collect::>(); + + let (status, detokenize_json) = post_json( + &mut app, + "/detokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "tokens": tokens, + }), + ) + .await; + + assert_eq!(status, StatusCode::OK); + assert_eq!(detokenize_json["prompt"], prompt); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn tokenize_completion_add_special_tokens_changes_ids() { + let mut app = test_app().await; + + let (_, with_special) = post_json( + &mut app, + "/tokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hi", + "add_special_tokens": true, + }), + ) + .await; + let (_, without_special) = post_json( + &mut app, + "/tokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hi", + "add_special_tokens": false, + }), + ) + .await; + + let with_ids: Vec = with_special["tokens"] + .as_array() + .unwrap() + .iter() + .map(|v| v.as_u64().unwrap() as u32) + .collect(); + let without_ids: Vec = without_special["tokens"] + .as_array() + .unwrap() + .iter() + .map(|v| v.as_u64().unwrap() as u32) + .collect(); + + assert_ne!(with_ids, without_ids); + assert_eq!(with_ids.first().copied(), Some(FAKE_BOS_TOKEN_ID)); + assert_eq!(without_ids.first().copied(), Some(b'h' as u32)); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn tokenize_completion_return_token_strs_matches_tokens() { + let mut app = test_app().await; + + let (status, json) = post_json( + &mut app, + "/tokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hi", + "add_special_tokens": false, + "return_token_strs": true, + }), + ) + .await; + + assert_eq!(status, StatusCode::OK); + let tokens = json["tokens"].as_array().expect("tokens"); + let token_strs = json["token_strs"].as_array().expect("token_strs"); + assert_eq!(tokens.len(), token_strs.len()); + assert_eq!(token_strs.len(), json["count"].as_u64().unwrap() as usize); + assert!(!token_strs[0].as_str().unwrap().is_empty()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn tokenize_completion_count_and_max_model_len() { + let mut app = test_app().await; + + let (status, json) = post_json( + &mut app, + "/tokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "prompt": "hello", + "add_special_tokens": false, + }), + ) + .await; + + assert_eq!(status, StatusCode::OK); + assert_eq!( + json["count"].as_u64().unwrap() as usize, + json["tokens"].as_array().unwrap().len() + ); + assert!(json["max_model_len"].as_u64().unwrap() > 0); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn tokenize_chat_includes_generation_prompt_in_token_count() { + let mut app = test_app().await; + let messages = json!([{"role": "user", "content": "hi"}]); + + let (_, with_prompt) = post_json( + &mut app, + "/tokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "messages": messages, + "add_generation_prompt": true, + "add_special_tokens": false, + }), + ) + .await; + let (_, without_prompt) = post_json( + &mut app, + "/tokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "messages": messages, + "add_generation_prompt": false, + "add_special_tokens": false, + }), + ) + .await; + + let with_len = with_prompt["tokens"].as_array().unwrap().len(); + let without_len = without_prompt["tokens"].as_array().unwrap().len(); + assert!(with_len > without_len); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn tokenize_chat_conflicting_generation_flags_returns_400() { + let mut app = test_app().await; + + let (status, json) = post_json( + &mut app, + "/tokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "messages": [{"role": "user", "content": "hi"}], + "add_generation_prompt": true, + "continue_final_message": true, + }), + ) + .await; + + assert_eq!(status, StatusCode::BAD_REQUEST); + assert_eq!(json["error"]["type"], "invalid_request_error"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn tokenize_chat_empty_messages_returns_400() { + let mut app = test_app().await; + + let (status, json) = post_json( + &mut app, + "/tokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "messages": [], + }), + ) + .await; + + assert_eq!(status, StatusCode::BAD_REQUEST); + assert_eq!(json["error"]["type"], "invalid_request_error"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn tokenize_chat_empty_message_content_returns_400() { + let mut app = test_app().await; + + let (status, json) = post_json( + &mut app, + "/tokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "messages": [{"role": "user", "content": ""}], + }), + ) + .await; + + assert_eq!(status, StatusCode::BAD_REQUEST); + assert_eq!(json["error"]["type"], "invalid_request_error"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn tokenize_unknown_model_returns_404() { + let mut app = test_app().await; + + let (status, json) = post_json( + &mut app, + "/tokenize", + json!({ + "model": "does-not-exist", + "prompt": "hello", + }), + ) + .await; + + assert_eq!(status, StatusCode::NOT_FOUND); + assert_eq!(json["error"]["type"], "invalid_request_error"); + assert_eq!(json["error"]["code"], "model_not_found"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn detokenize_unknown_model_returns_404() { + let mut app = test_app().await; + + let (status, json) = post_json( + &mut app, + "/detokenize", + json!({ + "model": "does-not-exist", + "tokens": [72, 101, 108, 108, 111], + }), + ) + .await; + + assert_eq!(status, StatusCode::NOT_FOUND); + assert_eq!(json["error"]["type"], "invalid_request_error"); + assert_eq!(json["error"]["code"], "model_not_found"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn detokenize_empty_tokens_returns_empty_prompt() { + let mut app = test_app().await; + + let (status, json) = post_json( + &mut app, + "/detokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "tokens": [], + }), + ) + .await; + + assert_eq!(status, StatusCode::OK); + assert_eq!(json["prompt"], ""); +} + +/// Decode an explicit token sequence — pins `/detokenize` independently of +/// `/tokenize` (the round-trip test alone would pass even if encode and decode +/// were both wrong in mirrored ways). +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn detokenize_decodes_known_token_ids() { + let mut app = test_app().await; + + let (status, json) = post_json( + &mut app, + "/detokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "tokens": [72, 101, 108, 108, 111], + }), + ) + .await; + + assert_eq!(status, StatusCode::OK); + assert_eq!(json["prompt"], "Hello"); +} + +/// `continue_final_message` without a trailing assistant message must 400. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn tokenize_chat_continue_without_assistant_returns_400() { + let mut app = test_app().await; + + let (status, json) = post_json( + &mut app, + "/tokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "messages": [{"role": "user", "content": "hi"}], + "add_generation_prompt": false, + "continue_final_message": true, + }), + ) + .await; + + assert_eq!(status, StatusCode::BAD_REQUEST); + assert_eq!(json["error"]["type"], "invalid_request_error"); +} + +/// `continue_final_message` must not append a new generation suffix vs `add_generation_prompt`. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn tokenize_chat_continue_final_vs_new_assistant_differs() { + let mut app = test_app().await; + let messages = json!([ + {"role": "user", "content": "hi"}, + {"role": "assistant", "content": "partial,"} + ]); + + let (_, continue_final) = post_json( + &mut app, + "/tokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "messages": messages, + "add_generation_prompt": false, + "continue_final_message": true, + "add_special_tokens": false, + }), + ) + .await; + let (_, new_assistant) = post_json( + &mut app, + "/tokenize", + json!({ + "model": "Qwen/Qwen1.5-0.5B-Chat", + "messages": messages, + "add_generation_prompt": true, + "continue_final_message": false, + "add_special_tokens": false, + }), + ) + .await; + + let continue_len = continue_final["tokens"].as_array().unwrap().len(); + let new_len = new_assistant["tokens"].as_array().unwrap().len(); + assert!(new_len > continue_len); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn world_size_endpoint_is_dev_mode_only() { + let mut app = test_app().await; + let response = app + .call( + Request::builder() + .uri("/get_world_size") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::NOT_FOUND); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn world_size_includes_data_parallelism_by_default() { + let ready = vllm_engine_core_client::protocol::handshake::EngineCoreReadyResponse { + world_size: 2, + data_parallel_size: 4, + ..default_ready_response() + }; + let (mut app, _engine_task) = test_dev_mode_app_with_ready(ready).await; + + let response = app + .call( + Request::builder() + .uri("/get_world_size") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json, json!({"world_size": 8})); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn world_size_excludes_data_parallelism_when_include_dp_false() { + let ready = vllm_engine_core_client::protocol::handshake::EngineCoreReadyResponse { + world_size: 2, + data_parallel_size: 4, + ..default_ready_response() + }; + let (mut app, _engine_task) = test_dev_mode_app_with_ready(ready).await; + + let response = app + .call( + Request::builder() + .uri("/get_world_size?include_dp=false") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + let json: serde_json::Value = serde_json::from_slice(&body).expect("decode json"); + assert_eq!(json, json!({"world_size": 2})); +} + +// ========================= Profiler route tests ========================= + +async fn test_profiling_app_with_engine_script(script: F) -> (axum::Router, MockEngineTask) +where + F: for<'a> FnOnce(&'a mut DealerSocket, &'a mut PushSocket) -> TestFuture<'a> + Send + 'static, +{ + let ipc = IpcNamespace::new().expect("create ipc namespace"); + let handshake_address = ipc.handshake_endpoint(); + let engine_id = b"engine-openai-profiler".to_vec(); + + let engine_task = MockEngineTask::new(spawn_mock_engine_task( + handshake_address.clone(), + engine_id.clone(), + move |dealer, push| script(dealer, push), + )); + + let client = EngineCoreClient::connect( + EngineCoreClientConfig::new_single(handshake_address) + .with_model_name("test-model") + .with_local_input_output_addresses( + Some(ipc.input_endpoint()), + Some(ipc.output_endpoint()), + ), + ) + .await + .expect("connect client"); + + let chat = ChatLlm::from_shared_backend(test_llm(client), Arc::new(FakeChatBackend::new())); + ( + build_router(Arc::new( + AppState::new(vec!["test-model".to_string()], chat) + .with_profiler(Some("torch".to_string())), + )), + engine_task, + ) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn start_profile_route_sends_expected_utility_call() { + let (app, engine_task) = test_profiling_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + + assert_eq!(array[2], Value::from("profile")); + assert_eq!(array[3], Value::Array(vec![Value::from(true), Value::Nil])); + + send_outputs(push, utility_outputs(call_id, utility_none_result())).await; + }) + }) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/start_profile") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + let status = response.status(); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + assert_eq!(status, StatusCode::OK, "{}", String::from_utf8_lossy(&body)); + assert!(body.is_empty()); + engine_task.await.expect("mock engine task"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn stop_profile_route_sends_expected_utility_call() { + let (app, engine_task) = test_profiling_app_with_engine_script(|dealer, push| { + boxed_test_future(async move { + let utility = recv_engine_message(dealer).await; + assert_eq!(utility[0].as_ref(), &[0x03]); + + let payload = decode_value(&utility[1]).expect("decode utility payload"); + let array = payload.as_array().expect("utility payload array"); + let call_id = array[1].as_u64().expect("call id"); + + assert_eq!(array[2], Value::from("profile")); + assert_eq!(array[3], Value::Array(vec![Value::from(false), Value::Nil])); + + send_outputs(push, utility_outputs(call_id, utility_none_result())).await; + }) + }) + .await; + + let response = app + .clone() + .call( + Request::builder() + .method("POST") + .uri("/stop_profile") + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + let status = response.status(); + let body = to_bytes(response.into_body(), usize::MAX).await.expect("read body"); + assert_eq!(status, StatusCode::OK, "{}", String::from_utf8_lossy(&body)); + assert!(body.is_empty()); + engine_task.await.expect("mock engine task"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[serial] +async fn profile_routes_are_hidden_when_profiling_is_disabled() { + let (chat, engine_task) = test_chat_with_engine_handle().await; + let app = build_router(Arc::new(AppState::new( + vec!["test-model".to_string()], + chat, + ))); + + for (method, uri) in [("POST", "/start_profile"), ("POST", "/stop_profile")] { + let response = app + .clone() + .call( + Request::builder() + .method(method) + .uri(uri) + .body(Body::empty()) + .expect("build request"), + ) + .await + .expect("call app"); + + assert_eq!(response.status(), StatusCode::NOT_FOUND, "{method} {uri}"); + } + + engine_task.abort_and_join().await; +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/tokenize.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/tokenize.rs new file mode 100644 index 0000000000000000000000000000000000000000..95895b7866839ec32feed712c87fb71de49de971 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/tokenize.rs @@ -0,0 +1,148 @@ +//! `POST /tokenize` and `POST /detokenize` (root paths, matching Python). +//! +//! Encode/decode runs entirely in-process via [`DynTokenizer`]; the inference +//! engine is not involved. + +mod types; + +use std::sync::Arc; + +use axum::Json; +use axum::extract::State; +use axum::http::HeaderMap; +use axum::response::{IntoResponse, Response}; +use thiserror_ext::AsReport as _; + +use crate::error::{ApiError, server_error}; +use crate::routes::openai::utils::validated_json::ValidatedJson; +use crate::routes::tokenize::types::{ + DetokenizeRequest, DetokenizeResponse, TokenizeChatRequest, TokenizeCompletionRequest, + TokenizeRequest, TokenizeResponse, +}; +use crate::state::AppState; +use crate::utils::resolve_base_request_id; + +/// Match Python `tokenize-{base}` where base is `X-Request-Id` or a new UUID. +fn tokenize_request_id(headers: &HeaderMap) -> String { + let base = resolve_base_request_id( + headers.get("X-Request-Id").and_then(|value| value.to_str().ok()), + None, + ); + format!("tokenize-{base}") +} + +/// Reject an unknown model name, matching the other handlers. +fn check_model(state: &AppState, model: Option<&str>) -> Result<(), ApiError> { + if let Some(model) = model + && !state.served_model_names().iter().any(|n| n == model) + { + return Err(ApiError::model_not_found(model.to_string())); + } + Ok(()) +} + +/// Build the `token_strs` vector when requested, via the tokenizer vocab. +fn token_strs(tokenizer: &vllm_text::tokenizer::DynTokenizer, ids: &[u32]) -> Vec { + // Unknown IDs yield "" — intentional; matches Python's convert_ids_to_tokens behaviour. + ids.iter().map(|&id| tokenizer.id_to_token(id).unwrap_or_default()).collect() +} + +pub async fn tokenize( + State(state): State>, + headers: HeaderMap, + ValidatedJson(body): ValidatedJson, +) -> Response { + let request_id = tokenize_request_id(&headers); + let tokenizer = state.chat.text().tokenizer(); + let max_model_len = state.chat.engine_core_client().max_model_len(); + + let result = match body { + // Completion form: encode the raw `prompt` string (no chat template). + TokenizeRequest::Completion(req) => tokenize_completion(&state, &tokenizer, req), + // Chat form: render `messages` through the template, then encode (see `tokenize_chat`). + TokenizeRequest::Chat(req) => tokenize_chat(&state, &request_id, req).await, + }; + + match result { + Ok((tokens, want_strs)) => { + let token_strs = want_strs.then(|| token_strs(&tokenizer, &tokens)); + Json(TokenizeResponse { + count: tokens.len(), + max_model_len, + tokens, + token_strs, + }) + .into_response() + } + Err(error) => error.into_response(), + } +} + +fn tokenize_completion( + state: &AppState, + tokenizer: &vllm_text::tokenizer::DynTokenizer, + req: TokenizeCompletionRequest, +) -> Result<(Vec, bool), ApiError> { + check_model(state, req.model.as_deref())?; + let tokens = tokenizer + .encode(&req.prompt, req.add_special_tokens) + .map_err(|e| server_error!("tokenize failed: {}", e.to_report_string()))?; + Ok((tokens, req.return_token_strs)) +} + +/// HTTP adapter for the chat-shaped `/tokenize` body. +/// +/// Not [`vllm_chat::ChatLlm::tokenize_chat`]: this checks the model name and maps +/// errors to [`ApiError`]; the chat-crate method does render → finalize → encode. +async fn tokenize_chat( + state: &AppState, + request_id: &str, + req: TokenizeChatRequest, +) -> Result<(Vec, bool), ApiError> { + check_model(state, req.model.as_deref())?; + let return_token_strs = req.return_token_strs; + // `continue_final_message` / `add_generation_prompt` mutual exclusion is + // enforced in `normalize_generation_prompt_mode` inside `into_chat_request`. + let tokens = state + .chat + .tokenize_chat(req.into_chat_request(request_id.to_string())?) + .await + .map_err(|e| server_error!("tokenize failed: {}", e.to_report_string()))?; + Ok((tokens, return_token_strs)) +} + +pub async fn detokenize( + State(state): State>, + ValidatedJson(body): ValidatedJson, +) -> Response { + if let Err(error) = check_model(&state, body.model.as_deref()) { + return error.into_response(); + } + let tokenizer = state.chat.text().tokenizer(); + match tokenizer.decode(&body.tokens, /* skip_special_tokens = */ false) { + Ok(prompt) => Json(DetokenizeResponse { prompt }).into_response(), + Err(e) => server_error!("detokenize failed: {}", e.to_report_string()).into_response(), + } +} + +#[cfg(test)] +mod tests { + use axum::http::{HeaderMap, HeaderValue}; + + use super::tokenize_request_id; + + #[test] + fn tokenize_request_id_prefers_x_request_id_header() { + let mut headers = HeaderMap::new(); + headers.insert("X-Request-Id", HeaderValue::from_static("client-req-1")); + assert_eq!(tokenize_request_id(&headers), "tokenize-client-req-1"); + } + + #[test] + fn tokenize_request_id_generates_uuid_when_header_missing() { + let headers = HeaderMap::new(); + let id = tokenize_request_id(&headers); + assert!(id.starts_with("tokenize-")); + assert_ne!(id, "tokenize-"); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/tokenize/types.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/tokenize/types.rs new file mode 100644 index 0000000000000000000000000000000000000000..b678b11ba29ac06e3b8330198974cadaaed8c06f --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/tokenize/types.rs @@ -0,0 +1,211 @@ +use std::collections::HashMap; + +use itertools::Itertools as _; +use serde::{Deserialize, Serialize}; +use serde_json::Value; +use validator::{Validate, ValidationErrors}; +use vllm_chat::{ChatOptions, ChatRequest, ChatToolChoice, SamplingParams}; +use vllm_text::output::TextDecodeOptions; + +use crate::error::ApiError; +use crate::routes::openai::chat_completions::convert::{ + convert_message, convert_tools, normalize_generation_prompt_mode, +}; +use crate::routes::openai::utils::types::{ + ChatMessage, Normalizable, Tool, default_true, validate_messages, +}; + +/// `POST /tokenize` body. Untagged: a JSON object with `messages` parses as the +/// chat variant; one with `prompt` parses as the completion variant. +#[derive(Debug, Clone, Deserialize)] +#[serde(untagged)] +pub enum TokenizeRequest { + Chat(TokenizeChatRequest), + Completion(TokenizeCompletionRequest), +} + +#[derive(Debug, Clone, Deserialize)] +pub struct TokenizeCompletionRequest { + pub model: Option, + pub prompt: String, + #[serde(default = "default_true")] + pub add_special_tokens: bool, + #[serde(default)] + pub return_token_strs: bool, +} + +#[derive(Debug, Clone, Deserialize, Validate)] +pub struct TokenizeChatRequest { + pub model: Option, + #[validate(custom(function = "validate_messages"))] + pub messages: Vec, + #[serde(default = "default_true")] + pub add_generation_prompt: bool, + #[serde(default)] + pub continue_final_message: bool, + #[serde(default)] // chat default is FALSE (template adds specials) + pub add_special_tokens: bool, + #[serde(default)] + pub return_token_strs: bool, + #[serde(default)] + pub chat_template: Option, + #[serde(default)] + pub chat_template_kwargs: Option>, + #[serde(default)] + pub tools: Option>, +} + +impl TokenizeChatRequest { + /// Lower this tokenize body into a [`ChatRequest`] for template rendering. + /// + /// Reuses [`convert_message`] and [`normalize_generation_prompt_mode`] from + /// `chat_completions/convert` so message lowering and generation-prompt + /// rules match chat completions. Only fields that affect rendering are set; + /// `sampling_params`, `decode_options`, etc. stay at default because + /// tokenize never generates. + pub fn into_chat_request(self, request_id: String) -> Result { + let messages: Vec<_> = self.messages.into_iter().map(convert_message).try_collect()?; + let generation_prompt_mode = normalize_generation_prompt_mode( + Some(self.add_generation_prompt), + self.continue_final_message, + &messages, + )?; + + Ok(ChatRequest { + request_id, + messages, + sampling_params: SamplingParams::default(), + chat_options: ChatOptions { + generation_prompt_mode, + chat_template: self.chat_template, + reasoning_effort: None, + template_kwargs: self.chat_template_kwargs.unwrap_or_default(), + }, + tools: convert_tools(self.tools)?, + tool_choice: ChatToolChoice::Auto, + parallel_tool_calls: true, + decode_options: TextDecodeOptions::default(), + intermediate: false, + priority: 0, + documents: None, + cache_salt: None, + add_special_tokens: self.add_special_tokens, + data_parallel_rank: None, + lora_request: None, + }) + } +} + +#[derive(Debug, Clone, Deserialize)] +pub struct DetokenizeRequest { + pub model: Option, + pub tokens: Vec, +} + +#[derive(Debug, Clone, Serialize)] +pub struct TokenizeResponse { + pub count: usize, + pub max_model_len: u32, + pub tokens: Vec, + #[serde(skip_serializing_if = "Option::is_none")] + pub token_strs: Option>, +} + +#[derive(Debug, Clone, Serialize)] +pub struct DetokenizeResponse { + pub prompt: String, +} + +// ---- trait impls required by ValidatedJson ---- +impl Validate for TokenizeRequest { + fn validate(&self) -> Result<(), ValidationErrors> { + if let Self::Chat(req) = self { + req.validate()?; + } + Ok(()) + } +} +impl Validate for DetokenizeRequest { + fn validate(&self) -> Result<(), ValidationErrors> { + Ok(()) + } +} +impl Normalizable for TokenizeRequest {} // default no-op normalize() +impl Normalizable for DetokenizeRequest {} + +#[cfg(test)] +mod tests { + use serde_json::json; + use vllm_chat::ChatTool; + + use super::*; + use crate::routes::openai::utils::types::{ChatMessage, MessageContent}; + + #[test] + fn tokenize_request_converts_openai_tools() { + // The untagged `TokenizeRequest` must resolve a messages+tools body to + // the chat variant and accept standard OpenAI tool objects + // (`{"type":"function",...}`), then convert them to `ChatTool`. + let request: TokenizeRequest = serde_json::from_value(json!({ + "messages": [{"role": "user", "content": "hi"}], + "tools": [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get weather", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + }, + }, + }], + })) + .expect("OpenAI tool JSON deserializes to the chat variant"); + + let TokenizeRequest::Chat(req) = request else { + panic!("messages+tools body should parse as the chat variant"); + }; + + let chat_request = + req.into_chat_request("tokenize-test".to_string()).expect("request is valid"); + + assert_eq!( + chat_request.tools, + vec![ChatTool { + name: "get_weather".to_string(), + description: Some("Get weather".to_string()), + parameters: json!({ + "type": "object", + "properties": {"city": {"type": "string"}}, + }), + strict: None, + }] + ); + } + + #[test] + fn into_chat_request_rejects_conflicting_generation_flags() { + let req = TokenizeChatRequest { + model: None, + messages: vec![ChatMessage::User { + content: MessageContent::Text("hi".to_string()), + name: None, + }], + add_generation_prompt: true, + continue_final_message: true, + add_special_tokens: false, + return_token_strs: false, + chat_template: None, + chat_template_kwargs: None, + tools: None, + }; + + let error = req + .into_chat_request("tokenize-test".to_string()) + .expect_err("conflicting flags"); + assert_eq!( + error.to_error_response().error.message, + "Cannot set both `continue_final_message` and `add_generation_prompt` to True." + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/version.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/version.rs new file mode 100644 index 0000000000000000000000000000000000000000..a28f0f975b55f06582ca6cd85ffbb97e96e997e5 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/version.rs @@ -0,0 +1,23 @@ +use std::sync::Arc; + +use axum::Json; +use axum::extract::State; +use serde::Serialize; + +use crate::state::AppState; + +#[derive(Serialize)] +pub(crate) struct VersionResponse { + version: String, + rust_frontend_version: &'static str, +} + +/// Get engine and Rust frontend version metadata. +pub async fn version(State(state): State>) -> Json { + let version = state.engine_core_client().vllm_version().to_string(); + + Json(VersionResponse { + version, + rust_frontend_version: env!("CARGO_PKG_VERSION"), + }) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/routes/world_size.rs b/asian_ai/performance/vLLM/rust/src/server/src/routes/world_size.rs new file mode 100644 index 0000000000000000000000000000000000000000..d44a9ca54c814b0b26af750ee5ca874fe09ced45 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/routes/world_size.rs @@ -0,0 +1,54 @@ +use std::sync::Arc; + +use axum::Json; +use axum::extract::{Query, State}; +use serde::{Deserialize, Serialize}; + +use crate::error::ApiError; +use crate::state::AppState; + +#[derive(Debug, Deserialize)] +pub(crate) struct WorldSizeParams { + /// If true (default), returns the world size including data parallelism + /// (TP * PP * DP). If false, returns the world size without data + /// parallelism (TP * PP). + #[serde(default = "default_true")] + include_dp: bool, +} + +const fn default_true() -> bool { + true +} + +#[derive(Serialize)] +pub(crate) struct WorldSizeResponse { + world_size: u64, +} + +/// Get the world size from the parallel config. +/// +/// Currently reads static values captured during the engine startup handshake. +/// +/// TODO: If the world size can change at runtime (e.g. elastic EP scaling, +/// DP rank recovery), this should be switched to either: +/// - A `call_utility("get_world_size", (include_dp,))` RPC to the Python +/// engine for live values (simple, adds one ZMQ round-trip per request), or +/// - A push-based approach where the engine sends config updates via the +/// output stream into shared state (zero per-request overhead, more complex). +pub async fn get_world_size( + State(state): State>, + Query(params): Query, +) -> Result, ApiError> { + let client = state.engine_core_client(); + + let ws = client.world_size(); + + let world_size = if params.include_dp { + let dp = client.data_parallel_size(); + ws * dp + } else { + ws + }; + + Ok(Json(WorldSizeResponse { world_size })) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/runtime.rs b/asian_ai/performance/vLLM/rust/src/server/src/runtime.rs new file mode 100644 index 0000000000000000000000000000000000000000..2f3eed2c3590b442585bb78dbf71c67277db0fd8 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/runtime.rs @@ -0,0 +1,54 @@ +use tokio::runtime::Builder; +use tracing::{info, warn}; +use vllm_engine_core_client::runtime::BackgroundShutdownRuntime; + +const REQUEST_WORKER_THREADS_ENV: &str = "VLLM_RS_REQUEST_WORKER_THREADS"; +const DEFAULT_MAX_REQUEST_WORKER_THREADS: usize = 32; + +/// Build a Tokio runtime for heavyweight request paths outside the HTTP runtime. +/// +/// The server middleware uses this runtime for inference and tokenization +/// routes so CPU-heavy request preparation does not monopolize the HTTP +/// runtime's worker queue. Dropping the wrapper shuts the runtime down in the +/// background. +pub(crate) fn build_request_runtime() -> BackgroundShutdownRuntime { + Builder::new_multi_thread() + .enable_all() + .thread_name("vllm-request") + .worker_threads(request_worker_threads()) + .build() + .expect("failed to build request runtime") + .into() +} + +/// Get the number of worker threads to use for the request runtime. +/// +/// If `VLLM_RS_REQUEST_WORKER_THREADS` is set to a valid positive integer, it is +/// used directly. Otherwise, the runtime uses available parallelism capped by +/// `DEFAULT_MAX_REQUEST_WORKER_THREADS`. +fn request_worker_threads() -> usize { + if let Some(value) = std::env::var_os(REQUEST_WORKER_THREADS_ENV) { + match value.to_string_lossy().parse::() { + Ok(worker_threads) if worker_threads > 0 => return worker_threads, + _ => warn!( + value = %value.to_string_lossy(), + "ignoring invalid {REQUEST_WORKER_THREADS_ENV}" + ), + } + } + + std::thread::available_parallelism() + .map(|parallelism| { + let available = parallelism.get(); + let worker_threads = available.min(DEFAULT_MAX_REQUEST_WORKER_THREADS); + if worker_threads < available { + info!( + available_parallelism = available, + capped_worker_threads = worker_threads, + "capping request runtime worker threads, set {REQUEST_WORKER_THREADS_ENV} to override" + ); + } + worker_threads + }) + .unwrap_or(DEFAULT_MAX_REQUEST_WORKER_THREADS) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/server_info.rs b/asian_ai/performance/vLLM/rust/src/server/src/server_info.rs new file mode 100644 index 0000000000000000000000000000000000000000..bd6f753979337a70f3195758d52d50d07f58bcf0 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/server_info.rs @@ -0,0 +1,144 @@ +use std::collections::BTreeMap; + +use serde_json::{Value, json}; + +use crate::config::Config; + +const SENSITIVE_VLLM_ENV_PATTERNS: &[&str] = + &["KEY", "SECRET", "TOKEN", "PASSWORD", "CREDENTIAL", "AUTH"]; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum ServerInfoConfigFormat { + Text, + Json, +} + +/// Snapshot returned by `/server_info`. +#[derive(Debug, Clone)] +pub(crate) struct ServerInfoSnapshot { + vllm_config_text: String, + vllm_config_json: Value, + vllm_env: BTreeMap, + system_env: BTreeMap, +} + +impl ServerInfoSnapshot { + /// Capture the runtime configuration fields available to the Rust frontend. + pub(crate) fn from_config(config: &Config) -> Self { + let vllm_config_json = + serde_json::to_value(config).expect("server info value must serialize"); + + Self { + vllm_config_text: render_config_text(&vllm_config_json), + vllm_config_json, + vllm_env: collect_vllm_env(), + system_env: collect_system_env(), + } + } + + pub(crate) fn response(&self, config_format: ServerInfoConfigFormat) -> Value { + let vllm_config = match config_format { + ServerInfoConfigFormat::Text => Value::String(self.vllm_config_text.clone()), + ServerInfoConfigFormat::Json => self.vllm_config_json.clone(), + }; + + json!({ + "vllm_config": vllm_config, + "vllm_env": self.vllm_env.clone(), + "system_env": self.system_env.clone(), + }) + } +} + +fn render_config_text(config: &Value) -> String { + match config { + Value::Object(fields) => fields + .iter() + .map(|(key, value)| format!("{key}={}", render_config_text_value(value))) + .collect::>() + .join("\n"), + _ => render_config_text_value(config), + } +} + +fn render_config_text_value(value: &Value) -> String { + match value { + Value::Null => "None".to_string(), + Value::String(value) => value.clone(), + _ => value.to_string(), + } +} + +fn collect_vllm_env() -> BTreeMap { + std::env::vars().filter(|(key, _)| is_public_vllm_env_key(key)).collect() +} + +fn is_public_vllm_env_key(key: &str) -> bool { + let key = key.to_ascii_uppercase(); + key.starts_with("VLLM_") + && !SENSITIVE_VLLM_ENV_PATTERNS.iter().any(|pattern| key.contains(pattern)) +} + +fn collect_system_env() -> BTreeMap { + BTreeMap::from([ + ("arch".to_string(), std::env::consts::ARCH.to_string()), + ("family".to_string(), std::env::consts::FAMILY.to_string()), + ("os".to_string(), std::env::consts::OS.to_string()), + ]) +} + +#[cfg(test)] +mod tests { + use std::collections::BTreeSet; + + use serde_json::{Value, json}; + + use super::{is_public_vllm_env_key, render_config_text}; + + #[test] + fn render_config_text_formats_config_snapshot() { + let rendered = render_config_text(&json!({ + "model": "test-model", + "served_model_name": ["served-model"], + "chat_template": null, + "enable_log_requests": true, + })); + let lines = rendered.lines().collect::>(); + + assert_eq!( + lines, + BTreeSet::from([ + "chat_template=None", + "enable_log_requests=true", + "model=test-model", + "served_model_name=[\"served-model\"]", + ]) + ); + assert_eq!( + render_config_text(&Value::String("inline".to_string())), + "inline" + ); + assert_eq!(render_config_text(&Value::Null), "None"); + } + + #[test] + fn server_info_env_filter_excludes_sensitive_vllm_keys() { + for key in [ + "VLLM_API_KEY", + "VLLM_AUTH_TOKEN", + "VLLM_SECRET", + "VLLM_PASSWORD", + "VLLM_CREDENTIAL_FILE", + "vllm_token", + ] { + assert!(!is_public_vllm_env_key(key), "{key}"); + } + } + + #[test] + fn server_info_env_filter_includes_public_vllm_keys() { + assert!(is_public_vllm_env_key("VLLM_LOGGING_LEVEL")); + assert!(is_public_vllm_env_key("VLLM_USE_MODELSCOPE")); + assert!(!is_public_vllm_env_key("OTHER_ENV")); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/state.rs b/asian_ai/performance/vLLM/rust/src/server/src/state.rs new file mode 100644 index 0000000000000000000000000000000000000000..0225502956eb0245d2cfb3dd6db8a8d77dada914 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/state.rs @@ -0,0 +1,262 @@ +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, OnceLock}; + +use serde_json::Value; +use sha2::{Digest, Sha256}; +use tokio::runtime::Runtime; +use tokio::time::{Duration, Instant, sleep_until}; +use tracing::warn; +use vllm_chat::ChatLlm; +use vllm_engine_core_client::EngineCoreClient; +use vllm_engine_core_client::protocol::lora::LoraRequest; +use vllm_engine_core_client::runtime::BackgroundShutdownRuntime; + +use crate::config::{ApiServerOptions, CorsConfig}; +use crate::lora::{LoadLoraError, LoraManager, LoraModelResolution, UnloadLoraError}; +use crate::runtime::build_request_runtime; +use crate::server_info::{ServerInfoConfigFormat, ServerInfoSnapshot}; + +const SHUTDOWN_REFCOUNT_POLL_INTERVAL: Duration = Duration::from_millis(100); + +pub(crate) type ApiKeyHash = [u8; 32]; + +pub(crate) fn hash_api_key(api_key: &str) -> ApiKeyHash { + Sha256::digest(api_key.as_bytes()).into() +} + +/// Shared router state for the minimal single-model OpenAI server. +pub struct AppState { + /// All public model IDs served by this frontend. The first entry is the + /// primary ID used in responses; all entries are valid in requests. + served_model_names: Vec, + /// Shared chat facade used by all requests. + pub chat: ChatLlm, + /// HTTP/API-server behavior switches. + pub api_server_options: ApiServerOptions, + /// CORS settings applied to every HTTP response. + pub cors: CorsConfig, + /// Runtime server information returned by `/server_info`, when available. + server_info: Option, + /// SHA-256 hashes of API keys accepted as bearer tokens for guarded routes. + api_key_hashes: Vec, + /// Number of in-flight inference requests currently owned by this frontend. + server_load: AtomicU64, + /// Dynamic LoRA adapter registry. + lora_manager: LoraManager, + /// Backend model path reported as `root` for base-model cards. + model_path: Option, + /// Lazily initialized runtime for heavyweight request paths. + request_runtime: OnceLock, + /// Profiler mode that registers `/start_profile` and `/stop_profile` + /// routes when present. + pub profiler: Option, +} + +impl AppState { + /// Construct one application state instance. + /// + /// `served_model_names` must be non-empty; the first entry is the primary + /// model ID returned in API responses. + /// + /// # Panics + /// + /// Panics if `served_model_names` is empty. + pub fn new(served_model_names: Vec, chat: ChatLlm) -> Self { + assert!( + !served_model_names.is_empty(), + "served_model_names must not be empty" + ); + Self { + served_model_names, + chat, + api_server_options: ApiServerOptions::default(), + cors: CorsConfig::default(), + server_info: None, + api_key_hashes: Vec::new(), + server_load: AtomicU64::new(0), + lora_manager: LoraManager::new(), + model_path: None, + request_runtime: OnceLock::new(), + profiler: None, + } + } + + /// Set HTTP/API-server behavior switches. + pub fn with_api_server_options(mut self, options: ApiServerOptions) -> Self { + self.api_server_options = options; + self + } + + /// Set the CORS settings applied to every HTTP response. + pub fn with_cors(mut self, cors: CorsConfig) -> Self { + self.cors = cors; + self + } + + /// Set the backend model path reported as `root` for base-model cards. + pub fn with_model_path(mut self, model_path: String) -> Self { + self.model_path = Some(model_path); + self + } + + /// Set the profiler mode that enables `/start_profile` and `/stop_profile`. + pub fn with_profiler(mut self, profiler: Option) -> Self { + self.profiler = profiler; + self + } + + /// Attach the runtime server information snapshot used by `/server_info`. + pub(crate) fn with_server_info(mut self, server_info: ServerInfoSnapshot) -> Self { + self.server_info = Some(server_info); + self + } + + /// Build a `/server_info` response payload. + pub(crate) fn server_info_response( + &self, + config_format: ServerInfoConfigFormat, + ) -> Option { + self.server_info.as_ref().map(|server_info| server_info.response(config_format)) + } + + /// Configure API keys accepted by guarded HTTP routes. + pub fn with_api_keys(mut self, api_keys: Vec) -> Self { + self.api_key_hashes = api_keys + .into_iter() + .filter(|key| !key.is_empty()) + .map(|key| hash_api_key(&key)) + .collect(); + self + } + + pub(crate) fn has_api_keys(&self) -> bool { + !self.api_key_hashes.is_empty() + } + + pub(crate) fn api_key_hashes(&self) -> &[ApiKeyHash] { + &self.api_key_hashes + } + + /// The primary model name echoed back in API responses (the first served + /// name). + pub fn primary_model_name(&self) -> &str { + self.served_model_names.first().map(String::as_str).unwrap_or_default() + } + + /// All model names served by this frontend. + pub fn served_model_names(&self) -> &[String] { + &self.served_model_names + } + + /// Backend model path reported as `root` for base-model cards, if known. + pub fn model_path(&self) -> Option<&str> { + self.model_path.as_deref() + } + + /// Snapshot the loaded LoRA adapters in load order, for `/v1/models` cards. + pub async fn served_lora_requests(&self) -> Vec { + self.lora_manager.served_lora_requests().await + } + + /// Resolve the requested model against one dynamic LoRA registry snapshot. + pub async fn resolve_model_with_loras(&self, model_name: Option<&str>) -> LoraModelResolution { + self.lora_manager.resolve_model(&self.served_model_names, model_name).await + } + + /// Load one dynamic LoRA adapter and register it as a public model name. + pub async fn load_lora( + &self, + lora_name: String, + lora_path: String, + load_inplace: bool, + is_3d_lora_weight: bool, + ) -> Result { + self.lora_manager + .load_lora( + self.engine_core_client(), + &self.served_model_names, + lora_name, + lora_path, + load_inplace, + is_3d_lora_weight, + ) + .await + } + + /// Remove one dynamic LoRA adapter from the engine and public model + /// registry. + pub async fn unload_lora( + &self, + lora_name: &str, + lora_int_id: Option, + ) -> Result { + self.lora_manager + .unload_lora(self.engine_core_client(), lora_name, lora_int_id) + .await + } + + /// Return a reference to the underlying engine core client for utility + /// calls. + pub(crate) fn engine_core_client(&self) -> &EngineCoreClient { + self.chat.engine_core_client() + } + + /// Runtime used by middleware to isolate heavyweight request handlers from + /// the HTTP reactor. + pub(crate) fn request_runtime(&self) -> &Runtime { + self.request_runtime.get_or_init(build_request_runtime) + } + + /// Return the current in-flight inference request count for the `/load` + /// endpoint. + pub fn server_load(&self) -> u64 { + self.server_load.load(Ordering::Relaxed) + } + + /// Increment the in-flight inference request count, called by the load + /// tracking middleware. + pub(crate) fn increment_server_load(&self) { + self.server_load.fetch_add(1, Ordering::Relaxed); + } + + /// Decrement the in-flight inference request count, called by the load + /// tracking middleware. + pub(crate) fn decrement_server_load(&self) { + self.server_load.fetch_sub(1, Ordering::Relaxed); + } + + /// Wait until all request-owned references are dropped, then shut down the + /// engine client. + /// + /// If the deadline elapses while request/connection tasks still hold state + /// references, skip the clean engine-client shutdown and let process + /// teardown reclaim the remaining resources. + pub async fn shutdown(mut self: Arc, deadline: Instant) -> anyhow::Result<()> { + loop { + match Arc::try_unwrap(self) { + Ok(state) => { + state.chat.shutdown().await?; + drop(state.request_runtime); // shutdown in background + return Ok(()); + } + Err(state) => self = state, + } + let ref_count = Arc::strong_count(&self); + + let now = Instant::now(); + if now >= deadline { + warn!( + ref_count, + "shutdown deadline elapsed before app state became idle; skipping engine-client shutdown" + ); + return Ok(()); + } + + sleep_until(std::cmp::min( + deadline, + now + SHUTDOWN_REFCOUNT_POLL_INTERVAL, + )) + .await; + } + } +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/tls.rs b/asian_ai/performance/vLLM/rust/src/server/src/tls.rs new file mode 100644 index 0000000000000000000000000000000000000000..446751fcbe854f6f94756dbf841cf9d77218ede9 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/tls.rs @@ -0,0 +1,119 @@ +//! OpenSSL server-config construction for TLS termination. +//! +//! Builds an OpenSSL [`SslContext`] from the uvicorn-style `ssl_*` arguments +//! (certificate chain, private key, mTLS client verifier, optional cipher list). +//! The `tls-listener` crate drives the handshake on each accepted connection. +//! +//! Crypto runs through whichever OpenSSL the binary links (system by default, +//! vendored when built with that feature). + +use std::path::Path; +use std::time::Duration; + +use anyhow::{Context as _, Result}; +use openssl::ssl::{ + AlpnError, SslAcceptor, SslAcceptorBuilder, SslContext, SslContextBuilder, SslFiletype, + SslMethod, SslOptions, SslVerifyMode, select_next_proto, +}; + +use crate::config::TlsConfig; + +/// Time a client has to complete the TLS handshake before the connection is dropped. +pub(crate) const TLS_HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(60); + +/// ALPN wire bytes for HTTP/2 (length-prefixed). +const ALPN_H2: &[u8] = b"\x02h2"; + +/// Build the shared OpenSSL acceptor from validated [`TlsConfig`]: the full +/// certificate chain, the private key (`key_file`, or the certificate file when +/// unset), the mTLS client verifier, and an optional cipher list. +/// +/// Starts from the Mozilla intermediate baseline (forward-secret AEAD suites, +/// TLS 1.2 floor, server cipher preference, no compression), a slightly +/// stricter subset of the Python frontend's default suites; `--ssl-ciphers` +/// overrides it. +fn build_server_builder(tls: &TlsConfig) -> Result { + let cert_file = tls.cert_file.as_deref().context("--ssl-certfile is required to enable TLS")?; + + let mut builder = SslAcceptor::mozilla_intermediate_v5(SslMethod::tls_server()) + .context("failed to initialize TLS")?; + builder.set_options(SslOptions::CIPHER_SERVER_PREFERENCE); + + // Load the whole chain (leaf + intermediates), not just the leaf, so + // deployments behind an intermediate CA serve a complete chain. + ensure_exists(cert_file, "--ssl-certfile")?; + builder.set_certificate_chain_file(cert_file).with_context(|| { + format!("failed to parse certificate chain in --ssl-certfile {cert_file:?}") + })?; + + // When `key_file` is unset the key is read from the certificate file + // (combined PEM). + let key_file = tls.key_file.as_deref().unwrap_or(cert_file); + ensure_exists(key_file, "private key file")?; + builder + .set_private_key_file(key_file, SslFiletype::PEM) + .with_context(|| format!("failed to parse private key in {key_file:?}"))?; + builder + .check_private_key() + .context("the certificate and private key do not match")?; + + configure_client_auth(&mut builder, tls)?; + + if let Some(ciphers) = tls.ciphers.as_deref().filter(|c| !c.is_empty()) { + builder + .set_cipher_list(ciphers) + .with_context(|| format!("invalid --ssl-ciphers {ciphers:?}"))?; + } + + Ok(builder) +} + +/// Build the HTTP [`SslContext`] (HTTP/1.1; no ALPN, matching uvicorn). +pub(crate) fn build_server_config(tls: &TlsConfig) -> Result { + Ok(build_server_builder(tls)?.build().into_context()) +} + +/// Build the gRPC [`SslContext`]: identical to [`build_server_config`] but +/// negotiates ALPN `h2`, which HTTP/2 over TLS requires. +pub(crate) fn build_grpc_server_config(tls: &TlsConfig) -> Result { + let mut builder = build_server_builder(tls)?; + builder.set_alpn_select_callback(|_ssl, client| { + select_next_proto(ALPN_H2, client).ok_or(AlpnError::NOACK) + }); + Ok(builder.build().into_context()) +} + +/// Fail loudly with a flag-named message when a configured file is missing, +/// distinguishing it from a malformed-PEM error raised later by OpenSSL (whose +/// `ErrorStack` does not name the offending file). +fn ensure_exists(path: &str, what: &str) -> Result<()> { + std::fs::metadata(Path::new(path)) + .map(drop) + .with_context(|| format!("failed to read {what} {path:?}")) +} + +/// Apply the `cert_reqs` client-certificate policy: 0 = none, 1 = optional +/// (verify if presented, allow anonymous), 2 = required. `PEER` without a custom +/// verify callback still rejects a presented-but-untrusted certificate. +fn configure_client_auth(builder: &mut SslContextBuilder, tls: &TlsConfig) -> Result<()> { + if tls.cert_reqs == 0 { + builder.set_verify(SslVerifyMode::NONE); + return Ok(()); + } + + let ca_file = tls + .ca_certs + .as_deref() + .context("--ssl-ca-certs is required for client certificate verification")?; + ensure_exists(ca_file, "--ssl-ca-certs")?; + builder + .set_ca_file(ca_file) + .with_context(|| format!("failed to parse --ssl-ca-certs {ca_file:?}"))?; + + let mut mode = SslVerifyMode::PEER; + if tls.cert_reqs == 2 { + mode |= SslVerifyMode::FAIL_IF_NO_PEER_CERT; + } + builder.set_verify(mode); + Ok(()) +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/tls_tests.rs b/asian_ai/performance/vLLM/rust/src/server/src/tls_tests.rs new file mode 100644 index 0000000000000000000000000000000000000000..6a8e7e78af22790429ab915d5917d1018f0a04e3 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/tls_tests.rs @@ -0,0 +1,677 @@ +//! TLS tests: `build_server_config` unit checks plus end-to-end OpenSSL handshakes +//! through the production listener/connection path, with a trivial router since TLS +//! terminates below the app. + +use std::pin::Pin; +use std::time::Duration; + +use axum::Router; +use axum::routing::get; +use openssl::asn1::Asn1Time; +use openssl::bn::{BigNum, MsbOption}; +use openssl::ec::{EcGroup, EcKey}; +use openssl::hash::MessageDigest; +use openssl::nid::Nid; +use openssl::pkey::{PKey, Private}; +use openssl::ssl::{SslConnector, SslFiletype, SslMethod, SslVersion}; +use openssl::x509::extension::{BasicConstraints, KeyUsage, SubjectAlternativeName}; +use openssl::x509::{X509, X509NameBuilder}; +use tempfile::TempDir; +use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; +use tokio::net::TcpStream; +use tokio_openssl::SslStream; +use tokio_util::sync::CancellationToken; + +use crate::config::{HttpListenerMode, TlsConfig}; +use crate::listener::{Listener, MaybeTlsListener}; +use crate::{ConnectionTimeouts, serve_connections, tls}; + +// ============================================================================ +// Test infrastructure +// ============================================================================ + +/// A throwaway CA + server/client/untrusted/chain cert set as PEM files in a +/// temp dir; dropping it deletes them. +pub(crate) struct TestCerts { + dir: TempDir, +} + +impl TestCerts { + pub(crate) fn generate() -> Self { + let dir = tempfile::tempdir().expect("tempdir"); + + let (ca, ca_key) = build_ca(); + let (server, server_key) = build_leaf("server", &["127.0.0.1", "localhost"], &ca, &ca_key); + let (client, client_key) = build_leaf("client", &[], &ca, &ca_key); + let (untrusted, untrusted_key) = build_self_signed("untrusted client"); + + // Leaf signed by an intermediate (itself signed by the root); the cert + // file holds leaf + intermediate, for the chain-serving test. + let (intermediate, intermediate_key) = build_intermediate(&ca, &ca_key); + let (chain_leaf, chain_leaf_key) = build_leaf( + "chain", + &["127.0.0.1", "localhost"], + &intermediate, + &intermediate_key, + ); + + let server_pem = pem(&server); + let server_key_pem = key_pem(&server_key); + let files = [ + ("ca.pem", pem(&ca)), + ("server.pem", server_pem.clone()), + ("server.key", server_key_pem.clone()), + ("client.pem", pem(&client)), + ("client.key", key_pem(&client_key)), + ("untrusted_client.pem", pem(&untrusted)), + ("untrusted_client.key", key_pem(&untrusted_key)), + ( + "server_combined.pem", + format!("{server_pem}{server_key_pem}"), + ), + ( + "server_chain.pem", + format!("{}{}", pem(&chain_leaf), pem(&intermediate)), + ), + ("server_chain.key", key_pem(&chain_leaf_key)), + ]; + for (name, contents) in files { + std::fs::write(dir.path().join(name), contents).expect("write fixture"); + } + Self { dir } + } + + /// Absolute path to a fixture by name; the file need not exist. + pub(crate) fn path(&self, name: &str) -> String { + self.dir.path().join(name).to_str().expect("utf-8 path").to_string() + } +} + +fn gen_key() -> PKey { + let group = EcGroup::from_curve_name(Nid::X9_62_PRIME256V1).expect("ec group"); + let ec = EcKey::generate(&group).expect("ec key"); + PKey::from_ec_key(ec).expect("pkey") +} + +fn serial() -> openssl::asn1::Asn1Integer { + let mut bn = BigNum::new().expect("bignum"); + bn.rand(159, MsbOption::MAYBE_ZERO, false).expect("rand serial"); + bn.to_asn1_integer().expect("asn1 serial") +} + +fn x509_name(cn: &str) -> openssl::x509::X509Name { + let mut builder = X509NameBuilder::new().expect("name builder"); + builder.append_entry_by_text("CN", cn).expect("cn"); + builder.build() +} + +fn pem(cert: &X509) -> String { + String::from_utf8(cert.to_pem().expect("cert pem")).expect("utf-8 cert") +} + +fn key_pem(key: &PKey) -> String { + String::from_utf8(key.private_key_to_pem_pkcs8().expect("key pem")).expect("utf-8 key") +} + +/// A self-signed CA used to sign the server/client leaf certs. +fn build_ca() -> (X509, PKey) { + let key = gen_key(); + let name = x509_name("vLLM Test CA"); + let mut builder = X509::builder().expect("x509 builder"); + builder.set_version(2).expect("version"); + builder.set_serial_number(&serial()).expect("serial"); + builder.set_subject_name(&name).expect("subject"); + builder.set_issuer_name(&name).expect("issuer"); + builder.set_pubkey(&key).expect("pubkey"); + builder + .set_not_before(&Asn1Time::days_from_now(0).expect("nb")) + .expect("set nb"); + builder + .set_not_after(&Asn1Time::days_from_now(3650).expect("na")) + .expect("set na"); + builder + .append_extension(BasicConstraints::new().critical().ca().build().expect("bc")) + .expect("ext bc"); + builder + .append_extension( + KeyUsage::new().critical().key_cert_sign().crl_sign().build().expect("ku"), + ) + .expect("ext ku"); + builder.sign(&key, MessageDigest::sha256()).expect("sign ca"); + (builder.build(), key) +} + +/// A CA-signed leaf cert with optional subject-alternative names (IP or DNS). +fn build_leaf(cn: &str, sans: &[&str], ca: &X509, ca_key: &PKey) -> (X509, PKey) { + let key = gen_key(); + let mut builder = X509::builder().expect("x509 builder"); + builder.set_version(2).expect("version"); + builder.set_serial_number(&serial()).expect("serial"); + builder.set_subject_name(&x509_name(cn)).expect("subject"); + builder.set_issuer_name(ca.subject_name()).expect("issuer"); + builder.set_pubkey(&key).expect("pubkey"); + builder + .set_not_before(&Asn1Time::days_from_now(0).expect("nb")) + .expect("set nb"); + builder + .set_not_after(&Asn1Time::days_from_now(3650).expect("na")) + .expect("set na"); + builder + .append_extension(BasicConstraints::new().build().expect("bc")) + .expect("ext bc"); + if !sans.is_empty() { + let mut san = SubjectAlternativeName::new(); + for entry in sans { + if entry.parse::().is_ok() { + san.ip(entry); + } else { + san.dns(entry); + } + } + let ext = san.build(&builder.x509v3_context(Some(ca), None)).expect("san"); + builder.append_extension(ext).expect("ext san"); + } + builder.sign(ca_key, MessageDigest::sha256()).expect("sign leaf"); + (builder.build(), key) +} + +/// A self-signed leaf not chained to the CA, for the untrusted-client test. +fn build_self_signed(cn: &str) -> (X509, PKey) { + let key = gen_key(); + let name = x509_name(cn); + let mut builder = X509::builder().expect("x509 builder"); + builder.set_version(2).expect("version"); + builder.set_serial_number(&serial()).expect("serial"); + builder.set_subject_name(&name).expect("subject"); + builder.set_issuer_name(&name).expect("issuer"); + builder.set_pubkey(&key).expect("pubkey"); + builder + .set_not_before(&Asn1Time::days_from_now(0).expect("nb")) + .expect("set nb"); + builder + .set_not_after(&Asn1Time::days_from_now(3650).expect("na")) + .expect("set na"); + builder + .append_extension(BasicConstraints::new().build().expect("bc")) + .expect("ext bc"); + builder.sign(&key, MessageDigest::sha256()).expect("sign self"); + (builder.build(), key) +} + +/// A CA-capable intermediate signed by the root, for the full-chain test. +fn build_intermediate(ca: &X509, ca_key: &PKey) -> (X509, PKey) { + let key = gen_key(); + let mut builder = X509::builder().expect("x509 builder"); + builder.set_version(2).expect("version"); + builder.set_serial_number(&serial()).expect("serial"); + builder + .set_subject_name(&x509_name("vLLM Test Intermediate CA")) + .expect("subject"); + builder.set_issuer_name(ca.subject_name()).expect("issuer"); + builder.set_pubkey(&key).expect("pubkey"); + builder + .set_not_before(&Asn1Time::days_from_now(0).expect("nb")) + .expect("set nb"); + builder + .set_not_after(&Asn1Time::days_from_now(3650).expect("na")) + .expect("set na"); + builder + .append_extension(BasicConstraints::new().critical().ca().build().expect("bc")) + .expect("ext bc"); + builder + .append_extension( + KeyUsage::new().critical().key_cert_sign().crl_sign().build().expect("ku"), + ) + .expect("ext ku"); + builder.sign(ca_key, MessageDigest::sha256()).expect("sign intermediate"); + (builder.build(), key) +} + +pub(crate) fn server_tls(certs: &TestCerts, cert_reqs: i32) -> TlsConfig { + TlsConfig { + cert_file: Some(certs.path("server.pem")), + key_file: Some(certs.path("server.key")), + ca_certs: (cert_reqs != 0).then(|| certs.path("ca.pem")), + cert_reqs, + ciphers: None, + } +} + +/// A plaintext-listener TLS config for `build_server_config` checks (`cert_reqs` +/// 0, no client auth), with the cert/key files chosen by the caller. +fn build_tls(certs: &TestCerts, cert: &str, key: Option<&str>) -> TlsConfig { + TlsConfig { + cert_file: Some(certs.path(cert)), + key_file: key.map(|k| certs.path(k)), + ca_certs: None, + cert_reqs: 0, + ciphers: None, + } +} + +/// Generous per-connection timeouts that never fire during the fast tests. +const TEST_TIMEOUTS: ConnectionTimeouts = ConnectionTimeouts { + header_read: Duration::from_secs(5), + keep_alive_enabled: true, +}; + +async fn spawn_server(tls_config: Option) -> (String, CancellationToken) { + spawn_server_with_timeouts(tls_config, TEST_TIMEOUTS).await +} + +/// Bind an ephemeral listener and serve a trivial router via the production +/// listener/connection path. The listener is bound (and thus accepting into the +/// backlog) before returning, so a client may connect immediately without a sleep. +async fn spawn_server_with_timeouts( + tls_config: Option, + timeouts: ConnectionTimeouts, +) -> (String, CancellationToken) { + let listener = Listener::bind(&HttpListenerMode::BindTcp { + host: "127.0.0.1".to_string(), + port: 0, + }) + .await + .expect("bind listener"); + let addr = listener.local_addr_display().expect("local addr"); + + let server_config = + tls_config.map(|cfg| tls::build_server_config(&cfg).expect("build server config")); + let app = Router::new().route("/health", get(|| async { "ok" })); + let shutdown = CancellationToken::new(); + let server_shutdown = shutdown.clone(); + tokio::spawn(async move { + let listener = match server_config { + Some(context) => MaybeTlsListener::tls(listener, context), + None => MaybeTlsListener::plain(listener), + }; + let _ = serve_connections(listener, app, server_shutdown.cancelled_owned(), timeouts).await; + }); + (addr, shutdown) +} + +/// Open a TLS connection trusting the test CA and finish the handshake, +/// optionally presenting a client identity (`.pem` + `.key`) for +/// mTLS. Hostname verification is disabled (the IP-SAN match is not under test); +/// chain verification stays on, so an untrusted server cert is still rejected. +async fn connect_tls( + certs: &TestCerts, + addr: &str, + identity: Option<&str>, +) -> std::io::Result>>> { + let tcp = TcpStream::connect(addr).await?; + + let mut builder = SslConnector::builder(SslMethod::tls_client()).expect("connector builder"); + builder.set_ca_file(certs.path("ca.pem")).expect("trust ca"); + if let Some(name) = identity { + builder + .set_certificate_chain_file(certs.path(&format!("{name}.pem"))) + .expect("client cert"); + builder + .set_private_key_file(certs.path(&format!("{name}.key")), SslFiletype::PEM) + .expect("client key"); + } + let connector = builder.build(); + let mut config = connector.configure().expect("configure"); + config.set_verify_hostname(false); + let ssl = config.into_ssl("127.0.0.1").expect("ssl"); + + let mut stream = Box::pin(SslStream::new(ssl, tcp).expect("client ssl stream")); + stream.as_mut().connect().await.map_err(std::io::Error::other)?; + Ok(stream) +} + +/// Issue an HTTPS GET (with `Connection: close`), optionally with an mTLS identity. +async fn https_get( + certs: &TestCerts, + addr: &str, + identity: Option<&str>, +) -> std::io::Result { + let mut stream = connect_tls(certs, addr, identity).await?; + stream + .write_all(b"GET /health HTTP/1.1\r\nHost: 127.0.0.1\r\nConnection: close\r\n\r\n") + .await?; + let mut response = String::new(); + stream.read_to_string(&mut response).await?; + Ok(response) +} + +/// Attempt a handshake offering only a legacy CBC+SHA1 suite over TLS 1.2, +/// capping the version so TLS 1.3 cannot rescue the negotiation. +async fn legacy_suite_handshake(certs: &TestCerts, addr: &str) -> std::io::Result<()> { + let tcp = TcpStream::connect(addr).await?; + + let mut builder = SslConnector::builder(SslMethod::tls_client()).expect("connector builder"); + builder.set_ca_file(certs.path("ca.pem")).expect("trust ca"); + builder.set_max_proto_version(Some(SslVersion::TLS1_2)).expect("cap tls1.2"); + builder + .set_cipher_list("ECDHE-ECDSA-AES256-SHA:@SECLEVEL=0") + .expect("legacy cipher"); + let connector = builder.build(); + let mut config = connector.configure().expect("configure"); + config.set_verify_hostname(false); + let ssl = config.into_ssl("127.0.0.1").expect("ssl"); + + let stream = SslStream::new(ssl, tcp).expect("client ssl stream"); + tokio::pin!(stream); + stream.as_mut().connect().await.map_err(std::io::Error::other) +} + +async fn plain_get(addr: &str) -> std::io::Result { + let mut tcp = TcpStream::connect(addr).await?; + tcp.write_all(b"GET /health HTTP/1.1\r\nHost: 127.0.0.1\r\nConnection: close\r\n\r\n") + .await?; + let mut response = String::new(); + tcp.read_to_string(&mut response).await?; + Ok(response) +} + +// ============================================================================ +// Tests +// ============================================================================ + +#[test] +fn builds_from_combined_pem() { + // Key omitted: it is read from the combined cert+key file. + let certs = TestCerts::generate(); + assert!(tls::build_server_config(&build_tls(&certs, "server_combined.pem", None)).is_ok()); +} + +#[test] +fn rejects_missing_cert_file() { + let certs = TestCerts::generate(); + assert!(tls::build_server_config(&build_tls(&certs, "does_not_exist.pem", None)).is_err()); +} + +#[test] +fn accepts_valid_cipher_list() { + let certs = TestCerts::generate(); + let mut cfg = build_tls(&certs, "server.pem", Some("server.key")); + cfg.ciphers = Some("ECDHE-ECDSA-AES256-GCM-SHA384".to_string()); + assert!(tls::build_server_config(&cfg).is_ok()); +} + +#[test] +fn rejects_invalid_cipher_list() { + let certs = TestCerts::generate(); + let mut cfg = build_tls(&certs, "server.pem", Some("server.key")); + cfg.ciphers = Some("THIS-IS-NOT-A-CIPHER".to_string()); + assert!(tls::build_server_config(&cfg).is_err()); +} + +#[test] +fn rejects_mismatched_cert_and_key() { + // check_private_key must reject a key that does not match the certificate. + let certs = TestCerts::generate(); + let tls = build_tls(&certs, "client.pem", Some("server.key")); + assert!(tls::build_server_config(&tls).is_err()); +} + +#[tokio::test] +async fn https_request_succeeds_over_tls() { + let certs = TestCerts::generate(); + let (addr, shutdown) = spawn_server(Some(server_tls(&certs, 0))).await; + let response = https_get(&certs, &addr, None).await.expect("https request"); + assert!(response.starts_with("HTTP/1.1 200"), "{response}"); + shutdown.cancel(); +} + +#[tokio::test] +async fn serves_full_certificate_chain() { + // Cert file holds leaf + intermediate; a client trusting only the root can + // verify only if the server sends the intermediate, guarding against a + // leaf-only load. + let certs = TestCerts::generate(); + let tls = TlsConfig { + cert_file: Some(certs.path("server_chain.pem")), + key_file: Some(certs.path("server_chain.key")), + ca_certs: None, + cert_reqs: 0, + ciphers: None, + }; + let (addr, shutdown) = spawn_server(Some(tls)).await; + let response = https_get(&certs, &addr, None).await.expect("chained https request"); + assert!(response.starts_with("HTTP/1.1 200"), "{response}"); + shutdown.cancel(); +} + +#[tokio::test] +async fn rejects_legacy_cipher_only_client() { + let certs = TestCerts::generate(); + let (addr, shutdown) = spawn_server(Some(server_tls(&certs, 0))).await; + let result = legacy_suite_handshake(&certs, &addr).await; + assert!(result.is_err(), "legacy-only client must be rejected"); + shutdown.cancel(); +} + +#[tokio::test] +async fn ssl_ciphers_override_widens_past_preset() { + // Counterpart to rejects_legacy_cipher_only_client: --ssl-ciphers set to that + // same legacy suite lets the client through, proving the override beats the preset. + let certs = TestCerts::generate(); + let mut tls = server_tls(&certs, 0); + tls.ciphers = Some("ECDHE-ECDSA-AES256-SHA:@SECLEVEL=0".to_string()); + let (addr, shutdown) = spawn_server(Some(tls)).await; + let result = legacy_suite_handshake(&certs, &addr).await; + assert!( + result.is_ok(), + "override must allow the legacy suite: {result:?}" + ); + shutdown.cancel(); +} + +#[tokio::test] +async fn mtls_required_rejects_client_without_certificate() { + let certs = TestCerts::generate(); + let (addr, shutdown) = spawn_server(Some(server_tls(&certs, 2))).await; + let result = https_get(&certs, &addr, None).await; + assert!( + result.is_err(), + "handshake must fail without a client certificate" + ); + shutdown.cancel(); +} + +#[tokio::test] +async fn mtls_required_accepts_valid_client_certificate() { + let certs = TestCerts::generate(); + let (addr, shutdown) = spawn_server(Some(server_tls(&certs, 2))).await; + let response = https_get(&certs, &addr, Some("client")).await.expect("mtls request"); + assert!(response.starts_with("HTTP/1.1 200"), "{response}"); + shutdown.cancel(); +} + +#[tokio::test] +async fn mtls_optional_allows_anonymous_and_authenticated() { + let certs = TestCerts::generate(); + let (addr, shutdown) = spawn_server(Some(server_tls(&certs, 1))).await; + let anonymous = https_get(&certs, &addr, None).await.expect("anonymous request"); + assert!(anonymous.starts_with("HTTP/1.1 200"), "{anonymous}"); + let authenticated = + https_get(&certs, &addr, Some("client")).await.expect("authenticated request"); + assert!(authenticated.starts_with("HTTP/1.1 200"), "{authenticated}"); + shutdown.cancel(); +} + +#[tokio::test] +async fn mtls_rejects_untrusted_client_certificate() { + // Optional (1) still verifies a presented cert, so a self-signed cert not + // chained to the CA is rejected in both modes, not just required (2). + let certs = TestCerts::generate(); + for cert_reqs in [1, 2] { + let (addr, shutdown) = spawn_server(Some(server_tls(&certs, cert_reqs))).await; + let result = https_get(&certs, &addr, Some("untrusted_client")).await; + assert!( + result.is_err(), + "cert_reqs={cert_reqs}: untrusted client cert must be rejected" + ); + shutdown.cancel(); + } +} + +#[tokio::test] +async fn plain_http_serves_when_tls_is_disabled() { + let (addr, shutdown) = spawn_server(None).await; + let response = plain_get(&addr).await.expect("http request"); + assert!(response.starts_with("HTTP/1.1 200"), "{response}"); + shutdown.cancel(); +} + +#[tokio::test(start_paused = true)] +async fn tls_handshake_timeout_drops_silent_client() { + // Silent client (no ClientHello) must be dropped at the handshake deadline. + let certs = TestCerts::generate(); + let (addr, shutdown) = spawn_server(Some(server_tls(&certs, 0))).await; + + let mut tcp = TcpStream::connect(&addr).await.expect("connect"); + tokio::task::yield_now().await; + tokio::time::advance(tls::TLS_HANDSHAKE_TIMEOUT + Duration::from_millis(1)).await; + tokio::task::yield_now().await; + + let mut buf = [0u8; 1]; + let read = tokio::time::timeout(Duration::from_secs(1), tcp.read(&mut buf)).await; + assert!( + matches!(read, Ok(Ok(0)) | Ok(Err(_))), + "server must drop a stalled TLS handshake (expected close, got {read:?})" + ); + shutdown.cancel(); +} + +#[tokio::test] +async fn keep_alive_timeout_closes_idle_connection() { + // Idle keep-alive connection must be closed at the deadline. + let timeouts = ConnectionTimeouts { + header_read: Duration::from_millis(150), + keep_alive_enabled: true, + }; + let (addr, shutdown) = spawn_server_with_timeouts(None, timeouts).await; + + let mut tcp = TcpStream::connect(&addr).await.expect("connect"); + // No `Connection: close`, so it stays alive until the idle deadline. + tcp.write_all(b"GET /health HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n") + .await + .expect("write request"); + + let drained = tokio::time::timeout(Duration::from_secs(5), async { + let mut buf = [0u8; 1024]; + loop { + match tcp.read(&mut buf).await { + Ok(0) => return Ok(()), + Ok(_) => continue, + Err(err) => return Err(err), + } + } + }) + .await; + assert!( + matches!(drained, Ok(Ok(()))), + "server must close an idle keep-alive connection (got {drained:?})" + ); + shutdown.cancel(); +} + +#[tokio::test] +async fn keep_alive_timeout_closes_idle_tls_connection() { + // The keep-alive idle bound lives in serve_connections, below TLS; assert it + // still fires through tls-listener's post-handshake SslStream, not just plaintext. + let certs = TestCerts::generate(); + let timeouts = ConnectionTimeouts { + header_read: Duration::from_millis(150), + keep_alive_enabled: true, + }; + let (addr, shutdown) = spawn_server_with_timeouts(Some(server_tls(&certs, 0)), timeouts).await; + + let mut stream = connect_tls(&certs, &addr, None).await.expect("handshake"); + // No `Connection: close`, so the connection stays alive until the idle deadline. + stream + .write_all(b"GET /health HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n") + .await + .expect("write request"); + + let closed = tokio::time::timeout(Duration::from_secs(5), async { + let mut buf = [0u8; 1024]; + loop { + // A clean close_notify (Ok(0)) or an abrupt TLS EOF both mean the + // server closed; only the outer timeout (still open) is a failure. + match stream.read(&mut buf).await { + Ok(0) | Err(_) => break, + Ok(_) => continue, + } + } + }) + .await; + assert!( + closed.is_ok(), + "server must close an idle keep-alive TLS connection at the deadline" + ); + shutdown.cancel(); +} + +#[tokio::test] +async fn idle_timeout_closes_silent_client() { + // Silent client closed by the header-read timeout (http1-only arms it from byte 0). + let timeouts = ConnectionTimeouts { + header_read: Duration::from_millis(150), + keep_alive_enabled: true, + }; + let (addr, shutdown) = spawn_server_with_timeouts(None, timeouts).await; + + let mut tcp = TcpStream::connect(&addr).await.expect("connect"); + let mut buf = [0u8; 1]; + let read = tokio::time::timeout(Duration::from_secs(5), tcp.read(&mut buf)).await; + assert!( + matches!(read, Ok(Ok(0)) | Ok(Err(_))), + "server must close a silent client (expected close, got {read:?})" + ); + shutdown.cancel(); +} + +#[tokio::test] +async fn keep_alive_zero_disables_keep_alive() { + // 0 disables keep-alive (serve, then close), like uvicorn's timeout_keep_alive=0. + let timeouts = ConnectionTimeouts { + header_read: Duration::from_secs(5), + keep_alive_enabled: false, + }; + let (addr, shutdown) = spawn_server_with_timeouts(None, timeouts).await; + + let mut tcp = TcpStream::connect(&addr).await.expect("connect"); + tcp.write_all(b"GET /health HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n") + .await + .expect("write request"); + + let mut response = String::new(); + let read = + tokio::time::timeout(Duration::from_secs(5), tcp.read_to_string(&mut response)).await; + assert!( + read.is_ok(), + "server must close after one response, not hang" + ); + assert!(response.starts_with("HTTP/1.1 200"), "{response}"); + // Assert `Connection: close`, not just 200: a 0 header-read timeout would also + // serve an immediate request, so 200 alone wouldn't prove keep-alive is off. + assert!( + response.to_ascii_lowercase().contains("connection: close"), + "keep-alive must be disabled (expected Connection: close): {response}" + ); + shutdown.cancel(); +} + +#[tokio::test] +async fn disabled_keep_alive_still_closes_silent_client() { + // Even with keep-alive off, the head read stays bounded, so a silent client + // is dropped rather than held open. + let timeouts = ConnectionTimeouts { + header_read: Duration::from_millis(150), + keep_alive_enabled: false, + }; + let (addr, shutdown) = spawn_server_with_timeouts(None, timeouts).await; + + let mut tcp = TcpStream::connect(&addr).await.expect("connect"); + let mut buf = [0u8; 1]; + let read = tokio::time::timeout(Duration::from_secs(5), tcp.read(&mut buf)).await; + assert!( + matches!(read, Ok(Ok(0)) | Ok(Err(_))), + "disabled keep-alive must still close a silent client (got {read:?})" + ); + shutdown.cancel(); +} diff --git a/asian_ai/performance/vLLM/rust/src/server/src/utils.rs b/asian_ai/performance/vLLM/rust/src/server/src/utils.rs new file mode 100644 index 0000000000000000000000000000000000000000..00b7b51a81857e20990941fc585487e6787b6c8b --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/server/src/utils.rs @@ -0,0 +1,106 @@ +use std::collections::HashMap; +use std::time::{SystemTime, UNIX_EPOCH}; + +use axum::http::HeaderMap; +use serde_json::Value; +use thiserror_ext::AsReport; +use uuid::Uuid; + +use crate::error::ApiError; + +#[derive(Debug, Clone, PartialEq, Eq, Default)] +pub struct ResolvedRequestContext { + pub request_id: String, + pub data_parallel_rank: Option, +} + +/// Return the current Unix timestamp in seconds for OpenAI response objects. +pub fn unix_timestamp() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|duration| duration.as_secs()) + .unwrap_or_default() +} + +/// Construct an API error for a failed utility call to the engine core. +pub fn utility_call_error(method: &str, error: impl AsReport) -> ApiError { + ApiError::server_error(format!("failed to call {method}: {}", error.as_report())) +} + +/// Merge `kv_transfer_params` into the `vllm_xargs` map, mirroring the Python +/// vLLM behavior where `kv_transfer_params` is injected into `extra_args` for +/// engine-core consumption. +pub fn merge_kv_transfer_params( + mut xargs: Option>, + kv_transfer_params: Option<&HashMap>, +) -> Option> { + if let Some(kv_params) = kv_transfer_params { + let map = xargs.get_or_insert_with(HashMap::new); + map.insert( + "kv_transfer_params".to_string(), + // This is safe because we know that `kv_params` is already valid JSON. + serde_json::to_value(kv_params).unwrap(), + ); + } + xargs +} + +/// Convert OpenAI-style `logit_bias` with string token-ID keys into the +/// internal `HashMap` representation, validating that every key +/// parses as a `u32`. +pub fn convert_logit_bias( + logit_bias: Option>, +) -> Result>, ApiError> { + logit_bias + .map(|bias| { + bias.into_iter() + .map(|(key, value)| { + key.parse().map(|k| (k, value)).map_err(|_| { + ApiError::invalid_request( + format!( + "Invalid key in 'logit_bias': '{key}' is not a valid token ID. \ + Token IDs must be non-negative integers." + ), + Some("logit_bias"), + ) + }) + }) + .collect() + }) + .transpose() +} + +/// Extract common request metadata from HTTP headers: the external request ID +/// and the optional data-parallel rank used for engine routing. +pub fn resolve_request_context( + headers: &HeaderMap, + request_id: Option<&str>, +) -> ResolvedRequestContext { + // `None` when the header is absent or cannot be parsed as a `u32`. + let data_parallel_rank = headers + .get("X-data-parallel-rank") + .and_then(|v| v.to_str().ok()) + .and_then(|s| s.trim().parse().ok()); + + // Extract request id from header. + let request_id_header = headers.get("X-Request-Id").and_then(|value| value.to_str().ok()); + let request_id = resolve_base_request_id(request_id_header, request_id); + + ResolvedRequestContext { + request_id, + data_parallel_rank, + } +} + +/// Resolve the base external request ID before API-specific prefixes such as +/// `chatcmpl-`. +pub fn resolve_base_request_id( + request_id_header: Option<&str>, + request_id: Option<&str>, +) -> String { + request_id_header.or(request_id).map(ToOwned::to_owned).unwrap_or_else(|| { + let mut id = Uuid::new_v4().simple().to_string(); + id.truncate(8); + id + }) +} diff --git a/asian_ai/performance/vLLM/rust/src/text/Cargo.toml b/asian_ai/performance/vLLM/rust/src/text/Cargo.toml new file mode 100644 index 0000000000000000000000000000000000000000..9f79b6eb59c26a3c66b29b85bbefa57dcf523ceb --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/text/Cargo.toml @@ -0,0 +1,37 @@ +[package] +name = "vllm-text" +version.workspace = true +edition.workspace = true +license.workspace = true + +[dependencies] +anyhow.workspace = true +asynk-strim-attr.workspace = true +easy-ext.workspace = true +enum-as-inner.workspace = true +futures.workspace = true +hf-hub.workspace = true +itertools.workspace = true +reqwest.workspace = true +serde.workspace = true +serde_json.workspace = true +serde_with.workspace = true +thiserror.workspace = true +thiserror-ext.workspace = true +tracing.workspace = true +trait-set.workspace = true +vllm-engine-core-client.workspace = true +vllm-llm.workspace = true +vllm-tokenizer.workspace = true + +[dev-dependencies] +expect-test.workspace = true +futures.workspace = true +serial_test.workspace = true +tempfile.workspace = true +tokio.workspace = true +vllm-llm = { workspace = true, features = ["test-util"] } +vllm-tokenizer = { workspace = true, features = ["test-utils"] } + +[lints] +workspace = true diff --git a/asian_ai/performance/vLLM/rust/src/text/src/backend/hf/config.rs b/asian_ai/performance/vLLM/rust/src/text/src/backend/hf/config.rs new file mode 100644 index 0000000000000000000000000000000000000000..03eed15d3ec292482a6391dd6a8e458e6e1f5c74 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/text/src/backend/hf/config.rs @@ -0,0 +1,369 @@ +use std::collections::BTreeSet; +use std::fs; +use std::path::Path; + +use serde::{Deserialize, Serialize}; +use thiserror_ext::AsReport as _; + +use crate::error::{Error, Result}; + +/// Minimal subset of `tokenizer_config.json` needed by chat/EOS handling. +#[derive(Debug, Default, Deserialize)] +#[serde(default)] +pub struct HfTokenizerConfig { + #[serde(flatten)] + pub special_tokens: HfSpecialTokens, + pub chat_template: Option, + /// The `tokenizer_class` field from HuggingFace tokenizer configs. Some + /// tiktoken-based models (e.g. DeepSeek, Kimi K2) set this to a value + /// containing "Tiktoken" which can be used as a hint for backend + /// selection. + pub tokenizer_class: Option, +} + +/// Hugging Face named special tokens may be serialized as a string or an +/// object carrying the token content. +#[derive(Debug, Clone, Deserialize)] +#[serde(untagged)] +pub enum NamedSpecialToken { + Text(String), + WithContent { content: String }, +} + +impl Serialize for NamedSpecialToken { + fn serialize(&self, serializer: S) -> std::result::Result + where + S: serde::Serializer, + { + serializer.serialize_str(self.as_str()) + } +} + +impl From for String { + fn from(value: NamedSpecialToken) -> Self { + match value { + NamedSpecialToken::Text(string) => string, + NamedSpecialToken::WithContent { content } => content, + } + } +} + +impl NamedSpecialToken { + pub fn as_str(&self) -> &str { + match self { + Self::Text(value) => value, + Self::WithContent { content } => content, + } + } +} + +/// Minimal set of special-token entries needed by chat/EOS handling. +#[serde_with::skip_serializing_none] +#[derive(Debug, Clone, Default, Deserialize, Serialize)] +#[serde(default)] +pub struct HfSpecialTokens { + pub bos_token: Option, + pub eos_token: Option, + pub unk_token: Option, + pub pad_token: Option, +} + +impl HfSpecialTokens { + /// Returns true if we don't discover any special tokens in the config. + pub fn is_empty(&self) -> bool { + self.bos_token.is_none() + && self.eos_token.is_none() + && self.unk_token.is_none() + && self.pad_token.is_none() + } +} + +/// Minimal subset of `config.json` (the model's main HF config). +/// +/// This intentionally supports only the two layouts we currently care about in +/// the Rust frontend: +/// - pure text models that keep text metadata at the top level +/// - composite models that expose a single nested `text_config` +/// +/// We do not support additional entry points such as `decoder`, `generator`, or +/// `text_encoder`. +#[derive(Debug, Default, Deserialize)] +#[serde(default)] +pub struct ModelConfig { + model_type: Option, + vocab_size: Option, + num_experts: Option, + moe_num_experts: Option, + n_routed_experts: Option, + num_local_experts: Option, + block_configs: Vec, + text_config: Option>, +} + +/// Minimal subset of `generation_config.json`. +#[derive(Debug, Default, Deserialize)] +#[serde(default)] +pub(super) struct GenerationConfig { + pub eos_token_id: Option, + pub temperature: Option, + pub top_p: Option, + pub top_k: Option, + pub min_p: Option, + pub repetition_penalty: Option, + pub max_new_tokens: Option, +} + +/// HF generation configs allow either one EOS id or a list of EOS ids. +#[derive(Debug, Clone, Deserialize)] +#[serde(untagged)] +pub(super) enum OneOrManyTokenIds { + One(u32), + Many(Vec), +} + +impl OneOrManyTokenIds { + pub(super) fn into_set(self) -> BTreeSet { + match self { + Self::One(id) => BTreeSet::from([id]), + Self::Many(ids) => ids.into_iter().collect(), + } + } +} + +/// Hugging Face configs may expose the expert count either as one integer or +/// as a list of repeated integers. +#[derive(Debug, Clone, Deserialize)] +#[serde(untagged)] +pub(super) enum OneOrManyExpertCount { + One(u32), + Many(Vec), +} + +impl OneOrManyExpertCount { + fn first_value(&self) -> u32 { + match self { + Self::One(value) => *value, + // Python currently takes the first value for list[int] expert + // counts in remote-code configs. + Self::Many(values) => values.first().copied().unwrap_or(0), + } + } +} + +/// Heterogeneous block-level MoE metadata used as a fallback when no top-level +/// expert-count field is available. +#[derive(Debug, Default, Deserialize)] +#[serde(default)] +pub(super) struct BlockConfig { + pub block_type: String, + pub n_routed_experts: u32, +} + +impl ModelConfig { + /// Return the config that the Rust frontend treats as the text/LLM config. + /// + /// This is deliberately narrower than Python/transformers: we only support + /// either the top-level config itself or a single nested `text_config`. + fn effective_text_config(&self) -> &Self { + self.text_config.as_deref().unwrap_or(self) + } + + /// Return the effective Hugging Face `model_type` used by the Rust + /// frontend. + /// + /// This follows the same simplified text-config selection as the rest of + /// this type: the top-level config wins, otherwise a single nested + /// `text_config` may provide the value. + pub fn model_type(&self) -> Option<&str> { + self.model_type.as_deref().or_else(|| self.text_config.as_deref()?.model_type()) + } + + /// Return the effective model vocabulary size, following the same + /// simplified text-config selection as `model_type`. + pub fn vocab_size(&self) -> Result { + if let Some(vocab_size) = self.vocab_size { + Ok(vocab_size) + } else if let Some(text_config) = self.text_config.as_deref() { + text_config.vocab_size() + } else { + Err(Error::Tokenizer( + "the model config does not define `vocab_size`".to_string(), + )) + } + } + + /// Match Python's current expert-count priority on the selected text + /// config. + /// + /// The only intentional simplification here is how we pick the text config: + /// Rust only looks at the top level or `text_config`, not the broader + /// transformers composite-config surface. + fn num_experts_from_block_configs(&self) -> u32 { + self.effective_text_config() + .block_configs + .iter() + .filter(|block| block.block_type == "moe") + .map(|block| block.n_routed_experts) + .max() + .unwrap_or(0) + } + + pub(super) fn num_experts(&self) -> u32 { + let config = self.effective_text_config(); + let direct = [ + config.num_experts.as_ref(), + config.moe_num_experts.as_ref(), + config.n_routed_experts.as_ref(), + config.num_local_experts.as_ref(), + ] + .into_iter() + .flatten() + .map(OneOrManyExpertCount::first_value) + .next() + .unwrap_or(0); + + if direct > 0 { + direct + } else { + self.num_experts_from_block_configs() + } + } + + pub(super) fn is_moe(&self) -> bool { + self.num_experts() > 0 + } +} + +/// Load the tokenizer-side EOS metadata if a config file is present. +pub fn load_tokenizer_config(path: Option<&Path>) -> Result { + read_json_file(path) +} + +/// Load the generation-side EOS metadata if a config file is present. +pub(super) fn load_generation_config(path: Option<&Path>) -> Result { + read_json_file(path) +} + +/// Load the model-side config (`config.json`) if present. +pub fn load_model_config(path: Option<&Path>) -> Result { + read_json_file(path) +} + +fn read_json_file(path: Option<&Path>) -> Result +where + T: for<'de> Deserialize<'de> + Default, +{ + let Some(path) = path else { + return Ok(T::default()); + }; + let content = fs::read_to_string(path).map_err(|error| { + Error::Tokenizer(format!( + "failed to read {}: {}", + path.display(), + error.as_report() + )) + })?; + serde_json::from_str(&content).map_err(|error| { + Error::Tokenizer(format!( + "failed to parse {}: {}", + path.display(), + error.as_report() + )) + }) +} + +#[cfg(test)] +mod tests { + use super::ModelConfig; + + #[test] + fn model_config_detects_moe_from_named_expert_fields() { + let field_names = [ + "num_experts", + "moe_num_experts", + "n_routed_experts", + "num_local_experts", + ]; + + for field_name in field_names { + let config: ModelConfig = + serde_json::from_str(&format!(r#"{{"{field_name}": 64}}"#)).unwrap(); + assert_eq!(config.num_experts(), 64, "field_name={field_name}"); + assert!(config.is_moe(), "field_name={field_name}"); + } + } + + #[test] + fn model_config_uses_first_value_for_list_expert_counts() { + let config: ModelConfig = serde_json::from_str(r#"{"num_experts":[16,16]}"#).unwrap(); + + assert_eq!(config.num_experts(), 16); + assert!(config.is_moe()); + } + + #[test] + fn model_config_falls_back_to_block_configs_maximum() { + let config: ModelConfig = serde_json::from_str( + r#"{ + "block_configs": [ + {"block_type":"attention","n_routed_experts":9}, + {"block_type":"moe","n_routed_experts":32}, + {"block_type":"moe","n_routed_experts":64} + ] + }"#, + ) + .unwrap(); + + assert_eq!(config.num_experts(), 64); + assert!(config.is_moe()); + } + + #[test] + fn model_config_prefers_nested_text_config_like_python_hf_text_config() { + let config: ModelConfig = serde_json::from_str( + r#"{ + "model_type": "top_level", + "num_experts": 64, + "text_config": { + "model_type": "nested", + "num_local_experts": 8 + } + }"#, + ) + .unwrap(); + + assert_eq!(config.num_experts(), 8); + assert_eq!(config.model_type(), Some("top_level")); + assert!(config.is_moe()); + } + + #[test] + fn model_config_uses_nested_vocab_size_when_top_level_is_absent() { + let config: ModelConfig = serde_json::from_str( + r#"{ + "text_config": { + "vocab_size": 151936 + } + }"#, + ) + .unwrap(); + + assert_eq!(config.vocab_size().unwrap(), 151936); + } + + #[test] + fn model_config_rejects_missing_vocab_size() { + let config: ModelConfig = serde_json::from_str(r#"{}"#).unwrap(); + + let error = config.vocab_size().unwrap_err(); + assert!(error.to_string().contains("does not define `vocab_size`")); + } + + #[test] + fn model_config_defaults_to_non_moe_when_no_expert_metadata_exists() { + let config: ModelConfig = serde_json::from_str(r#"{}"#).unwrap(); + + assert_eq!(config.num_experts(), 0); + assert!(!config.is_moe()); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/text/src/backend/hf/mod.rs b/asian_ai/performance/vLLM/rust/src/text/src/backend/hf/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..c6190f5c6dc89953b09b924ee1e3aaa48c2d2748 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/text/src/backend/hf/mod.rs @@ -0,0 +1,127 @@ +mod config; +mod model_files; + +use std::collections::BTreeSet; +use std::sync::Arc; + +use tracing::info; +use vllm_tokenizer::{DynTokenizer, HuggingFaceTokenizer, TekkenTokenizer, TiktokenTokenizer}; + +use self::config::{GenerationConfig, load_generation_config}; +pub use self::config::{ + HfSpecialTokens, HfTokenizerConfig, ModelConfig, NamedSpecialToken, load_model_config, + load_tokenizer_config, +}; +pub use self::model_files::{ResolvedModelFiles, TokenizerSource}; +use crate::backend::{SamplingHints, TextBackend}; +use crate::error::Result; + +fn load_tokenizer(tokenizer: &TokenizerSource) -> Result { + match tokenizer { + TokenizerSource::HuggingFace(path) => Ok(Arc::new(HuggingFaceTokenizer::new(path)?)), + TokenizerSource::Tiktoken(path) => Ok(Arc::new(TiktokenTokenizer::new(path)?)), + TokenizerSource::Tekken(path) => Ok(Arc::new(TekkenTokenizer::new(path)?)), + } +} + +/// [`TextBackend`] implementation built on Hugging Face model files. +pub struct HfTextBackend { + model_id: String, + files: ResolvedModelFiles, + tokenizer: DynTokenizer, + /// Primary EOS handled by engine-core's dedicated EOS path. + primary_eos_token_id: Option, + /// Additional EOS ids that should flow through stop-token handling. + extra_eos_token_ids: BTreeSet, + /// Generation-config for sampling defaults that may be inherited when the + /// user does not explicitly override them. + generation_config: GenerationConfig, + /// Model vocabulary size from the selected text config. + model_vocab_size: usize, + /// Model config (`config.json`). + model_config: ModelConfig, +} + +impl HfTextBackend { + /// Load the text backend with the given model id. + pub async fn from_model(model_id: &str) -> Result { + let files = ResolvedModelFiles::new(model_id).await?; + Self::from_resolved_model_files(files, model_id.to_string()) + } + + /// Load the text backend from resolved Hugging Face model files. + pub fn from_resolved_model_files(files: ResolvedModelFiles, model_id: String) -> Result { + let tokenizer_config = load_tokenizer_config(files.tokenizer_config_path.as_deref())?; + let tokenizer = load_tokenizer(&files.tokenizer)?; + let primary_eos_token_id = tokenizer_config + .special_tokens + .eos_token + .as_ref() + .and_then(|token| tokenizer.token_to_id(token.as_str())); + + let model_config = load_model_config(files.config_path.as_deref())?; + let model_vocab_size = model_config.vocab_size()? as usize; + let generation_config = load_generation_config(files.generation_config_path.as_deref())?; + let mut extra_eos_token_ids = generation_config + .eos_token_id + .clone() + .map(|value| value.into_set()) + .unwrap_or_default(); + if let Some(primary_eos_token_id) = primary_eos_token_id { + extra_eos_token_ids.remove(&primary_eos_token_id); + } + + info!( + model_id, + "loaded text backend with Hugging Face model files" + ); + + Ok(Self { + model_id, + files, + tokenizer, + primary_eos_token_id, + extra_eos_token_ids, + generation_config, + model_vocab_size, + model_config, + }) + } + + /// Expose the resolved model files for use by the chat backend to load the + /// chat template. + pub fn resolved_model_files(&self) -> &ResolvedModelFiles { + &self.files + } +} + +impl TextBackend for HfTextBackend { + fn tokenizer(&self) -> DynTokenizer { + self.tokenizer.clone() + } + + fn is_moe(&self) -> bool { + self.model_config.is_moe() + } + + fn model_vocab_size(&self) -> usize { + self.model_vocab_size + } + + fn model_id(&self) -> &str { + &self.model_id + } + + fn sampling_hints(&self) -> Result { + Ok(SamplingHints { + primary_eos_token_id: self.primary_eos_token_id, + extra_eos_token_ids: self.extra_eos_token_ids.clone(), + default_temperature: self.generation_config.temperature, + default_top_p: self.generation_config.top_p, + default_top_k: self.generation_config.top_k, + default_min_p: self.generation_config.min_p, + default_repetition_penalty: self.generation_config.repetition_penalty, + default_max_tokens: self.generation_config.max_new_tokens, + }) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/text/src/backend/hf/model_files.rs b/asian_ai/performance/vLLM/rust/src/text/src/backend/hf/model_files.rs new file mode 100644 index 0000000000000000000000000000000000000000..78448bbbc0daa328dcd44864f35ea6e08a91b824 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/text/src/backend/hf/model_files.rs @@ -0,0 +1,459 @@ +use std::path::{Path, PathBuf}; + +use hf_hub::Cache; +use hf_hub::api::tokio::{Api, ApiBuilder, ApiRepo}; +use thiserror_ext::AsReport as _; + +use super::config::{HfTokenizerConfig, load_tokenizer_config}; +use crate::error::{Error, Result}; + +const HF_TOKEN_ENV: &str = "HF_TOKEN"; + +/// The tokenizer source selected for a model. +#[derive(Debug, Clone)] +pub enum TokenizerSource { + /// Path to `tokenizer.json` in HuggingFace format. + HuggingFace(PathBuf), + /// Path to `tiktoken.model` or `*.tiktoken` file for tiktoken-based models. + Tiktoken(PathBuf), + /// Path to `tekken.json` when present (Mistral native tokenizer format). + /// + /// When set, the Tekken tokenizer should be preferred over the Hugging Face + /// tokenizer because the HuggingFace `tokenizer.json` for Mistral + /// models has a known regex bug that produces incorrect token IDs for + /// some inputs. + Tekken(PathBuf), +} + +impl TokenizerSource { + pub fn path(&self) -> &Path { + match self { + Self::HuggingFace(path) | Self::Tiktoken(path) | Self::Tekken(path) => path, + } + } +} + +/// Concrete tokenizer/config file locations resolved for one HF model id. +#[derive(Debug, Clone)] +pub struct ResolvedModelFiles { + /// The selected tokenizer source for this model. + pub tokenizer: TokenizerSource, + pub tokenizer_config_path: Option, + pub generation_config_path: Option, + pub preprocessor_config_path: Option, + pub chat_template_path: Option, + pub config_path: Option, +} + +impl ResolvedModelFiles { + /// Resolve tokenizer/config files from a local model directory first when + /// `model_id` points to one, otherwise consult the local HF cache and + /// finally the Hub. + pub async fn new(model_id: &str) -> Result { + if Path::new(model_id).is_dir() { + return resolve_local_model_files(Path::new(model_id)); + } + if let Some(files) = resolve_cached_model_files(model_id)? { + return Ok(files); + } + resolve_remote_model_files(model_id).await + } +} + +fn resolve_local_model_files(model_dir: &Path) -> Result { + let tokenizer_config_path = local_file_if_exists(model_dir, "tokenizer_config.json"); + let tokenizer_config = load_tokenizer_config(tokenizer_config_path.as_deref())?; + let tokenizer = resolve_local_tokenizer_source(model_dir, &tokenizer_config)?; + + Ok(ResolvedModelFiles { + tokenizer, + tokenizer_config_path, + generation_config_path: local_file_if_exists(model_dir, "generation_config.json"), + preprocessor_config_path: local_file_if_exists(model_dir, "preprocessor_config.json"), + chat_template_path: discover_chat_template_in_dir(model_dir), + config_path: local_file_if_exists(model_dir, "config.json"), + }) +} + +async fn resolve_remote_model_files(model_id: &str) -> Result { + let api = build_api().map_err(|error| Error::Tokenizer(error.to_report_string()))?; + let repo = api.model(model_id.to_string()); + let info = repo.info().await.map_err(|error| { + Error::Tokenizer(format!( + "failed to fetch model '{model_id}': {}", + error.as_report() + )) + })?; + + let siblings = info + .siblings + .iter() + .map(|sibling| sibling.rfilename.as_str()) + .collect::>(); + + let tokenizer_config_path = + download_if_present(&repo, model_id, &siblings, "tokenizer_config.json").await?; + let tokenizer_config = load_tokenizer_config(tokenizer_config_path.as_deref())?; + + let tokenizer = resolve_remote_tokenizer_source( + &repo, + model_id, + &siblings, + tokenizer_config.tokenizer_class.as_deref(), + ) + .await?; + + let generation_config_path = + download_if_present(&repo, model_id, &siblings, "generation_config.json").await?; + let preprocessor_config_path = + download_if_present(&repo, model_id, &siblings, "preprocessor_config.json").await?; + let chat_template_name = siblings + .contains("chat_template.json") + .then_some("chat_template.json") + .or_else(|| siblings.contains("chat_template.jinja").then_some("chat_template.jinja")) + .or_else(|| siblings.iter().copied().find(|name| name.ends_with(".jinja"))); + let chat_template_path = match chat_template_name { + Some(name) => Some(download_known_file(&repo, model_id, name).await?), + None => None, + }; + let config_path = download_if_present(&repo, model_id, &siblings, "config.json").await?; + + Ok(ResolvedModelFiles { + tokenizer, + tokenizer_config_path, + generation_config_path, + preprocessor_config_path, + chat_template_path, + config_path, + }) +} + +fn resolve_cached_model_files(model_id: &str) -> Result> { + let cache_repo = Cache::from_env().model(model_id.to_string()); + + let tokenizer_config_path = cache_repo.get("tokenizer_config.json"); + let tokenizer_config = load_tokenizer_config(tokenizer_config_path.as_deref())?; + let tokenizer = match resolve_cached_tokenizer_source(&cache_repo, &tokenizer_config)? { + Some(tokenizer) => tokenizer, + None => return Ok(None), + }; + + let model_dir = tokenizer.path().parent().ok_or_else(|| { + Error::Tokenizer("resolved tokenizer file has no parent directory".to_string()) + })?; + let generation_config_path = cache_repo.get("generation_config.json"); + let preprocessor_config_path = cache_repo.get("preprocessor_config.json"); + let chat_template_path = discover_chat_template_in_dir(model_dir); + let config_path = cache_repo.get("config.json"); + + Ok(Some(ResolvedModelFiles { + tokenizer, + tokenizer_config_path, + generation_config_path, + preprocessor_config_path, + chat_template_path, + config_path, + })) +} + +async fn resolve_remote_tokenizer_source( + repo: &ApiRepo, + model_id: &str, + siblings: &std::collections::BTreeSet<&str>, + tokenizer_class: Option<&str>, +) -> Result { + if let Some(tekken_path) = download_if_present(repo, model_id, siblings, "tekken.json").await? { + return Ok(TokenizerSource::Tekken(tekken_path)); + } + + let tokenizer_path = if siblings.contains("tokenizer.json") { + download_known_file(repo, model_id, "tokenizer.json").await? + } else if let Some(tiktoken_name) = find_tiktoken_sibling(siblings) { + download_known_file(repo, model_id, tiktoken_name).await? + } else { + return Err(Error::Tokenizer(format!( + "model '{model_id}' does not expose a supported tokenizer file \ + (tokenizer.json, tiktoken.model, or *.tiktoken) on Hugging Face" + ))); + }; + + Ok(resolve_tokenizer_source( + tokenizer_path, + tokenizer_class, + None, + )) +} + +fn resolve_cached_tokenizer_source( + cache_repo: &hf_hub::CacheRepo, + tokenizer_config: &HfTokenizerConfig, +) -> Result> { + let tekken_path = cache_repo.get("tekken.json"); + + if let Some(tekken_path) = tekken_path { + return Ok(Some(TokenizerSource::Tekken(tekken_path))); + } + + let Some(tokenizer_path) = cache_repo.get("tokenizer.json").or_else(|| { + // tiktoken.model is the most common name, try it first. + cache_repo.get("tiktoken.model").or_else(|| { + // Scan for any *.tiktoken file in the cache snapshot directory. + let snapshot_dir = cache_repo.get("config.json")?.parent()?.to_path_buf(); + discover_tiktoken_in_dir(&snapshot_dir) + }) + }) else { + return Ok(None); + }; + + Ok(Some(resolve_tokenizer_source( + tokenizer_path, + tokenizer_config.tokenizer_class.as_deref(), + None, + ))) +} + +fn resolve_local_tokenizer_source( + model_dir: &Path, + tokenizer_config: &HfTokenizerConfig, +) -> Result { + let tekken_path = local_file_if_exists(model_dir, "tekken.json"); + if let Some(tekken_path) = tekken_path { + return Ok(TokenizerSource::Tekken(tekken_path)); + } + + let tokenizer_path = local_file_if_exists(model_dir, "tokenizer.json") + .or_else(|| local_file_if_exists(model_dir, "tiktoken.model")) + .or_else(|| discover_tiktoken_in_dir(model_dir)) + .ok_or_else(|| { + Error::Tokenizer(format!( + "local model directory '{}' does not contain a supported tokenizer file \ + (tokenizer.json, tiktoken.model, or *.tiktoken)", + model_dir.display() + )) + })?; + + Ok(resolve_tokenizer_source( + tokenizer_path, + tokenizer_config.tokenizer_class.as_deref(), + None, + )) +} + +/// Choose the tokenizer. +/// +/// Selection order: +/// 1. `tekken.json` — Mistral native tokenizer (preferred over HF `tokenizer.json` because the HF +/// version has a known regex bug for Mistral models). +/// 2. File extension — `.tiktoken` / `tiktoken.model` files use tiktoken from BPE data. +/// 3. `tokenizer_class` in `tokenizer_config.json` — classes containing "Tiktoken" (case- +/// insensitive) trigger tiktoken loading from a sibling BPE file. +/// 4. Default — `tokenizer.json` in HuggingFace format. +fn resolve_tokenizer_source( + tokenizer_path: PathBuf, + tokenizer_class: Option<&str>, + tekken_path: Option, +) -> TokenizerSource { + if let Some(tekken_path) = tekken_path { + return TokenizerSource::Tekken(tekken_path); + } + + if is_tiktoken_file(&tokenizer_path) { + return TokenizerSource::Tiktoken(tokenizer_path); + } + + if tokenizer_class.is_some_and(|cls| cls.to_ascii_lowercase().contains("tiktoken")) + && let Some(dir) = tokenizer_path.parent() + && let Some(tiktoken_path) = discover_tiktoken_in_dir(dir) + { + return TokenizerSource::Tiktoken(tiktoken_path); + } + + TokenizerSource::HuggingFace(tokenizer_path) +} + +/// Download `filename` only if it exists in `siblings`. +async fn download_if_present( + repo: &ApiRepo, + model_id: &str, + siblings: &std::collections::BTreeSet<&str>, + filename: &str, +) -> Result> { + match siblings.contains(filename) { + true => download_known_file(repo, model_id, filename).await.map(Some), + false => Ok(None), + } +} + +async fn download_known_file(repo: &ApiRepo, model_id: &str, filename: &str) -> Result { + repo.get(filename).await.map_err(|error| { + Error::Tokenizer(format!( + "failed to download '{filename}' for model '{model_id}': {}", + error.as_report() + )) + }) +} + +fn build_api() -> anyhow::Result { + let mut builder = ApiBuilder::from_env().with_progress(true); + if let Ok(token) = std::env::var(HF_TOKEN_ENV) + && !token.is_empty() + { + builder = builder.with_token(Some(token)); + } + Ok(builder.build()?) +} + +fn local_file_if_exists(dir: &Path, filename: &str) -> Option { + let path = dir.join(filename); + path.is_file().then_some(path) +} + +/// Find a tiktoken file name among repo siblings, preferring `tiktoken.model`. +fn find_tiktoken_sibling<'a>(siblings: &std::collections::BTreeSet<&'a str>) -> Option<&'a str> { + if siblings.contains("tiktoken.model") { + return Some("tiktoken.model"); + } + siblings.iter().copied().find(|name| name.ends_with(".tiktoken")) +} + +/// Discover a tiktoken model file in a local directory. +pub(super) fn discover_tiktoken_in_dir(dir: &std::path::Path) -> Option { + let tiktoken_model = dir.join("tiktoken.model"); + if tiktoken_model.exists() { + return Some(tiktoken_model); + } + std::fs::read_dir(dir).ok()?.flatten().find_map(|entry| { + let path = entry.path(); + if path + .file_name() + .and_then(|n| n.to_str()) + .is_some_and(|n| n.ends_with(".tiktoken")) + { + Some(path) + } else { + None + } + }) +} + +/// Returns `true` if `path` points to a tiktoken-format file (by name). +pub(super) fn is_tiktoken_file(path: &std::path::Path) -> bool { + path.file_name() + .and_then(|n| n.to_str()) + .is_some_and(|name| name == "tiktoken.model" || name.ends_with(".tiktoken")) +} + +/// Chat templates are sometimes stored as dedicated .jinja files rather than as +/// a fixed-name config entry, so we scan the cached model dir. +fn discover_chat_template_in_dir(dir: &std::path::Path) -> Option { + let json_template_path = dir.join("chat_template.json"); + if json_template_path.exists() { + return Some(json_template_path); + } + + let jinja_path = dir.join("chat_template.jinja"); + if jinja_path.exists() { + return Some(jinja_path); + } + + std::fs::read_dir(dir).ok()?.flatten().map(|entry| entry.path()).find(|path| { + path.file_name() + .and_then(|name| name.to_str()) + .is_some_and(|name| name.ends_with(".jinja")) + }) +} + +#[cfg(test)] +mod tests { + use std::fs; + + use tempfile::tempdir; + use vllm_tokenizer::{TiktokenTokenizer, Tokenizer}; + + use super::{ResolvedModelFiles, TokenizerSource}; + + #[tokio::test] + async fn resolved_model_files_prefers_absolute_local_model_dir() { + let dir = tempdir().expect("create temp dir"); + fs::write(dir.path().join("tokenizer.json"), "{}").expect("write tokenizer"); + fs::write( + dir.path().join("tokenizer_config.json"), + r#"{"tokenizer_class":"PreTrainedTokenizerFast"}"#, + ) + .expect("write tokenizer config"); + fs::write(dir.path().join("config.json"), "{}").expect("write config"); + + let files = ResolvedModelFiles::new(dir.path().to_str().expect("utf8 path")) + .await + .expect("resolve local model files"); + + match files.tokenizer { + TokenizerSource::HuggingFace(path) => { + assert_eq!(path, dir.path().join("tokenizer.json")); + } + other => panic!("expected HuggingFace tokenizer, got {other:?}"), + } + assert_eq!(files.config_path, Some(dir.path().join("config.json"))); + assert_eq!( + files.tokenizer_config_path, + Some(dir.path().join("tokenizer_config.json")) + ); + } + + #[tokio::test] + #[ignore = "too slow for CI and requires network access to Hugging Face"] + async fn tiktoken_real_kimi_k25_tokenizer_files_load_and_handle_special_tokens() { + let files = ResolvedModelFiles::new("moonshotai/Kimi-K2.5") + .await + .expect("resolve real Kimi K2.5 model files"); + + let tokenizer_path = match &files.tokenizer { + TokenizerSource::Tiktoken(path) => path.clone(), + other => panic!("expected tiktoken tokenizer source, got {other:?}"), + }; + + for backend in [ + TiktokenTokenizer::new_riptoken(&tokenizer_path).expect("load riptoken backend"), + TiktokenTokenizer::new_tiktoken_rs(&tokenizer_path).expect("load tiktoken-rs backend"), + ] { + let think_id = backend.token_to_id("").expect("resolve "); + let end_think_id = backend.token_to_id("").expect("resolve "); + let tool_section_id = backend + .token_to_id("<|tool_calls_section_begin|>") + .expect("resolve tool call section marker"); + let contraction_heavy_text = + "I'm sure it's fine, but I can't say I'd trust that it's what we'd ship."; + let contraction_heavy_ids = backend.encode(contraction_heavy_text, false).unwrap(); + + assert_eq!( + (think_id, end_think_id, tool_section_id), + (163606, 163607, 163595) + ); + assert_eq!(backend.decode(&[think_id], true).unwrap(), ""); + assert_eq!(backend.decode(&[end_think_id], true).unwrap(), ""); + assert_eq!( + backend.decode(&[tool_section_id], true).unwrap(), + "<|tool_calls_section_begin|>" + ); + + // This demonstrates that we're using Kimi's custom BPE pattern. + // With CL100K this will be 23 tokens instead. + assert_eq!( + contraction_heavy_ids, + vec![ + 17172, 3287, 4643, 8201, 11, 996, 374, 8971, 3637, 20020, 8173, 473, 4643, + 1573, 56229, 13922, 13, + ] + ); + assert_eq!(contraction_heavy_ids.len(), 17); + assert_eq!( + backend.decode(&contraction_heavy_ids, false).unwrap(), + contraction_heavy_text + ); + + // Special-looking text that is not actually registered should fail gracefully. + assert_eq!(backend.token_to_id("◁think▷"), None); + assert_eq!(backend.token_to_id("<|definitely_not_registered|>"), None); + } + } +} diff --git a/asian_ai/performance/vLLM/rust/src/text/src/backend/mod.rs b/asian_ai/performance/vLLM/rust/src/text/src/backend/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..ccb305cade3dfd8ab61f0d7fa1b51001bc40f6a8 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/text/src/backend/mod.rs @@ -0,0 +1,90 @@ +pub mod hf; + +use std::sync::Arc; + +use vllm_tokenizer::DynTokenizer; + +use crate::error::Result; + +/// Tokenizer/model-derived defaults used to enrich text-generation requests +/// before they are lowered into engine-core. +#[derive(Debug, Clone, Default, PartialEq)] +pub struct SamplingHints { + pub primary_eos_token_id: Option, + pub extra_eos_token_ids: std::collections::BTreeSet, + pub default_temperature: Option, + pub default_top_p: Option, + pub default_top_k: Option, + pub default_min_p: Option, + pub default_repetition_penalty: Option, + pub default_max_tokens: Option, +} + +/// Effective bounds used to validate and lower sampling requests. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct SamplingLimits { + /// Runtime context window size reported by the engine startup handshake. + pub max_model_len: u32, + /// Maximum number of top log probabilities accepted by this frontend. + /// + /// `-1` means allowing requests up to the model vocabulary size. + pub max_logprobs: i32, + + /// Model vocabulary size from the model config, used to bound generated + /// token IDs and logits-domain sampling controls. + pub model_vocab_size: usize, + /// Tokenizer vocabulary size, used to bound `allowed_token_ids` and + /// token-ID prompts. + pub tokenizer_vocab_size: usize, +} + +impl SamplingLimits { + /// Original Python definition: + /// + pub const DEFAULT_MAX_LOGPROBS: i32 = 20; + /// Original Python definition: + /// + pub const MAX_LOGPROB_TOKEN_IDS: usize = 128; + + /// Return the union bound used to validate token-ID prompts. + pub fn prompt_token_vocab_size(&self) -> usize { + self.tokenizer_vocab_size.max(self.model_vocab_size) + } +} + +/// Minimal text-processing backend needed by `vllm-text`. +pub trait TextBackend: Send + Sync { + /// Return the tokenizer used by this backend. + fn tokenizer(&self) -> DynTokenizer; + + /// Return whether the loaded model is a mixture-of-experts model. + fn is_moe(&self) -> bool { + false + } + + /// Return the backend model ID. + fn model_id(&self) -> &str; + + /// Return tokenizer/model-derived hints used to enrich southbound sampling + /// parameters. + fn sampling_hints(&self) -> Result { + Ok(SamplingHints::default()) + } + + /// Return the model vocabulary size from the model config. + /// + /// The permissive default exists for lightweight test backends. Production + /// backends should override it with the resolved model config value. + fn model_vocab_size(&self) -> usize { + usize::MAX + } + + /// Return the full tokenizer vocabulary size (Python `len(tokenizer)`). + /// Used to range-check `allowed_token_ids` and token-id prompts. + fn tokenizer_vocab_size(&self) -> usize { + self.tokenizer().vocab_size() + } +} + +/// Shared trait-object form of [`TextBackend`]. +pub type DynTextBackend = Arc; diff --git a/asian_ai/performance/vLLM/rust/src/text/src/error.rs b/asian_ai/performance/vLLM/rust/src/text/src/error.rs new file mode 100644 index 0000000000000000000000000000000000000000..60e75024a04f1d14a9502c77753388a6fffa9a31 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/text/src/error.rs @@ -0,0 +1,65 @@ +use thiserror::Error; +use vllm_engine_core_client::Error as EngineCoreError; +use vllm_llm::Error as LlmError; + +pub use crate::lower::logprobs::LogprobsError; +pub use crate::lower::token_ids::TokenIdsError; + +#[derive(Debug, Error)] +pub enum Error { + #[error("tokenizer error: {0}")] + Tokenizer(String), + #[error("text request `{request_id}` must contain at least one prompt token ID")] + EmptyPromptTokenIds { request_id: String }, + #[error( + "this model's maximum context length is {max_model_len} tokens, \ + but the prompt contains {prompt_len} input tokens" + )] + PromptTooLong { max_model_len: u32, prompt_len: u32 }, + #[error(transparent)] + Logprobs(#[from] LogprobsError), + #[error(transparent)] + TokenIds(#[from] TokenIdsError), + #[error( + "`min_tokens` must be less than or equal to `max_tokens`, \ + got min_tokens={min_tokens}, max_tokens={max_tokens}" + )] + MinTokensExceedsMaxTokens { min_tokens: u32, max_tokens: u32 }, + #[error("`thinking_token_budget` must be a non-negative integer or -1 for unlimited.")] + InvalidThinkingTokenBudget, + #[error("invalid repetition detection params: {message}")] + InvalidRepetitionDetection { message: String }, + #[error("text request stream `{request_id}` closed before terminal output")] + StreamClosedBeforeTerminalOutput { request_id: String }, + #[error(transparent)] + Llm(#[from] LlmError), + #[error(transparent)] + EngineCore(#[from] EngineCoreError), +} + +pub type Result = std::result::Result; + +impl Error { + /// Whether this error represents invalid user request parameters. + pub fn is_request_validation_error(&self) -> bool { + match self { + Self::PromptTooLong { .. } + | Self::EmptyPromptTokenIds { .. } + | Self::Logprobs(_) + | Self::TokenIds(_) + | Self::MinTokensExceedsMaxTokens { .. } + | Self::InvalidThinkingTokenBudget + | Self::InvalidRepetitionDetection { .. } + // An empty tokenized prompt detected later, at request prepare + // time, surfaces through the transparent Llm wrapper. + | Self::Llm(LlmError::EmptyPromptTokenIds { .. }) => true, + _ => false, + } + } +} + +impl From for Error { + fn from(error: vllm_tokenizer::TokenizerError) -> Self { + Self::Tokenizer(error.0) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/text/src/lib.rs b/asian_ai/performance/vLLM/rust/src/text/src/lib.rs new file mode 100644 index 0000000000000000000000000000000000000000..a806849096943682fb31501d12c66713dc967416 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/text/src/lib.rs @@ -0,0 +1,177 @@ +//! Shared text-generation support used by chat and future raw completions. +//! +//! This crate intentionally stays below chat semantics: +//! prompt text handling, tokenizer/model loading, incremental detokenization, +//! and the thin generate-facing backend interface live here. + +use std::mem::take; + +pub use backend::{DynTextBackend, SamplingHints, SamplingLimits, TextBackend}; +pub use error::{Error, LogprobsError, Result, TokenIdsError}; +use futures::Stream; +pub use lower::{ + PreparedTextRequest, lower_sampling_params, lower_text_request, resolve_max_tokens, +}; +pub use output::{ + CollectedTextOutput, DecodedLogprobs, DecodedPositionLogprobs, DecodedPromptLogprobs, + DecodedTextEvent, DecodedTokenLogprob, Finished, TextDecodeOptions, TextOutputStreamExt, +}; +pub use request::{Prompt, SamplingParams, TextRequest}; +use trait_set::trait_set; +use vllm_engine_core_client::EngineCoreClient; +pub use vllm_llm::FinishReason; +use vllm_llm::{GenerateOutputStream, Llm}; +use vllm_tokenizer::DynTokenizer; + +pub mod backend; +mod error; +mod lower; +pub mod output; +mod request; +pub use vllm_tokenizer as tokenizer; + +trait_set! { + /// Shared streamed text output type used by raw completions and other text-only northbound paths. + pub trait TextOutputStream = Stream> + Send + 'static; +} + +/// Raw text facade above [`Llm`]. +/// +/// This layer stays below chat semantics: prompt text or prompt token IDs flow +/// in, decoded text deltas and terminal metadata flow out. +pub struct TextLlm { + /// Generate-only client owned by this text facade. + llm: Llm, + /// Tokenizer/model metadata backend responsible for prompt encode/decode + /// and sampling hints. + backend: DynTextBackend, + /// Runtime context window size reported by the engine startup handshake. + max_model_len: u32, + /// Maximum number of top log probabilities accepted by this text facade. + max_logprobs: i32, +} + +impl TextLlm { + /// Create a new text-generation facade from a shared LLM client plus a text + /// backend. + pub fn new(llm: Llm, backend: DynTextBackend) -> Self { + // The engine-reported value reflects the post-profiling, auto-fitted + // KV cache limit used at runtime. + let max_model_len = llm.engine_core_client().max_model_len(); + + Self { + llm, + backend, + max_model_len, + max_logprobs: SamplingLimits::DEFAULT_MAX_LOGPROBS, + } + } + + /// Override the maximum accepted logprobs count. + pub fn with_max_logprobs(mut self, max_logprobs: Option) -> Self { + if let Some(max_logprobs) = max_logprobs { + self.max_logprobs = max_logprobs; + } + self + } + + /// Return the backend model ID. + pub fn model_id(&self) -> &str { + self.backend.model_id() + } + + /// Expose the underlying engine-core client for low-level utility/admin + /// calls. + pub fn engine_core_client(&self) -> &EngineCoreClient { + self.llm.engine_core_client() + } + + /// Return the tokenizer used by this text backend. + pub fn tokenizer(&self) -> DynTokenizer { + self.backend.tokenizer() + } + + /// Tokenizer vocabulary size (the number of tokens the tokenizer knows), + /// used to bound `allowed_token_ids` like the Python frontend `len(tokenizer)`. + pub fn tokenizer_vocab_size(&self) -> usize { + self.backend.tokenizer_vocab_size() + } + + /// Model vocabulary size from the model config, used to bound generated + /// token IDs and logits-domain sampling controls. + pub fn model_vocab_size(&self) -> usize { + self.backend.model_vocab_size() + } + + /// Tokenize if needed, lower to a generate request, and return the raw + /// token stream. + pub async fn generate_raw(&self, request: TextRequest) -> Result { + let (_, raw_stream) = self.generate_inner(request).await?; + Ok(raw_stream) + } + + /// Tokenize if needed, lower to a generate request, and stream + /// incrementally decoded text. + pub async fn generate(&self, request: TextRequest) -> Result { + let (text_request, raw_stream) = self.generate_inner(request).await?; + let tokenizer = self.backend.tokenizer(); + let decoded_stream = output::decoded_text_event_stream( + text_request.request_id, + tokenizer, + raw_stream, + text_request.decode_options, + text_request.intermediate, + ); + + Ok(decoded_stream) + } + + async fn generate_inner( + &self, + mut request: TextRequest, + ) -> Result<(TextRequest, GenerateOutputStream)> { + request.validate()?; + + let tokenizer = self.backend.tokenizer(); + let prompt_token_ids = match take(&mut request.prompt) { + Prompt::Text(text) => tokenizer.encode(&text, request.add_special_tokens)?, + // Pre-tokenized prompts are the main completions-side escape hatch that lets benchmark + // and infra workloads bypass chat rendering and tokenizer overhead entirely. + Prompt::TokenIds(token_ids) => token_ids, + }; + + let sampling_hints = self.backend.sampling_hints()?; + let sampling_limits = SamplingLimits { + max_model_len: self.max_model_len, + max_logprobs: self.max_logprobs, + model_vocab_size: self.backend.model_vocab_size(), + tokenizer_vocab_size: self.backend.tokenizer_vocab_size(), + }; + + let PreparedTextRequest { + text_request, + generate_request, + } = lower_text_request( + request, + prompt_token_ids, + sampling_hints, + sampling_limits, + &*tokenizer, + )?; + + let raw_stream = self.llm.generate(generate_request).await?; + Ok((text_request, raw_stream)) + } + + /// Abort in-flight requests by their external (user-supplied) request ids. + pub async fn abort(&self, external_ids: &[String]) -> Result<()> { + self.llm.abort(external_ids).await?; + Ok(()) + } + + /// Shut down the underlying LLM client and its background tasks. + pub async fn shutdown(self) -> Result<()> { + self.llm.shutdown().await?; + Ok(()) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/text/src/lower.rs b/asian_ai/performance/vLLM/rust/src/text/src/lower.rs new file mode 100644 index 0000000000000000000000000000000000000000..19a2eb85068fa6189f9c8f1643a620e2d0d3c4bf --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/text/src/lower.rs @@ -0,0 +1,1160 @@ +use std::collections::BTreeSet; + +pub(crate) mod logprobs; +pub(crate) mod token_ids; + +use logprobs::validate_logprobs; +use token_ids::{validate_prompt_token_ids, validate_vocab_range}; +use vllm_engine_core_client::protocol::sampling::{ + EngineCoreSamplingParams, RepetitionDetectionParams, +}; +use vllm_llm::GenerateRequest; +use vllm_tokenizer::Tokenizer; + +use crate::backend::{SamplingHints, SamplingLimits}; +use crate::error::{Error, Result}; +use crate::request::{SamplingParams, TextRequest}; + +/// One text request after it has been lowered into the raw generate boundary. +#[derive(Debug)] +pub struct PreparedTextRequest { + /// The original high-level request, preserved for response-side metadata + /// and decoding options. + pub text_request: TextRequest, + /// The southbound request ready to be sent to `vllm-llm`. + pub generate_request: GenerateRequest, +} + +/// Convert a high-level [`TextRequest`] into one lower-level +/// [`GenerateRequest`] ready for the `llm` crate. +pub fn lower_text_request( + request: TextRequest, + prompt_token_ids: Vec, + sampling_hints: SamplingHints, + sampling_limits: SamplingLimits, + tokenizer: &dyn Tokenizer, +) -> Result { + let prompt_len = prompt_token_ids.len() as u32; + validate_prompt_token_ids(&prompt_token_ids, &sampling_limits)?; + + let generate_request = GenerateRequest { + request_id: request.request_id.clone(), + prompt_token_ids, + mm_features: request.mm_features.clone(), + sampling_params: lower_sampling_params( + request.sampling_params.clone(), + sampling_hints, + sampling_limits, + prompt_len, + tokenizer, + )?, + cache_salt: request.cache_salt.clone(), + priority: request.priority, + data_parallel_rank: request.data_parallel_rank, + reasoning_parser_kwargs: request.reasoning_parser_kwargs.clone(), + lora_request: request.lora_request.clone(), + arrival_time: None, + trace_headers: None, + }; + + Ok(PreparedTextRequest { + text_request: request, + generate_request, + }) +} + +/// Convert [`SamplingParams`] into [`EngineCoreSamplingParams`], enriching +/// omitted user values with tokenizer/model-derived hints when available. +pub fn lower_sampling_params( + sampling_params: SamplingParams, + SamplingHints { + primary_eos_token_id, + extra_eos_token_ids, + default_temperature, + default_top_p, + default_top_k, + default_min_p, + default_repetition_penalty, + default_max_tokens, + }: SamplingHints, + sampling_limits: SamplingLimits, + prompt_len: u32, + tokenizer: &dyn Tokenizer, +) -> Result { + let SamplingParams { + temperature, + top_p, + top_k, + seed, + max_tokens, + min_tokens, + thinking_token_budget, + logprobs, + prompt_logprobs, + min_p, + frequency_penalty, + presence_penalty, + repetition_penalty, + repetition_detection, + stop_token_ids, + ignore_eos, + logit_bias, + allowed_token_ids, + bad_words, + logprob_token_ids, + structured_outputs, + skip_reading_prefix_cache, + vllm_xargs, + } = sampling_params; + + validate_logprobs( + logprobs, + prompt_logprobs, + logprob_token_ids.as_deref(), + sampling_limits, + )?; + validate_repetition_detection(repetition_detection.as_ref())?; + + // Mirrors the model-generation-config inheritance used by vLLM's OpenAI chat + // path: https://github.com/vllm-project/vllm/blob/bc2c0c86efb28e77677a3cfb8687e976914a313a/vllm/entrypoints/openai/chat_completion/protocol.py#L424-L450 + // If neither the caller nor the model provides a value, fall back to 1.0 — the + // default used by the Python vLLM OpenAI-compatible API (via + // `_DEFAULT_SAMPLING_PARAMS`). + let temperature = temperature.or(default_temperature).unwrap_or(1.0); + let top_p = top_p.or(default_top_p).unwrap_or(1.0); + let top_k = top_k.or(default_top_k).unwrap_or(0); + let min_p = min_p.or(default_min_p).unwrap_or(0.0); + let repetition_penalty = repetition_penalty.or(default_repetition_penalty).unwrap_or(1.0); + let max_tokens = resolve_max_tokens( + max_tokens, + default_max_tokens, + sampling_limits.max_model_len, + prompt_len, + )?; + let min_tokens = min_tokens.unwrap_or(0); + if min_tokens > max_tokens { + return Err(Error::MinTokensExceedsMaxTokens { + min_tokens, + max_tokens, + }); + } + let thinking_token_budget = normalize_thinking_token_budget(thinking_token_budget)?; + let frequency_penalty = frequency_penalty.unwrap_or(0.0); + let presence_penalty = presence_penalty.unwrap_or(0.0); + + let mut stop_token_ids = stop_token_ids.unwrap_or_default(); + let mut all_stop_token_ids = BTreeSet::from_iter(stop_token_ids.iter().copied()); + if let Some(primary_eos_token_id) = primary_eos_token_id { + all_stop_token_ids.insert(primary_eos_token_id); + } + all_stop_token_ids.extend(extra_eos_token_ids.iter().copied()); + + if !ignore_eos { + merge_unique_token_ids(&mut stop_token_ids, extra_eos_token_ids.iter().copied()); + } + + let params = EngineCoreSamplingParams { + temperature, + top_p, + top_k, + seed, + max_tokens, + min_tokens, + thinking_token_budget, + logprobs, + prompt_logprobs, + min_p, + frequency_penalty, + presence_penalty, + repetition_penalty, + repetition_detection: repetition_detection.filter(|p| !p.is_disabled()), + stop_token_ids, + eos_token_id: (!ignore_eos).then_some(primary_eos_token_id).flatten(), + all_stop_token_ids, + logit_bias, + allowed_token_ids, + bad_words_token_ids: tokenize_bad_words(bad_words.as_deref(), tokenizer)?, + // TODO: Validate structured-output schemas and regexes before submitting requests to engine-core. + structured_outputs, + logprob_token_ids, + skip_reading_prefix_cache, + extra_args: vllm_xargs, + }; + validate_vocab_range(¶ms, &sampling_limits)?; + Ok(params) +} + +/// Normalize the user-facing `thinking_token_budget` into the engine value. +/// +/// Mirrors Python's `validate_thinking_token_budget` +/// (): +/// `None` and the `-1` "unlimited" sentinel both map to `None`; any other +/// negative value is rejected; non-negative values pass through unchanged. Like +/// Python's `int`, no upper bound is imposed. +fn normalize_thinking_token_budget(value: Option) -> Result> { + match value { + None | Some(-1) => Ok(None), + Some(budget) if budget >= 0 => Ok(Some(budget as u64)), + Some(_) => Err(Error::InvalidThinkingTokenBudget), + } +} + +fn validate_repetition_detection(params: Option<&RepetitionDetectionParams>) -> Result<()> { + let Some(params) = params else { + return Ok(()); + }; + + if params.min_pattern_size > params.max_pattern_size { + return Err(Error::InvalidRepetitionDetection { + message: format!( + "`min_pattern_size` must be less than or equal to \ + `max_pattern_size`, got min_pattern_size={}, \ + max_pattern_size={}", + params.min_pattern_size, params.max_pattern_size + ), + }); + } + if params.max_pattern_size > 0 && params.min_count < 2 { + return Err(Error::InvalidRepetitionDetection { + message: format!( + "`min_count` must be at least 2, got min_count={}", + params.min_count + ), + }); + } + + Ok(()) +} + +/// Convert bad-word strings into token-ID sequences, following the Python vLLM +/// logic in `SamplingParams.update_from_tokenizer()`. +/// +/// Each word is encoded both with and without a leading space so that the ban +/// applies regardless of whether the word appears at the beginning or in the +/// middle of generated text (this accounts for tokenizers that use an +/// `add_prefix_space` convention). +/// +/// Reference: +fn tokenize_bad_words( + bad_words: Option<&[String]>, + tokenizer: &dyn Tokenizer, +) -> Result>>> { + let bad_words = bad_words.filter(|w| !w.is_empty()); + let mut all_token_ids = Vec::new(); + + for bad_word in bad_words.into_iter().flatten() { + // Without a leading space we always keep the encoding. + // With a leading space we only keep it when the prefix-space variant produces a + // distinct first token but the same sequence length — this mirrors the Python + // dedup condition that avoids redundant entries. + let without_space = tokenizer.encode(bad_word, false)?; + let with_space = tokenizer.encode(&format!(" {}", bad_word.trim_start()), false)?; + + if !without_space.is_empty() { + all_token_ids.push(without_space); + } + if !with_space.is_empty() + && all_token_ids.last().is_some_and(|prev: &Vec| { + with_space[0] != prev[0] && with_space.len() == prev.len() + }) + { + all_token_ids.push(with_space); + } + } + + Ok((!all_token_ids.is_empty()).then_some(all_token_ids)) +} + +/// Resolve the effective `max_tokens` for generation, mirroring vLLM Python's +/// `get_max_tokens()` in `vllm/entrypoints/utils.py`. +/// +/// Takes the minimum of all available limits: user-specified, generation-config +/// default, and `max_model_len - prompt_len`. +pub fn resolve_max_tokens( + user_max_tokens: Option, + default_max_tokens: Option, + max_model_len: u32, + prompt_len: u32, +) -> Result { + let model_max_tokens = if prompt_len >= max_model_len { + return Err(Error::PromptTooLong { + max_model_len, + prompt_len, + }); + } else { + max_model_len - prompt_len + }; + + let request_max_tokens = user_max_tokens.or(default_max_tokens); + Ok(request_max_tokens.map_or(model_max_tokens, |n| n.min(model_max_tokens))) +} + +fn merge_unique_token_ids( + stop_token_ids: &mut Vec, + extra_token_ids: impl Iterator, +) { + // Keep user-provided ordering stable while still folding in backend-derived EOS + // aliases. + for token_id in extra_token_ids { + if !stop_token_ids.contains(&token_id) { + stop_token_ids.push(token_id); + } + } +} + +#[cfg(test)] +mod tests { + use std::collections::{BTreeSet, HashMap}; + + use serial_test::file_serial; + use vllm_tokenizer::test_utils::TestTokenizer; + + use super::*; + use crate::backend::hf::HfTextBackend; + use crate::backend::{SamplingHints, TextBackend as _}; + use crate::error::{LogprobsError, TokenIdsError}; + use crate::request::{Prompt, TextRequest}; + + fn stub_tokenizer() -> TestTokenizer { + TestTokenizer::new() + } + + fn sample_request() -> TextRequest { + TextRequest { + prompt: Prompt::TokenIds(vec![1, 2, 3]), + request_id: "text-1".to_string(), + ..TextRequest::for_test() + } + } + + fn sample_sampling_hints() -> SamplingHints { + SamplingHints { + primary_eos_token_id: Some(99), + extra_eos_token_ids: BTreeSet::from([77]), + default_temperature: None, + default_top_p: None, + default_top_k: None, + default_min_p: None, + default_repetition_penalty: None, + default_max_tokens: None, + } + } + + fn sample_sampling_limits() -> SamplingLimits { + SamplingLimits { + max_model_len: 1_000_000, + max_logprobs: SamplingLimits::DEFAULT_MAX_LOGPROBS, + model_vocab_size: 1000, + tokenizer_vocab_size: 2000, + } + } + + fn lower_sampling_params_with_limits( + sampling_params: SamplingParams, + sampling_limits: SamplingLimits, + ) -> Result { + lower_sampling_params( + sampling_params, + SamplingHints { + primary_eos_token_id: None, + extra_eos_token_ids: BTreeSet::new(), + default_temperature: None, + default_top_p: None, + default_top_k: None, + default_min_p: None, + default_repetition_penalty: None, + default_max_tokens: None, + }, + sampling_limits, + 3, + &stub_tokenizer(), + ) + } + + #[test] + fn lower_sampling_params_normalizes_thinking_token_budget() { + let lower = |budget: Option| { + lower_sampling_params_with_limits( + SamplingParams { + thinking_token_budget: budget, + ..SamplingParams::default() + }, + sample_sampling_limits(), + ) + }; + + // Non-negative budgets (including 0) pass through unchanged. + assert_eq!(lower(Some(256)).unwrap().thinking_token_budget, Some(256)); + assert_eq!(lower(Some(0)).unwrap().thinking_token_budget, Some(0)); + // `None` and the `-1` "unlimited" sentinel both disable the budget. + assert_eq!(lower(None).unwrap().thinking_token_budget, None); + assert_eq!(lower(Some(-1)).unwrap().thinking_token_budget, None); + // No upper bound is imposed, matching Python's `int`. + assert_eq!( + lower(Some(i64::from(u32::MAX) + 1)).unwrap().thinking_token_budget, + Some(u64::from(u32::MAX) + 1) + ); + // Other negatives are rejected. + assert!(matches!( + lower(Some(-2)), + Err(Error::InvalidThinkingTokenBudget) + )); + } + + #[test] + fn lower_sampling_params_rejects_min_tokens_above_resolved_max_tokens() { + let error = lower_sampling_params_with_limits( + SamplingParams { + max_tokens: Some(4), + min_tokens: Some(5), + ..SamplingParams::default() + }, + sample_sampling_limits(), + ) + .unwrap_err(); + + assert!(matches!( + error, + Error::MinTokensExceedsMaxTokens { + min_tokens: 5, + max_tokens: 4, + } + )); + } + + #[test] + fn lower_sampling_params_validates_repetition_detection() { + let lower = |repetition_detection| { + lower_sampling_params_with_limits( + SamplingParams { + repetition_detection, + ..SamplingParams::default() + }, + sample_sampling_limits(), + ) + }; + + let enabled = RepetitionDetectionParams { + max_pattern_size: 4, + min_pattern_size: 2, + min_count: 2, + }; + assert_eq!( + lower(Some(enabled.clone())).unwrap().repetition_detection, + Some(enabled) + ); + + let disabled = RepetitionDetectionParams { + max_pattern_size: 0, + min_pattern_size: 0, + min_count: 0, + }; + assert_eq!(lower(Some(disabled)).unwrap().repetition_detection, None); + + let error = lower(Some(RepetitionDetectionParams { + max_pattern_size: 1, + min_pattern_size: 2, + min_count: 2, + })) + .unwrap_err(); + let Error::InvalidRepetitionDetection { message } = error else { + panic!("expected repetition_detection validation error"); + }; + assert!(message.contains("min_pattern_size=2")); + assert!(message.contains("max_pattern_size=1")); + + let error = lower(Some(RepetitionDetectionParams { + max_pattern_size: 1, + min_pattern_size: 1, + min_count: 1, + })) + .unwrap_err(); + let Error::InvalidRepetitionDetection { message } = error else { + panic!("expected repetition_detection validation error"); + }; + assert!(message.contains("min_count=1")); + } + + #[test] + fn lower_text_request_applies_python_style_eos_hints() { + let prepared = lower_text_request( + sample_request(), + vec![1, 2, 3], + sample_sampling_hints(), + sample_sampling_limits(), + &stub_tokenizer(), + ) + .unwrap(); + + let params = prepared.generate_request.sampling_params; + expect_test::expect![[r#" + EngineCoreSamplingParams { + temperature: 1.0, + top_p: 1.0, + top_k: 0, + seed: None, + max_tokens: 999997, + min_tokens: 0, + thinking_token_budget: None, + logprobs: None, + prompt_logprobs: None, + min_p: 0.0, + frequency_penalty: 0.0, + presence_penalty: 0.0, + repetition_penalty: 1.0, + repetition_detection: None, + stop_token_ids: [ + 77, + ], + eos_token_id: Some( + 99, + ), + all_stop_token_ids: { + 77, + 99, + }, + logit_bias: None, + allowed_token_ids: None, + bad_words_token_ids: None, + structured_outputs: None, + logprob_token_ids: None, + skip_reading_prefix_cache: None, + extra_args: None, + } + "#]] + .assert_debug_eq(¶ms); + } + + #[test] + fn lower_text_request_respects_ignore_eos_for_stop_token_ids() { + let mut request = sample_request(); + request.sampling_params.ignore_eos = true; + + let prepared = lower_text_request( + request, + vec![1, 2, 3], + sample_sampling_hints(), + sample_sampling_limits(), + &stub_tokenizer(), + ) + .unwrap(); + + let params = prepared.generate_request.sampling_params; + expect_test::expect![[r#" + EngineCoreSamplingParams { + temperature: 1.0, + top_p: 1.0, + top_k: 0, + seed: None, + max_tokens: 999997, + min_tokens: 0, + thinking_token_budget: None, + logprobs: None, + prompt_logprobs: None, + min_p: 0.0, + frequency_penalty: 0.0, + presence_penalty: 0.0, + repetition_penalty: 1.0, + repetition_detection: None, + stop_token_ids: [], + eos_token_id: None, + all_stop_token_ids: { + 77, + 99, + }, + logit_bias: None, + allowed_token_ids: None, + bad_words_token_ids: None, + structured_outputs: None, + logprob_token_ids: None, + skip_reading_prefix_cache: None, + extra_args: None, + } + "#]] + .assert_debug_eq(¶ms); + } + + #[test] + fn lower_text_request_uses_union_vocab_for_prompt_token_ids() { + lower_text_request( + sample_request(), + vec![1500], + sample_sampling_hints(), + SamplingLimits { + model_vocab_size: 2000, + tokenizer_vocab_size: 1000, + ..sample_sampling_limits() + }, + &stub_tokenizer(), + ) + .expect("model vocab extends prompt token range"); + + lower_text_request( + sample_request(), + vec![1500], + sample_sampling_hints(), + SamplingLimits { + model_vocab_size: 1000, + tokenizer_vocab_size: 2000, + ..sample_sampling_limits() + }, + &stub_tokenizer(), + ) + .expect("tokenizer vocab extends prompt token range"); + + let error = lower_text_request( + sample_request(), + vec![2000], + sample_sampling_hints(), + SamplingLimits { + model_vocab_size: 1000, + tokenizer_vocab_size: 2000, + ..sample_sampling_limits() + }, + &stub_tokenizer(), + ) + .unwrap_err(); + + assert!(matches!( + error, + Error::TokenIds(TokenIdsError::OutOfVocab { + parameter: "prompt", + token_ids, + vocab_size: 2000, + }) if token_ids == vec![2000] + )); + } + + #[tokio::test] + #[file_serial(hf_qwen3)] + async fn lower_text_request_uses_real_qwen_generation_defaults() { + let backend = HfTextBackend::from_model("Qwen/Qwen3-0.6B") + .await + .expect("load qwen tokenizer and generation config"); + let hints = backend.sampling_hints().expect("collect sampling hints"); + + expect_test::expect![[r#" + SamplingHints { + primary_eos_token_id: Some( + 151645, + ), + extra_eos_token_ids: { + 151643, + }, + default_temperature: Some( + 0.6, + ), + default_top_p: Some( + 0.95, + ), + default_top_k: Some( + 20, + ), + default_min_p: None, + default_repetition_penalty: None, + default_max_tokens: None, + } + "#]] + .assert_debug_eq(&hints); + + let prepared = lower_text_request( + sample_request(), + vec![1, 2, 3], + hints, + SamplingLimits { + max_model_len: 40960, + max_logprobs: SamplingLimits::DEFAULT_MAX_LOGPROBS, + model_vocab_size: backend.model_vocab_size(), + tokenizer_vocab_size: backend.tokenizer_vocab_size(), + }, + &stub_tokenizer(), + ) + .expect("lower request"); + let params = prepared.generate_request.sampling_params; + + expect_test::expect![[r#" + EngineCoreSamplingParams { + temperature: 0.6, + top_p: 0.95, + top_k: 20, + seed: None, + max_tokens: 40957, + min_tokens: 0, + thinking_token_budget: None, + logprobs: None, + prompt_logprobs: None, + min_p: 0.0, + frequency_penalty: 0.0, + presence_penalty: 0.0, + repetition_penalty: 1.0, + repetition_detection: None, + stop_token_ids: [ + 151643, + ], + eos_token_id: Some( + 151645, + ), + all_stop_token_ids: { + 151643, + 151645, + }, + logit_bias: None, + allowed_token_ids: None, + bad_words_token_ids: None, + structured_outputs: None, + logprob_token_ids: None, + skip_reading_prefix_cache: None, + extra_args: None, + } + "#]] + .assert_debug_eq(¶ms); + } + + #[test] + fn lower_sampling_params_preserves_explicit_stop_token_ids_in_all_stop_set() { + let sampling_params = SamplingParams { + stop_token_ids: Some(vec![11, 77]), + ..SamplingParams::default() + }; + + let params = lower_sampling_params( + sampling_params, + SamplingHints { + primary_eos_token_id: Some(99), + extra_eos_token_ids: BTreeSet::from([77, 88]), + default_temperature: None, + default_top_p: None, + default_top_k: None, + default_min_p: None, + default_repetition_penalty: None, + default_max_tokens: None, + }, + sample_sampling_limits(), + 3, + &stub_tokenizer(), + ) + .unwrap(); + + expect_test::expect![[r#" + EngineCoreSamplingParams { + temperature: 1.0, + top_p: 1.0, + top_k: 0, + seed: None, + max_tokens: 999997, + min_tokens: 0, + thinking_token_budget: None, + logprobs: None, + prompt_logprobs: None, + min_p: 0.0, + frequency_penalty: 0.0, + presence_penalty: 0.0, + repetition_penalty: 1.0, + repetition_detection: None, + stop_token_ids: [ + 11, + 77, + 88, + ], + eos_token_id: Some( + 99, + ), + all_stop_token_ids: { + 11, + 77, + 88, + 99, + }, + logit_bias: None, + allowed_token_ids: None, + bad_words_token_ids: None, + structured_outputs: None, + logprob_token_ids: None, + skip_reading_prefix_cache: None, + extra_args: None, + } + "#]] + .assert_debug_eq(¶ms); + } + + #[test] + fn lower_sampling_params_prefers_user_values_over_generation_defaults() { + let sampling_params = SamplingParams { + temperature: Some(0.2), + top_p: Some(0.3), + top_k: Some(4), + max_tokens: Some(32), + min_tokens: Some(2), + ..Default::default() + }; + + let params = lower_sampling_params( + sampling_params, + SamplingHints { + primary_eos_token_id: None, + extra_eos_token_ids: BTreeSet::new(), + default_temperature: Some(0.8), + default_top_p: Some(0.9), + default_top_k: Some(12), + default_min_p: Some(0.1), + default_repetition_penalty: Some(1.2), + default_max_tokens: Some(128), + }, + sample_sampling_limits(), + 3, + &stub_tokenizer(), + ) + .unwrap(); + + expect_test::expect![[r#" + EngineCoreSamplingParams { + temperature: 0.2, + top_p: 0.3, + top_k: 4, + seed: None, + max_tokens: 32, + min_tokens: 2, + thinking_token_budget: None, + logprobs: None, + prompt_logprobs: None, + min_p: 0.1, + frequency_penalty: 0.0, + presence_penalty: 0.0, + repetition_penalty: 1.2, + repetition_detection: None, + stop_token_ids: [], + eos_token_id: None, + all_stop_token_ids: {}, + logit_bias: None, + allowed_token_ids: None, + bad_words_token_ids: None, + structured_outputs: None, + logprob_token_ids: None, + skip_reading_prefix_cache: None, + extra_args: None, + } + "#]] + .assert_debug_eq(¶ms); + } + + #[test] + fn lower_sampling_params_passes_logprobs_fields_through() { + let sampling_params = SamplingParams { + logprobs: Some(3), + prompt_logprobs: Some(-1), + ..Default::default() + }; + + let params = lower_sampling_params( + sampling_params, + SamplingHints { + primary_eos_token_id: None, + extra_eos_token_ids: BTreeSet::new(), + default_temperature: None, + default_top_p: None, + default_top_k: None, + default_min_p: None, + default_repetition_penalty: None, + default_max_tokens: None, + }, + SamplingLimits { + max_logprobs: -1, + ..sample_sampling_limits() + }, + 3, + &stub_tokenizer(), + ) + .unwrap(); + + assert_eq!(params.logprobs, Some(3)); + assert_eq!(params.prompt_logprobs, Some(-1)); + } + + #[test] + fn lower_sampling_params_rejects_full_vocab_logprobs_over_default_cap() { + let error = lower_sampling_params_with_limits( + SamplingParams { + logprobs: Some(-1), + ..Default::default() + }, + sample_sampling_limits(), + ) + .unwrap_err(); + + assert!(matches!( + error, + Error::Logprobs(LogprobsError::TooManyCount { + parameter: "logprobs", + requested: 1000, + max_allowed: 20, + }) + )); + } + + #[test] + fn lower_sampling_params_expands_full_vocab_logprobs_from_model_vocab() { + let params = lower_sampling_params_with_limits( + SamplingParams { + logprobs: Some(-1), + ..Default::default() + }, + SamplingLimits { + max_logprobs: 1500, + ..sample_sampling_limits() + }, + ) + .unwrap(); + + assert_eq!(params.logprobs, Some(-1)); + } + + #[test] + fn lower_sampling_params_rejects_invalid_logprob_token_ids() { + let error = lower_sampling_params_with_limits( + SamplingParams { + logprobs: Some(1), + logprob_token_ids: Some(vec![1000]), + ..Default::default() + }, + sample_sampling_limits(), + ) + .unwrap_err(); + + assert!(matches!( + error, + Error::TokenIds(TokenIdsError::OutOfVocab { + parameter: "logprob_token_ids", + token_ids, + vocab_size: 1000, + }) if token_ids == vec![1000] + )); + } + + #[test] + fn lower_sampling_params_rejects_out_of_vocab_stop_token_ids() { + let error = lower_sampling_params_with_limits( + SamplingParams { + stop_token_ids: Some(vec![999, 1000]), + ..Default::default() + }, + sample_sampling_limits(), + ) + .unwrap_err(); + + assert!(matches!( + error, + Error::TokenIds(TokenIdsError::OutOfVocab { + parameter: "stop_token_ids", + token_ids, + vocab_size: 1000, + }) if token_ids == vec![1000] + )); + } + + #[test] + fn lower_sampling_params_rejects_out_of_vocab_allowed_token_ids() { + let error = lower_sampling_params_with_limits( + SamplingParams { + allowed_token_ids: Some(vec![1999, 2000]), + ..Default::default() + }, + sample_sampling_limits(), + ) + .unwrap_err(); + + assert!(matches!( + error, + Error::TokenIds(TokenIdsError::OutOfVocab { + parameter: "allowed_token_ids", + token_ids, + vocab_size: 2000, + }) if token_ids == vec![2000] + )); + } + + #[test] + fn lower_sampling_params_rejects_empty_allowed_token_ids() { + let error = lower_sampling_params_with_limits( + SamplingParams { + allowed_token_ids: Some(vec![]), + ..Default::default() + }, + sample_sampling_limits(), + ) + .unwrap_err(); + + assert!(matches!( + error, + Error::TokenIds(TokenIdsError::EmptyAllowedTokenIds) + )); + } + + #[test] + fn lower_sampling_params_rejects_out_of_vocab_bad_words() { + let tokenizer = TestTokenizer::new().with_regular_token("blocked", 2000); + let error = lower_sampling_params( + SamplingParams { + bad_words: Some(vec!["blocked".to_string()]), + ..Default::default() + }, + SamplingHints::default(), + sample_sampling_limits(), + 3, + &tokenizer, + ) + .unwrap_err(); + + assert!(matches!( + error, + Error::TokenIds(TokenIdsError::OutOfVocab { + parameter: "bad_words", + token_ids, + vocab_size: 2000, + }) if token_ids == vec![2000] + )); + } + + #[test] + fn lower_sampling_params_rejects_out_of_vocab_logit_bias() { + let error = lower_sampling_params_with_limits( + SamplingParams { + logit_bias: Some(HashMap::from([(1000, 1.0)])), + ..Default::default() + }, + sample_sampling_limits(), + ) + .unwrap_err(); + + assert!(matches!( + error, + Error::TokenIds(TokenIdsError::OutOfVocab { + parameter: "logit_bias", + token_ids, + vocab_size: 1000, + }) if token_ids == vec![1000] + )); + } + + #[test] + fn lower_sampling_params_uses_generation_defaults_when_user_omits_values() { + let params = lower_sampling_params( + SamplingParams::default(), + SamplingHints { + primary_eos_token_id: None, + extra_eos_token_ids: BTreeSet::new(), + default_temperature: Some(0.8), + default_top_p: Some(0.9), + default_top_k: Some(12), + default_min_p: Some(0.1), + default_repetition_penalty: Some(1.2), + default_max_tokens: Some(128), + }, + sample_sampling_limits(), + 3, + &stub_tokenizer(), + ) + .unwrap(); + + expect_test::expect![[r#" + EngineCoreSamplingParams { + temperature: 0.8, + top_p: 0.9, + top_k: 12, + seed: None, + max_tokens: 128, + min_tokens: 0, + thinking_token_budget: None, + logprobs: None, + prompt_logprobs: None, + min_p: 0.1, + frequency_penalty: 0.0, + presence_penalty: 0.0, + repetition_penalty: 1.2, + repetition_detection: None, + stop_token_ids: [], + eos_token_id: None, + all_stop_token_ids: {}, + logit_bias: None, + allowed_token_ids: None, + bad_words_token_ids: None, + structured_outputs: None, + logprob_token_ids: None, + skip_reading_prefix_cache: None, + extra_args: None, + } + "#]] + .assert_debug_eq(¶ms); + } + + #[test] + fn resolve_max_tokens_caps_by_model_len() { + let result = resolve_max_tokens(Some(150), None, 200, 100); + assert_eq!(result.unwrap(), 100); + } + + #[test] + fn lower_text_request_preserves_non_streaming_request_metadata() { + let mut request = sample_request(); + request.intermediate = false; + + let prepared = lower_text_request( + request, + vec![1, 2, 3], + sample_sampling_hints(), + sample_sampling_limits(), + &stub_tokenizer(), + ) + .unwrap(); + + assert!(!prepared.text_request.intermediate); + assert_eq!(prepared.generate_request.request_id, "text-1"); + } + + #[test] + fn resolve_max_tokens_user_smaller_than_model_limit() { + let result = resolve_max_tokens(Some(50), None, 200, 100); + assert_eq!(result.unwrap(), 50); + } + + #[test] + fn resolve_max_tokens_uses_default_when_user_omits() { + let result = resolve_max_tokens(None, Some(64), 200, 100); + assert_eq!(result.unwrap(), 64); + } + + #[test] + fn resolve_max_tokens_default_capped_by_model_len() { + let result = resolve_max_tokens(None, Some(256), 200, 100); + assert_eq!(result.unwrap(), 100); + } + + #[test] + fn resolve_max_tokens_uses_model_limit_when_user_omits() { + let result = resolve_max_tokens(None, None, 200, 100); + assert_eq!(result.unwrap(), 100); + } + + #[test] + fn resolve_max_tokens_prompt_too_long() { + let result = resolve_max_tokens(Some(10), None, 100, 100); + assert!(matches!( + result, + Err(Error::PromptTooLong { + max_model_len: 100, + prompt_len: 100, + }) + )); + } + + #[test] + fn resolve_max_tokens_prompt_exceeds_model_len() { + let result = resolve_max_tokens(Some(10), None, 100, 200); + assert!(matches!( + result, + Err(Error::PromptTooLong { + max_model_len: 100, + prompt_len: 200, + }) + )); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/text/src/lower/logprobs.rs b/asian_ai/performance/vLLM/rust/src/text/src/lower/logprobs.rs new file mode 100644 index 0000000000000000000000000000000000000000..a2c7fbdce7201f6bf1f7c8083db60bcfff3ce60b --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/text/src/lower/logprobs.rs @@ -0,0 +1,113 @@ +//! Python-compatible validation for logprobs sampling params. +//! +//! `-1` is expanded only for bounds checks. The original request values are +//! passed through to engine-core. + +use thiserror::Error; + +use crate::backend::SamplingLimits; + +#[derive(Debug, Error)] +pub enum LogprobsError { + #[error("{parameter} must be non-negative or -1, got {value}")] + InvalidCount { parameter: &'static str, value: i32 }, + #[error( + "requested {parameter} of {requested}, which is greater than max allowed: {max_allowed}" + )] + TooManyCount { + parameter: &'static str, + requested: usize, + max_allowed: usize, + }, + #[error( + "requested logprob_token_ids of length {requested}, \ + which is greater than max allowed: {max_allowed}" + )] + TooManyTokenIds { + requested: usize, + max_allowed: usize, + }, + #[error( + "when both logprobs and logprob_token_ids are set, logprobs must equal \ + len(logprob_token_ids). Got logprobs={logprobs}, len(logprob_token_ids)={num_token_ids}." + )] + TokenIdsMismatch { logprobs: i32, num_token_ids: usize }, +} + +/// Validate logprobs count sampling parameters. +pub(super) fn validate_logprobs( + logprobs: Option, + prompt_logprobs: Option, + logprob_token_ids: Option<&[u32]>, + sampling_limits: SamplingLimits, +) -> Result<(), LogprobsError> { + let vocab_size = sampling_limits.model_vocab_size; + let max_logprobs = + normalize_logprobs_count(sampling_limits.max_logprobs, vocab_size, "max_logprobs")?; + + validate_logprobs_count(logprobs, max_logprobs, vocab_size, "logprobs")?; + validate_logprobs_count(prompt_logprobs, max_logprobs, vocab_size, "prompt_logprobs")?; + validate_logprob_token_ids(logprobs, logprob_token_ids) +} + +fn validate_logprobs_count( + requested: Option, + max_logprobs: usize, + vocab_size: usize, + parameter: &'static str, +) -> Result<(), LogprobsError> { + let Some(requested) = requested else { + return Ok(()); + }; + + let requested = normalize_logprobs_count(requested, vocab_size, parameter)?; + if requested > max_logprobs { + return Err(LogprobsError::TooManyCount { + parameter, + requested, + max_allowed: max_logprobs, + }); + } + + Ok(()) +} + +pub(super) fn validate_logprob_token_ids( + logprobs: Option, + logprob_token_ids: Option<&[u32]>, +) -> Result<(), LogprobsError> { + let Some(logprob_token_ids) = logprob_token_ids else { + return Ok(()); + }; + + let n = logprob_token_ids.len(); + if n > SamplingLimits::MAX_LOGPROB_TOKEN_IDS { + return Err(LogprobsError::TooManyTokenIds { + requested: n, + max_allowed: SamplingLimits::MAX_LOGPROB_TOKEN_IDS, + }); + } + + if let Some(logprobs) = logprobs + && logprobs != n as i32 + { + return Err(LogprobsError::TokenIdsMismatch { + logprobs, + num_token_ids: n, + }); + } + + Ok(()) +} + +fn normalize_logprobs_count( + value: i32, + vocab_size: usize, + parameter: &'static str, +) -> Result { + match value { + -1 => Ok(vocab_size), + value if value < 0 => Err(LogprobsError::InvalidCount { parameter, value }), + value => Ok(value as usize), + } +} diff --git a/asian_ai/performance/vLLM/rust/src/text/src/lower/token_ids.rs b/asian_ai/performance/vLLM/rust/src/text/src/lower/token_ids.rs new file mode 100644 index 0000000000000000000000000000000000000000..6cdd4d8256267c6f8ac9784f06ab25dc6329b677 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/text/src/lower/token_ids.rs @@ -0,0 +1,123 @@ +use std::result::Result; + +use thiserror::Error; +use vllm_engine_core_client::protocol::sampling::EngineCoreSamplingParams; + +use crate::SamplingLimits; + +#[derive(Debug, Error)] +pub enum TokenIdsError { + #[error("allowed_token_ids should not be empty")] + EmptyAllowedTokenIds, + #[error( + "token_id(s) {token_ids:?} in {parameter} contain out-of-vocab token ids. \ + Vocabulary size: {vocab_size}" + )] + OutOfVocab { + parameter: &'static str, + token_ids: Vec, + vocab_size: usize, + }, +} + +fn validate_param( + parameter: &'static str, + token_ids: impl IntoIterator, + vocab_size: usize, +) -> Result<(), TokenIdsError> { + let invalid_token_ids: Vec<_> = token_ids + .into_iter() + .filter(|&token_id| token_id as usize >= vocab_size) + .collect(); + if invalid_token_ids.is_empty() { + return Ok(()); + } + + Err(TokenIdsError::OutOfVocab { + parameter, + token_ids: invalid_token_ids, + vocab_size, + }) +} + +/// Validate that pre-tokenized prompt IDs are within the engine-visible prompt +/// vocabulary range. +pub(crate) fn validate_prompt_token_ids( + prompt_token_ids: &[u32], + limits: &SamplingLimits, +) -> Result<(), TokenIdsError> { + validate_param( + "prompt", + prompt_token_ids.iter().copied(), + limits.prompt_token_vocab_size(), + ) +} + +/// Validate that token IDs in text sampling parameters are within their +/// parameter-specific vocabulary ranges. +pub(crate) fn validate_vocab_range( + params: &EngineCoreSamplingParams, + limits: &SamplingLimits, +) -> Result<(), TokenIdsError> { + validate_param( + "stop_token_ids", + params.stop_token_ids.iter().copied(), + limits.model_vocab_size, + )?; + + if let Some(token_ids) = params.allowed_token_ids.as_deref() { + if token_ids.is_empty() { + return Err(TokenIdsError::EmptyAllowedTokenIds); + } + validate_param( + "allowed_token_ids", + token_ids.iter().copied(), + limits.tokenizer_vocab_size, + )?; + } + + if let Some(logit_bias) = params.logit_bias.as_ref() { + validate_param( + "logit_bias", + logit_bias.keys().copied(), + limits.model_vocab_size, + )?; + } + + if let Some(token_ids) = params.logprob_token_ids.as_deref() { + validate_param( + "logprob_token_ids", + token_ids.iter().copied(), + limits.model_vocab_size, + )?; + } + + if let Some(bad_words_token_ids) = params.bad_words_token_ids.as_deref() { + validate_param( + "bad_words", + bad_words_token_ids.iter().flatten().copied(), + limits.tokenizer_vocab_size, + )?; + } + + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn validate_vocab_range_rejects_out_of_vocab_ids() { + let error = validate_param("logprob_token_ids", [5_u32, 1000, 1001], 1000).unwrap_err(); + + assert!(matches!( + error, + TokenIdsError::OutOfVocab { + parameter: "logprob_token_ids", + token_ids, + vocab_size: 1000, + } if token_ids == vec![1000, 1001] + )); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/text/src/output/decoded.rs b/asian_ai/performance/vLLM/rust/src/text/src/output/decoded.rs new file mode 100644 index 0000000000000000000000000000000000000000..f008bec02d5f33a18b755c7e03fbec8e99e24aed --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/text/src/output/decoded.rs @@ -0,0 +1,592 @@ +use std::sync::Arc; + +use asynk_strim_attr::{TryYielder, try_stream}; +use futures::{Stream, StreamExt}; +use serde::{Deserialize, Serialize}; +use tracing::{Level, debug, trace}; +use vllm_engine_core_client::AbortCause; +use vllm_engine_core_client::protocol::output::StopReason; +use vllm_llm::{FinishReason, GenerateOutput, TokenUsage}; +use vllm_tokenizer::{DynTokenizer, IncrementalDecoder}; + +use super::logprobs::{ + DecodedLogprobs, DecodedPromptLogprobs, decode_logprobs, decode_prompt_logprobs, +}; +use crate::error::Error; + +/// Request-neutral options for incremental text decoding. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct TextDecodeOptions { + pub skip_special_tokens: bool, + pub include_stop_str_in_output: bool, + pub stop_strings: Option>, + /// Minimum number of tokens to generate before stop-string checking kicks + /// in. Stop strings found within the first `min_tokens` tokens are + /// ignored. + pub min_tokens: u32, +} + +impl Default for TextDecodeOptions { + fn default() -> Self { + Self { + skip_special_tokens: true, + include_stop_str_in_output: false, + stop_strings: None, + min_tokens: 0, + } + } +} + +/// Terminal metadata carried on the final [`DecodedTextEvent`]. +#[derive(Debug, Clone, PartialEq)] +pub struct Finished { + pub usage: TokenUsage, + pub finish_reason: FinishReason, + /// Connector-specific KV transfer parameters for disaggregated serving. + pub kv_transfer_params: Option, +} + +/// Internal decoded-text event emitted before higher-level assistant +/// adaptation. +#[derive(Debug, Clone, PartialEq)] +pub enum DecodedTextEvent { + /// The request has reached the point where prompt-scoped decoding metadata + /// is ready. + Start { + /// The actual prompt token IDs for this request. + prompt_token_ids: Arc<[u32]>, + /// Once-only prompt logprobs metadata, when requested. + /// + /// The first prompt token is carried separately because it has no left + /// context to score against; `scored_positions` covers the + /// remaining prompt positions. + prompt_logprobs: Option, + }, + /// A delta of text has been decoded, optionally alongside token-position + /// logprobs. + /// + /// `delta` is the newly visible decoded text fragment for this update. + /// + /// `logprobs` covers the newly generated token positions from the same + /// update, but is not guaranteed to align with `delta` by character + /// span. One update may carry token logprobs but no newly visible text + /// yet, and one visible text fragment may reflect multiple token + /// positions becoming decodable together. + /// + /// Upper-level may further parse `delta` as reasoning or tool calls. + /// + /// When `finished` is `Some`, this is the terminal event for the request. + TextDelta { + delta: String, + token_ids: Vec, + logprobs: Option, + finished: Option, + }, +} + +/// Convert the output token stream from the `vllm_llm` layer into incrementally +/// decoded text. +#[try_stream] +pub async fn decoded_text_event_stream( + request_id: String, + tokenizer: DynTokenizer, + mut raw_stream: impl Stream> + Unpin, + mut decode_options: TextDecodeOptions, + intermediate: bool, + mut y: TryYielder, +) -> crate::Result<()> { + let mut decoder: Option> = None; + let mut prompt_token_count = 0_usize; + let mut cached_token_count = 0_usize; + let mut token_ids = Vec::new(); + let mut output_token_count: usize = 0; + let mut logprobs: Option = None; + + while let Some(next) = raw_stream.next().await { + let output = next?; + cached_token_count = cached_token_count.max(output.cached_token_count); + + // If it's the first output, init states and yield `Start` event. + if decoder.is_none() { + let prompt_token_ids = + output.prompt_token_ids().expect("first llm output must carry prompt token ids"); + prompt_token_count = prompt_token_ids.len(); + + let dec = tokenizer.create_decode_stream( + prompt_token_ids, + decode_options.skip_special_tokens, + // If we are excluding stop strings from output, we need to buffer + // the output so that we don't return the beginning of a stop string + // when streaming the outputs. + match decode_options.include_stop_str_in_output { + true => 0, + false => { + decode_options + .stop_strings + .as_ref() + .and_then(|stops| stops.iter().map(|ss| ss.len()).max()) + .unwrap_or(1) + - 1 + } + }, + ); + decoder = Some(dec); + + y.yield_ok(DecodedTextEvent::Start { + prompt_token_ids: prompt_token_ids.clone(), + prompt_logprobs: output + .prompt_logprobs() + .map(|logprobs| { + decode_prompt_logprobs( + tokenizer.as_ref(), + prompt_token_ids, + logprobs, + decode_options.skip_special_tokens, + ) + }) + .transpose()?, + }) + .await; + }; + let decoder = decoder.as_mut().unwrap(); + + let kv_transfer_params = output.kv_transfer_params; + let mut finish_reason = output.finish_reason; + let mut stop_str_matched = false; + let suppress_terminal_stop_token = finish_reason.as_ref().is_some_and(|r| r.is_stop()) + && !decode_options.include_stop_str_in_output; + let decodable_token_ids = if suppress_terminal_stop_token { + // Match Python V1 token-stop detokenization by keeping the stop token + // in metadata while excluding it from user-visible text. + output.token_ids.split_last().map(|(_, rest)| rest).unwrap_or(&[]) + } else { + &output.token_ids + }; + + let mut delta: Option = None; + let mut truncate_output_to = None; + let mut truncate_tokens_to = None; + for (tok_idx, &token_id) in decodable_token_ids.iter().enumerate() { + let new_bytes = decoder.push_token(token_id)?; + if output_token_count + tok_idx + 1 > decode_options.min_tokens as usize + && let Some(stops) = decode_options.stop_strings.as_mut() + && let Some((idx, off)) = matches_stop_string(stops, decoder.output(), new_bytes) + { + let stop_str = stops.swap_remove(idx); + truncate_output_to = match decode_options.include_stop_str_in_output { + true => Some(off + stop_str.len()), + false => Some(off), + }; + finish_reason = Some(FinishReason::Stop(Some(StopReason::Text(stop_str)))); + truncate_tokens_to = Some(tok_idx + 1); + stop_str_matched = true; + + break; + } + + if intermediate && let Some(chunk) = decoder.next_chunk() { + if let Some(delta_str) = delta.as_mut() { + delta_str.push_str(&chunk); + } else { + delta = Some(chunk); + } + } + } + + let mut new_token_ids = output.token_ids; + let mut new_logprobs = output.logprobs; + + // Trim tokens and logprobs if we matched stop string. + if let Some(num_tokens) = truncate_tokens_to { + new_token_ids.truncate(num_tokens); + if let Some(logprobs) = &mut new_logprobs { + logprobs.positions.truncate(num_tokens); + } + } + + output_token_count += new_token_ids.len(); + + let decoded_logprobs = new_logprobs + .as_ref() + .map(|logprobs| { + decode_logprobs( + tokenizer.as_ref(), + logprobs, + decode_options.skip_special_tokens, + ) + }) + .transpose()?; + + if !intermediate { + token_ids.extend(&new_token_ids); + if let Some(dlp) = decoded_logprobs.as_ref() { + logprobs + .get_or_insert_with(|| DecodedLogprobs { positions: vec![] }) + .positions + .extend_from_slice(&dlp.positions); + } + } + + if let Some(reason) = finish_reason { + // Flush any remaining buffered text. + let (last_chunk, mut text) = decoder.flush(truncate_output_to)?; + let text_len = text.len(); + let full_text = tracing::enabled!(Level::TRACE).then(|| text.clone()); + + if intermediate { + if let Some(chunk) = last_chunk { + if let Some(delta_str) = delta.as_mut() { + delta_str.push_str(&chunk); + } else { + delta = Some(chunk); + } + } + token_ids = new_token_ids; + logprobs = decoded_logprobs; + text = delta.unwrap_or_default(); + } + + debug!( + finish_reason = ?reason, + text_length_bytes = text_len, + output_token_count = output_token_count, + "request finished with terminal output" + ); + if let Some(full_text) = full_text { + trace!(full_text, "request finished with terminal decoded text"); + } + + // Intentionally drop the stream with explicit cause, so that the engine core + // can distinguish between such normal completion vs an unexpected + // early drop. + if stop_str_matched { + AbortCause::StopStringMatched.drop_as(raw_stream); + } + + y.yield_ok(DecodedTextEvent::TextDelta { + delta: text, + token_ids, + logprobs, + finished: Some(Finished { + usage: TokenUsage { + prompt_token_count, + output_token_count, + cached_token_count, + }, + finish_reason: reason, + kv_transfer_params, + }), + }) + .await; + return Ok(()); + } + + if intermediate { + y.yield_ok(DecodedTextEvent::TextDelta { + delta: delta.unwrap_or_default(), + token_ids: new_token_ids, + logprobs: decoded_logprobs, + finished: None, + }) + .await; + } + } + + Err(Error::StreamClosedBeforeTerminalOutput { request_id }) +} + +/// If stop string matches, returns tuple +/// (index into stop string vec, byte index of first byte of stop string in +/// output) +fn matches_stop_string(stops: &[String], output: &str, new_bytes: usize) -> Option<(usize, usize)> { + // We compare byte subslices to avoid utf8 boundary problem + let output = output.as_bytes(); + let next_off = (output.len() + 1) - new_bytes; + stops + .iter() + .map(|ss| (ss.as_bytes(), ss.len(), next_off.saturating_sub(ss.len()))) + .enumerate() + .find_map(|(ss_idx, (ss, len, start_off))| { + output[start_off..] + .windows(len) + .position(|w| w == ss) + .map(|pos| (ss_idx, start_off + pos)) + }) +} + +#[cfg(test)] +mod tests { + use std::pin::Pin; + use std::sync::{Arc, Mutex}; + use std::task::{Context, Poll}; + + use futures::{Stream, stream}; + use vllm_engine_core_client::AbortCause; + use vllm_llm::GenerateOutput; + use vllm_tokenizer::test_utils::TestTokenizer; + + use super::*; + use crate::output::TextOutputStreamExt as _; + + /// Helper: run `decoded_text_event_stream` to completion and return the + /// collected output. + async fn run_to_completion( + token_ids: Vec, + decode_options: TextDecodeOptions, + ) -> crate::output::CollectedTextOutput { + let prompt: Arc<[u32]> = Arc::from([]); + let raw_stream = stream::iter(vec![Ok(GenerateOutput::for_test( + Some(prompt), + token_ids, + Some(FinishReason::Length), + ))]); + let tokenizer: DynTokenizer = Arc::new(TestTokenizer::new()); + decoded_text_event_stream("test".into(), tokenizer, raw_stream, decode_options, false) + .collect_output() + .await + .unwrap() + } + + /// Convert ASCII string to token IDs (one byte per token). + fn ascii_tokens(s: &str) -> Vec { + s.bytes().map(u32::from).collect() + } + + fn opts(stop: &[&str], min_tokens: u32) -> TextDecodeOptions { + TextDecodeOptions { + stop_strings: Some(stop.iter().map(|s| s.to_string()).collect()), + min_tokens, + ..Default::default() + } + } + + struct DropRecordingStream { + next: Option>, + dropped_cause: Arc>>, + } + + impl Stream for DropRecordingStream { + type Item = vllm_llm::Result; + + fn poll_next(mut self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll> { + Poll::Ready(self.next.take()) + } + } + + impl Drop for DropRecordingStream { + fn drop(&mut self) { + *self.dropped_cause.lock().unwrap() = Some(AbortCause::current()); + } + } + + // --- stop string stream tests --- + + #[tokio::test] + async fn stream_stop_string_sets_task_local_abort_cause_on_raw_stream_drop() { + let prompt: Arc<[u32]> = Arc::from([]); + let dropped_cause = Arc::new(Mutex::new(None)); + let raw_stream = DropRecordingStream { + next: Some(Ok(GenerateOutput::for_test( + Some(prompt), + ascii_tokens("hello"), + Some(FinishReason::Length), + ))), + dropped_cause: Arc::clone(&dropped_cause), + }; + let tokenizer: DynTokenizer = Arc::new(TestTokenizer::new()); + + let output = decoded_text_event_stream( + "test".into(), + tokenizer, + raw_stream, + opts(&["ll"], 0), + false, + ) + .collect_output() + .await + .unwrap(); + + assert_eq!(output.text, "he"); + assert!(output.finish_reason.is_stop()); + assert_eq!( + *dropped_cause.lock().unwrap(), + Some(AbortCause::StopStringMatched) + ); + } + + #[tokio::test] + async fn stream_stop_string_truncates_at_match() { + let output = run_to_completion(ascii_tokens("hello"), opts(&["e"], 0)).await; + assert_eq!(output.text, "h"); + assert!(output.finish_reason.is_stop()); + } + + #[tokio::test] + async fn stream_stop_string_at_end() { + let output = run_to_completion(ascii_tokens("abcxyz"), opts(&["xyz"], 0)).await; + assert_eq!(output.text, "abc"); + assert!(output.finish_reason.is_stop()); + } + + #[tokio::test] + async fn stream_stop_string_first_token() { + let output = run_to_completion(ascii_tokens("xhello"), opts(&["x"], 0)).await; + assert_eq!(output.text, ""); + assert!(output.finish_reason.is_stop()); + } + + #[tokio::test] + async fn stream_stop_string_no_match_runs_to_completion() { + let output = run_to_completion(ascii_tokens("hello"), opts(&["z"], 0)).await; + assert_eq!(output.text, "hello"); + assert_eq!(output.finish_reason, FinishReason::Length); + } + + #[tokio::test] + async fn stream_stop_string_multi_char() { + let output = run_to_completion(ascii_tokens("say hello world"), opts(&["lo"], 0)).await; + assert_eq!(output.text, "say hel"); + assert!(output.finish_reason.is_stop()); + } + + #[tokio::test] + async fn stream_stop_string_first_of_multiple_wins() { + // Both "ll" and "lo" are present; "ll" appears first in the output. + let output = run_to_completion(ascii_tokens("hello"), opts(&["ll", "lo"], 0)).await; + assert_eq!(output.text, "he"); + assert!(output.finish_reason.is_stop()); + } + + #[tokio::test] + async fn stream_stop_string_include_in_output() { + let output = run_to_completion( + ascii_tokens("hello"), + TextDecodeOptions { + stop_strings: Some(vec!["ll".to_string()]), + include_stop_str_in_output: true, + ..Default::default() + }, + ) + .await; + assert_eq!(output.text, "hell"); + assert!(output.finish_reason.is_stop()); + } + + // --- min_tokens + stop string interaction --- + + #[tokio::test] + async fn min_tokens_suppresses_early_stop_string() { + // stop="e", min_tokens=3: the 'e' at token 2 is within the first 3 tokens, + // so it should be skipped. No later 'e' exists, so output runs to completion. + let output = run_to_completion(ascii_tokens("hello"), opts(&["e"], 3)).await; + assert_eq!(output.text, "hello"); + assert_eq!(output.finish_reason, FinishReason::Length); + } + + #[tokio::test] + async fn min_tokens_allows_stop_string_after_threshold() { + // stop="e", min_tokens=2: the first 'e' at token 3 is past the threshold. + let output = run_to_completion(ascii_tokens("greet"), opts(&["e"], 2)).await; + assert_eq!(output.text, "gr"); + assert!(output.finish_reason.is_stop()); + } + + #[tokio::test] + async fn min_tokens_zero_behaves_like_absent() { + let output = run_to_completion(ascii_tokens("hello"), opts(&["e"], 0)).await; + assert_eq!(output.text, "h"); + assert!(output.finish_reason.is_stop()); + } + + #[test] + fn stop_string_matches_at_end() { + let stops = vec!["wor".to_string()]; + // Output: "say wor", last byte 'r' was just added (new_bytes=1) + let result = matches_stop_string(&stops, "say wor", 1); + assert_eq!(result, Some((0, 4))); + } + + #[test] + fn stop_string_no_match() { + let stops = vec!["xyz".to_string()]; + let result = matches_stop_string(&stops, "say wor", 1); + assert_eq!(result, None); + } + + #[test] + fn stop_string_matches_first_of_multiple() { + let stops = vec!["wor".to_string(), "say".to_string()]; + // "say" appears earlier but "wor" is checked first (index 0) + let result = matches_stop_string(&stops, "say wor", 1); + assert_eq!(result, Some((0, 4))); + } + + #[test] + fn stop_string_matches_second_of_multiple() { + let stops = vec!["xyz".to_string(), "wor".to_string()]; + let result = matches_stop_string(&stops, "say wor", 1); + assert_eq!(result, Some((1, 4))); + } + + #[test] + fn stop_string_matches_with_multiple_new_bytes() { + let stops = vec!["wor".to_string()]; + // "say wor" where last 3 bytes "wor" were added at once + let result = matches_stop_string(&stops, "say wor", 3); + assert_eq!(result, Some((0, 4))); + } + + #[test] + fn stop_string_matches_leftmost_with_multiple_new_bytes() { + let stops = vec!["\n".to_string()]; + let result = matches_stop_string(&stops, "Answer\n\n", 2); + assert_eq!(result, Some((0, 6))); + } + + #[test] + fn stop_string_matches_at_beginning() { + let stops = vec!["say".to_string()]; + let result = matches_stop_string(&stops, "say wor", 7); + assert_eq!(result, Some((0, 0))); + } + + #[test] + fn stop_string_exact_output() { + let stops = vec!["abc".to_string()]; + let result = matches_stop_string(&stops, "abc", 3); + assert_eq!(result, Some((0, 0))); + } + + #[test] + fn stop_string_single_char() { + let stops = vec!["!".to_string()]; + let result = matches_stop_string(&stops, "hello!", 1); + assert_eq!(result, Some((0, 5))); + } + + #[test] + fn stop_string_not_in_new_bytes_region() { + let stops = vec!["say".to_string()]; + // "say" is in the output but before the new byte region. + // new_bytes=1 means only 'r' was added; "say" ended at byte 3, + // but the search window starts at next_off - stop_len = 7+1-1 - 3 = 4. + let result = matches_stop_string(&stops, "say wor", 1); + assert_eq!(result, None); + } + + #[test] + fn stop_string_empty_list() { + let stops: Vec = vec![]; + let result = matches_stop_string(&stops, "hello", 1); + assert_eq!(result, None); + } + + #[test] + fn stop_string_multibyte_utf8() { + let stops = vec!["世界".to_string()]; + // "你好世界" is 12 bytes: 你(3) + 好(3) + 世(3) + 界(3) + // "世界" starts at byte 6 + let result = matches_stop_string(&stops, "你好世界", 3); + assert_eq!(result, Some((0, 6))); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/text/src/output/logprobs.rs b/asian_ai/performance/vLLM/rust/src/text/src/output/logprobs.rs new file mode 100644 index 0000000000000000000000000000000000000000..366e0dea902748a477ae2b01a88185083b9bf899 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/text/src/output/logprobs.rs @@ -0,0 +1,209 @@ +use itertools::Itertools as _; +use serde::{Deserialize, Serialize}; +use vllm_llm::{Logprobs, PositionLogprobs}; +use vllm_tokenizer::Tokenizer; + +use crate::error::Error; + +/// One decoded token candidate and its logprob metadata. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct DecodedTokenLogprob { + /// Original vocabulary token ID for this candidate. + pub token_id: u32, + /// Best-effort decoded token string for this candidate. + pub token: String, + /// Log probability of this token candidate. + pub logprob: f32, + /// Vocabulary rank of this token candidate. + pub rank: u32, +} + +/// One position's decoded token candidates and their logprobs. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct DecodedPositionLogprobs { + /// Candidate tokens for this position. + pub entries: Vec, +} + +/// Decoded sample logprobs for generated token positions. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct DecodedLogprobs { + /// Generated token positions covered by this payload. + pub positions: Vec, +} + +/// Decoded prompt logprobs for prompt token positions. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct DecodedPromptLogprobs { + /// Original vocabulary token ID for the first prompt token. + pub first_token_id: u32, + /// Best-effort decoded string for the first prompt token. + /// + /// The first prompt token has no left context to score against, so it is + /// stored separately instead of appearing in `scored_positions`. + pub first_token: String, + /// Scored prompt positions after the first prompt token. + /// + /// `scored_positions[i]` corresponds to the prompt token at position `i + + /// 1`. + pub scored_positions: Vec, +} + +/// Decode generated-token logprobs from the raw `llm` token-ID shape into the +/// text-layer decoded-token representation. +/// +/// Each returned position corresponds to one generated token position from the +/// same `llm` update. +pub(super) fn decode_logprobs( + tokenizer: &T, + logprobs: &Logprobs, + skip_special_tokens: bool, +) -> Result { + Ok(DecodedLogprobs { + positions: logprobs + .positions + .iter() + .map(|position| decode_position_logprobs(tokenizer, position, skip_special_tokens)) + .try_collect()?, + }) +} + +/// Decode prompt logprobs from the raw `llm` token-ID shape into the text-layer +/// decoded-token representation. +/// +/// The returned payload stores the first prompt token separately and decodes +/// the remaining scored prompt positions into `scored_positions`, matching +/// vLLM's prompt-logprobs semantics. +pub(super) fn decode_prompt_logprobs( + tokenizer: &T, + prompt_token_ids: &[u32], + logprobs: &Logprobs, + skip_special_tokens: bool, +) -> Result { + let first_token_id = prompt_token_ids + .first() + .copied() + .expect("prompt logprobs require at least one prompt token"); + let first_token = tokenizer.decode(&[first_token_id], skip_special_tokens)?; + let scored_positions = logprobs + .positions + .iter() + .map(|position| decode_position_logprobs(tokenizer, position, skip_special_tokens)) + .try_collect()?; + + Ok(DecodedPromptLogprobs { + first_token_id, + first_token, + scored_positions, + }) +} + +/// Decode one token position's raw candidate set into decoded token strings +/// plus logprob metadata. +/// +/// This decodes every candidate token ID independently through the active text +/// backend. +fn decode_position_logprobs( + tokenizer: &T, + position: &PositionLogprobs, + skip_special_tokens: bool, +) -> Result { + Ok(DecodedPositionLogprobs { + entries: position + .entries + .iter() + .map(|entry| { + tokenizer.decode(&[entry.token_id], skip_special_tokens).map(|token| { + DecodedTokenLogprob { + token_id: entry.token_id, + token, + logprob: entry.logprob, + rank: entry.rank, + } + }) + }) + .try_collect()?, + }) +} + +#[cfg(test)] +mod tests { + use vllm_llm::{Logprobs, PositionLogprobs, TokenLogprob}; + use vllm_tokenizer::test_utils::TestTokenizer; + + use super::*; + + #[test] + fn decode_logprobs_decodes_every_candidate_token() { + let tokenizer = TestTokenizer::new(); + let logprobs = Logprobs { + positions: vec![PositionLogprobs { + entries: vec![ + TokenLogprob { + token_id: b'a' as u32, + logprob: -0.1, + rank: 3, + }, + TokenLogprob { + token_id: b'b' as u32, + logprob: -0.2, + rank: 1, + }, + ], + }], + }; + + assert_eq!( + decode_logprobs(&tokenizer, &logprobs, false).unwrap(), + DecodedLogprobs { + positions: vec![DecodedPositionLogprobs { + entries: vec![ + DecodedTokenLogprob { + token_id: b'a' as u32, + token: "a".to_string(), + logprob: -0.1, + rank: 3, + }, + DecodedTokenLogprob { + token_id: b'b' as u32, + token: "b".to_string(), + logprob: -0.2, + rank: 1, + }, + ], + }], + } + ); + } + + #[test] + fn decode_prompt_logprobs_separates_first_prompt_token() { + let tokenizer = TestTokenizer::new(); + let logprobs = Logprobs { + positions: vec![PositionLogprobs { + entries: vec![TokenLogprob { + token_id: b'x' as u32, + logprob: -0.4, + rank: 1, + }], + }], + }; + + assert_eq!( + decode_prompt_logprobs(&tokenizer, &[b'p' as u32, b'x' as u32], &logprobs, false) + .unwrap(), + DecodedPromptLogprobs { + first_token_id: b'p' as u32, + first_token: "p".to_string(), + scored_positions: vec![DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: b'x' as u32, + token: "x".to_string(), + logprob: -0.4, + rank: 1, + }], + }], + } + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/text/src/output/mod.rs b/asian_ai/performance/vLLM/rust/src/text/src/output/mod.rs new file mode 100644 index 0000000000000000000000000000000000000000..5af2e091edeea1acf64da9c48dd595d95eee5fdc --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/text/src/output/mod.rs @@ -0,0 +1,332 @@ +//! Output processing helpers shared by text and chat layers. + +pub use decoded::{DecodedTextEvent, Finished, TextDecodeOptions, decoded_text_event_stream}; +pub use logprobs::{ + DecodedLogprobs, DecodedPositionLogprobs, DecodedPromptLogprobs, DecodedTokenLogprob, +}; + +mod decoded; +mod logprobs; + +use std::sync::Arc; + +use futures::{StreamExt as _, pin_mut}; + +use crate::{Error, FinishReason, Result, TextOutputStream}; + +/// Final decoded text plus terminal stream metadata. +#[derive(Debug, Clone, PartialEq)] +pub struct CollectedTextOutput { + pub text: String, + pub prompt_token_ids: Arc<[u32]>, + pub prompt_logprobs: Option, + pub logprobs: Option, + pub token_ids: Vec, + pub finish_reason: FinishReason, + pub usage: vllm_llm::TokenUsage, + /// Connector-specific KV transfer parameters for disaggregated serving. + pub kv_transfer_params: Option, +} + +#[allow(clippy::manual_async_fn, reason = "specify `Send` bound")] +#[easy_ext::ext(TextOutputStreamExt)] +impl T { + /// Collect the stream to completion and return the final decoded text plus + /// terminal metadata. + pub fn collect_output(self) -> impl Future> + Send { + async move { + let stream = self; + pin_mut!(stream); + let mut prompt_logprobs = None; + let mut prompt_token_ids: Arc<[u32]> = Arc::from([]); + let mut collected: Option = None; + + while let Some(event) = stream.next().await.transpose()? { + match event { + DecodedTextEvent::Start { + prompt_logprobs: start_prompt_logprobs, + prompt_token_ids: start_prompt_token_ids, + .. + } => { + prompt_logprobs = start_prompt_logprobs; + prompt_token_ids = start_prompt_token_ids; + } + DecodedTextEvent::TextDelta { + delta, + token_ids: delta_token_ids, + logprobs: mut delta_logprobs, + finished, + } => { + if let Some(c) = collected.as_mut() { + c.text.push_str(&delta); + c.token_ids.extend(delta_token_ids); + if let Some(dlp) = delta_logprobs.as_mut() { + if let Some(lp) = c.logprobs.as_mut() { + lp.positions.extend_from_slice(&dlp.positions); + } else { + c.logprobs = delta_logprobs; + } + } + } else { + collected = Some(CollectedTextOutput { + text: delta, + prompt_token_ids: Arc::clone(&prompt_token_ids), + prompt_logprobs: prompt_logprobs.take(), + logprobs: delta_logprobs, + token_ids: delta_token_ids, + finish_reason: FinishReason::Error, + usage: vllm_llm::TokenUsage::default(), + kv_transfer_params: None, + }) + }; + + if let Some(finished) = finished { + let mut collected = collected.unwrap(); + collected.finish_reason = finished.finish_reason; + collected.usage = finished.usage; + collected.kv_transfer_params = finished.kv_transfer_params; + return Ok(collected); + } + } + } + } + + // Note: this is actually unreachable, as the underlying stream always emit an + // error on unexpected close. + Err(Error::StreamClosedBeforeTerminalOutput { + request_id: "unknown".to_string(), + }) + } + } +} + +#[cfg(test)] +mod tests { + use futures::stream; + use vllm_llm::FinishReason; + + use super::*; + + #[tokio::test] + async fn collect_output_retains_prompt_and_sample_logprobs() { + let stream = stream::iter(vec![ + Ok(DecodedTextEvent::Start { + prompt_token_ids: vec![10, 11].into(), + prompt_logprobs: Some(DecodedPromptLogprobs { + first_token_id: 0, + first_token: "o".to_string(), + scored_positions: vec![DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "p".to_string(), + logprob: -0.1, + rank: 1, + }], + }], + }), + }), + Ok(DecodedTextEvent::TextDelta { + delta: "bc".to_string(), + token_ids: vec![1, 2], + logprobs: Some(DecodedLogprobs { + positions: vec![ + DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "a".to_string(), + logprob: -0.2, + rank: 1, + }], + }, + DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "bc".to_string(), + logprob: -0.3, + rank: 1, + }], + }, + ], + }), + finished: Some(Finished { + usage: vllm_llm::TokenUsage { + prompt_token_count: 2, + output_token_count: 2, + cached_token_count: 0, + }, + finish_reason: FinishReason::stop_eos(), + kv_transfer_params: None, + }), + }), + ]); + + let collected = stream.collect_output().await.unwrap(); + assert_eq!(collected.text, "bc"); + assert_eq!( + collected.prompt_logprobs, + Some(DecodedPromptLogprobs { + first_token_id: 0, + first_token: "o".to_string(), + scored_positions: vec![DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "p".to_string(), + logprob: -0.1, + rank: 1, + }], + }], + }) + ); + assert_eq!( + collected.logprobs, + Some(DecodedLogprobs { + positions: vec![ + DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "a".to_string(), + logprob: -0.2, + rank: 1, + }], + }, + DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "bc".to_string(), + logprob: -0.3, + rank: 1, + }], + }, + ], + }) + ); + } + + #[tokio::test] + async fn collect_output_accumulates_intermediate_deltas() { + let stream = stream::iter(vec![ + Ok(DecodedTextEvent::Start { + prompt_token_ids: vec![10, 11].into(), + prompt_logprobs: None, + }), + Ok(DecodedTextEvent::TextDelta { + delta: "he".to_string(), + token_ids: vec![1, 2], + logprobs: Some(DecodedLogprobs { + positions: vec![ + DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "h".to_string(), + logprob: -0.1, + rank: 1, + }], + }, + DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "e".to_string(), + logprob: -0.2, + rank: 1, + }], + }, + ], + }), + finished: None, + }), + Ok(DecodedTextEvent::TextDelta { + delta: "llo".to_string(), + token_ids: vec![3, 4, 5], + logprobs: Some(DecodedLogprobs { + positions: vec![ + DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "l".to_string(), + logprob: -0.3, + rank: 1, + }], + }, + DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "l".to_string(), + logprob: -0.4, + rank: 1, + }], + }, + DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "o".to_string(), + logprob: -0.5, + rank: 1, + }], + }, + ], + }), + finished: Some(Finished { + usage: vllm_llm::TokenUsage { + prompt_token_count: 2, + output_token_count: 5, + cached_token_count: 0, + }, + finish_reason: FinishReason::stop_eos(), + kv_transfer_params: None, + }), + }), + ]); + + let collected = stream.collect_output().await.unwrap(); + assert_eq!(collected.text, "hello"); + assert_eq!(collected.prompt_logprobs, None); + assert_eq!(collected.token_ids, vec![1, 2, 3, 4, 5]); + assert_eq!( + collected.logprobs, + Some(DecodedLogprobs { + positions: vec![ + DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "h".to_string(), + logprob: -0.1, + rank: 1, + }], + }, + DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "e".to_string(), + logprob: -0.2, + rank: 1, + }], + }, + DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "l".to_string(), + logprob: -0.3, + rank: 1, + }], + }, + DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "l".to_string(), + logprob: -0.4, + rank: 1, + }], + }, + DecodedPositionLogprobs { + entries: vec![DecodedTokenLogprob { + token_id: 0, + token: "o".to_string(), + logprob: -0.5, + rank: 1, + }], + }, + ], + }) + ); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/text/src/request.rs b/asian_ai/performance/vLLM/rust/src/text/src/request.rs new file mode 100644 index 0000000000000000000000000000000000000000..f2cd49c5c201b314c339d413cb50885155f8d446 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/text/src/request.rs @@ -0,0 +1,220 @@ +use std::collections::HashMap; + +use enum_as_inner::EnumAsInner; +use serde::{Deserialize, Serialize}; +use serde_json::Value; +use vllm_engine_core_client::protocol::lora::LoraRequest; +use vllm_engine_core_client::protocol::multimodal::MmFeatures; +use vllm_engine_core_client::protocol::request::ReasoningParserKwargs; +use vllm_engine_core_client::protocol::sampling::RepetitionDetectionParams; +use vllm_engine_core_client::protocol::structured_outputs::StructuredOutputsParams; + +use crate::error::{Error, Result}; +use crate::output::TextDecodeOptions; + +/// One raw text-generation prompt. +/// +/// This supports either ordinary text that still needs tokenization or +/// already-tokenized prompt IDs that should bypass tokenizer work entirely. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, EnumAsInner)] +#[serde(untagged)] +pub enum Prompt { + /// Untokenized prompt text that still needs tokenizer work before + /// generation. + Text(String), + /// Pre-tokenized prompt IDs that should be forwarded southbound without + /// re-encoding. + TokenIds(Vec), +} + +impl Default for Prompt { + fn default() -> Self { + Self::Text(String::new()) // placeholder + } +} + +/// User-facing sampling parameters accepted by `vllm-text`. +/// +/// This intentionally keeps only the subset that the current Rust text layer +/// supports as northbound request semantics. Engine-core-specific normalized +/// fields are derived later during lowering. +/// +/// Original Python definition: +/// +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(default)] +pub struct SamplingParams { + /// Controls randomness. Lower values are more deterministic; zero means + /// greedy sampling. `None` means no explicit user override. + pub temperature: Option, + /// Cumulative probability threshold for nucleus sampling. + pub top_p: Option, + /// Maximum number of top tokens to consider. `Some(0)` means all tokens. + pub top_k: Option, + /// Random seed used by the sampler when present. + pub seed: Option, + /// Maximum number of tokens to generate. `None` means no explicit user + /// override. + pub max_tokens: Option, + /// Minimum number of tokens to generate before EOS or stop-token handling. + pub min_tokens: Option, + /// Maximum number of reasoning ("thinking") tokens to emit before the + /// reasoning section is force-closed. `None` or the user-facing `-1` + /// "unlimited" sentinel both disable the budget. The raw value is carried + /// here; `-1` is normalized to `None` (and other negatives rejected) during + /// lowering (see `lower_sampling_params`). + pub thinking_token_budget: Option, + /// Number of log probabilities to return per generated token. + /// + /// `None` disables sample logprobs. `-1` requests the full vocabulary. + pub logprobs: Option, + /// Number of log probabilities to return per prompt token. + /// + /// `None` disables prompt logprobs. `-1` requests the full vocabulary. + pub prompt_logprobs: Option, + /// Minimum probability threshold for token sampling. `None` means no + /// explicit user override. + pub min_p: Option, + /// Frequency penalty applied by the sampler. `None` means no explicit user + /// override. + pub frequency_penalty: Option, + /// Presence penalty applied by the sampler. `None` means no explicit user + /// override. + pub presence_penalty: Option, + /// Repetition penalty applied by the sampler. `None` means no explicit user + /// override. + pub repetition_penalty: Option, + /// Parameters for detecting repetitive N-gram patterns. `None` means no + /// explicit user override. + pub repetition_detection: Option, + /// Explicit stop token IDs provided by the caller. `None` means no explicit + /// user override. + pub stop_token_ids: Option>, + /// If true, do not stop on the model's primary EOS token. + pub ignore_eos: bool, + /// Modify the likelihood of specified tokens appearing in the completion. + /// Keys are token IDs. + pub logit_bias: Option>, + /// Restrict output to these token IDs only. + pub allowed_token_ids: Option>, + /// Words to avoid during generation (tokenized to IDs during lowering). + pub bad_words: Option>, + /// Specific token IDs for which log probabilities should be returned at + /// each position. + /// + /// When set, the engine returns logprobs for exactly these tokens in + /// addition to the sampled/scored token. Mutually exclusive with + /// `logprobs` in practice. + pub logprob_token_ids: Option>, + /// Parameters for configuring structured outputs (guided decoding). + pub structured_outputs: Option, + /// If true, bypass reads from the prefix cache for this request (the prompt + /// will not reuse cached KV blocks from earlier requests, though newly + /// computed blocks may still populate the cache). `None` defers to + /// engine-core defaults. + pub skip_reading_prefix_cache: Option, + /// Additional request parameters for custom extensions. + pub vllm_xargs: Option>, +} + +#[allow(clippy::derivable_impls)] // more explicit +impl Default for SamplingParams { + fn default() -> Self { + Self { + temperature: None, + top_p: None, + top_k: None, + seed: None, + max_tokens: None, + min_tokens: None, + thinking_token_budget: None, + logprobs: None, + prompt_logprobs: None, + min_p: None, + frequency_penalty: None, + presence_penalty: None, + repetition_penalty: None, + repetition_detection: None, + stop_token_ids: None, + ignore_eos: false, + logit_bias: None, + allowed_token_ids: None, + bad_words: None, + logprob_token_ids: None, + structured_outputs: None, + skip_reading_prefix_cache: None, + vllm_xargs: None, + } + } +} + +/// One raw text-generation request ready to be tokenized or sent directly to +/// the engine. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct TextRequest { + /// Stable caller-supplied request ID. + pub request_id: String, + /// Prompt text or prompt token IDs for this request. + pub prompt: Prompt, + /// Multimodal features prepared by a higher-level frontend. Raw text + /// requests keep this empty; multimodal chat uses it with pre-tokenized + /// prompt IDs. + pub mm_features: Option, + /// User-facing sampling parameters accepted by `vllm-text`. + pub sampling_params: SamplingParams, + /// Incremental detokenization options for the response path. + pub decode_options: TextDecodeOptions, + /// Whether to emit intermediate northbound deltas before the terminal + /// result. + /// + /// If `false`, callers only observe the terminal accumulated output. If + /// `true`, callers may receive zero or more incremental decoded updates + /// before the final terminal event. + pub intermediate: bool, + /// Request scheduling priority (lower means earlier handling; default 0). + pub priority: i32, + /// Salt for prefix cache isolation in multi-user environments. + pub cache_salt: Option, + /// Whether to add special tokens (e.g. BOS) during prompt tokenization. + pub add_special_tokens: bool, + /// Override data parallel rank. + #[serde(default)] + pub data_parallel_rank: Option, + /// Optional reasoning-parser kwargs forwarded to engine-side structured + /// output logic. + #[serde(default)] + pub reasoning_parser_kwargs: Option, + /// LoRA adapter selected for this request. + #[serde(default)] + pub lora_request: Option, +} + +impl TextRequest { + /// Return one minimal valid request fixture for tests. + pub fn for_test() -> Self { + Self { + request_id: "test-request".to_string(), + prompt: Prompt::Text("test".to_string()), + mm_features: None, + sampling_params: SamplingParams::default(), + decode_options: TextDecodeOptions::default(), + intermediate: true, + priority: 0, + cache_salt: None, + add_special_tokens: false, + data_parallel_rank: None, + reasoning_parser_kwargs: None, + lora_request: None, + } + } + + /// Validate the minimum invariants before tokenization or request lowering. + pub fn validate(&self) -> Result<()> { + if matches!(&self.prompt, Prompt::TokenIds(ids) if ids.is_empty()) { + return Err(Error::EmptyPromptTokenIds { + request_id: self.request_id.clone(), + }); + } + Ok(()) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/tokenizer/Cargo.toml b/asian_ai/performance/vLLM/rust/src/tokenizer/Cargo.toml new file mode 100644 index 0000000000000000000000000000000000000000..74d8452509e7773b42b5b411140feade38f8642a --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/tokenizer/Cargo.toml @@ -0,0 +1,40 @@ +[package] +name = "vllm-tokenizer" +version.workspace = true +edition.workspace = true +license.workspace = true + +[features] +test-utils = [] + +[dependencies] +base64.workspace = true +fastokens.workspace = true +riptoken.workspace = true +rustc-hash.workspace = true +serde.workspace = true +serde_json.workspace = true +tekken.workspace = true +thiserror.workspace = true +thiserror-ext.workspace = true +tiktoken-rs.workspace = true +tokenizers.workspace = true +tracing.workspace = true + +[dev-dependencies] +criterion.workspace = true +hf-hub.workspace = true +reqwest.workspace = true +tempfile.workspace = true +tokio.workspace = true + +[[bench]] +name = "hf" +harness = false + +[[bench]] +name = "tiktoken" +harness = false + +[lints] +workspace = true diff --git a/asian_ai/performance/vLLM/rust/src/tokenizer/benches/hf.rs b/asian_ai/performance/vLLM/rust/src/tokenizer/benches/hf.rs new file mode 100644 index 0000000000000000000000000000000000000000..b11f0697d91c798e5e90dbaa2f7337206408fe72 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/tokenizer/benches/hf.rs @@ -0,0 +1,122 @@ +use criterion::{Criterion, Throughput, black_box, criterion_group, criterion_main}; +use hf_hub::api::tokio::ApiBuilder; +use tokio::runtime::Runtime; +use vllm_tokenizer::{HuggingFaceTokenizer, Tokenizer}; + +const MODEL_ID: &str = "Qwen/Qwen3.5-0.8B"; +const SAMPLE_TEXT: &str = "\ +<|im_start|>system +You are Qwen3.5, a helpful assistant. +<|im_end|> +<|im_start|>user +请用中英混合总结以下需求,并给出一个简短的 JSON 示例。 +The service should stop cleanly at EOS, avoid leaking the next template turn, and keep decode latency low. +Input: 4 concurrent requests, 10240 prompt tokens, 16 generated tokens. +<|im_end|> +<|im_start|>assistant +"; + +struct BenchFixture { + fastokens: HuggingFaceTokenizer, + hf: HuggingFaceTokenizer, + text: String, + token_ids: Vec, +} + +impl BenchFixture { + fn load() -> Self { + let path = tokenizer_json(); + let fastokens = + HuggingFaceTokenizer::new_fastokens(&path).expect("load fastokens tokenizer"); + let hf = HuggingFaceTokenizer::new_hf(&path).expect("load huggingface tokenizer"); + + let text = SAMPLE_TEXT.repeat(32); + let hf_token_ids = + hf.encode(text.as_str(), false).expect("encode sample text with hf tokenizer"); + let fastokens_token_ids = fastokens + .encode(text.as_str(), false) + .expect("encode sample text with fastokens"); + assert_eq!(fastokens_token_ids, hf_token_ids); + + let hf_decoded = hf + .decode(hf_token_ids.as_slice(), false) + .expect("decode sample token ids with hf tokenizer"); + let fastokens_decoded = fastokens + .decode(hf_token_ids.as_slice(), false) + .expect("decode sample token ids with fastokens"); + assert_eq!(fastokens_decoded, hf_decoded); + + Self { + fastokens, + hf, + text, + token_ids: hf_token_ids, + } + } +} + +fn tokenizer_json() -> std::path::PathBuf { + Runtime::new().expect("build tokio runtime").block_on(async { + ApiBuilder::from_env() + .with_progress(false) + .build() + .expect("build hf-hub api") + .model(MODEL_ID.to_string()) + .get("tokenizer.json") + .await + .expect("fetch tokenizer.json from hf-hub") + }) +} + +fn bench_encode(c: &mut Criterion) { + let fixture = BenchFixture::load(); + let mut group = c.benchmark_group("tokenizer_encode"); + group.throughput(Throughput::Bytes(fixture.text.len() as u64)); + + group.bench_function("fastokens", |b| { + b.iter(|| { + fixture + .fastokens + .encode(black_box(fixture.text.as_str()), black_box(false)) + .expect("encode sample text with fastokens") + }) + }); + group.bench_function("hf_tokenizers", |b| { + b.iter(|| { + fixture + .hf + .encode(black_box(fixture.text.as_str()), black_box(false)) + .expect("encode sample text with hf tokenizer") + }) + }); + + group.finish(); +} + +fn bench_decode(c: &mut Criterion) { + let fixture = BenchFixture::load(); + let mut group = c.benchmark_group("tokenizer_decode"); + group.throughput(Throughput::Elements(fixture.token_ids.len() as u64)); + + group.bench_function("fastokens", |b| { + b.iter(|| { + fixture + .fastokens + .decode(black_box(fixture.token_ids.as_slice()), black_box(false)) + .expect("decode sample token ids with fastokens") + }) + }); + group.bench_function("hf_tokenizers", |b| { + b.iter(|| { + fixture + .hf + .decode(black_box(fixture.token_ids.as_slice()), black_box(false)) + .expect("decode sample token ids with hf tokenizer") + }) + }); + + group.finish(); +} + +criterion_group!(benches, bench_encode, bench_decode); +criterion_main!(benches); diff --git a/asian_ai/performance/vLLM/rust/src/tokenizer/benches/tiktoken.rs b/asian_ai/performance/vLLM/rust/src/tokenizer/benches/tiktoken.rs new file mode 100644 index 0000000000000000000000000000000000000000..f46dd5983a51e07a0f28e0fe50b03404c0338d11 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/tokenizer/benches/tiktoken.rs @@ -0,0 +1,121 @@ +use criterion::{Criterion, Throughput, black_box, criterion_group, criterion_main}; +use hf_hub::api::tokio::ApiBuilder; +use tokio::runtime::Runtime; +use vllm_tokenizer::{TiktokenTokenizer, Tokenizer}; + +const MODEL_ID: &str = "moonshotai/Kimi-K2.5"; +const SAMPLE_TEXT: &str = "\ + +I'm sure it's fine, but I can't say I'd trust that it's what we'd ship. + +请用中英混合总结以下需求,并保留 tool-call marker: +<|tool_calls_section_begin|>{\"name\":\"summarize\",\"arguments\":{\"style\":\"brief\"}}<|tool_calls_section_end|> +The service should stop cleanly at EOS, avoid leaking the next template turn, and keep decode latency low. +"; + +struct BenchFixture { + riptoken: TiktokenTokenizer, + tiktoken_rs: TiktokenTokenizer, + text: String, + token_ids: Vec, +} + +impl BenchFixture { + fn load() -> Self { + let path = tiktoken_model(); + let riptoken = TiktokenTokenizer::new_riptoken(&path).expect("load riptoken tokenizer"); + let tiktoken_rs = + TiktokenTokenizer::new_tiktoken_rs(&path).expect("load tiktoken-rs tokenizer"); + + let text = SAMPLE_TEXT.repeat(32); + let riptoken_token_ids = + riptoken.encode(text.as_str(), false).expect("encode sample text with riptoken"); + let tiktoken_rs_token_ids = tiktoken_rs + .encode(text.as_str(), false) + .expect("encode sample text with tiktoken-rs"); + assert_eq!(riptoken_token_ids, tiktoken_rs_token_ids); + + let riptoken_decoded = riptoken + .decode(riptoken_token_ids.as_slice(), false) + .expect("decode sample token ids with riptoken"); + let tiktoken_rs_decoded = tiktoken_rs + .decode(riptoken_token_ids.as_slice(), false) + .expect("decode sample token ids with tiktoken-rs"); + assert_eq!(riptoken_decoded, tiktoken_rs_decoded); + + Self { + riptoken, + tiktoken_rs, + text, + token_ids: riptoken_token_ids, + } + } +} + +fn tiktoken_model() -> std::path::PathBuf { + Runtime::new().expect("build tokio runtime").block_on(async { + let repo = ApiBuilder::from_env() + .with_progress(false) + .build() + .expect("build hf-hub api") + .model(MODEL_ID.to_string()); + repo.get("config.json").await.expect("fetch config.json from hf-hub"); + repo.get("tokenizer_config.json") + .await + .expect("fetch tokenizer_config.json from hf-hub"); + repo.get("tiktoken.model").await.expect("fetch tiktoken.model from hf-hub") + }) +} + +fn bench_encode(c: &mut Criterion) { + let fixture = BenchFixture::load(); + let mut group = c.benchmark_group("tiktoken_encode"); + group.throughput(Throughput::Bytes(fixture.text.len() as u64)); + + group.bench_function("riptoken", |b| { + b.iter(|| { + fixture + .riptoken + .encode(black_box(fixture.text.as_str()), black_box(false)) + .expect("encode sample text with riptoken") + }) + }); + group.bench_function("tiktoken_rs", |b| { + b.iter(|| { + fixture + .tiktoken_rs + .encode(black_box(fixture.text.as_str()), black_box(false)) + .expect("encode sample text with tiktoken-rs") + }) + }); + + group.finish(); +} + +fn bench_decode(c: &mut Criterion) { + let fixture = BenchFixture::load(); + let mut group = c.benchmark_group("tiktoken_decode"); + group.throughput(Throughput::Elements(fixture.token_ids.len() as u64)); + + group.bench_function("riptoken", |b| { + b.iter(|| { + fixture + .riptoken + .decode(black_box(fixture.token_ids.as_slice()), black_box(false)) + .expect("decode sample token ids with riptoken") + }) + }); + group.bench_function("tiktoken_rs", |b| { + b.iter(|| { + fixture + .tiktoken_rs + .decode(black_box(fixture.token_ids.as_slice()), black_box(false)) + .expect("decode sample token ids with tiktoken-rs") + }) + }); + + group.finish(); +} + +criterion_group!(benches, bench_encode, bench_decode); +criterion_main!(benches); diff --git a/asian_ai/performance/vLLM/rust/src/tokenizer/src/byte_level_decode.rs b/asian_ai/performance/vLLM/rust/src/tokenizer/src/byte_level_decode.rs new file mode 100644 index 0000000000000000000000000000000000000000..f35fdd192141cb21f2e393fbfb6e520248ccb19a --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/tokenizer/src/byte_level_decode.rs @@ -0,0 +1,119 @@ +//! Fast GPT-2 byte-level detokenization that writes into a single `Vec`, +//! avoiding the `Vec` / `String::join` assembly in fastokens' generic +//! `Decoder::decode` pipeline. + +/// Reverse GPT-2 byte-to-unicode mapping: codepoint → original byte. The GPT-2 +/// table only emits codepoints in U+0000..U+0143, so a flat array suffices. +const CHAR_TO_BYTE: [u8; 324] = build_char_to_byte(); + +const fn is_nice(b: u8) -> bool { + (b >= b'!' && b <= b'~') || (b >= 0xA1 && b <= 0xAC) || b >= 0xAE +} + +const fn build_char_to_byte() -> [u8; 324] { + let mut table = [0u8; 324]; + let mut b: u16 = 0; + while b < 256 { + let cp = if is_nice(b as u8) { + b as u32 + } else { + 256 + nice_offset(b as u8) + }; + table[cp as usize] = b as u8; + b += 1; + } + table +} + +const fn nice_offset(b: u8) -> u32 { + let mut i: u16 = 0; + let mut n: u32 = 0; + while i < b as u16 { + if !is_nice(i as u8) { + n += 1; + } + i += 1; + } + n +} + +/// Decode byte-level encoded token strings into a single UTF-8 string, +/// matching `fastokens::decoders::ByteLevelDecoder`. +pub fn decode_byte_level<'a, I: IntoIterator>(tokens: I) -> String { + let iter = tokens.into_iter(); + let (lower, _) = iter.size_hint(); + let mut bytes: Vec = Vec::with_capacity(lower.saturating_mul(4)); + for token in iter { + for c in token.chars() { + let cp = c as usize; + if cp < CHAR_TO_BYTE.len() { + bytes.push(CHAR_TO_BYTE[cp]); + } else { + // Non-GPT2 codepoints (e.g. DeepSeek's U+FF5C, U+2581) pass through. + let mut buf = [0u8; 4]; + bytes.extend_from_slice(c.encode_utf8(&mut buf).as_bytes()); + } + } + } + String::from_utf8(bytes).unwrap_or_else(|e| String::from_utf8_lossy(e.as_bytes()).into_owned()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn build_byte_to_char_ref() -> [char; 256] { + let mut table = ['\0'; 256]; + let mut next: u32 = 256; + for b in 0..=255u8 { + let cp = if is_nice(b) { + b as u32 + } else { + let cp = next; + next += 1; + cp + }; + table[b as usize] = char::from_u32(cp).unwrap(); + } + table + } + + #[test] + fn char_to_byte_roundtrips_every_byte() { + let byte_to_char = build_byte_to_char_ref(); + for b in 0..=255u8 { + let cp = byte_to_char[b as usize] as usize; + assert!(cp < CHAR_TO_BYTE.len()); + assert_eq!(CHAR_TO_BYTE[cp], b, "mismatch for byte {b:#x}"); + } + } + + #[test] + fn decode_ascii() { + assert_eq!(decode_byte_level(["Hello"]), "Hello"); + } + + #[test] + fn decode_space_marker() { + // GPT-2 maps 0x20 → Ġ (U+0120). + assert_eq!( + decode_byte_level(["\u{120}Hello", "\u{120}world"]), + " Hello world", + ); + } + + #[test] + fn decode_multibyte_euro() { + // € → 0xE2 0x82 0xAC, each mapped to a specific GPT-2 char. + let byte_to_char = build_byte_to_char_ref(); + let encoded: String = + [0xE2u8, 0x82, 0xAC].iter().map(|&b| byte_to_char[b as usize]).collect(); + assert_eq!(decode_byte_level([encoded.as_str()]), "€"); + } + + #[test] + fn decode_preserves_non_gpt2_chars() { + let tok = "<\u{FF5C}begin\u{2581}of\u{2581}sentence\u{FF5C}>"; + assert_eq!(decode_byte_level([tok]), "<|begin▁of▁sentence|>"); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/tokenizer/src/error.rs b/asian_ai/performance/vLLM/rust/src/tokenizer/src/error.rs new file mode 100644 index 0000000000000000000000000000000000000000..83213b2c610b0c2319e41cacc2e3d6203d068358 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/tokenizer/src/error.rs @@ -0,0 +1,9 @@ +use thiserror::Error; +use thiserror_ext::Macro; + +pub type Result = std::result::Result; + +#[derive(Debug, Error, Macro)] +#[thiserror_ext(macro(path = "crate::error"))] +#[error("tokenizer error: {0}")] +pub struct TokenizerError(#[message] pub String); diff --git a/asian_ai/performance/vLLM/rust/src/tokenizer/src/hf.rs b/asian_ai/performance/vLLM/rust/src/tokenizer/src/hf.rs new file mode 100644 index 0000000000000000000000000000000000000000..d323ffd1389ce3fdc9af1e72e487432118053382 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/tokenizer/src/hf.rs @@ -0,0 +1,387 @@ +use std::path::Path; +use std::sync::Arc; + +use fastokens::Tokenizer as FastokensTokenizer; +use fastokens::decoders::Decoder as FastokensDecoder; +use thiserror_ext::AsReport as _; +use tokenizers::Tokenizer as HfTokenizer; +use tracing::{info, warn}; + +use crate::byte_level_decode::decode_byte_level; +use crate::hf::added_tokens::load_tokenizer_json_with_extra_tokens; +use crate::{Result, Tokenizer}; + +mod added_tokens; + +enum Backend { + Hf(Box), + Fastokens(Box), + /// Fastokens tokenizer whose decoder is pure GPT-2 byte-level, so we can + /// bypass `Decoder::decode`'s `Vec`/`join("")` assembly. + FastokensByteLevel(Box), +} + +/// True if `dec` is effectively a single `ByteLevel` stage — one `ByteLevel` +/// leaf in a tree of `Sequence`s (fastokens represents `Fuse` as an empty +/// `Sequence`, which is a no-op for our purposes). +fn is_byte_level_only(dec: &FastokensDecoder) -> bool { + fn count_byte_level(dec: &FastokensDecoder) -> usize { + match dec { + FastokensDecoder::ByteLevel(_) => 1, + FastokensDecoder::Sequence(steps) => steps.iter().map(count_byte_level).sum(), + } + } + count_byte_level(dec) == 1 +} + +fn decode_fastokens_byte_level( + t: &FastokensTokenizer, + token_ids: &[u32], + skip_special_tokens: bool, +) -> Result { + let tokens: Vec<&str> = token_ids + .iter() + .filter(|&&id| !(skip_special_tokens && t.is_special_token(id))) + .map(|&id| { + t.id_to_token(id) + .ok_or_else(|| tokenizer_error!("decoding failed: unknown token ID: {id}")) + }) + .collect::>()?; + Ok(decode_byte_level(tokens)) +} + +/// Tokenizer from `tokenizer.json` in HuggingFace format. +/// +/// This tries to load with `fastokens` first for better performance, then falls +/// back to HuggingFace's `tokenizers` if the former fails (e.g. due to +/// unsupported tokenizer features or file formats). +pub struct HuggingFaceTokenizer { + backend: Backend, + special_token_ids: Arc<[u32]>, +} + +impl HuggingFaceTokenizer { + fn from_hf_backend(tokenizer: HfTokenizer) -> Self { + let special_token_ids = { + let mut ids: Vec = tokenizer + .get_added_tokens_decoder() + .iter() + .filter(|(_id, token)| token.special) + .map(|(id, _token)| *id) + .collect(); + ids.sort_unstable(); + ids.dedup(); + Arc::from(ids) + }; + Self { + backend: Backend::Hf(Box::new(tokenizer)), + special_token_ids, + } + } + + fn from_fastokens_backend(tokenizer: FastokensTokenizer) -> Self { + let special_token_ids = { + let mut ids: Vec = tokenizer + .added_tokens() + .into_iter() + .flat_map(|added_tokens| added_tokens.iter()) + .filter(|token| token.special) + .map(|token| token.id) + .collect(); + ids.sort_unstable(); + ids.dedup(); + Arc::from(ids) + }; + let byte_level = tokenizer.decoder().is_some_and(is_byte_level_only); + let backend = if byte_level { + Backend::FastokensByteLevel(Box::new(tokenizer)) + } else { + Backend::Fastokens(Box::new(tokenizer)) + }; + Self { + backend, + special_token_ids, + } + } + + /// Load from `tokenizer.json` with `fastokens`. + pub fn new_fastokens(path: &Path) -> Result { + info!(path = %path.display(), "loading tokenizer with fastokens"); + let tokenizer_json = load_tokenizer_json_with_extra_tokens(path)?; + let t = FastokensTokenizer::from_json(tokenizer_json) + .map_err(|error| tokenizer_error!("failed to load tokenizer: {}", error.as_report()))?; + Ok(Self::from_fastokens_backend(t)) + } + + /// Load from `tokenizer.json` with Hugging Face `tokenizers`. + pub fn new_hf(path: &Path) -> Result { + info!(path = %path.display(), "loading tokenizer with huggingface tokenizers"); + let tokenizer_json = load_tokenizer_json_with_extra_tokens(path)?; + let t = serde_json::from_value::(tokenizer_json) + .map_err(|error| tokenizer_error!("failed to load tokenizer: {}", error.as_report()))?; + Ok(Self::from_hf_backend(t)) + } + + /// Load from `tokenizer.json` via fastokens or HuggingFace tokenizers. + pub fn new(path: &Path) -> Result { + match Self::new_fastokens(path) { + Ok(tokenizer) => Ok(tokenizer), + Err(error) => { + warn!( + path = %path.display(), + error = %error.as_report(), + "failed to load tokenizer with fastokens; falling back to HuggingFace tokenizers" + ); + Self::new_hf(path) + } + } + } +} + +impl Tokenizer for HuggingFaceTokenizer { + fn encode(&self, text: &str, add_special_tokens: bool) -> Result> { + match &self.backend { + Backend::Hf(t) => { + let encoding = t + .encode(text, add_special_tokens) + .map_err(|error| tokenizer_error!("encoding failed: {}", error.as_report()))?; + Ok(encoding.get_ids().to_vec()) + } + Backend::Fastokens(t) | Backend::FastokensByteLevel(t) => t + .encode_with_special_tokens(text, add_special_tokens) + .map_err(|error| tokenizer_error!("encoding failed: {}", error.as_report())), + } + } + + fn decode(&self, token_ids: &[u32], skip_special_tokens: bool) -> Result { + match &self.backend { + Backend::Hf(t) => t + .decode(token_ids, skip_special_tokens) + .map_err(|error| tokenizer_error!("decoding failed: {}", error.as_report())), + Backend::Fastokens(t) => t + .decode(token_ids, skip_special_tokens) + .map_err(|error| tokenizer_error!("decoding failed: {}", error.as_report())), + Backend::FastokensByteLevel(t) => { + decode_fastokens_byte_level(t, token_ids, skip_special_tokens) + } + } + } + + fn token_to_id(&self, token: &str) -> Option { + match &self.backend { + Backend::Hf(t) => t.token_to_id(token), + Backend::Fastokens(t) | Backend::FastokensByteLevel(t) => t.token_to_id(token), + } + } + + fn vocab_size(&self) -> usize { + match &self.backend { + Backend::Hf(t) => t.get_vocab_size(true), + Backend::Fastokens(t) | Backend::FastokensByteLevel(t) => t.vocab_size(), + } + } + + fn id_to_token(&self, id: u32) -> Option { + match &self.backend { + Backend::Hf(t) => t.id_to_token(id), + Backend::Fastokens(t) | Backend::FastokensByteLevel(t) => { + t.id_to_token(id).map(ToOwned::to_owned) + } + } + } + + fn is_special_id(&self, token_id: u32) -> bool { + self.special_token_ids.binary_search(&token_id).is_ok() + } +} + +#[cfg(test)] +mod tests { + use tempfile::tempdir; + use tokenizers::models::bpe::BPE; + use tokenizers::{AddedToken, Tokenizer as HfTokenizer}; + + use super::{HuggingFaceTokenizer, Tokenizer}; + + fn tiny_bpe_tokenizer() -> HfTokenizer { + let vocab = [ + ("".to_string(), 0), + ("h".to_string(), 1), + ("e".to_string(), 2), + ("l".to_string(), 3), + ("o".to_string(), 4), + ("he".to_string(), 5), + ("ll".to_string(), 6), + ("hell".to_string(), 7), + ("hello".to_string(), 8), + ]; + let merges = vec![ + ("h".to_string(), "e".to_string()), + ("l".to_string(), "l".to_string()), + ("he".to_string(), "ll".to_string()), + ("hell".to_string(), "o".to_string()), + ]; + let model = BPE::builder() + .vocab_and_merges(vocab, merges) + .unk_token("".to_string()) + .build() + .expect("build bpe tokenizer"); + HfTokenizer::new(model) + } + + #[test] + fn hf_constructor_resolves_added_token_ids() { + let mut tokenizer = tiny_bpe_tokenizer(); + tokenizer.add_special_tokens(&[AddedToken::from("<|im_end|>", true)]); + + let dir = tempdir().expect("create temp dir"); + let path = dir.path().join("tokenizer.json"); + tokenizer.save(&path, false).expect("save tokenizer json"); + + let wrapper = HuggingFaceTokenizer::new_hf(&path).expect("load hf wrapper"); + let special_id = wrapper.token_to_id("<|im_end|>").expect("resolve added special token id"); + assert!(wrapper.is_special_id(special_id)); + } + + #[test] + fn new_fastokens_preserves_special_ids_from_fastokens_metadata() { + let mut tokenizer = tiny_bpe_tokenizer(); + tokenizer.add_special_tokens(&[AddedToken::from("<|im_end|>", true)]); + + let dir = tempdir().expect("create temp dir"); + let path = dir.path().join("tokenizer.json"); + tokenizer.save(&path, false).expect("save tokenizer json"); + + let wrapper = HuggingFaceTokenizer::new_fastokens(&path) + .expect("load wrapper with fastokens backend"); + assert!(matches!( + wrapper.backend, + super::Backend::Fastokens(_) | super::Backend::FastokensByteLevel(_), + )); + let special_id = wrapper.token_to_id("<|im_end|>").expect("resolve added special token id"); + assert!(wrapper.is_special_id(special_id)); + } + + #[test] + fn constructors_merge_extra_added_tokens_from_tokenizer_config() { + let tokenizer = tiny_bpe_tokenizer(); + + let dir = tempdir().expect("create temp dir"); + let path = dir.path().join("tokenizer.json"); + tokenizer.save(&path, false).expect("save tokenizer json"); + std::fs::write( + dir.path().join("tokenizer_config.json"), + r#"{ + "added_tokens_decoder": { + "9": { + "content": "<|image_pad|>", + "special": true, + "normalized": false + } + } + }"#, + ) + .expect("write tokenizer config"); + + for wrapper in [ + HuggingFaceTokenizer::new_fastokens(&path).expect("load fastokens wrapper"), + HuggingFaceTokenizer::new_hf(&path).expect("load hf wrapper"), + ] { + assert_eq!(wrapper.token_to_id("<|image_pad|>"), Some(9)); + assert_eq!(wrapper.id_to_token(9).as_deref(), Some("<|image_pad|>")); + assert!(wrapper.is_special_id(9)); + } + } + + /// BPE tokenizer that round-trips through fastokens with a genuine + /// `ByteLevel` decoder; vocab covers both GPT-2 (Ġ U+0120) and non-GPT-2 + /// (| U+FF5C) codepoints. + fn tiny_byte_level_bpe() -> fastokens::Tokenizer { + let raw = r#"{ + "version": "1.0", + "truncation": null, + "padding": null, + "added_tokens": [ + {"id": 0, "content": "<|endoftext|>", "single_word": false, + "lstrip": false, "rstrip": false, "normalized": false, "special": true} + ], + "normalizer": null, + "pre_tokenizer": {"type": "ByteLevel", "add_prefix_space": false, + "trim_offsets": true, "use_regex": true}, + "post_processor": null, + "decoder": {"type": "ByteLevel", "add_prefix_space": false, + "trim_offsets": true, "use_regex": true}, + "model": { + "type": "BPE", + "dropout": null, + "unk_token": null, + "continuing_subword_prefix": null, + "end_of_word_suffix": null, + "fuse_unk": false, + "byte_fallback": false, + "ignore_merges": false, + "vocab": { + "<|endoftext|>": 0, + "H": 1, "e": 2, "l": 3, "o": 4, "w": 5, "r": 6, "d": 7, + "Ġ": 8, "!": 9, + "|": 10 + }, + "merges": [] + } + }"#; + let value: serde_json::Value = serde_json::from_str(raw).expect("parse tokenizer json"); + fastokens::Tokenizer::from_json(value).expect("build fastokens tokenizer") + } + + #[test] + fn byte_level_detected_direct() { + let t = tiny_byte_level_bpe(); + assert!(super::is_byte_level_only(t.decoder().expect("decoder"))); + } + + #[test] + fn byte_level_detected_inside_sequence() { + let raw = r#"{ + "type": "Sequence", + "decoders": [ + {"type": "ByteLevel", "add_prefix_space": false, + "trim_offsets": true, "use_regex": true}, + {"type": "Fuse"} + ] + }"#; + let config: fastokens::DecoderConfig = + serde_json::from_str(raw).expect("parse decoder config"); + let dec = + fastokens::decoders::Decoder::from_config(config).expect("build decoder from config"); + assert!(super::is_byte_level_only(&dec)); + } + + /// Fast path must produce byte-identical output to fastokens' own decode. + #[test] + fn fast_byte_level_matches_fastokens_decode() { + let t = tiny_byte_level_bpe(); + let cases: &[&[u32]] = &[ + &[], + &[1, 2, 3, 3, 4], // "Hello" + &[1, 2, 3, 3, 4, 8, 5, 4, 6, 3, 7], // "Hello world" + &[0, 1, 2, 3, 3, 4, 0, 9, 0], // specials interleaved + &[10, 1, 2, 3, 3, 4, 10], // |Hello| (non-GPT2 chars) + ]; + for ids in cases { + for &skip in &[false, true] { + let expected = t.decode(ids, skip).expect("fastokens decode"); + let got = + super::decode_fastokens_byte_level(&t, ids, skip).expect("fast-path decode"); + assert_eq!(got, expected, "ids={ids:?} skip={skip}"); + } + } + } + + #[test] + fn fast_byte_level_errors_on_unknown_id() { + let t = tiny_byte_level_bpe(); + let err = super::decode_fastokens_byte_level(&t, &[999], false) + .expect_err("unknown id must error"); + assert!(format!("{err:?}").contains("999")); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/tokenizer/src/hf/added_tokens.rs b/asian_ai/performance/vLLM/rust/src/tokenizer/src/hf/added_tokens.rs new file mode 100644 index 0000000000000000000000000000000000000000..78b6c7fc5812a9eaf043461404f45d8df2534a10 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/tokenizer/src/hf/added_tokens.rs @@ -0,0 +1,159 @@ +use std::fs; +use std::path::Path; + +use serde::{Deserialize, Serialize}; +use thiserror_ext::AsReport as _; +use tracing::warn; + +use crate::Result; + +/// Minimal `tokenizer.json` projection used to patch `added_tokens` while +/// preserving the rest of the tokenizer definition verbatim. +#[derive(Debug, Deserialize, Serialize)] +struct TokenizerJson { + #[serde(default)] + added_tokens: Vec, + #[serde(flatten)] + extra: serde_json::Map, +} + +/// Minimal `tokenizer_config.json` projection for Hugging Face's +/// `added_tokens_decoder` map. Other config keys are intentionally ignored. +#[derive(Debug, Deserialize)] +struct TokenizerConfigJson { + #[serde(default)] + added_tokens_decoder: std::collections::HashMap, +} + +/// Hugging Face added-token payload. `tokenizer.json` stores `id` inside each +/// item, while `tokenizer_config.json` stores it as the map key. +#[derive(Clone, Debug, Deserialize, Serialize)] +struct AddedTokenConfig { + #[serde(default, skip_serializing_if = "Option::is_none")] + id: Option, + content: String, + #[serde(default)] + single_word: bool, + #[serde(default)] + lstrip: bool, + #[serde(default)] + rstrip: bool, + #[serde(default)] + normalized: bool, + #[serde(default)] + special: bool, +} + +impl AddedTokenConfig { + /// Return this added-token payload in `tokenizer.json` shape by filling the + /// numeric token id that came from `added_tokens_decoder`'s string key. + fn with_id(mut self, id: u32) -> Self { + self.id = Some(id); + self + } +} + +/// Read `tokenizer.json`, then merge in extra added tokens from `tokenizer_config.json`. +pub(super) fn load_tokenizer_json_with_extra_tokens(path: &Path) -> Result { + let tokenizer_json = fs::read_to_string(path) + .map_err(|error| tokenizer_error!("failed to read {}: {}", path.display(), error))?; + let mut tokenizer_json: TokenizerJson = serde_json::from_str(&tokenizer_json) + .map_err(|error| tokenizer_error!("failed to parse {}: {}", path.display(), error))?; + + if let Some(parent) = path.parent() { + let config_path = parent.join("tokenizer_config.json"); + if config_path.exists() { + match load_tokenizer_config_json(&config_path) { + Ok(config_json) => merge_added_tokens_from_config(&mut tokenizer_json, config_json), + Err(error) => { + warn!( + path = %config_path.display(), + error = %error.as_report(), + "failed to load tokenizer_config.json; skipping extra added tokens" + ); + } + } + } + } + + serde_json::to_value(tokenizer_json) + .map_err(|error| tokenizer_error!("failed to serialize tokenizer json: {}", error)) +} + +/// Read and parse a sibling `tokenizer_config.json`. +fn load_tokenizer_config_json(path: &Path) -> Result { + let text = fs::read_to_string(path) + .map_err(|error| tokenizer_error!("failed to read {}: {}", path.display(), error))?; + serde_json::from_str(&text) + .map_err(|error| tokenizer_error!("failed to parse {}: {}", path.display(), error)) +} + +/// Merge added_tokens in `tokenizer.json` and `tokenizer_config.json`. +fn merge_added_tokens_from_config( + tokenizer_json: &mut TokenizerJson, + config_json: TokenizerConfigJson, +) { + use std::collections::HashSet; + + let mut existing_ids: HashSet = + tokenizer_json.added_tokens.iter().filter_map(|token| token.id).collect(); + + let mut extra_tokens = Vec::with_capacity(config_json.added_tokens_decoder.len()); + for (id_str, token) in config_json.added_tokens_decoder { + let id = match id_str.parse::() { + Ok(id) => id, + Err(_) => continue, + }; + extra_tokens.push((id, token)); + } + extra_tokens.sort_unstable_by_key(|(id, _)| *id); + + for (id, token) in extra_tokens { + if existing_ids.contains(&id) { + continue; + } + + // Convert from decoder format to added_tokens array format by adding the "id" field. + tokenizer_json.added_tokens.push(token.with_id(id)); + existing_ids.insert(id); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn merge_added_tokens_from_config_preserves_unmodeled_fields() { + let mut tokenizer_json: TokenizerJson = serde_json::from_value(serde_json::json!({ + "version": "1.0", + "added_tokens": [ + {"id": 0, "content": "", "special": true} + ], + "model": {"type": "WordLevel"} + })) + .expect("parse tokenizer json"); + + let config_json: TokenizerConfigJson = serde_json::from_value(serde_json::json!({ + "chat_template": "{{ messages }}", + "added_tokens_decoder": { + "1": { + "content": "<|image_pad|>", + "special": true, + "normalized": false + } + } + })) + .expect("parse tokenizer config"); + + merge_added_tokens_from_config(&mut tokenizer_json, config_json); + let merged = serde_json::to_value(tokenizer_json).expect("serialize tokenizer json"); + + assert_eq!(merged["version"], "1.0"); + assert_eq!(merged["model"]["type"], "WordLevel"); + assert_eq!(merged["added_tokens"][1]["id"], 1); + assert_eq!(merged["added_tokens"][1]["content"], "<|image_pad|>"); + assert_eq!(merged["added_tokens"][1]["special"], true); + assert_eq!(merged["added_tokens"][1]["normalized"], false); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/tokenizer/src/incremental.rs b/asian_ai/performance/vLLM/rust/src/tokenizer/src/incremental.rs new file mode 100644 index 0000000000000000000000000000000000000000..762821becb4514263e87f8be165babf78e603c85 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/tokenizer/src/incremental.rs @@ -0,0 +1,489 @@ +use std::mem::take; + +use crate::{Result, Tokenizer}; + +/// Stateful incremental decoder that emits text chunks one token at a time. +pub trait IncrementalDecoder: Send { + /// Push one generated token and return how many new string bytes were + /// added. + fn push_token(&mut self, token_id: u32) -> Result; + + /// Consume any text which is currently ready. + fn next_chunk(&mut self) -> Option; + + /// Flush any remaining buffered text that has not yet been emitted. + /// + /// Called after the final generated token to force out buffered/incomplete + /// fragments. + fn flush(&mut self, truncate_output_to: Option) -> Result<(Option, String)>; + + /// Return cumulative decoded text so far. + fn output(&self) -> &str; +} + +/// [`IncrementalDecoder`] built on [`Tokenizer::decode()`] with prefix-diffing. +/// +/// This is the same sliding-window algorithm used by `tokenizers::DecodeStream` +pub(crate) struct DecodeStream<'a, T: Tokenizer + ?Sized> { + tokenizer: &'a T, + skip_special_tokens: bool, + min_bytes_to_buffer: usize, + // mutated state + ids: Vec, + prefix: String, + prefix_index: usize, + prefix_seeded: bool, + cumulative_output: String, + output_index: usize, +} + +impl<'a, T: Tokenizer + ?Sized> DecodeStream<'a, T> { + pub(crate) fn new( + tokenizer: &'a T, + prompt_token_ids: &[u32], + skip_special_tokens: bool, + min_bytes_to_buffer: usize, + ) -> Self { + Self { + tokenizer, + skip_special_tokens, + min_bytes_to_buffer, + ids: prompt_token_ids.to_vec(), + prefix: String::new(), + prefix_index: 0, + prefix_seeded: prompt_token_ids.is_empty(), + cumulative_output: String::new(), + output_index: 0, + } + } +} + +/// Try a short tail suffix first (covers a CJK glyph straddling 1-2 token +/// boundaries); beyond 6 tokens the fallback full-prompt decode is no worse +/// than baseline so widening the sweep just adds overhead. +const SAFE_SUFFIX_MIN: usize = 4; +const SAFE_SUFFIX_MAX: usize = 6; + +impl DecodeStream<'_, T> { + /// Decode prompt-only context for prefix seeding. + /// + /// Prompt ids may come from the model vocabulary rather than the local + /// tokenizer vocabulary. For this seeding path, ids that cannot be mapped + /// back to raw token text are dropped before retrying strict decode. This + /// tolerance is intentionally limited to prompt context; generated ids are + /// decoded later through the normal strict path. + fn decode_prompt_context(&self, ids: &[u32]) -> Result<(String, Vec)> { + match self.tokenizer.decode(ids, self.skip_special_tokens) { + Ok(decoded) => Ok((decoded, ids.to_vec())), + Err(error) => { + let filtered = ids + .iter() + .copied() + .filter(|&id| self.tokenizer.id_to_token(id).is_some()) + .collect::>(); + if filtered.len() == ids.len() { + return Err(error); + } + self.tokenizer + .decode(&filtered, self.skip_special_tokens) + .map(|decoded| (decoded, filtered)) + } + } + } + + /// Seed `self.prefix` from the shortest trailing prompt suffix whose + /// filtered context is still long enough and whose decoded text has no + /// U+FFFD. A clean decode means the suffix starts and ends at valid + /// UTF-8/token boundaries, so priming from it is equivalent to priming from + /// the full prompt. + fn seed_prefix(&mut self) -> Result<()> { + let prompt_len = self.ids.len(); + if prompt_len > SAFE_SUFFIX_MIN { + let max_try = SAFE_SUFFIX_MAX.min(prompt_len - 1); + for suffix_len in SAFE_SUFFIX_MIN..=max_try { + let start = prompt_len - suffix_len; + let (decoded, context_ids) = self.decode_prompt_context(&self.ids[start..])?; + if !decoded.contains('\u{FFFD}') && context_ids.len() >= SAFE_SUFFIX_MIN { + self.prefix = decoded; + self.ids = context_ids; + self.prefix_index = self.ids.len(); + return Ok(()); + } + } + } + let (decoded, context_ids) = self.decode_prompt_context(&self.ids)?; + self.ids = context_ids; + if !decoded.ends_with('\u{FFFD}') { + self.prefix = decoded; + self.prefix_index = self.ids.len(); + } + Ok(()) + } +} + +impl IncrementalDecoder for DecodeStream<'_, T> { + fn push_token(&mut self, token_id: u32) -> Result { + if !self.prefix_seeded && !self.ids.is_empty() { + self.seed_prefix()?; + self.prefix_seeded = true; + } + + self.ids.push(token_id); + let string = self.tokenizer.decode(&self.ids, self.skip_special_tokens)?; + let prefix_len = self.prefix.len(); + if string.len() <= prefix_len || string.ends_with('\u{FFFD}') { + return Ok(0); + } + // Ensure we split at a utf-8 char boundary. + let new_chunk = &string[string.floor_char_boundary(prefix_len)..]; + self.cumulative_output.push_str(new_chunk); + self.ids.drain(..self.prefix_index); + self.prefix = self.tokenizer.decode(&self.ids, self.skip_special_tokens)?; + self.prefix_index = self.ids.len(); + Ok(new_chunk.len()) + } + + fn next_chunk(&mut self) -> Option { + let cutoff = self.cumulative_output.len().saturating_sub(self.min_bytes_to_buffer); + // Ensure we split at a utf-8 char boundary. + let cutoff = self.cumulative_output.floor_char_boundary(cutoff); + (cutoff > self.output_index).then(|| { + let chunk = self.cumulative_output[self.output_index..cutoff].to_string(); + self.output_index = cutoff; + chunk + }) + } + + fn flush(&mut self, truncate_output_to: Option) -> Result<(Option, String)> { + if !self.ids.is_empty() { + let string = self.tokenizer.decode(&self.ids, self.skip_special_tokens)?; + let prefix_len = self.prefix.len(); + self.ids.clear(); + self.prefix.clear(); + self.prefix_index = 0; + self.prefix_seeded = true; + // Ensure we split at a utf-8 char boundary. + self.cumulative_output + .push_str(&string[string.floor_char_boundary(prefix_len)..]); + } + if let Some(truncate_output_to) = truncate_output_to { + self.cumulative_output.truncate(truncate_output_to); + } + let last_chunk = (self.output_index < self.cumulative_output.len()) + .then(|| self.cumulative_output[self.output_index..].to_string()); + self.output_index = 0; + Ok((last_chunk, take(&mut self.cumulative_output))) + } + + fn output(&self) -> &str { + &self.cumulative_output + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::test_utils::TestTokenizer; + + /// Backend that treats each token ID as a raw byte, producing lossy UTF-8. + #[derive(Debug)] + struct Utf8Backend; + + impl Tokenizer for Utf8Backend { + fn encode(&self, _text: &str, _add_special_tokens: bool) -> Result> { + unreachable!() + } + + fn decode(&self, token_ids: &[u32], _skip_special_tokens: bool) -> Result { + let bytes = token_ids.iter().map(|id| *id as u8).collect::>(); + Ok(String::from_utf8_lossy(&bytes).into_owned()) + } + + fn token_to_id(&self, _token: &str) -> Option { + unreachable!() + } + + fn id_to_token(&self, _id: u32) -> Option { + unreachable!() + } + } + + #[test] + fn holds_incomplete_utf8_until_complete() { + let backend = Utf8Backend; + let mut decoder = backend.create_decode_stream(&[], false, 0); + + // 你 = U+4F60 = 0xE4 0xBD 0xA0 + assert_eq!(decoder.push_token(0xe4).unwrap(), 0); + assert_eq!(decoder.push_token(0xbd).unwrap(), 0); + assert_eq!(decoder.push_token(0xa0).unwrap(), 3); // "你" is 3 bytes + assert_eq!(decoder.output(), "你"); + } + + #[test] + fn emits_ascii_immediately() { + let backend = Utf8Backend; + let mut decoder = backend.create_decode_stream(&[], false, 0); + + assert_eq!(decoder.push_token(b'o' as u32).unwrap(), 1); + assert_eq!(decoder.push_token(b'k' as u32).unwrap(), 1); + assert_eq!(decoder.output(), "ok"); + } + + #[test] + fn flush_returns_none_when_fully_consumed() { + let backend = Utf8Backend; + let mut decoder = backend.create_decode_stream(&[], false, 0); + + assert_eq!(decoder.push_token(b'o' as u32).unwrap(), 1); + assert_eq!(decoder.next_chunk().as_deref(), Some("o")); + assert_eq!(decoder.push_token(b'k' as u32).unwrap(), 1); + assert_eq!(decoder.next_chunk().as_deref(), Some("k")); + // All text already consumed via next_chunk + let (last_chunk, full_text) = decoder.flush(None).unwrap(); + assert_eq!(last_chunk, None); + assert_eq!(full_text, "ok"); + } + + #[test] + fn flush_emits_buffered_incomplete_utf8() { + let backend = Utf8Backend; + let mut decoder = backend.create_decode_stream(&[], false, 0); + + // Push incomplete multi-byte sequence — step returns 0 bytes. + assert_eq!(decoder.push_token(0xe4).unwrap(), 0); + assert_eq!(decoder.push_token(0xbd).unwrap(), 0); + + // Flush forces out whatever the decoder can produce (lossy replacement). + let (last_chunk, _full_text) = decoder.flush(None).unwrap(); + assert!(last_chunk.is_some()); + } + + /// Backend where token 0 is a special token. + #[derive(Debug)] + struct SpecialTokenBackend; + + impl Tokenizer for SpecialTokenBackend { + fn encode(&self, _text: &str, _add_special_tokens: bool) -> Result> { + unreachable!() + } + + fn decode(&self, token_ids: &[u32], skip_special_tokens: bool) -> Result { + let mut text = String::new(); + for &token_id in token_ids { + match token_id { + 0 if !skip_special_tokens => text.push_str(""), + 0 => {} + 1 => text.push('a'), + _ => {} + } + } + Ok(text) + } + + fn token_to_id(&self, _token: &str) -> Option { + unreachable!() + } + + fn id_to_token(&self, _id: u32) -> Option { + unreachable!() + } + } + + #[test] + fn respects_skip_special_tokens() { + let backend = SpecialTokenBackend; + let mut skip_decoder = backend.create_decode_stream(&[], true, 0); + let mut keep_decoder = backend.create_decode_stream(&[], false, 0); + + assert_eq!(skip_decoder.push_token(0).unwrap(), 0); + assert_eq!(keep_decoder.push_token(0).unwrap(), 9); // "" is 9 bytes + assert_eq!(keep_decoder.output(), ""); + } + + #[test] + fn prompt_tokens_provide_context_without_re_emission() { + let backend = Utf8Backend; + let prompt = &[b'H' as u32, b'i' as u32]; + let mut decoder = backend.create_decode_stream(prompt, false, 0); + + // First generated token should not re-emit "Hi". + let added = decoder.push_token(b'!' as u32).unwrap(); + assert_eq!(added, 1); + assert_eq!(decoder.output(), "!"); + } + + #[test] + fn prompt_context_filters_unknown_ids() { + let tokenizer = TestTokenizer::new(); + assert_eq!(tokenizer.id_to_token(10_000), None); + + let cases: &[(&str, &[u32], u32, &str)] = &[ + ( + "suffix seed", + &[ + b'a' as u32, + b'b' as u32, + b'c' as u32, + 10_000, + b'H' as u32, + b'i' as u32, + ], + b'!' as u32, + "!", + ), + ("all unknown", &[10_000], b'!' as u32, "!"), + ( + "unknown before incomplete utf-8", + &[10_000, 0xe4, 0xbd], + 0xa0, + "你", + ), + ( + "incomplete utf-8 before filtered suffix", + &[0xe4, 0xbd, 10_000, 10_001, 10_002, 10_003, 10_004, 10_005], + 0xa0, + "你", + ), + ]; + + for &(name, prompt, token_id, output) in cases { + let mut decoder = tokenizer.create_decode_stream(prompt, false, 0); + assert_eq!( + decoder.push_token(token_id).unwrap(), + output.len(), + "{name}" + ); + assert_eq!(decoder.output(), output, "{name}"); + } + } + + #[test] + fn generated_unknown_ids_still_return_decode_error() { + let tokenizer = TestTokenizer::new(); + assert_eq!(tokenizer.id_to_token(10_000), None); + + let prompt = &[10_000, b'H' as u32, b'i' as u32]; + let mut decoder = tokenizer.create_decode_stream(prompt, false, 0); + + let error = decoder.push_token(10_000).unwrap_err(); + assert!( + error + .to_string() + .contains("test tokenizer cannot decode unknown token id 10000") + ); + } + + #[test] + fn chunks_concatenate_to_full_text() { + let backend = Utf8Backend; + let mut decoder = backend.create_decode_stream(&[], false, 0); + + let input = b"Hello, world!"; + let mut full = String::new(); + for &byte in input { + decoder.push_token(byte as u32).unwrap(); + if let Some(chunk) = decoder.next_chunk() { + full.push_str(&chunk); + } + } + let (last_chunk, full_text) = decoder.flush(None).unwrap(); + assert_eq!(last_chunk, None); // all consumed via next_chunk + assert_eq!(full, "Hello, world!"); + assert_eq!(full_text, "Hello, world!"); + } + + /// Backend simulating non-monotonic decode where adding a token changes how + /// earlier tokens decode (context-dependent normalization), causing + /// prefix_len to land mid-UTF-8. Reproduces the class of bug from + /// vllm-project/vllm#17448. + #[derive(Debug)] + struct NonMonotonicBackend; + + impl Tokenizer for NonMonotonicBackend { + fn encode(&self, _text: &str, _add_special_tokens: bool) -> Result> { + unreachable!() + } + + fn decode(&self, token_ids: &[u32], _skip_special_tokens: bool) -> Result { + match token_ids { + [1] => Ok("abc".into()), + [1, 2] => Ok("ab".into()), + // Token 3 triggers a normalization change: "ab" becomes emoji + "d". + // prefix_len=3 ("abc") lands inside the 4-byte emoji 🎉. + [1, 2, 3] => Ok("🎉d".into()), // 🎉 is 4 bytes + d = 5 bytes + [2, 3] => Ok("🎉d".into()), // prefix recompute after drain + [3] => Ok("d".into()), // after drain + _ => panic!("unexpected decode: {:?}", token_ids), + } + } + + fn token_to_id(&self, _token: &str) -> Option { + unreachable!() + } + + fn id_to_token(&self, _id: u32) -> Option { + unreachable!() + } + } + + /// Without the char-boundary fix, this panics slicing mid-emoji. + #[test] + fn non_monotonic_decode_does_not_panic() { + let backend = NonMonotonicBackend; + let mut decoder = backend.create_decode_stream(&[], false, 0); + + // Token 1: "abc", prefix="abc" + assert_eq!(decoder.push_token(1).unwrap(), 3); + // Token 2: "ab" (shorter), no emit + assert_eq!(decoder.push_token(2).unwrap(), 0); + // Token 3: "🎉d" — prefix_len=3 is mid-emoji. Without fix this panics. + let added = decoder.push_token(3).unwrap(); + assert!(added > 0); + } + + #[test] + fn next_chunk_with_hold_back() { + let backend = Utf8Backend; + // hold_back_bytes: 3 means we buffer the last 3 bytes + let mut decoder = backend.create_decode_stream(&[], false, 3); + + let input = b"Hello!"; + let mut chunks = String::new(); + for &byte in input { + decoder.push_token(byte as u32).unwrap(); + if let Some(chunk) = decoder.next_chunk() { + chunks.push_str(&chunk); + } + } + // With hold_back_bytes=3, last 3 bytes ("lo!") are held back + assert_eq!(chunks, "Hel"); + // Flush returns the rest + let (last_chunk, full_text) = decoder.flush(None).unwrap(); + assert_eq!(last_chunk.as_deref(), Some("lo!")); + assert_eq!(full_text, "Hello!"); + } + + #[test] + fn next_chunk_cutoff_respects_char_boundary() { + // Regression: next_chunk's cutoff (len - min_bytes_to_buffer) must be + // aligned to a UTF-8 char boundary like push_token/flush; otherwise + // streaming multi-byte output (CJK/emoji) with a hold-back buffer (set + // by a stop string) panics slicing cumulative_output mid-character. + let backend = Utf8Backend; + let mut decoder = backend.create_decode_stream(&[], false, 2); + let mut out = String::new(); + for byte in "你好A".bytes() { + decoder.push_token(u32::from(byte)).unwrap(); + if let Some(chunk) = decoder.next_chunk() { + out.push_str(&chunk); + } + } + let (last_chunk, full_text) = decoder.flush(None).unwrap(); + if let Some(chunk) = last_chunk { + out.push_str(&chunk); + } + assert_eq!(full_text, "你好A"); + assert_eq!(out, "你好A"); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/tokenizer/src/lib.rs b/asian_ai/performance/vLLM/rust/src/tokenizer/src/lib.rs new file mode 100644 index 0000000000000000000000000000000000000000..908c248c85db3e8c0610446a9a2e6f97ad16ceeb --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/tokenizer/src/lib.rs @@ -0,0 +1,66 @@ +use std::sync::Arc; + +use crate::incremental::DecodeStream; + +mod byte_level_decode; +#[macro_use] +mod error; +mod hf; +mod incremental; +mod tekken; +#[cfg(any(test, feature = "test-utils"))] +pub mod test_utils; +mod tiktoken; + +pub use error::{Result, TokenizerError}; +pub use hf::HuggingFaceTokenizer; +pub use incremental::IncrementalDecoder; +pub use tekken::TekkenTokenizer; +pub use tiktoken::TiktokenTokenizer; + +pub trait Tokenizer: Send + Sync { + /// Encode one prompt string into token IDs. + fn encode(&self, text: &str, add_special_tokens: bool) -> Result>; + + /// Decode one token sequence into text. + fn decode(&self, token_ids: &[u32], skip_special_tokens: bool) -> Result; + + /// Convert one token string into a token ID, returning `None` if the token + /// is not in the tokenizer vocabulary. + fn token_to_id(&self, token: &str) -> Option; + + /// Convert one token ID into the tokenizer's raw token string. + fn id_to_token(&self, id: u32) -> Option; + + /// Return the vocabulary size. Backends that cannot report it fall back to + /// `usize::MAX`, an effectively unbounded value used only by test stubs. + fn vocab_size(&self) -> usize { + usize::MAX + } + + /// Return whether the given token ID is special. + fn is_special_id(&self, _token_id: u32) -> bool { + false + } + + /// Create a stateful incremental decoder primed with the given prompt + /// tokens. + /// + /// The prompt tokens provide left context for the first generated token; + /// the decoder does not re-emit prompt text. + fn create_decode_stream( + &self, + prompt_token_ids: &[u32], + skip_special_tokens: bool, + min_bytes_to_buffer: usize, + ) -> Box { + Box::new(DecodeStream::new( + self, + prompt_token_ids, + skip_special_tokens, + min_bytes_to_buffer, + )) + } +} + +pub type DynTokenizer = Arc; diff --git a/asian_ai/performance/vLLM/rust/src/tokenizer/src/tekken.rs b/asian_ai/performance/vLLM/rust/src/tokenizer/src/tekken.rs new file mode 100644 index 0000000000000000000000000000000000000000..92db4cc9c3124468636741bf69a1ad4c38c14d7b --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/tokenizer/src/tekken.rs @@ -0,0 +1,66 @@ +use std::path::Path; + +use tekken::Tekkenizer; +use tracing::info; + +use crate::{Result, Tokenizer}; + +/// Mistral Tekken tokenizer from a `tekken.json` file. +pub struct TekkenTokenizer { + inner: Tekkenizer, +} + +impl TekkenTokenizer { + /// Load a Mistral Tekken tokenizer from a `tekken.json` file. + pub fn new(path: &Path) -> Result { + info!(path = %path.display(), "loading tokenizer with Mistral Tekken"); + + let inner = Tekkenizer::from_file(path).map_err(|error| { + tokenizer_error!( + "failed to load tekken tokenizer from {}: {error}", + path.display() + ) + })?; + Ok(Self { inner }) + } +} + +impl Tokenizer for TekkenTokenizer { + fn encode(&self, text: &str, add_special_tokens: bool) -> Result> { + self.inner + .encode(text, add_special_tokens, false) + .map_err(|error| tokenizer_error!("encoding failed: {error}")) + } + + fn decode(&self, token_ids: &[u32], skip_special_tokens: bool) -> Result { + let policy = if skip_special_tokens { + tekken::SpecialTokenPolicy::Ignore + } else { + tekken::SpecialTokenPolicy::Keep + }; + self.inner + .decode(token_ids, policy) + .map_err(|error| tokenizer_error!("decoding failed: {error}")) + } + + fn token_to_id(&self, token: &str) -> Option { + // tekken-rs exposes `get_control_token` for special tokens. Try that first, + // then fall back to encoding. + self.inner.get_control_token(token).ok().or_else(|| { + let ids = self.inner.encode(token, false, false).ok()?; + if ids.len() == 1 { Some(ids[0]) } else { None } + }) + } + + fn id_to_token(&self, id: u32) -> Option { + self.inner.id_to_piece(id).ok() + } + + fn vocab_size(&self) -> usize { + self.inner.vocab_size() + } + + fn is_special_id(&self, token_id: u32) -> bool { + self.inner.is_special_token(token_id) + } +} diff --git a/asian_ai/performance/vLLM/rust/src/tokenizer/src/test_utils.rs b/asian_ai/performance/vLLM/rust/src/tokenizer/src/test_utils.rs new file mode 100644 index 0000000000000000000000000000000000000000..57e65c0a29151415b31dbfdb3e49634f71682fb7 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/tokenizer/src/test_utils.rs @@ -0,0 +1,434 @@ +use std::collections::BTreeMap; + +use crate::{Result, Tokenizer, TokenizerError}; + +const FIRST_CONFIGURED_TOKEN_ID: u32 = 256; + +/// Whether a configured test token should be treated as special. +/// +/// Special tokens are skipped by [`Tokenizer::decode`] when +/// `skip_special_tokens` is set. Regular configured tokens are always emitted. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TestTokenKind { + /// Token is skipped when `skip_special_tokens = true`. + Special, + /// Token is emitted regardless of `skip_special_tokens`. + Regular, +} + +impl TestTokenKind { + fn is_special(self) -> bool { + matches!(self, Self::Special) + } +} + +/// Decode behavior for token ids that are neither configured tokens nor byte ids. +/// +/// The default is [`UnknownDecode::Error`] so tests notice missing tokenizer +/// fixtures instead of silently accepting impossible ids. Individual tests can +/// opt into empty or replacement output when they are explicitly modeling a +/// lenient detokenization path. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum UnknownDecode { + /// Return a tokenizer error on the first unknown id. + Error, + /// Drop unknown ids from decoded output. + Empty, + /// Emit U+FFFD for each unknown id. + Replacement, +} + +#[derive(Debug, Clone)] +struct TestToken { + text: String, + kind: TestTokenKind, +} + +/// Configurable tokenizer for Rust frontend tests. +/// +/// `TestTokenizer` is intentionally small, but its methods obey the same basic +/// contract as production tokenizers: +/// +/// - ordinary text encodes as UTF-8 byte ids; +/// - configured token ids start at 256, leaving `0..=255` for byte fallback; +/// - configured token ids and token text are unique; +/// - configured tokens are matched before ordinary bytes, using longest-prefix matching so +/// multi-character markers such as `` work naturally; +/// - `token_to_id` and `id_to_token` are consistent for configured tokens; +/// - `decode` is strict by default for ids outside the byte range and the configured token table; +/// - `vocab_size` is an exclusive upper bound covering byte ids and configured token ids unless a +/// test sets it explicitly. +/// +/// Prefer this helper over ad-hoc fake tokenizers for tests that rely on +/// tokenizer semantics. Keep dedicated tiny fakes for error injection or for +/// tests that deliberately need a degenerate tokenizer. +#[derive(Debug, Clone)] +pub struct TestTokenizer { + token_to_id: BTreeMap, + id_to_token: BTreeMap, + unknown_decode: UnknownDecode, + vocab_size: Option, + bos_token_id: Option, +} + +impl Default for TestTokenizer { + fn default() -> Self { + Self::new() + } +} + +impl TestTokenizer { + /// Create a byte-level test tokenizer with strict unknown-id decode. + pub fn new() -> Self { + Self { + token_to_id: BTreeMap::new(), + id_to_token: BTreeMap::new(), + unknown_decode: UnknownDecode::Error, + vocab_size: None, + bos_token_id: None, + } + } + + /// Add a configured token and return the updated tokenizer. + /// + /// Configured tokens must use ids outside the byte range and may be marked + /// special or regular. + pub fn with_token(mut self, token: impl Into, id: u32, kind: TestTokenKind) -> Self { + self.insert_token(token, id, kind); + self + } + + /// Add a special configured token and return the updated tokenizer. + pub fn with_special_token(self, token: impl Into, id: u32) -> Self { + self.with_token(token, id, TestTokenKind::Special) + } + + /// Add a regular configured token and return the updated tokenizer. + pub fn with_regular_token(self, token: impl Into, id: u32) -> Self { + self.with_token(token, id, TestTokenKind::Regular) + } + + /// Add a special BOS token inserted by `encode(..., true)`. + /// + /// This also registers the token in the normal token/id maps so + /// `token_to_id`, `id_to_token`, `decode`, and `is_special_id` stay + /// consistent for the inserted id. + pub fn with_bos_token(mut self, token: impl Into, id: u32) -> Self { + self.insert_token(token, id, TestTokenKind::Special); + self.bos_token_id = Some(id); + self + } + + /// Set decode behavior for unknown non-byte ids. + pub fn with_unknown_decode(mut self, behavior: UnknownDecode) -> Self { + self.unknown_decode = behavior; + self + } + + /// Set an explicit vocabulary size. + /// + /// Use this when a test needs a model-like vocabulary bound that differs + /// from the highest configured token id plus one. + pub fn with_vocab_size(mut self, vocab_size: usize) -> Self { + self.vocab_size = Some(vocab_size); + self + } + + fn insert_token(&mut self, token: impl Into, id: u32, kind: TestTokenKind) { + let token = token.into(); + assert!( + !token.is_empty(), + "configured test token text must be non-empty" + ); + assert!( + id >= FIRST_CONFIGURED_TOKEN_ID, + "configured test token id {id} overlaps byte fallback range 0..=255" + ); + assert!( + token.len() > 1, + "configured test token text {token:?} overlaps byte fallback token text" + ); + if self.token_to_id.insert(token.clone(), id).is_some() { + panic!("configured test token text {token:?} was registered more than once"); + } + if self.id_to_token.insert(id, TestToken { text: token, kind }).is_some() { + panic!("configured test token id {id} was registered more than once"); + } + } + + fn byte_to_token(id: u32) -> Option { + u8::try_from(id).ok().map(|byte| String::from_utf8_lossy(&[byte]).into_owned()) + } + + fn flush_bytes(bytes: &mut Vec, output: &mut String) { + if !bytes.is_empty() { + output.push_str(&String::from_utf8_lossy(bytes)); + bytes.clear(); + } + } + + fn configured_token_prefix(&self, text: &str) -> Option<(&str, u32)> { + self.token_to_id + .iter() + .filter_map(|(token, &id)| text.starts_with(token).then_some((token.as_str(), id))) + .max_by_key(|(token, _)| token.len()) + } + + fn inferred_vocab_size(&self) -> usize { + let max_configured = + self.id_to_token.last_key_value().map(|(&id, _)| id as usize + 1).unwrap_or(0); + 256.max(max_configured) + } +} + +impl Tokenizer for TestTokenizer { + fn encode(&self, text: &str, add_special_tokens: bool) -> Result> { + let mut ids = Vec::new(); + if add_special_tokens && let Some(bos_token_id) = self.bos_token_id { + ids.push(bos_token_id); + } + + let mut rest = text; + while !rest.is_empty() { + if let Some((token, id)) = self.configured_token_prefix(rest) { + ids.push(id); + rest = &rest[token.len()..]; + continue; + } + + let ch = rest.chars().next().expect("rest is not empty"); + let mut buf = [0_u8; 4]; + ids.extend(ch.encode_utf8(&mut buf).bytes().map(u32::from)); + rest = &rest[ch.len_utf8()..]; + } + + Ok(ids) + } + + fn decode(&self, token_ids: &[u32], skip_special_tokens: bool) -> Result { + let mut output = String::new(); + let mut pending_bytes = Vec::new(); + for &id in token_ids { + if let Some(token) = self.id_to_token.get(&id) { + Self::flush_bytes(&mut pending_bytes, &mut output); + if !(skip_special_tokens && token.kind.is_special()) { + output.push_str(&token.text); + } + } else if let Ok(byte) = u8::try_from(id) { + pending_bytes.push(byte); + } else { + Self::flush_bytes(&mut pending_bytes, &mut output); + match self.unknown_decode { + UnknownDecode::Error => { + return Err(TokenizerError(format!( + "test tokenizer cannot decode unknown token id {id}" + ))); + } + UnknownDecode::Empty => {} + UnknownDecode::Replacement => output.push('\u{FFFD}'), + } + } + } + Self::flush_bytes(&mut pending_bytes, &mut output); + Ok(output) + } + + fn token_to_id(&self, token: &str) -> Option { + self.token_to_id.get(token).copied().or_else(|| { + let bytes = token.as_bytes(); + (bytes.len() == 1).then(|| u32::from(bytes[0])) + }) + } + + fn id_to_token(&self, id: u32) -> Option { + self.id_to_token + .get(&id) + .map(|token| token.text.clone()) + .or_else(|| Self::byte_to_token(id)) + } + + fn vocab_size(&self) -> usize { + self.vocab_size.unwrap_or_else(|| self.inferred_vocab_size()) + } + + fn is_special_id(&self, token_id: u32) -> bool { + self.id_to_token.get(&token_id).is_some_and(|token| token.kind.is_special()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn byte_text_roundtrips_and_reports_byte_ids() { + let tokenizer = TestTokenizer::new(); + + let ids = tokenizer.encode("hi", false).unwrap(); + assert_eq!(ids, vec![b'h' as u32, b'i' as u32]); + assert_eq!(tokenizer.decode(&ids, false).unwrap(), "hi"); + assert_eq!(tokenizer.token_to_id("h"), Some(b'h' as u32)); + assert_eq!(tokenizer.id_to_token(b'h' as u32).as_deref(), Some("h")); + assert_eq!(tokenizer.vocab_size(), 256); + } + + #[test] + fn configured_tokens_use_longest_prefix_matching() { + let tokenizer = TestTokenizer::new() + .with_regular_token("", 999) + .with_regular_token("", 1000); + + assert_eq!( + tokenizer.encode("ab", false).unwrap(), + vec![b'a' as u32, 1000, b'b' as u32,] + ); + assert_eq!( + tokenizer.decode(&[b'a' as u32, 1000, b'b' as u32], false).unwrap(), + "ab" + ); + assert_eq!(tokenizer.token_to_id(""), Some(999)); + assert_eq!( + tokenizer.id_to_token(1000).as_deref(), + Some("") + ); + assert_eq!(tokenizer.vocab_size(), 1001); + } + + #[test] + fn non_ascii_text_roundtrips_through_buffered_byte_decode() { + let tokenizer = TestTokenizer::new(); + let text = "你好, café, 🚀"; + + let ids = tokenizer.encode(text, false).unwrap(); + assert_eq!( + ids, + text.as_bytes().iter().copied().map(u32::from).collect::>() + ); + assert_eq!(tokenizer.decode(&ids, false).unwrap(), text); + } + + #[test] + fn buffered_byte_decode_flushes_around_configured_tokens() { + let tokenizer = TestTokenizer::new() + .with_regular_token("", 999) + .with_special_token("", 1000); + + assert_eq!( + tokenizer.encode("你好🚀", false).unwrap(), + vec![228, 189, 160, 999, 229, 165, 189, 1000, 240, 159, 154, 128] + ); + assert_eq!( + tokenizer + .decode( + &[228, 189, 160, 999, 229, 165, 189, 1000, 240, 159, 154, 128], + false + ) + .unwrap(), + "你好🚀" + ); + assert_eq!( + tokenizer + .decode( + &[228, 189, 160, 999, 229, 165, 189, 1000, 240, 159, 154, 128], + true + ) + .unwrap(), + "你好🚀" + ); + } + + #[test] + fn invalid_utf8_bytes_decode_lossily_as_a_sequence() { + let tokenizer = TestTokenizer::new(); + + assert_eq!(tokenizer.decode(&[0xE4, 0xBD], false).unwrap(), "\u{FFFD}"); + assert_eq!( + tokenizer.decode(&[0xFF, b'a' as u32], false).unwrap(), + "\u{FFFD}a" + ); + } + + #[test] + fn special_tokens_respect_skip_special_tokens() { + let tokenizer = TestTokenizer::new() + .with_bos_token("", 256) + .with_special_token("", 0xF001) + .with_regular_token("", 0xF002); + + assert_eq!( + tokenizer.encode("x", true).unwrap(), + vec![256, 0xF001, b'x' as u32, 0xF002,] + ); + assert_eq!( + tokenizer.decode(&[256, 0xF001, b'x' as u32, 0xF002], false).unwrap(), + "x" + ); + assert_eq!( + tokenizer.decode(&[256, 0xF001, b'x' as u32, 0xF002], true).unwrap(), + "x" + ); + assert!(tokenizer.is_special_id(0xF001)); + assert!(!tokenizer.is_special_id(0xF002)); + } + + #[test] + #[should_panic(expected = "configured test token id 255 overlaps byte fallback range 0..=255")] + fn configured_token_id_must_stay_outside_byte_range() { + let _ = TestTokenizer::new().with_regular_token("", 255); + } + + #[test] + #[should_panic(expected = "configured test token text \"a\" overlaps byte fallback token text")] + fn configured_token_text_must_not_shadow_byte_tokens() { + let _ = TestTokenizer::new().with_regular_token("a", 256); + } + + #[test] + #[should_panic( + expected = "configured test token text \"\" was registered more than once" + )] + fn configured_token_text_must_be_unique() { + let _ = TestTokenizer::new() + .with_regular_token("", 256) + .with_regular_token("", 257); + } + + #[test] + #[should_panic(expected = "configured test token id 256 was registered more than once")] + fn configured_token_id_must_be_unique() { + let _ = TestTokenizer::new() + .with_regular_token("", 256) + .with_regular_token("", 256); + } + + #[test] + fn unknown_decode_is_strict_by_default_and_configurable() { + let strict = TestTokenizer::new(); + assert!(strict.decode(&[300], false).is_err()); + assert_eq!( + TestTokenizer::new() + .with_unknown_decode(UnknownDecode::Empty) + .decode(&[b'a' as u32, 300, b'b' as u32], false) + .unwrap(), + "ab" + ); + assert_eq!( + TestTokenizer::new() + .with_unknown_decode(UnknownDecode::Replacement) + .decode(&[300], false) + .unwrap(), + "\u{FFFD}" + ); + assert_eq!(strict.id_to_token(300), None); + } + + #[test] + fn explicit_vocab_size_overrides_inferred_bound() { + let tokenizer = TestTokenizer::new() + .with_regular_token("", 10_000) + .with_vocab_size(20_000); + + assert_eq!(tokenizer.vocab_size(), 20_000); + assert_eq!(tokenizer.id_to_token(10_000).as_deref(), Some("")); + } +} diff --git a/asian_ai/performance/vLLM/rust/src/tokenizer/src/tiktoken.rs b/asian_ai/performance/vLLM/rust/src/tokenizer/src/tiktoken.rs new file mode 100644 index 0000000000000000000000000000000000000000..68a3d7b13dff2e07d84cc4e7bb585e966f7b9f54 --- /dev/null +++ b/asian_ai/performance/vLLM/rust/src/tokenizer/src/tiktoken.rs @@ -0,0 +1,1031 @@ +use std::collections::HashSet; +use std::path::Path; +use std::sync::Mutex; + +use base64::Engine as _; +use rustc_hash::{FxHashMap, FxHashSet}; +use serde::Deserialize; +use thiserror_ext::AsReport as _; +use tracing::{info, warn}; + +use crate::{Result, Tokenizer}; + +/// Default regex pattern used when loading tiktoken from a BPE file. This is +/// the same `cl100k_base` pattern that HuggingFace transformers uses as its +/// default in `TikTokenConverter`. +const CL100K_BASE_PATTERN: &str = r"(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+"; + +/// Kimi BPE pattern from `moonshotai/Kimi-K2-Instruct/tokenization_kimi.py`. +const KIMI_PATTERN: &str = r"[\p{Han}]+|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]*[\p{Ll}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]+[\p{Ll}\p{Lm}\p{Lo}\p{M}&&[^\p{Han}]]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+"; + +/// Fallback number of reserved special-token slots to assume when the model's +/// `config.json` is not available (so we cannot read `vocab_size` directly). +/// +/// 256 is the value used by Kimi K2 / K2.5 (`tokenization_kimi.py`'s +/// `num_reserved_special_tokens`) and by Llama 3, and it appears to be the most +/// common convention among modern tiktoken-based HF tokenizers. When +/// `config.json` *is* present we honour the model's actual `vocab_size` instead +/// of this fallback — see `Self::new`. +const FALLBACK_NUM_RESERVED_SPECIAL_TOKENS: u32 = 256; +const DISABLE_RIPTOKEN_ENV: &str = "VLLM_RS_DISABLE_RIPTOKEN"; + +/// Parsed entry from `tokenizer_config.json`'s `added_tokens_decoder`. +#[derive(Debug, Clone, Deserialize)] +struct AddedToken { + content: String, + /// HuggingFace `added_tokens_decoder` entries can be marked `"special": + /// true|false`. Special tokens are dropped from output when `decode` is + /// called with `skip_special_tokens = true`. Defaults to `false` when + /// the field is omitted, matching HuggingFace's `AddedToken` default — + /// so only tokens explicitly marked special are stripped during normal + /// decode (where `skip_special_tokens` itself defaults to true). + #[serde(default)] + special: bool, +} + +/// Minimal subset of `tokenizer_config.json` needed by the tiktoken loader. +#[derive(Debug, Default, Deserialize)] +#[serde(default)] +struct TiktokenTokenizerConfig { + /// Format: + /// `{ "added_tokens_decoder": { "163584": { "content": "[BOS]", "special": + /// true }, ... } }` + #[serde(default)] + added_tokens_decoder: FxHashMap, +} + +/// Minimal subset of model `config.json` needed by the tiktoken loader. +#[derive(Debug, Default, Deserialize)] +#[serde(default)] +struct TiktokenModelConfig { + model_type: Option, + vocab_size: Option, + text_config: Option>, +} + +impl TiktokenModelConfig { + /// Read `model_type` from a model `config.json` value, falling back to a + /// single-level nested `text_config.model_type` for composite (e.g. + /// multimodal) configs that keep text metadata under a `text_config` + /// object. + fn effective_model_type(&self) -> Option<&str> { + self.model_type + .as_deref() + .or_else(|| self.text_config.as_deref()?.effective_model_type()) + } + + /// Read `vocab_size` from a model `config.json` value, falling back to a + /// single-level nested `text_config.vocab_size` for composite (e.g. + /// multimodal) configs that keep text metadata under a `text_config` + /// object — matching the same shape `ModelConfig` parses. + fn effective_vocab_size(&self) -> Option { + self.vocab_size.or_else(|| self.text_config.as_deref()?.effective_vocab_size()) + } +} + +/// Tiktoken tokenizer from `tiktoken.model` or `*.tiktoken` BPE files. +pub struct TiktokenTokenizer { + backend: Backend, + metadata: TokenMetadata, +} + +enum Backend { + Riptoken(RiptokenBackend), + TiktokenRs(TiktokenRsBackend), +} + +struct RiptokenBackend { + inner: Box, + allowed_special_tokens: Vec, +} + +struct TiktokenRsBackend { + inner: Box, + /// Reverse map for special / added token strings populated from the + /// reserved range. This lets `token_to_id` answer special-token lookups + /// directly without round-tripping through `tiktoken-rs`'s encoder, + /// which can panic for unknown special-looking strings. + special_token_ids_by_text: FxHashMap, + /// Set of out-of-vocab token IDs we have already warned about. The + /// reserved-slot population in the constructor should keep this empty + /// under normal operation; it only fills up if a model emits ids at or + /// above `vocab_upper_bound` (e.g. an engine sampling bug). We dedupe + /// so streaming decode (which calls `decode` repeatedly on the same prefix) + /// does not spam. + warned_unknown_ids: Mutex>, +} + +struct TokenMetadata { + /// Number of regular BPE tokens. Token ids in `[0, num_base_tokens)` are + /// BPE tokens that always decode to text; ids in `[num_base_tokens, + /// vocab_upper_bound)` live in the special-token slots and are subject + /// to `skip_special_tokens` filtering. + num_base_tokens: u32, + /// Exclusive upper bound on token IDs that `inner` is guaranteed to know + /// how to decode. + /// + /// The constructor registers every id in `[num_base_tokens, + /// vocab_upper_bound)` with the inner `CoreBPE` as a (named or + /// `<|reserved_token_{id}|>`) special token, and the BPE + /// encoder densely covers `[0, num_base_tokens)`. So any id below this + /// bound is in one of the inner `CoreBPE`'s decoder maps and + /// `_decode_native_and_split` will not panic on it. `decode` filters + /// out ids at or above this bound to keep that guarantee. + vocab_upper_bound: u32, + /// Ids in `[num_base_tokens, vocab_upper_bound)` whose + /// `added_tokens_decoder` entry was explicitly marked `"special": + /// false` — i.e. tokens that should still appear in output + /// even when `skip_special_tokens = true`. For Kimi K2 / K2.5 this + /// typically holds the tool-call markers and `` / ``. + /// Reserved-slot placeholders are not in this set (they default to + /// special and get skipped). + non_special_added_ids: FxHashSet, + /// Raw token string by token id. Base BPE tokens are represented with + /// lossy UTF-8, matching decode behavior for byte sequences that are not + /// valid UTF-8 on their own. + token_by_id: FxHashMap, +} + +impl TokenMetadata { + fn filter_special_tokens(&self, token_ids: &[u32]) -> Vec { + token_ids + .iter() + .copied() + .filter(|&id| { + id < self.num_base_tokens + || id >= self.vocab_upper_bound + || self.non_special_added_ids.contains(&id) + }) + .collect() + } + + fn is_special_id(&self, token_id: u32) -> bool { + token_id >= self.num_base_tokens + && token_id < self.vocab_upper_bound + && !self.non_special_added_ids.contains(&token_id) + } + + fn id_to_token(&self, token_id: u32) -> Option { + self.token_by_id.get(&token_id).cloned() + } +} + +impl RiptokenBackend { + fn encode(&self, text: &str) -> Vec { + // TODO: avoid collecting `allowed_special` every time this method is called. + let allowed_special: HashSet<&str> = + self.allowed_special_tokens.iter().map(String::as_str).collect(); + self.inner.encode(text, &allowed_special) + } + + fn decode(&self, token_ids: &[u32]) -> String { + let bytes = self.inner.decode_bytes(token_ids); + // TODO: use `from_utf8_lossy_owned` once it's stabilized. + String::from_utf8_lossy(&bytes).into_owned() + } + + fn token_to_id(&self, token: &str) -> Option { + self.inner.encode_single_token(token.as_bytes()) + } +} + +impl TiktokenRsBackend { + fn encode(&self, text: &str) -> Vec { + self.inner.encode_with_special_tokens(text) + } + + fn decode(&self, token_ids: &[u32], metadata: &TokenMetadata) -> String { + let safe_ids: Vec = token_ids + .iter() + .copied() + .filter(|&id| { + if id >= metadata.vocab_upper_bound { + self.warn_unknown_id(id); + return false; + } + true + }) + .collect(); + let bytes: Vec = self.inner._decode_native_and_split(safe_ids).flatten().collect(); + // TODO: use `from_utf8_lossy_owned` once it's stabilized. + String::from_utf8_lossy(&bytes).into_owned() + } + + fn token_to_id(&self, token: &str) -> Option { + if let Some(&token_id) = self.special_token_ids_by_text.get(token) { + return Some(token_id); + } + + // Fall back to ordinary encoding for regular vocabulary items. This + // deliberately avoids `encode_with_special_tokens`: older `tiktoken-rs` + // versions can panic if the input text merely *looks* like a special + // token but is not registered in `special_tokens_encoder`. + let ids = self.inner.encode_ordinary(token); + if ids.len() == 1 { Some(ids[0]) } else { None } + } + + /// Log a warning the first time an unknown token id is seen during decode, + /// deduped across calls so streaming decode does not spam the log for + /// the same id. + fn warn_unknown_id(&self, token_id: u32) { + let newly_inserted = self + .warned_unknown_ids + .lock() + .map(|mut set| set.insert(token_id)) + .unwrap_or(false); + if newly_inserted { + warn!( + token_id, + "tiktoken-rs decode encountered token id not in the vocabulary; skipping. \ + This typically indicates a sparse-vocab model whose `added_tokens_decoder` \ + does not list every reserved id in the special-token range." + ); + } + } +} + +impl TiktokenTokenizer { + /// Load a tiktoken tokenizer from a `.tiktoken` / `tiktoken.model` BPE + /// file. + /// + /// The BPE file format is one ` ` pair per line, + /// the same format used by OpenAI's tiktoken and by HuggingFace model + /// repos that ship tiktoken files (e.g. DeepSeek, Kimi K2). + /// + /// Special / added tokens are read from `tokenizer_config.json` in the same + /// directory when present. The `cl100k_base` regex pattern is used as a + /// reasonable default. + pub fn new(path: &Path) -> Result { + if std::env::var_os(DISABLE_RIPTOKEN_ENV).is_some() { + return Self::new_tiktoken_rs(path); + } + + match Self::new_riptoken(path) { + Ok(tokenizer) => Ok(tokenizer), + Err(error) => { + warn!( + path = %path.display(), + error = %error.as_report(), + "failed to load tokenizer with riptoken; falling back to tiktoken-rs" + ); + Self::new_tiktoken_rs(path) + } + } + } + + /// Load from `tiktoken.model` / `*.tiktoken` with riptoken. + pub fn new_riptoken(path: &Path) -> Result { + info!(path = %path.display(), "loading tokenizer with riptoken (BPE file)"); + + let config = LoadedTiktokenConfig::load(path)?; + let allowed_special_tokens = config.special_tokens_encoder.keys().cloned().collect(); + let inner = riptoken::CoreBPE::new( + config.encoder.into_iter().collect(), + config.special_tokens_encoder.into_iter().collect(), + config.pattern, + ) + .map_err(|error| { + tokenizer_error!( + "failed to create riptoken tokenizer from {}: {error}", + path.display() + ) + })?; + + Ok(Self { + backend: Backend::Riptoken(RiptokenBackend { + inner: Box::new(inner), + allowed_special_tokens, + }), + metadata: config.metadata, + }) + } + + /// Load from `tiktoken.model` / `*.tiktoken` with tiktoken-rs. + pub fn new_tiktoken_rs(path: &Path) -> Result { + info!(path = %path.display(), "loading tokenizer with tiktoken-rs (BPE file)"); + + let config = LoadedTiktokenConfig::load(path)?; + let special_token_ids_by_text = config.special_tokens_encoder.clone(); + let inner = tiktoken_rs::CoreBPE::new( + config.encoder, + config.special_tokens_encoder, + config.pattern, + ) + .map_err(|error| { + tokenizer_error!( + "failed to create tiktoken-rs tokenizer from {}: {error}", + path.display() + ) + })?; + + Ok(Self { + backend: Backend::TiktokenRs(TiktokenRsBackend { + inner: Box::new(inner), + special_token_ids_by_text, + warned_unknown_ids: Mutex::new(FxHashSet::default()), + }), + metadata: config.metadata, + }) + } +} + +struct LoadedTiktokenConfig { + encoder: FxHashMap, u32>, + special_tokens_encoder: FxHashMap, + metadata: TokenMetadata, + pattern: &'static str, +} + +impl LoadedTiktokenConfig { + fn load(path: &Path) -> Result { + let content = std::fs::read_to_string(path).map_err(|error| { + tokenizer_error!( + "failed to read tiktoken file {}: {}", + path.display(), + error.as_report() + ) + })?; + let mut encoder: FxHashMap, u32> = + FxHashMap::with_capacity_and_hasher(content.lines().count(), Default::default()); + for line in content.lines() { + if line.is_empty() { + continue; + } + let mut parts = line.split_whitespace(); + let token_b64 = + parts.next().ok_or_else(|| tokenizer_error!("missing token in tiktoken file"))?; + let rank_str = + parts.next().ok_or_else(|| tokenizer_error!("missing rank in tiktoken file"))?; + let token_bytes = base64::engine::general_purpose::STANDARD + .decode(token_b64) + .map_err(|error| tokenizer_error!("invalid base64 in tiktoken file: {error}"))?; + let rank: u32 = rank_str + .parse() + .map_err(|error| tokenizer_error!("invalid rank in tiktoken file: {error}"))?; + encoder.insert(token_bytes, rank); + } + + let parent_dir = path.parent(); + + // Read added/special tokens (id -> {name, special}) from + // tokenizer_config.json in the same dir. + let added_tokens_by_id = parent_dir + .map(|dir| dir.join("tokenizer_config.json")) + .filter(|p| p.exists()) + .and_then(|config_path| { + let content = std::fs::read_to_string(&config_path).ok()?; + serde_json::from_str(&content).ok() + }) + .map(|config: TiktokenTokenizerConfig| config.added_tokens_decoder) + .unwrap_or_default(); + + let model_config: Option = parent_dir + .map(|dir| dir.join("config.json")) + .filter(|p| p.exists()) + .and_then(|config_path| { + let content = std::fs::read_to_string(&config_path).ok()?; + serde_json::from_str(&content).ok() + }); + let vocab_size_from_config = model_config.as_ref().and_then(|c| c.effective_vocab_size()); + + // Build the full special-tokens encoder by populating the reserved + // range that follows the BPE vocabulary. Unknown reserved slots get + // Python-compatible placeholder names so sampled ids can still decode. + // + // Note: `*.tiktoken` ranks are token ids, and they are not guaranteed + // to be contiguous. The base-vocab boundary is therefore `max_rank + 1`, + // not `encoder.len()`. + let num_base_tokens = + encoder.values().copied().max().map_or(0, |max_rank| max_rank.saturating_add(1)); + let max_added_id = added_tokens_by_id.keys().copied().max().unwrap_or(0); + let reserved_end = vocab_size_from_config + .unwrap_or_else(|| num_base_tokens.saturating_add(FALLBACK_NUM_RESERVED_SPECIAL_TOKENS)) + .max(num_base_tokens) + .max(max_added_id.saturating_add(1)); + + let mut special_tokens_encoder: FxHashMap = + FxHashMap::with_capacity_and_hasher( + (reserved_end - num_base_tokens) as usize, + Default::default(), + ); + let mut non_special_added_ids: FxHashSet = FxHashSet::default(); + for id in num_base_tokens..reserved_end { + let name = match added_tokens_by_id.get(&id) { + Some(token) => { + if !token.special { + non_special_added_ids.insert(id); + } + token.content.clone() + } + None => format!("<|reserved_token_{id}|>"), + }; + special_tokens_encoder.insert(name, id); + } + + let mut token_by_id: FxHashMap = FxHashMap::with_capacity_and_hasher( + encoder.len() + special_tokens_encoder.len(), + Default::default(), + ); + for (token_bytes, &id) in &encoder { + token_by_id.insert(id, String::from_utf8_lossy(token_bytes).into_owned()); + } + for (token, &id) in &special_tokens_encoder { + token_by_id.insert(id, token.clone()); + } + + let pattern = model_config.as_ref().map_or(CL100K_BASE_PATTERN, detect_bpe_pattern); + + Ok(Self { + encoder, + special_tokens_encoder, + metadata: TokenMetadata { + num_base_tokens, + vocab_upper_bound: reserved_end, + non_special_added_ids, + token_by_id, + }, + pattern, + }) + } +} + +impl Tokenizer for TiktokenTokenizer { + fn encode(&self, text: &str, _add_special_tokens: bool) -> Result> { + // Tiktoken does not have a separate add_special_tokens toggle; both + // backends recognize registered special tokens in the input. + Ok(match &self.backend { + Backend::Riptoken(backend) => backend.encode(text), + Backend::TiktokenRs(backend) => backend.encode(text), + }) + } + + fn decode(&self, token_ids: &[u32], skip_special_tokens: bool) -> Result { + // Filter passes: + // + // 1. The constructor registers every id in `[num_base_tokens, vocab_upper_bound)` as a + // special token (named or `<|reserved_token_{id}|>` placeholder, matching + // `tokenization_kimi.py`). The tiktoken-rs backend additionally drops ids at or above + // that bound so `_decode_native_and_split` cannot panic; riptoken's `decode_bytes` + // already skips unknown ids. + // + // 2. When `skip_special_tokens = true`, ids in `[num_base_tokens, vocab_upper_bound)` are + // dropped *unless* they were marked `"special": false` in `added_tokens_decoder`. This + // matches HuggingFace's tokenizer semantics: tool-call markers and `` / + // `` (which Kimi K2 / K2.5 declare as non-special) stay in the output, while + // BOS/EOS/header tokens and reserved-slot placeholders are stripped. + // + // Lossy UTF-8 decoding (instead of strict `String::from_utf8`) is used so + // partial multi-byte sequences become `\u{FFFD}`, which `DecodeStream` + // relies on to detect incomplete characters during streaming. + let ids = if skip_special_tokens { + &self.metadata.filter_special_tokens(token_ids) + } else { + token_ids + }; + + Ok(match &self.backend { + Backend::Riptoken(backend) => backend.decode(ids), + Backend::TiktokenRs(backend) => backend.decode(ids, &self.metadata), + }) + } + + fn token_to_id(&self, token: &str) -> Option { + match &self.backend { + Backend::Riptoken(backend) => backend.token_to_id(token), + Backend::TiktokenRs(backend) => backend.token_to_id(token), + } + } + + fn id_to_token(&self, id: u32) -> Option { + self.metadata.id_to_token(id) + } + + fn is_special_id(&self, token_id: u32) -> bool { + self.metadata.is_special_id(token_id) + } + + fn vocab_size(&self) -> usize { + // Exclusive upper bound on token ids the tokenizer can decode (BPE base + // tokens plus the registered special/reserved slots), used to range-check + // `allowed_token_ids` so tiktoken models are not exempt from validation. + self.metadata.vocab_upper_bound as usize + } +} + +/// Select the BPE regex pattern for a tiktoken model based on `config.json`. +/// +/// Most tiktoken models use the `cl100k_base` regex. Kimi models ship a custom +/// regex in their Python tokenizer implementation; we mirror the explicit +/// `model_type` switch used by Dynamo instead of heuristically parsing Python +/// source files. +fn detect_bpe_pattern(config: &TiktokenModelConfig) -> &'static str { + let model_type = config.effective_model_type(); + + match model_type { + Some("kimi" | "kimi_k2" | "kimi_k25" | "deepseek_v3") => KIMI_PATTERN, + _ => CL100K_BASE_PATTERN, + } +} + +#[cfg(test)] +mod tests { + use std::fs; + use std::path::{Path, PathBuf}; + + use base64::Engine as _; + use tempfile::TempDir; + + use super::{ + CL100K_BASE_PATTERN, KIMI_PATTERN, TiktokenModelConfig, TiktokenTokenizer, + TiktokenTokenizerConfig, detect_bpe_pattern, + }; + use crate::Tokenizer; + + macro_rules! config_json { + ($($json:tt)+) => { + serde_json::from_value::(serde_json::json!($($json)+)).unwrap() + }; + } + + /// Write a minimal `*.tiktoken` BPE file (one token per byte 0..=255) into + /// `dir` and return its path. The single-byte vocab is enough to + /// exercise the multi-byte / streaming UTF-8 paths without depending on + /// any pretrained tokenizer asset. + fn write_synthetic_bpe_file(dir: &std::path::Path) -> PathBuf { + let mut content = String::new(); + for byte in 0u8..=255 { + let b64 = base64::engine::general_purpose::STANDARD.encode([byte]); + content.push_str(&format!("{b64} {}\n", byte as u32)); + } + let path = dir.join("test.tiktoken"); + fs::write(&path, content).expect("write tiktoken file"); + path + } + + /// Write a synthetic `*.tiktoken` file whose base-vocab ranks are + /// sparse/non-contiguous. + /// + /// This reproduces the important edge case for `num_base_tokens`: it must + /// be derived from `max_rank + 1`, not `encoder.len()`, otherwise + /// high-rank base tokens get misclassified as reserved/special ids. + fn write_sparse_rank_bpe_file(dir: &std::path::Path) -> PathBuf { + let mut content = String::new(); + for byte in 0u8..=255 { + let b64 = base64::engine::general_purpose::STANDARD.encode([byte]); + content.push_str(&format!("{b64} {}\n", byte as u32)); + } + + let high_rank_token = base64::engine::general_purpose::STANDARD.encode(b"SPARSE"); + content.push_str(&format!("{high_rank_token} 1000\n")); + + let path = dir.join("sparse-rank.tiktoken"); + fs::write(&path, content).expect("write sparse-rank tiktoken file"); + path + } + + /// Build a `TiktokenTokenizer` from the synthetic BPE file with no sibling + /// config files, so the constructor takes the + /// `FALLBACK_NUM_RESERVED_SPECIAL_TOKENS` (256) path. + fn explicit_backends(path: &Path) -> Vec { + vec![ + TiktokenTokenizer::new_riptoken(path).expect("load riptoken backend"), + TiktokenTokenizer::new_tiktoken_rs(path).expect("load tiktoken-rs backend"), + ] + } + + fn tiktoken_backends() -> (Vec, TempDir) { + let dir = tempfile::tempdir().expect("create temp dir"); + let path = write_synthetic_bpe_file(dir.path()); + (explicit_backends(&path), dir) + } + + /// Verify that tiktoken decode uses lossy UTF-8 (producing `\u{FFFD}`) + /// rather than returning an error for incomplete multi-byte sequences. + /// This is critical for streaming decode — `DecodeStream` relies on + /// `\u{FFFD}` to detect incomplete characters. + #[test] + fn tiktoken_decode_incomplete_utf8_produces_replacement_char() { + let (backends, _dir) = tiktoken_backends(); + + for backend in backends { + let ids = backend.encode("你", false).unwrap(); + let full = backend.decode(&ids, false).unwrap(); + assert_eq!(full, "你"); + + let text_with_multibyte = "Hello你好World"; + let all_ids = backend.encode(text_with_multibyte, false).unwrap(); + for &id in &all_ids { + let result = backend.decode(&[id], false); + assert!(result.is_ok(), "decode of token {id} should not error"); + } + } + } + + #[test] + fn tiktoken_vocab_size_reports_upper_bound() { + // The synthetic BPE file has 256 base tokens (bytes 0..=255) and ships no + // sibling config, so the constructor uses the 256-slot reserved fallback, + // giving a vocab upper bound of 512. + let (backends, _dir) = tiktoken_backends(); + for backend in backends { + assert_eq!(backend.vocab_size(), 512); + } + } + + /// When `config.json` exposes a `vocab_size`, the reserved-token range must + /// be sized to it rather than to the 256-slot fallback. This is the + /// general (non-Kimi-specific) path: any tiktoken model whose own + /// `config.json` says e.g. `vocab_size = 280` should populate + /// reserved slots for `[num_base_tokens, 280)` and nothing beyond. + #[test] + fn tiktoken_reserved_range_uses_vocab_size_from_config_json() { + let dir = tempfile::tempdir().expect("create temp dir"); + let bpe_path = write_synthetic_bpe_file(dir.path()); + // num_base_tokens = 256, vocab_size = 280 → reserved range = [256, 280) (24 + // slots, smaller than the 256 fallback so we can prove the config value + // is honoured). + fs::write(dir.path().join("config.json"), r#"{"vocab_size": 280}"#) + .expect("write config.json"); + + for backend in explicit_backends(&bpe_path) { + // Inside the configured range: reserved placeholder, round-trips both ways. + let in_range_id: u32 = 270; + let placeholder = format!("<|reserved_token_{in_range_id}|>"); + assert_eq!(backend.decode(&[in_range_id], false).unwrap(), placeholder); + assert_eq!( + backend.encode(&placeholder, false).unwrap(), + vec![in_range_id] + ); + assert_eq!( + backend.id_to_token(in_range_id).as_deref(), + Some(placeholder.as_str()) + ); + + // Outside the configured range: not registered as a reserved slot — falls + // through to the backend's unknown-id behavior. The point is that we *don't* + // over-populate beyond what the model actually exposes. + let out_of_range_id: u32 = 290; + let out_of_range_placeholder = format!("<|reserved_token_{out_of_range_id}|>"); + assert_eq!(backend.decode(&[out_of_range_id], false).unwrap(), ""); + assert_eq!(backend.token_to_id(&out_of_range_placeholder), None); + assert_eq!(backend.id_to_token(out_of_range_id), None); + } + } + + /// Sparse/non-contiguous BPE ranks must still count as base-vocab ids. + /// + /// Regression shape: + /// - base vocabulary contains ids 0..=255 and also a normal BPE token at id 1000 + /// - if `num_base_tokens` were computed as `encoder.len()` (257), id 1000 would be + /// misclassified as special/reserved and disappear under `skip_special_tokens = true` + #[test] + fn tiktoken_sparse_base_ranks_are_not_misclassified_as_special() { + let dir = tempfile::tempdir().expect("create temp dir"); + let bpe_path = write_sparse_rank_bpe_file(dir.path()); + fs::write(dir.path().join("config.json"), r#"{"vocab_size": 1002}"#) + .expect("write config.json"); + + for backend in explicit_backends(&bpe_path) { + let sparse_id = backend.token_to_id("SPARSE"); + assert_eq!(sparse_id, Some(1000)); + assert_eq!(backend.id_to_token(1000).as_deref(), Some("SPARSE")); + assert!(!backend.is_special_id(1000)); + assert_eq!(backend.decode(&[1000], false).unwrap(), "SPARSE"); + assert_eq!(backend.decode(&[1000], true).unwrap(), "SPARSE"); + } + } + + /// `skip_special_tokens` must: + /// * keep regular BPE token text unchanged, + /// * drop ids whose `added_tokens_decoder` entry says `"special": true`, + /// * drop reserved-slot placeholder ids (which default to special), + /// * keep ids whose `added_tokens_decoder` entry says `"special": false` — this is how Kimi K2 + /// / K2.5 marks tool-call markers and `` / ``. + /// + /// Synthetic backend has `num_base_tokens = 256`. We write a + /// `tokenizer_config.json` that names ids 257 (special) and 258 + /// (non-special), and a `config.json` with `vocab_size` covering both. + /// Id 259 stays a default reserved placeholder (special). + #[test] + fn tiktoken_skip_special_tokens_filters_special_but_keeps_non_special_added_tokens() { + let dir = tempfile::tempdir().expect("create temp dir"); + let bpe_path = write_synthetic_bpe_file(dir.path()); + fs::write( + dir.path().join("tokenizer_config.json"), + r#"{ + "added_tokens_decoder": { + "257": { "content": "<|im_end|>", "special": true }, + "258": { "content": "<|tool_call_begin|>", "special": false } + } + }"#, + ) + .expect("write tokenizer_config.json"); + fs::write(dir.path().join("config.json"), r#"{"vocab_size": 260}"#) + .expect("write config.json"); + + for backend in explicit_backends(&bpe_path) { + // Resolve the BPE ids for "Hi" so we can interleave them with special-token + // ids. + let h = backend.encode("H", false).unwrap()[0]; + let i = backend.encode("i", false).unwrap()[0]; + + let special_id: u32 = 257; // <|im_end|> + let non_special_id: u32 = 258; // <|tool_call_begin|> + let reserved_id: u32 = 259; // default <|reserved_token_259|> placeholder + + let ids = vec![h, special_id, i, non_special_id, reserved_id]; + + // skip_special_tokens = false: everything is rendered as-is. + let kept = backend.decode(&ids, false).unwrap(); + assert_eq!( + kept, + "H<|im_end|>i<|tool_call_begin|><|reserved_token_259|>" + ); + + // skip_special_tokens = true: special token (257) and reserved placeholder + // (259) are dropped; the non-special added token (258) survives. + let stripped = backend.decode(&ids, true).unwrap(); + assert_eq!(stripped, "Hi<|tool_call_begin|>"); + } + } + + /// `vocab_size` may live under `text_config` for composite (e.g. + /// multimodal) configs. + #[test] + fn tiktoken_reserved_range_reads_text_config_vocab_size() { + let dir = tempfile::tempdir().expect("create temp dir"); + let bpe_path = write_synthetic_bpe_file(dir.path()); + fs::write( + dir.path().join("config.json"), + r#"{"text_config": {"vocab_size": 270}}"#, + ) + .expect("write config.json"); + + for backend in explicit_backends(&bpe_path) { + let in_range_id: u32 = 260; + let placeholder = format!("<|reserved_token_{in_range_id}|>"); + assert_eq!(backend.decode(&[in_range_id], false).unwrap(), placeholder); + + // Just outside the nested vocab_size — should not be registered. + assert_eq!(backend.decode(&[270], false).unwrap(), ""); + } + } + + #[test] + fn tiktoken_detects_kimi_pattern_from_model_type() { + let kimi = config_json!({ "model_type": "kimi_k25" }); + let baseten_kimi = config_json!({ "model_type": "deepseek_v3" }); + let nested_kimi = config_json!({ + "model_type": "composite_wrapper", + "text_config": { "model_type": "kimi_k2" } + }); + let generic = config_json!({ "model_type": "gpt2" }); + let nested_generic = config_json!({ + "model_type": "composite_wrapper", + "text_config": { "model_type": "gpt2" } + }); + let missing = config_json!({ "text_config": {} }); + + assert_eq!(detect_bpe_pattern(&kimi), KIMI_PATTERN); + assert_eq!(detect_bpe_pattern(&baseten_kimi), KIMI_PATTERN); + assert_eq!(detect_bpe_pattern(&nested_kimi), CL100K_BASE_PATTERN); + assert_eq!(detect_bpe_pattern(&generic), CL100K_BASE_PATTERN); + assert_eq!(detect_bpe_pattern(&nested_generic), CL100K_BASE_PATTERN); + assert_eq!(detect_bpe_pattern(&missing), CL100K_BASE_PATTERN); + } + + #[test] + fn tiktoken_reads_model_type_from_text_config_when_top_level_missing() { + let nested_only = config_json!({ + "text_config": { "model_type": "kimi_k2" } + }); + let direct_and_nested = config_json!({ + "model_type": "kimi_k25", + "text_config": { "model_type": "kimi_k2" } + }); + let missing = config_json!({ + "text_config": {} + }); + + assert_eq!(nested_only.effective_model_type(), Some("kimi_k2")); + assert_eq!(direct_and_nested.effective_model_type(), Some("kimi_k25")); + assert_eq!(missing.effective_model_type(), None); + } + + #[test] + fn tiktoken_tokenizer_config_models_added_tokens_decoder() { + let config: TiktokenTokenizerConfig = serde_json::from_value(serde_json::json!({ + "added_tokens_decoder": { + "257": { "content": "" }, + "258": { "content": "", "special": true } + } + })) + .unwrap(); + + let added_tokens = config.added_tokens_decoder; + assert_eq!(added_tokens.len(), 2); + assert_eq!( + added_tokens.get(&257).map(|t| t.content.as_str()), + Some("") + ); + assert_eq!(added_tokens.get(&257).map(|t| t.special), Some(false)); + assert_eq!( + added_tokens.get(&258).map(|t| (t.content.as_str(), t.special)), + Some(("", true)) + ); + } + + /// Reserved token ids in `[num_base_tokens, num_base_tokens + 256)` must + /// decode to their placeholder name (matching `tokenization_kimi.py`'s + /// `<|reserved_token_{i}|>` format), even when the source + /// `tokenizer_config.json` does not list them in `added_tokens_decoder`. + /// + /// In our synthetic backend `num_base_tokens = 256` (256 single-byte BPE + /// tokens), so the reserved range is `[256, 512)`. Picking id 300 — + /// well inside that range and absent from any `added_tokens_decoder` — + /// should round-trip both ways. + #[test] + fn tiktoken_reserved_token_round_trip() { + let (backends, _dir) = tiktoken_backends(); + + for backend in backends { + let reserved_id: u32 = 300; + let placeholder = format!("<|reserved_token_{reserved_id}|>"); + + let decoded = backend.decode(&[reserved_id], false).unwrap(); + assert_eq!(decoded, placeholder); + + // The placeholder name should also encode back to the same single id, since + // the constructor registers it as a special token with `CoreBPE`. + let encoded = backend.encode(&placeholder, false).unwrap(); + assert_eq!(encoded, vec![reserved_id]); + + assert_eq!(backend.token_to_id(&placeholder), Some(reserved_id)); + assert_eq!( + backend.id_to_token(reserved_id).as_deref(), + Some(placeholder.as_str()) + ); + } + } + + /// Decoding a token id that is beyond even the reserved range must not + /// panic — it falls through to the warn-and-skip backstop instead of + /// crashing the worker thread. + #[test] + fn tiktoken_rs_decode_unknown_token_id_does_not_panic() { + let dir = tempfile::tempdir().expect("create temp dir"); + let path = write_synthetic_bpe_file(dir.path()); + let backend = TiktokenTokenizer::new_tiktoken_rs(&path).expect("load tiktoken-rs backend"); + + // ID well above num_base_tokens (256) + reserved (256) = 512 — guaranteed + // unknown. + let unknown_id: u32 = 999_999; + let result = backend.decode(&[unknown_id], false); + assert_eq!(result.unwrap(), ""); + + // Mixed: known bytes for "Hi" surrounding an unknown id should yield just "Hi". + let h = backend.encode("H", false).unwrap()[0]; + let i = backend.encode("i", false).unwrap()[0]; + let result = backend.decode(&[h, unknown_id, i], false).unwrap(); + assert_eq!(result, "Hi"); + } + + #[test] + fn riptoken_decode_unknown_token_id_does_not_panic() { + let dir = tempfile::tempdir().expect("create temp dir"); + let path = write_synthetic_bpe_file(dir.path()); + let backend = TiktokenTokenizer::new_riptoken(&path).expect("load riptoken backend"); + + let unknown_id: u32 = 999_999; + assert_eq!(backend.decode(&[unknown_id], false).unwrap(), ""); + + let h = backend.encode("H", false).unwrap()[0]; + let i = backend.encode("i", false).unwrap()[0]; + assert_eq!(backend.decode(&[h, unknown_id, i], false).unwrap(), "Hi"); + } + + /// Streaming decode of CJK text through tiktoken should produce the + /// original text without errors, even though individual tokens may + /// represent partial UTF-8 byte sequences. + #[test] + fn tiktoken_streaming_decode_multibyte() { + let (backends, _dir) = tiktoken_backends(); + for backend in backends { + let text = "你好世界"; // 4 CJK characters + let ids = backend.encode(text, false).unwrap(); + + let mut decoder = backend.create_decode_stream(&[], false, 0); + let mut output = String::new(); + for &id in &ids { + decoder.push_token(id).unwrap(); + if let Some(chunk) = decoder.next_chunk() { + output.push_str(&chunk); + } + } + let (last_chunk, full_text) = decoder.flush(None).unwrap(); + if let Some(chunk) = last_chunk { + output.push_str(&chunk); + } + + assert_eq!(output, text); + assert_eq!(full_text, text); + } + } + + /// Mixed ASCII and multi-byte text should stream correctly through + /// tiktoken. + #[test] + fn tiktoken_streaming_decode_mixed_ascii_and_multibyte() { + let (backends, _dir) = tiktoken_backends(); + for backend in backends { + let text = "Hello 你好 World 🌍"; + let ids = backend.encode(text, false).unwrap(); + + let mut decoder = backend.create_decode_stream(&[], false, 0); + let mut output = String::new(); + for &id in &ids { + decoder.push_token(id).unwrap(); + if let Some(chunk) = decoder.next_chunk() { + output.push_str(&chunk); + } + } + let (last_chunk, full_text) = decoder.flush(None).unwrap(); + if let Some(chunk) = last_chunk { + output.push_str(&chunk); + } + + assert_eq!(output, text); + assert_eq!(full_text, text); + } + } + + #[test] + fn tiktoken_token_to_id_resolves_added_special_tokens() { + let dir = tempfile::tempdir().expect("create temp dir"); + let bpe_path = write_synthetic_bpe_file(dir.path()); + fs::write( + dir.path().join("tokenizer_config.json"), + r#"{ + "added_tokens_decoder": { + "257": { "content": "", "special": false }, + "258": { "content": "", "special": false } + } + }"#, + ) + .expect("write tokenizer_config.json"); + fs::write(dir.path().join("config.json"), r#"{"vocab_size": 259}"#) + .expect("write config.json"); + + for backend in explicit_backends(&bpe_path) { + assert_eq!(backend.token_to_id(""), Some(257)); + assert_eq!(backend.token_to_id(""), Some(258)); + assert_eq!(backend.id_to_token(257).as_deref(), Some("")); + assert_eq!(backend.id_to_token(258).as_deref(), Some("")); + assert_eq!( + backend.decode(&[257, 258], true).unwrap(), + "" + ); + } + } + + #[test] + fn riptoken_token_to_id_uses_encode_single_token_path() { + let dir = tempfile::tempdir().expect("create temp dir"); + let bpe_path = write_synthetic_bpe_file(dir.path()); + fs::write( + dir.path().join("tokenizer_config.json"), + r#"{ + "added_tokens_decoder": { + "257": { "content": "", "special": false } + } + }"#, + ) + .expect("write tokenizer_config.json"); + fs::write(dir.path().join("config.json"), r#"{"vocab_size": 258}"#) + .expect("write config.json"); + let backend = TiktokenTokenizer::new_riptoken(&bpe_path).expect("load riptoken backend"); + + assert_eq!(backend.token_to_id("H"), Some(b'H' as u32)); + assert_eq!(backend.id_to_token(b'H' as u32).as_deref(), Some("H")); + assert_eq!(backend.token_to_id(""), Some(257)); + assert_eq!(backend.id_to_token(257).as_deref(), Some("")); + } + + #[test] + fn tiktoken_rs_token_to_id_handles_unknown_special_like_text_without_panicking() { + let dir = tempfile::tempdir().expect("create temp dir"); + let path = write_synthetic_bpe_file(dir.path()); + let backend = TiktokenTokenizer::new_tiktoken_rs(&path).expect("load tiktoken-rs backend"); + + assert_eq!(backend.token_to_id("<|definitely_not_registered|>"), None); + } + + #[test] + fn riptoken_token_to_id_handles_unknown_special_like_text_without_panicking() { + let dir = tempfile::tempdir().expect("create temp dir"); + let path = write_synthetic_bpe_file(dir.path()); + let backend = TiktokenTokenizer::new_riptoken(&path).expect("load riptoken backend"); + + assert_eq!(backend.token_to_id("<|definitely_not_registered|>"), None); + } +} diff --git a/asian_ai/performance/vLLM/scripts/autotune_helion_kernels.py b/asian_ai/performance/vLLM/scripts/autotune_helion_kernels.py new file mode 100644 index 0000000000000000000000000000000000000000..33303267a1cc287d9229a56170f530df29e26101 --- /dev/null +++ b/asian_ai/performance/vLLM/scripts/autotune_helion_kernels.py @@ -0,0 +1,435 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Autotune registered Helion kernels for optimal configurations. + +Usage: + # Autotune all registered kernels + python scripts/autotune_helion_kernels.py + + # Autotune specific kernel + python scripts/autotune_helion_kernels.py --kernels silu_mul_fp8 + + # Autotune multiple kernels + python scripts/autotune_helion_kernels.py --kernels silu_mul_fp8 rms_norm_fp8 + + # Force re-autotuning + python scripts/autotune_helion_kernels.py --force + + # List available kernels + python scripts/autotune_helion_kernels.py --list +""" + +import argparse +import sys +import time +from dataclasses import dataclass + +import torch +from torch._subclasses.fake_tensor import FakeTensorMode + +try: + import helion + + from vllm.kernels.helion import ( + ConfigManager, + get_kernel_by_name, + get_registered_kernels, + ) + from vllm.kernels.helion.utils import get_canonical_gpu_name + from vllm.logger import init_logger + from vllm.utils.import_utils import has_helion +except ImportError as e: + print(f"Error importing vLLM: {e}") + print("Please ensure vLLM is installed and in your Python path") + sys.exit(1) + +logger = init_logger("vllm.scripts.autotune_helion_kernels") + + +@dataclass +class AutotuneResult: + status: str # "success" | "partial" | "error" | "skipped" + successful: int + failed: int + configs: dict[str, "helion.Config"] + message: str = "" + + +def list_kernels() -> None: + kernels = get_registered_kernels() + + if not kernels: + print("No Helion kernels found in registry.") + return + + print("Available Helion kernels:") + print("=" * 50) + + for name in sorted(kernels.keys()): + print(f" {name}") + + print(f"\nTotal: {len(kernels)} kernels") + + +def check_requirements() -> bool: + if not torch.cuda.is_available(): + logger.error("CUDA is not available. Helion autotuning requires GPU.") + return False + + if not has_helion(): + logger.error("Helion is not installed. Please install Helion package.") + return False + + return True + + +def autotune_kernel( + kernel_name: str, + platform: str, + config_manager: ConfigManager, + force: bool = False, + autotune_effort: str = "quick", +) -> AutotuneResult: + logger.debug( + "Starting autotune for kernel '%s' with effort='%s'", + kernel_name, + autotune_effort, + ) + kernel_wrapper = get_kernel_by_name(kernel_name) + if kernel_wrapper is None: + error_msg = f"Kernel '{kernel_name}' not found in registry" + logger.error(error_msg) + return AutotuneResult( + status="error", + message=error_msg, + successful=0, + failed=0, + configs={}, + ) + + try: + with FakeTensorMode(): + all_config_keys = list(kernel_wrapper.get_inputs().keys()) + except NotImplementedError: + error_msg = f"Kernel '{kernel_name}' has no input generator registered" + logger.error(error_msg) + return AutotuneResult( + status="error", + message=error_msg, + successful=0, + failed=0, + configs={}, + ) + + try: + logger.info( + "Autotuning kernel '%s' for platform '%s' with %d configs", + kernel_name, + platform, + len(all_config_keys), + ) + + if not force: + existing_configs = config_manager.get_platform_configs( + kernel_name, platform + ) + keys_to_autotune = [] + for config_key in all_config_keys: + if config_key in existing_configs: + logger.debug( + "Config '%s' already exists for platform '%s', skipping", + config_key, + platform, + ) + else: + keys_to_autotune.append(config_key) + else: + logger.debug("Force mode enabled, will re-autotune all configs") + keys_to_autotune = all_config_keys + + if not keys_to_autotune: + logger.info( + "All configs already exist for kernel '%s' on platform '%s'. " + "Use --force to re-autotune.", + kernel_name, + platform, + ) + return AutotuneResult( + status="skipped", + message="All configs already exist", + successful=0, + failed=0, + configs={}, + ) + + inputs_dict = kernel_wrapper.get_inputs() + configs_to_autotune = {k: inputs_dict[k] for k in keys_to_autotune} + + total_start_time = time.time() + autotuned_configs = {} + failed_configs = [] + + for config_key, inputs in configs_to_autotune.items(): + logger.info("Autotuning config: %s", config_key) + logger.debug( + "Input shapes: %s", + [getattr(inp, "shape", type(inp).__name__) for inp in inputs], + ) + + try: + config_start_time = time.time() + config = kernel_wrapper.run_autotune(inputs, autotune_effort) + config_duration = time.time() - config_start_time + + # Save immediately for checkpointing + config_manager.save_configs(kernel_name, platform, {config_key: config}) + + autotuned_configs[config_key] = config + logger.debug("Config details: %s", config) + + logger.info( + "✓ Autotuned and saved config '%s' (%.2fs)", + config_key, + config_duration, + ) + + except (RuntimeError, ValueError, OSError) as e: + logger.exception( + "Failed to autotune config '%s': %s", + config_key, + e, + ) + failed_configs.append(config_key) + + total_duration = time.time() - total_start_time + successful = len(autotuned_configs) + failed = len(failed_configs) + + logger.info( + "Completed autotuning for kernel '%s': %d successful, %d failed (%.2fs)", + kernel_name, + successful, + failed, + total_duration, + ) + + status = "success" if failed == 0 else "partial" + return AutotuneResult( + status=status, + successful=successful, + failed=failed, + configs=autotuned_configs, + ) + + except (KeyError, RuntimeError, ValueError, OSError) as e: + error_msg = f"Unexpected error: {e}" + logger.exception("Failed to autotune kernel '%s': %s", kernel_name, e) + return AutotuneResult( + status="error", + message=error_msg, + successful=0, + failed=0, + configs={}, + ) + + +def summarize_results(results: dict[str, AutotuneResult]) -> bool: + logger.info("=" * 50) + logger.info("Autotuning Results Summary") + logger.info("=" * 50) + + total_successful = 0 + total_failed = 0 + success_kernels = [] + partial_kernels = [] + error_kernels = [] + skipped_kernels = [] + + for kernel_name, result in results.items(): + total_successful += result.successful + total_failed += result.failed + + if result.status == "success": + success_kernels.append(f"{kernel_name} ({result.successful} configs)") + logger.info("✓ %s: %d configs successful", kernel_name, result.successful) + elif result.status == "partial": + partial_kernels.append( + f"{kernel_name} ({result.successful} ok, {result.failed} failed)" + ) + logger.warning( + "⚠ %s: %d successful, %d failed", + kernel_name, + result.successful, + result.failed, + ) + elif result.status == "error": + error_kernels.append(f"{kernel_name}: {result.message or 'Unknown error'}") + logger.error("✗ %s: %s", kernel_name, result.message or "Unknown error") + elif result.status == "skipped": + skipped_kernels.append(f"{kernel_name}: {result.message or 'Skipped'}") + logger.info("- %s: %s", kernel_name, result.message or "Skipped") + + logger.info("=" * 50) + logger.info( + "Summary: %d total configs (%d successful, %d failed)", + total_successful + total_failed, + total_successful, + total_failed, + ) + logger.info( + "Kernels: %d success, %d partial, %d error, %d skipped", + len(success_kernels), + len(partial_kernels), + len(error_kernels), + len(skipped_kernels), + ) + + has_failures = bool(error_kernels or partial_kernels) + + if not has_failures: + if total_successful > 0: + logger.info("All configs autotuned successfully!") + else: + logger.info("No new configs were generated (all may already exist)") + + return not has_failures + + +def get_kernels_to_autotune(requested_kernels: list[str] | None) -> list[str]: + all_kernels = get_registered_kernels() + if not all_kernels: + logger.error("No Helion kernels found in registry") + sys.exit(1) + + if not requested_kernels: + return list(all_kernels.keys()) + + if len(requested_kernels) != len(set(requested_kernels)): + duplicates = [ + k for k in set(requested_kernels) if requested_kernels.count(k) > 1 + ] + logger.error("Duplicate kernel names in --kernels flag: %s", duplicates) + sys.exit(1) + + kernels_to_autotune = [] + missing_kernels = [] + + for kernel_name in requested_kernels: + if kernel_name in all_kernels: + kernels_to_autotune.append(kernel_name) + else: + missing_kernels.append(kernel_name) + + if missing_kernels: + logger.error("Kernel(s) not found: %s", missing_kernels) + logger.error("Available kernels: %s", list(all_kernels.keys())) + sys.exit(1) + + return kernels_to_autotune + + +def main(): + parser = argparse.ArgumentParser( + description="Autotune Helion kernels", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=__doc__.split("Usage:")[1] if "Usage:" in __doc__ else "", + ) + + parser.add_argument( + "--kernels", + nargs="+", + help="Kernel(s) to autotune (default: all kernels)", + ) + + parser.add_argument( + "--config-dir", + type=str, + help="Config directory for config files (default: vLLM helion configs dir)", + ) + + parser.add_argument( + "--list", + action="store_true", + help="List available Helion kernels and exit", + ) + + parser.add_argument( + "--force", + action="store_true", + help=( + "Force re-autotuning even if configs already exist for the " + "platform and config keys" + ), + ) + + parser.add_argument( + "--autotune-effort", + type=str, + default="quick", + help=( + "Helion autotune effort level: 'quick' (smaller search) or " + "'full' (full search budget) (default: quick)" + ), + ) + + parser.add_argument( + "--verbose", + action="store_true", + help="Enable verbose logging", + ) + + args = parser.parse_args() + + import logging + + if args.verbose: + logging.getLogger("vllm").setLevel(logging.DEBUG) + logger.debug("Verbose mode enabled") + logger.debug("Arguments: %s", vars(args)) + else: + logging.getLogger("vllm").setLevel(logging.INFO) + + if args.list: + list_kernels() + return + + if not check_requirements(): + sys.exit(1) + + platform = get_canonical_gpu_name() + logger.info("Detected GPU platform: %s", platform) + + config_manager = ( + ConfigManager(args.config_dir) if args.config_dir else ConfigManager() + ) + + try: + config_manager.ensure_base_dir_writable() + except OSError as e: + logger.error("Failed to access config directory: %s", e) + sys.exit(1) + + kernels_to_autotune = get_kernels_to_autotune(args.kernels) + + logger.info( + "Will autotune %d kernel(s) for platform '%s': %s", + len(kernels_to_autotune), + platform, + kernels_to_autotune, + ) + + results = {} + for kernel_name in kernels_to_autotune: + result = autotune_kernel( + kernel_name, platform, config_manager, args.force, args.autotune_effort + ) + results[kernel_name] = result + + success = summarize_results(results) + sys.exit(0 if success else 1) + + +if __name__ == "__main__": + main() diff --git a/asian_ai/performance/vLLM/setup.py b/asian_ai/performance/vLLM/setup.py new file mode 100644 index 0000000000000000000000000000000000000000..69950240f363cc2ebf9703690f2ca29318892fe6 --- /dev/null +++ b/asian_ai/performance/vLLM/setup.py @@ -0,0 +1,1280 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import ctypes +import importlib.util +import json +import logging +import os +import re +import shutil +import subprocess +import sys +import sysconfig +from pathlib import Path +from shutil import which + +import torch +from packaging.version import Version, parse +from setuptools import Extension, setup +from setuptools.command.build_ext import build_ext +from setuptools_rust.build import build_rust +from setuptools_scm import get_version +from torch.utils.cpp_extension import CUDA_HOME, ROCM_HOME + + +def load_module_from_path(module_name, path): + spec = importlib.util.spec_from_file_location(module_name, path) + module = importlib.util.module_from_spec(spec) + sys.modules[module_name] = module + spec.loader.exec_module(module) + return module + + +ROOT_DIR = Path(__file__).parent +logger = logging.getLogger(__name__) + +PRECOMPILED_RUST_FRONTEND_PATH = ROOT_DIR / "vllm" / "vllm-rs" +# setuptools-rust installs PyO3 artifacts as `.`, where the +# suffix ends with `.so` on Linux and macOS alike (e.g. `_rust_foo.abi3.so`). +PRECOMPILED_RUST_EXTENSION_MEMBER_REGEX = re.compile(r"vllm/_rust_[^/]*\.so$") + +# cannot import envs directly because it depends on vllm, +# which is not installed yet +envs = load_module_from_path("envs", os.path.join(ROOT_DIR, "vllm", "envs.py")) +rust_build = load_module_from_path( + "rust_build", os.path.join(ROOT_DIR, "tools", "build_rust.py") +) + +VLLM_TARGET_DEVICE = envs.VLLM_TARGET_DEVICE +USE_PRECOMPILED_EXTENSIONS = envs.VLLM_USE_PRECOMPILED +# VLLM_USE_PRECOMPILED implies precompiled rust frontend too. +USE_PRECOMPILED_RUST_FRONTEND = ( + envs.VLLM_USE_PRECOMPILED or envs.VLLM_USE_PRECOMPILED_RUST +) + + +def should_require_rust_frontend() -> bool: + value = os.getenv("VLLM_REQUIRE_RUST_FRONTEND", "") + return value.lower() not in ("", "0", "false", "no") + + +def get_precompiled_rust_extension_paths() -> list[Path]: + return sorted((ROOT_DIR / "vllm").glob("_rust_*.so")) + + +def get_missing_precompiled_rust_extension_modules() -> list[str]: + present = { + path.name.split(".", 1)[0] for path in get_precompiled_rust_extension_paths() + } + return [ + module_name + for module_name in rust_build.rust_py_extension_module_names() + if module_name not in present + ] + + +def has_precompiled_rust_extensions() -> bool: + return not get_missing_precompiled_rust_extension_modules() + + +if sys.platform.startswith("darwin") and VLLM_TARGET_DEVICE != "cpu": + logger.warning("VLLM_TARGET_DEVICE automatically set to `cpu` due to macOS") + VLLM_TARGET_DEVICE = "cpu" +elif not (sys.platform.startswith("linux") or sys.platform.startswith("darwin")): + logger.warning( + "vLLM only supports Linux platform (including WSL) and MacOS." + "Building on %s, " + "so vLLM may not be able to run correctly", + sys.platform, + ) + VLLM_TARGET_DEVICE = "empty" +elif sys.platform.startswith("linux") and os.getenv("VLLM_TARGET_DEVICE") is None: + if torch.version.hip is not None: + VLLM_TARGET_DEVICE = "rocm" + logger.info("Auto-detected ROCm") + elif torch.version.xpu is not None: + VLLM_TARGET_DEVICE = "xpu" + logger.info("Auto-detected XPU") + elif torch.version.cuda is not None: + VLLM_TARGET_DEVICE = "cuda" + logger.info("Auto-detected CUDA") + else: + VLLM_TARGET_DEVICE = "cpu" + + +def is_sccache_available() -> bool: + return which("sccache") is not None and not bool( + int(os.getenv("VLLM_DISABLE_SCCACHE", "0")) + ) + + +def is_ccache_available() -> bool: + return which("ccache") is not None + + +def is_ninja_available() -> bool: + return which("ninja") is not None + + +def is_freethreaded(): + return bool(sysconfig.get_config_var("Py_GIL_DISABLED")) + + +def should_bundle_tcmalloc() -> bool: + import platform + + return ( + VLLM_TARGET_DEVICE == "cpu" + and sys.platform.startswith("linux") + and platform.machine() in ("aarch64", "x86_64") + ) + + +def find_tcmalloc() -> Path | None: + try: + # get all shared libs the dynamic loader knows about + output = subprocess.check_output( + ["ldconfig", "-p"], + text=True, + stderr=subprocess.DEVNULL, + ) + except Exception: + return None + + # search for libtcmalloc and libtcmalloc_minimal + for library_pattern in ( + r"\blibtcmalloc_minimal\.so\.(\d+)\b", + r"\blibtcmalloc\.so\.(\d+)\b", + ): + candidates: list[tuple[int, Path]] = [] + for line in output.splitlines(): + match = re.search(library_pattern, line) + if match is None or "=>" not in line: + continue + candidate = Path(line.split("=>")[1].strip()) + if candidate.exists(): + candidates.append((int(match.group(1)), candidate)) + + if candidates: + # if multiple candidates are found, pick the one with the highest + # version number + return max(candidates, key=lambda item: item[0])[1] + + return None + + +def bundle_tcmalloc(build_lib: str) -> None: + tcmalloc_library = find_tcmalloc() + if tcmalloc_library is None: + logger.warning( + "Failed to locate tcmalloc. For best performance, " + "please install tcmalloc (e.g. `sudo apt-get " + "install -y --no-install-recommends libtcmalloc-minimal4`)" + ) + return + + bundle_dir = os.path.join(build_lib, "vllm", "libs") + os.makedirs(bundle_dir, exist_ok=True) + bundle_path = os.path.join(bundle_dir, tcmalloc_library.name) + shutil.copy2(tcmalloc_library, bundle_path) + logger.info("Bundled tcmalloc into wheel: %s", bundle_path) + + +class CMakeExtension(Extension): + def __init__(self, name: str, cmake_lists_dir: str = ".", **kwa) -> None: + super().__init__(name, sources=[], py_limited_api=not is_freethreaded(), **kwa) + self.cmake_lists_dir = os.path.abspath(cmake_lists_dir) + + +class cmake_build_ext(build_ext): + # A dict of extension directories that have been configured. + did_config: dict[str, bool] = {} + + # + # Determine number of compilation jobs and optionally nvcc compile threads. + # + def compute_num_jobs(self): + # `num_jobs` is either the value of the MAX_JOBS environment variable + # (if defined) or the number of CPUs available. + num_jobs = envs.MAX_JOBS + if num_jobs is not None: + num_jobs = int(num_jobs) + logger.info("Using MAX_JOBS=%d as the number of jobs.", num_jobs) + else: + try: + # os.sched_getaffinity() isn't universally available, so fall + # back to os.cpu_count() if we get an error here. + num_jobs = len(os.sched_getaffinity(0)) + except AttributeError: + num_jobs = os.cpu_count() + + nvcc_threads = None + if _is_cuda() and CUDA_HOME is not None: + try: + nvcc_version = get_nvcc_cuda_version() + if nvcc_version >= Version("11.2"): + # `nvcc_threads` is either the value of the NVCC_THREADS + # environment variable (if defined) or 1. + # when it is set, we reduce `num_jobs` to avoid + # overloading the system. + nvcc_threads = envs.NVCC_THREADS + if nvcc_threads is not None: + nvcc_threads = int(nvcc_threads) + logger.info( + "Using NVCC_THREADS=%d as the number of nvcc threads.", + nvcc_threads, + ) + else: + nvcc_threads = 1 + num_jobs = max(1, num_jobs // nvcc_threads) + except Exception as e: + logger.warning("Failed to get NVCC version: %s", e) + + return num_jobs, nvcc_threads + + # + # Perform cmake configuration for a single extension. + # + def configure(self, ext: CMakeExtension) -> None: + # If we've already configured using the CMakeLists.txt for + # this extension, exit early. + if ext.cmake_lists_dir in cmake_build_ext.did_config: + return + + cmake_build_ext.did_config[ext.cmake_lists_dir] = True + + # Select the build type. + # Note: optimization level + debug info are set by the build type + default_cfg = "Debug" if self.debug else "RelWithDebInfo" + cfg = envs.CMAKE_BUILD_TYPE or default_cfg + + cmake_args = [ + "-DCMAKE_BUILD_TYPE={}".format(cfg), + "-DVLLM_TARGET_DEVICE={}".format(VLLM_TARGET_DEVICE), + ] + + verbose = envs.VERBOSE + if verbose: + cmake_args += ["-DCMAKE_VERBOSE_MAKEFILE=ON"] + + if is_sccache_available(): + cmake_args += [ + "-DCMAKE_C_COMPILER_LAUNCHER=sccache", + "-DCMAKE_CXX_COMPILER_LAUNCHER=sccache", + "-DCMAKE_CUDA_COMPILER_LAUNCHER=sccache", + "-DCMAKE_HIP_COMPILER_LAUNCHER=sccache", + ] + elif is_ccache_available(): + cmake_args += [ + "-DCMAKE_C_COMPILER_LAUNCHER=ccache", + "-DCMAKE_CXX_COMPILER_LAUNCHER=ccache", + "-DCMAKE_CUDA_COMPILER_LAUNCHER=ccache", + "-DCMAKE_HIP_COMPILER_LAUNCHER=ccache", + ] + + # Pass the python executable to cmake so it can find an exact + # match. + cmake_args += ["-DVLLM_PYTHON_EXECUTABLE={}".format(sys.executable)] + + # Pass the python path to cmake so it can reuse the build dependencies + # on subsequent calls to python. + cmake_args += ["-DVLLM_PYTHON_PATH={}".format(":".join(sys.path))] + + # Override the base directory for FetchContent downloads to $ROOT/.deps + # This allows sharing dependencies between profiles, + # and plays more nicely with sccache. + # To override this, set the FETCHCONTENT_BASE_DIR environment variable. + fc_base_dir = os.path.join(ROOT_DIR, ".deps") + fc_base_dir = os.environ.get("FETCHCONTENT_BASE_DIR", fc_base_dir) + cmake_args += ["-DFETCHCONTENT_BASE_DIR={}".format(fc_base_dir)] + + # + # Setup parallelism and build tool + # + num_jobs, nvcc_threads = self.compute_num_jobs() + + if nvcc_threads: + cmake_args += ["-DNVCC_THREADS={}".format(nvcc_threads)] + + if is_ninja_available(): + build_tool = ["-G", "Ninja"] + cmake_args += [ + "-DCMAKE_JOB_POOL_COMPILE:STRING=compile", + "-DCMAKE_JOB_POOLS:STRING=compile={}".format(num_jobs), + ] + else: + # Default build tool to whatever cmake picks. + build_tool = [] + # Make sure we use the nvcc from CUDA_HOME + if _is_cuda() and CUDA_HOME is not None: + cmake_args += [f"-DCMAKE_CUDA_COMPILER={CUDA_HOME}/bin/nvcc"] + elif _is_hip() and ROCM_HOME is not None: + cmake_args += [f"-DROCM_PATH={ROCM_HOME}"] + + other_cmake_args = os.environ.get("CMAKE_ARGS") + if other_cmake_args: + cmake_args += other_cmake_args.split() + + subprocess.check_call( + ["cmake", ext.cmake_lists_dir, *build_tool, *cmake_args], + cwd=self.build_temp, + ) + + def build_extensions(self) -> None: + # Ensure that CMake is present and working + try: + subprocess.check_output(["cmake", "--version"]) + except OSError as e: + raise RuntimeError("Cannot find CMake executable") from e + + # Create build directory if it does not exist. + if not os.path.exists(self.build_temp): + os.makedirs(self.build_temp) + + targets = [] + + def target_name(s: str) -> str: + return s.removeprefix("vllm.").removeprefix("vllm_flash_attn.") + + # Build all the extensions + for ext in self.extensions: + self.configure(ext) + targets.append(target_name(ext.name)) + + num_jobs, _ = self.compute_num_jobs() + + build_args = [ + "--build", + ".", + f"-j={num_jobs}", + *[f"--target={name}" for name in targets], + ] + + subprocess.check_call(["cmake", *build_args], cwd=self.build_temp) + + # Install the libraries + for ext in self.extensions: + # Install the extension into the proper location + outdir = Path(self.get_ext_fullpath(ext.name)).parent.absolute() + + # Skip if the install directory is the same as the build directory + if outdir == self.build_temp: + continue + + # CMake appends the extension prefix to the install path, + # and outdir already contains that prefix, so we need to remove it. + prefix = outdir + for _ in range(ext.name.count(".")): + prefix = prefix.parent + + # prefix here should actually be the same for all components + install_args = [ + "cmake", + "--install", + ".", + "--prefix", + prefix, + "--component", + target_name(ext.name), + ] + subprocess.check_call(install_args, cwd=self.build_temp) + + def run(self): + # First, run the standard build_ext command to compile the extensions + super().run() + + # bundle tcmalloc into CPU wheels for best OOB perf + if should_bundle_tcmalloc(): + bundle_tcmalloc(self.build_lib) + + # copy vllm/vllm_flash_attn/**/*.py from self.build_lib to current + # directory so that they can be included in the editable build + import glob + + files = glob.glob( + os.path.join(self.build_lib, "vllm", "vllm_flash_attn", "**", "*.py"), + recursive=True, + ) + for file in files: + dst_file = os.path.join( + "vllm/vllm_flash_attn", file.split("vllm/vllm_flash_attn/")[-1] + ) + print(f"Copying {file} to {dst_file}") + os.makedirs(os.path.dirname(dst_file), exist_ok=True) + self.copy_file(file, dst_file) + + if _is_cuda() or _is_hip(): + # copy vllm/third_party/triton_kernels/**/*.py from self.build_lib + # to current directory so that they can be included in the editable + # build + print( + f"Copying {self.build_lib}/vllm/third_party/triton_kernels " + "to vllm/third_party/triton_kernels" + ) + shutil.copytree( + f"{self.build_lib}/vllm/third_party/triton_kernels", + "vllm/third_party/triton_kernels", + dirs_exist_ok=True, + ) + + if _is_cuda(): + # copy vendored deep_gemm package from build_lib to source tree + # for editable installs + deep_gemm_build = os.path.join( + self.build_lib, "vllm", "third_party", "deep_gemm" + ) + if os.path.exists(deep_gemm_build): + print(f"Copying {deep_gemm_build} to vllm/third_party/deep_gemm") + shutil.copytree( + deep_gemm_build, + "vllm/third_party/deep_gemm", + dirs_exist_ok=True, + ) + + # copy vendored fmha_sm100 package from build_lib to source tree + # for editable installs + fmha_sm100_build = os.path.join( + self.build_lib, "vllm", "third_party", "fmha_sm100" + ) + if os.path.exists(fmha_sm100_build): + print(f"Copying {fmha_sm100_build} to vllm/third_party/fmha_sm100") + shutil.copytree( + fmha_sm100_build, + "vllm/third_party/fmha_sm100", + dirs_exist_ok=True, + ) + + +class precompiled_build_ext(build_ext): + """Disables extension building when using precompiled binaries.""" + + def run(self) -> None: + return + + def build_extensions(self) -> None: + print("Skipping build_ext: using precompiled extensions.") + return + + +class precompiled_build_rust(build_rust): + """Skips local Rust builds when all precompiled Rust artifacts are present.""" + + def run(self) -> None: + missing = [] + if not PRECOMPILED_RUST_FRONTEND_PATH.exists(): + missing.append(str(PRECOMPILED_RUST_FRONTEND_PATH)) + missing_rust_extensions = get_missing_precompiled_rust_extension_modules() + if missing_rust_extensions: + missing.extend( + str(ROOT_DIR / "vllm" / f"{module_name}*.so") + for module_name in missing_rust_extensions + ) + + if not missing: + logger.info( + "Skipping local Rust build: using precompiled %s and %s", + PRECOMPILED_RUST_FRONTEND_PATH, + get_precompiled_rust_extension_paths(), + ) + return + + logger.warning( + "Precompiled wheel did not provide all Rust artifacts (%s); " + "falling back to local Rust build.", + ", ".join(missing), + ) + super().run() + + +class precompiled_wheel_utils: + """Extracts libraries and other files from an existing wheel.""" + + @staticmethod + def fetch_metadata_for_variant( + commit: str, variant: str | None + ) -> tuple[list[dict], str]: + """ + Fetches metadata for a specific variant of the precompiled wheel. + """ + variant_dir = f"{variant}/" if variant is not None else "" + repo_url = f"https://wheels.vllm.ai/{commit}/{variant_dir}vllm/" + meta_url = repo_url + "metadata.json" + print(f"Trying to fetch nightly build metadata from {meta_url}") + from urllib.request import urlopen + + with urlopen(meta_url) as resp: + # urlopen raises HTTPError on unexpected status code + wheels = json.loads(resp.read().decode("utf-8")) + return wheels, repo_url + + @staticmethod + def is_rocm_system() -> bool: + """Detect ROCm without relying on torch (for build environment).""" + if os.getenv("ROCM_PATH"): + return True + if os.path.isdir("/opt/rocm"): + return True + if which("rocminfo") is not None: + return True + try: + import torch + + return torch.version.hip is not None + except ImportError: + return False + + @staticmethod + def detect_system_cuda_variant() -> str: + """Auto-detect CUDA variant from torch, nvidia-smi, or env default.""" + + # Map CUDA major version to hosted wheel variants on wheels.vllm.ai + supported = {12: "cu129", 13: "cu130"} + + # Respect explicitly set VLLM_MAIN_CUDA_VERSION + if envs.is_set("VLLM_MAIN_CUDA_VERSION"): + v = envs.VLLM_MAIN_CUDA_VERSION + print(f"Using VLLM_MAIN_CUDA_VERSION={v}") + return "cu" + v.replace(".", "")[:3] + + # Try torch.version.cuda + cuda_version = None + try: + import torch + + cuda_version = torch.version.cuda + except Exception: + pass + + # Try nvidia-smi + if not cuda_version: + try: + out = subprocess.run( + ["nvidia-smi"], capture_output=True, text=True, timeout=10 + ) + if m := re.search(r"CUDA Version:\s*(\d+\.\d+)", out.stdout): + cuda_version = m.group(1) + except Exception: + pass + + # Fall back to default + if not cuda_version: + cuda_version = envs.VLLM_MAIN_CUDA_VERSION + + # Map to supported variant + major = int(cuda_version.split(".")[0]) + variant = supported.get(major, supported[max(supported)]) + print(f"Detected CUDA {cuda_version}, using variant {variant}") + return variant + + @staticmethod + def find_local_rocm_wheel() -> str | None: + """Search for a local vllm wheel in common locations.""" + import glob + + for pattern in ["/vllm-workspace/dist/vllm-*.whl", "./dist/vllm-*.whl"]: + wheels = glob.glob(pattern) + if wheels: + return sorted(wheels)[-1] + return None + + @staticmethod + def fetch_wheel_from_pypi_index(index_url: str, package: str = "vllm") -> str: + """Fetch the latest wheel URL from a PyPI-style simple index.""" + import platform + from html.parser import HTMLParser + from urllib.parse import urljoin + from urllib.request import urlopen + + arch = platform.machine() + + class WheelLinkParser(HTMLParser): + def __init__(self): + super().__init__() + self.wheels = [] + + def handle_starttag(self, tag, attrs): + if tag == "a": + for name, value in attrs: + if name == "href" and value.endswith(".whl"): + self.wheels.append(value) + + simple_url = f"{index_url.rstrip('/')}/{package}/" + print(f"Fetching wheel list from {simple_url}") + with urlopen(simple_url) as resp: + html = resp.read().decode("utf-8") + + parser = WheelLinkParser() + parser.feed(html) + + for wheel in reversed(parser.wheels): + if arch in wheel: + if wheel.startswith("http"): + return wheel + return urljoin(simple_url, wheel) + + raise ValueError(f"No compatible wheel found for {arch} at {simple_url}") + + @staticmethod + def determine_wheel_url_rocm() -> tuple[str, str | None]: + """Determine the precompiled wheel for ROCm.""" + # Search for local wheel first + local_wheel = precompiled_wheel_utils.find_local_rocm_wheel() + if local_wheel is not None: + print(f"Found local ROCm wheel: {local_wheel}") + return local_wheel, None + + # Fall back to AMD's PyPI index + index_url = os.getenv( + "VLLM_ROCM_WHEEL_INDEX", "https://pypi.amd.com/vllm-rocm/simple" + ) + print(f"Fetching ROCm precompiled wheel from {index_url}") + wheel_url = precompiled_wheel_utils.fetch_wheel_from_pypi_index(index_url) + download_filename = wheel_url.split("/")[-1].split("#")[0] + print(f"Using ROCm precompiled wheel: {wheel_url}") + return wheel_url, download_filename + + @staticmethod + def determine_wheel_url() -> tuple[str, str | None]: + """ + Try to determine the precompiled wheel URL or path to use. + The order of preference is: + 1. user-specified wheel location (can be either local or remote, via + VLLM_PRECOMPILED_WHEEL_LOCATION) + 2. user-specified variant (VLLM_PRECOMPILED_WHEEL_VARIANT) from nightly repo + or auto-detected CUDA variant based on system (torch, nvidia-smi) + 3. the default variant from nightly repo + + If downloading from the nightly repo, the commit can be specified via + VLLM_PRECOMPILED_WHEEL_COMMIT; otherwise, the head commit in the main branch + is used. + """ + wheel_location = os.getenv("VLLM_PRECOMPILED_WHEEL_LOCATION", None) + if wheel_location is not None: + print(f"Using user-specified precompiled wheel location: {wheel_location}") + return wheel_location, None + else: + # ROCm: use local wheel or AMD's PyPI index + # TODO: When we have ROCm nightly wheels, we can update this logic. + if precompiled_wheel_utils.is_rocm_system(): + return precompiled_wheel_utils.determine_wheel_url_rocm() + + import platform + + arch = platform.machine() + # try to fetch the wheel metadata from the nightly wheel repo, + # detecting CUDA variant from system if not specified + variant = os.getenv("VLLM_PRECOMPILED_WHEEL_VARIANT", None) + if variant is None: + variant = precompiled_wheel_utils.detect_system_cuda_variant() + commit = os.getenv("VLLM_PRECOMPILED_WHEEL_COMMIT", "").lower() + if not commit or len(commit) != 40: + print( + f"VLLM_PRECOMPILED_WHEEL_COMMIT not valid: {commit}" + ", trying to fetch base commit in main branch" + ) + commit = precompiled_wheel_utils.get_base_commit_in_main_branch() + print(f"Using precompiled wheel commit {commit} with variant {variant}") + try_default = False + wheels, repo_url, download_filename = None, None, None + try: + wheels, repo_url = precompiled_wheel_utils.fetch_metadata_for_variant( + commit, variant + ) + except Exception as e: + logger.warning( + "Failed to fetch precompiled wheel metadata for variant %s: %s", + variant, + e, + ) + try_default = True # try outside handler to keep the stacktrace simple + if try_default: + print("Trying the default variant from remote") + wheels, repo_url = precompiled_wheel_utils.fetch_metadata_for_variant( + commit, None + ) + # if this also fails, then we have nothing more to try / cache + assert wheels is not None and repo_url is not None, ( + "Failed to fetch precompiled wheel metadata" + ) + # The metadata.json has the following format: + # see .buildkite/scripts/generate-nightly-index.py for details + """[{ + "package_name": "vllm", + "version": "0.11.2.dev278+gdbc3d9991", + "build_tag": null, + "python_tag": "cp38", + "abi_tag": "abi3", + "platform_tag": "manylinux1_x86_64", + "variant": null, + "filename": "vllm-0.11.2.dev278+gdbc3d9991-cp38-abi3-manylinux1_x86_64.whl", + "path": "../vllm-0.11.2.dev278%2Bgdbc3d9991-cp38-abi3-manylinux1_x86_64.whl" + }, + ...]""" + from urllib.parse import urljoin + + for wheel in wheels: + # TODO: maybe check more compatibility later? (python_tag, abi_tag, etc) + if wheel.get("package_name") == "vllm" and arch in wheel.get( + "platform_tag", "" + ): + print(f"Found precompiled wheel metadata: {wheel}") + if "path" not in wheel: + raise ValueError(f"Wheel metadata missing path: {wheel}") + wheel_url = urljoin(repo_url, wheel["path"]) + download_filename = wheel.get("filename") + print(f"Using precompiled wheel URL: {wheel_url}") + break + else: + raise ValueError( + f"No precompiled vllm wheel found for architecture {arch} " + f"from repo {repo_url}. All available wheels: {wheels}" + ) + + return wheel_url, download_filename + + @staticmethod + def extract_precompiled_and_patch_package( + wheel_url_or_path: str, + download_filename: str | None, + *, + extract_extensions: bool, + extract_rust_frontend: bool, + ) -> dict: + import tempfile + import zipfile + + temp_dir = None + try: + if not os.path.isfile(wheel_url_or_path): + # use provided filename first, then derive from URL + wheel_filename = download_filename or wheel_url_or_path.split("/")[-1] + temp_dir = tempfile.mkdtemp(prefix="vllm-wheels") + wheel_path = os.path.join(temp_dir, wheel_filename) + print(f"Downloading wheel from {wheel_url_or_path} to {wheel_path}") + from urllib.request import urlretrieve + + urlretrieve(wheel_url_or_path, filename=wheel_path) + else: + wheel_path = wheel_url_or_path + print(f"Using existing wheel at {wheel_path}") + + package_data_patch = {} + + with zipfile.ZipFile(wheel_path) as wheel: + exact_members = set() + if extract_extensions: + exact_members.update( + { + "vllm/_C.abi3.so", + "vllm/_C_stable_libtorch.abi3.so", + "vllm/_moe_C_stable_libtorch.abi3.so", + "vllm/_qutlass_C.abi3.so", + "vllm/_flashmla_C.abi3.so", + "vllm/_flashmla_extension_C.abi3.so", + "vllm/_sparse_flashmla_C.abi3.so", + "vllm/vllm_flash_attn/_vllm_fa2_C.abi3.so", + "vllm/vllm_flash_attn/_vllm_fa3_C.abi3.so", + "vllm/cumem_allocator.abi3.so", + "vllm/spinloop.abi3.so", + "vllm/fs_io_C.abi3.so", + # ROCm-specific libraries + "vllm/_rocm_C.abi3.so", + } + ) + if extract_rust_frontend: + exact_members.add("vllm/vllm-rs") + + flash_attn_regex = re.compile( + r"vllm/vllm_flash_attn/(?:[^/.][^/]*/)*(?!\.)[^/]*\.py" + ) + # __init__.py and flash_attn_interface.py are source-controlled + # in vllm and should not be overwritten (matches cmake exclusions) + flash_attn_files_to_skip = { + "vllm/vllm_flash_attn/__init__.py", + "vllm/vllm_flash_attn/flash_attn_interface.py", + } + triton_kernels_regex = re.compile( + r"vllm/third_party/triton_kernels/(?:[^/.][^/]*/)*(?!\.)[^/]*\.py" + ) + flashmla_regex = re.compile( + r"vllm/third_party/flashmla/(?:[^/.][^/]*/)*(?!\.)[^/]*\.py" + ) + # DeepGEMM: extract all files (.py, .so, .cuh, .h, .hpp, etc.) + deep_gemm_regex = re.compile(r"vllm/third_party/deep_gemm/.*") + fmha_sm100_regex = re.compile(r"vllm/third_party/fmha_sm100/.*") + file_members = [] + for member in wheel.filelist: + if member.filename in exact_members: + file_members.append(member) + continue + if ( + extract_rust_frontend + and PRECOMPILED_RUST_EXTENSION_MEMBER_REGEX.match( + member.filename + ) + ): + file_members.append(member) + continue + + if not extract_extensions: + continue + + if ( + ( + flash_attn_regex.match(member.filename) + and member.filename not in flash_attn_files_to_skip + ) + or triton_kernels_regex.match(member.filename) + or flashmla_regex.match(member.filename) + or deep_gemm_regex.match(member.filename) + or fmha_sm100_regex.match(member.filename) + ): + file_members.append(member) + + for file in file_members: + print(f"[extract] {file.filename}") + target_path = os.path.join(".", file.filename) + os.makedirs(os.path.dirname(target_path), exist_ok=True) + with ( + wheel.open(file.filename) as src, + open(target_path, "wb") as dst, + ): + shutil.copyfileobj(src, dst) + mode = file.external_attr >> 16 + if mode: + os.chmod(target_path, mode) + + pkg = os.path.dirname(file.filename).replace("/", ".") + package_data_patch.setdefault(pkg, []).append( + os.path.basename(file.filename) + ) + + return package_data_patch + finally: + if temp_dir is not None: + print(f"Removing temporary directory {temp_dir}") + shutil.rmtree(temp_dir) + + @staticmethod + def get_base_commit_in_main_branch() -> str: + try: + # Get the latest commit hash of the upstream main branch. + curl_cmd = [ + "curl", + "-s", + "https://api.github.com/repos/vllm-project/vllm/commits/main", + ] + github_token = os.getenv("GH_TOKEN", os.getenv("GITHUB_TOKEN")) + if github_token: + curl_cmd += [ + "-H", + f"Authorization: token {github_token}", + ] + resp_json = subprocess.check_output(curl_cmd).decode("utf-8") + upstream_main_commit = json.loads(resp_json)["sha"] + print(f"Upstream main branch latest commit: {upstream_main_commit}") + + # In Docker build context, .git may be immutable or missing. + if envs.VLLM_DOCKER_BUILD_CONTEXT: + return upstream_main_commit + + # Check if the upstream_main_commit exists in the local repo + try: + subprocess.check_output( + ["git", "cat-file", "-e", f"{upstream_main_commit}"] + ) + except subprocess.CalledProcessError: + # If not present, fetch it from the remote repository. + # Note that this does not update any local branches, + # but ensures that this commit ref and its history are + # available in our local repo. + subprocess.check_call( + ["git", "fetch", "https://github.com/vllm-project/vllm", "main"] + ) + + # Then get the commit hash of the current branch that is the same as + # the upstream main commit. + current_branch = ( + subprocess.check_output(["git", "branch", "--show-current"]) + .decode("utf-8") + .strip() + ) + + base_commit = ( + subprocess.check_output( + ["git", "merge-base", f"{upstream_main_commit}", current_branch] + ) + .decode("utf-8") + .strip() + ) + return base_commit + except ValueError as err: + raise ValueError(err) from None + except Exception as err: + logger.warning( + "Failed to get the base commit in the main branch. " + "Using the nightly wheel. The libraries in this " + "wheel may not be compatible with your dev branch: %s", + err, + ) + return "nightly" + + +def _no_device() -> bool: + return VLLM_TARGET_DEVICE == "empty" + + +def _is_cuda() -> bool: + has_cuda = torch.version.cuda is not None + return VLLM_TARGET_DEVICE == "cuda" and has_cuda and not _is_tpu() + + +def _is_hip() -> bool: + return ( + VLLM_TARGET_DEVICE == "cuda" or VLLM_TARGET_DEVICE == "rocm" + ) and torch.version.hip is not None + + +def _is_tpu() -> bool: + return VLLM_TARGET_DEVICE == "tpu" + + +def _is_cpu() -> bool: + return VLLM_TARGET_DEVICE == "cpu" + + +def _is_xpu() -> bool: + return VLLM_TARGET_DEVICE == "xpu" + + +def _build_custom_ops() -> bool: + return _is_cuda() or _is_hip() + + +def get_rocm_version(): + # Get the Rocm version from the ROCM_HOME/bin/librocm-core.so + # see https://github.com/ROCm/rocm-core/blob/d11f5c20d500f729c393680a01fa902ebf92094b/rocm_version.cpp#L21 + try: + if ROCM_HOME is None: + return None + librocm_core_file = Path(ROCM_HOME) / "lib" / "librocm-core.so" + if not librocm_core_file.is_file(): + return None + librocm_core = ctypes.CDLL(librocm_core_file) + VerErrors = ctypes.c_uint32 + get_rocm_core_version = librocm_core.getROCmVersion + get_rocm_core_version.restype = VerErrors + get_rocm_core_version.argtypes = [ + ctypes.POINTER(ctypes.c_uint32), + ctypes.POINTER(ctypes.c_uint32), + ctypes.POINTER(ctypes.c_uint32), + ] + major = ctypes.c_uint32() + minor = ctypes.c_uint32() + patch = ctypes.c_uint32() + + if ( + get_rocm_core_version( + ctypes.byref(major), ctypes.byref(minor), ctypes.byref(patch) + ) + == 0 + ): + return f"{major.value}.{minor.value}.{patch.value}" + return None + except Exception: + return None + + +def get_nvcc_cuda_version() -> Version: + """Get the CUDA version from nvcc. + + Adapted from https://github.com/NVIDIA/apex/blob/8b7a1ff183741dd8f9b87e7bafd04cfde99cea28/setup.py + """ + assert CUDA_HOME is not None, "CUDA_HOME is not set" + nvcc_output = subprocess.check_output( + [CUDA_HOME + "/bin/nvcc", "-V"], universal_newlines=True + ) + output = nvcc_output.split() + release_idx = output.index("release") + 1 + nvcc_cuda_version = parse(output[release_idx].split(",")[0]) + return nvcc_cuda_version + + +def get_vllm_version() -> str: + # Allow overriding the version. This is useful to build platform-specific + # wheels (e.g. CPU, TPU) without modifying the source. + if env_version := os.getenv("VLLM_VERSION_OVERRIDE"): + print(f"Overriding VLLM version with {env_version} from VLLM_VERSION_OVERRIDE") + os.environ["SETUPTOOLS_SCM_PRETEND_VERSION"] = env_version + return get_version(write_to="vllm/_version.py") + + version = get_version(write_to="vllm/_version.py") + sep = "+" if "+" not in version else "." # dev versions might contain + + + if _no_device(): + if envs.VLLM_TARGET_DEVICE == "empty": + version += f"{sep}empty" + elif _is_cuda(): + if USE_PRECOMPILED_EXTENSIONS and not envs.VLLM_SKIP_PRECOMPILED_VERSION_SUFFIX: + version += f"{sep}precompiled" + else: + cuda_version = str(get_nvcc_cuda_version()) + if cuda_version != envs.VLLM_MAIN_CUDA_VERSION: + cuda_version_str = cuda_version.replace(".", "")[:3] + # skip this for source tarball, required for pypi + if "sdist" not in sys.argv: + version += f"{sep}cu{cuda_version_str}" + elif _is_hip(): + # Get the Rocm Version + rocm_version = get_rocm_version() or torch.version.hip + if rocm_version and rocm_version != envs.VLLM_MAIN_CUDA_VERSION: + version += f"{sep}rocm{rocm_version.replace('.', '')[:3]}" + elif _is_tpu(): + version += f"{sep}tpu" + elif _is_cpu(): + # Check the local VLLM_TARGET_DEVICE (may be set by auto-detect above), + # not envs.VLLM_TARGET_DEVICE, so CPU-only hosts still get `+cpu`. + if VLLM_TARGET_DEVICE == "cpu": + version += f"{sep}cpu" + elif _is_xpu(): + version += f"{sep}xpu" + else: + raise RuntimeError("Unknown runtime environment") + + return version + + +def get_requirements() -> list[str]: + """Get Python package dependencies from requirements.txt.""" + requirements_dir = ROOT_DIR / "requirements" + + def _read_requirements(filename: str) -> list[str]: + with open(requirements_dir / filename) as f: + requirements = f.read().strip().split("\n") + resolved_requirements = [] + for line in requirements: + if line.startswith("-r "): + resolved_requirements += _read_requirements(line.split()[1]) + elif ( + not line.startswith("--") + and not line.startswith("#") + and line.strip() != "" + ): + resolved_requirements.append(line) + return resolved_requirements + + if _no_device(): + requirements = _read_requirements("common.txt") + elif _is_cuda(): + requirements = _read_requirements("cuda.txt") + cuda_major, cuda_minor = torch.version.cuda.split(".") + modified_requirements = [] + for req in requirements: + if "vllm-flash-attn" in req and cuda_major != "12": + # vllm-flash-attn is built only for CUDA 12.x. + # Skip for other versions. + continue + if "nvidia-cutlass-dsl[cu13]" in req and cuda_major == "12": + # [cu13] extra is the default; strip it on CUDA 12 builds. + req = req.replace("nvidia-cutlass-dsl[cu13]", "nvidia-cutlass-dsl") + if "humming-kernels[cu13]" in req and cuda_major == "12": + req = req.replace("humming-kernels[cu13]", "humming-kernels[cu12]") + modified_requirements.append(req) + requirements = modified_requirements + elif _is_hip(): + requirements = _read_requirements("rocm.txt") + elif _is_tpu(): + requirements = _read_requirements("tpu.txt") + elif _is_cpu(): + requirements = _read_requirements("cpu.txt") + elif _is_xpu(): + requirements = _read_requirements("xpu.txt") + else: + raise ValueError("Unsupported platform, please use CUDA, ROCm, or CPU.") + return requirements + + +ext_modules = [] + +if _is_cuda() or _is_hip(): + ext_modules.append(CMakeExtension(name="vllm.cumem_allocator")) + # Optional since this doesn't get built (produce an .so file). This is just + # copying the relevant .py files from the source repository. + ext_modules.append(CMakeExtension(name="vllm.triton_kernels", optional=True)) + +if sys.version_info >= (3, 11): + ext_modules.append(CMakeExtension(name="vllm.spinloop")) + ext_modules.append(CMakeExtension(name="vllm.fs_io_C")) + +if _is_hip(): + ext_modules.append(CMakeExtension(name="vllm._rocm_C")) + +if _is_cuda(): + ext_modules.append(CMakeExtension(name="vllm.vllm_flash_attn._vllm_fa2_C")) + if USE_PRECOMPILED_EXTENSIONS or ( + CUDA_HOME and get_nvcc_cuda_version() >= Version("12.3") + ): + # FA3 requires CUDA 12.3 or later + ext_modules.append(CMakeExtension(name="vllm.vllm_flash_attn._vllm_fa3_C")) + # FA4 CuteDSL - Python-only component for FA4's cute DSL support + # Optional since this doesn't produce a .so file, just copies Python files + ext_modules.append( + CMakeExtension(name="vllm.vllm_flash_attn._vllm_fa4_cutedsl_C", optional=True) + ) + if USE_PRECOMPILED_EXTENSIONS or ( + CUDA_HOME and get_nvcc_cuda_version() >= Version("12.9") + ): + # FlashMLA requires CUDA 12.9 or later + # Optional since this doesn't get built (produce an .so file) when + # not targeting a hopper system + ext_modules.append(CMakeExtension(name="vllm._flashmla_C", optional=True)) + ext_modules.append( + CMakeExtension(name="vllm._flashmla_extension_C", optional=True) + ) + if envs.VLLM_USE_PRECOMPILED or ( + CUDA_HOME and get_nvcc_cuda_version() >= Version("12.3") + ): + # DeepGEMM requires CUDA 12.3+ (SM90/SM100) + # Optional since it won't build on unsupported architectures + ext_modules.append(CMakeExtension(name="vllm._deep_gemm_C", optional=True)) + ext_modules.append(CMakeExtension(name="vllm._qutlass_C", optional=True)) + # fmha_sm100 is a Python/CuTe-DSL package installed into vllm.third_party. + ext_modules.append(CMakeExtension(name="vllm.fmha_sm100", optional=True)) + +if _is_cpu(): + import platform + + if platform.machine() in ("x86_64", "AMD64"): + ext_modules.append(CMakeExtension(name="vllm._C")) + ext_modules.append(CMakeExtension(name="vllm._C_AVX512")) + ext_modules.append(CMakeExtension(name="vllm._C_AVX2")) + else: + ext_modules.append(CMakeExtension(name="vllm._C")) + +if _build_custom_ops(): + if _is_hip(): + ext_modules.append(CMakeExtension(name="vllm._C")) + if _is_cuda() or _is_hip(): + ext_modules.append(CMakeExtension(name="vllm._C_stable_libtorch")) + ext_modules.append(CMakeExtension(name="vllm._moe_C_stable_libtorch")) + +package_data = { + "vllm": [ + "py.typed", + "libs/*.so*", + "model_executor/layers/fused_moe/configs/*.json", + "model_executor/layers/quantization/utils/configs/*.json", + "entrypoints/serve/instrumentator/static/*.js", + "entrypoints/serve/instrumentator/static/*.css", + "distributed/kv_transfer/kv_connector/v1/hf3fs/utils/*.cpp", + # DeepGEMM JIT include headers (vendored via cmake) + "third_party/deep_gemm/include/**/*.cuh", + "third_party/deep_gemm/include/**/*.h", + "third_party/deep_gemm/include/**/*.hpp", + # fmha_sm100 sparse CuTe-DSL helper kernels (vendored via cmake) + "third_party/fmha_sm100/csrc/**/*.cu", + "third_party/fmha_sm100/csrc/**/*.h", + "third_party/fmha_sm100/csrc/**/*.jinja", + "third_party/fmha_sm100/csrc/**/*.cu.jinja", + "third_party/fmha_sm100/cute/**/*.cu", + "third_party/fmha_sm100/cutlass/include/**/*.h", + "third_party/fmha_sm100/cutlass/include/**/*.hpp", + "third_party/fmha_sm100/cutlass/tools/util/include/**/*.h", + "third_party/fmha_sm100/cutlass/tools/util/include/**/*.hpp", + ] +} + + +def add_vllm_package_data(filename: str) -> None: + vllm_files = package_data.setdefault("vllm", []) + if filename not in vllm_files: + vllm_files.append(filename) + + +# If using precompiled artifacts, extract and patch package_data in advance. +if USE_PRECOMPILED_RUST_FRONTEND: + wheel_url, download_filename = precompiled_wheel_utils.determine_wheel_url() + patch = precompiled_wheel_utils.extract_precompiled_and_patch_package( + wheel_url, + download_filename, + extract_extensions=USE_PRECOMPILED_EXTENSIONS, + extract_rust_frontend=True, + ) + for pkg, files in patch.items(): + package_data.setdefault(pkg, []).extend(files) + +# If the rust frontend binary is already present in the source tree (e.g., +# pre-built in a separate Docker build stage), ship it as-is. +if PRECOMPILED_RUST_FRONTEND_PATH.exists(): + add_vllm_package_data("vllm-rs") +for rust_extension_path in get_precompiled_rust_extension_paths(): + add_vllm_package_data(rust_extension_path.name) + +if _no_device(): + ext_modules = [] + +if not ext_modules: + cmdclass = {} +else: + cmdclass = { + "build_ext": precompiled_build_ext + if USE_PRECOMPILED_EXTENSIONS + else cmake_build_ext, + } +if ( + USE_PRECOMPILED_RUST_FRONTEND + or PRECOMPILED_RUST_FRONTEND_PATH.exists() + or has_precompiled_rust_extensions() +): + cmdclass["build_rust"] = precompiled_build_rust + +# Rust artifacts, built via setuptools-rust and installed into the package +# directory alongside the Python modules. +rust_extensions = rust_build.rust_extensions( + optional=not should_require_rust_frontend() +) + +setup( + # static metadata should rather go in pyproject.toml + version=get_vllm_version(), + ext_modules=ext_modules, + rust_extensions=rust_extensions, + install_requires=get_requirements(), + extras_require={ + # AMD Zen CPU optimizations via zentorch + "zen": ["zentorch==2.11.0.0"], + "bench": ["pandas", "matplotlib", "seaborn", "datasets", "scipy", "plotly"], + "tensorizer": ["tensorizer==2.10.1"], + "fastsafetensors": ["fastsafetensors >= 0.3.2"], + "instanttensor": ["instanttensor >= 0.1.5"], + "runai": ["runai-model-streamer[s3,gcs,azure] >= 0.15.7"], + "audio": [ + "av", + "scipy", + "soundfile", + "soxr", + "mistral_common[audio]", + ], # Required for audio processing + "video": [], # Kept for backwards compatibility + "flashinfer": [], # Kept for backwards compatibility + # Optional deps for Helion kernel development + # NOTE: When updating helion version, also update CI files: + # - .buildkite/test_areas/kernels.yaml + # - .buildkite/test-amd.yaml + "helion": ["helion==1.1.0"], + # Optional deps for gRPC server (vllm serve --grpc) + "grpc": ["smg-grpc-servicer[vllm] >= 0.5.2"], + # Optional deps for OpenTelemetry tracing + "otel": [ + "opentelemetry-sdk>=1.26.0", + "opentelemetry-api>=1.26.0", + "opentelemetry-exporter-otlp>=1.26.0", + "opentelemetry-semantic-conventions-ai>=0.4.1", + ], + # extra quantization plugin + "extra-quant": ["vllm-gguf-plugin>=0.0.2"], + }, + cmdclass=cmdclass, + package_data=package_data, +) diff --git a/asian_ai/performance/vLLM/tests/__init__.py b/asian_ai/performance/vLLM/tests/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/basic_correctness/__init__.py b/asian_ai/performance/vLLM/tests/basic_correctness/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/basic_correctness/test_basic_correctness.py b/asian_ai/performance/vLLM/tests/basic_correctness/test_basic_correctness.py new file mode 100644 index 0000000000000000000000000000000000000000..0cfd2c892b730adefc696872f95c5284f40f73e0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/basic_correctness/test_basic_correctness.py @@ -0,0 +1,295 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Compare the short outputs of HF and vLLM when using greedy sampling. + +Run `pytest tests/basic_correctness/test_basic_correctness.py`. +""" + +import os +import weakref +from unittest.mock import Mock + +import pytest +import torch +from packaging.version import Version +from transformers import __version__ as TRANSFORMERS_VERSION + +from vllm import LLM +from vllm.platforms import current_platform +from vllm.v1.engine.llm_engine import LLMEngine + +from ..conftest import HfRunner, VllmRunner +from ..models.utils import check_outputs_equal +from ..utils import multi_gpu_test + +ATTN_BACKEND = ["ROCM_ATTN"] if current_platform.is_rocm() else ["FLASH_ATTN"] + +MODELS = [ + "hmellor/tiny-random-Gemma2ForCausalLM", + "meta-llama/Llama-3.2-1B-Instruct", +] + +TARGET_TEST_SUITE_ENV = "VLLM_TARGET_TEST_SUITE" +LEGACY_TARGET_TEST_SUITE_ENV = "TARGET_TEST_SUITE" + +GENERIC_DISTRIBUTED_TEST_SUITES = ("L4", "MI250", "MI300", "MI325", "MI355") +ALL_DISTRIBUTED_TEST_SUITES = (*GENERIC_DISTRIBUTED_TEST_SUITES, "A100") + + +def _default_target_test_suite() -> str: + if not current_platform.is_rocm(): + return "L4" + + try: + device_name = current_platform.get_device_name().upper() + except Exception: + device_name = "" + + if "MI355" in device_name: + return "MI355" + if "MI300" in device_name: + return "MI300" + if "MI325" in device_name: + return "MI325" + if "MI250" in device_name: + return "MI250" + + try: + from vllm.platforms import rocm as rocm_platform + + if rocm_platform.on_gfx950(): + return "MI355" + if rocm_platform.on_gfx942(): + return "MI300" + except Exception: + pass + + return "MI250" + + +def _resolve_target_test_suite() -> str: + for env_name in (TARGET_TEST_SUITE_ENV, LEGACY_TARGET_TEST_SUITE_ENV): + value = os.environ.get(env_name, "").strip().upper() + if value: + return value + return _default_target_test_suite() + + +TARGET_TEST_SUITE = _resolve_target_test_suite() + + +def test_vllm_gc_ed(): + """Verify vllm instance is GC'ed when it is deleted""" + llm = LLM("hmellor/tiny-random-LlamaForCausalLM") + weak_llm = weakref.ref(llm) + del llm + # If there's any circular reference to vllm, this fails + # because llm instance is not GC'ed. + assert weak_llm() is None + + +def _fix_prompt_embed_outputs( + vllm_outputs: list[tuple[list[int], str]], + hf_model: HfRunner, + example_prompts: list[str], +) -> list[tuple[list[int], str]]: + fixed_vllm_outputs = [] + for vllm_output, hf_input, prompt in zip( + vllm_outputs, hf_model.get_inputs(example_prompts), example_prompts + ): + hf_input_ids = hf_input["input_ids"].tolist()[0] + fixed_vllm_outputs.append( + ( + hf_input_ids + vllm_output[0][len(hf_input_ids) :], + prompt + vllm_output[1], + ) + ) + return fixed_vllm_outputs + + +@pytest.mark.parametrize("model", MODELS) +@pytest.mark.parametrize("backend", ATTN_BACKEND) +@pytest.mark.parametrize("max_tokens", [5]) +@pytest.mark.parametrize("enforce_eager", [False]) +@pytest.mark.parametrize("async_scheduling", [True, False]) +@pytest.mark.parametrize("model_executor", ["uni", "mp"]) +@pytest.mark.parametrize("enable_prompt_embeds", [True, False]) +def test_models( + hf_runner, + model: str, + backend: str, + max_tokens: int, + enforce_eager: bool, + async_scheduling: bool, + model_executor: str, + enable_prompt_embeds: bool, +) -> None: + # 5042 tokens for gemma2 + # gemma2 has alternating sliding window size of 4096 + # we need a prompt with more than 4096 tokens to test the sliding window + prompt = ( + "The following numbers of the sequence " + + ", ".join(str(i) for i in range(1024)) + + " are:" + ) + example_prompts = [prompt] + + with hf_runner(model) as hf_model: + hf_outputs = hf_model.generate_greedy(example_prompts, max_tokens) + if enable_prompt_embeds: + with torch.no_grad(): + prompt_embeds = hf_model.get_prompt_embeddings(example_prompts) + if model == "hmellor/tiny-random-Gemma2ForCausalLM" and ( + Version(TRANSFORMERS_VERSION) < Version("5.3.0.dev0") + ): + # For Gemma 1/2 models with Transformers 5.4.0+, the prompt embeddings + # are normalised in `get_prompt_embeddings`, like Gemma 3. + # For older versions, we need to manually normalise. + embed_scale = hf_model.config.hidden_size**0.5 + normalizer = torch.tensor(embed_scale, dtype=prompt_embeds[0].dtype) + prompt_embeds = [p_e * normalizer for p_e in prompt_embeds] + + with VllmRunner( + model, + max_model_len=8192, + enforce_eager=enforce_eager, + enable_prompt_embeds=enable_prompt_embeds, + gpu_memory_utilization=0.7, + async_scheduling=async_scheduling, + distributed_executor_backend=model_executor, + attention_config={"backend": backend}, + ) as vllm_model: + if enable_prompt_embeds: + vllm_outputs = vllm_model.generate_greedy(prompt_embeds, max_tokens) + vllm_outputs = _fix_prompt_embed_outputs( + vllm_outputs, hf_model, example_prompts + ) + else: + vllm_outputs = vllm_model.generate_greedy(example_prompts, max_tokens) + + check_outputs_equal( + outputs_0_lst=hf_outputs, + outputs_1_lst=vllm_outputs, + name_0="hf", + name_1="vllm", + ) + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize( + ( + "model, distributed_executor_backend, attention_backend, " + "target_test_suites, extra_env" + ), + [ + ("facebook/opt-125m", "ray", "", ALL_DISTRIBUTED_TEST_SUITES, {}), + ("facebook/opt-125m", "mp", "", ALL_DISTRIBUTED_TEST_SUITES, {}), + ( + "meta-llama/Llama-3.2-1B-Instruct", + "ray", + "", + GENERIC_DISTRIBUTED_TEST_SUITES, + {}, + ), + ( + "meta-llama/Llama-3.2-1B-Instruct", + "mp", + "", + GENERIC_DISTRIBUTED_TEST_SUITES, + {}, + ), + ], +) +@pytest.mark.parametrize("enable_prompt_embeds", [True, False]) +def test_models_distributed( + monkeypatch: pytest.MonkeyPatch, + hf_runner, + vllm_runner, + example_prompts, + model: str, + distributed_executor_backend: str, + attention_backend: str, + target_test_suites: tuple[str, ...], + extra_env: dict[str, str], + enable_prompt_embeds: bool, +) -> None: + if TARGET_TEST_SUITE and TARGET_TEST_SUITE not in target_test_suites: + pytest.skip(f"Skip test for {TARGET_TEST_SUITE}") + + with monkeypatch.context() as monkeypatch_context: + if ( + model == "meta-llama/Llama-3.2-1B-Instruct" + and distributed_executor_backend == "ray" + and attention_backend == "" + and TARGET_TEST_SUITE == "L4" + and enable_prompt_embeds + ): # noqa + pytest.skip("enable_prompt_embeds does not work with ray compiled dag.") + + for k, v in extra_env.items(): + monkeypatch_context.setenv(k, v) + + dtype = "half" + max_tokens = 5 + + # NOTE: take care of the order. run vLLM first, and then run HF. + # vLLM needs a fresh new process without cuda initialization. + # if we run HF first, the cuda initialization will be done and it + # will hurt multiprocessing backend with fork method + # (the default method). + attention_config = {"backend": attention_backend} if attention_backend else None + with vllm_runner( + model, + dtype=dtype, + tensor_parallel_size=2, + distributed_executor_backend=distributed_executor_backend, + enable_prompt_embeds=enable_prompt_embeds, + gpu_memory_utilization=0.7, + attention_config=attention_config, + ) as vllm_model: + if enable_prompt_embeds: + with hf_runner(model, dtype=dtype) as hf_model: + with torch.no_grad(): + prompt_embeds = hf_model.get_prompt_embeddings(example_prompts) + vllm_outputs = vllm_model.generate_greedy(prompt_embeds, max_tokens) + vllm_outputs = _fix_prompt_embed_outputs( + vllm_outputs, hf_model, example_prompts + ) + hf_outputs = hf_model.generate_greedy(example_prompts, max_tokens) + else: + vllm_outputs = vllm_model.generate_greedy(example_prompts, max_tokens) + with hf_runner(model, dtype=dtype) as hf_model: + hf_outputs = hf_model.generate_greedy(example_prompts, max_tokens) + + check_outputs_equal( + outputs_0_lst=hf_outputs, + outputs_1_lst=vllm_outputs, + name_0="hf", + name_1="vllm", + ) + + +def test_failed_model_execution(vllm_runner, monkeypatch) -> None: + # Needed to mock an error in the same process + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + + with vllm_runner("facebook/opt-125m", enforce_eager=True) as vllm_model: + if isinstance(vllm_model.llm.llm_engine, LLMEngine): + v1_test_failed_model_execution(vllm_model) + + +def v1_test_failed_model_execution(vllm_model): + engine = vllm_model.llm.llm_engine + mocked_execute_model = Mock(side_effect=RuntimeError("Mocked Critical Error")) + engine.engine_core.engine_core.model_executor.execute_model = mocked_execute_model + + with pytest.raises(RuntimeError) as exc_info: + prompts = [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + "The future of AI is", + ] + vllm_model.generate_greedy(prompts, 200, use_tqdm=False) + assert isinstance(exc_info.value, RuntimeError) + assert "Mocked Critical Error" in str(exc_info.value) diff --git a/asian_ai/performance/vLLM/tests/basic_correctness/test_cpu_offload.py b/asian_ai/performance/vLLM/tests/basic_correctness/test_cpu_offload.py new file mode 100644 index 0000000000000000000000000000000000000000..b49879f3015584061a2a4de7f7e8ce843ebaf88f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/basic_correctness/test_cpu_offload.py @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from ..utils import compare_two_settings + + +@pytest.mark.parametrize("disable_pin_memory", [False, True]) +@pytest.mark.parametrize("disable_uva", [False, True]) +def test_cpu_offload(disable_pin_memory, disable_uva): + env_vars = { + "VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY": str(int(disable_pin_memory)), + "VLLM_WEIGHT_OFFLOADING_DISABLE_UVA": str(int(disable_uva)), + } + + args = ["--cpu-offload-gb", "1"] + + # cuda graph only works with UVA offloading + if disable_uva: + args.append("--enforce-eager") + + compare_two_settings( + model="hmellor/tiny-random-LlamaForCausalLM", + arg1=[], + arg2=args, + env1=None, + env2=env_vars, + ) diff --git a/asian_ai/performance/vLLM/tests/basic_correctness/test_mem.py b/asian_ai/performance/vLLM/tests/basic_correctness/test_mem.py new file mode 100644 index 0000000000000000000000000000000000000000..9df8a130706566100027bef44469332a29945c9c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/basic_correctness/test_mem.py @@ -0,0 +1,283 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import asyncio + +import pytest +import torch + +from vllm import LLM, AsyncEngineArgs, AsyncLLMEngine, SamplingParams +from vllm.device_allocator import get_mem_allocator_instance +from vllm.platforms import current_platform +from vllm.utils.mem_constants import GiB_bytes + +from ..utils import create_new_process_for_each_test, requires_fp8 + +DEVICE_TYPE = current_platform.device_type + + +@create_new_process_for_each_test("fork" if current_platform.is_cuda() else "spawn") +def test_python_error(): + """ + Test if Python error occurs when there's low-level + error happening from the C++ side. + """ + allocator = get_mem_allocator_instance() + total_bytes = torch.accelerator.get_memory_info()[1] + alloc_bytes = int(total_bytes * 0.7) + tensors = [] + with allocator.use_memory_pool(): + # allocate 70% of the total memory + x = torch.empty(alloc_bytes, dtype=torch.uint8, device=DEVICE_TYPE) + tensors.append(x) + # release the memory + allocator.sleep() + + # allocate more memory than the total memory + y = torch.empty(alloc_bytes, dtype=torch.uint8, device=DEVICE_TYPE) + tensors.append(y) + with pytest.raises(RuntimeError): + # when the allocator is woken up, it should raise an error + # because we don't have enough memory + allocator.wake_up() + + +@create_new_process_for_each_test("fork" if current_platform.is_cuda() else "spawn") +def test_basic_cumem(): + # some tensors from default memory pool + shape = (1024, 1024) + x = torch.empty(shape, device=DEVICE_TYPE) + x.zero_() + + # some tensors from custom memory pool + allocator = get_mem_allocator_instance() + with allocator.use_memory_pool(): + # custom memory pool + y = torch.empty(shape, device=DEVICE_TYPE) + y.zero_() + y += 1 + z = torch.empty(shape, device=DEVICE_TYPE) + z.zero_() + z += 2 + + # they can be used together + output = x + y + z + assert torch.allclose(output, torch.ones_like(output) * 3) + + free_bytes = torch.accelerator.get_memory_info()[0] + allocator.sleep() + free_bytes_after_sleep = torch.accelerator.get_memory_info()[0] + assert free_bytes_after_sleep > free_bytes + allocator.wake_up() + + # they can be used together + output = x + y + z + assert torch.allclose(output, torch.ones_like(output) * 3) + + +@create_new_process_for_each_test("fork" if current_platform.is_cuda() else "spawn") +@pytest.mark.skipif(current_platform.is_xpu(), reason="CUDA graph not supported on XPU") +def test_cumem_with_cudagraph(): + allocator = get_mem_allocator_instance() + with allocator.use_memory_pool(): + weight = torch.eye(1024, device=DEVICE_TYPE) + with allocator.use_memory_pool(tag="discard"): + cache = torch.empty(1024, 1024, device=DEVICE_TYPE) + + def model(x): + out = x @ weight + cache[: out.size(0)].copy_(out) + return out + 1 + + x = torch.empty(128, 1024, device=DEVICE_TYPE) + + # warmup + model(x) + + # capture cudagraph + model_graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(model_graph): + y = model(x) + + free_bytes = torch.accelerator.get_memory_info()[0] + allocator.sleep() + free_bytes_after_sleep = torch.accelerator.get_memory_info()[0] + assert free_bytes_after_sleep > free_bytes + allocator.wake_up() + + # after waking up, the content in the weight tensor + # should be restored, but the content in the cache tensor + # should be discarded + + # this operation is also compatible with cudagraph + + x.random_() + model_graph.replay() + + # cache content is as expected + assert torch.allclose(x, cache[: x.size(0)]) + + # output content is as expected + assert torch.allclose(y, x + 1) + + +@create_new_process_for_each_test("fork" if current_platform.is_cuda() else "spawn") +@pytest.mark.parametrize( + "model", + [ + # sleep mode with safetensors + "hmellor/tiny-random-LlamaForCausalLM", + # sleep mode with pytorch checkpoint + "facebook/opt-125m", + ], +) +def test_end_to_end(model: str): + free, total = torch.accelerator.get_memory_info() + used_bytes_baseline = total - free # in case other process is running + llm = LLM(model, enable_sleep_mode=True) + prompt = "How are you?" + sampling_params = SamplingParams(temperature=0, max_tokens=10) + output = llm.generate(prompt, sampling_params) + + # the benefit of `llm.sleep(level=2)` is mainly CPU memory usage, + # which is difficult to measure in the test. therefore, we only + # test sleep level 1 here. + llm.sleep(level=1) + + free_gpu_bytes_after_sleep, total = torch.accelerator.get_memory_info() + used_bytes = total - free_gpu_bytes_after_sleep - used_bytes_baseline + # now the memory usage is mostly cudagraph memory pool, + # and it should be less than the model weights (1B model, 2GiB weights) + + # NOTE: In V1, the memory buffer for logits (max_num_reqs x vocab_size) + # is captured but cannot be releasesd from PyTorch due to a known bug, + # therefore high memory usage after `llm.sleep` is called is expected. + # FIXME(youkaichao & ywang96): Fix memory buffer issue with sleep mode + # in V1. + assert used_bytes < 7 * GiB_bytes + + llm.wake_up() + output2 = llm.generate(prompt, sampling_params) + # cmp output + assert output[0].outputs[0].text == output2[0].outputs[0].text + + llm.sleep(level=1) + llm.wake_up(tags=["weights"]) + + free_gpu_bytes_wake_up_w, total = torch.accelerator.get_memory_info() + used_bytes = total - free_gpu_bytes_wake_up_w - used_bytes_baseline + + # should just reallocate memory for weights (1B model, ~2GiB weights) + assert used_bytes < 10 * GiB_bytes + + # now allocate kv cache memory + llm.wake_up(tags=["kv_cache"]) + output3 = llm.generate(prompt, sampling_params) + + # cmp output + assert output[0].outputs[0].text == output3[0].outputs[0].text + + +@create_new_process_for_each_test() +def test_deep_sleep(): + model = "hmellor/tiny-random-LlamaForCausalLM" + free, total = torch.accelerator.get_memory_info() + used_bytes_baseline = total - free # in case other process is running + llm = LLM(model, enable_sleep_mode=True) + prompt = "How are you?" + sampling_params = SamplingParams(temperature=0, max_tokens=10) + output = llm.generate(prompt, sampling_params) + + # Put the engine to deep sleep + llm.sleep(level=2) + + free_gpu_bytes_after_sleep, total = torch.accelerator.get_memory_info() + used_bytes = total - free_gpu_bytes_after_sleep - used_bytes_baseline + assert used_bytes < 3 * GiB_bytes + + llm.wake_up(tags=["weights"]) + llm.collective_rpc("reload_weights") + free_gpu_bytes_wake_up_w, total = torch.accelerator.get_memory_info() + used_bytes = total - free_gpu_bytes_wake_up_w - used_bytes_baseline + assert used_bytes < 4 * GiB_bytes + + # now allocate kv cache and cuda graph memory + llm.wake_up(tags=["kv_cache"]) + output2 = llm.generate(prompt, sampling_params) + + # cmp output + assert output[0].outputs[0].text == output2[0].outputs[0].text + + +@create_new_process_for_each_test() +def test_deep_sleep_async(): + async def test(): + model = "hmellor/tiny-random-LlamaForCausalLM" + free, total = torch.accelerator.get_memory_info() + used_bytes_baseline = total - free # in case other process is running + engine_args = AsyncEngineArgs( + model=model, + enable_sleep_mode=True, + ) + + llm = AsyncLLMEngine.from_engine_args(engine_args) + prompt = "How are you?" + sampling_params = SamplingParams(temperature=0, max_tokens=10) + outputs = llm.generate(prompt, sampling_params, request_id="test_request_id1") + async for output in outputs: + pass + + # Put the engine to deep sleep + await llm.sleep(level=2) + + await llm.wake_up(tags=["weights"]) + await llm.collective_rpc("reload_weights") + free_gpu_bytes_wake_up_w, total = torch.accelerator.get_memory_info() + used_bytes = total - free_gpu_bytes_wake_up_w - used_bytes_baseline + assert used_bytes < 4 * GiB_bytes + + # now allocate kv cache and cuda graph memory + await llm.wake_up(tags=["kv_cache"]) + outputs2 = llm.generate(prompt, sampling_params, request_id="test_request_id2") + async for output2 in outputs2: + pass + + # cmp output + assert output.outputs[0].text == output2.outputs[0].text + + asyncio.run(test()) + + +@requires_fp8 +def test_deep_sleep_fp8_kvcache(): + model = "Qwen/Qwen2-0.5B" + used_bytes_baseline = current_platform.get_current_memory_usage() + + llm = LLM(model, enable_sleep_mode=True, kv_cache_dtype="fp8") + prompt = "How are you?" + sampling_params = SamplingParams(temperature=0, max_tokens=10) + output = llm.generate(prompt, sampling_params) + + # Put the engine to deep sleep + llm.sleep(level=2) + + used_bytes = current_platform.get_current_memory_usage() - used_bytes_baseline + + # Rocm uses more memory for CudaGraphs, so we add 2 GiB more for the threshold + rocm_extra_mem_bytes = 2 * GiB_bytes if current_platform.is_rocm() else 0 + mem_threshold_after_sleep = 3 * GiB_bytes + rocm_extra_mem_bytes + assert used_bytes < mem_threshold_after_sleep + + llm.wake_up(tags=["weights"]) + llm.collective_rpc("reload_weights") + + used_bytes = current_platform.get_current_memory_usage() - used_bytes_baseline + mem_threshold_after_wake_up = 4 * GiB_bytes + rocm_extra_mem_bytes + assert used_bytes < mem_threshold_after_wake_up + + # now allocate kv cache and cuda graph memory + llm.wake_up(tags=["kv_cache"]) + output2 = llm.generate(prompt, sampling_params) + + # cmp output + assert output[0].outputs[0].text == output2[0].outputs[0].text diff --git a/asian_ai/performance/vLLM/tests/basic_correctness/test_prefetch_offload.py b/asian_ai/performance/vLLM/tests/basic_correctness/test_prefetch_offload.py new file mode 100644 index 0000000000000000000000000000000000000000..4c0e390a04621426044cb78f47180b84900f051d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/basic_correctness/test_prefetch_offload.py @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Test prefetch offloading correctness with Llama model.""" + +from ..utils import compare_two_settings + + +def test_prefetch_offload_llama(): + """Test prefetch CPU offloading with Llama-3.2-1B-Instruct. + + Compares outputs between: + 1. Baseline (no offloading) + 2. Prefetch offloading (group_size=8, num_in_group=2, prefetch_step=1) + + This tests prefetching-based offloading on a dense model. + """ + compare_two_settings( + "meta-llama/Llama-3.2-1B-Instruct", + [ + # Prefetch offloading configuration + "--offload-group-size", + "8", + "--offload-num-in-group", + "2", + "--offload-prefetch-step", + "1", + # Selective offloading: only MLP weights + "--offload-params", + "gate_up_proj", + "down_proj", + ], + [], # Baseline: no offloading + ) diff --git a/asian_ai/performance/vLLM/tests/benchmarks/__init__.py b/asian_ai/performance/vLLM/tests/benchmarks/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/benchmarks/sweep/__init__.py b/asian_ai/performance/vLLM/tests/benchmarks/sweep/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/benchmarks/sweep/test_param_sweep.py b/asian_ai/performance/vLLM/tests/benchmarks/sweep/test_param_sweep.py new file mode 100644 index 0000000000000000000000000000000000000000..4c2caa7d08c350848185c8a6cdad3af8faeafb8a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/sweep/test_param_sweep.py @@ -0,0 +1,249 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import json +import tempfile +from pathlib import Path + +import pytest + +from vllm.benchmarks.sweep.param_sweep import ParameterSweep, ParameterSweepItem + + +class TestParameterSweepItem: + """Test ParameterSweepItem functionality.""" + + @pytest.mark.parametrize( + "input_dict,expected", + [ + ( + {"compilation_config.use_inductor_graph_partition": False}, + "--compilation-config.use_inductor_graph_partition=false", + ), + ( + {"compilation_config.use_inductor_graph_partition": True}, + "--compilation-config.use_inductor_graph_partition=true", + ), + ], + ) + def test_nested_boolean_params(self, input_dict, expected): + """Test that nested boolean params use =true/false syntax.""" + item = ParameterSweepItem.from_record(input_dict) + cmd = item.apply_to_cmd(["vllm", "serve", "model"]) + assert expected in cmd + + @pytest.mark.parametrize( + "input_dict,expected", + [ + ({"enable_prefix_caching": False}, "--no-enable-prefix-caching"), + ({"enable_prefix_caching": True}, "--enable-prefix-caching"), + ({"disable_log_stats": False}, "--no-disable-log-stats"), + ({"disable_log_stats": True}, "--disable-log-stats"), + ], + ) + def test_non_nested_boolean_params(self, input_dict, expected): + """Test that non-nested boolean params use --no- prefix.""" + item = ParameterSweepItem.from_record(input_dict) + cmd = item.apply_to_cmd(["vllm", "serve", "model"]) + assert expected in cmd + + @pytest.mark.parametrize( + "compilation_config", + [ + {"cudagraph_mode": "full", "mode": 2, "use_inductor_graph_partition": True}, + { + "cudagraph_mode": "piecewise", + "mode": 3, + "use_inductor_graph_partition": False, + }, + ], + ) + def test_nested_dict_value(self, compilation_config): + """Test that nested dict values are serialized as JSON.""" + item = ParameterSweepItem.from_record( + {"compilation_config": compilation_config} + ) + cmd = item.apply_to_cmd(["vllm", "serve", "model"]) + assert "--compilation-config" in cmd + # The dict should be JSON serialized + idx = cmd.index("--compilation-config") + assert json.loads(cmd[idx + 1]) == compilation_config + + @pytest.mark.parametrize( + "input_dict,expected_key,expected_value", + [ + ({"model": "test-model"}, "--model", "test-model"), + ({"max_tokens": 100}, "--max-tokens", "100"), + ({"temperature": 0.7}, "--temperature", "0.7"), + ], + ) + def test_string_and_numeric_values(self, input_dict, expected_key, expected_value): + """Test that string and numeric values are handled correctly.""" + item = ParameterSweepItem.from_record(input_dict) + cmd = item.apply_to_cmd(["vllm", "serve"]) + assert expected_key in cmd + assert expected_value in cmd + + @pytest.mark.parametrize( + "input_dict,expected_key,key_idx_offset", + [ + ({"max_tokens": 200}, "--max-tokens", 1), + ({"enable_prefix_caching": False}, "--no-enable-prefix-caching", 0), + ], + ) + def test_replace_existing_parameter(self, input_dict, expected_key, key_idx_offset): + """Test that existing parameters in cmd are replaced.""" + item = ParameterSweepItem.from_record(input_dict) + + if key_idx_offset == 1: + # Key-value pair + cmd = item.apply_to_cmd(["vllm", "serve", "--max-tokens", "100", "model"]) + assert expected_key in cmd + idx = cmd.index(expected_key) + assert cmd[idx + 1] == "200" + assert "100" not in cmd + else: + # Boolean flag + cmd = item.apply_to_cmd( + ["vllm", "serve", "--enable-prefix-caching", "model"] + ) + assert expected_key in cmd + assert "--enable-prefix-caching" not in cmd + + +class TestParameterSweep: + """Test ParameterSweep functionality.""" + + def test_from_records_list(self): + """Test creating ParameterSweep from a list of records.""" + records = [ + {"max_tokens": 100, "temperature": 0.7}, + {"max_tokens": 200, "temperature": 0.9}, + ] + sweep = ParameterSweep.from_records(records) + assert len(sweep) == 2 + assert sweep[0]["max_tokens"] == 100 + assert sweep[1]["max_tokens"] == 200 + + def test_read_from_dict(self): + """Test creating ParameterSweep from a dict format.""" + data = { + "experiment1": {"max_tokens": 100, "temperature": 0.7}, + "experiment2": {"max_tokens": 200, "temperature": 0.9}, + } + sweep = ParameterSweep.read_from_dict(data) + assert len(sweep) == 2 + + # Check that items have the _benchmark_name field + names = {item["_benchmark_name"] for item in sweep} + assert names == {"experiment1", "experiment2"} + + # Check that parameters are preserved + for item in sweep: + if item["_benchmark_name"] == "experiment1": + assert item["max_tokens"] == 100 + assert item["temperature"] == 0.7 + elif item["_benchmark_name"] == "experiment2": + assert item["max_tokens"] == 200 + assert item["temperature"] == 0.9 + + def test_read_json_list_format(self): + """Test reading JSON file with list format.""" + records = [ + {"max_tokens": 100, "temperature": 0.7}, + {"max_tokens": 200, "temperature": 0.9}, + ] + + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f: + json.dump(records, f) + temp_path = Path(f.name) + + try: + sweep = ParameterSweep.read_json(temp_path) + assert len(sweep) == 2 + assert sweep[0]["max_tokens"] == 100 + assert sweep[1]["max_tokens"] == 200 + finally: + temp_path.unlink() + + def test_read_json_dict_format(self): + """Test reading JSON file with dict format.""" + data = { + "experiment1": {"max_tokens": 100, "temperature": 0.7}, + "experiment2": {"max_tokens": 200, "temperature": 0.9}, + } + + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f: + json.dump(data, f) + temp_path = Path(f.name) + + try: + sweep = ParameterSweep.read_json(temp_path) + assert len(sweep) == 2 + + # Check that items have the _benchmark_name field + names = {item["_benchmark_name"] for item in sweep} + assert names == {"experiment1", "experiment2"} + finally: + temp_path.unlink() + + def test_unique_benchmark_names_validation(self): + """Test that duplicate _benchmark_name values raise an error.""" + # Test with duplicate names in list format + records = [ + {"_benchmark_name": "exp1", "max_tokens": 100}, + {"_benchmark_name": "exp1", "max_tokens": 200}, + ] + + with pytest.raises(ValueError, match="Duplicate _benchmark_name values"): + ParameterSweep.from_records(records) + + def test_unique_benchmark_names_multiple_duplicates(self): + """Test validation with multiple duplicate names.""" + records = [ + {"_benchmark_name": "exp1", "max_tokens": 100}, + {"_benchmark_name": "exp1", "max_tokens": 200}, + {"_benchmark_name": "exp2", "max_tokens": 300}, + {"_benchmark_name": "exp2", "max_tokens": 400}, + ] + + with pytest.raises(ValueError, match="Duplicate _benchmark_name values"): + ParameterSweep.from_records(records) + + def test_no_benchmark_names_allowed(self): + """Test that records without _benchmark_name are allowed.""" + records = [ + {"max_tokens": 100, "temperature": 0.7}, + {"max_tokens": 200, "temperature": 0.9}, + ] + sweep = ParameterSweep.from_records(records) + assert len(sweep) == 2 + + def test_mixed_benchmark_names_allowed(self): + """Test that mixing records with and without _benchmark_name is allowed.""" + records = [ + {"_benchmark_name": "exp1", "max_tokens": 100}, + {"max_tokens": 200, "temperature": 0.9}, + ] + sweep = ParameterSweep.from_records(records) + assert len(sweep) == 2 + + +class TestParameterSweepItemKeyNormalization: + """Test key normalization in ParameterSweepItem.""" + + def test_underscore_to_hyphen_conversion(self): + """Test that underscores are converted to hyphens in CLI.""" + item = ParameterSweepItem.from_record({"max_tokens": 100}) + cmd = item.apply_to_cmd(["vllm", "serve"]) + assert "--max-tokens" in cmd + + def test_nested_key_preserves_suffix(self): + """Test that nested keys preserve the suffix format.""" + # The suffix after the dot should preserve underscores + item = ParameterSweepItem.from_record( + {"compilation_config.some_nested_param": "value"} + ) + cmd = item.apply_to_cmd(["vllm", "serve"]) + # The prefix (compilation_config) gets converted to hyphens, + # but the suffix (some_nested_param) is preserved + assert any("compilation-config.some_nested_param" in arg for arg in cmd) diff --git a/asian_ai/performance/vLLM/tests/benchmarks/test_audio_dataset.py b/asian_ai/performance/vLLM/tests/benchmarks/test_audio_dataset.py new file mode 100644 index 0000000000000000000000000000000000000000..5ecacf2c925436782a0b172d2e187214892da44f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/test_audio_dataset.py @@ -0,0 +1,271 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import asyncio +from pathlib import Path +from typing import Protocol, cast + +import numpy as np +import pytest +import soundfile as sf + +import vllm.benchmarks.datasets.datasets as datasets_module +import vllm.benchmarks.lib.endpoint_request_func as request_func_module +from vllm.benchmarks.lib.endpoint_request_func import RequestFuncInput + +pytestmark = pytest.mark.skip_global_cleanup + + +class _ReadableBinary(Protocol): + def read(self, size: int = -1) -> bytes: ... + + +class _TokenizedPrompt: + def __init__(self, prompt: str) -> None: + self.input_ids = prompt.split() + + +class _Tokenizer: + def __init__(self, name_or_path: str = "openai/whisper-large-v3") -> None: + self.name_or_path = name_or_path + + def __call__(self, prompt: str) -> _TokenizedPrompt: + return _TokenizedPrompt(prompt) + + +class CohereAsrTokenizer(_Tokenizer): + def __init__(self, name_or_path: str = "/models/cohere-transcribe") -> None: + super().__init__(name_or_path) + + +class _CohereNameOnlyTokenizer(_Tokenizer): + def __init__(self) -> None: + super().__init__("cohere/some-local-checkpoint") + + +def _write_wav(path: Path, duration_s: float = 0.1, sample_rate: int = 16_000) -> None: + num_samples = int(duration_s * sample_rate) + sf.write(path, np.zeros(num_samples, dtype=np.float32), sample_rate) + + +class _FakeFormData: + def __init__(self) -> None: + self.fields: list[tuple[str, object, dict[str, str]]] = [] + + def add_field(self, name: str, value: object, **kwargs: str) -> None: + self.fields.append((name, value, kwargs)) + + +class _FakeContent: + async def iter_any(self): + yield b'data: {"choices":[{"delta":{"content":"hello"}}]}\n\n' + yield b'data: {"usage":{"completion_tokens":1}}\n\n' + yield b"data: [DONE]\n\n" + + +class _FakeResponse: + def __init__(self) -> None: + self.status = 200 + self.reason = "OK" + self.content = _FakeContent() + + async def __aenter__(self): + return self + + async def __aexit__(self, exc_type, exc, tb): + return False + + +class _FakeSession: + def __init__(self) -> None: + self.uploaded_bytes: bytes | None = None + self.upload_filename: str | None = None + self.fields: list[tuple[str, object, dict[str, str]]] | None = None + + def post(self, *, url: str, data: _FakeFormData, headers: dict[str, str]): + del url, headers + self.fields = list(data.fields) + _, file_obj, file_kwargs = self.fields[0] + file_obj = cast(_ReadableBinary, file_obj) + self.uploaded_bytes = file_obj.read() + self.upload_filename = file_kwargs.get("filename") + return _FakeResponse() + + +def test_asr_dataset_sample_handles_local_audio_paths(tmp_path: Path) -> None: + audio_path = tmp_path / "earnings.wav" + _write_wav(audio_path, duration_s=0.1) + + dataset = object.__new__(datasets_module.ASRDataset) + dataset.data = [ + { + "audio": { + "path": str(audio_path), + "bytes": None, + }, + "text": "quarterly earnings call", + } + ] + + samples = dataset.sample( + tokenizer=_Tokenizer(), + num_requests=1, + output_len=32, + asr_min_audio_len_sec=0.0, + asr_max_audio_len_sec=1.0, + ) + + assert len(samples) == 1 + assert samples[0].multi_modal_data == {"audio_path": str(audio_path)} + assert ( + samples[0].prompt == "<|startoftranscript|><|en|><|transcribe|><|notimestamps|>" + ) + + +def test_asr_dataset_sample_handles_embedded_audio_bytes(tmp_path: Path) -> None: + audio_path = tmp_path / "earnings.wav" + _write_wav(audio_path, duration_s=0.1) + + dataset = object.__new__(datasets_module.ASRDataset) + dataset.data = [ + { + "audio": { + "path": None, + "bytes": audio_path.read_bytes(), + }, + "text": "quarterly earnings call", + } + ] + + samples = dataset.sample( + tokenizer=_Tokenizer(), + num_requests=1, + output_len=32, + asr_min_audio_len_sec=0.0, + asr_max_audio_len_sec=1.0, + ) + + assert len(samples) == 1 + assert isinstance(samples[0].multi_modal_data, dict) + audio, sample_rate = samples[0].multi_modal_data["audio"] + assert sample_rate == 16_000 + assert isinstance(audio, np.ndarray) + assert audio.size > 0 + + +def test_async_request_openai_audio_handles_local_audio_paths( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + audio_path = tmp_path / "earnings.wav" + _write_wav(audio_path, duration_s=0.25) + + monkeypatch.setattr(request_func_module.aiohttp, "FormData", _FakeFormData) + session = _FakeSession() + request_input = RequestFuncInput( + prompt="", + api_url="http://localhost:8000/v1/audio/transcriptions", + prompt_len=1, + output_len=32, + model="openai/whisper-large-v3", + multi_modal_content={"audio_path": str(audio_path)}, + ) + + output = asyncio.run( + request_func_module.async_request_openai_audio(request_input, session) + ) + + assert session.upload_filename == audio_path.name + assert session.uploaded_bytes == audio_path.read_bytes() + assert output.success is True + assert output.generated_text == "hello" + assert output.output_tokens == 1 + assert output.input_audio_duration == pytest.approx(0.25, abs=1e-2) + + +def test_async_request_openai_audio_handles_decoded_audio_arrays( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(request_func_module.aiohttp, "FormData", _FakeFormData) + session = _FakeSession() + request_input = RequestFuncInput( + prompt="", + api_url="http://localhost:8000/v1/audio/transcriptions", + prompt_len=1, + output_len=32, + model="openai/whisper-large-v3", + multi_modal_content={ + "audio": (np.zeros(1_600, dtype=np.float32), 16_000), + }, + ) + + output = asyncio.run( + request_func_module.async_request_openai_audio(request_input, session) + ) + + assert session.upload_filename == "audio.wav" + assert session.uploaded_bytes is not None + assert output.success is True + assert output.generated_text == "hello" + + +_COHERE_ASR_PROMPT = ( + "<|startofcontext|><|startoftranscript|>" + "<|emo:undefined|><|en|><|en|><|pnc|><|noitn|>" + "<|notimestamp|><|nodiarize|>" +) + + +def _make_asr_dataset(tmp_path: Path) -> datasets_module.ASRDataset: + audio_path = tmp_path / "sample.wav" + _write_wav(audio_path, duration_s=0.1) + dataset = object.__new__(datasets_module.ASRDataset) + dataset.data = [ + { + "audio": {"path": str(audio_path), "bytes": None}, + "text": "hello world", + } + ] + return dataset + + +def test_asr_dataset_cohere_class_name_gets_decoder_prompt(tmp_path: Path) -> None: + dataset = _make_asr_dataset(tmp_path) + samples = dataset.sample( + tokenizer=CohereAsrTokenizer(), + num_requests=1, + output_len=32, + asr_min_audio_len_sec=0.0, + asr_max_audio_len_sec=1.0, + ) + assert len(samples) == 1 + assert samples[0].prompt == _COHERE_ASR_PROMPT + + +def test_asr_dataset_cohere_name_or_path_fallback_gets_decoder_prompt( + tmp_path: Path, +) -> None: + dataset = _make_asr_dataset(tmp_path) + samples = dataset.sample( + tokenizer=_CohereNameOnlyTokenizer(), + num_requests=1, + output_len=32, + asr_min_audio_len_sec=0.0, + asr_max_audio_len_sec=1.0, + ) + assert len(samples) == 1 + assert samples[0].prompt == _COHERE_ASR_PROMPT + + +def test_asr_dataset_unknown_tokenizer_gets_empty_prompt(tmp_path: Path) -> None: + dataset = _make_asr_dataset(tmp_path) + samples = dataset.sample( + tokenizer=_Tokenizer(name_or_path="some-other/asr-model"), + num_requests=1, + output_len=32, + asr_min_audio_len_sec=0.0, + asr_max_audio_len_sec=1.0, + ) + assert len(samples) == 1 + assert samples[0].prompt == "" diff --git a/asian_ai/performance/vLLM/tests/benchmarks/test_bench_startup.py b/asian_ai/performance/vLLM/tests/benchmarks/test_bench_startup.py new file mode 100644 index 0000000000000000000000000000000000000000..efae19e9cfd074eb9f949290face99da0756da4c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/test_bench_startup.py @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import subprocess + +import pytest + + +@pytest.mark.benchmark +def test_bench_startup(): + command = [ + "vllm", + "bench", + "startup", + ] + result = subprocess.run(command, capture_output=True, text=True) + print(result.stdout) + print(result.stderr) + + assert result.returncode == 0, f"Benchmark failed: {result.stderr}" diff --git a/asian_ai/performance/vLLM/tests/benchmarks/test_bfcl_dataset.py b/asian_ai/performance/vLLM/tests/benchmarks/test_bfcl_dataset.py new file mode 100644 index 0000000000000000000000000000000000000000..50b8b09c57d126107ef59a29ca89ed9e234a49df --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/test_bfcl_dataset.py @@ -0,0 +1,341 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import argparse +import json +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest +from transformers import AutoTokenizer, PreTrainedTokenizerBase + +from vllm.benchmarks.datasets import BFCLDataset, get_samples + + +def _patch_hf_api(side_effect): + """Return a patch context that swaps `hf_api()` to a stub whose + `.hf_hub_download` attribute uses `side_effect`.""" + fake_api = MagicMock() + fake_api.hf_hub_download.side_effect = side_effect + return patch("vllm.benchmarks.datasets.datasets.hf_api", return_value=fake_api) + + +@pytest.fixture(scope="session") +def hf_tokenizer() -> PreTrainedTokenizerBase: + return AutoTokenizer.from_pretrained("gpt2") + + +_FAKE_ROWS = { + "simple": [ + { + "id": "simple_0", + "question": [ + [ + { + "role": "user", + "content": "What is 2+2?", + } + ] + ], + "function": [ + { + "name": "add", + "description": "Add two numbers.", + "parameters": { + "type": "dict", + "properties": { + "a": {"type": "integer", "description": "first"}, + "b": {"type": "float", "description": "second"}, + }, + "required": ["a", "b"], + }, + } + ], + }, + ], + "live_simple": [ + { + "id": "live_simple_0", + "question": [[{"role": "user", "content": "Tell me the weather."}]], + "function": [ + { + "name": "get_weather", + "description": "Get weather.", + "parameters": { + "type": "dict", + "properties": { + "city": {"type": "any", "description": "city"}, + "coords": {"type": "tuple", "description": "coords"}, + }, + "required": ["city"], + }, + } + ], + }, + ], +} + + +def _write_fake_files(tmp_path: Path) -> dict[str, Path]: + """Write fake BFCL JSONL files mimicking the HF repo layout.""" + paths = {} + for category, rows in _FAKE_ROWS.items(): + p = tmp_path / f"BFCL_v3_{category}.json" + with p.open("w") as f: + for row in rows: + f.write(json.dumps(row) + "\n") + paths[category] = p + return paths + + +def _args_for_bfcl(categories: list[str] | None) -> argparse.Namespace: + return argparse.Namespace( + dataset_name="hf", + dataset_path="gorilla-llm/Berkeley-Function-Calling-Leaderboard", + hf_name=None, + hf_subset=None, + hf_split=None, + hf_output_len=64, + disable_shuffle=True, + num_prompts=2, + no_oversample=False, + no_stream=True, + seed=0, + request_id_prefix="", + trust_remote_code=False, + skip_chat_template=False, + enable_multimodal_chat=False, + backend="openai-chat", + bfcl_categories=categories, + ) + + +@pytest.mark.benchmark +def test_bfcl_dataset_translates_schema_and_attaches_tools( + hf_tokenizer: PreTrainedTokenizerBase, tmp_path: Path +) -> None: + """BFCLDataset should translate schemas to OpenAI tool format, set + `messages` directly on SampleRequest, and attach tools/tool_choice via + request_overrides.""" + paths = _write_fake_files(tmp_path) + + def fake_download(_repo, filename, **_kwargs): + category = filename.removeprefix("BFCL_v3_").removesuffix(".json") + return str(paths[category]) + + args = _args_for_bfcl(categories=["simple", "live_simple"]) + + with _patch_hf_api(fake_download): + samples = get_samples(args, hf_tokenizer) + + assert len(samples) == 2 + for s in samples: + assert s.chat_messages is not None + assert isinstance(s.chat_messages, list) + assert s.chat_messages[0]["role"] == "user" + assert s.request_overrides is not None + assert "tools" in s.request_overrides + assert s.request_overrides["tool_choice"] == "auto" + # messages must NOT leak into request_overrides — it has its own + # typed field on SampleRequest. + assert "messages" not in s.request_overrides + tools = s.request_overrides["tools"] + assert len(tools) == 1 + tool = tools[0] + assert tool["type"] == "function" + # Translated schema: dict -> object, float -> number, + # any -> string, tuple -> array. + params = tool["function"]["parameters"] + assert params["type"] == "object" + for prop in params["properties"].values(): + assert prop["type"] in {"integer", "number", "string", "array"} + + +@pytest.mark.benchmark +def test_bfcl_dataset_requires_openai_chat_backend( + hf_tokenizer: PreTrainedTokenizerBase, +) -> None: + args = _args_for_bfcl(categories=["simple"]) + args.backend = "openai" + + with pytest.raises(ValueError, match="openai-chat"): + get_samples(args, hf_tokenizer) + + +@pytest.mark.benchmark +def test_bfcl_dataset_missing_category_raises_clear_error( + hf_tokenizer: PreTrainedTokenizerBase, +) -> None: + """A typo'd category should produce an actionable ValueError, not an + opaque huggingface_hub exception.""" + from huggingface_hub.errors import EntryNotFoundError + + args = _args_for_bfcl(categories=["simpl"]) # typo + + def raise_missing(_repo, filename, **_kwargs): + raise EntryNotFoundError(f"404 Not Found: {filename}") + + with ( + _patch_hf_api(raise_missing), + pytest.raises(ValueError, match=r"BFCL category 'simpl' not found"), + ): + get_samples(args, hf_tokenizer) + + +@pytest.mark.benchmark +def test_chat_backend_uses_messages_field_when_set() -> None: + """When RequestFuncInput.chat_messages is set, the chat backend must use + it verbatim and skip default content construction from `prompt`.""" + import asyncio + + from vllm.benchmarks.lib.endpoint_request_func import ( + RequestFuncInput, + async_request_openai_chat_completions, + ) + + captured: dict = {} + + class _FakeResp: + status = 500 + reason = "stop-after-capture" + content = None + + async def __aenter__(self): + return self + + async def __aexit__(self, *exc): + return False + + class _FakeSession: + def post(self, url, json, headers): # noqa: A002 + captured["url"] = url + captured["payload"] = json + return _FakeResp() + + messages = [ + {"role": "system", "content": "you are helpful"}, + {"role": "user", "content": "call add(3, 4)"}, + ] + req = RequestFuncInput( + prompt="IGNORED", + api_url="http://localhost:0/v1/chat/completions", + prompt_len=10, + output_len=16, + model="test-model", + chat_messages=messages, + extra_body={"tools": [{"type": "function", "function": {"name": "add"}}]}, + ) + + asyncio.run( + async_request_openai_chat_completions( + request_func_input=req, session=_FakeSession() + ) + ) + + payload = captured["payload"] + assert payload["messages"] is messages, ( + "chat backend must forward RequestFuncInput.chat_messages verbatim " + "instead of constructing a default user message from `prompt`" + ) + # extra_body still merges in as before (shallow, per-request wins). + assert payload["tools"][0]["function"]["name"] == "add" + + +@pytest.mark.benchmark +def test_bfcl_prompt_len_includes_tools(tmp_path: Path) -> None: + """prompt_len must reflect tokens from both messages *and* tool schemas, + so percentile buckets and input-distribution summaries aren't biased + low for tool-heavy traffic.""" + paths = _write_fake_files(tmp_path) + + def fake_download(_repo, filename, **_kwargs): + category = filename.removeprefix("BFCL_v3_").removesuffix(".json") + return str(paths[category]) + + captured: dict = {} + + class _FakeTokenizer: + def apply_chat_template( + self, messages, tools=None, tokenize=False, add_generation_prompt=True + ): + captured["tools"] = tools + base = " ".join(m.get("content", "") for m in messages) + tool_text = json.dumps(tools) if tools else "" + return base + " " + tool_text + + def __call__(self, text): + # 1 "token" per whitespace-separated word. + return type("Enc", (), {"input_ids": text.split()})() + + fake = _FakeTokenizer() + args = _args_for_bfcl(categories=["simple"]) + args.num_prompts = 1 + + with _patch_hf_api(fake_download): + samples = get_samples(args, fake) + + assert len(samples) == 1 + assert captured["tools"] is not None, ( + "apply_chat_template must be called with tools= so the schema " + "contributes to the prompt-length estimate" + ) + assert len(captured["tools"]) == 1 + assert captured["tools"][0]["function"]["name"] == "add" + + # Sanity: prompt_len exceeds a messages-only estimate. The fake row's + # user message is "What is 2+2?" (3 whitespace-separated tokens). + assert samples[0].prompt_len > 3 + + +@pytest.mark.benchmark +def test_bfcl_prompt_len_falls_back_when_tokenizer_rejects_tools( + tmp_path: Path, +) -> None: + """Older tokenizers don't accept tools=; fallback must still produce a + non-zero prompt_len without crashing.""" + paths = _write_fake_files(tmp_path) + + def fake_download(_repo, filename, **_kwargs): + category = filename.removeprefix("BFCL_v3_").removesuffix(".json") + return str(paths[category]) + + class _LegacyTokenizer: + def apply_chat_template(self, messages, **kwargs): + if "tools" in kwargs: + raise TypeError("unexpected keyword argument 'tools'") + return " ".join(m.get("content", "") for m in messages) + + def __call__(self, text): + return type("Enc", (), {"input_ids": text.split()})() + + args = _args_for_bfcl(categories=["simple"]) + args.num_prompts = 1 + + with _patch_hf_api(fake_download): + samples = get_samples(args, _LegacyTokenizer()) + + assert len(samples) == 1 + assert samples[0].prompt_len > 0 + + +@pytest.mark.benchmark +def test_bfcl_schema_translation_is_recursive() -> None: + """_translate_schema must recurse into nested properties.""" + input_schema = { + "type": "dict", + "properties": { + "nested": { + "type": "dict", + "properties": { + "value": {"type": "float"}, + "tags": {"type": "tuple", "items": {"type": "any"}}, + }, + } + }, + } + out = BFCLDataset._translate_schema(input_schema) + assert out["type"] == "object" + assert out["properties"]["nested"]["type"] == "object" + assert out["properties"]["nested"]["properties"]["value"]["type"] == "number" + assert out["properties"]["nested"]["properties"]["tags"]["type"] == "array" + nested_props = out["properties"]["nested"]["properties"] + assert nested_props["tags"]["items"]["type"] == "string" diff --git a/asian_ai/performance/vLLM/tests/benchmarks/test_custom_dataset_chat_template_kwargs.py b/asian_ai/performance/vLLM/tests/benchmarks/test_custom_dataset_chat_template_kwargs.py new file mode 100644 index 0000000000000000000000000000000000000000..805da92851026bd9376912d871e8ea015e1a9a60 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/test_custom_dataset_chat_template_kwargs.py @@ -0,0 +1,75 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import argparse +import json +from pathlib import Path + +import pytest + +from vllm.benchmarks.datasets import get_samples + + +class _RecordingTokenizer: + """Minimal tokenizer stub that records the kwargs forwarded to + apply_chat_template, so we can assert chat_template_kwargs propagation + without loading a real model/template.""" + + def __init__(self) -> None: + self.captured_kwargs: dict | None = None + self.chat_template = "dummy-template" + + def apply_chat_template( + self, + conversation, + add_generation_prompt: bool = True, + tokenize: bool = False, + **kwargs, + ) -> str: + self.captured_kwargs = kwargs + return conversation[0]["content"] + + def __call__(self, text: str): + return argparse.Namespace(input_ids=list(range(len(text.split())))) + + +def _args(dataset_path: str, chat_template_kwargs) -> argparse.Namespace: + return argparse.Namespace( + dataset_name="custom", + dataset_path=dataset_path, + disable_shuffle=True, + num_prompts=1, + custom_output_len=32, + skip_chat_template=False, + chat_template_kwargs=chat_template_kwargs, + no_oversample=False, + seed=0, + request_id_prefix="", + ) + + +def _write_one(path: Path) -> None: + path.write_text(json.dumps({"prompt": "hello world"}) + "\n") + + +@pytest.mark.benchmark +def test_chat_template_kwargs_forwarded(tmp_path: Path) -> None: + """--chat-template-kwargs must reach the client-side apply_chat_template.""" + jsonl = tmp_path / "data.jsonl" + _write_one(jsonl) + + tok = _RecordingTokenizer() + get_samples(_args(str(jsonl), {"thinking": True}), tok) + + assert tok.captured_kwargs == {"thinking": True} + + +@pytest.mark.benchmark +def test_chat_template_kwargs_default_is_noop(tmp_path: Path) -> None: + """When not provided, no extra kwargs are passed (existing behavior).""" + jsonl = tmp_path / "data.jsonl" + _write_one(jsonl) + + tok = _RecordingTokenizer() + get_samples(_args(str(jsonl), None), tok) + + assert tok.captured_kwargs == {} diff --git a/asian_ai/performance/vLLM/tests/benchmarks/test_custom_dataset_seed.py b/asian_ai/performance/vLLM/tests/benchmarks/test_custom_dataset_seed.py new file mode 100644 index 0000000000000000000000000000000000000000..b36b6dd97e999f9ef0d989d05e747f6b50dff682 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/test_custom_dataset_seed.py @@ -0,0 +1,77 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import argparse +import json +from pathlib import Path + +import pytest +from transformers import AutoTokenizer, PreTrainedTokenizerBase + +from vllm.benchmarks.datasets import get_samples + + +@pytest.fixture(scope="session") +def hf_tokenizer() -> PreTrainedTokenizerBase: + return AutoTokenizer.from_pretrained("gpt2") + + +def _write_jsonl(path: Path, n_rows: int) -> None: + with path.open("w") as f: + for i in range(n_rows): + f.write(json.dumps({"prompt": f"row {i}: unique prompt content."}) + "\n") + + +def _args_for_custom(dataset_path: str, seed: int) -> argparse.Namespace: + return argparse.Namespace( + dataset_name="custom", + dataset_path=dataset_path, + disable_shuffle=False, + num_prompts=30, + custom_output_len=32, + skip_chat_template=True, + no_oversample=False, + seed=seed, + request_id_prefix="", + ) + + +@pytest.mark.benchmark +def test_custom_dataset_seed_propagates( + hf_tokenizer: PreTrainedTokenizerBase, tmp_path: Path +) -> None: + """--seed must control the CustomDataset shuffle used by get_samples. + + Without the fix, CustomDataset was instantiated without random_seed, + so its load-time shuffle always used DEFAULT_SEED=0 regardless of + args.seed, causing every run with --dataset-name custom to pick the + same subset of rows from a larger file. + """ + jsonl = tmp_path / "data.jsonl" + _write_jsonl(jsonl, n_rows=60) + + samples_a = get_samples(_args_for_custom(str(jsonl), seed=0), hf_tokenizer) + samples_b = get_samples(_args_for_custom(str(jsonl), seed=42), hf_tokenizer) + + prompts_a = {s.prompt for s in samples_a} + prompts_b = {s.prompt for s in samples_b} + + assert len(prompts_a) == 30 + assert len(prompts_b) == 30 + assert prompts_a != prompts_b + + +@pytest.mark.benchmark +def test_custom_dataset_same_seed_is_deterministic( + hf_tokenizer: PreTrainedTokenizerBase, tmp_path: Path +) -> None: + """Same --seed must yield the same CustomDataset subset.""" + jsonl = tmp_path / "data.jsonl" + _write_jsonl(jsonl, n_rows=60) + + samples_a = get_samples(_args_for_custom(str(jsonl), seed=7), hf_tokenizer) + samples_b = get_samples(_args_for_custom(str(jsonl), seed=7), hf_tokenizer) + + prompts_a = [s.prompt for s in samples_a] + prompts_b = [s.prompt for s in samples_b] + + assert prompts_a == prompts_b diff --git a/asian_ai/performance/vLLM/tests/benchmarks/test_custom_image_dataset.py b/asian_ai/performance/vLLM/tests/benchmarks/test_custom_image_dataset.py new file mode 100644 index 0000000000000000000000000000000000000000..4e1e0c23bdd6a15285115a420f5119e58959c4fd --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/test_custom_image_dataset.py @@ -0,0 +1,386 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import json +from argparse import Namespace +from io import BytesIO +from pathlib import Path +from typing import Any + +import pybase64 as base64 +import pytest +from PIL import Image + +import vllm.benchmarks.datasets.datasets as datasets_module +from vllm.benchmarks.datasets import CustomImageDataset, get_samples +from vllm.benchmarks.lib.endpoint_request_func import ( + RequestFuncInput, + _get_chat_content, + _get_chat_messages, +) + +pytestmark = pytest.mark.skip_global_cleanup + + +class _TokenizedPrompt: + def __init__(self, prompt: str) -> None: + self.input_ids = prompt.split() + + +class _Tokenizer: + def __call__(self, prompt: str) -> _TokenizedPrompt: + return _TokenizedPrompt(prompt) + + +def _write_jsonl(path: Path, rows: list[dict[str, Any]]) -> None: + with path.open("w") as f: + for row in rows: + f.write(json.dumps(row) + "\n") + + +def _write_png(path: Path, color: tuple[int, int, int] = (255, 0, 0)) -> None: + Image.new("RGB", (1, 1), color=color).save(path) + + +def _decode_data_url(data_url: str) -> tuple[str, bytes]: + prefix, image_base64 = data_url.split(",", 1) + return prefix, base64.b64decode(image_base64) + + +def _assert_png_data_url(data_url: str) -> None: + prefix, image_bytes = _decode_data_url(data_url) + assert prefix == "data:image/png;base64" + with Image.open(BytesIO(image_bytes)) as image: + image.verify() + + +def _args_for_custom_image(dataset_path: Path) -> Namespace: + return Namespace( + dataset_name="custom_image", + dataset_path=str(dataset_path), + disable_shuffle=True, + seed=0, + num_prompts=2, + custom_output_len=32, + enable_multimodal_chat=False, + custom_ensure_client_side_data=False, + request_id_prefix="req-", + no_oversample=False, + ) + + +@pytest.mark.benchmark +def test_get_samples_custom_image_cli_path_supports_multi_image_and_content( + tmp_path: Path, +) -> None: + image_a = tmp_path / "chart_a.png" + image_b = tmp_path / "chart_b.png" + image_c = tmp_path / "chart_c.png" + jsonl = tmp_path / "images.jsonl" + _write_jsonl( + jsonl, + [ + { + "prompt": "Compare the first two charts.", + "image_files": [str(image_a), str(image_b)], + }, + { + "content": [ + {"type": "text", "text": "Now compare "}, + {"type": "image", "image": str(image_c)}, + ], + }, + ], + ) + + samples = get_samples(_args_for_custom_image(jsonl), _Tokenizer()) + + assert len(samples) == 2 + assert samples[0].request_id == "req-0" + assert isinstance(samples[0].multi_modal_data, list) + assert [part["image_url"]["url"] for part in samples[0].multi_modal_data] == [ + f"file://{image_a}", + f"file://{image_b}", + ] + + assert samples[1].request_id == "req-1" + assert samples[1].multi_modal_data is None + assert isinstance(samples[1].prompt, list) + assert samples[1].prompt[0] == {"type": "text", "text": "Now compare "} + assert samples[1].prompt[1]["image_url"]["url"] == f"file://{image_c}" + + +@pytest.mark.benchmark +def test_custom_image_dataset_uses_all_image_files(tmp_path: Path) -> None: + image_a = tmp_path / "chart_a.png" + image_b = tmp_path / "chart_b.png" + jsonl = tmp_path / "images.jsonl" + _write_jsonl( + jsonl, + [ + { + "prompt": "Compare the charts.", + "image_files": [str(image_a), str(image_b)], + } + ], + ) + + dataset = CustomImageDataset(dataset_path=str(jsonl), disable_shuffle=True) + samples = dataset.sample( + tokenizer=_Tokenizer(), + num_requests=1, + output_len=32, + ) + + assert len(samples) == 1 + sample = samples[0] + assert sample.prompt == "Compare the charts." + assert sample.prompt_len == 3 + assert isinstance(sample.multi_modal_data, list) + assert [part["image_url"]["url"] for part in sample.multi_modal_data] == [ + f"file://{image_a}", + f"file://{image_b}", + ] + + +@pytest.mark.benchmark +def test_custom_image_dataset_preserves_interleaved_content_order( + tmp_path: Path, +) -> None: + image_a = tmp_path / "chart_a.png" + image_b = tmp_path / "chart_b.png" + jsonl = tmp_path / "images.jsonl" + _write_jsonl( + jsonl, + [ + { + "content": [ + {"type": "text", "text": "Compare "}, + {"type": "image", "image": str(image_a)}, + {"type": "text", "text": " with "}, + { + "type": "image_url", + "image_url": { + "url": str(image_b), + "detail": "low", + }, + }, + ], + } + ], + ) + + dataset = CustomImageDataset(dataset_path=str(jsonl), disable_shuffle=True) + samples = dataset.sample( + tokenizer=_Tokenizer(), + num_requests=1, + output_len=32, + ) + + assert len(samples) == 1 + sample = samples[0] + assert sample.multi_modal_data is None + assert sample.prompt_len == 2 + assert isinstance(sample.prompt, list) + assert [part["type"] for part in sample.prompt] == [ + "text", + "image_url", + "text", + "image_url", + ] + assert sample.prompt[1]["image_url"]["url"] == f"file://{image_a}" + assert sample.prompt[3]["image_url"] == { + "url": f"file://{image_b}", + "detail": "low", + } + + request_input = RequestFuncInput( + prompt=sample.prompt, + api_url="http://localhost:8000/v1/chat/completions", + prompt_len=sample.prompt_len, + output_len=32, + model="test-model", + ) + assert _get_chat_content(request_input) == sample.prompt + + +@pytest.mark.benchmark +def test_custom_image_dataset_wraps_interleaved_content_for_multimodal_chat( + tmp_path: Path, +) -> None: + image = tmp_path / "chart.png" + jsonl = tmp_path / "images.jsonl" + _write_jsonl( + jsonl, + [ + { + "content": [ + {"type": "text", "text": "Describe "}, + {"type": "image", "image": str(image)}, + ], + } + ], + ) + + dataset = CustomImageDataset(dataset_path=str(jsonl), disable_shuffle=True) + samples = dataset.sample( + tokenizer=_Tokenizer(), + num_requests=1, + output_len=32, + enable_multimodal_chat=True, + ) + + sample = samples[0] + assert sample.multi_modal_data is None + assert sample.prompt == [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Describe "}, + {"type": "image_url", "image_url": {"url": f"file://{image}"}}, + ], + } + ] + + request_input = RequestFuncInput( + prompt=sample.prompt, + api_url="http://localhost:8000/v1/chat/completions", + prompt_len=sample.prompt_len, + output_len=32, + model="test-model", + ) + assert _get_chat_messages(request_input) == sample.prompt + + +@pytest.mark.benchmark +def test_custom_image_dataset_encodes_image_media_when_requested( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + image_a = tmp_path / "chart_a.png" + image_b = tmp_path / "chart b.png" + _write_png(image_a, color=(255, 0, 0)) + _write_png(image_b, color=(0, 255, 0)) + data_url = "data:image/png;base64,Zm9v" + remote_url = "https://example.com/chart.png" + original_fetch_image = datasets_module.fetch_image + + def fake_fetch_image(image_url: str) -> Image.Image: + if image_url == remote_url: + return Image.new("RGB", (1, 1), color=(0, 0, 255)) + return original_fetch_image(image_url) + + monkeypatch.setattr(datasets_module, "fetch_image", fake_fetch_image) + + jsonl = tmp_path / "images.jsonl" + _write_jsonl( + jsonl, + [ + { + "prompt": "Compare the charts.", + "image_files": [ + str(image_a), + image_b.as_uri(), + remote_url, + data_url, + ], + } + ], + ) + + dataset = CustomImageDataset(dataset_path=str(jsonl), disable_shuffle=True) + samples = dataset.sample( + tokenizer=_Tokenizer(), + num_requests=1, + output_len=32, + ensure_client_side_data=True, + ) + + assert len(samples) == 1 + assert isinstance(samples[0].multi_modal_data, list) + image_urls = [part["image_url"]["url"] for part in samples[0].multi_modal_data] + + _assert_png_data_url(image_urls[0]) + _assert_png_data_url(image_urls[1]) + _assert_png_data_url(image_urls[2]) + assert image_urls[3] == data_url + + +@pytest.mark.benchmark +def test_custom_image_dataset_encodes_interleaved_image_media( + tmp_path: Path, +) -> None: + image_a = tmp_path / "chart_a.png" + image_b = tmp_path / "chart_b.png" + _write_png(image_a, color=(255, 0, 0)) + _write_png(image_b, color=(0, 255, 0)) + jsonl = tmp_path / "images.jsonl" + _write_jsonl( + jsonl, + [ + { + "content": [ + {"type": "text", "text": "Compare "}, + {"type": "image", "image": str(image_a)}, + { + "type": "image_url", + "image_url": { + "url": image_b.as_uri(), + "detail": "low", + }, + }, + ], + } + ], + ) + + dataset = CustomImageDataset(dataset_path=str(jsonl), disable_shuffle=True) + samples = dataset.sample( + tokenizer=_Tokenizer(), + num_requests=1, + output_len=32, + ensure_client_side_data=True, + ) + + sample = samples[0] + assert isinstance(sample.prompt, list) + _assert_png_data_url(sample.prompt[1]["image_url"]["url"]) + _assert_png_data_url(sample.prompt[2]["image_url"]["url"]) + assert sample.prompt[2]["image_url"]["detail"] == "low" + + +@pytest.mark.benchmark +def test_custom_image_dataset_rejects_invalid_image_media( + tmp_path: Path, +) -> None: + invalid_image = tmp_path / "not_an_image.png" + invalid_image.write_text("not an image") + jsonl = tmp_path / "images.jsonl" + _write_jsonl( + jsonl, + [{"prompt": "Describe the image.", "image_files": [str(invalid_image)]}], + ) + + dataset = CustomImageDataset(dataset_path=str(jsonl), disable_shuffle=True) + with pytest.raises(ValueError, match="Invalid image URL"): + dataset.sample( + tokenizer=_Tokenizer(), + num_requests=1, + output_len=32, + ensure_client_side_data=True, + ) + + +@pytest.mark.benchmark +def test_custom_image_dataset_rejects_invalid_content_part( + tmp_path: Path, +) -> None: + jsonl = tmp_path / "images.jsonl" + _write_jsonl(jsonl, [{"content": [{"type": "audio", "audio": "clip.wav"}]}]) + + dataset = CustomImageDataset(dataset_path=str(jsonl), disable_shuffle=True) + with pytest.raises(ValueError, match="type 'text', 'image', or 'image_url'"): + dataset.sample( + tokenizer=_Tokenizer(), + num_requests=1, + output_len=32, + ) diff --git a/asian_ai/performance/vLLM/tests/benchmarks/test_latency_cli.py b/asian_ai/performance/vLLM/tests/benchmarks/test_latency_cli.py new file mode 100644 index 0000000000000000000000000000000000000000..c7343e36885e5b508ac6a9a7b9d91be8878b35a7 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/test_latency_cli.py @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import subprocess + +import pytest + +MODEL_NAME = "meta-llama/Llama-3.2-1B-Instruct" + + +@pytest.mark.benchmark +def test_bench_latency(): + command = [ + "vllm", + "bench", + "latency", + "--model", + MODEL_NAME, + "--input-len", + "32", + "--output-len", + "1", + "--enforce-eager", + "--load-format", + "dummy", + ] + result = subprocess.run(command, capture_output=True, text=True) + print(result.stdout) + print(result.stderr) + + assert result.returncode == 0, f"Benchmark failed: {result.stderr}" diff --git a/asian_ai/performance/vLLM/tests/benchmarks/test_plot_filters.py b/asian_ai/performance/vLLM/tests/benchmarks/test_plot_filters.py new file mode 100644 index 0000000000000000000000000000000000000000..623b882241f6a3f6ea1e89e0129d3142f3bb2316 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/test_plot_filters.py @@ -0,0 +1,171 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pandas as pd +import pytest + +from vllm.benchmarks.sweep.plot import ( + PlotEqualTo, + PlotFilterBase, + PlotFilters, + PlotGreaterThan, + PlotGreaterThanOrEqualTo, + PlotLessThan, + PlotLessThanOrEqualTo, + PlotNotEqualTo, +) + + +class TestPlotFilters: + """Test PlotFilter functionality including 'inf' edge case.""" + + def setup_method(self): + """Create sample DataFrames for testing.""" + # DataFrame with numeric values + self.df_numeric = pd.DataFrame( + { + "request_rate": [1.0, 5.0, 10.0, 50.0, 100.0], + "value": [10, 20, 30, 40, 50], + } + ) + + # DataFrame with float('inf') - note: string "inf" values are coerced + # to float when loading data, so we only test with float('inf') + self.df_inf_float = pd.DataFrame( + { + "request_rate": [1.0, 5.0, 10.0, float("inf"), float("inf")], + "value": [10, 20, 30, 40, 50], + } + ) + + @pytest.mark.parametrize( + "target,expected_count", + [ + ("5.0", 1), + ("10.0", 1), + ("1.0", 1), + ], + ) + def test_equal_to_numeric(self, target, expected_count): + """Test PlotEqualTo with numeric values.""" + filter_obj = PlotEqualTo("request_rate", target) + result = filter_obj.apply(self.df_numeric) + assert len(result) == expected_count + + def test_equal_to_inf_float(self): + """Test PlotEqualTo with float('inf').""" + filter_obj = PlotEqualTo("request_rate", "inf") + result = filter_obj.apply(self.df_inf_float) + # Should match both float('inf') entries because float('inf') == float('inf') + assert len(result) == 2 + + @pytest.mark.parametrize( + "target,expected_count", + [ + ("5.0", 4), # All except 5.0 + ("1.0", 4), # All except 1.0 + ], + ) + def test_not_equal_to_numeric(self, target, expected_count): + """Test PlotNotEqualTo with numeric values.""" + filter_obj = PlotNotEqualTo("request_rate", target) + result = filter_obj.apply(self.df_numeric) + assert len(result) == expected_count + + def test_not_equal_to_inf_float(self): + """Test PlotNotEqualTo with float('inf').""" + filter_obj = PlotNotEqualTo("request_rate", "inf") + result = filter_obj.apply(self.df_inf_float) + # Should exclude float('inf') entries + assert len(result) == 3 + + @pytest.mark.parametrize( + "target,expected_count", + [ + ("10.0", 2), # 1.0, 5.0 + ("50.0", 3), # 1.0, 5.0, 10.0 + ("5.0", 1), # 1.0 + ], + ) + def test_less_than(self, target, expected_count): + """Test PlotLessThan with numeric values.""" + filter_obj = PlotLessThan("request_rate", target) + result = filter_obj.apply(self.df_numeric) + assert len(result) == expected_count + + @pytest.mark.parametrize( + "target,expected_count", + [ + ("10.0", 3), # 1.0, 5.0, 10.0 + ("5.0", 2), # 1.0, 5.0 + ], + ) + def test_less_than_or_equal_to(self, target, expected_count): + """Test PlotLessThanOrEqualTo with numeric values.""" + filter_obj = PlotLessThanOrEqualTo("request_rate", target) + result = filter_obj.apply(self.df_numeric) + assert len(result) == expected_count + + @pytest.mark.parametrize( + "target,expected_count", + [ + ("10.0", 2), # 50.0, 100.0 + ("5.0", 3), # 10.0, 50.0, 100.0 + ], + ) + def test_greater_than(self, target, expected_count): + """Test PlotGreaterThan with numeric values.""" + filter_obj = PlotGreaterThan("request_rate", target) + result = filter_obj.apply(self.df_numeric) + assert len(result) == expected_count + + @pytest.mark.parametrize( + "target,expected_count", + [ + ("10.0", 3), # 10.0, 50.0, 100.0 + ("5.0", 4), # 5.0, 10.0, 50.0, 100.0 + ], + ) + def test_greater_than_or_equal_to(self, target, expected_count): + """Test PlotGreaterThanOrEqualTo with numeric values.""" + filter_obj = PlotGreaterThanOrEqualTo("request_rate", target) + result = filter_obj.apply(self.df_numeric) + assert len(result) == expected_count + + @pytest.mark.parametrize( + "filter_str,expected_var,expected_target,expected_type", + [ + ("request_rate==5.0", "request_rate", "5.0", PlotEqualTo), + ("request_rate!=10.0", "request_rate", "10.0", PlotNotEqualTo), + ("request_rate<50.0", "request_rate", "50.0", PlotLessThan), + ("request_rate<=50.0", "request_rate", "50.0", PlotLessThanOrEqualTo), + ("request_rate>10.0", "request_rate", "10.0", PlotGreaterThan), + ("request_rate>=10.0", "request_rate", "10.0", PlotGreaterThanOrEqualTo), + ("request_rate==inf", "request_rate", "inf", PlotEqualTo), + ("request_rate!='inf'", "request_rate", "inf", PlotNotEqualTo), + ], + ) + def test_parse_str(self, filter_str, expected_var, expected_target, expected_type): + """Test parsing filter strings.""" + filter_obj = PlotFilterBase.parse_str(filter_str) + assert isinstance(filter_obj, expected_type) + assert filter_obj.var == expected_var + assert filter_obj.target == expected_target + + def test_parse_str_inf_edge_case(self): + """Test parsing 'inf' string in filter.""" + filter_obj = PlotFilterBase.parse_str("request_rate==inf") + assert isinstance(filter_obj, PlotEqualTo) + assert filter_obj.var == "request_rate" + assert filter_obj.target == "inf" + + def test_parse_multiple_filters(self): + """Test parsing multiple filters.""" + filters = PlotFilters.parse_str("request_rate>5.0,value<=40") + assert len(filters) == 2 + assert isinstance(filters[0], PlotGreaterThan) + assert isinstance(filters[1], PlotLessThanOrEqualTo) + + def test_parse_empty_filter(self): + """Test parsing empty filter string.""" + filters = PlotFilters.parse_str("") + assert len(filters) == 0 diff --git a/asian_ai/performance/vLLM/tests/benchmarks/test_random_dataset.py b/asian_ai/performance/vLLM/tests/benchmarks/test_random_dataset.py new file mode 100644 index 0000000000000000000000000000000000000000..61d8dbb7fb97ebe85d5542da82ded9f963ac1b6e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/test_random_dataset.py @@ -0,0 +1,484 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import random +from typing import Any, NamedTuple, cast + +import numpy as np +import pytest +from transformers import AutoTokenizer, PreTrainedTokenizerBase + +from vllm.benchmarks.datasets import ( + RandomDataset, + RandomMultiModalDataset, + SampleRequest, +) + + +@pytest.fixture(scope="session") +def hf_tokenizer() -> PreTrainedTokenizerBase: + # Use a small, commonly available tokenizer + return AutoTokenizer.from_pretrained("gpt2") + + +class Params(NamedTuple): + num_requests: int + prefix_len: int + range_ratio: float + input_len: int + output_len: int + + +@pytest.fixture(scope="session") +def random_dataset_params() -> Params: + return Params( + num_requests=16, prefix_len=7, range_ratio=0.3, input_len=50, output_len=20 + ) + + +def _fingerprint_sample(req: SampleRequest) -> tuple[str, int, int]: + """Project a SampleRequest into a comparable tuple.""" + return (req.prompt, req.prompt_len, req.expected_output_len) + + +def _collect_samples( + dataset: RandomDataset, + tokenizer: PreTrainedTokenizerBase, + num_requests: int = 16, + prefix_len: int = 7, + range_ratio: float = 0.3, + input_len: int = 50, + output_len: int = 20, +) -> list[tuple[str, int, int]]: + samples = dataset.sample( + tokenizer=tokenizer, + num_requests=num_requests, + prefix_len=prefix_len, + range_ratio=range_ratio, + input_len=input_len, + output_len=output_len, + ) + return [_fingerprint_sample(s) for s in samples] + + +@pytest.mark.benchmark +def test_random_dataset_same_seed( + hf_tokenizer: PreTrainedTokenizerBase, random_dataset_params: Params +) -> None: + """Same seed should yield identical outputs, even if global RNGs change. + + This guards against accidental reliance on Python's random or np.random + in RandomDataset after moving to numpy.default_rng. + """ + p = random_dataset_params + common_seed = 123 + dataset_a = RandomDataset(random_seed=common_seed) + dataset_b = RandomDataset(random_seed=common_seed) + a = _collect_samples( + dataset_a, + hf_tokenizer, + num_requests=p.num_requests, + prefix_len=p.prefix_len, + range_ratio=p.range_ratio, + input_len=p.input_len, + output_len=p.output_len, + ) + + # Perturb global RNG state to ensure isolation + random.seed(999) + _ = [random.random() for _ in range(100)] + np.random.seed(888) + _ = [np.random.random() for _ in range(100)] + + b = _collect_samples( + dataset_b, + hf_tokenizer, + num_requests=p.num_requests, + prefix_len=p.prefix_len, + range_ratio=p.range_ratio, + input_len=p.input_len, + output_len=p.output_len, + ) + assert a == b + + +@pytest.mark.benchmark +def test_random_dataset_different_seeds( + hf_tokenizer: PreTrainedTokenizerBase, random_dataset_params: Params +) -> None: + """Different seeds should change outputs with overwhelming likelihood.""" + p = random_dataset_params + seed_a = 0 + dataset_a = RandomDataset(random_seed=seed_a) + a = _collect_samples( + dataset_a, + hf_tokenizer, + num_requests=p.num_requests, + prefix_len=p.prefix_len, + range_ratio=p.range_ratio, + input_len=p.input_len, + output_len=p.output_len, + ) + + seed_b = 999 + dataset_b = RandomDataset(random_seed=seed_b) + # Perturb global RNG with same seed as dataset_a to ensure isolation + random.seed(seed_a) + np.random.seed(seed_a) + b = _collect_samples( + dataset_b, + hf_tokenizer, + num_requests=p.num_requests, + prefix_len=p.prefix_len, + range_ratio=p.range_ratio, + input_len=p.input_len, + output_len=p.output_len, + ) + assert a != b + + +# ----------------------------- +# RandomMultiModalDataset tests +# ----------------------------- + + +def _mm_fingerprint_sample( + req: SampleRequest, +) -> tuple[str, int, int, int, list[str]]: + """Create a compact fingerprint for multimodal samples. + + Includes: + - prompt string + - prompt_len + - expected_output_len + - count of multimodal items + - per-item type and URL prefix (e.g., 'data:image/jpeg;base64,') + """ + items = req.multi_modal_data or [] + item_prefixes: list[str] = [] + for it in items: + if isinstance(it, dict) and it.get("type") == "image_url": + url = it.get("image_url", {}).get("url", "") + # Only keep a short identifying prefix to avoid huge strings + item_prefixes.append(f"image:{url[:22]}") + elif isinstance(it, dict) and it.get("type") == "video_url": + url = it.get("video_url", {}).get("url", "") + item_prefixes.append(f"video:{url[:22]}") + else: + item_prefixes.append("unknown:") + return ( + req.prompt, + req.prompt_len, + req.expected_output_len, + len(items), + item_prefixes, + ) + + +def _collect_mm_samples( + dataset: RandomMultiModalDataset, + tokenizer: PreTrainedTokenizerBase, + *, + num_requests: int = 8, + prefix_len: int = 3, + range_ratio: float = 0.0, + input_len: int = 20, + output_len: int = 5, + base_items_per_request: int = 2, + num_mm_items_range_ratio: float = 0.0, + limit_mm_per_prompt: dict[str, int] | None = None, + bucket_config: dict[tuple[int, int, int], float] | None = None, + enable_multimodal_chat: bool = False, +) -> list[SampleRequest]: + if limit_mm_per_prompt is None: + limit_mm_per_prompt = {"image": 5, "video": 0} + if bucket_config is None: + bucket_config = {(32, 32, 1): 0.5, (52, 64, 1): 0.5} + return dataset.sample( + tokenizer=tokenizer, + num_requests=num_requests, + prefix_len=prefix_len, + range_ratio=range_ratio, + input_len=input_len, + output_len=output_len, + base_items_per_request=base_items_per_request, + num_mm_items_range_ratio=num_mm_items_range_ratio, + limit_mm_per_prompt=limit_mm_per_prompt, + bucket_config=bucket_config, + enable_multimodal_chat=enable_multimodal_chat, + ) + + +@pytest.mark.benchmark +def test_random_mm_same_seed(hf_tokenizer: PreTrainedTokenizerBase) -> None: + seed = 42 + ds_a = RandomMultiModalDataset(random_seed=seed) + ds_b = RandomMultiModalDataset(random_seed=seed) + a = _collect_mm_samples(ds_a, hf_tokenizer) + b = _collect_mm_samples(ds_b, hf_tokenizer) + fa = [_mm_fingerprint_sample(s) for s in a] + fb = [_mm_fingerprint_sample(s) for s in b] + assert fa == fb + + +@pytest.mark.benchmark +def test_random_mm_different_seeds( + hf_tokenizer: PreTrainedTokenizerBase, +) -> None: + ds_a = RandomMultiModalDataset(random_seed=0) + ds_b = RandomMultiModalDataset(random_seed=999) + a = _collect_mm_samples(ds_a, hf_tokenizer) + b = _collect_mm_samples(ds_b, hf_tokenizer) + fa = [_mm_fingerprint_sample(s) for s in a] + fb = [_mm_fingerprint_sample(s) for s in b] + assert fa != fb + + +@pytest.mark.benchmark +def test_random_mm_respects_limits( + hf_tokenizer: PreTrainedTokenizerBase, +) -> None: + ds = RandomMultiModalDataset(random_seed=0) + # Requesting 3 items with a per-prompt limit of 1 should error per current + # design (dataset refuses to silently clamp below the requested baseline). + with pytest.raises(ValueError): + _collect_mm_samples( + ds, + hf_tokenizer, + num_requests=12, + base_items_per_request=3, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt={"image": 1, "video": 0}, + bucket_config={(32, 32, 1): 1.0}, + ) + + +@pytest.mark.benchmark +def test_random_mm_zero_prob_entries_are_removed( + hf_tokenizer: PreTrainedTokenizerBase, +) -> None: + ds = RandomMultiModalDataset(random_seed=0) + # Second bucket has zero probability and should be ignored after + # normalization + samples = _collect_mm_samples( + ds, + hf_tokenizer, + num_requests=6, + base_items_per_request=2, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt={"image": 10, "video": 0}, + bucket_config={(32, 32, 1): 1.0, (52, 64, 1): 0.0}, + ) + for s in samples: + assert isinstance(s.multi_modal_data, list) + typed_mm = cast(list[dict[str, Any]], s.multi_modal_data) + for it in typed_mm: + assert it.get("type") == "image_url" + + +@pytest.mark.benchmark +def test_random_mm_zero_items(hf_tokenizer: PreTrainedTokenizerBase) -> None: + ds = RandomMultiModalDataset(random_seed=0) + samples = _collect_mm_samples( + ds, + hf_tokenizer, + num_requests=5, + base_items_per_request=0, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt={"image": 5, "video": 0}, + bucket_config={(32, 32, 1): 1.0}, + ) + for s in samples: + assert s.multi_modal_data == [] + + +@pytest.mark.benchmark +def test_random_mm_num_items_per_prompt(hf_tokenizer: PreTrainedTokenizerBase) -> None: + ds = RandomMultiModalDataset(random_seed=0) + # Fixed number of images per prompt + # set num_mm_items_range_ratio to 0.0 + # TODO: modify video values when video sampling is implemented + samples_fixed_items = _collect_mm_samples( + ds, + hf_tokenizer, + num_requests=5, + base_items_per_request=3, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt={"image": 3, "video": 0}, + bucket_config={(32, 32, 1): 1.0}, + ) + # Must have 5 requests each with 3 mm items per prompt + assert len(samples_fixed_items) == 5 + for s in samples_fixed_items: + mm_data = cast(list[dict[str, Any]], s.multi_modal_data) + assert len(mm_data) == 3 + for it in mm_data: + assert it.get("type") == "image_url" + + +@pytest.mark.benchmark +def test_random_mm_bucket_config_not_mutated( + hf_tokenizer: PreTrainedTokenizerBase, +) -> None: + ds = RandomMultiModalDataset(random_seed=0) + # This bucket config is not normalized to sum to 1 + # and has more buckets than requested images + original = {(32, 32, 1): 0.2, (52, 64, 1): 6, (25, 64, 1): 3} + # Keep a snapshot to compare after sampling + snapshot = dict(original) + + _ = _collect_mm_samples( + ds, + hf_tokenizer, + num_requests=4, + base_items_per_request=1, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt={"image": 1, "video": 0}, + bucket_config=original, + ) + + # Ensure the original dict content is unchanged + assert original == snapshot + + # Vary number of mm items per prompt + # set num_mm_items_range_ratio to 0.5 + samples_varying_items = _collect_mm_samples( + ds, + hf_tokenizer, + num_requests=5, + base_items_per_request=2, + num_mm_items_range_ratio=0.5, + limit_mm_per_prompt={"image": 4, "video": 0}, + bucket_config={(32, 32, 1): 1.0}, + ) + # Must have 5 requests each with less than 4 mm items per prompt + # but at least 1 mm item per prompt + assert len(samples_varying_items) == 5 + for s in samples_varying_items: + mm_data = cast(list[dict[str, Any]], s.multi_modal_data) + assert len(mm_data) <= 4 + assert len(mm_data) >= 1 + for it in mm_data: + assert it.get("type") == "image_url" + + +@pytest.mark.benchmark +def test_random_mm_video_sampling(hf_tokenizer: PreTrainedTokenizerBase) -> None: + """Test video sampling functionality in RandomMultiModalDataset.""" + ds = RandomMultiModalDataset(random_seed=42) + + # Test with video bucket configuration + bucket_config = { + (64, 64, 1): 0.3, # Images + (64, 64, 8): 0.7, # Videos + } + + limit_mm_per_prompt = {"image": 2, "video": 2} + + samples = _collect_mm_samples( + ds, + hf_tokenizer, + num_requests=5, + base_items_per_request=1, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt=limit_mm_per_prompt, + bucket_config=bucket_config, + ) + + assert len(samples) == 5 + + # Check that we have both images and videos + video_count = 0 + image_count = 0 + + for s in samples: + mm_data = cast(list[dict[str, Any]], s.multi_modal_data) + assert len(mm_data) == 1 + + item = mm_data[0] + if item.get("type") == "video_url": + video_count += 1 + # Verify video URL format + url = item.get("video_url", {}).get("url", "") + assert url.startswith("data:video/mp4;base64,") + elif item.get("type") == "image_url": + image_count += 1 + # Verify image URL format + url = item.get("image_url", {}).get("url", "") + assert url.startswith("data:image/jpeg;base64,") + + # Should have some videos due to 0.7 probability + assert video_count > 0 + assert image_count > 0 + + +@pytest.mark.benchmark +def test_random_mm_video_only_sampling(hf_tokenizer: PreTrainedTokenizerBase) -> None: + """Test sampling with only video buckets.""" + ds = RandomMultiModalDataset(random_seed=42) + + bucket_config = { + (64, 64, 8): 1.0, # Only videos + } + + limit_mm_per_prompt = {"image": 0, "video": 1} + + samples = _collect_mm_samples( + ds, + hf_tokenizer, + num_requests=3, + base_items_per_request=1, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt=limit_mm_per_prompt, + bucket_config=bucket_config, + ) + + assert len(samples) == 3 + + for s in samples: + mm_data = cast(list[dict[str, Any]], s.multi_modal_data) + assert len(mm_data) == 1 + + item = mm_data[0] + assert item.get("type") == "video_url" + url = item.get("video_url", {}).get("url", "") + assert url.startswith("data:video/mp4;base64,") + + +@pytest.mark.benchmark +def test_random_mm_video_deterministic_sampling( + hf_tokenizer: PreTrainedTokenizerBase, +) -> None: + """Test that video sampling is deterministic with same seed.""" + seed = 123 + ds_a = RandomMultiModalDataset(random_seed=seed) + ds_b = RandomMultiModalDataset(random_seed=seed) + + bucket_config = { + (64, 64, 8): 1.0, # Only videos + } + + limit_mm_per_prompt = {"image": 0, "video": 1} + + a = _collect_mm_samples( + ds_a, + hf_tokenizer, + num_requests=3, + base_items_per_request=1, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt=limit_mm_per_prompt, + bucket_config=bucket_config, + ) + + b = _collect_mm_samples( + ds_b, + hf_tokenizer, + num_requests=3, + base_items_per_request=1, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt=limit_mm_per_prompt, + bucket_config=bucket_config, + ) + + fa = [_mm_fingerprint_sample(s) for s in a] + fb = [_mm_fingerprint_sample(s) for s in b] + assert fa == fb diff --git a/asian_ai/performance/vLLM/tests/benchmarks/test_random_multimodal_dataset_video.py b/asian_ai/performance/vLLM/tests/benchmarks/test_random_multimodal_dataset_video.py new file mode 100644 index 0000000000000000000000000000000000000000..2346bb97529b1a439f75ba6bb29c9cb8126aecdf --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/test_random_multimodal_dataset_video.py @@ -0,0 +1,398 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import os +from tempfile import NamedTemporaryFile +from typing import Any, cast + +import cv2 +import pybase64 as base64 +import pytest +from transformers import AutoTokenizer, PreTrainedTokenizerBase + +from vllm.benchmarks.datasets import RandomMultiModalDataset, SampleRequest + + +@pytest.fixture(scope="session") +def hf_tokenizer() -> PreTrainedTokenizerBase: + """Use a small, commonly available tokenizer.""" + return AutoTokenizer.from_pretrained("gpt2") + + +@pytest.fixture +def video_dataset() -> RandomMultiModalDataset: + """Create a RandomMultiModalDataset instance for testing.""" + return RandomMultiModalDataset(random_seed=42) + + +@pytest.mark.benchmark +def test_generate_synthetic_video_different_seeds(): + """Test that different seeds produce different videos.""" + dataset1 = RandomMultiModalDataset(random_seed=123) + dataset2 = RandomMultiModalDataset(random_seed=456) + + width, height, num_frames = 64, 48, 8 + + video1 = dataset1.generate_synthetic_video(width, height, num_frames) + video2 = dataset2.generate_synthetic_video(width, height, num_frames) + + # Videos should be different due to different seeds + assert video1["bytes"] != video2["bytes"] + + +@pytest.mark.benchmark +def test_map_config_to_modality(video_dataset: RandomMultiModalDataset): + """Test modality mapping for different configurations.""" + # Test image configuration (num_frames = 1) + assert video_dataset.map_config_to_modality((256, 256, 1)) == "image" + assert video_dataset.map_config_to_modality((720, 1280, 1)) == "image" + + # Test video configurations (num_frames > 1) + assert video_dataset.map_config_to_modality((256, 256, 8)) == "video" + assert video_dataset.map_config_to_modality((720, 1280, 16)) == "video" + assert video_dataset.map_config_to_modality((64, 64, 32)) == "video" + + # Test invalid configurations + with pytest.raises(ValueError, match="Invalid multimodal item configuration"): + video_dataset.map_config_to_modality((256, 256, 0)) + + with pytest.raises(ValueError, match="Invalid multimodal item configuration"): + video_dataset.map_config_to_modality((256, 256, -1)) + + +@pytest.mark.benchmark +def test_generate_mm_item_video(video_dataset: RandomMultiModalDataset): + """Test generating multimodal items for video configurations.""" + # Test video item generation + video_config = (64, 48, 8) # height, width, num_frames + result = video_dataset.generate_mm_item(video_config) + + # Check the result structure matches OpenAI API format + assert isinstance(result, dict) + assert result["type"] == "video_url" + assert "video_url" in result + assert "url" in result["video_url"] + + # Check that the URL is a data URL with base64 encoded video + url = result["video_url"]["url"] + assert url.startswith("data:video/mp4;base64,") + + # Decode and verify the video content + base64_data = url.split(",")[1] + video_bytes = base64.b64decode(base64_data) + assert len(video_bytes) > 0 + + # Verify the video can be decoded + with NamedTemporaryFile(suffix=".mp4", delete=False) as temp_file: + temp_path = temp_file.name + temp_file.write(video_bytes) + + try: + cap = cv2.VideoCapture(temp_path) + assert cap.isOpened() + + frame_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) + frame_width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) + frame_height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) + + assert frame_count == 8 + assert frame_width == 48 + assert frame_height == 64 + + cap.release() + finally: + if os.path.exists(temp_path): + os.unlink(temp_path) + + +@pytest.mark.benchmark +def test_generate_mm_item_image(video_dataset: RandomMultiModalDataset): + """Test generating multimodal items for image configurations.""" + # Test image item generation + image_config = (64, 48, 1) # height, width, num_frames=1 + result = video_dataset.generate_mm_item(image_config) + + # Check the result structure matches OpenAI API format + assert isinstance(result, dict) + assert result["type"] == "image_url" + assert "image_url" in result + assert "url" in result["image_url"] + + # Check that the URL is a data URL with base64 encoded image + url = result["image_url"]["url"] + assert url.startswith("data:image/jpeg;base64,") + + +@pytest.mark.benchmark +def test_generate_mm_item_invalid_config(video_dataset: RandomMultiModalDataset): + """Test error handling for invalid configurations.""" + with pytest.raises(ValueError, match="Invalid multimodal item configuration"): + video_dataset.generate_mm_item((256, 256, 0)) + + +@pytest.mark.benchmark +def test_sample_with_video_buckets( + video_dataset: RandomMultiModalDataset, hf_tokenizer: PreTrainedTokenizerBase +): + """Test sampling with video bucket configurations.""" + # Configure bucket with video probability > 0 + bucket_config = { + (64, 64, 1): 0.3, # Images + (64, 64, 8): 0.7, # Videos + } + + limit_mm_per_prompt = {"image": 5, "video": 3} + + samples = video_dataset.sample( + tokenizer=hf_tokenizer, + num_requests=5, + base_items_per_request=2, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt=limit_mm_per_prompt, + bucket_config=bucket_config, + input_len=20, + output_len=5, + ) + + assert len(samples) == 5 + + # Check that samples contain both images and videos + video_count = 0 + image_count = 0 + + for sample in samples: + assert isinstance(sample, SampleRequest) + assert sample.multi_modal_data is not None + assert isinstance(sample.multi_modal_data, list) + + mm_data = cast(list[dict[str, Any]], sample.multi_modal_data) + assert len(mm_data) == 2 # base_items_per_request + + for item in mm_data: + if item["type"] == "video_url": + video_count += 1 + # Verify video URL format + url = item["video_url"]["url"] + assert url.startswith("data:video/mp4;base64,") + elif item["type"] == "image_url": + image_count += 1 + # Verify image URL format + url = item["image_url"]["url"] + assert url.startswith("data:image/jpeg;base64,") + + # Should have some videos due to 0.7 probability + assert video_count > 0 + assert image_count > 0 + + +@pytest.mark.benchmark +def test_sample_video_only_buckets( + video_dataset: RandomMultiModalDataset, hf_tokenizer: PreTrainedTokenizerBase +): + """Test sampling with only video buckets.""" + bucket_config = { + (64, 64, 8): 1.0, # Only videos + } + + limit_mm_per_prompt = {"image": 0, "video": 2} + + samples = video_dataset.sample( + tokenizer=hf_tokenizer, + num_requests=3, + base_items_per_request=1, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt=limit_mm_per_prompt, + bucket_config=bucket_config, + input_len=20, + output_len=5, + ) + + assert len(samples) == 3 + + for sample in samples: + assert isinstance(sample, SampleRequest) + assert sample.multi_modal_data is not None + assert isinstance(sample.multi_modal_data, list) + + mm_data = cast(list[dict[str, Any]], sample.multi_modal_data) + assert len(mm_data) == 1 + + item = mm_data[0] + assert item["type"] == "video_url" + url = item["video_url"]["url"] + assert url.startswith("data:video/mp4;base64,") + + +@pytest.mark.benchmark +def test_sample_respects_video_limits( + video_dataset: RandomMultiModalDataset, hf_tokenizer: PreTrainedTokenizerBase +): + """Test that sampling respects video limits per prompt.""" + bucket_config = { + (64, 64, 8): 1.0, # Only videos + } + + # Set very low video limit + limit_mm_per_prompt = {"image": 0, "video": 1} + + samples = video_dataset.sample( + tokenizer=hf_tokenizer, + num_requests=3, + base_items_per_request=1, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt=limit_mm_per_prompt, + bucket_config=bucket_config, + input_len=20, + output_len=5, + ) + + assert len(samples) == 3 + + for sample in samples: + mm_data = cast(list[dict[str, Any]], sample.multi_modal_data) + assert len(mm_data) <= 1 # Should respect video limit + + +@pytest.mark.benchmark +def test_sample_mixed_buckets_with_zero_probability( + video_dataset: RandomMultiModalDataset, hf_tokenizer: PreTrainedTokenizerBase +): + """Test sampling with mixed buckets including zero probability entries.""" + bucket_config = { + (64, 64, 1): 0.5, # Images + (64, 64, 8): 0.5, # Videos + (128, 128, 16): 0.0, # Zero probability videos (should be ignored) + } + + limit_mm_per_prompt = {"image": 2, "video": 2} + + samples = video_dataset.sample( + tokenizer=hf_tokenizer, + num_requests=4, + base_items_per_request=2, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt=limit_mm_per_prompt, + bucket_config=bucket_config, + input_len=20, + output_len=5, + ) + + assert len(samples) == 4 + + # Should only see 64x64 videos, not 128x128 videos + for sample in samples: + mm_data = cast(list[dict[str, Any]], sample.multi_modal_data) + for item in mm_data: + if item["type"] == "video_url": + # Decode video to verify dimensions + url = item["video_url"]["url"] + base64_data = url.split(",")[1] + video_bytes = base64.b64decode(base64_data) + + with NamedTemporaryFile(suffix=".mp4", delete=False) as temp_file: # noqa + temp_path = temp_file.name + temp_file.write(video_bytes) + + try: + cap = cv2.VideoCapture(temp_path) + frame_width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) + frame_height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) + cap.release() + + # Should be 64x64, not 128x128 + assert frame_width == 64 + assert frame_height == 64 + finally: + if os.path.exists(temp_path): + os.unlink(temp_path) + + +@pytest.mark.benchmark +def test_sample_deterministic_with_videos(hf_tokenizer: PreTrainedTokenizerBase): + """Test that sampling with videos is deterministic with same seed.""" + dataset1 = RandomMultiModalDataset(random_seed=123) + dataset2 = RandomMultiModalDataset(random_seed=123) + + bucket_config = { + (64, 64, 1): 0.3, # Images + (64, 64, 8): 0.7, # Videos + } + + limit_mm_per_prompt = {"image": 2, "video": 2} + + samples1 = dataset1.sample( + tokenizer=hf_tokenizer, + num_requests=3, + base_items_per_request=1, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt=limit_mm_per_prompt, + bucket_config=bucket_config, + input_len=20, + output_len=5, + ) + + samples2 = dataset2.sample( + tokenizer=hf_tokenizer, + num_requests=3, + base_items_per_request=1, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt=limit_mm_per_prompt, + bucket_config=bucket_config, + input_len=20, + output_len=5, + ) + + assert len(samples1) == len(samples2) + + # Compare multimodal data + for s1, s2 in zip(samples1, samples2): + assert s1.multi_modal_data == s2.multi_modal_data + + +@pytest.mark.benchmark +def test_sample_different_seeds_produce_different_videos( + hf_tokenizer: PreTrainedTokenizerBase, +): + """Test that different seeds produce different video content.""" + dataset1 = RandomMultiModalDataset(random_seed=123) + dataset2 = RandomMultiModalDataset(random_seed=456) + + bucket_config = { + (64, 64, 8): 1.0, # Only videos + } + + limit_mm_per_prompt = {"image": 0, "video": 1} + + samples1 = dataset1.sample( + tokenizer=hf_tokenizer, + num_requests=2, + base_items_per_request=1, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt=limit_mm_per_prompt, + bucket_config=bucket_config, + input_len=20, + output_len=5, + ) + + samples2 = dataset2.sample( + tokenizer=hf_tokenizer, + num_requests=2, + base_items_per_request=1, + num_mm_items_range_ratio=0.0, + limit_mm_per_prompt=limit_mm_per_prompt, + bucket_config=bucket_config, + input_len=20, + output_len=5, + ) + + # Video content should be different + for s1, s2 in zip(samples1, samples2): + mm_data1 = cast(list[dict[str, Any]], s1.multi_modal_data) + mm_data2 = cast(list[dict[str, Any]], s2.multi_modal_data) + + assert len(mm_data1) == len(mm_data2) == 1 + + url1 = mm_data1[0]["video_url"]["url"] + url2 = mm_data2[0]["video_url"]["url"] + + assert url1 != url2 # Different video content diff --git a/asian_ai/performance/vLLM/tests/benchmarks/test_sampling_params.py b/asian_ai/performance/vLLM/tests/benchmarks/test_sampling_params.py new file mode 100644 index 0000000000000000000000000000000000000000..15337641a4e7640088767fe411b5429b6244042d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/test_sampling_params.py @@ -0,0 +1,258 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import numpy as np +import pytest + +from vllm.benchmarks.datasets.utils import get_sampling_params +from vllm.tokenizers import TokenizerLike + + +class _FakeTokenizer(TokenizerLike): + """Minimal tokenizer implementing the TokenizerLike protocol + for testing get_sampling_params.""" + + def __init__(self, vocab_size: int = 1000, num_special_tokens: int = 0) -> None: + self._vocab_size = vocab_size + self._num_special_tokens = num_special_tokens + + # -- Properties required by TokenizerLike -- + + @classmethod + def from_pretrained(cls, path_or_repo_id, *a, **kw): # type: ignore[override] + return cls() + + @property + def vocab_size(self) -> int: + return self._vocab_size + + @property + def all_special_tokens(self) -> list[str]: + return [] + + @property + def all_special_ids(self) -> list[int]: + return [] + + @property + def bos_token_id(self) -> int: + return 0 + + @property + def eos_token_id(self) -> int: + return 1 + + @property + def pad_token_id(self) -> int: + return 2 + + @property + def is_fast(self) -> bool: + return False + + @property + def max_token_id(self) -> int: + return self._vocab_size - 1 + + @property + def max_chars_per_token(self) -> int: + return 4 + + @property + def truncation_side(self) -> str: + return "right" + + def num_special_tokens_to_add(self) -> int: + return self._num_special_tokens + + def __call__(self, text, text_pair=None, **kw): # type: ignore[override] + raise NotImplementedError + + def get_vocab(self) -> dict[str, int]: + return {} + + def get_added_vocab(self) -> dict[str, int]: + return {} + + def encode(self, text, **kw) -> list[int]: # type: ignore[override] + raise NotImplementedError + + def apply_chat_template(self, messages, **kw): # type: ignore[override] + raise NotImplementedError + + def convert_tokens_to_ids(self, tokens): # type: ignore[override] + raise NotImplementedError + + def convert_tokens_to_string(self, tokens: list[str]) -> str: + raise NotImplementedError + + def decode(self, ids, skip_special_tokens: bool = False) -> str: # type: ignore[override] + raise NotImplementedError + + def convert_ids_to_tokens( # type: ignore[override] + self, ids, skip_special_tokens: bool = False + ) -> list[str]: + raise NotImplementedError + + +class TestGetSamplingParams: + """Tests for ``get_sampling_params`` in ``vllm.benchmarks.datasets.shared``.""" + + # -- helpers -- + + @staticmethod + def _tok(vocab_size: int = 1000, num_special: int = 0) -> _FakeTokenizer: + return _FakeTokenizer(vocab_size=vocab_size, num_special_tokens=num_special) + + # -- return shape / dtype -- + + def test_returns_three_arrays(self): + rng = np.random.default_rng(0) + result = get_sampling_params(rng, 5, 0.0, 100, 50, self._tok()) + assert len(result) == 3 + for arr in result: + assert isinstance(arr, np.ndarray) + + @pytest.mark.parametrize("n", [1, 10, 100]) + def test_output_length_matches_num_requests(self, n: int): + rng = np.random.default_rng(42) + input_lens, output_lens, offsets = get_sampling_params( + rng, n, 0.0, 64, 32, self._tok() + ) + assert input_lens.shape == (n,) + assert output_lens.shape == (n,) + assert offsets.shape == (n,) + + # -- fixed lengths (range_ratio = 0) -- + + def test_zero_range_ratio_gives_constant_lengths(self): + rng = np.random.default_rng(7) + input_lens, output_lens, _ = get_sampling_params( + rng, 20, 0.0, 128, 64, self._tok() + ) + assert np.all(input_lens == 128) + assert np.all(output_lens == 64) + + def test_special_tokens_subtracted_from_input_only(self): + rng = np.random.default_rng(7) + input_lens, output_lens, _ = get_sampling_params( + rng, 10, 0.0, 100, 50, self._tok(num_special=4) + ) + # real_input_len = 100 - 4 = 96, range_ratio 0 → all 96 + assert np.all(input_lens == 96) + # special tokens are not subtracted from output length + assert np.all(output_lens == 50) + + # -- range ratios -- + + def test_input_range_bounds(self): + rng = np.random.default_rng(0) + ratio = 0.5 + base = 200 + input_lens, _, _ = get_sampling_params( + rng, 500, {"input": ratio, "output": 0.0}, base, 50, self._tok() + ) + lo = int(np.floor(base * (1 - ratio))) + hi = int(np.ceil(base * (1 + ratio))) + assert np.all(input_lens >= lo) + assert np.all(input_lens <= hi) + + def test_output_range_bounds(self): + rng = np.random.default_rng(0) + ratio = 0.3 + base = 100 + _, output_lens, _ = get_sampling_params( + rng, 500, {"input": 0.0, "output": ratio}, 50, base, self._tok() + ) + lo = max(1, int(np.floor(base * (1 - ratio)))) + hi = int(np.ceil(base * (1 + ratio))) + assert np.all(output_lens >= lo) + assert np.all(output_lens <= hi) + + def test_output_low_clamped_to_one(self): + """Even with a high ratio that would push output_low to 0, + the function clamps it to 1.""" + rng = np.random.default_rng(0) + # output_len=1, ratio=0.99 → floor(1*0.01)=0, should clamp to 1 + _, output_lens, _ = get_sampling_params( + rng, 50, {"input": 0.0, "output": 0.99}, 100, 1, self._tok() + ) + assert np.all(output_lens >= 1) + + # -- offsets bounded by vocab_size -- + + @pytest.mark.parametrize("vocab", [100, 32000, 128256]) + def test_offsets_within_vocab(self, vocab: int): + rng = np.random.default_rng(0) + _, _, offsets = get_sampling_params( + rng, 200, 0.0, 64, 32, self._tok(vocab_size=vocab) + ) + assert np.all(offsets >= 0) + assert np.all(offsets < vocab) + + # -- reproducibility -- + + def test_same_seed_same_results(self): + tok = self._tok() + rr = {"input": 0.3, "output": 0.2} + a = get_sampling_params(np.random.default_rng(42), 50, rr, 256, 64, tok) + b = get_sampling_params(np.random.default_rng(42), 50, rr, 256, 64, tok) + for arr_a, arr_b in zip(a, b): + np.testing.assert_array_equal(arr_a, arr_b) + + def test_different_seed_different_results(self): + tok = self._tok() + rr = {"input": 0.3, "output": 0.2} + a = get_sampling_params(np.random.default_rng(0), 50, rr, 256, 64, tok) + b = get_sampling_params(np.random.default_rng(1), 50, rr, 256, 64, tok) + # Extremely unlikely all three arrays match with different seeds + assert not all(np.array_equal(arr_a, arr_b) for arr_a, arr_b in zip(a, b)) + + # -- validation / error paths -- + + @pytest.mark.parametrize("bad_ratio", [-0.1, 1.0, 1.5]) + def test_invalid_input_range_ratio(self, bad_ratio: float): + rng = np.random.default_rng(0) + with pytest.raises(ValueError, match="input_range_ratio"): + get_sampling_params( + rng, 10, {"input": bad_ratio, "output": 0.0}, 100, 50, self._tok() + ) + + @pytest.mark.parametrize("bad_ratio", [-0.1, 1.0, 1.5]) + def test_invalid_output_range_ratio(self, bad_ratio: float): + rng = np.random.default_rng(0) + with pytest.raises(ValueError, match="output_range_ratio"): + get_sampling_params( + rng, 10, {"input": 0.0, "output": bad_ratio}, 100, 50, self._tok() + ) + + def test_invalid_dict_missing_keys(self): + rng = np.random.default_rng(0) + with pytest.raises(ValueError, match="input.*output"): + get_sampling_params(rng, 10, {"input": 0.1}, 100, 50, self._tok()) + + def test_input_len_zero_with_special_tokens(self): + """input_len < num_special_tokens → real_input_len = 0, which is fine + (range [0, 0]).""" + rng = np.random.default_rng(0) + input_lens, _, _ = get_sampling_params( + rng, 5, 0.0, 5, 50, self._tok(num_special=10) + ) + # real_input_len = max(0, 5 - 10) = 0 + assert np.all(input_lens == 0) + + # -- edge cases -- + + def test_single_request(self): + rng = np.random.default_rng(0) + i, o, off = get_sampling_params(rng, 1, 0.0, 100, 50, self._tok()) + assert i.shape == (1,) + assert o.shape == (1,) + assert off.shape == (1,) + + def test_large_num_requests(self): + rng = np.random.default_rng(0) + i, o, off = get_sampling_params(rng, 10_000, 0.5, 512, 128, self._tok()) + assert i.shape == (10_000,) + assert o.shape == (10_000,) + assert off.shape == (10_000,) diff --git a/asian_ai/performance/vLLM/tests/benchmarks/test_serve_cli.py b/asian_ai/performance/vLLM/tests/benchmarks/test_serve_cli.py new file mode 100644 index 0000000000000000000000000000000000000000..8e99fd0d784ef721ee4d746a48722b248c4ff67a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/test_serve_cli.py @@ -0,0 +1,181 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import subprocess +import tempfile +import time +from pathlib import Path + +import pytest +import requests +import urllib3 + +from ..utils import RemoteOpenAIServer + +MODEL_NAME = "meta-llama/Llama-3.2-1B-Instruct" + + +def generate_self_signed_cert(cert_dir: Path) -> tuple[Path, Path]: + """Generate a self-signed certificate for testing.""" + cert_file = cert_dir / "cert.pem" + key_file = cert_dir / "key.pem" + + # Generate self-signed certificate using openssl + subprocess.run( + [ + "openssl", + "req", + "-x509", + "-newkey", + "rsa:2048", + "-keyout", + str(key_file), + "-out", + str(cert_file), + "-days", + "1", + "-nodes", + "-subj", + "/CN=localhost", + ], + check=True, + capture_output=True, + ) + return cert_file, key_file + + +class RemoteOpenAIServerSSL(RemoteOpenAIServer): + """RemoteOpenAIServer subclass that supports SSL with self-signed certs.""" + + @property + def url_root(self) -> str: + return f"https://{self.host}:{self.port}" + + def _wait_for_server(self, *, url: str, timeout: float): + """Override to use HTTPS with SSL verification disabled.""" + # Suppress InsecureRequestWarning for self-signed certs + urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + + start = time.time() + while True: + try: + if requests.get(url, verify=False).status_code == 200: + break + except Exception: + result = self._poll() + if result is not None and result != 0: + raise RuntimeError("Server exited unexpectedly.") from None + + time.sleep(0.5) + if time.time() - start > timeout: + raise RuntimeError("Server failed to start in time.") from None + + +@pytest.fixture(scope="function") +def server(): + args = ["--max-model-len", "1024", "--enforce-eager", "--load-format", "dummy"] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.fixture(scope="function") +def ssl_server(): + """Start a vLLM server with SSL enabled using a self-signed certificate.""" + with tempfile.TemporaryDirectory() as cert_dir: + cert_file, key_file = generate_self_signed_cert(Path(cert_dir)) + args = [ + "--max-model-len", + "1024", + "--enforce-eager", + "--load-format", + "dummy", + "--ssl-certfile", + str(cert_file), + "--ssl-keyfile", + str(key_file), + ] + + with RemoteOpenAIServerSSL(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.mark.benchmark +def test_bench_serve(server): + # Test default model detection and input/output len + command = [ + "vllm", + "bench", + "serve", + "--host", + server.host, + "--port", + str(server.port), + "--input-len", + "32", + "--output-len", + "4", + "--num-prompts", + "5", + ] + result = subprocess.run(command, capture_output=True, text=True) + print(result.stdout) + print(result.stderr) + + assert result.returncode == 0, f"Benchmark failed: {result.stderr}" + + +@pytest.mark.benchmark +def test_bench_serve_insecure(ssl_server): + """Test --insecure flag with an HTTPS server using a self-signed certificate.""" + base_url = f"https://{ssl_server.host}:{ssl_server.port}" + command = [ + "vllm", + "bench", + "serve", + "--base-url", + base_url, + "--input-len", + "32", + "--output-len", + "4", + "--num-prompts", + "5", + "--insecure", + ] + result = subprocess.run(command, capture_output=True, text=True) + print(result.stdout) + print(result.stderr) + + assert result.returncode == 0, f"Benchmark failed: {result.stderr}" + + +@pytest.mark.benchmark +def test_bench_serve_chat(server): + command = [ + "vllm", + "bench", + "serve", + "--model", + MODEL_NAME, + "--host", + server.host, + "--port", + str(server.port), + "--dataset-name", + "random", + "--random-input-len", + "32", + "--random-output-len", + "4", + "--num-prompts", + "5", + "--endpoint", + "/v1/chat/completions", + "--backend", + "openai-chat", + ] + result = subprocess.run(command, capture_output=True, text=True) + print(result.stdout) + print(result.stderr) + + assert result.returncode == 0, f"Benchmark failed: {result.stderr}" diff --git a/asian_ai/performance/vLLM/tests/benchmarks/test_throughput_cli.py b/asian_ai/performance/vLLM/tests/benchmarks/test_throughput_cli.py new file mode 100644 index 0000000000000000000000000000000000000000..7e852a4c59530b60572c4c06ad97fa3cd5dcc87f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/test_throughput_cli.py @@ -0,0 +1,104 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import subprocess + +import pytest + +from vllm.benchmarks.datasets import SampleRequest +from vllm.benchmarks.throughput import ( + _run_vllm_chat_requests, + add_cli_args, +) +from vllm.utils.argparse_utils import FlexibleArgumentParser + +MODEL_NAME = "meta-llama/Llama-3.2-1B-Instruct" + + +@pytest.mark.benchmark +def test_bench_throughput(): + command = [ + "vllm", + "bench", + "throughput", + "--model", + MODEL_NAME, + "--input-len", + "32", + "--output-len", + "1", + "--enforce-eager", + "--load-format", + "dummy", + ] + result = subprocess.run(command, capture_output=True, text=True) + print(result.stdout) + print(result.stderr) + + assert result.returncode == 0, f"Benchmark failed: {result.stderr}" + + +def test_bench_throughput_accepts_custom_audio_args(): + parser = FlexibleArgumentParser() + add_cli_args(parser) + + args = parser.parse_args( + [ + "--dataset-name", + "custom_audio", + "--dataset-path", + "audio.jsonl", + "--no-oversample", + "--custom-output-len", + "32", + "--enable-multimodal-chat", + ] + ) + + assert args.dataset_name == "custom_audio" + assert args.no_oversample + assert args.custom_output_len == 32 + assert args.enable_multimodal_chat + + +def test_vllm_chat_requests_include_multimodal_content(): + class FakeLLM: + def __init__(self): + self.prompts = None + + def chat(self, prompts, sampling_params, use_tqdm): + del sampling_params, use_tqdm + self.prompts = prompts + return [] + + llm = FakeLLM() + audio_content = { + "type": "input_audio", + "input_audio": {"data": "abc", "format": "wav"}, + } + request = SampleRequest( + prompt="Transcribe this audio.", + prompt_len=1, + expected_output_len=8, + multi_modal_data=audio_content, + ) + + _run_vllm_chat_requests( + llm, + [request], + n=1, + disable_detokenize=False, + do_profile=False, + prequeue_requests=False, + ) + + assert llm.prompts == [ + [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Transcribe this audio."}, + audio_content, + ], + } + ] + ] diff --git a/asian_ai/performance/vLLM/tests/benchmarks/test_txt_slices_dataset.py b/asian_ai/performance/vLLM/tests/benchmarks/test_txt_slices_dataset.py new file mode 100644 index 0000000000000000000000000000000000000000..9d23fe695442a18c7b8020a41ec400ba5ae77e89 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/benchmarks/test_txt_slices_dataset.py @@ -0,0 +1,68 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import json +from pathlib import Path + +import pytest +from transformers import AutoTokenizer, PreTrainedTokenizerBase + +from vllm.benchmarks.datasets import CustomDataset +from vllm.benchmarks.datasets.create_txt_slices_dataset import create_txt_slices_jsonl + + +@pytest.fixture(scope="session") +def hf_tokenizer() -> PreTrainedTokenizerBase: + # Use a small, commonly available tokenizer + return AutoTokenizer.from_pretrained("gpt2") + + +text_content = """ +Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor +incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud +exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. +Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat +nulla pariatur. Excepteur sint occaecat cupidatat non proident, +sunt in culpa qui officia deserunt mollit anim id est laborum. +""" + + +@pytest.mark.benchmark +def test_create_txt_slices_jsonl( + hf_tokenizer: PreTrainedTokenizerBase, tmp_path: Path +) -> None: + """Test that create_txt_slices_jsonl produces valid JSONL for CustomDataset.""" + txt_path = tmp_path / "input.txt" + jsonl_path = tmp_path / "input.txt.jsonl" + + txt_path.write_text(text_content) + + create_txt_slices_jsonl( + input_path=str(txt_path), + output_path=str(jsonl_path), + tokenizer_name="gpt2", + num_prompts=10, + input_len=10, + output_len=10, + ) + + # Verify the JSONL file is valid and has the expected structure + records = [json.loads(line) for line in jsonl_path.read_text().splitlines()] + + assert len(records) == 10 + for record in records: + assert "prompt" in record + assert "output_tokens" in record + assert isinstance(record["prompt"], str) + assert record["output_tokens"] == 10 + + # Verify the JSONL file can be loaded by CustomDataset + dataset = CustomDataset(dataset_path=str(jsonl_path)) + samples = dataset.sample( + tokenizer=hf_tokenizer, + num_requests=10, + output_len=10, + skip_chat_template=True, + ) + + assert len(samples) == 10 + assert all(sample.expected_output_len == 10 for sample in samples) diff --git a/asian_ai/performance/vLLM/tests/ci_envs.py b/asian_ai/performance/vLLM/tests/ci_envs.py new file mode 100644 index 0000000000000000000000000000000000000000..6a38c728f70143333e84debd80eba424fa992fa8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/ci_envs.py @@ -0,0 +1,52 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +These envs only work for a small part of the tests, fix what you need! +""" + +import os +from collections.abc import Callable +from typing import TYPE_CHECKING, Any + +from vllm.envs import maybe_convert_bool + +if TYPE_CHECKING: + VLLM_CI_NO_SKIP: bool = False + VLLM_CI_DTYPE: str | None = None + VLLM_CI_HEAD_DTYPE: str | None = None + VLLM_CI_HF_DTYPE: str | None = None + +environment_variables: dict[str, Callable[[], Any]] = { + # A model family has many models with the same architecture. + # By default, a model family tests only one model. + # Through this flag, all models can be tested. + "VLLM_CI_NO_SKIP": lambda: bool(int(os.getenv("VLLM_CI_NO_SKIP", "0"))), + # Allow changing the dtype used by vllm in tests + "VLLM_CI_DTYPE": lambda: os.getenv("VLLM_CI_DTYPE", None), + # Allow changing the head dtype used by vllm in tests + "VLLM_CI_HEAD_DTYPE": lambda: os.getenv("VLLM_CI_HEAD_DTYPE", None), + # Allow changing the head dtype used by transformers in tests + "VLLM_CI_HF_DTYPE": lambda: os.getenv("VLLM_CI_HF_DTYPE", None), + # Allow control over whether tests use enforce_eager + "VLLM_CI_ENFORCE_EAGER": lambda: maybe_convert_bool( + os.getenv("VLLM_CI_ENFORCE_EAGER", None) + ), +} + + +def __getattr__(name: str): + # lazy evaluation of environment variables + if name in environment_variables: + return environment_variables[name]() + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") + + +def __dir__(): + return list(environment_variables.keys()) + + +def is_set(name: str): + """Check if an environment variable is explicitly set.""" + if name in environment_variables: + return name in os.environ + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") diff --git a/asian_ai/performance/vLLM/tests/compile/README.md b/asian_ai/performance/vLLM/tests/compile/README.md new file mode 100644 index 0000000000000000000000000000000000000000..8f5e4c778d3a9a54d1714162e2572ddce3daaa8d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/README.md @@ -0,0 +1,5 @@ +# compile test folder structure + +- `compile/test_*.py` : various unit tests meant for testing particular code path/features. Future tests are most likely added here. New test files added here will be included in CI automatically +- `compile/fullgraph/` : full model tests, including all tests previously in compile/piecewise. These tests do not target particular features. New test files added here will be included in CI automatically +- `compile/distributed/` : tests that require multiple GPUs. New test files added here will **NOT** be included in CI automatically as these tests generally need to be manually configured to run in runners with particular number/type of GPUs. diff --git a/asian_ai/performance/vLLM/tests/compile/__init__.py b/asian_ai/performance/vLLM/tests/compile/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/compile/backend.py b/asian_ai/performance/vLLM/tests/compile/backend.py new file mode 100644 index 0000000000000000000000000000000000000000..8cf33397dfa0b0a155bcd5f0134994c0b90978e6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/backend.py @@ -0,0 +1,132 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import weakref +from collections.abc import Callable, Sequence +from contextlib import nullcontext +from copy import deepcopy + +import depyf +from torch import fx +from torch._ops import OpOverload, OpOverloadPacket +from torch.fx._utils import lazy_format_graph_code + +from vllm.compilation.passes.fx_utils import find_op_nodes +from vllm.compilation.passes.inductor_pass import ( + InductorPass, + pass_context, +) +from vllm.compilation.passes.ir.inplace_functionalization import ( + VllmIRInplaceFunctionalizationPass, +) +from vllm.compilation.passes.pass_manager import with_pattern_match_debug +from vllm.compilation.passes.vllm_inductor_pass import VllmInductorPass +from vllm.config import VllmConfig, get_current_vllm_config +from vllm.config.utils import Range +from vllm.logger import init_logger + +logger = init_logger("vllm.tests.compile.backend") + + +class LazyInitPass(InductorPass): + """ + If there's a pass that we want to initialize lazily in a test, + we can wrap it in LazyInitPass, which will initialize the pass when invoked + and then immediately invoke it. + """ + + def __init__(self, pass_cls: type[VllmInductorPass], vllm_config: VllmConfig): + self.pass_cls = pass_cls + self.vllm_config = weakref.proxy(vllm_config) # avoid cycle + + def __call__(self, graph: fx.Graph) -> None: + self.pass_ = self.pass_cls(self.vllm_config) + self.pass_(graph) + + +class TestBackend: + """ + This class provides a simple Inductor backend that can be used for testing. + It takes a list of custom passes and runs them after Inductor's passes. + It also saves the graph before and after the custom passes for inspection. + + Inductor config can be modified directly by editing the inductor_config + property. This can be helpful for adding passes like the + 'pre_grad_custom_pass' and the 'post_grad_custom_pre_pass'. + Inductor config is default-initialized from VllmConfig.CompilationConfig. + """ + + def __init__(self, *passes: InductorPass | Callable[[fx.Graph], None]): + self.custom_passes = list(passes) + vllm_config = get_current_vllm_config() + compile_config = vllm_config.compilation_config + self.range = Range(1, vllm_config.scheduler_config.max_num_batched_tokens) + # Deepcopy to allow multiple TestBackend instances to use the same VllmConfig + self.inductor_config = deepcopy(compile_config.inductor_compile_config) + self.inductor_config["force_disable_caches"] = True + self.inductor_config["post_grad_custom_post_pass"] = self.post_pass + + # Add VllmIRInplaceFunctionalizationPass as pre-grad pass by default + self.inductor_config["pre_grad_custom_pass"] = ( + VllmIRInplaceFunctionalizationPass(vllm_config) + ) + + if debug_dump_path := vllm_config.compile_debug_dump_path(): + logger.debug("Dumping depyf output to %s", debug_dump_path) + self.debug_ctx = depyf.prepare_debug(debug_dump_path.as_posix()) + else: + self.debug_ctx = nullcontext() + + def __call__(self, graph: fx.GraphModule, example_inputs): + self.graph_pre_compile = deepcopy(graph) + from torch._inductor.compile_fx import compile_fx + + with self.debug_ctx, pass_context(self.range): + return compile_fx( + graph, example_inputs, config_patches=self.inductor_config + ) + + @with_pattern_match_debug + def post_pass(self, graph: fx.Graph): + self.graph_pre_pass = deepcopy(graph) + lazy_format_graph_code("graph_pre_pass", graph.owning_module) + + VllmInductorPass.dump_prefix = 0 + for pass_ in self.custom_passes: + pass_(graph) + VllmInductorPass.dump_prefix += 1 + + VllmInductorPass.dump_prefix = None + + self.graph_post_pass = deepcopy(graph) + lazy_format_graph_code("graph_post_pass", graph.owning_module) + # assign by reference, will reflect the final state of the graph + self.final_graph = graph + + def check_before_ops( + self, ops: Sequence[OpOverload | OpOverloadPacket], fully_replaced=True + ): + for op in ops: + num_pre = len(list(find_op_nodes(op, self.graph_pre_pass))) + num_post = len(list(find_op_nodes(op, self.graph_post_pass))) + assert num_pre > 0, f"Op {op.name()} not found in pre-pass graph" + assert num_pre > num_post, f"All nodes remain for op {op.name()}" + if fully_replaced: + assert num_post == 0, f"Unexpected op {op.name()} in post-pass graph" + + def check_after_ops(self, ops: Sequence[OpOverload | OpOverloadPacket]): + for op in ops: + num_pre = len(list(find_op_nodes(op, self.graph_pre_pass))) + num_post = len(list(find_op_nodes(op, self.graph_post_pass))) + assert num_pre == 0, f"Unexpected op {op.name()} in pre-pass graph" + assert num_post > 0, f"Op {op.name()} not found in post-pass graph" + + def op_count(self, op: OpOverload | OpOverloadPacket, before=False) -> int: + graph = self.graph_pre_pass if before else self.graph_post_pass + return len(list(find_op_nodes(op, graph))) + + def print_graphs(self): + print("=== Graph before custom passes ===") + print(self.graph_pre_pass.python_code(root_module="self", verbose=True).src) + print("=== Graph after custom passes ===") + print(self.graph_post_pass.python_code(root_module="self", verbose=True).src) diff --git a/asian_ai/performance/vLLM/tests/compile/conftest.py b/asian_ai/performance/vLLM/tests/compile/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..02dbab304a63af9a1aa97c0a6f2a400d6d5ffbe0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/conftest.py @@ -0,0 +1,71 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from contextlib import contextmanager +from unittest.mock import MagicMock, patch + +import pytest + +from vllm.platforms.interface import DeviceCapability + + +@pytest.fixture +def mock_cuda_platform(): + """ + Fixture that returns a factory for creating mocked CUDA platforms. + + Usage: + def test_something(mock_cuda_platform): + with mock_cuda_platform(is_cuda=True, capability=(9, 0)): + # test code + """ + + @contextmanager + def _mock_platform(is_cuda: bool = True, capability: tuple[int, int] | None = None): + mock_platform = MagicMock() + mock_platform.is_cuda.return_value = is_cuda + mock_platform.is_xpu.return_value = False + device_capability = ( + DeviceCapability(*capability) if capability is not None else None + ) + mock_platform.get_device_capability.return_value = device_capability + + def is_device_capability_family( + requested_capability: int, device_id: int = 0 + ) -> bool: + current_capability = mock_platform.get_device_capability( + device_id=device_id + ) + if current_capability is None: + return False + return current_capability.major == (requested_capability // 10) + + mock_platform.is_device_capability_family.side_effect = ( + is_device_capability_family + ) + with patch("vllm.platforms.current_platform", mock_platform): + yield mock_platform + + return _mock_platform + + +@pytest.fixture +def mock_xpu_platform(): + """ + Fixture that returns a factory for creating mocked XPU platforms. + + Usage: + def test_something(mock_xpu_platform): + with mock_xpu_platform(): + # test code + """ + + @contextmanager + def _mock_platform(): + mock_platform = MagicMock() + mock_platform.is_cuda.return_value = False + mock_platform.is_xpu.return_value = True + with patch("vllm.platforms.current_platform", mock_platform): + yield mock_platform + + return _mock_platform diff --git a/asian_ai/performance/vLLM/tests/compile/correctness_e2e/__init__.py b/asian_ai/performance/vLLM/tests/compile/correctness_e2e/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/compile/correctness_e2e/test_async_tp.py b/asian_ai/performance/vLLM/tests/compile/correctness_e2e/test_async_tp.py new file mode 100644 index 0000000000000000000000000000000000000000..d8db3a1ba7c0762d9ebeef9be7bfc3bd0db4470d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/correctness_e2e/test_async_tp.py @@ -0,0 +1,169 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import pytest + +from tests.models.registry import HF_EXAMPLE_MODELS +from tests.utils import ( + compare_two_settings, + create_new_process_for_each_test, +) +from vllm.config import ( + CompilationMode, +) +from vllm.platforms import current_platform +from vllm.utils.flashinfer import has_flashinfer + +NVFP4_MODEL_ID = "nvidia/Llama-3.1-8B-Instruct-NVFP4" +NVFP4_HF_OVERRIDES = { + "num_hidden_layers": 4, + "hidden_size": 512, + "intermediate_size": 800, + "num_attention_heads": 4, + "num_key_value_heads": 1, +} + + +@create_new_process_for_each_test() +@pytest.mark.parametrize( + "model_id", + ["meta-llama/Llama-3.2-1B-Instruct", "RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8"], +) +@pytest.mark.parametrize("tp_size", [2]) +@pytest.mark.parametrize("async_tp_enabled", [True]) +@pytest.mark.parametrize("distributed_backend", ["mp"]) +@pytest.mark.parametrize("eager_mode", [False, True]) +def test_async_tp_pass_correctness( + model_id: str, + tp_size: int, + async_tp_enabled: bool, + distributed_backend: str, + eager_mode: bool, + num_gpus_available: int, + monkeypatch, +): + # Disable FlashInfer FP8 scaled_mm kernel as it is incompatible with + # async TP patterns. No-op on H100 (kernel requires CC >= 100). + monkeypatch.setenv("VLLM_DISABLED_KERNELS", "FlashInferFP8ScaledMMLinearKernel") + + model_info = HF_EXAMPLE_MODELS.find_hf_info(model_id) + model_info.check_transformers_version(on_fail="skip") + model_info.check_available_online(on_fail="skip") + + pp_size = 1 + if num_gpus_available < tp_size: + pytest.skip(f"Need at least {tp_size} x {pp_size} GPUs") + + common_args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "2048", + "--max-num-seqs", + "8", + ] + if eager_mode: + common_args.append("--enforce-eager") + + compilation_config = { + "mode": CompilationMode.VLLM_COMPILE, + "compile_sizes": [2, 4, 8], + "splitting_ops": [], + "pass_config": {"fuse_gemm_comms": async_tp_enabled}, + } + + async_tp_args = [ + *common_args, + "--tensor-parallel-size", + str(tp_size), + "--distributed-executor-backend", + distributed_backend, + "--compilation_config", + json.dumps(compilation_config), + ] + + tp_args = [ + *common_args, + "--tensor-parallel-size", + str(tp_size), + "--distributed-executor-backend", + "mp", + ] + + compare_two_settings( + model_id, + async_tp_args, + tp_args, + method="generate", + force_v1_runner=True, + ) + + +@create_new_process_for_each_test() +def test_async_tp_pass_nvfp4_correctness(num_gpus_available: int): + if ( + not current_platform.is_cuda() + or not current_platform.is_device_capability_family(100) + ): + pytest.skip("NVFP4 requires Blackwell") + if not has_flashinfer(): + pytest.skip("FlashInfer is required for the NVFP4 AsyncTP path") + + tp_size = 2 + if num_gpus_available < tp_size: + pytest.skip(f"Need at least {tp_size} GPUs") + + common_args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "2048", + "--max-num-seqs", + "8", + "--load-format", + "dummy", + "--linear-backend", + "flashinfer_cutlass", + "--hf-overrides", + json.dumps(NVFP4_HF_OVERRIDES), + ] + + compilation_config = { + "mode": CompilationMode.VLLM_COMPILE, + "compile_sizes": [2, 4, 8], + "splitting_ops": [], + "pass_config": { + "enable_sp": True, + "fuse_gemm_comms": True, + "fuse_allreduce_rms": False, + "sp_min_token_num": 1, + }, + } + + async_tp_args = [ + *common_args, + "--tensor-parallel-size", + str(tp_size), + "--distributed-executor-backend", + "mp", + "--compilation_config", + json.dumps(compilation_config), + ] + + tp_args = [ + *common_args, + "--tensor-parallel-size", + str(tp_size), + "--distributed-executor-backend", + "mp", + ] + + compare_two_settings( + NVFP4_MODEL_ID, + async_tp_args, + tp_args, + method="generate", + force_v1_runner=True, + ) diff --git a/asian_ai/performance/vLLM/tests/compile/correctness_e2e/test_sequence_parallel.py b/asian_ai/performance/vLLM/tests/compile/correctness_e2e/test_sequence_parallel.py new file mode 100644 index 0000000000000000000000000000000000000000..e1cd3ed562714a5361e15f57380bf927800bb04a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/correctness_e2e/test_sequence_parallel.py @@ -0,0 +1,448 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +WARNING: This test runs in both single-node (4 GPUs) and multi-node + (2 node with 2 GPUs each) modes. If the test only uses 2 GPUs, it is + important to set the distributed backend to "mp" to avoid Ray scheduling + all workers in a node other than the head node, which can cause the test + to fail. +""" + +import json +import os +from dataclasses import dataclass +from typing import Literal, NamedTuple + +import pytest + +from vllm.config.compilation import CompilationMode +from vllm.config.model import RunnerOption +from vllm.logger import init_logger +from vllm.platforms import current_platform +from vllm.utils.torch_utils import is_torch_equal_or_newer + +from ...models.registry import HF_EXAMPLE_MODELS, _HfExamplesInfo +from ...utils import compare_two_settings, create_new_process_for_each_test + +logger = init_logger("test_sequence_parallel") + +VLLM_MULTI_NODE = os.getenv("VLLM_MULTI_NODE", "0") == "1" +NVFP4_MODEL_ID = "nvidia/Llama-3.1-8B-Instruct-NVFP4" +NVFP4_MODEL_INFO = _HfExamplesInfo(NVFP4_MODEL_ID) + + +class ParallelSetup(NamedTuple): + tp_size: int + pp_size: int + fuse_norm_quant: bool + fuse_act_quant: bool + eager_mode: bool + chunked_prefill: bool + + +class SPTestOptions(NamedTuple): + multi_node_only: bool + load_format: str | None = None + model_info: _HfExamplesInfo | None = None + + +@dataclass +class SPTestSettings: + parallel_setups: list[ParallelSetup] + distributed_backends: list[str] + runner: RunnerOption + test_options: SPTestOptions + + @staticmethod + def detailed( + *, + tp_base: int = 2, + pp_base: int = 1, + multi_node_only: bool = False, + runner: RunnerOption = "auto", + load_format: str | None = None, + ): + parallel_setups = [] + for eager_mode_val in [False, True]: + for pp_multiplier in [1, 2]: + for chunked_prefill_val in [False, True]: + parallel_setups.append( + ParallelSetup( + tp_size=tp_base, + pp_size=pp_multiplier * pp_base, + fuse_norm_quant=False, + fuse_act_quant=False, + eager_mode=eager_mode_val, + chunked_prefill=chunked_prefill_val, + ) + ) + return SPTestSettings( + parallel_setups=parallel_setups, + distributed_backends=["mp", "ray"], + runner=runner, + test_options=SPTestOptions( + multi_node_only=multi_node_only, load_format=load_format + ), + ) + + @staticmethod + def fast( + *, + tp_base: int = 2, + pp_base: int = 1, + runner: RunnerOption = "auto", + multi_node_only: bool = False, + load_format: str | None = None, + ): + parallel_setups = [] + for eager_mode_val in [False, True]: + for pp_multiplier in [1, 2]: + for chunked_prefill_val in [False, True]: + parallel_setups.append( + ParallelSetup( + tp_size=tp_base, + pp_size=pp_multiplier * pp_base, + fuse_norm_quant=False, + fuse_act_quant=False, + eager_mode=eager_mode_val, + chunked_prefill=chunked_prefill_val, + ) + ) + return SPTestSettings( + parallel_setups=parallel_setups, + distributed_backends=["mp", "ray"], + runner=runner, + test_options=SPTestOptions( + multi_node_only=multi_node_only, load_format=load_format + ), + ) + + @staticmethod + def fp8_quant( + *, + tp_base: int = 2, + pp_base: int = 1, + runner: RunnerOption = "auto", + multi_node_only: bool = False, + load_format: str | None = None, + ): + parallel_setups = [] + for fusion_val in [False, True]: + parallel_setups.append( + ParallelSetup( + tp_size=tp_base, + pp_size=pp_base, + fuse_norm_quant=fusion_val, + fuse_act_quant=fusion_val, + eager_mode=True, + chunked_prefill=False, + ) + ) + return SPTestSettings( + parallel_setups=parallel_setups, + distributed_backends=["mp", "ray"], + runner=runner, + test_options=SPTestOptions( + multi_node_only=multi_node_only, load_format=load_format + ), + ) + + def iter_params(self, model_id: str): + opts = self.test_options + + for parallel_setup in self.parallel_setups: + for backend in self.distributed_backends: + yield ( + model_id, + parallel_setup, + backend, + self.runner, + opts, + ) + + +def _compare_sp( + model_id: str, + parallel_setup: ParallelSetup, + distributed_backend: str, + runner: RunnerOption, + test_options: SPTestOptions, + num_gpus_available: int, + use_inductor_graph_partition: bool, + fuse_gemm_comms: bool, + enable_prompt_embeds: bool, + *, + method: Literal["generate", "encode"], + is_multimodal: bool, + dtype: str = "float16", +): + ( + tp_size, + pp_size, + fuse_norm_quant, + fuse_act_quant, + eager_mode, + chunked_prefill, + ) = parallel_setup + + multi_node_only = test_options.multi_node_only + load_format = test_options.load_format + + model_info = test_options.model_info or HF_EXAMPLE_MODELS.find_hf_info(model_id) + model_info.check_transformers_version(on_fail="skip") + + trust_remote_code = model_info.trust_remote_code + tokenizer_mode = model_info.tokenizer_mode + hf_overrides = dict(model_info.hf_overrides) + require_embed_inputs = model_info.require_embed_inputs + + if load_format == "dummy": + # Avoid OOM + text_overrides = { + "num_hidden_layers": 4, + "hidden_size": 512, + "intermediate_size": 800, + "num_attention_heads": 4, + "num_key_value_heads": 1, + } + + if is_multimodal: + hf_overrides.update({"text_config": text_overrides}) + else: + hf_overrides.update(text_overrides) + else: + model_info.check_available_online(on_fail="skip") + + if num_gpus_available < tp_size * pp_size: + pytest.skip(f"Need at least {tp_size} x {pp_size} GPUs") + if VLLM_MULTI_NODE and distributed_backend == "mp": + pytest.skip( + "Skipping multi-node pipeline parallel test for " + "multiprocessing distributed backend" + ) + if multi_node_only and not VLLM_MULTI_NODE: + pytest.skip("Not in multi-node setting") + + common_args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + dtype, + "--max-model-len", + "2048", + "--max-num-seqs", + "8", + ] + if chunked_prefill: + common_args.append("--enable-chunked-prefill") + if eager_mode: + common_args.append("-cc.cudagraph_mode=none") + if runner != "auto": + common_args.extend(["--runner", runner]) + if trust_remote_code: + common_args.append("--trust-remote-code") + if tokenizer_mode: + common_args.extend(["--tokenizer-mode", tokenizer_mode]) + if load_format: + common_args.extend(["--load-format", load_format]) + if hf_overrides: + common_args.extend(["--hf-overrides", json.dumps(hf_overrides)]) + if require_embed_inputs: + common_args.extend( + [ + "--skip-tokenizer-init", + "--enable-prompt-embeds", + "--enable-mm-embeds", + ] + ) + elif enable_prompt_embeds: + common_args.append("--enable-prompt-embeds") + + compilation_config = { + "mode": CompilationMode.VLLM_COMPILE, + "compile_sizes": [4, 8], + "pass_config": { + "enable_sp": True, + "fuse_gemm_comms": fuse_gemm_comms, + "fuse_norm_quant": fuse_norm_quant, + "fuse_act_quant": fuse_act_quant, + "fuse_allreduce_rms": False, + "eliminate_noops": True, + "sp_min_token_num": 0, + }, + "use_inductor_graph_partition": use_inductor_graph_partition, + } + if not use_inductor_graph_partition: + compilation_config["splitting_ops"] = [] + + tp_sp_args = [ + *common_args, + "--tensor-parallel-size", + str(tp_size), + "--pipeline-parallel-size", + str(pp_size), + "--distributed-executor-backend", + distributed_backend, + "--compilation_config", + json.dumps(compilation_config), + ] + + tp_args = [ + *common_args, + "--tensor-parallel-size", + str(tp_size), + "--distributed-executor-backend", + "mp", + ] + + compare_two_settings( + model_id, + tp_sp_args, + tp_args, + method=method, + force_v1_runner=True, + ) + + +SP_TEXT_GENERATION_MODELS = { + # [Decoder-only] + "hmellor/tiny-random-LlamaForCausalLM": SPTestSettings.fast(), + "RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8": SPTestSettings.fp8_quant(), +} + +SP_TEST_MODELS = [ + # TODO support other models + # [LANGUAGE GENERATION] + "hmellor/tiny-random-LlamaForCausalLM", + "RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8", +] + + +@pytest.mark.parametrize( + ( + "model_id", + "parallel_setup", + "distributed_backend", + "runner", + "test_options", + ), + [ + params + for model_id, settings in SP_TEXT_GENERATION_MODELS.items() + for params in settings.iter_params(model_id) + if model_id in SP_TEST_MODELS + ], +) +@pytest.mark.parametrize("use_inductor_graph_partition", [True, False]) +@pytest.mark.parametrize("fuse_gemm_comms", [False]) # TODO: enable async TP +@create_new_process_for_each_test() +def test_tp_sp_generation( + model_id: str, + parallel_setup: ParallelSetup, + distributed_backend: str, + runner: RunnerOption, + test_options: SPTestOptions, + num_gpus_available, + use_inductor_graph_partition: bool, + fuse_gemm_comms: bool, +): + if use_inductor_graph_partition and not is_torch_equal_or_newer("2.9.0.dev"): + pytest.skip("inductor graph partition is only available in PyTorch 2.9+") + + # Skip FP8 SP-only test on sm89 (compute capability 8.9) + if ( + "fp8" in model_id.lower() + and current_platform.get_device_capability() < (9, 0) + and (not fuse_gemm_comms) + ): + pytest.skip("FP8 reduction support begins with sm90 capable devices.") + + _compare_sp( + model_id, + parallel_setup, + distributed_backend, + runner, + test_options, + num_gpus_available, + use_inductor_graph_partition, + fuse_gemm_comms=fuse_gemm_comms, + enable_prompt_embeds=False, + method="generate", + is_multimodal=False, + ) + + +# Focused regression test for the SP + prompt_embeds graph-rewrite path. +# Covers pp_size=1 (SP only) and pp_size=2 (SP + PP); kept small on purpose so +# we don't double the matrix of `test_tp_sp_generation` above. +SP_PROMPT_EMBEDS_PARALLEL_SETUPS = [ + ParallelSetup( + tp_size=2, + pp_size=pp_size, + fuse_norm_quant=False, + fuse_act_quant=False, + eager_mode=False, + chunked_prefill=False, + ) + for pp_size in [1, 2] +] + + +@pytest.mark.parametrize("parallel_setup", SP_PROMPT_EMBEDS_PARALLEL_SETUPS) +@pytest.mark.parametrize("use_inductor_graph_partition", [True, False]) +@create_new_process_for_each_test() +def test_tp_sp_generation_prompt_embeds( + parallel_setup: ParallelSetup, + num_gpus_available, + use_inductor_graph_partition: bool, +): + if use_inductor_graph_partition and not is_torch_equal_or_newer("2.9.0.dev"): + pytest.skip("inductor graph partition is only available in PyTorch 2.9+") + + _compare_sp( + "hmellor/tiny-random-LlamaForCausalLM", + parallel_setup, + distributed_backend="mp", + runner="auto", + test_options=SPTestOptions(multi_node_only=False, load_format=None), + num_gpus_available=num_gpus_available, + use_inductor_graph_partition=use_inductor_graph_partition, + fuse_gemm_comms=False, + enable_prompt_embeds=True, + method="generate", + is_multimodal=False, + ) + + +@create_new_process_for_each_test() +def test_tp_sp_nvfp4_generation(num_gpus_available: int): + if ( + not current_platform.is_cuda() + or not current_platform.is_device_capability_family(100) + ): + pytest.skip("NVFP4 requires Blackwell") + + _compare_sp( + NVFP4_MODEL_ID, + ParallelSetup( + tp_size=2, + pp_size=1, + fuse_norm_quant=True, + fuse_act_quant=True, + eager_mode=True, + chunked_prefill=False, + ), + "mp", + "auto", + SPTestOptions( + multi_node_only=False, + load_format="dummy", + model_info=NVFP4_MODEL_INFO, + ), + num_gpus_available, + use_inductor_graph_partition=False, + fuse_gemm_comms=False, + enable_prompt_embeds=False, + method="generate", + is_multimodal=False, + dtype="bfloat16", + ) diff --git a/asian_ai/performance/vLLM/tests/compile/fullgraph/__init__.py b/asian_ai/performance/vLLM/tests/compile/fullgraph/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/compile/fullgraph/test_basic_correctness.py b/asian_ai/performance/vLLM/tests/compile/fullgraph/test_basic_correctness.py new file mode 100644 index 0000000000000000000000000000000000000000..9035a0fac41c07211b9b83dedff56b2b7f3338a4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/fullgraph/test_basic_correctness.py @@ -0,0 +1,162 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import dataclasses + +import pytest + +from vllm.config import CompilationMode +from vllm.platforms import current_platform + +from ...utils import compare_all_settings + +ATTN_BACKEND = "FLASH_ATTN" if not current_platform.is_rocm() else "ROCM_ATTN" + + +@dataclasses.dataclass +class TestSetting: + model: str + model_args: list[str] + pp_size: int + tp_size: int + attn_backend: str + method: str + + +# we cannot afford testing the full Cartesian product +# of all models and all modes +@pytest.mark.parametrize( + "test_setting", + [ + # basic llama model + TestSetting( + model="meta-llama/Llama-3.2-1B-Instruct", + model_args=["--max-model-len", "2048"], + pp_size=2, + tp_size=2, + attn_backend=ATTN_BACKEND, + method="generate", + ), + # llama model with quantization + TestSetting( + model="TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ", + model_args=["--quantization", "gptq", "--max-model-len", "2048"], + pp_size=1, + tp_size=1, + attn_backend=ATTN_BACKEND, + method="generate", + ), + # MoE model + TestSetting( + model="ibm/PowerMoE-3b", + model_args=["--max-model-len", "2048"], + pp_size=1, + tp_size=2, + attn_backend=ATTN_BACKEND, + method="generate", + ), + # embedding model + TestSetting( + model="BAAI/bge-multilingual-gemma2", + model_args=[ + "--runner", + "pooling", + "--dtype", + "bfloat16", + "--max-model-len", + "2048", + ], + pp_size=1, + tp_size=1, + attn_backend=ATTN_BACKEND, + method="encode", + ), + pytest.param( + TestSetting( + model="BAAI/bge-base-en-v1.5", + model_args=["--runner", "pooling"], + pp_size=1, + tp_size=1, + attn_backend="FLASH_ATTN", + method="encode", + ), + marks=pytest.mark.skipif( + current_platform.is_rocm(), + reason="Encoder self-attention is not implemented for ROCm", + ), + ), + # vision language model + # See https://github.com/vllm-project/vllm/issues/26716. + # TestSetting( + # model="microsoft/Phi-3.5-vision-instruct", + # model_args=["--trust-remote-code", "--max-model-len", "2048"], + # pp_size=2, + # tp_size=1, + # attn_backend="FLASH_ATTN", + # method="generate_with_image", + # ), + ], +) +def test_compile_correctness( + test_setting: TestSetting, +): + # this test is run under multiple suits, with different GPUs. + # make sure we only run the test with correct CUDA devices. + # don't use "<", as it will duplicate the tests. + model = test_setting.model + model_args = test_setting.model_args + pp_size = test_setting.pp_size + tp_size = test_setting.tp_size + attn_backend = test_setting.attn_backend + method = test_setting.method + if current_platform.device_count() < pp_size * tp_size: + pytest.skip( + f"Need at least {pp_size}*{tp_size} CUDA gpus but got " + f"{current_platform.device_count()}" + ) + + final_args = [ + *model_args, + "-pp", + str(pp_size), + "-tp", + str(tp_size), + "-cc.cudagraph_mode=none", + f"--attention-backend={attn_backend}", + ] + + all_args: list[list[str]] = [] + all_envs: list[dict[str, str] | None] = [] + + for comp_mode in [ + CompilationMode.STOCK_TORCH_COMPILE, + CompilationMode.DYNAMO_TRACE_ONCE, + CompilationMode.VLLM_COMPILE, + ]: + for mode in [CompilationMode.NONE, comp_mode]: + all_args.append( + final_args + [f"-cc.mode={mode.name}", "-cc.backend=inductor"] + ) + all_envs.append({}) + + # inductor will change the output, so we only compare if the output + # is close, not exactly the same. + compare_all_settings( + model, + all_args, + all_envs, + method=method if method != "generate" else "generate_close", + force_v1_runner=True, + ) + all_envs.clear() + all_args.clear() + + for mode in [ + CompilationMode.NONE, + CompilationMode.STOCK_TORCH_COMPILE, + CompilationMode.DYNAMO_TRACE_ONCE, + CompilationMode.VLLM_COMPILE, + ]: + all_args.append(final_args + [f"-cc.mode={mode.name}", "-cc.backend=eager"]) + all_envs.append({}) + + compare_all_settings(model, all_args, all_envs, method=method, force_v1_runner=True) diff --git a/asian_ai/performance/vLLM/tests/compile/fullgraph/test_full_cudagraph.py b/asian_ai/performance/vLLM/tests/compile/fullgraph/test_full_cudagraph.py new file mode 100644 index 0000000000000000000000000000000000000000..eecd455600f0cbef5ad79bf676de58f2b59305ac --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/fullgraph/test_full_cudagraph.py @@ -0,0 +1,172 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import contextlib +import os +import weakref + +import pytest + +from tests.utils import wait_for_gpu_memory_to_clear +from tests.v1.attention.utils import full_cg_backend_configs as backend_configs +from vllm import LLM, SamplingParams +from vllm.config import CompilationConfig +from vllm.platforms import current_platform +from vllm.utils.torch_utils import is_torch_equal_or_newer +from vllm.v1.attention.backends.registry import AttentionBackendEnum + + +@contextlib.contextmanager +def temporary_environ(env_vars): + """ + Temporarily set environment variables and restore them afterward. + We have to do this vs monkeypatch because monkeypatch doesn't work + with "module" scoped fixtures. + """ + original_env = {k: os.environ.get(k) for k in env_vars} + try: + os.environ.update(env_vars) + yield + finally: + for k, v in original_env.items(): + if v is None: + os.environ.pop(k, None) + else: + os.environ[k] = v + + +model_backends_full_cudagraph = [] + +# deepseek-ai/DeepSeek-V2-Lite with MLA +MLA_backends = ["FlashMLA", "FlashAttentionMLA", "CutlassMLA"] +for mla_backend in MLA_backends: + model_backends_full_cudagraph.append( + ("deepseek-ai/DeepSeek-V2-Lite", backend_configs[mla_backend]) + ) + +# Qwen/Qwen2-1.5B-Instruct with other backends +other_backend_configs = [ + backend_configs[c] for c in backend_configs if c not in MLA_backends +] +for backend_config in other_backend_configs: + model_backends_full_cudagraph.append(("Qwen/Qwen2-1.5B-Instruct", backend_config)) + + +@pytest.fixture(scope="class") +def llm_pair(request): + model, backend_config, use_inductor_graph_partition = request.param + backend_config.comp_config["use_inductor_graph_partition"] = ( + use_inductor_graph_partition + ) + + if use_inductor_graph_partition and not is_torch_equal_or_newer("2.9.0.dev"): + pytest.skip("Inductor graph partition only supported in torch>=2.9") + + # Dynamically skip test if GPU capability is not met + if ( + backend_config.specific_gpu_arch + and backend_config.specific_gpu_arch != current_platform.get_device_capability() + ): + if backend_config.specific_gpu_arch == (9, 0): + pytest.skip("Only Hopper GPUs support FA3 and FlashMLA") + elif backend_config.specific_gpu_arch == (10, 0): + pytest.skip("Only Blackwell GPUs support Cutlass MLA") + + # FlashInfer is not supported on ROCm + if backend_config == AttentionBackendEnum.FLASHINFER and current_platform.is_rocm(): + pytest.skip("FlashInfer is not supported on ROCm") + + env_vars = { + # Force native sampler to avoid potential nondeterminism in FlashInfer + # when per-request generators are not used in V1. + "VLLM_USE_FLASHINFER_SAMPLER": "0", + } + with temporary_environ(env_vars): + full = LLM( + model=model, + gpu_memory_utilization=0.43, + trust_remote_code=True, + max_model_len=1024, + max_num_seqs=128, + compilation_config=CompilationConfig(**backend_config.comp_config), + generation_config="vllm", + seed=42, + ) + piecewise = LLM( + model=model, + gpu_memory_utilization=0.43, + trust_remote_code=True, + max_model_len=1024, + max_num_seqs=128, + compilation_config=CompilationConfig(cudagraph_mode="PIECEWISE"), + generation_config="vllm", + seed=42, + ) + + # PyTest caches the fixture values so we use weakref.proxy to enable GC + yield weakref.proxy(full), weakref.proxy(piecewise) + del full + del piecewise + + wait_for_gpu_memory_to_clear( + devices=[0], + threshold_ratio=0.1, + ) + + +@pytest.mark.parametrize( + "llm_pair", + [ + pytest.param((model, backend_config, use_inductor_graph_partition)) + for model, backend_config in model_backends_full_cudagraph + for use_inductor_graph_partition in [True, False] + ], + indirect=True, +) +class TestFullCUDAGraph: + """ + Use a class such that an llm pair is constructed once for all + batch_size/max_tokens combinations and released immediately after. + + Module-scope fixtures would stick around the whole time, + meaning there would be multiple LLM instances hogging memory simultaneously. + """ + + @pytest.mark.parametrize( + ("batch_size", "max_tokens"), + [ + (1, 10), + (7, 10), + (16, 10), + (25, 10), + (32, 10), + (45, 10), + (64, 10), + (123, 10), + (8, 5), + (8, 30), + ], + ) + def test_full_cudagraph(self, batch_size, max_tokens, llm_pair: tuple[LLM, LLM]): + """ + Test various batch sizes and max_tokens to ensure that the + full cudagraph compilation works for padded cases too. + """ + + full_cudagraph_llm, piecewise_llm = llm_pair + + prompts = ["the quick brown fox"] * batch_size + # Use purely greedy decoding to avoid top-p truncation sensitivity + # that can amplify tiny numeric differences across runtimes. + sampling_params = SamplingParams( + temperature=0.0, max_tokens=max_tokens, top_p=1.0 + ) + + piecewise_responses = piecewise_llm.generate(prompts, sampling_params) + full_responses = full_cudagraph_llm.generate(prompts, sampling_params) + + # Check that all responses are the same + for piecewise_res, full_res in zip(piecewise_responses, full_responses): + assert ( + piecewise_res.outputs[0].text.lower() + == full_res.outputs[0].text.lower() + ) diff --git a/asian_ai/performance/vLLM/tests/compile/fullgraph/test_full_graph.py b/asian_ai/performance/vLLM/tests/compile/fullgraph/test_full_graph.py new file mode 100644 index 0000000000000000000000000000000000000000..ec208634eed58dc16b31b2026b09d81f9ad14b5f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/fullgraph/test_full_graph.py @@ -0,0 +1,249 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import tempfile +from pathlib import Path +from typing import Any + +import pytest +import torch + +from tests.quantization.utils import is_quant_method_supported +from vllm import LLM, SamplingParams +from vllm.config import CompilationConfig, CompilationMode, CUDAGraphMode, PassConfig +from vllm.platforms import current_platform +from vllm.utils.torch_utils import is_torch_equal_or_newer +from vllm.v1.attention.backends.registry import AttentionBackendEnum + +from ...utils import create_new_process_for_each_test + + +def models_list(*, all: bool = True, keywords: list[str] | None = None): + TEST_MODELS: list[tuple[str, dict[str, Any]]] = [ + ("facebook/opt-125m", {}), + ( + "neuralmagic/Llama-3.2-1B-Instruct-FP8-dynamic", + {"dtype": torch.float16}, + ), + ("meta-llama/Llama-3.2-1B-Instruct", {}), + ] + + if all: + TEST_MODELS.extend( + [ + ("neuralmagic/Llama-3.2-1B-Instruct-quantized.w8a8", {}), + ( + "nm-testing/tinyllama-oneshot-w8w8-test-static-shape-change", + {"dtype": torch.float16}, + ), + ] + ) + + if is_quant_method_supported("gptq"): + TEST_MODELS.append( + ("TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ", {"quantization": "gptq"}) + ) + + if is_quant_method_supported("gptq_marlin"): + TEST_MODELS.append( + ( + "TheBloke/TinyLlama-1.1B-Chat-v1.0-GPTQ", + {"quantization": "gptq_marlin"}, + ) + ) + + if not current_platform.is_rocm() and is_quant_method_supported("awq"): + TEST_MODELS.append( + ("TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ", {"quantization": "AWQ"}) + ) + + if keywords is None: + return TEST_MODELS + + # filter by keywords + pred = lambda model: any(keyword in model[0] for keyword in keywords) + return list(filter(pred, TEST_MODELS)) + + +@pytest.mark.parametrize( + "compilation_mode", + [CompilationMode.DYNAMO_TRACE_ONCE, CompilationMode.VLLM_COMPILE], +) +@pytest.mark.parametrize("model, model_kwargs", models_list(all=True)) +@create_new_process_for_each_test() +def test_full_graph( + monkeypatch: pytest.MonkeyPatch, + model: str, + model_kwargs: dict[str, Any], + compilation_mode: int, +): + if ( + "w8a8" in model + or "w8w8" in model + and current_platform.has_device_capability((10, 0)) + ): + # int8 removed on Blackwell: + pytest.skip("int8 support removed on Blackwell") + + with monkeypatch.context(): + print(f"MODEL={model}") + + run_model(compilation_mode, model, **model_kwargs) + + +# TODO(luka) add other supported compilation config scenarios here +@pytest.mark.parametrize( + "compilation_config, model, model_kwargs", + [ + # additional compile sizes, only some of the models + ( + CompilationConfig(mode=CompilationMode.VLLM_COMPILE, compile_sizes=[1, 2]), + *model_info, + ) + for model_info in models_list(all=False) + ] + + [ + # RMSNorm + quant fusion, only 8-bit quant models + ( + CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=["+rms_norm"], + pass_config=PassConfig( + fuse_norm_quant=True, fuse_act_quant=True, eliminate_noops=True + ), + ), + *model_info, + ) + for model_info in models_list(keywords=["FP8-dynamic", "quantized.w8a8"]) + ] + + [ + # Test depyf integration works + ( + CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + debug_dump_path=Path(tempfile.gettempdir()), + ), + "facebook/opt-125m", + {}, + ), + ] + + [ + # graph inductor partition + ( + CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + # inductor graph partition uses + # torch._C.Tag.cudagraph_unsafe to specify splitting ops + use_inductor_graph_partition=True, + cudagraph_mode=CUDAGraphMode.PIECEWISE, + compile_sizes=[1, 2], + ), + *model_info, + ) + for model_info in models_list(all=False) + if is_torch_equal_or_newer("2.9.0.dev") + ] + + [ + # Test get_raw_stream patch with compile_sizes + # This tests that TorchInductor autotune works correctly with get_raw_stream + # patch in torch 2.9 and without patch in torch 2.10+ + ( + CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + compile_sizes=[1, 2], # Triggers autotune which uses get_raw_stream + cudagraph_mode=CUDAGraphMode.NONE, + ), + "facebook/opt-125m", + {}, + ), + ], +) +# only test some of the models +@create_new_process_for_each_test() +def test_custom_compile_config( + compilation_config: CompilationConfig, + model: str, + model_kwargs: dict[str, Any], +): + if ( + "w8a8" in model + or "w8w8" in model + and current_platform.has_device_capability((10, 0)) + ): + # int8 removed on Blackwell: + pytest.skip("int8 support removed on Blackwell") + + if compilation_config.use_inductor_graph_partition and not is_torch_equal_or_newer( + "2.9.0.dev" + ): + pytest.skip("inductor graph partition is only available in PyTorch 2.9+") + + print(f"MODEL={model}") + run_model(compilation_config, model, **model_kwargs) + + +@pytest.mark.parametrize( + "compilation_mode", + [CompilationMode.NONE, CompilationMode.VLLM_COMPILE], +) +@pytest.mark.parametrize( + "model, backend", + [ + ("Qwen/Qwen2-0.5B", None), # Standard attention model + ( + "deepseek-ai/DeepSeek-V2-Lite", + AttentionBackendEnum.FLASHINFER_MLA, + ), # MLA (Multi-head Latent Attention) model + ], +) +def test_fp8_kv_scale_compile( + compilation_mode: int, + model: str, + backend: AttentionBackendEnum | None, +): + model_kwargs = { + "quantization": "fp8", + "kv_cache_dtype": "fp8_e4m3", + "calculate_kv_scales": True, + "max_model_len": 512, + } + if backend: + model_kwargs["attention_config"] = {"backend": backend.name} + + run_model(compilation_mode, model, **model_kwargs) + + +def run_model(compile_config: int | CompilationConfig, model: str, **model_kwargs): + compilation_config = ( + compile_config + if isinstance(compile_config, CompilationConfig) + else CompilationConfig(mode=compile_config) + ) + + prompts = [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + "The future of AI is", + ] + sampling_params = SamplingParams(temperature=0) + # Allow override from model_kwargs + model_kwargs = {"tensor_parallel_size": 1, **model_kwargs} + model_kwargs = {"disable_custom_all_reduce": True, **model_kwargs} + + # No cudagraphs by default + if compilation_config.cudagraph_mode is None: + compilation_config.cudagraph_mode = CUDAGraphMode.NONE + + llm = LLM( + model=model, + compilation_config=compilation_config, + **model_kwargs, + ) + outputs = llm.generate(prompts, sampling_params) + + # Print the outputs. + for output in outputs: + prompt = output.prompt + generated_text = output.outputs[0].text + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") diff --git a/asian_ai/performance/vLLM/tests/compile/fullgraph/test_multimodal_compile.py b/asian_ai/performance/vLLM/tests/compile/fullgraph/test_multimodal_compile.py new file mode 100644 index 0000000000000000000000000000000000000000..0ead5fe462ed3fbbbc076ab28b6151a0325bce2e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/fullgraph/test_multimodal_compile.py @@ -0,0 +1,110 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest + +from vllm.compilation.counter import compilation_counter +from vllm.config import VllmConfig +from vllm.config.compilation import CompilationMode +from vllm.platforms import current_platform + + +def test_compile(): + vllm_config = VllmConfig() + # Default configuration does not compile mm encoder + assert not vllm_config.compilation_config.compile_mm_encoder + + +# forked needed to workaround https://github.com/vllm-project/vllm/issues/21073 +@pytest.mark.forked +@pytest.mark.skipif(not current_platform.is_cuda(), reason="Skip if not cuda") +def test_qwen2_5_vl_compilation(vllm_runner, monkeypatch): + """Test that Qwen2.5-VL vision submodules are compiled. + + This test verifies that the 3 vision submodules (Qwen2_5_VisionPatchEmbed, + Qwen2_5_VisionBlock, and Qwen2_5_VisionPatchMerger) are properly tagged + for compilation by checking that num_models_seen increases by at least 3. + """ + # Disable multiprocessing so that the counter is in the same process + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + + with ( + # NOTE: Qwen2.5-VL has 35 models in total - the LLM backend + # Vision Patch Embed, Vision Patch Merger, and then 32 Vision Blocks + # (one for each layer) - in the future, we should fix vLLM compilation + # logic to handle this case and only compile the Vision submodules once + # and reuse the compiled code for all layers + # See https://github.com/vllm-project/vllm/issues/27590 + compilation_counter.expect(num_models_seen=35), + vllm_runner( + "Qwen/Qwen2.5-VL-3B-Instruct", + max_model_len=2048, + gpu_memory_utilization=0.8, + compilation_config={ + "mode": CompilationMode.VLLM_COMPILE, + "compile_mm_encoder": True, + }, + ) as _, + ): + pass + + +# forked needed to workaround https://github.com/vllm-project/vllm/issues/21073 +@pytest.mark.forked +@pytest.mark.skipif(not current_platform.is_cuda(), reason="Skip if not cuda") +def test_qwen2_5_vl_no_vit_compilation(vllm_runner, monkeypatch): + """Test that Qwen2.5-VL vision submodules are not compiled when the + config is passed off + """ + # Disable multiprocessing so that the counter is in the same process + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + + with ( + compilation_counter.expect(num_models_seen=1), + vllm_runner( + "Qwen/Qwen2.5-VL-3B-Instruct", + max_model_len=2048, + gpu_memory_utilization=0.8, + compilation_config={ + "mode": CompilationMode.VLLM_COMPILE, + "compile_mm_encoder": False, + }, + ) as _, + ): + pass + + +# forked needed to workaround https://github.com/vllm-project/vllm/issues/21073 +# Requires Cuda and 8 gpus as well +@pytest.mark.forked +@pytest.mark.skip(reason="Skipping due to CI resource constraints") +def test_mllama4_vit_compilation(vllm_runner, monkeypatch): + """Test that Mllama4 vision submodules are compiled. + + This test verifies that the 2 vision submodules (Llama4VisionEncoder, + Llama4VisionPixelShuffleMLP) are properly tagged + for compilation by checking that num_models_seen increases to 3. + + However since we are using TP=8, we compilation_counter will not + work properly so we will just check the run succeeds rn + """ + # Disable multiprocessing so that the counter is in the same process + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + + with ( + monkeypatch.context(), + # TODO: Since we require TP=8, this messes with the compilation + # counter. We should fix this in the future, but leave for now + # to make sure that compilation runs (no crash) with llama vision encoder + compilation_counter.expect(num_models_seen=0), + vllm_runner( + "meta-llama/Llama-4-Scout-17B-16E-Instruct", + max_model_len=512, + gpu_memory_utilization=0.8, + tensor_parallel_size=8, + compilation_config={ + "mode": CompilationMode.VLLM_COMPILE, + "compile_mm_encoder": True, + }, + ), + ): + pass diff --git a/asian_ai/performance/vLLM/tests/compile/fullgraph/test_multiple_graphs.py b/asian_ai/performance/vLLM/tests/compile/fullgraph/test_multiple_graphs.py new file mode 100644 index 0000000000000000000000000000000000000000..0fc987ce0866cfdc9fb5b83bffced54a2dbef5b8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/fullgraph/test_multiple_graphs.py @@ -0,0 +1,326 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Test (piecewise) compilation with a simple model where multiple submodules +are compiled and graph captured separately. +""" + +import pytest +import torch +from torch import nn + +from vllm.compilation.backends import set_model_tag +from vllm.compilation.counter import compilation_counter +from vllm.compilation.decorators import ignore_torch_compile, support_torch_compile +from vllm.config import ( + CompilationConfig, + CompilationMode, + CUDAGraphMode, + VllmConfig, + set_current_vllm_config, +) +from vllm.forward_context import BatchDescriptor, set_forward_context +from vllm.utils.torch_utils import is_torch_equal_or_newer + +from ...utils import create_new_process_for_each_test + +# This import automatically registers `torch.ops.silly.attention` +from .. import silly_attention # noqa: F401 + +BATCH_SIZE = 32 +MLP_SIZE = 128 +HIDDEN_SIZE = 1024 +RANDOM_SEED = 0 + + +@support_torch_compile +class ParentModel(nn.Module): + def __init__(self, *, vllm_config: VllmConfig, prefix: str = "", **kwargs) -> None: + super().__init__() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return x + + +class Attention(nn.Module): + def __init__(self, mlp_size: int, hidden_size: int) -> None: + super().__init__() + self.pre_attn = nn.Linear(mlp_size, hidden_size, bias=False) + self.post_attn = nn.Linear(hidden_size, mlp_size, bias=False) + self.rms_norm_weight = nn.Parameter(torch.ones(hidden_size)) + + # Initialize to same weights for testing + nn.init.xavier_normal_( + self.pre_attn.weight.data, + generator=torch.Generator().manual_seed(RANDOM_SEED), + gain=0.001, + ) + nn.init.xavier_normal_( + self.post_attn.weight.data, + generator=torch.Generator().manual_seed(RANDOM_SEED), + gain=0.001, + ) + + def rms_norm_ref(self, x: torch.Tensor) -> torch.Tensor: + x_f32 = x.float() + return ( + x_f32 + * torch.rsqrt(torch.mean(x_f32.square(), dim=-1, keepdim=True) + 1e-6) + * self.rms_norm_weight + ).to(x.dtype) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = self.pre_attn(x) + x = self.rms_norm_ref(x) + attn_output = torch.empty_like(x) + torch.ops.silly.attention(x, x, x, attn_output) + x = attn_output + x = self.rms_norm_ref(x) + x = self.post_attn(x) + return x + + +@support_torch_compile +class CompiledAttention(nn.Module): + def __init__( + self, + *, + mlp_size: int, + hidden_size: int, + vllm_config: VllmConfig, + prefix: str = "", + **kwargs, + ) -> None: + super().__init__() + self.attn = Attention(mlp_size, hidden_size) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.attn(x) + + +@support_torch_compile +class CompiledAttentionTwo(CompiledAttention): + def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.attn(x) + x + + +@ignore_torch_compile +class SimpleModelWithTwoGraphs(ParentModel): + def __init__( + self, + *, + mlp_size: int, + hidden_size: int, + vllm_config: VllmConfig, + prefix: str = "", + **kwargs, + ) -> None: + super().__init__(vllm_config=vllm_config, prefix=prefix) + # Test will fail without set_model_tag here with error: + # "ValueError: too many values to unpack (expected 3)" + # This is because CompiledAttention and CompiledAttentionTwo + # have different implementations but the same torch.compile + # cache dir will be used as default prefix is 'model_tag' + with set_model_tag("attn_one"): + self.attn_one = CompiledAttention( + mlp_size=mlp_size, + hidden_size=hidden_size, + vllm_config=vllm_config, + prefix=f"{prefix}.attn_one", + ) + with set_model_tag("attn_two"): + self.attn_two = CompiledAttentionTwo( + mlp_size=mlp_size, + hidden_size=hidden_size, + vllm_config=vllm_config, + prefix=f"{prefix}.attn_two", + ) + + self.hidden_states = torch.zeros((BATCH_SIZE, MLP_SIZE)).cuda() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + bsz = x.shape[0] + # CUDAGraph expects same tensor addresses for each run + self.hidden_states[:bsz].copy_(x) + x = self.attn_one(self.hidden_states[:bsz]) + self.hidden_states[:bsz].copy_(x) + x = self.attn_two(self.hidden_states[:bsz]) + return x + + +@torch.inference_mode +def run_model( + vllm_config: VllmConfig, + model: nn.Module, + inputs: torch.Tensor, + cudagraph_runtime_mode: CUDAGraphMode, +): + with set_forward_context({}, vllm_config=vllm_config): + # warmup for the model with cudagraph_mode NONE + model(inputs) + + # simulate cudagraphs capturing + with set_forward_context( + {}, + vllm_config=vllm_config, + cudagraph_runtime_mode=cudagraph_runtime_mode, + batch_descriptor=BatchDescriptor( + num_tokens=2, + ), + ): + model(inputs[:2]) + with set_forward_context( + {}, + vllm_config=vllm_config, + cudagraph_runtime_mode=cudagraph_runtime_mode, + batch_descriptor=BatchDescriptor( + num_tokens=1, + ), + ): + model(inputs[:1]) + + # simulate cudagraphs replay + with set_forward_context( + {}, + vllm_config=vllm_config, + cudagraph_runtime_mode=cudagraph_runtime_mode, + batch_descriptor=BatchDescriptor( + num_tokens=2, + ), + ): + output = model(inputs[:2]) + + output = output.cpu() + return output.cpu() + + +@pytest.mark.parametrize("use_inductor_graph_partition", [False, True]) +@pytest.mark.parametrize("use_bytecode_hook", [True, False]) +@create_new_process_for_each_test("spawn") +def test_multi_graph_piecewise_compile( + use_inductor_graph_partition: bool, use_bytecode_hook: bool, monkeypatch +): + # Set the environment variable for this test + monkeypatch.setenv("VLLM_USE_BYTECODE_HOOK", "1" if use_bytecode_hook else "0") + + if use_inductor_graph_partition and not is_torch_equal_or_newer("2.9.0.dev"): + pytest.skip("inductor graph partition is only available in PyTorch 2.9+") + + outputs = [] + + # vllmcompile compile + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + cudagraph_mode=CUDAGraphMode.PIECEWISE, + splitting_ops=["silly::attention"], + cudagraph_capture_sizes=[1, 2], + use_inductor_graph_partition=use_inductor_graph_partition, + ) + ) + cudagraph_runtime_mode = CUDAGraphMode.PIECEWISE + + with set_current_vllm_config(vllm_config): + model = ( + SimpleModelWithTwoGraphs( + mlp_size=MLP_SIZE, + hidden_size=HIDDEN_SIZE, + vllm_config=vllm_config, + prefix="", + ) + .eval() + .cuda() + ) + + # Pre-allocate memory for CUDAGraph which expects + # static tensor addresses + inputs = torch.randn(BATCH_SIZE, MLP_SIZE).cuda() + + if use_inductor_graph_partition: + # Splitting happens at Inductor lowering level, + # total piecewise fx graphs is equal to total graphs + num_piecewise_fx = 2 + num_piecewise_capturable_fx = 2 + else: + # attn_one, attn_two each has 3 piecewise graphs + # (pre attn, post attn, silly_attention) each + num_piecewise_fx = 6 + # attn_one, attn_two has pre attn and post attn each, total=4 + num_piecewise_capturable_fx = 4 + + with compilation_counter.expect( + num_graphs_seen=2, # two graphs for the model + num_piecewise_graphs_seen=num_piecewise_fx, + num_piecewise_capturable_graphs_seen=num_piecewise_capturable_fx, + num_backend_compilations=num_piecewise_capturable_fx, + num_cudagraph_captured=8, # num_cudagraph_sizes * num_partitions + ): + outputs.append(run_model(vllm_config, model, inputs, cudagraph_runtime_mode)) + + # no compile or cudagraph + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.NONE, + ) + ) + cudagraph_runtime_mode = CUDAGraphMode.NONE + + with set_current_vllm_config(vllm_config): + model = ( + SimpleModelWithTwoGraphs( + mlp_size=MLP_SIZE, + hidden_size=HIDDEN_SIZE, + vllm_config=vllm_config, + prefix="", + ) + .eval() + .cuda() + ) + + with compilation_counter.expect( + num_graphs_seen=0, + num_piecewise_graphs_seen=0, + num_piecewise_capturable_graphs_seen=0, + num_backend_compilations=0, + num_cudagraph_captured=0, + ): + outputs.append(run_model(vllm_config, model, inputs, cudagraph_runtime_mode)) + + # piecewise compile without CUDA graph + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + cudagraph_mode=CUDAGraphMode.NONE, + splitting_ops=["silly::attention"], + use_inductor_graph_partition=use_inductor_graph_partition, + ) + ) + cudagraph_runtime_mode = CUDAGraphMode.PIECEWISE + + with set_current_vllm_config(vllm_config): + model = ( + SimpleModelWithTwoGraphs( + mlp_size=MLP_SIZE, + hidden_size=HIDDEN_SIZE, + vllm_config=vllm_config, + prefix="", + ) + .eval() + .cuda() + ) + + with compilation_counter.expect( + num_graphs_seen=2, + num_piecewise_graphs_seen=num_piecewise_fx, + num_piecewise_capturable_graphs_seen=num_piecewise_capturable_fx, + num_backend_compilations=num_piecewise_capturable_fx, + num_cudagraph_captured=0, # no cudagraph captured + ): + outputs.append(run_model(vllm_config, model, inputs, cudagraph_runtime_mode)) + + # Generally don't expect outputs with and without inductor + # to be bitwise equivalent + assert torch.allclose(outputs[0], outputs[1]) + + # Expect bitwise equivalence using inductor w/ and w/o cudagraph + assert torch.equal(outputs[0], outputs[2]) diff --git a/asian_ai/performance/vLLM/tests/compile/fullgraph/test_simple.py b/asian_ai/performance/vLLM/tests/compile/fullgraph/test_simple.py new file mode 100644 index 0000000000000000000000000000000000000000..5ce420408dd18d9dbeb02d0396be5e924886d30c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/fullgraph/test_simple.py @@ -0,0 +1,209 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Test the piecewise compilation with a simple model so that we +can exactly calculate the expected output and side effects. +""" + +import pytest +import torch +from torch import nn + +from vllm.compilation.counter import compilation_counter +from vllm.compilation.decorators import support_torch_compile +from vllm.config import ( + CompilationConfig, + CompilationMode, + CUDAGraphMode, + VllmConfig, + set_current_vllm_config, +) +from vllm.forward_context import BatchDescriptor, set_forward_context +from vllm.utils.torch_utils import is_torch_equal_or_newer + +from ...utils import create_new_process_for_each_test + +# This import automatically registers `torch.ops.silly.attention` +from ..silly_attention import get_global_counter, reset_global_counter + + +# Custom op that returns an unbacked symint during graph capture +@torch.library.custom_op("mylib::foo", mutates_args=()) +def foo(x: torch.Tensor) -> int: + return 3 + + +@foo.register_fake +def _(x): + return torch.library.get_ctx().new_dynamic_size() + + +@support_torch_compile +class SillyModel(nn.Module): + def __init__( + self, + *, + vllm_config: VllmConfig, + prefix: str = "", + intermediate_unbacked=False, + **kwargs, + ) -> None: + super().__init__() + self.intermediate_unbacked = intermediate_unbacked + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """ + Overall effect: + x = 3 * x + 19 + global_counter += 2 + """ + x = x + 1 + x = x + 2 + out = torch.empty_like(x) + torch.ops.silly.attention(x, x, x, out) + x = out + x = x - 2 + + if self.intermediate_unbacked: + # Test for unbacked symints: the following is a fancy way to multiply by 1 + u0 = foo(x) + ones = x.new_ones(x.shape[0], u0).sum(-1) / 3 + x = x * ones + + x = x - 1 + out = torch.empty_like(x) + torch.ops.silly.attention(x, x, x, out) + x = out + x = x + 1 + return x + + +@torch._dynamo.config.patch(capture_dynamic_output_shape_ops=True) +def _run_simple_model( + splitting_ops, + use_inductor_graph_partition, + backend, + expected_num_piecewise_graphs_seen, + expected_num_piecewise_capturable_graphs_seen, + expected_num_backend_compilations, + expected_num_cudagraph_captured, + *, + intermediate_unbacked=False, +): + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + backend=backend, + splitting_ops=splitting_ops, + use_inductor_graph_partition=use_inductor_graph_partition, + cudagraph_copy_inputs=True, + cudagraph_capture_sizes=[1, 2], + ) + ) + with set_current_vllm_config(vllm_config): + model = SillyModel( + vllm_config=vllm_config, + prefix="", + intermediate_unbacked=intermediate_unbacked, + ) + + inputs = torch.randn(100).cuda() + + with ( + compilation_counter.expect( + num_graphs_seen=1, # one graph for the model + num_piecewise_graphs_seen=expected_num_piecewise_graphs_seen, + num_piecewise_capturable_graphs_seen=expected_num_piecewise_capturable_graphs_seen, + num_backend_compilations=expected_num_backend_compilations, + num_cudagraph_captured=expected_num_cudagraph_captured, + ), + set_forward_context(None, vllm_config=vllm_config), + ): # background context + # warm up with background context + model(inputs) + + # capturing/replaying should under context of cudagraph dispatching + with set_forward_context( + None, + vllm_config=vllm_config, + cudagraph_runtime_mode=CUDAGraphMode.PIECEWISE, + batch_descriptor=BatchDescriptor( + num_tokens=2, + ), + ): + model(torch.randn(2).cuda()) + with set_forward_context( + None, + vllm_config=vllm_config, + cudagraph_runtime_mode=CUDAGraphMode.PIECEWISE, + batch_descriptor=BatchDescriptor( + num_tokens=1, + ), + ): + model(torch.randn(1).cuda()) + + input = torch.zeros(2).cuda() + reset_global_counter() + with set_forward_context( + None, + vllm_config=vllm_config, + cudagraph_runtime_mode=CUDAGraphMode.PIECEWISE, + batch_descriptor=BatchDescriptor( + num_tokens=2, + ), + ): + output = model(input) + assert get_global_counter() == 2 + assert torch.allclose(output.cpu(), torch.tensor([19.0, 19.0])) + + +@pytest.mark.parametrize("backend", ["inductor", "eager"]) +@pytest.mark.parametrize("intermediate_unbacked", [True, False]) +@torch.inference_mode() +@create_new_process_for_each_test("spawn") +def test_simple_piecewise_compile(backend, intermediate_unbacked, monkeypatch): + # `intermediate_unbacked` flips a control-flow branch inside + # `SillyModel.forward`, but the AOT-compile cache key only hashes the + # forward function's qualname + line number, so both parametrize variants + # share the same cache slot. Disabling the cache forces each variant to + # compile fresh; otherwise the second-running variant loads the first's + # artifact and segfaults with an illegal memory access. + monkeypatch.setenv("VLLM_DISABLE_COMPILE_CACHE", "1") + _run_simple_model( + splitting_ops=["silly::attention"], + use_inductor_graph_partition=False, + backend=backend, + # 2 * num_layers + 1 + expected_num_piecewise_graphs_seen=5, + # 1 + num_layers + expected_num_piecewise_capturable_graphs_seen=3, + # num_piecewise_capturable_graphs_seen + expected_num_backend_compilations=3, + # num_cudagraph_sizes * num_piecewise_capturable_graphs_seen + expected_num_cudagraph_captured=6, + intermediate_unbacked=intermediate_unbacked, + ) + + +@torch.inference_mode() +def test_simple_inductor_graph_partition(monkeypatch): + if not is_torch_equal_or_newer("2.9.0.dev"): + pytest.skip("inductor graph partition is only available in PyTorch 2.9+") + + # disable compile cache so that we run separately for different splitting_ops + # and get the expected number of cudagraphs captured. + monkeypatch.setenv("VLLM_DISABLE_COMPILE_CACHE", "1") + + _run_simple_model( + splitting_ops=["silly::attention"], + use_inductor_graph_partition=True, + backend="inductor", + # Since not splitting at fx graph level + expected_num_piecewise_graphs_seen=1, + # Since not splitting at fx graph level + expected_num_piecewise_capturable_graphs_seen=1, + # Since not splitting at fx graph level + expected_num_backend_compilations=1, + # Inductor graph partition still captures 6 graph, same as fx graph partition + expected_num_cudagraph_captured=6, + ) diff --git a/asian_ai/performance/vLLM/tests/compile/fullgraph/test_toy_llama.py b/asian_ai/performance/vLLM/tests/compile/fullgraph/test_toy_llama.py new file mode 100644 index 0000000000000000000000000000000000000000..98b67dbc5219ff1caa09f800e8447b1384b714d3 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/fullgraph/test_toy_llama.py @@ -0,0 +1,524 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Test the piecewise compilation with a simple model, comparing the output +with and without the piecewise compilation. + +This is a tractable model, the weights and computation are specially designed +if the config `tractable_init` is set to True. Otherwise, the weights are +initialized randomly with a fixed seed. +""" + +from copy import deepcopy +from dataclasses import dataclass +from typing import Any + +import pytest +import torch +from torch import nn + +from vllm.compilation.decorators import support_torch_compile +from vllm.config import ( + CompilationConfig, + CompilationMode, + CUDAGraphMode, + VllmConfig, + set_current_vllm_config, +) +from vllm.forward_context import BatchDescriptor, set_forward_context +from vllm.utils.torch_utils import is_torch_equal_or_newer + +from ...utils import create_new_process_for_each_test + +# This import automatically registers `torch.ops.silly.attention` +from .. import silly_attention # noqa: F401 + + +@dataclass +class LlamaConfig: + hidden_size: int = 128 + mlp_size: int = 256 + vocab_size: int = 128 + num_layers: int = 2 + init_value: float = 1.0 + tractable_init: bool = False + random_seed: int = 0 + + def compute_hash(self) -> str: + factors: list[Any] = [] + for k, v in self.__dict__.items(): + if k == "random_seed": + continue + factors.append((k, v)) + factors.sort() + import hashlib + + return hashlib.md5(str(factors).encode(), usedforsecurity=False).hexdigest() + + def __post_init__(self): + assert self.mlp_size >= self.hidden_size + + +class LlamaMLP(nn.Module): + def __init__(self, config: LlamaConfig) -> None: + super().__init__() + self.gate_up_projection = nn.Linear( + in_features=config.hidden_size, + out_features=config.mlp_size * 2, + bias=False, + ) + self.down_projection = nn.Linear( + in_features=config.mlp_size, + out_features=config.hidden_size, + bias=False, + ) + + if config.tractable_init: + nn.init.eye_(self.gate_up_projection.weight.data[: config.mlp_size]) + nn.init.eye_(self.gate_up_projection.weight.data[config.mlp_size :]) + nn.init.eye_(self.down_projection.weight.data) + else: + nn.init.xavier_normal_( + self.gate_up_projection.weight.data, + generator=torch.Generator().manual_seed(config.random_seed), + gain=0.001, + ) + nn.init.xavier_normal_( + self.down_projection.weight.data, + generator=torch.Generator().manual_seed(config.random_seed), + gain=0.001, + ) + + def forward(self, x): + # for tractable_init and positive input, this is + # essentially an elementwise-square + x = self.gate_up_projection(x) + x = x[:, : x.size(1) // 2] * torch.nn.functional.relu(x[:, x.size(1) // 2 :]) + x = self.down_projection(x) + return x + + +class LlamaAttention(nn.Module): + def __init__(self, config: LlamaConfig) -> None: + super().__init__() + self.qkv_projection = nn.Linear( + in_features=config.hidden_size, + out_features=config.hidden_size * 3, + bias=False, + ) + + self.output_projection = nn.Linear( + in_features=config.hidden_size, + out_features=config.hidden_size, + bias=False, + ) + + if config.tractable_init: + nn.init.eye_(self.qkv_projection.weight.data[: config.hidden_size]) + nn.init.eye_( + self.qkv_projection.weight.data[ + config.hidden_size : 2 * config.hidden_size + ] + ) + nn.init.eye_(self.qkv_projection.weight.data[2 * config.hidden_size :]) + nn.init.eye_(self.output_projection.weight.data) + else: + nn.init.xavier_normal_( + self.qkv_projection.weight.data, + generator=torch.Generator().manual_seed(config.random_seed), + gain=0.001, + ) + nn.init.xavier_normal_( + self.output_projection.weight.data, + generator=torch.Generator().manual_seed(config.random_seed), + gain=0.001, + ) + + def forward( + self, + positions: torch.Tensor, + hidden_states: torch.Tensor, + ) -> torch.Tensor: + # for tractable_init, this is: + # output = (hidden_states * 3 + positions * 2) + qkv = self.qkv_projection(hidden_states) + hidden_size = qkv.size(-1) // 3 + q, k, v = qkv.split([hidden_size, hidden_size, hidden_size], dim=-1) + + q = q + positions.unsqueeze(1) + k = k + positions.unsqueeze(1) + + attn_output = torch.empty_like(q) + torch.ops.silly.attention(q, k, v, attn_output) + + output = self.output_projection(attn_output) + return output + + +class LlamaDecoderLayer(nn.Module): + def __init__(self, config: LlamaConfig) -> None: + super().__init__() + self.self_attention = LlamaAttention(config) + self.mlp = LlamaMLP(config) + + def forward( + self, + positions: torch.Tensor, + hidden_states: torch.Tensor, + residual: torch.Tensor | None, + ) -> tuple[torch.Tensor, torch.Tensor]: + """ + For tractable computation: + - if residual is None, the outputs are: + - residual = (hidden_states + 1) * 3 + positions * 2 + hidden_states = hidden_states * 4 + positions * 2 + 3 + - hidden_states = (residual + 1) ** 2 + - if residual is not None, the outputs are: + - residual = (hidden_states + residual + 1) * 3 + positions * 2 + hidden_states + residual = (hidden_states + residual) * 4 + positions * 2 + 3 + - hidden_states = (residual + 1) ** 2 + """ # noqa + if residual is None: + residual = hidden_states + hidden_states = hidden_states + 1 + else: + hidden_states = hidden_states + residual + residual = hidden_states + hidden_states = hidden_states + 1 + + hidden_states = self.self_attention( + positions=positions, hidden_states=hidden_states + ) + + hidden_states = hidden_states + residual + residual = hidden_states + hidden_states = hidden_states + 1 + hidden_states = self.mlp(hidden_states) + + return hidden_states, residual + + +@support_torch_compile +class LlamaModel(nn.Module): + def __init__( + self, + *, + vllm_config: VllmConfig, + config: LlamaConfig, + prefix: str = "", + **kwargs, + ) -> None: + super().__init__() + self.embedding_tokens = nn.Embedding( + num_embeddings=config.vocab_size, + embedding_dim=config.hidden_size, + ) + self.layers = nn.ModuleList( + [LlamaDecoderLayer(config) for _ in range(config.num_layers)] + ) + + # this is the initial value of the hidden states + self.embedding_tokens.weight.data.fill_(config.init_value) + + def forward( + self, + input_ids: torch.Tensor | None, + positions: torch.Tensor, + ) -> torch.Tensor: + hidden_states = self.embedding_tokens(input_ids) + residual = None + for layer in self.layers: + hidden_states, residual = layer(positions, hidden_states, residual) + return hidden_states + + +def tractable_computation( + input_ids: torch.Tensor, + positions: torch.Tensor, + config: LlamaConfig, + init_value: float = 1.0, +) -> torch.Tensor: + hidden_states = ( + torch.ones( + input_ids.size(0), + config.hidden_size, + device=input_ids.device, + dtype=input_ids.dtype, + ) + * init_value + ) + + # first layer + residual = hidden_states * 4 + positions.unsqueeze(1) * 2 + 3 + hidden_states = (residual + 1) ** 2 + + # following layers + for _ in range(config.num_layers - 1): + hidden_states = hidden_states + residual + residual = hidden_states * 4 + positions.unsqueeze(1) * 2 + 3 + hidden_states = (residual + 1) ** 2 + + return hidden_states + + +@torch.inference_mode +def run_model(llama_config, compile_config: CompilationConfig) -> torch.Tensor: + # Start with a fresh copy to make sure there's no cache dir sharing + compile_config = deepcopy(compile_config) + cudagraph_runtime_mode = compile_config.cudagraph_mode + + vllm_config = VllmConfig( + compilation_config=compile_config, additional_config=llama_config + ) + with set_current_vllm_config(vllm_config): + model = ( + LlamaModel(config=llama_config, vllm_config=vllm_config, prefix="") + .eval() + .cuda() + ) + + with set_forward_context({}, vllm_config=vllm_config): # background context + B = 16 # max batch size + input_ids = torch.randint(0, llama_config.vocab_size, (B,)).cuda() + positions = torch.arange(B).cuda() + + # warmup for the model with cudagraph_mode NONE + model(input_ids, positions) + + # simulate cudagraphs capturing + with set_forward_context( + {}, + vllm_config=vllm_config, + cudagraph_runtime_mode=cudagraph_runtime_mode, + batch_descriptor=BatchDescriptor( + num_tokens=2, + ), + ): + model(input_ids[:2], positions[:2]) + with set_forward_context( + {}, + vllm_config=vllm_config, + cudagraph_runtime_mode=cudagraph_runtime_mode, + batch_descriptor=BatchDescriptor( + num_tokens=1, + ), + ): + model(input_ids[:1], positions[:1]) + + input_ids[:2].zero_() + # simulate cudagraphs replay + with set_forward_context( + {}, + vllm_config=vllm_config, + cudagraph_runtime_mode=cudagraph_runtime_mode, + batch_descriptor=BatchDescriptor( + num_tokens=2, + ), + ): + output = model(input_ids[:2], positions[:2]) + + output = output.cpu() + + if llama_config.tractable_init: + expected_output = tractable_computation( + input_ids[:2], positions[:2], llama_config + ).cpu() + + assert torch.allclose(output, expected_output) + else: + return output.cpu() + + +@pytest.mark.parametrize( + "backend, use_inductor_graph_partition", + [ + ("eager", False), # No inductor + ("inductor", False), # Inductor, Dynamo partition + ("inductor", True), # Inductor, Inductor partition + ], +) +@create_new_process_for_each_test("spawn") +def test_toy_llama( + backend: str, use_inductor_graph_partition: bool, monkeypatch, tmp_path +): + from vllm.compilation.counter import compilation_counter + + # We disable the vLLM compile cache into a new tmp dir for 1 reason: + # 1. To make sure we can properly track the number of Inductor compilations. + monkeypatch.setenv("VLLM_DISABLE_COMPILE_CACHE", "1") + + if use_inductor_graph_partition and not is_torch_equal_or_newer("2.9.0.dev"): + pytest.skip("Inductor graph partition only supported in torch>=2.9") + + # compare output with and without piecewise compilation + + llama_config = LlamaConfig( + hidden_size=128, mlp_size=256, vocab_size=128, num_layers=12 + ) + + tractable_config = LlamaConfig( + hidden_size=128, mlp_size=256, vocab_size=128, num_layers=2, tractable_init=True + ) + + compile_config_no_compile = CompilationConfig( + mode=CompilationMode.NONE, + cudagraph_mode=CUDAGraphMode.NONE, + backend="eager", + ) + + compile_config_no_split = CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + use_inductor_graph_partition=use_inductor_graph_partition, + cudagraph_mode=CUDAGraphMode.PIECEWISE, + backend=backend, + cudagraph_capture_sizes=[1, 2], + ) + + compile_config_split = deepcopy(compile_config_no_split) + compile_config_split.splitting_ops = ["silly::attention"] + + outputs = [] + with compilation_counter.expect( + num_graphs_seen=0, + num_piecewise_graphs_seen=0, + num_piecewise_capturable_graphs_seen=0, + num_backend_compilations=0, + num_cudagraph_captured=0, + ): + outputs.append(run_model(llama_config, compile_config_no_compile)) + + run_model(tractable_config, compile_config_no_compile) + + if backend == "inductor": + kwargs = {"num_inductor_compiles": 1, "num_eager_compiles": 0} + else: + kwargs = {"num_eager_compiles": 1, "num_inductor_compiles": 0} + + with compilation_counter.expect( + num_graphs_seen=1, # one graph for the model + num_piecewise_graphs_seen=1, + num_piecewise_capturable_graphs_seen=1, + num_backend_compilations=1, # num_piecewise_capturable_graphs_seen + num_cudagraph_captured=2, + **kwargs, + ): + outputs.append(run_model(llama_config, compile_config_no_split)) + + run_model(tractable_config, compile_config_no_split) + + if use_inductor_graph_partition: + num_piecewise_fx = 1 + num_piecewise_capturable_fx = 1 + else: + num_piecewise_fx = 2 * llama_config.num_layers + 1 + num_piecewise_capturable_fx = 1 + llama_config.num_layers + + with compilation_counter.expect( + num_graphs_seen=1, # one graph for the model + num_piecewise_graphs_seen=num_piecewise_fx, + num_piecewise_capturable_graphs_seen=num_piecewise_capturable_fx, + num_backend_compilations=num_piecewise_capturable_fx, + # num_cudagraph_sizes * num_partitions + num_cudagraph_captured=2 * (1 + llama_config.num_layers), + ): + outputs.append(run_model(llama_config, compile_config_split)) + run_model(tractable_config, compile_config_split) + + for i in range(1, len(outputs)): + assert torch.allclose(outputs[0], outputs[i]) + + +@torch.inference_mode +def benchmark(): + from triton.testing import do_bench + + # similar to llama 3.1-8B + llama_config = LlamaConfig( + hidden_size=4096, mlp_size=14336, vocab_size=128 * 1024, num_layers=32 + ) + + # a tiny model to measure the overhead + # of piecewise cudagraph + llama_config = LlamaConfig( + hidden_size=40, mlp_size=80, vocab_size=128, num_layers=2 + ) + + cudagraph_sizes = [1, 2, 4] + [i * 8 for i in range(1, 33)] + + eager_time = {} + full_cudagraph_time = {} + piecewise_cudagraph_time = {} + + pool = torch.cuda.graph_pool_handle() + + for piecewise in [False, True]: + if piecewise: + compilation_config = CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + splitting_ops=["silly::attention"], + cudagraph_capture_sizes=cudagraph_sizes, + ) + else: + compilation_config = CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + cudagraph_capture_sizes=cudagraph_sizes, + ) + + vllm_config = VllmConfig(compilation_config=compilation_config) + with set_current_vllm_config(vllm_config): + model = ( + LlamaModel(config=llama_config, vllm_config=vllm_config, prefix="") + .eval() + .cuda() + .to(torch.bfloat16) + ) + + B = 256 # max batch size + input_ids = torch.randint(0, llama_config.vocab_size, (B,)).cuda() + positions = torch.arange(B).cuda().to(torch.bfloat16) + + graphs = {} + + model(input_ids, positions) + for b in cudagraph_sizes[::-1]: + if not piecewise: + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, pool=pool): + output = model(input_ids[:b], positions[:b]) + graphs[b] = (graph, output) + else: + output = model(input_ids[:b], positions[:b]) + graphs[b] = (model, output) + for b in cudagraph_sizes: + if piecewise: + # noqa is for `Function definition does not bind loop variable` + # it will be problematic if we save the created lambda function + # and use it later, because it will look up the name `b` in the + # enclosing scope, and the value of `b` will always be 256. + # it is fine here, because we only use the lambda function once. + runtime = do_bench( + lambda: graphs[b][0]( # noqa + input_ids[:b], # noqa + positions[:b], # noqa + ) + ) + piecewise_cudagraph_time[b] = runtime + else: + runtime = do_bench(lambda: graphs[b][0].replay()) # noqa + eager_runtime = do_bench(lambda: model(input_ids[:b], positions[:b])) # noqa + full_cudagraph_time[b] = runtime + eager_time[b] = eager_runtime + + # print in tabular format + print("batch size\teager mode\tfull cudagraph\tpiecewise cudagraph") + for b in cudagraph_sizes: + print( + f"{b}\t{eager_time[b]:.3f}\t{full_cudagraph_time[b]:.3f}" + f"\t{piecewise_cudagraph_time[b]:.3f}" + ) + + +if __name__ == "__main__": + # Protect against subprocess reimport when using spawn_new_process_for_each_test + import os + + if os.environ.get("RUNNING_IN_SUBPROCESS") != "1": + benchmark() diff --git a/asian_ai/performance/vLLM/tests/compile/fusions_e2e/__init__.py b/asian_ai/performance/vLLM/tests/compile/fusions_e2e/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/compile/fusions_e2e/common.py b/asian_ai/performance/vLLM/tests/compile/fusions_e2e/common.py new file mode 100644 index 0000000000000000000000000000000000000000..0864d76751c61b190e7c58abb5e05ee7c368c764 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/fusions_e2e/common.py @@ -0,0 +1,108 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import itertools +from collections.abc import Callable, Iterable +from typing import Any, NamedTuple + +import pytest +import regex as re + +from vllm.platforms import current_platform +from vllm.v1.attention.backends.registry import AttentionBackendEnum + + +class Matches(NamedTuple): + # simple pointwise + aiter_rms_quant_fusion: int = 0 + rms_quant_fusion: int = 0 + act_quant_fusion: int = 0 + norm_rope_fusion: int = 0 + attn_quant_fusion: int = 0 + # distributed + ar_rms_fusion: int = 0 + aiter_ar_rms_fusion: int = 0 + sequence_parallel: int = 0 + async_tp: int = 0 + + +class ModelFusionInfo(NamedTuple): + model_name: str + matches: Callable[[int], Matches] + """Given number of hidden layers, produces the matches object""" + model_kwargs: dict[str, Any] = {} + hf_overrides: Callable[[int], dict] = lambda n: {"num_hidden_layers": n} + + +class AttentionBackendCase(NamedTuple): + backend: AttentionBackendEnum + model_kwargs: dict[str, Any] = {} + """Additional args required for attn+quant fusion""" + + +is_blackwell = lambda: current_platform.is_device_capability_family(100) +"""Are we running on Blackwell, a lot of tests depend on it""" + + +def custom_ops_combos(*custom_ops: str) -> Iterable[str]: + """Generate all combinations of custom ops for parametrization.""" + custom_ops_lists = [[f"-{op}", f"+{op}"] for op in custom_ops] + for op_list in itertools.product(*custom_ops_lists): + yield ",".join(op_list) + + +# Quick inline validation +assert list(custom_ops_combos("silu_and_mul")) == ["-silu_and_mul", "+silu_and_mul"] +assert list(custom_ops_combos("quant_fp8", "rms_norm")) == [ + "-quant_fp8,-rms_norm", + "-quant_fp8,+rms_norm", + "+quant_fp8,-rms_norm", + "+quant_fp8,+rms_norm", +] + + +def has_cuda_graph_wrapper_metadata() -> bool: + from importlib import import_module + + try: + module = import_module("torch._inductor.utils") + module.CUDAGraphWrapperMetadata # noqa B018 + except AttributeError: + return False + return True + + +INDUCTOR_GRAPH_PARTITION = [ + pytest.param( + True, + marks=pytest.mark.skipif( + not has_cuda_graph_wrapper_metadata(), + reason="torch version does not support Inductor partition", + ), + id="inductor_partition", + ), + pytest.param(False, id="dynamo_partition"), +] + +FUSION_LOG_PATTERNS: dict[str, re.Pattern] = { + "aiter_rms_quant_fusion": re.compile( + r"RocmAiterRMSNormQuantFusionPass Replaced (\d+) patterns" + ), + "rms_quant_fusion": re.compile(r"rms_quant_fusion.py:\d+] Replaced (\d+) patterns"), + "act_quant_fusion": re.compile(r"act_quant_fusion.py:\d+] Replaced (\d+) patterns"), + "norm_rope_fusion": re.compile( + r"qk_norm_rope_fusion.py:\d+] Fused QK Norm\+RoPE on (\d+) sites" + ), + "attn_quant_fusion": re.compile( + r"attn_quant_fusion.py:\d+] Fused quant onto (\d+) attention nodes" + ), + "ar_rms_fusion": re.compile( + r"allreduce_rms_fusion.py:\d+] Replaced (\d+) patterns" + ), + "aiter_ar_rms_fusion": re.compile( + r"RocmAiterAllReduceFusionPass Replaced (\d+) patterns" + ), + "sequence_parallel": re.compile( + r"sequence_parallelism.py:\d+] Replaced (\d+) patterns" + ), + "async_tp": re.compile(r"collective_fusion.py:\d+] Replaced (\d+) patterns"), +} diff --git a/asian_ai/performance/vLLM/tests/compile/fusions_e2e/conftest.py b/asian_ai/performance/vLLM/tests/compile/fusions_e2e/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..e6746976015b35b45312e8a313701b9567b6e9c8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/fusions_e2e/conftest.py @@ -0,0 +1,309 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import logging +from collections import defaultdict + +import pytest +import regex as re + +from vllm import LLM, SamplingParams +from vllm.config import CompilationConfig, CompilationMode, CUDAGraphMode + +from .common import FUSION_LOG_PATTERNS, AttentionBackendCase, Matches + + +def run_model(compile_config: int | CompilationConfig, model: str, **model_kwargs): + """Run a model with the given compilation config for E2E fusion tests.""" + compilation_config = ( + compile_config + if isinstance(compile_config, CompilationConfig) + else CompilationConfig(mode=compile_config) + ) + + prompts = [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + "The future of AI is", + ] + sampling_params = SamplingParams(temperature=0) + # Allow override from model_kwargs + model_kwargs = {"tensor_parallel_size": 1, **model_kwargs} + model_kwargs = {"disable_custom_all_reduce": True, **model_kwargs} + + # No cudagraphs by default + if compilation_config.cudagraph_mode is None: + compilation_config.cudagraph_mode = CUDAGraphMode.NONE + llm = LLM( + model=model, + compilation_config=compilation_config, + **model_kwargs, + ) + outputs = llm.generate(prompts, sampling_params) + + # Print the outputs. + for output in outputs: + prompt = output.prompt + generated_text = output.outputs[0].text + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") + + # Get the compile ranges endpoints after vllm config post init + # in order to compute compile ranges correctly + compilation_config.compile_ranges_endpoints = ( + llm.llm_engine.vllm_config.compilation_config.compile_ranges_endpoints + ) + + # Fetch match table from each worker via RPC and sum across workers. + worker_tables = llm.llm_engine.engine_core.collective_rpc( + "get_compilation_match_table" + ) + combined: defaultdict[str, int] = defaultdict(int) + for table in worker_tables: + for k, v in table.items(): + combined[k] += v + return dict(combined) + + +@pytest.fixture +def run_e2e_fusion_test(monkeypatch, caplog_mp_spawn): + def run( + model_name: str, + matches: Matches, + model_kwargs: dict, + attn_backend: AttentionBackendCase, + compilation_config: dict, + matches_check: list[str], + use_deepgemm: bool = False, + use_aiter: bool = False, + tp_size: int = 1, + ): + monkeypatch.setenv("VLLM_USE_DEEP_GEMM", "1" if use_deepgemm else "0") + monkeypatch.setenv("VLLM_ROCM_USE_AITER", "1" if use_aiter else "0") + from vllm._aiter_ops import rocm_aiter_ops + + rocm_aiter_ops.refresh_env_variables() + + # Filter here to reduce code duplication + backend_name = attn_backend.backend.name.lower() + requires_mla = "deepseek" in model_name.lower() + is_mla = "mla" in backend_name + # DeepSeek V3.2 uses sparse MLA + requires_sparse = "v3.2" in model_name.lower() + is_sparse = "sparse" in backend_name + + if requires_mla != is_mla or requires_sparse != is_sparse: + pytest.skip( + f"Incompatible model '{model_name}' and " + f"attention backend '{attn_backend.backend.name}'" + ) + + if backend_name == "rocm_attn" and model_name == "openai/gpt-oss-20b": + pytest.skip( + "ROCM_ATTN does not support attention sinks (required by gpt-oss-20b)" + ) + + if attn_backend.backend.name == "FLASHINFER": + from vllm.utils.flashinfer import supports_trtllm_attention + + if not supports_trtllm_attention(): + matches = matches._replace(attn_quant_fusion=0) + + # Disable, compile cache to make sure custom passes run. + # Otherwise, we can't verify fusion happened through the logs. + monkeypatch.setenv("VLLM_DISABLE_COMPILE_CACHE", "1") + + # To capture subprocess logs, we need to know whether spawn or fork is used. + # Force spawn as it is more general. + monkeypatch.setenv("VLLM_WORKER_MULTIPROC_METHOD", "spawn") + + model_kwargs = {**attn_backend.model_kwargs, **model_kwargs} + model_kwargs["attention_config"] = {"backend": attn_backend.backend.name} + model_kwargs["tensor_parallel_size"] = tp_size + + # Cap warmup memory: tests use small max_model_len (1024) but the + # engine default max_num_batched_tokens is 16384. Warming up large + # models (e.g. Llama-4-Scout-FP8) at 16384 tokens may trigger OOM. + model_kwargs.setdefault("max_num_batched_tokens", 8192) + + # Sparse MLA models (DSv3.2) hit an over-strict inductor assertion in + # decompose_auto_functionalized when +rotary_embedding is forced into + # the compile graph. Disable qk_norm+rope fusion (which auto-enables + # +rotary_embedding) for this combo to avoid the known torch bug. + # TODO: remove once upstream torch fix lands. + if requires_sparse: + if "pass_config" in compilation_config: + compilation_config["pass_config"].enable_qk_norm_rope_fusion = False + matches_check = [m for m in matches_check if m != "norm_rope_fusion"] + # DSv3.2 sparse indexer uses persistent_topk with k=config.index_topk + # (2048 for the default config). max_model_len must be >= index_topk + # or the topk kernel raises "k out of range" at runtime. + model_kwargs["max_model_len"] = max( + model_kwargs.get("max_model_len", 0), 2048 + ) + + # Always compile the full graph instead of piecewise + if not compilation_config["use_inductor_graph_partition"]: + compilation_config["splitting_ops"] = [] + + full_compilation_config = CompilationConfig( + cudagraph_mode=CUDAGraphMode.NONE, + mode=CompilationMode.VLLM_COMPILE, + inductor_compile_config={"force_disable_caches": True}, + **compilation_config, + ) + + with caplog_mp_spawn(logging.DEBUG) as log_holder: + match_table = run_model(full_compilation_config, model_name, **model_kwargs) + + num_compile_ranges = len(full_compilation_config.get_compile_ranges()) + assert num_compile_ranges in [1, 2, 3] + + print(f"Compile ranges: {full_compilation_config.get_compile_ranges()}") + print("Fusion results:") + + # Iterate through all so printing happens before asserting + log_matches_dict = {} + for match_name, pattern in FUSION_LOG_PATTERNS.items(): + log_matches_dict[match_name] = list(pattern.findall(log_holder.text)) + print(f"- {match_name}={','.join(log_matches_dict[match_name])}") + + # Now check the matches + for match_name in matches_check: + log_matches = list(int(ms) for ms in log_matches_dict[match_name]) + + # AR+RMS skips the largest range; SP skips the smallest. + # When both are enabled, AR+RMS activation count is + # model-dependent (hidden_size affects threshold), so derive + # from log data. + if ( + match_name == "ar_rms_fusion" + and "sequence_parallel" in matches_check + and num_compile_ranges >= 2 + ): + assert ( + len(log_matches) >= tp_size and len(log_matches) % tp_size == 0 + ), ( + f"Expected multiple of {tp_size} ar_rms log entries, " + f"found {len(log_matches)}" + ) + num_ranges_activated = len(log_matches) // tp_size + elif ( + match_name in ("ar_rms_fusion", "sequence_parallel") + and num_compile_ranges >= 2 + ): + num_ranges_activated = num_compile_ranges - 1 + else: + num_ranges_activated = num_compile_ranges + + # TODO: Remove log counting in unit tests + # once all matchers implement VllmFusionPatternMatcherPass + n_expected = tp_size * num_ranges_activated + if match_name not in ("attn_quant_fusion", "act_quant_fusion"): + assert len(log_matches) == n_expected, ( + f"Could not find {n_expected} {match_name} " + f"(found {len(log_matches)}) in:\n {log_holder.text}" + ) + + expected_matches = getattr(matches, match_name) + + if match_name == "rms_quant_fusion" and "ar_rms_fusion" in matches_check: + # AR+rms+quant takes precedence over rms+quant if activated. + # That means we get full matching where ar+rms+quant was not + # activated, and less where it was (only the smallest range). + assert sum(m == expected_matches for m in log_matches) == tp_size * ( + num_ranges_activated - 1 + ), "Expecting full rms+quant fusion where ar+rms+quant not activated" + + assert all( + expected_matches - matches.ar_rms_fusion <= m <= expected_matches + for m in log_matches + ), ( + f"Expecting at least {expected_matches - matches.ar_rms_fusion} " + f"where ar+rms+quant was activated" + ) + elif ( + match_name == "async_tp" + and "sequence_parallel" in matches_check + and num_compile_ranges >= 2 + ): + # AsyncTP only finds patterns on ranges where SP ran. + n_sp_ranges = num_compile_ranges - 1 + assert ( + sum(m == expected_matches for m in log_matches) + == tp_size * n_sp_ranges + ), ( + f"Expecting {expected_matches} async_tp on " + f"{tp_size * n_sp_ranges} SP-range entries, " + f"found: {log_matches}" + ) + assert sum(m == 0 for m in log_matches) == tp_size, ( + f"Expecting 0 async_tp on {tp_size} small-range entries " + f"(no SP), found: {log_matches}" + ) + elif ( + match_name == "ar_rms_fusion" + and "sequence_parallel" in matches_check + and num_compile_ranges >= 2 + ): + # SP consumes allreduce patterns first, so AR+RMS finds + # full matches only on the smallest range (no SP). + assert sum(m == expected_matches for m in log_matches) == tp_size, ( + f"Expecting {expected_matches} ar_rms on " + f"{tp_size} small-range entries, found: {log_matches}" + ) + assert sum(m == 0 for m in log_matches) == tp_size * ( + num_ranges_activated - 1 + ), ( + f"Expecting 0 ar_rms on " + f"{tp_size * (num_ranges_activated - 1)} large-range " + f"entries (SP took precedence), found: {log_matches}" + ) + + elif match_name == "act_quant_fusion": + actual_match = match_table.get("activation_quant_fusion_pass", 0) + assert actual_match == expected_matches * n_expected, ( + f"Could not find {expected_matches * n_expected} " + f"{match_name} (found {actual_match})." + ) + elif match_name == "attn_quant_fusion": + actual_match = match_table.get( + "attn_quant_fusion", 0 + ) + match_table.get("mla_attn_quant_fusion", 0) + assert actual_match == expected_matches * n_expected, ( + f"Could not find {expected_matches * n_expected} " + f"{match_name} (found {actual_match})." + ) + else: + expected_matches_list = [expected_matches] * n_expected + assert sorted(log_matches) == expected_matches_list, ( + f"{match_name} expected: {expected_matches_list}, " + f"found: {sorted(log_matches)}" + ) + + if match_name == "ar_rms_fusion" and num_compile_ranges >= 2: + log_matches = re.findall( + r"pass_manager.py:\d+] Skipping " + r".*AllReduceFusionPass.* with compile range", + log_holder.text, + ) + + n_expected = tp_size * (num_compile_ranges - num_ranges_activated) + assert len(log_matches) == n_expected, ( + f'Could not find {n_expected} "Skipping AllReduceFusionPass" ' + f"(found {len(log_matches)}) in:\n {log_holder.text}" + ) + + if match_name == "sequence_parallel" and num_compile_ranges >= 2: + log_matches = re.findall( + r"pass_manager.py:\d+] Skipping " + r".*SequenceParallelismPass.* with compile range", + log_holder.text, + ) + + n_expected = tp_size * (num_compile_ranges - num_ranges_activated) + assert len(log_matches) == n_expected, ( + f'Could not find {n_expected} "Skipping SequenceParallelismPass" ' + f"(found {len(log_matches)}) in:\n {log_holder.text}" + ) + + return run diff --git a/asian_ai/performance/vLLM/tests/compile/fusions_e2e/models.py b/asian_ai/performance/vLLM/tests/compile/fusions_e2e/models.py new file mode 100644 index 0000000000000000000000000000000000000000..4169e5dba03a154a0594509bfda3279c70502ec5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/fusions_e2e/models.py @@ -0,0 +1,225 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest + +from vllm._aiter_ops import is_aiter_found_and_supported +from vllm.platforms import current_platform +from vllm.utils.flashinfer import has_flashinfer +from vllm.v1.attention.backends.registry import AttentionBackendEnum + +from .common import AttentionBackendCase, Matches, ModelFusionInfo, is_blackwell + +# Attn backends +FLASHINFER_ATTN = pytest.param( + AttentionBackendCase( + backend=AttentionBackendEnum.FLASHINFER, + model_kwargs=dict(kv_cache_dtype="fp8"), + ), + id="FLASHINFER", + marks=pytest.mark.skipif( + not is_blackwell() or not has_flashinfer(), + reason="FI backend requires Blackwell and FlashInfer", + ), +) + +TRITON_ATTN = pytest.param( + AttentionBackendCase(backend=AttentionBackendEnum.TRITON_ATTN), id="TRITON_ATTN" +) + +ROCM_ATTN = pytest.param( + AttentionBackendCase(backend=AttentionBackendEnum.ROCM_ATTN), + id="ROCM_ATTN", + marks=pytest.mark.skipif( + not current_platform.is_rocm(), + reason="ROCm attention only for AMD", + ), +) + +ROCM_AITER_UNIFIED_ATTN = pytest.param( + AttentionBackendCase(backend=AttentionBackendEnum.ROCM_AITER_UNIFIED_ATTN), + id="ROCM_AITER_UNIFIED_ATTN", + marks=pytest.mark.skipif( + not is_aiter_found_and_supported(), + reason="ROCM_AITER_UNIFIED_ATTN only for AMD when AITER is installed", + ), +) + +FLASHINFER_MLA_ATTN = pytest.param( + AttentionBackendCase(backend=AttentionBackendEnum.FLASHINFER_MLA), + id="FLASHINFER_MLA", + marks=pytest.mark.skipif( + not is_blackwell() or not has_flashinfer(), + reason="FI backend requires Blackwell and FlashInfer", + ), +) + +TRITON_MLA_ATTN = pytest.param( + AttentionBackendCase(backend=AttentionBackendEnum.TRITON_MLA), + id="TRITON_MLA", +) + +FLASHMLA_SPARSE_ATTN = pytest.param( + AttentionBackendCase( + backend=AttentionBackendEnum.FLASHMLA_SPARSE, + model_kwargs=dict(kv_cache_dtype="fp8_ds_mla"), + ), + id="FLASHMLA_SPARSE", + marks=pytest.mark.skipif( + not is_blackwell(), + reason="FlashMLA Sparse requires Blackwell", + ), +) + +# Models +llama3_8b = ModelFusionInfo( + model_name="meta-llama/Llama-3.1-8B-Instruct", + matches=lambda n_layers: Matches( + ar_rms_fusion=n_layers * 2 + 1, + aiter_ar_rms_fusion=n_layers * 2, + sequence_parallel=n_layers * 2 + 1, + async_tp=n_layers * 4, + ), +) + +llama3_8b_fp8 = ModelFusionInfo( + model_name="RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8", + matches=lambda n_layers: Matches( + rms_quant_fusion=n_layers * 2, + act_quant_fusion=n_layers, + attn_quant_fusion=n_layers, + ar_rms_fusion=n_layers * 2 + 1, + sequence_parallel=n_layers * 2 + 1, + async_tp=n_layers * 4, + ), +) + +llama3_8b_fp4 = ModelFusionInfo( + model_name="nvidia/Llama-3.1-8B-Instruct-FP4", + matches=lambda n_layers: Matches( + act_quant_fusion=n_layers, + attn_quant_fusion=n_layers, + ar_rms_fusion=n_layers * 2 + 1, + sequence_parallel=n_layers * 2 + 1, + async_tp=n_layers * 4, + ), +) + +# MoEs cannot do act+quant fusion because those ops are hidden from torch.compile. +# MoEs also only expose 1 rms+quant fusion because the quant for up_proj is hidden. +# TODO(luka): https://github.com/vllm-project/vllm/issues/31985 +# Also, for MoEs, gemm+collective fusion only happens for dense GEMMs (o_proj/qkv proj) + +llama4_scout_fp8 = ModelFusionInfo( + model_name="nvidia/Llama-4-Scout-17B-16E-Instruct-FP8", + hf_overrides=lambda n_layers: {"text_config": {"num_hidden_layers": n_layers}}, + matches=lambda n_layers: Matches( + rms_quant_fusion=n_layers, + attn_quant_fusion=n_layers, + ar_rms_fusion=n_layers * 2, + sequence_parallel=n_layers * 2, + async_tp=n_layers * 2 - 1, + ), +) + +llama4_scout_fp4 = ModelFusionInfo( + model_name="nvidia/Llama-4-Scout-17B-16E-Instruct-NVFP4", + hf_overrides=lambda n_layers: {"text_config": {"num_hidden_layers": n_layers}}, + matches=lambda n_layers: Matches( + attn_quant_fusion=n_layers, + ar_rms_fusion=n_layers * 2, + sequence_parallel=n_layers * 2, + async_tp=n_layers * 2 - 1, + ), +) + +qwen3_a3b = ModelFusionInfo( + model_name="Qwen/Qwen3-30B-A3B", + matches=lambda n_layers: Matches( + norm_rope_fusion=n_layers, + ar_rms_fusion=n_layers * 2 + 1, + aiter_ar_rms_fusion=n_layers * 2, + sequence_parallel=n_layers * 2 + 1, + async_tp=n_layers * 2, + ), +) + +qwen3_a3b_fp8 = ModelFusionInfo( + model_name="Qwen/Qwen3-30B-A3B-FP8", + matches=lambda n_layers: Matches( + rms_quant_fusion=n_layers, + norm_rope_fusion=n_layers, + attn_quant_fusion=0, # attn + group quant not supported + ar_rms_fusion=n_layers * 2 + 1, + sequence_parallel=n_layers * 2 + 1, + async_tp=n_layers * 2, + ), +) + +deepseek_coder_v2_lite_fp8 = ModelFusionInfo( + model_name="RedHatAI/DeepSeek-Coder-V2-Lite-Instruct-FP8", + matches=lambda n_layers: Matches( + # first_k_dense_replace=1; MoE hides most rms+quant sites + rms_quant_fusion=1, + act_quant_fusion=min(1, n_layers), # dense layers only + # MLA attn + static FP8 quant + attn_quant_fusion=n_layers, + ar_rms_fusion=n_layers * 2 + 1, + ), +) + +deepseek_v3_fp8 = ModelFusionInfo( + model_name="deepseek-ai/DeepSeek-V3", + matches=lambda n_layers: Matches( + # 3 per dense layer (first 3): + # - input_rms + qkv_proj + # - q_a_layernorm + q_b_proj (inside MLA wrapper) + # - post_attn_layernorm + MLP + # 2 per MoE layer (remaining) due to MoE wrapping + rms_quant_fusion=n_layers * 2 + min(3, n_layers), # add for 3 dense layers + # silu+block quant + act_quant_fusion=min(3, n_layers), # dense layers only + # MLA attn + per-group FP8 quant + attn_quant_fusion=n_layers, + ar_rms_fusion=n_layers * 2 + 1, + # TODO + # sequence_parallel= n_layers * 2 + 1, + # async_tp=n_layers * 2, + ), +) + +deepseek_r1_fp4 = ModelFusionInfo( + model_name="nvidia/DeepSeek-R1-0528-NVFP4-v2", + matches=lambda n_layers: Matches( + rms_quant_fusion=0, + act_quant_fusion=min(3, n_layers), + attn_quant_fusion=n_layers, + ar_rms_fusion=n_layers * 2 + 1, + ), +) + +deepseek_v32_fp4 = ModelFusionInfo( + model_name="nvidia/DeepSeek-V3.2-NVFP4", + matches=lambda n_layers: Matches( + rms_quant_fusion=0, + # silu+quant on dense layers only; MoE hides the act+quant site + act_quant_fusion=min(3, n_layers), + # MLA attn + NVFP4 output quant fuses on sparse MLA output path + attn_quant_fusion=n_layers, + ar_rms_fusion=n_layers * 2 + 1, + ), +) + +gpt_oss_20b = ModelFusionInfo( + model_name="openai/gpt-oss-20b", + matches=lambda n_layers: Matches( + ar_rms_fusion=n_layers * 2 + 1, + aiter_ar_rms_fusion=n_layers + 1, + sequence_parallel=n_layers * 2 + 1, + async_tp=n_layers * 2, + ), + model_kwargs=( + {"quantization_config": {"moe": {"activation": "mxfp8"}}} + if is_blackwell() + else {} + ), +) diff --git a/asian_ai/performance/vLLM/tests/compile/fusions_e2e/test_tp1_quant.py b/asian_ai/performance/vLLM/tests/compile/fusions_e2e/test_tp1_quant.py new file mode 100644 index 0000000000000000000000000000000000000000..17d1c024b39b8dea89d03f652127a52f8a546303 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/fusions_e2e/test_tp1_quant.py @@ -0,0 +1,201 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from collections.abc import Callable + +import pytest + +from vllm.config import PassConfig +from vllm.platforms import current_platform +from vllm.utils.flashinfer import is_flashinfer_fp8_blockscale_gemm_supported + +from .common import ( + INDUCTOR_GRAPH_PARTITION, + AttentionBackendCase, + Matches, + custom_ops_combos, + is_blackwell, +) +from .models import ( + FLASHINFER_ATTN, + FLASHINFER_MLA_ATTN, + FLASHMLA_SPARSE_ATTN, + ROCM_AITER_UNIFIED_ATTN, + ROCM_ATTN, + TRITON_ATTN, + TRITON_MLA_ATTN, + deepseek_coder_v2_lite_fp8, + deepseek_r1_fp4, + deepseek_v3_fp8, + deepseek_v32_fp4, + llama3_8b_fp4, + llama3_8b_fp8, + llama4_scout_fp4, + llama4_scout_fp8, + qwen3_a3b_fp8, +) + + +@pytest.mark.parametrize( + "model_name, matches_fn, model_kwargs, hf_overrides, use_deepgemm", + [ + (*llama3_8b_fp8, False), + (*qwen3_a3b_fp8, False), + (*qwen3_a3b_fp8, True), + (*deepseek_coder_v2_lite_fp8, False), + (*deepseek_v3_fp8, False), + (*deepseek_v3_fp8, True), + pytest.param( + *llama4_scout_fp8, + False, + marks=pytest.mark.skipif( + not current_platform.is_cuda(), + reason="Llama4 Scout FP8 only supported on CUDA", + ), + ), + ], +) +@pytest.mark.parametrize( + "attn_backend", + [ + TRITON_ATTN, + FLASHINFER_ATTN, + ROCM_ATTN, + ROCM_AITER_UNIFIED_ATTN, + FLASHINFER_MLA_ATTN, + TRITON_MLA_ATTN, + ], +) +@pytest.mark.parametrize("n_layers", [6]) +@pytest.mark.parametrize("custom_ops", custom_ops_combos("quant_fp8", "rms_norm")) +@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) +def test_tp1_fp8_fusions( + model_name: str, + matches_fn: Callable[[int], Matches], + model_kwargs: dict, + hf_overrides: Callable[[int], dict], + attn_backend: AttentionBackendCase, + n_layers: int, + custom_ops: str, + inductor_graph_partition: bool, + use_deepgemm: bool, + run_e2e_fusion_test, + monkeypatch, +): + if use_deepgemm and not current_platform.is_cuda(): + pytest.skip("DeepGemm only supported on CUDA") + + if use_deepgemm and is_flashinfer_fp8_blockscale_gemm_supported(): + # Flashinfer block FP8 GEMM has internal quantization, so it can't + # be fused with other ops. + pytest.skip("FlashInfer block FP8 GEMM not supported") + if use_deepgemm and is_blackwell(): + # TODO(luka) DeepGEMM uses different quants, matching not supported + # - on Blackwell, uses a special quant fp8, currently not supported + pytest.skip("DeepGEMM & quant matching not currently supported") + + matches = matches_fn(n_layers) + + block_fp8 = "qwen" in model_name.lower() or "deepseek" in model_name.lower() + if block_fp8 and "-quant_fp8" in custom_ops: + # This is why config forces +quant_fp8 by default + pytest.skip("native QuantFP8 matching not supported for group quant") + + # Reduce size of model and skip weight loading time + model_kwargs["hf_overrides"] = hf_overrides(n_layers) + model_kwargs["load_format"] = "dummy" + model_kwargs["max_model_len"] = 1024 + model_kwargs["kernel_config"] = {"enable_flashinfer_autotune": False} + + compilation_config = dict( + use_inductor_graph_partition=inductor_graph_partition, + custom_ops=custom_ops.split(","), + pass_config=PassConfig( + fuse_norm_quant=True, + fuse_act_quant=True, + fuse_attn_quant=True, + enable_qk_norm_rope_fusion=True, + ), + ) + + use_aiter = current_platform.is_rocm() and ("qwen" in model_name.lower()) + + matches_check = [ + "rms_quant_fusion", + "act_quant_fusion", + "norm_rope_fusion", + "attn_quant_fusion", + ] + + if use_aiter: + matches_check[0] = "aiter_rms_quant_fusion" + + matches = matches._replace(aiter_rms_quant_fusion=matches.rms_quant_fusion) + # TODO: enable the `norm_rope_fusion` test, + # On ROCm norm_rope_fusion is only supported without + # enabling AITER. + matches_check.remove("norm_rope_fusion") + + run_e2e_fusion_test( + model_name, + matches, + model_kwargs, + attn_backend, + compilation_config, + matches_check, + use_deepgemm=use_deepgemm, + use_aiter=use_aiter, + ) + + +@pytest.mark.parametrize( + "model_name, matches_fn, model_kwargs, hf_overrides", + [llama3_8b_fp4, llama4_scout_fp4, deepseek_r1_fp4, deepseek_v32_fp4], +) +@pytest.mark.parametrize( + "attn_backend", + [FLASHINFER_ATTN, FLASHINFER_MLA_ATTN, FLASHMLA_SPARSE_ATTN], +) +@pytest.mark.parametrize("n_layers", [6]) +@pytest.mark.parametrize("custom_ops", custom_ops_combos("rms_norm")) +@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) +@pytest.mark.skipif(not is_blackwell(), reason="Blackwell required for fp4") +def test_tp1_fp4_fusions( + model_name: str, + matches_fn: Callable[[int], Matches], + model_kwargs: dict, + hf_overrides: Callable[[int], dict], + attn_backend: AttentionBackendCase, + n_layers: int, + custom_ops: str, + inductor_graph_partition: bool, + run_e2e_fusion_test, +): + matches = matches_fn(n_layers) + + # Reduce size of model and skip weight loading time + model_kwargs["hf_overrides"] = hf_overrides(n_layers) + model_kwargs["load_format"] = "dummy" + model_kwargs["max_model_len"] = 1024 + model_kwargs["kernel_config"] = {"enable_flashinfer_autotune": False} + + compilation_config = dict( + use_inductor_graph_partition=inductor_graph_partition, + custom_ops=custom_ops.split(","), + pass_config=PassConfig( + fuse_norm_quant=True, + fuse_act_quant=True, + fuse_attn_quant=True, + enable_qk_norm_rope_fusion=True, + ), + ) + + matches_check = ["act_quant_fusion", "attn_quant_fusion", "norm_rope_fusion"] + + run_e2e_fusion_test( + model_name, + matches, + model_kwargs, + attn_backend, + compilation_config, + matches_check, + ) diff --git a/asian_ai/performance/vLLM/tests/compile/fusions_e2e/test_tp2_ar_rms.py b/asian_ai/performance/vLLM/tests/compile/fusions_e2e/test_tp2_ar_rms.py new file mode 100644 index 0000000000000000000000000000000000000000..a06377c4e642d0dd78d40ac266de4e304172f474 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/fusions_e2e/test_tp2_ar_rms.py @@ -0,0 +1,247 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from collections.abc import Callable + +import pytest + +from vllm.config import PassConfig +from vllm.platforms import current_platform + +from ...utils import multi_gpu_test +from .common import ( + INDUCTOR_GRAPH_PARTITION, + AttentionBackendCase, + Matches, + custom_ops_combos, + is_blackwell, +) +from .models import ( + FLASHINFER_ATTN, + FLASHINFER_MLA_ATTN, + FLASHMLA_SPARSE_ATTN, + ROCM_AITER_UNIFIED_ATTN, + ROCM_ATTN, + TRITON_ATTN, + deepseek_coder_v2_lite_fp8, + deepseek_r1_fp4, + deepseek_v3_fp8, + deepseek_v32_fp4, + gpt_oss_20b, + llama3_8b, + llama3_8b_fp4, + llama3_8b_fp8, + llama4_scout_fp4, + llama4_scout_fp8, + qwen3_a3b, + qwen3_a3b_fp8, +) + +pytestmark = pytest.mark.skipif( + not current_platform.is_cuda_alike(), reason="Only test CUDA/ROCm" +) + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize( + "model_name, matches_fn, model_kwargs, hf_overrides", + # qwen3 & dsv3 should still fuse AR+rms even though group quant is not yet supported + [ + llama3_8b_fp8, + llama4_scout_fp8, + qwen3_a3b_fp8, + deepseek_coder_v2_lite_fp8, + deepseek_v3_fp8, + ], +) +@pytest.mark.parametrize( + "attn_backend", [TRITON_ATTN, FLASHINFER_ATTN, FLASHINFER_MLA_ATTN] +) +@pytest.mark.parametrize("n_layers", [4]) +@pytest.mark.parametrize("custom_ops", custom_ops_combos("quant_fp8", "rms_norm")) +@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) +@pytest.mark.skipif(not current_platform.is_cuda(), reason="Only test CUDA") +def test_tp2_ar_rms_fp8_fusions( + model_name: str, + matches_fn: Callable[[int], Matches], + model_kwargs: dict, + hf_overrides: Callable[[int], dict], + attn_backend: AttentionBackendCase, + n_layers: int, + custom_ops: str, + inductor_graph_partition: bool, + run_e2e_fusion_test, + monkeypatch, +): + matches = matches_fn(n_layers) + + block_fp8 = "qwen" in model_name.lower() or "deepseek" in model_name.lower() + if block_fp8 and "-quant_fp8" in custom_ops: + # This is why config forces +quant_fp8 by default + pytest.skip("native QuantFP8 matching not supported for group quant") + + # Reduce size of model and skip weight loading time + model_kwargs["hf_overrides"] = hf_overrides(n_layers) + model_kwargs["load_format"] = "dummy" + model_kwargs["max_model_len"] = 1024 + model_kwargs["kernel_config"] = {"enable_flashinfer_autotune": False} + model_kwargs["disable_custom_all_reduce"] = False + + compilation_config = dict( + use_inductor_graph_partition=inductor_graph_partition, + custom_ops=custom_ops.split(","), + pass_config=PassConfig( + fuse_norm_quant=True, + fuse_act_quant=True, + fuse_attn_quant=True, + enable_qk_norm_rope_fusion=True, + fuse_allreduce_rms=True, + ), + ) + + matches_check = [ + "rms_quant_fusion", + "act_quant_fusion", + "norm_rope_fusion", + "attn_quant_fusion", + "ar_rms_fusion", + ] + + run_e2e_fusion_test( + model_name, + matches, + model_kwargs, + attn_backend, + compilation_config, + matches_check, + tp_size=2, + ) + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize( + "model_name, matches_fn, model_kwargs, hf_overrides", + [llama3_8b_fp4, llama4_scout_fp4, deepseek_r1_fp4, deepseek_v32_fp4], +) +@pytest.mark.parametrize( + "attn_backend", + [FLASHINFER_ATTN, FLASHINFER_MLA_ATTN, FLASHMLA_SPARSE_ATTN], +) +@pytest.mark.parametrize("n_layers", [4]) +@pytest.mark.parametrize("custom_ops", custom_ops_combos("rms_norm")) +@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) +@pytest.mark.skipif(not is_blackwell(), reason="Blackwell required for fp4") +@pytest.mark.skipif(not current_platform.is_cuda(), reason="Only test CUDA") +def test_tp2_ar_rms_fp4_fusions( + model_name: str, + matches_fn: Callable[[int], Matches], + model_kwargs: dict, + hf_overrides: Callable[[int], dict], + attn_backend: AttentionBackendCase, + n_layers: int, + custom_ops: str, + inductor_graph_partition: bool, + run_e2e_fusion_test, + monkeypatch, +): + matches = matches_fn(n_layers) + + # Reduce size of model and skip weight loading time + model_kwargs["hf_overrides"] = hf_overrides(n_layers) + model_kwargs["load_format"] = "dummy" + model_kwargs["max_model_len"] = 1024 + model_kwargs["kernel_config"] = {"enable_flashinfer_autotune": False} + model_kwargs["disable_custom_all_reduce"] = False + + compilation_config = dict( + use_inductor_graph_partition=inductor_graph_partition, + custom_ops=custom_ops.split(","), + pass_config=PassConfig( + fuse_act_quant=True, + fuse_attn_quant=True, + fuse_allreduce_rms=True, + ), + ) + + matches_check = [ + "act_quant_fusion", + "attn_quant_fusion", + "ar_rms_fusion", + ] + + run_e2e_fusion_test( + model_name, + matches, + model_kwargs, + attn_backend, + compilation_config, + matches_check, + tp_size=2, + ) + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize( + "model_name, matches_fn, model_kwargs, hf_overrides", + [llama3_8b, qwen3_a3b, gpt_oss_20b], +) +@pytest.mark.parametrize( + "attn_backend", + [ + TRITON_ATTN, + FLASHINFER_ATTN, + ROCM_ATTN, + ROCM_AITER_UNIFIED_ATTN, + ], +) +@pytest.mark.parametrize("n_layers", [4]) +@pytest.mark.parametrize("custom_ops", tuple(custom_ops_combos("rms_norm"))) +@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) +@pytest.mark.skipif(not current_platform.is_cuda_alike(), reason="Only test CUDA/ROCm") +def test_tp2_ar_rms_fusions( + model_name: str, + matches_fn: Callable[[int], Matches], + model_kwargs: dict, + hf_overrides: Callable[[int], dict], + attn_backend: AttentionBackendCase, + n_layers: int, + custom_ops: str, + inductor_graph_partition: bool, + run_e2e_fusion_test, +): + matches = matches_fn(n_layers) + + # Reduce size of model and skip weight loading time + model_kwargs["hf_overrides"] = hf_overrides(n_layers) + model_kwargs["load_format"] = "dummy" + model_kwargs["max_model_len"] = 1024 + model_kwargs["kernel_config"] = {"enable_flashinfer_autotune": False} + model_kwargs["disable_custom_all_reduce"] = False + + compilation_config = dict( + use_inductor_graph_partition=inductor_graph_partition, + custom_ops=custom_ops.split(","), + pass_config=PassConfig( + enable_qk_norm_rope_fusion=True, + fuse_allreduce_rms=True, + ), + ) + + matches_check = [ + "norm_rope_fusion", + ] + + if current_platform.is_rocm(): + matches_check.append("aiter_ar_rms_fusion") + else: + matches_check.append("ar_rms_fusion") + + run_e2e_fusion_test( + model_name, + matches, + model_kwargs, + attn_backend, + compilation_config, + matches_check, + tp_size=2, + use_aiter=current_platform.is_rocm(), + ) diff --git a/asian_ai/performance/vLLM/tests/compile/fusions_e2e/test_tp2_async_tp.py b/asian_ai/performance/vLLM/tests/compile/fusions_e2e/test_tp2_async_tp.py new file mode 100644 index 0000000000000000000000000000000000000000..860e8df8d7acc435a81b8979711005d68ca258de --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/fusions_e2e/test_tp2_async_tp.py @@ -0,0 +1,335 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from collections.abc import Callable + +import pytest + +from vllm.config import PassConfig +from vllm.platforms import current_platform + +from ...utils import multi_gpu_test +from .common import ( + INDUCTOR_GRAPH_PARTITION, + AttentionBackendCase, + Matches, + custom_ops_combos, + is_blackwell, +) +from .models import ( + FLASHINFER_ATTN, + TRITON_ATTN, + llama3_8b, + llama3_8b_fp4, + llama3_8b_fp8, + llama4_scout_fp8, + qwen3_a3b, +) + +pytestmark = pytest.mark.skipif(not current_platform.is_cuda(), reason="Only test CUDA") + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize( + "model_name, matches_fn, model_kwargs, hf_overrides", + [llama3_8b_fp8, llama4_scout_fp8], +) +@pytest.mark.parametrize("attn_backend", [TRITON_ATTN, FLASHINFER_ATTN]) +@pytest.mark.parametrize("n_layers", [4]) +@pytest.mark.parametrize("custom_ops", custom_ops_combos("quant_fp8", "rms_norm")) +@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) +def test_tp2_async_tp_fp8_fusions( + model_name: str, + matches_fn: Callable[[int], Matches], + model_kwargs: dict, + hf_overrides: Callable[[int], dict], + attn_backend: AttentionBackendCase, + n_layers: int, + custom_ops: str, + inductor_graph_partition: bool, + run_e2e_fusion_test, +): + matches = matches_fn(n_layers) + + # Reduce size of model and skip weight loading time + model_kwargs["hf_overrides"] = hf_overrides(n_layers) + model_kwargs["load_format"] = "dummy" + model_kwargs["max_model_len"] = 1024 + model_kwargs["kernel_config"] = {"enable_flashinfer_autotune": False} + + compilation_config = dict( + use_inductor_graph_partition=inductor_graph_partition, + custom_ops=custom_ops.split(","), + pass_config=PassConfig( + fuse_norm_quant=True, + fuse_act_quant=True, + fuse_attn_quant=True, + enable_qk_norm_rope_fusion=True, + enable_sp=True, + fuse_gemm_comms=True, + fuse_allreduce_rms=False, + # Override threshold for testing (models have small hidden_size) + sp_min_token_num=512, + ), + ) + + matches_check = [ + "rms_quant_fusion", + "act_quant_fusion", + "norm_rope_fusion", + "attn_quant_fusion", + "sequence_parallel", + "async_tp", + ] + + run_e2e_fusion_test( + model_name, + matches, + model_kwargs, + attn_backend, + compilation_config, + matches_check, + tp_size=2, + ) + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize( + "model_name, matches_fn, model_kwargs, hf_overrides", + [llama3_8b_fp4], +) +@pytest.mark.parametrize("attn_backend", [FLASHINFER_ATTN]) +@pytest.mark.parametrize("n_layers", [4]) +@pytest.mark.parametrize("custom_ops", custom_ops_combos("rms_norm")) +@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) +@pytest.mark.skipif(not is_blackwell(), reason="Blackwell required for fp4") +@pytest.mark.skipif(not current_platform.is_cuda(), reason="Only test CUDA") +def test_tp2_async_tp_nvfp4_fusions( + model_name: str, + matches_fn: Callable[[int], Matches], + model_kwargs: dict, + hf_overrides: Callable[[int], dict], + attn_backend: AttentionBackendCase, + n_layers: int, + custom_ops: str, + inductor_graph_partition: bool, + run_e2e_fusion_test, +): + # NVFP4 currently wires the all-gather + GEMM path only. + matches = matches_fn(n_layers)._replace(async_tp=n_layers * 2) + + # Reduce size of model and skip weight loading time + model_kwargs["hf_overrides"] = hf_overrides(n_layers) + model_kwargs["load_format"] = "dummy" + model_kwargs["max_model_len"] = 1024 + model_kwargs["kernel_config"] = {"enable_flashinfer_autotune": False} + + compilation_config = dict( + use_inductor_graph_partition=inductor_graph_partition, + custom_ops=custom_ops.split(","), + pass_config=PassConfig( + fuse_act_quant=True, + fuse_attn_quant=True, + enable_sp=True, + fuse_gemm_comms=True, + fuse_allreduce_rms=False, + # Override threshold for testing (models have small hidden_size) + sp_min_token_num=512, + ), + ) + + matches_check = [ + "act_quant_fusion", + "attn_quant_fusion", + "sequence_parallel", + "async_tp", + ] + + run_e2e_fusion_test( + model_name, + matches, + model_kwargs, + attn_backend, + compilation_config, + matches_check, + tp_size=2, + ) + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize( + "model_name, matches_fn, model_kwargs, hf_overrides", + [llama3_8b, qwen3_a3b], +) +@pytest.mark.parametrize("attn_backend", [TRITON_ATTN]) +@pytest.mark.parametrize("n_layers", [4]) +@pytest.mark.parametrize("custom_ops", custom_ops_combos("rms_norm")) +@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) +def test_tp2_async_tp_fusions( + model_name: str, + matches_fn: Callable[[int], Matches], + model_kwargs: dict, + hf_overrides: Callable[[int], dict], + attn_backend: AttentionBackendCase, + n_layers: int, + custom_ops: str, + inductor_graph_partition: bool, + run_e2e_fusion_test, +): + matches = matches_fn(n_layers) + + # Reduce size of model and skip weight loading time + model_kwargs["hf_overrides"] = hf_overrides(n_layers) + model_kwargs["load_format"] = "dummy" + model_kwargs["max_model_len"] = 1024 + model_kwargs["kernel_config"] = {"enable_flashinfer_autotune": False} + + compilation_config = dict( + use_inductor_graph_partition=inductor_graph_partition, + custom_ops=custom_ops.split(","), + pass_config=PassConfig( + enable_qk_norm_rope_fusion=True, + enable_sp=True, + fuse_gemm_comms=True, + fuse_allreduce_rms=False, + # Override threshold for testing (models have small hidden_size) + sp_min_token_num=512, + ), + ) + + matches_check = [ + "norm_rope_fusion", + "sequence_parallel", + "async_tp", + ] + + run_e2e_fusion_test( + model_name, + matches, + model_kwargs, + attn_backend, + compilation_config, + matches_check, + tp_size=2, + ) + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize( + "model_name, matches_fn, model_kwargs, hf_overrides", + [llama3_8b_fp8, llama4_scout_fp8], +) +@pytest.mark.parametrize("attn_backend", [TRITON_ATTN, FLASHINFER_ATTN]) +@pytest.mark.parametrize("n_layers", [4]) +@pytest.mark.parametrize("custom_ops", custom_ops_combos("quant_fp8", "rms_norm")) +@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) +def test_tp2_sp_ar_rms_fp8_fusions( + model_name: str, + matches_fn: Callable[[int], Matches], + model_kwargs: dict, + hf_overrides: Callable[[int], dict], + attn_backend: AttentionBackendCase, + n_layers: int, + custom_ops: str, + inductor_graph_partition: bool, + run_e2e_fusion_test, +): + matches = matches_fn(n_layers) + + # Reduce size of model and skip weight loading time + model_kwargs["hf_overrides"] = hf_overrides(n_layers) + model_kwargs["load_format"] = "dummy" + model_kwargs["max_model_len"] = 1024 + + compilation_config = dict( + use_inductor_graph_partition=inductor_graph_partition, + custom_ops=custom_ops.split(","), + pass_config=PassConfig( + fuse_norm_quant=True, + fuse_act_quant=True, + fuse_attn_quant=True, + enable_qk_norm_rope_fusion=True, + enable_sp=True, + fuse_gemm_comms=True, + fuse_allreduce_rms=True, + # Override threshold for testing (models have small hidden_size) + sp_min_token_num=512, + ), + ) + + matches_check = [ + "rms_quant_fusion", + "act_quant_fusion", + "norm_rope_fusion", + "attn_quant_fusion", + "ar_rms_fusion", + "sequence_parallel", + "async_tp", + ] + + run_e2e_fusion_test( + model_name, + matches, + model_kwargs, + attn_backend, + compilation_config, + matches_check, + tp_size=2, + ) + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize( + "model_name, matches_fn, model_kwargs, hf_overrides", + [llama3_8b, qwen3_a3b], +) +@pytest.mark.parametrize("attn_backend", [TRITON_ATTN]) +@pytest.mark.parametrize("n_layers", [4]) +@pytest.mark.parametrize("custom_ops", custom_ops_combos("rms_norm")) +@pytest.mark.parametrize("inductor_graph_partition", INDUCTOR_GRAPH_PARTITION) +def test_tp2_sp_ar_rms_fusions( + model_name: str, + matches_fn: Callable[[int], Matches], + model_kwargs: dict, + hf_overrides: Callable[[int], dict], + attn_backend: AttentionBackendCase, + n_layers: int, + custom_ops: str, + inductor_graph_partition: bool, + run_e2e_fusion_test, +): + matches = matches_fn(n_layers) + + # Reduce size of model and skip weight loading time + model_kwargs["hf_overrides"] = hf_overrides(n_layers) + model_kwargs["load_format"] = "dummy" + model_kwargs["max_model_len"] = 1024 + + compilation_config = dict( + use_inductor_graph_partition=inductor_graph_partition, + custom_ops=custom_ops.split(","), + pass_config=PassConfig( + enable_qk_norm_rope_fusion=True, + enable_sp=True, + fuse_gemm_comms=True, + fuse_allreduce_rms=True, + # Override threshold for testing (models have small hidden_size) + sp_min_token_num=512, + ), + ) + + matches_check = [ + "norm_rope_fusion", + "ar_rms_fusion", + "sequence_parallel", + "async_tp", + ] + + run_e2e_fusion_test( + model_name, + matches, + model_kwargs, + attn_backend, + compilation_config, + matches_check, + tp_size=2, + ) diff --git a/asian_ai/performance/vLLM/tests/compile/h100/__init__.py b/asian_ai/performance/vLLM/tests/compile/h100/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/compile/h100/test_startup.py b/asian_ai/performance/vLLM/tests/compile/h100/test_startup.py new file mode 100644 index 0000000000000000000000000000000000000000..a4b892ee9c28f48af147ea37b798a6f1a11f89dc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/h100/test_startup.py @@ -0,0 +1,252 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Cold start and warm start tests for vLLM-compile. + +Cold start runs in a forked child (must fork before CUDA init) which +populates on-disk caches and asserts cold-start counters. Warm start +then runs in the parent with clean in-memory state but populated caches. +""" + +import multiprocessing as mp +from typing import NamedTuple + +import pytest +from torch._dynamo.utils import counters + +import vllm.envs as envs +from vllm.compilation.counter import compilation_counter +from vllm.config import CompilationConfig, CompilationMode, CUDAGraphMode, PassConfig +from vllm.utils.torch_utils import is_torch_equal_or_newer + +from ...utils import fork_new_process_for_each_test + +MODEL = "microsoft/Phi-tiny-MoE-instruct" + + +def _run_vllm(vllm_runner): + with vllm_runner( + MODEL, + trust_remote_code=False, + max_model_len=256, + max_num_batched_tokens=1024, + load_format="dummy", + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + cudagraph_mode=CUDAGraphMode.NONE, + ), + # Phi-tiny-MoE uses SWA, whose admission cap is `cdiv(L, block_size) + 1` + # at default block_size=16 — i.e. 17 blocks for max_model_len=256. Use + # 32 for headroom. + num_gpu_blocks_override=32, + ): + pass + + +def _cold_start(vllm_runner): + counters.clear() + with compilation_counter.expect( + num_compiled_artifacts_saved=3, + num_compiled_artifacts_loaded=0, + ): + _run_vllm(vllm_runner) + assert counters["aot_autograd"]["total"] == 33 + assert counters["aot_autograd"]["autograd_cache_miss"] == 3 + assert counters["aot_autograd"]["autograd_cache_hit"] == 0 + + +@fork_new_process_for_each_test +@pytest.mark.parametrize("mega_aot_artifact", ["0", "1"]) +def test_moe_startup(monkeypatch, vllm_runner, fresh_vllm_cache, mega_aot_artifact): + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + monkeypatch.setenv("VLLM_USE_MEGA_AOT_ARTIFACT", mega_aot_artifact) + monkeypatch.setenv("VLLM_DEEP_GEMM_WARMUP", "skip") + + # Cold start in a forked child (must fork before CUDA init). + # This model has 32 identical transformer layers which produce + # 33 subgraphs after splitting on attention — only 3 are unique. + ctx = mp.get_context("fork") + p = ctx.Process(target=_cold_start, args=(vllm_runner,)) + p.start() + p.join() + assert p.exitcode == 0, "Cold-start child failed" + + # Warm start — compiled artifacts loaded from disk cache. + counters.clear() + with compilation_counter.expect( + num_compiled_artifacts_loaded=3, + num_compiled_artifacts_saved=0, + ): + _run_vllm(vllm_runner) + mega_aot_active = envs.VLLM_USE_MEGA_AOT_ARTIFACT and is_torch_equal_or_newer( + "2.10.0" + ) + if mega_aot_active: + # MEGA_AOT_ARTIFACT is enabled, so we expect no aot_autograd running on + # subgraphs. + assert counters["aot_autograd"]["total"] == 0 + else: + assert counters["aot_autograd"]["total"] == 30 + assert counters["aot_autograd"]["autograd_cache_miss"] == 0 + assert ( + counters["aot_autograd"]["autograd_cache_hit"] == 0 + ) # No miss at aot_autograd level causing disk I/O. + + +# --------------------------------------------------------------------------- +# Parametrized model startup tests +# --------------------------------------------------------------------------- + + +class ModelStartupSpec(NamedTuple): + model: str + hf_overrides: dict + cold_artifacts_saved: int + warm_artifacts_saved: int + warm_artifacts_loaded: int + + +_SMALL_MOE_OVERRIDES = { + "num_hidden_layers": 8, + "hidden_size": 256, + "intermediate_size": 512, + "num_attention_heads": 8, + "num_key_value_heads": 1, + "n_routed_experts": 8, +} + +MODEL_SPECS = [ + pytest.param( + ModelStartupSpec( + model="openai/gpt-oss-120b", + hf_overrides={ + "num_hidden_layers": 8, + "hidden_size": 256, + "intermediate_size": 512, + "num_attention_heads": 8, + "num_key_value_heads": 1, + "num_local_experts": 8, + }, + cold_artifacts_saved=3, + warm_artifacts_saved=0, + warm_artifacts_loaded=3, + ), + id="gpt_oss_120b", + ), + # NOTE: DeepSeek-V3.2 requires sparse MLA (index_topk) which needs + # Hopper+ GPUs. This test must run on H100 (see pytorch.yaml). + pytest.param( + ModelStartupSpec( + model="deepseek-ai/DeepSeek-V3.2", + hf_overrides=_SMALL_MOE_OVERRIDES, + cold_artifacts_saved=9, + # https://github.com/vllm-project/vllm/issues/38051 + warm_artifacts_saved=0 if is_torch_equal_or_newer("2.12.0") else 9, + warm_artifacts_loaded=9 if is_torch_equal_or_newer("2.12.0") else 0, + ), + id="deepseek_v3.2", + ), + pytest.param( + ModelStartupSpec( + model="moonshotai/Kimi-K2.5", + hf_overrides={"text_config": _SMALL_MOE_OVERRIDES}, + cold_artifacts_saved=4, + # https://github.com/vllm-project/vllm/issues/38051 + warm_artifacts_saved=0 if is_torch_equal_or_newer("2.12.0") else 4, + warm_artifacts_loaded=4 if is_torch_equal_or_newer("2.12.0") else 0, + ), + id="kimi_k2.5", + ), + pytest.param( + ModelStartupSpec( + model="zai-org/GLM-4.5", + hf_overrides=_SMALL_MOE_OVERRIDES, + cold_artifacts_saved=4, + warm_artifacts_saved=0, + warm_artifacts_loaded=4, + ), + id="glm_4.5", + ), + pytest.param( + ModelStartupSpec( + model="MiniMaxAI/MiniMax-M2.5", + hf_overrides=_SMALL_MOE_OVERRIDES, + cold_artifacts_saved=3, + warm_artifacts_saved=0, + warm_artifacts_loaded=3, + ), + id="minimax_m2.5", + ), +] + + +def _run_model(vllm_runner, spec: ModelStartupSpec): + with vllm_runner( + spec.model, + trust_remote_code=True, + max_model_len=256, + max_num_batched_tokens=1024, + block_size=64, + load_format="dummy", + hf_overrides=spec.hf_overrides, + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + cudagraph_mode=CUDAGraphMode.NONE, + pass_config=PassConfig(fuse_allreduce_rms=False), + ), + num_gpu_blocks_override=16, + ): + pass + + +def _check_model_run(vllm_runner, spec: ModelStartupSpec, is_cold_start: bool): + """Runs a model and checks the number of compiled artifacts.""" + old = compilation_counter.clone() + _run_model(vllm_runner, spec) + saved = ( + compilation_counter.num_compiled_artifacts_saved + - old.num_compiled_artifacts_saved + ) + loaded = ( + compilation_counter.num_compiled_artifacts_loaded + - old.num_compiled_artifacts_loaded + ) + + start_type = "COLD" if is_cold_start else "WARM" + # Print actual values for debugging — intentional, helps diagnose + # failures and calibrate expected counts when adding new models. + print(f"\n=== {start_type} START for {spec.model} ===") + print(f" num_compiled_artifacts_saved={saved}") + print(f" num_compiled_artifacts_loaded={loaded}") + + if is_cold_start: + expected_saved = spec.cold_artifacts_saved + expected_loaded = 0 + else: + expected_saved = spec.warm_artifacts_saved + expected_loaded = spec.warm_artifacts_loaded + + assert saved == expected_saved, f"{start_type.lower()}_artifacts_saved: got {saved}" + assert loaded == expected_loaded, ( + f"{start_type.lower()}_artifacts_loaded: got {loaded}" + ) + + +def _cold_start_model(vllm_runner, spec: ModelStartupSpec): + _check_model_run(vllm_runner, spec, is_cold_start=True) + + +@pytest.mark.parametrize("spec", MODEL_SPECS) +@fork_new_process_for_each_test +def test_model_startup(monkeypatch, vllm_runner, fresh_vllm_cache, spec): + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + monkeypatch.setenv("VLLM_DEEP_GEMM_WARMUP", "skip") + + # Cold start in a forked child (must fork before CUDA init). + ctx = mp.get_context("fork") + p = ctx.Process(target=_cold_start_model, args=(vllm_runner, spec)) + p.start() + p.join() + assert p.exitcode == 0, "Cold-start child failed" + + # Warm start — compiled artifacts loaded from disk cache. + _check_model_run(vllm_runner, spec, is_cold_start=False) diff --git a/asian_ai/performance/vLLM/tests/compile/passes/__init__.py b/asian_ai/performance/vLLM/tests/compile/passes/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/compile/passes/distributed/__init__.py b/asian_ai/performance/vLLM/tests/compile/passes/distributed/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/compile/passes/distributed/test_async_tp.py b/asian_ai/performance/vLLM/tests/compile/passes/distributed/test_async_tp.py new file mode 100644 index 0000000000000000000000000000000000000000..3d61274f9dd9a9c6228bf207230595ba1e34b94f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/distributed/test_async_tp.py @@ -0,0 +1,405 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import pytest +import torch + +import vllm.envs as envs +from tests.compile.backend import TestBackend +from tests.utils import ( + multi_gpu_test, +) +from vllm.compilation.passes.fusion.collective_fusion import AsyncTPPass +from vllm.config import ( + CompilationConfig, + DeviceConfig, + ModelConfig, + PassConfig, + VllmConfig, + set_current_vllm_config, +) +from vllm.config.utils import Range +from vllm.distributed import ( + tensor_model_parallel_all_gather, + tensor_model_parallel_reduce_scatter, +) +from vllm.distributed.parallel_state import ( + init_distributed_environment, + initialize_model_parallel, +) +from vllm.platforms import current_platform +from vllm.utils.network_utils import get_open_port +from vllm.utils.system_utils import update_environment_variables +from vllm.utils.torch_utils import set_random_seed + +DEVICE_TYPE = current_platform.device_type +FP8_DTYPE = current_platform.fp8_dtype() + +prompts = [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + "The future of AI is", +] + + +class TestMMRSModel(torch.nn.Module): + def __init__(self, hidden_size=16, dtype=torch.float16): + super().__init__() + self.hidden_size = hidden_size + self.dtype = dtype + self.gate_proj = torch.nn.Parameter( + torch.empty((self.hidden_size * 2, hidden_size)), requires_grad=False + ) + # Initialize weights + torch.nn.init.normal_(self.gate_proj, std=0.02) + + def forward(self, hidden_states): + """ + Forward pass implementing the mm + reduce scatter in the FX graph + + """ + # Reshape input + view = hidden_states.reshape(-1, self.hidden_size) + + # matrix multiplication + permute = self.gate_proj.permute(1, 0) + mm = torch.mm(view, permute) + reduce_scatter = tensor_model_parallel_reduce_scatter(mm, dim=0) + return reduce_scatter + + def ops_in_model_before(self): + return [torch.ops.vllm.reduce_scatter.default] + + def ops_in_model_after(self): + return [torch.ops.symm_mem.fused_matmul_reduce_scatter.default] + + +class TestAGMMModel(torch.nn.Module): + def __init__(self, hidden_size=16, dtype=torch.float16): + super().__init__() + self.hidden_size = hidden_size + self.dtype = dtype + self.weight = torch.nn.Parameter( + torch.empty((hidden_size, hidden_size)), requires_grad=False + ) + # Initialize weights + torch.nn.init.normal_(self.weight, std=0.02) + + def forward(self, hidden_states): + """ + Forward pass implementing the mm + all gather in the FX graph + """ + # Reshape input + view = hidden_states.reshape(-1, self.hidden_size) + all_gather = tensor_model_parallel_all_gather(view, dim=0) + permute = self.weight.permute(1, 0) + mm = torch.mm(all_gather, permute) + return mm + + def ops_in_model_before(self): + return [torch.ops.vllm.all_gather.default] + + def ops_in_model_after(self): + return [torch.ops.symm_mem.fused_all_gather_matmul.default] + + +class _BaseScaledMMModel(torch.nn.Module): + def __init__(self, hidden_size=16, dtype=torch.float16): + super().__init__() + self.hidden_size = hidden_size + self.dtype = dtype + self.weight = ( + torch.empty([hidden_size, hidden_size], dtype=FP8_DTYPE) + .contiguous() + .transpose(0, 1) + ) + + # Initialize scale_b for _scaled_mm. + self.scale_b = torch.ones(1, self.hidden_size, dtype=torch.float32) + + +class TestScaledMMRSModel(_BaseScaledMMModel): + def forward(self, input: torch.Tensor): + """ + Forward pass implementing the scaled_mm + reduce scatter in the FX graph + + """ + fp8_input = input.to(FP8_DTYPE) + scale_a = torch.ones(input.shape[0], 1, dtype=torch.float32) + scaled_mm = torch._scaled_mm( + fp8_input, + self.weight, + scale_a=scale_a, + scale_b=self.scale_b, + out_dtype=self.dtype, + ) + reduce_scatter = tensor_model_parallel_reduce_scatter(scaled_mm, dim=0) + return reduce_scatter + + def ops_in_model_before(self): + return [torch.ops.vllm.reduce_scatter.default] + + def ops_in_model_after(self): + return [torch.ops.vllm.patched_fused_scaled_matmul_reduce_scatter.default] + + +class TestAGScaledMMModel(_BaseScaledMMModel): + def forward(self, input: torch.Tensor): + """ + Forward pass implementing the all gather + scaled_mm in the FX graph + """ + # Reshape input + fp8_input = input.to(FP8_DTYPE) + all_gather = tensor_model_parallel_all_gather(fp8_input, dim=0) + + scale_a = torch.ones(all_gather.shape[0], 1, dtype=torch.float32) + scaled_mm = torch._scaled_mm( + all_gather, + self.weight, + scale_a=scale_a, + scale_b=self.scale_b, + out_dtype=self.dtype, + ) + return scaled_mm + + def ops_in_model_before(self): + return [torch.ops.vllm.all_gather.default] + + def ops_in_model_after(self): + return [torch.ops.symm_mem.fused_all_gather_scaled_matmul.default] + + +class TestCutlassScaledMMRSModel(_BaseScaledMMModel): + def forward(self, input: torch.Tensor): + """ + Forward pass implementing the cutlass_scaled_mm + reduce scatter + in the FX graph + + """ + fp8_input = input.to(FP8_DTYPE) + scale_a = torch.ones(input.shape[0], 1, dtype=torch.float32) + mm_out = torch.empty( + (fp8_input.shape[0], self.weight.shape[1]), + dtype=self.dtype, + device=input.device, + ) + torch.ops._C.cutlass_scaled_mm( + mm_out, fp8_input, self.weight, scale_a, self.scale_b, None + ) + reduce_scatter = tensor_model_parallel_reduce_scatter(mm_out, dim=0) + return reduce_scatter + + def ops_in_model_before(self): + return [torch.ops.vllm.reduce_scatter.default] + + def ops_in_model_after(self): + return [torch.ops.vllm.patched_fused_scaled_matmul_reduce_scatter.default] + + +class TestAGCutlassScaledMMModel(_BaseScaledMMModel): + def forward(self, input: torch.Tensor): + """ + Forward pass implementing the all gather + cutlass_scaled_mm + in the FX graph + """ + # Reshape input + fp8_input = input.to(FP8_DTYPE) + all_gather = tensor_model_parallel_all_gather(fp8_input, dim=0) + + scale_a = torch.ones(all_gather.shape[0], 1, dtype=torch.float32) + + mm_out = torch.empty( + (all_gather.shape[0], self.weight.shape[1]), + dtype=self.dtype, + device=all_gather.device, + ) + torch.ops._C.cutlass_scaled_mm( + mm_out, all_gather, self.weight, scale_a, self.scale_b, None + ) + return mm_out + + def ops_in_model_before(self): + return [torch.ops.vllm.all_gather.default] + + def ops_in_model_after(self): + return [torch.ops.symm_mem.fused_all_gather_scaled_matmul.default] + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize( + "test_model", + [ + TestMMRSModel, + TestAGMMModel, + TestScaledMMRSModel, + TestAGScaledMMModel, + pytest.param( + TestCutlassScaledMMRSModel, + marks=pytest.mark.skipif( + not hasattr(torch.ops._C, "cutlass_scaled_mm"), + reason="Requires cutlass_scaled_mm", + ), + ), + pytest.param( + TestAGCutlassScaledMMModel, + marks=pytest.mark.skipif( + not hasattr(torch.ops._C, "cutlass_scaled_mm"), + reason="Requires cutlass_scaled_mm", + ), + ), + ], +) +@pytest.mark.parametrize("batch_size", [8]) +@pytest.mark.parametrize("seq_len", [16]) +@pytest.mark.parametrize("hidden_size", [16]) +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@pytest.mark.parametrize("dynamic", [True, False]) +@pytest.mark.skipif(envs.VLLM_TARGET_DEVICE not in ["cuda"], reason="Only test on CUDA") +def test_async_tp_pass_replace( + test_model: str, + batch_size: int, + seq_len: int, + hidden_size: int, + dtype: torch.dtype, + dynamic: bool, +): + if ( + test_model + in ( + TestScaledMMRSModel, + TestAGScaledMMModel, + TestCutlassScaledMMRSModel, + TestAGCutlassScaledMMModel, + ) + and dtype == torch.float16 + ): + pytest.skip( + "Only bf16 high precision output types are supported for " + "per-token (row-wise) scaling" + ) + + num_processes = 2 + master_port = str(get_open_port()) + + def run_torch_spawn(fn, nprocs): + # need to use torch.mp.spawn otherwise will have problems with + # torch.distributed and cuda + torch.multiprocessing.spawn( + fn, + args=( + num_processes, + test_model, + batch_size, + seq_len, + hidden_size, + dtype, + dynamic, + master_port, + ), + nprocs=nprocs, + ) + + run_torch_spawn(async_tp_pass_on_test_model, num_processes) + + +def test_async_tp_pass_requires_full_graph_compilation(): + vllm_config = VllmConfig() + vllm_config.compilation_config.use_inductor_graph_partition = False + vllm_config.compilation_config.splitting_ops = [ + "vllm::unified_attention_with_output" + ] + + async_tp_pass = object.__new__(AsyncTPPass) + async_tp_pass.compilation_config = vllm_config.compilation_config + + with pytest.raises( + AssertionError, match="AsyncTPPass requires full-graph compilation" + ): + async_tp_pass.is_applicable_for_range(Range(start=8, end=8)) + + +def async_tp_pass_on_test_model( + local_rank: int, + world_size: int, + test_model_cls: torch.nn.Module, + batch_size: int, + seq_len: int, + hidden_size: int, + dtype: torch.dtype, + dynamic: bool, + master_port: str = "0", +): + set_random_seed(0) + + device = torch.device(f"{DEVICE_TYPE}:{local_rank}") + torch.accelerator.set_device_index(device) + torch.set_default_device(device) + torch.set_default_dtype(dtype) + + update_environment_variables( + { + "RANK": str(local_rank), + "LOCAL_RANK": str(local_rank), + "WORLD_SIZE": str(world_size), + "MASTER_ADDR": "localhost", + "MASTER_PORT": master_port, + } + ) + + # initialize distributed + init_distributed_environment() + + # configure vllm config for SequenceParallelismPass + vllm_config = VllmConfig() + vllm_config.compilation_config = CompilationConfig( + pass_config=PassConfig( + fuse_gemm_comms=True, + ), + ) + vllm_config.device_config = DeviceConfig(device=torch.device(DEVICE_TYPE)) + + # this is a fake model name to construct the model config + # in the vllm_config, it's not really used. + model_name = "RedHatAI/Llama-3.2-1B-Instruct-FP8" + vllm_config.model_config = ModelConfig( + model=model_name, trust_remote_code=True, dtype=dtype, seed=42 + ) + + with set_current_vllm_config(vllm_config): + initialize_model_parallel(tensor_model_parallel_size=world_size) + + async_tp_pass = AsyncTPPass(vllm_config) + backend = TestBackend(async_tp_pass) + + assert ( + async_tp_pass.compilation_config.splitting_ops + == vllm_config.compilation_config.splitting_ops + ) + assert ( + async_tp_pass.compilation_config.use_inductor_graph_partition + == vllm_config.compilation_config.use_inductor_graph_partition + ) + + model = test_model_cls(hidden_size, dtype) # Pass dtype to model constructor + + hidden_states = torch.randn( + (batch_size * seq_len, hidden_size), dtype=dtype, requires_grad=False + ) + + if dynamic: + torch._dynamo.mark_dynamic(hidden_states, 0) + + compiled_model = torch.compile(model, backend=backend) + compiled_model(hidden_states) + + assert async_tp_pass.matched_count == 1 + + # In pre-nodes, all gather or reduce scatter should exist, + # fused_matmul_reduce_scatter or fused_all_gather_matmul should not + backend.check_before_ops(model.ops_in_model_before(), fully_replaced=False) + + # In post-nodes, fused_matmul_reduce_scatter or \ + # fused_all_gather_matmul should exist + backend.check_after_ops(model.ops_in_model_after()) diff --git a/asian_ai/performance/vLLM/tests/compile/passes/distributed/test_fusion_all_reduce.py b/asian_ai/performance/vLLM/tests/compile/passes/distributed/test_fusion_all_reduce.py new file mode 100644 index 0000000000000000000000000000000000000000..cdd58997ad22a85becbceb1a4c1e678927016239 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/distributed/test_fusion_all_reduce.py @@ -0,0 +1,741 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from importlib.util import find_spec + +import pytest +import torch + +import vllm.envs as envs +from tests.compile.backend import TestBackend +from tests.utils import TestFP8Layer, has_module_attribute, multi_gpu_test +from vllm._aiter_ops import IS_AITER_FOUND, rocm_aiter_ops +from vllm._custom_ops import cutlass_scaled_fp4_mm, scaled_fp4_quant +from vllm.compilation.passes.fusion.allreduce_rms_fusion import ( + AllReduceFusionPass, + RocmAiterAllReduceFusionPass, +) +from vllm.compilation.passes.fx_utils import find_op_nodes +from vllm.compilation.passes.utility.fix_functionalization import ( + FixFunctionalizationPass, +) +from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass +from vllm.compilation.passes.utility.post_cleanup import PostCleanupPass +from vllm.config import ( + CompilationConfig, + CompilationMode, + DeviceConfig, + ModelConfig, + PassConfig, + VllmConfig, + set_current_vllm_config, +) +from vllm.distributed import tensor_model_parallel_all_reduce +from vllm.distributed.parallel_state import ( + init_distributed_environment, + initialize_model_parallel, +) +from vllm.model_executor.layers.layernorm import GemmaRMSNorm, RMSNorm +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + kFp8StaticTensorSym, +) +from vllm.platforms import current_platform +from vllm.utils.system_utils import update_environment_variables +from vllm.utils.torch_utils import set_random_seed + +DEVICE_TYPE = current_platform.device_type + + +class TestAllReduceRMSNormModel(torch.nn.Module): + def __init__( + self, + hidden_size=16, + token_num=16, + eps=1e-6, + dtype: torch.dtype = torch.float16, + use_aiter: bool = False, + ): + super().__init__() + self.hidden_size = hidden_size + self.eps = eps + self.norm = [RMSNorm(hidden_size, eps) for i in range(4)] + self.w = [torch.rand(hidden_size, hidden_size) for _ in range(3)] + self.use_aiter = use_aiter + + def forward(self, x): + # avoid having graph input be an arg to a pattern directly + z = torch.relu(x) + x = resid = tensor_model_parallel_all_reduce(z) + y = self.norm[0](x) + + z2 = torch.mm(y, self.w[0]) + x2 = tensor_model_parallel_all_reduce(z2) + + y2, resid = self.norm[1](x2, resid) + + z3 = torch.mm(y2, self.w[1]) + x3 = tensor_model_parallel_all_reduce(z3) + + y3, resid = self.norm[2](x3, resid) + + z4 = torch.mm(y3, self.w[2]) + x4 = tensor_model_parallel_all_reduce(z4) + + y4, resid = self.norm[3](x4, resid) + return y4 + + def ops_in_model_before(self): + return [torch.ops.vllm.all_reduce.default] + + def ops_in_model_after(self): + if self.use_aiter: + return [rocm_aiter_ops.get_fused_allreduce_rmsnorm_op()] + return [torch.ops.vllm.flashinfer_trtllm_fused_allreduce_norm.default] + + +class TestAllReduceGemmaRMSNormModel(torch.nn.Module): + def __init__( + self, + hidden_size=16, + token_num=16, + eps=1e-6, + dtype: torch.dtype = torch.float16, + ): + super().__init__() + self.hidden_size = hidden_size + self.eps = eps + self.norm = [GemmaRMSNorm(hidden_size, eps) for _ in range(4)] + # Non-trivial weight (~Gemma range) so (1 + w) exercises the scale path. + for n in self.norm: + n.weight.data.normal_(mean=0.0, std=0.1) + self.w = [torch.rand(hidden_size, hidden_size) for _ in range(3)] + + def forward(self, x): + # avoid having graph input be an arg to a pattern directly + z = torch.relu(x) + x = resid = tensor_model_parallel_all_reduce(z) + y = self.norm[0](x) + + z2 = torch.mm(y, self.w[0]) + x2 = tensor_model_parallel_all_reduce(z2) + y2, resid = self.norm[1](x2, resid) + + z3 = torch.mm(y2, self.w[1]) + x3 = tensor_model_parallel_all_reduce(z3) + y3, resid = self.norm[2](x3, resid) + + z4 = torch.mm(y3, self.w[2]) + x4 = tensor_model_parallel_all_reduce(z4) + y4, resid = self.norm[3](x4, resid) + return y4 + + def ops_in_model_before(self): + return [torch.ops.vllm.all_reduce.default] + + def ops_in_model_after(self): + return [torch.ops.vllm.flashinfer_trtllm_fused_allreduce_norm.default] + + +class TestAllReduceRMSNormStaticQuantFP8Model(torch.nn.Module): + quant_key = kFp8StaticTensorSym + + def __init__( + self, hidden_size=16, token_num=16, eps=1e-6, dtype: torch.dtype = torch.float16 + ): + super().__init__() + self.hidden_size = hidden_size + self.eps = eps + self.norm = [RMSNorm(hidden_size, eps) for i in range(4)] + self.fp8_linear_layers = [ + TestFP8Layer( + weight_shape=(hidden_size, hidden_size), + activation_quant_key=self.quant_key, + weight_quant_key=self.quant_key, + input_dtype=dtype, + ) + for i in range(3) + ] + + def forward(self, hidden_states): + # avoid having graph input be an arg to a pattern directly + z = torch.relu(hidden_states) + x = resid = tensor_model_parallel_all_reduce(z) + y = self.norm[0](x) + + z2 = self.fp8_linear_layers[0](y) + + x2 = tensor_model_parallel_all_reduce(z2) + y2, resid = self.norm[1](x2, resid) + + z3 = self.fp8_linear_layers[1](y2) + + x3 = tensor_model_parallel_all_reduce(z3) + y3, resid = self.norm[2](x3, resid) # use resid here + + z4 = self.fp8_linear_layers[2](y3) + + x4 = tensor_model_parallel_all_reduce(z4) + y4, resid = self.norm[3](x4, resid) # use resid here + return y4 + + def ops_in_model_after(self): + return [torch.ops.vllm.flashinfer_trtllm_fused_allreduce_norm.default] + + def ops_in_model_before(self): + return [ + torch.ops.vllm.all_reduce.default, + torch.ops._C.static_scaled_fp8_quant.default + if self.fp8_linear_layers[0].is_quant_fp8_enabled() + else torch.ops.aten.reciprocal.default, + ] + + +class TestAiterAllReduceRMSNormGroupQuantFP8Model(torch.nn.Module): + """Exercises the new ROCm AITER AR+RMS+per-group-FP8-quant patterns. + + Four ``rms_norm`` sites that together hit every pattern registered by + ``RocmAiterAllReduceFusionPass`` for the per-group FP8 quant path: + + * ``norm[0]``: ``all_reduce -> rms_norm -> group_fp8_quant`` (no residual) + -> ``AiterAllreduceFusedRMSNormGroupQuantFP8Pattern`` + * ``norm[1]``: ``all_reduce -> fused_add_rms_norm -> group_fp8_quant`` + (single ``rms`` consumer) + -> ``AiterAllreduceFusedAddRMSNormGroupQuantFP8Pattern`` + * ``norm[2..3]``: ``all_reduce -> fused_add_rms_norm + -> (group_fp8_quant + rocm_unquantized_gemm)`` (two ``rms`` consumers, + modeling the DSv3.2 indexer fan-out) + -> ``AiterAllreduceFusedAddRMSNormGroupQuantWithIndexerPattern`` + + The chain feeds the next AllReduce by dequantizing the FP8 output (FP8 + cast back to bf16 multiplied by the per-group scale), which is enough to + keep the matmul chain bf16 without depending on a real FP8 block-scaled + GEMM kernel. + """ + + quant_group_size = 128 + indexer_out_dim = 8 + + def __init__( + self, + hidden_size=128, + token_num=16, + eps=1e-6, + dtype: torch.dtype = torch.bfloat16, + use_triton_quant: bool = False, + ): + super().__init__() + self.hidden_size = hidden_size + self.eps = eps + self.use_triton_quant = use_triton_quant + assert hidden_size % self.quant_group_size == 0, ( + f"hidden_size ({hidden_size}) must be a multiple of " + f"quant_group_size ({self.quant_group_size}) for per-group FP8 quant" + ) + self.norm = [RMSNorm(hidden_size, eps) for _ in range(4)] + self.w = [torch.rand(hidden_size, hidden_size, dtype=dtype) for _ in range(3)] + self.indexer_w = [ + torch.rand(self.indexer_out_dim, hidden_size, dtype=dtype) for _ in range(2) + ] + + def _group_quant(self, rms: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + if self.use_triton_quant: + return torch.ops.vllm.triton_per_token_group_quant_fp8( + rms, self.quant_group_size + ) + return torch.ops.vllm.rocm_aiter_group_fp8_quant.default( + rms, self.quant_group_size + ) + + def _dequantize_to_bf16( + self, q: torch.Tensor, s: torch.Tensor, ref: torch.Tensor + ) -> torch.Tensor: + # Broadcast the per-group scale across each group of `quant_group_size` + # so we can chain the FP8 output back into a bf16 matmul. This avoids + # depending on a real FP8 block-scaled GEMM kernel in the test. + s_full = s.repeat_interleave(self.quant_group_size, dim=-1).to(ref.dtype) + return q.to(ref.dtype) * s_full + + def forward(self, hidden_states): + z = torch.relu(hidden_states) + x = resid = tensor_model_parallel_all_reduce(z) + rms = self.norm[0](x) + q0, s0 = self._group_quant(rms) + y = self._dequantize_to_bf16(q0, s0, rms) + + z2 = torch.mm(y, self.w[0]) + x2 = tensor_model_parallel_all_reduce(z2) + rms2, resid = self.norm[1](x2, resid) + q1, s1 = self._group_quant(rms2) + y2 = self._dequantize_to_bf16(q1, s1, rms2) + + z3 = torch.mm(y2, self.w[1]) + x3 = tensor_model_parallel_all_reduce(z3) + rms3, resid = self.norm[2](x3, resid) + q2, s2 = self._group_quant(rms3) + # Second consumer of ``rms3``: forces the with-indexer pattern. + idx2 = torch.ops.vllm.rocm_unquantized_gemm(rms3, self.indexer_w[0], None) + y3 = self._dequantize_to_bf16(q2, s2, rms3) + + z4 = torch.mm(y3, self.w[2]) + x4 = tensor_model_parallel_all_reduce(z4) + rms4, resid = self.norm[3](x4, resid) + q3, s3 = self._group_quant(rms4) + # Second consumer of ``rms4``: forces the with-indexer pattern. + idx3 = torch.ops.vllm.rocm_unquantized_gemm(rms4, self.indexer_w[1], None) + y4 = self._dequantize_to_bf16(q3, s3, rms4) + return y4, idx2, idx3 + + def ops_in_model_before(self): + return [ + torch.ops.vllm.all_reduce.default, + ( + torch.ops.vllm.triton_per_token_group_quant_fp8.default + if self.use_triton_quant + else torch.ops.vllm.rocm_aiter_group_fp8_quant.default + ), + ] + + def ops_in_model_after(self): + return [ + rocm_aiter_ops.get_fused_allreduce_rmsnorm_quant_per_group_op(), + rocm_aiter_ops.get_fused_allreduce_rmsnorm_quant_per_group_with_bf16_norm_op(), # noqa: E501 + ] + + +class TestAllReduceFusedAddRMSNormStaticQuantFP4Model(torch.nn.Module): + def __init__( + self, hidden_size=16, token_num=16, eps=1e-6, dtype: torch.dtype = torch.float16 + ): + super().__init__() + self.hidden_size = hidden_size + self.eps = eps + self.norm = [RMSNorm(hidden_size, eps) for i in range(4)] + + self.w = [torch.rand(hidden_size, hidden_size) for _ in range(3)] + self.agscale = [torch.rand(1, dtype=torch.float32) for _ in range(3)] + wgscale = [torch.rand(1, dtype=torch.float32) for _ in range(3)] + self.alpha = [1 / (w * a) for w, a in zip(wgscale, self.agscale)] + + wq_gen, wscale_gen = zip( + *(scaled_fp4_quant(w, wg) for w, wg in zip(self.w, wgscale)) + ) + self.wq, self.wscale = list(wq_gen), list(wscale_gen) + + def forward(self, hidden_states): + # avoid having graph input be an arg to a pattern directly + z = torch.relu(hidden_states) + x = resid = tensor_model_parallel_all_reduce(z) + y = self.norm[0](x) + + yq, y_scale = scaled_fp4_quant(y, self.agscale[0]) + z2 = cutlass_scaled_fp4_mm( + yq, self.wq[0], y_scale, self.wscale[0], self.alpha[0], out_dtype=y.dtype + ) + + x2 = tensor_model_parallel_all_reduce(z2) + y2, resid = self.norm[1](x2, resid) + + yq2, y_scale2 = scaled_fp4_quant(y2, self.agscale[1]) + z3 = cutlass_scaled_fp4_mm( + yq2, self.wq[1], y_scale2, self.wscale[1], self.alpha[1], out_dtype=y2.dtype + ) + + x3 = tensor_model_parallel_all_reduce(z3) + y3, resid = self.norm[2](x3, resid) # use resid here + + yq3, y_scale3 = scaled_fp4_quant(y3, self.agscale[2]) + z4 = cutlass_scaled_fp4_mm( + yq3, self.wq[2], y_scale3, self.wscale[2], self.alpha[2], out_dtype=y3.dtype + ) + x4 = tensor_model_parallel_all_reduce(z4) + y4, resid = self.norm[3](x4, resid) # use resid here + return y4 + + def ops_in_model_after(self): + return [torch.ops.vllm.flashinfer_trtllm_fused_allreduce_norm.default] + + def ops_in_model_before(self): + return [ + torch.ops.vllm.all_reduce.default, + torch.ops._C.scaled_fp4_quant.out, + ] + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize( + "test_model, enable_quant_fp8_custom_op, use_aiter", + [ + (TestAllReduceRMSNormModel, False, IS_AITER_FOUND), + pytest.param( + TestAllReduceGemmaRMSNormModel, + False, + False, + marks=pytest.mark.skipif( + current_platform.is_rocm(), + reason="Not supported on ROCm platform", + ), + ), + pytest.param( + TestAllReduceRMSNormStaticQuantFP8Model, + True, + False, + marks=pytest.mark.skipif( + current_platform.is_rocm(), + reason="Not supported on ROCm platform", + ), + ), + pytest.param( + TestAllReduceRMSNormStaticQuantFP8Model, + False, + False, + marks=pytest.mark.skipif( + current_platform.is_rocm(), + reason="Not supported on ROCm platform", + ), + ), + pytest.param( + TestAllReduceFusedAddRMSNormStaticQuantFP4Model, + False, + False, + marks=pytest.mark.skipif( + current_platform.is_rocm(), + reason="Not supported on ROCm platform", + ), + ), + ], +) +@pytest.mark.parametrize("batch_size", [8]) +@pytest.mark.parametrize("seq_len", [8]) +@pytest.mark.parametrize("hidden_size", [64]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("enable_rms_norm_custom_op", [True, False]) +@pytest.mark.parametrize("flashinfer_allreduce_backend", ["trtllm", "mnnvl"]) +@pytest.mark.skipif(envs.VLLM_TARGET_DEVICE not in ["cuda"], reason="Only test on CUDA") +@pytest.mark.skipif( + current_platform.is_rocm() and not IS_AITER_FOUND, + reason="aiter is not found", +) +@pytest.mark.skipif( + current_platform.is_cuda() + and ( + not find_spec("flashinfer") + or not has_module_attribute("flashinfer.comm", "allreduce_fusion") + or not has_module_attribute( + "flashinfer.comm", "create_allreduce_fusion_workspace" + ) + ), + reason="flashinfer is not found or flashinfer " + "is not compiled with allreduce_fusion", +) +def test_all_reduce_fusion_pass_replace( + test_model: torch.nn.Module, + batch_size: int, + seq_len: int, + hidden_size: int, + dtype: torch.dtype, + enable_rms_norm_custom_op, + enable_quant_fp8_custom_op, + flashinfer_allreduce_backend, + use_aiter: bool, + monkeypatch: pytest.MonkeyPatch, +): + if use_aiter: + with monkeypatch.context() as m: + m.setenv("VLLM_ROCM_USE_AITER", str(use_aiter)) + rocm_aiter_ops.refresh_env_variables() + + num_processes = 2 + if ( + test_model == TestAllReduceFusedAddRMSNormStaticQuantFP4Model + and not current_platform.has_device_capability(100) + ): + pytest.skip( + "Skip as nvfp4 is only supported on " + "devices with compute capability 10.0 (Blackwell)" + ) + + def run_torch_spawn(fn, nprocs): + torch.multiprocessing.spawn( + fn, + args=( + num_processes, + test_model, + batch_size, + seq_len, + hidden_size, + dtype, + enable_rms_norm_custom_op, + enable_quant_fp8_custom_op, + flashinfer_allreduce_backend, + use_aiter, + monkeypatch, + ), + nprocs=nprocs, + ) + + run_torch_spawn(all_reduce_fusion_pass_on_test_model, num_processes) + + +def all_reduce_fusion_pass_on_test_model( + local_rank: int, + world_size: int, + test_model_cls: torch.nn.Module, + batch_size: int, + seq_len: int, + hidden_size: int, + dtype: torch.dtype, + enable_rms_norm_custom_op, + enable_quant_fp8_custom_op, + flashinfer_allreduce_backend, + use_aiter: bool, + monkeypatch: pytest.MonkeyPatch, +): + set_random_seed(0) + + device = torch.device(f"{DEVICE_TYPE}:{local_rank}") + torch.accelerator.set_device_index(device) + torch.set_default_device(device) + torch.set_default_dtype(dtype) + + update_environment_variables( + { + "RANK": str(local_rank), + "LOCAL_RANK": str(local_rank), + "WORLD_SIZE": str(world_size), + "MASTER_ADDR": "localhost", + "MASTER_PORT": "12345", + "VLLM_FLASHINFER_ALLREDUCE_BACKEND": flashinfer_allreduce_backend, + } + ) + + init_distributed_environment() + + custom_ops = [] + if enable_rms_norm_custom_op: + custom_ops.append("+rms_norm") + if enable_quant_fp8_custom_op: + custom_ops.append("+quant_fp8") + + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, custom_ops=custom_ops + ) + ) + vllm_config.compilation_config.pass_config = PassConfig( + fuse_allreduce_rms=True, eliminate_noops=True + ) + vllm_config.device_config = DeviceConfig(device=torch.device(DEVICE_TYPE)) + vllm_config.parallel_config.rank = local_rank # Setup rank for debug path + + # this is a fake model name to construct the model config + # in the vllm_config, it's not really used. + model_name = "RedHatAI/Llama-3.2-1B-Instruct-FP8" + vllm_config.model_config = ModelConfig( + model=model_name, trust_remote_code=True, dtype=dtype, seed=42 + ) + with set_current_vllm_config(vllm_config): + initialize_model_parallel(tensor_model_parallel_size=world_size) + all_reduce_fusion_pass = ( + RocmAiterAllReduceFusionPass(vllm_config) + if use_aiter + else AllReduceFusionPass(vllm_config) + ) + noop_pass = NoOpEliminationPass(vllm_config) + func_pass = FixFunctionalizationPass(vllm_config) + cleanup_pass = PostCleanupPass(vllm_config) + + backend = TestBackend( + noop_pass, all_reduce_fusion_pass, func_pass, cleanup_pass + ) + + token_num = batch_size * seq_len + if test_model_cls is TestAllReduceRMSNormModel: + model = test_model_cls( + hidden_size, token_num, dtype=dtype, use_aiter=use_aiter + ) + else: + model = test_model_cls(hidden_size, token_num, dtype=dtype) + + hidden_states = torch.randn((token_num, hidden_size), requires_grad=False) + + compiled_model = torch.compile(model, backend=backend) + compiled_model(hidden_states) + + results_unfused = model(hidden_states) + results_fused = compiled_model(hidden_states) + torch.testing.assert_close(results_unfused, results_fused, atol=1e-2, rtol=1e-2) + + assert all_reduce_fusion_pass.matched_count == 4, ( + f"{all_reduce_fusion_pass.matched_count=}" + ) + backend.check_before_ops(model.ops_in_model_before(), fully_replaced=False) + backend.check_after_ops(model.ops_in_model_after()) + if test_model_cls is TestAllReduceGemmaRMSNormModel: + fused_op = torch.ops.vllm.flashinfer_trtllm_fused_allreduce_norm.default + fused_nodes = list(find_op_nodes(fused_op, backend.graph_post_pass)) + assert fused_nodes + assert all(n.kwargs.get("weight_bias") == 1.0 for n in fused_nodes) + del all_reduce_fusion_pass + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize("use_triton_quant", [True, False]) +@pytest.mark.parametrize("batch_size", [8]) +@pytest.mark.parametrize("seq_len", [8]) +@pytest.mark.parametrize("hidden_size", [128]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("enable_rms_norm_custom_op", [True, False]) +@pytest.mark.skipif( + not current_platform.is_rocm(), + reason="ROCm AITER AR+RMS+per-group-FP8-quant fusion is ROCm-only", +) +@pytest.mark.skipif(not IS_AITER_FOUND, reason="aiter is not found") +def test_rocm_aiter_all_reduce_rmsnorm_group_quant_fp8_fusion_pass_replace( + batch_size: int, + seq_len: int, + hidden_size: int, + dtype: torch.dtype, + enable_rms_norm_custom_op: bool, + use_triton_quant: bool, + monkeypatch: pytest.MonkeyPatch, +): + """Sibling of ``test_all_reduce_fusion_pass_replace`` for the new + ROCm AITER AR+RMS+per-group-FP8-quant fusion patterns. + + Validates the three new ``VllmPatternReplacement`` patterns added to + ``RocmAiterAllReduceFusionPass``: + + * ``AiterAllreduceFusedRMSNormGroupQuantFP8Pattern`` (no-residual) + * ``AiterAllreduceFusedAddRMSNormGroupQuantFP8Pattern`` (with-residual, + single ``rms`` consumer) + * ``AiterAllreduceFusedAddRMSNormGroupQuantWithIndexerPattern`` (with- + residual, DSv3.2 indexer fan-out; parametrized over both + ``triton_per_token_group_quant_fp8`` and ``rocm_aiter_group_fp8_quant`` + producers). + """ + with monkeypatch.context() as m: + m.setenv("VLLM_ROCM_USE_AITER", "1") + rocm_aiter_ops.refresh_env_variables() + + if not rocm_aiter_ops.has_fused_allreduce_rmsnorm_quant_per_group(): + pytest.skip( + "aiter build is missing 'fused_ar_rms_per_group_quant' (needs " + "ROCm/aiter PR #2823); the new patterns aren't registered." + ) + + num_processes = 2 + + def run_torch_spawn(fn, nprocs): + torch.multiprocessing.spawn( + fn, + args=( + num_processes, + TestAiterAllReduceRMSNormGroupQuantFP8Model, + batch_size, + seq_len, + hidden_size, + dtype, + enable_rms_norm_custom_op, + use_triton_quant, + monkeypatch, + ), + nprocs=nprocs, + ) + + run_torch_spawn(rocm_aiter_group_quant_fusion_pass_on_test_model, num_processes) + + +def rocm_aiter_group_quant_fusion_pass_on_test_model( + local_rank: int, + world_size: int, + test_model_cls: torch.nn.Module, + batch_size: int, + seq_len: int, + hidden_size: int, + dtype: torch.dtype, + enable_rms_norm_custom_op: bool, + use_triton_quant: bool, + monkeypatch: pytest.MonkeyPatch, +): + set_random_seed(0) + + device = torch.device(f"{DEVICE_TYPE}:{local_rank}") + torch.accelerator.set_device_index(device) + torch.set_default_device(device) + torch.set_default_dtype(dtype) + + update_environment_variables( + { + "RANK": str(local_rank), + "LOCAL_RANK": str(local_rank), + "WORLD_SIZE": str(world_size), + "MASTER_ADDR": "localhost", + "MASTER_PORT": "12345", + "VLLM_ROCM_USE_AITER": "1", + } + ) + rocm_aiter_ops.refresh_env_variables() + + init_distributed_environment() + + custom_ops = [] + if enable_rms_norm_custom_op: + custom_ops.append("+rms_norm") + # ``triton_per_token_group_quant_fp8`` is emitted by ``QuantFP8.forward_hip`` + # only when QuantFP8 is enabled as a custom op (and ``use_triton=True`` at + # the call site). The patterns in this PR are robust to both Triton and + # rocm_aiter forms; we always enable +quant_fp8 so the matcher's example + # trace finds the same form the test model uses. + custom_ops.append("+quant_fp8") + + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, custom_ops=custom_ops + ) + ) + vllm_config.compilation_config.pass_config = PassConfig( + fuse_allreduce_rms=True, eliminate_noops=True + ) + vllm_config.device_config = DeviceConfig(device=torch.device(DEVICE_TYPE)) + vllm_config.parallel_config.rank = local_rank + + model_name = "RedHatAI/Llama-3.2-1B-Instruct-FP8" + vllm_config.model_config = ModelConfig( + model=model_name, trust_remote_code=True, dtype=dtype, seed=42 + ) + with set_current_vllm_config(vllm_config): + initialize_model_parallel(tensor_model_parallel_size=world_size) + all_reduce_fusion_pass = RocmAiterAllReduceFusionPass(vllm_config) + noop_pass = NoOpEliminationPass(vllm_config) + func_pass = FixFunctionalizationPass(vllm_config) + cleanup_pass = PostCleanupPass(vllm_config) + + backend = TestBackend( + noop_pass, all_reduce_fusion_pass, func_pass, cleanup_pass + ) + + token_num = batch_size * seq_len + model = test_model_cls( + hidden_size, token_num, dtype=dtype, use_triton_quant=use_triton_quant + ) + + hidden_states = torch.randn((token_num, hidden_size), requires_grad=False) + + compiled_model = torch.compile(model, backend=backend) + compiled_model(hidden_states) + + results_unfused = model(hidden_states) + results_fused = compiled_model(hidden_states) + # The fused per-group AR+RMS+QUANT op is bit-equivalent to the unfused + # chain modulo the small AllReduce + RMSNorm reordering inside aiter. + # Per-group FP8 quant introduces step noise <=1 per group; use the + # same tolerance as the sibling FP8 static test. + torch.testing.assert_close(results_unfused, results_fused, atol=1e-2, rtol=1e-2) + + # Four pattern firings: norm[0] (no-add quant), norm[1] (add quant, + # single ``rms`` consumer), norm[2..3] (add quant + indexer fan-out). + assert all_reduce_fusion_pass.matched_count == 4, ( + f"{all_reduce_fusion_pass.matched_count=}" + ) + backend.check_before_ops(model.ops_in_model_before(), fully_replaced=False) + backend.check_after_ops(model.ops_in_model_after()) + del all_reduce_fusion_pass diff --git a/asian_ai/performance/vLLM/tests/compile/passes/distributed/test_sequence_parallelism.py b/asian_ai/performance/vLLM/tests/compile/passes/distributed/test_sequence_parallelism.py new file mode 100644 index 0000000000000000000000000000000000000000..0c02149d56ad1c1c42e612961514e599be612410 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/distributed/test_sequence_parallelism.py @@ -0,0 +1,343 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +import vllm.envs as envs +from tests.compile.backend import TestBackend +from tests.utils import TestFP8Layer, multi_gpu_test +from vllm.compilation.passes.fusion.rms_quant_fusion import RMSNormQuantFusionPass +from vllm.compilation.passes.fusion.sequence_parallelism import SequenceParallelismPass +from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass +from vllm.compilation.passes.utility.post_cleanup import PostCleanupPass +from vllm.compilation.passes.vllm_inductor_pass import VllmInductorPass +from vllm.config import ( + CompilationConfig, + CUDAGraphMode, + DeviceConfig, + ModelConfig, + PassConfig, + VllmConfig, + get_current_vllm_config, + set_current_vllm_config, +) +from vllm.config.utils import Range +from vllm.distributed import tensor_model_parallel_all_reduce +from vllm.distributed.parallel_state import ( + init_distributed_environment, + initialize_model_parallel, +) +from vllm.model_executor.layers.layernorm import RMSNorm +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + kFp8StaticTensorSym, +) +from vllm.platforms import current_platform +from vllm.utils.system_utils import update_environment_variables +from vllm.utils.torch_utils import set_random_seed + +DEVICE_TYPE = current_platform.device_type + +pytestmark = pytest.mark.skipif(not current_platform.is_cuda(), reason="Only test CUDA") + +FP8_DTYPE = current_platform.fp8_dtype() +prompts = [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + "The future of AI is", +] + + +class TestAllReduceRMSNormModel(torch.nn.Module): + def __init__(self, hidden_size=16, eps=1e-6): + super().__init__() + self.hidden_size = hidden_size + self.eps = eps + self.norm = [RMSNorm(hidden_size, eps) for i in range(4)] + self.w = [torch.rand(hidden_size, hidden_size) for _ in range(3)] + + def forward(self, x): + z = torch.relu(x) + x = resid = tensor_model_parallel_all_reduce(z) + y = self.norm[0](x) + + z2 = torch.mm(y, self.w[0]) + x2 = tensor_model_parallel_all_reduce(z2) + + y2, resid = self.norm[1](x2, resid) + + z3 = torch.mm(y2, self.w[1]) + x3 = tensor_model_parallel_all_reduce(z3) + + y3, resid = self.norm[2](x3, resid) + + z4 = torch.mm(y3, self.w[2]) + x4 = tensor_model_parallel_all_reduce(z4) + + y4, resid = self.norm[3](x4, resid) + return y4 + + def ops_in_model_before(self): + return [torch.ops.vllm.all_reduce.default] + + def ops_in_model_after(self): + return [ + torch.ops.vllm.all_gather.default, + torch.ops.vllm.reduce_scatter.default, + ] + + def ops_in_model(self): + return [ + torch.ops.vllm_ir.rms_norm, + torch.ops.vllm_ir.fused_add_rms_norm, + ] + + +class TestAllReduceRMSNormStaticQuantFP8Model(torch.nn.Module): + quant_key = kFp8StaticTensorSym + + def __init__(self, hidden_size=16, eps=1e-6): + super().__init__() + self.vllm_config = get_current_vllm_config() + self.hidden_size = hidden_size + self.eps = eps + self.norm = [RMSNorm(hidden_size, eps) for i in range(4)] + self.fp8_linear_layers = [ + TestFP8Layer( + weight_shape=(hidden_size, hidden_size), + activation_quant_key=self.quant_key, + weight_quant_key=self.quant_key, + input_dtype=self.vllm_config.model_config.dtype, + ) + for i in range(3) + ] + + def forward(self, hidden_states): + # avoid having graph input be an arg to a pattern directly + z = torch.relu(hidden_states) + x = resid = tensor_model_parallel_all_reduce(z) + y = self.norm[0](x) + + z2 = self.fp8_linear_layers[0](y) + + x2 = tensor_model_parallel_all_reduce(z2) + y2, resid = self.norm[1](x2, resid) + + z3 = self.fp8_linear_layers[1](y2) + + x3 = tensor_model_parallel_all_reduce(z3) + y3, resid = self.norm[2](x3, resid) # use resid here + + z4 = self.fp8_linear_layers[2](y3) + x4 = tensor_model_parallel_all_reduce(z4) + y4, resid = self.norm[3](x4, resid) # use resid here + return y4 + + def ops_in_model_after(self): + return [ + torch.ops.vllm.all_gather.default, + torch.ops.vllm.reduce_scatter.default, + ] + + def ops_in_model_before(self): + return [ + torch.ops.vllm.all_reduce.default, + ] + + def ops_in_model(self): + if self.vllm_config.compilation_config.pass_config.fuse_norm_quant: + return [torch.ops._C.fused_add_rms_norm_static_fp8_quant.default] + else: + quant_ops = ( + [torch.ops._C.static_scaled_fp8_quant.default] + if any(layer.is_quant_fp8_enabled() for layer in self.fp8_linear_layers) + else [torch.ops.aten.reciprocal] + ) + return [ + torch.ops.vllm_ir.rms_norm, + torch.ops.vllm_ir.fused_add_rms_norm, + *quant_ops, + ] + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize( + "test_model_cls, custom_ops", + [ + (TestAllReduceRMSNormModel, "+rms_norm"), + (TestAllReduceRMSNormModel, "-rms_norm"), + (TestAllReduceRMSNormStaticQuantFP8Model, "+rms_norm,+quant_fp8"), + (TestAllReduceRMSNormStaticQuantFP8Model, "+rms_norm,-quant_fp8"), + (TestAllReduceRMSNormStaticQuantFP8Model, "-rms_norm,+quant_fp8"), + (TestAllReduceRMSNormStaticQuantFP8Model, "-rms_norm,-quant_fp8"), + ], +) +@pytest.mark.parametrize("batch_size", [8]) +@pytest.mark.parametrize("seq_len", [16]) +@pytest.mark.parametrize("hidden_size", [16]) +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@pytest.mark.parametrize("fuse_norm_quant", [True, False]) +@pytest.mark.parametrize("dynamic", [False, True]) +@pytest.mark.skipif(envs.VLLM_TARGET_DEVICE not in ["cuda"], reason="Only test on CUDA") +def test_sequence_parallelism_pass( + test_model_cls: type[torch.nn.Module], + custom_ops: str, + batch_size: int, + seq_len: int, + hidden_size: int, + dtype: torch.dtype, + fuse_norm_quant: bool, + dynamic: bool, +): + num_processes = 2 + + def run_torch_spawn(fn, nprocs): + # need to use torch.mp.spawn otherwise will have problems with + # torch.distributed and cuda + torch.multiprocessing.spawn( + fn, + args=( + num_processes, + test_model_cls, + custom_ops, + batch_size, + seq_len, + hidden_size, + dtype, + fuse_norm_quant, + dynamic, + ), + nprocs=nprocs, + ) + + run_torch_spawn(sequence_parallelism_pass_on_test_model, num_processes) + + +def test_sequence_parallelism_pass_requires_full_graph_compilation(): + vllm_config = VllmConfig() + vllm_config.compilation_config.use_inductor_graph_partition = False + vllm_config.compilation_config.splitting_ops = [ + "vllm::unified_attention_with_output" + ] + + sequence_parallelism_pass = object.__new__(SequenceParallelismPass) + sequence_parallelism_pass.compilation_config = vllm_config.compilation_config + sequence_parallelism_pass.min_token_num = 1 + + with pytest.raises( + AssertionError, + match="SequenceParallelismPass requires full-graph compilation", + ): + sequence_parallelism_pass.is_applicable_for_range(Range(start=8, end=8)) + + +def sequence_parallelism_pass_on_test_model( + local_rank: int, + world_size: int, + test_model_cls: type[torch.nn.Module], + custom_ops: str, + batch_size: int, + seq_len: int, + hidden_size: int, + dtype: torch.dtype, + fuse_norm_quant: bool, + dynamic: bool, +): + set_random_seed(0) + + device = torch.device(f"{DEVICE_TYPE}:{local_rank}") + torch.accelerator.set_device_index(device) + torch.set_default_device(device) + torch.set_default_dtype(dtype) + + update_environment_variables( + { + "RANK": str(local_rank), + "LOCAL_RANK": str(local_rank), + "WORLD_SIZE": str(world_size), + "MASTER_ADDR": "localhost", + "MASTER_PORT": "12345", + } + ) + + # initialize distributed + init_distributed_environment() + + # configure vllm config for SequenceParallelismPass + custom_ops_list = custom_ops.split(",") if custom_ops else [] + compilation_config = CompilationConfig( + splitting_ops=[], # avoid automatic rms_norm enablement + cudagraph_mode=CUDAGraphMode.NONE, # avoid piecewise warnings + custom_ops=custom_ops_list, + pass_config=PassConfig( + enable_sp=True, + fuse_norm_quant=fuse_norm_quant, + eliminate_noops=True, + ), + ) # NoOp needed for fusion + device_config = DeviceConfig(device=torch.device(DEVICE_TYPE)) + + # this is a fake model name to construct the model config + # in the vllm_config, it's not really used. + model_name = "RedHatAI/Llama-3.2-1B-Instruct-FP8" + model_config = ModelConfig( + model=model_name, trust_remote_code=True, dtype=dtype, seed=42 + ) + + vllm_config = VllmConfig( + model_config=model_config, + device_config=device_config, + compilation_config=compilation_config, + ) + + with set_current_vllm_config(vllm_config): + initialize_model_parallel(tensor_model_parallel_size=world_size) + noop_pass = NoOpEliminationPass(vllm_config) + sequence_parallelism_pass = SequenceParallelismPass(vllm_config) + cleanup_pass = PostCleanupPass(vllm_config) + assert ( + sequence_parallelism_pass.compilation_config.splitting_ops + == vllm_config.compilation_config.splitting_ops + ) + assert ( + sequence_parallelism_pass.compilation_config.use_inductor_graph_partition + == vllm_config.compilation_config.use_inductor_graph_partition + ) + passes_for_backend: list[VllmInductorPass] = [ + noop_pass, + sequence_parallelism_pass, + ] + + if fuse_norm_quant: + fusion_pass = RMSNormQuantFusionPass(vllm_config) + passes_for_backend.append(fusion_pass) + + passes_for_backend.append(cleanup_pass) + + backend = TestBackend(*passes_for_backend) + + model = test_model_cls(hidden_size) + + hidden_states = torch.randn((batch_size * seq_len, hidden_size), dtype=dtype) + + if dynamic: + torch._dynamo.mark_dynamic(hidden_states, 0) + + compiled_model = torch.compile(model, backend=backend) + compiled_model(hidden_states) + + assert sequence_parallelism_pass.matched_count == 4 + + # In pre-nodes, all reduce should be there, + # reduce scatter and all gather should not + for op in model.ops_in_model_before(): + assert backend.op_count(op, before=True) == 4 + + # In post-nodes, reduce scatter and all gather should be there, + # all reduce should not + for op in model.ops_in_model_after(): + assert backend.op_count(op, before=False) == 4 + + for op in model.ops_in_model(): + assert backend.op_count(op, before=False) > 0 diff --git a/asian_ai/performance/vLLM/tests/compile/passes/ir/__init__.py b/asian_ai/performance/vLLM/tests/compile/passes/ir/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/compile/passes/ir/test_clone_cleanup.py b/asian_ai/performance/vLLM/tests/compile/passes/ir/test_clone_cleanup.py new file mode 100644 index 0000000000000000000000000000000000000000..98cfe461044b7f64da65fd8032736d6e0323e6ce --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/ir/test_clone_cleanup.py @@ -0,0 +1,431 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Comprehensive tests for UnsafeCloneEliminationPass. + +This test suite exercises all possible valid FX graph patterns involving clones: +1. Clone with no users (dead code) +2. Clone with read-only users +3. Clone with mutation users +4. Clone of graph input +5. Clone with original used after mutation +6. Clone chains +""" + +import pytest +import torch +from torch import fx +from torch.fx.experimental.proxy_tensor import make_fx + +from vllm.compilation.passes.fx_utils import find_op_nodes +from vllm.compilation.passes.inductor_pass import get_pass_context, pass_context +from vllm.compilation.passes.ir.clone_elimination import ( + UnsafeCloneEliminationPass, + user_writes_to_node, +) +from vllm.config import VllmConfig +from vllm.config.utils import Range + + +def count_clones(graph: fx.Graph) -> int: + """Count clone nodes in a graph.""" + return len(list(find_op_nodes(torch.ops.aten.clone.default, graph))) + + +@pytest.fixture(scope="function") +def clone_cleanup_pass(): + return UnsafeCloneEliminationPass(VllmConfig()) + + +@pytest.fixture(autouse=True) +def setup_pass_context(): + """Set up pass context for each test.""" + with pass_context(compile_range=Range(1, 8192)): + yield + + +class TestCloneCleanup: + """Test UnsafeCloneEliminationPass behavior on various graph patterns.""" + + def test_remove_clone_readonly_users(self, clone_cleanup_pass): + """Clone with only read-only users should be removed.""" + + def f(x: torch.Tensor) -> torch.Tensor: + x_clone = x.clone() + return x_clone + 1 + + inp = torch.randn(2, 3) + graph_module = make_fx(f)(inp) + assert count_clones(graph_module.graph) == 1 + + expected = graph_module(inp) + clone_cleanup_pass(graph_module.graph) + graph_module.recompile() + actual = graph_module(inp) + + assert count_clones(graph_module.graph) == 0 + torch.testing.assert_close(actual, expected) + + def test_keep_clone_with_mutation_and_original_used_after(self, clone_cleanup_pass): + """Clone must be kept if it's mutated AND original is used after mutation.""" + + def f(x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + x = x.relu() # not a graph param + x_clone = x.clone() + x_clone.add_(1) + return x, x_clone + + inp = torch.randn(2, 3) + graph_module = make_fx(f)(inp) + assert count_clones(graph_module.graph) == 1 + + expected = graph_module(inp) + clone_cleanup_pass(graph_module.graph) + graph_module.recompile() + actual = graph_module(inp) + + # Clone should be KEPT because original is used after mutation + assert count_clones(graph_module.graph) == 1 + torch.testing.assert_close(actual[0], expected[0]) + torch.testing.assert_close(actual[1], expected[1]) + + def test_remove_clone_with_mutation_no_original_use(self, clone_cleanup_pass): + """Clone can be removed if it's mutated but original is not used after.""" + + def f(x: torch.Tensor) -> torch.Tensor: + x = x.relu() # not a graph param + x_clone = x.clone() + x_clone.add_(1) + return x_clone + + inp = torch.randn(2, 3) + graph_module = make_fx(f)(inp) + assert count_clones(graph_module.graph) == 1 + + expected = graph_module(inp) + clone_cleanup_pass(graph_module.graph) + graph_module.recompile() + actual = graph_module(inp) + + assert count_clones(graph_module.graph) == 0 + torch.testing.assert_close(actual, expected) + + def test_clone_chain(self, clone_cleanup_pass): + """Test handling of clone chains: x -> clone1 -> clone2.""" + + def f(x: torch.Tensor) -> torch.Tensor: + x = x.relu() # not a graph param + x1 = x.clone() + x2 = x1.clone() + return x2 + 1 + + inp = torch.randn(2, 3) + graph_module = make_fx(f)(inp) + assert count_clones(graph_module.graph) == 2 + + expected = graph_module(inp) + clone_cleanup_pass(graph_module.graph) + graph_module.recompile() + actual = graph_module(inp) + + # Both clones should be removed + assert count_clones(graph_module.graph) == 0 + torch.testing.assert_close(actual, expected) + + def test_keep_clone_that_changes_layout(self, clone_cleanup_pass): + """Clone must be kept when it materializes a compact slice layout.""" + + def f(x: torch.Tensor) -> torch.Tensor: + return x[:, :3].contiguous() + + inp = torch.randn(4, 5) + graph_module = make_fx(f)(inp) + assert count_clones(graph_module.graph) == 1 + + expected = graph_module(inp) + clone_cleanup_pass(graph_module.graph) + graph_module.recompile() + actual = graph_module(inp) + + assert count_clones(graph_module.graph) == 1 + assert actual.stride() == expected.stride() == (3, 1) + torch.testing.assert_close(actual, expected) + + def test_multiple_clones_of_same_input(self, clone_cleanup_pass): + """Test multiple independent clones of the same input.""" + + def f(x: torch.Tensor) -> torch.Tensor: + x1 = x.clone() + x2 = x.clone() + return x1 + x2 + + inp = torch.randn(2, 3) + graph_module = make_fx(f)(inp) + assert count_clones(graph_module.graph) == 2 + + expected = graph_module(inp) + clone_cleanup_pass(graph_module.graph) + graph_module.recompile() + actual = graph_module(inp) + + # Both clones should be removed (only readonly uses) + assert count_clones(graph_module.graph) == 0 + torch.testing.assert_close(actual, expected) + + def test_no_clones_in_graph(self, clone_cleanup_pass): + """Test pass behavior when graph has no clones.""" + + def f(x: torch.Tensor) -> torch.Tensor: + return x + 1 + + inp = torch.randn(2, 3) + graph_module = make_fx(f)(inp) + assert count_clones(graph_module.graph) == 0 + + expected = graph_module(inp) + clone_cleanup_pass(graph_module.graph) + graph_module.recompile() + actual = graph_module(inp) + + assert count_clones(graph_module.graph) == 0 + torch.testing.assert_close(actual, expected) + + def test_multiple_passes(self, clone_cleanup_pass): + """Test running the pass multiple times (should be idempotent).""" + + def f(x: torch.Tensor) -> torch.Tensor: + x1 = x.clone() + return x1 + 1 + + inp = torch.randn(2, 3) + graph_module = make_fx(f)(inp) + assert count_clones(graph_module.graph) == 1 + + expected = graph_module(inp) + + clone_cleanup_pass(graph_module.graph) + assert count_clones(graph_module.graph) == 0 + graph_module.recompile() + actual = graph_module(inp) + torch.testing.assert_close(actual, expected) + + clone_cleanup_pass(graph_module.graph) + assert count_clones(graph_module.graph) == 0 + graph_module.recompile() + actual = graph_module(inp) + torch.testing.assert_close(actual, expected) + + def test_output_node_no_write(self): + """Output nodes never write to their inputs.""" + + def f(x: torch.Tensor) -> torch.Tensor: + return x + + graph_module = make_fx(f)(torch.randn(2, 3)) + x_node = [n for n in graph_module.graph.nodes if n.op == "placeholder"][0] + output_node = [n for n in graph_module.graph.nodes if n.op == "output"][0] + + assert not user_writes_to_node(output_node, x_node) + + def test_readonly_op_no_write(self): + """Readonly operations don't write to inputs.""" + + def f(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return x + y + + graph_module = make_fx(f)(torch.randn(2, 3), torch.randn(2, 3)) + placeholders = [n for n in graph_module.graph.nodes if n.op == "placeholder"] + add_node = [ + n + for n in graph_module.graph.nodes + if n.op == "call_function" and n.target == torch.ops.aten.add.Tensor + ][0] + + assert not user_writes_to_node(add_node, placeholders[0]) + assert not user_writes_to_node(add_node, placeholders[1]) + + def test_inplace_op_writes(self): + """Inplace operations write to first argument.""" + + def f(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + x.add_(y) + return x + + graph_module = make_fx(f)(torch.randn(2, 3), torch.randn(2, 3)) + placeholders = [n for n in graph_module.graph.nodes if n.op == "placeholder"] + add_node = [ + n + for n in graph_module.graph.nodes + if n.op == "call_function" and "add_" in str(n.target) + ][0] + + # add_ writes to first arg but not second + assert user_writes_to_node(add_node, placeholders[0]) + assert not user_writes_to_node(add_node, placeholders[1]) + + def test_copy_writes(self): + """copy_ operation writes to first argument.""" + + def f(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + x.copy_(y) + return x + + graph_module = make_fx(f)(torch.randn(2, 3), torch.randn(2, 3)) + placeholders = [n for n in graph_module.graph.nodes if n.op == "placeholder"] + copy_node = [ + n + for n in graph_module.graph.nodes + if n.op == "call_function" and "copy_" in str(n.target) + ][0] + + assert user_writes_to_node(copy_node, placeholders[0]) + assert not user_writes_to_node(copy_node, placeholders[1]) + + def test_auto_functionalized_not_a_write(self): + """auto_functionalized ops are follow-up uses, not writes.""" + from torch._higher_order_ops.auto_functionalize import auto_functionalized + + def f(x: torch.Tensor) -> torch.Tensor: + return x + + graph_module = make_fx(f)(torch.randn(2, 3)) + x_node = [n for n in graph_module.graph.nodes if n.op == "placeholder"][0] + + # Create an auto_functionalized node in the graph + with graph_module.graph.inserting_before(None): + af_node = graph_module.graph.call_function( + auto_functionalized, kwargs={"input": x_node} + ) + + # auto_functionalized should not be treated as a write + assert not user_writes_to_node(af_node, x_node) + + def test_higher_order_op_conservatively_writes(self): + """Other higher-order operators are conservatively treated as writes.""" + from torch._ops import HigherOrderOperator + + def f(x: torch.Tensor) -> torch.Tensor: + return x + + graph_module = make_fx(f)(torch.randn(2, 3)) + x_node = [n for n in graph_module.graph.nodes if n.op == "placeholder"][0] + + # Create a concrete higher-order operator subclass + class MockHigherOrderOp(HigherOrderOperator): + def __call__(self, *args, **kwargs): + return args[0] if args else None + + mock_hoo = MockHigherOrderOp("mock_higher_order_op") + + with graph_module.graph.inserting_before(None): + hoo_node = graph_module.graph.call_function(mock_hoo, args=(x_node,)) + + # Should be conservative and assume it could write + assert user_writes_to_node(hoo_node, x_node) + + +class TestCloneCleanupWithDonatedInputs: + """Test UnsafeCloneEliminationPass with donated input tracking via PassContext.""" + + @pytest.fixture(autouse=True) + def setup_pass_context(self): + """Set up pass context for each test.""" + with pass_context(compile_range=Range(1, 8192)): + yield + + def test_donated_input_clone_removed(self, clone_cleanup_pass): + """Clone of donated input should be removed.""" + + def f(x: torch.Tensor) -> torch.Tensor: + x_clone = x.clone() + x_clone.add_(1) + return x_clone + + inp = torch.randn(2, 3) + graph_module = make_fx(f)(inp) + assert count_clones(graph_module.graph) == 1 + + # Mark first parameter as donated + get_pass_context().donated_input_ids = {0} + + expected = graph_module(inp.clone()) + clone_cleanup_pass(graph_module.graph) + graph_module.recompile() + + # Clone should be removed since input is donated + assert count_clones(graph_module.graph) == 0 + + # Input can be mutated (donated) + inp_copy = inp.clone() + actual = graph_module(inp_copy) + torch.testing.assert_close(actual, expected) + + def test_non_donated_input_clone_kept(self, clone_cleanup_pass): + """Clone of non-donated input with mutation should be kept.""" + + def f(x: torch.Tensor, y: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + x_clone = x.clone() + x_clone.add_(1) + return x, x_clone + + inp_x = torch.randn(2, 3) + inp_y = torch.randn(2, 3) + graph_module = make_fx(f)(inp_x, inp_y) + assert count_clones(graph_module.graph) == 1 + + # No donated inputs + get_pass_context().donated_input_ids = set() + + expected = graph_module(inp_x.clone(), inp_y.clone()) + clone_cleanup_pass(graph_module.graph) + graph_module.recompile() + + # Clone should be kept since input is not donated and original is used + assert count_clones(graph_module.graph) == 1 + + # Verify inputs are not mutated + inp_x_before = inp_x.clone() + inp_y_before = inp_y.clone() + actual = graph_module(inp_x, inp_y) + torch.testing.assert_close( + inp_x, inp_x_before, msg="Input x should not be mutated" + ) + torch.testing.assert_close( + inp_y, inp_y_before, msg="Input y should not be mutated" + ) + torch.testing.assert_close(actual[0], expected[0]) + torch.testing.assert_close(actual[1], expected[1]) + + def test_mixed_donated_inputs(self, clone_cleanup_pass): + """Test with some inputs donated and some not.""" + + def f(x: torch.Tensor, y: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + x_clone = x.clone() + x_clone.add_(1) + y_clone = y.clone() + y_clone.add_(2) + return x_clone, y_clone + + inp_x = torch.randn(2, 3) + inp_y = torch.randn(2, 3) + graph_module = make_fx(f)(inp_x, inp_y) + assert count_clones(graph_module.graph) == 2 + + # Only x is donated + get_pass_context().donated_input_ids = {0} + + expected = graph_module(inp_x.clone(), inp_y.clone()) + clone_cleanup_pass(graph_module.graph) + graph_module.recompile() + + # x_clone removed (x is donated), y_clone kept (y is not donated) + assert count_clones(graph_module.graph) == 1 + + # Verify y is not mutated (x can be mutated since it's donated) + inp_y_before = inp_y.clone() + actual = graph_module(inp_x.clone(), inp_y) + torch.testing.assert_close( + inp_y, inp_y_before, msg="Input y should not be mutated" + ) + torch.testing.assert_close(actual[0], expected[0]) + torch.testing.assert_close(actual[1], expected[1]) diff --git a/asian_ai/performance/vLLM/tests/compile/passes/ir/test_inplace_functionalization.py b/asian_ai/performance/vLLM/tests/compile/passes/ir/test_inplace_functionalization.py new file mode 100644 index 0000000000000000000000000000000000000000..ab10549233e8d39340f3556988d6390d39033dca --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/ir/test_inplace_functionalization.py @@ -0,0 +1,465 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Tests for IR inplace functionalization pass integration. + +This test suite verifies that the inplace functionalization pass, lowering pass, +and clone cleanup pass work together correctly with donated buffer tracking. +""" + +from collections.abc import Callable + +import pytest +import torch +import torch._dynamo.exc +from torch import nn + +import vllm.kernels # noqa: F401 to register kernels +from vllm.compilation.passes.inductor_pass import InductorPass, get_pass_context +from vllm.compilation.passes.ir.clone_elimination import ( + UnsafeCloneEliminationPass, +) +from vllm.compilation.passes.ir.inplace_functionalization import ( + VllmIRInplaceFunctionalizationPass, +) +from vllm.compilation.passes.ir.lowering_pass import VllmIRLoweringPass +from vllm.config import VllmConfig +from vllm.ir import ops +from vllm.platforms import current_platform +from vllm.triton_utils import HAS_TRITON, tl, triton + +from ...backend import TestBackend + + +class StoreDonationInfoPass(InductorPass): + def __init__(self): + self.donated_input_ids_sets: list[set[int]] = [] + + def __call__(self, *args, **kwargs): + ctx = get_pass_context() + self.donated_input_ids_sets += [ctx.donated_input_ids] + + +class MaybeInplaceModel(nn.Module): + """Model using only maybe_inplace variants.""" + + def __init__(self, hidden_size=16): + super().__init__() + self.weight1 = nn.Parameter(torch.ones(hidden_size, dtype=torch.bfloat16)) + self.weight2 = nn.Parameter(torch.ones(hidden_size, dtype=torch.bfloat16)) + + def forward( + self, x: torch.Tensor, residual1: torch.Tensor, residual2: torch.Tensor + ): + # First maybe_inplace - x & residual1 are donated + x_normed1, residual_out1 = ops.fused_add_rms_norm.maybe_inplace( + x, residual1, self.weight1, 1e-5 + ) + # Second maybe_inplace - residual2 is donated + x_normed2, residual_out2 = ops.fused_add_rms_norm.maybe_inplace( + x_normed1, residual2, self.weight2, 1e-5 + ) + return x_normed2, residual_out1, residual_out2 + + +class FunctionalModel(nn.Module): + """Model using only functional (default) variants.""" + + def __init__(self, hidden_size=16): + super().__init__() + self.weight1 = nn.Parameter(torch.ones(hidden_size, dtype=torch.bfloat16)) + self.weight2 = nn.Parameter(torch.ones(hidden_size, dtype=torch.bfloat16)) + + def forward( + self, x: torch.Tensor, residual1: torch.Tensor, residual2: torch.Tensor + ): + # First functional - no donation + x_normed1, residual_out1 = ops.fused_add_rms_norm( + x, residual1, self.weight1, 1e-5 + ) + # Second functional - no donation + x_normed2, residual_out2 = ops.fused_add_rms_norm( + x_normed1, residual2, self.weight2, 1e-5 + ) + return x_normed2, residual_out1, residual_out2 + + +class MixedModel(nn.Module): + """Model mixing maybe_inplace and functional variants.""" + + def __init__(self, hidden_size=16): + super().__init__() + self.weight1 = nn.Parameter(torch.ones(hidden_size, dtype=torch.bfloat16)) + self.weight2 = nn.Parameter(torch.ones(hidden_size, dtype=torch.bfloat16)) + + def forward( + self, x: torch.Tensor, residual1: torch.Tensor, residual2: torch.Tensor + ): + # First maybe_inplace - x & residual1 are donated + x_normed1, residual_out1 = ops.fused_add_rms_norm.maybe_inplace( + x, residual1, self.weight1, 1e-5 + ) + # Second functional - no donation, x_normed1 must be preserved as it's returned + x_normed2, residual_out2 = ops.fused_add_rms_norm( + x_normed1, residual2, self.weight2, 1e-5 + ) + # Return both to prevent x_normed1 from being optimized away + return x_normed1, x_normed2, residual_out1, residual_out2 + + +class ModelWithTritonAfterMaybeInplace(nn.Module): + """ + Model using maybe_inplace followed by a Triton kernel. + Test clone elimination can handle Triton in the graph + """ + + def __init__(self, hidden_size=16): + super().__init__() + self.weight = nn.Parameter(torch.ones(hidden_size, dtype=torch.bfloat16)) + + @triton.jit + def _triton_add_kernel( + x_ptr, + y_ptr, + n_elements, + BLOCK_SIZE: tl.constexpr, + ): + pid = tl.program_id(axis=0) + block_start = pid * BLOCK_SIZE + offsets = block_start + tl.arange(0, BLOCK_SIZE) + mask = offsets < n_elements + x = tl.load(x_ptr + offsets, mask=mask) + y = x + 0.1 + tl.store(y_ptr + offsets, y, mask=mask) + + def triton_add(x: torch.Tensor) -> torch.Tensor: + """Simple Triton add kernel.""" + y = torch.empty_like(x) + n_elements = x.numel() + grid = (triton.cdiv(n_elements, 256),) + _triton_add_kernel[grid](x, y, n_elements, BLOCK_SIZE=256) + return y + + self.triton_add = triton_add + + def forward(self, x: torch.Tensor, residual: torch.Tensor, residual2: torch.Tensor): + x_normed, residual_out = ops.fused_add_rms_norm.maybe_inplace( + x, residual, self.weight, 1e-5 + ) + + x_processed = self.triton_add(x_normed) + + # x_processed does not need to be cloned, residual2 does + x_normed2, residual_out2 = ops.fused_add_rms_norm( + x_processed, residual2, self.weight, 1e-5 + ) + return x_normed2, residual_out2 + + +skipif_no_triton = pytest.mark.skipif(not HAS_TRITON, reason="Requires Triton") + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="Only test on cuda and rocm platform", +) +@pytest.mark.parametrize( + "model_class,expected_functionalized,expected_donated,expected_clones", + [ + # 2 inplace calls, all activations donated, all clones eliminated + (MaybeInplaceModel, 2, 3, 0), + # No inplace calls, no donations, 3 clones (one eliminated) + (FunctionalModel, 0, 0, 3), + # One inplace call, two donated activations, 2 clones + (MixedModel, 1, 2, 2), + # One inplace call, two donated, 1 clone remaining + pytest.param(ModelWithTritonAfterMaybeInplace, 1, 2, 1, marks=skipif_no_triton), + ], +) +def test_inplace_functionalization( + default_vllm_config: VllmConfig, + model_class, + expected_functionalized: int, + expected_clones: int, + expected_donated: int, +): + """Test inplace functionalization, lowering, and clone cleanup.""" + torch.set_default_device(current_platform.device_type) + + # Use vllm_c so inplace path is triggered + default_vllm_config.kernel_config.ir_op_priority.fused_add_rms_norm = [ + "vllm_c", + "native", + ] + + # Create passes in order they run during compilation + functionalization_pass = VllmIRInplaceFunctionalizationPass(default_vllm_config) + lowering_pass = VllmIRLoweringPass(default_vllm_config) + donated_info_pass = StoreDonationInfoPass() + cleanup_pass = UnsafeCloneEliminationPass(default_vllm_config) + + # Set up backend with pre-grad pass + backend = TestBackend(lowering_pass, donated_info_pass, cleanup_pass) + backend.inductor_config["pre_grad_custom_pass"] = functionalization_pass + + model = model_class() + x = torch.randn(8, 16, dtype=torch.bfloat16) + residual1 = torch.randn(8, 16, dtype=torch.bfloat16) + residual2 = torch.randn(8, 16, dtype=torch.bfloat16) + + with default_vllm_config.kernel_config.ir_op_priority.set_priority(): + # Reference output without optimization + ref_output = model(x.clone(), residual1.clone(), residual2.clone()) + + # Compile with inplace optimization + compiled_model = torch.compile(model, backend=backend, fullgraph=True) + output = compiled_model(x.clone(), residual1.clone(), residual2.clone()) + + # Verify correctness (relaxed tolerance for bfloat16) + for i in range(len(ref_output)): + torch.testing.assert_close(output[i], ref_output[i], rtol=1e-2, atol=1e-2) + + # Verify expected number of ops were functionalized + func_ops = functionalization_pass.functionalized_ops + assert len(func_ops) == int(bool(expected_functionalized)) + if expected_functionalized > 0: + assert "fused_add_rms_norm" in func_ops + assert func_ops["fused_add_rms_norm"] == expected_functionalized + + # Verify lowering happened (2 ops in all cases) + assert "fused_add_rms_norm" in lowering_pass.selected_impls + assert len(lowering_pass.selected_impls["fused_add_rms_norm"]) == 2 + assert all( + provider == "vllm_c" + for node, provider in lowering_pass.selected_impls["fused_add_rms_norm"].items() + ), lowering_pass.selected_impls + + # Verify correct number of donated IDs + assert len(donated_info_pass.donated_input_ids_sets) == 1 + assert len(donated_info_pass.donated_input_ids_sets[0]) == expected_donated + + # Verify expected number of clones after cleanup + actual_clones = backend.op_count(torch.ops.aten.clone.default, before=False) + assert actual_clones == expected_clones, ( + f"Expected {expected_clones} clones, got {actual_clones}:" + f"{backend.print_graphs()}" + ) + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="Only test on cuda and rocm platform", +) +def test_donated_buffer_context_propagation(default_vllm_config): + """Test that donated_input_ids propagates correctly through pass_context.""" + torch.set_default_device(current_platform.device_type) + + # Create a custom backend that inspects pass_context in cleanup pass + functionalization_pass = VllmIRInplaceFunctionalizationPass(default_vllm_config) + lowering_pass = VllmIRLoweringPass(default_vllm_config) + + donation_info_pass = StoreDonationInfoPass() + cleanup_pass = UnsafeCloneEliminationPass(default_vllm_config) + + backend = TestBackend(lowering_pass, donation_info_pass, cleanup_pass) + backend.inductor_config["pre_grad_custom_pass"] = functionalization_pass + + model = MaybeInplaceModel() + x = torch.randn(8, 16, dtype=torch.bfloat16) + residual1 = torch.randn(8, 16, dtype=torch.bfloat16) + residual2 = torch.randn(8, 16, dtype=torch.bfloat16) + + compiled_model = torch.compile(model, backend=backend, fullgraph=True) + compiled_model(x.clone(), residual1.clone(), residual2.clone()) + + donated_ids_seen = donation_info_pass.donated_input_ids_sets + # Verify donated_input_ids was set and propagated + assert len(donated_ids_seen) == 1 + # Should have donated inputs (exact indices depend on AOTAutograd) + assert len(donated_ids_seen[0]) == 3 + # All donated ids should be valid non-negative integers + for idx in donated_ids_seen[0]: + assert isinstance(idx, int) and idx >= 0, f"Invalid donated index: {idx}" + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="Only test on cuda and rocm platform", +) +def test_maybe_inplace_reuse_error(default_vllm_config): + """Test that reusing a donated activation input raises ValueError.""" + torch.set_default_device(current_platform.device_type) + + class ReuseModel(nn.Module): + """Model that incorrectly reuses a donated activation input.""" + + def __init__(self, hidden_size=16): + super().__init__() + self.weight = nn.Parameter(torch.ones(hidden_size, dtype=torch.bfloat16)) + + def forward(self, x: torch.Tensor, residual: torch.Tensor): + # x is donated to maybe_inplace + x_normed, residual_out = ops.fused_add_rms_norm.maybe_inplace( + x, residual, self.weight, 1e-5 + ) + # ERROR: x is used again after being donated + return x_normed + x # This should raise ValueError + + functionalization_pass = VllmIRInplaceFunctionalizationPass(default_vllm_config) + lowering_pass = VllmIRLoweringPass(default_vllm_config) + cleanup_pass = UnsafeCloneEliminationPass(default_vllm_config) + + backend = TestBackend(lowering_pass, cleanup_pass) + backend.inductor_config["pre_grad_custom_pass"] = functionalization_pass + + model = ReuseModel() + x = torch.randn(8, 16, dtype=torch.bfloat16) + residual = torch.randn(8, 16, dtype=torch.bfloat16) + + # Compilation should raise BackendCompilerFailed wrapping ValueError + with pytest.raises( + torch._dynamo.exc.BackendCompilerFailed, + match="is used again after the node", + ): + compiled_model = torch.compile(model, backend=backend, fullgraph=True) + compiled_model(x.clone(), residual.clone()) + + +# Piecewise compilation tests with graph splitting + + +@torch.library.custom_op("vllm::test_split_marker", mutates_args=()) +def test_split_marker(x: torch.Tensor) -> torch.Tensor: + """Identity op that marks a split point for piecewise compilation.""" + return x.clone() + + +@test_split_marker.register_fake +def _fake_split_marker(x: torch.Tensor) -> torch.Tensor: + return torch.empty_like(x) + + +class TransformerBlockWithSplits(nn.Module): + """Transformer block with explicit split points for piecewise compilation.""" + + def __init__(self, hidden_size=32, intermediate_size=128): + super().__init__() + self.hidden_size = hidden_size + self.intermediate_size = intermediate_size + + # Attention-like projection + self.attn_proj = nn.Linear( + hidden_size, hidden_size, bias=False, dtype=torch.bfloat16 + ) + + # Post-attention norm + self.post_attn_norm = nn.Parameter( + torch.ones(hidden_size, dtype=torch.bfloat16) + ) + + # MLP + self.gate_proj = nn.Linear( + hidden_size, intermediate_size, bias=False, dtype=torch.bfloat16 + ) + self.up_proj = nn.Linear( + hidden_size, intermediate_size, bias=False, dtype=torch.bfloat16 + ) + self.down_proj = nn.Linear( + intermediate_size, hidden_size, bias=False, dtype=torch.bfloat16 + ) + + # Post-MLP norm + self.post_mlp_norm = nn.Parameter(torch.ones(hidden_size, dtype=torch.bfloat16)) + + def forward(self, x: torch.Tensor): + # Attention block with residual + residual1 = x + attn_out = self.attn_proj(x) + + # Fused add + norm (maybe_inplace: residual1 is donated) + normed1, residual1 = ops.fused_add_rms_norm.maybe_inplace( + attn_out, residual1, self.post_attn_norm, 1e-5 + ) + + # Force a graph split here + normed1 = torch.ops.vllm.test_split_marker(normed1) + + # MLP block + gate = self.gate_proj(normed1) + up = self.up_proj(normed1) + mlp_out = self.down_proj(gate * torch.nn.functional.silu(up)) + + # Fused add + norm (maybe_inplace: residual1 is donated) + normed2, residual2 = ops.fused_add_rms_norm.maybe_inplace( + mlp_out, residual1, self.post_mlp_norm, 1e-5 + ) + + return normed2, residual2 + + +def with_dyn_arg(fn: Callable, arg_index: int, dim_index: int): + def inner(*args): + torch._dynamo.mark_dynamic(args[arg_index], dim_index) + return fn(*args) + + return inner + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="Only test on cuda and rocm platform", +) +def test_piecewise_compilation_with_donated_buffers(monkeypatch, fresh_vllm_cache): + """ + Test piecewise compilation with donated buffers across graph splits. + Utilizes a custom splitting op. Uses fresh cache to avoid compilation caching. + """ + torch.set_default_device(current_platform.device_type) + + # Disable compilation cache to avoid serialization issues + monkeypatch.setenv("VLLM_DISABLE_COMPILE_CACHE", "1") + + from vllm.compilation.backends import VllmBackend + from vllm.config import CompilationConfig, VllmConfig + + # Create config with custom splitting op + store_donation_info = StoreDonationInfoPass() + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + custom_ops=["all"], + splitting_ops=["vllm::test_split_marker"], + inductor_compile_config={"post_grad_custom_post_pass": store_donation_info}, + ) + ) + + backend = VllmBackend(vllm_config) + + model = TransformerBlockWithSplits() + x = torch.randn(8, 32, dtype=torch.bfloat16) + + # Reference output + ref_output = with_dyn_arg(model, 0, 0)(x.clone()) + + # Compile with piecewise compilation (graph will split at split_marker) + compiled_model = torch.compile(model, backend=backend, fullgraph=False) + output = with_dyn_arg(compiled_model, 0, 0)(x.clone()) + + # Verify correctness (relaxed tolerance for bfloat16) + torch.testing.assert_close(output[0], ref_output[0], rtol=1e-2, atol=1e-2) + torch.testing.assert_close(output[1], ref_output[1], rtol=1e-2, atol=1e-2) + + # Verify the model was split into multiple submodules + assert hasattr(backend, "split_gm"), "Backend should have split graph module" + + # Should have at least 2 submodules (split by test_split_marker op) + submodules = list(backend.split_gm.named_children()) + num_submodules = len(submodules) + assert num_submodules >= 2, ( + f"Expected at least 2 submodules (split), got {num_submodules}" + ) + + # Check that donation info was propagated correctly + donated_inputs_sets = store_donation_info.donated_input_ids_sets + assert len(donated_inputs_sets) == 2 + assert len(donated_inputs_sets[0]) == 1 + assert len(donated_inputs_sets[1]) == 1 diff --git a/asian_ai/performance/vLLM/tests/compile/passes/ir/test_lowering.py b/asian_ai/performance/vLLM/tests/compile/passes/ir/test_lowering.py new file mode 100644 index 0000000000000000000000000000000000000000..8b40f3197bd0eba42ddc6dd79afa4380f7c8f85c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/ir/test_lowering.py @@ -0,0 +1,69 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch +from torch import nn + +import vllm.kernels # noqa: F401 to register kernels +from vllm import ir +from vllm.compilation.passes.ir.lowering_pass import ( + VllmIRLoweringPass, +) +from vllm.config import get_current_vllm_config +from vllm.ir import ops +from vllm.platforms import current_platform + +from ...backend import TestBackend + + +class Model(nn.Module): + def __init__(self, hidden_size=16, *args, **kwargs): + super().__init__(*args, **kwargs) + self.hidden_size = hidden_size + self.weight = torch.ones(hidden_size, dtype=torch.bfloat16) + + def forward(self, x): + x1 = x + 4.0 + x2 = ops.rms_norm(x1, self.weight, 1e-5) + x3 = x2 * 5.0 + # no weight + x4 = ops.rms_norm(x3, None, 1e-5) + x5 = x4 / 2.0 + # dispatch to native due to variance_size parameter + x6 = ops.rms_norm(x5, self.weight, 1e-5, self.hidden_size // 2) + return x6 + 3.0 + + +@pytest.mark.parametrize("rms_provider", ops.rms_norm.supported_providers()) +def test_lowering_rms_norm(rms_provider, default_vllm_config): + torch.set_default_device(current_platform.device_type) + + lowering_pass = VllmIRLoweringPass(get_current_vllm_config()) + backend = TestBackend(lowering_pass) + backend_unlowered = TestBackend() + + model = Model() + x = torch.randn(8, 16, dtype=torch.bfloat16) + with ( + ops.rms_norm.set_priority([rms_provider, "native"]), + ir.enable_torch_wrap(True), + ): + compiled_model = torch.compile(model, backend=backend, fullgraph=True) + compiled_unlowered_model = torch.compile( + model, backend=backend_unlowered, fullgraph=True + ) + output = compiled_model(x) + output_unlowered = compiled_unlowered_model(x) + + selected = lowering_pass.selected_impls["rms_norm"] + assert len(selected) == 3 + assert selected["rms_norm"] == rms_provider + assert selected["rms_norm_1"] == rms_provider + assert selected["rms_norm_2"] == "native" + + # Compiled function guards on global value, avoid recompilation + with ir.enable_torch_wrap(True): + output2 = compiled_model(x) + + torch.testing.assert_close(output_unlowered, output) + torch.testing.assert_close(output_unlowered, output2) diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_double_aiter_rms_quant_fusion.py b/asian_ai/performance/vLLM/tests/compile/passes/test_double_aiter_rms_quant_fusion.py new file mode 100644 index 0000000000000000000000000000000000000000..9f4f4004a6c6897c7ae30f38096851bb1c3f09cb --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_double_aiter_rms_quant_fusion.py @@ -0,0 +1,165 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Unit tests for the DoubleQuant fan-out variants registered by +``RocmAiterRMSNormQuantFusionPass``. + +Both variants target a 1-to-2 fan-out where one ``rms_norm`` output feeds +two distinct ``rocm_aiter_group_fp8_quant`` consumers and rewrite it into +two independent fused ``rms_norm + group_fp8_quant`` ops: + +* ``DoubleAiterRMSFp8GroupQuantPattern`` matches the un-viewed shape + (e.g. Kimi-K2.5 / DSR1). +* ``DoubleAiterRMSFp8GroupQuantViewPattern`` (this PR) is the view-tolerant + sibling that additionally matches the + ``rms_norm -> view -> group_fp8_quant`` shape that DSv3.2's MLA indexer + q_c norm exposes through ``Fp8BlockScaledMMLinearKernel.apply_weights``'s + 2D-flatten boilerplate. +""" + +import pytest +import torch + +import vllm.config +from tests.compile.backend import TestBackend +from vllm._aiter_ops import rocm_aiter_ops +from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass +from vllm.compilation.passes.utility.post_cleanup import PostCleanupPass +from vllm.config import ( + CompilationConfig, + CompilationMode, + ModelConfig, + PassConfig, + VllmConfig, +) + +EPS = 1e-5 +HIDDEN_SIZE = 256 +GROUP_SIZE = 128 + + +class _NoViewDoubleQuantModel(torch.nn.Module): + """``rms_norm -> 2x group_fp8_quant`` fan-out (Kimi-K2.5 / DSR1 shape).""" + + def __init__(self) -> None: + super().__init__() + self.weight = torch.nn.Parameter(torch.ones(HIDDEN_SIZE, dtype=torch.bfloat16)) + + def forward( + self, x: torch.Tensor + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + # avoid graph input being a direct arg to a matched pattern node + x = torch.relu(x) + rms = torch.ops.vllm_ir.rms_norm(x, self.weight, EPS) + q1, s1 = torch.ops.vllm.rocm_aiter_group_fp8_quant.default(rms, GROUP_SIZE) + q2, s2 = torch.ops.vllm.rocm_aiter_group_fp8_quant.default(rms, GROUP_SIZE) + return q1, s1, q2, s2 + + +class _ViewDoubleQuantModel(torch.nn.Module): + """``rms_norm -> view -> 2x group_fp8_quant`` fan-out (DSv3.2 shape). + + Reproduces the FX-graph shape produced by ``Fp8BlockScaledMMLinearKernel``'s + 2D-flatten before the FP8 group quant op. + """ + + def __init__(self) -> None: + super().__init__() + self.weight = torch.nn.Parameter(torch.ones(HIDDEN_SIZE, dtype=torch.bfloat16)) + + def forward( + self, x: torch.Tensor + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + x = torch.relu(x) + rms = torch.ops.vllm_ir.rms_norm(x, self.weight, EPS) + view = rms.view(-1, rms.shape[-1]) + q1, s1 = torch.ops.vllm.rocm_aiter_group_fp8_quant.default(view, GROUP_SIZE) + q2, s2 = torch.ops.vllm.rocm_aiter_group_fp8_quant.default(view, GROUP_SIZE) + return q1, s1, q2, s2 + + +@pytest.mark.parametrize( + "model_cls", + [_NoViewDoubleQuantModel, _ViewDoubleQuantModel], + ids=["no_view", "with_view"], +) +@pytest.mark.skip( + reason="Skipping for now because pytorch compiler removes one the two quant ops" +) +def test_double_aiter_rms_fp8_group_quant_fusion( + model_cls: type[torch.nn.Module], + monkeypatch: pytest.MonkeyPatch, +) -> None: + """ + Both fan-out shapes (with and without an intermediate view) must fuse + into ``rocm_aiter_rmsnorm_fp8_group_quant``: the no-view shape via + ``DoubleAiterRMSFp8GroupQuantPattern`` and the viewed shape via the + new ``DoubleAiterRMSFp8GroupQuantViewPattern`` sibling. + + A failure on the ``with_view`` parametrization is a regression on the + DSv3.2 q_c norm path that this PR's view-tolerant pattern is intended + to cover. + """ + torch._dynamo.reset() + + vllm_config = VllmConfig( + model_config=ModelConfig(dtype=torch.bfloat16), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=["+rms_norm", "+quant_fp8"], + pass_config=PassConfig( + fuse_norm_quant=True, + eliminate_noops=True, + ), + ), + ) + + with vllm.config.set_current_vllm_config(vllm_config), monkeypatch.context() as m: + from vllm.compilation.passes.fusion.rocm_aiter_fusion import ( + RocmAiterRMSNormQuantFusionPass, + ) + + torch.set_default_device("cuda") + torch.set_default_dtype(torch.bfloat16) + torch.manual_seed(0) + + m.setenv("VLLM_ROCM_USE_AITER", "1") + rocm_aiter_ops.refresh_env_variables() + + fusion_pass = RocmAiterRMSNormQuantFusionPass(vllm_config) + passes = [ + NoOpEliminationPass(vllm_config), + fusion_pass, + PostCleanupPass(vllm_config), + ] + backend = TestBackend(*passes) + model = model_cls() + + x = torch.randn(8, HIDDEN_SIZE) + torch._dynamo.mark_dynamic(x, 0) + + outputs_unfused = model(x) + model_fused = torch.compile(model, backend=backend) + outputs_fused = model_fused(x) + + # Both consumers must be rewritten into the fused op (one + # ``register_replacement`` rewrite covers the whole 1-to-2 fan-out). + assert fusion_pass.matched_count == 1, ( + f"Expected the {model_cls.__name__} fan-out to fuse via the " + f"DoubleQuant pattern (matched_count == 1), got " + f"{fusion_pass.matched_count}" + ) + + fused_op = rocm_aiter_ops.get_rmsnorm_group_fused_quant_op() + backend.check_after_ops([fused_op]) + + # Numerical parity sanity-check: the fused pair must match the + # unfused pair on FP8 outputs (exact byte-equality is the goal, + # but allow a tiny tolerance for any residual numeric noise). + for fused_t, unfused_t in zip(outputs_fused, outputs_unfused): + torch.testing.assert_close( + fused_t.to(torch.float32), + unfused_t.to(torch.float32), + atol=1e-2, + rtol=1e-2, + ) diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_functionalization.py b/asian_ai/performance/vLLM/tests/compile/passes/test_functionalization.py new file mode 100644 index 0000000000000000000000000000000000000000..131939b81dcd74eff6fdeff504eb55d9e915d29a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_functionalization.py @@ -0,0 +1,340 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import copy + +import pytest +import torch + +from tests.compile.backend import TestBackend +from tests.utils import TestFP8Layer +from vllm.compilation.passes.fusion.act_quant_fusion import ( + ActivationQuantFusionPass, +) +from vllm.compilation.passes.fusion.rms_quant_fusion import RMSNormQuantFusionPass +from vllm.compilation.passes.fx_utils import find_auto_fn, find_auto_fn_maybe, is_func +from vllm.compilation.passes.utility.fix_functionalization import ( + FixFunctionalizationPass, +) +from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass +from vllm.compilation.passes.utility.post_cleanup import PostCleanupPass +from vllm.config import ( + CompilationConfig, + ModelConfig, + PassConfig, + VllmConfig, + get_current_vllm_config, + set_current_vllm_config, +) +from vllm.model_executor.layers.activation import SiluAndMul +from vllm.model_executor.layers.layernorm import RMSNorm +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + kFp8StaticTensorSym, +) +from vllm.model_executor.layers.rotary_embedding import get_rope +from vllm.platforms import current_platform +from vllm.utils.torch_utils import direct_register_custom_op + +TEST_FP8 = current_platform.supports_fp8() +FP8_DTYPE = current_platform.fp8_dtype() + + +class TestSiluMul(torch.nn.Module): + quant_key = kFp8StaticTensorSym + + def __init__(self, hidden_size: int = 128): + super().__init__() + self.silu_and_mul = SiluAndMul() + if TEST_FP8: + self.fp8_linear = TestFP8Layer( + weight_shape=(hidden_size, hidden_size), + activation_quant_key=self.quant_key, + weight_quant_key=self.quant_key, + input_dtype=get_current_vllm_config().model_config.dtype, + ) + + def forward(self, x): + y = self.silu_and_mul(x) + if TEST_FP8: + return self.fp8_linear(y) + else: + return y + + def example_inputs(self, num_tokens=32, hidden_size=128): + return (torch.rand(num_tokens, hidden_size * 2),) + + def ops_in_model(self, do_fusion): + if TEST_FP8 and do_fusion: + return [torch.ops._C.silu_and_mul_quant.default] + else: + return [torch.ops._C.silu_and_mul.default] + + def ops_not_in_model(self): + return [] + + +class TestFusedAddRMSNorm(torch.nn.Module): + quant_key = kFp8StaticTensorSym + + def __init__(self, hidden_size=16, intermediate_size=32): + super().__init__() + self.hidden_size = hidden_size + self.intermediate_size = intermediate_size + + self.gate_proj = torch.nn.Parameter( + torch.empty((intermediate_size, hidden_size)) + ) + self.norm = RMSNorm(intermediate_size, 1e-05) + self.norm.weight = torch.nn.Parameter(torch.ones(intermediate_size)) + + torch.nn.init.normal_(self.gate_proj, std=0.02) + + if TEST_FP8: + self.fp8_linear = TestFP8Layer( + weight_shape=(hidden_size, intermediate_size), + activation_quant_key=self.quant_key, + weight_quant_key=self.quant_key, + input_dtype=get_current_vllm_config().model_config.dtype, + ) + + def forward(self, hidden_states, residual): + # Reshape input + view = hidden_states.reshape(-1, self.hidden_size) + + # matrix multiplication + permute = self.gate_proj.permute(1, 0) + mm = torch.mm(view, permute) + + # layer normalization + norm_output, residual_output = self.norm(mm, residual) + + if TEST_FP8: + # scaled_mm with static input quantization + fp8_linear_result = self.fp8_linear(norm_output) + + return fp8_linear_result, residual_output + + else: + return norm_output, residual_output + + def example_inputs(self, batch_size=8, seq_len=16): + hidden_states = torch.randn((batch_size * seq_len, self.hidden_size)) + residual = torch.randn((batch_size * seq_len, self.intermediate_size)) + return (hidden_states, residual) + + def ops_in_model(self, do_fusion): + if TEST_FP8 and do_fusion: + return [torch.ops._C.fused_add_rms_norm_static_fp8_quant.default] + else: + return [] + + def ops_not_in_model(self): + return [] + + +class TestRotaryEmbedding(torch.nn.Module): + def __init__(self, head_dim=64, max_position=2048, base=10000): + super().__init__() + self.head_dim = head_dim + + self.rotary_emb = get_rope( + self.head_dim, + max_position=max_position, + rope_parameters={"rope_type": "default", "rope_theta": base}, + ) + + def forward(self, positions, q, k): + q_rotated, k_rotated = self.rotary_emb(positions, q, k) + return q_rotated, k_rotated + + def example_inputs(self, num_tokens=32, head_dim=64): + positions = torch.arange(num_tokens, dtype=torch.long) + q = torch.randn(num_tokens, head_dim) + k = torch.randn(num_tokens, head_dim) + return (positions, q, k) + + def ops_in_model(self, do_fusion): + return [torch.ops._C.rotary_embedding.default] + + def ops_not_in_model(self): + return [] + + +class TestRotaryEmbeddingSliceScatter(torch.nn.Module): + def __init__(self, head_dim=64, num_heads=4, max_position=2048, base=10000): + super().__init__() + self.head_dim = head_dim + self.num_heads = num_heads + self.hidden_size = head_dim * num_heads + + self.qkv_proj = torch.nn.Linear( + self.hidden_size, self.hidden_size * 3, bias=False + ) + + self.rotary_emb = get_rope( + self.head_dim, + max_position=max_position, + rope_parameters={"rope_type": "default", "rope_theta": base}, + ) + + def forward(self, positions, hidden_states): + # Simulate the pattern: mm -> split_with_sizes -> rotary_embedding + # -> slice_scatter -> split_with_sizes + + qkv = self.qkv_proj(hidden_states) + split_sizes = [self.hidden_size, self.hidden_size, self.hidden_size] + q, k, v = torch.split(qkv, split_sizes, dim=-1) + + q_rotated, k_rotated = self.rotary_emb(positions, q, k) + + qkv_updated = torch.cat([q_rotated, k_rotated, v], dim=-1) + return qkv_updated + + def example_inputs(self, num_tokens=32, head_dim=64, num_heads=4): + hidden_size = head_dim * num_heads + positions = torch.arange(num_tokens, dtype=torch.long) + hidden_states = torch.randn(num_tokens, hidden_size) + return (positions, hidden_states) + + def ops_in_model(self, do_fusion): + return [torch.ops._C.rotary_embedding.default] + + def ops_not_in_model(self): + return [torch.ops.aten.slice_scatter.default] + + +class TestFunctionWithMutatedArgsAndReturn(torch.nn.Module): + OP_REGISTERED = False + + def __init__(self): + super().__init__() + self.register_test_custom_op() + + @classmethod + def register_test_custom_op(cls): + if not cls.OP_REGISTERED: + + def function_with_mutated_args_and_return_impl( + x: torch.Tensor, + ) -> torch.Tensor: + ret = x + 1 + x.add_(2) + return ret + + def function_with_mutated_args_and_return_fake( + x: torch.Tensor, + ) -> torch.Tensor: + return torch.empty_like(x) + + direct_register_custom_op( + op_name="function_with_mutated_args_and_return", + op_func=function_with_mutated_args_and_return_impl, + mutates_args=["x"], + fake_impl=function_with_mutated_args_and_return_fake, + ) + + cls.OP_REGISTERED = True + + def forward(self, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + # Clone x to avoid mutating the original tensor + ret = torch.ops.vllm.function_with_mutated_args_and_return(x) + return x, ret + + def example_inputs(self, num_tokens=32): + hidden_states = torch.randn(num_tokens) + return (hidden_states,) + + def ops_in_model(self, do_fusion): + return [torch.ops.vllm.function_with_mutated_args_and_return.default] + + def ops_not_in_model(self): + return [] + + +MODELS_AND_DO_FUSION = { + TestSiluMul: [True, False], + TestFusedAddRMSNorm: [True, False], + TestRotaryEmbedding: [False], + TestRotaryEmbeddingSliceScatter: [False], + TestFunctionWithMutatedArgsAndReturn: [False], +} + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@pytest.mark.parametrize( + "model_class, do_fusion", + [ + (model_class, do_fusion) + for model_class, fusions in MODELS_AND_DO_FUSION.items() + for do_fusion in fusions + ], +) +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="Only test on cuda and rocm platform", +) +def test_fix_functionalization( + model_class: torch.nn.Module, do_fusion: bool, dtype: torch.dtype +): + torch.set_default_device("cuda") + torch.set_default_dtype(dtype) + torch.manual_seed(0) + + vllm_config = VllmConfig( + model_config=ModelConfig(dtype=dtype), + compilation_config=CompilationConfig( + custom_ops=["all"], + pass_config=PassConfig( + fuse_norm_quant=do_fusion, + fuse_act_quant=do_fusion, + eliminate_noops=True, + ), + ), + ) + + with set_current_vllm_config(vllm_config): + assert RMSNorm.enabled() + noop_pass = NoOpEliminationPass(vllm_config) + fusion_pass = RMSNormQuantFusionPass(vllm_config) + cleanup_pass = PostCleanupPass(vllm_config) + act_quant_fusion_pass = ActivationQuantFusionPass(vllm_config) + + passes = ( + [noop_pass, fusion_pass, act_quant_fusion_pass, cleanup_pass] + if do_fusion + else [noop_pass, cleanup_pass] + ) + func_pass = FixFunctionalizationPass(vllm_config) + + backend_func = TestBackend(*passes, func_pass) + backend_no_func = TestBackend(*passes) + + model = model_class() + inputs_func = model.example_inputs() + inputs_no_func = copy.deepcopy(inputs_func) + model_func = copy.deepcopy(model) + model_no_func = copy.deepcopy(model) + model_func = torch.compile(model_func, backend=backend_func) + model_no_func = torch.compile(model_no_func, backend=backend_no_func) + + # deepcopy inputs to prevent potential in place mutation + outputs_func = model_func(*copy.deepcopy(inputs_func)) + outputs_no_func = model_no_func(*copy.deepcopy(inputs_no_func)) + torch.testing.assert_close(outputs_func, outputs_no_func) + + # check if the functionalization pass is applied + for op in model.ops_in_model(do_fusion): + find_auto_fn(backend_no_func.graph_post_pass.nodes, op) + assert find_auto_fn_maybe(backend_func.graph_post_pass.nodes, op) is None + + # make sure the ops were all de-functionalized + found = dict() + for node in backend_func.graph_post_pass.nodes: + for op in model.ops_in_model(do_fusion): + if is_func(node, op): + found[op] = True + for op in model.ops_not_in_model(): + if is_func(node, op): + found[op] = True + assert all(found[op] for op in model.ops_in_model(do_fusion)) + assert all(not found.get(op) for op in model.ops_not_in_model()) diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_fuse_act_padding.py b/asian_ai/performance/vLLM/tests/compile/passes/test_fuse_act_padding.py new file mode 100644 index 0000000000000000000000000000000000000000..bef0a8c66de72cc91e930794bb6f0d766d133ee9 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_fuse_act_padding.py @@ -0,0 +1,128 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import pytest +import torch + +import vllm.config +from tests.compile.backend import TestBackend +from vllm._aiter_ops import rocm_aiter_ops +from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass +from vllm.compilation.passes.utility.post_cleanup import PostCleanupPass +from vllm.config import ( + CompilationConfig, + CompilationMode, + ModelConfig, + PassConfig, + VllmConfig, +) +from vllm.model_executor.layers.layernorm import RMSNorm +from vllm.model_executor.layers.utils import rocm_unquantized_gemm + + +class TestModel(torch.nn.Module): + def __init__( + self, + num_layers: int, + hidden_size: int, + num_local_experts: int, + x_pad_to_multiple: int, + ): + super().__init__() + self.num_layers = num_layers + self.hidden_size = hidden_size + self.x_pad_to_multiple = x_pad_to_multiple + self.pad_dim = x_pad_to_multiple - (hidden_size % x_pad_to_multiple) + + self.norm = [RMSNorm(hidden_size, eps=1e-5) for _ in range(num_layers)] + self.router = [ + torch.nn.Linear(hidden_size, num_local_experts) for _ in range(4) + ] + + def forward(self, x): + # avoid having graph input be an arg to a pattern directly + x = resid = torch.relu(x) + all_router_logits = [] + for layer in range(self.num_layers): + x = x[:, : self.hidden_size] + x, resid = self.norm[layer](x, resid) + router_logits = rocm_unquantized_gemm( + self, x, self.router[layer].weight, self.router[layer].bias + ) + x = torch.nn.functional.pad( + x, (0, self.pad_dim), mode="constant", value=0.0 + ) + all_router_logits.append(router_logits) + + return x, resid, *all_router_logits + + def ops_in_model_before(self): + return [ + torch.ops.vllm_ir.fused_add_rms_norm, + torch.ops.aten.constant_pad_nd, + ] + + def ops_in_model_after(self): + return [rocm_aiter_ops.get_triton_add_rmsnorm_pad_op()] + + +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("num_layers", [3]) +@pytest.mark.parametrize("hidden_size", [2880]) +@pytest.mark.parametrize("num_local_experts", [128]) +@pytest.mark.parametrize("x_pad_to_multiple", [256]) +@pytest.mark.skip( + reason="Skipping for now because of the accuracy issue. See: https://github.com/ROCm/aiter/issues/2614" +) +def test_fuse_act_padding( + dtype: torch.dtype, + num_layers: int, + hidden_size: int, + num_local_experts: int, + x_pad_to_multiple: int, + monkeypatch: pytest.MonkeyPatch, +): + vllm_config = VllmConfig( + model_config=ModelConfig(dtype=dtype), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=["+rms_norm"], + pass_config=PassConfig(fuse_act_padding=True, eliminate_noops=True), + ), + ) + + with vllm.config.set_current_vllm_config(vllm_config), monkeypatch.context() as m: + from vllm.compilation.passes.fusion.rocm_aiter_fusion import ( + RocmAiterTritonAddRMSNormPadFusionPass, + ) + + torch.set_default_device("cuda") + torch.set_default_dtype(dtype) + torch.manual_seed(1) + + m.setenv("VLLM_ROCM_USE_AITER", "1") + rocm_aiter_ops.refresh_env_variables() + + fusion_pass = RocmAiterTritonAddRMSNormPadFusionPass(vllm_config) + passes = [ + NoOpEliminationPass(vllm_config), + fusion_pass, + PostCleanupPass(vllm_config), + ] + backend = TestBackend(*passes) + model = TestModel(num_layers, hidden_size, num_local_experts, x_pad_to_multiple) + + x = torch.rand(1, hidden_size) + torch._dynamo.mark_dynamic(x, 0) + outputs_unfused = model(x) + + model_fused = torch.compile(model, backend=backend) + outputs_fused = model_fused(x) + + torch.testing.assert_close(outputs_unfused, outputs_fused) + + assert fusion_pass.matched_count == num_layers + + backend.check_before_ops(model.ops_in_model_before()) + backend.check_after_ops(model.ops_in_model_after()) diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_fuse_mla_dual_rms_norm.py b/asian_ai/performance/vLLM/tests/compile/passes/test_fuse_mla_dual_rms_norm.py new file mode 100644 index 0000000000000000000000000000000000000000..fc477abb10495442f12b0c720a7d1e5d7cdd26a4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_fuse_mla_dual_rms_norm.py @@ -0,0 +1,290 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Unit test for the MLADualRMSNormFusionPass. + +The pass fuses paired q/kv RMS norms in MLA attention into a single +fused_mla_dual_rms_norm op backed by AITER's fused_qk_rmsnorm kernel. +""" + +import pytest +import torch + +import vllm.config +from tests.compile.backend import TestBackend +from vllm._aiter_ops import ( + is_aiter_found_and_supported, + rocm_aiter_ops, +) +from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass +from vllm.compilation.passes.utility.post_cleanup import PostCleanupPass +from vllm.config import ( + CompilationConfig, + CompilationMode, + ModelConfig, + PassConfig, + VllmConfig, +) +from vllm.model_executor.layers.layernorm import RMSNorm +from vllm.platforms import current_platform + +# MLA attention geometry for DeepSeek-V3 / Kimi-K2 +Q_DIM = 1536 +KV_C_DIM = 512 +K_PE_DIM = 64 +EPS = 1e-6 + +FP8_DTYPE = current_platform.fp8_dtype() + + +class MLADualRMSNormTestModel(torch.nn.Module): + """ + Minimal model reproducing the MLA dual RMS norm pattern: + linear -> split([q_dim, kv_dim]) + +-- q_c (getitem 0) -> rms_norm(q_w, eps) -> linear + +-- kv_lora (getitem 1) -> split([kv_c_dim, k_pe_dim]) + +-- kv_c (getitem 0) -> rms_norm(kv_w, eps) + +-- k_pe + """ + + def __init__( + self, + hidden_size: int, + q_dim: int = Q_DIM, + kv_c_dim: int = KV_C_DIM, + k_pe_dim: int = K_PE_DIM, + eps: float = EPS, + ): + super().__init__() + self.q_dim = q_dim + self.kv_dim = kv_c_dim + k_pe_dim + self.kv_c_dim = kv_c_dim + self.k_pe_dim = k_pe_dim + + self.proj = torch.nn.Linear(hidden_size, q_dim + self.kv_dim, bias=False) + self.q_norm = RMSNorm(q_dim, eps=eps) + self.kv_norm = RMSNorm(kv_c_dim, eps=eps) + self.q_b_proj = torch.nn.Linear(q_dim, hidden_size, bias=False) + + def forward( + self, x: torch.Tensor + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + # Avoid graph input being a direct arg to a matched pattern node + x = torch.relu(x) + + projected = self.proj(x) + + q_c, kv_lora = projected.split([self.q_dim, self.kv_dim], dim=-1) + kv_c, k_pe = kv_lora.split([self.kv_c_dim, self.k_pe_dim], dim=-1) + + q_normed = self.q_norm(q_c) + kv_normed = self.kv_norm(kv_c) + + q_out = self.q_b_proj(q_normed) + return q_out, kv_normed, k_pe + + def ops_in_model_before(self): + return [torch.ops.vllm_ir.rms_norm.default] + + def ops_in_model_after(self): + return [torch.ops.vllm.fused_mla_dual_rms_norm.default] + + +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("hidden_size", [7168]) +@pytest.mark.skipif( + not is_aiter_found_and_supported(), + reason="Only test on ROCm with AITER installed and supported", +) +def test_fuse_mla_dual_rms_norm( + dtype: torch.dtype, + hidden_size: int, + monkeypatch: pytest.MonkeyPatch, +): + torch._dynamo.reset() + + vllm_config = VllmConfig( + model_config=ModelConfig(dtype=dtype), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=["+rms_norm"], + pass_config=PassConfig( + fuse_mla_dual_rms_norm=True, + eliminate_noops=True, + ), + ), + ) + + with vllm.config.set_current_vllm_config(vllm_config), monkeypatch.context() as m: + from vllm.compilation.passes.fusion.rocm_aiter_fusion import ( + MLADualRMSNormFusionPass, + ) + + torch.set_default_device("cuda") + torch.set_default_dtype(dtype) + torch.manual_seed(42) + + m.setenv("VLLM_ROCM_USE_AITER", "1") + rocm_aiter_ops.refresh_env_variables() + + fusion_pass = MLADualRMSNormFusionPass(vllm_config) + passes = [ + NoOpEliminationPass(vllm_config), + fusion_pass, + PostCleanupPass(vllm_config), + ] + backend = TestBackend(*passes) + model = MLADualRMSNormTestModel(hidden_size) + + x = torch.randn(1, hidden_size) + torch._dynamo.mark_dynamic(x, 0) + + outputs_unfused = model(x) + + model_fused = torch.compile(model, backend=backend) + outputs_fused = model_fused(x) + + torch.testing.assert_close(outputs_unfused, outputs_fused, atol=1e-2, rtol=1e-2) + + assert fusion_pass.matched_count == 1, ( + f"Expected 1 fused pair, got {fusion_pass.matched_count}" + ) + + backend.check_before_ops(model.ops_in_model_before()) + backend.check_after_ops(model.ops_in_model_after()) + + +class MLADualRMSNormFp8PerTokenTestModel(torch.nn.Module): + """ + Minimal model reproducing the FP8 MLA attention path with *per-token* quant: + linear -> split([q_dim, kv_dim]) + +-- q_c (getitem 0) -> rocm_aiter_rmsnorm_fused_dynamic_quant -> dequant + +-- kv_lora (getitem 1) -> split([kv_c_dim, k_pe_dim]) + +-- kv_c (getitem 0) -> rms_norm (bf16) + +-- k_pe + """ + + def __init__( + self, + hidden_size: int, + q_dim: int = Q_DIM, + kv_c_dim: int = KV_C_DIM, + k_pe_dim: int = K_PE_DIM, + eps: float = EPS, + ): + super().__init__() + self.q_dim = q_dim + self.kv_dim = kv_c_dim + k_pe_dim + self.kv_c_dim = kv_c_dim + self.k_pe_dim = k_pe_dim + self.eps = eps + + self.proj = torch.nn.Linear(hidden_size, q_dim + self.kv_dim, bias=False) + self.q_weight = torch.nn.Parameter(torch.ones(q_dim)) + self.kv_norm = RMSNorm(kv_c_dim, eps=eps) + + def _dequant(self, x_fp8: torch.Tensor, scale: torch.Tensor) -> torch.Tensor: + # Per-token: a single (M, 1) scale broadcast across the row. + return (x_fp8.to(torch.float32) * scale).to(torch.bfloat16) + + def forward(self, x: torch.Tensor): + # Avoid graph input being a direct arg to a matched pattern node + x = torch.relu(x) + + projected = self.proj(x) + + q_c, kv_lora = projected.split([self.q_dim, self.kv_dim], dim=-1) + kv_c, k_pe = kv_lora.split([self.kv_c_dim, self.k_pe_dim], dim=-1) + + q_fp8, q_scale = torch.ops.vllm.rocm_aiter_rmsnorm_fused_dynamic_quant( + q_c, self.q_weight, self.eps, FP8_DTYPE + ) + kv_normed = self.kv_norm(kv_c) + + return self._dequant(q_fp8, q_scale), kv_normed, k_pe + + def ops_in_model_before(self): + return [ + torch.ops.vllm.rocm_aiter_rmsnorm_fused_dynamic_quant.default, + torch.ops.vllm_ir.rms_norm.default, + ] + + def ops_in_model_after(self): + return [torch.ops.vllm.fused_mla_dual_rms_norm_per_token_quant.default] + + +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("hidden_size", [7168]) +@pytest.mark.skipif( + not is_aiter_found_and_supported(), + reason="Only test on ROCm with AITER installed and supported", +) +def test_fuse_mla_dual_rms_norm_fp8_per_token( + dtype: torch.dtype, + hidden_size: int, + monkeypatch: pytest.MonkeyPatch, +): + torch._dynamo.reset() + + vllm_config = VllmConfig( + model_config=ModelConfig(dtype=dtype), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=["+rms_norm"], + pass_config=PassConfig( + fuse_mla_dual_rms_norm=True, + eliminate_noops=True, + ), + ), + ) + + with vllm.config.set_current_vllm_config(vllm_config), monkeypatch.context() as m: + from vllm.compilation.passes.fusion.rocm_aiter_fusion import ( + MLADualRMSNormFusionPass, + ) + + torch.set_default_device("cuda") + torch.set_default_dtype(dtype) + torch.manual_seed(42) + + m.setenv("VLLM_ROCM_USE_AITER", "1") + rocm_aiter_ops.refresh_env_variables() + + fusion_pass = MLADualRMSNormFusionPass(vllm_config) + passes = [ + NoOpEliminationPass(vllm_config), + fusion_pass, + PostCleanupPass(vllm_config), + ] + backend = TestBackend(*passes) + model = MLADualRMSNormFp8PerTokenTestModel(hidden_size) + + x = torch.randn(4, hidden_size) + torch._dynamo.mark_dynamic(x, 0) + + with torch.inference_mode(): + outputs_unfused = model(x) + + model_fused = torch.compile(model, backend=backend) + outputs_fused = model_fused(x) + + q_deq_u, kv_normed_u, k_pe_u = outputs_unfused + q_deq_f, kv_normed_f, k_pe_f = outputs_fused + + torch.testing.assert_close(k_pe_u, k_pe_f, atol=0, rtol=0) + + torch.testing.assert_close(kv_normed_u, kv_normed_f, atol=1e-2, rtol=1e-2) + + E4M3_STEP = 0.125 + exact_frac = (q_deq_u == q_deq_f).float().mean().item() + assert exact_frac > 0.99, ( + f"q: only {exact_frac:.4f} of elements bit-exact; scales likely differ" + ) + torch.testing.assert_close(q_deq_u, q_deq_f, atol=1e-2, rtol=E4M3_STEP) + + assert fusion_pass.matched_count == 1, ( + f"Expected 1 fused pair, got {fusion_pass.matched_count}" + ) + + backend.check_before_ops(model.ops_in_model_before()) + backend.check_after_ops(model.ops_in_model_after()) diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_fusion.py b/asian_ai/performance/vLLM/tests/compile/passes/test_fusion.py new file mode 100644 index 0000000000000000000000000000000000000000..2f48d301a68b6020b05d413ed7b06633e335a19f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_fusion.py @@ -0,0 +1,680 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import pytest +import torch + +import vllm.config +import vllm.ir.ops +import vllm.plugins +from tests.compile.backend import TestBackend +from tests.utils import TestFP8Layer +from vllm._aiter_ops import IS_AITER_FOUND, rocm_aiter_ops +from vllm.compilation.passes.fusion.matcher_utils import QUANT_OPS +from vllm.compilation.passes.fusion.rms_quant_fusion import ( + FUSED_OPS, + FusedRMSQuantKey, + RMSNormQuantFusionPass, +) +from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass +from vllm.compilation.passes.utility.post_cleanup import PostCleanupPass +from vllm.config import ( + CompilationConfig, + CompilationMode, + ModelConfig, + PassConfig, + VllmConfig, +) +from vllm.model_executor.kernels.linear import ( + AiterFp8BlockScaledMMKernel, + ChannelWiseTorchFP8ScaledMMLinearKernel, + CutlassFp8BlockScaledMMKernel, + CutlassFP8ScaledMMLinearKernel, + DeepGemmFp8BlockScaledMMKernel, + FlashInferFp8DeepGEMMDynamicBlockScaledKernel, + FlashInferFP8ScaledMMLinearKernel, + PerTensorTorchFP8ScaledMMLinearKernel, + ROCmFP8ScaledMMLinearKernel, + RowWiseTorchFP8ScaledMMLinearKernel, + TritonFp8BlockScaledMMKernel, + _KernelT, +) +from vllm.model_executor.layers.layernorm import RMSNorm, RMSNormGated +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + GroupShape, + create_fp8_quant_key, +) +from vllm.model_executor.layers.quantization.utils.w8a8_utils import ( + cutlass_block_fp8_supported, +) +from vllm.platforms import current_platform +from vllm.utils.deep_gemm import ( + is_deep_gemm_e8m0_used, + is_deep_gemm_supported, +) + +FP8_DTYPE = current_platform.fp8_dtype() + +RMS_ADD_OP = torch.ops._C.fused_add_rms_norm.default + +# Kernel and group_shape combinations: (kernel, group_shape) +# CUDA kernels +CUDA_KERNEL_GROUPSHAPE_COMBINATIONS = [ + # FlashInferFP8ScaledMMLinearKernel supports both per-tensor only + (FlashInferFP8ScaledMMLinearKernel, GroupShape.PER_TENSOR), + # CutlassFP8ScaledMMLinearKernel supports both per-tensor and per-token + (CutlassFP8ScaledMMLinearKernel, GroupShape.PER_TOKEN), + (CutlassFP8ScaledMMLinearKernel, GroupShape.PER_TENSOR), + # PerTensorTorchFP8ScaledMMLinearKernel only supports per-tensor + (PerTensorTorchFP8ScaledMMLinearKernel, GroupShape.PER_TENSOR), + # ChannelWiseTorchFP8ScaledMMLinearKernel only supports per-token + (ChannelWiseTorchFP8ScaledMMLinearKernel, GroupShape.PER_TOKEN), + # Blockwise group shapes + (FlashInferFp8DeepGEMMDynamicBlockScaledKernel, GroupShape(1, 128)), + (CutlassFp8BlockScaledMMKernel, GroupShape(1, 128)), + (DeepGemmFp8BlockScaledMMKernel, GroupShape(1, 128)), + (TritonFp8BlockScaledMMKernel, GroupShape(1, 128)), + (TritonFp8BlockScaledMMKernel, GroupShape(1, 64)), +] + +# ROCm kernels +ROCM_KERNEL_GROUPSHAPE_COMBINATIONS = [ + # ROCmFP8ScaledMMLinearKernel supports per-tensor only + (ROCmFP8ScaledMMLinearKernel, GroupShape.PER_TENSOR), + # RowWiseTorchFP8ScaledMMLinearKernel only supports per-token + (RowWiseTorchFP8ScaledMMLinearKernel, GroupShape.PER_TOKEN), + # ChannelWiseTorchFP8ScaledMMLinearKernel only supports per-token + (ChannelWiseTorchFP8ScaledMMLinearKernel, GroupShape.PER_TOKEN), + # Blockwise group shapes (no kernel abstraction) + (TritonFp8BlockScaledMMKernel, GroupShape(1, 128)), + (TritonFp8BlockScaledMMKernel, GroupShape(1, 64)), +] + +KERNEL_GROUPSHAPE_COMBINATIONS = ( + CUDA_KERNEL_GROUPSHAPE_COMBINATIONS + if current_platform.is_cuda() + else ROCM_KERNEL_GROUPSHAPE_COMBINATIONS +) + +# For Aiter tests we toggle use_aiter_quant_op +AITER_KERNEL_GROUPSHAPE_COMBINATIONS = [ + # Per-token with RowWiseTorchFP8ScaledMMLinearKernel + (RowWiseTorchFP8ScaledMMLinearKernel, GroupShape.PER_TOKEN, True), + (RowWiseTorchFP8ScaledMMLinearKernel, GroupShape.PER_TOKEN, False), + # Per-token with ChannelWiseTorchFP8ScaledMMLinearKernel + (ChannelWiseTorchFP8ScaledMMLinearKernel, GroupShape.PER_TOKEN, True), + (ChannelWiseTorchFP8ScaledMMLinearKernel, GroupShape.PER_TOKEN, False), + # Blockwise + (AiterFp8BlockScaledMMKernel, GroupShape(1, 128), True), +] + + +class TestModel(torch.nn.Module): + def __init__( + self, + hidden_size: int, + eps: float, + force_kernel: type[_KernelT] | None, + group_shape: GroupShape, + dtype: torch.dtype, + use_aiter_fusion: bool = False, + use_aiter_quant: bool = False, + *args, + **kwargs, + ): + super().__init__(*args, **kwargs) + self.fp8_linear_layers: list[torch.nn.Module] + self.group_shape = group_shape + self.use_aiter_quant_op = use_aiter_quant + self.use_aiter_fusion = use_aiter_fusion + self.norm = [RMSNorm(hidden_size, eps) for _ in range(4)] + self.enable_rms_norm_custom_op = self.norm[0].enabled() + + # Determine if blockwise based on group_shape + is_blockwise = group_shape.is_per_group() + + if is_blockwise: + block_size = group_shape.col + self.activation_quant_key = create_fp8_quant_key( + static=False, group_shape=group_shape + ) + self.weight_quant_key = create_fp8_quant_key( + static=True, group_shape=GroupShape(block_size, block_size) + ) + + else: + is_static = group_shape == GroupShape.PER_TENSOR + self.activation_quant_key = create_fp8_quant_key( + is_static, group_shape=group_shape + ) + self.weight_quant_key = create_fp8_quant_key( + static=True, group_shape=group_shape + ) + + self.fp8_linear_layers = [ + TestFP8Layer( + weight_shape=(hidden_size, hidden_size), + activation_quant_key=self.activation_quant_key, + weight_quant_key=self.weight_quant_key, + force_kernel=force_kernel, + transpose_weights=use_aiter_fusion, + input_dtype=dtype, + ) + for _ in range(3) + ] + + # Enable aiter quantization if requested + for layer in self.fp8_linear_layers: + layer.kernel.quant_fp8.use_aiter = use_aiter_quant + + self.enable_quant_fp8_custom_op = self.fp8_linear_layers[ + 0 + ].is_quant_fp8_enabled() + + def forward(self, x): + # avoid having graph input be an arg to a pattern directly + x = resid = torch.relu(x) + y = self.norm[0](x) + + x2 = self.fp8_linear_layers[0](y) + # make sure resid is used for replacement to work + y2, resid = self.norm[1](x2, resid) + + x3 = self.fp8_linear_layers[1](y2) + + y3, resid = self.norm[2](x3, resid) # use resid here + + x4 = self.fp8_linear_layers[2](y3) + + y4, resid = self.norm[3](x4, resid) # use resid here + return y4 + + def ops_in_model_before(self): + if self.group_shape.is_per_group(): + # Blockwise path + if self.use_aiter_fusion and self.use_aiter_quant_op: + return [rocm_aiter_ops.get_group_quant_op()] + if self.use_aiter_fusion: + return [torch.ops.vllm.triton_per_token_group_quant_fp8.default] + else: + if self.use_aiter_quant_op: + return [rocm_aiter_ops.get_per_token_quant_op()] + + # Common path + return ( + [QUANT_OPS[self.activation_quant_key]] + if self.enable_quant_fp8_custom_op + else [torch.ops.aten.reciprocal] + ) + + def ops_in_model_after(self): + if self.use_aiter_fusion: + if self.group_shape.is_per_group(): + # Blockwise aiter fusion + from vllm.compilation.passes.fusion.rocm_aiter_fusion import ( + AiterFusedAddRMSFp8GroupQuantPattern, + AiterRMSFp8GroupQuantPattern, + ) + + return [ + AiterFusedAddRMSFp8GroupQuantPattern.FUSED_OP, + AiterRMSFp8GroupQuantPattern.FUSED_OP, + ] + else: + # Per-token aiter fusion + from vllm.compilation.passes.fusion.rocm_aiter_fusion import ( + AiterFusedAddRMSNormDynamicQuantPattern, + AiterRMSNormDynamicQuantPattern, + ) + + return [ + AiterFusedAddRMSNormDynamicQuantPattern.FUSED_OP, + AiterRMSNormDynamicQuantPattern.FUSED_OP, + ] + + # Regular fusion + return [ + FUSED_OPS[FusedRMSQuantKey(self.activation_quant_key, True)], + FUSED_OPS[FusedRMSQuantKey(self.activation_quant_key, False)], + ] + + def ops_in_model_before_partial(self): + return [ + torch.ops.vllm_ir.rms_norm, + torch.ops.vllm_ir.fused_add_rms_norm.default, + ] + + +def _run_fusion_test( + model, + fusion_pass, + vllm_config, + dtype, + hidden_size, + num_tokens, +): + """Helper function for common fusion test logic. + + Must be called within vllm_config context. + """ + noop_pass = NoOpEliminationPass(vllm_config) + cleanup_pass = PostCleanupPass(vllm_config) + + backend = TestBackend(noop_pass, fusion_pass, cleanup_pass) + backend2 = TestBackend(noop_pass, cleanup_pass) + + x = torch.rand(num_tokens, hidden_size) + torch._dynamo.mark_dynamic(x, 0) + + model_fused = torch.compile(model, backend=backend) + result_fused = model_fused(x) + + model_unfused = torch.compile(model, backend=backend2) + result_unfused = model_unfused(x) + + if dtype == torch.float16: + ATOL, RTOL = (2e-3, 2e-3) + else: + ATOL, RTOL = (1e-2, 1e-2) + + torch.testing.assert_close(result_fused, result_unfused, atol=ATOL, rtol=RTOL) + + assert fusion_pass.matched_count == 3 + backend.check_before_ops(model.ops_in_model_before()) + backend.check_after_ops(model.ops_in_model_after()) + + return backend, backend2 + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@pytest.mark.parametrize("hidden_size", [256]) +@pytest.mark.parametrize("num_tokens", [257]) +@pytest.mark.parametrize("eps", [1e-5, 1e-6]) +@pytest.mark.parametrize("kernel_groupshape", KERNEL_GROUPSHAPE_COMBINATIONS) +@pytest.mark.parametrize("enable_rms_norm_custom_op", [True, False]) +@pytest.mark.parametrize("enable_quant_fp8_custom_op", [True, False]) +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), reason="Only test on CUDA and ROCm" +) +def test_fusion_rmsnorm_quant( + dtype, + hidden_size, + num_tokens, + eps, + kernel_groupshape, + enable_rms_norm_custom_op, + enable_quant_fp8_custom_op, +): + force_kernel, group_shape = kernel_groupshape + + if not enable_quant_fp8_custom_op and group_shape.is_per_group(): + pytest.skip("Unsupported unwrapped quant fp8 op for blockwise quantization") + + if group_shape == GroupShape(1, 64) and ( + cutlass_block_fp8_supported() or is_deep_gemm_supported() + ): + pytest.skip("Unsupported group shape 64 for CUTLASS/DeepGemm") + + # TODO(quant-rms-fusion): DeepGEMM UE8M0 activation quant on B200 lowers + # to a packed int32-scale op (per_token_group_quant_fp8_packed_for_deepgemm), + # but the rms+quant fusion pattern only matches the fp32-scale variant, so + # the fused output gets a mismatched scale layout and produces NaN. Only + # reproduces on bf16 (DeepGEMM UE8M0 on B200 is bf16-only). + # To re-enable: make rms_norm_per_block_quant emit packed UE8M0 scales + # and extend the fusion pattern to rewrite the packed activation quant. + deepgemm_kernels = ( + DeepGemmFp8BlockScaledMMKernel, + FlashInferFp8DeepGEMMDynamicBlockScaledKernel, + ) + if ( + dtype == torch.bfloat16 + and force_kernel in deepgemm_kernels + and is_deep_gemm_e8m0_used() + ): + pytest.skip( + "rms+quant fusion does not yet match the packed UE8M0 DeepGEMM path" + ) + + custom_ops = [] + if enable_rms_norm_custom_op: + custom_ops.append("+rms_norm") + if enable_quant_fp8_custom_op: + custom_ops.append("+quant_fp8") + + vllm_config = VllmConfig( + model_config=ModelConfig(dtype=dtype), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=custom_ops, + pass_config=PassConfig( + fuse_norm_quant=True, fuse_act_quant=True, eliminate_noops=True + ), + ), + ) + + with ( + vllm.config.set_current_vllm_config(vllm_config), + vllm_config.kernel_config.ir_op_priority.set_priority(), + ): + # Setup device before model creation + torch.set_default_device("cuda") + torch.set_default_dtype(dtype) + torch.manual_seed(1) + + fusion_pass = RMSNormQuantFusionPass(vllm_config) + + model = TestModel( + hidden_size=hidden_size, + eps=eps, + force_kernel=force_kernel, + group_shape=group_shape, + dtype=dtype, + use_aiter_fusion=False, + use_aiter_quant=False, + ) + + backend, _ = _run_fusion_test( + model, fusion_pass, vllm_config, dtype, hidden_size, num_tokens + ) + backend.check_before_ops( + model.ops_in_model_before_partial(), fully_replaced=False + ) + + +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("hidden_size", [256]) +@pytest.mark.parametrize("num_tokens", [257]) +@pytest.mark.parametrize("eps", [1e-5, 1e-6]) +@pytest.mark.parametrize( + "kernel_groupshape_quant", AITER_KERNEL_GROUPSHAPE_COMBINATIONS +) +@pytest.mark.skipif( + (not current_platform.is_rocm() or not IS_AITER_FOUND), + reason="Only test on ROCm with aiter package installed", +) +def test_aiter_fusion_rmsnorm_quant( + dtype: torch.dtype, + hidden_size: int, + num_tokens: int, + eps: float, + kernel_groupshape_quant: tuple, + monkeypatch: pytest.MonkeyPatch, +): + force_kernel, group_shape, use_aiter_quant_op = kernel_groupshape_quant + vllm_config = VllmConfig( + model_config=ModelConfig(dtype=dtype), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=["+rms_norm", "+quant_fp8"], + pass_config=PassConfig(fuse_norm_quant=True, eliminate_noops=True), + ), + ) + + with vllm.config.set_current_vllm_config(vllm_config), monkeypatch.context() as m: + from vllm.compilation.passes.fusion.rocm_aiter_fusion import ( + RocmAiterRMSNormQuantFusionPass, + ) + + m.setenv("VLLM_ROCM_USE_AITER", "1") + + rocm_aiter_ops.refresh_env_variables() + + torch.set_default_device("cuda") + torch.set_default_dtype(dtype) + torch.manual_seed(1) + + fusion_pass = RocmAiterRMSNormQuantFusionPass(vllm_config) + + model = TestModel( + hidden_size=hidden_size, + eps=eps, + force_kernel=force_kernel, + group_shape=group_shape, + dtype=dtype, + use_aiter_fusion=True, # Always use aiter fusion ops in aiter test + use_aiter_quant=use_aiter_quant_op, # Toggle aiter quantization + ) + + _run_fusion_test( + model, fusion_pass, vllm_config, dtype, hidden_size, num_tokens + ) + + +class TestGatedModel(torch.nn.Module): + """Model that uses RMSNormGated + reshape + group FP8 quant + linear. + + Mimics GatedDeltaNetAttention's output projection path where: + - RMSNormGated operates on per-head tensors (N*H, D) + - Output is reshaped to (N, H*D) before group quantization + linear + """ + + def __init__( + self, + num_heads: int, + head_dim: int, + eps: float, + force_kernel: type[_KernelT], + group_shape: GroupShape, + dtype: torch.dtype, + use_aiter_quant: bool = True, + ): + super().__init__() + self.num_heads = num_heads + self.head_dim = head_dim + hidden_dim = num_heads * head_dim + + self.norm = RMSNormGated( + head_dim, + eps=eps, + group_size=None, + norm_before_gate=True, + ) + + self.activation_quant_key = create_fp8_quant_key( + static=False, group_shape=group_shape + ) + self.weight_quant_key = create_fp8_quant_key( + static=True, group_shape=GroupShape(group_shape.col, group_shape.col) + ) + + self.fp8_linear = TestFP8Layer( + weight_shape=(hidden_dim, hidden_dim), + activation_quant_key=self.activation_quant_key, + weight_quant_key=self.weight_quant_key, + force_kernel=force_kernel, + transpose_weights=True, + input_dtype=dtype, + ) + self.fp8_linear.kernel.quant_fp8.use_aiter = use_aiter_quant + + def forward(self, x, z): + num_heads = self.num_heads + head_dim = self.head_dim + hidden_dim = num_heads * head_dim + x = torch.relu(x) + z = torch.relu(z) + x_heads = x.reshape(-1, num_heads, head_dim).reshape(-1, head_dim) + z_heads = z.reshape(-1, num_heads, head_dim).reshape(-1, head_dim) + normed = self.norm(x_heads, z_heads) + merged = normed.reshape(-1, hidden_dim) + out = self.fp8_linear(merged) + return out + + def ops_in_model_after(self): + from vllm.compilation.passes.fusion.rocm_aiter_fusion import ( + AiterRMSNormGatedFp8GroupQuantPattern, + ) + + return [AiterRMSNormGatedFp8GroupQuantPattern.FUSED_OP] + + +class _MockGDNLayer: + """Minimal mock to populate static_forward_context for pass discovery. + + Uses __class__ assignment to pass isinstance checks against + GatedDeltaNetAttention without requiring a full config-based init. + """ + + def __init__(self, num_v_heads: int, head_v_dim: int, tp_size: int = 1): + self.num_v_heads = num_v_heads + self.head_v_dim = head_v_dim + self.tp_size = tp_size + + from vllm.model_executor.layers.mamba.gdn.base import ( + GatedDeltaNetAttention, + ) + + self.__class__ = GatedDeltaNetAttention + + +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("num_heads", [2]) +@pytest.mark.parametrize("head_dim", [128]) +@pytest.mark.parametrize("num_tokens", [8]) +@pytest.mark.parametrize("eps", [1e-5, 1e-6]) +@pytest.mark.skipif( + (not current_platform.is_rocm() or not IS_AITER_FOUND), + reason="Only test on ROCm with aiter package installed", +) +def test_aiter_fusion_rmsnorm_gated_quant( + dtype: torch.dtype, + num_heads: int, + head_dim: int, + num_tokens: int, + eps: float, + monkeypatch: pytest.MonkeyPatch, +): + group_shape = GroupShape(1, 128) + vllm_config = VllmConfig( + model_config=ModelConfig(dtype=dtype), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=["-rms_norm", "-silu_and_mul", "-quant_fp8"], + pass_config=PassConfig(fuse_norm_quant=True, eliminate_noops=True), + ), + ) + + with vllm.config.set_current_vllm_config(vllm_config), monkeypatch.context() as m: + from vllm.compilation.passes.fusion.rocm_aiter_fusion import ( + RocmAiterRMSNormQuantFusionPass, + ) + + m.setenv("VLLM_ROCM_USE_AITER", "1") + rocm_aiter_ops.refresh_env_variables() + + # Register a mock GDN layer so the pass discovers num_heads/head_dim + mock_gdn = _MockGDNLayer(num_v_heads=num_heads, head_v_dim=head_dim, tp_size=1) + vllm_config.compilation_config.static_forward_context["mock_gdn_layer"] = ( + mock_gdn + ) + + torch.set_default_device("cuda") + torch.set_default_dtype(dtype) + torch.manual_seed(1) + + fusion_pass = RocmAiterRMSNormQuantFusionPass(vllm_config) + + model = TestGatedModel( + num_heads=num_heads, + head_dim=head_dim, + eps=eps, + force_kernel=AiterFp8BlockScaledMMKernel, + group_shape=group_shape, + dtype=dtype, + use_aiter_quant=True, + ) + + noop_pass = NoOpEliminationPass(vllm_config) + cleanup_pass = PostCleanupPass(vllm_config) + + backend = TestBackend(noop_pass, fusion_pass, cleanup_pass) + backend2 = TestBackend(noop_pass, cleanup_pass) + + hidden_dim = num_heads * head_dim + x = torch.rand(num_tokens, hidden_dim) + z = torch.rand(num_tokens, hidden_dim) + torch._dynamo.mark_dynamic(x, 0) + torch._dynamo.mark_dynamic(z, 0) + + model_fused = torch.compile(model, backend=backend) + result_fused = model_fused(x, z) + + model_unfused = torch.compile(model, backend=backend2) + result_unfused = model_unfused(x, z) + + torch.testing.assert_close(result_fused, result_unfused, atol=1e-2, rtol=1e-2) + + assert fusion_pass.matched_count == 1 + backend.check_after_ops(model.ops_in_model_after()) + + +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("num_heads", [2]) +@pytest.mark.parametrize("head_dim", [128]) +@pytest.mark.parametrize("num_tokens", [8]) +@pytest.mark.parametrize("eps", [1e-6]) +@pytest.mark.skipif( + (not current_platform.is_rocm() or not IS_AITER_FOUND), + reason="Only test on ROCm with aiter package installed", +) +def test_aiter_fusion_rmsnorm_gated_quant_no_gdn_layers( + dtype: torch.dtype, + num_heads: int, + head_dim: int, + num_tokens: int, + eps: float, + monkeypatch: pytest.MonkeyPatch, +): + """Verify that without GDN layers in static_forward_context, + the gated pattern is not registered and no matches occur.""" + group_shape = GroupShape(1, 128) + vllm_config = VllmConfig( + model_config=ModelConfig(dtype=dtype), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=["-rms_norm", "-silu_and_mul", "-quant_fp8"], + pass_config=PassConfig(fuse_norm_quant=True, eliminate_noops=True), + ), + ) + + with vllm.config.set_current_vllm_config(vllm_config), monkeypatch.context() as m: + from vllm.compilation.passes.fusion.rocm_aiter_fusion import ( + RocmAiterRMSNormQuantFusionPass, + ) + + m.setenv("VLLM_ROCM_USE_AITER", "1") + rocm_aiter_ops.refresh_env_variables() + + torch.set_default_device("cuda") + torch.set_default_dtype(dtype) + torch.manual_seed(1) + + # No mock GDN layer registered -- pass should not register gated pattern + fusion_pass = RocmAiterRMSNormQuantFusionPass(vllm_config) + + model = TestGatedModel( + num_heads=num_heads, + head_dim=head_dim, + eps=eps, + force_kernel=AiterFp8BlockScaledMMKernel, + group_shape=group_shape, + dtype=dtype, + use_aiter_quant=True, + ) + + noop_pass = NoOpEliminationPass(vllm_config) + cleanup_pass = PostCleanupPass(vllm_config) + + backend = TestBackend(noop_pass, fusion_pass, cleanup_pass) + + hidden_dim = num_heads * head_dim + x = torch.rand(num_tokens, hidden_dim) + z = torch.rand(num_tokens, hidden_dim) + torch._dynamo.mark_dynamic(x, 0) + torch._dynamo.mark_dynamic(z, 0) + + model_fused = torch.compile(model, backend=backend) + model_fused(x, z) + + assert fusion_pass.matched_count == 0 diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_fusion_attn.py b/asian_ai/performance/vLLM/tests/compile/passes/test_fusion_attn.py new file mode 100644 index 0000000000000000000000000000000000000000..155f7294ee8f2fe20eb248bded57500b0dd6239b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_fusion_attn.py @@ -0,0 +1,487 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import copy + +import pytest +import torch._dynamo + +from tests.compile.backend import LazyInitPass, TestBackend +from tests.utils import TestFP8Layer, flat_product +from tests.v1.attention.utils import BatchSpec, create_common_attn_metadata +from vllm._custom_ops import cutlass_scaled_fp4_mm, scaled_fp4_quant +from vllm.compilation.passes.fusion.attn_quant_fusion import ( + ATTN_OP, + AttnQuantFusionPass, +) +from vllm.compilation.passes.fusion.matcher_utils import QUANT_OPS +from vllm.compilation.passes.fx_utils import find_op_nodes +from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass +from vllm.compilation.passes.utility.post_cleanup import PostCleanupPass +from vllm.config import ( + AttentionConfig, + CacheConfig, + CompilationConfig, + CompilationMode, + ModelConfig, + PassConfig, + SchedulerConfig, + VllmConfig, + set_current_vllm_config, +) +from vllm.forward_context import get_forward_context, set_forward_context +from vllm.model_executor.layers.attention import Attention +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + QuantKey, + kFp8StaticTensorSym, + kNvfp4Dynamic, +) +from vllm.platforms import current_platform +from vllm.utils.flashinfer import has_flashinfer +from vllm.v1.attention.backend import AttentionMetadata +from vllm.v1.attention.backends.registry import AttentionBackendEnum +from vllm.v1.kv_cache_interface import AttentionSpec, get_kv_quant_mode + +DEVICE_TYPE = current_platform.device_type +FP8_DTYPE = current_platform.fp8_dtype() +FP4_DTYPE = torch.uint8 + + +class AttentionQuantPatternModel(torch.nn.Module): + """Base model for AttentionQuantPattern fusion.""" + + def __init__( + self, + num_qo_heads: int, + num_kv_heads: int, + head_size: int, + device: torch.device, + vllm_config: VllmConfig, + block_size: int, + **kwargs, + ): + super().__init__() + self.num_qo_heads = num_qo_heads + self.num_kv_heads = num_kv_heads + self.head_size = head_size + self.device = device + self.vllm_config = vllm_config + self.dtype = vllm_config.model_config.dtype + + self.attn = Attention( + num_heads=self.num_qo_heads, + head_size=self.head_size, + scale=1.0 / (self.head_size**0.5), + num_kv_heads=self.num_kv_heads, + cache_config=vllm_config.cache_config, + prefix="model.layers.0.self_attn.attn", + ) + self.attn._k_scale = self.attn._k_scale.to(device) + self.attn._v_scale = self.attn._v_scale.to(device) + + self.block_size = block_size + + # Initialize attn MetadataBuilder (match Attention.get_kv_cache_spec) + self.builder = self.attn.attn_backend.get_builder_cls()( + kv_cache_spec=AttentionSpec( + block_size=self.block_size, + num_kv_heads=self.num_kv_heads, + head_size=self.head_size, + dtype=self.attn.kv_cache_torch_dtype, + kv_quant_mode=get_kv_quant_mode(self.attn.kv_cache_dtype), + ), + layer_names=[self.attn.layer_name], + vllm_config=self.vllm_config, + device=self.device, + ) + + def build_attn_metadata(self, batch_size: int) -> AttentionMetadata: + """Initialize attention metadata.""" + + # TODO (Rohan138) reuse utils from vllm/v1/worker/gpu/attn_utils.py + + # Create common attn metadata + batch_spec = BatchSpec(seq_lens=[1] * batch_size, query_lens=[1] * batch_size) + common_attn_metadata = create_common_attn_metadata( + batch_spec, self.block_size, self.device, arange_block_indices=True + ) + + max_blocks = (max(batch_spec.seq_lens) + self.block_size - 1) // self.block_size + num_blocks = batch_size * max_blocks + + # Fetch the attention backend and kv cache shape and stride order + attn_backend = self.attn.attn_backend + kv_cache_shape = attn_backend.get_kv_cache_shape( + num_blocks, self.block_size, self.num_kv_heads, self.head_size + ) + try: + kv_cache_stride_order = attn_backend.get_kv_cache_stride_order() + except (AttributeError, NotImplementedError): + kv_cache_stride_order = tuple(range(len(kv_cache_shape))) + + kv_cache_shape = tuple(kv_cache_shape[i] for i in kv_cache_stride_order) + inv_order = [ + kv_cache_stride_order.index(i) for i in range(len(kv_cache_stride_order)) + ] + + # Create dummy KV cache + raw_tensor = torch.zeros( + 2 * num_blocks * self.block_size * self.num_kv_heads * self.head_size, + dtype=self.attn.kv_cache_torch_dtype, + device=self.device, + ) + raw_tensor = raw_tensor.view(kv_cache_shape) + kv_cache = raw_tensor.permute(*inv_order) + + self.attn.kv_cache = kv_cache + + # Build attn metadata + self.attn_metadata = self.builder.build( + common_prefix_len=0, common_attn_metadata=common_attn_metadata + ) + + return self.attn_metadata + + +class TestAttentionFp8StaticQuantPatternModel(AttentionQuantPatternModel): + """Test model for AttentionFp8StaticQuantPattern fusion.""" + + quant_key = kFp8StaticTensorSym + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + + hidden_size = self.num_qo_heads * self.head_size + self.fp8_linear = TestFP8Layer( + weight_shape=(hidden_size, hidden_size), + activation_quant_key=self.quant_key, + weight_quant_key=self.quant_key, + device=self.device, + input_dtype=self.dtype, + ) + + w = kwargs.get("w") + if w is not None: + self.fp8_linear.weight = w["weight"] + self.fp8_linear.weight_scale = w["wscale"] + self.fp8_linear.input_scale = w["scale"] + + self.w = { + "weight": self.fp8_linear.weight, + "wscale": self.fp8_linear.weight_scale, + "scale": self.fp8_linear.input_scale, + } + + def forward(self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor): + """Forward pass that creates the pattern to be fused.""" + attn_output = self.attn(q, k, v) + return self.fp8_linear(attn_output) + + +class TestAttentionNvfp4QuantPatternModel(AttentionQuantPatternModel): + """Test model for AttentionNvfp4QuantPattern fusion.""" + + quant_key = kNvfp4Dynamic + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + + hidden_size = self.num_qo_heads * self.head_size + self.w = kwargs.get( + "w", + { + "weight": torch.randint( + 256, + (hidden_size, hidden_size // 2), + dtype=FP4_DTYPE, + device=self.device, + ), + "wscale_swizzled": torch.randn(hidden_size, hidden_size // 16).to( + dtype=FP8_DTYPE, device=self.device + ), + "wscale": torch.tensor([500], dtype=torch.float32, device=self.device), + "scale": torch.tensor([0.002], dtype=torch.float32, device=self.device), + }, + ) + + def forward(self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor): + """Forward pass that creates the pattern to be fused.""" + attn_output = self.attn(q, k, v) + quant_output, output_block_scale = scaled_fp4_quant( + attn_output, 1 / self.w["scale"] + ) + return cutlass_scaled_fp4_mm( + a=quant_output, + b=self.w["weight"], + block_scale_a=output_block_scale, + block_scale_b=self.w["wscale_swizzled"], + alpha=self.w["scale"] * self.w["wscale"], + out_dtype=attn_output.dtype, + ) + + +PATTERN_TEST_MODELS_FP8: list[tuple[str, type]] = [] +PATTERN_TEST_MODELS_FP4: list[tuple[str, type]] = [] +HEADS: list[tuple[int, int]] = [] +SPLIT_ATTENTION: list[bool] = [] +BACKENDS_FP8: list[AttentionBackendEnum] = [] +BACKENDS_FP4: list[AttentionBackendEnum] = [] + +if current_platform.is_cuda(): + HEADS = [(64, 8), (40, 8)] + PATTERN_TEST_MODELS_FP8 = [ + ( + "RedHatAI/Meta-Llama-3.1-8B-FP8", + TestAttentionFp8StaticQuantPatternModel, + ) + ] + PATTERN_TEST_MODELS_FP4 = [ + ( + "nvidia/Llama-3.1-8B-Instruct-NVFP4", + TestAttentionNvfp4QuantPatternModel, + ) + ] + BACKENDS_FP8 = [AttentionBackendEnum.TRITON_ATTN, AttentionBackendEnum.FLASHINFER] + BACKENDS_FP4 = [AttentionBackendEnum.FLASHINFER] + +elif current_platform.is_rocm(): + HEADS = [(32, 8), (40, 8)] + PATTERN_TEST_MODELS_FP8 = [ + ("amd/Llama-3.1-8B-Instruct-FP8-KV", TestAttentionFp8StaticQuantPatternModel) + ] + BACKENDS_FP8 = [ + AttentionBackendEnum.ROCM_AITER_UNIFIED_ATTN, + AttentionBackendEnum.ROCM_ATTN, + AttentionBackendEnum.TRITON_ATTN, + ] + + +@pytest.mark.parametrize("num_qo_heads, num_kv_heads", HEADS) +@pytest.mark.parametrize("head_size", [128]) +@pytest.mark.parametrize( + "batch_size", [7, 256, 533] if current_platform.is_cuda() else [8] +) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) +@pytest.mark.parametrize( + "backend, model_name, model_class, custom_ops", + # Test attention+quant_fp8 fusion with custom and torch impls of QuantFP8 + list( + flat_product( + BACKENDS_FP8, PATTERN_TEST_MODELS_FP8, ["+quant_fp8", "-quant_fp8"] + ) + ) + # quant_fp4 only has the custom impl + + list(flat_product(BACKENDS_FP4, PATTERN_TEST_MODELS_FP4, [""])), +) +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), reason="Only test ROCm or CUDA" +) +@pytest.mark.skipif(not current_platform.supports_fp8(), reason="Need FP8") +def test_attention_quant_pattern( + num_qo_heads: int, + num_kv_heads: int, + head_size: int, + batch_size: int, + dtype: torch.dtype, + custom_ops: str, + model_name: str, + model_class: type[AttentionQuantPatternModel], + backend: AttentionBackendEnum, + dist_init, + monkeypatch, + use_fresh_inductor_cache, +): + """Test AttentionStaticQuantPattern fusion pass""" + monkeypatch.setenv("VLLM_DISABLE_COMPILE_CACHE", "1") + + if backend == AttentionBackendEnum.FLASHINFER and ( + not current_platform.is_device_capability((10, 0)) or not has_flashinfer() + ): + # This also captures the FP4 case + pytest.skip("FlashInfer attn fusion requires Blackwell and flashinfer") + + custom_ops_list = custom_ops.split(",") if custom_ops else [] + + device = torch.device(f"{DEVICE_TYPE}:0") + torch.set_default_dtype(dtype) + torch.manual_seed(42) + + backend_cls = backend.get_class() + + # TODO: drop once AITER reenables fp16 unified attention. + if dtype not in backend_cls.supported_dtypes: + pytest.skip(f"{backend.name} does not support dtype {dtype}") + + block_size = backend_cls.get_preferred_block_size(16) + + model_config = ModelConfig( + model=model_name, + max_model_len=2048, + dtype=dtype, + ) + vllm_config = VllmConfig( + model_config=model_config, + scheduler_config=SchedulerConfig( + max_num_seqs=1024, + max_model_len=model_config.max_model_len, + is_encoder_decoder=model_config.is_encoder_decoder, + ), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=custom_ops_list, + ), + cache_config=CacheConfig(cache_dtype="fp8"), + attention_config=AttentionConfig(backend=backend), + ) + + # Create test inputs + q = torch.randn(batch_size, num_qo_heads * head_size, dtype=dtype, device=device) + k = torch.randn(batch_size, num_kv_heads * head_size, dtype=dtype, device=device) + v = torch.randn(batch_size, num_kv_heads * head_size, dtype=dtype, device=device) + + # Mark first dimension as dynamic for realistic testing + torch._dynamo.mark_dynamic(q, 0) + torch._dynamo.mark_dynamic(k, 0) + torch._dynamo.mark_dynamic(v, 0) + + # Run model directly without compilation and fusion + vllm_config_unfused = copy.deepcopy(vllm_config) + with ( + set_current_vllm_config(vllm_config_unfused), + set_forward_context(attn_metadata=None, vllm_config=vllm_config_unfused), + ): + model_unfused = model_class( + num_qo_heads=num_qo_heads, + num_kv_heads=num_kv_heads, + head_size=head_size, + device=device, + vllm_config=vllm_config_unfused, + block_size=block_size, + ) + model_unfused = model_unfused.to(device) + result_unfused_0 = model_unfused(q, k, v) # noqa: F841 HACK: See #131044 + + forward_ctx = get_forward_context() + forward_ctx.attn_metadata = model_unfused.build_attn_metadata(batch_size) + + # Run model directly without fusion + # Still compile so query QuantFP8 has closer numerics + compiled_unfused = torch.compile(model_unfused, fullgraph=True) + result_unfused = compiled_unfused(q, k, v) + + # Run model with attn fusion enabled + vllm_config.compilation_config.pass_config = PassConfig( + fuse_attn_quant=True, eliminate_noops=True + ) + with ( + set_current_vllm_config(vllm_config), + set_forward_context(attn_metadata=None, vllm_config=vllm_config), + ): + model_fused = model_class( + num_qo_heads=num_qo_heads, + num_kv_heads=num_kv_heads, + head_size=head_size, + device=device, + vllm_config=vllm_config, + w=model_unfused.w, + block_size=block_size, + ) + model_fused = model_fused.to(device) + + forward_ctx = get_forward_context() + forward_ctx.attn_metadata = model_fused.build_attn_metadata(batch_size) + + # Create test backend with fusion passes enabled + noop_pass = NoOpEliminationPass(vllm_config) + attn_pass = LazyInitPass(AttnQuantFusionPass, vllm_config) + cleanup_pass = PostCleanupPass(vllm_config) + + test_backend = TestBackend(noop_pass, attn_pass, cleanup_pass) + # HACK: See https://github.com/vllm-project/vllm/issues/31044 + result_fused_0 = model_fused(q, k, v) # noqa: F841 + + # Compile model with fusion enabled + compiled_fused = torch.compile( + model_fused, backend=test_backend, fullgraph=True + ) + assert compiled_fused.attn._o_scale_float is None + + result_fused = compiled_fused(q, k, v) + + if backend == AttentionBackendEnum.FLASHINFER: + # With the Flashinfer backend after the 1st round of the forward + # pass, output quant scale should be loaded into the attn layer's + # _o_scale_float, the 2nd round should reuse the loaded + # _o_scale_float + assert compiled_fused.attn._o_scale_float is not None + result_fused_2 = compiled_fused(q, k, v) + + assert compiled_fused.attn._o_scale_float is not None + + torch.testing.assert_close( + result_unfused, result_fused_2, atol=1e-2, rtol=1e-2 + ) + + # Check attn fusion support + quant_key: QuantKey = model_class.quant_key + attn_fusion_supported = [ + layer.impl.fused_output_quant_supported(quant_key) + for key, layer in vllm_config.compilation_config.static_forward_context.items() + ] + assert sum(attn_fusion_supported) == len(attn_fusion_supported), ( + "All layers should support attention fusion" + ) + + # Check quantization ops in the graph before and after fusion + quant_op = ( + torch.ops.aten.reciprocal + if "-quant_fp8" in custom_ops_list + else QUANT_OPS[quant_key] + ) + + # Note: for fp8, fully_replaced=False because query quant ops remain in graph. + # Only output quant ops are fused into attention. + test_backend.check_before_ops([quant_op], fully_replaced=quant_key is kNvfp4Dynamic) + + # access the underlying `AttnQuantFusionPass` on the `LazyInitPass` + assert attn_pass.pass_.matched_count == sum(attn_fusion_supported) + + # Check attention ops in the graph before and after fusion + attn_nodes_pre = list(find_op_nodes(ATTN_OP, test_backend.graph_pre_pass)) + attn_nodes_post = list(find_op_nodes(ATTN_OP, test_backend.graph_post_pass)) + + assert len(attn_nodes_pre) > 0, "Should have attention nodes before fusion" + assert len(attn_nodes_pre) == len(attn_nodes_post), ( + "Should have same number of attention nodes before and after fusion" + ) + assert attn_nodes_pre[0].kwargs.get("output_scale") is None, ( + "Attention should not have output_scale before fusion" + ) + assert attn_nodes_post[0].kwargs.get("output_scale") is not None, ( + "Attention should have output_scale after fusion" + ) + + assert attn_nodes_pre[0].kwargs.get("output_block_scale") is None, ( + "Attention should not have output_block_scale before fusion" + ) + + kv_cache_dummy_dep_pre_is_none = ( + attn_nodes_pre[0].kwargs.get("kv_cache_dummy_dep") is None + ) + kv_cache_dummy_dep_post_is_none = ( + attn_nodes_post[0].kwargs.get("kv_cache_dummy_dep") is None + ) + assert not (kv_cache_dummy_dep_pre_is_none ^ kv_cache_dummy_dep_post_is_none), ( + "The kv_cache_dummy_dep should be consistent before and after fusion" + ) + + if quant_key.dtype == FP8_DTYPE: + assert attn_nodes_post[0].kwargs.get("output_block_scale") is None, ( + "Attention should not have output_block_scale after FP8 fusion" + ) + elif quant_key.dtype == FP4_DTYPE: + assert attn_nodes_post[0].kwargs.get("output_block_scale") is not None, ( + "Attention should have output_block_scale after FP4 fusion" + ) + + # Check that results are close + torch.testing.assert_close(result_unfused, result_fused, atol=1e-2, rtol=1e-2) diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_mla_attn_quant_fusion.py b/asian_ai/performance/vLLM/tests/compile/passes/test_mla_attn_quant_fusion.py new file mode 100644 index 0000000000000000000000000000000000000000..80a71b5db0672703072bdc38e1d6e57090040666 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_mla_attn_quant_fusion.py @@ -0,0 +1,587 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import copy + +import pytest +import torch._dynamo + +from tests.compile.backend import LazyInitPass, TestBackend +from tests.utils import TestFP8Layer, flat_product +from tests.v1.attention.utils import BatchSpec, create_common_attn_metadata +from vllm._custom_ops import cutlass_scaled_fp4_mm, scaled_fp4_quant +from vllm.compilation.passes.fusion.matcher_utils import QUANT_OPS +from vllm.compilation.passes.fusion.mla_attn_quant_fusion import ( + MLA_ATTN_OP, + MLAAttnQuantFusionPass, +) +from vllm.compilation.passes.fx_utils import find_op_nodes +from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass +from vllm.compilation.passes.utility.post_cleanup import PostCleanupPass +from vllm.config import ( + AttentionConfig, + CacheConfig, + CompilationConfig, + CompilationMode, + ModelConfig, + PassConfig, + SchedulerConfig, + VllmConfig, + set_current_vllm_config, +) +from vllm.forward_context import get_forward_context, set_forward_context +from vllm.model_executor.kernels.linear.scaled_mm.cutlass import ( + CutlassFp8BlockScaledMMKernel, +) +from vllm.model_executor.layers.attention import MLAAttention +from vllm.model_executor.layers.linear import ColumnParallelLinear +from vllm.model_executor.layers.quantization.fp8 import Fp8Config +from vllm.model_executor.layers.quantization.modelopt import ModelOptNvFp4Config +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + GroupShape, + QuantKey, + create_fp8_quant_key, + kFp8Dynamic128Sym, + kFp8StaticTensorSym, + kNvfp4Dynamic, +) +from vllm.platforms import current_platform +from vllm.v1.attention.backend import AttentionMetadata +from vllm.v1.attention.backends.registry import AttentionBackendEnum +from vllm.v1.kv_cache_interface import MLAAttentionSpec + +FP8_DTYPE = current_platform.fp8_dtype() +FP4_DTYPE = torch.uint8 +DEVICE_TYPE = current_platform.device_type + + +class MLAAttentionQuantPatternModel(torch.nn.Module): + """Base model for MLA AttentionQuantPattern fusion.""" + + def __init__( + self, + num_heads: int, + qk_nope_head_dim: int, + qk_rope_head_dim: int, + v_head_dim: int, + kv_lora_rank: int, + kv_cache_dtype: torch.dtype, + device: torch.device, + vllm_config: VllmConfig, + **kwargs, + ): + super().__init__() + self.num_heads = num_heads + self.qk_nope_head_dim = qk_nope_head_dim + self.qk_rope_head_dim = qk_rope_head_dim + self.qk_head_dim = qk_nope_head_dim + qk_rope_head_dim + self.v_head_dim = v_head_dim + self.kv_lora_rank = kv_lora_rank + self.output_dim = num_heads * v_head_dim + self.head_size = kv_lora_rank + qk_rope_head_dim + self.kv_cache_dtype = kv_cache_dtype + self.device = device + self.vllm_config = vllm_config + self.dtype = vllm_config.model_config.dtype + + kv_b_proj = ColumnParallelLinear( + input_size=kv_lora_rank, + output_size=num_heads * (qk_nope_head_dim + v_head_dim), + bias=False, + prefix="model.layers.0.self_attn.kv_b_proj", + ).to(device) + kv_b_proj_weight = kwargs.get("kv_b_proj_weight") + if kv_b_proj_weight is not None: + kv_b_proj.weight.data.copy_(kv_b_proj_weight) + else: + kv_b_proj.weight.data.normal_() + + # Create MLAAttention + self.mla_attn = MLAAttention( + num_heads=num_heads, + scale=1.0 / (self.qk_head_dim**0.5), + qk_nope_head_dim=qk_nope_head_dim, + qk_rope_head_dim=qk_rope_head_dim, + v_head_dim=v_head_dim, + q_lora_rank=None, + kv_lora_rank=kv_lora_rank, + kv_b_proj=kv_b_proj, + cache_config=vllm_config.cache_config, + quant_config=self.quant_config, + prefix="model.layers.0.self_attn.attn", + ) + self.mla_attn._k_scale = self.mla_attn._k_scale.to(device) + self.mla_attn._v_scale = self.mla_attn._v_scale.to(device) + + # Initialize W_UK_T and W_UV from kv_b_proj weights + self.mla_attn.process_weights_after_loading(torch.get_default_dtype()) + self.kv_b_proj_weight = kv_b_proj.weight.data.clone() + + self.block_size = 16 + + # Initialize MLA MetadataBuilder + self.builder = self.mla_attn.attn_backend.get_builder_cls()( + kv_cache_spec=MLAAttentionSpec( + block_size=self.block_size, + num_kv_heads=1, + head_size=self.head_size, + dtype=self.kv_cache_dtype, + ), + layer_names=[self.mla_attn.layer_name], + vllm_config=self.vllm_config, + device=self.device, + ) + + def build_attn_metadata(self, batch_size: int) -> AttentionMetadata: + """Initialize MLA attention metadata. + + NOTE: Uses decode-only batch (query_len=1 per request). The prefill + (forward_mha) path is not separately tested here because it requires + FlashAttention availability and different input tensor shapes. The + quant logic in forward_impl is identical for both paths — it quantizes + the full output[:num_actual_toks] buffer after both forward_mha and + forward_mqa have written their results. + """ + + batch_spec = BatchSpec(seq_lens=[1] * batch_size, query_lens=[1] * batch_size) + common_attn_metadata = create_common_attn_metadata( + batch_spec, self.block_size, self.device, arange_block_indices=True + ) + + max_blocks = (max(batch_spec.seq_lens) + self.block_size - 1) // self.block_size + num_blocks = batch_size * max_blocks + + # MLA KV cache is 3D: (num_blocks, block_size, head_size) + attn_backend = self.mla_attn.attn_backend + kv_cache_shape = attn_backend.get_kv_cache_shape( + num_blocks, self.block_size, 1, self.head_size + ) + try: + kv_cache_stride_order = attn_backend.get_kv_cache_stride_order() + except (AttributeError, NotImplementedError): + kv_cache_stride_order = tuple(range(len(kv_cache_shape))) + + ordered_shape = tuple(kv_cache_shape[i] for i in kv_cache_stride_order) + inv_order = [ + kv_cache_stride_order.index(i) for i in range(len(kv_cache_stride_order)) + ] + + raw_tensor = torch.zeros( + ordered_shape, dtype=self.kv_cache_dtype, device=self.device + ) + kv_cache = raw_tensor.permute(*inv_order) + + self.mla_attn.kv_cache = kv_cache + + self.attn_metadata = self.builder.build( + common_prefix_len=0, common_attn_metadata=common_attn_metadata + ) + + return self.attn_metadata + + +class TestMLAAttentionFp8StaticQuantPatternModel(MLAAttentionQuantPatternModel): + """Test model for MLA Attention + FP8 static quant fusion.""" + + quant_key = kFp8StaticTensorSym + quant_config = Fp8Config() + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + + self.fp8_linear = TestFP8Layer( + weight_shape=(self.output_dim, self.output_dim), + activation_quant_key=self.quant_key, + weight_quant_key=self.quant_key, + device=self.device, + input_dtype=self.dtype, + ) + + w = kwargs.get("w") + if w is not None: + self.fp8_linear.weight = w["weight"] + self.fp8_linear.weight_scale = w["wscale"] + self.fp8_linear.input_scale = w["scale"] + + self.w = { + "weight": self.fp8_linear.weight, + "wscale": self.fp8_linear.weight_scale, + "scale": self.fp8_linear.input_scale, + } + + def forward( + self, + q: torch.Tensor, + kv_c_normed: torch.Tensor, + k_pe: torch.Tensor, + ): + """Forward pass that creates the MLA attention + FP8 quant pattern.""" + attn_output = self.mla_attn( + q, + kv_c_normed, + k_pe, + output_shape=(q.shape[0], self.output_dim), + ) + return self.fp8_linear(attn_output) + + +class TestMLAAttentionNvfp4QuantPatternModel(MLAAttentionQuantPatternModel): + """Test model for MLA Attention + NVFP4 quant fusion.""" + + quant_key = kNvfp4Dynamic + quant_config = ModelOptNvFp4Config( + is_checkpoint_nvfp4_serialized=False, + kv_cache_quant_algo=None, + exclude_modules=[], + ) + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + + self.w = kwargs.get( + "w", + { + "weight": torch.randint( + 256, + (self.output_dim, self.output_dim // 2), + dtype=FP4_DTYPE, + device=self.device, + ), + "wscale_swizzled": torch.randn( + self.output_dim, self.output_dim // 16 + ).to(dtype=FP8_DTYPE, device=self.device), + "wscale": torch.tensor([500], dtype=torch.float32, device=self.device), + "scale": torch.tensor([0.002], dtype=torch.float32, device=self.device), + }, + ) + + def forward( + self, + q: torch.Tensor, + kv_c_normed: torch.Tensor, + k_pe: torch.Tensor, + ): + """Forward pass that creates the MLA attention + NVFP4 quant pattern.""" + attn_output = self.mla_attn( + q, + kv_c_normed, + k_pe, + output_shape=(q.shape[0], self.output_dim), + ) + quant_output, output_block_scale = scaled_fp4_quant( + attn_output, 1 / self.w["scale"] + ) + return cutlass_scaled_fp4_mm( + a=quant_output, + b=self.w["weight"], + block_scale_a=output_block_scale, + block_scale_b=self.w["wscale_swizzled"], + alpha=self.w["scale"] * self.w["wscale"], + out_dtype=attn_output.dtype, + ) + + +class TestMLAAttentionFp8GroupQuantPatternModel(MLAAttentionQuantPatternModel): + """Test model for MLA Attention + per-group FP8 (block quant) fusion.""" + + quant_key = kFp8Dynamic128Sym + quant_config = Fp8Config( + is_checkpoint_fp8_serialized=True, + weight_block_size=[128, 128], + ) + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + + weight_quant_key = create_fp8_quant_key( + static=True, group_shape=GroupShape(128, 128) + ) + device = kwargs.get("device", torch.device("cuda:0")) + + # Subclass to set weight_block_size before process_weights_after_loading + class _BlockFP8Layer(TestFP8Layer): + def __init__(self, *a, **kw): + self.weight_block_size = [128, 128] + super().__init__(*a, **kw) + + # Force CutlassFp8BlockScaledMMKernel to ensure the graph uses + # per_token_group_fp8_quant (not the deepgemm packed variant). + self.block_fp8_linear = _BlockFP8Layer( + weight_shape=(self.output_dim, self.output_dim), + activation_quant_key=self.quant_key, + weight_quant_key=weight_quant_key, + input_dtype=self.dtype, + device=device, + force_kernel=CutlassFp8BlockScaledMMKernel, + ) + + w = kwargs.get("w") + if w is not None: + self.block_fp8_linear.weight = w["weight"] + # Block-wise uses weight_scale_inv, not weight_scale + self.block_fp8_linear.weight_scale_inv = w["wscale"] + + self.w = { + "weight": self.block_fp8_linear.weight, + "wscale": self.block_fp8_linear.weight_scale_inv, + } + + def forward( + self, + q: torch.Tensor, + kv_c_normed: torch.Tensor, + k_pe: torch.Tensor, + ): + """Forward pass: MLA attention -> block FP8 linear (group quant).""" + attn_output = self.mla_attn( + q, + kv_c_normed, + k_pe, + output_shape=(q.shape[0], self.output_dim), + ) + return self.block_fp8_linear(attn_output) + + +def is_nvfp4_supported(): + return current_platform.has_device_capability(100) + + +# MLA test configuration +MLA_DIMS: list[tuple[int, int, int, int, int]] = [] +PATTERN_TEST_MODELS_MLA_FP8: list[tuple[str, type]] = [] +PATTERN_TEST_MODELS_MLA_GROUP_FP8: list[tuple[str, type]] = [] +PATTERN_TEST_MODELS_MLA_FP4: list[tuple[str, type]] = [] +BACKENDS_MLA_FP8: list[AttentionBackendEnum] = [] +BACKENDS_MLA_FP4: list[AttentionBackendEnum] = [] + +if current_platform.is_cuda(): + # (num_heads, qk_nope_head_dim, qk_rope_head_dim, v_head_dim, kv_lora_rank) + MLA_DIMS = [(16, 128, 64, 128, 512)] + PATTERN_TEST_MODELS_MLA_FP8 = [ + ( + "deepseek-ai/DeepSeek-V2-Lite", + TestMLAAttentionFp8StaticQuantPatternModel, + ) + ] + PATTERN_TEST_MODELS_MLA_GROUP_FP8 = [ + ( + "deepseek-ai/DeepSeek-V3", + TestMLAAttentionFp8GroupQuantPatternModel, + ) + ] + PATTERN_TEST_MODELS_MLA_FP4 = [ + ( + "deepseek-ai/DeepSeek-V2-Lite", + TestMLAAttentionNvfp4QuantPatternModel, + ) + ] + BACKENDS_MLA_FP8 = [AttentionBackendEnum.TRITON_MLA] + BACKENDS_MLA_FP4 = [AttentionBackendEnum.TRITON_MLA] + + +@pytest.mark.parametrize( + "num_heads, qk_nope_head_dim, qk_rope_head_dim, v_head_dim, kv_lora_rank", + MLA_DIMS, +) +@pytest.mark.parametrize("batch_size", [7, 256] if current_platform.is_cuda() else [8]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize( + "backend, model_name, model_class, custom_ops", + list( + flat_product( + BACKENDS_MLA_FP8, + PATTERN_TEST_MODELS_MLA_FP8, + ["+quant_fp8", "-quant_fp8"], + ) + ) + + list( + flat_product( + BACKENDS_MLA_FP8, + PATTERN_TEST_MODELS_MLA_GROUP_FP8, + ["+quant_fp8"], + ) + ) + + list(flat_product(BACKENDS_MLA_FP4, PATTERN_TEST_MODELS_MLA_FP4, [""])), +) +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), reason="Only test ROCm or CUDA" +) +@pytest.mark.skipif(not current_platform.supports_fp8(), reason="Need FP8") +def test_mla_attention_quant_pattern( + num_heads: int, + qk_nope_head_dim: int, + qk_rope_head_dim: int, + v_head_dim: int, + kv_lora_rank: int, + batch_size: int, + dtype: torch.dtype, + custom_ops: str, + model_name: str, + model_class: type[MLAAttentionQuantPatternModel], + backend: AttentionBackendEnum, + dist_init, + monkeypatch, + use_fresh_inductor_cache, +): + """Test MLA AttentionQuantPattern fusion pass""" + if ( + model_class is TestMLAAttentionNvfp4QuantPatternModel + and not is_nvfp4_supported() + ): + pytest.skip("NVFP4 is not supported on this GPU (requires SM 100+).") + + monkeypatch.setenv("VLLM_DISABLE_COMPILE_CACHE", "1") + + custom_ops_list = custom_ops.split(",") if custom_ops else [] + + device = torch.device(f"{DEVICE_TYPE}:0") + torch.set_default_dtype(dtype) + torch.manual_seed(42) + + model_config = ModelConfig( + model=model_name, + max_model_len=2048, + dtype=dtype, + ) + vllm_config = VllmConfig( + model_config=model_config, + scheduler_config=SchedulerConfig( + max_num_seqs=1024, + max_model_len=model_config.max_model_len, + is_encoder_decoder=model_config.is_encoder_decoder, + ), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=custom_ops_list, + ), + cache_config=CacheConfig(cache_dtype="auto"), + attention_config=AttentionConfig(backend=backend), + ) + + # MLA inputs: q(B, N, qk_head_dim), kv_c_normed(B, L), k_pe(B, 1, R) + qk_head_dim = qk_nope_head_dim + qk_rope_head_dim + q = torch.randn(batch_size, num_heads, qk_head_dim, dtype=dtype, device=device) + kv_c_normed = torch.randn(batch_size, kv_lora_rank, dtype=dtype, device=device) + k_pe = torch.randn(batch_size, 1, qk_rope_head_dim, dtype=dtype, device=device) + + # Mark first dimension as dynamic + torch._dynamo.mark_dynamic(q, 0) + torch._dynamo.mark_dynamic(kv_c_normed, 0) + torch._dynamo.mark_dynamic(k_pe, 0) + + # Run model without fusion + vllm_config_unfused = copy.deepcopy(vllm_config) + with ( + set_current_vllm_config(vllm_config_unfused), + set_forward_context(attn_metadata=None, vllm_config=vllm_config_unfused), + ): + model_unfused = model_class( + num_heads=num_heads, + qk_nope_head_dim=qk_nope_head_dim, + qk_rope_head_dim=qk_rope_head_dim, + v_head_dim=v_head_dim, + kv_lora_rank=kv_lora_rank, + kv_cache_dtype=dtype, + device=device, + vllm_config=vllm_config_unfused, + ) + model_unfused = model_unfused.to(device) + # HACK: See #131044 + result_unfused_0 = model_unfused(q, kv_c_normed, k_pe) # noqa: F841 + + forward_ctx = get_forward_context() + forward_ctx.attn_metadata = model_unfused.build_attn_metadata(batch_size) + + compiled_unfused = torch.compile(model_unfused, fullgraph=True) + result_unfused = compiled_unfused(q, kv_c_normed, k_pe) + + # Run model with attn fusion enabled + vllm_config.compilation_config.pass_config = PassConfig( + fuse_attn_quant=True, eliminate_noops=True + ) + with ( + set_current_vllm_config(vllm_config), + set_forward_context(attn_metadata=None, vllm_config=vllm_config), + ): + model_fused = model_class( + num_heads=num_heads, + qk_nope_head_dim=qk_nope_head_dim, + qk_rope_head_dim=qk_rope_head_dim, + v_head_dim=v_head_dim, + kv_lora_rank=kv_lora_rank, + kv_cache_dtype=dtype, + device=device, + vllm_config=vllm_config, + w=model_unfused.w, + kv_b_proj_weight=model_unfused.kv_b_proj_weight, + ) + model_fused = model_fused.to(device) + + forward_ctx = get_forward_context() + forward_ctx.attn_metadata = model_fused.build_attn_metadata(batch_size) + + # Create test backend with fusion passes + noop_pass = NoOpEliminationPass(vllm_config) + attn_pass = LazyInitPass(MLAAttnQuantFusionPass, vllm_config) + cleanup_pass = PostCleanupPass(vllm_config) + + test_backend = TestBackend(noop_pass, attn_pass, cleanup_pass) + # HACK: See https://github.com/vllm-project/vllm/issues/31044 + result_fused_0 = model_fused(q, kv_c_normed, k_pe) # noqa: F841 + + compiled_fused = torch.compile( + model_fused, backend=test_backend, fullgraph=True + ) + + result_fused = compiled_fused(q, kv_c_normed, k_pe) + + # Check attn fusion support + quant_key: QuantKey = model_class.quant_key + attn_fusion_supported = [ + layer.impl.fused_output_quant_supported(quant_key) + for key, layer in vllm_config.compilation_config.static_forward_context.items() + if isinstance(layer, MLAAttention) + ] + assert sum(attn_fusion_supported) == len(attn_fusion_supported), ( + "All MLA layers should support attention fusion" + ) + + # Check quantization ops in the graph + is_per_group = quant_key.scale.group_shape.is_per_group() + quant_op = ( + torch.ops.aten.reciprocal + if "-quant_fp8" in custom_ops_list + else QUANT_OPS[quant_key] + ) + test_backend.check_before_ops([quant_op], fully_replaced=is_per_group) + + assert attn_pass.pass_.matched_count == sum(attn_fusion_supported) + + # Check MLA attention ops in the graph + attn_nodes_pre = list(find_op_nodes(MLA_ATTN_OP, test_backend.graph_pre_pass)) + attn_nodes_post = list(find_op_nodes(MLA_ATTN_OP, test_backend.graph_post_pass)) + + assert len(attn_nodes_pre) > 0, "Should have MLA attention nodes before fusion" + assert len(attn_nodes_pre) == len(attn_nodes_post), ( + "Should have same number of MLA attention nodes before and after fusion" + ) + + # Before fusion: neither scale should be set + assert attn_nodes_pre[0].kwargs.get("output_scale") is None + assert attn_nodes_pre[0].kwargs.get("output_block_scale") is None + + # After fusion: derive expected scale presence from quant_key properties. + # - output_scale: present for static quant or non-FP8 (NVFP4 carries input_scale) + # - output_block_scale: present when quant uses per-group/block scaling + has_output_scale = attn_nodes_post[0].kwargs.get("output_scale") is not None + has_block_scale = attn_nodes_post[0].kwargs.get("output_block_scale") is not None + + expects_output_scale = quant_key.scale.static or quant_key.dtype != FP8_DTYPE + assert has_output_scale == expects_output_scale, ( + f"output_scale: expected present={expects_output_scale}, got {has_output_scale}" + ) + assert has_block_scale == is_per_group, ( + f"output_block_scale: expected present={is_per_group}, got {has_block_scale}" + ) + + # Check numerical correctness + torch.testing.assert_close(result_unfused, result_fused, atol=1e-2, rtol=1e-2) diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_mla_rope_kvcache_cat_fusion.py b/asian_ai/performance/vLLM/tests/compile/passes/test_mla_rope_kvcache_cat_fusion.py new file mode 100644 index 0000000000000000000000000000000000000000..545000353592d73a8ba2fbb023b86452f7f8f3f1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_mla_rope_kvcache_cat_fusion.py @@ -0,0 +1,413 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +import vllm.config +from tests.compile.backend import TestBackend +from tests.v1.attention.utils import BatchSpec, create_common_attn_metadata +from vllm._aiter_ops import is_aiter_found_and_supported, rocm_aiter_ops +from vllm.compilation.passes.fusion.mla_rope_kvcache_cat_fusion import ( + MLARoPEKVCacheCatFusionPass, +) +from vllm.compilation.passes.utility.fix_functionalization import ( + FixFunctionalizationPass, +) +from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass +from vllm.compilation.passes.utility.post_cleanup import PostCleanupPass +from vllm.config import ( + CacheConfig, + CompilationConfig, + CompilationMode, + ModelConfig, + PassConfig, + VllmConfig, +) +from vllm.forward_context import get_forward_context, set_forward_context +from vllm.model_executor.layers.attention import MLAAttention +from vllm.model_executor.layers.linear import ColumnParallelLinear +from vllm.model_executor.layers.rotary_embedding import ( + DeepseekScalingRotaryEmbedding, + RotaryEmbedding, +) +from vllm.platforms import current_platform +from vllm.utils.torch_utils import _encode_layer_name +from vllm.v1.attention.backend import ( + AttentionBackend, + CommonAttentionMetadata, +) +from vllm.v1.attention.backends.fa_utils import flash_attn_supports_mla +from vllm.v1.attention.backends.registry import AttentionBackendEnum + +INDEX_SELECT_OP = torch.ops.aten.index.Tensor +VLLM_UNIFIED_MLA_KV_CACHE_UPDATE_OP = torch.ops.vllm.unified_mla_kv_cache_update +FP8_DTYPE = current_platform.fp8_dtype() + + +class MLARoPEKVCacheCatTestModel(torch.nn.Module): + def __init__( + self, + vllm_config: VllmConfig, + attn_backend: AttentionBackendEnum, + use_deepseek_scaling_rope: bool, + num_heads: int, + qk_nope_head_dim: int, + qk_rope_head_dim: int, + v_head_dim: int, + q_lora_rank: int, + kv_lora_rank: int, + is_neox: bool, + dtype: torch.dtype, + device: torch.device, + prefix: str = "model.layers.0.self_attn.attn", + ): + super().__init__() + self.num_heads = num_heads + self.qk_nope_head_dim = qk_nope_head_dim + self.qk_rope_head_dim = qk_rope_head_dim + self.qk_head_dim = qk_nope_head_dim + qk_rope_head_dim + self.v_head_dim = v_head_dim + self.q_lora_rank = q_lora_rank + self.kv_lora_rank = kv_lora_rank + self.dtype = dtype + self.device = device + self.layer_name = prefix + + self.num_kv_heads = 1 + self.head_size = kv_lora_rank + qk_rope_head_dim + self.block_size = vllm_config.cache_config.block_size + self.scale = self.qk_head_dim**-0.5 + + if use_deepseek_scaling_rope: + self.rotary_emb = DeepseekScalingRotaryEmbedding( + head_size=qk_rope_head_dim, + rotary_dim=qk_rope_head_dim, + max_position_embeddings=4096, + base=10000, + is_neox_style=is_neox, + scaling_factor=1.0, + dtype=dtype, + ) + else: + self.rotary_emb = RotaryEmbedding( + head_size=qk_rope_head_dim, + rotary_dim=qk_rope_head_dim, + max_position_embeddings=4096, + base=10000, + is_neox_style=is_neox, + dtype=dtype, + ) + + # Initialize intermediate mm layers for unit test + self.q_b_proj = ColumnParallelLinear( + self.q_lora_rank, + self.num_heads * self.qk_head_dim, + bias=False, + prefix=f"{prefix}.q_b_proj", + ).to(device) + self.kv_b_proj = ColumnParallelLinear( + self.kv_lora_rank, + self.num_heads * (self.qk_nope_head_dim + self.v_head_dim), + bias=False, + prefix=f"{prefix}.kv_b_proj", + ).to(device) + + # ColumnParallelLinear default init in bf16 with seed 0 produces + # near-zero weights (7/4.7M nonzero), making the GEMM output almost + # entirely zero and masking correctness bugs. Reinitialize to get + # dense outputs. + with torch.no_grad(): + torch.nn.init.normal_(self.q_b_proj.weight, std=0.02) + torch.nn.init.normal_(self.kv_b_proj.weight, std=0.02) + + # Register layer metadata for the fusion pass via MLAAttention + self.mla_attn = MLAAttention( + num_heads=self.num_heads, + scale=self.scale, + qk_nope_head_dim=self.qk_nope_head_dim, + qk_rope_head_dim=self.qk_rope_head_dim, + v_head_dim=self.v_head_dim, + q_lora_rank=self.q_lora_rank, + kv_lora_rank=self.kv_lora_rank, + kv_b_proj=self.kv_b_proj, + cache_config=vllm_config.cache_config, + quant_config=vllm_config.quant_config, + prefix=prefix, + attn_backend=attn_backend.get_class(), + ) + self.attn_backend: type[AttentionBackend] = self.mla_attn.get_attn_backend() + self.mla_attn._k_scale = self.mla_attn._k_scale.to(device) + self.mla_attn._v_scale = self.mla_attn._v_scale.to(device) + + # Keep both the string dtype (for ops) and torch dtype (for tensors) + self.kv_cache_dtype_str = vllm_config.cache_config.cache_dtype + self.kv_cache_dtype = ( + FP8_DTYPE if self.kv_cache_dtype_str.startswith("fp8") else self.dtype + ) + + # Initialize attn MetadataBuilder + self.builder = self.attn_backend.get_builder_cls()( + kv_cache_spec=self.mla_attn.get_kv_cache_spec(vllm_config), + layer_names=[self.mla_attn.layer_name], + vllm_config=vllm_config, + device=device, + ) + + def build_attn_metadata(self, batch_size: int) -> CommonAttentionMetadata: + """Initialize attention metadata.""" + # Create common attn metadata + batch_spec = BatchSpec(seq_lens=[1] * batch_size, query_lens=[1] * batch_size) + common_attn_metadata = create_common_attn_metadata( + batch_spec, self.block_size, self.device, arange_block_indices=True + ) + + max_blocks = (max(batch_spec.seq_lens) + self.block_size - 1) // self.block_size + num_blocks = batch_size * max_blocks + + # Fetch the attention backend and kv cache shape and stride order + kv_cache_shape = self.attn_backend.get_kv_cache_shape( + num_blocks, self.block_size, self.num_kv_heads, self.head_size + ) + try: + kv_cache_stride_order = self.attn_backend.get_kv_cache_stride_order() + except (AttributeError, NotImplementedError): + kv_cache_stride_order = tuple(range(len(kv_cache_shape))) + + kv_cache_shape = tuple(kv_cache_shape[i] for i in kv_cache_stride_order) + inv_order = [ + kv_cache_stride_order.index(i) for i in range(len(kv_cache_stride_order)) + ] + + raw_tensor = torch.zeros( + num_blocks * self.block_size * self.num_kv_heads * self.head_size, + dtype=self.kv_cache_dtype, + device=self.device, + ) + raw_tensor = raw_tensor.view(kv_cache_shape) + kv_cache = raw_tensor.permute(*inv_order) + + self.mla_attn.kv_cache = kv_cache + + # Build attn metadata + attn_metadata = self.builder.build( + common_prefix_len=0, common_attn_metadata=common_attn_metadata + ) + + return attn_metadata + + def forward( + self, qkv_lora: torch.Tensor, positions: torch.Tensor + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + qkv_lora = qkv_lora.clone() + q_c, kv_lora = qkv_lora.split( + [self.q_lora_rank, self.kv_lora_rank + self.qk_rope_head_dim], + dim=-1, + ) + q = self.q_b_proj(q_c)[0] + kv_c, k_pe = kv_lora.split([self.kv_lora_rank, self.qk_rope_head_dim], dim=-1) + + q = q.view(-1, self.num_heads, self.qk_head_dim) + k_pe = k_pe.unsqueeze(1) + + q[..., self.qk_nope_head_dim :], k_pe = self.rotary_emb( + positions, q[..., self.qk_nope_head_dim :], k_pe + ) + + dummy = torch.ops.vllm.unified_mla_kv_cache_update( + kv_c, + k_pe, + _encode_layer_name(self.layer_name), + self.kv_cache_dtype_str, + self.mla_attn._k_scale, + ) + return q, kv_c, k_pe, dummy + + def ops_in_model_before(self) -> list[torch._ops.OpOverload]: + ops = [ + INDEX_SELECT_OP, + torch.ops.vllm.unified_mla_kv_cache_update.default, + ] + return ops + + def ops_in_model_after(self) -> list[torch._ops.OpOverload]: + return [torch.ops.vllm.fused_rope_unified_mla_kv_cache_update.default] + + +MLA_BACKENDS = [AttentionBackendEnum.TRITON_MLA] +if flash_attn_supports_mla(): + MLA_BACKENDS += [AttentionBackendEnum.FLASH_ATTN_MLA] +if is_aiter_found_and_supported(): + MLA_BACKENDS += [AttentionBackendEnum.ROCM_AITER_MLA] + + +@pytest.mark.parametrize("attn_backend", MLA_BACKENDS) +@pytest.mark.parametrize("use_deepseek_scaling_rope", [True]) +@pytest.mark.parametrize("num_heads", [16]) +@pytest.mark.parametrize("qk_nope_head_dim", [128]) +@pytest.mark.parametrize("qk_rope_head_dim", [64]) +@pytest.mark.parametrize("v_head_dim", [128]) +@pytest.mark.parametrize("q_lora_rank", [1536]) +@pytest.mark.parametrize("kv_lora_rank", [512]) +@pytest.mark.parametrize("block_size", [16]) +@pytest.mark.parametrize("is_neox", [True, False]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8"]) +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="MLA RoPE+KVCache+Cat fusion is only supported on CUDA and ROCm.", +) +def test_mla_rope_kvcache_cat_fusion( + attn_backend: AttentionBackendEnum, + use_deepseek_scaling_rope: bool, + num_heads: int, + qk_nope_head_dim: int, + qk_rope_head_dim: int, + v_head_dim: int, + q_lora_rank: int, + kv_lora_rank: int, + block_size: int, + is_neox: bool, + dtype: torch.dtype, + kv_cache_dtype: str, + monkeypatch: pytest.MonkeyPatch, +): + torch.set_default_device("cuda") + torch.set_default_dtype(dtype) + torch.manual_seed(0) + + vllm_config = VllmConfig( + model_config=ModelConfig( + model="deepseek-ai/DeepSeek-V2-Lite", + dtype=dtype, + ), + cache_config=CacheConfig( + block_size=block_size, + cache_dtype=kv_cache_dtype, + ), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + pass_config=PassConfig( + fuse_rope_kvcache_cat_mla=True, + eliminate_noops=True, + ), + ), + ) + + with vllm.config.set_current_vllm_config(vllm_config), monkeypatch.context() as m: + if not torch.distributed.is_initialized(): + from vllm.distributed.parallel_state import ( + init_distributed_environment, + initialize_model_parallel, + ) + from vllm.utils.system_utils import update_environment_variables + + update_environment_variables( + { + "RANK": "0", + "LOCAL_RANK": "0", + "WORLD_SIZE": "1", + "MASTER_ADDR": "localhost", + "MASTER_PORT": "54321", + } + ) + init_distributed_environment() + initialize_model_parallel() + + if attn_backend == AttentionBackendEnum.ROCM_AITER_MLA: + m.setenv("VLLM_ROCM_USE_AITER", "1") + rocm_aiter_ops.refresh_env_variables() + + model = MLARoPEKVCacheCatTestModel( + vllm_config=vllm_config, + attn_backend=attn_backend, + use_deepseek_scaling_rope=use_deepseek_scaling_rope, + num_heads=num_heads, + qk_nope_head_dim=qk_nope_head_dim, + qk_rope_head_dim=qk_rope_head_dim, + v_head_dim=v_head_dim, + q_lora_rank=q_lora_rank, + kv_lora_rank=kv_lora_rank, + is_neox=is_neox, + dtype=dtype, + device=torch.get_default_device(), + ) + + fusion_pass = MLARoPEKVCacheCatFusionPass(vllm_config) + # note: FixFunctionalizationPass is required to correctly lower + # the fused op to its inplace version with auto-functionalization v1. + # Without it, decompose_auto_functionalized calls clone_preserve_strides + # on the non-contiguous q_pe slice directly, and inductor's lowering + # of the resulting as_strided chain incorrectly drops the storage offset. + # auto-functionalization v2 avoids this: it clones the contiguous base + # tensor (_all_bases) and reconstructs the slice as a view, so the + # offset is never passed through as_strided lowering. + passes = [ + NoOpEliminationPass(vllm_config), + fusion_pass, + PostCleanupPass(vllm_config), + FixFunctionalizationPass(vllm_config), + ] + backend = TestBackend(*passes) + + T = 5 + + qkv_lora = torch.randn( + T, + q_lora_rank + kv_lora_rank + qk_rope_head_dim, + dtype=dtype, + ) + pos = torch.arange(T, dtype=torch.long) + + qkv_unfused = qkv_lora.clone() + pos_unfused = pos.clone() + + # Run unfused version + with set_forward_context(None, vllm_config): + forward_context = get_forward_context() + attn_metadata = model.build_attn_metadata(T) + forward_context.slot_mapping = { + model.layer_name: attn_metadata.slot_mapping + } + q_unfused, kv_c_unfused, k_pe_unfused, dummy = model( + qkv_unfused, pos_unfused + ) + attn_layer = forward_context.no_compile_layers[model.layer_name] + kv_cache_unfused = attn_layer.kv_cache.clone() + del dummy + + # Run fused version (compiled) + torch._dynamo.mark_dynamic(qkv_lora, 0) + torch._dynamo.mark_dynamic(pos, 0) + with set_forward_context(None, vllm_config): + model_fused = torch.compile(model, backend=backend) + forward_context = get_forward_context() + attn_metadata = model.build_attn_metadata(T) + forward_context.slot_mapping = { + model.layer_name: attn_metadata.slot_mapping + } + q_fused, kv_c_fused, k_pe_fused, dummy = model_fused(qkv_lora, pos) + attn_layer = forward_context.no_compile_layers[model.layer_name] + kv_cache_fused = attn_layer.kv_cache + del dummy + + assert fusion_pass.matched_count == 1 + + backend.check_before_ops(model.ops_in_model_before()) + backend.check_after_ops(model.ops_in_model_after()) + + if dtype == torch.float16: + ATOL, RTOL = (2e-3, 2e-3) + else: + ATOL, RTOL = (1e-2, 1e-2) + + torch.testing.assert_close(q_unfused, q_fused, atol=ATOL, rtol=RTOL) + torch.testing.assert_close(kv_c_unfused, kv_c_fused, atol=ATOL, rtol=RTOL) + torch.testing.assert_close(k_pe_unfused, k_pe_fused, atol=ATOL, rtol=RTOL) + # Cannot compare fp8_* directly here, cast to model dtype instead + torch.testing.assert_close( + kv_cache_unfused.view(dtype), + kv_cache_fused.view(dtype), + atol=ATOL, + rtol=RTOL, + ) diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_noop_elimination.py b/asian_ai/performance/vLLM/tests/compile/passes/test_noop_elimination.py new file mode 100644 index 0000000000000000000000000000000000000000..ee0ce268d32c1bde18fba775265de3a8ce8ffec4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_noop_elimination.py @@ -0,0 +1,120 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +import vllm +from tests.compile.backend import TestBackend +from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass +from vllm.config import CompilationConfig, CompilationMode, PassConfig, VllmConfig +from vllm.platforms import current_platform + +DEVICE_TYPE = current_platform.device_type + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32]) +# Important edge case is when `num_tokens == buffer_size` +@pytest.mark.parametrize( + ("num_tokens", "buffer_size"), [(256, 256), (256, 512), (1024, 1024), (1024, 1025)] +) +@pytest.mark.parametrize("hidden_size", [64, 4096]) +def test_noop_elimination(dtype, num_tokens, hidden_size, buffer_size): + torch.set_default_device(DEVICE_TYPE) + torch.set_default_dtype(dtype) + torch.manual_seed(1) + + class Model(torch.nn.Module): + def __init__(self) -> None: + super().__init__() + # Avoid using empty, since on rocm torch.empty + # does not initialize the memory. + self.pos_embed = torch.randn(buffer_size, hidden_size, dtype=dtype) + + def forward(self, x): + # Avoid += to prevent inplace addition. + x = x + self.pos_embed[: x.shape[0]] + # Chain of reshapes + y = x.reshape(-1, 128, 32) + z = y.reshape(-1, 4096) + # No-op reshape + a = z.reshape(-1, 4096) + # Final reshape that should remain + b = a.reshape(-1, 128, 32) + # No-op slice + c = b[0 : b.shape[0]] + # The pass should replace the result of this op with `c`. + d = torch.slice_scatter( + torch.ones_like(c), # Dummy tensor to be scattered into + c, # Source tensor + 0, # dim + 0, # start + c.shape[0], # end + ) + return d + + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + pass_config=PassConfig(eliminate_noops=True), + ) + ) + with vllm.config.set_current_vllm_config(vllm_config): + noop_pass = NoOpEliminationPass(vllm_config) + + backend = TestBackend(noop_pass) + + model = Model() + # First dimension dynamic + x = torch.rand(num_tokens, hidden_size) + torch._dynamo.mark_dynamic(x, 0) + + result = model(x) + + model2 = torch.compile(model, backend=backend) + result2 = model2(x) + + ATOL, RTOL = (2e-3, 2e-3) + torch.testing.assert_close(result, result2, atol=ATOL, rtol=RTOL) + + # The no-op reshape and slice should be eliminated. + # The initial slice on the positional embedding should remain. + # The chain of reshapes should be fused into a single reshape. + assert backend.op_count(torch.ops.aten.reshape.default) == 1 + assert backend.op_count(torch.ops.aten.slice.Tensor) == 1 + assert backend.op_count(torch.ops.aten.slice_scatter.default) == 0 + + +def test_non_noop_slice_preserved(): + """Ensure that a slice with end=-1 (dropping last row) is NOT eliminated. + + Regression test for a bug where end=-1 was treated like an inferred + dimension (reshape semantics) leading to incorrect elimination. + """ + torch.set_default_device(DEVICE_TYPE) + x = torch.randn(16, 16) + + class SliceModel(torch.nn.Module): + def forward(self, x): + base = x.clone() + src = torch.ones(15, 16) + y = torch.slice_scatter(base, src, dim=0, start=0, end=-1) + return x[0:-1, :], y + + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + pass_config=PassConfig(eliminate_noops=True), + ) + ) + with vllm.config.set_current_vllm_config(vllm_config): + noop_pass = NoOpEliminationPass(vllm_config) + backend = TestBackend(noop_pass) + model = SliceModel() + ref = model(x) + compiled = torch.compile(model, backend=backend) + out = compiled(x) + torch.testing.assert_close(ref, out) + # The slice should remain (not a no-op). + assert backend.op_count(torch.ops.aten.slice.Tensor) == 1 + assert backend.op_count(torch.ops.aten.slice_scatter.default) == 1 diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_pass_manager.py b/asian_ai/performance/vLLM/tests/compile/passes/test_pass_manager.py new file mode 100644 index 0000000000000000000000000000000000000000..634e848d7dea141ba18026cddecb76ce64ad2bfc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_pass_manager.py @@ -0,0 +1,83 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import copy + +import pytest +import torch + +from vllm.compilation.passes.inductor_pass import ( + CallableInductorPass, + InductorPass, + pass_context, +) +from vllm.compilation.passes.pass_manager import PostGradPassManager +from vllm.config import ModelConfig, VllmConfig +from vllm.config.utils import Range + + +# dummy custom pass that doesn't inherit +def simple_callable(graph: torch.fx.Graph): + pass + + +# Should fail to add directly to the pass manager +def test_bad_callable(): + config = VllmConfig() + + pass_manager = PostGradPassManager() + pass_manager.configure(config) + + with pytest.raises(AssertionError): + pass_manager.add(simple_callable) # type: ignore[arg-type] + + +# Pass that inherits from InductorPass +class ProperPass(InductorPass): + def __call__(self, graph: torch.fx.graph.Graph) -> None: + pass + + +@pytest.mark.parametrize( + "callable", + [ + ProperPass(), + # Can also wrap callables in CallableInductorPass for compliance + CallableInductorPass(simple_callable), + CallableInductorPass(simple_callable, InductorPass.hash_source(__file__)), + ], +) +def test_pass_manager_uuid(callable): + # Set the pass context as PassManager uuid uses it + with pass_context(Range(start=1, end=8)): + # Some passes need dtype to be set + config = VllmConfig(model_config=ModelConfig(dtype=torch.bfloat16)) + + pass_manager = PostGradPassManager() + pass_manager.configure(config) + + # Check that UUID is different if the same pass is added 2x + pass_manager.add(callable) + uuid1 = pass_manager.uuid() + pass_manager.add(callable) + uuid2 = pass_manager.uuid() + assert uuid1 != uuid2 + + # UUID should be the same as the original one, + # as we constructed in the same way. + pass_manager2 = PostGradPassManager() + pass_manager2.configure(config) + pass_manager2.add(callable) + assert uuid1 == pass_manager2.uuid() + + # UUID should be different due to config change + config2 = copy.deepcopy(config) + config2.compilation_config.pass_config.fuse_norm_quant = ( + not config2.compilation_config.pass_config.fuse_norm_quant + ) + config2.compilation_config.pass_config.fuse_act_quant = ( + not config2.compilation_config.pass_config.fuse_act_quant + ) + pass_manager3 = PostGradPassManager() + pass_manager3.configure(config2) + pass_manager3.add(callable) + assert uuid1 != pass_manager3.uuid() diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_qk_norm_rope_fusion.py b/asian_ai/performance/vLLM/tests/compile/passes/test_qk_norm_rope_fusion.py new file mode 100644 index 0000000000000000000000000000000000000000..117730216b189d0cc7bf71561eb27cfb8935811c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_qk_norm_rope_fusion.py @@ -0,0 +1,214 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch +from torch._ops import OpOverload, OpOverloadPacket + +from tests.compile.backend import TestBackend +from vllm.compilation.passes.fusion.matcher_utils import ( + FLASHINFER_ROTARY_OP, + ROTARY_OP, +) +from vllm.compilation.passes.fusion.qk_norm_rope_fusion import ( + FUSED_QK_ROPE_OP, + QKNormRoPEFusionPass, +) +from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass +from vllm.compilation.passes.utility.post_cleanup import PostCleanupPass +from vllm.compilation.passes.utility.split_coalescing import SplitCoalescingPass +from vllm.config import ( + CompilationConfig, + CompilationMode, + ModelConfig, + PassConfig, + VllmConfig, + set_current_vllm_config, +) +from vllm.model_executor.layers.attention import Attention +from vllm.model_executor.layers.layernorm import RMSNorm +from vllm.model_executor.layers.rotary_embedding import RotaryEmbedding +from vllm.platforms import current_platform +from vllm.v1.attention.backend import AttentionType + +RSQRT_OP = torch.ops.aten.rsqrt.default +INDEX_SELECT_OP = torch.ops.aten.index.Tensor + + +class QKNormRoPETestModel(torch.nn.Module): + def __init__( + self, + *, + num_heads: int, + num_kv_heads: int, + head_dim: int, + eps: float, + is_neox: bool, + vllm_config: VllmConfig, + dtype: torch.dtype, + test_scattered_split: bool = False, + prefix: str = "model.layers.0.self_attn.attn", + ) -> None: + super().__init__() + self.num_heads = num_heads + self.num_kv_heads = num_kv_heads + self.head_dim = head_dim + self.q_size = num_heads * head_dim + self.kv_size = num_kv_heads * head_dim + self.rotary_dim = head_dim + self.eps = eps + self.dtype = dtype + + # Register layer metadata for the fusion pass via Attention. + self.attn = Attention( + num_heads=self.num_heads, + head_size=self.head_dim, + scale=1.0 / self.head_dim**0.5, + num_kv_heads=self.num_kv_heads, + cache_config=vllm_config.cache_config, + prefix=prefix, + attn_type=AttentionType.DECODER, + ) + + self.q_norm = RMSNorm(self.head_dim, eps=self.eps) + self.k_norm = RMSNorm(self.head_dim, eps=self.eps) + self.rotary_emb = RotaryEmbedding( + self.head_dim, + rotary_dim=self.rotary_dim, + max_position_embeddings=4096, + base=10000, + is_neox_style=is_neox, + dtype=self.dtype, + ) + self.test_scattered_split = test_scattered_split + self.enable_rms_norm_custom_op = self.q_norm.enabled() + self.enable_rope_custom_op = self.rotary_emb.enabled() + + def forward(self, qkv: torch.Tensor, positions: torch.Tensor): + if self.test_scattered_split: + q, _, _ = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1) + _, k, _ = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1) + _, _, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1) + else: + q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1) + q_by_head = q.view(*q.shape[:-1], q.shape[-1] // self.head_dim, self.head_dim) + q_by_head = self.q_norm(q_by_head) + q = q_by_head.view(q.shape) + k_by_head = k.view(*k.shape[:-1], k.shape[-1] // self.head_dim, self.head_dim) + k_by_head = self.k_norm(k_by_head) + k = k_by_head.view(k.shape) + q, k = self.rotary_emb(positions, q, k) + return q, k, v + + def ops_in_model_before(self) -> list[OpOverload | OpOverloadPacket]: + ops: list[OpOverload | OpOverloadPacket] = [torch.ops.vllm_ir.rms_norm] + if self.enable_rope_custom_op: + if self.rotary_emb.use_flashinfer: + ops.append(FLASHINFER_ROTARY_OP) + else: + ops.append(ROTARY_OP) + else: + ops.append(INDEX_SELECT_OP) + return ops + + def ops_in_model_after(self) -> list[OpOverload | OpOverloadPacket]: + return [FUSED_QK_ROPE_OP] + + +@pytest.mark.parametrize("scattered_split", [True, False]) +@pytest.mark.parametrize("eps", [1e-5, 1e-6]) +@pytest.mark.parametrize("is_neox", [True, False]) +@pytest.mark.parametrize("enable_rms_norm_custom_op", [True, False]) +@pytest.mark.parametrize("enable_rope_custom_op", [True]) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) +@pytest.mark.skipif( + not (current_platform.is_cuda_alike() or current_platform.is_xpu()), + reason="Only test on cuda and rocm platform", +) +def test_qk_norm_rope_fusion( + eps, + is_neox, + enable_rms_norm_custom_op, + enable_rope_custom_op, + dtype, + scattered_split, +): + if not hasattr(torch.ops._C, "fused_qk_norm_rope"): + pytest.skip("fused_qk_norm_rope custom op not available") + + torch.set_default_device(current_platform.device_type) + torch.set_default_dtype(dtype) + torch.manual_seed(0) + + custom_ops: list[str] = [] + if enable_rms_norm_custom_op: + custom_ops.append("+rms_norm") + if enable_rope_custom_op: + custom_ops.append("+rotary_embedding") + + vllm_config = VllmConfig( + model_config=ModelConfig(dtype=dtype), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=custom_ops, + pass_config=PassConfig( + enable_qk_norm_rope_fusion=True, + eliminate_noops=True, + ), + ), + ) + + num_heads, num_kv_heads, head_dim = 16, 4, 128 + T = 5 + + with ( + set_current_vllm_config(vllm_config), + vllm_config.kernel_config.ir_op_priority.set_priority(), + ): + model = QKNormRoPETestModel( + num_heads=num_heads, + num_kv_heads=num_kv_heads, + head_dim=head_dim, + eps=eps, + is_neox=is_neox, + vllm_config=vllm_config, + dtype=dtype, + test_scattered_split=scattered_split, + ) + + noop_pass = NoOpEliminationPass(vllm_config) + coalesce_pass = SplitCoalescingPass(vllm_config) + fusion_pass = QKNormRoPEFusionPass(vllm_config) + cleanup_pass = PostCleanupPass(vllm_config) + + backend = TestBackend(noop_pass, coalesce_pass, fusion_pass, cleanup_pass) + backend_baseline = TestBackend(noop_pass, cleanup_pass) + + qkv = torch.randn(T, model.q_size + 2 * model.kv_size) + pos = torch.arange(T, dtype=torch.long, device=qkv.device) + qkv_unfused = qkv.clone() + pos_unfused = pos.clone() + + torch._dynamo.mark_dynamic(qkv, 0) + torch._dynamo.mark_dynamic(pos, 0) + model_fused = torch.compile(model, backend=backend) + q_fused, k_fused, v_fused = model_fused(qkv, pos) + + torch._dynamo.mark_dynamic(qkv_unfused, 0) + torch._dynamo.mark_dynamic(pos_unfused, 0) + model_unfused = torch.compile(model, backend=backend_baseline) + q_unfused, k_unfused, v_unfused = model_unfused(qkv_unfused, pos_unfused) + + if dtype == torch.float16: + ATOL, RTOL = (2e-3, 2e-3) + else: + ATOL, RTOL = (1e-2, 1e-2) + + torch.testing.assert_close(q_unfused, q_fused, atol=ATOL, rtol=RTOL) + torch.testing.assert_close(k_unfused, k_fused, atol=ATOL, rtol=RTOL) + torch.testing.assert_close(v_unfused, v_fused, atol=ATOL, rtol=RTOL) + + assert fusion_pass.matched_count == 1 + + backend.check_before_ops(model.ops_in_model_before()) + backend.check_after_ops(model.ops_in_model_after()) diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_rope_kvcache_fusion.py b/asian_ai/performance/vLLM/tests/compile/passes/test_rope_kvcache_fusion.py new file mode 100644 index 0000000000000000000000000000000000000000..ad11f255d0fde6a6c1eb385ac658e43f79837c66 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_rope_kvcache_fusion.py @@ -0,0 +1,588 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch +from torch._higher_order_ops import auto_functionalized + +import vllm.config +from tests.compile.backend import TestBackend +from tests.v1.attention.utils import BatchSpec, create_common_attn_metadata +from vllm._aiter_ops import is_aiter_found_and_supported, rocm_aiter_ops +from vllm.compilation.passes.fusion import rope_kvcache_fusion +from vllm.compilation.passes.fusion.matcher_utils import ROTARY_OP +from vllm.compilation.passes.fusion.rope_kvcache_fusion import RopeKVCacheFusionPass +from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass +from vllm.compilation.passes.utility.post_cleanup import PostCleanupPass +from vllm.compilation.passes.utility.scatter_split_replace import ( + ScatterSplitReplacementPass, +) +from vllm.compilation.passes.utility.split_coalescing import SplitCoalescingPass +from vllm.config import ( + CacheConfig, + CompilationConfig, + CompilationMode, + ModelConfig, + PassConfig, + VllmConfig, +) +from vllm.config.utils import Range +from vllm.forward_context import get_forward_context, set_forward_context +from vllm.model_executor.layers.attention import Attention +from vllm.model_executor.layers.rotary_embedding import RotaryEmbedding +from vllm.platforms import current_platform +from vllm.utils.torch_utils import _encode_layer_name +from vllm.v1.attention.backend import ( + AttentionBackend, + CommonAttentionMetadata, +) +from vllm.v1.attention.backends.registry import AttentionBackendEnum + +INDEX_SELECT_OP = torch.ops.aten.index.Tensor +VLLM_UNIFIED_KV_CACHE_UPDATE_OP = torch.ops.vllm.unified_kv_cache_update +FP8_DTYPE = current_platform.fp8_dtype() + + +def test_rope_kvcache_fusion_default_keeps_large_ranges_unfused(): + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + pass_config=PassConfig(fuse_rope_kvcache=True), + ), + ) + fusion_pass = RopeKVCacheFusionPass(vllm_config) + + assert fusion_pass.is_applicable_for_range(Range(1, 256)) + assert not fusion_pass.is_applicable_for_range(Range(257, 11650)) + assert not fusion_pass.is_applicable_for_range(Range(11651, 16384)) + + +class QKRoPEKVCacheTestModel(torch.nn.Module): + def __init__( + self, + vllm_config: VllmConfig, + attn_backend: AttentionBackendEnum, + num_heads: int, + num_kv_heads: int, + head_size: int, + is_neox: bool, + dtype: torch.dtype, + device: torch.device, + prefix: str = "model.layers.0.self_attn.attn", + ): + super().__init__() + self.num_heads = num_heads + self.num_kv_heads = num_kv_heads + self.head_size = head_size + self.block_size = vllm_config.cache_config.block_size + self.q_size = num_heads * head_size + self.kv_size = num_kv_heads * head_size + self.is_neox = is_neox + self.dtype = dtype + self.device = device + self.layer_name = prefix + + self.rotary_emb = RotaryEmbedding( + head_size, + rotary_dim=head_size, + max_position_embeddings=4096, + base=10000, + is_neox_style=is_neox, + dtype=self.dtype, + ) + + # Whether to check for the RoPE custom op or component index_select + self.enable_rope_custom_op = self.rotary_emb.enabled() + + # Register layer metadata for the fusion pass via Attention. + self.attn = Attention( + num_heads=num_heads, + head_size=head_size, + scale=1.0 / head_size**0.5, + num_kv_heads=num_kv_heads, + cache_config=vllm_config.cache_config, + quant_config=vllm_config.quant_config, + prefix=prefix, + attn_backend=attn_backend.get_class(), + ) + self.attn_backend: type[AttentionBackend] = self.attn.get_attn_backend() + assert not self.attn_backend.forward_includes_kv_cache_update, ( + f"Attention backend {self.attn_backend} does not support fuse_rope_kvcache." + ) + self.attn._k_scale = self.attn._k_scale.to(device) + self.attn._v_scale = self.attn._v_scale.to(device) + + kv_cache_dtype_str = vllm_config.cache_config.cache_dtype + self.kv_cache_dtype = ( + FP8_DTYPE if kv_cache_dtype_str.startswith("fp8") else self.dtype + ) + + # Initialize attn MetadataBuilder + self.builder = self.attn_backend.get_builder_cls()( + kv_cache_spec=self.attn.get_kv_cache_spec(vllm_config), + layer_names=[self.attn.layer_name], + vllm_config=vllm_config, + device=device, + ) + + def build_attn_metadata(self, batch_size: int) -> CommonAttentionMetadata: + """Initialize attention metadata.""" + # Create common attn metadata + batch_spec = BatchSpec(seq_lens=[1] * batch_size, query_lens=[1] * batch_size) + common_attn_metadata = create_common_attn_metadata( + batch_spec, self.block_size, self.device, arange_block_indices=True + ) + + max_blocks = (max(batch_spec.seq_lens) + self.block_size - 1) // self.block_size + num_blocks = batch_size * max_blocks + + # Fetch the attention backend and kv cache shape and stride order + kv_cache_shape = self.attn_backend.get_kv_cache_shape( + num_blocks, self.block_size, self.num_kv_heads, self.head_size + ) + try: + kv_cache_stride_order = self.attn_backend.get_kv_cache_stride_order() + except (AttributeError, NotImplementedError): + kv_cache_stride_order = tuple(range(len(kv_cache_shape))) + + kv_cache_shape = tuple(kv_cache_shape[i] for i in kv_cache_stride_order) + inv_order = [ + kv_cache_stride_order.index(i) for i in range(len(kv_cache_stride_order)) + ] + + # Create dummy KV cache + raw_tensor = torch.zeros( + 2 * num_blocks * self.block_size * self.num_kv_heads * self.head_size, + dtype=self.kv_cache_dtype, + device=self.device, + ) + raw_tensor = raw_tensor.view(kv_cache_shape) + kv_cache = raw_tensor.permute(*inv_order) + + self.attn.kv_cache = kv_cache + + # Build attn metadata + attn_metadata = self.builder.build( + common_prefix_len=0, common_attn_metadata=common_attn_metadata + ) + + return attn_metadata + + def forward( + self, qkv: torch.Tensor, positions: torch.Tensor + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + # Create copy so inplace ops do not modify the original tensors + qkv = qkv.clone() + q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1) + q, k = self.rotary_emb(positions, q, k) + + # Instead of a full forward pass, match only the KV cache update op here + q = q.view(-1, self.num_heads, self.head_size) + k = k.view(-1, self.num_kv_heads, self.head_size) + v = v.view(-1, self.num_kv_heads, self.head_size) + kv_cache_dummy_dep = torch.ops.vllm.unified_kv_cache_update( + k, v, _encode_layer_name(self.layer_name) + ) + return q, k, v, kv_cache_dummy_dep + + def ops_in_model_before(self) -> list[torch._ops.OpOverload]: + ops = [] + if self.enable_rope_custom_op: + if rocm_aiter_ops.is_triton_rotary_embed_enabled(): + ops.append(torch.ops.vllm.rocm_aiter_triton_rotary_embedding.default) + else: + ops.append(ROTARY_OP) + else: + ops.append(INDEX_SELECT_OP) + ops.append(torch.ops.vllm.unified_kv_cache_update.default) + return ops + + def ops_in_model_after(self) -> list[torch._ops.OpOverload]: + return [torch.ops.vllm.fused_rope_and_unified_kv_cache_update.default] + + +class QKRoPEStaticQKVCacheTestModel(QKRoPEKVCacheTestModel): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self.q_scale = torch.ones((), dtype=torch.float32, device=self.device) + + def forward( + self, qkv: torch.Tensor, positions: torch.Tensor + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + # Create copy so inplace ops do not modify the original tensors + qkv = qkv.clone() + q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1) + q, k = self.rotary_emb(positions, q, k) + + q_fp8 = torch.empty(q.shape, device=q.device, dtype=FP8_DTYPE) + _, q_fp8 = auto_functionalized( + torch.ops._C.static_scaled_fp8_quant.default, + result=q_fp8, + input=q, + scale=self.q_scale, + group_shape=(-1, -1), + ) + q = q_fp8.view(-1, self.num_heads, self.head_size) + k = k.view(-1, self.num_kv_heads, self.head_size) + v = v.view(-1, self.num_kv_heads, self.head_size) + kv_cache_dummy_dep = torch.ops.vllm.unified_kv_cache_update( + k, v, _encode_layer_name(self.layer_name) + ) + return q, k, v, kv_cache_dummy_dep + + def ops_in_model_before(self) -> list[torch._ops.OpOverload]: + ops = [] + if self.enable_rope_custom_op: + if rocm_aiter_ops.is_triton_rotary_embed_enabled(): + ops.append(torch.ops.vllm.rocm_aiter_triton_rotary_embedding.default) + else: + ops.append(ROTARY_OP) + else: + ops.append(INDEX_SELECT_OP) + ops.append(torch.ops.vllm.unified_kv_cache_update.default) + return ops + + def ops_in_model_after(self) -> list[torch._ops.OpOverload]: + return [torch.ops.vllm.fused_rope_and_unified_kv_cache_update.default] + + +@pytest.mark.parametrize( + "attn_backend", + [ + AttentionBackendEnum.ROCM_AITER_UNIFIED_ATTN, + AttentionBackendEnum.TRITON_ATTN, + AttentionBackendEnum.ROCM_ATTN, + AttentionBackendEnum.ROCM_AITER_FA, + ], +) +@pytest.mark.parametrize("enable_rope_custom_op", [True]) # [True, False]) +@pytest.mark.parametrize("enable_aiter_triton_rope", [True, False]) +@pytest.mark.parametrize("num_heads", [64]) +@pytest.mark.parametrize("num_kv_heads", [8]) +@pytest.mark.parametrize("head_size", [64]) +@pytest.mark.parametrize("block_size", [16]) +@pytest.mark.parametrize("is_neox", [True, False]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8"]) +@pytest.mark.skipif( + not is_aiter_found_and_supported(), + reason="Only test on ROCm with AITER installed and supported", +) +def test_rope_kvcache_fusion( + attn_backend: AttentionBackendEnum, + enable_rope_custom_op: bool, + enable_aiter_triton_rope: bool, + num_heads: int, + num_kv_heads: int, + head_size: int, + block_size: int, + is_neox: bool, + dtype: torch.dtype, + kv_cache_dtype: str, + monkeypatch: pytest.MonkeyPatch, +): + torch.set_default_device("cuda") + torch.set_default_dtype(dtype) + torch.manual_seed(0) + + custom_ops: list[str] = [] + if enable_rope_custom_op: + custom_ops.append("+rotary_embedding") + + vllm_config = VllmConfig( + model_config=ModelConfig(dtype=dtype), + cache_config=CacheConfig( + block_size=block_size, + cache_dtype=kv_cache_dtype, + ), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=custom_ops, + pass_config=PassConfig( + fuse_rope_kvcache=True, + eliminate_noops=True, + ), + ), + ) + + with vllm.config.set_current_vllm_config(vllm_config), monkeypatch.context() as m: + m.setenv("VLLM_ROCM_USE_AITER", "1") + m.setenv( + "VLLM_ROCM_USE_AITER_TRITON_ROPE", "1" if enable_aiter_triton_rope else "0" + ) + rocm_aiter_ops.refresh_env_variables() + + model = QKRoPEKVCacheTestModel( + vllm_config=vllm_config, + attn_backend=attn_backend, + num_heads=num_heads, + num_kv_heads=num_kv_heads, + head_size=head_size, + is_neox=is_neox, + dtype=dtype, + device=torch.get_default_device(), + ) + + fusion_pass = RopeKVCacheFusionPass(vllm_config) + passes = [ + NoOpEliminationPass(vllm_config), + SplitCoalescingPass(vllm_config), + ScatterSplitReplacementPass(vllm_config), + fusion_pass, + PostCleanupPass(vllm_config), + ] + backend = TestBackend(*passes) + + T = 5 + + qkv = torch.randn( + T, num_heads * head_size + 2 * num_kv_heads * head_size, dtype=dtype + ) + pos = torch.arange(T, dtype=torch.long) + + qkv_unfused = qkv.clone() + pos_unfused = pos.clone() + + with set_forward_context(None, vllm_config): + forward_context = get_forward_context() + attn_metadata = model.build_attn_metadata(T) + forward_context.slot_mapping = { + model.layer_name: attn_metadata.slot_mapping + } + q_unfused, k_unfused, v_unfused, dummy = model(qkv_unfused, pos_unfused) + attn_layer = forward_context.no_compile_layers[model.layer_name] + kv_cache_unfused = attn_layer.kv_cache + del dummy + + torch._dynamo.mark_dynamic(qkv, 0) + torch._dynamo.mark_dynamic(pos, 0) + with set_forward_context(None, vllm_config): + model_fused = torch.compile(model, backend=backend) + forward_context = get_forward_context() + attn_metadata = model_fused.build_attn_metadata(T) + forward_context.slot_mapping = { + model.layer_name: attn_metadata.slot_mapping + } + q_fused, k_fused, v_fused, dummy = model_fused(qkv, pos) + attn_layer = forward_context.no_compile_layers[model.layer_name] + kv_cache_fused = attn_layer.kv_cache + del dummy + + assert fusion_pass.matched_count == 1 + + backend.check_before_ops(model.ops_in_model_before()) + backend.check_after_ops(model.ops_in_model_after()) + + if dtype == torch.float16: + ATOL, RTOL = (2e-3, 2e-3) + else: + ATOL, RTOL = (1e-2, 1e-2) + + torch.testing.assert_close(q_unfused, q_fused, atol=ATOL, rtol=RTOL) + torch.testing.assert_close(k_unfused, k_fused, atol=ATOL, rtol=RTOL) + torch.testing.assert_close(v_unfused, v_fused, atol=ATOL, rtol=RTOL) + # Cannot compare fp8_* directly here, cast to model dtype instead + # TODO(charlifu): switch back to ATOL, RTOL after aiter fix is merged. + torch.testing.assert_close( + kv_cache_unfused.to(dtype), + kv_cache_fused.to(dtype), + atol=1e-1, + rtol=1e-1, + ) + + +@pytest.mark.parametrize( + "attn_backend", + [AttentionBackendEnum.ROCM_AITER_UNIFIED_ATTN], +) +@pytest.mark.parametrize("enable_rope_custom_op", [True]) +@pytest.mark.parametrize("enable_aiter_triton_rope", [True, False]) +@pytest.mark.parametrize("num_heads", [64]) +@pytest.mark.parametrize("num_kv_heads", [8]) +@pytest.mark.parametrize("head_size", [64]) +@pytest.mark.parametrize("block_size", [16]) +@pytest.mark.parametrize("is_neox", [True, False]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8"]) +@pytest.mark.skipif( + not is_aiter_found_and_supported(), + reason="Only test on ROCm with AITER installed and supported", +) +@pytest.mark.skipif( + not hasattr(torch.ops._C, "static_scaled_fp8_quant"), + reason="static fp8 quant op not available on this build", +) +def test_rope_static_qquant_kvcache_fusion( + attn_backend: AttentionBackendEnum, + enable_rope_custom_op: bool, + enable_aiter_triton_rope: bool, + num_heads: int, + num_kv_heads: int, + head_size: int, + block_size: int, + is_neox: bool, + dtype: torch.dtype, + kv_cache_dtype: str, + monkeypatch: pytest.MonkeyPatch, +): + torch.set_default_device("cuda") + torch.set_default_dtype(dtype) + torch.manual_seed(0) + + custom_ops: list[str] = [] + if enable_rope_custom_op: + custom_ops.append("+rotary_embedding") + + vllm_config = VllmConfig( + model_config=ModelConfig(dtype=dtype), + cache_config=CacheConfig( + block_size=block_size, + cache_dtype=kv_cache_dtype, + ), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=custom_ops, + pass_config=PassConfig( + fuse_rope_kvcache=True, + eliminate_noops=True, + ), + ), + ) + + with vllm.config.set_current_vllm_config(vllm_config), monkeypatch.context() as m: + m.setenv("VLLM_ROCM_USE_AITER", "1") + m.setenv( + "VLLM_ROCM_USE_AITER_TRITON_ROPE", "1" if enable_aiter_triton_rope else "0" + ) + rocm_aiter_ops.refresh_env_variables() + + model = QKRoPEStaticQKVCacheTestModel( + vllm_config=vllm_config, + attn_backend=attn_backend, + num_heads=num_heads, + num_kv_heads=num_kv_heads, + head_size=head_size, + is_neox=is_neox, + dtype=dtype, + device=torch.get_default_device(), + ) + + fusion_pass = RopeKVCacheFusionPass(vllm_config) + passes = [ + NoOpEliminationPass(vllm_config), + SplitCoalescingPass(vllm_config), + ScatterSplitReplacementPass(vllm_config), + fusion_pass, + PostCleanupPass(vllm_config), + ] + backend = TestBackend(*passes) + + T = 5 + qkv = torch.randn( + T, num_heads * head_size + 2 * num_kv_heads * head_size, dtype=dtype + ) + pos = torch.arange(T, dtype=torch.long) + + qkv_unfused = qkv.clone() + pos_unfused = pos.clone() + + with set_forward_context(None, vllm_config): + forward_context = get_forward_context() + attn_metadata = model.build_attn_metadata(T) + forward_context.slot_mapping = { + model.layer_name: attn_metadata.slot_mapping + } + q_unfused, k_unfused, v_unfused, dummy = model(qkv_unfused, pos_unfused) + attn_layer = forward_context.no_compile_layers[model.layer_name] + kv_cache_unfused = attn_layer.kv_cache + del dummy + + torch._dynamo.mark_dynamic(qkv, 0) + torch._dynamo.mark_dynamic(pos, 0) + with set_forward_context(None, vllm_config): + model_fused = torch.compile(model, backend=backend) + forward_context = get_forward_context() + attn_metadata = model_fused.build_attn_metadata(T) + forward_context.slot_mapping = { + model.layer_name: attn_metadata.slot_mapping + } + q_fused, k_fused, v_fused, dummy = model_fused(qkv, pos) + attn_layer = forward_context.no_compile_layers[model.layer_name] + kv_cache_fused = attn_layer.kv_cache + del dummy + + assert fusion_pass.matched_count == 1 + backend.check_before_ops(model.ops_in_model_before()) + backend.check_after_ops(model.ops_in_model_after()) + static_quant_pre = backend.op_count( + torch.ops._C.static_scaled_fp8_quant.default, before=True + ) + static_quant_post = backend.op_count( + torch.ops._C.static_scaled_fp8_quant.default + ) + assert static_quant_pre > 0 + # The replacement still emits static quant, so count is expected to + # remain non-zero after fusion. + assert static_quant_post > 0 + + # Negative control: without the static-Q pattern, the generic RoPE+KV + # pattern cannot match this rope -> static-quant -> kv graph, so the + # fusion above is attributable solely to RopeStaticQQuantKVCachePattern. + # This is a structural property independent of the rope/dtype/neox axes, + # so run the (extra compile) check only once on a representative combo. + if is_neox and enable_aiter_triton_rope and kv_cache_dtype == "auto": + m.setattr( + rope_kvcache_fusion, + "_supports_static_q_fp8_quant_fusion", + lambda: False, + ) + generic_pass = RopeKVCacheFusionPass(vllm_config) + generic_backend = TestBackend( + NoOpEliminationPass(vllm_config), + SplitCoalescingPass(vllm_config), + ScatterSplitReplacementPass(vllm_config), + generic_pass, + PostCleanupPass(vllm_config), + ) + # Reset dynamo so the model is recompiled through generic_backend + # instead of reusing the cached compilation from above. + torch._dynamo.reset() + with set_forward_context(None, vllm_config): + model_generic = torch.compile(model, backend=generic_backend) + forward_context = get_forward_context() + attn_metadata = model_generic.build_attn_metadata(T) + forward_context.slot_mapping = { + model.layer_name: attn_metadata.slot_mapping + } + model_generic(qkv, pos) + # op_count reads the post-pass graph, so it also confirms the pass ran + # (a no-op pass would raise instead of silently passing on count 0). + assert generic_pass.matched_count == 0 + assert ( + generic_backend.op_count( + torch.ops.vllm.fused_rope_and_unified_kv_cache_update.default + ) + == 0 + ) + + if dtype == torch.float16: + ATOL, RTOL = (2e-3, 2e-3) + else: + ATOL, RTOL = (1e-2, 1e-2) + + # TODO(charlifu): switch back to ATOL, RTOL after aiter fix is merged. + torch.testing.assert_close( + q_unfused.to(torch.float32), + q_fused.to(torch.float32), + atol=1e-1, + rtol=1e-1, + ) + torch.testing.assert_close(k_unfused, k_fused, atol=ATOL, rtol=RTOL) + torch.testing.assert_close(v_unfused, v_fused, atol=ATOL, rtol=RTOL) + # TODO(charlifu): switch back to ATOL, RTOL after aiter fix is merged. + torch.testing.assert_close( + kv_cache_unfused.to(dtype), + kv_cache_fused.to(dtype), + atol=1e-1, + rtol=1e-1, + ) diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_scatter_split_replace.py b/asian_ai/performance/vLLM/tests/compile/passes/test_scatter_split_replace.py new file mode 100644 index 0000000000000000000000000000000000000000..b88df0f51104223347fdc3d153589db57018682a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_scatter_split_replace.py @@ -0,0 +1,110 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch +import torch.nn as nn + +import vllm +from tests.compile.backend import TestBackend +from vllm.compilation.passes.utility.scatter_split_replace import ( + ScatterSplitReplacementPass, +) +from vllm.compilation.passes.utility.split_coalescing import SplitCoalescingPass +from vllm.config import CompilationConfig, CompilationMode, VllmConfig +from vllm.model_executor.layers.rotary_embedding import RotaryEmbedding +from vllm.platforms import current_platform + +DEVICE_TYPE = current_platform.device_type + + +class ScatterSplitReplacementModel(nn.Module): + """Model with a rope+getitem+slice_scatter+split_with_sizes sequence.""" + + def __init__( + self, + num_heads: int, + num_kv_heads: int, + head_size: int, + dtype: torch.dtype, + ): + super().__init__() + self.q_size = num_heads * head_size + self.kv_size = num_kv_heads * head_size + + self.rotary_emb = RotaryEmbedding( + head_size, + rotary_dim=head_size, + max_position_embeddings=4096, + base=10000, + is_neox_style=True, + dtype=dtype, + ) + + def forward(self, qkv: torch.Tensor, positions: torch.Tensor): + # Create copy so inplace ops do not modify the original tensors + qkv = qkv.clone() + q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1) + q, k = self.rotary_emb(positions, q, k) + q = q + 1 + k = k + 2 + v = v + 3 + return q, k, v + + def ops_in_model_before(self) -> list[torch._ops.OpOverload]: + return [ + torch.ops.aten.slice_scatter.default, + torch.ops.aten.split_with_sizes.default, + torch.ops.aten.getitem.default, + ] + + def ops_in_model_after(self) -> list[torch._ops.OpOverload]: + return [torch.ops.aten.getitem.default] + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +def test_scatter_split_replace(dtype): + torch.set_default_device(DEVICE_TYPE) + torch.set_default_dtype(dtype) + torch.manual_seed(0) + + num_heads = 8 + num_kv_heads = 4 + head_size = 64 + + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=["+rotary_embedding"], + ), + ) + with vllm.config.set_current_vllm_config(vllm_config): + # ScatterSplitReplacementPass requires SplitCoalescingPass to be run before it + coalesce_pass = SplitCoalescingPass(vllm_config) + replace_pass = ScatterSplitReplacementPass(vllm_config) + passes = [coalesce_pass, replace_pass] + backend = TestBackend(*passes) + + model = ScatterSplitReplacementModel(num_heads, num_kv_heads, head_size, dtype) + + T = 5 + qkv = torch.randn( + T, num_heads * head_size + 2 * num_kv_heads * head_size, dtype=dtype + ) + pos = torch.arange(T, dtype=torch.long) + + qkv_eager = qkv.clone() + pos_eager = pos.clone() + result_eager = model(qkv_eager, pos_eager) + + torch._dynamo.mark_dynamic(qkv, 0) + torch._dynamo.mark_dynamic(pos, 0) + + model_compiled = torch.compile(model, backend=backend) + result_compiled = model_compiled(qkv, pos) + + for eager, compiled in zip(result_eager, result_compiled): + torch.testing.assert_close(eager, compiled) + + assert backend.op_count(torch.ops.aten.slice_scatter.default) == 0 + assert backend.op_count(torch.ops.aten.split_with_sizes.default) == 1 diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_silu_mul_quant_fusion.py b/asian_ai/performance/vLLM/tests/compile/passes/test_silu_mul_quant_fusion.py new file mode 100644 index 0000000000000000000000000000000000000000..713d00720d0373385e437537c440368cd4ed2137 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_silu_mul_quant_fusion.py @@ -0,0 +1,379 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import itertools +from functools import partial + +import pytest +import torch + +import vllm.envs as envs +from tests.compile.backend import TestBackend +from tests.kernels.quantization.nvfp4_utils import quant_nvfp4_tensor +from tests.utils import TestFP8Layer +from vllm._aiter_ops import IS_AITER_FOUND, rocm_aiter_ops +from vllm._custom_ops import cutlass_scaled_fp4_mm, scaled_fp4_quant +from vllm.compilation.passes.fusion.act_quant_fusion import ( + FUSED_OPS, + SILU_MUL_OP, + ActivationQuantFusionPass, +) +from vllm.compilation.passes.fusion.rms_quant_fusion import QUANT_OPS +from vllm.compilation.passes.utility.noop_elimination import NoOpEliminationPass +from vllm.compilation.passes.utility.post_cleanup import PostCleanupPass +from vllm.config import ( + CompilationConfig, + CompilationMode, + PassConfig, + VllmConfig, + set_current_vllm_config, +) +from vllm.model_executor.kernels.linear import ( + CutlassFP8ScaledMMLinearKernel, + FlashInferFP8ScaledMMLinearKernel, + FP8ScaledMMLinearKernel, + PerTensorTorchFP8ScaledMMLinearKernel, + ROCmFP8ScaledMMLinearKernel, +) +from vllm.model_executor.layers.activation import SiluAndMul +from vllm.model_executor.layers.quantization.input_quant_fp8 import QuantFP8 +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + GroupShape, + create_fp8_quant_key, + kFp8Dynamic128Sym, + kFp8StaticTensorSym, + kNvfp4Dynamic, +) +from vllm.platforms import current_platform +from vllm.utils.deep_gemm import is_deep_gemm_supported + +FP8_DTYPE = current_platform.fp8_dtype() +FP4_DTYPE = torch.uint8 + + +def is_nvfp4_supported(): + return current_platform.has_device_capability(100) + + +class TestSiluMulFp8QuantModel(torch.nn.Module): + quant_key = kFp8StaticTensorSym + + def __init__( + self, + hidden_size: int, + force_kernel: FP8ScaledMMLinearKernel, + dtype: torch.dtype, + **kwargs, + ): + super().__init__() + self.silu_and_mul = SiluAndMul() + + self.fp8_linear = TestFP8Layer( + weight_shape=(hidden_size, hidden_size), + activation_quant_key=self.quant_key, + weight_quant_key=self.quant_key, + force_kernel=force_kernel, + input_dtype=dtype, + ) + + self.enable_silu_mul_custom_op = self.silu_and_mul.enabled() + self.enable_quant_fp8_custom_op = self.fp8_linear.is_quant_fp8_enabled() + + def forward(self, x): + y = self.silu_and_mul(x) + x2 = self.fp8_linear(y) + return x2 + + def ops_in_model_before(self): + return [ + SILU_MUL_OP if self.enable_silu_mul_custom_op else torch.ops.aten.mul, + ( + QUANT_OPS[kFp8StaticTensorSym] + if self.enable_quant_fp8_custom_op + else torch.ops.aten.reciprocal + ), + ] + + def ops_in_model_after(self): + return [FUSED_OPS[kFp8StaticTensorSym]] + + +class TestSiluMulNvfp4QuantModel(torch.nn.Module): + def __init__(self, hidden_size: int, x: torch.Tensor, **kwargs): + super().__init__() + from vllm.compilation.passes.fusion.act_quant_fusion import ( + silu_and_mul_nvfp4_quant_supported, + ) + + assert silu_and_mul_nvfp4_quant_supported + + self.silu_and_mul = SiluAndMul() + self.enable_silu_mul_custom_op = self.silu_and_mul.enabled() + + # create nvfp4 weight + w = torch.rand((hidden_size, hidden_size)) + self.w, self.w_block_scale, self.w_global_scale = quant_nvfp4_tensor(w) + + # get global scale offline + _, _, self.y_global_scale = quant_nvfp4_tensor(self.silu_and_mul(x)) + + self.alpha = 1.0 / (self.w_global_scale * self.y_global_scale) + + def forward(self, x): + y = self.silu_and_mul(x) + y_quant, y_block_scale = scaled_fp4_quant(y, self.y_global_scale) + out = cutlass_scaled_fp4_mm( + a=y_quant, + b=self.w, + block_scale_a=y_block_scale, + block_scale_b=self.w_block_scale, + alpha=self.alpha, + out_dtype=y.dtype, + ) + return out + + def ops_in_model_before(self): + return [ + SILU_MUL_OP if self.enable_silu_mul_custom_op else torch.ops.aten.mul, + QUANT_OPS[kNvfp4Dynamic], + ] + + def ops_in_model_after(self): + return [FUSED_OPS[kNvfp4Dynamic]] + + +class TestSiluMulGroupFp8QuantModel(torch.nn.Module): + act_quant_key = kFp8Dynamic128Sym + + def __init__(self, hidden_size: int, dtype: torch.dtype, **kwargs): + super().__init__() + self.silu_and_mul = SiluAndMul() + self.weight_quant_key = create_fp8_quant_key( + static=True, group_shape=GroupShape(hidden_size, hidden_size) + ) + + self.w8a8_block_fp8_linear = TestFP8Layer( + weight_shape=(hidden_size, hidden_size), + weight_quant_key=self.weight_quant_key, + activation_quant_key=self.act_quant_key, + input_dtype=dtype, + ) + + if not current_platform.is_fp8_fnuz(): + kernel = self.w8a8_block_fp8_linear.kernel + orig_quant = kernel.quant_fp8 + kernel.quant_fp8 = lambda *a, use_triton=False, **kw: orig_quant( + *a, use_triton=True, **kw + ) + + self.enable_silu_mul_custom_op = self.silu_and_mul.enabled() + + def forward(self, x): + y = self.silu_and_mul(x) + x2 = self.w8a8_block_fp8_linear(y) + return x2 + + def ops_in_model_before(self): + return [ + SILU_MUL_OP if self.enable_silu_mul_custom_op else torch.ops.aten.mul, + rocm_aiter_ops.get_group_quant_op() + if current_platform.is_fp8_fnuz() + else torch.ops.vllm.triton_per_token_group_quant_fp8.default, + ] + + def ops_in_model_after(self): + return [torch.ops.vllm.rocm_aiter_act_mul_and_fp8_group_quant] + + +class TestSiluMulBlockQuantModel(torch.nn.Module): + quant_key = kFp8Dynamic128Sym + + def __init__(self, hidden_size: int, is_scale_transposed: bool = False, **kwargs): + super().__init__() + self.silu_and_mul = SiluAndMul() + self.is_scale_transposed = is_scale_transposed + self.quant_fp8 = QuantFP8( + static=False, + group_shape=GroupShape(1, 128), + column_major_scales=is_scale_transposed, + compile_native=False, + ) + + self.enable_silu_mul_custom_op = self.silu_and_mul.enabled() + self.enable_quant_fp8_custom_op = self.quant_fp8.enabled() + + def forward(self, x): + y = self.silu_and_mul(x) + out, scale = self.quant_fp8(y) + group_size = self.quant_key.scale.group_shape[1] + scale_expanded = scale.repeat_interleave(group_size, dim=1) + dequant = out.to(dtype=torch.float32) * scale_expanded + return (dequant,) + + def ops_in_model_before(self): + ops = [] + if self.enable_silu_mul_custom_op: + ops.append(SILU_MUL_OP) + # When silu custom op is disabled, aten.mul.Tensor also appears + # in dequant code, so we skip checking it to avoid false positives. + ops.append( + QUANT_OPS[self.quant_key] + if self.enable_quant_fp8_custom_op + else torch.ops.aten.reciprocal.default + ) + return ops + + def ops_in_model_after(self): + return [FUSED_OPS[self.quant_key]] + + +ROCM_KERNELS = [ROCmFP8ScaledMMLinearKernel, PerTensorTorchFP8ScaledMMLinearKernel] +CUDA_KERNELS = [ + FlashInferFP8ScaledMMLinearKernel, + CutlassFP8ScaledMMLinearKernel, + PerTensorTorchFP8ScaledMMLinearKernel, +] +TEST_KERNELS = ROCM_KERNELS if current_platform.is_rocm() else CUDA_KERNELS + + +@pytest.mark.parametrize("num_tokens", [32, 64]) +@pytest.mark.parametrize("hidden_size", [128, 256]) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) +@pytest.mark.parametrize("enable_silu_mul_custom_op", [True, False]) +@pytest.mark.parametrize( + "model_class, enable_quant_fp8_custom_op, force_kernel", + list(itertools.product([TestSiluMulFp8QuantModel], [True, False], TEST_KERNELS)) + + [ + pytest.param( + TestSiluMulNvfp4QuantModel, + False, + None, + marks=pytest.mark.skipif( + not current_platform.is_cuda(), reason="CUDA only" + ), + ), + # GroupFP8Quant fusion only works with AITER on ROCm. + # and the enable_quant_fp8_custom_op must be True. + pytest.param( + TestSiluMulGroupFp8QuantModel, + True, + None, + marks=pytest.mark.skipif( + not current_platform.is_rocm(), reason="ROCm only" + ), + ), + # Block quant fusion for per-group FP8 (CUDA only). + *[ + pytest.param( + partial(TestSiluMulBlockQuantModel, is_scale_transposed=transposed), + True, + None, + marks=pytest.mark.skipif( + not current_platform.is_cuda(), reason="CUDA only" + ), + id=f"TestSiluMulBlockQuant-transposed={transposed}", + ) + for transposed in [False, True] + ], + ], +) +@pytest.mark.skipif( + envs.VLLM_TARGET_DEVICE not in ["cuda", "rocm"], reason="Only test on CUDA and ROCm" +) +def test_fusion_silu_and_mul_quant( + num_tokens: int, + hidden_size: int, + dtype: torch.dtype, + model_class: type[ + TestSiluMulFp8QuantModel + | TestSiluMulNvfp4QuantModel + | TestSiluMulGroupFp8QuantModel + | TestSiluMulBlockQuantModel + ], + enable_silu_mul_custom_op: bool, + enable_quant_fp8_custom_op: bool, + force_kernel: FP8ScaledMMLinearKernel | None, + monkeypatch: pytest.MonkeyPatch, +): + if model_class is TestSiluMulNvfp4QuantModel and not is_nvfp4_supported(): + pytest.skip("NVFP4 is not supported on this GPU.") + if model_class is TestSiluMulGroupFp8QuantModel and not IS_AITER_FOUND: + pytest.skip("AITER is not supported on this GPU.") + if ( + isinstance(model_class, partial) + and model_class.func is TestSiluMulBlockQuantModel + and is_deep_gemm_supported() + ): + pytest.skip("SiluMul+BlockQuant fusion not applicable with DeepGemm") + + torch.set_default_device("cuda") + torch.set_default_dtype(dtype) + + x = torch.rand(num_tokens, hidden_size * 2) + + # Reshape pass is needed for the fusion pass to work + custom_ops = ["none"] + if enable_silu_mul_custom_op: + custom_ops.append("+silu_and_mul") + if enable_quant_fp8_custom_op: + custom_ops.append("+quant_fp8") + config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=custom_ops, + backend="eager", # avoid compilation for SiluAndMul and QuantFP8 + pass_config=PassConfig(fuse_act_quant=True, eliminate_noops=True), + ), + ) + + with set_current_vllm_config(config), monkeypatch.context() as m: + fusion_passes = [ActivationQuantFusionPass(config)] + if IS_AITER_FOUND and model_class is TestSiluMulGroupFp8QuantModel: + from vllm.compilation.passes.fusion.rocm_aiter_fusion import ( + RocmAiterSiluMulFp8GroupQuantFusionPass, + ) + + m.setenv("VLLM_ROCM_USE_AITER", "1") + rocm_aiter_ops.refresh_env_variables() + fusion_passes += [RocmAiterSiluMulFp8GroupQuantFusionPass(config)] + + passes = [NoOpEliminationPass(config), *fusion_passes, PostCleanupPass(config)] + backend = TestBackend(*passes) + model = model_class( + hidden_size=hidden_size, force_kernel=force_kernel, x=x, dtype=dtype + ) + + # First dimension dynamic + torch._dynamo.mark_dynamic(x, 0) + + result = model(x) + + model2 = torch.compile(model, backend=backend) + result2 = model2(x) + + # Check that it gives the same answer + if isinstance(model, TestSiluMulFp8QuantModel): + atol, rtol = 1e-3, 1e-3 + elif isinstance(model, TestSiluMulNvfp4QuantModel): + atol, rtol = 1e-1, 1e-1 + elif isinstance(model, TestSiluMulGroupFp8QuantModel): + atol, rtol = 5e-2, 5e-2 + elif isinstance(model, TestSiluMulBlockQuantModel): + if current_platform.is_rocm(): + atol, rtol = 1e-3, 1e-3 + else: + # CUDA fused kernel computes silu*mul in fp32 while the reference + # goes through bf16/fp16 storage, so group maxima (and thus scales) + # can shift by one FP8-e4m3 code (~1/8 relative step). + atol, rtol = 5e-2, 5e-2 + + torch.testing.assert_close( + result[0].to(dtype=dtype), result2[0].to(dtype=dtype), atol=atol, rtol=rtol + ) + + assert sum([p.matched_count for p in fusion_passes]) == 1 + + # In pre-nodes, quant op should be present and fused kernels should not + backend.check_before_ops(model.ops_in_model_before()) + + # In post-nodes, fused kernels should be present and quant op should not + backend.check_after_ops(model.ops_in_model_after()) diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_split_coalescing.py b/asian_ai/performance/vLLM/tests/compile/passes/test_split_coalescing.py new file mode 100644 index 0000000000000000000000000000000000000000..472e3af6eeb006c3b6346d4059f2455b9503de2b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_split_coalescing.py @@ -0,0 +1,65 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +import vllm +from tests.compile.backend import TestBackend +from vllm.compilation.passes.utility.split_coalescing import SplitCoalescingPass +from vllm.config import CompilationConfig, CompilationMode, PassConfig, VllmConfig +from vllm.platforms import current_platform + +DEVICE_TYPE = current_platform.device_type + + +class SplitCoalescingModel(torch.nn.Module): + """Model with 3 separate split_with_sizes calls on the same input, + simulating the B200+FP8 graph where CSE fails to merge them.""" + + def __init__(self, q_size: int, kv_size: int) -> None: + super().__init__() + self.q_size = q_size + self.kv_size = kv_size + + def forward(self, qkv: torch.Tensor): + q, _, _ = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1) + _, k, _ = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1) + _, _, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1) + return q + 1, k + 2, v + 3 + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +def test_split_coalescing(dtype): + torch.set_default_device(DEVICE_TYPE) + torch.set_default_dtype(dtype) + torch.manual_seed(0) + + q_size, kv_size = 2048, 512 + + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + pass_config=PassConfig(), + ) + ) + with vllm.config.set_current_vllm_config(vllm_config): + coalesce_pass = SplitCoalescingPass(vllm_config) + backend = TestBackend(coalesce_pass) + + model = SplitCoalescingModel(q_size, kv_size) + + T = 5 + qkv = torch.randn(T, q_size + 2 * kv_size) + torch._dynamo.mark_dynamic(qkv, 0) + + result_eager = model(qkv) + + model_compiled = torch.compile(model, backend=backend) + result_compiled = model_compiled(qkv) + + ATOL, RTOL = (2e-3, 2e-3) + for eager, compiled in zip(result_eager, result_compiled): + torch.testing.assert_close(eager, compiled, atol=ATOL, rtol=RTOL) + + assert backend.op_count(torch.ops.aten.split_with_sizes.default) == 1 diff --git a/asian_ai/performance/vLLM/tests/compile/passes/test_vllm_fusion_pattern_matcher_pass.py b/asian_ai/performance/vLLM/tests/compile/passes/test_vllm_fusion_pattern_matcher_pass.py new file mode 100644 index 0000000000000000000000000000000000000000..2d2958d9807913b0f3ac23f325b7a9763a4e588a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/passes/test_vllm_fusion_pattern_matcher_pass.py @@ -0,0 +1,126 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +import vllm.config +from tests.compile.backend import TestBackend +from vllm.compilation.passes.vllm_inductor_pass import ( + VllmFusionPatternMatcherPass, + VllmPatternMatcherPass, + VllmPatternReplacement, +) +from vllm.config import CompilationConfig, CompilationMode, VllmConfig +from vllm.platforms import current_platform + + +class ReluToAbsPattern(VllmPatternReplacement): + """Replaces relu(x) with abs(x) — a minimal test fixture.""" + + @property + def pattern(self): + def _pattern(x: torch.Tensor) -> torch.Tensor: + return torch.ops.aten.relu.default(x) + + return _pattern + + @property + def replacement(self): + def _replacement(x: torch.Tensor) -> torch.Tensor: + return torch.ops.aten.abs.default(x) + + return _replacement + + def get_inputs(self) -> list[torch.Tensor]: + return [self.empty_fp32(4)] + + +class ExpToSqrtPattern(VllmPatternReplacement): + """A second distinct pattern type — used to test uuid differentiation.""" + + @property + def pattern(self): + def _pattern(x: torch.Tensor) -> torch.Tensor: + return torch.ops.aten.exp.default(x) + + return _pattern + + @property + def replacement(self): + def _replacement(x: torch.Tensor) -> torch.Tensor: + return torch.ops.aten.sqrt.default(x) + + return _replacement + + def get_inputs(self) -> list[torch.Tensor]: + return [self.empty_fp32(4)] + + +class ReluFusionPass(VllmFusionPatternMatcherPass): + def __init__(self, config: VllmConfig) -> None: + super().__init__(config, "test_relu_fusion") + self.register(ReluToAbsPattern()) + + +class TwoPatternFusionPass(VllmFusionPatternMatcherPass): + def __init__(self, config: VllmConfig) -> None: + super().__init__(config, "test_two_pattern_fusion") + self.register(ReluToAbsPattern()) + self.register(ExpToSqrtPattern()) + + +@pytest.fixture +def vllm_config(): + return VllmConfig( + compilation_config=CompilationConfig(mode=CompilationMode.VLLM_COMPILE), + ) + + +@pytest.mark.skipif(not current_platform.is_cuda_alike(), reason="Requires CUDA") +def test_register_tracks_patterns(vllm_config): + """register() appends each VllmPatternReplacement to _pattern_replacements.""" + with vllm.config.set_current_vllm_config(vllm_config): + single = ReluFusionPass(vllm_config) + two = TwoPatternFusionPass(vllm_config) + + assert len(single._pattern_replacements) == 1 + assert len(two._pattern_replacements) == 2 + + +@pytest.mark.skipif(not current_platform.is_cuda_alike(), reason="Requires CUDA") +def test_uuid_stable(vllm_config): + """Two instances of the same pass class produce identical uuids.""" + with vllm.config.set_current_vllm_config(vllm_config): + p1 = ReluFusionPass(vllm_config) + p2 = ReluFusionPass(vllm_config) + p3 = TwoPatternFusionPass(vllm_config) + + assert p1.uuid() == p2.uuid() + assert p1.uuid() != p3.uuid() + assert p2.uuid() != p3.uuid() + + +@pytest.mark.skipif(not current_platform.is_cuda_alike(), reason="Requires CUDA") +@pytest.mark.parametrize("N", [1, 2, 4]) +def test_matched_count_and_match_table(vllm_config, N): + """matched_count and match_table reflect the number of matched patterns.""" + + class Model(torch.nn.Module): + def forward(self, *inputs): + # N independent relus + return sum(torch.relu(x) for x in inputs) + + with vllm.config.set_current_vllm_config(vllm_config): + torch.set_default_device("cuda") + torch.set_default_dtype(torch.float32) + + fusion_pass = ReluFusionPass(vllm_config) + backend = TestBackend(fusion_pass) + model = torch.compile(Model(), backend=backend) + + inputs = [torch.rand(8) for _ in range(N)] + model(*inputs) + + assert fusion_pass.matched_count == N + assert VllmPatternMatcherPass.match_table["test_relu_fusion"] >= N diff --git a/asian_ai/performance/vLLM/tests/compile/silly_attention.py b/asian_ai/performance/vLLM/tests/compile/silly_attention.py new file mode 100644 index 0000000000000000000000000000000000000000..2b391e18266500c73b335b0b9aaa4a3d0dae1386 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/silly_attention.py @@ -0,0 +1,65 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Shared PyTorch custom silly attention for compilation tests. +Centralizes custom operation definitions to avoid duplicate registrations. +""" + +import torch +from torch.library import Library + +from vllm.utils.torch_utils import direct_register_custom_op + +# Shared library for all compilation test operations +# Using "silly" namespace to match existing test expectations +# import this file will automatically register +# torch ops for testing (like silly.attention) +silly_lib = Library("silly", "FRAGMENT") + +# Global counter that counts the number of times attention is invoked +_global_counter = 0 + + +def get_global_counter(): + """Get the current global counter value""" + return _global_counter + + +def reset_global_counter(): + """Reset the global counter to 0""" + global _global_counter + _global_counter = 0 + + +def silly_attention( + q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, out: torch.Tensor +) -> None: + """ + Unified attention implementation that depends on + all inputs and affects the output. + Always increments a global counter that tests can use or ignore. + """ + global _global_counter + + # Always increment the global counter + _global_counter += 1 + + # Unified implementation that depends on all inputs + out.copy_(q + k + v) + + +def silly_attention_fake( + q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, out: torch.Tensor +) -> None: + """Fake implementation for testing""" + return + + +# Register the unified attention operation +direct_register_custom_op( + op_name="attention", + op_func=silly_attention, + mutates_args=["out"], + fake_impl=silly_attention_fake, + target_lib=silly_lib, +) diff --git a/asian_ai/performance/vLLM/tests/compile/test_aot_compile.py b/asian_ai/performance/vLLM/tests/compile/test_aot_compile.py new file mode 100644 index 0000000000000000000000000000000000000000..a387dbae20c8167fce61ba14afedb532cf6f57d9 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/test_aot_compile.py @@ -0,0 +1,912 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import hashlib +import os +import pickle +import tempfile +from contextlib import contextmanager +from pathlib import Path +from unittest.mock import Mock, patch + +import pytest +import torch + +import vllm.envs as envs +from vllm.compilation.backends import VllmBackend +from vllm.compilation.caching import ( + StandaloneCompiledArtifacts, + VllmSerializableFunction, +) +from vllm.compilation.counter import compilation_counter +from vllm.compilation.decorators import support_torch_compile +from vllm.config import ( + CompilationConfig, + CompilationMode, + VllmConfig, + set_current_vllm_config, +) +from vllm.envs import disable_envs_cache +from vllm.forward_context import set_forward_context +from vllm.utils.torch_utils import is_torch_equal_or_newer + +from ..utils import create_new_process_for_each_test + + +@pytest.fixture +def vllm_tmp_cache(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + """Fixture that sets VLLM_CACHE_ROOT to a temporary directory.""" + monkeypatch.setenv("VLLM_CACHE_ROOT", str(tmp_path / "vllm_cache")) + return tmp_path + + +def reference_fn(x: torch.Tensor): + assert x.shape[0] <= 42 + assert x.shape[0] % 2 == 0 + for _ in range(3000): + x = x + x.shape[0] + return x + + +def reference_fn_tuple(x: torch.Tensor): + """Reference function that returns a tuple of tensors.""" + assert x.shape[0] <= 42 + assert x.shape[0] % 2 == 0 + for _ in range(3000): + x = x + x.shape[0] + return x, x * 2 + + +@support_torch_compile +class CompiledMod(torch.nn.Module): + def __init__(self, **kwargs): + super().__init__() + + def forward(self, x: torch.Tensor): + return reference_fn(x) + + +@support_torch_compile +class CompiledModTuple(torch.nn.Module): + """A compiled module that returns a tuple of tensors.""" + + def __init__(self, **kwargs): + super().__init__() + + def forward(self, x: torch.Tensor): + return reference_fn_tuple(x) + + +def make_vllm_config() -> VllmConfig: + return VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + backend="inductor", + ) + ) + + +@contextmanager +def use_vllm_config(vllm_config: VllmConfig): + with set_forward_context({}, vllm_config), set_current_vllm_config(vllm_config): + yield + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +def test_no_dynamo_cache_entry(monkeypatch: pytest.MonkeyPatch): + with monkeypatch.context() as m: + vllm_config = make_vllm_config() + args = (torch.randn(10, 10),) + expected = reference_fn(*args) + with use_vllm_config(vllm_config): + m.setenv("VLLM_USE_AOT_COMPILE", "0") + m.setenv("VLLM_USE_MEGA_AOT_ARTIFACT", "1") + m.setenv("VLLM_USE_STANDALONE_COMPILE", "1") + with ( + pytest.raises(RuntimeError, match="Detected recompile"), + torch.compiler.set_stance("fail_on_recompile"), + ): + CompiledMod(vllm_config=vllm_config)(*args) + disable_envs_cache() + + m.setenv("VLLM_USE_AOT_COMPILE", "1") + torch._dynamo.reset() + with torch.compiler.set_stance("fail_on_recompile"): + actual = CompiledMod(vllm_config=vllm_config)(*args) + assert torch.allclose(actual, expected) + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +def test_force_aot_load(monkeypatch: pytest.MonkeyPatch): + with tempfile.TemporaryDirectory() as tmpdirname, monkeypatch.context() as m: + args = (torch.randn(10, 10),) + m.setenv("VLLM_USE_AOT_COMPILE", "1") + m.setenv("VLLM_USE_MEGA_AOT_ARTIFACT", "1") + m.setenv("VLLM_USE_STANDALONE_COMPILE", "1") + m.setenv("VLLM_FORCE_AOT_LOAD", "1") + m.setenv("VLLM_CACHE_ROOT", tmpdirname) + vllm_config = make_vllm_config() + with use_vllm_config(vllm_config), pytest.raises(FileNotFoundError): + CompiledMod(vllm_config=vllm_config)(*args) + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +def test_save_and_load(monkeypatch: pytest.MonkeyPatch): + with monkeypatch.context() as m: + args = (torch.randn(10, 10),) + + with tempfile.TemporaryDirectory() as tmpdirname: + m.setenv("VLLM_CACHE_ROOT", tmpdirname) + m.setenv("VLLM_USE_AOT_COMPILE", "1") + m.setenv("VLLM_USE_MEGA_AOT_ARTIFACT", "1") + m.setenv("VLLM_USE_STANDALONE_COMPILE", "1") + vllm_config = make_vllm_config() + with use_vllm_config(vllm_config): + compiled_mod = CompiledMod(vllm_config=vllm_config) + expected = compiled_mod(*args) + + disable_envs_cache() + + m.setenv("VLLM_FORCE_AOT_LOAD", "1") + vllm_config = make_vllm_config() + with use_vllm_config(vllm_config): + cached_mod = CompiledMod(vllm_config=vllm_config) + ret = cached_mod(*args) + assert cached_mod.was_aot_compile_fn_loaded_from_disk, ( + "Expected was_aot_compile_fn_loaded_from_disk to be True" + ) + assert torch.allclose(ret, expected) + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +def test_save_and_load_slice(monkeypatch: pytest.MonkeyPatch): + from torch._subclasses import FakeTensorMode + from torch.fx.experimental.symbolic_shapes import ShapeEnv + + def foo(x: torch.Tensor): + return x[slice(0, x.shape[0])] + + vllm_config = make_vllm_config() + + example_input = torch.randn(10, 10) + torch._dynamo.mark_dynamic(example_input, 0) + gm = torch.fx.symbolic_trace(foo) + assert "getitem_1 = x[slice(0, getitem, None)]" in gm.code + with use_vllm_config(vllm_config): + payload = VllmSerializableFunction.serialize_graph_module(gm) + fake_mode = FakeTensorMode(shape_env=ShapeEnv()) + loaded_gm = VllmSerializableFunction.deserialize_graph_module( + payload, fake_mode + ) + + assert gm.code == loaded_gm.code + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +def test_cache_load_returns_tuple_consistency(monkeypatch: pytest.MonkeyPatch): + """ + Test that cache loading correctly handles the returns_tuple logic. + + This verifies that when a model returns a single tensor (not a tuple), + the output type is consistent between fresh compilation and cache load. + Without the fix, cached artifacts would return [tensor] instead of tensor. + """ + with monkeypatch.context() as m: + args = (torch.randn(10, 10),) + + with tempfile.TemporaryDirectory() as tmpdirname: + m.setenv("VLLM_CACHE_ROOT", tmpdirname) + m.setenv("VLLM_USE_AOT_COMPILE", "1") + m.setenv("VLLM_USE_MEGA_AOT_ARTIFACT", "1") + m.setenv("VLLM_USE_STANDALONE_COMPILE", "1") + vllm_config = make_vllm_config() + + # Fresh compilation + with use_vllm_config(vllm_config): + compiled_mod = CompiledMod(vllm_config=vllm_config) + fresh_result = compiled_mod(*args) + fresh_result_type = type(fresh_result) + + # Verify fresh result is a tensor, not a tuple/list + assert isinstance(fresh_result, torch.Tensor), ( + f"Fresh compile should return tensor, got {fresh_result_type}" + ) + + disable_envs_cache() + + # Load from cache + m.setenv("VLLM_FORCE_AOT_LOAD", "1") + vllm_config = make_vllm_config() + with use_vllm_config(vllm_config): + cached_mod = CompiledMod(vllm_config=vllm_config) + cached_result = cached_mod(*args) + cached_result_type = type(cached_result) + + # Verify cache was actually loaded + assert cached_mod.was_aot_compile_fn_loaded_from_disk, ( + "Expected was_aot_compile_fn_loaded_from_disk to be True after " + "loading from cache" + ) + + # Verify cached result has same type as fresh result + assert isinstance(cached_result, torch.Tensor), ( + f"Cache load should return tensor, got {cached_result_type}. " + "This indicates the returns_tuple logic is not being applied " + "correctly when loading from cache." + ) + + # Verify values match + assert torch.allclose(cached_result, fresh_result), ( + "Cached result values should match fresh compilation" + ) + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +def test_cache_load_returns_tuple_consistency_tuple_output( + monkeypatch: pytest.MonkeyPatch, +): + """ + Test that cache loading correctly handles models that return tuples. + + This verifies that when a model returns a tuple of tensors, the output + type is preserved as a tuple between fresh compilation and cache load. + """ + with monkeypatch.context() as m: + args = (torch.randn(10, 10),) + + with tempfile.TemporaryDirectory() as tmpdirname: + m.setenv("VLLM_CACHE_ROOT", tmpdirname) + m.setenv("VLLM_USE_AOT_COMPILE", "1") + m.setenv("VLLM_USE_MEGA_AOT_ARTIFACT", "1") + m.setenv("VLLM_USE_STANDALONE_COMPILE", "1") + vllm_config = make_vllm_config() + + # Fresh compilation with tuple-returning model + with use_vllm_config(vllm_config): + compiled_mod = CompiledModTuple(vllm_config=vllm_config) + fresh_result = compiled_mod(*args) + fresh_result_type = type(fresh_result) + + # Verify fresh result is a tuple + assert isinstance(fresh_result, tuple), ( + f"Fresh compile should return tuple, got {fresh_result_type}" + ) + assert len(fresh_result) == 2, ( + f"Fresh compile should return 2-tuple, got {len(fresh_result)}" + ) + + disable_envs_cache() + + # Load from cache + m.setenv("VLLM_FORCE_AOT_LOAD", "1") + vllm_config = make_vllm_config() + with use_vllm_config(vllm_config): + cached_mod = CompiledModTuple(vllm_config=vllm_config) + cached_result = cached_mod(*args) + cached_result_type = type(cached_result) + + # Verify cache was actually loaded + assert cached_mod.was_aot_compile_fn_loaded_from_disk, ( + "Expected was_aot_compile_fn_loaded_from_disk to be True after " + "loading from cache" + ) + + # Verify cached result is also a tuple + assert isinstance(cached_result, tuple), ( + f"Cache load should return tuple, got {cached_result_type}. " + "This indicates the returns_tuple logic is not preserving " + "tuple outputs when loading from cache." + ) + assert len(cached_result) == 2, ( + f"Cache load should return 2-tuple, got {len(cached_result)}" + ) + + # Verify values match + assert torch.allclose(cached_result[0], fresh_result[0]), ( + "Cached result[0] values should match fresh compilation" + ) + assert torch.allclose(cached_result[1], fresh_result[1]), ( + "Cached result[1] values should match fresh compilation" + ) + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +def test_shape_env(monkeypatch: pytest.MonkeyPatch): + """ + Test that the shape environment is correctly serialized and preserved + when loading from cache. + """ + with monkeypatch.context() as m: + args = (torch.randn(10, 10),) + + with tempfile.TemporaryDirectory() as tmpdirname: + m.setenv("VLLM_CACHE_ROOT", tmpdirname) + m.setenv("VLLM_USE_AOT_COMPILE", "1") + m.setenv("VLLM_USE_MEGA_AOT_ARTIFACT", "1") + m.setenv("VLLM_USE_STANDALONE_COMPILE", "1") + vllm_config = make_vllm_config() + with use_vllm_config(vllm_config): + compiled_mod = CompiledMod(vllm_config=vllm_config) + compiled_mod(*args) + artifacts = compiled_mod.aot_compiled_fn._artifacts + guards_string = artifacts.compiled_fn.shape_env.format_guards() + assert guards_string == " - s77 <= 42\n - Eq(Mod(s77, 2), 0)" + + disable_envs_cache() + + m.setenv("VLLM_FORCE_AOT_LOAD", "1") + vllm_config = make_vllm_config() + with use_vllm_config(vllm_config): + compiled_mod = CompiledMod(vllm_config=vllm_config) + compiled_mod(*args) + assert compiled_mod.was_aot_compile_fn_loaded_from_disk, ( + "Expected was_aot_compile_fn_loaded_from_disk to be True" + ) + artifacts = compiled_mod.aot_compiled_fn._artifacts + guards_string = artifacts.compiled_fn.shape_env.format_guards() + assert guards_string == " - s77 <= 42\n - Eq(Mod(s77, 2), 0)" + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +def test_partition_wrapper_applied_on_aot_load( + monkeypatch: pytest.MonkeyPatch, vllm_tmp_cache: Path, mocker +): + """ + Test that partition wrappers are applied when loading AOT cached functions. + + This test verifies the fix for GitHub issue #31439 where AOT compile + caused 2x latency regression when use_inductor_graph_partition=True. + The root cause was that partition wrapper context was bypassed when + loading from AOT cache. + """ + from vllm.config import CUDAGraphMode + + args = (torch.randn(10, 10),) + monkeypatch.setenv("VLLM_USE_AOT_COMPILE", "1") + + # Create config with partition enabled + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + use_inductor_graph_partition=True, + cudagraph_mode=CUDAGraphMode.PIECEWISE, + ) + ) + + # First compilation - save to cache + with use_vllm_config(vllm_config): + compiled_mod = CompiledMod(vllm_config=vllm_config) + compiled_mod(*args) + + disable_envs_cache() + + # Second run - load from cache, verify partition wrapper applied + monkeypatch.setenv("VLLM_FORCE_AOT_LOAD", "1") + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + use_inductor_graph_partition=True, + cudagraph_mode=CUDAGraphMode.PIECEWISE, + ) + ) + + # Use mocker to spy on set_customized_partition_wrappers + spy = mocker.spy(torch._inductor.utils, "set_customized_partition_wrappers") + + with use_vllm_config(vllm_config): + compiled_mod = CompiledMod(vllm_config=vllm_config) + + # First call after restart: loads from AOT cache. + # This tests the fix for the first call after a restart. + compiled_mod(*args) + + # Verify cache was loaded + assert compiled_mod.was_aot_compile_fn_loaded_from_disk, ( + "Expected was_aot_compile_fn_loaded_from_disk to be True" + ) + + # Verify partition wrapper was called on AOT load. + assert spy.call_count >= 2, ( + "Expected partition wrapper to be set and cleared on AOT load, " + f"got {spy.call_count} calls" + ) + # First call should set a wrapper, last call should clear it + assert spy.call_args_list[0][0][0] is not None, ( + "First call on AOT load should set a wrapper function" + ) + assert spy.call_args_list[-1][0][0] is None, ( + "Last call on AOT load should clear the wrapper" + ) + + # Reset for the next check. + spy.reset_mock() + + # Subsequent call: uses the cached `aot_compiled_fn`. + # This tests the fix for subsequent calls. + compiled_mod(*args) + + # Verify partition wrapper was called on the subsequent call. + assert spy.call_count >= 2, ( + "Expected partition wrapper set and cleared on subsequent " + f"call, got {spy.call_count} calls" + ) + assert spy.call_args_list[0][0][0] is not None, ( + "First call on subsequent call should set a wrapper function" + ) + assert spy.call_args_list[-1][0][0] is None, ( + "Last call on subsequent call should clear the wrapper" + ) + + +@create_new_process_for_each_test("spawn") +def test_standalone_compile_correctness(): + """Outputs must match regardless of VLLM_USE_STANDALONE_COMPILE.""" + import json + + from ..utils import compare_two_settings + + compilation_config = json.dumps( + { + "mode": CompilationMode.VLLM_COMPILE, + } + ) + + common_args = [ + "--dtype", + "float16", + "--max-model-len", + "256", + "--compilation_config", + compilation_config, + ] + + compare_two_settings( + "facebook/opt-125m", + common_args, + common_args, + env1={"VLLM_USE_STANDALONE_COMPILE": "1"}, + env2={ + "VLLM_USE_STANDALONE_COMPILE": "0", + "VLLM_USE_MEGA_AOT_ARTIFACT": "0", + }, + ) + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +@create_new_process_for_each_test("spawn") +def test_gpt2_cache_hit(monkeypatch: pytest.MonkeyPatch): + """ + Test that compiling gpt2 twice results in a cache hit. + + Counter values are read from the EngineCore subprocess via + ``LLM.collective_rpc`` so the test works under default V1 + multiprocessing (no shared memory between test and engine). + """ + + from vllm import LLM + + def _snap(self): + from vllm.compilation.counter import compilation_counter + + return ( + compilation_counter.num_aot_compiles, + compilation_counter.num_aot_artifacts_saved, + compilation_counter.num_aot_artifacts_loaded, + ) + + # collective_rpc(callable) requires pickle-based serialization. + monkeypatch.setenv("VLLM_ALLOW_INSECURE_SERIALIZATION", "1") + + with monkeypatch.context() as m, tempfile.TemporaryDirectory() as tmpdirname: + m.setenv("VLLM_CACHE_ROOT", tmpdirname) + m.setenv("VLLM_USE_AOT_COMPILE", "1") + # First compilation - initialize model and generate + llm_model = LLM( + model="gpt2", + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + ), + max_model_len=256, + ) + + llm_model.generate("Hello, my name is") + assert llm_model.collective_rpc(_snap)[0] == (1, 1, 0) + + # Clean up first model + del llm_model + disable_envs_cache() + + # Second compilation - should hit cache + m.setenv("VLLM_FORCE_AOT_LOAD", "1") + llm_model = LLM( + model="gpt2", + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + ), + max_model_len=256, + ) + llm_model.generate("Hello, my name is") + assert llm_model.collective_rpc(_snap)[0] == (0, 0, 1) + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +class TestStandaloneCompiledArtifacts: + def test_init(self): + cache = StandaloneCompiledArtifacts() + assert cache.submodule_bytes == {} + assert cache.submodule_bytes_store == {} + assert cache.loaded_submodule_store == {} + + def test_insert_new_artifact(self): + cache = StandaloneCompiledArtifacts() + test_data = b"test_artifact_data" + submod_name = "test_submod" + shape = "s1" + + hasher = hashlib.sha256() + hasher.update(test_data) + expected_hash = hasher.hexdigest() + + cache.insert(submod_name, shape, test_data) + + assert f"{submod_name}_{shape}" in cache.submodule_bytes + assert cache.submodule_bytes[f"{submod_name}_{shape}"] == expected_hash + assert expected_hash in cache.submodule_bytes_store + assert cache.submodule_bytes_store[expected_hash] == test_data + + def test_insert_duplicate_artifact(self): + cache = StandaloneCompiledArtifacts() + + test_data = b"duplicate_test_data" + submod_name1 = "submod1" + submod_name2 = "submod2" + shape = "s2" + + cache.insert(submod_name1, shape, test_data) + cache.insert(submod_name2, shape, test_data) + + hash1 = cache.submodule_bytes[f"{submod_name1}_{shape}"] + hash2 = cache.submodule_bytes[f"{submod_name2}_{shape}"] + assert hash1 == hash2 + + assert len(cache.submodule_bytes_store) == 1 + assert len(cache.submodule_bytes) == 2 + + def test_get_artifact(self): + cache = StandaloneCompiledArtifacts() + test_data = b"retrievable_data" + submod_name = "mod1" + shape = "shape16" + + cache.insert(submod_name, shape, test_data) + retrieved_data = cache.get(submod_name, shape) + + assert retrieved_data == test_data + + def test_get_nonexistent_artifact(self): + cache = StandaloneCompiledArtifacts() + + with pytest.raises(KeyError): + cache.get("nonexistent", "shape") + + def test_size_bytes(self): + cache = StandaloneCompiledArtifacts() + + assert cache.size_bytes() == 0 + + data1 = b"x" * 100 + data2 = b"y" * 200 + cache.insert("mod1", "shape1", data1) + cache.insert("mod2", "shape2", data2) + + assert cache.size_bytes() == 300 + + def test_num_artifacts_and_entries(self): + cache = StandaloneCompiledArtifacts() + + assert cache.num_artifacts() == 0 + assert cache.num_entries() == 0 + + cache.insert("mod1", "shape1", b"data1") + cache.insert("mod2", "shape2", b"data2") + assert cache.num_artifacts() == 2 + assert cache.num_entries() == 2 + + cache.insert("mod3", "shape3", b"data1") + assert cache.num_artifacts() == 2 + assert cache.num_entries() == 3 + + @patch("torch._inductor.standalone_compile.AOTCompiledArtifact.deserialize") + def test_load_all_success(self, mock_deserialize): + """Test successful loading of all artifacts""" + cache = StandaloneCompiledArtifacts() + + mock_artifact1 = Mock() + mock_artifact2 = Mock() + mock_deserialize.side_effect = [mock_artifact1, mock_artifact2] + + cache.insert("mod1", "shape1", pickle.dumps(b"data1")) + cache.insert("mod2", "shape2", pickle.dumps(b"data2")) + + cache.load_all() + + assert len(cache.loaded_submodule_store) == 2 + assert mock_deserialize.call_count == 2 + + @patch("torch._inductor.standalone_compile.AOTCompiledArtifact.deserialize") + def test_load_all_already_loaded(self, mock_deserialize): + """Test that load_all skips if already loaded""" + cache = StandaloneCompiledArtifacts() + + mock_artifact = Mock() + cache.submodule_bytes_store["hash1"] = pickle.dumps(b"data1") + cache.loaded_submodule_store["hash1"] = mock_artifact + + cache.load_all() + + mock_deserialize.assert_not_called() + + @patch("torch._inductor.standalone_compile.AOTCompiledArtifact.deserialize") + def test_get_loaded_artifact(self, mock_deserialize): + """Test retrieving loaded artifacts""" + cache = StandaloneCompiledArtifacts() + + mock_artifact = Mock() + mock_deserialize.return_value = mock_artifact + + submod_name = "test_mod" + shape = "test_shape" + cache.insert(submod_name, shape, pickle.dumps(b"test_data")) + cache.load_all() + + retrieved_artifact = cache.get_loaded(submod_name, shape) + assert retrieved_artifact == mock_artifact + + def test_getstate_setstate(self): + cache = StandaloneCompiledArtifacts() + + cache.insert("mod1", "shape1", b"data1") + cache.insert("mod2", "shape2", b"data2") + + cache.loaded_submodule_store["hash1"] = Mock() + + state = cache.__getstate__() + + assert "submodule_bytes" in state + assert "submodule_bytes_store" in state + assert "loaded_submodule_store" not in state + + new_cache = StandaloneCompiledArtifacts() + new_cache.__setstate__(state) + + assert new_cache.submodule_bytes == cache.submodule_bytes + assert new_cache.submodule_bytes_store == cache.submodule_bytes_store + assert new_cache.loaded_submodule_store == {} + + def test_pickle_roundtrip(self): + cache = StandaloneCompiledArtifacts() + + test_data1 = b"pickle_test_data_1" + test_data2 = b"pickle_test_data_2" + cache.insert("mod1", "shape1", test_data1) + cache.insert("mod2", "shape2", test_data2) + + pickled_data = pickle.dumps(cache) + restored_cache = pickle.loads(pickled_data) + + assert restored_cache.get("mod1", "shape1") == test_data1 + assert restored_cache.get("mod2", "shape2") == test_data2 + assert restored_cache.num_artifacts() == cache.num_artifacts() + assert restored_cache.num_entries() == cache.num_entries() + assert restored_cache.size_bytes() == cache.size_bytes() + + assert len(restored_cache.loaded_submodule_store) == 0 + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +class TestStandaloneCompiledArtifactsIntegration: + def test_add_pickle_unpickle(self): + cache = StandaloneCompiledArtifacts() + + artifacts = { + ("mod1", "shape1"): b"m1s1_artifact", + ("mod1", "shape2"): b"m1s2_artifact", + ("mod2", "shape1"): b"m2s1_artifact", + ("mod2", "shape2"): b"m2s2_artifact", + } + + for (submod, shape), data in artifacts.items(): + cache.insert(submod, shape, data) + + assert cache.num_entries() == 4 + assert cache.num_artifacts() == 4 + + for (submod, shape), expected_data in artifacts.items(): + retrieved_data = cache.get(submod, shape) + assert retrieved_data == expected_data + + pickled = pickle.dumps(cache) + restored_cache = pickle.loads(pickled) + + for (submod, shape), expected_data in artifacts.items(): + retrieved_data = restored_cache.get(submod, shape) + assert retrieved_data == expected_data + + def test_deduplication(self): + cache = StandaloneCompiledArtifacts() + + shared_data = b"shared_artifact_data" * 1000 + + cache.insert("mod1", "shape1", shared_data) + cache.insert("mod2", "shape1", shared_data) + cache.insert("mod1", "shape2", shared_data) + cache.insert("mod3", "shape3", shared_data) + + assert cache.num_entries() == 4 + assert cache.num_artifacts() == 1 + assert cache.size_bytes() == len(shared_data) + + for submod, shape in [ + ("mod1", "shape1"), + ("mod2", "shape1"), + ("mod1", "shape2"), + ("mod3", "shape3"), + ]: + assert cache.get(submod, shape) == shared_data + + @pytest.mark.skipif( + envs.VLLM_USE_MEGA_AOT_ARTIFACT, + reason="There's no AOT Autograd run with mega artifact", + ) + def test_functorch_config(self): + vllm_config = make_vllm_config() + example_inputs = (torch.randn(10, 10),) + + def add_1(x: torch.Tensor): + return x + 1 + + gm = torch._dynamo.functional_export.dynamo_graph_capture_for_export(add_1)( + *example_inputs + ) + + gm.graph._codegen = torch.fx.graph.CodeGen() + gm._dynamo_bytecode_flatten = None + gm._dynamo_bytecode_unflatten = None + + with ( + torch._functorch.config.patch(bundled_autograd_cache=False), + set_current_vllm_config(vllm_config), + ): + with torch._functorch.config.patch(bundled_autograd_cache=True): + fn = VllmSerializableFunction(gm, example_inputs, "", add_1) + + payload = VllmSerializableFunction.serialize_compile_artifacts(fn) + + config = None + + def backend(*args, **kwargs) -> VllmSerializableFunction: + nonlocal config + # bundled_autograd_cache should be True even compiler backend + # runs with bundled_autograd_cache=False in ambient context. + config = torch._functorch.config.save_config_portable() + return fn + + loaded_fn = VllmSerializableFunction.deserialize_compile_artifacts(payload) + with patch.object(VllmBackend, "__call__", backend): + loaded_fn(*example_inputs) + + assert isinstance(config, dict) + assert "bundled_autograd_cache" in config + assert config["bundled_autograd_cache"] is True + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +def test_disable_compile_cache_skips_aot_save( + monkeypatch: pytest.MonkeyPatch, fresh_vllm_cache: str +): + """When VLLM_DISABLE_COMPILE_CACHE=1, AOT artifacts must not be saved.""" + monkeypatch.setenv("VLLM_DISABLE_COMPILE_CACHE", "1") + monkeypatch.setenv("VLLM_USE_AOT_COMPILE", "1") + disable_envs_cache() + + args = (torch.randn(10, 10),) + expected = reference_fn(*args) + vllm_config = make_vllm_config() + + with ( + use_vllm_config(vllm_config), + compilation_counter.expect( + num_aot_compiles=1, + num_aot_artifacts_saved=0, + num_aot_artifacts_loaded=0, + ), + ): + mod = CompiledMod(vllm_config=vllm_config) + actual = mod(*args) + + assert torch.allclose(actual, expected) + + # No cached artifact should exist on disk + aot_dir = os.path.join(fresh_vllm_cache, "torch_compile_cache", "torch_aot_compile") + if os.path.isdir(aot_dir): + for root, _dirs, files in os.walk(aot_dir): + for f in files: + assert f != "model", ( + f"AOT artifact unexpectedly saved at {os.path.join(root, f)}" + ) + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +def test_disable_compile_cache_skips_aot_load( + monkeypatch: pytest.MonkeyPatch, fresh_vllm_cache: str +): + """When VLLM_DISABLE_COMPILE_CACHE=1, AOT artifacts must not be loaded.""" + # Phase 1: compile and save with cache enabled + monkeypatch.setenv("VLLM_USE_AOT_COMPILE", "1") + disable_envs_cache() + + args = (torch.randn(10, 10),) + vllm_config = make_vllm_config() + + with ( + use_vllm_config(vllm_config), + compilation_counter.expect(num_aot_artifacts_saved=1), + ): + CompiledMod(vllm_config=vllm_config)(*args) + + # Phase 2: disable cache, compile again — should NOT load from disk + monkeypatch.setenv("VLLM_DISABLE_COMPILE_CACHE", "1") + disable_envs_cache() + torch._dynamo.reset() + + vllm_config = make_vllm_config() + with ( + use_vllm_config(vllm_config), + compilation_counter.expect( + num_aot_compiles=1, + num_aot_artifacts_saved=0, + num_aot_artifacts_loaded=0, + ), + ): + mod = CompiledMod(vllm_config=vllm_config) + mod(*args) + + assert not mod.was_aot_compile_fn_loaded_from_disk + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +def test_aot_counters_on_save_and_load( + monkeypatch: pytest.MonkeyPatch, fresh_vllm_cache: str +): + """Verify AOT counters are incremented correctly on save and load.""" + monkeypatch.setenv("VLLM_USE_AOT_COMPILE", "1") + disable_envs_cache() + + args = (torch.randn(10, 10),) + + # Phase 1: fresh compile + save + vllm_config = make_vllm_config() + with ( + use_vllm_config(vllm_config), + compilation_counter.expect( + num_aot_compiles=1, + num_aot_artifacts_saved=1, + num_aot_artifacts_loaded=0, + ), + ): + CompiledMod(vllm_config=vllm_config)(*args) + + # Phase 2: load from cache + monkeypatch.setenv("VLLM_FORCE_AOT_LOAD", "1") + disable_envs_cache() + + vllm_config = make_vllm_config() + with ( + use_vllm_config(vllm_config), + compilation_counter.expect( + num_aot_compiles=0, + num_aot_artifacts_saved=0, + num_aot_artifacts_loaded=1, + ), + ): + CompiledMod(vllm_config=vllm_config)(*args) diff --git a/asian_ai/performance/vLLM/tests/compile/test_codegen.py b/asian_ai/performance/vLLM/tests/compile/test_codegen.py new file mode 100644 index 0000000000000000000000000000000000000000..095859f91f8ae1e5e8b4268aadd0c0e3b2b7c228 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/test_codegen.py @@ -0,0 +1,376 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for vllm.compilation.codegen — execution code generation. + +Each test runs a real Python function through the same pipeline vLLM uses +in production: ``make_fx`` to obtain an aten-level fx graph, ``split_graph`` +to split it into the stitching layer + submodules, and then +``generate_execution_code``/``compile_execution_fn`` for codegen. +""" + +from collections.abc import Callable + +import pytest +import regex as re +import torch +import torch.fx as fx +from torch.fx.experimental.proxy_tensor import make_fx + +from vllm.compilation.backends import split_graph +from vllm.compilation.codegen import ( + _node_ref, + compile_execution_fn, + generate_execution_code, + generate_execution_code_with_name, +) +from vllm.utils.torch_utils import is_torch_equal_or_newer + + +def _trace_and_split( + model_fn: Callable[..., torch.Tensor], + example_inputs: tuple[torch.Tensor, ...], + split_ops: list[str], +) -> fx.GraphModule: + """Trace ``model_fn`` with make_fx, then split on the named aten ops.""" + gm = make_fx(model_fn)(*example_inputs) + split_gm, _ = split_graph(gm, split_ops) + return split_gm + + +def _to_copy_model(x: torch.Tensor) -> torch.Tensor: + """Traces to ``aten._to_copy.default`` with device + dtype kwargs.""" + return x.to(device=torch.device("cpu"), dtype=torch.float16) + + +def _empty_model(x: torch.Tensor) -> torch.Tensor: + """Traces to ``aten.empty.memory_format`` with device + dtype kwargs.""" + buf = torch.empty(x.shape, device=torch.device("cpu"), dtype=torch.float16) + return buf.fill_(0).add(x.to(dtype=torch.float16)) + + +@pytest.fixture +def x() -> torch.Tensor: + return torch.zeros(2, 3) + + +@pytest.mark.parametrize( + "model_fn,split_ops", + [ + (_to_copy_model, ["aten::_to_copy.default"]), + (_empty_model, []), + ], + ids=["aten::_to_copy.default", "aten::empty.memory_format"], +) +def test_non_primitive_kwargs_lifted_to_consts( + model_fn: Callable[[torch.Tensor], torch.Tensor], + split_ops: list[str], + x: torch.Tensor, +) -> None: + """Regression: arguments whose ``repr()`` is not a valid Python + expression in the generated function's namespace (notably + ``torch.device``) used to be inlined via ``repr()``, producing source + like + + out = torch.ops.aten._to_copy.default(x, device=device(type='cpu')) + + which fails at call time — only ``torch`` and ``operator`` are imported + into the namespace, so ``device`` is unbound. The fix collects such + objects into ``__vllm_consts__`` and references them by index. The + unqualified ``device(type=...)`` form must never appear in the + generated source.""" + split_gm = _trace_and_split(model_fn, (x,), split_ops) + code, submod_names, consts = generate_execution_code(split_gm) + + assert "device(type=" not in code, ( + "Generated code contains unqualified `device(type=...)` from repr(); " + "torch.device should be lifted into __vllm_consts__" + ) + assert torch.device("cpu") in consts, "torch.device kwarg not lifted to consts" + assert torch.float16 in consts, "torch.dtype kwarg not lifted to consts" + + fn = compile_execution_fn(code, {}, submod_names, consts) + out = fn(x) + expected = model_fn(x) + assert torch.equal(out, expected), "Compiled output does not match reference" + + +def test_dtype_singleton_deduped(x: torch.Tensor) -> None: + """``torch.float16`` is a process-wide singleton, so two ops referring + to it in the traced graph share a single consts slot via ``id()``-based + dedup. Distinct expressions (``x.to(...)`` vs ``(x*2).to(...)``) ensure + the tracer can't CSE the two ops into a single node.""" + + def model_fn(x: torch.Tensor) -> torch.Tensor: + return x.to(dtype=torch.float16) + (x * 2).to(dtype=torch.float16) + + split_gm = _trace_and_split(model_fn, (x,), []) + code, submod_names, consts = generate_execution_code(split_gm) + + # The traced graph must have two distinct _to_copy nodes (otherwise the + # dedup assertion below is trivially satisfied). + n_to_copy = sum( + 1 + for n in split_gm.graph.nodes + if n.op == "call_module" + for sn in getattr(split_gm, n.target).graph.nodes + if sn.op == "call_function" and "to_copy" in sn.name + ) + assert n_to_copy >= 2, ( + f"Test setup failed: expected ≥2 _to_copy nodes, got {n_to_copy}" + ) + + assert consts.count(torch.float16) == 1, ( + f"torch.float16 should occupy exactly one slot, got consts={consts}" + ) + assert code.count("__vllm_consts__[0]") >= 2, ( + "Deduped const slot should be referenced from both _to_copy nodes" + ) + + fn = compile_execution_fn(code, {}, submod_names, consts) + assert torch.equal(fn(x), model_fn(x)) + + +def test_distinct_dtypes_get_distinct_slots(x: torch.Tensor) -> None: + """Distinct dtype singletons in the traced graph occupy distinct slots.""" + + def model_fn(x: torch.Tensor) -> torch.Tensor: + return x.to(dtype=torch.float16) + x.to(dtype=torch.bfloat16) + + split_gm = _trace_and_split(model_fn, (x,), []) + _, _, consts = generate_execution_code(split_gm) + + assert torch.float16 in consts + assert torch.bfloat16 in consts + assert len(consts) == 2, f"Expected 2 distinct dtype slots, got {consts}" + + +def test_consts_ordering_deterministic(x: torch.Tensor) -> None: + """Two independent traces of the same model must produce equal consts + lists *in the same order*. Cache artifacts identify const slots by + index, so a non-deterministic order would invalidate cached code.""" + + def model_fn(x: torch.Tensor) -> torch.Tensor: + # Multiple distinct non-primitives encountered in a fixed graph order. + a = x.to(device=torch.device("cpu"), dtype=torch.float16) + return a.to(dtype=torch.bfloat16) + + _, _, consts1 = generate_execution_code(_trace_and_split(model_fn, (x,), [])) + _, _, consts2 = generate_execution_code(_trace_and_split(model_fn, (x,), [])) + + assert len(consts1) >= 2, "Test setup: model should produce ≥2 const slots" + assert consts1 == consts2, ( + f"consts ordering must be reproducible across traces; " + f"got {consts1} vs {consts2}" + ) + + +def test_primitive_args_inlined(x: torch.Tensor) -> None: + """Primitive args (int dim, etc.) stay inline as repr — no consts.""" + + def model_fn(x: torch.Tensor) -> torch.Tensor: + return torch.transpose(x, 0, 1).relu() + + split_gm = _trace_and_split(model_fn, (x,), []) + code, submod_names, consts = generate_execution_code(split_gm) + + assert consts == [], "Primitive-only graph must produce empty consts" + + fn = compile_execution_fn(code, {}, submod_names, consts) + assert torch.equal(fn(x), model_fn(x)) + + +def test_consts_shared_across_split_submods(x: torch.Tensor) -> None: + """Dedup must apply across inlined submodules, not just within one. + + The function below splits into three inlined submods, two of which + independently reference ``torch.float16``. The shared ``const_index`` + threaded through recursive ``generate_execution_code_with_name`` calls + must collapse the dtype to a single slot used from both submods.""" + + def model_fn(x: torch.Tensor) -> torch.Tensor: + a = x.to(dtype=torch.float16) # submod_0: _to_copy(fp16) + b = a.relu() # submod_1: relu (split point) + c = b.to(dtype=torch.float32) # submod_2: _to_copy(fp32) + return c.to(dtype=torch.float16) + 1 # submod_2: another _to_copy(fp16) + + split_gm = _trace_and_split(model_fn, (x,), ["aten::relu.default"]) + + n_submods = sum(1 for _ in split_gm.named_children()) + assert n_submods >= 3, ( + f"Test setup failed: expected ≥3 submods after split, got {n_submods}" + ) + + code, submod_names, consts = generate_execution_code(split_gm) + + assert consts.count(torch.float16) == 1, ( + f"fp16 singleton must dedup across submods, got consts={consts}" + ) + + # Find the consts index for fp16 and confirm at least two distinct + # inlined submods reference it. This rules out the false-positive where + # one submod references it twice and the other not at all. + fp16_idx = consts.index(torch.float16) + submod_bodies = re.findall( + r"def __vllm_inlined_submods__(\d+)\([^)]*\):\n((?: .*\n)+)", code + ) + assert len(submod_bodies) >= 2 + referencing_submods = [ + name for name, body in submod_bodies if f"__vllm_consts__[{fp16_idx}]" in body + ] + assert len(referencing_submods) >= 2, ( + f"fp16 slot should be referenced from ≥2 inlined submods, " + f"got {referencing_submods}" + ) + + fn = compile_execution_fn(code, {}, submod_names, consts) + assert torch.equal(fn(x), model_fn(x)) + + +def test_non_graphmodule_submod_uses_indexed_callable(x: torch.Tensor) -> None: + """When a child of split_gm is *not* a ``torch.fx.GraphModule`` — as + happens in production once ``PiecewiseBackend`` replaces submods — + codegen emits ``__vllm_submods__[idx](...)`` instead of inlining, and + the runtime callable is bound from ``submod_callables``.""" + + def model_fn(x: torch.Tensor) -> torch.Tensor: + return x.relu().sigmoid() + + split_gm = _trace_and_split(model_fn, (x,), ["aten::relu.default"]) + + # Find a GraphModule child and wrap it in a non-GraphModule nn.Module + # that delegates to the original — this is the structural shape vLLM + # produces after PiecewiseBackend takes over a submod. + child_names = [name for name, _ in split_gm.named_children()] + target_name = child_names[0] + + class NonGMWrapper(torch.nn.Module): + def __init__(self, gm: fx.GraphModule) -> None: + super().__init__() + self.gm = gm + + def forward(self, *args, **kwargs): + return self.gm(*args, **kwargs) + + original = getattr(split_gm, target_name) + del split_gm._modules[target_name] + split_gm.add_module(target_name, NonGMWrapper(original)) + + code, submod_names, consts = generate_execution_code(split_gm) + + assert "__vllm_submods__[" in code, ( + "Non-GraphModule submod should produce an indexed callable reference" + ) + assert target_name in submod_names + + submod_callables = { + name: getattr(split_gm, name) + for name in submod_names + if not isinstance(getattr(split_gm, name), fx.GraphModule) + } + fn = compile_execution_fn(code, submod_callables, submod_names, consts) + assert torch.equal(fn(x), model_fn(x)) + + +# split_graph only passes tuple_return=True to split_module on PyTorch >= 2.12, +# so getitem nodes only appear in the stitching graph from that version onward. +@pytest.mark.skipif( + not is_torch_equal_or_newer("2.12.0.dev"), + reason="split_module tuple_return requires PyTorch >= 2.12", +) +def test_getitem_in_stitching_graph(x: torch.Tensor) -> None: + """``operator.getitem`` on submod tuple returns is the ``call_function`` + special case at codegen.py — emitted as ``name = source[index]`` + rather than a function call.""" + + def model_fn(x: torch.Tensor) -> torch.Tensor: + return x.relu().sigmoid() + + split_gm = _trace_and_split(model_fn, (x,), ["aten::relu.default"]) + code, _, _ = generate_execution_code(split_gm) + + # split_module wraps each submod return in a tuple, so the stitching + # graph unpacks via getitem. The codegen must emit it as indexing. + assert re.search(r"\b\w+ = \w+\[\d+\]\n", code), ( + "Stitching graph should emit `name = source[N]` for getitem nodes" + ) + + +def test_del_emitted_for_intermediate_values(x: torch.Tensor) -> None: + """The codegen schedules ``del`` after a value's last use to free + memory early. Multi-submod splits naturally have intermediates whose + last use is not the output node.""" + + def model_fn(x: torch.Tensor) -> torch.Tensor: + return x.relu().sigmoid().tanh() + + split_gm = _trace_and_split( + model_fn, (x,), ["aten::relu.default", "aten::sigmoid.default"] + ) + code, _, _ = generate_execution_code(split_gm) + + assert re.search(r"^ del \w+", code, re.MULTILINE), ( + "Liveness analysis should emit `del` for intermediates with " + "last-use before the output" + ) + + +def test_with_submod_false_rejects_call_module() -> None: + """``generate_execution_code_with_name(with_submod=False)`` is the + recursive entry for inlining a GraphModule into its parent. It must + refuse a graph that itself contains ``call_module`` nodes — the parent + is responsible for handling those.""" + g = fx.Graph() + x_node = g.placeholder("x") + root = torch.nn.Module() + root.add_module("inner", torch.nn.Identity()) + call = g.call_module("inner", args=(x_node,)) + g.output(call) + gm = fx.GraphModule(root, g) + + with pytest.raises(RuntimeError, match="call_module is not allowed"): + generate_execution_code_with_name(gm, "f", with_submod=False) + + +def test_node_ref_recurses_through_containers() -> None: + """``_node_ref`` is the recursive walker that lifts non-primitives + nested inside list/tuple/dict args. Real aten ops rarely produce such + structures, but the path is needed for DTensor placement lists and + other future cases — unit-test the walker directly.""" + consts: list = [] + const_index: dict[int, int] = {} + cpu = torch.device("cpu") + + # Non-primitive in a list, primitive alongside. + assert _node_ref([cpu, 1], consts, const_index) == "[__vllm_consts__[0], 1]" + assert consts == [cpu] + + # Same object in a tuple — id-based dedup reuses the existing slot. + assert _node_ref((cpu, 2), consts, const_index) == "(__vllm_consts__[0], 2)" + assert consts == [cpu] + + # Single-element tuple uses the trailing-comma form. + assert _node_ref((cpu,), consts, const_index) == "(__vllm_consts__[0],)" + + # Dict value lifts the same way. + ref = _node_ref({"k": cpu}, consts, const_index) + assert ref == "{'k': __vllm_consts__[0]}" + + +def test_legacy_code_without_consts() -> None: + """``compile_execution_fn(consts=None)`` must still load code that has + no ``__vllm_consts__`` reference, so older serialized cache artifacts + keep working.""" + # Pre-consts codegen: no __vllm_consts__ reference, only torch/operator. + legacy_code = ( + "import torch\n" + "def execution_fn(x, *, __vllm_submods__):\n" + " return __vllm_submods__[0](x) + 1\n" + ) + + class AddOne(torch.nn.Module): + def forward(self, x: torch.Tensor) -> torch.Tensor: + return x + 1 + + fn = compile_execution_fn(legacy_code, {"sub": AddOne()}, ["sub"], consts=None) + out = fn(torch.zeros(3)) + assert torch.equal(out, torch.full((3,), 2.0)) diff --git a/asian_ai/performance/vLLM/tests/compile/test_compile_ranges.py b/asian_ai/performance/vLLM/tests/compile/test_compile_ranges.py new file mode 100644 index 0000000000000000000000000000000000000000..a0c3c780ccc61fc9ef0f0066b4623c59f05b1bcd --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/test_compile_ranges.py @@ -0,0 +1,257 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from typing import Any + +import torch +from torch import fx as fx +from torch import nn + +# This import automatically registers `torch.ops.silly.attention` +import tests.compile.silly_attention # noqa +from vllm.compilation.counter import compilation_counter +from vllm.compilation.decorators import support_torch_compile +from vllm.compilation.passes.inductor_pass import ( + InductorPass, + get_pass_context, +) +from vllm.config import ( + VllmConfig, + set_current_vllm_config, +) +from vllm.config.compilation import CompilationConfig, CompilationMode +from vllm.config.scheduler import SchedulerConfig +from vllm.config.utils import Range +from vllm.forward_context import set_forward_context + +BATCH_SIZE = 64 +MLP_SIZE = 128 + + +@support_torch_compile +class TestModel(nn.Module): + def __init__(self, *, vllm_config: VllmConfig, prefix: str = "", **kwargs) -> None: + super().__init__() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = x + x + attn_output = torch.empty_like(x) + torch.ops.silly.attention(x, x, x, attn_output) + x = attn_output + x = x * 3 + return x + + +@torch.inference_mode +def run_model(vllm_config: VllmConfig, model: nn.Module, batch_sizes: list[int]): + with set_forward_context({}, vllm_config=vllm_config): + model(torch.randn(BATCH_SIZE, MLP_SIZE)) + for batch_size in batch_sizes: + model(torch.randn(batch_size, MLP_SIZE)) + + +class PostGradRangeChecker(InductorPass): + def __init__(self, ranges: list[Range]): + self.ranges = ranges + self.num_calls = 0 + + def __call__(self, graph: fx.Graph): + compile_range = get_pass_context().compile_range + assert compile_range in self.ranges, ( + f"Compile range {compile_range} not in {self.ranges}" + ) + self.num_calls += 1 + + def uuid(self) -> str: + state: dict[str, Any] = {} + return InductorPass.hash_dict(state) + + +def test_compile_ranges(use_fresh_inductor_cache): + post_grad_range_checker = PostGradRangeChecker( + [ + Range(start=1, end=8), + Range(start=16, end=16), + Range(start=9, end=32), + Range(start=64, end=64), + Range(start=128, end=128), + Range(start=33, end=8192), + ] + ) + torch.set_default_device("cuda") + vllm_config = VllmConfig( + scheduler_config=SchedulerConfig( + max_num_batched_tokens=8192, + max_model_len=8192, + is_encoder_decoder=False, + ), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + compile_ranges_endpoints=[8, 32], + compile_sizes=[16, 64, 128], + inductor_compile_config={ + "post_grad_custom_post_pass": post_grad_range_checker, + }, + ), + ) + + with set_current_vllm_config(vllm_config): + model = TestModel(vllm_config=vllm_config, prefix="").eval() + # Number of compilations: 3 compile ranges + 3 compile sizes + batch_sizes = [1, 4, 16, 24, 48, 64, 8192] + + with compilation_counter.expect( + num_graphs_seen=1, + num_piecewise_graphs_seen=1, + num_backend_compilations=6, + ): + run_model(vllm_config, model, batch_sizes) + assert post_grad_range_checker.num_calls == 6 + + +def test_compile_config_get_compile_ranges(): + compilation_config = CompilationConfig( + compile_ranges_endpoints=[8, 32], + ) + VllmConfig( + scheduler_config=SchedulerConfig( + max_num_batched_tokens=8192, + max_model_len=8192, + is_encoder_decoder=False, + ), + compilation_config=compilation_config, + ) + assert compilation_config.get_compile_ranges() == [ + Range(start=1, end=8), + Range(start=9, end=32), + Range(start=33, end=8192), + ] + + +class PostGradStaticShapeChecker(InductorPass): + """Asserts that compile_sizes entries produce graphs with fully concrete + (non-symbolic) shapes, and compile_ranges entries have symbolic shapes.""" + + def __init__(self): + self.num_static_calls = 0 + self.num_dynamic_calls = 0 + + def __call__(self, graph: fx.Graph): + from torch.fx.experimental.symbolic_shapes import is_symbolic + + compile_range = get_pass_context().compile_range + is_single = compile_range.is_single_size() + + for node in graph.nodes: + val = node.meta.get("val") + if val is None: + val = node.meta.get("example_value") + if isinstance(val, torch.Tensor): + has_symbolic = any(is_symbolic(d) for d in val.shape) + if is_single: + assert not has_symbolic, ( + f"compile_sizes entry {compile_range}: " + f"node '{node.name}' has symbolic shape " + f"{val.shape}" + ) + else: + # compile_ranges should have at least some + # symbolic shapes (the batch dimension) + if has_symbolic: + self.num_dynamic_calls += 1 + return + + if is_single: + self.num_static_calls += 1 + + def uuid(self) -> str: + state: dict[str, Any] = {} + return InductorPass.hash_dict(state) + + +def test_compile_sizes_produce_static_shapes(use_fresh_inductor_cache): + """Verify that compile_sizes entries are compiled with fully concrete + shapes (no SymInts), while compile_ranges entries retain dynamic shapes.""" + checker = PostGradStaticShapeChecker() + torch.set_default_device("cuda") + vllm_config = VllmConfig( + scheduler_config=SchedulerConfig( + max_num_batched_tokens=8192, + max_model_len=8192, + is_encoder_decoder=False, + ), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + compile_ranges_endpoints=[8], + compile_sizes=[16], + inductor_compile_config={ + "post_grad_custom_post_pass": checker, + }, + ), + ) + + with set_current_vllm_config(vllm_config): + model = TestModel(vllm_config=vllm_config, prefix="").eval() + # 3 compilations: Range(1,8), Range(9,8192), single-size 16 + with compilation_counter.expect( + num_graphs_seen=1, + num_piecewise_graphs_seen=1, + num_backend_compilations=3, + ): + run_model(vllm_config, model, [1, 16, 64]) + + # compile_sizes=16 should produce static shapes + assert checker.num_static_calls == 1, ( + f"Expected 1 static compilation, got {checker.num_static_calls}" + ) + # compile_ranges should produce dynamic shapes + assert checker.num_dynamic_calls == 2, ( + f"Expected 2 dynamic compilations, got {checker.num_dynamic_calls}" + ) + + +def test_inductor_cache_compile_ranges(monkeypatch, use_fresh_inductor_cache): + # To force multiple compilations, we disable the compile cache + monkeypatch.setenv("VLLM_DISABLE_COMPILE_CACHE", "1") + + post_grad_range_checker = PostGradRangeChecker( + ranges=[ + Range(start=1, end=8), + Range(start=9, end=8192), + ] + ) + scheduler_config = SchedulerConfig( + max_num_batched_tokens=8192, + max_model_len=8192, + is_encoder_decoder=False, + ) + torch.set_default_device("cuda") + + def create_vllm_config(): + return VllmConfig( + scheduler_config=scheduler_config, + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + compile_ranges_endpoints=[8], + inductor_compile_config={ + "post_grad_custom_post_pass": post_grad_range_checker, + }, + ), + ) + + vllm_config_1 = create_vllm_config() + with set_current_vllm_config(vllm_config_1): + model1 = TestModel(vllm_config=vllm_config_1, prefix="").eval() + batch_sizes = [1, 16] + run_model(vllm_config_1, model1, batch_sizes) + assert post_grad_range_checker.num_calls == 2 + + post_grad_range_checker.num_calls = 0 + # Create a new vllm config with the new pass context + vllm_config_2 = create_vllm_config() + with set_current_vllm_config(vllm_config_2): + model2 = TestModel(vllm_config=vllm_config_2, prefix="").eval() + batch_sizes = [4, 32] + run_model(vllm_config_2, model2, batch_sizes) + # Check that cache is used, so the number of calls + # should be 0 + assert post_grad_range_checker.num_calls == 0 diff --git a/asian_ai/performance/vLLM/tests/compile/test_config.py b/asian_ai/performance/vLLM/tests/compile/test_config.py new file mode 100644 index 0000000000000000000000000000000000000000..a0b8ef4736648a6dac4d600dcf5549d886942876 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/test_config.py @@ -0,0 +1,790 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import copy +from contextlib import nullcontext +from unittest.mock import MagicMock, patch + +import pytest +import torch +from pydantic import ValidationError + +from vllm.compilation.counter import compilation_counter +from vllm.compilation.passes.utility.fix_functionalization import ( + FixFunctionalizationPass, +) +from vllm.config import ( + CompilationConfig, + CUDAGraphMode, + ParallelConfig, + SchedulerConfig, + VllmConfig, +) +from vllm.config.compilation import CompilationMode, PassConfig +from vllm.engine.arg_utils import EngineArgs +from vllm.platforms import current_platform +from vllm.utils.torch_utils import ( + _is_torch_equal_or_newer, + is_torch_equal, +) +from vllm.v1.cudagraph_dispatcher import CudagraphDispatcher + +# This import automatically registers `torch.ops.silly.attention` +from . import silly_attention # noqa: F401 + +DEVICE_TYPE = current_platform.device_type + + +def test_version(): + # Test the version comparison logic using the private function + assert _is_torch_equal_or_newer("2.8.0.dev20250624+cu128", "2.8.0.dev") + assert _is_torch_equal_or_newer("2.8.0a0+gitc82a174", "2.8.0.dev") + assert _is_torch_equal_or_newer("2.8.0", "2.8.0.dev") + assert _is_torch_equal_or_newer("2.8.1", "2.8.0.dev") + assert not _is_torch_equal_or_newer("2.7.1", "2.8.0.dev") + + +def test_get_raw_stream_patch(): + """Test that get_raw_stream patch is applied only for torch 2.9.0 or 2.9.1.""" + import builtins + + # Check if get_raw_stream exists in builtins + has_patch = hasattr(builtins, "get_raw_stream") + + # Import torch to get actual version + + is_torch_2_9 = is_torch_equal("2.9.0") or is_torch_equal("2.9.1") + + if is_torch_2_9: + # For torch 2.9.x, the patch should be applied + assert has_patch, "get_raw_stream should be patched for torch 2.9.x" + # Verify it's callable (it should be the _cuda_getCurrentRawStream function) + get_raw_stream = builtins.get_raw_stream # type: ignore[attr-defined] + assert callable(get_raw_stream) + # Verify it's the correct function from torch._C + from torch._C import _cuda_getCurrentRawStream + + assert get_raw_stream is _cuda_getCurrentRawStream + + +def test_copy_pass(): + vllm_config = VllmConfig() + inductor_pass = FixFunctionalizationPass(vllm_config) + copied_inductor_pass = copy.deepcopy(inductor_pass) + assert ( + copied_inductor_pass.compilation_config.use_inductor_graph_partition + == vllm_config.compilation_config.use_inductor_graph_partition + ) + assert ( + copied_inductor_pass.compilation_config.splitting_ops + == vllm_config.compilation_config.splitting_ops + ) + + +def test_custom_op(): + # proper syntax + _ = CompilationConfig(custom_ops=["+quant_fp8", "-silu_and_mul"]) + + with pytest.raises(ValueError, match="Invalid syntax '"): + _ = CompilationConfig(custom_ops=["quant_fp8"]) + + +# forked needed to workaround https://github.com/vllm-project/vllm/issues/21073 +@pytest.mark.forked +# NB: We don't test VLLM_DISABLE_COMPILE_CACHE=0 because that depends +# on the state of the cache directory on the current machine, which +# may be influenced by other tests. +@pytest.mark.parametrize("val", ["1"]) +def test_VLLM_DISABLE_COMPILE_CACHE(vllm_runner, monkeypatch, val): + # Disable multiprocessing so that the counter is in the same process + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + monkeypatch.setenv("VLLM_DISABLE_COMPILE_CACHE", val) + + compilation_config = { + "cudagraph_mode": CUDAGraphMode.NONE, # speed things up a bit + } + with ( + compilation_counter.expect( + num_cache_entries_updated=0, num_compiled_artifacts_saved=0 + ), + # loading the model causes compilation (if enabled) to happen + vllm_runner( + "facebook/opt-125m", + compilation_config=compilation_config, + gpu_memory_utilization=0.4, + ) as _, + ): + pass + + +# forked needed to workaround https://github.com/vllm-project/vllm/issues/21073 +@pytest.mark.forked +@pytest.mark.parametrize( + "cudagraph_mode,num_cudagraph_captured", + [ + (CUDAGraphMode.NONE, 0), + (CUDAGraphMode.FULL_DECODE_ONLY, 1), + (CUDAGraphMode.PIECEWISE, 13), + (CUDAGraphMode.FULL_AND_PIECEWISE, 14), + ], +) +def test_use_cudagraphs( + vllm_runner, monkeypatch, cudagraph_mode, num_cudagraph_captured +): + # Disable multiprocessing so that the counter is in the same process + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + + compilation_config = { + "cudagraph_capture_sizes": [100], + "cudagraph_mode": cudagraph_mode, + } + num_gpu_runner_capture_triggers = 1 if cudagraph_mode != CUDAGraphMode.NONE else 0 + with ( + compilation_counter.expect( + num_graphs_seen=1, + num_gpu_runner_capture_triggers=num_gpu_runner_capture_triggers, + num_cudagraph_captured=num_cudagraph_captured, + ), + # loading the model causes compilation (if enabled) to happen + vllm_runner( + "facebook/opt-125m", + compilation_config=compilation_config, + gpu_memory_utilization=0.4, + ) as _, + ): + pass + + +# forked needed to workaround https://github.com/vllm-project/vllm/issues/21073 +@pytest.mark.forked +def test_stock_torch_compile(vllm_runner, monkeypatch): + # Disable multiprocessing so that the counter is in the same process + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + + with ( + compilation_counter.expect(stock_torch_compile_count=1), + # loading the model causes compilation (if enabled) to happen + vllm_runner( + "facebook/opt-125m", + compilation_config={"mode": CompilationMode.STOCK_TORCH_COMPILE}, + gpu_memory_utilization=0.4, + ) as _, + ): + pass + + +# forked needed to workaround https://github.com/vllm-project/vllm/issues/21073 +@pytest.mark.forked +def test_no_compilation(vllm_runner, monkeypatch): + # Disable multiprocessing so that the counter is in the same process + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + with ( + compilation_counter.expect(num_graphs_seen=0, stock_torch_compile_count=0), + # loading the model causes compilation (if enabled) to happen + vllm_runner( + "facebook/opt-125m", + compilation_config={"mode": CompilationMode.NONE}, + gpu_memory_utilization=0.4, + ) as _, + ): + pass + + +# forked needed to workaround https://github.com/vllm-project/vllm/issues/21073 +@pytest.mark.forked +def test_enforce_eager(vllm_runner, monkeypatch): + # Disable multiprocessing so that the counter is in the same process + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + + with ( + compilation_counter.expect(num_graphs_seen=0, stock_torch_compile_count=0), + # loading the model causes compilation (if enabled) to happen + vllm_runner( + "facebook/opt-125m", enforce_eager=True, gpu_memory_utilization=0.4 + ) as _, + ): + pass + + +@pytest.mark.forked +def test_torch_compile_disable(vllm_runner, monkeypatch): + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + monkeypatch.setenv("TORCH_COMPILE_DISABLE", "1") + monkeypatch.setenv("VLLM_DISABLE_COMPILE_CACHE", "1") + + with ( + compilation_counter.expect(num_graphs_seen=0, stock_torch_compile_count=0), + vllm_runner( + "facebook/opt-125m", + gpu_memory_utilization=0.4, + ) as _, + ): + pass + + +def test_splitting_ops_dynamic(): + # Default config + config = VllmConfig() + # Default V1 config leaves cudagraph mode unset; splitting ops are only + # populated when the engine decides to use piecewise compilation. + assert config.compilation_config.cudagraph_mode == CUDAGraphMode.FULL_AND_PIECEWISE + assert config.compilation_config.splitting_ops_contain_attention() + + # When use_inductor_graph_partition=True + config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + use_inductor_graph_partition=True, + splitting_ops=["vllm::unified_attention_with_output"], + ) + ) + # with inductor partition we use splitting_ops directly for + # partition rules + assert config.compilation_config.splitting_ops == [ + "vllm::unified_attention_with_output" + ] + + # When attn_fusion pass enabled. + config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + pass_config=PassConfig(fuse_attn_quant=True, eliminate_noops=True), + custom_ops=["+quant_fp8"], + cudagraph_mode=CUDAGraphMode.PIECEWISE, + ) + ) + assert config.compilation_config.splitting_ops == [] + # cudagraph mode also fall back to FULL + assert config.compilation_config.cudagraph_mode == CUDAGraphMode.FULL + + # splitting_ops can not contain attention ops when attn_fusion + # pass enabled. + with pytest.raises(ValidationError): + config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + pass_config=PassConfig(fuse_attn_quant=True, eliminate_noops=True), + custom_ops=["+quant_fp8"], + cudagraph_mode=CUDAGraphMode.PIECEWISE, + # work around for accessing all attntion ops + splitting_ops=CompilationConfig()._attention_ops, + ) + ) + + # When both use_inductor_graph_partition and attn_fusion pass enabled. + config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + use_inductor_graph_partition=True, + pass_config=PassConfig(fuse_attn_quant=True, eliminate_noops=True), + custom_ops=["+quant_fp8"], + cudagraph_mode=CUDAGraphMode.PIECEWISE, + ) + ) + # With inductor graph partition, attn_fusion and splitting_ops + # work together. Default splitting_ops include attention ops. + assert config.compilation_config.splitting_ops_contain_attention() + # fuse_attn_quant is directly supported under + # use_inductor_graph_partition=True, and cudagraph_mode + # is unchanged. + assert config.compilation_config.cudagraph_mode == CUDAGraphMode.PIECEWISE + + +def test_moe_splitting_ops_deepep_ht_inductor_partition(): + # Inductor partition case: user-provided splitting_ops should be + # preserved and MoE ops should be appended for DeepEP HT with dp>1. + config = VllmConfig( + parallel_config=ParallelConfig( + all2all_backend="deepep_high_throughput", + data_parallel_size=8, + ), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + use_inductor_graph_partition=True, + splitting_ops=[ + "vllm::unified_attention_with_output", + "vllm::moe_forward", + "vllm::moe_forward_shared", + ], + ), + ) + splitting_ops = config.compilation_config.splitting_ops + assert splitting_ops == [ + "vllm::unified_attention_with_output", + "vllm::moe_forward", + "vllm::moe_forward_shared", + ] + + +def test_should_split(): + import torch + + from vllm.compilation.partition_rules import should_split + + graph = torch.fx.Graph() + node = torch.fx.Node( + graph=graph, + name="dummy_node", + op="call_function", + target=torch.ops.aten.add.default, + args=(), + kwargs={}, + ) + + # supports OpOverloadPacket + splitting_ops = ["aten::add"] + assert should_split(node, splitting_ops) + + # supports OpOverload + splitting_ops = ["aten::add.default"] + assert should_split(node, splitting_ops) + + # supports OpOverload + splitting_ops = ["aten::add.Tensor"] + assert not should_split(node, splitting_ops) + + q, k, v, out = [torch.randn(1)] * 4 + + # supports custom ops as OpOverloadPacket + node = torch.fx.Node( + graph=graph, + name="dummy_node", + op="call_function", + target=torch.ops.silly.attention, + args=(q, k, v, out), + kwargs={}, + ) + + splitting_ops = ["silly::attention"] + assert should_split(node, splitting_ops) + + # supports custom ops as OpOverload + node = torch.fx.Node( + graph=graph, + name="dummy_node", + op="call_function", + target=torch.ops.silly.attention.default, + args=(q, k, v, out), + kwargs={}, + ) + + splitting_ops = ["silly::attention"] + assert should_split(node, splitting_ops) + + splitting_ops = ["silly::attention.default"] + assert should_split(node, splitting_ops) + + +@pytest.mark.skipif( + not current_platform.support_static_graph_mode(), + reason="Skip if not cudagraph mode supported", +) +@pytest.mark.parametrize( + ( + "cudagraph_capture_sizes", + "max_cudagraph_capture_size", + "tp_size", + "enable_sp", + "max_num_batched_tokens", + "cudagraph_mode", + "expected_max_size", + ), + [ + (None, None, 1, False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 256), + ([1, 2, 4], 4, 1, False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 4), + ( + [1, 2, 4], + 8, + 1, + False, + 2048, + CUDAGraphMode.FULL_AND_PIECEWISE, + ValidationError, + ), + ([1, 256], None, 1, False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 256), + ([], None, 1, False, 2048, CUDAGraphMode.NONE, 0), + (None, 0, 1, False, 2048, CUDAGraphMode.NONE, 0), + # truncated to nearest multiple of 8 or 16 + (None, 257, 1, False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 256), + # max_num_batched_tokens <= max_cudagraph_capture_size should always be + # captured even if not landing on a 16-stride step + (None, 2048, 1, False, 257, CUDAGraphMode.FULL_AND_PIECEWISE, 257), + # max from list + ([1, 2, 4, 15], None, 1, False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 15), + # SP forces full-graph compilation, sizes are filtered by TP + ([1, 2, 4, 15], None, 2, True, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, 4), + # limited by the max_tokens + ([1, 2, 4, 15], None, 1, False, 8, CUDAGraphMode.FULL_AND_PIECEWISE, 4), + # the list should contain at least 1 element when use cudagraph + ([], None, 1, False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, ValidationError), + # the max capturing size should be >= 1 when use cudagraph + (None, 0, 1, False, 2048, CUDAGraphMode.FULL_AND_PIECEWISE, ValidationError), + ], +) +def test_cudagraph_sizes_post_init( + cudagraph_capture_sizes, + max_cudagraph_capture_size, + tp_size, + enable_sp, + max_num_batched_tokens, + cudagraph_mode, + expected_max_size, +): + ctx = nullcontext() + if expected_max_size == ValidationError: + ctx = pytest.raises(expected_max_size) + + with ( + ctx, + patch.object(current_platform, "device_count", return_value=tp_size), + ): + kwargs = {} + if cudagraph_capture_sizes is not None: + kwargs["cudagraph_capture_sizes"] = cudagraph_capture_sizes + if max_cudagraph_capture_size is not None: + kwargs["max_cudagraph_capture_size"] = max_cudagraph_capture_size + compilation_config = CompilationConfig( + pass_config=PassConfig( + enable_sp=enable_sp, + fuse_norm_quant=True, + fuse_act_quant=True, + eliminate_noops=True, + sp_min_token_num=512 if enable_sp else None, + ), + cudagraph_mode=cudagraph_mode, + **kwargs, + ) + engine_args = EngineArgs( + model="facebook/opt-125m", + tensor_parallel_size=tp_size, + max_num_seqs=min(max_num_batched_tokens, 128), + max_num_batched_tokens=max_num_batched_tokens, + compilation_config=compilation_config, + ) + vllm_config = engine_args.create_engine_config() + + assert ( + vllm_config.compilation_config.max_cudagraph_capture_size + == expected_max_size + ) + + +@pytest.mark.skipif( + not current_platform.support_static_graph_mode(), + reason="Skip if not cudagraph mode supported", +) +@pytest.mark.parametrize( + ( + "cudagraph_mode", + "use_inductor_graph_partition", + "expected_enable_sp", + "expected_cudagraph_mode", + "expected_piecewise_compile", + "expected_capture_sizes", + "expected_max_size", + ), + [ + (CUDAGraphMode.PIECEWISE, False, True, CUDAGraphMode.FULL, False, [2, 4], 4), + ( + CUDAGraphMode.FULL_DECODE_ONLY, + False, + True, + CUDAGraphMode.FULL_DECODE_ONLY, + False, + [2, 4], + 4, + ), + ( + CUDAGraphMode.FULL_AND_PIECEWISE, + False, + True, + CUDAGraphMode.FULL, + False, + [2, 4], + 4, + ), + ( + CUDAGraphMode.FULL_AND_PIECEWISE, + True, + True, + CUDAGraphMode.FULL_AND_PIECEWISE, + True, + [2, 4], + 4, + ), + ], +) +def test_sequence_parallelism_requires_full_graph_compilation( + cudagraph_mode: CUDAGraphMode, + use_inductor_graph_partition: bool, + expected_enable_sp: bool, + expected_cudagraph_mode: CUDAGraphMode, + expected_piecewise_compile: bool, + expected_capture_sizes: list[int], + expected_max_size: int, +): + with patch.object(current_platform, "device_count", return_value=2): + vllm_config = VllmConfig( + parallel_config=ParallelConfig(tensor_parallel_size=2), + scheduler_config=SchedulerConfig( + max_num_seqs=128, + max_num_batched_tokens=2048, + max_model_len=2048, + is_encoder_decoder=False, + ), + ) + vllm_config.model_config = MagicMock( + dtype=torch.float16, + enforce_eager=False, + is_moe=False, + disable_cascade_attn=False, + get_hidden_size=MagicMock(return_value=4096), + ) + vllm_config.compilation_config = CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + cudagraph_capture_sizes=[1, 2, 4, 15], + max_cudagraph_capture_size=None, + compile_sizes=["cudagraph_capture_sizes"], + use_inductor_graph_partition=use_inductor_graph_partition, + pass_config=PassConfig( + enable_sp=True, + fuse_gemm_comms=True, + fuse_norm_quant=True, + fuse_act_quant=True, + eliminate_noops=True, + sp_min_token_num=512, + ), + cudagraph_mode=cudagraph_mode, + ) + vllm_config.compilation_config.set_splitting_ops_for_v1( + all2all_backend=vllm_config.parallel_config.all2all_backend, + data_parallel_size=1, + ) + vllm_config._set_compile_ranges() + vllm_config._set_cudagraph_sizes() + + assert ( + vllm_config.compilation_config.use_inductor_graph_partition + == use_inductor_graph_partition + ) + assert ( + bool(vllm_config.compilation_config.splitting_ops) == expected_piecewise_compile + ) + assert vllm_config.compilation_config.pass_config.enable_sp == expected_enable_sp + assert ( + vllm_config.compilation_config.pass_config.fuse_gemm_comms == expected_enable_sp + ) + assert vllm_config.compilation_config.cudagraph_mode == expected_cudagraph_mode + assert ( + vllm_config.compilation_config.cudagraph_capture_sizes == expected_capture_sizes + ) + assert ( + vllm_config.compilation_config.max_cudagraph_capture_size == expected_max_size + ) + assert ( + 511 in vllm_config.compilation_config.compile_ranges_endpoints + ) == expected_enable_sp + + +def test_cached_compilation_config(default_vllm_config): + import torch + from torch._inductor.utils import run_and_get_code + + from vllm.config import get_cached_compilation_config, set_current_vllm_config + from vllm.model_executor.layers.quantization.input_quant_fp8 import QuantFP8 + from vllm.model_executor.layers.quantization.utils.quant_utils import GroupShape + + dtype = torch.bfloat16 + device = torch.device(f"{DEVICE_TYPE}:0") + batch_size, num_qo_heads, head_size = 8, 16, 128 + + # access and cache default compilation config + # default compilation config does not contain +quant_fp8 custom op. If this is + # used, the generated code would use inductor-generated triton kernel instead + # of the custom op `torch.ops._C.static_scaled_fp8_quant`. + get_cached_compilation_config() + + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + custom_ops=["+quant_fp8"], + ) + ) + + # set_current_vllm_config should clear cached compilation config and + # use the new compilation_config in vllm_config + with set_current_vllm_config(vllm_config): + query_quant = QuantFP8(static=True, group_shape=GroupShape.PER_TENSOR) + query_quant = torch.compile(query_quant) + + _q_scale = torch.tensor(1.0, dtype=torch.float32, device=DEVICE_TYPE) + query = torch.randn( + batch_size, num_qo_heads * head_size, dtype=dtype, device=device + ) + + _, code = run_and_get_code(query_quant, query, _q_scale) + + code = " ".join(code) + assert "torch.ops._C.static_scaled_fp8_quant.default(" in code + + +def _create_vllm_config_for_validation( + compilation_config: CompilationConfig, +) -> MagicMock: + """Helper to create a mock VllmConfig for padding validation testing.""" + mock_config = MagicMock(spec=VllmConfig) + mock_config.compilation_config = compilation_config + mock_config.scheduler_config = SchedulerConfig.default_factory(max_num_seqs=8) + mock_config.parallel_config = ParallelConfig() + mock_config.speculative_config = None + mock_config.lora_config = None + return mock_config + + +def test_compile_sizes_padding_validation(): + """Test that compile_sizes with values that would be padded raises an error.""" + # cudagraph_capture_sizes=[1, 2, 4, 8] means: + # - size 1 -> padded to 1 + # - size 2 -> padded to 2 + # - size 3 -> padded to 4 + # - size 4 -> padded to 4 + # - size 5 -> padded to 8 + # etc. + # So compile_sizes=[3] should fail because 3 would be padded to 4 + + with pytest.raises(ValueError, match="would be padded to"): + config = CompilationConfig( + cudagraph_capture_sizes=[1, 2, 4, 8], + max_cudagraph_capture_size=8, + compile_sizes=[3], + cudagraph_mode=CUDAGraphMode.FULL, + ) + config.post_init_cudagraph_sizes() + dispatcher = CudagraphDispatcher(_create_vllm_config_for_validation(config)) + dispatcher.initialize_cudagraph_keys(CUDAGraphMode.FULL) + + with pytest.raises(ValueError, match="would be padded to"): + config = CompilationConfig( + cudagraph_capture_sizes=[1, 2, 4, 8], + max_cudagraph_capture_size=8, + compile_sizes=[5], + cudagraph_mode=CUDAGraphMode.FULL, + ) + config.post_init_cudagraph_sizes() + dispatcher = CudagraphDispatcher(_create_vllm_config_for_validation(config)) + dispatcher.initialize_cudagraph_keys(CUDAGraphMode.FULL) + + config = CompilationConfig( + cudagraph_capture_sizes=[1, 2, 4, 8], + max_cudagraph_capture_size=8, + compile_sizes=[1, 2, 4, 8], + cudagraph_mode=CUDAGraphMode.FULL, + ) + config.post_init_cudagraph_sizes() + assert sorted(config.compile_sizes) == [1, 2, 4, 8] + dispatcher = CudagraphDispatcher(_create_vllm_config_for_validation(config)) + dispatcher.initialize_cudagraph_keys(CUDAGraphMode.FULL) # Should not raise + + config = CompilationConfig( + cudagraph_capture_sizes=[1, 2, 4, 8], + max_cudagraph_capture_size=8, + compile_sizes=["cudagraph_capture_sizes"], + cudagraph_mode=CUDAGraphMode.FULL, + ) + config.post_init_cudagraph_sizes() + assert sorted(config.compile_sizes) == [1, 2, 4, 8] + + # When cudagraphs are disabled (max_cudagraph_capture_size=0), + # padding validation should be skipped + config = CompilationConfig( + cudagraph_capture_sizes=[], + max_cudagraph_capture_size=0, + compile_sizes=[3, 5, 7], # would be invalid with cudagraphs + ) + config.post_init_cudagraph_sizes() + assert sorted(config.compile_sizes) == [3, 5, 7] + + # When cudagraph_mode is NONE but capture_sizes is non-empty, + # padding validation should still be skipped + config = CompilationConfig( + cudagraph_capture_sizes=[1, 2, 4, 8], + max_cudagraph_capture_size=8, + cudagraph_mode=CUDAGraphMode.NONE, + compile_sizes=[3, 5, 7], # would be invalid if cudagraphs were enabled + ) + config.post_init_cudagraph_sizes() + assert sorted(config.compile_sizes) == [3, 5, 7] + dispatcher = CudagraphDispatcher(_create_vllm_config_for_validation(config)) + dispatcher.initialize_cudagraph_keys(CUDAGraphMode.NONE) # Should not raise + + +def test_inductor_asserts_default_disabled(monkeypatch): + """Test that inductor runtime asserts are disabled by default + (INFO logging level) on torch < 2.12.""" + monkeypatch.setenv("VLLM_LOGGING_LEVEL", "INFO") + + import importlib + + import vllm.envs + + importlib.reload(vllm.envs) + + config = CompilationConfig() + if not _is_torch_equal_or_newer(torch.__version__, "2.12.0.dev"): + assert config.inductor_compile_config.get("size_asserts") is False + assert config.inductor_compile_config.get("alignment_asserts") is False + assert config.inductor_compile_config.get("scalar_asserts") is False + + +def test_inductor_asserts_enabled_in_debug(monkeypatch): + """Test that VLLM_LOGGING_LEVEL=DEBUG enables inductor runtime asserts + on torch < 2.12.""" + monkeypatch.setenv("VLLM_LOGGING_LEVEL", "DEBUG") + + import importlib + + import vllm.envs + + importlib.reload(vllm.envs) + + config = CompilationConfig() + if not _is_torch_equal_or_newer(torch.__version__, "2.12.0.dev"): + assert config.inductor_compile_config.get("size_asserts") is True + assert config.inductor_compile_config.get("alignment_asserts") is True + assert config.inductor_compile_config.get("scalar_asserts") is True + + +def test_get_inductor_factors_includes_configs(): + """Changing inductor or functorch config must change the cache key factors.""" + from torch._functorch import config as functorch_config + from torch._inductor import config as inductor_config + + from vllm.compilation.compiler_interface import get_inductor_factors + + baseline = get_inductor_factors() + + with inductor_config.patch("max_autotune", not inductor_config.max_autotune): + patched = get_inductor_factors() + assert baseline != patched, "inductor config change was not reflected" + + with functorch_config.patch("donated_buffer", not functorch_config.donated_buffer): + patched = get_inductor_factors() + assert baseline != patched, "functorch config change was not reflected" + + +def test_inductor_asserts_user_override(monkeypatch): + """Test that explicit inductor_compile_config overrides the + debug-logging default.""" + monkeypatch.setenv("VLLM_LOGGING_LEVEL", "INFO") + + import importlib + + import vllm.envs + + importlib.reload(vllm.envs) + + config = CompilationConfig( + inductor_compile_config={"size_asserts": True}, + ) + assert config.inductor_compile_config.get("size_asserts") is True + if not _is_torch_equal_or_newer(torch.__version__, "2.12.0.dev"): + assert config.inductor_compile_config.get("alignment_asserts") is False diff --git a/asian_ai/performance/vLLM/tests/compile/test_decorator.py b/asian_ai/performance/vLLM/tests/compile/test_decorator.py new file mode 100644 index 0000000000000000000000000000000000000000..2cf78df0d24f9dd8a083f2ae3f2bf712c680076b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/test_decorator.py @@ -0,0 +1,286 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch +from torch import nn + +from vllm.compilation.counter import compilation_counter +from vllm.compilation.decorators import ignore_torch_compile, support_torch_compile +from vllm.config import ( + CacheConfig, + CompilationConfig, + CompilationMode, + CUDAGraphMode, + VllmConfig, + set_current_vllm_config, +) +from vllm.forward_context import BatchDescriptor, set_forward_context +from vllm.utils.torch_utils import is_torch_equal_or_newer + +# This import automatically registers `torch.ops.silly.attention` +from . import silly_attention # noqa: F401 + +BATCH_SIZE = 32 +MLP_SIZE = 128 + + +@torch.inference_mode +def run_model( + vllm_config: VllmConfig, model: nn.Module, cudagraph_runtime_mode: CUDAGraphMode +): + with set_forward_context({}, vllm_config=vllm_config): + # warmup for the model with cudagraph_mode NONE + model(torch.randn(BATCH_SIZE, MLP_SIZE).cuda()) + + # simulate cudagraphs capturing + with set_forward_context( + {}, + vllm_config=vllm_config, + cudagraph_runtime_mode=cudagraph_runtime_mode, + batch_descriptor=BatchDescriptor( + num_tokens=2, + ), + ): + model(torch.randn(2, MLP_SIZE).cuda()) + with set_forward_context( + {}, + vllm_config=vllm_config, + cudagraph_runtime_mode=cudagraph_runtime_mode, + batch_descriptor=BatchDescriptor( + num_tokens=1, + ), + ): + model(torch.randn(1, MLP_SIZE).cuda()) + + # simulate cudagraphs replay + with set_forward_context( + {}, + vllm_config=vllm_config, + cudagraph_runtime_mode=cudagraph_runtime_mode, + batch_descriptor=BatchDescriptor( + num_tokens=2, + ), + ): + output = model(torch.randn(2, MLP_SIZE).cuda()) + + output = output.cpu() + return output.cpu() + + +@pytest.mark.parametrize("use_inductor_graph_partition", [True, False]) +def test_ignore_torch_compile_decorator(use_inductor_graph_partition, monkeypatch): + # disable compile cache so that we can count the number of compilations + # appropriately + monkeypatch.setenv("VLLM_DISABLE_COMPILE_CACHE", "1") + + if use_inductor_graph_partition and not is_torch_equal_or_newer("2.9.0.dev"): + pytest.skip("inductor graph partition is only available in PyTorch 2.9+") + + # piecewise + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + splitting_ops=["silly::attention"], + cudagraph_capture_sizes=[1, 2], + use_inductor_graph_partition=use_inductor_graph_partition, + ) + ) + cudagraph_runtime_mode = CUDAGraphMode.PIECEWISE + + expected_num_graphs_seen = 1 + expected_num_cudagraph_captured = ( + 4 # num_cudagraph_sizes * num cudagraphs to capture + ) + if use_inductor_graph_partition: + expected_num_piecewise_graphs_seen = 1 + expected_num_piecewise_capturable_graphs_seen = 1 + expected_num_backend_compilations = 1 + else: + expected_num_piecewise_graphs_seen = 3 + expected_num_piecewise_capturable_graphs_seen = 2 + expected_num_backend_compilations = 2 + + @support_torch_compile + class A(nn.Module): + def __init__( + self, *, vllm_config: VllmConfig, prefix: str = "", **kwargs + ) -> None: + super().__init__() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = x + x + attn_output = torch.empty_like(x) + torch.ops.silly.attention(x, x, x, attn_output) + x = attn_output + x = x * 3 + return x + + @ignore_torch_compile + class B(A): ... + + @support_torch_compile + class C(B): ... + + with set_current_vllm_config(vllm_config): + mod_A = A(vllm_config=vllm_config, prefix="").eval().cuda() + + # A has support_torch_compile + with compilation_counter.expect( + num_graphs_seen=expected_num_graphs_seen, + num_piecewise_graphs_seen=expected_num_piecewise_graphs_seen, + num_piecewise_capturable_graphs_seen=expected_num_piecewise_capturable_graphs_seen, + num_backend_compilations=expected_num_backend_compilations, + num_cudagraph_captured=expected_num_cudagraph_captured, + ): + run_model(vllm_config, mod_A, cudagraph_runtime_mode) + + with set_current_vllm_config(vllm_config): + mod_B = B(vllm_config=vllm_config, prefix="").eval().cuda() + + # B's ignore_torch_compile should override A's support_torch_compile + with compilation_counter.expect( + num_graphs_seen=0, + num_piecewise_graphs_seen=0, + num_piecewise_capturable_graphs_seen=0, + num_backend_compilations=0, + num_cudagraph_captured=0, + ): + run_model(vllm_config, mod_B, cudagraph_runtime_mode) + + with set_current_vllm_config(vllm_config): + mod_C = C(vllm_config=vllm_config, prefix="").eval().cuda() + + # C's support_torch_compile should override B's ignore_torch_compile + with compilation_counter.expect( + num_graphs_seen=expected_num_graphs_seen, + num_piecewise_graphs_seen=expected_num_piecewise_graphs_seen, + num_piecewise_capturable_graphs_seen=expected_num_piecewise_capturable_graphs_seen, + num_backend_compilations=expected_num_backend_compilations, + num_cudagraph_captured=expected_num_cudagraph_captured, + ): + run_model(vllm_config, mod_C, cudagraph_runtime_mode) + + +# Only enable torch.compile if +# vllm_config.cache_config.kv_sharing_fast_prefill=True +@support_torch_compile( + enable_if=lambda vllm_config: vllm_config.cache_config.kv_sharing_fast_prefill +) +class B(nn.Module): + def __init__(self, *, vllm_config: VllmConfig, prefix: str = "", **kwargs) -> None: + super().__init__() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = x + x + attn_output = torch.empty_like(x) + torch.ops.silly.attention(x, x, x, attn_output) + x = attn_output + x = x + x + return x + + +# Only enable torch.compile if +# vllm_config.cache_config.kv_sharing_fast_prefill=False +@support_torch_compile( + enable_if=lambda vllm_config: not vllm_config.cache_config.kv_sharing_fast_prefill +) +class A(nn.Module): + def __init__(self, *, vllm_config: VllmConfig, prefix: str = "", **kwargs) -> None: + super().__init__() + self.mod1 = B(vllm_config=vllm_config, prefix=prefix, **kwargs) + self.mod2 = B(vllm_config=vllm_config, prefix=prefix, **kwargs) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = self.mod1(x) + attn_output = torch.empty_like(x) + torch.ops.silly.attention(x, x, x, attn_output) + x = attn_output + x = self.mod2(x) + return x + + +@pytest.mark.parametrize("use_inductor_graph_partition", [True, False]) +def test_conditional_compile_enable_if(use_inductor_graph_partition, monkeypatch): + # disable compile cache so that we can count the number of compilations + # appropriately + monkeypatch.setenv("VLLM_DISABLE_COMPILE_CACHE", "1") + + if use_inductor_graph_partition and not is_torch_equal_or_newer("2.9.0.dev"): + pytest.skip("inductor graph partition is only available in PyTorch 2.9+") + + vllm_config = VllmConfig( + cache_config=CacheConfig( + kv_sharing_fast_prefill=True, + ), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + splitting_ops=["silly::attention"], + cudagraph_capture_sizes=[1, 2], + use_inductor_graph_partition=use_inductor_graph_partition, + ), + ) + cudagraph_runtime_mode = CUDAGraphMode.PIECEWISE + + with set_current_vllm_config(vllm_config): + mod_A = A(vllm_config=vllm_config, prefix="").eval().cuda() + + if use_inductor_graph_partition: + expected_num_piecewise_graphs_seen = 2 + expected_num_piecewise_capturable_graphs_seen = 2 + expected_num_backend_compilations = 2 + else: + expected_num_piecewise_graphs_seen = 6 + expected_num_piecewise_capturable_graphs_seen = 4 + expected_num_backend_compilations = 4 + + # A has support_torch_compile but enable_if fn returns False + # enable_if will be True for B, so we expect mod1 and mod2 + # to be compiled + with compilation_counter.expect( + num_graphs_seen=2, + num_piecewise_graphs_seen=expected_num_piecewise_graphs_seen, + # 3 piecewise graphs per instance of B() + num_piecewise_capturable_graphs_seen=expected_num_piecewise_capturable_graphs_seen, + num_backend_compilations=expected_num_backend_compilations, + num_cudagraph_captured=8, + # num_cudagraph_sizes * num cudagraphable graphs to capture + ): + run_model(vllm_config, mod_A, cudagraph_runtime_mode) + + # Set kv_sharing_fast_prefill=False + # which will cause A to be compiled and B to not be compiled + vllm_config = VllmConfig( + cache_config=CacheConfig( + kv_sharing_fast_prefill=False, + ), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + splitting_ops=["silly::attention"], + cudagraph_capture_sizes=[1, 2], + use_inductor_graph_partition=use_inductor_graph_partition, + ), + ) + + with set_current_vllm_config(vllm_config): + mod_A = A(vllm_config=vllm_config, prefix="").eval().cuda() + + if use_inductor_graph_partition: + expected_num_piecewise_graphs_seen = 1 + expected_num_piecewise_capturable_graphs_seen = 1 + expected_num_backend_compilations = 1 + else: + # 3 attn ops and 4 non-attn ops + expected_num_piecewise_graphs_seen = 7 + expected_num_piecewise_capturable_graphs_seen = 4 + expected_num_backend_compilations = 4 + + with compilation_counter.expect( + num_graphs_seen=1, + num_piecewise_graphs_seen=expected_num_piecewise_graphs_seen, + # 3 attn ops and 4 non-attn ops + num_piecewise_capturable_graphs_seen=expected_num_piecewise_capturable_graphs_seen, + num_backend_compilations=expected_num_backend_compilations, + num_cudagraph_captured=8, + # num_cudagraph_sizes * num cudagraphable graphs to capture + ): + run_model(vllm_config, mod_A, cudagraph_runtime_mode) diff --git a/asian_ai/performance/vLLM/tests/compile/test_dynamic_shapes_compilation.py b/asian_ai/performance/vLLM/tests/compile/test_dynamic_shapes_compilation.py new file mode 100644 index 0000000000000000000000000000000000000000..f35b4396f3e2e77161266758c910792d7b17eb0e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/test_dynamic_shapes_compilation.py @@ -0,0 +1,282 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import gc +import tempfile +from contextlib import contextmanager + +import pytest +import torch + +from tests.models.utils import check_logprobs_close +from vllm import LLM, SamplingParams +from vllm.compilation.decorators import support_torch_compile +from vllm.config import CompilationConfig, VllmConfig, set_current_vllm_config +from vllm.config.compilation import ( + CompilationMode, + DynamicShapesConfig, + DynamicShapesType, +) +from vllm.forward_context import set_forward_context +from vllm.utils.torch_utils import is_torch_equal_or_newer + + +def get_test_models(): + """Get list of models to test based on PyTorch version""" + models = [ + "gpt2", + "Qwen/Qwen2-7B-Instruct", + "meta-llama/Llama-3.1-8B", + ] + if is_torch_equal_or_newer("2.12.0.dev"): + models.append("Qwen/Qwen3-4B-Instruct-2507") + return models + + +@pytest.mark.parametrize("model_name", get_test_models()) +@pytest.mark.parametrize( + "shapes_type", + [ + DynamicShapesType.BACKED, + DynamicShapesType.UNBACKED, + DynamicShapesType.BACKED_SIZE_OBLIVIOUS, + ], +) +@pytest.mark.parametrize("use_aot_compile", ["0", "1"]) +@pytest.mark.parametrize("use_bytecode_hook", [True, False]) +@pytest.mark.parametrize("evaluate_guards", [False, True]) +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +def test_dynamic_shapes_compilation( + monkeypatch, + model_name, + shapes_type, + use_aot_compile, + use_bytecode_hook, + evaluate_guards, +): + """Test that all dynamic shapes types compile successfully""" + if shapes_type == DynamicShapesType.UNBACKED and not is_torch_equal_or_newer( + "2.11.0" + ): + # NOTE[ROCm]: shape_id (used by Qwen2/Llama to relate input dims) only + # landed in torch 2.11, but the ROCm CI still runs torch 2.10.x. On + # older torch there's no way to express it, so unbacked shapes go + # data-dependent and compilation blows up -- nothing to test. + pytest.skip("unbacked dynamic shapes with shape_id require torch>=2.11") + + if evaluate_guards and shapes_type == DynamicShapesType.UNBACKED: + pytest.skip("unbacked dynamic shapes do not add guards") + + # TODO is this still a requirement? + if evaluate_guards and use_aot_compile: + pytest.skip("evaluate_guards requires use_aot_compile=0") + + monkeypatch.setenv("VLLM_USE_AOT_COMPILE", use_aot_compile) + monkeypatch.setenv("VLLM_USE_BYTECODE_HOOK", "1" if use_bytecode_hook else "0") + + prompt = "Hello, my name is" + + print(f"Testing {shapes_type.name} dynamic shapes...") + + # Initialize the model with specific dynamic shapes configuration + model = LLM( + model=model_name, + compilation_config={ + "mode": CompilationMode.VLLM_COMPILE, + "dynamic_shapes_config": { + "type": shapes_type.value, + "evaluate_guards": evaluate_guards, + }, + }, + max_model_len=1024, + ) + + sampling_params = SamplingParams(max_tokens=5, temperature=0, logprobs=10) + test_prompts = [prompt, "The capital of France is"] + + compiled_outputs = [] + for p in test_prompts: + output = model.generate(p, sampling_params)[0].outputs[0] + assert len(output.text.strip()) > 0, "Compiled model produced empty output" + compiled_outputs.append((output.token_ids, output.text, output.logprobs)) + + del model + gc.collect() + torch.accelerator.empty_cache() + torch.accelerator.synchronize() + + eager_model = LLM(model=model_name, enforce_eager=True, max_model_len=1024) + eager_outputs = [] + for p in test_prompts: + output = eager_model.generate(p, sampling_params)[0].outputs[0] + assert len(output.text.strip()) > 0, "Eager model produced empty output" + eager_outputs.append((output.token_ids, output.text, output.logprobs)) + del eager_model + gc.collect() + torch.accelerator.empty_cache() + torch.accelerator.synchronize() + + check_logprobs_close( + outputs_0_lst=eager_outputs, + outputs_1_lst=compiled_outputs, + name_0="eager", + name_1="compiled", + ) + + +@pytest.mark.parametrize("use_aot_compile", ["0", "1"]) +@pytest.mark.parametrize( + "dynamic_shapes_type", + [ + DynamicShapesType.BACKED, + DynamicShapesType.BACKED_SIZE_OBLIVIOUS, + ], +) +@pytest.mark.parametrize("evaluate_guards", [False, True]) +def test_model_specialization_with_evaluate_guards( + monkeypatch, use_aot_compile, dynamic_shapes_type, evaluate_guards +): + """Test that evaluate_guards correctly detects shape specialization + violations. + """ + + if ( + use_aot_compile == "1" + and dynamic_shapes_type == DynamicShapesType.BACKED + and evaluate_guards + ): + pytest.skip("evaluate_guards for backed does not work with aot_compile=1") + + @support_torch_compile + class ModelWithSizeCheck(torch.nn.Module): + def __init__(self, **kwargs): + super().__init__() + + def forward(self, x: torch.Tensor): + # This will cause specialization - torch.compile will guard on + # sx.shape[0] + if x.shape[0] >= 10: + return x * 10 + else: + return x * 10 + + @support_torch_compile + class ModelWithOneSizeCheck(torch.nn.Module): + def __init__(self, **kwargs): + super().__init__() + + def forward(self, x: torch.Tensor): + # This will cause 0/1 specializations. + if x.shape[0] == 0: + return x * 10 + if x.shape[0] == 1: + return x * 10 + else: + return x * 10 + + @contextmanager + def use_vllm_config(vllm_config: VllmConfig): + with set_forward_context({}, vllm_config), set_current_vllm_config(vllm_config): + yield + + monkeypatch.setenv("TOKENIZERS_PARALLELISM", "true") + monkeypatch.setenv("VLLM_USE_AOT_COMPILE", use_aot_compile) + monkeypatch.setenv("VLLM_USE_BYTECODE_HOOK", "0") + + # Create vllm config with the desired settings + from vllm.config import CompilationMode + + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + dynamic_shapes_config=DynamicShapesConfig( + type=dynamic_shapes_type, + evaluate_guards=evaluate_guards, + ), + ) + ) + + def test(model_class, input1, input2, is_01_specialization=False): + with ( + torch.no_grad(), + use_vllm_config(vllm_config), + tempfile.TemporaryDirectory() as tmpdirname, + ): + monkeypatch.setenv("VLLM_CACHE_ROOT", tmpdirname) + + model = model_class(vllm_config=vllm_config).cuda() + + model(input1) + + if evaluate_guards and ( + not ( + is_01_specialization + and dynamic_shapes_type == DynamicShapesType.BACKED + ) + ): + # This should fail because guards were added. + with pytest.raises(RuntimeError) as excinfo: + model(input2) + + # Expected failure - guard was violated + error_msg = str(excinfo.value) + assert ( + "GuardManager check failed" in error_msg + or "Detected recompile when torch.compile stance" in error_msg + ), error_msg + + else: + model(input2) + + test(ModelWithSizeCheck, torch.randn(20, 10).cuda(), torch.randn(5, 10).cuda()) + test(ModelWithSizeCheck, torch.randn(5, 10).cuda(), torch.randn(20, 10).cuda()) + test( + ModelWithOneSizeCheck, + torch.randn(20, 10).cuda(), + torch.randn(1, 10).cuda(), + is_01_specialization=True, + ) + + +@pytest.mark.skipif(not is_torch_equal_or_newer("2.10.0"), reason="requires torch 2.10") +def test_piecewise_backend_empty_sym_shape_indices(): + """Test that PiecewiseBackend handles empty sym_shape_indices correctly. + + When all inputs have static shapes (no torch.SymInt), sym_shape_indices + will be empty. The fix in PiecewiseBackend.__call__ handles this case + by using the first compiled range_entry. + """ + gc.collect() + torch.accelerator.empty_cache() + torch.accelerator.synchronize() + + # Use small max_model_len and max_num_batched_tokens to encourage + # static shape compilation with empty sym_shape_indices + llm = LLM( + model="Qwen/Qwen3-0.6B", + max_model_len=512, + max_num_batched_tokens=1, + compilation_config={ + "mode": CompilationMode.VLLM_COMPILE, + "dynamic_shapes_config": { + "type": DynamicShapesType.BACKED.value, + }, + }, + ) + + sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=10) + + # Generate with static shape inputs + output = llm.generate("Hello, my name is", sampling_params=sampling_params) + result = output[0].outputs[0].text + assert len(result) > 0, "Should generate non-empty output" + + # Generate again to verify compilation works with empty sym_shape_indices + output = llm.generate("The capital of France is", sampling_params=sampling_params) + result = output[0].outputs[0].text + assert len(result) > 0, "Should generate non-empty output on second run" + + del llm + gc.collect() + torch.accelerator.empty_cache() + torch.accelerator.synchronize() diff --git a/asian_ai/performance/vLLM/tests/compile/test_graph_partition.py b/asian_ai/performance/vLLM/tests/compile/test_graph_partition.py new file mode 100644 index 0000000000000000000000000000000000000000..3977f44046e32898b9643efcef276ddba6a31b5e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/test_graph_partition.py @@ -0,0 +1,703 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import operator + +import pytest +import torch +import torch._dynamo +import torch.fx as fx +from torch.fx.experimental.proxy_tensor import make_fx + +from vllm.compilation.backends import ( + _decompose_size_nodes, + _is_empty_allocation_node, + split_graph, +) +from vllm.compilation.passes.fx_utils import find_op_nodes +from vllm.platforms import current_platform + +# This import automatically registers `torch.ops.silly.attention` +from . import silly_attention # noqa: F401 + +DEVICE_TYPE = current_platform.device_type + + +def test_getitem_moved_to_producer_subgraph(): + """ + Test that getitem operations are moved to the same subgraph as their input, + preventing tuple inputs to submodules. + """ + + def model_fn(x: torch.Tensor) -> torch.Tensor: + # torch.split returns a tuple, creating real getitem operations + # Should become first submodule that produces tuple + chunks = torch.split(x, x.shape[0] // 2, dim=0) + + # Following ops should become second submodule that consumes tuple + result_0 = torch.relu(chunks[0]) + result_1 = torch.relu(chunks[1]) + return torch.cat([result_0, result_1], dim=0) + + x = torch.randn(4, 3) + gm = make_fx(model_fn)(x) + + has_getitem = any( + node.op == "call_function" and node.target == operator.getitem + for node in gm.graph.nodes + ) + assert has_getitem, "Test setup failed: graph should contain getitem operations" + + # Split on tuple producer aten::split + split_ops = ["aten::split.Tensor"] + split_gm, split_items = split_graph(gm, split_ops) + assert len(split_items) == 2, "Graph should be split into 2 submodules" + + for split_item in split_items: + submodule = split_item.graph + + getitem_on_placeholder = [] + for node in submodule.graph.nodes: + if ( + node.op == "call_function" + and node.target == operator.getitem + and node.args[0].op == "placeholder" + ): + getitem_on_placeholder.append(node) + + assert len(getitem_on_placeholder) == 0, ( + f"Submodule {split_item.submod_name} has getitem operations on " + f"placeholder nodes: {[n.name for n in getitem_on_placeholder]}. " + "This means tuple inputs were not properly eliminated." + ) + + new_x = torch.randn(4, 3) + output_original = gm(new_x) + output_split = split_gm(new_x) + + assert torch.allclose(output_original, output_split), "Output mismatch" + + +def test_no_tuple_inputs_with_multiple_consumers(): + """ + Test that when a tuple is consumed by multiple split operations, + getitem operations are properly moved to avoid tuple inputs. + """ + + def model_fn(x: torch.Tensor) -> torch.Tensor: + # torch.split returns a tuple, creating real getitem operations + # Should become first submodule that produces tuple + chunks = torch.split(x, x.shape[0] // 2, dim=0) + + # These should become second submodule consuming tuple + result_1 = torch.relu(chunks[0]) + result_2 = torch.relu(chunks[1]) + + # Artificial graph splitting point to create another + # independent submodule that consumes tuple later + # This would become the third submodule + result_1 = torch.sigmoid(result_1) + + # Fourth submodule that consumes tuple + result = torch.cat([chunks[0], chunks[1], result_1, result_2]) + return result + + x = torch.randn(4, 3) + gm = make_fx(model_fn)(x) + + has_getitem = any( + node.op == "call_function" and node.target == operator.getitem + for node in gm.graph.nodes + ) + assert has_getitem, "Test setup failed: graph should contain getitem operations" + + split_ops = ["aten::split.Tensor", "aten::sigmoid"] + split_gm, split_items = split_graph(gm, split_ops) + assert len(split_items) == 4, "Graph should be split into 4 submodules" + + for split_item in split_items: + submodule = split_item.graph + + for node in submodule.graph.nodes: + if ( + node.op == "call_function" + and node.target == operator.getitem + and node.args[0].op == "placeholder" + ): + pytest.fail( + f"Submodule {split_item.submod_name} has getitem on " + f"placeholder {node.args[0].name}, indicating it receives " + "a tuple input" + ) + + new_x = torch.randn(4, 3) + output_original = gm(new_x) + output_split = split_gm(new_x) + + assert torch.allclose(output_original, output_split), "Output mismatch after split" + + +def test_consecutive_ops_in_split(): + """ + Test that consecutive splitting operations are grouped into the same subgraph + """ + + def model_fn(x: torch.Tensor) -> torch.Tensor: + """ + Define a simple model where consecutive operations create opportunities + for splitting subgraphs. + """ + # Apply silly attention followed by consecutive operations + intermediate = torch.relu(x) + attn_inout = torch.sqrt(intermediate) + torch.ops.silly.attention(intermediate, intermediate, attn_inout, attn_inout) + final_result = torch.sigmoid(attn_inout) + return final_result + + torch.set_default_device(DEVICE_TYPE) + + # Create the traced FX graph for the model + x = torch.randn(8, 4) + + gm = make_fx(model_fn)(x) + + # Assert presence of the expected operations in the setup + assert ( + len(list(find_op_nodes(torch.ops.aten.relu, gm.graph))) == 1 + and len(list(find_op_nodes(torch.ops.aten.sqrt, gm.graph))) == 1 + ), "Test setup failed: Expected sqrt and relu operations in the graph." + + # Configure split operations to test + splitting_ops = ["silly::attention", "aten::sqrt"] + split_gm, split_items = split_graph(gm, splitting_ops) + + # Validate the number of partitions + assert len(split_items) == 3, ( + "Consecutive splitting operations were not grouped correctly." + ) + + # Validate that correctness is preserved + new_x = torch.randn(8, 4) + output_original = gm(new_x) + output_split = split_gm(new_x) + assert torch.allclose(output_original, output_split), ( + "Output mismatch after splitting." + ) + + # Check the splitting item has 2 nodes exactly (relu and attn) + splitting_items = list(s for s in split_items if s.is_splitting_graph) + assert len(splitting_items) == 1, "Expecting a single splitting graph" + print(splitting_items[0].graph.graph) + splitting_gm = splitting_items[0].graph + assert len(splitting_gm.graph.nodes) == 4, "Expecting 4 nodes in splitting graph" + assert [node.op for node in splitting_gm.graph.nodes] == ["placeholder"] + 2 * [ + "call_function" + ] + ["output"] + + +def _get_empty_nodes(split_item): + return [ + node for node in split_item.graph.graph.nodes if _is_empty_allocation_node(node) + ] + + +def _subgraphs_with_empty_nodes(split_items, *, is_splitting_graph): + return [ + split_item + for split_item in split_items + if split_item.is_splitting_graph == is_splitting_graph + and _get_empty_nodes(split_item) + ] + + +def test_empty_only_partition_stays_separate_after_splitting_predecessor(): + """ + Empty-only subgraphs should not be merged when the only predecessor is + a splitting-op subgraph. + """ + + def model_fn(x: torch.Tensor) -> torch.Tensor: + y = torch.sin(x) + out = torch.empty_like(y) + torch.ops.aten.cos.out(y, out=out) + return out + + x = torch.randn(4, 3) + gm = make_fx(model_fn)(x) + + split_ops = ["aten::sin", "aten::cos.out"] + split_gm, split_items = split_graph(gm, split_ops) + + # Graph partitioning for this pattern is: + # [sin], [empty_like], [cos.out]. + assert len(split_items) == 3, ( + "Empty-only partition should not merge into splitting-op subgraph" + ) + + splitting_with_empty = _subgraphs_with_empty_nodes( + split_items, is_splitting_graph=True + ) + assert len(splitting_with_empty) == 0, ( + "Splitting-op subgraphs should not contain empty allocation nodes: " + f"{[item.submod_name for item in splitting_with_empty]}" + ) + + output_original = gm(x) + output_split = split_gm(x) + assert torch.allclose(output_original, output_split), "Output mismatch after split" + + +def test_empty_only_partition_is_merged(): + """ + Empty-only subgraphs should still be merged when a non-splitting predecessor + exists. The merged empty node must remain outside splitting-op subgraphs. + """ + + def model_fn(x: torch.Tensor) -> torch.Tensor: + base = x + 1 + y = torch.sin(base) + out = torch.empty_like(base) + torch.ops.aten.cos.out(base, out=out) + return out + y + + x = torch.randn(4, 3) + gm = make_fx(model_fn)(x) + split_gm, split_items = split_graph(gm, ["aten::sin", "aten::cos.out"]) + + # Partitioning should be: + # [add, empty_like], [sin], [cos.out], [add]. + assert len(split_items) == 4, ( + "Empty-only partition should be merged into non-splitting predecessor" + ) + + splitting_with_empty = _subgraphs_with_empty_nodes( + split_items, is_splitting_graph=True + ) + assert len(splitting_with_empty) == 0, ( + "Splitting-op subgraphs should not contain empty allocation nodes: " + f"{[item.submod_name for item in splitting_with_empty]}" + ) + + non_splitting_with_empty = _subgraphs_with_empty_nodes( + split_items, is_splitting_graph=False + ) + assert len(non_splitting_with_empty) == 1, ( + "Exactly one non-splitting subgraph should contain the merged empty node" + ) + assert len(_get_empty_nodes(non_splitting_with_empty[0])) == 1, ( + "Expected exactly one empty allocation node in merged subgraph" + ) + + output_original = gm(x) + output_split = split_gm(x) + assert torch.allclose(output_original, output_split), "Output mismatch after split" + + +def test_builtin_empty_only_partition_is_merged(): + """ + In Dynamo graphs, torch.empty/empty_like may appear as builtin call targets + (not aten OpOverload). Ensure empty-only partitions are still merged. + """ + + def model_fn(x: torch.Tensor) -> torch.Tensor: + hidden = x + 1 + out1 = torch.empty_like(hidden) + torch.ops.silly.attention(hidden, hidden, hidden, out1) + out2 = torch.empty_like(hidden) + torch.ops.silly.attention(out1, out1, hidden, out2) + return out2 + hidden + + gm = torch.fx.symbolic_trace(model_fn) + split_gm, split_items = split_graph(gm, ["silly::attention"]) + + # Without empty-only merge, this graph would split into: + # [add, empty_like], [attention], [empty_like], [attention], [add]. + assert len(split_items) == 4, "Builtin empty-only partition should be merged" + + splitting_with_empty = _subgraphs_with_empty_nodes( + split_items, is_splitting_graph=True + ) + assert len(splitting_with_empty) == 0, ( + "Splitting-op subgraphs should not contain empty allocation nodes: " + f"{[item.submod_name for item in splitting_with_empty]}" + ) + + non_splitting_with_empty = _subgraphs_with_empty_nodes( + split_items, is_splitting_graph=False + ) + assert len(non_splitting_with_empty) == 1, ( + "Exactly one non-splitting subgraph should contain merged empty nodes" + ) + assert len(_get_empty_nodes(non_splitting_with_empty[0])) == 2, ( + "Expected two builtin empty_like nodes in merged non-splitting subgraph" + ) + + x = torch.randn(2, 3, device=DEVICE_TYPE) + output_original = gm(x) + output_split = split_gm(x) + assert torch.allclose(output_original, output_split), "Output mismatch after split" + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="requires CUDA") +def test_sym_size_whole_shape_boundary(): + """ + Test that using x.size() (whole shape) across a split boundary can be + compiled by standalone_compile. + + The dynamo graph looks like: + shape = x.size() + y = sigmoid(x) # split point + z = y.clone().view(shape) + + Which splits into: + subgraph0(x) -> shape # returns torch.Size — problematic + subgraph1(x) -> y # sigmoid + subgraph2(y, shape) -> z # view + + Two approaches to fix the torch.Size crossing: + + Approach 1 — move sym_size to consumer (memory implication: x passed to + subgraph2 just for .size()): + subgraph0(x) -> # empty + subgraph1(x) -> y + subgraph2(y, x) -> z # computes shape locally from x + + Approach 2 — decompose shape into individual int/SymInt values: + subgraph0(x) -> s0, val # returns individual scalars, not Size + subgraph1(x) -> y + subgraph2(y, s0, val) -> z # reconstructs view args from scalars + """ + from torch._inductor import standalone_compile + + captured_graph = None + + def capturing_backend(gm: fx.GraphModule, example_inputs: list) -> fx.GraphModule: + nonlocal captured_graph + captured_graph = gm + return gm + + def model_fn(x: torch.Tensor) -> torch.Tensor: + shape = x.size() + x = torch.ops.aten.sigmoid.default(x) + x = x.clone().view(shape) + return x + + x = torch.randn(4, 8) + torch._dynamo.mark_dynamic(x, 0) + compiled_fn = torch.compile(model_fn, backend=capturing_backend) + compiled_fn(x) + + split_gm, split_items = split_graph(captured_graph, ["aten::sigmoid"]) + assert len(split_items) == 3 + + submod_0 = split_gm.submod_0 + example_input = torch.randn(4, 8) + compiled = standalone_compile( + submod_0, [example_input, 4], dynamic_shapes="from_example_inputs" + ) + assert compiled is not None + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="requires CUDA") +def test_symint_crosses_split_boundary(): + """ + Test that SymInt placeholders from torch.compile + mark_dynamic + cross split boundaries safely via split_module's natural threading. + + SymInt values are threaded through subgraphs by split_module and + handled correctly by inductor — no special replacement is needed. + """ + captured_graph = None + + def capturing_backend(gm: fx.GraphModule, example_inputs: list) -> fx.GraphModule: + nonlocal captured_graph + captured_graph = gm + return gm + + def model_fn(x: torch.Tensor) -> torch.Tensor: + batch_size = x.shape[0] + hidden_size = x.shape[1] + x = torch.ops.aten.sigmoid.default(x) + x = x.clone().view(batch_size, hidden_size) + x = torch.ops.aten.sigmoid.default(x) + x = x.clone().view(batch_size, hidden_size) + x = torch.ops.aten.sigmoid.default(x) + x = x.clone().view(batch_size, hidden_size) + return x + + x = torch.randn(4, 8) + torch._dynamo.mark_dynamic(x, 0) + + compiled_fn = torch.compile(model_fn, backend=capturing_backend) + compiled_fn(x) + + assert captured_graph is not None, "Graph should be captured by backend" + + # SymInt placeholders should exist in the captured graph + symint_placeholders = [ + node + for node in captured_graph.graph.nodes + if node.op == "placeholder" + and isinstance(node.meta.get("example_value"), torch.SymInt) + ] + assert len(symint_placeholders) > 0, ( + "Captured graph should have SymInt placeholders from mark_dynamic." + ) + + # split_graph should handle SymInt placeholders without error + split_gm, split_items = split_graph(captured_graph, ["aten::sigmoid"]) + + # Should have 3 splitting subgraphs (3 sigmoids) + splitting_subgraphs = [item for item in split_items if item.is_splitting_graph] + assert len(splitting_subgraphs) == 3, ( + f"Expected 3 splitting subgraphs (3 sigmoids), got {len(splitting_subgraphs)}" + ) + assert len(split_items) >= 6, ( + f"Expected at least 6 total subgraphs, got {len(split_items)}" + ) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="requires CUDA") +def test_shape_boundary_standalone_compile(): + """ + Repro for the original production bug: + + AssertionError: out_spec mismatch + TreeSpec(tuple, None, [*, *, TreeSpec(Size, None, [*, *]), *]) + vs + TreeSpec(tuple, None, [*, *, *, *]) + + A subgraph outputs torch.Size (e.g. torch.Size([s72, 2048])) as one of + its values when shape info crosses a split boundary. aot_autograd / inductor + expect all submodule outputs to be flat tensors or scalars, not torch.Size. + + With the fix, x.size() is decomposed into individual sym_size.int calls + so only scalar SymInts cross the boundary — not the torch.Size. + """ + from torch._inductor import standalone_compile + + captured_graph = None + + def capturing_backend(gm: fx.GraphModule, example_inputs: list) -> fx.GraphModule: + nonlocal captured_graph + captured_graph = gm + return gm + + def model_fn(x: torch.Tensor) -> torch.Tensor: + shape = x.size() + x = torch.ops.aten.sigmoid.default(x) + x = x.clone().view(shape) + return x + + x = torch.randn(4, 8) + torch._dynamo.mark_dynamic(x, 0) + torch.compile(model_fn, backend=capturing_backend)(x) + + split_gm, split_items = split_graph(captured_graph, ["aten::sigmoid"]) + assert len(split_items) == 3 + + # Verify that the consumer subgraph only has a placeholder for the dynamic + # dim (SymInt) — the static dim (8) should be inlined as a literal, not + # threaded as a placeholder. + consumer = split_items[-1] # valid since len == 3: [producer, sigmoid, consumer] + symint_placeholders = [ + n + for n in consumer.graph.graph.nodes + if n.op == "placeholder" + and isinstance(n.meta.get("example_value"), torch.SymInt) + ] + static_int_placeholders = [ + n + for n in consumer.graph.graph.nodes + if n.op == "placeholder" + and isinstance(n.meta.get("example_value"), int) + and not isinstance(n.meta.get("example_value"), torch.SymInt) + ] + assert len(symint_placeholders) >= 1, ( + "Consumer should have a SymInt placeholder for the dynamic dim." + ) + assert len(static_int_placeholders) == 0, ( + "Static dims should be inlined as literals, not threaded as placeholders." + ) + + submod_0 = split_gm.submod_0 + + standalone_compile( + submod_0, [torch.randn(4, 8), 4], dynamic_shapes="from_example_inputs" + ) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="requires CUDA") +def test_size_used_in_multiple_consumer_subgraphs(): + """ + Validates that x.size() (whole shape) used by multiple downstream subgraphs + does not cause torch.Size to cross split boundaries. + + Model: + shape = x.size() # whole shape — must not cross as torch.Size + z1 = sigmoid(x) # split point 1 + y1 = y.view(shape) # consumer 1 uses shape + z2 = sigmoid(z1) # split point 2 + y2 = y.view(shape) # consumer 2 uses shape again + + Without the fix, torch.Size crosses the boundary as a submodule output, + which aot_autograd / standalone_compile rejects. + """ + captured_graph = None + captured_inputs = None + + def capturing_backend(gm: fx.GraphModule, example_inputs: list) -> fx.GraphModule: + nonlocal captured_graph, captured_inputs + captured_graph = gm + captured_inputs = example_inputs + return gm + + def model_fn(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + shape = x.size() + z1 = torch.ops.aten.sigmoid.default(x) + y1 = y.view(shape) + z2 = torch.ops.aten.sigmoid.default(z1) + y2 = y.view(shape) + return z2 + y1 + y2 + + x = torch.randn(4, 8) + y = torch.randn(4, 8) # same shape as x so view(shape) doesn't specialize dim 0 + torch._dynamo.mark_dynamic(x, 0) + torch._dynamo.mark_dynamic(y, 0) + torch.compile(model_fn, backend=capturing_backend)(x, y) + assert captured_graph is not None, "Graph should be captured by backend" + assert captured_inputs is not None, "Example inputs should be captured by backend" + + split_gm, split_items = split_graph(captured_graph, ["aten::sigmoid"]) + + splitting_items = [item for item in split_items if item.is_splitting_graph] + assert len(splitting_items) == 2 + + # Verify functional correctness — fails without the fix because torch.Size + # would cross a split boundary as a submodule output + output_original = model_fn(x, y) + output_split = split_gm(*captured_inputs) + if isinstance(output_split, tuple): + output_split = next(o for o in output_split if isinstance(o, torch.Tensor)) + assert torch.allclose(output_original, output_split), "Output mismatch after split" + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="requires CUDA") +def test_sym_size_metadata_propagated(): + """ + Validates that new sym_size.int nodes created by the pre-pass have + example_value metadata set. Without it, placeholder metadata in consumer + subgraphs would be None, breaking any code that dynamically builds + example inputs from metadata (e.g. standalone_compile per-submodule). + """ + from torch._inductor import standalone_compile + + captured_graph = None + + def capturing_backend(gm: fx.GraphModule, example_inputs: list) -> fx.GraphModule: + nonlocal captured_graph + captured_graph = gm + return gm + + def model_fn(x: torch.Tensor) -> torch.Tensor: + shape = x.size() + x = torch.ops.aten.sigmoid.default(x) + x = x.clone().view(shape) + return x + + x = torch.randn(4, 8) + torch._dynamo.mark_dynamic(x, 0) + torch.compile(model_fn, backend=capturing_backend)(x) + + split_gm, split_items = split_graph(captured_graph, ["aten::sigmoid"]) + + # For each submodule, build example inputs purely from placeholder metadata. + # This fails if example_value is None on any placeholder (i.e. metadata + # was not propagated to the sym_size.int nodes we created). + for item in split_items: + submod = item.graph + example_inputs = [] + for n in submod.graph.nodes: + if n.op != "placeholder": + continue + ev = n.meta.get("example_value") + assert ev is not None, ( + f"Placeholder '{n.name}' in {item.submod_name} has no " + "example_value metadata. sym_size.int nodes must propagate " + "metadata so consumer subgraphs can be introspected." + ) + if isinstance(ev, torch.Tensor): + example_inputs.append(torch.randn(*(int(d) for d in ev.shape))) + else: + example_inputs.append(int(ev)) + standalone_compile(submod, example_inputs, dynamic_shapes="from_example_inputs") + + +def test_decompose_size_with_getitem_user(): + """ + Regression test: _decompose_size_nodes must handle getitem users of size() + correctly. + + When a graph contains x.shape[i], it can appear as: + + %size = call_method[target="size"](args = (%x,)) + %getitem = call_function[target=operator.getitem](args = (%size, 1)) + + The old code spliced *all* per-dim values into every user's args + unconditionally, turning the 2-arg getitem into a malformed 3-arg node: + + %getitem(args = (%sym_size_int, 5120, 1)) # TypeError at runtime + + The fix detects getitem users and replaces them with dims[idx] directly. + """ + # Build a graph manually to guarantee the size() + getitem pattern. + # + # Graph: + # %x = placeholder + # %size = x.size() + # %dim1 = getitem(%size, 1) <-- the getitem branch we're testing + # %relu = relu(%x) + # %view = view(%relu, -1, %dim1) + # return %view + graph = fx.Graph() + x = graph.placeholder("x") + size_node = graph.call_method("size", args=(x,)) + getitem_node = graph.call_function(operator.getitem, args=(size_node, 1)) + relu_node = graph.call_function(torch.ops.aten.relu.default, args=(x,)) + view_node = graph.call_function( + torch.ops.aten.view.default, args=(relu_node, [-1, getitem_node]) + ) + graph.output(view_node) + + # Attach example_value metadata so _decompose_size_nodes can inspect dims. + # dim 0 is dynamic (SymInt), dim 1 is static (8). + from torch._dynamo.source import LocalSource + from torch._subclasses.fake_tensor import FakeTensorMode + from torch.fx.experimental.symbolic_shapes import ShapeEnv + + shape_env = ShapeEnv() + src = LocalSource("batch_size") + sym_batch = shape_env.create_symintnode(shape_env.create_symbol(4, src), hint=4) + fake_mode = FakeTensorMode(shape_env=shape_env) + with fake_mode: + fake_x = torch.empty_strided((sym_batch, 8), (8, 1)) + x.meta["example_value"] = fake_x + + gm = fx.GraphModule(torch.nn.Module(), graph) + + # Run decomposition — this would produce a 3-arg getitem without the fix + _decompose_size_nodes(gm) + + # Verify no size() nodes remain + remaining_size_nodes = list(gm.graph.find_nodes(op="call_method", target="size")) + assert len(remaining_size_nodes) == 0, ( + f"size() nodes should be fully decomposed, found {len(remaining_size_nodes)}" + ) + + # Verify no malformed getitem nodes (3+ args) + for node in gm.graph.nodes: + if node.op == "call_function" and node.target is operator.getitem: + assert len(node.args) == 2, ( + f"getitem node '{node.name}' has {len(node.args)} args " + f"(expected 2): {node.args}" + ) diff --git a/asian_ai/performance/vLLM/tests/compile/test_inductor_fallback_allow_list_patch.py b/asian_ai/performance/vLLM/tests/compile/test_inductor_fallback_allow_list_patch.py new file mode 100644 index 0000000000000000000000000000000000000000..e8c7cf8d78b1ab2cb00c74226ee1bcbccb4aa056 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/test_inductor_fallback_allow_list_patch.py @@ -0,0 +1,250 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the Inductor FALLBACK_ALLOW_LIST patch in env_override.py. + +The patch wraps ``torch._inductor.lowering.FALLBACK_ALLOW_LIST`` in a thin +proxy that auto-allows any custom op in the ``vllm::`` or ``vllm_aiter::`` +namespaces. This routes those ops through Inductor's fast-path +``make_fallback(target, warn=False, override_decomp=True)`` and avoids the +expensive ``error.operator_str(target, args, kwargs)`` formatting that +recursively stringifies every input ``TensorBox``. + +The slow path is what made ``torch.compile`` effectively hang on Kimi-K2.6 +TP=8 (deep MoE/TP IR provenance trees). These tests cover both the proxy's +semantics in isolation and the membership-check fast-path that Inductor's +``GraphLowering.call_function`` actually performs, so we can validate the +optimization without needing a full GPU compile. +""" + +import time + +import pytest + +from vllm.env_override import ( + _patch_inductor_fallback_allow_list, + _VllmFallbackAllowList, +) + + +class TestVllmFallbackAllowListProxy: + """Unit tests for the membership-proxy semantics.""" + + def test_vllm_namespace_auto_allowed(self): + proxy = _VllmFallbackAllowList(set()) + assert "vllm::all_reduce" in proxy + assert "vllm::fused_add_rms_norm" in proxy + assert "vllm::all_reduce.default" in proxy + + def test_vllm_aiter_namespace_auto_allowed(self): + proxy = _VllmFallbackAllowList(set()) + assert "vllm_aiter::fused_add_rms_norm" in proxy + assert "vllm_aiter::rocm_aiter_fused_moe" in proxy + + def test_unknown_namespace_falls_through(self): + proxy = _VllmFallbackAllowList({"torchvision::roi_align"}) + assert "torchvision::roi_align" in proxy + assert "made_up_ns::nonexistent_op" not in proxy + + def test_non_string_falls_through_to_inner(self): + sentinel = object() + inner = {sentinel} + proxy = _VllmFallbackAllowList(inner) + assert sentinel in proxy + assert object() not in proxy + + def test_prefix_only_match_not_substring(self): + proxy = _VllmFallbackAllowList(set()) + assert "not_vllm::something" not in proxy + assert " vllm::space_prefixed" not in proxy + + def test_standard_entries_preserved(self): + base = {"torchvision::roi_align", "aten::index_add"} + proxy = _VllmFallbackAllowList(base) + assert "torchvision::roi_align" in proxy + assert "aten::index_add" in proxy + assert "aten::__not_present__" not in proxy + + def test_add_and_discard_delegate_to_inner(self): + inner: set[str] = set() + proxy = _VllmFallbackAllowList(inner) + proxy.add("custom::op") + assert "custom::op" in inner + proxy.discard("custom::op") + assert "custom::op" not in inner + + def test_iter_len_repr(self): + base = {"torchvision::roi_align", "aten::index_add"} + proxy = _VllmFallbackAllowList(base) + assert set(iter(proxy)) == base + assert len(proxy) == len(base) + assert "torchvision::roi_align" in repr(proxy) + + def test_getattr_delegates_to_inner(self): + class _Inner: + sentinel = "i_am_inner" + + def some_method(self): + return 42 + + inner = _Inner() + proxy = _VllmFallbackAllowList(inner) + assert proxy.sentinel == "i_am_inner" + assert proxy.some_method() == 42 + + def test_sentinel_attribute(self): + proxy = _VllmFallbackAllowList(set()) + assert proxy._vllm_patched is True + + +class TestPatchApplication: + """Integration tests verifying the patch reaches ``torch._inductor``.""" + + def test_patch_applied_to_lowering(self): + import torch._inductor.lowering as _lowering + + assert getattr(_lowering.FALLBACK_ALLOW_LIST, "_vllm_patched", False), ( + "env_override._patch_inductor_fallback_allow_list did not run" + ) + + def test_graph_module_local_binding_rebound(self): + # ``torch/_inductor/graph.py`` does: + # from torch._inductor.lowering import FALLBACK_ALLOW_LIST + # so the patch has to overwrite the graph module's local binding too, + # otherwise the fast-path check in GraphLowering.call_function still + # sees the original (unwrapped) OrderedSet. + import torch._inductor.graph as _graph + import torch._inductor.lowering as _lowering + + if not hasattr(_graph, "FALLBACK_ALLOW_LIST"): + pytest.skip( + "torch._inductor.graph no longer imports FALLBACK_ALLOW_LIST " + "as a module-level symbol; nothing to rebind." + ) + + assert _graph.FALLBACK_ALLOW_LIST is _lowering.FALLBACK_ALLOW_LIST + + def test_patch_is_idempotent(self): + import torch._inductor.lowering as _lowering + + first = _lowering.FALLBACK_ALLOW_LIST + _patch_inductor_fallback_allow_list() + _patch_inductor_fallback_allow_list() + assert _lowering.FALLBACK_ALLOW_LIST is first + + def test_real_vllm_ops_in_real_allow_list(self): + # End-to-end membership check using the live (already-patched) object. + import torch._inductor.lowering as _lowering + + allow_list = _lowering.FALLBACK_ALLOW_LIST + assert "vllm::all_reduce" in allow_list + assert "vllm::fused_add_rms_norm" in allow_list + assert "vllm_aiter::fused_add_rms_norm" in allow_list + + +class TestInductorFallbackFastPath: + """Emulates ``GraphLowering.call_function``'s FALLBACK_ALLOW_LIST check. + + The relevant snippet in ``torch/_inductor/graph.py`` is roughly:: + + base_name = target.name() + if base_name not in FALLBACK_ALLOW_LIST: + log.info( + "Creating implicit fallback for:\\n%s", + error.operator_str(target, args, kwargs), + ) + out = make_fallback(target, ...) + + On a deep MoE/TP graph (Kimi-K2.6 at TP=4/8) ``operator_str`` recurses + through every input ``TensorBox.__str__`` and ends up taking many minutes + of CPU per encountered op. The patch ensures the membership test + short-circuits for ``vllm::*``/``vllm_aiter::*`` ops so the slow path is + never entered. These tests pin that behaviour without needing a real + GPU compile. + """ + + def _simulate_graph_lowering(self, target_names: list[str]): + """Returns the set of target names that would have hit the slow + operator_str() path under the patched FALLBACK_ALLOW_LIST. + """ + import torch._inductor.lowering as _lowering + + allow_list = _lowering.FALLBACK_ALLOW_LIST + slow_path_hits: list[str] = [] + for name in target_names: + if name not in allow_list: + slow_path_hits.append(name) + return slow_path_hits + + def test_vllm_ops_skip_slow_path(self): + slow = self._simulate_graph_lowering( + [ + "vllm::all_reduce", + "vllm::fused_add_rms_norm", + "vllm_aiter::rocm_aiter_fused_moe", + "vllm_aiter::asm_moe", + ] + ) + assert slow == [], ( + "Patched FALLBACK_ALLOW_LIST must short-circuit for all " + f"vllm::*/vllm_aiter::* ops; got slow-path hits: {slow}" + ) + + def test_non_vllm_ops_still_hit_slow_path(self): + # Without the patch this is also what would happen; with the patch + # the behaviour for non-vllm namespaces must be unchanged. + slow = self._simulate_graph_lowering( + ["my_user_ns::custom_op", "fancy_ns::something_else"] + ) + assert "my_user_ns::custom_op" in slow + assert "fancy_ns::something_else" in slow + + def test_kimi_k2_6_style_op_stream(self): + """Emulates one decoder layer's worth of fallback hits. + + Kimi-K2.6 at TP=4 lowers a stream of ``vllm::all_reduce`` + + ``vllm_aiter::fused_add_rms_norm`` calls (one per residual block) + plus a handful of fused-MoE ops. Pre-patch every one of these would + invoke ``operator_str`` and stringify a hundreds-deep IR provenance + tree; post-patch they must all short-circuit. + """ + n_layers = 64 # Kimi-K2.6 has ~64 decoder layers per replica + op_stream: list[str] = [] + for _ in range(n_layers): + op_stream.extend( + [ + "vllm::all_reduce", + "vllm_aiter::fused_add_rms_norm", + "vllm_aiter::rocm_aiter_fused_moe", + ] + ) + + start = time.perf_counter() + slow = self._simulate_graph_lowering(op_stream) + elapsed_s = time.perf_counter() - start + + assert slow == [], ( + f"Expected all {len(op_stream)} vllm/vllm_aiter ops to take " + f"the fast path; got {len(slow)} slow-path hits." + ) + # ``__contains__`` is O(1) per call, so a Kimi-sized stream should + # complete in well under a second even on a slow runner. The + # pre-patch slow path took many minutes per op on Kimi-K2.6 TP=8. + assert elapsed_s < 1.0, ( + f"FALLBACK_ALLOW_LIST membership check is unexpectedly slow: " + f"{elapsed_s:.3f}s for {len(op_stream)} ops" + ) + + def test_inner_set_membership_still_works_for_standard_ops(self): + """The patch must not break Inductor's existing fallback decisions + for non-vllm ops such as ``torchvision::roi_align``.""" + import torch._inductor.lowering as _lowering + + allow_list = _lowering.FALLBACK_ALLOW_LIST + # ``torchvision::roi_align`` has been a member of the upstream + # FALLBACK_ALLOW_LIST since the original Inductor implementation. + # If the proxy ever broke pass-through, this would regress. + if "torchvision::roi_align" not in allow_list: + pytest.skip( + "Upstream FALLBACK_ALLOW_LIST no longer ships " + "torchvision::roi_align; nothing to verify." + ) diff --git a/asian_ai/performance/vLLM/tests/compile/test_rotary_embedding_compile.py b/asian_ai/performance/vLLM/tests/compile/test_rotary_embedding_compile.py new file mode 100644 index 0000000000000000000000000000000000000000..73f6957641813774610b7d21534378f31e31607c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/test_rotary_embedding_compile.py @@ -0,0 +1,70 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +import vllm.envs as envs +from vllm.compilation.decorators import support_torch_compile +from vllm.config import ( + CompilationConfig, + ModelConfig, + VllmConfig, + set_current_vllm_config, +) +from vllm.config.compilation import CompilationMode, CUDAGraphMode +from vllm.model_executor.layers.rotary_embedding import get_rope +from vllm.platforms import current_platform + +DEVICE_TYPE = current_platform.device_type + + +@support_torch_compile +class RotaryEmbeddingCompileModule(torch.nn.Module): + def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None: + super().__init__() + self.rotary_emb = get_rope( + head_size=32, + max_position=128, + dtype=torch.float32, + rope_parameters={"rope_type": "default", "rope_theta": 10000}, + is_neox_style=True, + ) + + def forward( + self, positions: torch.Tensor, query: torch.Tensor, key: torch.Tensor + ) -> torch.Tensor: + q_rot, k_rot = self.rotary_emb(positions, query, key) + return q_rot + k_rot + + +@pytest.mark.skipif(current_platform.is_cpu(), reason="Requires GPU for torch.compile") +def test_rotary_embedding_torch_compile_with_custom_op(monkeypatch): + # Ensure env toggles take effect for this test only. + # The bytecode hook is required to detect buffer mutation in compiled code, + # and AOT compile bypasses that hook entirely. + envs.disable_envs_cache() + monkeypatch.setenv("VLLM_USE_BYTECODE_HOOK", "1") + monkeypatch.setenv("VLLM_USE_AOT_COMPILE", "0") + + device = DEVICE_TYPE + positions = torch.arange(16, device=device) + query = torch.randn(16, 32, device=device, dtype=torch.bfloat16) + key = torch.randn(16, 32, device=device, dtype=torch.bfloat16) + + vllm_config = VllmConfig( + model_config=ModelConfig(dtype=torch.bfloat16), + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + backend="inductor", + custom_ops=["+rotary_embedding"], + cudagraph_mode=CUDAGraphMode.NONE, + cudagraph_num_of_warmups=0, + ), + ) + + with set_current_vllm_config(vllm_config): + model = RotaryEmbeddingCompileModule(vllm_config=vllm_config) + model(positions, query, key) + assert model._compiled_bytecode is not None + assert "update" not in model._compiled_bytecode.co_names diff --git a/asian_ai/performance/vLLM/tests/compile/test_sequence_parallelism_threshold.py b/asian_ai/performance/vLLM/tests/compile/test_sequence_parallelism_threshold.py new file mode 100644 index 0000000000000000000000000000000000000000..5c4c2aa56d1c2d8a34bcb11b33c516587cb71459 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/test_sequence_parallelism_threshold.py @@ -0,0 +1,192 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from vllm.compilation.passes.fusion.sequence_parallelism import ( + SP_MIN_HIDDEN_SIZE, + SP_MIN_PER_GPU_SIZE_MB, + get_sequence_parallelism_threshold, +) + + +class TestGetSequenceParallelismThreshold: + """Tests for get_sequence_parallelism_threshold function.""" + + def test_non_cuda_returns_none(self, mock_cuda_platform): + """Non-CUDA platforms should return None.""" + with mock_cuda_platform(is_cuda=False): + result = get_sequence_parallelism_threshold( + hidden_size=8192, tp_size=2, element_size=2 + ) + assert result is None + + def test_unsupported_device_capability_returns_none(self, mock_cuda_platform): + """Unsupported device capabilities (e.g., sm80) should return None.""" + with mock_cuda_platform(capability=(8, 0)): + result = get_sequence_parallelism_threshold( + hidden_size=8192, tp_size=2, element_size=2 + ) + assert result is None + + def test_small_hidden_size_returns_none(self, mock_cuda_platform): + """H100 with hidden_size below threshold should return None.""" + with mock_cuda_platform(capability=(9, 0)): + result = get_sequence_parallelism_threshold( + hidden_size=4096, + tp_size=2, + element_size=2, # 4096 < 8192 + ) + assert result is None + + def test_h100_large_model_returns_threshold(self, mock_cuda_platform): + """H100 with large enough hidden_size should return calculated threshold.""" + with mock_cuda_platform(capability=(9, 0)): + hidden_size = 8192 + tp_size = 2 + element_size = 2 # float16/bfloat16 + + result = get_sequence_parallelism_threshold( + hidden_size=hidden_size, + tp_size=tp_size, + element_size=element_size, + ) + + # Verify calculation: (8 * 2 * 1024 * 1024) // (8192 * 2) = 1024 + MiB = 1024 * 1024 + expected = int( + (SP_MIN_PER_GPU_SIZE_MB[90] * tp_size * MiB) + // (hidden_size * element_size) + ) + assert result == expected + assert result == 1024 + + @pytest.mark.parametrize( + "hidden_size,tp_size,element_size,expected", + [ + # Boundary: exactly at min hidden size threshold, tp_size=1 + # (8 * 1 * 1024 * 1024) // (8192 * 2) = 512 + (8192, 1, 2, 512), + # Larger hidden size reduces token threshold + # (8 * 1 * 1024 * 1024) // (16384 * 2) = 256 + (16384, 1, 2, 256), + # Larger tp_size increases token threshold + # (8 * 4 * 1024 * 1024) // (8192 * 2) = 2048 + (8192, 4, 2, 2048), + # Larger element_size (fp32) reduces token threshold + # (8 * 2 * 1024 * 1024) // (8192 * 4) = 512 + (8192, 2, 4, 512), + ], + ) + def test_threshold_calculation_variations( + self, mock_cuda_platform, hidden_size, tp_size, element_size, expected + ): + """Test threshold calculation with various parameter combinations.""" + with mock_cuda_platform(capability=(9, 0)): + result = get_sequence_parallelism_threshold( + hidden_size=hidden_size, + tp_size=tp_size, + element_size=element_size, + ) + assert result == expected + + def test_hidden_size_boundary(self, mock_cuda_platform): + """Test behavior at the exact hidden_size boundary.""" + with mock_cuda_platform(capability=(9, 0)): + # Just below threshold + result = get_sequence_parallelism_threshold( + hidden_size=SP_MIN_HIDDEN_SIZE[90] - 1, + tp_size=2, + element_size=2, + ) + assert result is None + + # Exactly at threshold + result = get_sequence_parallelism_threshold( + hidden_size=SP_MIN_HIDDEN_SIZE[90], + tp_size=2, + element_size=2, + ) + assert result is not None + + +# XPU-specific constants (must match sequence_parallelism.py values) +_XPU_MIN_HIDDEN_SIZE = 4096 +_XPU_MIN_PER_GPU_SIZE_MB = 8.0 + + +class TestGetSequenceParallelismThresholdXPU: + """Tests for get_sequence_parallelism_threshold on XPU platform.""" + + def test_xpu_small_hidden_size_returns_none(self, mock_xpu_platform): + """XPU with hidden_size below threshold should return None.""" + with mock_xpu_platform(): + result = get_sequence_parallelism_threshold( + hidden_size=_XPU_MIN_HIDDEN_SIZE - 1, + tp_size=2, + element_size=2, + ) + assert result is None + + def test_xpu_large_model_returns_threshold(self, mock_xpu_platform): + """XPU with hidden_size >= threshold should return calculated value.""" + with mock_xpu_platform(): + hidden_size = _XPU_MIN_HIDDEN_SIZE + tp_size = 2 + element_size = 2 + result = get_sequence_parallelism_threshold( + hidden_size=hidden_size, + tp_size=tp_size, + element_size=element_size, + ) + # (8 * 2 * 1024 * 1024) // (4096 * 2) = 2048 + MiB = 1024 * 1024 + expected = int( + (_XPU_MIN_PER_GPU_SIZE_MB * tp_size * MiB) // (hidden_size * element_size) + ) + assert result == expected + assert result == 2048 + + @pytest.mark.parametrize( + "hidden_size,tp_size,element_size,expected", + [ + # (8 * 1 * 1024 * 1024) // (4096 * 2) = 1024 + (4096, 1, 2, 1024), + # (8 * 4 * 1024 * 1024) // (4096 * 2) = 4096 + (4096, 4, 2, 4096), + # (8 * 2 * 1024 * 1024) // (8192 * 2) = 1024 + (8192, 2, 2, 1024), + # (8 * 2 * 1024 * 1024) // (4096 * 4) = 1024 + (4096, 2, 4, 1024), + ], + ) + def test_xpu_threshold_calculation_variations( + self, mock_xpu_platform, hidden_size, tp_size, element_size, expected + ): + """Test XPU threshold calculation with various parameter combinations.""" + with mock_xpu_platform(): + result = get_sequence_parallelism_threshold( + hidden_size=hidden_size, + tp_size=tp_size, + element_size=element_size, + ) + assert result == expected + + def test_xpu_hidden_size_boundary(self, mock_xpu_platform): + """Test behavior at the exact XPU hidden_size boundary.""" + with mock_xpu_platform(): + # Just below threshold + result = get_sequence_parallelism_threshold( + hidden_size=_XPU_MIN_HIDDEN_SIZE - 1, + tp_size=2, + element_size=2, + ) + assert result is None + + # Exactly at threshold + result = get_sequence_parallelism_threshold( + hidden_size=_XPU_MIN_HIDDEN_SIZE, + tp_size=2, + element_size=2, + ) + assert result is not None diff --git a/asian_ai/performance/vLLM/tests/compile/test_structured_logging.py b/asian_ai/performance/vLLM/tests/compile/test_structured_logging.py new file mode 100644 index 0000000000000000000000000000000000000000..384c3e4227e226765a291bc3ac4626e5b8e31d8e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/test_structured_logging.py @@ -0,0 +1,123 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from unittest.mock import patch + +import pytest +import regex as re +import torch +from torch import nn + +import tests.compile.silly_attention # noqa +from vllm.compilation.decorators import support_torch_compile +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.config.compilation import ( + CompilationConfig, + CompilationMode, + CUDAGraphMode, +) +from vllm.config.scheduler import SchedulerConfig +from vllm.forward_context import set_forward_context +from vllm.platforms import current_platform + +MLP_SIZE = 64 +DEVICE_TYPE = current_platform.device_type + + +@support_torch_compile +class SimpleModel(nn.Module): + """A simple model with a splitting op for piecewise compilation.""" + + def __init__(self, *, vllm_config: VllmConfig, prefix: str = "", **kwargs): + super().__init__() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = x + x + attn_output = torch.empty_like(x) + torch.ops.silly.attention(x, x, x, attn_output) + x = attn_output * 2 + return x + + +class TraceStructuredCapture: + """Captures trace_structured calls for testing.""" + + def __init__(self): + self.calls: list[dict] = [] + + def __call__(self, event_type: str, metadata_fn=None, payload_fn=None, **kwargs): + """Capture a trace_structured call.""" + metadata = metadata_fn() if metadata_fn else {} + self.calls.append( + { + "event_type": event_type, + "metadata": metadata, + } + ) + + def get(self, event_type: str, name_pattern: str) -> list[dict]: + """Get all calls with the given event type and name matching pattern. + + Args: + event_type: The event type to filter by (e.g., "artifact", "graph_dump") + name_pattern: Regex pattern to match against the artifact name + """ + regex = re.compile(name_pattern) + return [ + c + for c in self.calls + if c["event_type"] == event_type + and regex.fullmatch(c.get("metadata", {}).get("name", "")) + ] + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required") +def test_vllm_structured_logging_artifacts(use_fresh_inductor_cache): + """Test that all expected vLLM artifacts are logged during compilation.""" + torch.set_default_device(DEVICE_TYPE) + + capture = TraceStructuredCapture() + + vllm_config = VllmConfig( + compilation_config=CompilationConfig( + mode=CompilationMode.VLLM_COMPILE, + cudagraph_mode=CUDAGraphMode.PIECEWISE, + compile_sizes=[8], + splitting_ops=["silly::attention"], + ), + scheduler_config=SchedulerConfig( + max_num_seqs=8, + max_model_len=8192, + is_encoder_decoder=False, + ), + ) + + # Patch trace_structured to capture calls + with ( + patch("vllm.compilation.backends.trace_structured", capture), + patch("vllm.compilation.piecewise_backend.trace_structured", capture), + set_current_vllm_config(vllm_config), + ): + model = SimpleModel(vllm_config=vllm_config, prefix="test") + with set_forward_context({}, vllm_config=vllm_config): + model(torch.randn(8, MLP_SIZE)) + + config_artifacts = capture.get("artifact", "vllm_compilation_config") + assert len(config_artifacts) == 1, ( + f"Expected 1 vllm_compilation_config, got {len(config_artifacts)}" + ) + vllm_piecewise_split_graph = capture.get("graph_dump", "vllm_piecewise_split_graph") + assert len(vllm_piecewise_split_graph) == 1, ( + "Expected 1 toplevel piecewise split graph, " + f"got {len(vllm_piecewise_split_graph)}" + ) + compile_start_artifacts = capture.get("artifact", "vllm_piecewise_compile_start") + assert len(compile_start_artifacts) == 4, ( + "Expected 4 vllm_piecewise_compile_start " + "(2 subgraphs x 2 ranges each: dynamic + compile size), " + f"got {len(compile_start_artifacts)}" + ) + submod_dumps = capture.get("graph_dump", r"vllm_submod_.*") + assert len(submod_dumps) == 2, ( + "Expected 2 submods (one before attention, one after attention), " + f"got {len(submod_dumps)}" + ) diff --git a/asian_ai/performance/vLLM/tests/compile/test_wrapper.py b/asian_ai/performance/vLLM/tests/compile/test_wrapper.py new file mode 100644 index 0000000000000000000000000000000000000000..23471d9de50a8cdc7e6312502ae114ee0829da93 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/compile/test_wrapper.py @@ -0,0 +1,135 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import os + +import pytest +import torch + +from vllm.compilation.wrapper import TorchCompileWithNoGuardsWrapper +from vllm.config import ( + CompilationConfig, + CompilationMode, + VllmConfig, + set_current_vllm_config, +) + + +class MyMod(torch.nn.Module): + def forward(self, x: torch.Tensor, cache: torch.Tensor | None = None): + if x.size()[0] >= 4: + return x * 2 + else: + return x * 100 + + +class MyWrapper(TorchCompileWithNoGuardsWrapper): + def __init__(self, model): + self.model = model + super().__init__() + + def forward(self, x: torch.Tensor): # type: ignore[override] + # this is the function to be compiled + return self.model(x) + + +@pytest.mark.parametrize("use_bytecode_hook", [True, False]) +def test_torch_compile_wrapper(use_bytecode_hook, monkeypatch): + """Test basic functionality of TorchCompileWithNoGuardsWrapper.""" + # Set the environment variable for this test + monkeypatch.setenv("VLLM_USE_BYTECODE_HOOK", "1" if use_bytecode_hook else "0") + + # Create a proper vLLM config instead of mocking + vllm_config = VllmConfig() + vllm_config.compilation_config = CompilationConfig() + vllm_config.compilation_config.mode = CompilationMode.DYNAMO_TRACE_ONCE + vllm_config.compilation_config.backend = "inductor" + + # Test DYNAMO_TRACE_ONCE + with set_current_vllm_config(vllm_config): + torch._dynamo.reset() + mod = MyMod() + wrapper = MyWrapper(mod) + + # First call should trigger compilation + x = torch.tensor([1, 2, 3, 4]) + torch._dynamo.mark_dynamic(x, 0) + + result1 = wrapper(x) + expected1 = torch.tensor([2, 4, 6, 8]) + assert torch.allclose(result1, expected1), ( + f"Expected {expected1}, got {result1}" + ) + + # Second call should use compiled code + x2 = torch.tensor([1, 2, 3]) + result2 = wrapper(x2) + expected2 = torch.tensor([2, 4, 6]) + assert torch.allclose(result2, expected2), ( + f"Expected {expected2}, got {result2}" + ) + + # without the wrapper result would be different. + result3 = mod(x2) + expected3 = torch.tensor([100, 200, 300]) + + assert torch.allclose(result3, expected3), ( + f"Expected {result3}, got {expected3}" + ) + + # with STOCK_TORCH_COMPILE we do not remove guards. + vllm_config.compilation_config.mode = CompilationMode.STOCK_TORCH_COMPILE + torch._dynamo.reset() + with set_current_vllm_config(vllm_config): + mod = MyMod() + wrapper = MyWrapper(mod) + + # First call should trigger compilation + x = torch.tensor([1, 2, 3, 4]) + torch._dynamo.mark_dynamic(x, 0) + + result1 = wrapper(x) + expected1 = torch.tensor([2, 4, 6, 8]) + assert torch.allclose(result1, expected1), ( + f"Expected {expected1}, got {result1}" + ) + + # Second call should trigger another compilation + x2 = torch.tensor([1, 2, 3]) + result2 = wrapper(x2) + expected2 = torch.tensor([100, 200, 300]) + assert torch.allclose(result2, expected2), ( + f"Expected {expected2}, got {result2}" + ) + + # NO_COMPILATION level not supported. + vllm_config.compilation_config.mode = None + torch._dynamo.reset() + with set_current_vllm_config(vllm_config): + torch._dynamo.reset() + mod = MyMod() + + try: + wrapper = MyWrapper(mod) + except Exception: + return + raise AssertionError("expected an exception to be raised") + + +if __name__ == "__main__": + # Run with both parameter values + + class MockMonkeypatch: + def setenv(self, name, value): + os.environ[name] = value + + mp = MockMonkeypatch() + + print("Testing with VLLM_USE_BYTECODE_HOOK=False") + test_torch_compile_wrapper(False, mp) + + print("Testing with VLLM_USE_BYTECODE_HOOK=True") + test_torch_compile_wrapper(True, mp) + + print("All tests passed!") diff --git a/asian_ai/performance/vLLM/tests/config/base_model_arch_groundtruth.json b/asian_ai/performance/vLLM/tests/config/base_model_arch_groundtruth.json new file mode 100644 index 0000000000000000000000000000000000000000..06bd384fb8cded5fc16046ab3d01744533e9148b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/config/base_model_arch_groundtruth.json @@ -0,0 +1,393 @@ +{ + "state-spaces/mamba-130m-hf": { + "architectures": [ + "MambaForCausalLM" + ], + "model_type": "mamba", + "text_model_type": "mamba", + "hidden_size": 768, + "total_num_hidden_layers": 24, + "total_num_attention_heads": 0, + "head_size": 0, + "vocab_size": 50280, + "total_num_kv_heads": 0, + "num_experts": 0, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.float32" + }, + "mistralai/Mamba-Codestral-7B-v0.1": { + "architectures": [ + "Mamba2ForCausalLM" + ], + "model_type": "mamba", + "text_model_type": "mamba", + "hidden_size": 4096, + "total_num_hidden_layers": 64, + "total_num_attention_heads": 0, + "head_size": 0, + "vocab_size": 32768, + "total_num_kv_heads": 0, + "num_experts": 0, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11": { + "architectures": [ + "Terratorch" + ], + "model_type": "timm_wrapper", + "text_model_type": "timm_wrapper", + "hidden_size": 0, + "total_num_hidden_layers": 0, + "total_num_attention_heads": 0, + "head_size": 0, + "vocab_size": 0, + "total_num_kv_heads": 0, + "num_experts": 0, + "is_deepseek_mla": false, + "is_multimodal_model": true, + "dtype": "torch.float32" + }, + "tiiuae/falcon-mamba-7b-instruct": { + "architectures": [ + "FalconMambaForCausalLM" + ], + "model_type": "falcon_mamba", + "text_model_type": "falcon_mamba", + "hidden_size": 4096, + "total_num_hidden_layers": 64, + "total_num_attention_heads": 0, + "head_size": 0, + "vocab_size": 65024, + "total_num_kv_heads": 0, + "num_experts": 0, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "Zyphra/Zamba2-7B-instruct": { + "architectures": [ + "Zamba2ForCausalLM" + ], + "model_type": "zamba2", + "text_model_type": "zamba2", + "hidden_size": 3584, + "total_num_hidden_layers": 81, + "total_num_attention_heads": 32, + "head_size": 224, + "vocab_size": 32000, + "total_num_kv_heads": 32, + "num_experts": 0, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "mosaicml/mpt-7b": { + "architectures": [ + "MPTForCausalLM" + ], + "model_type": "mpt", + "text_model_type": "mpt", + "hidden_size": 4096, + "total_num_hidden_layers": 32, + "total_num_attention_heads": 32, + "head_size": 128, + "vocab_size": 50432, + "total_num_kv_heads": 32, + "num_experts": 0, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "databricks/dbrx-instruct": { + "architectures": [ + "DbrxForCausalLM" + ], + "model_type": "dbrx", + "text_model_type": "dbrx", + "hidden_size": 6144, + "total_num_hidden_layers": 40, + "total_num_attention_heads": 48, + "head_size": 128, + "vocab_size": 100352, + "total_num_kv_heads": 8, + "num_experts": 0, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "tiiuae/falcon-7b": { + "architectures": [ + "FalconForCausalLM" + ], + "model_type": "falcon", + "text_model_type": "falcon", + "hidden_size": 4544, + "total_num_hidden_layers": 32, + "total_num_attention_heads": 71, + "head_size": 64, + "vocab_size": 65024, + "total_num_kv_heads": 1, + "num_experts": 0, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "tiiuae/falcon-40b": { + "architectures": [ + "FalconForCausalLM" + ], + "model_type": "falcon", + "text_model_type": "falcon", + "hidden_size": 8192, + "total_num_hidden_layers": 60, + "total_num_attention_heads": 128, + "head_size": 64, + "vocab_size": 65024, + "total_num_kv_heads": 8, + "num_experts": 0, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "luccafong/deepseek_mtp_main_random": { + "architectures": [ + "DeepseekV3ForCausalLM" + ], + "model_type": "deepseek_v3", + "text_model_type": "deepseek_v3", + "hidden_size": 2560, + "total_num_hidden_layers": 5, + "total_num_attention_heads": 32, + "head_size": 576, + "vocab_size": 129280, + "total_num_kv_heads": 32, + "num_experts": 72, + "is_deepseek_mla": true, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "luccafong/deepseek_mtp_draft_random": { + "architectures": [ + "DeepseekV3ForCausalLM" + ], + "model_type": "deepseek_v3", + "text_model_type": "deepseek_v3", + "hidden_size": 2560, + "total_num_hidden_layers": 10, + "total_num_attention_heads": 32, + "head_size": 576, + "vocab_size": 129280, + "total_num_kv_heads": 32, + "num_experts": 72, + "is_deepseek_mla": true, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "Qwen/Qwen3-Next-80B-A3B-Instruct": { + "architectures": [ + "Qwen3NextForCausalLM" + ], + "model_type": "qwen3_next", + "text_model_type": "qwen3_next", + "hidden_size": 2048, + "total_num_hidden_layers": 48, + "total_num_attention_heads": 16, + "head_size": 256, + "vocab_size": 151936, + "total_num_kv_heads": 2, + "num_experts": 512, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "tiny-random/qwen3-next-moe": { + "architectures": [ + "Qwen3NextForCausalLM" + ], + "model_type": "qwen3_next", + "text_model_type": "qwen3_next", + "hidden_size": 8, + "total_num_hidden_layers": 4, + "total_num_attention_heads": 16, + "head_size": 32, + "vocab_size": 151936, + "total_num_kv_heads": 8, + "num_experts": 32, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "zai-org/GLM-4.5": { + "architectures": [ + "Glm4MoeForCausalLM" + ], + "model_type": "glm4_moe", + "text_model_type": "glm4_moe", + "hidden_size": 5120, + "total_num_hidden_layers": 92, + "total_num_attention_heads": 96, + "head_size": 128, + "vocab_size": 151552, + "total_num_kv_heads": 8, + "num_experts": 160, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "baidu/ERNIE-4.5-21B-A3B-PT": { + "architectures": [ + "Ernie4_5_MoeForCausalLM" + ], + "model_type": "ernie4_5_moe", + "text_model_type": "ernie4_5_moe", + "hidden_size": 2560, + "total_num_hidden_layers": 28, + "total_num_attention_heads": 20, + "head_size": 128, + "vocab_size": 103424, + "total_num_kv_heads": 4, + "num_experts": 64, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "lmsys/gpt-oss-20b-bf16": { + "architectures": [ + "GptOssForCausalLM" + ], + "model_type": "gpt_oss", + "text_model_type": "gpt_oss", + "hidden_size": 2880, + "total_num_hidden_layers": 24, + "total_num_attention_heads": 64, + "head_size": 64, + "vocab_size": 201088, + "total_num_kv_heads": 8, + "num_experts": 32, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "deepseek-ai/DeepSeek-V3.2-Exp": { + "architectures": [ + "DeepseekV32ForCausalLM" + ], + "model_type": "deepseek_v32", + "text_model_type": "deepseek_v32", + "hidden_size": 7168, + "total_num_hidden_layers": 61, + "total_num_attention_heads": 128, + "head_size": 576, + "vocab_size": 129280, + "total_num_kv_heads": 128, + "num_experts": 256, + "is_deepseek_mla": true, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "meta-llama/Llama-4-Scout-17B-16E-Instruct": { + "architectures": [ + "Llama4ForConditionalGeneration" + ], + "model_type": "llama4", + "text_model_type": "llama4_text", + "hidden_size": 5120, + "total_num_hidden_layers": 48, + "total_num_attention_heads": 40, + "head_size": 128, + "vocab_size": 202048, + "total_num_kv_heads": 8, + "num_experts": 16, + "is_deepseek_mla": false, + "is_multimodal_model": true, + "dtype": "torch.bfloat16" + }, + "nvidia/Llama-3_3-Nemotron-Super-49B-v1": { + "architectures": [ + "DeciLMForCausalLM" + ], + "model_type": "nemotron-nas", + "text_model_type": "nemotron-nas", + "hidden_size": 8192, + "total_num_hidden_layers": 80, + "total_num_attention_heads": 64, + "head_size": 128, + "vocab_size": 128256, + "total_num_kv_heads": 8, + "num_experts": 0, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "XiaomiMiMo/MiMo-7B-RL": { + "architectures": [ + "MiMoForCausalLM" + ], + "model_type": "mimo", + "text_model_type": "mimo", + "hidden_size": 4096, + "total_num_hidden_layers": 36, + "total_num_attention_heads": 32, + "head_size": 128, + "vocab_size": 151680, + "total_num_kv_heads": 8, + "num_experts": 0, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "meituan-longcat/LongCat-Flash-Chat": { + "architectures": [ + "LongcatFlashForCausalLM" + ], + "model_type": "longcat_flash", + "text_model_type": "longcat_flash", + "hidden_size": 6144, + "total_num_hidden_layers": 28, + "total_num_attention_heads": 64, + "head_size": 576, + "vocab_size": 131072, + "total_num_kv_heads": 64, + "num_experts": 512, + "is_deepseek_mla": true, + "is_multimodal_model": false, + "dtype": "torch.float32" + }, + "stepfun-ai/Step-3.5-Flash": { + "architectures": [ + "Step3p5ForCausalLM" + ], + "model_type": "step3p5", + "text_model_type": "step3p5", + "hidden_size": 4096, + "total_num_hidden_layers": 45, + "total_num_attention_heads": 64, + "head_size": 128, + "vocab_size": 128896, + "total_num_kv_heads": 8, + "num_experts": 288, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16": { + "architectures": [ + "NemotronHForCausalLM" + ], + "model_type": "nemotron_h", + "text_model_type": "nemotron_h", + "hidden_size": 2688, + "total_num_hidden_layers": 52, + "total_num_attention_heads": 32, + "head_size": 128, + "vocab_size": 131072, + "total_num_kv_heads": 2, + "num_experts": 128, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + } +} diff --git a/asian_ai/performance/vLLM/tests/config/draft_model_arch_groundtruth.json b/asian_ai/performance/vLLM/tests/config/draft_model_arch_groundtruth.json new file mode 100644 index 0000000000000000000000000000000000000000..f06fed1912dcbfbec4aca04e0799f7f606e97555 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/config/draft_model_arch_groundtruth.json @@ -0,0 +1,87 @@ +{ + "abhigoyal/vllm-medusa-llama-68m-random": { + "architectures": [ + "MedusaModel" + ], + "model_type": "medusa", + "text_model_type": "medusa", + "hidden_size": 768, + "total_num_hidden_layers": 1, + "total_num_attention_heads": 0, + "head_size": "Error: integer division or modulo by zero", + "vocab_size": 32000, + "total_num_kv_heads": 0, + "num_experts": 0, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "torch.float32" + }, + "luccafong/deepseek_mtp_draft_random": { + "architectures": [ + "DeepSeekMTPModel" + ], + "model_type": "deepseek_mtp", + "text_model_type": "deepseek_mtp", + "hidden_size": 2560, + "total_num_hidden_layers": 1, + "total_num_attention_heads": 32, + "head_size": 576, + "vocab_size": 129280, + "total_num_kv_heads": 32, + "num_experts": 72, + "is_deepseek_mla": true, + "is_multimodal_model": false, + "dtype": "torch.bfloat16" + }, + "eagle618/eagle-deepseek-v3-random": { + "architectures": [ + "EagleDeepSeekMTPModel" + ], + "model_type": "eagle", + "text_model_type": "eagle", + "hidden_size": 2560, + "total_num_hidden_layers": 1, + "total_num_attention_heads": 32, + "head_size": 576, + "vocab_size": 129280, + "total_num_kv_heads": 32, + "num_experts": 72, + "is_deepseek_mla": true, + "is_multimodal_model": false, + "dtype": "bfloat16" + }, + "yuhuili/EAGLE-LLaMA3-Instruct-8B": { + "architectures": [ + "EagleLlamaForCausalLM" + ], + "model_type": "eagle", + "text_model_type": "eagle", + "hidden_size": 4096, + "total_num_hidden_layers": 1, + "total_num_attention_heads": 32, + "head_size": 128, + "vocab_size": 128256, + "total_num_kv_heads": 8, + "num_experts": 0, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "float16" + }, + "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B": { + "architectures": [ + "Eagle3LlamaForCausalLM" + ], + "model_type": "eagle", + "text_model_type": "eagle", + "hidden_size": 4096, + "total_num_hidden_layers": 1, + "total_num_attention_heads": 32, + "head_size": 128, + "vocab_size": 128256, + "total_num_kv_heads": 8, + "num_experts": 0, + "is_deepseek_mla": false, + "is_multimodal_model": false, + "dtype": "float16" + } +} diff --git a/asian_ai/performance/vLLM/tests/config/test_config.yaml b/asian_ai/performance/vLLM/tests/config/test_config.yaml new file mode 100644 index 0000000000000000000000000000000000000000..0788ac82ee383e3d8fc8c2d36729d34c335fc377 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/config/test_config.yaml @@ -0,0 +1,4 @@ +port: 12312 +served_model_name: mymodel +tensor_parallel_size: 2 +trust_remote_code: true diff --git a/asian_ai/performance/vLLM/tests/config/test_config_generation.py b/asian_ai/performance/vLLM/tests/config/test_config_generation.py new file mode 100644 index 0000000000000000000000000000000000000000..d7a318c3ec8be1b17a7f991a798ee27d6db12a5c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/config/test_config_generation.py @@ -0,0 +1,111 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest + +from vllm.engine.arg_utils import EngineArgs +from vllm.model_executor.layers.quantization.quark.utils import deep_compare + + +def test_cuda_empty_vs_unset_configs(monkeypatch: pytest.MonkeyPatch): + """Test that configs created with normal (untouched) CUDA_VISIBLE_DEVICES + and CUDA_VISIBLE_DEVICES="" are equivalent. This ensures consistent + behavior regardless of whether GPU visibility is disabled via empty string + or left in its normal state. + """ + + def create_config(): + engine_args = EngineArgs( + model="deepseek-ai/DeepSeek-V2-Lite", trust_remote_code=True + ) + return engine_args.create_engine_config() + + # Create config with CUDA_VISIBLE_DEVICES set normally + normal_config = create_config() + + # Create config with CUDA_VISIBLE_DEVICES="" + with monkeypatch.context() as m: + m.setenv("CUDA_VISIBLE_DEVICES", "") + empty_config = create_config() + + normal_config_dict = vars(normal_config) + empty_config_dict = vars(empty_config) + + # Remove instance_id before comparison as it's expected to be different + normal_config_dict.pop("instance_id", None) + empty_config_dict.pop("instance_id", None) + + assert deep_compare(normal_config_dict, empty_config_dict), ( + 'Configs with normal CUDA_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES=""' + " should be equivalent" + ) + + +def test_ray_runtime_env(monkeypatch: pytest.MonkeyPatch): + # In testing, this method needs to be nested inside as ray does not + # see the test module. + def create_config(): + engine_args = EngineArgs( + model="deepseek-ai/DeepSeek-V2-Lite", trust_remote_code=True + ) + return engine_args.create_engine_config() + + config = create_config() + parallel_config = config.parallel_config + assert parallel_config.ray_runtime_env is None + + import ray + + ray.init() + + runtime_env = { + "env_vars": { + "TEST_ENV_VAR": "test_value", + # In future ray versions, this will be default, so when setting a + # task or actor with num_gpus=None/0, the visible devices env var + # won't be overridden resulting in no GPUs being visible on a gpu + # machine. + "RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO": "0", + }, + } + + config_ref = ray.remote(create_config).options(runtime_env=runtime_env).remote() + + config = ray.get(config_ref) + parallel_config = config.parallel_config + assert parallel_config.ray_runtime_env is not None + assert ( + parallel_config.ray_runtime_env.env_vars().get("TEST_ENV_VAR") == "test_value" + ) + + ray.shutdown() + + +def test_unrecognized_env(monkeypatch): + import os + + from vllm.envs import environment_variables + + # Remove any existing unrecognized VLLM env vars that might interfere + for env in list(os.environ): + if env.startswith("VLLM_") and env not in environment_variables: + monkeypatch.delenv(env, raising=False) + + # Test that if fail_on_environ_validation is True, then an error + # is raised when an unrecognized vLLM environment variable is set + monkeypatch.setenv("VLLM_UNRECOGNIZED_ENV_VAR", "some_value") + engine_args = EngineArgs( + fail_on_environ_validation=True, + ) + with pytest.raises(ValueError, match="Unknown vLLM environment variable detected"): + engine_args.create_engine_config() + + # Test that if fail_on_environ_validation is False, then no error is raised + engine_args = EngineArgs() + engine_args.create_engine_config() + + # Test that when the unrecognized env var is removed, no error is raised + monkeypatch.delenv("VLLM_UNRECOGNIZED_ENV_VAR") + engine_args = EngineArgs( + fail_on_environ_validation=True, + ) + engine_args.create_engine_config() diff --git a/asian_ai/performance/vLLM/tests/config/test_config_utils.py b/asian_ai/performance/vLLM/tests/config/test_config_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..834cc6435418bb0d8311edf998cbe54ee5449aac --- /dev/null +++ b/asian_ai/performance/vLLM/tests/config/test_config_utils.py @@ -0,0 +1,203 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from dataclasses import dataclass +from enum import Enum + +import pytest + +from vllm.config.utils import get_hash_factors, hash_factors, normalize_value + +# Helpers + + +def endswith_fqname(obj, suffix: str) -> bool: + # normalize_value(type) returns fully-qualified name + # Compare suffix to avoid brittle import paths. + out = normalize_value(obj) + return isinstance(out, str) and out.endswith(suffix) + + +def expected_path(p_str: str = ".") -> str: + import pathlib + + p = pathlib.Path(p_str) + return p.expanduser().resolve().as_posix() + + +# Minimal dataclass to test get_hash_factors. +# Avoid importing heavy vLLM configs. +@dataclass +class SimpleConfig: + a: object + b: object | None = None + + +class DummyLogprobsMode(Enum): + RAW_LOGITS = "raw_logits" + + +def test_hash_factors_deterministic(): + """Test that hash_factors produces consistent SHA-256 hashes""" + factors = {"a": 1, "b": "test"} + hash1 = hash_factors(factors) + hash2 = hash_factors(factors) + + assert hash1 == hash2 + # Dict key insertion order should not affect the hash. + factors_reordered = {"b": "test", "a": 1} + assert hash_factors(factors_reordered) == hash1 + assert len(hash1) == 64 + assert all(c in "0123456789abcdef" for c in hash1) + + +@pytest.mark.parametrize( + "inp, expected", + [ + (None, None), + (True, True), + (1, 1), + (1.0, 1.0), + ("x", "x"), + (b"ab", "6162"), + (bytearray(b"ab"), "6162"), + ([1, 2], (1, 2)), + ({"b": 2, "a": 1}, (("a", 1), ("b", 2))), + ], +) +def test_normalize_value_matrix(inp, expected): + """Parametric input→expected normalization table.""" + assert normalize_value(inp) == expected + + +def test_normalize_value_enum(): + # Enums normalize to (module.QualName, value). + # DummyLogprobsMode uses a string payload. + out = normalize_value(DummyLogprobsMode.RAW_LOGITS) + assert isinstance(out, tuple) + assert out[0].endswith("DummyLogprobsMode") + # Expect string payload 'raw_logits'. + assert out[1] == "raw_logits" + + +def test_normalize_value_set_order_insensitive(): + # Sets are unordered; normalize_value sorts elements for determinism. + assert normalize_value({3, 1, 2}) == normalize_value({1, 2, 3}) + + +def test_normalize_value_path_normalization(): + from pathlib import Path # local import to avoid global dependency + + # Paths expand/resolve to absolute strings. + # Stabilizes hashing across working dirs. + assert normalize_value(Path(".")) == expected_path(".") + + +def test_normalize_value_uuid_and_to_json(): + # Objects may normalize via uuid() or to_json_string(). + class HasUUID: + def uuid(self): + return "test-uuid" + + class ToJson: + def to_json_string(self): + return '{"x":1}' + + assert normalize_value(HasUUID()) == "test-uuid" + assert normalize_value(ToJson()) == '{"x":1}' + + +@pytest.mark.parametrize( + "bad", + [ + (lambda x: x), + (type("CallableInstance", (), {"__call__": lambda self: 0}))(), + (lambda: (lambda: 0))(), # nested function instance + ], +) +def test_error_cases(bad): + """Inputs expected to raise TypeError.""" + # Reject functions/lambdas/callable instances + # to avoid under-hashing. + with pytest.raises(TypeError): + normalize_value(bad) + + +def test_enum_vs_int_disambiguation(): + # int stays primitive + nf_int = normalize_value(1) + assert nf_int == 1 + + # enum becomes ("module.QualName", value) + nf_enum = normalize_value(DummyLogprobsMode.RAW_LOGITS) + assert isinstance(nf_enum, tuple) and len(nf_enum) == 2 + enum_type, enum_val = nf_enum + assert enum_type.endswith(".DummyLogprobsMode") + assert enum_val == "raw_logits" + + # Build factor dicts from configs with int vs enum + f_int = get_hash_factors(SimpleConfig(1), set()) + f_enum = get_hash_factors(SimpleConfig(DummyLogprobsMode.RAW_LOGITS), set()) + # The int case remains a primitive value + assert f_int["a"] == 1 + # The enum case becomes a tagged tuple ("module.QualName", "raw_logits") + assert isinstance(f_enum["a"], tuple) and f_enum["a"][1] == "raw_logits" + # Factor dicts must differ so we don't collide primitives with Enums. + assert f_int != f_enum + # Hash digests must differ correspondingly + assert hash_factors(f_int) != hash_factors(f_enum) + + # Hash functions produce stable hex strings + h_int = hash_factors(f_int) + h_enum = hash_factors(f_enum) + assert isinstance(h_int, str) and len(h_int) == 64 + assert isinstance(h_enum, str) and len(h_enum) == 64 + + +def test_classes_are_types(): + """Types normalize to FQNs; include real vLLM types.""" + # Only classes allowed; functions/lambdas are rejected. + # Canonical form is the fully-qualified name. + assert isinstance(normalize_value(str), str) + + class LocalDummy: + pass + + assert endswith_fqname(LocalDummy, ".LocalDummy") + + +def test_envs_compile_factors_stable(): + """Test that envs.compile_factors() hash is stable across fresh initializations. + + Uses subprocesses to ensure env vars with dynamic defaults (like UUIDs) + are freshly generated each time, verifying they're properly ignored. + """ + import subprocess + import sys + + code = """ +import sys +import logging +logging.disable(logging.CRITICAL) +from vllm import envs +from vllm.config.utils import hash_factors +print(hash_factors(envs.compile_factors())) +""" + + def get_hash_in_subprocess(): + result = subprocess.run( + [sys.executable, "-c", code], + capture_output=True, + text=True, + check=True, + env={**dict(__import__("os").environ), "VLLM_LOGGING_LEVEL": "ERROR"}, + ) + return result.stdout.strip() + + hash1 = get_hash_in_subprocess() + hash2 = get_hash_in_subprocess() + + assert hash1 == hash2, ( + "compile_factors hash differs between fresh initializations - " + "dynamic env vars may not be properly ignored" + ) diff --git a/asian_ai/performance/vLLM/tests/config/test_config_with_model.yaml b/asian_ai/performance/vLLM/tests/config/test_config_with_model.yaml new file mode 100644 index 0000000000000000000000000000000000000000..09549436dac93803b82bffd25a8cdf3ffe99b34d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/config/test_config_with_model.yaml @@ -0,0 +1,6 @@ +# Same as test_config.yaml but with model specified +model: config-model +port: 12312 +served_model_name: mymodel +tensor_parallel_size: 2 +trust_remote_code: true diff --git a/asian_ai/performance/vLLM/tests/config/test_model_arch_config.py b/asian_ai/performance/vLLM/tests/config/test_model_arch_config.py new file mode 100644 index 0000000000000000000000000000000000000000..551f8c107e2fb44a5f5ed775dc3f7511bb6016b6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/config/test_model_arch_config.py @@ -0,0 +1,174 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for ModelArchitectureConfig and its integration with ModelConfig.""" + +import json +from pathlib import Path + +import pytest +from transformers import PretrainedConfig + +from vllm.config import ModelConfig, ParallelConfig, SpeculativeConfig +from vllm.transformers_utils.model_arch_config_convertor import ( + ModelArchConfigConvertorBase, +) + +BASE_TRUST_REMOTE_CODE_MODELS = { + "nvidia/Llama-3_3-Nemotron-Super-49B-v1", + "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16", + "XiaomiMiMo/MiMo-7B-RL", + "stepfun-ai/Step-3.5-Flash", + # Excluded: Not available online right now + # "FreedomIntelligence/openPangu-Ultra-MoE-718B-V1.1", + "meituan-longcat/LongCat-Flash-Chat", +} + +BASE_MODELS_TO_TEST = [ + "state-spaces/mamba-130m-hf", + "mistralai/Mamba-Codestral-7B-v0.1", + # Excluded: terratorch/torchgeo version mismatch in CPU CI environment + # (NonGeoDataset import error). Tested in model initialization tests. + # "ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11", + "Zyphra/Zamba2-7B-instruct", + # FIXME: mosaicml/mpt-7b has been deleted + # "mosaicml/mpt-7b", + # FIXME: databricks/dbrx-instruct has been deleted + # "databricks/dbrx-instruct", + "tiiuae/falcon-7b", + "tiiuae/falcon-40b", + "luccafong/deepseek_mtp_main_random", + "Qwen/Qwen3-Next-80B-A3B-Instruct", + "tiny-random/qwen3-next-moe", + "zai-org/GLM-4.5", + "baidu/ERNIE-4.5-21B-A3B-PT", + # Models using base convertor + "lmsys/gpt-oss-20b-bf16", + "deepseek-ai/DeepSeek-V3.2-Exp", + "meta-llama/Llama-4-Scout-17B-16E-Instruct", +] + list(BASE_TRUST_REMOTE_CODE_MODELS) + +# (target_model, draft_model, trust_remote_code) +SPECULATIVE_MODELS = [ + ("JackFram/llama-68m", "abhigoyal/vllm-medusa-llama-68m-random", False), + ("luccafong/deepseek_mtp_main_random", "luccafong/deepseek_mtp_draft_random", True), + ("eagle618/deepseek-v3-random", "eagle618/eagle-deepseek-v3-random", True), + ("meta-llama/Meta-Llama-3-8B-Instruct", "yuhuili/EAGLE-LLaMA3-Instruct-8B", True), + ("meta-llama/Llama-3.1-8B-Instruct", "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B", True), +] + + +def _load_groundtruth(filename: str) -> dict: + """Load groundtruth JSON from the test directory.""" + groundtruth_path = Path(__file__).parent / filename + with open(groundtruth_path) as f: + return json.load(f) + + +def _assert_model_arch_config( + model_config, expected: dict, check_head_size: bool = True +): + """Assert model_arch_config matches expected values.""" + model_arch_config = model_config.model_arch_config + assert model_arch_config.architectures == expected["architectures"] + assert model_arch_config.model_type == expected["model_type"] + assert model_arch_config.text_model_type == expected["text_model_type"] + assert model_arch_config.hidden_size == expected["hidden_size"] + assert ( + model_arch_config.total_num_hidden_layers == expected["total_num_hidden_layers"] + ) + assert ( + model_arch_config.total_num_attention_heads + == expected["total_num_attention_heads"] + ) + assert model_arch_config.vocab_size == expected["vocab_size"] + assert model_arch_config.total_num_kv_heads == expected["total_num_kv_heads"] + assert model_arch_config.num_experts == expected["num_experts"] + assert model_arch_config.is_deepseek_mla == expected["is_deepseek_mla"] + + torch_dtype = ModelArchConfigConvertorBase.get_torch_dtype( + model_config.hf_config, + model_config.model, + revision=model_config.revision, + config_format="hf", + ) + assert str(torch_dtype) == expected["dtype"] + + if check_head_size: + assert model_arch_config.head_size == expected["head_size"] + + +def _assert_model_config_methods( + model_config, expected: dict, check_head_size: bool = True +): + """Assert model_config methods return expected values.""" + assert model_config.architectures == expected["architectures"] + assert model_config.get_vocab_size() == expected["vocab_size"] + assert model_config.get_hidden_size() == expected["hidden_size"] + assert model_config.get_total_num_kv_heads() == expected["total_num_kv_heads"] + assert model_config.get_num_experts() == expected["num_experts"] + assert ( + model_config.get_total_num_hidden_layers() + == expected["total_num_hidden_layers"] + ) + + if check_head_size: + assert model_config.get_head_size() == expected["head_size"] + + +def test_head_size_falls_back_when_head_dim_is_zero(): + """Regression test for configs that materialize missing head_dim as 0.""" + hf_config = PretrainedConfig( + model_type="deepseek_vl_v2", + hidden_size=1280, + num_attention_heads=10, + num_key_value_heads=10, + head_dim=0, + kv_lora_rank=None, + ) + + convertor = ModelArchConfigConvertorBase(hf_config, hf_config) + + assert convertor.get_head_size() == 128 + + +@pytest.mark.parametrize("model", BASE_MODELS_TO_TEST) +def test_base_model_arch_config(model: str): + """Test model architecture config for base models.""" + groundtruth = _load_groundtruth("base_model_arch_groundtruth.json") + expected = groundtruth[model] + + model_config = ModelConfig( + model, trust_remote_code=model in BASE_TRUST_REMOTE_CODE_MODELS + ) + + _assert_model_arch_config(model_config, expected) + _assert_model_config_methods(model_config, expected) + + +@pytest.mark.parametrize( + "target_model,draft_model,trust_remote_code", SPECULATIVE_MODELS +) +def test_draft_model_arch_config( + target_model: str, draft_model: str, trust_remote_code: bool +): + """Test model architecture config for draft/speculative models.""" + groundtruth = _load_groundtruth("draft_model_arch_groundtruth.json") + expected = groundtruth[draft_model] + + target_model_config = ModelConfig(target_model, trust_remote_code=trust_remote_code) + speculative_config = SpeculativeConfig( + model=draft_model, + num_speculative_tokens=1, + target_model_config=target_model_config, + target_parallel_config=ParallelConfig(), + ) + model_config = speculative_config.draft_model_config + + # For medusa models, head_size may cause division by zero before + # model_arch_config was introduced, so we conditionally check it + check_head_size = isinstance(expected["head_size"], int) + + _assert_model_arch_config(model_config, expected, check_head_size=check_head_size) + _assert_model_config_methods( + model_config, expected, check_head_size=check_head_size + ) diff --git a/asian_ai/performance/vLLM/tests/config/test_mp_reducer.py b/asian_ai/performance/vLLM/tests/config/test_mp_reducer.py new file mode 100644 index 0000000000000000000000000000000000000000..4d5ef10d972c76ba1621c19a3b89e19aee4bd333 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/config/test_mp_reducer.py @@ -0,0 +1,53 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import sys +from unittest.mock import patch + +from vllm.config import VllmConfig +from vllm.engine.arg_utils import AsyncEngineArgs +from vllm.v1.engine.async_llm import AsyncLLM + + +def test_mp_reducer(): + """ + Test that _reduce_config reducer is registered when AsyncLLM is instantiated + without transformers_modules. This is a regression test for + https://github.com/vllm-project/vllm/pull/18640. + """ + + # Ensure transformers_modules is not in sys.modules + if "transformers_modules" in sys.modules: + del sys.modules["transformers_modules"] + + with patch("multiprocessing.reducer.register") as mock_register: + engine_args = AsyncEngineArgs( + model="facebook/opt-125m", + max_model_len=32, + gpu_memory_utilization=0.1, + disable_log_stats=True, + ) + + async_llm = AsyncLLM.from_engine_args( + engine_args, + start_engine_loop=False, + ) + + assert mock_register.called, ( + "multiprocessing.reducer.register should have been called" + ) + + vllm_config_registered = False + for call_args in mock_register.call_args_list: + # Verify that a reducer for VllmConfig was registered + if len(call_args[0]) >= 2 and call_args[0][0] == VllmConfig: + vllm_config_registered = True + + reducer_func = call_args[0][1] + assert callable(reducer_func), "Reducer function should be callable" + break + + assert vllm_config_registered, ( + "VllmConfig should have been registered to multiprocessing.reducer" + ) + + async_llm.shutdown() diff --git a/asian_ai/performance/vLLM/tests/config/test_multimodal_config.py b/asian_ai/performance/vLLM/tests/config/test_multimodal_config.py new file mode 100644 index 0000000000000000000000000000000000000000..0a82f48b5f91cebec150d8f00a1ce6f136eadbf7 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/config/test_multimodal_config.py @@ -0,0 +1,61 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from vllm.config.model import ModelConfig +from vllm.config.multimodal import MultiModalConfig +from vllm.v1.attention.backends.registry import AttentionBackendEnum + + +def test_mm_encoder_attn_backend_str_conversion(): + config = MultiModalConfig(mm_encoder_attn_backend="FLASH_ATTN") + assert config.mm_encoder_attn_backend == AttentionBackendEnum.FLASH_ATTN + + +def test_mm_encoder_attn_backend_invalid(): + with pytest.raises(ValueError): + MultiModalConfig(mm_encoder_attn_backend="not_a_backend") + + +def test_mm_encoder_attn_backend_hash_updates(): + base_hash = MultiModalConfig().compute_hash() + overridden_hash = MultiModalConfig( + mm_encoder_attn_backend=AttentionBackendEnum.FLASH_ATTN + ).compute_hash() + assert base_hash != overridden_hash + + +def test_language_model_only_does_not_affect_mm_hash(): + """language_model_only does not affect the ViT computation graph, + so it should not change the multimodal config hash.""" + base_hash = MultiModalConfig().compute_hash() + lm_only_hash = MultiModalConfig(language_model_only=True).compute_hash() + assert base_hash == lm_only_hash + + +def test_language_model_only_affects_model_hash(): + """language_model_only affects the LM computation graph, + so it should change the model config hash.""" + model = "llava-hf/llava-1.5-7b-hf" + base_hash = ModelConfig(model).compute_hash() + lm_only_hash = ModelConfig(model, language_model_only=True).compute_hash() + assert base_hash != lm_only_hash + + +def test_mm_encoder_fp8_scale_path_requires_fp8(): + with pytest.raises(ValueError, match="mm_encoder_attn_dtype"): + MultiModalConfig(mm_encoder_fp8_scale_path="/tmp/scales.json") + + +def test_mm_encoder_attn_dtype_hash_updates(tmp_path): + scale_file = tmp_path / "scales.json" + scale_file.write_text("{}") + base_hash = MultiModalConfig().compute_hash() + fp8_hash = MultiModalConfig(mm_encoder_attn_dtype="fp8").compute_hash() + fp8_static_hash = MultiModalConfig( + mm_encoder_attn_dtype="fp8", + mm_encoder_fp8_scale_path=str(scale_file), + ).compute_hash() + assert base_hash != fp8_hash + assert fp8_hash != fp8_static_hash diff --git a/asian_ai/performance/vLLM/tests/conftest.py b/asian_ai/performance/vLLM/tests/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..7cd129e0c71bf5ee6b37f31da4965b48a4a84b4e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/conftest.py @@ -0,0 +1,1810 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import contextlib +import pathlib +from copy import deepcopy + +from tblib import pickling_support + +# ruff: noqa + +# Install support for pickling exceptions so that we can nicely propagate +# failures from tests running in a subprocess. +# This should be run before any custom exception subclasses are defined. +pickling_support.install() + +import http.server +import json +import math +import mimetypes +import os +import socket +import tempfile +import threading +from collections.abc import Generator +from contextlib import nullcontext +from enum import Enum +from typing import Any, Callable, TypedDict, TypeVar, cast, TYPE_CHECKING, Optional + +import numpy as np +import pytest +import torch +import torch.nn as nn +import torch.nn.functional as F +from huggingface_hub import snapshot_download +from PIL import Image +from transformers import ( + AutoConfig, + AutoModelForCausalLM, + AutoTokenizer, + BatchEncoding, + BatchFeature, +) +from transformers.models.auto.auto_factory import _BaseAutoModelClass + +from tests.models.utils import ( + TokensTextLogprobs, + TokensTextLogprobsPromptLogprobs, + softmax, +) +from vllm import LLM, SamplingParams, envs +from vllm.assets.audio import AudioAsset +from vllm.assets.image import ImageAsset +from vllm.assets.video import VideoAsset +from vllm.config.cache import CacheConfig +from vllm.config.model import ConvertOption, RunnerOption, _get_and_verify_dtype +from vllm.connections import global_http_connection +from vllm.distributed import ( + cleanup_dist_env_and_memory, + init_distributed_environment, + initialize_model_parallel, +) +from vllm.logger import init_logger +from vllm.logprobs import Logprob +from vllm.multimodal.media import MediaWithBytes +from vllm.multimodal.utils import fetch_image +from vllm.outputs import RequestOutput +from vllm.platforms import current_platform +from vllm.sampling_params import BeamSearchParams +from vllm.transformers_utils.utils import maybe_model_redirect +from vllm.utils.collection_utils import is_list_of +from vllm.utils.torch_utils import set_default_torch_num_threads + +from torch._inductor.utils import fresh_cache + + +if TYPE_CHECKING: + from transformers import PreTrainedTokenizer, PreTrainedTokenizerFast + from transformers.generation.utils import GenerateOutput + + +logger = init_logger(__name__) + + +@pytest.fixture +def sample_json_schema(): + return { + "type": "object", + "properties": { + "name": {"type": "string"}, + "age": {"type": "integer"}, + "skills": { + "type": "array", + "items": { + "type": "string", + }, + }, + "grade": { + "type": "string", + "pattern": "^[A-D]$", + }, + "email": { + "type": "string", + "pattern": "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$", + }, + "work_history": { + "type": "array", + "items": { + "type": "object", + "properties": { + "company": {"type": "string"}, + "duration": { + "type": "number", + "minimum": 0.0, + "maximum": 100.0, + }, + "position": {"type": "string"}, + }, + "required": ["company", "duration", "position"], + "additionalProperties": False, + }, + "minItems": 0, + "maxItems": 3, + }, + }, + "required": ["name", "age", "skills", "grade", "email", "work_history"], + "additionalProperties": False, + "minProperties": 1, + "maxProperties": 10, + } + + +_TEST_DIR = os.path.dirname(__file__) +_TEST_PROMPTS = [os.path.join(_TEST_DIR, "prompts", "example.txt")] +_LONG_PROMPTS = [os.path.join(_TEST_DIR, "prompts", "summary.txt")] +_SYS_MSG = os.path.join(_TEST_DIR, "system_messages", "sonnet3.5_nov2024.txt") + +_M = TypeVar("_M") + +_PromptMultiModalInput = list[_M] | list[list[_M]] + +PromptImageInput = _PromptMultiModalInput[Image.Image] +PromptAudioInput = _PromptMultiModalInput[tuple[np.ndarray, int]] +PromptVideoInput = _PromptMultiModalInput[np.ndarray] + + +def _read_prompts(filename: str) -> list[str]: + with open(filename) as f: + prompts = f.readlines() + return prompts + + +class ImageAssetPrompts(TypedDict): + stop_sign: str + cherry_blossom: str + + +class ImageTestAssets(list[ImageAsset]): + def __init__(self) -> None: + super().__init__( + [ + ImageAsset("stop_sign"), + ImageAsset("cherry_blossom"), + ] + ) + + def prompts(self, prompts: ImageAssetPrompts) -> list[str]: + """ + Convenience method to define the prompt for each test image. + + The order of the returned prompts matches the order of the + assets when iterating through this object. + """ + return [prompts["stop_sign"], prompts["cherry_blossom"]] + + +class VideoAssetPrompts(TypedDict): + baby_reading: str + + +class VideoTestAssets(list[VideoAsset]): + def __init__(self) -> None: + super().__init__( + [ + VideoAsset("baby_reading"), + ] + ) + + def prompts(self, prompts: VideoAssetPrompts) -> list[str]: + return [prompts["baby_reading"]] + + +class AudioAssetPrompts(TypedDict): + mary_had_lamb: str + winning_call: str + + +class AudioTestAssets(list[AudioAsset]): + def __init__(self) -> None: + super().__init__( + [ + AudioAsset("mary_had_lamb"), + AudioAsset("winning_call"), + ] + ) + + def prompts(self, prompts: AudioAssetPrompts) -> list[str]: + return [prompts["mary_had_lamb"], prompts["winning_call"]] + + +IMAGE_ASSETS = ImageTestAssets() +"""Singleton instance of {class}`ImageTestAssets`.""" +VIDEO_ASSETS = VideoTestAssets() +"""Singleton instance of {class}`VideoTestAssets`.""" +AUDIO_ASSETS = AudioTestAssets() +"""Singleton instance of {class}`AudioTestAssets`.""" + + +@pytest.fixture(autouse=True) +def init_test_http_connection(): + # pytest_asyncio may use a different event loop per test + # so we need to make sure the async client is created anew + global_http_connection.reuse_client = False + + +@pytest.fixture +def dist_init(): + from tests.utils import ensure_current_vllm_config + + # Close the fd returned by mkstemp; FileStore opens the path itself. + # Leaving it open leaks one FD per test and eventually exhausts the + # ulimit, causing FileStore's destructor to throw c10::DistStoreError + # ("Too many open files") during gc and abort the process. + fd, temp_file = tempfile.mkstemp() + os.close(fd) + + try: + with ensure_current_vllm_config(): + init_distributed_environment( + world_size=1, + rank=0, + distributed_init_method=f"file://{temp_file}", + local_rank=0, + backend="nccl", + ) + initialize_model_parallel(1, 1) + yield + cleanup_dist_env_and_memory() + finally: + with contextlib.suppress(OSError): + os.unlink(temp_file) + + +@pytest.fixture +def default_vllm_config(): + """Set a default VllmConfig for tests that directly test CustomOps or pathways + that use get_current_vllm_config() outside of a full engine context. + """ + from vllm.config import VllmConfig, set_current_vllm_config + + config = VllmConfig() + with set_current_vllm_config(config): + yield config + + +@pytest.fixture() +def should_do_global_cleanup_after_test(request) -> bool: + """Allow subdirectories to skip global cleanup by overriding this fixture. + This can provide a ~10x speedup for non-GPU unit tests since they don't need + to initialize torch. + """ + + return not request.node.get_closest_marker("skip_global_cleanup") + + +@pytest.fixture(autouse=True) +def cleanup_fixture(should_do_global_cleanup_after_test: bool): + yield + if should_do_global_cleanup_after_test: + cleanup_dist_env_and_memory() + + +@pytest.fixture +def workspace_init(): + """Initialize the workspace manager for tests that need it. + + This fixture initializes the workspace manager with a CUDA device + if available, and resets it after the test completes. Tests that + create a full vLLM engine should NOT use this fixture as the engine + will initialize the workspace manager itself. + """ + from vllm.v1.worker.workspace import ( + init_workspace_manager, + reset_workspace_manager, + ) + + if torch.cuda.is_available(): + device = torch.device("cuda:0") + init_workspace_manager(device) + yield + reset_workspace_manager() + + +@pytest.fixture(autouse=True) +def dynamo_reset(): + yield + torch._dynamo.reset() + + +@pytest.fixture +def example_prompts() -> list[str]: + return [prompt for filename in _TEST_PROMPTS for prompt in _read_prompts(filename)] + + +@pytest.fixture +def example_system_message() -> str: + with open(_SYS_MSG) as f: + return f.read() + + +class DecoderPromptType(Enum): + """For encoder/decoder models only.""" + + CUSTOM = 1 + NONE = 2 + EMPTY_STR = 3 + + +@pytest.fixture +def example_long_prompts() -> list[str]: + return [prompt for filename in _LONG_PROMPTS for prompt in _read_prompts(filename)] + + +@pytest.fixture(scope="session") +def image_assets() -> ImageTestAssets: + return IMAGE_ASSETS + + +@pytest.fixture(scope="session") +def video_assets() -> VideoTestAssets: + return VIDEO_ASSETS + + +@pytest.fixture(scope="session") +def audio_assets() -> AudioTestAssets: + return AUDIO_ASSETS + + +_T = TypeVar("_T", nn.Module, torch.Tensor, BatchEncoding, BatchFeature, dict) +_R = TypeVar("_R") + + +class HfRunner: + def get_default_device(self): + from vllm.platforms import current_platform + + return "cpu" if current_platform.is_cpu() else current_platform.device_type + + def wrap_device(self, x: _T, device: str | None = None) -> _T: + if x is None or isinstance(x, (bool,)): + return x + + if device is None: + device = self.device + + if isinstance(x, dict): + return {k: self.wrap_device(v, device) for k, v in x.items()} + + if hasattr(x, "device") and x.device.type == device: + return x + + return x.to(device) + + def __init__( + self, + model_name: str, + dtype: str = "auto", + *, + revision: str | None = None, + model_kwargs: dict[str, Any] | None = None, + trust_remote_code: bool = True, + is_sentence_transformer: bool = False, + is_cross_encoder: bool = False, + skip_tokenizer_init: bool = False, + auto_cls: type[_BaseAutoModelClass] = AutoModelForCausalLM, + tokenizer_name: str | None = None, + processor: Any | None = None, + # Set this to avoid hanging issue + default_torch_num_threads: int | None = None, + ) -> None: + init_ctx = ( + nullcontext() + if default_torch_num_threads is None + else set_default_torch_num_threads(default_torch_num_threads) + ) + + with init_ctx: + self._init( + model_name=model_name, + dtype=dtype, + revision=revision, + model_kwargs=model_kwargs, + trust_remote_code=trust_remote_code, + is_sentence_transformer=is_sentence_transformer, + is_cross_encoder=is_cross_encoder, + skip_tokenizer_init=skip_tokenizer_init, + auto_cls=auto_cls, + tokenizer_name=tokenizer_name, + processor=processor, + ) + + def _init( + self, + model_name: str, + dtype: str = "auto", + *, + revision: str | None = None, + model_kwargs: dict[str, Any] | None = None, + trust_remote_code: bool = True, + is_sentence_transformer: bool = False, + is_cross_encoder: bool = False, + skip_tokenizer_init: bool = False, + auto_cls: type[_BaseAutoModelClass] = AutoModelForCausalLM, + tokenizer_name: str | None = None, + processor: Any | None = None, + ) -> None: + model_name = maybe_model_redirect(model_name) + self.model_name = model_name + + self.config = AutoConfig.from_pretrained( + model_name, + trust_remote_code=trust_remote_code, + ) + # HF runner should use the HF config so that it's consistent with the HF model + if self.config.__module__.startswith("vllm.transformers_utils.configs"): + from transformers.models.auto.configuration_auto import CONFIG_MAPPING + + del CONFIG_MAPPING._extra_content[self.config.model_type] + self.config = AutoConfig.from_pretrained( + model_name, + trust_remote_code=trust_remote_code, + ) + self.device = self.get_default_device() + self.dtype = dtype = _get_and_verify_dtype( + self.model_name, + self.config, + dtype=dtype, + is_pooling_model=is_sentence_transformer or is_cross_encoder, + config_format="hf", + ) + + model_kwargs = model_kwargs if model_kwargs is not None else {} + model_kwargs.setdefault("dtype", dtype) + + if is_sentence_transformer: + # Lazy init required for AMD CI + from sentence_transformers import SentenceTransformer + + self.model = SentenceTransformer( + model_name, + revision=revision, + device=self.device, + model_kwargs=model_kwargs, + trust_remote_code=trust_remote_code, + ) + elif is_cross_encoder: + # Lazy init required for AMD CI + from sentence_transformers import CrossEncoder + + self.model = CrossEncoder( + model_name, + revision=revision, + device=self.device, + automodel_args=model_kwargs, + trust_remote_code=trust_remote_code, + ) + else: + model = cast( + nn.Module, + auto_cls.from_pretrained( + model_name, + revision=revision, + trust_remote_code=trust_remote_code, + **model_kwargs, + ), + ) + + # in case some unquantized custom models are not in same dtype + if getattr(model, "quantization_method", None) is None and any( + p.dtype != self.dtype for p in model.parameters() + ): + model = model.to(dtype=self.dtype) + + if ( + getattr(model, "quantization_method", None) != "bitsandbytes" + and len({p.device for p in model.parameters()}) < 2 + ): + model = model.to(device=self.device) + + self.model = model + + if not skip_tokenizer_init: + self.tokenizer: "PreTrainedTokenizer | PreTrainedTokenizerFast" = ( + AutoTokenizer.from_pretrained( + tokenizer_name or model_name, + trust_remote_code=trust_remote_code, + ) + ) + + if processor is not None: + self.processor = processor + else: + # don't put this import at the top level + # it will call torch.accelerator.device_count() + from transformers import AutoProcessor + + self.processor = AutoProcessor.from_pretrained( + model_name, + trust_remote_code=trust_remote_code, + ) + if skip_tokenizer_init: + if self.processor is None: + raise ValueError( + "skip_tokenizer_init=True requires processor initialization." + ) + self.tokenizer = self.processor.tokenizer + + def get_inputs( + self, + prompts: list[str] | list[list[int]], + images: PromptImageInput | None = None, + videos: PromptVideoInput | None = None, + audios: PromptAudioInput | None = None, + tokenization_kwargs: dict[str, Any] | None = None, + ) -> list[BatchFeature | BatchEncoding | dict[str, torch.Tensor]]: + if images is not None: + assert len(prompts) == len(images) + + if videos is not None: + assert len(prompts) == len(videos) + + if audios is not None: + assert len(prompts) == len(audios) + + all_inputs: list[BatchFeature | BatchEncoding | dict[str, torch.Tensor]] = [] + for i, prompt in enumerate(prompts): + if isinstance(prompt, str): + if self.processor is None: + raise RuntimeError( + "HfRunner.processor is not initialized. " + "Pass processor=... to HfRunner or set " + "hf_model.processor before generation." + ) + # Create a copy to avoid modifying the original dict + processor_kwargs = ( + tokenization_kwargs.copy() + if tokenization_kwargs is not None + else {} + ) + processor_kwargs.update( + { + "text": prompt, + "return_tensors": "pt", + } + ) + if images is not None and (image := images[i]) is not None: + processor_kwargs["images"] = image + if videos is not None and (video := videos[i]) is not None: + processor_kwargs["videos"] = video + if audios is not None and (audio_inputs := audios[i]) is not None: + # HACK - not all processors take sampling_rate; we should + # clean this up in the future. + if len(audio_inputs) == 2: + audio, sr = audio_inputs + processor_kwargs["audio"] = audio + processor_kwargs["sampling_rate"] = sr + else: + processor_kwargs["audio"] = audio_inputs + + inputs = self.processor(**processor_kwargs) + if isinstance(inputs, BatchFeature): + inputs = inputs.to(dtype=self.dtype) + all_inputs.append(inputs) + else: + # check that prompt is (batched) list of integers (token ids) + if not is_list_of(prompt, typ=int, check="all"): + raise ValueError( + "Prompt must be a list of ints corresponding to the prompt token ids." + ) + # check that no multimodal input is provided + if images or videos or audios: + raise ValueError( + "When providing prompt token ids multimodal inputs are not supported." + ) + input_dict = { + "input_ids": torch.tensor(prompt, dtype=torch.long).unsqueeze(0), + } + all_inputs.append(input_dict) + + return all_inputs + + def get_prompt_embeddings(self, prompts: list[str]) -> list[torch.Tensor]: + all_inputs = self.get_inputs(prompts) + embeddings = [] + for inputs in all_inputs: + input_ids = self.wrap_device(inputs)["input_ids"] + embedding = self.model.get_input_embeddings()(input_ids).squeeze(0) + embeddings.append(embedding) + return embeddings + + def classify(self, prompts: list[str]) -> list[list[float]]: + # output is final logits + all_inputs = self.get_inputs(prompts) + outputs: list[list[float]] = [] + problem_type = getattr(self.config, "problem_type", "") + + for inputs in all_inputs: + output = self.model(**self.wrap_device(inputs)) + + assert isinstance(output.logits, torch.Tensor) + + if problem_type == "regression": + logits = output.logits[0].tolist() + elif problem_type == "multi_label_classification": + logits = output.logits.sigmoid()[0].tolist() + else: + logits = softmax(output.logits)[0].tolist() + outputs.append(logits) + + return outputs + + def generate( + self, + prompts: list[str] | list[list[int]], + images: PromptImageInput | None = None, + videos: PromptVideoInput | None = None, + audios: PromptAudioInput | None = None, + **kwargs: Any, + ) -> list[tuple[list[list[int]], list[str]]]: + all_inputs = self.get_inputs( + prompts, images=images, videos=videos, audios=audios + ) + + outputs: list[tuple[list[list[int]], list[str]]] = [] + for inputs in all_inputs: + output_ids: torch.Tensor = self.model.generate( + **self.wrap_device(inputs), + use_cache=True, + **kwargs, + ) + if self.processor is None: + raise RuntimeError( + "HfRunner.processor is not initialized; cannot decode output." + ) + output_str = self.processor.batch_decode( + output_ids, + skip_special_tokens=True, + clean_up_tokenization_spaces=False, + ) + outputs.append((output_ids.cpu().tolist(), output_str)) + return outputs + + def generate_greedy( + self, + prompts: list[str] | list[list[int]], + max_tokens: int, + images: PromptImageInput | None = None, + videos: PromptVideoInput | None = None, + audios: PromptAudioInput | None = None, + **kwargs: Any, + ) -> list[tuple[list[int], str]]: + outputs = self.generate( + prompts, + do_sample=False, + max_new_tokens=max_tokens, + images=images, + videos=videos, + audios=audios, + **kwargs, + ) + + return [(output_ids[0], output_str[0]) for output_ids, output_str in outputs] + + def generate_beam_search( + self, + prompts: list[str], + beam_width: int, + max_tokens: int, + images: PromptImageInput | None = None, + videos: PromptVideoInput | None = None, + audios: PromptAudioInput | None = None, + ) -> list[tuple[list[list[int]], list[str]]]: + outputs = self.generate( + prompts, + do_sample=False, + max_new_tokens=max_tokens, + num_beams=beam_width, + num_return_sequences=beam_width, + images=images, + videos=videos, + audios=audios, + ) + + for i in range(len(outputs)): + output_ids, output_str = outputs[i] + for j in range(len(output_ids)): + output_ids[j] = [ + x for x in output_ids[j] if x != self.tokenizer.pad_token_id + ] + outputs[i] = (output_ids, output_str) + return outputs + + def generate_greedy_logprobs( + self, + prompts: list[str], + max_tokens: int, + images: PromptImageInput | None = None, + videos: PromptVideoInput | None = None, + audios: PromptAudioInput | None = None, + **kwargs: Any, + ) -> list[list[torch.Tensor]]: + all_inputs = self.get_inputs( + prompts, images=images, videos=videos, audios=audios + ) + + all_logprobs: list[list[torch.Tensor]] = [] + for inputs in all_inputs: + output: "GenerateOutput" = self.model.generate( + **self.wrap_device(inputs), + use_cache=True, + do_sample=False, + max_new_tokens=max_tokens, + output_hidden_states=True, + return_dict_in_generate=True, + **kwargs, + ) + seq_logprobs = self._hidden_states_to_seq_logprobs(output.hidden_states) + all_logprobs.append(seq_logprobs) + return all_logprobs + + def _hidden_states_to_seq_logprobs( + self, + hidden_states: tuple[tuple[torch.Tensor, ...], ...], + ) -> list[torch.Tensor]: + output_embeddings = self.model.get_output_embeddings() + + seq_logprobs: list[torch.Tensor] = [] + for _, hidden_state in enumerate(hidden_states): + last_hidden_states = hidden_state[-1][0] + logits = torch.matmul( + last_hidden_states.to( + device=output_embeddings.weight.device, + dtype=output_embeddings.weight.dtype, + ), + output_embeddings.weight.t(), + ) + if getattr(output_embeddings, "bias", None) is not None: + logits += output_embeddings.bias.unsqueeze(0) + logprobs = F.log_softmax(logits, dim=-1, dtype=torch.float32) + seq_logprobs.append(logprobs) + + return seq_logprobs + + def _hidden_states_to_logprobs( + self, + hidden_states: tuple[tuple[torch.Tensor, ...], ...], + num_logprobs: int | None, + ) -> tuple[list[dict[int, float]], int]: + seq_logprobs = self._hidden_states_to_seq_logprobs(hidden_states) + output_len = len(hidden_states) + + # convert to dict + seq_logprobs_lst: list[dict[int, float]] = [] + for tok_idx, tok_logprobs in enumerate(seq_logprobs): + # drop prompt logprobs + if tok_idx == 0: + tok_logprobs = tok_logprobs[-1, :].reshape(1, -1) + topk = tok_logprobs.topk(num_logprobs) + + tok_logprobs_dct = {} + for token_id, logprob in zip(topk.indices[0], topk.values[0]): + tok_logprobs_dct[token_id.item()] = logprob.item() + + seq_logprobs_lst.append(tok_logprobs_dct) + + return ( + seq_logprobs_lst, + output_len, + ) + + def generate_greedy_logprobs_limit( + self, + prompts: list[str], + max_tokens: int, + num_logprobs: int | None, + images: PromptImageInput | None = None, + audios: PromptAudioInput | None = None, + videos: PromptVideoInput | None = None, + use_cache: bool = True, + tokenization_kwargs: dict[str, Any] | None = None, + **kwargs: Any, + ) -> list[TokensTextLogprobs]: + all_inputs = self.get_inputs( + prompts, + images=images, + videos=videos, + audios=audios, + tokenization_kwargs=tokenization_kwargs, + ) + + all_logprobs: list[list[dict[int, float]]] = [] + all_output_ids: list[list[int]] = [] + all_output_strs: list[str] = [] + + for inputs in all_inputs: + output: "GenerateOutput" = self.model.generate( + **self.wrap_device(inputs), + use_cache=use_cache, + do_sample=False, + max_new_tokens=max_tokens, + output_hidden_states=True, + return_dict_in_generate=True, + **kwargs, + ) + + # Encoder-decoder models return decoder_hidden_states instead of + # hidden_states + hidden_states = ( + getattr(output, "hidden_states", None) or output.decoder_hidden_states + ) + + ( + seq_logprobs_lst, + output_len, + ) = self._hidden_states_to_logprobs(hidden_states, num_logprobs) + + all_logprobs.append(seq_logprobs_lst) + seq_ids = output.sequences[0] + output_len = len(seq_logprobs_lst) + output_ids = seq_ids[-output_len:] + all_output_ids.append(output_ids.tolist()) + all_output_strs.append(self.tokenizer.decode(output_ids)) + + outputs = zip(all_output_ids, all_output_strs, all_logprobs) + return [ + (output_ids, output_str, output_logprobs) + for output_ids, output_str, output_logprobs in outputs + ] + + def encode(self, prompts: list[str], *args, **kwargs) -> list[list[torch.Tensor]]: + return self.model.encode(prompts, *args, **kwargs) + + def predict(self, prompts: list[list[str]], *args, **kwargs) -> torch.Tensor: + return self.model.predict(prompts, *args, convert_to_tensor=True, **kwargs) + + def __enter__(self): + if current_platform.is_rocm(): + # Record starting memory usage stats on ROCm so that we can wait for + # memory to roughly settle back below these levels on shutdown. This is + # helpful in cases where the HfRunner is initialized after significant GPU + # memory is already occupied, e.g. in + # tests/basic_correctness/test_basic_correctness.py::test_models_distributed + from tests.utils import ( + get_physical_device_indices, + record_gpu_memory_usage_stats, + ) + + if (device_count := current_platform.device_count()) > 0: + devices = get_physical_device_indices(devices=list(range(device_count))) + mem_usage_stats = record_gpu_memory_usage_stats(devices=devices) + self.threshold_ratios = { + device: 0.05 + mem_used / mem_tot + for device, (mem_used, mem_tot) in mem_usage_stats.items() + } + return self + + def __exit__(self, exc_type, exc_value, traceback): + from tests.utils import wait_for_rocm_memory_to_settle + + del self.model + cleanup_dist_env_and_memory() + # ROCm frees VRAM lazily; wait so a runner started right after this HF + # model exits does not OOM on its startup memory guard. + wait_for_rocm_memory_to_settle( + threshold_ratio=getattr(self, "threshold_ratios", None) + ) + if hasattr(self, "threshold_ratios"): + del self.threshold_ratios + + +@pytest.fixture(scope="session") +def hf_runner(): + return HfRunner + + +class VllmRunner: + """ + The default value of some arguments have been modified from + {class}`~vllm.LLM` as follows: + + - `trust_remote_code`: Set to `True` instead of `False` for convenience. + - `seed`: Set to `0` instead of `None` for test reproducibility. + - `max_model_len`: Set to `1024` instead of `None` to reduce memory usage. + - `block_size`: To reduce memory usage, set default to `64` if on XPU + devices, otherwise default to `16`. + - `enable_chunked_prefill`: Set to `False` instead of `None` for + test reproducibility. + - `enforce_eager`: Set to `False` to test CUDA graph. + """ + + def __init__( + self, + model_name: str, + runner: RunnerOption = "auto", + convert: ConvertOption = "auto", + tokenizer_name: str | None = None, + tokenizer_mode: str = "auto", + trust_remote_code: bool = True, + seed: int = 0, + max_model_len: int | None = 1024, + dtype: str = "auto", + disable_log_stats: bool = True, + tensor_parallel_size: int = 1, + block_size: int = 16 if not torch.xpu.is_available() else 64, + enable_chunked_prefill: bool | None = False, + enforce_eager: bool | None = False, + # Set this to avoid hanging issue + default_torch_num_threads: int | None = None, + **kwargs, + ) -> None: + init_ctx = ( + nullcontext() + if default_torch_num_threads is None + else set_default_torch_num_threads(default_torch_num_threads) + ) + + if not kwargs.get("compilation_config", None): + # Note(@tdoublep): This is set to 4 because some tests (e.g., hybrid + # model tests) may set max_num_seqs=4. If min cudagraph_capture_size is + # set to larger than max_num_seqs, then it will lead to *no* graphs + # being captured which can trigger edge cases that we don't handle yet. + kwargs["compilation_config"] = {"cudagraph_capture_sizes": [4]} + + # Make sure we have atleast one cudagraph large enough for a single decode. + if (speculative_config := kwargs.get("speculative_config")) and ( + num_speculative_tokens := speculative_config["num_speculative_tokens"] + ): + kwargs["compilation_config"]["cudagraph_capture_sizes"].append( + num_speculative_tokens + 1 + ) + + from vllm.platforms import current_platform + + if current_platform.is_rocm(): + gpu_memory_utilization = kwargs.get( + "gpu_memory_utilization", + CacheConfig.gpu_memory_utilization, + ) + # V1 startup requires free_memory >= total * gpu_memory_utilization. + # ROCm CI can hand a test a device that is still lazily releasing + # VRAM from a previous process, so wait before constructing LLM. + from tests.utils import wait_for_rocm_memory_to_settle + + wait_for_rocm_memory_to_settle(threshold_ratio=1.0 - gpu_memory_utilization) + + with init_ctx: + self.llm = LLM( + model=model_name, + runner=runner, + convert=convert, + tokenizer=tokenizer_name, + tokenizer_mode=tokenizer_mode, + trust_remote_code=trust_remote_code, + dtype=dtype, + seed=seed, + enforce_eager=enforce_eager, + disable_log_stats=disable_log_stats, + tensor_parallel_size=tensor_parallel_size, + max_model_len=max_model_len, + block_size=block_size, + enable_chunked_prefill=enable_chunked_prefill, + **kwargs, + ) + + def get_inputs( + self, + prompts: list[str] + | list[torch.Tensor] + | list[list[int]] + | list[dict[str, Any]], + images: PromptImageInput | None = None, + videos: PromptVideoInput | None = None, + audios: PromptAudioInput | None = None, + ) -> list[dict[str, Any]]: + if any( + x is not None and len(x) != len(prompts) for x in [images, videos, audios] + ): + raise ValueError( + "All non-None multimodal inputs must have the same length as prompts" + ) + + inputs = list[dict[str, Any]]() + for i, prompt in enumerate(prompts): + # If we're passing an encoder/decoder prompt, we assume it + # already contains the multimodal data in the prompt + if isinstance(prompt, dict): + assert images is None and audios is None and videos is None + inputs.append(prompt.copy()) + else: + prompt_dict = dict[str, Any]() + if isinstance(prompt, str): + prompt_dict["prompt"] = prompt + elif isinstance(prompt, list): + prompt_dict["prompt_token_ids"] = prompt + else: + prompt_dict["prompt_embeds"] = prompt + + multi_modal_data = dict[str, Any]() + if images is not None and (image := images[i]) is not None: + multi_modal_data["image"] = image + if videos is not None and (video := videos[i]) is not None: + multi_modal_data["video"] = video + if audios is not None and (audio := audios[i]) is not None: + multi_modal_data["audio"] = audio + + if multi_modal_data: + prompt_dict["multi_modal_data"] = multi_modal_data + + inputs.append(prompt_dict) + + return inputs + + def generate( + self, + prompts: list[str] | list[torch.Tensor] | list[list[int]], + sampling_params: SamplingParams, + images: PromptImageInput | None = None, + videos: PromptVideoInput | None = None, + audios: PromptAudioInput | None = None, + return_logprobs: bool = False, + **kwargs: Any, + ) -> list[tuple[list[list[int]], list[str]]] | tuple[list, list]: + inputs = self.get_inputs(prompts, images=images, videos=videos, audios=audios) + + req_outputs = self.llm.generate( + inputs, sampling_params=sampling_params, **kwargs + ) + + outputs: list[tuple[list[list[int]], list[str]]] = [] + logprobs = [] + for req_output in req_outputs: + prompt_str = req_output.prompt + prompt_ids = req_output.prompt_token_ids + req_sample_output_ids: list[list[int]] = [] + req_sample_output_strs: list[str] = [] + req_logprobs = [] + if req_output.prompt_logprobs: + req_logprobs.extend(req_output.prompt_logprobs) + for sample in req_output.outputs: + output_str = sample.text + output_ids = list(sample.token_ids) + req_sample_output_ids.append(prompt_ids + output_ids) + req_sample_output_strs.append((prompt_str or "") + output_str) + if sample.logprobs: + req_logprobs.extend(sample.logprobs) + outputs.append((req_sample_output_ids, req_sample_output_strs)) + logprobs.append(req_logprobs) + return outputs if not return_logprobs else (outputs, logprobs) + + @staticmethod + def _final_steps_generate_w_logprobs( + req_outputs: list[RequestOutput], + include_prompt_token_ids: bool = False, + ) -> list[TokensTextLogprobsPromptLogprobs]: + outputs: list[TokensTextLogprobsPromptLogprobs] = [] + for req_output in req_outputs: + assert len(req_output.outputs) > 0 + for sample in req_output.outputs: + output_str = sample.text + output_ids = list(sample.token_ids) + output_logprobs = sample.logprobs + if include_prompt_token_ids: + outputs.append( + ( # type: ignore[arg-type] + output_ids, + output_str, + output_logprobs, + req_output.prompt_token_ids, + req_output.prompt_logprobs, + ) + ) + else: + outputs.append( + ( + output_ids, + output_str, + output_logprobs, + req_output.prompt_logprobs, + ) + ) + + return outputs + + def generate_w_logprobs( + self, + prompts: list[str], + sampling_params: SamplingParams, + images: PromptImageInput | None = None, + audios: PromptAudioInput | None = None, + videos: PromptVideoInput | None = None, + include_prompt_token_ids: bool = False, + **kwargs: Any, + ) -> list[TokensTextLogprobs] | list[TokensTextLogprobsPromptLogprobs]: + inputs = self.get_inputs(prompts, images=images, videos=videos, audios=audios) + + req_outputs = self.llm.generate( + inputs, sampling_params=sampling_params, **kwargs + ) + + toks_str_logsprobs_prompt_logprobs = self._final_steps_generate_w_logprobs( + req_outputs, include_prompt_token_ids + ) + # Omit prompt logprobs if not required by sampling params + return ( + [x[0:-1] for x in toks_str_logsprobs_prompt_logprobs] + if sampling_params.prompt_logprobs is None + else toks_str_logsprobs_prompt_logprobs + ) + + def generate_greedy( + self, + prompts: list[str] | list[torch.Tensor] | list[list[int]], + max_tokens: int, + images: PromptImageInput | None = None, + videos: PromptVideoInput | None = None, + audios: PromptAudioInput | None = None, + **kwargs: Any, + ) -> list[tuple[list[int], str]]: + greedy_params = SamplingParams(temperature=0.0, max_tokens=max_tokens) + outputs = self.generate( + prompts, + greedy_params, + images=images, + videos=videos, + audios=audios, + **kwargs, + ) + return [(output_ids[0], output_str[0]) for output_ids, output_str in outputs] + + def generate_greedy_logprobs( + self, + prompts: list[str], + max_tokens: int, + num_logprobs: int | None, + num_prompt_logprobs: int | None = None, + images: PromptImageInput | None = None, + audios: PromptAudioInput | None = None, + videos: PromptVideoInput | None = None, + stop_token_ids: list[int] | None = None, + stop: list[str] | None = None, + **kwargs: Any, + ) -> list[TokensTextLogprobs] | list[TokensTextLogprobsPromptLogprobs]: + greedy_logprobs_params = SamplingParams( + temperature=0.0, + max_tokens=max_tokens, + logprobs=num_logprobs, + prompt_logprobs=num_prompt_logprobs, + stop_token_ids=stop_token_ids, + stop=stop, + ) + + return self.generate_w_logprobs( + prompts, + greedy_logprobs_params, + images=images, + audios=audios, + videos=videos, + **kwargs, + ) + + def generate_prompt_perplexity( + self, prompts: list[str], mask: Optional[list[str]] = None + ) -> list[float]: + """ + Return the perplexity score associated with generating the prompts + + :param prompts: list of prompts to score + :return: perplexity score of each prompt + """ + outputs = self.generate_greedy_logprobs( + prompts, max_tokens=1, num_logprobs=None, num_prompt_logprobs=0 + ) + + mask_prefix_lens = ( + [len(self.llm.get_tokenizer()(prefix)["input_ids"]) for prefix in mask] + if mask is not None + else [0 for _ in range(len(prompts))] + ) + + perplexities = [] + for output, mask_prefix_len in zip(outputs, mask_prefix_lens): + output = cast(TokensTextLogprobsPromptLogprobs, output) + token_datas = cast(list[dict[int, Logprob] | None], output[3]) + assert token_datas[0] is None + + token_log_probs = [] + for token_data in token_datas[mask_prefix_len + 1 :]: + assert token_data is not None + assert len(token_data) == 1 + token_log_prob = list(token_data.values())[0].logprob + token_log_probs.append(token_log_prob) + + perplexity = math.exp(-sum(token_log_probs) / len(token_log_probs)) + perplexities.append(perplexity) + + return perplexities + + def generate_beam_search( + self, + prompts: list[str], + beam_width: int, + max_tokens: int, + images: PromptImageInput | None = None, + videos: PromptVideoInput | None = None, + audios: PromptAudioInput | None = None, + concurrency_limit: int | None = None, + ) -> list[tuple[list[list[int]], list[str]]]: + inputs = self.get_inputs(prompts, images=images, videos=videos, audios=audios) + + outputs = self.llm.beam_search( + inputs, + BeamSearchParams(beam_width=beam_width, max_tokens=max_tokens), + concurrency_limit=concurrency_limit, + ) + returned_outputs = [] + for output in outputs: + token_ids = [x.tokens for x in output.sequences] + texts = [x.text for x in output.sequences] + returned_outputs.append((token_ids, texts)) + return returned_outputs + + def classify(self, prompts: list[str]) -> list[list[float]]: + req_outputs = self.llm.classify(prompts) + return [req_output.outputs.probs for req_output in req_outputs] + + def embed( + self, + prompts: list[str], + images: PromptImageInput | None = None, + videos: PromptVideoInput | None = None, + audios: PromptAudioInput | None = None, + *args, + **kwargs, + ) -> list[list[float]]: + inputs = self.get_inputs(prompts, images=images, videos=videos, audios=audios) + + req_outputs = self.llm.embed(inputs, *args, **kwargs) + return [req_output.outputs.embedding for req_output in req_outputs] + + def token_embed(self, prompts: list[str]) -> list[list[float]]: + req_outputs = self.llm.encode(prompts, pooling_task="token_embed") + return [req_output.outputs.data for req_output in req_outputs] + + def token_classify(self, prompts: list[str]) -> list[list[float]]: + req_outputs = self.llm.encode(prompts, pooling_task="token_classify") + return [req_output.outputs.data for req_output in req_outputs] + + def score( + self, + text_1: list[str] | str, + text_2: list[str] | str, + *args, + **kwargs, + ) -> list[float]: + req_outputs = self.llm.score(text_1, text_2, *args, **kwargs) + return [req_output.outputs.score for req_output in req_outputs] + + def apply_model(self, func: Callable[[nn.Module], _R]) -> list[_R]: + return self.llm.apply_model(func) + + def get_llm(self) -> LLM: + return self.llm + + def collective_rpc(self, *args, **kwargs): + return self.llm.collective_rpc(*args, **kwargs) + + def __enter__(self): + return self + + def _wait_for_rocm_memory_release(self, gpu_memory_utilization: float) -> None: + from tests.utils import wait_for_rocm_memory_to_settle + + # V1 startup requires free_memory >= total * gpu_memory_utilization. + # Wait for the complementary used-memory ratio so the next runner does + # not fail the startup guard immediately after this runner exits. The + # wait is bounded so cleanup failures fail this test instead of hanging. + wait_for_rocm_memory_to_settle(threshold_ratio=1.0 - gpu_memory_utilization) + + def __exit__(self, exc_type, exc_value, traceback): + # Explicitly shutdown the engine core to release GPU resources + # This is needed because when executing consecutive tests, the GC + # might not be fast enough in shutting down the llm engine. This can lead to OOMs + # because when the next test starts some GPU memory is still in use. + gpu_memory_utilization = ( + self.llm.llm_engine.vllm_config.cache_config.gpu_memory_utilization + ) + from vllm.platforms import current_platform + + try: + # Give the engine core time to run its own graceful shutdown + # (model_executor teardown + empty_cache + process-group destroy) + # before the process manager SIGKILLs it at the default 5s. On ROCm + # a hard kill leaves the whole allocation for the driver's slow async + # VRAM reclamation, which starves the next test's startup. + shutdown_timeout = 60.0 if current_platform.is_rocm() else None + self.llm.llm_engine.engine_core.shutdown(timeout=shutdown_timeout) + except Exception: + # Ignore shutdown errors as cleanup will still proceed + pass + del self.llm + torch._dynamo.reset() + cleanup_dist_env_and_memory() + self._wait_for_rocm_memory_release(gpu_memory_utilization) + + +@pytest.fixture(scope="session") +def vllm_runner(): + return VllmRunner + + +@pytest.fixture() +def temporary_enable_log_propagate(): + import logging + + logger = logging.getLogger("vllm") + logger.propagate = True + yield + logger.propagate = False + + +@pytest.fixture() +def caplog_vllm(temporary_enable_log_propagate, caplog): + # To capture vllm log, we should enable propagate=True temporarily + # because caplog depends on logs propagated to the root logger. + yield caplog + + +@pytest.fixture() +def caplog_mp_fork(): + """ + This fixture enables capturing logs from a forked MP subprocess. + It should be used in conjunction with caplog_vllm. + + By default, subprocess logs do not go through the parent process. + We instead create a queue listener in the parent process which + forwards logs to the logger's other handlers, and add a QueueHandler + to the root logger. Forked subprocesses will inherit the root logger + and pass their messages to the queue, which the listener will forward + to the root logger, which can be captured by caplog. + + Note that this workaround only works for fork; with spawn, the subprocess + reinitializes logging and does not automatically inherit the queue. + We'd have to manually pass the queue to the subprocess at the spawn point. + See caplog_mp_spawn below. + """ + + @contextlib.contextmanager + def ctx(): + import logging.handlers + import multiprocessing as mp + + logger_queue: mp.Queue[logging.LogRecord] = mp.Queue() + logger = logging.getLogger() + handlers = logger.handlers + + # The listener works on a background thread, not inherited by the child. + queue_listener = logging.handlers.QueueListener(logger_queue, *handlers) + queue_listener.start() + + # Add queue handler after creating the listener to avoid cycle + logger.addHandler(logging.handlers.QueueHandler(logger_queue)) + yield + queue_listener.stop() + + return ctx + + +class LogHolder: + def __init__(self): + self.text = None + + +@pytest.fixture() +def caplog_mp_spawn(tmp_path, monkeypatch): + """ + This fixture enables capturing logs from a forked MP subprocess. + It does not require caplog_vllm (but it only contains logs from the child). + + By default, subprocess logs do not go through the parent process. + We instead add a FileHandler to the config so the spawned child process + writes its logs to a temp file. + In the parent, we read the file and return the contents. + + Note: this method could be extended to fork by either reconfiguring logging + in the parent or using a SocketHandler: + https://docs.python.org/3/howto/logging-cookbook.html#sending-and-receiving-logging-events-across-a-network # noqa: E501 + """ + + @contextlib.contextmanager + def ctx(level: int | str): + from vllm.logger import DEFAULT_LOGGING_CONFIG + + config_path = tmp_path / "vllm_logging_config.json" + log_path = tmp_path / "vllm.log" + log_holder = LogHolder() + + config = deepcopy(DEFAULT_LOGGING_CONFIG) + if envs.VLLM_LOGGING_CONFIG_PATH: + path = pathlib.Path(envs.VLLM_LOGGING_CONFIG_PATH) + assert path.exists() + config = json.loads(path.read_text()) + + config["loggers"]["vllm"]["handlers"] += ["vllm_file"] + config["handlers"]["vllm_file"] = { + "class": "logging.FileHandler", + "formatter": "vllm", + "level": level, + "filename": log_path.as_posix(), + } + config["loggers"]["vllm"]["level"] = level + + config_path.write_text(json.dumps(config)) + + with monkeypatch.context() as monkeypatch_ctx: + monkeypatch_ctx.setenv("VLLM_LOGGING_CONFIG_PATH", config_path.as_posix()) + monkeypatch_ctx.setenv("VLLM_CONFIGURE_LOGGING", "1") + yield log_holder + + log_holder.text = log_path.read_text() + + return ctx + + +@pytest.fixture(scope="session") +def num_gpus_available(): + """Get number of GPUs without initializing the CUDA context + in current process.""" + + from vllm.platforms import current_platform + + return current_platform.device_count() + + +temp_dir = tempfile.gettempdir() +_dummy_opt_path = os.path.join(temp_dir, "dummy_opt") +_dummy_llava_path = os.path.join(temp_dir, "dummy_llava") +_dummy_gemma2_embedding_path = os.path.join(temp_dir, "dummy_gemma2_embedding") + + +@pytest.fixture +def dummy_opt_path(): + json_path = os.path.join(_dummy_opt_path, "config.json") + if not os.path.exists(_dummy_opt_path): + snapshot_download( + repo_id="facebook/opt-125m", + local_dir=_dummy_opt_path, + ignore_patterns=["*.bin", "*.bin.index.json", "*.pt", "*.h5", "*.msgpack"], + ) + assert os.path.exists(json_path) + with open(json_path) as f: + config = json.load(f) + config["architectures"] = ["MyOPTForCausalLM"] + with open(json_path, "w") as f: + json.dump(config, f) + return _dummy_opt_path + + +@pytest.fixture +def dummy_llava_path(): + json_path = os.path.join(_dummy_llava_path, "config.json") + if not os.path.exists(_dummy_llava_path): + snapshot_download( + repo_id="llava-hf/llava-1.5-7b-hf", + local_dir=_dummy_llava_path, + ignore_patterns=[ + "*.bin", + "*.bin.index.json", + "*.pt", + "*.h5", + "*.msgpack", + "*.safetensors", + ], + ) + assert os.path.exists(json_path) + with open(json_path) as f: + config = json.load(f) + config["architectures"] = ["MyLlava"] + with open(json_path, "w") as f: + json.dump(config, f) + return _dummy_llava_path + + +@pytest.fixture +def dummy_gemma2_embedding_path(): + json_path = os.path.join(_dummy_gemma2_embedding_path, "config.json") + if not os.path.exists(_dummy_gemma2_embedding_path): + snapshot_download( + repo_id="BAAI/bge-multilingual-gemma2", + local_dir=_dummy_gemma2_embedding_path, + ignore_patterns=[ + "*.bin", + "*.bin.index.json", + "*.pt", + "*.h5", + "*.msgpack", + "*.safetensors", + ], + ) + assert os.path.exists(json_path) + with open(json_path) as f: + config = json.load(f) + config["architectures"] = ["MyGemma2Embedding"] + with open(json_path, "w") as f: + json.dump(config, f) + return _dummy_gemma2_embedding_path + + +# Add the flag `--optional` to allow run tests +# that are marked with @pytest.mark.optional +def pytest_addoption(parser): + parser.addoption( + "--optional", action="store_true", default=False, help="run optional test" + ) + + +def pytest_collection_modifyitems(config, items): + if config.getoption("--optional"): + # --optional given in cli: do not skip optional tests + return + skip_optional = pytest.mark.skip(reason="need --optional option to run") + for item in items: + if "optional" in item.keywords: + item.add_marker(skip_optional) + + +@pytest.fixture(scope="session") +def cli_config_file(): + """Return the path to the CLI config file.""" + return os.path.join(_TEST_DIR, "config", "test_config.yaml") + + +@pytest.fixture(scope="session") +def cli_config_file_with_model(): + """Return the path to the CLI config file with model.""" + return os.path.join(_TEST_DIR, "config", "test_config_with_model.yaml") + + +class AssetHandler(http.server.BaseHTTPRequestHandler): + # _IMAGE_CACHE : Dict[str, bytes] = {} + + def log_message(self, *args, **kwargs): + pass + + def do_GET(self): + # Accepts paths like: /1280px-Venn_diagram_rgb.jpg + filename = self.path.lstrip("/") + if not filename or "." not in filename: + self.send_error(404, "Missing filename (expected /.)") + return + + base, ext = filename.rsplit(".", 1) + ext = ext.lower() + + if ext not in ["jpg", "png"]: + self.send_error(404, f"Unsupported extension: .{ext}") + return + + try: + data = ImageAsset(base).read_bytes(ext=ext) + except Exception as e: + self.send_error(500, f"Failed to load asset: {ext} {base} {e} ") + return + + ctype, _ = mimetypes.guess_type(filename) + if ctype is None: + ctype = {"jpg": "image/jpg", "png": "image/png"}[ext] + self.send_response(200) + self.send_header("Content-Type", ctype) + self.send_header("Content-Length", str(len(data))) + self.end_headers() + try: + self.wfile.write(data) + except (BrokenPipeError, ConnectionResetError) as e: + logger.debug( + "Client disconnected while serving test asset %s: %r", filename, e + ) + self.close_connection = True + + +def _find_free_port() -> int: + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + return s.getsockname()[1] + + +class LocalAssetServer: + address: str + port: int + server: http.server.ThreadingHTTPServer | None + thread: threading.Thread | None + + def __init__(self, address: str = "127.0.0.1") -> None: + self.address = address + self.port = -1 + self.server = None + self.thread = None + + def __enter__(self): + self.port = _find_free_port() + self.server = http.server.ThreadingHTTPServer( + (self.address, self.port), AssetHandler + ) + self.thread = threading.Thread(target=self.server.serve_forever, daemon=True) + self.thread.start() + return self + + def __exit__(self, exc_type, exc_value, traceback): + if self.server: + self.server.shutdown() + del self.server + + if self.thread: + self.thread.join() + del self.thread + + if exc_type is None: + return None + + return False + + @property + def base_url(self) -> str: + assert self.port is not None + return f"http://{self.address}:{self.port}" + + def url_for(self, name: str) -> str: + """e.g., name='RGBA_comp.png' -> 'http://127.0.0.1:PORT/RGBA_comp.png'""" + return f"{self.base_url}/{name}" + + def get_image_asset(self, name: str) -> Image.Image: + image = fetch_image(self.url_for(name)) + # Unwrap MediaWithBytes if present + if isinstance(image, MediaWithBytes): + image = image.media + return image + + +@pytest.fixture(scope="session") +def local_asset_server() -> Generator[LocalAssetServer, None, None]: + """ + Starts a thread based HTTP server bound to 127.0.0.1 on a random free port. + The server currently servers images at: + http://127.0.0.1:/. + """ + with LocalAssetServer() as srv: + yield srv + + +@pytest.fixture +def image_url(request, local_asset_server) -> str: + # request.param is one of the IMAGE_ASSETS filenames + name = request.param + return local_asset_server.url_for(name) + + +@pytest.fixture +def image_urls(request, local_asset_server) -> list[str]: + """Indirect fixture: takes a list of names, returns list of full URLs.""" + names: list[str] = request.param + return [local_asset_server.url_for(name) for name in names] + + +@pytest.fixture +def disable_deepgemm_ue8m0(monkeypatch): + from vllm.utils.deep_gemm import is_deep_gemm_e8m0_used + + with monkeypatch.context() as monkeypatch_ctx: + monkeypatch_ctx.setenv("VLLM_USE_DEEP_GEMM_E8M0", "0") + is_deep_gemm_e8m0_used.cache_clear() + yield + # Clear cache so the next time it is used it is processed with the + # default VLLM_USE_DEEP_GEMM_E8M0 setting. + is_deep_gemm_e8m0_used.cache_clear() + + +@pytest.fixture(autouse=True) +def clean_gpu_memory_between_tests(): + if os.getenv("VLLM_TEST_CLEAN_GPU_MEMORY", "0") != "1": + yield + return + + # Wait for GPU memory to be cleared before starting the test + import gc + + from tests.utils import wait_for_gpu_memory_to_clear + + num_gpus = torch.accelerator.device_count() + if num_gpus > 0: + try: + wait_for_gpu_memory_to_clear( + devices=list(range(num_gpus)), + threshold_ratio=0.1, + ) + except ValueError as e: + logger.info("Failed to clean GPU memory: %s", e) + + yield + + # Clean up GPU memory after the test + if torch.cuda.is_available(): + torch.accelerator.empty_cache() + gc.collect() + + +@pytest.fixture +def use_fresh_inductor_cache(): + """ + Use a fresh inductor cache for the test. + This is useful to ensure that the test is not affected by the + previous test calls. + """ + with fresh_cache(): + yield + + +@pytest.fixture +def fresh_vllm_cache(monkeypatch, use_fresh_inductor_cache): + """Temporary VLLM_CACHE_ROOT combined with a fresh inductor cache.""" + with tempfile.TemporaryDirectory() as tmp_dir: + monkeypatch.setenv("VLLM_CACHE_ROOT", tmp_dir) + yield tmp_dir + + +@pytest.fixture(scope="function") +def enable_pickle(monkeypatch): + """`LLM.apply_model` requires pickling a function.""" + monkeypatch.setenv("VLLM_ALLOW_INSECURE_SERIALIZATION", "1") + + +@pytest.fixture(scope="function") +def disable_log_dedup(monkeypatch): + """ + Disable log deduplication such that warning_once and info_once always print. + """ + + # Patch logger._print_warning_once to remove the lru_cache decorator + from vllm import logger + + original_print_warning_once = logger._print_warning_once + original_print_info_once = logger._print_info_once + original_print_debug_once = logger._print_debug_once + + logger._print_warning_once = original_print_warning_once.__wrapped__ + logger._print_info_once = original_print_info_once.__wrapped__ + logger._print_debug_once = original_print_debug_once.__wrapped__ + + yield + logger._print_warning_once = original_print_warning_once + logger._print_info_once = original_print_info_once + logger._print_debug_once = original_print_debug_once + + +@pytest.fixture(scope="function") +def fake_vllm_ir(monkeypatch): + """ + Pytest fixture to allow isolated IR op registration in tests. + + Replaces IrOp.registry with an empty dict and swaps ``vllm_ir_torch_lib`` for a + fresh ``Library`` with a unique namespace per test (see ``Library.ns``). + + Torch keeps registrations for the process lifetime; reusing the fragment + name ``vllm_ir`` and defining the same op string again can segfault. A + random library name keeps each fixture run on a disjoint namespace. + + The test Library is kept alive until after monkeypatch teardown so PyTorch's + C++ state is not freed while references may still exist. + + Usage: + def test_my_ir_op(fake_vllm_ir): + @vllm.ir.register_op + def my_test_op(x: torch.Tensor) -> torch.Tensor: + return x * 2 + + result = my_test_op(torch.tensor([1, 2, 3])) + # Registry and library cleaned up automatically after the test + """ + import secrets + + from torch.library import Library + from vllm.ir.op import IrOp + + monkeypatch.setattr(IrOp, "registry", {}) + + # Keep a local reference so the Library is not GC'd before monkeypatch + # teardown restores the original reference. + test_lib = Library(f"vllm_ir_{secrets.token_hex(8)}", "FRAGMENT") + monkeypatch.setattr("vllm.ir.op.vllm_ir_torch_lib", test_lib) + + yield + + del test_lib diff --git a/asian_ai/performance/vLLM/tests/cuda/scripts/check_device_count_respects_env.py b/asian_ai/performance/vLLM/tests/cuda/scripts/check_device_count_respects_env.py new file mode 100644 index 0000000000000000000000000000000000000000..2b7578d3c3cc51baad773abfd5298285732ebe6b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/cuda/scripts/check_device_count_respects_env.py @@ -0,0 +1,23 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Check that device_count respects CUDA_VISIBLE_DEVICES after platform import.""" + +import os +import sys + +for key in ["CUDA_VISIBLE_DEVICES", "HIP_VISIBLE_DEVICES", "ROCR_VISIBLE_DEVICES"]: + os.environ.pop(key, None) + +import torch # noqa: E402 + +from vllm.platforms import current_platform # noqa: F401, E402 + +os.environ["CUDA_VISIBLE_DEVICES"] = "0" +count = torch.accelerator.device_count() + +if count == 0: + sys.exit(0) # Skip: no GPUs available + +assert count == 1, f"device_count()={count}, expected 1" +print("OK") diff --git a/asian_ai/performance/vLLM/tests/cuda/scripts/check_platform_no_cuda_init.py b/asian_ai/performance/vLLM/tests/cuda/scripts/check_platform_no_cuda_init.py new file mode 100644 index 0000000000000000000000000000000000000000..edccc80a0bcf77a136122ffc214705e51a547f12 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/cuda/scripts/check_platform_no_cuda_init.py @@ -0,0 +1,20 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Check that vllm.platforms import does not initialize CUDA.""" + +import os + +for key in ["CUDA_VISIBLE_DEVICES", "HIP_VISIBLE_DEVICES", "ROCR_VISIBLE_DEVICES"]: + os.environ.pop(key, None) + +import torch # noqa: E402 + +assert not torch.cuda.is_initialized(), "CUDA initialized before import" + +from vllm.platforms import current_platform # noqa: E402 + +assert not torch.cuda.is_initialized(), ( + f"CUDA was initialized during vllm.platforms import on {current_platform}" +) +print("OK") diff --git a/asian_ai/performance/vLLM/tests/cuda/test_cuda_compatibility_path.py b/asian_ai/performance/vLLM/tests/cuda/test_cuda_compatibility_path.py new file mode 100644 index 0000000000000000000000000000000000000000..083dfc4626b6a5e06514591827a73d35974d8180 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/cuda/test_cuda_compatibility_path.py @@ -0,0 +1,187 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for CUDA forward compatibility path logic in env_override.py. + +Verifies the opt-in LD_LIBRARY_PATH manipulation for CUDA compat libs, +including env var parsing, path detection, and deduplication. +""" + +import os +from unittest.mock import patch + +import pytest + +# Import the functions directly (they're module-level in env_override) +# We must import them without triggering the module-level side effects, +# so we import the functions by name after the module is already loaded. +from vllm.env_override import ( + _get_torch_cuda_version, + _maybe_set_cuda_compatibility_path, +) + + +class TestCudaCompatibilityEnvParsing: + """Test VLLM_ENABLE_CUDA_COMPATIBILITY env var parsing.""" + + def test_disabled_by_default(self, monkeypatch): + """Compat path is NOT set when env var is absent.""" + monkeypatch.delenv("VLLM_ENABLE_CUDA_COMPATIBILITY", raising=False) + monkeypatch.delenv("LD_LIBRARY_PATH", raising=False) + _maybe_set_cuda_compatibility_path() + assert ( + "LD_LIBRARY_PATH" not in os.environ + or os.environ.get("LD_LIBRARY_PATH", "") == "" + ) + + @pytest.mark.parametrize("value", ["0", "false", "False", "no", ""]) + def test_disabled_values(self, monkeypatch, value): + """Various falsy values should not activate compat path.""" + monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", value) + monkeypatch.delenv("LD_LIBRARY_PATH", raising=False) + _maybe_set_cuda_compatibility_path() + # LD_LIBRARY_PATH should not be set (or remain empty) + ld_path = os.environ.get("LD_LIBRARY_PATH", "") + assert "compat" not in ld_path + + @pytest.mark.parametrize("value", ["1", "true", "True", " 1 ", " TRUE "]) + def test_enabled_values_with_valid_path(self, monkeypatch, tmp_path, value): + """Truthy values activate compat path when a valid path exists.""" + compat_dir = tmp_path / "compat" + compat_dir.mkdir() + monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", value) + monkeypatch.setenv("VLLM_CUDA_COMPATIBILITY_PATH", str(compat_dir)) + monkeypatch.delenv("LD_LIBRARY_PATH", raising=False) + _maybe_set_cuda_compatibility_path() + ld_path = os.environ.get("LD_LIBRARY_PATH", "") + assert str(compat_dir) in ld_path + + +class TestCudaCompatibilityPathDetection: + """Test path detection: custom override, conda, default.""" + + def test_custom_path_override(self, monkeypatch, tmp_path): + """VLLM_CUDA_COMPATIBILITY_PATH takes highest priority.""" + custom_dir = tmp_path / "my-compat" + custom_dir.mkdir() + monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1") + monkeypatch.setenv("VLLM_CUDA_COMPATIBILITY_PATH", str(custom_dir)) + monkeypatch.delenv("LD_LIBRARY_PATH", raising=False) + _maybe_set_cuda_compatibility_path() + ld_path = os.environ.get("LD_LIBRARY_PATH", "") + assert ld_path.startswith(str(custom_dir)) + + def test_conda_prefix_fallback(self, monkeypatch, tmp_path): + """Falls back to $CONDA_PREFIX/cuda-compat if custom not set.""" + conda_dir = tmp_path / "conda-env" + compat_dir = conda_dir / "cuda-compat" + compat_dir.mkdir(parents=True) + monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1") + monkeypatch.delenv("VLLM_CUDA_COMPATIBILITY_PATH", raising=False) + monkeypatch.setenv("CONDA_PREFIX", str(conda_dir)) + monkeypatch.delenv("LD_LIBRARY_PATH", raising=False) + _maybe_set_cuda_compatibility_path() + ld_path = os.environ.get("LD_LIBRARY_PATH", "") + assert str(compat_dir) in ld_path + + def test_no_valid_path_does_nothing(self, monkeypatch): + """When enabled but no valid path exists, LD_LIBRARY_PATH unchanged.""" + monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1") + monkeypatch.setenv("VLLM_CUDA_COMPATIBILITY_PATH", "/nonexistent/path") + monkeypatch.delenv("CONDA_PREFIX", raising=False) + monkeypatch.delenv("LD_LIBRARY_PATH", raising=False) + with patch("vllm.env_override._get_torch_cuda_version", return_value=None): + _maybe_set_cuda_compatibility_path() + assert os.environ.get("LD_LIBRARY_PATH", "") == "" + + def test_default_cuda_path_fallback(self, monkeypatch, tmp_path): + """Falls back to /usr/local/cuda-{ver}/compat via torch version.""" + fake_cuda = tmp_path / "cuda-12.8" / "compat" + fake_cuda.mkdir(parents=True) + monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1") + monkeypatch.delenv("VLLM_CUDA_COMPATIBILITY_PATH", raising=False) + monkeypatch.delenv("CONDA_PREFIX", raising=False) + monkeypatch.delenv("LD_LIBRARY_PATH", raising=False) + with ( + patch("vllm.env_override._get_torch_cuda_version", return_value="12.8"), + patch( + "vllm.env_override.os.path.isdir", + side_effect=lambda p: p == "/usr/local/cuda-12.8/compat" + or os.path.isdir(p), + ), + ): + _maybe_set_cuda_compatibility_path() + ld_path = os.environ.get("LD_LIBRARY_PATH", "") + assert "/usr/local/cuda-12.8/compat" in ld_path + + +class TestCudaCompatibilityLdPathManipulation: + """Test LD_LIBRARY_PATH prepend and deduplication logic.""" + + def test_prepends_to_empty_ld_path(self, monkeypatch, tmp_path): + """Compat path is set when LD_LIBRARY_PATH is empty.""" + compat_dir = tmp_path / "compat" + compat_dir.mkdir() + monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1") + monkeypatch.setenv("VLLM_CUDA_COMPATIBILITY_PATH", str(compat_dir)) + monkeypatch.delenv("LD_LIBRARY_PATH", raising=False) + _maybe_set_cuda_compatibility_path() + assert os.environ["LD_LIBRARY_PATH"] == str(compat_dir) + + def test_prepends_to_existing_ld_path(self, monkeypatch, tmp_path): + """Compat path is prepended before existing entries.""" + compat_dir = tmp_path / "compat" + compat_dir.mkdir() + monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1") + monkeypatch.setenv("VLLM_CUDA_COMPATIBILITY_PATH", str(compat_dir)) + monkeypatch.setenv("LD_LIBRARY_PATH", "/usr/lib:/other/lib") + _maybe_set_cuda_compatibility_path() + ld_path = os.environ["LD_LIBRARY_PATH"] + parts = ld_path.split(os.pathsep) + assert parts[0] == str(compat_dir) + assert "/usr/lib" in parts + assert "/other/lib" in parts + + def test_deduplicates_existing_compat_path(self, monkeypatch, tmp_path): + """If compat path already in LD_LIBRARY_PATH, move to front.""" + compat_dir = tmp_path / "compat" + compat_dir.mkdir() + monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1") + monkeypatch.setenv("VLLM_CUDA_COMPATIBILITY_PATH", str(compat_dir)) + monkeypatch.setenv( + "LD_LIBRARY_PATH", + f"/usr/lib:{compat_dir}:/other/lib", + ) + _maybe_set_cuda_compatibility_path() + ld_path = os.environ["LD_LIBRARY_PATH"] + parts = ld_path.split(os.pathsep) + assert parts[0] == str(compat_dir) + assert parts.count(str(compat_dir)) == 1 + + def test_already_at_front_is_noop(self, monkeypatch, tmp_path): + """If compat path is already first, don't modify LD_LIBRARY_PATH.""" + compat_dir = tmp_path / "compat" + compat_dir.mkdir() + original = f"{compat_dir}:/usr/lib" + monkeypatch.setenv("VLLM_ENABLE_CUDA_COMPATIBILITY", "1") + monkeypatch.setenv("VLLM_CUDA_COMPATIBILITY_PATH", str(compat_dir)) + monkeypatch.setenv("LD_LIBRARY_PATH", original) + _maybe_set_cuda_compatibility_path() + assert os.environ["LD_LIBRARY_PATH"] == original + + +class TestGetTorchCudaVersion: + """Test _get_torch_cuda_version() helper.""" + + def test_returns_string_when_torch_available(self): + """Should return a CUDA version string like '12.8'.""" + version = _get_torch_cuda_version() + # torch is installed in vllm's environment + assert version is None or isinstance(version, str) + + def test_returns_none_when_torch_missing(self): + """Should return None when torch is not importable.""" + with patch( + "vllm.env_override.importlib.util.find_spec", + return_value=None, + ): + assert _get_torch_cuda_version() is None diff --git a/asian_ai/performance/vLLM/tests/cuda/test_cuda_context.py b/asian_ai/performance/vLLM/tests/cuda/test_cuda_context.py new file mode 100644 index 0000000000000000000000000000000000000000..3e9597a6188628694fe488800a758cb1ff9abae3 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/cuda/test_cuda_context.py @@ -0,0 +1,81 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import ctypes +from concurrent.futures import ThreadPoolExecutor + +import pytest +import torch + +from vllm.platforms import current_platform + + +def check_cuda_context(): + """Check CUDA driver context status""" + try: + cuda = ctypes.CDLL("libcuda.so") + device = ctypes.c_int() + result = cuda.cuCtxGetDevice(ctypes.byref(device)) + return (True, device.value) if result == 0 else (False, None) + except Exception: + return False, None + + +def run_cuda_test_in_thread(device_input, expected_device_id): + """Run CUDA context test in separate thread for isolation""" + try: + # New thread should have no CUDA context initially + valid_before, device_before = check_cuda_context() + if valid_before: + return ( + False, + "CUDA context should not exist in new thread, " + f"got device {device_before}", + ) + + # Test setting CUDA context + current_platform.set_device(device_input) + + # Verify context is created correctly + valid_after, device_id = check_cuda_context() + if not valid_after: + return False, "CUDA context should be valid after set_cuda_context" + if device_id != expected_device_id: + return False, f"Expected device {expected_device_id}, got {device_id}" + + return True, "Success" + except Exception as e: + return False, f"Exception in thread: {str(e)}" + + +class TestSetCudaContext: + """Test suite for the set_cuda_context function.""" + + @pytest.mark.skipif(not current_platform.is_cuda(), reason="CUDA not available") + @pytest.mark.parametrize( + argnames="device_input,expected_device_id", + argvalues=[ + (0, 0), + (torch.device("cuda:0"), 0), + ("cuda:0", 0), + ], + ids=["int", "torch_device", "string"], + ) + def test_set_cuda_context_parametrized(self, device_input, expected_device_id): + """Test setting CUDA context in isolated threads.""" + with ThreadPoolExecutor(max_workers=1) as executor: + future = executor.submit( + run_cuda_test_in_thread, device_input, expected_device_id + ) + success, message = future.result(timeout=30) + assert success, message + + @pytest.mark.skipif(not current_platform.is_cuda(), reason="CUDA not available") + def test_set_cuda_context_invalid_device_type(self): + """Test error handling for invalid device type.""" + with pytest.raises(ValueError, match="Expected a cuda device"): + current_platform.set_device(torch.device("cpu")) + + +if __name__ == "__main__": + pytest.main([__file__, "-v"]) diff --git a/asian_ai/performance/vLLM/tests/cuda/test_platform_no_cuda_init.py b/asian_ai/performance/vLLM/tests/cuda/test_platform_no_cuda_init.py new file mode 100644 index 0000000000000000000000000000000000000000..093e40571d21afd7665ff97b91b01a2d43f64d92 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/cuda/test_platform_no_cuda_init.py @@ -0,0 +1,48 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Test that platform imports do not prematurely initialize CUDA. + +This is critical for Ray-based multi-GPU setups where workers need to +set CUDA_VISIBLE_DEVICES after importing vLLM but before CUDA is initialized. +If CUDA is initialized during import, device_count() gets locked and ignores +subsequent env var changes. +""" + +import subprocess +import sys +from pathlib import Path + +import pytest + +SCRIPTS_DIR = Path(__file__).parent / "scripts" + + +def run_script(script_name: str) -> subprocess.CompletedProcess: + """Run a test script in a subprocess with clean CUDA state.""" + script_path = SCRIPTS_DIR / script_name + return subprocess.run( + [sys.executable, str(script_path)], + capture_output=True, + text=True, + ) + + +def test_platform_import_does_not_init_cuda(): + """Test that importing vllm.platforms does not initialize CUDA.""" + result = run_script("check_platform_no_cuda_init.py") + if result.returncode != 0: + pytest.fail(f"Platform import initialized CUDA:\n{result.stderr}") + + +def test_device_count_respects_env_after_platform_import(): + """Test that device_count respects CUDA_VISIBLE_DEVICES after import.""" + result = run_script("check_device_count_respects_env.py") + if result.returncode != 0: + pytest.fail( + f"device_count does not respect env var after import:\n{result.stderr}" + ) + + +if __name__ == "__main__": + pytest.main([__file__, "-v"]) diff --git a/asian_ai/performance/vLLM/tests/detokenizer/__init__.py b/asian_ai/performance/vLLM/tests/detokenizer/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/detokenizer/test_disable_detokenization.py b/asian_ai/performance/vLLM/tests/detokenizer/test_disable_detokenization.py new file mode 100644 index 0000000000000000000000000000000000000000..99459c805765e30a887ac4677637498a9d1992c1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/detokenizer/test_disable_detokenization.py @@ -0,0 +1,31 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from vllm.entrypoints.llm import LLM +from vllm.sampling_params import SamplingParams + + +@pytest.mark.parametrize("model", ["distilbert/distilgpt2"]) +def test_computed_prefix_blocks(model: str): + # This test checks if the engine generates completions both with and + # without optional detokenization, that detokenization includes text + # and no-detokenization doesn't, and that both completions have the same + # token_ids. + prompt = ( + "You are a helpful assistant. How do I build a car from cardboard and " + "paper clips? Is there an easy to follow video tutorial available " + "online for free?" + ) + + llm = LLM(model=model) + sampling_params = SamplingParams(max_tokens=10, temperature=0.0, detokenize=False) + + outputs_no_detokenization = llm.generate(prompt, sampling_params)[0].outputs[0] + sampling_params.detokenize = True + outputs_with_detokenization = llm.generate(prompt, sampling_params)[0].outputs[0] + + assert outputs_no_detokenization.text == "" + assert outputs_with_detokenization.text != "" + assert outputs_no_detokenization.token_ids == outputs_with_detokenization.token_ids diff --git a/asian_ai/performance/vLLM/tests/detokenizer/test_min_tokens.py b/asian_ai/performance/vLLM/tests/detokenizer/test_min_tokens.py new file mode 100644 index 0000000000000000000000000000000000000000..9134123d23c31609f5c7df561bf21afbd1dbdfbd --- /dev/null +++ b/asian_ai/performance/vLLM/tests/detokenizer/test_min_tokens.py @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +from transformers import AutoTokenizer + +from vllm import SamplingParams +from vllm.v1.engine import EngineCoreRequest +from vllm.v1.engine.detokenizer import FastIncrementalDetokenizer + +PROMPT = "Hello, my name is Lee, and I'm a student in the " + "college of engineering" + + +@pytest.mark.parametrize( + "min_tokens,stop,truth", + [ + (0, None, " is Lee, and I'm a student in the college of engineering"), + (0, "e", " is L"), + (5, "e", " is Lee, and I'm a stud"), + ], +) +def test_min_tokens_with_stop(min_tokens: int, stop: str, truth: str): + """Test for a specific min_tokens and stop. + + See https://github.com/vllm-project/vllm/pull/22014 + """ + tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m") + all_prompt_ids = tokenizer(PROMPT, add_special_tokens=False).input_ids + + # The prompt is "Hello, my name is" + prompt_token_ids = all_prompt_ids[:4] + params = SamplingParams( + stop=stop, + min_tokens=min_tokens, + ) + request = EngineCoreRequest( + request_id="", + prompt_token_ids=prompt_token_ids, + mm_features=None, + sampling_params=params, + pooling_params=None, + arrival_time=0.0, + lora_request=None, + cache_salt=None, + data_parallel_rank=None, + ) + + detokenizer = FastIncrementalDetokenizer(tokenizer, request) + + detokenizer.update(all_prompt_ids[4:], False) + assert detokenizer.output_text == truth diff --git a/asian_ai/performance/vLLM/tests/detokenizer/test_stop_reason.py b/asian_ai/performance/vLLM/tests/detokenizer/test_stop_reason.py new file mode 100644 index 0000000000000000000000000000000000000000..0dfebb7953372cdb969d9b7b020cdbf9fe082ad7 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/detokenizer/test_stop_reason.py @@ -0,0 +1,69 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Test the different finish_reason="stop" situations during generation: + 1. One of the provided stop strings + 2. One of the provided stop tokens + 3. The EOS token + +Run `pytest tests/engine/test_stop_reason.py`. +""" + +import pytest +import transformers + +from vllm import SamplingParams + +MODEL = "distilbert/distilgpt2" +STOP_STR = "." +SEED = 42 +MAX_TOKENS = 1024 + + +@pytest.fixture +def vllm_model(vllm_runner): + with vllm_runner(MODEL) as vllm_model: + yield vllm_model + + +def test_stop_reason(vllm_model, example_prompts): + tokenizer = transformers.AutoTokenizer.from_pretrained(MODEL) + stop_token_id = tokenizer.convert_tokens_to_ids(STOP_STR) + llm = vllm_model.llm + + # test stop token + outputs = llm.generate( + example_prompts, + sampling_params=SamplingParams( + ignore_eos=True, + seed=SEED, + max_tokens=MAX_TOKENS, + stop_token_ids=[stop_token_id], + ), + ) + for output in outputs: + output = output.outputs[0] + assert output.finish_reason == "stop" + assert output.stop_reason == stop_token_id + + # test stop string + outputs = llm.generate( + example_prompts, + sampling_params=SamplingParams( + ignore_eos=True, seed=SEED, max_tokens=MAX_TOKENS, stop="." + ), + ) + for output in outputs: + output = output.outputs[0] + assert output.finish_reason == "stop" + assert output.stop_reason == STOP_STR + + # test EOS token + outputs = llm.generate( + example_prompts, + sampling_params=SamplingParams(seed=SEED, max_tokens=MAX_TOKENS), + ) + for output in outputs: + output = output.outputs[0] + assert output.finish_reason == "length" or ( + output.finish_reason == "stop" and output.stop_reason is None + ) diff --git a/asian_ai/performance/vLLM/tests/detokenizer/test_stop_string_while_stop_model_terminates.py b/asian_ai/performance/vLLM/tests/detokenizer/test_stop_string_while_stop_model_terminates.py new file mode 100644 index 0000000000000000000000000000000000000000..34ac336dbb65361bbabdd9ce2cd5f809543447d8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/detokenizer/test_stop_string_while_stop_model_terminates.py @@ -0,0 +1,101 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from vllm.sampling_params import SamplingParams +from vllm.v1.engine import EngineCoreRequest +from vllm.v1.engine.detokenizer import BaseIncrementalDetokenizer + + +@pytest.fixture(params=[True, False]) +def include_stop_str_in_output(request): + return request.param + + +class _DummyDetokenizer(BaseIncrementalDetokenizer): + def __init__(self, request: EngineCoreRequest): + super().__init__(request) + + def decode_next(self, next_token_id: int) -> str: + # Map token id to single ASCII character for deterministic testing. + return chr(next_token_id) + + +def _make_request(stop, include_stop_str_in_output: bool, min_tokens: int = 0): + params = SamplingParams( + stop=stop, + include_stop_str_in_output=include_stop_str_in_output, + min_tokens=min_tokens, + ) + # Keep other fields minimal for unit test purposes. + req = EngineCoreRequest( + request_id="test", + prompt_token_ids=[], + mm_features=None, + sampling_params=params, + pooling_params=None, + arrival_time=0.0, + lora_request=None, + cache_salt=None, + data_parallel_rank=None, + ) + return req + + +def test_stop_string_while_stop_token_terminates(include_stop_str_in_output: bool): + """ + This test verifies that the detokenizer correctly handles the case where + the generated token sequence contains both: + - a stop token + - an token + + The detokenizer should respect the stop string and truncate the output + accordingly. + + Imagine the following sequence: + - "abcdeZ" is generated, where "Z" is the token. + - "cd" is the stop string. + + If include_stop_str_in_output=False, the detokenizer should truncate the + output to "ab" because the stop string "cd" is excluded. + If include_stop_str_in_output=True, the detokenizer should include the stop + string "cd" in the output, resulting in "abcd". + + + This verifies the behavioral change introduced in BaseIncrementalDetokenizer + where stop-string evaluation occurs before the early-return on + stop_terminated. + """ + + # Generate text "abcdeZ" and tokenize it. + generated_text = "abcde" + eos_token = "Z" + stop_string = "cd" + generated_text = generated_text + eos_token + token_ids = [ord(c) for c in generated_text] + + # Create a request with the stop string and initialize the detokenizer. + req = _make_request( + stop=[stop_string], include_stop_str_in_output=include_stop_str_in_output + ) + detok = _DummyDetokenizer(req) + + # Simulate that the last token ('Z') is a stop token (stop_terminated=True). + result = detok.update(new_token_ids=token_ids, stop_terminated=True) + + # The update should not report a stop string + assert result == stop_string + + # Output text should reflect stop-string handling: + # - include_stop_str_in_output=False => exclude "cd" => "ab" + # - include_stop_str_in_output=True => include "cd" => "abcd" + expected_text = "abcd" if include_stop_str_in_output else "ab" + assert detok.output_text == expected_text + + # The skipped final token should still be recorded in token_ids. + assert detok.output_token_ids == token_ids + + # get_next_output_text should return the full text when finished=True. + # (Buffering only applies during streaming when finished=False.) + assert detok.get_next_output_text(finished=True, delta=False) == expected_text diff --git a/asian_ai/performance/vLLM/tests/detokenizer/test_stop_strings.py b/asian_ai/performance/vLLM/tests/detokenizer/test_stop_strings.py new file mode 100644 index 0000000000000000000000000000000000000000..c71eba852f2274b7d0fb617e530aa7d497d4bb73 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/detokenizer/test_stop_strings.py @@ -0,0 +1,120 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from typing import Any + +import pytest + +from vllm import LLM, SamplingParams + +MODEL = "meta-llama/llama-2-7b-hf" +MAX_TOKENS = 200 + + +def _test_stopping( + llm: LLM, + expected_output: str, + expected_reason: Any, + stop: list[str] | None = None, + stop_token_ids: list[int] | None = None, + include_in_output: bool = False, +) -> None: + output = llm.generate( + "A story about vLLM:\n", + SamplingParams( + temperature=0.0, + max_tokens=MAX_TOKENS, + stop=stop, + stop_token_ids=stop_token_ids, + include_stop_str_in_output=include_in_output, + ), + )[0].outputs[0] + + assert output is not None + assert output.text == expected_output + assert output.stop_reason == expected_reason + + +def _stop_basic(llm): + _test_stopping( + llm, + stop=["."], + include_in_output=False, + expected_output="VLLM is a 100% volunteer organization", + expected_reason=".", + ) + + _test_stopping( + llm, + stop=["."], + include_in_output=True, + expected_output="VLLM is a 100% volunteer organization.", + expected_reason=".", + ) + + +def _stop_multi_tokens(llm): + _test_stopping( + llm, + stop=["group of peo", "short"], + include_in_output=False, + expected_output="VLLM is a 100% volunteer organization. We are a ", + expected_reason="group of peo", + ) + + _test_stopping( + llm, + stop=["group of peo", "short"], + include_in_output=True, + expected_output="VLLM is a 100% volunteer organization. We are a group of peo", + expected_reason="group of peo", + ) + + +def _stop_partial_token(llm): + _test_stopping( + llm, + stop=["gani"], + include_in_output=False, + expected_output="VLLM is a 100% volunteer or", + expected_reason="gani", + ) + + _test_stopping( + llm, + stop=["gani"], + include_in_output=True, + expected_output="VLLM is a 100% volunteer organi", + expected_reason="gani", + ) + + +def _stop_token_id(llm): + # token id 13013 => " organization" + + _test_stopping( + llm, + stop_token_ids=[13013], + include_in_output=False, + expected_output="VLLM is a 100% volunteer", + expected_reason=13013, + ) + + _test_stopping( + llm, + stop_token_ids=[13013], + include_in_output=True, + expected_output="VLLM is a 100% volunteer organization", + expected_reason=13013, + ) + + +@pytest.mark.skip_global_cleanup +def test_stop_strings(): + llm = LLM(MODEL, enforce_eager=True) + + _stop_basic(llm) + _stop_multi_tokens(llm) + _stop_partial_token(llm) + # FIXME: this does not respect include_in_output=False + # _stop_token_id(llm) diff --git a/asian_ai/performance/vLLM/tests/distributed/__init__.py b/asian_ai/performance/vLLM/tests/distributed/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/distributed/conftest.py b/asian_ai/performance/vLLM/tests/distributed/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..618e51e8cbad0620929e54b9c797a3dbf39e5ed6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/conftest.py @@ -0,0 +1,197 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import os +import random + +import msgspec +import msgspec.msgpack +import pytest +import zmq + +from vllm.config.kv_events import KVEventsConfig +from vllm.distributed.kv_events import EventPublisherFactory + +from .test_events import SampleBatch + +DP_RANK = 0 + + +@pytest.fixture +def random_port(): + """Generate a random port number for testing""" + return random.randint(10000, 59900) + + +@pytest.fixture +def publisher_config(random_port, request): + """Create a publisher config with inproc transport""" + how = request.param if hasattr(request, "param") else "inproc" + + if how == "inproc": + endpoint = f"inproc://test-{random_port}" + replay_endpoint = endpoint + "-replay" + else: + endpoint = f"tcp://*:{random_port}" + replay_endpoint = f"tcp://*:{random_port + 100}" + + return KVEventsConfig( + enable_kv_cache_events=True, + publisher="zmq", + endpoint=endpoint, + replay_endpoint=replay_endpoint, + buffer_steps=100, + hwm=1000, + topic="test", + ) + + +@pytest.fixture +def publisher(publisher_config): + """Create and return a publisher instance""" + pub = EventPublisherFactory.create(publisher_config, DP_RANK) + yield pub + pub.shutdown() + + +@pytest.fixture +def subscriber(publisher_config): + """Create and return a subscriber for testing""" + endpoint = publisher_config.endpoint + replay_endpoint = publisher_config.replay_endpoint + + if endpoint.startswith("tcp://*"): + endpoint = endpoint.replace("*", "127.0.0.1") + if replay_endpoint and replay_endpoint.startswith("tcp://*"): + replay_endpoint = replay_endpoint.replace("*", "127.0.0.1") + + sub = MockSubscriber( + [endpoint], + [replay_endpoint] if replay_endpoint else None, + publisher_config.topic, + ) + yield sub + sub.close() + + +class MockSubscriber: + """Helper class to receive and verify published events""" + + def __init__( + self, + pub_endpoints: str | list[str], + replay_endpoints: str | list[str] | None = None, + topic: str = "", + decode_type=SampleBatch, + ): + self.ctx = zmq.Context.instance() + + # Convert single endpoint to list for consistency + if isinstance(pub_endpoints, str): + pub_endpoints = [pub_endpoints] + if isinstance(replay_endpoints, str): + replay_endpoints = [replay_endpoints] + + # Set up subscriber socket - connect to all endpoints + self.sub = self.ctx.socket(zmq.SUB) + self.sub.setsockopt(zmq.SUBSCRIBE, topic.encode("utf-8")) + for endpoint in pub_endpoints: + self.sub.connect(endpoint) + + # Set up replay sockets if provided + self.replay_sockets = [] + if replay_endpoints: + for replay_endpoint in replay_endpoints: + replay = self.ctx.socket(zmq.REQ) + replay.connect(replay_endpoint) + self.replay_sockets.append(replay) + + self.topic = topic + self.topic_bytes = topic.encode("utf-8") + self.received_msgs: list[tuple[int, SampleBatch]] = [] + self.last_seq = -1 + self.decoder = msgspec.msgpack.Decoder(type=decode_type) + + def receive_one(self, timeout=1000) -> tuple[int, SampleBatch] | None: + """Receive a single message with timeout""" + if not self.sub.poll(timeout): + return None + + topic_bytes, seq_bytes, payload = self.sub.recv_multipart() + assert topic_bytes == self.topic_bytes + + seq = int.from_bytes(seq_bytes, "big") + data = self.decoder.decode(payload) + self.last_seq = seq + self.received_msgs.append((seq, data)) + return seq, data + + def request_replay(self, start_seq: int, socket_idx: int = 0) -> None: + """Request replay of messages starting from start_seq""" + if not self.replay_sockets: + raise ValueError("Replay sockets not initialized") + if socket_idx >= len(self.replay_sockets): + raise ValueError(f"Invalid socket index {socket_idx}") + + self.replay_sockets[socket_idx].send(start_seq.to_bytes(8, "big")) + + def receive_replay(self, socket_idx: int = 0) -> list[tuple[int, SampleBatch]]: + """Receive replayed messages from a specific replay socket""" + if not self.replay_sockets: + raise ValueError("Replay sockets not initialized") + if socket_idx >= len(self.replay_sockets): + raise ValueError(f"Invalid socket index {socket_idx}") + + replay_socket = self.replay_sockets[socket_idx] + replayed: list[tuple[int, SampleBatch]] = [] + while True: + try: + if not replay_socket.poll(1000): + break + + frames = replay_socket.recv_multipart() + if not frames or not frames[-1]: + # End of replay marker + break + + seq_bytes, payload = frames + seq = int.from_bytes(seq_bytes, "big") + data = self.decoder.decode(payload) + replayed.append((seq, data)) + except zmq.ZMQError as _: + break + + return replayed + + def close(self): + """Clean up resources""" + self.sub.close() + for replay in self.replay_sockets: + replay.close() + + +@pytest.fixture +def enable_ray_v2_backend(): + """Set env vars for the Ray V2 executor backend and shut down Ray + between tests.""" + import ray + + saved = { + "VLLM_USE_RAY_V2_EXECUTOR_BACKEND": os.environ.get( + "VLLM_USE_RAY_V2_EXECUTOR_BACKEND" + ), + "VLLM_ENABLE_V1_MULTIPROCESSING": os.environ.get( + "VLLM_ENABLE_V1_MULTIPROCESSING" + ), + } + os.environ["VLLM_USE_RAY_V2_EXECUTOR_BACKEND"] = "1" + os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0" + if ray.is_initialized(): + ray.shutdown() + try: + yield + finally: + if ray.is_initialized(): + ray.shutdown() + os.environ.update({k: v for k, v in saved.items() if v is not None}) + for key in (k for k, v in saved.items() if v is None): + os.environ.pop(key, None) diff --git a/asian_ai/performance/vLLM/tests/distributed/eplb_utils.py b/asian_ai/performance/vLLM/tests/distributed/eplb_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..2466f85f34b3e61bf4e24f6a94ebb3e4f0decb47 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/eplb_utils.py @@ -0,0 +1,91 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import atexit +import os +import random + +import pytest +import torch +import torch.multiprocessing as mp + +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.distributed.parallel_state import ( + init_distributed_environment, +) +from vllm.utils.system_utils import update_environment_variables + +mp.set_start_method("spawn", force=True) + + +def _distributed_worker_wrapper(fn, env, world_size, args, rank, skip_queue): + try: + fn(env, world_size, *args) + except BaseException as exc: + if isinstance(exc, pytest.skip.Exception): + skip_queue.put((rank, str(exc))) + return + raise + + +def distributed_run(fn, world_size, *args): + number_of_processes = world_size + processes: list[mp.Process] = [] + skip_queue: mp.SimpleQueue = mp.SimpleQueue() + for i in range(number_of_processes): + env: dict[str, str] = {} + env["RANK"] = str(i) + env["LOCAL_RANK"] = str(i) + env["WORLD_SIZE"] = str(number_of_processes) + env["LOCAL_WORLD_SIZE"] = str(number_of_processes) + env["MASTER_ADDR"] = "localhost" + env["MASTER_PORT"] = "12345" + p = mp.Process( + target=_distributed_worker_wrapper, + args=(fn, env, world_size, args, i, skip_queue), + ) + processes.append(p) + p.start() + + for p in processes: + p.join() + + skipped: list[tuple[int, str]] = [] + while not skip_queue.empty(): + rank, reason = skip_queue.get() + skipped.append((rank, reason)) + + if len(skipped) == number_of_processes: + reason = skipped[0][1] + pytest.skip(reason) + if 0 < len(skipped) < number_of_processes: + skipped_ranks = sorted(rank for rank, _ in skipped) + raise AssertionError( + "Distributed test had partial skips; expected either all ranks " + f"to skip or none. Skipped ranks: {skipped_ranks}, " + f"total ranks: {number_of_processes}" + ) + + for p in processes: + assert p.exitcode == 0 + + +def set_env_vars_and_device(env: dict[str, str]) -> None: + update_environment_variables(env) + local_rank = os.environ["LOCAL_RANK"] + device = torch.device(f"cuda:{local_rank}") + torch.accelerator.set_device_index(device) + + # Create a minimal vllm config for init_distributed_environment + vllm_config = VllmConfig() + with set_current_vllm_config(vllm_config): + init_distributed_environment() + atexit.register(_destroy_process_group_if_initialized) + # Ensure each worker process has the same random seed + random.seed(42) + torch.manual_seed(42) + + +def _destroy_process_group_if_initialized() -> None: + if torch.distributed.is_available() and torch.distributed.is_initialized(): + torch.distributed.destroy_process_group() diff --git a/asian_ai/performance/vLLM/tests/distributed/test_ca_buffer_sharing.py b/asian_ai/performance/vLLM/tests/distributed/test_ca_buffer_sharing.py new file mode 100644 index 0000000000000000000000000000000000000000..5206ef5b716a1cb1f09e671c7e6afcdc60781f56 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_ca_buffer_sharing.py @@ -0,0 +1,64 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +# can only run on machines with p2p access across GPUs +# can only run with torchrun: +# torchrun --nproc_per_node=2 tests/distributed/test_ca_buffer_sharing.py + +import ctypes + +import torch +import torch.distributed as dist + +from vllm.distributed.device_communicators.cuda_wrapper import CudaRTLibrary +from vllm.distributed.device_communicators.custom_all_reduce import ( # noqa + CustomAllreduce, +) + +# create a cpu process group for communicating metadata (ipc handle) +dist.init_process_group(backend="gloo") +rank = local_rank = dist.get_rank() +world_size = dist.get_world_size() + +# every process sets its own device (differently) +lib = CudaRTLibrary() +lib.cudaSetDevice(rank) + +buffer_size_in_bytes = 1024 +byte_value = 2 # the value we write to the buffer for verification + +pointers = CustomAllreduce.create_shared_buffer(buffer_size_in_bytes) + +print(f"Rank {rank} has pointers {pointers}") + +dist.barrier() +torch.accelerator.synchronize() + +if rank == 0: + # the first rank tries to write to all buffers + for p in pointers: + pointer = ctypes.c_void_p(p) + lib.cudaMemset(pointer, byte_value, buffer_size_in_bytes) + +dist.barrier() +torch.accelerator.synchronize() + +host_data = (ctypes.c_char * buffer_size_in_bytes)() + +# all ranks read from all buffers, and check if the data is correct +for p in pointers: + pointer = ctypes.c_void_p(p) + lib.cudaMemcpy(host_data, pointer, buffer_size_in_bytes) + for i in range(buffer_size_in_bytes): + assert ord(host_data[i]) == byte_value, ( + f"Rank {rank} failed" + f" to verify buffer {p}. Expected {byte_value}, " + f"got {ord(host_data[i])}" + ) + +print(f"Rank {rank} verified all buffers") + +dist.barrier() +torch.accelerator.synchronize() + +CustomAllreduce.free_shared_buffer(pointers) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_comm_ops.py b/asian_ai/performance/vLLM/tests/distributed/test_comm_ops.py new file mode 100644 index 0000000000000000000000000000000000000000..2217ddcc466ee3889d3cb3cd1859ec4f212bdebc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_comm_ops.py @@ -0,0 +1,382 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Test the communication operators. + +Run `pytest tests/distributed/test_comm_ops.py`. +""" + +from collections.abc import Callable +from typing import Any + +import pytest +import ray +import torch + +from vllm.distributed import ( + broadcast_tensor_dict, + get_pp_group, + tensor_model_parallel_all_gather, + tensor_model_parallel_all_reduce, + tensor_model_parallel_reduce_scatter, +) +from vllm.distributed.parallel_state import GroupCoordinator, TensorMetadata +from vllm.v1.worker.gpu_worker import AsyncIntermediateTensors + +from ..utils import ( + init_test_distributed_environment, + multi_gpu_test, + multi_process_parallel, +) + + +@ray.remote(num_gpus=1, max_calls=1) +def all_reduce_test_worker( + monkeypatch: pytest.MonkeyPatch, + tp_size: int, + pp_size: int, + rank: int, + distributed_init_port: str, +): + # it is important to delete the CUDA_VISIBLE_DEVICES environment variable + # so that each worker can see all the GPUs + # they will be able to set the device to the correct GPU + monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising=False) + + device = torch.device(f"cuda:{rank}") + torch.accelerator.set_device_index(device) + init_test_distributed_environment(tp_size, pp_size, rank, distributed_init_port) + num_elements = 8 + all_tensors = [ + torch.arange(num_elements, dtype=torch.float32, device="cuda") * (r + 1) + for r in range(tp_size) + ] + expected = torch.sum(torch.stack(all_tensors, dim=0), dim=0) + t = all_tensors[rank % tp_size] + t = tensor_model_parallel_all_reduce(t) + torch.testing.assert_close(t, expected) + + +@ray.remote(num_gpus=1, max_calls=1) +def reduce_scatter_test_worker( + monkeypatch: pytest.MonkeyPatch, + tp_size: int, + pp_size: int, + rank: int, + distributed_init_port: str, +): + # it is important to delete the CUDA_VISIBLE_DEVICES environment variable + # so that each worker can see all the GPUs + # they will be able to set the device to the correct GPU + monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising=False) + device = torch.device(f"cuda:{rank}") + torch.accelerator.set_device_index(device) + init_test_distributed_environment(tp_size, pp_size, rank, distributed_init_port) + + num_elements = 8 + all_tensors = [ + torch.arange(num_elements, dtype=torch.float32, device="cuda") * (r + 1) + for r in range(tp_size) + ] + + index = rank % tp_size + partition_size = num_elements // tp_size + all_reduce = torch.sum(torch.stack(all_tensors, dim=0), dim=0) + expected = all_reduce[index * partition_size : (index + 1) * partition_size] + t = all_tensors[index] + t = tensor_model_parallel_reduce_scatter(t, 0) + torch.testing.assert_close(t, expected) + + +@ray.remote(num_gpus=1, max_calls=1) +def all_gather_test_worker( + monkeypatch: pytest.MonkeyPatch, + tp_size: int, + pp_size: int, + rank: int, + distributed_init_port: str, +): + # it is important to delete the CUDA_VISIBLE_DEVICES environment variable + # so that each worker can see all the GPUs + # they will be able to set the device to the correct GPU + monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising=False) + device = torch.device(f"cuda:{rank}") + torch.accelerator.set_device_index(device) + init_test_distributed_environment(tp_size, pp_size, rank, distributed_init_port) + num_dimensions = 3 + tensor_size = list(range(2, num_dimensions + 2)) + total_size = 1 + for s in tensor_size: + total_size *= s + for all_gather_dimension in range(num_dimensions): + all_tensors = [ + torch.arange(total_size, dtype=torch.float32, device="cuda").reshape( + tensor_size + ) + * (r + 1) + for r in range(tp_size) + ] + expected = torch.cat(all_tensors, dim=all_gather_dimension) + t = all_tensors[rank % tp_size] + t = tensor_model_parallel_all_gather(t, all_gather_dimension) + torch.testing.assert_close(t, expected) + + +@ray.remote(num_gpus=1, max_calls=1) +def broadcast_tensor_dict_test_worker( + monkeypatch: pytest.MonkeyPatch, + tp_size: int, + pp_size: int, + rank: int, + distributed_init_port: str, +): + # it is important to delete the CUDA_VISIBLE_DEVICES environment variable + # so that each worker can see all the GPUs + # they will be able to set the device to the correct GPU + monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising=False) + device = torch.device(f"cuda:{rank}") + torch.accelerator.set_device_index(device) + init_test_distributed_environment(tp_size, pp_size, rank, distributed_init_port) + test_dict = { + # device tensor + "a": torch.arange(8, dtype=torch.float32, device="cuda"), + # CPU tensor + "b": torch.arange(16, dtype=torch.int8, device="cpu"), + "c": "test", + "d": [1, 2, 3], + "e": {"a": 1, "b": 2}, + # empty tensor + "f": torch.tensor([], dtype=torch.float32, device="cuda"), + } + + if (rank % tp_size) == 0: + broadcast_tensor_dict(test_dict, src=0) + else: + recv_dict = broadcast_tensor_dict(src=0) + assert len(recv_dict) == len(test_dict) + torch.testing.assert_close(recv_dict["a"], test_dict["a"]) + torch.testing.assert_close(recv_dict["b"], test_dict["b"]) + assert recv_dict["c"] == test_dict["c"] + assert recv_dict["d"] == test_dict["d"] + assert recv_dict["e"] == test_dict["e"] + torch.testing.assert_close(recv_dict["f"], test_dict["f"]) + + +@ray.remote(num_gpus=1, max_calls=1) +def send_recv_tensor_dict_test_worker( + monkeypatch: pytest.MonkeyPatch, + tp_size: int, + pp_size: int, + rank: int, + distributed_init_port: str, +): + monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising=False) + device = torch.device(f"cuda:{rank}") + torch.accelerator.set_device_index(device) + init_test_distributed_environment(tp_size, pp_size, rank, distributed_init_port) + + test_dict = { + # device tensor + "a": torch.arange(8, dtype=torch.float32, device="cuda"), + # CPU tensor + "b": torch.arange(16, dtype=torch.int8, device="cpu"), + "c": "test", + "d": [1, 2, 3], + "e": {"a": 1, "b": 2}, + # empty tensor + "f": torch.tensor([], dtype=torch.float32, device="cuda"), + } + + if not get_pp_group().is_first_rank: + recv_dict = get_pp_group().recv_tensor_dict() + + if not get_pp_group().is_last_rank: + get_pp_group().send_tensor_dict(test_dict) + + if not get_pp_group().is_first_rank: + assert len(recv_dict) == len(test_dict) + torch.testing.assert_close(recv_dict["a"], test_dict["a"]) + torch.testing.assert_close(recv_dict["b"], test_dict["b"]) + assert recv_dict["c"] == test_dict["c"] + assert recv_dict["d"] == test_dict["d"] + assert recv_dict["e"] == test_dict["e"] + torch.testing.assert_close(recv_dict["f"], test_dict["f"]) + + +class _DummyWork: + def __init__(self) -> None: + self.wait_calls = 0 + + def wait(self) -> None: + self.wait_calls += 1 + + +class _DummyAllGatherGroup: + def __init__(self, world_size: int, rank_in_group: int) -> None: + self.world_size = world_size + self.rank_in_group = rank_in_group + + def all_gather(self, t: torch.Tensor, dim: int = 0) -> torch.Tensor: + # duplicate local slice across ranks. + assert dim == 0 + return torch.cat([t for _ in range(self.world_size)], dim=0) + + +def _make_group_for_unit_test( + rank_in_group: int = 0, world_size: int = 2 +) -> GroupCoordinator: + # avoid running GroupCoordinator.__init__ (it wires up real process groups). + g = GroupCoordinator.__new__(GroupCoordinator) + g.world_size = world_size + g.rank_in_group = rank_in_group + g.ranks = list(range(world_size)) + g.use_cpu_custom_send_recv = False + g.device_group = None + g.cpu_group = None + return g + + +def test_irecv_tensor_dict_send_allgather_postprocess_binds_keys( + monkeypatch: pytest.MonkeyPatch, +) -> None: + def fake_irecv(t: torch.Tensor, *args: Any, **kwargs: Any) -> _DummyWork: + t.fill_(1) + return _DummyWork() + + monkeypatch.setattr(torch.distributed, "is_initialized", lambda: True) + monkeypatch.setattr(torch.distributed, "irecv", fake_irecv) + + g = _make_group_for_unit_test(rank_in_group=0, world_size=2) + # 2 tensors so we can catch late-binding bugs in postprocess closures. + metadata_list = [ + ("a", TensorMetadata("cpu", torch.int32, torch.Size([4]))), + ("b", TensorMetadata("cpu", torch.int32, torch.Size([4]))), + ] + g.recv_object = lambda src=None: metadata_list # type: ignore[method-assign] + + ag = _DummyAllGatherGroup(world_size=2, rank_in_group=0) + td, handles, postprocess = g.irecv_tensor_dict(all_gather_group=ag) + + assert td is not None + assert len(handles) == 2 + assert len(postprocess) == 2 + + # before postprocess, dict holds the TP slice (shape 2). + assert td["a"].shape == torch.Size([2]) + assert td["b"].shape == torch.Size([2]) + + # simulate worker-side "defer wait": wait + postprocess later. + for handle in handles: + handle.wait() + for fn in postprocess: + fn() + + # after postprocess, dict values are reconstructed to full shape (shape 4), + # and each key should be updated independently + assert td["a"].shape == torch.Size([4]) + assert td["b"].shape == torch.Size([4]) + torch.testing.assert_close(td["a"], torch.ones(4, dtype=torch.int32)) + torch.testing.assert_close(td["b"], torch.ones(4, dtype=torch.int32)) + + +def test_async_intermediate_tensors_lazy_wait() -> None: + work = _DummyWork() + post_calls = {"n": 0} + + def post() -> None: + post_calls["n"] += 1 + + it = AsyncIntermediateTensors( + {"x": torch.tensor([1])}, + comm_handles=[work], + comm_postprocess=[post], + ) + + # accessing non-tensor attributes should not trigger wait. + assert it._comm_handles is not None + assert work.wait_calls == 0 + assert post_calls["n"] == 0 + + # first access of `.tensors` triggers wait + postprocess. + _ = it.tensors + assert work.wait_calls == 1 + assert post_calls["n"] == 1 + + # subsequent access should not re-wait. + _ = it.tensors + assert work.wait_calls == 1 + assert post_calls["n"] == 1 + + +@ray.remote(num_gpus=1, max_calls=1) +def send_recv_test_worker( + monkeypatch: pytest.MonkeyPatch, + tp_size: int, + pp_size: int, + rank: int, + distributed_init_port: str, +): + monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising=False) + device = torch.device(f"cuda:{rank}") + torch.accelerator.set_device_index(device) + init_test_distributed_environment(tp_size, pp_size, rank, distributed_init_port) + + size = 64 + test_tensor = torch.arange(64, dtype=torch.float32, device="cuda") + + if not get_pp_group().is_first_rank: + recv_tensor = get_pp_group().recv(size, dtype=torch.float32) + + if not get_pp_group().is_last_rank: + get_pp_group().send(test_tensor) + + if not get_pp_group().is_first_rank: + torch.testing.assert_close(test_tensor, recv_tensor) + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize("tp_size", [2]) +@pytest.mark.parametrize( + "test_target", + [all_reduce_test_worker, all_gather_test_worker, broadcast_tensor_dict_test_worker], +) +def test_multi_process_tensor_parallel( + monkeypatch: pytest.MonkeyPatch, + tp_size: int, + test_target: Callable[..., Any], +): + multi_process_parallel(monkeypatch, tp_size, 1, test_target) + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize("pp_size", [2]) +@pytest.mark.parametrize( + "test_target", [send_recv_test_worker, send_recv_tensor_dict_test_worker] +) +def test_multi_process_pipeline_parallel( + monkeypatch: pytest.MonkeyPatch, + pp_size: int, + test_target: Callable[..., Any], +): + multi_process_parallel(monkeypatch, 1, pp_size, test_target) + + +@multi_gpu_test(num_gpus=4) +@pytest.mark.parametrize("tp_size", [2]) +@pytest.mark.parametrize("pp_size", [2]) +@pytest.mark.parametrize( + "test_target", + [ + send_recv_test_worker, + send_recv_tensor_dict_test_worker, + all_reduce_test_worker, + all_gather_test_worker, + broadcast_tensor_dict_test_worker, + ], +) +def test_multi_process_tensor_parallel_pipeline_parallel( + tp_size: int, + pp_size: int, + test_target: Callable[..., Any], + monkeypatch: pytest.MonkeyPatch, +): + multi_process_parallel(monkeypatch, tp_size, pp_size, test_target) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_context_parallel.py b/asian_ai/performance/vLLM/tests/distributed/test_context_parallel.py new file mode 100644 index 0000000000000000000000000000000000000000..eaa5223d787b21912f105c85a84fa88797f2b25f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_context_parallel.py @@ -0,0 +1,311 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +WARNING: This test runs in both single-node (4 GPUs) and multi-node + (2 node with 2 GPUs each) modes. If the test only uses 2 GPUs, it is + important to set the distributed backend to "mp" to avoid Ray scheduling + all workers in a node other than the head node, which can cause the test + to fail. +""" + +import json +import os +from dataclasses import dataclass +from typing import Literal, NamedTuple + +import lm_eval +import pytest +import torch + +from tests.utils import RemoteOpenAIServer, create_new_process_for_each_test +from vllm.config.model import RunnerOption +from vllm.logger import init_logger +from vllm.platforms import current_platform + +from ..models.registry import HF_EXAMPLE_MODELS + +logger = init_logger("test_context_parallel") + +VLLM_MULTI_NODE = os.getenv("VLLM_MULTI_NODE", "0") == "1" + +CP_TEST_MODELS = [ + # TODO support other models + # [LANGUAGE GENERATION] + "deepseek-ai/DeepSeek-V2-Lite-Chat", + "Qwen/Qwen2.5-1.5B-Instruct", +] + +# GSM8K eval configuration +NUM_SHOTS = 5 # Few-shot examples +TASK = "gsm8k" +FILTER = "exact_match,strict-match" +NUM_CONCURRENT = 128 +# tp accuracy with 2% buffer +MIN_ACCURACY = { + # .buildkite/lm-eval-harness/configs/DeepSeek-V2-Lite-Chat.yaml + "deepseek-ai/DeepSeek-V2-Lite-Chat": 0.64, + # .buildkite/lm-eval-harness/configs/Qwen2.5-1.5B-Instruct.yaml + "Qwen/Qwen2.5-1.5B-Instruct": 0.52, +} + + +class ParallelSetup(NamedTuple): + tp_size: int + pp_size: int + dcp_size: int + cp_kv_cache_interleave_size: int + eager_mode: bool + chunked_prefill: bool + + +class CPTestOptions(NamedTuple): + multi_node_only: bool + attn_backend: str | None = None + + +@dataclass +class CPTestSettings: + parallel_setups: list[ParallelSetup] + distributed_backends: list[str] + runner: RunnerOption + test_options: CPTestOptions + + @staticmethod + def detailed( + *, + tp_base: int = 4, + pp_base: int = 1, + dcp_multipliers: list[float] | None = None, + cp_kv_cache_interleave_size: int = 1, + multi_node_only: bool = False, + runner: RunnerOption = "auto", + attn_backend: str | None = None, + ): + parallel_setups = [] + if dcp_multipliers is None: + dcp_multipliers = [ + 0.5, + ] + for eager_mode_val in [False]: + for pp_multiplier in [1]: + for dcp_multiplier in dcp_multipliers: + for chunked_prefill_val in [True]: + parallel_setups.append( + ParallelSetup( + tp_size=tp_base, + pp_size=pp_multiplier * pp_base, + dcp_size=int(dcp_multiplier * tp_base), + cp_kv_cache_interleave_size=cp_kv_cache_interleave_size, + eager_mode=eager_mode_val, + chunked_prefill=chunked_prefill_val, + ) + ) + return CPTestSettings( + parallel_setups=parallel_setups, + distributed_backends=["mp"], + runner=runner, + test_options=CPTestOptions( + multi_node_only=multi_node_only, + attn_backend=attn_backend, + ), + ) + + def iter_params(self, model_id: str): + opts = self.test_options + + for parallel_setup in self.parallel_setups: + for backend in self.distributed_backends: + yield ( + model_id, + parallel_setup, + backend, + self.runner, + opts, + ) + + +if current_platform.is_rocm(): + CP_TEXT_GENERATION_MODELS = { + "deepseek-ai/DeepSeek-V2-Lite-Chat": [ + CPTestSettings.detailed(dcp_multipliers=[1]), + ], + "Qwen/Qwen2.5-1.5B-Instruct": [ + CPTestSettings.detailed(dcp_multipliers=[1]), + ], + } +else: + CP_TEXT_GENERATION_MODELS = { + "deepseek-ai/DeepSeek-V2-Lite-Chat": [ + CPTestSettings.detailed(dcp_multipliers=[1]), + CPTestSettings.detailed( + dcp_multipliers=[0.5], + cp_kv_cache_interleave_size=64, + attn_backend="FLASHMLA", + ), + ], + "Qwen/Qwen2.5-1.5B-Instruct": [ + CPTestSettings.detailed( + cp_kv_cache_interleave_size=16, attn_backend="FLASH_ATTN" + ), + CPTestSettings.detailed( + cp_kv_cache_interleave_size=16, attn_backend="FLASHINFER" + ), + ], + } + + +def _test_cp_gsm8k( + model_id: str, + parallel_setup: ParallelSetup, + distributed_backend: str, + runner: RunnerOption, + test_options: CPTestOptions, + num_gpus_available: int, + *, + method: Literal["generate"], + is_multimodal: bool, +): + ( + tp_size, + pp_size, + dcp_size, + cp_kv_cache_interleave_size, + eager_mode, + chunked_prefill, + ) = parallel_setup + + multi_node_only, attn_backend = test_options + + model_info = HF_EXAMPLE_MODELS.find_hf_info(model_id) + model_info.check_transformers_version(on_fail="skip") + + trust_remote_code = model_info.trust_remote_code + tokenizer_mode = model_info.tokenizer_mode + hf_overrides = model_info.hf_overrides + + model_info.check_available_online(on_fail="skip") + + if num_gpus_available < tp_size * pp_size: + pytest.skip(f"Need at least {tp_size} x {pp_size} GPUs") + if VLLM_MULTI_NODE and distributed_backend == "mp": + pytest.skip( + "Skipping multi-node pipeline parallel test for " + "multiprocessing distributed backend" + ) + if multi_node_only and not VLLM_MULTI_NODE: + pytest.skip("Not in multi-node setting") + + server_args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-model-len", + "4096", + "--max-num-seqs", + "64", + ] + if chunked_prefill: + server_args.append("--enable-chunked-prefill") + if eager_mode: + server_args.append("--enforce-eager") + if runner != "auto": + server_args.extend(["--runner", runner]) + if trust_remote_code: + server_args.append("--trust-remote-code") + if tokenizer_mode: + server_args.extend(["--tokenizer-mode", tokenizer_mode]) + if hf_overrides: + server_args.extend(["--hf-overrides", json.dumps(hf_overrides)]) + + server_args.extend( + [ + "--tensor-parallel-size", + str(tp_size), + "--pipeline-parallel-size", + str(pp_size), + "--decode-context-parallel-size", + str(dcp_size), + "--dcp-kv-cache-interleave-size", + str(cp_kv_cache_interleave_size), + "--distributed-executor-backend", + distributed_backend, + ] + ) + + if attn_backend: + server_args.append(f"--attention-backend={attn_backend}") + + with RemoteOpenAIServer( + model_id, + server_args, + max_wait_seconds=720, + ) as remote_server: + url = f"{remote_server.url_for('v1')}/completions" + + model_args = ( + f"model={model_id}," + f"base_url={url}," + f"num_concurrent={NUM_CONCURRENT},tokenized_requests=False" + ) + + results = lm_eval.simple_evaluate( + model="local-completions", + model_args=model_args, + tasks=TASK, + num_fewshot=NUM_SHOTS, + ) + + # Validate accuracy is reasonable + accuracy = results["results"][TASK][FILTER] + min_accuracy = MIN_ACCURACY[model_id] + assert accuracy >= min_accuracy, ( + f"TP+DCP accuracy too low: {accuracy:.3f} < {min_accuracy:.3f}" + ) + + +@pytest.mark.parametrize( + ( + "model_id", + "parallel_setup", + "distributed_backend", + "runner", + "test_options", + ), + [ + params + for model_id, settings in CP_TEXT_GENERATION_MODELS.items() + for setting in settings + for params in setting.iter_params(model_id) + if model_id in CP_TEST_MODELS + ], +) +@create_new_process_for_each_test() +def test_cp_generation( + model_id: str, + parallel_setup: ParallelSetup, + distributed_backend: str, + runner: RunnerOption, + test_options: CPTestOptions, + num_gpus_available, +): + if ( + model_id == "deepseek-ai/DeepSeek-V2-Lite-Chat" + and torch.cuda.get_device_capability() < (9, 0) + ): + pytest.skip(reason="MLA+DCP requires compute capability of 9.0 or higher") + if ( + model_id == "Qwen/Qwen2.5-1.5B-Instruct" + and torch.cuda.get_device_capability() != (9, 0) + ): + pytest.skip(reason="GQA+DCP currently requires compute capability of 9.0") + + _test_cp_gsm8k( + model_id, + parallel_setup, + distributed_backend, + runner, + test_options, + num_gpus_available, + method="generate", + is_multimodal=False, + ) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_custom_all_reduce.py b/asian_ai/performance/vLLM/tests/distributed/test_custom_all_reduce.py new file mode 100644 index 0000000000000000000000000000000000000000..324236e99e11c751116b694a955c72fac99826ed --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_custom_all_reduce.py @@ -0,0 +1,132 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import random + +import pytest +import ray +import torch +import torch.distributed as dist + +from vllm.distributed.communication_op import tensor_model_parallel_all_reduce # noqa +from vllm.distributed.parallel_state import get_tp_group, graph_capture + +from ..utils import ( + ensure_model_parallel_initialized, + init_test_distributed_environment, + multi_process_parallel, +) + +random.seed(42) +test_sizes = [random.randint(1024, 2048 * 1024) for _ in range(8)] +for i, v in enumerate(test_sizes): + test_sizes[i] -= v % 8 + + +@ray.remote(num_gpus=1, max_calls=1) +def graph_allreduce( + monkeypatch: pytest.MonkeyPatch, + tp_size, + pp_size, + rank, + distributed_init_port, +): + with monkeypatch.context() as m: + m.delenv("CUDA_VISIBLE_DEVICES", raising=False) + m.delenv("HIP_VISIBLE_DEVICES", raising=False) + device = torch.device(f"cuda:{rank}") + torch.accelerator.set_device_index(device) + init_test_distributed_environment(tp_size, pp_size, rank, distributed_init_port) + ensure_model_parallel_initialized(tp_size, pp_size) + group = get_tp_group().device_group + + # A small all_reduce for warmup. + # this is needed because device communicators might be created lazily + # (e.g. NCCL). This will ensure that the communicator is initialized + # before any communication happens, so that this group can be used for + # graph capture immediately. + data = torch.zeros(1) + data = data.to(device=device) + torch.distributed.all_reduce(data, group=group) + torch.accelerator.synchronize() + del data + + # we use the first group to communicate once + # and the second group to communicate twice + # and so on + # this is used to demonstrate that each group can + # communicate independently + num_communication = rank // tp_size + 1 + + for sz in test_sizes: + for dtype in [torch.float32, torch.float16, torch.bfloat16]: + with graph_capture(device=device) as graph_capture_context: + # use integers so result matches NCCL exactly + device_idx = torch.accelerator.current_device_index() + inp1 = torch.randint(1, 16, (sz,), dtype=dtype, device=device_idx) + inp2 = torch.randint(1, 16, (sz,), dtype=dtype, device=device_idx) + + torch.accelerator.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=graph_capture_context.stream): + for i in range(num_communication): + out1 = tensor_model_parallel_all_reduce(inp1) + # the input buffer is immediately modified to test + # synchronization + dist.all_reduce(inp1, group=group) + out2 = tensor_model_parallel_all_reduce(inp2) + dist.all_reduce(inp2, group=group) + graph.replay() + torch.testing.assert_close(out1, inp1) + torch.testing.assert_close(out2, inp2) + + +@ray.remote(num_gpus=1, max_calls=1) +def eager_allreduce( + monkeypatch: pytest.MonkeyPatch, + tp_size, + pp_size, + rank, + distributed_init_port, +): + with monkeypatch.context() as m: + m.delenv("CUDA_VISIBLE_DEVICES", raising=False) + m.delenv("HIP_VISIBLE_DEVICES", raising=False) + device = torch.device(f"cuda:{rank}") + torch.accelerator.set_device_index(device) + init_test_distributed_environment(tp_size, pp_size, rank, distributed_init_port) + + # we use the first group to communicate once + # and the second group to communicate twice + # and so on + # this is used to demonstrate that each group can + # communicate independently + num_communication = rank // tp_size + 1 + sz = 1024 + fa = get_tp_group().device_communicator.ca_comm + inp = torch.ones(sz, dtype=torch.float32, device=device) + out = inp + for _ in range(num_communication): + out = fa.all_reduce(out, registered=False) + torch.testing.assert_close(out, inp * (tp_size**num_communication)) + + inp = torch.ones(sz * 4, dtype=torch.bfloat16, device=device) + out = inp + for _ in range(num_communication): + out = fa.all_reduce(out, registered=False) + torch.testing.assert_close(out, inp * (tp_size**num_communication)) + + +@pytest.mark.parametrize("tp_size", [2]) +@pytest.mark.parametrize("pipeline_parallel_size", [1, 2]) +@pytest.mark.parametrize("test_target", [eager_allreduce, graph_allreduce]) +def test_custom_allreduce( + monkeypatch: pytest.MonkeyPatch, + tp_size, + pipeline_parallel_size, + test_target, +): + world_size = tp_size * pipeline_parallel_size + if world_size > torch.accelerator.device_count(): + pytest.skip("Not enough GPUs to run the test.") + multi_process_parallel(monkeypatch, tp_size, pipeline_parallel_size, test_target) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_dcp_a2a.py b/asian_ai/performance/vLLM/tests/distributed/test_dcp_a2a.py new file mode 100644 index 0000000000000000000000000000000000000000..1b972420478ffaed8a5e0383e7c9e84c4b1c1f83 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_dcp_a2a.py @@ -0,0 +1,502 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for DCP A2A communication backend (no GPU required). + +Tests cover: +1. DCP A2A config validation (--dcp-comm-backend) +2. KVP group function exists +3. LSE-weighted combination correctness +""" + +import math + +import multiprocess as mp +import pytest +import torch +import torch.distributed as dist + +import vllm.envs as envs +from vllm.config.parallel import ParallelConfig +from vllm.utils.network_utils import get_open_port +from vllm.utils.system_utils import update_environment_variables + +mp.set_start_method("spawn", force=True) + + +class _FakeCPGroup: + def __init__(self, world_size: int, device_group: dist.ProcessGroup): + self.world_size = world_size + self.device_group = device_group + + +def _dtype_from_name(dtype_name: str) -> torch.dtype: + return { + "float16": torch.float16, + "bfloat16": torch.bfloat16, + "float32": torch.float32, + }[dtype_name] + + +def _packed_a2a_reference( + cp_attn_out: torch.Tensor, + cp_attn_lse: torch.Tensor, + world_size: int, + h_per_rank: int, + is_lse_base_on_e: bool, +) -> tuple[torch.Tensor, torch.Tensor]: + from vllm.v1.attention.ops.dcp_alltoall import _lse_weighted_combine + + B, _H, D = cp_attn_out.shape + outputs = ( + cp_attn_out.view(B, world_size, h_per_rank, D) + .permute(1, 0, 2, 3) + .contiguous() + .float() + ) + lses = cp_attn_lse.view(B, world_size, h_per_rank).permute(1, 0, 2).contiguous() + return _lse_weighted_combine( + outputs, + lses, + return_lse=True, + is_lse_base_on_e=is_lse_base_on_e, + ) + + +def _assert_packed_a2a_close( + actual: torch.Tensor, + expected: torch.Tensor, + dtype: torch.dtype, +) -> None: + if dtype == torch.float32: + torch.testing.assert_close(actual, expected, rtol=1e-5, atol=1e-5) + else: + torch.testing.assert_close( + actual.float(), expected.float(), rtol=3e-2, atol=3e-2 + ) + + +def _distributed_run(fn, world_size: int, extra_env: dict[str, str]) -> None: + port = str(get_open_port()) + processes: list[mp.Process] = [] + for rank in range(world_size): + env = { + "RANK": str(rank), + "LOCAL_RANK": str(rank), + "WORLD_SIZE": str(world_size), + "LOCAL_WORLD_SIZE": str(world_size), + "MASTER_ADDR": "localhost", + "MASTER_PORT": port, + **extra_env, + } + process = mp.Process(target=fn, args=(env,)) + processes.append(process) + process.start() + + for process in processes: + process.join(timeout=120) + + for process in processes: + if process.is_alive(): + process.kill() + process.join() + assert process.exitcode == 0 + + +class TestDCPCommBackendConfig: + """Test --dcp-comm-backend config validation.""" + + def test_default_is_ag_rs(self): + """Default comm backend is ag_rs.""" + config = ParallelConfig() + assert config.dcp_comm_backend == "ag_rs" + + def test_a2a_requires_dcp_greater_than_1(self): + """A2A backend requires decode_context_parallel_size > 1.""" + with pytest.raises( + ValueError, match="requires decode_context_parallel_size > 1" + ): + ParallelConfig( + dcp_comm_backend="a2a", + decode_context_parallel_size=1, + ) + + def test_a2a_with_dcp_valid(self): + """A2A backend is valid when DCP > 1.""" + config = ParallelConfig( + dcp_comm_backend="a2a", + tensor_parallel_size=4, + decode_context_parallel_size=4, + ) + assert config.dcp_comm_backend == "a2a" + + def test_invalid_backend_rejected(self): + """Invalid backend values are rejected.""" + with pytest.raises(ValueError, match="must be one of|Input should be"): + ParallelConfig( + dcp_comm_backend="invalid", + ) + + def test_ag_rs_with_dcp_1_valid(self): + """ag_rs backend is valid with DCP=1 (no DCP).""" + config = ParallelConfig( + dcp_comm_backend="ag_rs", + decode_context_parallel_size=1, + ) + assert config.dcp_comm_backend == "ag_rs" + + +class TestLSEWeightedCombine: + """Test LSE-weighted combination logic (CPU only, no GPU). + + The _lse_weighted_combine function is the reference implementation + that verifies the Triton kernel's correctness. It computes: + + result[b,h,d] = sum_n(w_n * output_n[b,h,d]) + + where w_n = softmax(lse_n) = exp(lse_n) / sum_k(exp(lse_k)) + """ + + def test_importable(self): + """Verify _lse_weighted_combine is importable.""" + from vllm.v1.attention.ops.dcp_alltoall import _lse_weighted_combine + + assert callable(_lse_weighted_combine) + + def test_single_rank(self): + """Single rank: output unchanged.""" + from vllm.v1.attention.ops.dcp_alltoall import _lse_weighted_combine + + # N=1, B=2, H=4, D=8 + outputs = torch.randn(1, 2, 4, 8) + lses = torch.randn(1, 2, 4) + + result = _lse_weighted_combine(outputs, lses) + + assert result.shape == (2, 4, 8) + torch.testing.assert_close(result, outputs.squeeze(0), rtol=1e-5, atol=1e-5) + + def test_equal_lse(self): + """Equal LSE values: outputs averaged equally.""" + from vllm.v1.attention.ops.dcp_alltoall import _lse_weighted_combine + + _N, B, H, D = 2, 1, 1, 4 + outputs = torch.tensor( + [ + [[[1.0, 2.0, 3.0, 4.0]]], # Rank 0 + [[[5.0, 6.0, 7.0, 8.0]]], # Rank 1 + ] + ) + lses = torch.tensor( + [ + [[0.0]], # Rank 0 + [[0.0]], # Rank 1 + ] + ) + + result = _lse_weighted_combine(outputs, lses) + + expected = (outputs[0] + outputs[1]) / 2 + assert result.shape == (B, H, D) + torch.testing.assert_close(result, expected, rtol=1e-5, atol=1e-5) + + def test_dominant_rank(self): + """Different LSE values: larger LSE gets more weight.""" + from vllm.v1.attention.ops.dcp_alltoall import _lse_weighted_combine + + B, H, D = 1, 1, 2 + outputs = torch.tensor( + [ + [[[0.0, 0.0]]], # Rank 0 + [[[1.0, 1.0]]], # Rank 1 + ] + ) + lses = torch.tensor( + [ + [[-100.0]], # Rank 0: negligible contribution + [[0.0]], # Rank 1: dominant + ] + ) + + result = _lse_weighted_combine(outputs, lses) + + assert result.shape == (B, H, D) + torch.testing.assert_close(result, outputs[1], atol=1e-5, rtol=1e-5) + + def test_mathematically_correct(self): + """Verify mathematical correctness of LSE combination.""" + from vllm.v1.attention.ops.dcp_alltoall import _lse_weighted_combine + + outputs = torch.tensor( + [ + [[[2.0, 4.0]]], + [[[6.0, 8.0]]], + ] + ) + lses = torch.tensor( + [ + [[1.0]], # exp(1) ≈ 2.718 + [[2.0]], # exp(2) ≈ 7.389 + ] + ) + + result = _lse_weighted_combine(outputs, lses) + + w0 = math.exp(1) / (math.exp(1) + math.exp(2)) + w1 = math.exp(2) / (math.exp(1) + math.exp(2)) + expected = torch.tensor([[[w0 * 2.0 + w1 * 6.0, w0 * 4.0 + w1 * 8.0]]]) + + torch.testing.assert_close(result, expected, rtol=1e-4, atol=1e-4) + + def test_return_lse(self): + """return_lse=True returns global LSE (logsumexp of inputs).""" + from vllm.v1.attention.ops.dcp_alltoall import _lse_weighted_combine + + B, H, D = 1, 1, 2 + outputs = torch.tensor( + [ + [[[1.0, 2.0]]], + [[[3.0, 4.0]]], + ] + ) + lses = torch.tensor( + [ + [[1.0]], + [[2.0]], + ] + ) + + result, global_lse = _lse_weighted_combine(outputs, lses, return_lse=True) + + expected_global_lse = math.log(math.exp(1) + math.exp(2)) + + assert result.shape == (B, H, D) + assert global_lse.shape == (B, H) + assert abs(global_lse.item() - expected_global_lse) < 1e-5 + + def test_base2_return_lse(self): + """Base-2 LSE mode returns log2-sum-exp2 global LSE.""" + from vllm.v1.attention.ops.dcp_alltoall import _lse_weighted_combine + + outputs = torch.tensor( + [ + [[[1.0, 2.0]]], + [[[3.0, 4.0]]], + ] + ) + lses = torch.tensor( + [ + [[1.0]], + [[2.0]], + ] + ) + + result, global_lse = _lse_weighted_combine( + outputs, + lses, + return_lse=True, + is_lse_base_on_e=False, + ) + + expected_global_lse = math.log2(2**1 + 2**2) + w0 = 2**1 / (2**1 + 2**2) + w1 = 2**2 / (2**1 + 2**2) + expected = torch.tensor([[[w0 * 1.0 + w1 * 3.0, w0 * 2.0 + w1 * 4.0]]]) + + torch.testing.assert_close(result, expected, rtol=1e-5, atol=1e-5) + torch.testing.assert_close( + global_lse, + torch.tensor([[expected_global_lse]]), + rtol=1e-5, + atol=1e-5, + ) + + def test_lse_pack_dim(self): + """Packed A2A stores one fp32 LSE in output-dtype lanes.""" + from vllm.v1.attention.ops.dcp_alltoall import _dcp_a2a_lse_pack_dim + + assert _dcp_a2a_lse_pack_dim(torch.bfloat16) == 2 + assert _dcp_a2a_lse_pack_dim(torch.float16) == 2 + assert _dcp_a2a_lse_pack_dim(torch.float32) == 1 + + +class TestPackedA2AKernels: + @pytest.mark.skipif( + torch.accelerator.device_count() < 1, reason="CUDA is required." + ) + @pytest.mark.parametrize("dtype_name", ["float16", "bfloat16", "float32"]) + @pytest.mark.parametrize("return_lse", [False, True]) + @pytest.mark.parametrize("is_lse_base_on_e", [False, True]) + def test_pack_unpack_combine_matches_reference( + self, + dtype_name: str, + return_lse: bool, + is_lse_base_on_e: bool, + ): + from vllm.v1.attention.ops.dcp_alltoall import ( + _dcp_a2a_lse_pack_dim, + _dcp_a2a_pack_send, + _dcp_a2a_unpack_combine, + ) + + torch.manual_seed(0) + dtype = _dtype_from_name(dtype_name) + device = torch.device("cuda") + world_size, B, h_per_rank, D = 4, 7, 2, 32 + H = world_size * h_per_rank + cp_attn_out = torch.randn(B, H, D, device=device, dtype=dtype) + cp_attn_lse = torch.randn(B, H, device=device, dtype=torch.float32) + lse_pack_dim = _dcp_a2a_lse_pack_dim(dtype) + send_buffer = torch.empty( + (world_size, B, h_per_rank, D + lse_pack_dim), + device=device, + dtype=dtype, + ) + + _dcp_a2a_pack_send( + cp_attn_out, + cp_attn_lse, + send_buffer, + world_size, + h_per_rank, + D, + lse_pack_dim, + ) + actual = _dcp_a2a_unpack_combine( + send_buffer, D, lse_pack_dim, return_lse, is_lse_base_on_e + ) + expected_out, expected_lse = _packed_a2a_reference( + cp_attn_out, cp_attn_lse, world_size, h_per_rank, is_lse_base_on_e + ) + + if return_lse: + actual_out, actual_lse = actual + _assert_packed_a2a_close(actual_out, expected_out, dtype) + torch.testing.assert_close(actual_lse, expected_lse, rtol=1e-4, atol=1e-4) + else: + _assert_packed_a2a_close(actual, expected_out, dtype) + + +def _distributed_packed_a2a_worker(env: dict[str, str]) -> None: + update_environment_variables(env) + local_rank = int(env["LOCAL_RANK"]) + torch.accelerator.set_device_index(local_rank) + if envs.VLLM_DISTRIBUTED_USE_SPLIT_GROUP: + dist.init_process_group( + backend="cpu:gloo,cuda:nccl", + device_id=torch.device(f"cuda:{local_rank}"), + ) + else: + dist.init_process_group(backend="nccl") + use_workspace = env.get("USE_WORKSPACE") == "1" + if use_workspace: + from vllm.v1.worker.workspace import init_workspace_manager + + init_workspace_manager(torch.device(f"cuda:{local_rank}")) + try: + from vllm.v1.attention.ops.dcp_alltoall import dcp_a2a_lse_reduce + + dtype = _dtype_from_name(env["TEST_DTYPE"]) + return_lse = env["RETURN_LSE"] == "1" + is_lse_base_on_e = env["LSE_BASE_E"] == "1" + rank = dist.get_rank() + world_size = dist.get_world_size() + B, h_per_rank, D = 5, 2, 32 + H = world_size * h_per_rank + + generator = torch.Generator(device=f"cuda:{local_rank}") + generator.manual_seed(1234 + rank) + cp_attn_out = torch.randn( + B, + H, + D, + device=f"cuda:{local_rank}", + dtype=dtype, + generator=generator, + ) + cp_attn_lse = torch.randn( + B, + H, + device=f"cuda:{local_rank}", + dtype=torch.float32, + generator=generator, + ) + actual = dcp_a2a_lse_reduce( + cp_attn_out, + cp_attn_lse, + _FakeCPGroup(world_size, dist.group.WORLD), + return_lse=return_lse, + is_lse_base_on_e=is_lse_base_on_e, + ) + + gathered_out = [torch.empty_like(cp_attn_out) for _ in range(world_size)] + gathered_lse = [torch.empty_like(cp_attn_lse) for _ in range(world_size)] + dist.all_gather(gathered_out, cp_attn_out) + dist.all_gather(gathered_lse, cp_attn_lse) + outputs = torch.stack( + [ + t[:, rank * h_per_rank : (rank + 1) * h_per_rank, :] + for t in gathered_out + ], + dim=0, + ).float() + lses = torch.stack( + [t[:, rank * h_per_rank : (rank + 1) * h_per_rank] for t in gathered_lse], + dim=0, + ) + from vllm.v1.attention.ops.dcp_alltoall import _lse_weighted_combine + + expected_out, expected_lse = _lse_weighted_combine( + outputs, + lses, + return_lse=True, + is_lse_base_on_e=is_lse_base_on_e, + ) + + if return_lse: + actual_out, actual_lse = actual + _assert_packed_a2a_close(actual_out, expected_out, dtype) + torch.testing.assert_close(actual_lse, expected_lse, rtol=1e-4, atol=1e-4) + else: + _assert_packed_a2a_close(actual, expected_out, dtype) + finally: + if use_workspace: + from vllm.v1.worker.workspace import reset_workspace_manager + + reset_workspace_manager() + dist.destroy_process_group() + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 4, reason="Need at least 4 GPUs." +) +@pytest.mark.parametrize("dtype_name", ["float16", "bfloat16", "float32"]) +def test_distributed_packed_a2a_matches_reference(dtype_name: str): + _distributed_run( + _distributed_packed_a2a_worker, + world_size=4, + extra_env={ + "TEST_DTYPE": dtype_name, + "RETURN_LSE": "1", + "LSE_BASE_E": "1", + }, + ) + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 4, reason="Need at least 4 GPUs." +) +def test_distributed_packed_a2a_with_workspace_matches_reference(): + _distributed_run( + _distributed_packed_a2a_worker, + world_size=4, + extra_env={ + "TEST_DTYPE": "bfloat16", + "RETURN_LSE": "1", + "LSE_BASE_E": "1", + "USE_WORKSPACE": "1", + }, + ) + + +if __name__ == "__main__": + pytest.main([__file__, "-v"]) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_distributed_oot.py b/asian_ai/performance/vLLM/tests/distributed/test_distributed_oot.py new file mode 100644 index 0000000000000000000000000000000000000000..4e76d8692030a3379d3cfc2f2d93b0bd30efff2d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_distributed_oot.py @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from tests.plugins_tests.test_oot_registration_online import ( + run_and_test_dummy_opt_api_server, +) + + +def test_distributed_oot(dummy_opt_path: str): + run_and_test_dummy_opt_api_server(dummy_opt_path, tp=2) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_elastic_ep.py b/asian_ai/performance/vLLM/tests/distributed/test_elastic_ep.py new file mode 100644 index 0000000000000000000000000000000000000000..382e3a8eafbed02a4c8ffecb1340a2a275f0d619 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_elastic_ep.py @@ -0,0 +1,205 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import os +import subprocess +import time + +import pytest +import requests + +from ..evals.gsm8k.gsm8k_eval import evaluate_gsm8k +from ..utils import RemoteOpenAIServer, multi_gpu_test + + +@pytest.fixture(autouse=True) +def cleanup_ray_between_tests(): + """Force-stop any lingering Ray processes between tests.""" + subprocess.run(["ray", "stop", "--force"], timeout=30, capture_output=True) + time.sleep(5) + yield + + +MODEL_NAME = "deepseek-ai/DeepSeek-V2-Lite-Chat" + +NUM_GSM8K_QUESTIONS = 256 +EXPECTED_ACCURACY = 0.58 +ACCURACY_TOL = 0.08 +MAX_NUM_SEQS = 32 + + +def _send_scale_command(server: RemoteOpenAIServer, new_dp_size: int) -> bool: + url = server.url_for("scale_elastic_ep") + payload = {"new_data_parallel_size": new_dp_size} + headers = {"Content-Type": "application/json"} + + try: + response = requests.post(url, json=payload, headers=headers, timeout=300) + return response.status_code == 200 + except requests.exceptions.RequestException: + return False + + +def _run_gsm8k_eval(server: RemoteOpenAIServer, stage: str) -> float: + assert server.port is not None + result = evaluate_gsm8k( + num_questions=NUM_GSM8K_QUESTIONS, + host=f"http://{server.host}", + port=server.port, + ) + accuracy = result["accuracy"] + print( + f"[{stage}] GSM8K accuracy: {accuracy:.3f} " + f"({result['num_questions']} questions)" + ) + assert accuracy >= EXPECTED_ACCURACY, ( + f"[{stage}] GSM8K accuracy {accuracy:.3f} is below " + f"expected threshold {EXPECTED_ACCURACY}" + ) + return accuracy + + +def _base_serve_args(use_async_eplb: bool = False) -> list[str]: + args = [ + "--trust-remote-code", + "--tensor-parallel-size", + "1", + "--gpu-memory-utilization", + "0.8", + "--max-model-len", + "4096", + "--max-num-seqs", + str(MAX_NUM_SEQS), + "--enable-expert-parallel", + "--all2all-backend", + "allgather_reducescatter", + "--enable-elastic-ep", + "--enable-eplb", + "--eplb-config.num_redundant_experts", + "0", + "--eplb-config.use_async", + "true" if use_async_eplb else "false", + "--eplb-config.step_interval", + "10", + "--eplb-config.window_size", + "5", + "--data-parallel-backend", + "ray", + "--data-parallel-size", + "2", + "--api-server-count", + "1", + ] + + leader_address = os.environ.get("LEADER_ADDRESS") + if leader_address: + args.extend(["--data-parallel-address", leader_address]) + + return args + + +@pytest.mark.parametrize( + "use_async_eplb", [False, True], ids=["sync_eplb", "async_eplb"] +) +@multi_gpu_test(num_gpus=4) +def test_elastic_ep_scaling(use_async_eplb: bool): + if use_async_eplb: + from vllm.distributed.eplb.eplb_communicator import has_nixl + + if not has_nixl(): + pytest.skip("Async EPLB with elastic EP requires NIXL (not installed)") + + vllm_serve_args = _base_serve_args(use_async_eplb) + + with RemoteOpenAIServer( + MODEL_NAME, vllm_serve_args, env_dict={}, max_wait_seconds=1200 + ) as server: + initial_accuracy = _run_gsm8k_eval(server, "Initial (2 GPUs)") + + assert _send_scale_command(server, 4) + time.sleep(10) + scale_up_accuracy = _run_gsm8k_eval(server, "After scale up (4 GPUs)") + + assert scale_up_accuracy >= initial_accuracy - ACCURACY_TOL, ( + f"Scale up accuracy {scale_up_accuracy:.3f} dropped more than " + f"{ACCURACY_TOL} below initial accuracy {initial_accuracy:.3f}" + ) + + assert _send_scale_command(server, 2) + time.sleep(5) + scale_down_accuracy = _run_gsm8k_eval(server, "After scale down (2 GPUs)") + + assert scale_down_accuracy >= initial_accuracy - ACCURACY_TOL, ( + f"Scale down accuracy {scale_down_accuracy:.3f} dropped more than " + f"{ACCURACY_TOL} below initial accuracy {initial_accuracy:.3f}" + ) + + print("\nAccuracy Summary:") + print(f" Initial: {initial_accuracy:.3f}") + print( + f" Scale up: {scale_up_accuracy:.3f} " + f"(diff: {scale_up_accuracy - initial_accuracy:+.3f})" + ) + print( + f" Scale down: {scale_down_accuracy:.3f} " + f"(diff: {scale_down_accuracy - initial_accuracy:+.3f})" + ) + print(f" Tolerance: {ACCURACY_TOL:.3f}") + + +@pytest.mark.parametrize( + "use_async_eplb", [False, True], ids=["sync_eplb", "async_eplb"] +) +@multi_gpu_test(num_gpus=4) +def test_elastic_ep_scaling_uneven(use_async_eplb: bool): + """Test scale up with uneven worker distribution. + + This tests the case where num_new_workers % old_dp_size != 0, + specifically 2 -> 3 where remainder = 1 % 2 = 1. + This exercises the remainder handling in sender-receiver pairing. + """ + if use_async_eplb: + from vllm.distributed.eplb.eplb_communicator import has_nixl + + if not has_nixl(): + pytest.skip("Async EPLB with elastic EP requires NIXL (not installed)") + + vllm_serve_args = _base_serve_args(use_async_eplb) + + with RemoteOpenAIServer( + MODEL_NAME, vllm_serve_args, env_dict={}, max_wait_seconds=1200 + ) as server: + initial_accuracy = _run_gsm8k_eval(server, "Initial (2 GPUs)") + + # Scale 2 -> 3: This has remainder = 1 % 2 = 1 + # Tests uneven sender-receiver pairing + assert _send_scale_command(server, 3) + time.sleep(10) + scale_up_accuracy = _run_gsm8k_eval(server, "After scale up (3 GPUs)") + + assert scale_up_accuracy >= initial_accuracy - ACCURACY_TOL, ( + f"Scale up accuracy {scale_up_accuracy:.3f} dropped more than " + f"{ACCURACY_TOL} below initial accuracy {initial_accuracy:.3f}" + ) + + # Scale back down to 2 + assert _send_scale_command(server, 2) + time.sleep(5) + scale_down_accuracy = _run_gsm8k_eval(server, "After scale down (2 GPUs)") + + assert scale_down_accuracy >= initial_accuracy - ACCURACY_TOL, ( + f"Scale down accuracy {scale_down_accuracy:.3f} dropped more than " + f"{ACCURACY_TOL} below initial accuracy {initial_accuracy:.3f}" + ) + + print("\nAccuracy Summary (Uneven Scaling):") + print(f" Initial: {initial_accuracy:.3f}") + print( + f" Scale up: {scale_up_accuracy:.3f} " + f"(diff: {scale_up_accuracy - initial_accuracy:+.3f})" + ) + print( + f" Scale down: {scale_down_accuracy:.3f} " + f"(diff: {scale_down_accuracy - initial_accuracy:+.3f})" + ) + print(f" Tolerance: {ACCURACY_TOL:.3f}") diff --git a/asian_ai/performance/vLLM/tests/distributed/test_eplb_algo.py b/asian_ai/performance/vLLM/tests/distributed/test_eplb_algo.py new file mode 100644 index 0000000000000000000000000000000000000000..53e0538a53dfa72a4f7e0fbfba7fb296c8955a26 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_eplb_algo.py @@ -0,0 +1,498 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import numpy as np +import pytest +import torch + +from vllm.distributed.eplb.eplb_state import compute_logical_maps +from vllm.distributed.eplb.policy.default import DefaultEplbPolicy + + +def test_basic_rebalance(): + """Test basic rebalancing functionality""" + # Example from https://github.com/deepseek-ai/eplb + weight = torch.tensor( + [ + [90, 132, 40, 61, 104, 165, 39, 4, 73, 56, 183, 86], + [20, 107, 104, 64, 19, 197, 187, 157, 172, 86, 16, 27], + ] + ) + + num_layers = weight.shape[0] + num_replicas = 16 + num_groups = 4 + num_nodes = 2 + num_gpus = 8 + + phy2log = DefaultEplbPolicy.rebalance_experts( + weight, num_replicas, num_groups, num_nodes, num_gpus + ) + log2phy, logcnt = compute_logical_maps(phy2log, weight.shape[-1]) + + # Verify output shapes + assert phy2log.shape == ( + 2, + 16, + ), f"Expected `phy2log` shape (2, 16), got {phy2log.shape}" + assert log2phy.shape[0] == 2, ( + f"Expected `log2phy` first dimension 2, got {log2phy.shape[0]}" + ) + assert log2phy.shape[1] == 12, ( + f"Expected `log2phy` second dimension 12, got {log2phy.shape[1]}" + ) + assert logcnt.shape == ( + 2, + 12, + ), f"Expected `logcnt` shape (2, 12), got {logcnt.shape}" + + # Verify physical to logical expert mapping range is correct + assert torch.all(phy2log >= 0) and torch.all(phy2log < 12), ( + "Physical to logical mapping should be in range [0, 12)" + ) + + # Verify expert count reasonableness + assert torch.all(logcnt >= 1), "Each logical expert should have at least 1 replica" + assert torch.sum(logcnt, dim=1).sum() == num_replicas * num_layers, ( + f"Total replicas should be {num_replicas * num_layers}" + ) + + # Verify expected output + expected_phy2log = torch.tensor( + [ + [5, 6, 5, 7, 8, 4, 3, 4, 10, 9, 10, 2, 0, 1, 11, 1], + [7, 10, 6, 8, 6, 11, 8, 9, 2, 4, 5, 1, 5, 0, 3, 1], + ] + ) + assert torch.all(phy2log == expected_phy2log) + + expected_logcnt = torch.tensor( + [[1, 2, 1, 1, 2, 2, 1, 1, 1, 1, 2, 1], [1, 2, 1, 1, 1, 2, 2, 1, 2, 1, 1, 1]] + ) + assert torch.all(logcnt == expected_logcnt) + + +def test_single_gpu_case(): + """Test single GPU case""" + weight = torch.tensor([[10, 20, 30, 40]]) + num_replicas = 4 + num_groups = 1 + num_nodes = 1 + num_gpus = 1 + + phy2log = DefaultEplbPolicy.rebalance_experts( + weight, num_replicas, num_groups, num_nodes, num_gpus + ) + log2phy, logcnt = compute_logical_maps(phy2log, weight.shape[-1]) + + # Verify shapes + assert phy2log.shape == (1, 4) + assert log2phy.shape[0] == 1 + assert log2phy.shape[1] == 4 + assert logcnt.shape == (1, 4) + + # Verify all logical experts are mapped + assert set(phy2log[0].tolist()) == {0, 1, 2, 3} + + +def test_equal_weights(): + """Test case with equal weights""" + weight = torch.tensor([[50, 50, 50, 50, 50, 50, 50, 50]]) + num_replicas = 8 + num_groups = 2 + num_nodes = 2 + num_gpus = 4 + + phy2log = DefaultEplbPolicy.rebalance_experts( + weight, num_replicas, num_groups, num_nodes, num_gpus + ) + _, logcnt = compute_logical_maps(phy2log, weight.shape[-1]) + + # Verify shapes + assert phy2log.shape == (1, 8) + assert logcnt.shape == (1, 8) + + # With equal weights, each expert should have exactly one replica + assert torch.all(logcnt == 1), ( + "With equal weights and no replication, " + "each expert should have exactly 1 replica" + ) + + +def test_extreme_weight_imbalance(): + """Test extreme weight imbalance case""" + weight = torch.tensor([[1000, 1, 1, 1, 1, 1, 1, 1]]) + num_replicas = 12 + num_groups = 2 + num_nodes = 2 + num_gpus = 4 + + phy2log = DefaultEplbPolicy.rebalance_experts( + weight, num_replicas, num_groups, num_nodes, num_gpus + ) + _, logcnt = compute_logical_maps(phy2log, weight.shape[-1]) + + # Verify shapes + assert phy2log.shape == (1, 12) + assert logcnt.shape == (1, 8) + + # Expert with highest weight (index 0) should have more replicas + assert logcnt[0, 0] > logcnt[0, 1], ( + "Expert with highest weight should have more replicas" + ) + + +def test_multiple_layers(): + """Test multiple layers case""" + weight = torch.tensor( + [ + [10, 20, 30, 40, 50, 60], # First layer + [60, 50, 40, 30, 20, 10], # Second layer (opposite weight pattern) + [25, 25, 25, 25, 25, 25], # Third layer (equal weights) + ] + ) + num_replicas = 8 + num_groups = 2 + num_nodes = 2 + num_gpus = 4 + + phy2log = DefaultEplbPolicy.rebalance_experts( + weight, num_replicas, num_groups, num_nodes, num_gpus + ) + _, logcnt = compute_logical_maps(phy2log, weight.shape[-1]) + + # Verify shapes + assert phy2log.shape == (3, 8) + assert logcnt.shape == (3, 6) + + # Verify expert allocation is reasonable for each layer + for layer in range(3): + assert torch.all(phy2log[layer] >= 0) and torch.all(phy2log[layer] < 6), ( + f"Layer {layer} physical to logical mappingshould be in range [0, 6)" + ) + assert torch.sum(logcnt[layer]) == num_replicas, ( + f"Layer {layer} total replicas should be {num_replicas}" + ) + + +def test_parameter_validation(): + """Test parameter validation""" + weight = torch.tensor([[10, 20, 30, 40]]) + + # Test non-divisible case - this should handle normally without throwing + # errors because the function will fall back to global load balancing + # strategy + phy2log = DefaultEplbPolicy.rebalance_experts(weight, 8, 3, 2, 4) + _, logcnt = compute_logical_maps(phy2log, weight.shape[-1]) + assert phy2log.shape == (1, 8) + assert logcnt.shape == (1, 4) + + # Test cases that will actually cause errors: + # num_physical_experts not divisible by num_gpus + with pytest.raises(AssertionError): + DefaultEplbPolicy.rebalance_experts(weight, 7, 2, 2, 4) # 7 not divisible by 4 + + +def test_small_scale_hierarchical(): + """Test small-scale hierarchical load balancing""" + weight = torch.tensor( + [ + [100, 50, 200, 75, 150, 25, 300, 80], # 8 experts + ] + ) + num_replicas = 12 + num_groups = 4 # 4 groups, 2 experts each + num_nodes = 2 # 2 nodes + num_gpus = 4 # 4 GPUs + + phy2log = DefaultEplbPolicy.rebalance_experts( + weight, num_replicas, num_groups, num_nodes, num_gpus + ) + _, logcnt = compute_logical_maps(phy2log, weight.shape[-1]) + + # Verify basic constraints + assert phy2log.shape == (1, 12) + assert logcnt.shape == (1, 8) + assert torch.sum(logcnt) == num_replicas + assert torch.all(logcnt >= 1) + + # Expert with highest weight should have more replicas + max_weight_expert = torch.argmax(weight[0]) + assert logcnt[0, max_weight_expert] >= 2, ( + "Highest weight expert should have multiple replicas" + ) + + +def test_global_load_balance_fallback(): + """Test global load balancing fallback case""" + # When num_groups % num_nodes != 0, should fall back to global load + # balancing + weight = torch.tensor([[10, 20, 30, 40, 50, 60]]) + num_replicas = 8 + num_groups = 3 # Cannot be divided evenly by num_nodes=2 + num_nodes = 2 + num_gpus = 4 + + phy2log = DefaultEplbPolicy.rebalance_experts( + weight, num_replicas, num_groups, num_nodes, num_gpus + ) + _, logcnt = compute_logical_maps(phy2log, weight.shape[-1]) + + # Should work normally, just using global load balancing strategy + assert phy2log.shape == (1, 8) + assert logcnt.shape == (1, 6) + assert torch.sum(logcnt) == num_replicas + + +@pytest.mark.parametrize("device", ["cpu", "cuda"]) +def test_device_compatibility(device): + """Test device compatibility""" + if device == "cuda" and not torch.cuda.is_available(): + pytest.skip("CUDA not available") + + weight = torch.tensor([[10, 20, 30, 40]], device=device) + num_replicas = 6 + num_groups = 2 + num_nodes = 1 + num_gpus = 2 + + phy2log = DefaultEplbPolicy.rebalance_experts( + weight, num_replicas, num_groups, num_nodes, num_gpus + ) + _, logcnt = compute_logical_maps(phy2log, weight.shape[-1]) + + # Function will convert to CPU internally, but should handle different + # device inputs normally + assert phy2log.shape == (1, 6) + assert logcnt.shape == (1, 4) + + +def test_additional_cases(): + """Test more edge cases and different parameter combinations""" + + # Test case 1: Large-scale distributed setup + weight1 = torch.tensor( + [[50, 100, 75, 120, 90, 60, 80, 110, 40, 70, 95, 85, 65, 55, 45, 35]] + ) + phy2log1 = DefaultEplbPolicy.rebalance_experts(weight1, 24, 8, 4, 8) + _, logcnt1 = compute_logical_maps(phy2log1, weight1.shape[-1]) + + assert phy2log1.shape == (1, 24) + assert logcnt1.shape == (1, 16) + assert torch.sum(logcnt1) == 24 + + # Test case 2: Different weight distributions + weight2 = torch.tensor( + [ + [200, 150, 100, 50, 25, 12], # Decreasing weights + [12, 25, 50, 100, 150, 200], # Increasing weights + ] + ) + phy2log2 = DefaultEplbPolicy.rebalance_experts(weight2, 10, 3, 1, 2) + _, logcnt2 = compute_logical_maps(phy2log2, weight2.shape[-1]) + + assert phy2log2.shape == (2, 10) + assert logcnt2.shape == (2, 6) + + # Verify high-weight experts have more replicas + for layer in range(2): + max_weight_idx = torch.argmax(weight2[layer]) + assert logcnt2[layer, max_weight_idx] >= 2 + + +def test_compute_logical_maps_with_negative_indices(): + """ + Test that compute_logical_maps correctly handles physical slots containing + -1 (unused slots). + """ + # 2 layers, 6 physical slots, 4 logical experts. + # Slots 2 and 5 are unused (-1). + phy2log = torch.tensor( + [ + [0, 1, -1, 2, 3, -1], + [3, -1, 2, 1, 0, -1], + ] + ) + num_layers = 2 + num_logical_experts = 4 + + log2phy, logcnt = compute_logical_maps(phy2log, num_logical_experts) + + assert logcnt.shape == (num_layers, num_logical_experts) + assert log2phy.shape == (num_layers, num_logical_experts, 1) + + expected_logcnt = torch.ones(num_layers, num_logical_experts, dtype=phy2log.dtype) + assert torch.all(logcnt == expected_logcnt), ( + f"Expected that all replica counts == 1, got {logcnt}" + ) + + assert torch.all(log2phy >= 0), ( + "log2phy should only contain valid physical indices, not -1" + ) + + assert log2phy[0, 0, 0] == 0 + assert log2phy[0, 1, 0] == 1 + assert log2phy[0, 2, 0] == 3 + assert log2phy[0, 3, 0] == 4 + + +if __name__ == "__main__": + weight = torch.tensor( + [ + [90, 132, 40, 61, 104, 165, 39, 4, 73, 56, 183, 86], + [20, 107, 104, 64, 19, 197, 187, 157, 172, 86, 16, 27], + ] + ) + + num_replicas = 16 + num_groups = 4 + num_nodes = 2 + num_gpus = 8 + + phy2log = DefaultEplbPolicy.rebalance_experts( + weight, num_replicas, num_groups, num_nodes, num_gpus + ) + print(phy2log) + + test_basic_rebalance() + + +def _make_phy_replicas_idx_from_phy2log(phy2log: np.ndarray) -> np.ndarray: + """Create replicas indices mapping from phy2log.""" + pr = np.zeros_like(phy2log, dtype=np.int64) + for layer in range(phy2log.shape[0]): + seen: dict[int, int] = {} + row = phy2log[layer].tolist() + for i, expert in enumerate(row): + r = seen.get(expert, 0) + pr[layer, i] = r + seen[expert] = r + 1 + return pr + + +def _validate_intragpu_rearrangement( + old_global_expert_indices: np.ndarray, + new_phy2log: np.ndarray, + new_phy_replicas_idx: np.ndarray, + post_phy2log: np.ndarray, + post_phy_replicas_idx: np.ndarray, + num_ranks: int, + slots_per_gpu: int, +): + # Per-GPU checks + for gpu_idx in range(num_ranks): + start = gpu_idx * slots_per_gpu + end = start + slots_per_gpu + old_seg = old_global_expert_indices[0, start:end] + new_seg = new_phy2log[0, start:end] + new_rnk = new_phy_replicas_idx[0, start:end] + post_seg = post_phy2log[0, start:end] + post_rnk = post_phy_replicas_idx[0, start:end] + + # Pairwise equality for (expert, rank) pairs to ensure nothing is lost + def sorted_pairs(seg, rnk): + pairs = list(zip(seg.tolist(), rnk.tolist())) + pairs.sort() + return pairs + + assert sorted_pairs(post_seg, post_rnk) == sorted_pairs(new_seg, new_rnk), ( + f"Per-GPU pairs of (expert,rank) must match new mapping for GPU {gpu_idx}" + ) + + # For experts that remain on the same GPU, the old slot is preserved + # for at least one occurrence; rank at that slot must be valid for that expert + old_list = old_seg.tolist() + new_list = new_seg.tolist() + post_list = post_seg.tolist() + remained = set(old_list) & set(new_list) + new_ranks_for_expert: dict[int, list[int]] = {} + for v, r in zip(new_list, new_rnk.tolist()): + new_ranks_for_expert.setdefault(v, []).append(r) + for expert in remained: + old_pos = old_list.index(expert) + assert post_list[old_pos] == expert, ( + f"Expert {expert} on GPU {gpu_idx} should stay at old slot {old_pos}" + ) + # Rank at preserved slot must be one of the ranks + # the expert has in new mapping + assert post_rnk.tolist()[old_pos] in new_ranks_for_expert[expert], ( + f"Rank for expert {expert} at preserved slot on GPU {gpu_idx} " + "must come from new mapping" + ) + + +@pytest.mark.parametrize( + "num_ranks, slots_per_gpu, old_phy2log, new_phy2log", + [ + pytest.param( + # Setup: 2 GPUs, 4 slots each, 1 layer + # Old mapping: GPU0 -> [0,1,2,3], GPU1 -> [4,5,6,7] + # New mapping shuffles within GPU0 and brings 4,5 into GPU0. + # GPU0 new -> [1,5,0,4]; GPU1 new -> [6,2,7,3] + 2, + 4, + np.array([[0, 1, 2, 3, 4, 5, 6, 7]]), + np.array([[1, 5, 0, 4, 6, 2, 7, 3]]), + id="simple", + ), + pytest.param( + # Setup: 2 GPUs, 5 slots each (total 10 physical experts), 1 layer + # Old mapping: + # GPU0 -> [0, 1, 0, 2, 3] (expert 0 duplicated) + # GPU1 -> [4, 5, 6, 1, 2] + # New mapping reorders within GPUs and moves some experts across GPUs, + # while still including duplicates: + # GPU0 new -> [0, 5, 4, 0, 1] (expert 0 duplicated, 4/5 incoming) + # GPU1 new -> [6, 2, 3, 2, 1] (expert 2 duplicated) + 2, + 5, + np.array([[0, 1, 0, 2, 3, 4, 5, 6, 1, 2]]), + np.array([[0, 5, 4, 0, 1, 6, 2, 3, 2, 1]]), + id="duplicates", + ), + pytest.param( + # Setup: 3 GPUs, 4 slots each (total 12 physical experts), 1 layer + # Old mapping: + # GPU0 -> [0, 1, 2, 3] + # GPU1 -> [0, 1, 2, 3] + # GPU2 -> [0, 1, 2, 3] + # New mapping decides to use one expert on 2 GPUs and shuffles + # experts on the third GPU, + # GPU0 new -> [0, 0, 0, 0] + # GPU1 new -> [0, 0, 0, 0] + # GPU2 new -> [1, 2, 3, 0] + 3, + 4, + np.array([[0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3]]), + np.array([[0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 0]]), + id="skewed_expert", + ), + ], +) +def test_preserve_intragpu_slots( + num_ranks: int, + slots_per_gpu: int, + old_phy2log: torch.Tensor, + new_phy2log: torch.Tensor, +): + """Experts that stay on a GPU keep their old slots; incoming not lost.""" + phy_replicas_idx = _make_phy_replicas_idx_from_phy2log(new_phy2log) + + post_phy2log = DefaultEplbPolicy.preserve_intragpu_slots( + new_phy2log, num_ranks, old_phy2log + ) + post_phy_replicas_idx = _make_phy_replicas_idx_from_phy2log(post_phy2log) + + # Shapes preserved + assert post_phy2log.shape == new_phy2log.shape + assert post_phy_replicas_idx.shape == phy_replicas_idx.shape + + _validate_intragpu_rearrangement( + old_phy2log, + new_phy2log, + phy_replicas_idx, + post_phy2log, + post_phy_replicas_idx, + num_ranks, + slots_per_gpu, + ) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_eplb_events.py b/asian_ai/performance/vLLM/tests/distributed/test_eplb_events.py new file mode 100644 index 0000000000000000000000000000000000000000..83a5077f078774f6d74db482947c2494ef8703b9 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_eplb_events.py @@ -0,0 +1,98 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import threading +import time + +import torch + +from vllm.distributed.eplb.eplb_utils import CpuGpuEvent + + +def test_wait_blocks_until_record(): + event = CpuGpuEvent() + record_stream = torch.cuda.Stream() + wait_stream = torch.cuda.Stream() + wait_returned = threading.Event() + + def waiter(): + event.wait(stream=wait_stream) + wait_returned.set() + + t = threading.Thread(target=waiter) + t.start() + + time.sleep(0.05) + assert not wait_returned.is_set(), "wait() returned before record() was called" + + event.record(stream=record_stream) + t.join(timeout=5.0) + + assert not event._recorded.is_set() + + +def test_reuse_across_multiple_cycles(): + wrapper = CpuGpuEvent() + record_stream = torch.cuda.Stream() + wait_stream = torch.cuda.Stream() + NUM_CYCLES = 8 + completed_cycles = [] + barriers = [threading.Barrier(2) for _ in range(NUM_CYCLES)] + + def waiter(): + for i in range(NUM_CYCLES): + wrapper.wait(stream=wait_stream) + completed_cycles.append(True) + barriers[i].wait() + + t = threading.Thread(target=waiter) + t.start() + + for i in range(NUM_CYCLES): + wrapper.record(stream=record_stream) + barriers[i].wait() + + t.join(timeout=10.0) + assert len(completed_cycles) == NUM_CYCLES + + +def test_producer_consumer(): + """ + This test uses the CpuGpuEvent to synchronize reads and writes to/from a shared GPU + tensor on multiple CPU threads. + """ + worker_stream = torch.cuda.Stream() + # Create a single element counter that will be shared between two threads + buf = torch.zeros(1, device="cuda") + NUM_ROUNDS = 5 + + ready_cpu = [threading.Event() for _ in range(NUM_ROUNDS)] + events = [CpuGpuEvent() for _ in range(NUM_ROUNDS)] + errors: list[str] = [] + + # For each round, the worker thread (writer) sets the counter in buf and waits for + # the main thread to read it. + def worker(): + for i in range(NUM_ROUNDS): + if i > 0: + events[i - 1].wait(stream=worker_stream) + + with torch.cuda.stream(worker_stream): + buf.fill_(float(i + 1)) + + worker_stream.synchronize() + ready_cpu[i].set() + + t = threading.Thread(target=worker) + t.start() + + for i in range(NUM_ROUNDS): + ready_cpu[i].wait() + snapshot = buf.clone() + events[i].record() + val = snapshot.item() + if val != float(i + 1): + errors.append(f"round {i}: expected {i + 1:.1f}, got {val:.1f}") + + t.join(timeout=10.0) + assert not errors, f"Buffer ordering errors: {errors}" diff --git a/asian_ai/performance/vLLM/tests/distributed/test_eplb_execute.py b/asian_ai/performance/vLLM/tests/distributed/test_eplb_execute.py new file mode 100644 index 0000000000000000000000000000000000000000..f9759449b13aaaf497d5d37f74f4a7c03970ea79 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_eplb_execute.py @@ -0,0 +1,906 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import random + +import pytest +import torch +import torch.distributed + +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.distributed.eplb.eplb_communicator import ( + create_eplb_communicator, + has_nixl, +) +from vllm.distributed.eplb.rebalance_execute import ( + move_from_buffer, + rearrange_expert_weights_inplace, + transfer_layer, +) +from vllm.distributed.parallel_state import ( + ensure_model_parallel_initialized, + get_tp_group, +) + +from .eplb_utils import distributed_run, set_env_vars_and_device + + +def create_expert_indices_with_redundancy( + num_layers: int, + num_logical_experts: int, + total_physical_experts: int, + redundancy_config: list[int], # redundancy for each logical expert +) -> torch.Tensor: + """ + Create expert indices with redundancy. + + Args: + num_layers: number of layers + num_logical_experts: number of logical experts + total_physical_experts: total number of physical experts + redundancy_config: redundancy for each logical expert + + Returns: + indices: Shape (num_layers, total_physical_experts) + """ + assert sum(redundancy_config) == total_physical_experts + assert len(redundancy_config) == num_logical_experts + + indices = torch.zeros(num_layers, total_physical_experts, dtype=torch.long) + + for layer in range(num_layers): + physical_pos = 0 + for logical_expert_id, redundancy in enumerate(redundancy_config): + for _ in range(redundancy): + indices[layer, physical_pos] = logical_expert_id + physical_pos += 1 + + # Shuffle the indices at dim 1 + for layer in range(num_layers): + indices[layer] = indices[layer][torch.randperm(indices.shape[1])] + + return indices + + +def create_expert_weights( + num_layers: int, + num_local_experts: int, + hidden_sizes: list[int], + rank: int, + device: torch.device, + physical_to_logical_mapping: torch.Tensor, +) -> list[list[torch.Tensor]]: + """ + Create fake expert weights tensor for testing. + + Use `arange` to generate predictable weights values, based on logical + expert ID. + All replicas of the same logical expert should have the same weights. + + Args: + physical_to_logical_mapping: Shape (num_layers, num_local_experts) + mapping[layer, physical_pos] = logical_expert_id + """ + expert_weights = [] + + for layer in range(num_layers): + layer_weights = [] + for weight_idx, hidden_size in enumerate(hidden_sizes): + weight_tensor = torch.zeros( + num_local_experts, hidden_size, device=device, dtype=torch.float32 + ) + + for local_expert in range(num_local_experts): + # Get the logical expert ID for this physical expert + global_pos = rank * num_local_experts + local_expert + logical_expert_id = physical_to_logical_mapping[ + layer, global_pos + ].item() + + # Generate weights based on logical expert ID + # (so that all replicas of the same logical expert have the + # same weights) + base_value = logical_expert_id * 1000 + layer * 100 + weight_idx * 10 + weight_tensor[local_expert] = torch.arange( + base_value, + base_value + hidden_size, + device=device, + dtype=torch.float32, + ) + + layer_weights.append(weight_tensor) + expert_weights.append(layer_weights) + + return expert_weights + + +def create_redundancy_config( + num_logical_experts: int, + num_physical_experts: int, +) -> list[int]: + """Create a redundancy configuration.""" + redundancy_config = [1] * num_logical_experts + remaining = num_physical_experts - num_logical_experts + # Randomly assign the remaining physical experts to the logical experts + for _ in range(remaining): + redundancy_config[random.choice(range(num_logical_experts))] += 1 + return redundancy_config + + +def verify_expert_weights_after_shuffle( + expert_weights: list[list[torch.Tensor]], + new_indices: torch.Tensor, + hidden_sizes: list[int], + ep_rank: int, + num_local_experts: int, +) -> bool: + """Verify the weights after shuffling are correct.""" + num_layers = len(expert_weights) + ok = True + + for layer in range(num_layers): + for weight_idx, hidden_size in enumerate(hidden_sizes): + weight_tensor = expert_weights[layer][weight_idx] + + for local_expert in range(num_local_experts): + # Calculate the global expert ID for this local expert + global_pos = ep_rank * num_local_experts + local_expert + expected_logical_expert = new_indices[layer, global_pos].item() + + # Check if the weights are correct + actual_weights = weight_tensor[local_expert] + expected_base = ( + expected_logical_expert * 1000 + layer * 100 + weight_idx * 10 + ) + expected_weights = torch.arange( + expected_base, + expected_base + hidden_size, + device=actual_weights.device, + dtype=actual_weights.dtype, + ) + + if not torch.equal(actual_weights, expected_weights): + ok = False + actual_head = actual_weights[:8].detach().cpu().tolist() + expected_head = expected_weights[:8].detach().cpu().tolist() + print( + "verify_expert_weights_after_shuffle failed: " + f"rank={ep_rank}, " + f"layer={layer}, weight_idx={weight_idx}, " + f"local_expert={local_expert}, " + f"expected_logical_expert={expected_logical_expert}, " + f"actual_head={actual_head}, expected_head={expected_head}", + flush=True, + ) + + return ok + + +def verify_redundant_experts_have_same_weights( + expert_weights: list[list[torch.Tensor]], + indices: torch.Tensor, + hidden_sizes: list[int], + ep_rank: int, + world_size: int, + num_local_experts: int, +) -> bool: + """ + Verify that all replicas of the same logical expert have the same weights. + """ + num_layers = len(expert_weights) + total_physical_experts = world_size * num_local_experts + + ok = True + for layer in range(num_layers): + # Collect weights for all physical experts for each weight matrix + all_weights: list[torch.Tensor] = [] + + for weight_idx, hidden_size in enumerate(hidden_sizes): + # Create tensor to store all expert weights + # Shape: [total_physical_experts, hidden_size] + gathered_weights = torch.zeros( + total_physical_experts, + hidden_size, + device=expert_weights[layer][weight_idx].device, + dtype=expert_weights[layer][weight_idx].dtype, + ) + + # Use all_gather to collect expert weights from current node + # expert_weights[layer][weight_idx] shape: + # [num_local_experts, hidden_size] + local_weights = expert_weights[layer][ + weight_idx + ] # [num_local_experts, hidden_size] + + # Split tensor along dim 0 into a list for all_gather + gathered_weights_list = torch.chunk(gathered_weights, world_size, dim=0) + + torch.distributed.all_gather( + # Output list: each element corresponds to one rank's weights + list(gathered_weights_list), + local_weights, # Input: current rank's local weights + ) + + all_weights.append(gathered_weights) + + # Verify that all replicas of the same logical expert have the same + # weights + logical_expert_weights: dict[int, dict[int, torch.Tensor]] = {} + + for physical_pos in range(total_physical_experts): + logical_expert_id = int(indices[layer, physical_pos].item()) + + if logical_expert_id not in logical_expert_weights: + # First time encountering this logical expert, save its weights + logical_expert_weights[logical_expert_id] = { + weight_idx: all_weights[weight_idx][physical_pos] + for weight_idx in range(len(hidden_sizes)) + } + else: + # Verify that current physical expert's weights match the + # previously saved logical expert weights + for weight_idx in range(len(hidden_sizes)): + if not torch.equal( + all_weights[weight_idx][physical_pos], + logical_expert_weights[logical_expert_id][weight_idx], + ): + ok = False + actual_head = ( + all_weights[weight_idx][physical_pos][:8] + .detach() + .cpu() + .tolist() + ) + reference_head = ( + logical_expert_weights[logical_expert_id][weight_idx][:8] + .detach() + .cpu() + .tolist() + ) + print( + "verify_redundant_experts_have_same_weights failed: " + f"rank={ep_rank}, " + f"layer={layer}, weight_idx={weight_idx}, " + f"logical_expert={logical_expert_id}, " + f"physical_pos={physical_pos}, " + f"actual_head={actual_head}, " + f"reference_head={reference_head}", + flush=True, + ) + + return ok + + +def assert_verification_synced(local_ok: bool, msg: str) -> None: + ok_tensor = torch.tensor([1 if local_ok else 0], device="cuda", dtype=torch.int32) + torch.distributed.all_reduce(ok_tensor, op=torch.distributed.ReduceOp.MIN) + assert bool(ok_tensor.item()), msg + + +def create_eplb_communicator_or_raise( + *, group_coordinator, backend, expert_weights, expert_buffer +): + try: + return create_eplb_communicator( + group_coordinator=group_coordinator, + backend=backend, + expert_weights=expert_weights, + expert_buffer=expert_buffer, + ) + except Exception as exc: + raise RuntimeError( + f"Failed to create EPLB communicator for backend={backend}: {exc}" + ) from exc + + +def _test_async_transfer_layer_without_mtp_worker( + env, + world_size: int, + num_layers: int, + num_local_experts: int, + num_logical_experts: int, + eplb_communicator: str, +) -> None: + set_env_vars_and_device(env) + + vllm_config = VllmConfig() + vllm_config.parallel_config.tensor_parallel_size = world_size + + with set_current_vllm_config(vllm_config): + ensure_model_parallel_initialized( + tensor_model_parallel_size=world_size, pipeline_model_parallel_size=1 + ) + + ep_group_coordinator = get_tp_group() + ep_group = ep_group_coordinator.device_group + ep_rank = torch.distributed.get_rank() + device = torch.device(f"cuda:{ep_rank}") + + total_physical_experts = world_size * num_local_experts + hidden_sizes = [16, 32] + + redundancy_config = create_redundancy_config( + num_logical_experts, + total_physical_experts, + ) + old_indices = create_expert_indices_with_redundancy( + num_layers, + num_logical_experts, + total_physical_experts, + redundancy_config, + ) + + new_redundancy_config = create_redundancy_config( + num_logical_experts, + total_physical_experts, + ) + new_indices = create_expert_indices_with_redundancy( + num_layers, + num_logical_experts, + total_physical_experts, + new_redundancy_config, + ) + + expert_weights = create_expert_weights( + num_layers, + num_local_experts, + hidden_sizes, + ep_rank, + device, + old_indices, + ) + old_indices_cpu = old_indices.cpu() + new_indices_cpu = new_indices.cpu() + + expert_buffer = [torch.empty_like(w) for w in expert_weights[0]] + cuda_stream = torch.cuda.Stream(device=device) + + communicator = create_eplb_communicator_or_raise( + group_coordinator=ep_group_coordinator, + backend=eplb_communicator, + expert_weights=expert_weights, + expert_buffer=expert_buffer, + ) + communicator.set_stream(cuda_stream) + + for layer_idx in range(num_layers): + transfer_metadata = transfer_layer( + old_layer_indices=old_indices_cpu[layer_idx], + new_layer_indices=new_indices_cpu[layer_idx], + expert_weights=expert_weights[layer_idx], + expert_weights_buffer=expert_buffer, + ep_group=ep_group, + communicator=communicator, + cuda_stream=cuda_stream, + layer_idx=layer_idx, + ) + cuda_stream.synchronize() + move_from_buffer( + expert_weights=expert_weights[layer_idx], + expert_weights_buffers=expert_buffer, + transfer_metadata=transfer_metadata, + new_indices=new_indices_cpu[layer_idx].numpy(), + ep_rank=ep_rank, + ) + + local_ok = verify_expert_weights_after_shuffle( + expert_weights, + new_indices, + hidden_sizes, + ep_rank, + num_local_experts, + ) + local_ok = ( + verify_redundant_experts_have_same_weights( + expert_weights, + new_indices, + hidden_sizes, + ep_rank, + world_size, + num_local_experts, + ) + and local_ok + ) + assert_verification_synced( + local_ok, + "Async transfer verification failed on at least one rank. " + "See logs for details.", + ) + + +def _test_rearrange_expert_weights_with_redundancy( + env, + world_size, + num_layers, + num_local_experts, + num_logical_experts, + eplb_communicator: str, +) -> None: + # Initialize model parallel (using tensor parallel as an entrypoint + # to expert parallel) + set_env_vars_and_device(env) + + vllm_config = VllmConfig() + vllm_config.parallel_config.tensor_parallel_size = world_size + + with set_current_vllm_config(vllm_config): + ensure_model_parallel_initialized( + tensor_model_parallel_size=world_size, pipeline_model_parallel_size=1 + ) + + ep_group_coordinator = get_tp_group() + ep_group = ep_group_coordinator.cpu_group + ep_rank = torch.distributed.get_rank() + device = torch.device(f"cuda:{ep_rank}") + + # Test parameters + total_physical_experts = world_size * num_local_experts + hidden_sizes = [32, 64] # Two different weight matrices + + # Create old expert indices (with redundancy) + redundancy_config = create_redundancy_config( + num_logical_experts, total_physical_experts + ) + + old_indices = create_expert_indices_with_redundancy( + num_layers, + num_logical_experts, + total_physical_experts, + redundancy_config, + ) + + # Create new expert indices (with redundancy) + new_redundancy_config = create_redundancy_config( + num_logical_experts, total_physical_experts + ) + new_indices = create_expert_indices_with_redundancy( + num_layers, + num_logical_experts, + total_physical_experts, + new_redundancy_config, + ) + + # Create expert weights + expert_weights = create_expert_weights( + num_layers, num_local_experts, hidden_sizes, ep_rank, device, old_indices + ) + + expert_buffer = [torch.empty_like(w) for w in expert_weights[0]] + communicator = create_eplb_communicator_or_raise( + group_coordinator=ep_group_coordinator, + backend=eplb_communicator, + expert_weights=expert_weights, + expert_buffer=expert_buffer, + ) + + # Execute weight rearrangement + rearrange_expert_weights_inplace( + old_indices, + new_indices, + expert_weights, + expert_buffer, + ep_group, + communicator, + ) + + # Verify the rearrangement result + local_ok = verify_expert_weights_after_shuffle( + expert_weights, + new_indices, + hidden_sizes, + ep_rank, + num_local_experts, + ) + + local_ok = ( + verify_redundant_experts_have_same_weights( + expert_weights, + new_indices, + hidden_sizes, + ep_rank, + world_size, + num_local_experts, + ) + and local_ok + ) + assert_verification_synced( + local_ok, + "Rearrange verification failed on at least one rank. See logs for details.", + ) + + +@pytest.mark.parametrize( + "world_size,num_layers,num_local_experts,num_logical_experts", + [ + # 2 GPU, 2 experts per GPU + # 3 logical experts, 4 physical experts, 1 redundant experts + (2, 1, 2, 3), + # 2 GPU, 3 experts per GPU + # 4 logical experts, 6 physical experts, 2 redundant experts + (2, 2, 3, 4), + # 2 GPU, 8 experts per GPU + # 16 logical experts, 16 physical experts, 0 redundant experts + (2, 4, 8, 16), + # 4 GPU, 2 experts per GPU + # 6 logical experts, 8 physical experts, 2 redundant experts + (4, 1, 2, 6), + # 4 GPU, 2 experts per GPU + # 5 logical experts, 8 physical experts, 3 redundant experts + (4, 2, 2, 5), + # 4 GPU, 8 experts per GPU + # 16 logical experts, 32 physical experts, 16 redundant experts + (4, 8, 8, 16), + ], +) +@pytest.mark.parametrize( + "eplb_communicator", ["torch_nccl", "torch_gloo", "pynccl", "nixl"] +) +def test_rearrange_expert_weights_with_redundancy( + world_size, + num_layers, + num_local_experts, + num_logical_experts, + eplb_communicator, +): + """Test the functionality of rearranging expert weights with redundancy.""" + + if eplb_communicator == "nixl" and not has_nixl(): + pytest.skip("NIXL is not available") + if torch.accelerator.device_count() < world_size: + pytest.skip(f"Need at least {world_size} GPUs to run the test") + distributed_run( + _test_rearrange_expert_weights_with_redundancy, + world_size, + num_layers, + num_local_experts, + num_logical_experts, + eplb_communicator, + ) + + +def _test_rearrange_expert_weights_no_change(env, world_size) -> None: + set_env_vars_and_device(env) + + vllm_config = VllmConfig() + vllm_config.parallel_config.tensor_parallel_size = world_size + + with set_current_vllm_config(vllm_config): + ensure_model_parallel_initialized( + tensor_model_parallel_size=world_size, pipeline_model_parallel_size=1 + ) + + ep_group_coordinator = get_tp_group() + ep_group = ep_group_coordinator.cpu_group + ep_rank = torch.distributed.get_rank() + device = torch.device(f"cuda:{ep_rank}") + + num_layers = 2 + num_local_experts = 2 + total_physical_experts = world_size * num_local_experts + num_logical_experts = total_physical_experts // 2 # Some redundancy + hidden_sizes = [32, 64] + + # Create redundancy configuration + redundancy_config = [2] * num_logical_experts + + # Same indices - no change + indices = create_expert_indices_with_redundancy( + num_layers, num_logical_experts, total_physical_experts, redundancy_config + ) + + expert_weights = create_expert_weights( + num_layers, num_local_experts, hidden_sizes, ep_rank, device, indices + ) + + # Save original weights + original_weights = [] + for layer_weights in expert_weights: + layer_copy = [] + for weight in layer_weights: + layer_copy.append(weight.clone()) + original_weights.append(layer_copy) + + expert_buffer = [torch.empty_like(w) for w in expert_weights[0]] + communicator = create_eplb_communicator_or_raise( + group_coordinator=ep_group_coordinator, + backend="torch_nccl", + expert_weights=expert_weights, + expert_buffer=expert_buffer, + ) + + # Execute rearrangement (should be no change) + rearrange_expert_weights_inplace( + indices, + indices, # Same indices + expert_weights, + expert_buffer, + ep_group, + communicator, + ) + + # Verify that the weights have not changed + local_ok = True + for layer in range(num_layers): + for weight_idx in range(len(hidden_sizes)): + if not torch.equal( + expert_weights[layer][weight_idx], + original_weights[layer][weight_idx], + ): + local_ok = False + print( + "test_rearrange_expert_weights_no_change failed: " + f"layer={layer}, weight_idx={weight_idx}", + flush=True, + ) + assert_verification_synced( + local_ok, + "No-change EPLB verification failed on at least one rank.", + ) + + +@pytest.mark.parametrize( + "world_size,num_layers,num_local_experts,num_logical_experts", + [ + (2, 2, 2, 3), + ], +) +@pytest.mark.parametrize("eplb_communicator", ["torch_gloo", "nixl"]) +def test_async_transfer_layer_without_mtp( + world_size: int, + num_layers: int, + num_local_experts: int, + num_logical_experts: int, + eplb_communicator: str, +): + """Exercise async EPLB transfer path without MTP/spec decode.""" + + if eplb_communicator == "nixl" and not has_nixl(): + pytest.skip("NIXL is not available") + if torch.accelerator.device_count() < world_size: + pytest.skip(f"Need at least {world_size} GPUs to run the test") + + distributed_run( + _test_async_transfer_layer_without_mtp_worker, + world_size, + num_layers, + num_local_experts, + num_logical_experts, + eplb_communicator, + ) + + +@pytest.mark.parametrize("world_size", [2, 4]) +def test_rearrange_expert_weights_no_change(world_size): + """ + Test that when the indices do not change, the weights should remain + unchanged. + """ + + if torch.accelerator.device_count() < world_size: + pytest.skip(f"Need at least {world_size} GPUs to run the test") + distributed_run( + _test_rearrange_expert_weights_no_change, + world_size, + ) + + +def _test_rearrange_expert_weights_profile_mode(env, world_size) -> None: + set_env_vars_and_device(env) + + vllm_config = VllmConfig() + vllm_config.parallel_config.tensor_parallel_size = world_size + + with set_current_vllm_config(vllm_config): + ensure_model_parallel_initialized( + tensor_model_parallel_size=world_size, pipeline_model_parallel_size=1 + ) + + ep_group_coordinator = get_tp_group() + ep_group = ep_group_coordinator.cpu_group + ep_rank = torch.distributed.get_rank() + device = torch.device(f"cuda:{ep_rank}") + + num_layers = 1 + num_local_experts = 2 + total_physical_experts = world_size * num_local_experts + num_logical_experts = total_physical_experts // 2 + hidden_sizes = [32] + + # Create different index distributions + old_redundancy = create_redundancy_config( + num_logical_experts, total_physical_experts + ) + new_redundancy = create_redundancy_config( + num_logical_experts, total_physical_experts + ) + + old_indices = create_expert_indices_with_redundancy( + num_layers, num_logical_experts, total_physical_experts, old_redundancy + ) + new_indices = create_expert_indices_with_redundancy( + num_layers, num_logical_experts, total_physical_experts, new_redundancy + ) + + expert_weights = create_expert_weights( + num_layers, num_local_experts, hidden_sizes, ep_rank, device, old_indices + ) + + # Save original weights + original_weights = [] + for layer_weights in expert_weights: + layer_copy = [] + for weight in layer_weights: + layer_copy.append(weight.clone()) + original_weights.append(layer_copy) + + expert_buffer = [torch.empty_like(w) for w in expert_weights[0]] + communicator = create_eplb_communicator_or_raise( + group_coordinator=ep_group_coordinator, + backend="torch_nccl", + expert_weights=expert_weights, + expert_buffer=expert_buffer, + ) + + # Execute profile mode rearrangement + rearrange_expert_weights_inplace( + old_indices, + new_indices, + expert_weights, + expert_buffer, + ep_group, + communicator, + is_profile=True, + ) + + # In profile mode, the weights should remain unchanged + local_ok = True + for layer in range(num_layers): + for weight_idx in range(len(hidden_sizes)): + if not torch.equal( + expert_weights[layer][weight_idx], + original_weights[layer][weight_idx], + ): + local_ok = False + print( + "test_rearrange_expert_weights_profile_mode failed: " + f"layer={layer}, weight_idx={weight_idx}", + flush=True, + ) + assert_verification_synced( + local_ok, + "Profile-mode EPLB verification failed on at least one rank.", + ) + + +@pytest.mark.parametrize("world_size", [2, 4]) +def test_rearrange_expert_weights_profile_mode(world_size): + """Test profile mode (should not copy actual weights)""" + + if torch.accelerator.device_count() < world_size: + pytest.skip(f"Need at least {world_size} GPUs to run the test") + distributed_run( + _test_rearrange_expert_weights_profile_mode, + world_size, + ) + + +def _test_nixl_deferred_init_worker( + env, + world_size: int, + num_layers: int, + num_local_experts: int, + num_logical_experts: int, +) -> None: + """Exercise NixlEplbCommunicator with defer_remote_setup=True (elastic EP path).""" + from vllm.distributed.eplb.eplb_communicator import NixlEplbCommunicator + + set_env_vars_and_device(env) + + vllm_config = VllmConfig() + vllm_config.parallel_config.tensor_parallel_size = world_size + + with set_current_vllm_config(vllm_config): + ensure_model_parallel_initialized( + tensor_model_parallel_size=world_size, pipeline_model_parallel_size=1 + ) + + ep_group_coordinator = get_tp_group() + ep_group = ep_group_coordinator.cpu_group + ep_rank = torch.distributed.get_rank() + device = torch.device(f"cuda:{ep_rank}") + + total_physical_experts = world_size * num_local_experts + hidden_sizes = [32, 64] + + redundancy_config = create_redundancy_config( + num_logical_experts, total_physical_experts + ) + old_indices = create_expert_indices_with_redundancy( + num_layers, + num_logical_experts, + total_physical_experts, + redundancy_config, + ) + + new_redundancy_config = create_redundancy_config( + num_logical_experts, total_physical_experts + ) + new_indices = create_expert_indices_with_redundancy( + num_layers, + num_logical_experts, + total_physical_experts, + new_redundancy_config, + ) + + expert_weights = create_expert_weights( + num_layers, num_local_experts, hidden_sizes, ep_rank, device, old_indices + ) + + expert_buffer = [torch.empty_like(w) for w in expert_weights[0]] + + communicator = NixlEplbCommunicator( + cpu_group=ep_group_coordinator.cpu_group, + all_expert_weights=expert_weights, + expert_buffer=expert_buffer, + defer_remote_setup=True, + ) + assert not communicator._remote_state_initialized + + rearrange_expert_weights_inplace( + old_indices, + new_indices, + expert_weights, + expert_buffer, + ep_group, + communicator, + ) + + assert communicator._remote_state_initialized + + local_ok = verify_expert_weights_after_shuffle( + expert_weights, + new_indices, + hidden_sizes, + ep_rank, + num_local_experts, + ) + + local_ok = ( + verify_redundant_experts_have_same_weights( + expert_weights, + new_indices, + hidden_sizes, + ep_rank, + world_size, + num_local_experts, + ) + and local_ok + ) + assert_verification_synced( + local_ok, + "Deferred NIXL init verification failed on at least one rank.", + ) + + +@pytest.mark.skipif(not has_nixl(), reason="NIXL is not available") +@pytest.mark.parametrize( + "world_size,num_layers,num_local_experts,num_logical_experts", + [(2, 2, 3, 4)], +) +def test_nixl_deferred_init( + world_size, + num_layers, + num_local_experts, + num_logical_experts, +): + """Test NixlEplbCommunicator with defer_remote_setup=True (elastic EP path).""" + + if torch.accelerator.device_count() < world_size: + pytest.skip(f"Need at least {world_size} GPUs to run the test") + distributed_run( + _test_nixl_deferred_init_worker, + world_size, + num_layers, + num_local_experts, + num_logical_experts, + ) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_eplb_fused_moe_layer.py b/asian_ai/performance/vLLM/tests/distributed/test_eplb_fused_moe_layer.py new file mode 100644 index 0000000000000000000000000000000000000000..af982910b9b5e53b2e7062c2b20a02ef279d1010 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_eplb_fused_moe_layer.py @@ -0,0 +1,295 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +# Test that the interaction between EPLB and FusedMoE Layer is okay + +from dataclasses import dataclass + +import pytest +import torch + +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.distributed.eplb.eplb_communicator import create_eplb_communicator +from vllm.distributed.eplb.rebalance_execute import rearrange_expert_weights_inplace +from vllm.distributed.parallel_state import ( + ensure_model_parallel_initialized, + get_eplb_group, + get_tp_group, +) +from vllm.model_executor.layers.fused_moe.layer import FusedMoE + +from .eplb_utils import distributed_run, set_env_vars_and_device + + +@dataclass +class TestConfig: + num_layers: int + num_experts: int + num_local_experts: int + num_topk: int + hidden_size: int + intermediate_size: int + weight_dtype: torch.dtype + weight_scale_dtype: torch.dtype | None + column_major_scales: bool + + +def make_expert_weights( + layer_idx: int, + global_expert_idx: int, + global_num_experts: int, + tensor_shape: tuple[int, ...], + tensor_dtype: torch.dtype, + tensor_device: torch.device, + is_column_major: bool, +) -> torch.Tensor: + assert len(tensor_shape) == 2 + + if is_column_major: + tensor_shape = (tensor_shape[1], tensor_shape[0]) + + x = torch.empty(tensor_shape, dtype=tensor_dtype, device=tensor_device) + value_offset = (layer_idx * global_num_experts + global_expert_idx) * x.numel() + x.view(-1).copy_( + torch.arange( + value_offset, + value_offset + x.numel(), + dtype=tensor_dtype, + device=tensor_device, + ) + ) + + if is_column_major: + x = torch.transpose(x, 1, 0) + assert not x.is_contiguous() + return x + + +def make_fused_moe_layer( + rank: int, + layer_idx: int, + test_config: TestConfig, +) -> FusedMoE: + fml = FusedMoE( + num_experts=test_config.num_experts, + top_k=test_config.num_topk, + hidden_size=test_config.hidden_size, + intermediate_size=test_config.intermediate_size, + prefix=f"dummy_layer_{layer_idx}", + activation="silu", + params_dtype=test_config.weight_dtype, + ) + re = fml.routed_experts + + device = torch.device(f"cuda:{rank}") + + from functools import partial + + _make_expert_weights = partial( + make_expert_weights, + layer_idx=layer_idx, + global_num_experts=test_config.num_experts, + tensor_device=device, + ) + + assert isinstance(re.w13_weight.data, torch.Tensor) + assert isinstance(re.w2_weight.data, torch.Tensor) + re.w13_weight.data = re.w13_weight.data.to(device=device) + re.w2_weight.data = re.w2_weight.data.to(device=device) + w13_weight = re.w13_weight.data + w2_weight = re.w2_weight.data + assert w13_weight.size(0) == test_config.num_local_experts + for i in range(test_config.num_local_experts): + g_i = rank * test_config.num_local_experts + i + w13_weight_e = w13_weight[i] + w2_weight_e = w2_weight[i] + w13_weight_e.copy_( + _make_expert_weights( + global_expert_idx=g_i, + tensor_shape=w13_weight_e.shape, + tensor_dtype=w13_weight_e.dtype, + is_column_major=False, + ) + ) + w2_weight_e.copy_( + _make_expert_weights( + global_expert_idx=g_i, + tensor_shape=w2_weight_e.shape, + tensor_dtype=w2_weight_e.dtype, + is_column_major=False, + ) + ) + + block_size = 16 + + def block_quant_scales_shape( + shape: tuple[int, ...], is_column_major: bool + ) -> tuple[int, ...]: + assert len(shape) == 3 + if not is_column_major: + return (shape[0], shape[1] // block_size, shape[2] // block_size) + else: + return (shape[0], shape[2] // block_size, shape[1] // block_size) + + is_column_major = test_config.column_major_scales + w13_weight_scale_inv = torch.empty( + block_quant_scales_shape(w13_weight.shape, is_column_major), + dtype=test_config.weight_dtype, + device=device, + ) + w2_weight_scale_inv = torch.empty( + block_quant_scales_shape(w2_weight.shape, is_column_major), + dtype=test_config.weight_dtype, + device=device, + ) + + for i in range(test_config.num_local_experts): + g_i = rank * test_config.num_local_experts + i + w13_s_e = w13_weight_scale_inv[i] + w2_s_e = w2_weight_scale_inv[i] + w13_s_e.copy_( + _make_expert_weights( + global_expert_idx=g_i, + tensor_shape=w13_s_e.shape, + tensor_dtype=w13_s_e.dtype, + # Fill data in row-major and then + # transpose if test_config requires col-major. + is_column_major=False, + ) + ) + w2_s_e.copy_( + _make_expert_weights( + global_expert_idx=g_i, + tensor_shape=w2_s_e.shape, + tensor_dtype=w2_s_e.dtype, + is_column_major=False, + ) + ) + if is_column_major: + w13_weight_scale_inv = torch.transpose(w13_weight_scale_inv, 1, 2) + w2_weight_scale_inv = torch.transpose(w2_weight_scale_inv, 1, 2) + assert not w13_weight_scale_inv.is_contiguous() + assert not w2_weight_scale_inv.is_contiguous() + + # Add scales to the parameter list + re.w13_weight_scale_inv = torch.nn.Parameter( + w13_weight_scale_inv, requires_grad=False + ) + re.w2_weight_scale_inv = torch.nn.Parameter( + w2_weight_scale_inv, requires_grad=False + ) + + return fml + + +def _test_eplb_fml(env, world_size: int, test_config: TestConfig): + # Initialize model parallel (using tensor parallel as an entrypoint + # to expert parallel) + set_env_vars_and_device(env) + + vllm_config = VllmConfig() + vllm_config.parallel_config.tensor_parallel_size = world_size + vllm_config.parallel_config.enable_expert_parallel = True + + with set_current_vllm_config(vllm_config): + ensure_model_parallel_initialized( + tensor_model_parallel_size=world_size, pipeline_model_parallel_size=1 + ) + + ep_group = get_tp_group().cpu_group + ep_rank = torch.distributed.get_rank() + + fml_layers = [ + make_fused_moe_layer(ep_rank, layer_idx, test_config) + for layer_idx in range(test_config.num_layers) + ] + rank_expert_weights = [fml.get_expert_weights() for fml in fml_layers] + + indices = torch.zeros( + test_config.num_layers, test_config.num_experts, dtype=torch.long + ) + for lidx in range(test_config.num_layers): + indices[lidx] = torch.Tensor(range(test_config.num_experts)) + + shuffled_indices = torch.zeros_like(indices) + for lidx in range(test_config.num_layers): + shuffled_indices[lidx] = torch.randperm(test_config.num_experts) + + expert_buffer = [torch.empty_like(w) for w in rank_expert_weights[0]] + communicator = create_eplb_communicator( + group_coordinator=get_eplb_group(), + backend="torch_nccl", + expert_weights=rank_expert_weights, + expert_buffer=expert_buffer, + ) + rearrange_expert_weights_inplace( + indices, + shuffled_indices, + rank_expert_weights, + expert_buffer, + ep_group, + communicator, + ) + + num_local_experts = test_config.num_local_experts + num_global_experts = test_config.num_experts + for lidx, fml in enumerate(fml_layers): + for name, w in fml.named_parameters(): + for e in range(num_local_experts): + g_e = shuffled_indices[lidx][ep_rank * num_local_experts + e] + ref = make_expert_weights( + layer_idx=lidx, + global_expert_idx=int(g_e.item()), + global_num_experts=num_global_experts, + tensor_shape=w[e].shape, + tensor_dtype=w[e].dtype, + tensor_device=w[e].device, + is_column_major=not w[e].is_contiguous(), + ) + assert w[e].shape == ref.shape and w[e].stride() == ref.stride(), ( + f"w[{e}] {w[e].size()} {w[e].stride()} vs " + f"ref {ref.size()} {ref.stride()}" + ) + torch.testing.assert_close(w[e], ref) + + +@pytest.mark.parametrize("world_size", [2]) +@pytest.mark.parametrize("num_layers", [4]) +@pytest.mark.parametrize("num_experts", [16]) +@pytest.mark.parametrize("hidden_size", [256]) +@pytest.mark.parametrize("intermediate_size", [256]) +@pytest.mark.parametrize("column_major_scales", [True, False]) +def test_eplb_fml( + world_size: int, + num_layers: int, + num_experts: int, + hidden_size: int, + intermediate_size: int, + column_major_scales: bool, +): + if torch.accelerator.device_count() < world_size: + pytest.skip(f"Need at least {world_size} GPUs to run the test") + + num_local_experts = num_experts // world_size + num_topk = 4 + # The dtypes are fine as we are essentially just checking data-copies + weight_dtype = torch.bfloat16 + weight_scale_dtype = torch.bfloat16 + + test_config = TestConfig( + num_layers=num_layers, + num_experts=num_experts, + num_local_experts=num_local_experts, + num_topk=num_topk, + hidden_size=hidden_size, + intermediate_size=intermediate_size, + weight_dtype=weight_dtype, + weight_scale_dtype=weight_scale_dtype, + column_major_scales=column_major_scales, + ) + + distributed_run( + _test_eplb_fml, + world_size, + test_config, + ) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_eplb_fused_moe_layer_dep_nvfp4.py b/asian_ai/performance/vLLM/tests/distributed/test_eplb_fused_moe_layer_dep_nvfp4.py new file mode 100644 index 0000000000000000000000000000000000000000..bdd4a90a3076eaf85a6d25689a0aed786079ebef --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_eplb_fused_moe_layer_dep_nvfp4.py @@ -0,0 +1,292 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +# Test that the interaction between EPLB and FusedMoE Layer is okay for DP w/ NVFP4 + +from dataclasses import dataclass + +import pytest +import torch + +from tests.kernels.moe.utils import make_test_quant_config +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.distributed.eplb.eplb_communicator import create_eplb_communicator +from vllm.distributed.eplb.eplb_state import EplbLayerState +from vllm.distributed.eplb.rebalance_execute import rearrange_expert_weights_inplace +from vllm.distributed.parallel_state import ( + ensure_model_parallel_initialized, + get_dp_group, + get_eplb_group, +) +from vllm.forward_context import set_forward_context +from vllm.model_executor.layers.fused_moe.layer import FusedMoE +from vllm.model_executor.layers.quantization.modelopt import ( + ModelOptNvFp4Config, + ModelOptNvFp4FusedMoE, +) + +from .eplb_utils import distributed_run, set_env_vars_and_device + + +@dataclass +class TestConfig: + num_layers: int + num_experts: int + num_local_experts: int + num_topk: int + hidden_size: int + intermediate_size: int + num_tokens: int + moe_backend: str + + +def make_fused_moe_layer( + rank: int, + layer_idx: int, + test_config: TestConfig, +) -> FusedMoE: + quant_config = None + + device = torch.device(f"cuda:{rank}") + + quant_config = ModelOptNvFp4Config( + is_checkpoint_nvfp4_serialized=True, + kv_cache_quant_algo=None, + exclude_modules=[], + ) + + fml = FusedMoE( + num_experts=test_config.num_experts, + top_k=test_config.num_topk, + hidden_size=test_config.hidden_size, + intermediate_size=test_config.intermediate_size, + prefix=f"dummy_layer_{layer_idx}", + activation="silu", + params_dtype=torch.bfloat16, + quant_config=quant_config, + ) + + nvfp4_fused_moe = ModelOptNvFp4FusedMoE(quant_config, fml) + nvfp4_fused_moe.create_weights( + fml, + test_config.num_local_experts, + test_config.hidden_size, + test_config.intermediate_size, + params_dtype=torch.uint8, + global_num_experts=test_config.num_experts, + ) + + fml = fml.to(device) + re = fml.routed_experts + w1_q, w2_q, quant_config = make_test_quant_config( + test_config.num_local_experts, + test_config.intermediate_size, + test_config.hidden_size, + in_dtype=torch.bfloat16, + quant_dtype="nvfp4", + block_shape=None, + per_act_token_quant=False, + ) + + re.w13_weight.data = w1_q + re.w2_weight.data = w2_q + + re.w2_input_scale.data = torch.randn_like(re.w2_input_scale.data) / 5 + re.w13_input_scale.data = torch.randn_like(re.w13_input_scale.data) / 5 + re.w2_weight_scale_2.data = torch.randn_like(re.w2_weight_scale_2.data) / 5 + re.w13_weight_scale_2.data = torch.randn_like(re.w13_weight_scale_2.data) / 5 + re.w2_weight_scale.data = ( + torch.randn(re.w2_weight_scale.data.shape, device=device) / 5 + ).to(re.w2_weight_scale.data.dtype) + re.w13_weight_scale.data = ( + torch.randn(re.w13_weight_scale.data.shape, device=device) / 5 + ).to(re.w13_weight_scale.data.dtype) + + nvfp4_fused_moe.process_weights_after_loading(re) + + fml.maybe_init_modular_kernel() + + return fml + + +def _test_eplb_fml(env, world_size: int, test_config: TestConfig): + set_env_vars_and_device(env) + + vllm_config = VllmConfig() + vllm_config.parallel_config.data_parallel_size = world_size + vllm_config.parallel_config.enable_expert_parallel = True + vllm_config.kernel_config.moe_backend = test_config.moe_backend + + with set_current_vllm_config(vllm_config): + ensure_model_parallel_initialized( + tensor_model_parallel_size=1, pipeline_model_parallel_size=1 + ) + + ep_group = get_dp_group().cpu_group + ep_rank = torch.distributed.get_rank() + + device = torch.device(f"cuda:{ep_rank}") + + fml_layers = [ + make_fused_moe_layer(ep_rank, layer_idx, test_config).to(device) + for layer_idx in range(test_config.num_layers) + ] + rank_expert_weights = [fml.get_expert_weights() for fml in fml_layers] + + hidden_states = [] + router_logits = [] + for layer_idx in range(test_config.num_layers): + hidden_states.append( + torch.randn( + (test_config.num_tokens, test_config.hidden_size), + dtype=torch.bfloat16, + device=device, + ) + ) + router_logits.append( + torch.randn( + (test_config.num_tokens, test_config.num_experts), + dtype=torch.bfloat16, + device=device, + ) + ) + + out_before_shuffle = [] + with set_forward_context( + {}, + num_tokens=test_config.num_tokens, + num_tokens_across_dp=torch.tensor( + [test_config.num_tokens] * world_size, device="cpu", dtype=torch.int + ), + vllm_config=vllm_config, + ): + for lidx, fml in enumerate(fml_layers): + out_before_shuffle.append( + fml(hidden_states[lidx].clone(), router_logits[lidx].clone()) + ) + + indices = torch.zeros( + test_config.num_layers, test_config.num_experts, dtype=torch.long + ) + for lidx in range(test_config.num_layers): + indices[lidx] = torch.Tensor(range(test_config.num_experts)) + + shuffled_indices = torch.zeros_like(indices) + for lidx in range(test_config.num_layers): + shuffled_indices[lidx] = torch.randperm(test_config.num_experts) + + expert_buffer = [torch.empty_like(w) for w in rank_expert_weights[0]] + communicator = create_eplb_communicator( + group_coordinator=get_eplb_group(), + backend="torch_nccl", + expert_weights=rank_expert_weights, + expert_buffer=expert_buffer, + ) + rearrange_expert_weights_inplace( + indices, + shuffled_indices, + rank_expert_weights, + expert_buffer, + ep_group, + communicator, + ) + + num_global_experts = test_config.num_experts + + logical_to_physical_map_list = [] + for lidx, fml in enumerate(fml_layers): + physical_to_logical_map = shuffled_indices[lidx].to(device) + logical_to_physical_map = torch.empty( + (num_global_experts,), dtype=torch.int32, device=device + ) + logical_to_physical_map[physical_to_logical_map] = torch.arange( + 0, num_global_experts, dtype=torch.int32, device=device + ) + logical_to_physical_map_list.append( + logical_to_physical_map.reshape(num_global_experts, 1) + ) + + logical_to_physical_map = torch.stack(logical_to_physical_map_list) + + for lidx, fml in enumerate(fml_layers): + logical_replica_count = torch.ones( + (test_config.num_layers, num_global_experts), + dtype=torch.int32, + device=device, + ) + fml.eplb_state = EplbLayerState() + fml.set_eplb_state( + lidx, + torch.zeros( + (test_config.num_layers, num_global_experts), + dtype=torch.int32, + device=device, + ), + logical_to_physical_map, + logical_replica_count, + ) + fml.router.eplb_state.should_record_tensor = torch.ones( + (), dtype=torch.bool, device=device + ) + fml.router.eplb_state.num_unpadded_tokens_tensors = [ + torch.tensor(0, dtype=torch.int32, device=device) + ] + + out_after_shuffle = [] + with set_forward_context( + {}, + num_tokens=test_config.num_tokens, + num_tokens_across_dp=torch.tensor( + [test_config.num_tokens] * world_size, device="cpu", dtype=torch.int + ), + vllm_config=vllm_config, + ): + for lidx, fml in enumerate(fml_layers): + out_after_shuffle.append( + fml(hidden_states[lidx].clone(), router_logits[lidx].clone()) + ) + + for lidx in range(test_config.num_layers): + torch.testing.assert_close( + out_before_shuffle[lidx], out_after_shuffle[lidx], atol=1e-1, rtol=1e-1 + ) + + +@pytest.mark.parametrize("world_size", [2, 4]) +@pytest.mark.parametrize("num_layers", [8]) +@pytest.mark.parametrize("num_experts", [32]) +@pytest.mark.parametrize("hidden_size", [256]) +@pytest.mark.parametrize("intermediate_size", [256]) +@pytest.mark.parametrize("num_tokens", [256]) +@pytest.mark.parametrize("moe_backend", ["flashinfer_trtllm", "flashinfer_cutlass"]) +def test_eplb_fml( + world_size: int, + num_layers: int, + num_experts: int, + hidden_size: int, + intermediate_size: int, + num_tokens: int, + moe_backend: str, +): + if torch.accelerator.device_count() < world_size: + pytest.skip(f"Need at least {world_size} GPUs to run the test") + + num_local_experts = num_experts // world_size + num_topk = 4 + + test_config = TestConfig( + num_layers=num_layers, + num_experts=num_experts, + num_local_experts=num_local_experts, + num_topk=num_topk, + hidden_size=hidden_size, + intermediate_size=intermediate_size, + num_tokens=num_tokens, + moe_backend=moe_backend, + ) + + distributed_run( + _test_eplb_fml, + world_size, + test_config, + ) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_eplb_spec_decode.py b/asian_ai/performance/vLLM/tests/distributed/test_eplb_spec_decode.py new file mode 100644 index 0000000000000000000000000000000000000000..9799966ddbd28f79b8ddb3c1ab51870a2c32f90d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_eplb_spec_decode.py @@ -0,0 +1,141 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from __future__ import annotations + +import lm_eval +import pytest + +from tests.utils import large_gpu_mark +from vllm.platforms import current_platform + + +def get_model_args( + model_name: str, + spec_model_name: str | None, + spec_method: str, + tp_size: int, + model_max_len: int, + use_async: bool = True, +) -> dict: + speculative_config = { + "method": spec_method, + "model": spec_model_name, + "num_speculative_tokens": 1, + "max_model_len": model_max_len, + } + eplb_config = { + "num_redundant_experts": tp_size, + "window_size": 128, + "step_interval": 1024, + "log_balancedness": False, + "use_async": use_async, + } + model_args = { + "pretrained": model_name, + "dtype": "auto", + "add_bos_token": True, + "tensor_parallel_size": tp_size, + "gpu_memory_utilization": 0.7, + "speculative_config": speculative_config, + "enable_expert_parallel": True, + "eplb_config": eplb_config, + "enable_eplb": True, + "max_model_len": model_max_len, + } + return model_args + + +pytestmark = pytest.mark.skipif( + current_platform.is_rocm(), + reason="EPLB with Spec Decode is a work in progress on ROCm.", +) + + +@pytest.mark.parametrize( + "model_setup", + [ + pytest.param( + ("mtp", "Qwen/Qwen3-Next-80B-A3B-Instruct", None, 4, 0.86), + marks=large_gpu_mark(min_gb=80), + ), + pytest.param( + ( + "eagle", + "meta-llama/Llama-4-Scout-17B-16E-Instruct", + "morgendave/EAGLE-Llama-4-Scout-17B-16E-Instruct", + 4, + 0.92, + ), + marks=pytest.mark.skip(reason="Skipping due to CI OOM issues"), + ), + ], + ids=["qwen3_next_mtp", "llama4_eagle"], +) +def test_eplb_spec_decode( + monkeypatch: pytest.MonkeyPatch, + model_setup: tuple[str, str, str, int, float], +): + """ + Test the correctness of EPLB speculative decoding with GSM8K dataset. + Applicable to MoE models with mtp or eagle spec decode. + """ + method, model_name, spec_model_name, tp_size, expected_gsm8k_value = model_setup + + TASK = "gsm8k" + FILTER = "exact_match,strict-match" + RTOL = 0.03 + + model_args = get_model_args( + model_name=model_name, + spec_model_name=spec_model_name, + spec_method=method, + tp_size=tp_size, + model_max_len=4096, + ) + + results = lm_eval.simple_evaluate( + model="vllm", + model_args=model_args, + tasks=TASK, + batch_size=64, + num_fewshot=8, + ) + measured_value = results["results"][TASK][FILTER] + assert ( + measured_value - RTOL < expected_gsm8k_value + and measured_value + RTOL > expected_gsm8k_value + ), f"Expected: {expected_gsm8k_value} | Measured: {measured_value}" + + +@large_gpu_mark(min_gb=80) +def test_eplb_spec_decode_qwen3_next_mtp_async() -> None: + """ + Ensure async EPLB works with MTP speculative decoding for Qwen3-Next. + """ + + TASK = "gsm8k" + FILTER = "exact_match,strict-match" + RTOL = 0.03 + expected_gsm8k_value = 0.86 + + model_args = get_model_args( + model_name="Qwen/Qwen3-Next-80B-A3B-Instruct", + spec_model_name=None, + spec_method="mtp", + tp_size=4, + model_max_len=4096, + use_async=True, + ) + + results = lm_eval.simple_evaluate( + model="vllm", + model_args=model_args, + tasks=TASK, + batch_size=64, + num_fewshot=8, + ) + measured_value = results["results"][TASK][FILTER] + assert ( + measured_value - RTOL < expected_gsm8k_value + and measured_value + RTOL > expected_gsm8k_value + ), f"Expected: {expected_gsm8k_value} | Measured: {measured_value}" diff --git a/asian_ai/performance/vLLM/tests/distributed/test_eplb_utils.py b/asian_ai/performance/vLLM/tests/distributed/test_eplb_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..20a800fb5d8936e18f98f01c3dff237ca2849495 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_eplb_utils.py @@ -0,0 +1,154 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from unittest.mock import MagicMock + +import pytest +import torch + +from vllm.distributed.eplb.eplb_state import ( + _commit_eplb_maps, + _commit_eplb_maps_for_layer, +) + + +def _make_model_state( + phy2log: torch.Tensor, + log2phy: torch.Tensor, + logcnt: torch.Tensor, +) -> MagicMock: + """Build a minimal EplbModelState mock with only the three map tensors.""" + state = MagicMock() + state.physical_to_logical_map = phy2log + state.logical_to_physical_map = log2phy + state.logical_replica_count = logcnt + return state + + +def test_commit_eplb_maps_shape_change(): + """ + The normal path copies the physical_to_logical map in-place. When the number of + physical experts changes, the old map should be replaced entirely. + """ + num_layers, num_logical, num_physical = 2, 4, 6 + max_replicas = 3 + + # Build current state tensors + model_state = _make_model_state( + phy2log=torch.zeros(num_layers, num_physical, dtype=torch.long), + log2phy=torch.full( + (num_layers, num_logical, max_replicas), -1, dtype=torch.long + ), + logcnt=torch.zeros(num_layers, num_logical, dtype=torch.long), + ) + + # The new map has two more physical experts. These new physical experts will + # automatically map to the first two logical experts + new_phy2log_larger = ( + (torch.arange(num_physical + 2, dtype=torch.long) % num_logical) + .unsqueeze(0) + .expand(num_layers, -1) + ) + _commit_eplb_maps(model_state, new_phy2log_larger) + + # Check that the number of physical experts has been updated and that the values + # match + assert model_state.physical_to_logical_map.shape[1] == num_physical + 2 + assert torch.equal(model_state.physical_to_logical_map, new_phy2log_larger) + + +def test_commit_eplb_maps_for_layer_logical_padding(): + """ + Test that logical_to_physical_map is padded with -1 to fill the + pre-allocated slots when the new map has fewer replicas than the max. + """ + num_layers, num_logical, num_physical = 2, 4, 6 + max_replicas = 3 + + model_state = _make_model_state( + phy2log=torch.zeros(num_layers, num_physical, dtype=torch.long), + log2phy=torch.full( + (num_layers, num_logical, max_replicas), -1, dtype=torch.long + ), + logcnt=torch.zeros(num_layers, num_logical, dtype=torch.long), + ) + + new_phy2log = ( + (torch.arange(num_physical, dtype=torch.long) % num_logical) + .unsqueeze(0) + .expand(num_layers, -1) + .contiguous() + ) + layer = 0 + _commit_eplb_maps_for_layer(model_state, new_phy2log[layer], layer) + + assert torch.all(model_state.logical_to_physical_map[layer, :, 2] == -1) + + +def test_commit_eplb_maps_for_layer_shape_assert(): + """Test that a mismatched number of physical experts triggers an assertion error.""" + num_layers, num_logical, num_physical = 2, 4, 6 + + model_state = _make_model_state( + phy2log=torch.zeros(num_layers, num_physical, dtype=torch.long), + log2phy=torch.full((num_layers, num_logical, 2), -1, dtype=torch.long), + logcnt=torch.zeros(num_layers, num_logical, dtype=torch.long), + ) + bad_phy2log = torch.zeros(num_layers, num_physical + 1, dtype=torch.long) + with pytest.raises(AssertionError): + _commit_eplb_maps_for_layer(model_state, bad_phy2log, layer=0) + + +def test_commit_eplb_maps(): + """Test that all values are copied correctly into model_state.""" + num_layers, num_logical, num_physical, max_replicas = 2, 3, 4, 2 + + model_state = _make_model_state( + phy2log=torch.zeros(num_layers, num_physical, dtype=torch.long), + log2phy=torch.full( + (num_layers, num_logical, max_replicas), -1, dtype=torch.long + ), + logcnt=torch.zeros(num_layers, num_logical, dtype=torch.long), + ) + + new_phy2log = torch.tensor([[0, 1, 2, 0], [1, 2, 0, 1]], dtype=torch.long) + new_log2phy = torch.tensor( + [[[0, 3], [1, -1], [2, -1]], [[2, -1], [0, 3], [1, -1]]], dtype=torch.long + ) + new_logcnt = torch.tensor([[2, 1, 1], [1, 2, 1]], dtype=torch.long) + + _commit_eplb_maps(model_state, new_phy2log) + + assert torch.equal(model_state.physical_to_logical_map, new_phy2log) + assert torch.equal(model_state.logical_to_physical_map, new_log2phy) + assert torch.equal(model_state.logical_replica_count, new_logcnt) + + +def test_commit_eplb_maps_for_layer(): + """Test that only the target layer is updated""" + num_layers, num_logical, max_replicas = 2, 3, 2 + + original_phy2log = torch.tensor([[9, 9, 9, 9], [8, 8, 8, 8]], dtype=torch.long) + model_state = _make_model_state( + phy2log=original_phy2log.clone(), + log2phy=torch.full( + (num_layers, num_logical, max_replicas), -1, dtype=torch.long + ), + logcnt=torch.zeros(num_layers, num_logical, dtype=torch.long), + ) + + new_phy2log = torch.tensor([[0, 1, 2, 0], [1, 2, 0, 1]], dtype=torch.long) + new_log2phy = torch.tensor( + [[[0, 3], [1, -1], [2, -1]], [[2, -1], [0, 3], [1, -1]]], dtype=torch.long + ) + new_logcnt = torch.tensor([[2, 1, 1], [1, 2, 1]], dtype=torch.long) + + _commit_eplb_maps_for_layer(model_state, new_phy2log[0], layer=0) + + # Layer 0 updated + assert torch.equal(model_state.physical_to_logical_map[0], new_phy2log[0]) + assert torch.equal(model_state.logical_to_physical_map[0], new_log2phy[0]) + assert torch.equal(model_state.logical_replica_count[0], new_logcnt[0]) + + # Layer 1 untouched + assert torch.equal(model_state.physical_to_logical_map[1], original_phy2log[1]) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_events.py b/asian_ai/performance/vLLM/tests/distributed/test_events.py new file mode 100644 index 0000000000000000000000000000000000000000..39cfe1b3552002bce1e2a5e0dcd8fd5f4030a949 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_events.py @@ -0,0 +1,314 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import threading +import time + +import msgspec +import pytest + +from vllm.distributed.kv_events import ( + EventBatch, + EventPublisherFactory, + NullEventPublisher, +) + +DP_RANK = 0 + + +class EventSample( + msgspec.Struct, + tag=True, # type: ignore + array_like=True, # type: ignore +): + """Test event for publisher testing""" + + id: int + value: str + + +class SampleBatch(EventBatch): + """Test event batch for publisher testing""" + + events: list[EventSample] + + +def create_test_events(count: int) -> SampleBatch: + """Create a batch of test events""" + events = [EventSample(id=i, value=f"test-{i}") for i in range(count)] + return SampleBatch(ts=time.time(), events=events) + + +def test_basic_publishing(publisher, subscriber): + """Test basic event publishing works""" + + test_batch = create_test_events(5) + publisher.publish(test_batch) + + result = subscriber.receive_one(timeout=1000) + assert result is not None, "No message received" + + seq, received = result + assert seq == 0, "Sequence number mismatch" + assert received.ts == pytest.approx(test_batch.ts, abs=0.1), "Timestamp mismatch" + assert len(received.events) == len(test_batch.events), "Number of events mismatch" + + for i, event in enumerate(received.events): + assert event.id == i, "Event id mismatch" + assert event.value == f"test-{i}", "Event value mismatch" + + +def test_multiple_events(publisher, subscriber): + """Test publishing and receiving multiple event batches""" + for _ in range(10): + batch = create_test_events(2) + publisher.publish(batch) + + received = [] + for _ in range(10): + data = subscriber.receive_one(timeout=100) + if data: + received.append(data) + + assert len(received) == 10, "Number of messages mismatch" + seqs = [seq for seq, _ in received] + assert seqs == list(range(10)), "Sequence numbers mismatch" + + +def test_replay_mechanism(publisher, subscriber): + """Test the replay mechanism works correctly""" + for _ in range(19): + batch = create_test_events(1) + publisher.publish(batch) + + time.sleep(0.5) # Need publisher to process above requests + subscriber.request_replay(10) + + batch = create_test_events(1) + publisher.publish(batch) # 20th message + + replayed = subscriber.receive_replay() + + assert len(replayed) > 0, "No replayed messages received" + seqs = [seq for seq, _ in replayed] + assert all(seq >= 10 for seq in seqs), "Replayed messages not in order" + assert seqs == list(range(min(seqs), max(seqs) + 1)), ( + "Replayed messages not consecutive" + ) + + +def test_buffer_limit(publisher, subscriber, publisher_config): + """Test buffer limit behavior""" + buffer_size = publisher_config.buffer_steps + + # Publish more events than the buffer can hold + for i in range(buffer_size + 10): + batch = create_test_events(1) + publisher.publish(batch) + + time.sleep(0.5) # Need publisher to process above requests + subscriber.request_replay(0) + + batch = create_test_events(1) + publisher.publish(batch) + + replayed = subscriber.receive_replay() + + assert len(replayed) <= buffer_size, "Can't replay more than buffer size" + + oldest_seq = min(seq for seq, _ in replayed) + assert oldest_seq >= 10, "The oldest sequence should be at least 10" + + +def test_topic_filtering(publisher_config): + """ + Test that a subscriber only receives messages matching its topic filter + """ + publisher_config.replay_endpoint = None + + publisher_config.topic = "foo" + pub = EventPublisherFactory.create(publisher_config, DP_RANK) + + from .conftest import MockSubscriber + + sub_foo = MockSubscriber(publisher_config.endpoint, None, "foo") + sub_bar = MockSubscriber(publisher_config.endpoint, None, "bar") + + try: + time.sleep(0.1) + + for _ in range(3): + pub.publish(create_test_events(1)) + + foo_received = [sub_foo.receive_one(timeout=200) for _ in range(3)] + assert all(msg is not None for msg in foo_received), ( + "Subscriber with matching topic should receive messages" + ) + + bar_received = [sub_bar.receive_one(timeout=200) for _ in range(3)] + assert all(msg is None for msg in bar_received), ( + "Subscriber with non-matching topic should receive no messages" + ) + finally: + pub.shutdown() + sub_foo.close() + sub_bar.close() + + +def test_high_volume(publisher, subscriber): + """Test publishing and receiving a high volume of events""" + num_batches = 10_000 + events_per_batch = 100 + + # Publish events in a separate thread to not block + def publish_events(): + for i in range(num_batches): + batch = create_test_events(events_per_batch) + publisher.publish(batch) + # Small delay to avoid overwhelming + if i % 100 == 0: + time.sleep(0.01) + + received: list[tuple[int, SampleBatch]] = [] + + publisher_thread = threading.Thread(target=publish_events) + publisher_thread.start() + + start_time = time.time() + while len(received) < num_batches: + if time.time() - start_time > 10: # Timeout after 10 seconds + break + + result = subscriber.receive_one(timeout=100) + if result: + received.append(result) + + publisher_thread.join() + + assert len(received) >= num_batches * 0.9, "We should have received most messages" + + seqs = [seq for seq, _ in received] + assert sorted(seqs) == seqs, "Sequence numbers should be in order" + + +def test_null_publisher(): + """Test that NullEventPublisher can be used without errors""" + publisher = NullEventPublisher(DP_RANK) + + # This should not raise any errors + batch = create_test_events(5) + publisher.publish(batch) + publisher.shutdown() + + +def test_data_parallel_rank_tagging(publisher_config): + """Test that events are properly tagged with their data parallel rank""" + + publisher_config.topic = "foo" + pub_0 = EventPublisherFactory.create(publisher_config, DP_RANK) + pub_1 = EventPublisherFactory.create(publisher_config, DP_RANK + 1) + + # Hardcode the expected endpoints based on port offsetting behavior + # Both ranks get offsets according to _offset_endpoint_port function + base_endpoint = publisher_config.endpoint + if "tcp://" in base_endpoint: + # For TCP endpoints: tcp://localhost:5557 -> tcp://localhost:5557, tcp://localhost:5558 + expected_endpoint_0 = base_endpoint # rank 0 gets port + 0 = same port + expected_endpoint_1 = base_endpoint.replace( + ":5557", ":5558" + ) # rank 1 gets port + 1 + else: + # For inproc endpoints: inproc://test -> inproc://test_dp0, inproc://test_dp1 + expected_endpoint_0 = base_endpoint # rank 0 gets base + expected_endpoint_1 = base_endpoint + "_dp1" # rank 1 gets _dp1 + + from .conftest import MockSubscriber + + sub_0 = MockSubscriber(expected_endpoint_0, None, publisher_config.topic) + sub_1 = MockSubscriber(expected_endpoint_1, None, publisher_config.topic) + + try: + time.sleep(0.1) # Let publishers start up + + # Publish events from different ranks + batch_0 = create_test_events(2) + batch_1 = create_test_events(3) + + pub_0.publish(batch_0) + pub_1.publish(batch_1) + + # Receive events from rank 0 + result_0 = sub_0.receive_one(timeout=200) + assert result_0 is not None, "No message received from rank 0" + seq_0, received_0 = result_0 + + # Receive events from rank 1 + result_1 = sub_1.receive_one(timeout=200) + assert result_1 is not None, "No message received from rank 1" + seq_1, received_1 = result_1 + + # Verify DP rank tagging + assert received_0.data_parallel_rank == 0, ( + f"Expected DP rank 0, got {received_0.data_parallel_rank}" + ) + assert received_1.data_parallel_rank == 1, ( + f"Expected DP rank 1, got {received_1.data_parallel_rank}" + ) + + # Verify event content is correct + assert len(received_0.events) == 2, "Wrong number of events from rank 0" + assert len(received_1.events) == 3, "Wrong number of events from rank 1" + + finally: + pub_0.shutdown() + pub_1.shutdown() + sub_0.close() + sub_1.close() + + +def test_event_publisher_factory(): + """Test event publisher factory creation behavior under different configurations""" + from vllm.config.kv_events import KVEventsConfig + from vllm.distributed.kv_events import ZmqEventPublisher + + # test config is None + publisher = EventPublisherFactory.create(None, DP_RANK) + assert isinstance(publisher, NullEventPublisher) + publisher.shutdown() + + # test disable kv cache events + config = KVEventsConfig( + enable_kv_cache_events=False, + publisher="zmq", # Even if zmq is specified, should return NullEventPublisher + endpoint="tcp://localhost:5557", + ) + publisher = EventPublisherFactory.create(config, DP_RANK) + assert isinstance(publisher, NullEventPublisher) + publisher.shutdown() + + # test zmq publisher + config = KVEventsConfig( + enable_kv_cache_events=True, + publisher="zmq", + endpoint="inproc://test-factory-true", + ) + publisher = EventPublisherFactory.create(config, DP_RANK) + assert isinstance(publisher, ZmqEventPublisher) + publisher.shutdown() + + # test unknown publisher + with pytest.raises(ValueError, match="Input should be"): + KVEventsConfig( + enable_kv_cache_events=True, + publisher="unknown_publisher", + endpoint="tcp://localhost:5557", + ) + + # test publisher not specified + config = KVEventsConfig( + enable_kv_cache_events=True, + # publisher not specified, should default to "zmq" + endpoint="tcp://localhost:5557", + ) + publisher = EventPublisherFactory.create(config, DP_RANK) + assert isinstance(publisher, ZmqEventPublisher) + publisher.shutdown() diff --git a/asian_ai/performance/vLLM/tests/distributed/test_expert_parallel.py b/asian_ai/performance/vLLM/tests/distributed/test_expert_parallel.py new file mode 100644 index 0000000000000000000000000000000000000000..599131ff620567eb7bfbe84a82d17b633362addd --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_expert_parallel.py @@ -0,0 +1,231 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from dataclasses import dataclass +from typing import Literal, NamedTuple + +import pytest + +from vllm.config.model import RunnerOption +from vllm.logger import init_logger + +from ..utils import compare_two_settings, create_new_process_for_each_test + +logger = init_logger("test_expert_parallel") + + +class ParallelSetup(NamedTuple): + tp_size: int + eager_mode: bool + chunked_prefill: bool + + +class EPTestOptions(NamedTuple): + trust_remote_code: bool + tokenizer_mode: str | None + load_format: str | None = None + hf_overrides: str | None = None + + +@dataclass +class EPTestSettings: + parallel_setups: list[ParallelSetup] + distributed_backends: list[str] + runner: RunnerOption + test_options: EPTestOptions + + @staticmethod + def detailed( + *, + tp_base: int = 2, + runner: RunnerOption = "auto", + trust_remote_code: bool = False, + tokenizer_mode: str | None = None, + load_format: str | None = None, + hf_overrides: str | None = None, + ): + return EPTestSettings( + parallel_setups=[ + ParallelSetup(tp_size=tp_base, eager_mode=False, chunked_prefill=False), + ParallelSetup(tp_size=tp_base, eager_mode=False, chunked_prefill=True), + ParallelSetup(tp_size=tp_base, eager_mode=True, chunked_prefill=False), + ParallelSetup( + tp_size=2 * tp_base, eager_mode=False, chunked_prefill=True + ), + ParallelSetup( + tp_size=2 * tp_base, eager_mode=True, chunked_prefill=False + ), + ], + distributed_backends=["mp", "ray"], + runner=runner, + test_options=EPTestOptions( + trust_remote_code=trust_remote_code, + tokenizer_mode=tokenizer_mode, + load_format=load_format, + hf_overrides=hf_overrides, + ), + ) + + @staticmethod + def fast( + *, + tp_base: int = 2, + runner: RunnerOption = "auto", + trust_remote_code: bool = False, + tokenizer_mode: str | None = None, + load_format: str | None = None, + hf_overrides: str | None = None, + ): + return EPTestSettings( + parallel_setups=[ + ParallelSetup(tp_size=tp_base, eager_mode=True, chunked_prefill=False), + ], + distributed_backends=["mp"], + runner=runner, + test_options=EPTestOptions( + trust_remote_code=trust_remote_code, + tokenizer_mode=tokenizer_mode, + load_format=load_format, + hf_overrides=hf_overrides, + ), + ) + + def iter_params(self, model_name: str): + opts = self.test_options + + for parallel_setup in self.parallel_setups: + for distributed_backend in self.distributed_backends: + yield ( + model_name, + parallel_setup, + distributed_backend, + self.runner, + opts, + ) + + +# NOTE: You can adjust tp_base locally to fit the model in GPU +# The values displayed here are only a rough indicator of the size of the model + +TEST_MODELS = { + "deepseek-ai/DeepSeek-V2-Lite-Chat": EPTestSettings.fast(trust_remote_code=True), + "mistralai/Mixtral-8x7B-Instruct-v0.1": EPTestSettings.fast(tp_base=4), +} + + +def _compare_tp( + model_name: str, + parallel_setup: ParallelSetup, + distributed_backend: str, + runner: RunnerOption, + test_options: EPTestOptions, + num_gpus_available: int, + *, + method: Literal["generate"], +): + ( + tp_size, + eager_mode, + chunked_prefill, + ) = parallel_setup + ( + trust_remote_code, + tokenizer_mode, + load_format, + hf_overrides, + ) = test_options + + if num_gpus_available < tp_size: + pytest.skip(f"Need at least {tp_size} GPUs") + + common_args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "float16", + "--max-model-len", + "2048", + "--max-num-seqs", + "8", + "--load-format", + "auto", + ] + if chunked_prefill: + common_args.append("--enable-chunked-prefill") + if eager_mode: + common_args.append("--enforce-eager") + if runner != "auto": + common_args.extend(["--runner", runner]) + if trust_remote_code: + common_args.append("--trust-remote-code") + if tokenizer_mode: + common_args.extend(["--tokenizer-mode", tokenizer_mode]) + if load_format: + common_args.extend(["--load-format", load_format]) + if hf_overrides: + common_args.extend(["--hf-overrides", hf_overrides]) + + ep_env = { + "VLLM_TEST_ENABLE_EP": "1", + } + + ep_args = [ + *common_args, + "--tensor-parallel-size", + str(tp_size), + "--distributed-executor-backend", + distributed_backend, + ] + + # compare without expert parallelism + tp_env = { + "VLLM_TEST_ENABLE_EP": "0", + } + + tp_args = [ + *common_args, + "--tensor-parallel-size", + str(tp_size), + "--distributed-executor-backend", + "mp", + ] + + try: + compare_two_settings( + model_name, + ep_args, + tp_args, + ep_env, + tp_env, + method=method, + max_wait_seconds=360, + ) + except Exception: + raise + + +@pytest.mark.parametrize( + ("model_name", "parallel_setup", "distributed_backend", "runner", "test_options"), + [ + params + for model_name, settings in TEST_MODELS.items() + for params in settings.iter_params(model_name) + ], +) +@create_new_process_for_each_test() +def test_ep( + model_name: str, + parallel_setup: ParallelSetup, + distributed_backend: str, + runner: RunnerOption, + test_options: EPTestOptions, + num_gpus_available, +): + _compare_tp( + model_name, + parallel_setup, + distributed_backend, + runner, + test_options, + num_gpus_available, + method="generate", + ) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_expert_placement.py b/asian_ai/performance/vLLM/tests/distributed/test_expert_placement.py new file mode 100644 index 0000000000000000000000000000000000000000..88dc138ec6d93324303aea5038f4b79dbcc8246e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_expert_placement.py @@ -0,0 +1,246 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from vllm.model_executor.layers.fused_moe.expert_map_manager import ( + determine_expert_map, +) + + +def verify_round_robin_pattern(expert_map, ep_rank, ep_size, global_num_experts): + """Verify that the expert map follows the round_robin pattern.""" + # Calculate expected local experts (supporting non-divisible cases) + base_experts = global_num_experts // ep_size + remainder = global_num_experts % ep_size + + local_num_experts = base_experts + 1 if ep_rank < remainder else base_experts + + # Expected expert IDs for this rank in round_robin pattern + # For non-divisible cases, ranks with extra experts start earlier + expected_expert_ids = [] + for expert_idx in range(local_num_experts): + global_expert_id = ep_rank + expert_idx * ep_size + expected_expert_ids.append(global_expert_id) + + # Check that only expected experts are mapped to this rank + for global_expert_id in range(global_num_experts): + if global_expert_id in expected_expert_ids: + local_expert_id = expert_map[global_expert_id] + expected_local_id = expected_expert_ids.index(global_expert_id) + assert local_expert_id == expected_local_id, ( + f"Global expert {global_expert_id} should map to local expert " + f"{expected_local_id}, got {local_expert_id}" + ) + else: + assert expert_map[global_expert_id] == -1, ( + f"Global expert {global_expert_id} should not be mapped to this rank" + ) + + # Verify that all local expert IDs are consecutive starting from 0 + local_expert_ids = [expert_map[global_id] for global_id in expected_expert_ids] + expected_local_ids = list(range(local_num_experts)) + assert local_expert_ids == expected_local_ids, ( + f"Expected local expert IDs {expected_local_ids}, got {local_expert_ids}" + ) + + +@pytest.mark.parametrize("expert_placement_strategy", ["round_robin"]) +@pytest.mark.parametrize("world_size", [2, 4]) +def test_expert_placement_various_sizes(expert_placement_strategy, world_size): + """Test round_robin expert placement with various expert counts.""" + + # Test with different global_num_experts values + # Include both divisible and non-divisible cases + if world_size == 2: + test_cases = [ + (4, 2), # 4 experts (divisible) + (8, 2), # 8 experts (divisible) + (9, 2), # 9 experts (non-divisible) + (16, 2), # 16 experts (divisible) + (17, 2), # 17 experts (non-divisible) + ] + elif world_size == 4: + test_cases = [ + (8, 4), # 8 experts (divisible) + (16, 4), # 16 experts (divisible) + (18, 4), # 18 experts (non-divisible) + (32, 4), # 32 experts (divisible) + (33, 4), # 33 experts (non-divisible) + ] + else: + test_cases = [] + + for test_global_experts, test_ep_size in test_cases: + # Ensure ep_size matches world_size + assert test_ep_size == world_size, ( + f"ep_size {test_ep_size} must equal world_size {world_size}" + ) + + # Test each rank + for ep_rank in range(world_size): + # Calculate expected local experts + base_experts = test_global_experts // test_ep_size + remainder = test_global_experts % test_ep_size + if ep_rank < remainder: + expected_test_local = base_experts + 1 + else: + expected_test_local = base_experts + + test_local_experts, test_expert_map, _ = determine_expert_map( + ep_size=test_ep_size, + ep_rank=ep_rank, + global_num_experts=test_global_experts, + expert_placement_strategy=expert_placement_strategy, + ) + + assert test_local_experts == expected_test_local, ( + f"For {test_global_experts} experts on {test_ep_size} ranks, " + f"rank {ep_rank}: expected {expected_test_local} local" + f"experts, got {test_local_experts}" + ) + + if test_expert_map is not None: + assert test_expert_map.shape == (test_global_experts,), ( + f"Expected expert map shape ({test_global_experts},), " + f"got {test_expert_map.shape}" + ) + + # Verify round_robin pattern for this test case + verify_round_robin_pattern( + test_expert_map, ep_rank, test_ep_size, test_global_experts + ) + + +@pytest.mark.parametrize("expert_placement_strategy", ["round_robin"]) +@pytest.mark.parametrize("world_size", [2, 4]) +def test_expert_placement_edge_cases(expert_placement_strategy, world_size): + """Test edge cases for round_robin expert placement.""" + + # Test case 1: ep_size = 1 (should return None for expert_map) + local_num_experts, expert_map, _ = determine_expert_map( + ep_size=1, + ep_rank=0, + global_num_experts=8, + expert_placement_strategy=expert_placement_strategy, + ) + assert local_num_experts == 8, "For ep_size=1, should get all experts" + assert expert_map is None, "For ep_size=1, expert_map should be None" + + # Test case 2: ep_size = 0 (should raise assertion) + with pytest.raises(AssertionError): + determine_expert_map( + ep_size=0, + ep_rank=0, + global_num_experts=8, + expert_placement_strategy=expert_placement_strategy, + ) + + +def test_determine_expert_map_comprehensive(): + """Test of determine_expert_map function with various configurations.""" + + # Test cases: (ep_size, ep_rank, global_num_experts, + # expert_placement_strategy, expected_local, expected_map_pattern) + test_cases = [ + # Round robin placement tests + ( + 2, + 0, + 8, + "round_robin", + 4, + [0, -1, 1, -1, 2, -1, 3, -1], + ), # rank 0 gets even experts + ( + 2, + 1, + 8, + "round_robin", + 4, + [-1, 0, -1, 1, -1, 2, -1, 3], + ), # rank 1 gets odd experts + ( + 2, + 0, + 9, + "round_robin", + 5, + [0, -1, 1, -1, 2, -1, 3, -1, 4], + ), # rank 0 gets 5 experts (even + last) + ( + 2, + 1, + 9, + "round_robin", + 4, + [-1, 0, -1, 1, -1, 2, -1, 3, -1], + ), # rank 1 gets 4 experts (odd) + # 4-rank tests + ( + 4, + 0, + 8, + "round_robin", + 2, + [0, -1, -1, -1, 1, -1, -1, -1], + ), # rank 0 gets experts 0, 4 + ( + 4, + 1, + 8, + "round_robin", + 2, + [-1, 0, -1, -1, -1, 1, -1, -1], + ), # rank 1 gets experts 1, 5 + ( + 4, + 2, + 8, + "round_robin", + 2, + [-1, -1, 0, -1, -1, -1, 1, -1], + ), # rank 2 gets experts 2, 6 + ( + 4, + 3, + 8, + "round_robin", + 2, + [-1, -1, -1, 0, -1, -1, -1, 1], + ), # rank 3 gets experts 3, 7 + ] + + for ( + ep_size, + ep_rank, + global_num_experts, + expert_placement_strategy, + expected_local, + expected_map_pattern, + ) in test_cases: + local_num_experts, expert_map, _ = determine_expert_map( + ep_size=ep_size, + ep_rank=ep_rank, + global_num_experts=global_num_experts, + expert_placement_strategy=expert_placement_strategy, + ) + + assert local_num_experts == expected_local, ( + f"ep_size={ep_size}, ep_rank={ep_rank}, " + f"global_num_experts={global_num_experts}, " + f"expert_placement_strategy={expert_placement_strategy}: " + f"expected {expected_local} local experts, got {local_num_experts}" + ) + + if expected_map_pattern is None: + assert expert_map is None, "Expected expert_map to be None" + else: + assert expert_map is not None, "Expected expert_map to not be None" + actual_map = expert_map.tolist() + assert actual_map == expected_map_pattern, ( + f"ep_size={ep_size}, ep_rank={ep_rank}, " + f"global_num_experts={global_num_experts}, " + f"expert_placement_strategy={expert_placement_strategy}: " + f"expected map {expected_map_pattern}, got {actual_map}" + ) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_kv_cache_events.py b/asian_ai/performance/vLLM/tests/distributed/test_kv_cache_events.py new file mode 100644 index 0000000000000000000000000000000000000000..80e48c06cb1d8702d4723a83084b52748a41174a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_kv_cache_events.py @@ -0,0 +1,86 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from vllm.distributed.kv_events import BlockRemoved, BlockStored + +# Minimal ExternalBlockHash for testing (bytes are a valid ExternalBlockHash). +_FAKE_HASH: bytes = b"\xab" * 32 + + +def _make_block_stored( + group_idx: int | None = None, + kv_cache_spec_sliding_window: int | None = None, +) -> BlockStored: + return BlockStored( + block_hashes=[_FAKE_HASH], + parent_block_hash=None, + token_ids=[1, 2, 3, 4], + block_size=4, + lora_id=None, + medium="GPU", + lora_name=None, + group_idx=group_idx, + kv_cache_spec_sliding_window=kv_cache_spec_sliding_window, + ) + + +def _make_block_removed( + group_idx: int | None = None, +) -> BlockRemoved: + return BlockRemoved( + block_hashes=[_FAKE_HASH], + medium="GPU", + group_idx=group_idx, + ) + + +def test_block_stored_default_group_idx_is_none(): + """group_idx defaults to None when not provided.""" + event = _make_block_stored() + assert event.group_idx is None + + +def test_block_removed_default_group_idx_is_none(): + """group_idx defaults to None when not provided.""" + event = _make_block_removed() + assert event.group_idx is None + + +@pytest.mark.parametrize("group_idx", [1, 2, 3]) +def test_block_stored_hash_differs_by_group_idx(group_idx: int): + """BlockStored events that differ only in group_idx must hash differently.""" + other_group_idx = group_idx + 1 + event_a = _make_block_stored(group_idx=group_idx) + event_b = _make_block_stored(group_idx=other_group_idx) + assert hash(event_a) != hash(event_b) + + +def test_block_stored_hash_same_for_equal_group_idx(): + """Two BlockStored events with identical fields produce the same hash.""" + event_a = _make_block_stored(group_idx=1) + event_b = _make_block_stored(group_idx=1) + assert hash(event_a) == hash(event_b) + + +@pytest.mark.parametrize("group_idx", [1, 2, 3]) +def test_block_removed_hash_differs_by_group_idx(group_idx: int): + """BlockRemoved events that differ only in group_idx must hash differently.""" + other_group_idx = group_idx + 1 + event_a = _make_block_removed(group_idx=group_idx) + event_b = _make_block_removed(group_idx=other_group_idx) + assert hash(event_a) != hash(event_b) + + +def test_block_removed_hash_same_for_equal_group_idx(): + """Two BlockRemoved events with identical fields produce the same hash.""" + event_a = _make_block_removed(group_idx=1) + event_b = _make_block_removed(group_idx=1) + assert hash(event_a) == hash(event_b) + + +def test_block_stored_hash_differs_by_sliding_window(): + event_a = _make_block_stored(group_idx=1, kv_cache_spec_sliding_window=128) + event_b = _make_block_stored(group_idx=1, kv_cache_spec_sliding_window=256) + assert hash(event_a) != hash(event_b) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_kvlayout.py b/asian_ai/performance/vLLM/tests/distributed/test_kvlayout.py new file mode 100644 index 0000000000000000000000000000000000000000..77816b5781f5f6a7879477768b75761d8ae06b5a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_kvlayout.py @@ -0,0 +1,78 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from vllm.config import ( + DeviceConfig, + KVTransferConfig, + ModelConfig, + VllmConfig, + set_current_vllm_config, +) +from vllm.distributed.kv_transfer.kv_connector.utils import ( + get_kv_connector_cache_layout, +) +from vllm.logger import init_logger + +logger = init_logger("test_expert_parallel") + + +def test_get_kv_connector_cache_layout_without_kv_connector(): + vllm_config = VllmConfig(device_config=DeviceConfig("cpu")) + with set_current_vllm_config(vllm_config): + # Test with default settings + layout = get_kv_connector_cache_layout() + assert layout == "NHD" + + +def test_get_kv_connector_cache_layout_with_lmcache_connector(): + kv_transfer_config = KVTransferConfig( + kv_connector="LMCacheConnectorV1", + kv_role="kv_both", + ) + vllm_config = VllmConfig( + device_config=DeviceConfig("cpu"), kv_transfer_config=kv_transfer_config + ) + with set_current_vllm_config(vllm_config): + # Test with default settings + layout = get_kv_connector_cache_layout() + assert layout == "NHD" + + +def test_get_kv_connector_cache_layout_with_nixl_connector(): + kv_transfer_config = KVTransferConfig( + kv_connector="NixlConnector", + kv_role="kv_both", + ) + model_config = ModelConfig() + vllm_config = VllmConfig( + device_config=DeviceConfig("cpu"), + model_config=model_config, + kv_transfer_config=kv_transfer_config, + ) + with set_current_vllm_config(vllm_config): + # Test with default settings + layout = get_kv_connector_cache_layout() + assert layout == "HND" + + +def test_get_kv_connector_cache_layout_with_multi_connector(): + kv_transfer_config = KVTransferConfig( + kv_connector="MultiConnector", + kv_role="kv_both", + kv_connector_extra_config={ + "connectors": [ + {"kv_connector": "ExampleConnector", "kv_role": "kv_both"}, + {"kv_connector": "NixlConnector", "kv_role": "kv_both"}, + ] + }, + ) + model_config = ModelConfig() + vllm_config = VllmConfig( + device_config=DeviceConfig("cpu"), + model_config=model_config, + kv_transfer_config=kv_transfer_config, + ) + with set_current_vllm_config(vllm_config): + # Test with default settings + layout = get_kv_connector_cache_layout() + assert layout == "HND" diff --git a/asian_ai/performance/vLLM/tests/distributed/test_mnnvl_alltoall.py b/asian_ai/performance/vLLM/tests/distributed/test_mnnvl_alltoall.py new file mode 100644 index 0000000000000000000000000000000000000000..658f9b57b3bd357be4229fc19bcc593df2edc66e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_mnnvl_alltoall.py @@ -0,0 +1,941 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Tests for MNNVL AllToAll operations. + +Requires: docker run ... --cap-add=SYS_PTRACE ... +Run: pytest tests/distributed/test_mnnvl_alltoall.py -v +""" + +import os +import traceback + +import pytest +import torch +import torch.multiprocessing as mp + +from vllm.distributed import get_ep_group +from vllm.utils.flashinfer import ( + has_flashinfer_nvlink_one_sided, + has_flashinfer_nvlink_two_sided, +) +from vllm.utils.import_utils import has_deep_ep_v2 +from vllm.utils.network_utils import get_open_port + +from ..utils import init_test_distributed_environment + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _has_sys_ptrace() -> bool: + """Check for SYS_PTRACE capability (bit 19 in CapEff).""" + try: + with open("/proc/self/status") as f: + for line in f: + if line.startswith("CapEff:"): + return bool(int(line.split()[1], 16) & (1 << 19)) + except Exception: + pass + return False + + +def _spawn_workers(worker_fn, world_size, *, dp_size=None): + """Spawn one process per GPU, run worker_fn, assert all succeed. + + Uses an mp.Queue to propagate worker tracebacks back to the parent + so pytest shows the actual failure, not just an exit code. + """ + if mp.get_start_method(allow_none=True) is None: + mp.set_start_method("spawn") + + port = str(get_open_port()) + # Allocate a second port for DP master when dp_size is set, so the + # distributed init port and DP port can't collide even under xdist. + dp_port = str(get_open_port()) if dp_size is not None else None + err_queue: mp.Queue = mp.Queue() + procs = [] + for rank in range(world_size): + p = mp.Process( + target=_run_worker, + args=(rank, world_size, port, worker_fn, dp_size, dp_port, err_queue), + ) + p.start() + procs.append(p) + for p in procs: + p.join() + + # Collect any errors from workers before asserting. + errors = [] + while not err_queue.empty(): + errors.append(err_queue.get_nowait()) + err_queue.close() + err_queue.join_thread() + if errors: + pytest.fail("Worker(s) failed:\n" + "\n---\n".join(errors)) + + +def _run_worker(rank, world_size, port, worker_fn, dp_size, dp_port, err_queue): + """Per-process setup: device, distributed env, then call worker_fn. + + Args: + dp_size: If set, initialize with tp=1 and data_parallel_size=dp_size. + Otherwise use tp=world_size (default for EP-based tests). + dp_port: Separate port for the DP master (only used when dp_size is set). + err_queue: Queue for propagating tracebacks to the parent process. + """ + try: + os.environ.pop("CUDA_VISIBLE_DEVICES", None) + torch.accelerator.set_device_index(rank) + if dp_size is not None: + _init_dp_environment(world_size, rank, port, dp_size, dp_port) + else: + init_test_distributed_environment(world_size, 1, rank, port) + worker_fn(rank, world_size) + torch.distributed.barrier() + except Exception: + err_queue.put(f"[Rank {rank}]\n{traceback.format_exc()}") + # Don't re-raise: the parent reads errors from err_queue. + # A non-zero exit from the re-raise would be redundant. + import sys + + sys.exit(1) + + +def _init_dp_environment(world_size, rank, port, dp_size, dp_port): + """Initialize distributed env with data parallelism. + + Sets up tp=1, pp=1, dp=dp_size. Each process is one DP rank + with local rank 0 within its (trivial) tp*pp group. + + Args: + port: Port for torch.distributed init. + dp_port: Separate port for the DP master group init. + """ + from vllm.config import VllmConfig, set_current_vllm_config + from vllm.config.parallel import ParallelConfig + from vllm.distributed.parallel_state import ( + ensure_model_parallel_initialized, + init_distributed_environment, + ) + + vllm_config = VllmConfig() + vllm_config.parallel_config = ParallelConfig( + data_parallel_size=dp_size, + data_parallel_rank=rank, + # Pre-populate port list so __post_init__ doesn't auto-generate + # random ports. All DP ranks must agree on the same port. + _data_parallel_master_port_list=[int(dp_port)], + ) + with set_current_vllm_config(vllm_config): + # rank=0 here because each DP rank has a single (tp=1,pp=1) process, + # so the local rank within the tp*pp group is always 0. + # init_distributed_environment will offset by data_parallel_rank. + init_distributed_environment( + world_size=1, # tp * pp = 1 + rank=0, + distributed_init_method=f"tcp://localhost:{port}", + local_rank=rank, + ) + ensure_model_parallel_initialized(1, 1) + + +def _make_forward_context(rank, world_size, num_tokens_per_rank): + """Create a forward context with mock DP metadata for AgRs tests. + + Returns a context manager suitable for ``with`` statements. + The real DPMetadata (with sp_local_sizes etc.) is created internally + by set_forward_context from num_tokens_across_dp; the attn_metadata + placeholder just satisfies the "attn_metadata is not None" guard. + """ + from vllm.config.parallel import ParallelConfig + from vllm.config.vllm import VllmConfig + from vllm.forward_context import set_forward_context + + class _AttnMeta: + """Minimal placeholder so set_forward_context's + ``attn_metadata is not None`` guard (forward_context.py:334) + is satisfied. The real DPMetadata is built from num_tokens_across_dp.""" + + dp_metadata = None + + vllm_config = VllmConfig() + vllm_config.parallel_config = ParallelConfig( + data_parallel_size=world_size, + is_moe_model=True, + data_parallel_rank=rank, + ) + return set_forward_context( + _AttnMeta(), + vllm_config, + num_tokens=num_tokens_per_rank, + num_tokens_across_dp=torch.tensor( + [num_tokens_per_rank] * world_size, dtype=torch.int + ), + ) + + +# --------------------------------------------------------------------------- +# Skip conditions +# --------------------------------------------------------------------------- + +requires_multi_gpu = pytest.mark.skipif( + torch.accelerator.device_count() < 2, reason="Need >= 2 GPUs" +) +requires_two_sided = pytest.mark.skipif( + not has_flashinfer_nvlink_two_sided(), + reason="FlashInfer NVLink two-sided not available", +) +requires_one_sided = pytest.mark.skipif( + not has_flashinfer_nvlink_one_sided(), + reason="FlashInfer NVLink one-sided not available", +) +requires_ptrace = pytest.mark.skipif( + not _has_sys_ptrace(), + reason="SYS_PTRACE required (docker run --cap-add=SYS_PTRACE)", +) +requires_deep_ep_v2 = pytest.mark.skipif( + not has_deep_ep_v2(), + reason="DeepEP v2 (ElasticBuffer) not available or NCCL < 2.30.4", +) + +# NOTE: No module-level pytestmark here. The FlashInfer lifecycle tests have +# their own @requires_two_sided / @requires_one_sided decorators, and +# test_args_dispatch_combine uses only standard torch.distributed ops and +# should run even when FlashInfer NVLink backends are not installed. + + +# --------------------------------------------------------------------------- +# Test 1: Two-sided manager lifecycle (init, cleanup, reinit, ensure_init) +# --------------------------------------------------------------------------- +# +# Tests FlashInferNVLinkTwoSidedManager which wraps FlashInfer's MnnvlMoe. +# initialize() allocates MNNVL shared workspaces via MnnvlMoe.get_moe_workspaces, +# which uses pidfd_getfd() to share memory file descriptors across processes — +# hence the SYS_PTRACE requirement. +# +# Uses EP group (get_ep_group) because the two-sided manager is constructed +# with an EP-scoped communicator in production. With tp=world_size the EP +# group spans all ranks, giving us a multi-rank group for testing. +# --------------------------------------------------------------------------- + + +def _two_sided_lifecycle_worker(rank, world_size): + from vllm.distributed.device_communicators.all2all import ( + FlashInferNVLinkTwoSidedManager, + ) + + cpu_group = get_ep_group().cpu_group + num_gpus = torch.accelerator.device_count() + manager = FlashInferNVLinkTwoSidedManager(cpu_group) + + # Not initialized yet + assert not manager.initialized + assert manager.rank == rank + assert manager.world_size == world_size + + # Initialize + manager.initialize(world_size=world_size, rank=rank, gpus_per_node=num_gpus) + assert manager.initialized + assert manager.workspace_tensor is not None + assert manager.prepare_workspace_tensor is not None + assert manager.mapping is not None + + torch.distributed.barrier() + + # Cleanup + manager.cleanup() + assert not manager.initialized + assert manager.workspace_tensor is None + assert manager.prepare_workspace_tensor is None + + torch.distributed.barrier() + + # Reinitialize + manager.initialize(world_size=world_size, rank=rank, gpus_per_node=num_gpus) + assert manager.initialized + + torch.distributed.barrier() + + # ensure_alltoall_workspace_initialized is idempotent when already init'd + assert manager.ensure_alltoall_workspace_initialized() + assert manager.initialized + + manager.cleanup() + assert not manager.initialized + + +@requires_multi_gpu +@requires_two_sided +@requires_ptrace +@pytest.mark.parametrize("world_size", [2]) +def test_two_sided_manager_lifecycle(world_size): + """Test init, cleanup, reinit, and ensure_initialized idempotency.""" + _spawn_workers(_two_sided_lifecycle_worker, world_size) + + +# --------------------------------------------------------------------------- +# Test 2: One-sided manager lifecycle (init, cleanup, reinit) +# --------------------------------------------------------------------------- +# +# Tests FlashInferNVLinkOneSidedManager which wraps FlashInfer's MoeAlltoAll. +# initialize() creates MoeAlltoAll with an MnnvlConfig, which allocates MNNVL +# shared workspaces — same cross-process memory sharing as two-sided, hence +# the SYS_PTRACE requirement. +# +# Uses DP group (get_dp_group) because the one-sided manager's initialize() +# internally calls get_dp_group() to set up the MnnvlConfig communicator. +# We therefore need a real DP group with world_size > 1, which requires +# dp_size=world_size via _init_dp_environment. +# --------------------------------------------------------------------------- + + +def _one_sided_lifecycle_worker(rank, world_size): + from vllm.distributed.device_communicators.all2all import ( + FlashInferNVLinkOneSidedManager, + ) + from vllm.distributed.parallel_state import get_dp_group + + cpu_group = get_dp_group().cpu_group + manager = FlashInferNVLinkOneSidedManager(cpu_group) + + assert not manager.initialized + assert manager.rank == rank + assert manager.world_size == world_size + + init_kwargs = dict( + max_num_tokens=1024, + top_k=2, + num_experts=world_size * 8, + hidden_size=4096, + ) + + # Initialize + manager.initialize(**init_kwargs) + assert manager.initialized + assert manager.moe_alltoall is not None + assert manager.mapping is not None + + torch.distributed.barrier() + + # Cleanup + manager.cleanup() + assert not manager.initialized + assert manager.moe_alltoall is None + + torch.distributed.barrier() + + # Reinitialize with different token count + manager.initialize(**{**init_kwargs, "max_num_tokens": 2048}) + assert manager.initialized + + torch.distributed.barrier() + manager.cleanup() + + +@requires_multi_gpu +@requires_one_sided +@requires_ptrace +@pytest.mark.parametrize("world_size", [2]) +def test_one_sided_manager_lifecycle(world_size): + """Test init, cleanup, and reinit with different params.""" + _spawn_workers( + _one_sided_lifecycle_worker, + world_size, + dp_size=world_size, + ) + + +# --------------------------------------------------------------------------- +# Test 2b: One-sided manager grows workspace across heterogeneous MoE layers +# --------------------------------------------------------------------------- +# +# Models with heterogeneous MoE quantization — most notably a quantized base +# MoE combined with an unquantized MTP head — can call initialize() multiple +# times with different per-token dispatch payload sizes. The shared workspace +# must grow to the union and the MoeAlltoAll must be rebuilt; otherwise a +# later layer's combine call overruns the workspace sized for the first +# layer's smaller payload and trips FlashInfer's combinePayloadOffset assert. +# --------------------------------------------------------------------------- + + +def _one_sided_workspace_grow_worker(rank, world_size): + from vllm.distributed.device_communicators.all2all import ( + FlashInferNVLinkOneSidedManager, + ) + from vllm.distributed.parallel_state import get_dp_group + + cpu_group = get_dp_group().cpu_group + manager = FlashInferNVLinkOneSidedManager(cpu_group) + + base_kwargs = dict( + max_num_tokens=1024, + top_k=2, + num_experts=world_size * 8, + hidden_size=4096, + ) + nvfp4_kwargs = dict( + dispatch_dtype_bytes_per_elem=0, + dispatch_scale_bytes_per_token=base_kwargs["hidden_size"] // 16, + ) + bf16_kwargs = dict( + dispatch_dtype_bytes_per_elem=2, + dispatch_scale_bytes_per_token=0, + ) + + # First init: NVFP4-like (hidden_bytes = hidden // 2 + hidden // 16). + manager.initialize(**base_kwargs, **nvfp4_kwargs) + assert manager.initialized + nvfp4_workspace_size = manager.workspace_size + nvfp4_moe_alltoall = manager.moe_alltoall + + torch.distributed.barrier() + + # Second init: bf16-like (hidden_bytes = hidden * 2). Models the case of + # a quantized base MoE followed by an unquantized MoE layer (e.g. an MTP + # head). Per-token dispatch payload is ~4x larger, so the union workspace + # must grow and MoeAlltoAll must be rebuilt. + manager.initialize(**base_kwargs, **bf16_kwargs) + assert manager.initialized + assert manager.workspace_size > nvfp4_workspace_size + assert manager.moe_alltoall is not nvfp4_moe_alltoall + bf16_workspace_size = manager.workspace_size + bf16_moe_alltoall = manager.moe_alltoall + + torch.distributed.barrier() + + # Third init: back to NVFP4-like shape. Existing workspace already covers + # it, so initialize() must no-op — no shrink, no rebuild. + manager.initialize(**base_kwargs, **nvfp4_kwargs) + assert manager.initialized + assert manager.workspace_size == bf16_workspace_size + assert manager.moe_alltoall is bf16_moe_alltoall + + torch.distributed.barrier() + manager.cleanup() + + +@requires_multi_gpu +@requires_one_sided +@requires_ptrace +@pytest.mark.parametrize("world_size", [2]) +def test_one_sided_manager_workspace_grow(world_size): + """A later initialize() with a larger per-token payload must grow the + workspace and rebuild MoeAlltoAll; a later initialize() with a smaller + payload must no-op.""" + _spawn_workers( + _one_sided_workspace_grow_worker, + world_size, + dp_size=world_size, + ) + + +# --------------------------------------------------------------------------- +# Test 3: AgRs dispatch/combine with value validation +# --------------------------------------------------------------------------- +# +# Tests AgRsAll2AllManager which uses only standard torch.distributed +# all_gatherv / reduce_scatterv — no FlashInfer or MNNVL dependency. +# This test validates the reference all-to-all implementation that other +# backends are compared against. +# --------------------------------------------------------------------------- + + +def _args_dispatch_combine_worker(rank, world_size): + from vllm.distributed.device_communicators.all2all import AgRsAll2AllManager + from vllm.forward_context import get_forward_context + + cpu_group = get_ep_group().cpu_group + device = torch.device(f"cuda:{rank}") + + hidden_size = 64 + tokens_per_rank = 16 + experts_per_token = 2 + num_experts = world_size * 4 + total_tokens = world_size * tokens_per_rank + + # Deterministic per-rank data: rank r has value (r + 1) + hidden = torch.full( + (tokens_per_rank, hidden_size), + float(rank + 1), + device=device, + dtype=torch.float32, + ) + router = torch.full( + (tokens_per_rank, num_experts), + float(rank + 1) * 10, + device=device, + dtype=torch.float32, + ) + weights = torch.full( + (tokens_per_rank, experts_per_token), + float(rank + 1) * 100, + device=device, + dtype=torch.float32, + ) + ids = torch.full( + (tokens_per_rank, experts_per_token), + rank, + device=device, + dtype=torch.long, + ) + + with _make_forward_context(rank, world_size, tokens_per_rank): + manager = AgRsAll2AllManager(cpu_group) + dp_metadata = get_forward_context().dp_metadata + + with dp_metadata.sp_local_sizes(sequence_parallel_size=1): + # -- dispatch_router_logits -- + d_hidden, d_router = manager.dispatch_router_logits( + hidden.clone(), + router.clone(), + is_sequence_parallel=True, + ) + assert d_hidden.shape == (total_tokens, hidden_size) + assert d_router.shape == (total_tokens, num_experts) + + for r in range(world_size): + s = r * tokens_per_rank + e = (r + 1) * tokens_per_rank + torch.testing.assert_close( + d_hidden[s:e], + torch.full_like(d_hidden[s:e], float(r + 1)), + ) + torch.testing.assert_close( + d_router[s:e], + torch.full_like(d_router[s:e], float(r + 1) * 10), + ) + + # -- dispatch -- + d_hidden2, d_weights, d_ids = manager.dispatch( + hidden.clone(), + weights.clone(), + ids.clone(), + is_sequence_parallel=True, + ) + assert d_hidden2.shape == (total_tokens, hidden_size) + assert d_weights.shape == (total_tokens, experts_per_token) + assert d_ids.shape == (total_tokens, experts_per_token) + + for r in range(world_size): + s = r * tokens_per_rank + e = (r + 1) * tokens_per_rank + torch.testing.assert_close( + d_weights[s:e], + torch.full_like(d_weights[s:e], float(r + 1) * 100), + ) + assert (d_ids[s:e] == r).all() + + # -- combine (reduce-scatter) -- + # Each token i has value i in all columns; after reduce-scatter + # each rank gets its slice, summed across ranks. + expert_out = ( + torch.arange(total_tokens, device=device, dtype=torch.float32) + .unsqueeze(1) + .expand(total_tokens, hidden_size) + .contiguous() + ) + + combined = manager.combine(expert_out, is_sequence_parallel=True) + assert combined.shape == (tokens_per_rank, hidden_size) + + for i in range(tokens_per_rank): + expected_val = float(rank * tokens_per_rank + i) * world_size + torch.testing.assert_close( + combined[i], + torch.full_like(combined[i], expected_val), + ) + + torch.distributed.barrier() + + +@requires_multi_gpu +@pytest.mark.parametrize("world_size", [2]) +def test_args_dispatch_combine(world_size): + """Validate dispatch gathers all-rank data and combine reduces correctly.""" + _spawn_workers(_args_dispatch_combine_worker, world_size) + + +# --------------------------------------------------------------------------- +# Test 4: FlashInfer two-sided dispatch/combine data communication +# --------------------------------------------------------------------------- +# +# Tests actual data flow through the FlashInfer NVLink two-sided backend +# by calling flashinfer_alltoall_dispatch (with defer_input_quant=True to +# skip quantization) and flashinfer_alltoall_combine, then verifying exact +# round-trip values. Dispatch sends each token once per distinct expert +# rank, and combine performs an unweighted sum, so: +# dispatch(hidden) → identity → combine = hidden * num_distinct_ranks(i) +# --------------------------------------------------------------------------- + + +def _two_sided_data_worker(rank, world_size): + from vllm.distributed.device_communicators.all2all import ( + FlashInferNVLinkTwoSidedManager, + ) + from vllm.distributed.parallel_state import get_dp_group + from vllm.forward_context import get_forward_context + from vllm.model_executor.layers.fused_moe.config import ( + FusedMoEQuantConfig, + FusedMoEQuantDesc, + ) + from vllm.model_executor.layers.fused_moe.prepare_finalize.flashinfer_nvlink_two_sided import ( # noqa: E501 + flashinfer_alltoall_combine, + flashinfer_alltoall_dispatch, + ) + + # Use DP group because MnnvlMoe workspace allocation calls get_dp_group() + # internally and requires dp_size == ep_size. + cpu_group = get_dp_group().cpu_group + device = torch.device(f"cuda:{rank}") + num_gpus = torch.accelerator.device_count() + + hidden_size = 128 + tokens_per_rank = 32 + experts_per_token = 2 + num_experts = world_size * 4 + + # Initialize the FlashInfer two-sided manager + manager = FlashInferNVLinkTwoSidedManager(cpu_group) + manager.initialize(world_size=world_size, rank=rank, gpus_per_node=num_gpus) + assert manager.initialized + + torch.distributed.barrier() + + # Create deterministic per-rank test data + torch.manual_seed(rank + 42) + hidden = torch.randn( + tokens_per_rank, + hidden_size, + device=device, + dtype=torch.bfloat16, + ) + # Assign each token to experts spread across ranks so tokens move between GPUs + topk_ids = torch.randint( + 0, + num_experts, + (tokens_per_rank, experts_per_token), + device=device, + dtype=torch.int32, + ) + topk_weights = torch.rand( + tokens_per_rank, + experts_per_token, + device=device, + dtype=torch.float32, + ) + + # Unquantized config: quant_dtype=None means moe_kernel_quantize_input is a no-op + no_quant = FusedMoEQuantDesc() + quant_config = FusedMoEQuantConfig( + _a1=no_quant, + _a2=no_quant, + _w1=no_quant, + _w2=no_quant, + ) + assert quant_config.quant_dtype is None # sanity: no quantization + + with _make_forward_context(rank, world_size, tokens_per_rank): + dp_metadata = get_forward_context().dp_metadata + + with dp_metadata.sp_local_sizes(sequence_parallel_size=1): + local_sizes = dp_metadata.get_chunk_sizes_across_dp_rank() + + # --- FlashInfer two-sided dispatch --- + alltoall_info, fi_topk_ids, fi_topk_weights, fi_hidden, fi_scale = ( + flashinfer_alltoall_dispatch( + manager, + local_sizes, + hidden.clone(), + None, # no global scale + topk_ids.clone(), + topk_weights.clone(), + experts_per_token, + num_experts, + quant_config, + defer_input_quant=True, + ) + ) + assert fi_scale is None # deferred quant: no scale produced + assert fi_hidden is not None + assert fi_hidden.shape[1] == hidden_size + assert fi_hidden.numel() > 0 + + # --- Round-trip exact verification --- + # The all-to-all sends each token once per *distinct* expert + # rank. Combine performs an unweighted sum of the per-rank + # contributions. With identity expert (feeding dispatched + # hidden straight back): + # result[i] = hidden[i] * num_distinct_expert_ranks(i) + combined = flashinfer_alltoall_combine( + manager, + fi_hidden, + top_k=experts_per_token, + token_count=tokens_per_rank, + alltoall_info=alltoall_info, + ) + assert combined.shape == (tokens_per_rank, hidden_size) + + experts_per_rank = num_experts // world_size + expert_ranks = topk_ids // experts_per_rank # (tokens, top_k) + num_distinct = torch.tensor( + [len(set(row.tolist())) for row in expert_ranks], + device=device, + dtype=torch.float32, + ).unsqueeze(1) # (tokens, 1) + expected = (hidden.float() * num_distinct).to(hidden.dtype) + torch.testing.assert_close(combined, expected) + + # --- Linearity check with scaled expert output --- + # Scaling the expert output by a constant should scale the + # combined result by the same constant. + scale = 3.0 + combined_scaled = flashinfer_alltoall_combine( + manager, + fi_hidden * scale, + top_k=experts_per_token, + token_count=tokens_per_rank, + alltoall_info=alltoall_info, + ) + expected_scaled = (hidden.float() * num_distinct * scale).to(hidden.dtype) + torch.testing.assert_close(combined_scaled, expected_scaled) + + torch.distributed.barrier() + + manager.cleanup() + + +@requires_multi_gpu +@requires_two_sided +@requires_ptrace +@pytest.mark.parametrize("world_size", [2]) +def test_two_sided_dispatch_combine(world_size): + """Test FlashInfer two-sided dispatch/combine with exact value verification.""" + _spawn_workers(_two_sided_data_worker, world_size, dp_size=world_size) + + +# --------------------------------------------------------------------------- +# Test 5: FlashInfer one-sided dispatch/combine data communication +# --------------------------------------------------------------------------- +# +# Tests actual data flow through the FlashInfer NVLink one-sided backend +# by calling MoeAlltoAll.dispatch() and MoeAlltoAll.combine() directly +# with synthetic payloads, then verifying shapes and round-trip consistency. +# --------------------------------------------------------------------------- + + +def _one_sided_data_worker(rank, world_size): + from vllm.distributed.device_communicators.all2all import ( + FlashInferNVLinkOneSidedManager, + ) + from vllm.distributed.parallel_state import get_dp_group + from vllm.forward_context import get_forward_context + + cpu_group = get_dp_group().cpu_group + device = torch.device(f"cuda:{rank}") + + hidden_size = 256 + tokens_per_rank = 32 + experts_per_token = 2 + num_experts = world_size * 8 + + # Initialize the one-sided manager + manager = FlashInferNVLinkOneSidedManager(cpu_group) + manager.initialize( + max_num_tokens=tokens_per_rank, + top_k=experts_per_token, + num_experts=num_experts, + hidden_size=hidden_size, + # Account for the fp8 block-scale payload (a1q_scale: hidden//16 bytes + # per token) that is dispatched alongside the nvfp4 hidden states. + # Without this the dispatch region is under-reserved and the combine + # payload overflows the per-rank workspace. + dispatch_scale_bytes_per_token=hidden_size // 16, + ) + assert manager.initialized + assert manager.moe_alltoall is not None + + with _make_forward_context(rank, world_size, tokens_per_rank): + dp_metadata = get_forward_context().dp_metadata + + with dp_metadata.sp_local_sizes(sequence_parallel_size=1): + local_sizes = dp_metadata.get_chunk_sizes_across_dp_rank() + runtime_max_tokens = max(local_sizes) + + # Create test data with raw tensors matching the nvfp4 payload + # sizes the workspace was allocated for: + # a1q: (tokens, hidden_size // 2) — nvfp4 hidden states + # a1q_scale: (tokens, hidden_size // 16) — fp8 scaling factors + torch.manual_seed(rank + 42) + a1q = torch.randint( + 0, + 256, + (tokens_per_rank, hidden_size // 2), + device=device, + dtype=torch.uint8, + ) + a1q_scale = torch.randint( + 0, + 256, + (tokens_per_rank, hidden_size // 16), + device=device, + dtype=torch.uint8, + ) + topk_ids = torch.randint( + 0, + num_experts, + (tokens_per_rank, experts_per_token), + device=device, + dtype=torch.int32, + ) + topk_weights = torch.rand( + tokens_per_rank, + experts_per_token, + device=device, + dtype=torch.float32, + ) + + # --- One-sided dispatch --- + payloads = [a1q, a1q_scale, topk_ids, topk_weights] + recv_payloads = manager.moe_alltoall.dispatch( + token_selected_experts=topk_ids, + input_payloads=payloads, + runtime_max_tokens_per_rank=runtime_max_tokens, + ) + assert len(recv_payloads) == 4 + recv_a1q, recv_scale, recv_ids, recv_weights = recv_payloads + assert recv_a1q.numel() > 0 + assert recv_ids.numel() > 0 + + # --- Round-trip exact verification --- + # The dispatch routes each token once per *distinct* expert + # rank. Combine performs an unweighted sum of per-rank + # contributions. With constant expert output (all 1s): + # result[i] = 1.0 * num_distinct_expert_ranks(i) + expert_output = torch.ones( + world_size, + runtime_max_tokens, + hidden_size, + device=device, + dtype=torch.bfloat16, + ) + combined = manager.moe_alltoall.combine( + payload=expert_output, + runtime_max_tokens_per_rank=runtime_max_tokens, + ) + assert combined.shape == (tokens_per_rank, hidden_size) + + experts_per_rank = num_experts // world_size + expert_ranks = topk_ids // experts_per_rank # (tokens, top_k) + num_distinct = torch.tensor( + [len(set(row.tolist())) for row in expert_ranks], + device=device, + dtype=torch.bfloat16, + ).unsqueeze(1) # (tokens, 1) + expected = num_distinct.expand_as(combined) + torch.testing.assert_close(combined, expected) + + # --- Linearity check with scaled expert output --- + # Scaling the expert output by a constant should scale the + # combined result by the same constant. + # Re-dispatch to reset internal state (one-sided requires a + # fresh dispatch before each combine). + manager.moe_alltoall.dispatch( + token_selected_experts=topk_ids, + input_payloads=payloads, + runtime_max_tokens_per_rank=runtime_max_tokens, + ) + scale = 3.0 + combined_scaled = manager.moe_alltoall.combine( + payload=expert_output * scale, + runtime_max_tokens_per_rank=runtime_max_tokens, + ) + expected_scaled = (expected * scale).to(torch.bfloat16) + torch.testing.assert_close(combined_scaled, expected_scaled) + + torch.distributed.barrier() + + manager.cleanup() + + +@requires_multi_gpu +@requires_one_sided +@requires_ptrace +@pytest.mark.parametrize("world_size", [2]) +def test_one_sided_dispatch_combine(world_size): + """Test FlashInfer one-sided dispatch/combine with actual data flow.""" + _spawn_workers(_one_sided_data_worker, world_size, dp_size=world_size) + + +# --------------------------------------------------------------------------- +# Test 6: DeepEP v2 (ElasticBuffer) manager lifecycle +# --------------------------------------------------------------------------- +# +# Tests DeepEPV2All2AllManager which wraps DeepEP's ElasticBuffer API using +# the NCCL GIN backend. Requires DeepEP >= 2.0 and NCCL >= 2.30.4. +# +# Uses EP group because the DeepEP v2 manager is constructed with an +# EP-scoped communicator in production. With tp=world_size the EP group +# spans all ranks. +# --------------------------------------------------------------------------- + + +def _deepep_v2_lifecycle_worker(rank, world_size): + from vllm.distributed.device_communicators.all2all import ( + DeepEPV2All2AllManager, + ) + + cpu_group = get_ep_group().cpu_group + manager = DeepEPV2All2AllManager(cpu_group) + + assert manager.rank == rank + assert manager.world_size == world_size + assert manager._num_sms is None + + hidden_size = 7168 + num_experts = world_size * 32 + num_topk = 8 + max_tokens = 256 + + handle_kwargs = dict( + num_max_tokens_per_rank=max_tokens, + hidden=hidden_size, + num_topk=num_topk, + num_experts=num_experts, + use_fp8_dispatch=False, + ) + + handle = manager.get_handle(handle_kwargs) + assert handle is not None + assert manager._num_sms is not None + assert manager._num_sms > 0 + + torch.distributed.barrier() + + # get_handle again with same args should return cached handle + handle2 = manager.get_handle(dict(handle_kwargs)) + assert handle2 is handle + + torch.distributed.barrier() + + # Destroy clears the cache + manager.destroy() + assert len(manager.handle_cache._cache) == 0 + + torch.distributed.barrier() + + # Re-create after destroy + handle3 = manager.get_handle(dict(handle_kwargs)) + assert handle3 is not None + + torch.distributed.barrier() + manager.destroy() + + +@requires_multi_gpu +@requires_deep_ep_v2 +@pytest.mark.parametrize("world_size", [2]) +def test_deepep_v2_manager_lifecycle(world_size): + """Test DeepEP v2 ElasticBuffer manager init, caching, and destroy.""" + _spawn_workers(_deepep_v2_lifecycle_worker, world_size) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_mq_connect_ip.py b/asian_ai/performance/vLLM/tests/distributed/test_mq_connect_ip.py new file mode 100644 index 0000000000000000000000000000000000000000..27c65d1ccff839c91960d11d450922c63d31a6b8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_mq_connect_ip.py @@ -0,0 +1,79 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Test that MessageQueue uses the local node's IP for binding, +not a remote master_addr. This validates the fix for cross-node +data-parallel where each DP group leader must bind to its own IP. + +The bug: multiproc_executor used `parallel_config.master_addr` as +`connect_ip` for every DP group's MessageQueue. For DP groups whose +leader is NOT on the master node, binding to master_addr fails with +"Cannot assign requested address". + +The fix: use `get_ip()` (local node IP) instead of `master_addr`. +""" + +import pytest +import zmq + +from vllm.distributed.device_communicators.shm_broadcast import MessageQueue +from vllm.utils.network_utils import get_ip + + +def test_mq_bind_with_local_ip(): + """MessageQueue with remote readers should successfully bind + when connect_ip is the local node's IP.""" + # n_reader=2, n_local_reader=1 means 1 remote reader, + # which triggers the remote ZMQ socket bind. + mq = MessageQueue( + n_reader=2, + n_local_reader=1, + connect_ip=get_ip(), + ) + handle = mq.export_handle() + assert handle.remote_subscribe_addr is not None + # The bound address should contain our local IP + local_ip = get_ip() + assert ( + local_ip in handle.remote_subscribe_addr + or f"[{local_ip}]" in handle.remote_subscribe_addr + ) + del mq + + +def test_mq_bind_with_non_local_ip_fails(): + """MessageQueue should fail to bind when connect_ip is a + non-local IP address (simulating the bug where master_addr + from a different node was used).""" + # Use a non-local IP that we definitely can't bind to. + # 198.51.100.1 is from TEST-NET-2 (RFC 5737), never locally assigned. + non_local_ip = "198.51.100.1" + with pytest.raises(zmq.error.ZMQError, match="Cannot assign requested address"): + MessageQueue( + n_reader=2, + n_local_reader=1, + connect_ip=non_local_ip, + ) + + +def test_mq_bind_defaults_to_local_ip(): + """When connect_ip is None, MessageQueue should auto-detect + the local IP and bind successfully.""" + mq = MessageQueue( + n_reader=2, + n_local_reader=1, + connect_ip=None, # should fallback to get_ip() + ) + handle = mq.export_handle() + assert handle.remote_subscribe_addr is not None + del mq + + +if __name__ == "__main__": + test_mq_bind_with_local_ip() + print("PASSED: test_mq_bind_with_local_ip") + test_mq_bind_with_non_local_ip_fails() + print("PASSED: test_mq_bind_with_non_local_ip_fails") + test_mq_bind_defaults_to_local_ip() + print("PASSED: test_mq_bind_defaults_to_local_ip") + print("\nAll tests passed!") diff --git a/asian_ai/performance/vLLM/tests/distributed/test_multi_node_assignment.py b/asian_ai/performance/vLLM/tests/distributed/test_multi_node_assignment.py new file mode 100644 index 0000000000000000000000000000000000000000..0060ba097f06ff5c2f5bb4c5055a37d6977ad3ab --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_multi_node_assignment.py @@ -0,0 +1,64 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Make sure ray assigns GPU workers to the correct node. + +Run: +```sh +cd $VLLM_PATH/tests + +pytest distributed/test_multi_node_assignment.py +``` +""" + +import os + +import pytest +import ray +from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy + +from vllm import initialize_ray_cluster +from vllm.config import ParallelConfig +from vllm.utils.network_utils import get_ip +from vllm.v1.executor.ray_utils import _wait_until_pg_removed + +VLLM_MULTI_NODE = os.getenv("VLLM_MULTI_NODE", "0") == "1" + + +@pytest.mark.skipif( + not VLLM_MULTI_NODE, reason="Need at least 2 nodes to run the test." +) +def test_multi_node_assignment() -> None: + # NOTE: important to keep this class definition here + # to let ray use cloudpickle to serialize it. + class Actor: + def get_ip(self): + return get_ip() + + for _ in range(10): + config = ParallelConfig(1, 2) + initialize_ray_cluster(config) + + current_ip = get_ip() + workers = [] + for bundle_id, bundle in enumerate(config.placement_group.bundle_specs): + if not bundle.get("GPU", 0): + continue + scheduling_strategy = PlacementGroupSchedulingStrategy( + placement_group=config.placement_group, + placement_group_capture_child_tasks=True, + placement_group_bundle_index=bundle_id, + ) + + worker = ray.remote( + num_cpus=0, + num_gpus=1, + scheduling_strategy=scheduling_strategy, + )(Actor).remote() + worker_ip = ray.get(worker.get_ip.remote()) + assert worker_ip == current_ip + workers.append(worker) + + for worker in workers: + ray.kill(worker) + + _wait_until_pg_removed(config.placement_group) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_multiproc_executor.py b/asian_ai/performance/vLLM/tests/distributed/test_multiproc_executor.py new file mode 100644 index 0000000000000000000000000000000000000000..992fe3c20c3c53cefac52d9d5b6e9ec958d3a820 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_multiproc_executor.py @@ -0,0 +1,444 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +""" +Integration tests for MultiprocExecutor at the executor level. +This test directly tests the executor without going through the LLM interface, +focusing on executor initialization, RPC calls, and distributed execution. +""" + +import multiprocessing +import os +import socket + +from tests.utils import multi_gpu_test +from vllm.config import VllmConfig +from vllm.engine.arg_utils import EngineArgs +from vllm.v1.core.sched.output import SchedulerOutput +from vllm.v1.executor.multiproc_executor import MultiprocExecutor + +MODEL = "facebook/opt-125m" + + +def create_vllm_config( + tensor_parallel_size: int = 1, + pipeline_parallel_size: int = 1, + max_model_len: int = 256, + gpu_memory_utilization: float = 0.3, + distributed_executor_backend: str = "mp", + nnodes: int = 1, + node_rank: int = 0, + master_port: int = 0, +) -> VllmConfig: + """Create a VllmConfig for testing using EngineArgs.""" + engine_args = EngineArgs( + model=MODEL, + tensor_parallel_size=tensor_parallel_size, + pipeline_parallel_size=pipeline_parallel_size, + max_model_len=max_model_len, + gpu_memory_utilization=gpu_memory_utilization, + distributed_executor_backend=distributed_executor_backend, + enforce_eager=True, + ) + vllm_config = engine_args.create_engine_config() + + # Override distributed node settings if needed + if nnodes > 1 or node_rank > 0: + vllm_config.parallel_config.nnodes = nnodes + vllm_config.parallel_config.node_rank = node_rank + vllm_config.parallel_config.master_port = master_port + if nnodes > 1: + vllm_config.parallel_config.disable_custom_all_reduce = True + + return vllm_config + + +def create_test_scheduler_output(num_requests: int = 1) -> SchedulerOutput: + """Create a minimal SchedulerOutput for testing.""" + # This is a simplified version - in practice you'd need proper + # SchedulerOutput construction based on the actual vLLM v1 API + return SchedulerOutput( + scheduled_new_reqs=[], + scheduled_resumed_reqs=[], + scheduled_running_reqs=[], + num_scheduled_tokens={}, + total_num_scheduled_tokens=0, + ) + + +def test_multiproc_executor_initialization(): + """Test that MultiprocExecutor can be initialized with proper config.""" + vllm_config = create_vllm_config( + tensor_parallel_size=1, + pipeline_parallel_size=1, + ) + + # Create executor - this should initialize workers + executor = MultiprocExecutor(vllm_config=vllm_config) + + # Verify executor properties + assert executor.world_size == 1, "World size should be 1 for single GPU" + assert executor.local_world_size == 1, "Local world size should be 1" + assert hasattr(executor, "workers"), "Executor should have workers" + assert len(executor.workers) == 1, "Should have 1 worker for single GPU" + + # Clean up + executor.shutdown() + + +@multi_gpu_test(num_gpus=2) +def test_multiproc_executor_initialization_tensor_parallel(): + """Test MultiprocExecutor initialization with tensor parallelism.""" + vllm_config = create_vllm_config( + tensor_parallel_size=2, + pipeline_parallel_size=1, + ) + + # Create executor + executor = MultiprocExecutor(vllm_config=vllm_config) + + # Verify executor properties + assert executor.world_size == 2, "World size should be 2 for TP=2" + assert executor.local_world_size == 2, "Local world size should be 2" + assert len(executor.workers) == 2, "Should have 2 workers for TP=2" + + # Verify output rank calculation + output_rank = executor._get_output_rank() + assert output_rank == 0, "Output rank should be 0 for TP=2, PP=1" + + # Clean up + executor.shutdown() + + +@multi_gpu_test(num_gpus=2) +def test_multiproc_executor_collective_rpc(): + """Test collective RPC calls to all workers.""" + vllm_config = create_vllm_config( + tensor_parallel_size=2, + pipeline_parallel_size=1, + ) + + # Create executor + executor = MultiprocExecutor(vllm_config=vllm_config) + + try: + # Test check_health RPC - should work without errors + executor.check_health() + + # Test that RPC works correctly + # Note: We're just testing that the RPC mechanism works, + # not testing actual model execution here + assert not executor.is_failed, "Executor should not be in failed state" + + finally: + # Clean up + executor.shutdown() + + +def test_multiproc_executor_failure_callback(): + """Test failure callback registration and invocation.""" + vllm_config = create_vllm_config( + tensor_parallel_size=1, + pipeline_parallel_size=1, + ) + + executor = MultiprocExecutor(vllm_config=vllm_config) + + try: + # Test callback registration + callback_invoked = [] + + def test_callback(): + callback_invoked.append(True) + + # Register callback + executor.register_failure_callback(test_callback) + + # Callback should not be invoked yet + assert len(callback_invoked) == 0, "Callback should not be invoked immediately" + + # Simulate failure + executor.is_failed = True + + # Register another callback - should be invoked immediately + executor.register_failure_callback(test_callback) + assert len(callback_invoked) == 1, ( + "Callback should be invoked when executor is failed" + ) + + finally: + # Clean up + executor.shutdown() + + +@multi_gpu_test(num_gpus=2) +def test_multiproc_executor_worker_monitor(): + """Test that worker monitor is set up correctly.""" + vllm_config = create_vllm_config( + tensor_parallel_size=2, + pipeline_parallel_size=1, + ) + + executor = MultiprocExecutor(vllm_config=vllm_config) + + try: + # Verify all worker processes are alive + for worker in executor.workers: + assert worker.proc.is_alive(), f"Worker rank {worker.rank} should be alive" + + # Verify executor is not in failed state + assert not executor.is_failed, "Executor should not be in failed state" + + finally: + # Clean up + executor.shutdown() + + # After shutdown, workers should be terminated + import time + + time.sleep(0.5) # Give processes time to terminate + for worker in executor.workers: + assert not worker.proc.is_alive(), ( + f"Worker rank {worker.rank} should terminate after shutdown" + ) + + +@multi_gpu_test(num_gpus=2) +def test_multiproc_executor_get_response_message_queues(): + """Test message queue retrieval for different ranks.""" + vllm_config = create_vllm_config( + tensor_parallel_size=2, + pipeline_parallel_size=1, + ) + + executor = MultiprocExecutor(vllm_config=vllm_config) + + try: + # Get all message queues + all_queues = executor.get_response_mqs() + assert len(all_queues) == 2, "Should have 2 message queues for 2 workers" + + # Get message queue for specific rank + rank0_queue = executor.get_response_mqs(unique_reply_rank=0) + assert len(rank0_queue) == 1, "Should have 1 message queue for rank 0" + + rank1_queue = executor.get_response_mqs(unique_reply_rank=1) + assert len(rank1_queue) == 1, "Should have 1 message queue for rank 1" + + finally: + # Clean up + executor.shutdown() + + +def test_multiproc_executor_shutdown_cleanup(): + """Test that shutdown properly cleans up resources.""" + vllm_config = create_vllm_config( + tensor_parallel_size=1, + pipeline_parallel_size=1, + ) + + executor = MultiprocExecutor(vllm_config=vllm_config) + + # Verify executor is set up + assert hasattr(executor, "workers"), "Executor should have workers" + assert len(executor.workers) > 0, "Should have at least one worker" + + # Shutdown + executor.shutdown() + + # Verify cleanup + import time + + time.sleep(0.5) # Give processes time to terminate + + for worker in executor.workers: + assert not worker.proc.is_alive(), "Worker processes should be terminated" + + # Verify shutdown event is set + assert executor.shutdown_event.is_set(), "Shutdown event should be set" + + # Multiple shutdowns should be safe (idempotent) + executor.shutdown() + executor.shutdown() + + +@multi_gpu_test(num_gpus=4) +def test_multiproc_executor_pipeline_parallel(): + """Test MultiprocExecutor with pipeline parallelism.""" + vllm_config = create_vllm_config( + tensor_parallel_size=2, + pipeline_parallel_size=2, + ) + + executor = MultiprocExecutor(vllm_config=vllm_config) + + try: + # Verify executor properties + assert executor.world_size == 4, "World size should be 4 for TP=2, PP=2" + assert len(executor.workers) == 4, "Should have 4 workers" + + # Verify output rank calculation + # For TP=2, PP=2: output should be from the last PP stage (ranks 2-3) + # Specifically rank 2 (first rank of last PP stage) + output_rank = executor._get_output_rank() + assert output_rank == 2, "Output rank should be 2 (first rank of last PP stage)" + + # V2 model runner uses one extra batch to overlap async scheduling. + expected_concurrent_batches = 2 + int( + vllm_config.scheduler_config.async_scheduling + and vllm_config.use_v2_model_runner + ) + assert vllm_config.max_concurrent_batches == expected_concurrent_batches, ( + "Max concurrent batches should follow the configured PP/async " + "scheduling policy" + ) + + finally: + # Clean up + executor.shutdown() + + +def test_multiproc_executor_properties(): + """Test various executor properties and configurations.""" + vllm_config = create_vllm_config( + tensor_parallel_size=1, + pipeline_parallel_size=1, + ) + + executor = MultiprocExecutor(vllm_config=vllm_config) + + try: + # Test supports_pp property + assert MultiprocExecutor.supports_pp is True, ( + "MultiprocExecutor should support pipeline parallelism" + ) + + # Test world_size calculation + assert executor.world_size == ( + executor.parallel_config.tensor_parallel_size + * executor.parallel_config.pipeline_parallel_size + ), "World size should equal TP * PP" + + # Test local_world_size calculation + assert executor.local_world_size == ( + executor.parallel_config.world_size // executor.parallel_config.nnodes + ), "Local world size should be world_size / nnodes" + + finally: + # Clean up + executor.shutdown() + + +@multi_gpu_test(num_gpus=4) +def test_multiproc_executor_multi_node(): + """ + Test MultiprocExecutor with multi-node configuration. + This simulates 2 nodes with TP=4: + - Node 0 (rank 0): Uses GPUs 0,1 (CUDA_VISIBLE_DEVICES=0,1) with TP=2 + - Node 1 (rank 1): Uses GPUs 2,3 (CUDA_VISIBLE_DEVICES=2,3) with TP=2 + Total world_size = 4, nnodes = 2 + """ + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.bind(("", 0)) + port = s.getsockname()[1] + # symm_mem does not work for simulating multi instance in single node + os.environ["VLLM_ALLREDUCE_USE_SYMM_MEM"] = "0" + + def run_node(node_rank: int, result_queue: multiprocessing.Queue, port: int): + """Run a single node's executor.""" + executor = None + try: + # Set CUDA_VISIBLE_DEVICES for this node + if node_rank == 0: + os.environ["CUDA_VISIBLE_DEVICES"] = "0,1" + else: + os.environ["CUDA_VISIBLE_DEVICES"] = "2,3" + + # Create config for this node + vllm_config = create_vllm_config( + tensor_parallel_size=4, # Total TP across all nodes + pipeline_parallel_size=1, + nnodes=2, # 2 nodes + node_rank=node_rank, + master_port=port, # same port + ) + + # Create executor for this node + executor = MultiprocExecutor(vllm_config=vllm_config) + + # Verify node-specific properties + assert executor.world_size == 4, ( + f"World size should be 4 on node {node_rank}" + ) + assert executor.local_world_size == 2, ( + f"Local world size should be 2 on node {node_rank}" + ) + assert len(executor.workers) == 2, ( + f"Should have 2 local workers on node {node_rank}" + ) + + # Verify worker ranks are correct for this node + expected_ranks = [node_rank * 2, node_rank * 2 + 1] + actual_ranks = sorted([w.rank for w in executor.workers]) + assert actual_ranks == expected_ranks, ( + f"Node {node_rank} should have workers " + f"with ranks {expected_ranks}, got {actual_ranks}" + ) + # Verify all workers are alive + for worker in executor.workers: + assert worker.proc.is_alive(), ( + f"Worker rank {worker.rank} should be alive on node {node_rank}" + ) + # executor.gen + # Put success result in queue BEFORE shutdown to avoid hanging + result_queue.put({"node": node_rank, "success": True}) + import time + + time.sleep(2) + executor.shutdown() + except Exception as e: + # Put failure result in queue + result_queue.put({"node": node_rank, "success": False, "error": str(e)}) + raise e + finally: + if executor is not None: + executor.shutdown() + + # Create a queue to collect results from both processes + result_queue: multiprocessing.Queue[dict[str, int | bool]] = multiprocessing.Queue() + + # Start both node processes + processes = [] + for node_rank in range(2): + p = multiprocessing.Process( + target=run_node, + args=(node_rank, result_queue, port), + name=f"Node{node_rank}", + ) + p.start() + processes.append(p) + + # Wait for both processes to complete + all_completed = True + for p in processes: + p.join(timeout=60) + if p.is_alive(): + p.terminate() + p.join(timeout=20) + if p.is_alive(): + p.kill() + p.join() + all_completed = False + + # Check results from both nodes + results: list[dict[str, int | bool]] = [] + while len(results) < 2: + try: + result = result_queue.get(timeout=1) + results.append(result) + except Exception: + pass + assert all_completed, "Not all processes completed successfully" + assert len(results) == 2, f"Expected 2 results, got {len(results)}" + assert results[0]["success"], f"Node 0 failed: {results[0]}" + assert results[1]["success"], f"Node 1 failed: {results[1]}" diff --git a/asian_ai/performance/vLLM/tests/distributed/test_nccl_symm_mem.py b/asian_ai/performance/vLLM/tests/distributed/test_nccl_symm_mem.py new file mode 100644 index 0000000000000000000000000000000000000000..b8f5b7b474244e99f36e5c430e838675c7b733dd --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_nccl_symm_mem.py @@ -0,0 +1,223 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import random +import typing + +import pytest +import torch +import torch.distributed as dist +import torch.multiprocessing as mp + +import vllm.envs as envs +from tests.utils import ensure_current_vllm_config +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.distributed.device_communicators.cuda_communicator import CudaCommunicator +from vllm.distributed.device_communicators.pynccl import register_nccl_symmetric_ops +from vllm.distributed.device_communicators.pynccl_allocator import ( + get_nccl_mem_pool, + is_symmetric_memory_enabled, +) +from vllm.distributed.parallel_state import ( + get_tp_group, + init_distributed_environment, + initialize_model_parallel, +) +from vllm.platforms import current_platform +from vllm.utils.system_utils import update_environment_variables + +torch.manual_seed(42) +random.seed(44) + +test_size_elements = 4 * 1024 * 1024 + + +def nccl_symm_mem_allreduce_worker(local_rank: int, world_size: int): + monkeypatch = pytest.MonkeyPatch() + with monkeypatch.context() as m: + m.delenv("CUDA_VISIBLE_DEVICES", raising=False) + dtype = torch.bfloat16 + device = torch.device(f"cuda:{local_rank}") + torch.accelerator.set_device_index(device) + torch.set_default_device(device) + torch.set_default_dtype(dtype) + update_environment_variables( + { + "RANK": str(local_rank), + "LOCAL_RANK": str(local_rank), + "WORLD_SIZE": str(world_size), + "MASTER_ADDR": "localhost", + "MASTER_PORT": "12345", + } + ) + + init_distributed_environment() + with ensure_current_vllm_config(): + initialize_model_parallel(tensor_model_parallel_size=world_size) + + cuda_communicator = typing.cast( + CudaCommunicator, get_tp_group().device_communicator + ) + pynccl_comm = cuda_communicator.pynccl_comm + if get_nccl_mem_pool() is None: + pytest.skip( + "NCCL allocator compilation failed (probably missing NCCL headers)." + ) + if not is_symmetric_memory_enabled(): + pytest.skip("NCCL symmetric memory allreduce is disabled.") + + register_nccl_symmetric_ops(pynccl_comm) + input = torch.randint(1, 23, (test_size_elements,), dtype=dtype, device=device) + input_clone = input.clone() + output = torch.ops.vllm.all_reduce_symmetric_with_copy(input) + assert output is not None + + group = get_tp_group().device_group + dist.all_reduce(input_clone, group=group) + torch.testing.assert_close(output, input_clone, atol=2.5, rtol=0.1) + + +@pytest.mark.skipif( + not current_platform.is_cuda(), + reason="NCCLSymmMemAllreduce is only available for CUDA platforms.", +) +@pytest.mark.parametrize("world_size", [2]) +@pytest.mark.skipif(envs.VLLM_TARGET_DEVICE not in ["cuda"], reason="Only test on CUDA") +def test_nccl_symm_mem_allreduce(monkeypatch: pytest.MonkeyPatch, world_size): + if world_size > torch.accelerator.device_count(): + pytest.skip("Not enough GPUs to run the test.") + + # Enable SymmMemCommunicator + monkeypatch.setenv("VLLM_USE_NCCL_SYMM_MEM", "1") + monkeypatch.setenv("NCCL_NVLS_ENABLE", "1") + monkeypatch.setenv("NCCL_CUMEM_ENABLE", "1") + + mp.spawn(nccl_symm_mem_allreduce_worker, args=(world_size,), nprocs=world_size) + cleanup_dist_env_and_memory() + + +def nccl_symm_mem_allgather_worker(local_rank: int, world_size: int): + monkeypatch = pytest.MonkeyPatch() + with monkeypatch.context() as m: + m.delenv("CUDA_VISIBLE_DEVICES", raising=False) + dtype = torch.bfloat16 + device = torch.device(f"cuda:{local_rank}") + torch.accelerator.set_device_index(device) + torch.set_default_device(device) + torch.set_default_dtype(dtype) + update_environment_variables( + { + "RANK": str(local_rank), + "LOCAL_RANK": str(local_rank), + "WORLD_SIZE": str(world_size), + "MASTER_ADDR": "localhost", + "MASTER_PORT": "12346", + } + ) + + init_distributed_environment() + with ensure_current_vllm_config(): + initialize_model_parallel(tensor_model_parallel_size=world_size) + + cuda_communicator = typing.cast( + CudaCommunicator, get_tp_group().device_communicator + ) + if get_nccl_mem_pool() is None: + pytest.skip( + "NCCL allocator compilation failed (probably missing NCCL headers)." + ) + if not is_symmetric_memory_enabled(): + pytest.skip("NCCL symmetric memory is disabled.") + + per_rank_size = test_size_elements // world_size + input_tensor = torch.randint( + 1, 23, (per_rank_size,), dtype=dtype, device=device + ) + output = cuda_communicator.all_gatherv(input_tensor, dim=0) + + group = get_tp_group().device_group + expected = torch.empty(test_size_elements, dtype=dtype, device=device) + dist.all_gather_into_tensor(expected, input_tensor, group=group) + torch.testing.assert_close(output, expected, atol=0.0, rtol=0.0) + + +@pytest.mark.skipif( + not current_platform.is_cuda(), + reason="NCCL symmetric memory is only available for CUDA platforms.", +) +@pytest.mark.parametrize("world_size", [2]) +@pytest.mark.skipif(envs.VLLM_TARGET_DEVICE not in ["cuda"], reason="Only test on CUDA") +def test_nccl_symm_mem_allgather(monkeypatch: pytest.MonkeyPatch, world_size): + if world_size > torch.accelerator.device_count(): + pytest.skip("Not enough GPUs to run the test.") + + monkeypatch.setenv("VLLM_USE_NCCL_SYMM_MEM", "1") + monkeypatch.setenv("NCCL_NVLS_ENABLE", "1") + monkeypatch.setenv("NCCL_CUMEM_ENABLE", "1") + + mp.spawn(nccl_symm_mem_allgather_worker, args=(world_size,), nprocs=world_size) + cleanup_dist_env_and_memory() + + +def nccl_symm_mem_reduce_scatter_worker(local_rank: int, world_size: int): + monkeypatch = pytest.MonkeyPatch() + with monkeypatch.context() as m: + m.delenv("CUDA_VISIBLE_DEVICES", raising=False) + dtype = torch.bfloat16 + device = torch.device(f"cuda:{local_rank}") + torch.accelerator.set_device_index(device) + torch.set_default_device(device) + torch.set_default_dtype(dtype) + update_environment_variables( + { + "RANK": str(local_rank), + "LOCAL_RANK": str(local_rank), + "WORLD_SIZE": str(world_size), + "MASTER_ADDR": "localhost", + "MASTER_PORT": "12347", + } + ) + + init_distributed_environment() + with ensure_current_vllm_config(): + initialize_model_parallel(tensor_model_parallel_size=world_size) + + cuda_communicator = typing.cast( + CudaCommunicator, get_tp_group().device_communicator + ) + if get_nccl_mem_pool() is None: + pytest.skip( + "NCCL allocator compilation failed (probably missing NCCL headers)." + ) + if not is_symmetric_memory_enabled(): + pytest.skip("NCCL symmetric memory is disabled.") + + per_rank_size = test_size_elements // world_size + input_tensor = torch.randint( + 1, 23, (test_size_elements,), dtype=dtype, device=device + ) + input_clone = input_tensor.clone() + output = cuda_communicator.reduce_scatter(input_tensor, dim=0) + + group = get_tp_group().device_group + expected = torch.empty(per_rank_size, dtype=dtype, device=device) + dist.reduce_scatter_tensor(expected, input_clone, group=group) + torch.testing.assert_close(output, expected, atol=2.5, rtol=0.1) + + +@pytest.mark.skipif( + not current_platform.is_cuda(), + reason="NCCL symmetric memory is only available for CUDA platforms.", +) +@pytest.mark.parametrize("world_size", [2]) +@pytest.mark.skipif(envs.VLLM_TARGET_DEVICE not in ["cuda"], reason="Only test on CUDA") +def test_nccl_symm_mem_reduce_scatter(monkeypatch: pytest.MonkeyPatch, world_size): + if world_size > torch.accelerator.device_count(): + pytest.skip("Not enough GPUs to run the test.") + + monkeypatch.setenv("VLLM_USE_NCCL_SYMM_MEM", "1") + monkeypatch.setenv("NCCL_NVLS_ENABLE", "1") + monkeypatch.setenv("NCCL_CUMEM_ENABLE", "1") + + mp.spawn(nccl_symm_mem_reduce_scatter_worker, args=(world_size,), nprocs=world_size) + cleanup_dist_env_and_memory() diff --git a/asian_ai/performance/vLLM/tests/distributed/test_node_count.py b/asian_ai/performance/vLLM/tests/distributed/test_node_count.py new file mode 100644 index 0000000000000000000000000000000000000000..c85d7976305e725eaa08a75043849035828848da --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_node_count.py @@ -0,0 +1,46 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import os + +import torch.distributed as dist + +from vllm.distributed.parallel_state import _node_count +from vllm.distributed.utils import StatelessProcessGroup +from vllm.utils.network_utils import get_ip, get_open_port + +if __name__ == "__main__": + dist.init_process_group(backend="gloo") + + rank = dist.get_rank() + world_size = dist.get_world_size() + + if rank == 0: + port = get_open_port() + ip = get_ip() + dist.broadcast_object_list([ip, port], src=0) + else: + recv = [None, None] + dist.broadcast_object_list(recv, src=0) + ip, port = recv + + stateless_pg = StatelessProcessGroup.create(ip, port, rank, world_size) + + for pg in [dist.group.WORLD, stateless_pg]: + test_result = _node_count(pg) + + # Expected node count based on environment variable) + expected = int(os.environ.get("NUM_NODES", "1")) + + assert test_result == expected, f"Expected {expected} nodes, got {test_result}" + + if pg == dist.group.WORLD: + print( + f"Node count test passed! Got {test_result} nodes " + f"when using torch distributed!" + ) + else: + print( + f"Node count test passed! Got {test_result} nodes " + f"when using StatelessProcessGroup!" + ) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_packed_tensor.py b/asian_ai/performance/vLLM/tests/distributed/test_packed_tensor.py new file mode 100644 index 0000000000000000000000000000000000000000..826156808b187224d44af05ff02a1db1fa903c83 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_packed_tensor.py @@ -0,0 +1,775 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for packed tensor broadcasting functionality. + +Unit tests for packed_nccl_broadcast_producer and packed_nccl_broadcast_consumer. +These utilities enable efficient batched tensor transfer over NCCL. +""" + +import pytest +import torch + +from vllm.distributed.weight_transfer.nccl_engine import NCCLWeightTransferUpdateInfo +from vllm.distributed.weight_transfer.packed_tensor import ( + pack_tensors, + packed_ipc_consumer, + packed_ipc_producer, + packed_nccl_broadcast_consumer, + packed_nccl_broadcast_producer, + unpack_tensor, +) + + +class MockCommunicationGroup: + """Mock communication group for testing producer broadcast operations.""" + + def __init__(self): + self.broadcasted_tensors: list[torch.Tensor] = [] + self.broadcast_count = 0 + self.device = torch.device("cuda:0") + + def broadcast(self, tensor, src): + """Mock broadcast that stores the tensor for later verification.""" + self.broadcasted_tensors.append(tensor.clone()) + self.broadcast_count += 1 + + +class MockConsumerCommunicationGroup: + """Mock communication group for consumer that returns pre-stored tensors.""" + + def __init__(self, tensors_to_return: list[torch.Tensor]): + self.tensors_to_return = tensors_to_return + self.current_index = 0 + self.device = torch.device("cuda:0") + + def broadcast(self, tensor, src): + """Mock broadcast that fills the tensor with pre-stored data.""" + if self.current_index < len(self.tensors_to_return): + tensor.copy_(self.tensors_to_return[self.current_index]) + self.current_index += 1 + + +def create_mock_model_params( + num_layers: int = 3, + dtype: torch.dtype = torch.float32, +) -> list[tuple[str, torch.Tensor]]: + """Create mock model parameters for testing.""" + params = [] + for i in range(num_layers): + params.append((f"layer{i}.weight", torch.randn(10, 20, dtype=dtype))) + params.append((f"layer{i}.bias", torch.randn(10, dtype=dtype))) + return params + + +def create_state_dict_info( + params: list[tuple[str, torch.Tensor]], +) -> dict[str, tuple[tuple[int, ...], torch.dtype]]: + """Create state dict info (name -> (shape, dtype)) from params.""" + return {name: (tuple(tensor.shape), tensor.dtype) for name, tensor in params} + + +# --- Unit Tests: NCCLWeightTransferUpdateInfo packed field --- + + +class TestNCCLWeightTransferUpdateInfoPacked: + """Test NCCLWeightTransferUpdateInfo dataclass packed field.""" + + def test_packed_default_false(self): + """Test that packed defaults to False.""" + info = NCCLWeightTransferUpdateInfo( + names=["layer.weight"], + dtype_names=["float32"], + shapes=[[10, 10]], + ) + assert info.packed is False + + def test_packed_can_be_set_true(self): + """Test that packed can be set to True.""" + info = NCCLWeightTransferUpdateInfo( + names=["layer.weight"], + dtype_names=["float32"], + shapes=[[10, 10]], + packed=True, + ) + assert info.packed is True + + +# --- Unit Tests: packed_nccl_broadcast_producer --- + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA not available") +class TestPackedBroadcastProducer: + """Test packed_nccl_broadcast_producer function.""" + + def test_producer_empty_iterator(self): + """Test producer handles empty iterator gracefully.""" + mock_group = MockCommunicationGroup() + + packed_nccl_broadcast_producer( + iterator=iter([]), + group=mock_group, + src=0, + post_iter_func=lambda x: x[1], + buffer_size_bytes=1000, + ) + + # No broadcasts for empty iterator + assert mock_group.broadcast_count == 0 + + +# --- Integration Tests: Producer-Consumer Roundtrip --- + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA not available") +class TestPackedBroadcastRoundtrip: + """Test producer-consumer roundtrip behavior.""" + + @pytest.mark.parametrize("dtype", [torch.float32, torch.float16, torch.bfloat16]) + def test_roundtrip_different_dtypes(self, dtype): + """Test roundtrip with different data types.""" + params = create_mock_model_params(num_layers=2, dtype=dtype) + params_cuda = [(name, tensor.cuda()) for name, tensor in params] + + buffer_size = 1000 + producer_group = MockCommunicationGroup() + + packed_nccl_broadcast_producer( + iterator=iter(params_cuda), + group=producer_group, + src=0, + post_iter_func=lambda x: x[1], + buffer_size_bytes=buffer_size, + ) + + consumer_group = MockConsumerCommunicationGroup( + producer_group.broadcasted_tensors + ) + + state_dict_info = create_state_dict_info(params_cuda) + unpacked_tensors = {} + + def post_unpack_func(tensor_list): + for name, tensor in tensor_list: + unpacked_tensors[name] = tensor.clone() + + packed_nccl_broadcast_consumer( + iterator=iter(state_dict_info.items()), + group=consumer_group, + src=0, + post_unpack_func=post_unpack_func, + buffer_size_bytes=buffer_size, + ) + + # Verify roundtrip preserves data + for name, original_tensor in params_cuda: + assert name in unpacked_tensors + unpacked = unpacked_tensors[name] + assert unpacked.dtype == dtype + assert torch.allclose(unpacked, original_tensor, rtol=1e-4, atol=1e-6) + + def test_roundtrip_mixed_dtypes(self): + """Test roundtrip with mixed data types.""" + # Create params with mixed dtypes + params = [ + ("layer1.weight", torch.randn(10, 20, dtype=torch.float32).cuda()), + ("layer1.bias", torch.randn(10, dtype=torch.float16).cuda()), + ("layer2.weight", torch.randn(20, 30, dtype=torch.bfloat16).cuda()), + ] + + buffer_size = 500 + producer_group = MockCommunicationGroup() + + packed_nccl_broadcast_producer( + iterator=iter(params), + group=producer_group, + src=0, + post_iter_func=lambda x: x[1], + buffer_size_bytes=buffer_size, + ) + + consumer_group = MockConsumerCommunicationGroup( + producer_group.broadcasted_tensors + ) + + state_dict_info = create_state_dict_info(params) + unpacked_tensors = {} + + def post_unpack_func(tensor_list): + for name, tensor in tensor_list: + unpacked_tensors[name] = tensor.clone() + + packed_nccl_broadcast_consumer( + iterator=iter(state_dict_info.items()), + group=consumer_group, + src=0, + post_unpack_func=post_unpack_func, + buffer_size_bytes=buffer_size, + ) + + # Verify all params roundtrip correctly with correct dtypes + for name, original_tensor in params: + assert name in unpacked_tensors + unpacked = unpacked_tensors[name] + assert unpacked.shape == original_tensor.shape + assert unpacked.dtype == original_tensor.dtype + assert torch.allclose(unpacked, original_tensor, rtol=1e-4, atol=1e-6) + + @pytest.mark.parametrize("target_size", [100, 100000]) + def test_roundtrip_different_batch_sizes(self, target_size): + """Test roundtrip with different target batch sizes.""" + params = create_mock_model_params(num_layers=5) + params_cuda = [(name, tensor.cuda()) for name, tensor in params] + + producer_group = MockCommunicationGroup() + + packed_nccl_broadcast_producer( + iterator=iter(params_cuda), + group=producer_group, + src=0, + post_iter_func=lambda x: x[1], + buffer_size_bytes=target_size, + ) + + consumer_group = MockConsumerCommunicationGroup( + producer_group.broadcasted_tensors + ) + + state_dict_info = create_state_dict_info(params_cuda) + unpacked_tensors = {} + + def post_unpack_func(tensor_list): + for name, tensor in tensor_list: + unpacked_tensors[name] = tensor.clone() + + packed_nccl_broadcast_consumer( + iterator=iter(state_dict_info.items()), + group=consumer_group, + src=0, + post_unpack_func=post_unpack_func, + buffer_size_bytes=target_size, + ) + + # Verify all params roundtrip correctly + assert len(unpacked_tensors) == len(params) + for name, original_tensor in params_cuda: + assert name in unpacked_tensors + assert torch.allclose( + unpacked_tensors[name], original_tensor, rtol=1e-5, atol=1e-7 + ) + + def test_roundtrip_non_contiguous_tensors(self): + """Test roundtrip with non-contiguous tensors from the trainer.""" + # Create non-contiguous tensors (simulating trainer outputs) + # Transposed tensors are non-contiguous + weight1 = torch.randn(20, 10, dtype=torch.float32).cuda().T + # Sliced tensors with step are non-contiguous + weight2 = torch.randn(40, 30, dtype=torch.float16).cuda()[::2, ::2] + # Permuted tensors are non-contiguous + weight3 = torch.randn(5, 10, 15, dtype=torch.bfloat16).cuda().permute(2, 0, 1) + + params = [ + ("layer1.weight", weight1), + ("layer2.weight", weight2), + ("layer3.weight", weight3), + ] + + # Verify tensors are indeed non-contiguous + for name, tensor in params: + assert not tensor.is_contiguous(), f"{name} should be non-contiguous" + + buffer_size = 500 + producer_group = MockCommunicationGroup() + + packed_nccl_broadcast_producer( + iterator=iter(params), + group=producer_group, + src=0, + post_iter_func=lambda x: x[1], + buffer_size_bytes=buffer_size, + ) + + consumer_group = MockConsumerCommunicationGroup( + producer_group.broadcasted_tensors + ) + + state_dict_info = create_state_dict_info(params) + unpacked_tensors = {} + + def post_unpack_func(tensor_list): + for name, tensor in tensor_list: + unpacked_tensors[name] = tensor.clone() + + packed_nccl_broadcast_consumer( + iterator=iter(state_dict_info.items()), + group=consumer_group, + src=0, + post_unpack_func=post_unpack_func, + buffer_size_bytes=buffer_size, + ) + + # Verify all non-contiguous params roundtrip correctly + for name, original_tensor in params: + assert name in unpacked_tensors + unpacked = unpacked_tensors[name] + assert unpacked.shape == original_tensor.shape + assert unpacked.dtype == original_tensor.dtype + assert torch.allclose(unpacked, original_tensor, rtol=1e-4, atol=1e-6) + + +# --- Unit Tests: unpack_tensor --- + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA not available") +class TestUnpackTensor: + """Test the shared unpack_tensor function.""" + + def test_unpack_produces_independent_copies(self): + """Verify unpacked tensors don't share memory with packed buffer.""" + original = torch.randn(10, dtype=torch.float32).cuda() + packed = original.contiguous().view(torch.uint8).view(-1) + + result = unpack_tensor( + packed, + names=["w"], + shapes=[[10]], + dtypes=[torch.float32], + tensor_sizes=[packed.numel()], + ) + + # Mutate the packed buffer + packed.zero_() + + # Unpacked tensor should be unaffected + assert torch.allclose(result[0][1], original) + + +# --- Unit Tests: pack_tensors --- + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA not available") +class TestPackTensors: + """Test the shared pack_tensors function.""" + + def test_pack_basic(self): + """Test packing a few tensors into one buffer.""" + params = [ + ("w1", torch.randn(10, 20, dtype=torch.float32).cuda()), + ("w2", torch.randn(5, dtype=torch.float16).cuda()), + ] + + chunk = pack_tensors( + iterator=iter(params), + post_iter_func=lambda x: x[1], + buffer_size_bytes=10_000_000, + ) + + assert chunk is not None + assert len(chunk.names) == 2 + assert chunk.names == ["w1", "w2"] + assert chunk.shapes == [[10, 20], [5]] + assert chunk.dtypes == [torch.float32, torch.float16] + assert chunk.packed_tensor.dtype == torch.uint8 + + def test_pack_respects_buffer_limit(self): + """Test that packing stops when buffer_size_bytes is exceeded.""" + params = [ + (f"w{i}", torch.randn(100, 100, dtype=torch.float32).cuda()) + for i in range(10) + ] + + chunk = pack_tensors( + iterator=iter(params), + post_iter_func=lambda x: x[1], + buffer_size_bytes=50_000, + ) + + assert chunk is not None + assert len(chunk.names) < 10 + + def test_pack_empty_iterator(self): + """Test that an empty iterator returns None.""" + chunk = pack_tensors( + iterator=iter([]), + post_iter_func=lambda x: x[1], + buffer_size_bytes=1000, + ) + assert chunk is None + + def test_pack_single_tensor_larger_than_buffer_warns(self): + """Test that a tensor exceeding buffer_size_bytes emits a warning.""" + big = torch.randn(1000, 1000, dtype=torch.float32).cuda() + params = [("big", big)] + + import warnings + + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter("always") + chunk = pack_tensors( + iterator=iter(params), + post_iter_func=lambda x: x[1], + buffer_size_bytes=100, + ) + assert chunk is not None + assert len(chunk.names) == 1 + assert any("exceeds buffer_size_bytes" in str(wi.message) for wi in w) + + def test_pack_unpack_roundtrip(self): + """Test pack then unpack produces identical tensors.""" + params = [ + ("a", torch.randn(8, 16, dtype=torch.float32).cuda()), + ("b", torch.randn(4, dtype=torch.float16).cuda()), + ("c", torch.randn(3, 5, 7, dtype=torch.bfloat16).cuda()), + ] + + chunk = pack_tensors( + iterator=iter(params), + post_iter_func=lambda x: x[1], + buffer_size_bytes=10_000_000, + ) + + assert chunk is not None + result = unpack_tensor( + chunk.packed_tensor, + chunk.names, + chunk.shapes, + chunk.dtypes, + chunk.tensor_sizes, + ) + + assert len(result) == len(params) + for (orig_name, orig_tensor), (res_name, res_tensor) in zip(params, result): + assert orig_name == res_name + assert res_tensor.shape == orig_tensor.shape + assert res_tensor.dtype == orig_tensor.dtype + assert torch.allclose(res_tensor, orig_tensor, rtol=1e-4, atol=1e-6) + + def test_pack_multiple_chunks(self): + """Test consuming an iterator across multiple pack_tensors calls.""" + params = [ + (f"w{i}", torch.randn(50, 50, dtype=torch.float32).cuda()) for i in range(6) + ] + it = iter(params) + + all_names = [] + chunks = [] + while True: + chunk = pack_tensors(it, lambda x: x[1], buffer_size_bytes=12_000) + if chunk is None: + break + chunks.append(chunk) + all_names.extend(chunk.names) + + assert len(chunks) > 1 + assert all_names == [f"w{i}" for i in range(6)] + + +# --- Unit Tests: packed_ipc_producer --- + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA not available") +class TestPackedIpcProducer: + """Test the packed_ipc_producer generator.""" + + def test_producer_yields_chunks(self): + """Test that the producer yields PackedIpcChunk objects.""" + params = [ + (f"w{i}", torch.randn(50, 50, dtype=torch.float32).cuda()) for i in range(6) + ] + + chunks = list( + packed_ipc_producer( + iterator=iter(params), + gpu_uuid="test-uuid", + post_iter_func=lambda x: x[1], + buffer_size_bytes=12_000, + ) + ) + + assert len(chunks) > 1 + + def test_producer_ipc_handle_has_uuid(self): + """Test that each chunk's ipc_handle is keyed by the given UUID.""" + params = [("w", torch.randn(10, dtype=torch.float32).cuda())] + + chunks = list( + packed_ipc_producer( + iterator=iter(params), + gpu_uuid="my-gpu-uuid", + post_iter_func=lambda x: x[1], + buffer_size_bytes=10_000_000, + ) + ) + + assert "my-gpu-uuid" in chunks[0].ipc_handle + + def test_producer_dtype_names_are_strings(self): + """Test that dtype_names are string representations.""" + params = [ + ("a", torch.randn(10, dtype=torch.float32).cuda()), + ("b", torch.randn(10, dtype=torch.float16).cuda()), + ] + + chunks = list( + packed_ipc_producer( + iterator=iter(params), + gpu_uuid="uuid", + post_iter_func=lambda x: x[1], + buffer_size_bytes=10_000_000, + ) + ) + + assert chunks[0].dtype_names == ["float32", "float16"] + + def test_producer_empty_iterator(self): + """Test producer with empty iterator yields nothing.""" + chunks = list( + packed_ipc_producer( + iterator=iter([]), + gpu_uuid="uuid", + post_iter_func=lambda x: x[1], + buffer_size_bytes=1000, + ) + ) + assert len(chunks) == 0 + + +# --- Integration Tests: IPC Producer-Consumer Roundtrip --- + + +def _ipc_consumer_worker(cmd_q, ack_q, result_q, done_event, device_index): + """Worker that consumes chunks streamed one at a time from the parent. + + CUDA IPC requires the consumer to be in a separate process from the + producer. The producer reuses a single IPC buffer between chunks, so + the parent must wait for our ack (sent after we copy the chunk to + CPU) before advancing the producer. + """ + try: + torch.accelerator.set_device_index(device_index) + all_results = [] + while True: + cd = cmd_q.get() + if cd is None: + break + result = packed_ipc_consumer( + ipc_handle=cd["ipc_handle"], + names=cd["names"], + shapes=cd["shapes"], + dtype_names=cd["dtype_names"], + tensor_sizes=cd["tensor_sizes"], + device_index=device_index, + ) + # .cpu() forces a GPU→CPU copy off the shared IPC buffer, so + # the producer is free to overwrite it once we ack. + all_results.extend([(name, tensor.cpu()) for name, tensor in result]) + del result + ack_q.put("ack") + result_q.put(("ok", all_results)) + except Exception as e: + result_q.put(("error", str(e))) + # Keep the process alive until the parent has finished reading from + # the result queue — torch serializes CPU tensors via fd sharing, + # which requires this process's resource-sharer server to be running. + done_event.wait(timeout=60) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA not available") +class TestPackedIpcRoundtrip: + """Test IPC producer-consumer roundtrip using real CUDA IPC. + + These tests spawn a child process for the consumer because + rebuild_cuda_tensor requires a separate process from the one that + called reduce_tensor. + """ + + def _get_gpu_uuid(self) -> str: + device_index = torch.cuda.current_device() + props = torch.cuda.get_device_properties(device_index) + return str(props.uuid) + + def _run_roundtrip(self, chunk_iter, device_index, timeout=30): + """Stream chunks through a child consumer one at a time. + + ``packed_ipc_producer`` reuses a single IPC buffer for every + chunk, so the producer must not be advanced until the consumer + has finished reading the current chunk. We enforce that with an + ack queue: the consumer puts ``"ack"`` after it has copied the + chunk to CPU, and only then do we pull the next chunk from the + generator. + + Returns ``(num_chunks, results)``. + """ + import multiprocessing as mp + + ctx = mp.get_context("spawn") + cmd_q = ctx.Queue() + ack_q = ctx.Queue() + result_q = ctx.Queue() + done_event = ctx.Event() + proc = ctx.Process( + target=_ipc_consumer_worker, + args=(cmd_q, ack_q, result_q, done_event, device_index), + ) + proc.start() + + num_chunks = 0 + try: + for chunk in chunk_iter: + cmd_q.put( + { + "ipc_handle": chunk.ipc_handle, + "names": chunk.names, + "shapes": chunk.shapes, + "dtype_names": chunk.dtype_names, + "tensor_sizes": chunk.tensor_sizes, + } + ) + if ack_q.get(timeout=timeout) != "ack": + raise RuntimeError("Consumer did not ack chunk") + num_chunks += 1 + cmd_q.put(None) + status, payload = result_q.get(timeout=timeout) + finally: + done_event.set() + proc.join(timeout=10) + if proc.is_alive(): + proc.kill() + + if status == "error": + raise RuntimeError(f"Consumer process failed: {payload}") + # Reclaim IPC-shared memory now that the child has released it + torch.cuda.ipc_collect() + return num_chunks, payload + + def test_roundtrip_basic(self): + """Test basic IPC producer -> consumer roundtrip.""" + params = [ + ("w1", torch.randn(10, 20, dtype=torch.float32).cuda()), + ("w2", torch.randn(5, dtype=torch.float16).cuda()), + ] + gpu_uuid = self._get_gpu_uuid() + device_index = torch.cuda.current_device() + + num_chunks, result = self._run_roundtrip( + packed_ipc_producer( + iterator=iter(params), + gpu_uuid=gpu_uuid, + post_iter_func=lambda x: x[1], + buffer_size_bytes=10_000_000, + ), + device_index, + ) + + assert num_chunks == 1 + assert len(result) == 2 + for (orig_name, orig_tensor), (res_name, res_tensor) in zip(params, result): + assert orig_name == res_name + assert res_tensor.shape == orig_tensor.shape + assert res_tensor.dtype == orig_tensor.dtype + assert torch.allclose(res_tensor, orig_tensor.cpu(), rtol=1e-4, atol=1e-6) + + @pytest.mark.parametrize("dtype", [torch.float32, torch.float16, torch.bfloat16]) + def test_roundtrip_dtypes(self, dtype): + """Test IPC roundtrip with different dtypes.""" + params = create_mock_model_params(num_layers=2, dtype=dtype) + params_cuda = [(n, t.cuda()) for n, t in params] + gpu_uuid = self._get_gpu_uuid() + device_index = torch.cuda.current_device() + + _, result = self._run_roundtrip( + packed_ipc_producer( + iterator=iter(params_cuda), + gpu_uuid=gpu_uuid, + post_iter_func=lambda x: x[1], + buffer_size_bytes=10_000_000, + ), + device_index, + ) + + assert len(result) == len(params_cuda) + for (orig_name, orig_tensor), (res_name, res_tensor) in zip( + params_cuda, result + ): + assert orig_name == res_name + assert res_tensor.dtype == dtype + assert torch.allclose(res_tensor, orig_tensor.cpu(), rtol=1e-4, atol=1e-6) + + def test_roundtrip_multiple_chunks(self): + """Test IPC roundtrip across multiple chunks.""" + params = [ + (f"layer{i}.weight", torch.randn(100, 100, dtype=torch.float32).cuda()) + for i in range(8) + ] + gpu_uuid = self._get_gpu_uuid() + device_index = torch.cuda.current_device() + + num_chunks, result = self._run_roundtrip( + packed_ipc_producer( + iterator=iter(params), + gpu_uuid=gpu_uuid, + post_iter_func=lambda x: x[1], + buffer_size_bytes=50_000, + ), + device_index, + ) + + assert num_chunks > 1 + assert len(result) == len(params) + for (orig_name, orig_tensor), (res_name, res_tensor) in zip(params, result): + assert orig_name == res_name + assert torch.allclose(res_tensor, orig_tensor.cpu(), rtol=1e-5, atol=1e-7) + + def test_roundtrip_non_contiguous(self): + """Test IPC roundtrip with non-contiguous tensors.""" + params = [ + ("transposed", torch.randn(20, 10, dtype=torch.float32).cuda().T), + ("sliced", torch.randn(40, 30, dtype=torch.float16).cuda()[::2, ::2]), + ] + gpu_uuid = self._get_gpu_uuid() + device_index = torch.cuda.current_device() + + for _, t in params: + assert not t.is_contiguous() + + _, result = self._run_roundtrip( + packed_ipc_producer( + iterator=iter(params), + gpu_uuid=gpu_uuid, + post_iter_func=lambda x: x[1], + buffer_size_bytes=10_000_000, + ), + device_index, + ) + + for (orig_name, orig_tensor), (res_name, res_tensor) in zip(params, result): + assert orig_name == res_name + assert res_tensor.shape == orig_tensor.shape + assert res_tensor.dtype == orig_tensor.dtype + assert torch.allclose(res_tensor, orig_tensor.cpu(), rtol=1e-4, atol=1e-6) + + def test_consumer_wrong_uuid_raises(self): + """Test that consumer raises ValueError for unknown GPU UUID.""" + params = [("w", torch.randn(10, dtype=torch.float32).cuda())] + gpu_uuid = self._get_gpu_uuid() + + chunks = list( + packed_ipc_producer( + iterator=iter(params), + gpu_uuid=gpu_uuid, + post_iter_func=lambda x: x[1], + buffer_size_bytes=10_000_000, + ) + ) + + c = chunks[0] + fake_handle = {"fake-uuid-12345": c.ipc_handle[gpu_uuid]} + + with pytest.raises(ValueError, match="IPC handle not found"): + packed_ipc_consumer( + ipc_handle=fake_handle, + names=c.names, + shapes=c.shapes, + dtype_names=c.dtype_names, + tensor_sizes=c.tensor_sizes, + device_index=torch.cuda.current_device(), + ) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_pipeline_parallel.py b/asian_ai/performance/vLLM/tests/distributed/test_pipeline_parallel.py new file mode 100644 index 0000000000000000000000000000000000000000..8a431ef8f0c3cb9bdca7324e75577b2cdc191a61 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_pipeline_parallel.py @@ -0,0 +1,436 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +WARNING: This test runs in both single-node (4 GPUs) and multi-node + (2 node with 2 GPUs each) modes. If the test only uses 2 GPUs, it is + important to set the distributed backend to "mp" to avoid Ray scheduling + all workers in a node other than the head node, which can cause the test + to fail. +""" + +import json +import os +from dataclasses import dataclass +from typing import Literal, NamedTuple + +import pytest + +from vllm.config.model import _FLOAT16_NOT_SUPPORTED_MODELS, RunnerOption +from vllm.logger import init_logger +from vllm.transformers_utils.config import get_config + +from ..models.registry import HF_EXAMPLE_MODELS +from ..utils import compare_two_settings, create_new_process_for_each_test + +logger = init_logger("test_pipeline_parallel") + +VLLM_MULTI_NODE = os.getenv("VLLM_MULTI_NODE", "0") == "1" + + +class ParallelSetup(NamedTuple): + tp_size: int + pp_size: int + eager_mode: bool + + +class PPTestOptions(NamedTuple): + multi_node_only: bool + load_format: str | None = None + + +@dataclass +class PPTestSettings: + parallel_setups: list[ParallelSetup] + distributed_backends: list[str] + runner: RunnerOption + test_options: PPTestOptions + + @staticmethod + def detailed( + *, + tp_base: int = 1, + pp_base: int = 2, + multi_node_only: bool = False, + runner: RunnerOption = "auto", + load_format: str | None = None, + ): + return PPTestSettings( + parallel_setups=[ + ParallelSetup(tp_size=tp_base, pp_size=pp_base, eager_mode=False), + ParallelSetup(tp_size=tp_base, pp_size=2 * pp_base, eager_mode=False), + ParallelSetup(tp_size=tp_base, pp_size=2 * pp_base, eager_mode=True), + ParallelSetup(tp_size=2 * tp_base, pp_size=pp_base, eager_mode=False), + ParallelSetup(tp_size=2 * tp_base, pp_size=pp_base, eager_mode=True), + ], + distributed_backends=["mp", "ray"], + runner=runner, + test_options=PPTestOptions( + multi_node_only=multi_node_only, load_format=load_format + ), + ) + + @staticmethod + def fast( + *, + tp_base: int = 1, + pp_base: int = 2, + runner: RunnerOption = "auto", + multi_node_only: bool = False, + load_format: str | None = None, + ): + return PPTestSettings( + parallel_setups=[ + ParallelSetup(tp_size=tp_base, pp_size=pp_base, eager_mode=True), + ], + distributed_backends=["mp"], + runner=runner, + test_options=PPTestOptions( + multi_node_only=multi_node_only, load_format=load_format + ), + ) + + def iter_params(self, model_id: str): + opts = self.test_options + + for parallel_setup in self.parallel_setups: + for backend in self.distributed_backends: + yield (model_id, parallel_setup, backend, self.runner, opts) + + +# NOTE: You can adjust tp_base and/or pp_base locally to fit the model in GPU +# The values displayed here are only a rough indicator of the size of the model + +TEXT_GENERATION_MODELS = { + # [Decoder-only] + "Snowflake/snowflake-arctic-instruct": PPTestSettings.fast(load_format="dummy"), + "bigscience/bloomz-1b1": PPTestSettings.fast(), + "zai-org/chatglm3-6b": PPTestSettings.fast(), + "CohereLabs/c4ai-command-r-v01": PPTestSettings.fast(load_format="dummy"), + "databricks/dbrx-instruct": PPTestSettings.fast(load_format="dummy"), + "Deci/DeciLM-7B-instruct": PPTestSettings.fast(), + "deepseek-ai/deepseek-llm-7b-chat": PPTestSettings.fast(), + "deepseek-ai/DeepSeek-V2-Lite-Chat": PPTestSettings.fast(tp_base=2), + "LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct": PPTestSettings.fast(), + "tiiuae/falcon-7b": PPTestSettings.fast(), + "google/gemma-1.1-2b-it": PPTestSettings.fast(), + "google/gemma-2-9b": PPTestSettings.fast(), + "gpt2": PPTestSettings.fast(), + "EleutherAI/gpt-j-6b": PPTestSettings.fast(), + "EleutherAI/pythia-1.4b": PPTestSettings.fast(), + "ibm/PowerLM-3b": PPTestSettings.fast(), + "ibm/PowerMoE-3b": PPTestSettings.fast(), + "internlm/internlm2-chat-7b": PPTestSettings.fast(), + "ai21labs/Jamba-tiny-dev": PPTestSettings.fast(), + "pfnet/plamo-2-1b": PPTestSettings.fast(), + "pfnet/plamo-3-nict-2b-base": PPTestSettings.fast(), + "meta-llama/Llama-3.2-1B-Instruct": PPTestSettings.detailed(), + # Tests TransformersForCausalLM + "hmellor/Ilama-3.2-1B": PPTestSettings.fast(), + "openbmb/MiniCPM-2B-sft-bf16": PPTestSettings.fast(), + "openbmb/MiniCPM3-4B": PPTestSettings.fast(), + # Uses Llama + # "mistralai/Mistral-7B-Instruct-v0.1": PPTestSettings.fast(), + "state-spaces/mamba-130m-hf": PPTestSettings.fast(), + "mistralai/Mixtral-8x7B-Instruct-v0.1": PPTestSettings.fast(load_format="dummy"), + "mosaicml/mpt-7b": PPTestSettings.fast(), + "nvidia/Minitron-8B-Base": PPTestSettings.fast(), + "allenai/OLMo-1B-hf": PPTestSettings.fast(), + "allenai/OLMo-2-0425-1B": PPTestSettings.fast(), + "allenai/OLMoE-1B-7B-0924-Instruct": PPTestSettings.fast(), + "facebook/opt-iml-max-1.3b": PPTestSettings.fast(), + "OrionStarAI/Orion-14B-Chat": PPTestSettings.fast(), + "adept/persimmon-8b-chat": PPTestSettings.fast(), + "microsoft/phi-2": PPTestSettings.fast(), + "microsoft/Phi-3-small-8k-instruct": PPTestSettings.fast(), + "microsoft/Phi-3.5-MoE-instruct": PPTestSettings.detailed( + multi_node_only=True, load_format="dummy" + ), + "Qwen/Qwen2.5-0.5B-Instruct": PPTestSettings.fast(), + "Qwen/Qwen1.5-MoE-A2.7B-Chat": PPTestSettings.fast(), + "stabilityai/stablelm-3b-4e1t": PPTestSettings.fast(), + "bigcode/starcoder2-3b": PPTestSettings.fast(), + "upstage/solar-pro-preview-instruct": PPTestSettings.fast(load_format="dummy"), + # [Encoder-only] + # TODO: Implement PP + # "facebook/bart-base": PPTestSettings.fast(), +} + +EMBEDDING_MODELS = { # type: ignore[var-annotated] + # [Text-only] + "intfloat/e5-mistral-7b-instruct": PPTestSettings.fast(runner="pooling"), + "BAAI/bge-multilingual-gemma2": PPTestSettings.fast(runner="pooling"), + "Qwen/Qwen2.5-Math-RM-72B": PPTestSettings.fast( + load_format="dummy", runner="pooling" + ), +} + +MULTIMODAL_MODELS = { + # [Decoder-only] + "Salesforce/blip2-opt-6.7b": PPTestSettings.fast(), + "facebook/chameleon-7b": PPTestSettings.fast(), + "adept/fuyu-8b": PPTestSettings.fast(), + "zai-org/glm-4v-9b": PPTestSettings.fast(), + "OpenGVLab/InternVL3-1B": PPTestSettings.fast(), + "llava-hf/llava-1.5-7b-hf": PPTestSettings.fast(), + "llava-hf/llava-v1.6-mistral-7b-hf": PPTestSettings.fast(), + "llava-hf/LLaVA-NeXT-Video-7B-hf": PPTestSettings.fast(), + "llava-hf/llava-onevision-qwen2-0.5b-ov-hf": PPTestSettings.fast(), + "openbmb/MiniCPM-Llama3-V-2_5": PPTestSettings.fast(), + "allenai/Molmo-7B-D-0924": PPTestSettings.fast(), + "AIDC-AI/Ovis2-1B": PPTestSettings.fast(), + "AIDC-AI/Ovis2.5-2B": PPTestSettings.fast(), + "microsoft/Phi-3.5-vision-instruct": PPTestSettings.fast(), + "mistralai/Pixtral-12B-2409": PPTestSettings.fast(load_format="dummy"), + "Qwen/Qwen2-Audio-7B-Instruct": PPTestSettings.fast(), + "Qwen/Qwen2-VL-2B-Instruct": PPTestSettings.fast(), + "fixie-ai/ultravox-v0_5-llama-3_2-1b": PPTestSettings.fast(), +} + +# NOTE: You can update this on your local machine to run specific tests +TEST_MODELS = [ + # [LANGUAGE GENERATION] + "microsoft/Phi-3.5-MoE-instruct", + "meta-llama/Llama-3.2-1B-Instruct", + "hmellor/Ilama-3.2-1B", + "ibm/PowerLM-3b", + "deepseek-ai/DeepSeek-V2-Lite-Chat", + # [LANGUAGE EMBEDDING] + "intfloat/e5-mistral-7b-instruct", + "BAAI/bge-multilingual-gemma2", + # [MULTIMODAL GENERATION] + "OpenGVLab/InternVL3-1B", + "microsoft/Phi-3.5-vision-instruct", + "fixie-ai/ultravox-v0_5-llama-3_2-1b", + # [LANGUAGE GENERATION - HYBRID ARCH] + "ai21labs/Jamba-tiny-dev", +] + + +def _compare_tp( + model_id: str, + parallel_setup: ParallelSetup, + distributed_backend: str, + runner: RunnerOption, + test_options: PPTestOptions, + num_gpus_available: int, + *, + method: Literal["generate", "encode"], + is_multimodal: bool, +): + ( + tp_size, + pp_size, + eager_mode, + ) = parallel_setup + + multi_node_only, load_format = test_options + + model_info = HF_EXAMPLE_MODELS.find_hf_info(model_id) + model_info.check_transformers_version(on_fail="skip") + + trust_remote_code = model_info.trust_remote_code + tokenizer_mode = model_info.tokenizer_mode + hf_overrides = model_info.hf_overrides + hf_config = get_config(model_id, trust_remote_code) + require_embed_inputs = model_info.require_embed_inputs + max_num_seqs = model_info.max_num_seqs + enable_prefix_caching = model_info.enable_prefix_caching + + dtype = "float16" + if hf_config.model_type in _FLOAT16_NOT_SUPPORTED_MODELS: + dtype = "bfloat16" + + if load_format == "dummy": + # Avoid OOM + text_overrides = { + "num_hidden_layers": 4, + "hidden_size": 512, + "intermediate_size": 800, + "num_attention_heads": 4, + "num_key_value_heads": 1, + } + + if is_multimodal: + hf_overrides.update({"text_config": text_overrides}) + else: + hf_overrides.update(text_overrides) + else: + model_info.check_available_online(on_fail="skip") + + if num_gpus_available < tp_size * pp_size: + pytest.skip(f"Need at least {tp_size} x {pp_size} GPUs") + if VLLM_MULTI_NODE and distributed_backend == "mp": + pytest.skip( + "Skipping multi-node pipeline parallel test for " + "multiprocessing distributed backend" + ) + if multi_node_only and not VLLM_MULTI_NODE: + pytest.skip("Not in multi-node setting") + + common_args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + dtype, + "--max-model-len", + "2048", + "--max-num-seqs", + "8", + ] + if eager_mode: + common_args.append("--enforce-eager") + if runner != "auto": + common_args.extend(["--runner", runner]) + if trust_remote_code: + common_args.append("--trust-remote-code") + if tokenizer_mode: + common_args.extend(["--tokenizer-mode", tokenizer_mode]) + if load_format: + common_args.extend(["--load-format", load_format]) + if hf_overrides: + common_args.extend(["--hf-overrides", json.dumps(hf_overrides)]) + if not enable_prefix_caching: + common_args.append("--no-enable-prefix-caching") + if require_embed_inputs: + common_args.extend( + [ + "--skip-tokenizer-init", + "--enable-prompt-embeds", + "--enable-mm-embeds", + ] + ) + if max_num_seqs: + common_args.extend(["--max-num-seqs", f"{max_num_seqs}"]) + + if distributed_backend == "ray": + # Test Ray Compiled Graph for all the tests + pp_env = { + "VLLM_USE_RAY_COMPILED_DAG_NCCL_CHANNEL": "1", + } + elif distributed_backend == "mp": + pp_env = None + else: + pp_env = None + + tp_env = None + + pp_args = [ + *common_args, + "--pipeline-parallel-size", + str(pp_size), + "--tensor-parallel-size", + str(tp_size), + "--distributed-executor-backend", + distributed_backend, + ] + + # compare without pipeline parallelism + # NOTE: use mp backend for TP + # PP tests might involve multiple nodes, and ray might + # schedule all workers in a node other than the head node, + # which can cause the test to fail. + tp_args = [ + *common_args, + "--tensor-parallel-size", + str(tp_size), + "--distributed-executor-backend", + "mp", + ] + + compare_two_settings( + model_id, + pp_args, + tp_args, + pp_env, + tp_env, + method=method, + ) + + +@pytest.mark.parametrize( + ("model_id", "parallel_setup", "distributed_backend", "runner", "test_options"), + [ + params + for model_id, settings in TEXT_GENERATION_MODELS.items() + for params in settings.iter_params(model_id) + if model_id in TEST_MODELS + ], +) +@create_new_process_for_each_test() +def test_tp_language_generation( + model_id: str, + parallel_setup: ParallelSetup, + distributed_backend: str, + runner: RunnerOption, + test_options: PPTestOptions, + num_gpus_available, +): + _compare_tp( + model_id, + parallel_setup, + distributed_backend, + runner, + test_options, + num_gpus_available, + method="generate", + is_multimodal=False, + ) + + +@pytest.mark.parametrize( + ("model_id", "parallel_setup", "distributed_backend", "runner", "test_options"), + [ + params + for model_id, settings in EMBEDDING_MODELS.items() + for params in settings.iter_params(model_id) + if model_id in TEST_MODELS + ], +) +@create_new_process_for_each_test() +def test_tp_language_embedding( + model_id: str, + parallel_setup: ParallelSetup, + distributed_backend: str, + runner: RunnerOption, + test_options: PPTestOptions, + num_gpus_available, +): + _compare_tp( + model_id, + parallel_setup, + distributed_backend, + runner, + test_options, + num_gpus_available, + method="encode", + is_multimodal=False, + ) + + +@pytest.mark.parametrize( + ("model_id", "parallel_setup", "distributed_backend", "runner", "test_options"), + [ + params + for model_id, settings in MULTIMODAL_MODELS.items() + for params in settings.iter_params(model_id) + if model_id in TEST_MODELS + ], +) +@create_new_process_for_each_test() +def test_tp_multimodal_generation( + model_id: str, + parallel_setup: ParallelSetup, + distributed_backend: str, + runner: RunnerOption, + test_options: PPTestOptions, + num_gpus_available, +): + _compare_tp( + model_id, + parallel_setup, + distributed_backend, + runner, + test_options, + num_gpus_available, + method="generate", + is_multimodal=True, + ) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_pipeline_partition.py b/asian_ai/performance/vLLM/tests/distributed/test_pipeline_partition.py new file mode 100644 index 0000000000000000000000000000000000000000..1d3a56ebc104b761f1851c91437fccf4905a2959 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_pipeline_partition.py @@ -0,0 +1,67 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import os + +import pytest + +from vllm.distributed.utils import get_pp_indices + + +def test_custom_layer_partition(monkeypatch: pytest.MonkeyPatch): + with monkeypatch.context() as m: + + def _verify(partition_str, num_layers, pp_size, goldens): + bak = os.environ.get("VLLM_PP_LAYER_PARTITION", None) + m.setenv("VLLM_PP_LAYER_PARTITION", partition_str) + for pp_rank, golden in enumerate(goldens): + assert get_pp_indices(num_layers, pp_rank, pp_size) == golden + if bak is not None: + m.setenv("VLLM_PP_LAYER_PARTITION", bak) + + # Even partition + _verify("5,5,5,5", 20, 4, [(0, 5), (5, 10), (10, 15), (15, 20)]) + # Balanced partition + _verify("4,6,6,4", 20, 4, [(0, 4), (4, 10), (10, 16), (16, 20)]) + # Put reminder somewhere + _verify("5,6,5,6", 22, 4, [(0, 5), (5, 11), (11, 16), (16, 22)]) + # Invalid partition strings + with pytest.raises(ValueError): + _verify("5,5,5,5,", 20, 4, [(0, 5), (5, 10), (10, 15), (15, 20)]) + with pytest.raises(ValueError): + _verify("5,5,5,a", 20, 4, [(0, 5), (5, 10), (10, 15), (15, 20)]) + # Wrong number of partitions + with pytest.raises(ValueError): + _verify("5,5,5", 20, 4, [(0, 5), (5, 10), (10, 15), (15, 20)]) + # Wrong number of layers + with pytest.raises(ValueError): + _verify("5,5,5,5", 21, 4, [(0, 5), (5, 10), (10, 15), (15, 20)]) + + +@pytest.mark.parametrize( + "num_hidden_layers,pp_size,pp_rank,indices", + [ + # pp_size 2 + (2, 2, 0, (0, 1)), + (2, 2, 1, (1, 2)), + (3, 2, 0, (0, 2)), + (3, 2, 1, (2, 3)), + # pp_size 3 + (3, 3, 0, (0, 1)), + (3, 3, 1, (1, 2)), + (3, 3, 2, (2, 3)), + (4, 3, 0, (0, 1)), + (4, 3, 1, (1, 3)), + (4, 3, 2, (3, 4)), + (5, 3, 0, (0, 2)), + (5, 3, 1, (2, 4)), + (5, 3, 2, (4, 5)), + ], +) +def test_uneven_auto_partition( + num_hidden_layers: int, + pp_size: int, + pp_rank: int, + indices: tuple[int, int], +): + assert indices == get_pp_indices(num_hidden_layers, pp_rank, pp_size) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_pp_cudagraph.py b/asian_ai/performance/vLLM/tests/distributed/test_pp_cudagraph.py new file mode 100644 index 0000000000000000000000000000000000000000..d8f0d0c6baef2f01f341b00a07f1f6f799671305 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_pp_cudagraph.py @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest + +from vllm.platforms import current_platform + +from ..utils import compare_two_settings, create_new_process_for_each_test + + +@pytest.mark.parametrize( + "PP_SIZE, MODEL_NAME", + [ + (2, "JackFram/llama-160m"), + ], +) +@pytest.mark.parametrize( + "ATTN_BACKEND", + [None] if current_platform.is_rocm() else ["FLASH_ATTN"], +) +@create_new_process_for_each_test() +def test_pp_cudagraph( + PP_SIZE: int, + MODEL_NAME: str, + ATTN_BACKEND: str | None, +): + cudagraph_args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "float16", + "--pipeline-parallel-size", + str(PP_SIZE), + "--distributed-executor-backend", + "mp", + ] + # On ROCm, defer to the platform attention selector instead of forcing a backend. + if ATTN_BACKEND is not None: + cudagraph_args.append(f"--attention-backend={ATTN_BACKEND}") + + eager_args = cudagraph_args + ["--enforce-eager"] + + compare_two_settings(MODEL_NAME, eager_args, cudagraph_args) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_pynccl.py b/asian_ai/performance/vLLM/tests/distributed/test_pynccl.py new file mode 100644 index 0000000000000000000000000000000000000000..0df710e10934428fb7f0f81fffdbc8e40bbb2c1a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_pynccl.py @@ -0,0 +1,427 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import os + +import multiprocess as mp +import numpy as np +import pytest +import torch +import torch.distributed + +import vllm.envs as envs +from tests.utils import ensure_current_vllm_config +from vllm.distributed.communication_op import tensor_model_parallel_all_reduce # noqa +from vllm.distributed.device_communicators.pynccl import PyNcclCommunicator +from vllm.distributed.device_communicators.pynccl_wrapper import NCCLLibrary +from vllm.distributed.parallel_state import ( + ensure_model_parallel_initialized, + get_world_group, + graph_capture, + init_distributed_environment, +) +from vllm.utils.system_utils import update_environment_variables + +mp.set_start_method("spawn", force=True) + + +def distributed_run(fn, world_size): + number_of_processes = world_size + processes: list[mp.Process] = [] + for i in range(number_of_processes): + env: dict[str, str] = {} + env["RANK"] = str(i) + env["LOCAL_RANK"] = str(i) + env["WORLD_SIZE"] = str(number_of_processes) + env["LOCAL_WORLD_SIZE"] = str(number_of_processes) + env["MASTER_ADDR"] = "localhost" + env["MASTER_PORT"] = "12345" + p = mp.Process(target=fn, args=(env,)) + processes.append(p) + p.start() + + for p in processes: + p.join() + + for p in processes: + assert p.exitcode == 0 + + +def worker_fn_wrapper(fn): + # `multiprocessing.Process` cannot accept environment variables directly + # so we need to pass the environment variables as arguments + # and update the environment variables in the function + def wrapped_fn(env): + update_environment_variables(env) + local_rank = os.environ["LOCAL_RANK"] + device = torch.device(f"cuda:{local_rank}") + torch.accelerator.set_device_index(device) + init_distributed_environment() + fn() + + return wrapped_fn + + +@worker_fn_wrapper +def worker_fn(): + pynccl_comm = PyNcclCommunicator( + get_world_group().cpu_group, device=get_world_group().device + ) + tensor = torch.ones(16, 1024, 1024, dtype=torch.float32).cuda(pynccl_comm.rank) + tensor = pynccl_comm.all_reduce(tensor) + torch.accelerator.synchronize() + assert torch.all(tensor == pynccl_comm.world_size).cpu().item() + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 2, reason="Need at least 2 GPUs to run the test." +) +def test_pynccl(): + distributed_run(worker_fn, 2) + + +@worker_fn_wrapper +def multiple_allreduce_worker_fn(): + device = torch.device(f"cuda:{torch.distributed.get_rank()}") + if envs.VLLM_DISTRIBUTED_USE_SPLIT_GROUP: + # Eager-init path: parent PG has bound_device_id + a CPU backend, + # so split_group is supported. + group = torch.distributed.split_group( + split_ranks=[[0, 1], [2, 3]], backend="cpu:gloo,cuda:nccl" + ) + else: + groups = [ + torch.distributed.new_group(ranks=[0, 1], backend="gloo"), + torch.distributed.new_group(ranks=[2, 3], backend="gloo"), + ] + group = groups[0] if torch.distributed.get_rank() in [0, 1] else groups[1] + pynccl_comm = PyNcclCommunicator(group=group, device=device) + tensor = torch.ones(16, 1024, 1024, dtype=torch.float32, device=device) + # two groups can communicate independently + if torch.distributed.get_rank() in [0, 1]: + tensor = pynccl_comm.all_reduce(tensor) + tensor = pynccl_comm.all_reduce(tensor) + torch.accelerator.synchronize() + assert torch.all(tensor == 4).cpu().item() + else: + tensor = pynccl_comm.all_reduce(tensor) + torch.accelerator.synchronize() + assert torch.all(tensor == 2).cpu().item() + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 4, reason="Need at least 4 GPUs to run the test." +) +def test_pynccl_multiple_allreduce(): + # this tests pynccl for multiple tp groups, in a standalone way + # i.e. call `pynccl_comm.all_reduce` directly + distributed_run(multiple_allreduce_worker_fn, 4) + + +@worker_fn_wrapper +def multiple_allreduce_with_vllm_worker_fn(): + device = torch.device(f"cuda:{torch.distributed.get_rank()}") + with ensure_current_vllm_config(): + ensure_model_parallel_initialized(2, 2) + tensor = torch.ones(16, 1024, 1024, dtype=torch.float32, device=device) + with graph_capture(device=device): + # two tp groups can communicate independently + if torch.distributed.get_rank() in [0, 1]: + tensor = tensor_model_parallel_all_reduce(tensor) + tensor = tensor_model_parallel_all_reduce(tensor) + torch.accelerator.synchronize() + assert torch.all(tensor == 4).cpu().item() + else: + tensor = tensor_model_parallel_all_reduce(tensor) + torch.accelerator.synchronize() + assert torch.all(tensor == 2).cpu().item() + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 4, reason="Need at least 4 GPUs to run the test." +) +def test_pynccl_multiple_allreduce_with_vllm(): + # this tests pynccl for multiple tp groups, together with vllm + # i.e. call `tensor_model_parallel_all_reduce` + distributed_run(multiple_allreduce_with_vllm_worker_fn, 4) + + +@worker_fn_wrapper +def worker_fn_with_cudagraph(): + with torch.no_grad(): + graph = torch.cuda.CUDAGraph() + pynccl_comm = PyNcclCommunicator( + get_world_group().cpu_group, device=get_world_group().device + ) + # run something in the default stream to initialize torch engine + a = torch.ones((4, 4), device=f"cuda:{pynccl_comm.rank}") + torch.accelerator.synchronize() + with torch.cuda.graph(graph): + a_out = pynccl_comm.all_reduce(a) + torch.accelerator.synchronize() + graph.replay() + torch.accelerator.synchronize() + assert torch.all(a_out == pynccl_comm.world_size).cpu().item() + + +@worker_fn_wrapper +def all_gather_worker_fn(): + pynccl_comm = PyNcclCommunicator( + get_world_group().cpu_group, device=get_world_group().device + ) + + rank = pynccl_comm.rank + world_size = pynccl_comm.world_size + device = f"cuda:{pynccl_comm.rank}" + + num_elems = 1000 + tensor = ( + torch.arange(num_elems, dtype=torch.float32, device=device) + rank * num_elems + ) + result = torch.zeros(num_elems * world_size, dtype=torch.float32, device=device) + + expected = torch.cat( + [ + torch.arange(num_elems, dtype=torch.float32) + r * num_elems + for r in range(world_size) + ] + ).to(device) + + pynccl_comm.all_gather(result, tensor) + torch.accelerator.synchronize() + torch.testing.assert_close(result, expected, rtol=1e-5, atol=1e-8) + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 2, reason="Need at least 2 GPUs to run the test." +) +def test_pynccl_all_gather(): + distributed_run(all_gather_worker_fn, 2) + + +@worker_fn_wrapper +def all_gatherv_worker_fn(): + pynccl_comm = PyNcclCommunicator( + get_world_group().cpu_group, device=get_world_group().device + ) + + rank = pynccl_comm.rank + world_size = pynccl_comm.world_size + device = f"cuda:{pynccl_comm.rank}" + + assert world_size <= 8 + sizes = [81, 20, 57, 52, 81, 5, 49, 49][:world_size] + num_elems = sizes[rank] + tensor = torch.arange(num_elems, dtype=torch.float32, device=device) + rank * 100 + result = torch.zeros(sum(sizes), dtype=torch.float32, device=device) + + expected = torch.cat( + [ + torch.arange(sizes[r], dtype=torch.float32) + r * 100 + for r in range(world_size) + ] + ).to(device) + + pynccl_comm.all_gatherv(result, tensor, sizes=sizes) + torch.accelerator.synchronize() + torch.testing.assert_close(result, expected, rtol=1e-5, atol=1e-8) + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 2, reason="Need at least 2 GPUs to run the test." +) +def test_pynccl_all_gatherv(): + distributed_run(all_gatherv_worker_fn, 2) + + +@worker_fn_wrapper +def reduce_scatter_worker_fn(): + pynccl_comm = PyNcclCommunicator( + get_world_group().cpu_group, device=get_world_group().device + ) + + rank = pynccl_comm.rank + world_size = pynccl_comm.world_size + device = f"cuda:{pynccl_comm.rank}" + + num_elems = 1000 + tensor = ( + torch.arange(num_elems, dtype=torch.float32, device=device) + rank * num_elems + ) + assert num_elems % world_size == 0 + result = torch.zeros(num_elems // world_size, dtype=torch.float32, device=device) + + # Calculate expected result for this rank's chunk + scattered_size = num_elems // world_size + all_tensors = [ + torch.arange(num_elems, dtype=torch.float32) + r * num_elems + for r in range(world_size) + ] + expected = sum( + tensor[rank * scattered_size : (rank + 1) * scattered_size] + for tensor in all_tensors + ).to(device) + + pynccl_comm.reduce_scatter(result, tensor) + torch.accelerator.synchronize() + torch.testing.assert_close(result, expected, rtol=1e-5, atol=1e-8) + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 2, reason="Need at least 2 GPUs to run the test." +) +def test_pynccl_reduce_scatter(): + distributed_run(reduce_scatter_worker_fn, 2) + + +@worker_fn_wrapper +def reduce_scatterv_worker_fn(): + pynccl_comm = PyNcclCommunicator( + get_world_group().cpu_group, device=get_world_group().device + ) + + rank = pynccl_comm.rank + world_size = pynccl_comm.world_size + device = f"cuda:{pynccl_comm.rank}" + + assert world_size <= 8 + sizes = [81, 20, 57, 52, 81, 5, 49, 49][:world_size] + num_elems = sum(sizes) + tensor = torch.arange(num_elems, dtype=torch.float32, device=device) + rank * 100 + result = torch.zeros(sizes[rank], dtype=torch.float32, device=device) + + # Calculate expected result for this rank's chunk + all_tensors = [ + torch.arange(num_elems, dtype=torch.float32) + r * 100 + for r in range(world_size) + ] + sizes_cumsum = np.cumsum(sizes) + start = 0 if rank == 0 else sizes_cumsum[rank - 1] + end = sizes_cumsum[rank] + expected = sum(tensor[start:end] for tensor in all_tensors).to(device) + + pynccl_comm.reduce_scatterv(result, tensor, sizes=sizes) + torch.accelerator.synchronize() + torch.testing.assert_close(result, expected, rtol=1e-5, atol=1e-8) + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 2, reason="Need at least 2 GPUs to run the test." +) +def test_pynccl_reduce_scatterv(): + distributed_run(reduce_scatterv_worker_fn, 2) + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 2, reason="Need at least 2 GPUs to run the test." +) +def test_pynccl_with_cudagraph(): + distributed_run(worker_fn_with_cudagraph, 2) + + +@worker_fn_wrapper +def send_recv_worker_fn(): + pynccl_comm = PyNcclCommunicator( + get_world_group().cpu_group, device=get_world_group().device + ) + if pynccl_comm.rank == 0: + tensor = torch.ones(16, 1024, 1024, dtype=torch.float32).cuda(pynccl_comm.rank) + else: + tensor = torch.empty(16, 1024, 1024, dtype=torch.float32).cuda(pynccl_comm.rank) + + if pynccl_comm.rank == 0: + pynccl_comm.send(tensor, dst=(pynccl_comm.rank + 1) % pynccl_comm.world_size) + else: + pynccl_comm.recv(tensor, src=(pynccl_comm.rank - 1) % pynccl_comm.world_size) + torch.accelerator.synchronize() + assert torch.all(tensor == 1).cpu().item() + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 2, reason="Need at least 2 GPUs to run the test." +) +def test_pynccl_send_recv(): + distributed_run(send_recv_worker_fn, 2) + + +@worker_fn_wrapper +def multiple_send_recv_worker_fn(): + device = torch.device(f"cuda:{torch.distributed.get_rank()}") + if envs.VLLM_DISTRIBUTED_USE_SPLIT_GROUP: + group = torch.distributed.split_group( + split_ranks=[[0, 2], [1, 3]], backend="cpu:gloo,cuda:nccl" + ) + else: + groups = [ + torch.distributed.new_group(ranks=[0, 2], backend="gloo"), + torch.distributed.new_group(ranks=[1, 3], backend="gloo"), + ] + group = groups[0] if torch.distributed.get_rank() in [0, 2] else groups[1] + pynccl_comm = PyNcclCommunicator(group=group, device=device) + if torch.distributed.get_rank() == 0: + tensor = torch.ones(16, 1024, 1024, dtype=torch.float32, device=device) + elif torch.distributed.get_rank() == 1: + tensor = 2 * torch.ones(16, 1024, 1024, dtype=torch.float32, device=device) + else: + tensor = torch.empty(16, 1024, 1024, dtype=torch.float32, device=device) + if torch.distributed.get_rank() in [0, 1]: + pynccl_comm.send(tensor, dst=(pynccl_comm.rank + 1) % pynccl_comm.world_size) + else: + pynccl_comm.recv(tensor, src=(pynccl_comm.rank - 1) % pynccl_comm.world_size) + torch.accelerator.synchronize() + if torch.distributed.get_rank() in [0, 2]: + assert torch.all(tensor == 1).cpu().item() + else: + assert torch.all(tensor == 2).cpu().item() + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 4, reason="Need at least 4 GPUs to run the test." +) +def test_pynccl_multiple_send_recv(): + distributed_run(multiple_send_recv_worker_fn, 4) + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 4, reason="Need at least 4 GPUs to run the test." +) +def test_pynccl_broadcast(): + distributed_run(broadcast_worker_fn, 4) + + +@worker_fn_wrapper +def broadcast_worker_fn(): + # Test broadcast for every root rank. + # Essentially this is an all-gather operation. + pynccl_comm = PyNcclCommunicator( + get_world_group().cpu_group, device=get_world_group().device + ) + recv_tensors = [ + torch.empty(16, 1024, 1024, dtype=torch.float32, device=pynccl_comm.device) + for i in range(pynccl_comm.world_size) + ] + recv_tensors[pynccl_comm.rank] = ( + torch.ones(16, 1024, 1024, dtype=torch.float32, device=pynccl_comm.device) + * pynccl_comm.rank + ) + + for i in range(pynccl_comm.world_size): + pynccl_comm.broadcast(recv_tensors[i], src=i) + # the broadcast op might be launched in a different stream + # need to synchronize to make sure the tensor is ready + torch.accelerator.synchronize() + assert torch.all(recv_tensors[i] == i).cpu().item() + + +def test_ncclGetUniqueId(): + lib = NCCLLibrary() + unique_id = lib.ncclGetUniqueId() + # `list(unique_id.internal)` is something like this: + # [34, -16, 23, 83, 109, -19, 59, 95, 2, 0, -86, 55, 10, -128, 0, 29, 0, + # 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + # 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + # 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + # 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + # 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] + # as long as the function doesn't raise an exception, we're good + assert unique_id is not None diff --git a/asian_ai/performance/vLLM/tests/distributed/test_quick_all_reduce.py b/asian_ai/performance/vLLM/tests/distributed/test_quick_all_reduce.py new file mode 100644 index 0000000000000000000000000000000000000000..13b6d0caa7a5ee1fc1e2b4387e04be478671b511 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_quick_all_reduce.py @@ -0,0 +1,483 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import multiprocessing +import random + +import pytest +import ray +import torch +import torch.distributed as dist + +import vllm.envs as envs +from vllm import _custom_ops as ops +from vllm.distributed.communication_op import tensor_model_parallel_all_reduce # noqa +from vllm.distributed.device_communicators.quick_all_reduce import ( + KB, + MB, + QuickAllReduce, + QuickReduceRegime, +) +from vllm.distributed.parallel_state import get_tp_group, graph_capture +from vllm.envs import disable_envs_cache +from vllm.platforms import current_platform + +from ..utils import ( + ensure_model_parallel_initialized, + init_test_distributed_environment, + multi_process_parallel, + set_random_seed, +) + + +def on_gfx942() -> bool: + if current_platform.is_rocm(): + from vllm.platforms.rocm import on_gfx942 as rocm_on_gfx942 + + return rocm_on_gfx942() + return False + + +set_random_seed(42) +_test_size_rng = random.Random(44) +# Size over 8MB is sufficient for custom quick allreduce. +test_sizes = [ + _test_size_rng.randint(8 * 1024 * 1024, 10 * 1024 * 1024) for _ in range(8) +] +for i, v in enumerate(test_sizes): + test_sizes[i] -= v % 8 + + +def _assert_quickreduce(fa, inp): + assert fa is not None + assert not fa.disabled + assert fa.should_quick_allreduce(inp) + + +@pytest.fixture +def envs_cache_disabled(): + disable_envs_cache() + yield + disable_envs_cache() + + +def _make_quick_allreduce_for_test( + min_size_mb: int | None = None, + quantization_min_size: int | None = None, +) -> QuickAllReduce: + quick_reduce = QuickAllReduce.__new__(QuickAllReduce) + quick_reduce.disabled = False + quick_reduce.qr_max_size = 16 * MB + quick_reduce.qr_min_size = min_size_mb * MB if min_size_mb is not None else None + quick_reduce.qr_quant_level = QuickReduceRegime.INT4 + quick_reduce.qr_quantization_min_size = quantization_min_size + quick_reduce.use_fp16_kernels = False + quick_reduce.world_size = 2 + return quick_reduce + + +def test_should_quick_allreduce_uses_builtin_min_size_when_unset(): + quick_reduce = _make_quick_allreduce_for_test(min_size_mb=None) + + below_builtin_min = torch.empty(MB // 4, dtype=torch.float16) + at_builtin_min = torch.empty(MB // 2, dtype=torch.float16) + + assert not quick_reduce.should_quick_allreduce(below_builtin_min) + assert quick_reduce.should_quick_allreduce(at_builtin_min) + + +def test_should_quick_allreduce_uses_min_size_override(): + quick_reduce = _make_quick_allreduce_for_test(min_size_mb=0) + + below_builtin_min = torch.empty(8, dtype=torch.float16) + + assert quick_reduce.should_quick_allreduce(below_builtin_min) + + +def test_quick_allreduce_min_size_env_unset( + monkeypatch: pytest.MonkeyPatch, + envs_cache_disabled, +): + monkeypatch.delenv("VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB", raising=False) + + assert QuickAllReduce._get_qr_min_size(qr_max_size=16 * MB) is None + + +def test_quick_allreduce_min_size_env_converts_mb_to_bytes( + monkeypatch: pytest.MonkeyPatch, + envs_cache_disabled, +): + monkeypatch.setenv("VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB", "4") + + assert QuickAllReduce._get_qr_min_size(qr_max_size=16 * MB) == 4 * MB + + +def test_quick_allreduce_min_size_env_rejects_negative( + monkeypatch: pytest.MonkeyPatch, + envs_cache_disabled, +): + monkeypatch.setenv("VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB", "-1") + + with pytest.raises(ValueError, match="must be non-negative"): + QuickAllReduce._get_qr_min_size(qr_max_size=16 * MB) + + +def test_quick_allreduce_min_size_env_allows_equal_to_max( + monkeypatch: pytest.MonkeyPatch, + envs_cache_disabled, +): + monkeypatch.setenv("VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB", "16") + + assert QuickAllReduce._get_qr_min_size(qr_max_size=16 * MB) == 16 * MB + + +def test_quick_allreduce_min_size_env_rejects_larger_than_max( + monkeypatch: pytest.MonkeyPatch, + envs_cache_disabled, +): + monkeypatch.setenv("VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB", "17") + + with pytest.raises(ValueError, match="effective QuickReduce max size"): + QuickAllReduce._get_qr_min_size(qr_max_size=16 * MB) + + +def test_quick_allreduce_quantization_min_size_env_unset( + monkeypatch: pytest.MonkeyPatch, + envs_cache_disabled, +): + monkeypatch.delenv("VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB", raising=False) + + assert QuickAllReduce._get_qr_quantization_min_size() is None + + +def test_quick_allreduce_quantization_min_size_env_converts_kb_to_bytes( + monkeypatch: pytest.MonkeyPatch, + envs_cache_disabled, +): + monkeypatch.setenv("VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB", "2048") + + assert QuickAllReduce._get_qr_quantization_min_size() == 2048 * KB + + +def test_quick_allreduce_quantization_min_size_env_rejects_negative( + monkeypatch: pytest.MonkeyPatch, + envs_cache_disabled, +): + monkeypatch.setenv("VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB", "-1") + + with pytest.raises(ValueError, match="must be non-negative"): + QuickAllReduce._get_qr_quantization_min_size() + + +def test_quick_allreduce_quantization_min_size_unset_uses_configured_codec(): + quick_reduce = _make_quick_allreduce_for_test(quantization_min_size=None) + inp = torch.empty(8, dtype=torch.float16) + + assert quick_reduce._get_qr_quant_level(inp) == QuickReduceRegime.INT4.value + + +def test_quick_allreduce_quantization_min_size_uses_fp_below_threshold(): + quick_reduce = _make_quick_allreduce_for_test(quantization_min_size=2048) + inp = torch.empty(1024 // 2, dtype=torch.float16) + + assert quick_reduce._get_qr_quant_level(inp) == QuickReduceRegime.FP.value + + +def test_quick_allreduce_quantization_min_size_uses_configured_codec_at_threshold(): + quick_reduce = _make_quick_allreduce_for_test(quantization_min_size=2048) + inp = torch.empty(2048 // 2, dtype=torch.float16) + + assert quick_reduce._get_qr_quant_level(inp) == QuickReduceRegime.INT4.value + + +def test_quick_allreduce_quantization_min_size_does_not_change_eligibility(): + quick_reduce = _make_quick_allreduce_for_test(quantization_min_size=2 * MB) + + below_builtin_min = torch.empty(MB // 4, dtype=torch.float16) + at_builtin_min = torch.empty(MB // 2, dtype=torch.float16) + + assert not quick_reduce.should_quick_allreduce(below_builtin_min) + assert quick_reduce.should_quick_allreduce(at_builtin_min) + + +def test_quick_allreduce_passes_dynamic_quant_level( + monkeypatch: pytest.MonkeyPatch, +): + quick_reduce = _make_quick_allreduce_for_test(quantization_min_size=2 * KB) + quick_reduce._ptr = object() + inp = torch.empty(KB // 2, dtype=torch.float16) + called_quant_level = None + + def fake_qr_all_reduce( + fa, + inp, + out, + quant_level, + cast_bf2half, + ): + nonlocal called_quant_level + called_quant_level = quant_level + + monkeypatch.setattr(ops, "qr_all_reduce", fake_qr_all_reduce) + + quick_reduce.quick_all_reduce(inp) + + assert called_quant_level == QuickReduceRegime.FP.value + + +@ray.remote(num_gpus=1, max_calls=1) +def graph_quickreduce( + monkeypatch: pytest.MonkeyPatch, + tp_size, + pp_size, + rank, + distributed_init_port, +): + with monkeypatch.context() as m: + m.delenv("CUDA_VISIBLE_DEVICES", raising=False) + m.delenv("HIP_VISIBLE_DEVICES", raising=False) + m.delenv("ROCR_VISIBLE_DEVICES", raising=False) + device = torch.device(f"cuda:{rank}") + torch.accelerator.set_device_index(device) + init_test_distributed_environment(tp_size, pp_size, rank, distributed_init_port) + ensure_model_parallel_initialized(tp_size, pp_size) + group = get_tp_group().device_group + fa = get_tp_group().device_communicator.qr_comm + + # A small all_reduce for warmup. + # this is needed because device communicators might be created lazily + # (e.g. NCCL). This will ensure that the communicator is initialized + # before any communication happens, so that this group can be used for + # graph capture immediately. + data = torch.zeros(1) + data = data.to(device=device) + torch.distributed.all_reduce(data, group=group) + torch.accelerator.synchronize() + del data + + # we use the first group to communicate once + # and the second group to communicate twice + # and so on + # this is used to demonstrate that each group can + # communicate independently + num_communication = rank // tp_size + 1 + + for sz in test_sizes: + for dtype in [torch.float16, torch.bfloat16]: + with graph_capture(device=device) as graph_capture_context: + device_idx = torch.accelerator.current_device_index() + inp1 = torch.randint(1, 23, (sz,), dtype=dtype, device=device_idx) + inp2 = torch.randint(-23, 1, (sz,), dtype=dtype, device=device_idx) + _assert_quickreduce(fa, inp1) + _assert_quickreduce(fa, inp2) + + torch.accelerator.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=graph_capture_context.stream): + for _ in range(num_communication): + out1 = tensor_model_parallel_all_reduce(inp1) + dist.all_reduce(inp1, group=group) + out2 = tensor_model_parallel_all_reduce(inp2) + dist.all_reduce(inp2, group=group) + graph.replay() + torch.testing.assert_close(out1, inp1, atol=2.5, rtol=0.1) + torch.testing.assert_close(out2, inp2, atol=2.5, rtol=0.1) + + +@ray.remote(num_gpus=1, max_calls=1) +def eager_quickreduce( + monkeypatch: pytest.MonkeyPatch, + tp_size, + pp_size, + rank, + distributed_init_port, +): + with monkeypatch.context() as m: + m.delenv("CUDA_VISIBLE_DEVICES", raising=False) + m.delenv("HIP_VISIBLE_DEVICES", raising=False) + m.delenv("ROCR_VISIBLE_DEVICES", raising=False) + device = torch.device(f"cuda:{rank}") + torch.accelerator.set_device_index(device) + + init_test_distributed_environment(tp_size, pp_size, rank, distributed_init_port) + + # Size over 8MB is sufficient for custom quick allreduce. + sz = 16 * 1024 * 1024 + fa = get_tp_group().device_communicator.qr_comm + inp = torch.tensor( + [1.0 * ((i) % 23) for i in range(sz)], dtype=torch.float16, device=device + ) + _assert_quickreduce(fa, inp) + out = fa.quick_all_reduce(inp) + torch.testing.assert_close(out, inp * tp_size, atol=2.5, rtol=0.1) + + inp = torch.tensor( + [1.0 * ((i) % 23) for i in range(sz)], dtype=torch.bfloat16, device=device + ) + _assert_quickreduce(fa, inp) + out = fa.quick_all_reduce(inp) + torch.testing.assert_close(out, inp * tp_size, atol=2.5, rtol=0.1) + + +@ray.remote(num_gpus=1, max_calls=1) +def bf16_cast_quickreduce( + monkeypatch: pytest.MonkeyPatch, + tp_size, + pp_size, + rank, + distributed_init_port, +): + with monkeypatch.context() as m: + m.delenv("CUDA_VISIBLE_DEVICES", raising=False) + m.delenv("HIP_VISIBLE_DEVICES", raising=False) + m.delenv("ROCR_VISIBLE_DEVICES", raising=False) + m.setenv("VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16", "1") + device = torch.device(f"cuda:{rank}") + torch.accelerator.set_device_index(device) + init_test_distributed_environment(tp_size, pp_size, rank, distributed_init_port) + + sz = 16 * 1024 * 1024 + fa = get_tp_group().device_communicator.qr_comm + inp = torch.tensor( + [1.0 * (i % 23) for i in range(sz)], dtype=torch.bfloat16, device=device + ) + _assert_quickreduce(fa, inp) + assert fa.use_fp16_kernels + out = fa.quick_all_reduce(inp) + torch.testing.assert_close(out, inp * tp_size, atol=2.5, rtol=0.1) + + +@pytest.mark.skipif( + not current_platform.is_rocm(), reason="only test quick allreduce for rocm" +) +@pytest.mark.parametrize("quant_mode", ["FP", "INT8", "INT6", "INT4", "INT3"]) +@pytest.mark.parametrize("tp_size", [2]) +@pytest.mark.parametrize("pipeline_parallel_size", [1, 2]) +@pytest.mark.parametrize("test_target", [graph_quickreduce, eager_quickreduce]) +def test_custom_quick_allreduce( + monkeypatch: pytest.MonkeyPatch, + tp_size, + pipeline_parallel_size, + test_target, + quant_mode, +): + world_size = tp_size * pipeline_parallel_size + if world_size > torch.accelerator.device_count(): + pytest.skip("Not enough GPUs to run the test.") + if test_target is graph_quickreduce and on_gfx942(): + pytest.xfail( + "CUDA graph capture with quick reduce hits " + "hipErrorStreamCaptureInvalidated on gfx942" + ) + + monkeypatch.setenv("VLLM_ROCM_QUICK_REDUCE_QUANTIZATION", quant_mode) + + multi_process_parallel(monkeypatch, tp_size, pipeline_parallel_size, test_target) + + +@pytest.mark.skipif( + not current_platform.is_rocm(), reason="only test quick allreduce for rocm" +) +def test_custom_quick_allreduce_bf16_cast(monkeypatch: pytest.MonkeyPatch): + if torch.accelerator.device_count() < 2: + pytest.skip("Not enough GPUs to run the test.") + monkeypatch.setenv("VLLM_ROCM_QUICK_REDUCE_QUANTIZATION", "FP") + multi_process_parallel(monkeypatch, 2, 1, bf16_cast_quickreduce) + + +def qr_variable_input(rank, world_size): + """ + When the tensor parallelism is set to 4 or 8, frequent changes + in the input shape can cause QuickReduce to hang (this issue + has been observed with the gpt_oss model). + """ + device = torch.device(f"cuda:{rank}") + torch.accelerator.set_device_index(device) + qr_max_size = None # MB + _ptr = ops.init_custom_qr(rank, world_size, qr_max_size) + ranks = [] + for i in range(world_size): + ranks.append(i) + if envs.VLLM_DISTRIBUTED_USE_SPLIT_GROUP: + dist.init_process_group( + backend="cpu:gloo,cuda:nccl", + init_method="tcp://127.0.0.1:29500", + rank=rank, + world_size=world_size, + device_id=device, + ) + else: + dist.init_process_group( + backend="nccl", + init_method="tcp://127.0.0.1:29500", + rank=rank, + world_size=world_size, + ) + if envs.VLLM_DISTRIBUTED_USE_SPLIT_GROUP: + cpu_group = torch.distributed.split_group( + split_ranks=[ranks], backend="cpu:gloo,cuda:nccl" + ) + else: + cpu_group = torch.distributed.new_group(ranks, backend="nccl") + + handle = ops.qr_get_handle(_ptr) + world_size = dist.get_world_size(group=cpu_group) + handles = [None] * world_size + dist.all_gather_object(handles, handle, group=cpu_group) + ops.qr_open_handles(_ptr, handles) + + num = 1 + s1 = 1024 + while num < 50000: # 50000 is sufficient to identify issues. + dtype = torch.float16 + device_idx = torch.accelerator.current_device_index() + if num % 2 == 0: + s2 = 1024 + inp1 = torch.zeros((s1, s2), dtype=dtype, device=device_idx) + else: + s2 = 2048 + inp1 = torch.ones((s1, s2), dtype=dtype, device=device_idx) + result = torch.empty_like(inp1) + # FP = 0 INT8 = 1 INT6 = 2 INT4 = 3 INT3 = 4 + ops.qr_all_reduce(_ptr, inp1, result, 3, cast_bf2half=True) + try: + if inp1[0, 0] == 0: + assert torch.all(result == 0) + else: + assert torch.all(result == world_size) + except AssertionError: + print("Assertion failed! Allreduce results are incorrect.") + raise + num += 1 + + +@pytest.mark.skipif( + not current_platform.is_rocm(), reason="only test quick allreduce for rocm" +) +@pytest.mark.parametrize("tp_size", [4, 8]) +@pytest.mark.parametrize("pipeline_parallel_size", [1]) +def test_custom_quick_allreduce_variable_input(tp_size, pipeline_parallel_size): + world_size = tp_size * pipeline_parallel_size + if world_size > torch.accelerator.device_count(): + pytest.skip("Not enough GPUs to run the test.") + + multiprocessing.set_start_method("spawn", force=True) + # 60s is enough + timeout = 60 + processes = [] + for rank in range(tp_size): + p = multiprocessing.Process(target=qr_variable_input, args=(rank, tp_size)) + p.start() + processes.append((rank, p)) + for rank, p in processes: + p.join(timeout=timeout) + if p.is_alive(): + for r, proc in processes: + if proc.is_alive(): + proc.terminate() + proc.join() + raise RuntimeError(f"QuickReduce hang detected after {timeout} seconds!") + + +if __name__ == "__main__": + test_custom_quick_allreduce_variable_input(tp_size=4, pipeline_parallel_size=1) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_ray_v2_executor.py b/asian_ai/performance/vLLM/tests/distributed/test_ray_v2_executor.py new file mode 100644 index 0000000000000000000000000000000000000000..850284144de7a9407703cba89219437eb6d7199c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_ray_v2_executor.py @@ -0,0 +1,389 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +""" +Integration tests for RayExecutorV2 at the executor level. +Validates executor initialization, placement group support, RPC calls, +and distributed execution with various TP/PP configurations. +""" + +import gc +import threading +from unittest.mock import patch + +import pytest +import ray + +from vllm import LLM +from vllm.config import VllmConfig +from vllm.engine.arg_utils import EngineArgs +from vllm.v1.executor import ray_executor_v2 +from vllm.v1.executor.ray_executor_v2 import RayExecutorV2 + +pytestmark = pytest.mark.usefixtures("enable_ray_v2_backend") + +MODEL = "facebook/opt-125m" + + +def create_vllm_config( + tensor_parallel_size: int = 1, + pipeline_parallel_size: int = 1, + max_model_len: int = 256, + gpu_memory_utilization: float = 0.3, + placement_group=None, +) -> VllmConfig: + engine_args = EngineArgs( + model=MODEL, + tensor_parallel_size=tensor_parallel_size, + pipeline_parallel_size=pipeline_parallel_size, + max_model_len=max_model_len, + gpu_memory_utilization=gpu_memory_utilization, + distributed_executor_backend="ray", + enforce_eager=True, + ) + vllm_config = engine_args.create_engine_config() + + if placement_group is not None: + vllm_config.parallel_config.placement_group = placement_group + + return vllm_config + + +def ensure_ray_initialized(): + if not ray.is_initialized(): + ray.init(ignore_reinit_error=True) + + +@pytest.fixture +def create_placement_group(request): + ensure_ray_initialized() + num_gpus = request.param + bundles = [{"GPU": 1, "CPU": 1} for _ in range(num_gpus)] + pg = ray.util.placement_group(bundles, strategy="PACK") + ray.get(pg.ready()) + yield pg + ray.util.remove_placement_group(pg) + + +@pytest.fixture +def executor(request): + """Create a RayExecutorV2 and shut it down after the test.""" + executor = RayExecutorV2(vllm_config=request.param) + yield executor + executor.shutdown() + + +def assert_executor(executor, tp_size, pp_size): + """Common assertions for executor initialization tests.""" + world_size = tp_size * pp_size + expected_output_rank = (pp_size - 1) * tp_size + + assert executor.world_size == world_size + assert len(executor.ray_worker_handles) == world_size + assert len(executor.response_mqs) == world_size + assert executor._get_output_rank() == expected_output_rank + + if pp_size > 1: + expected_concurrent_batches = pp_size + int( + executor.vllm_config.scheduler_config.async_scheduling + and executor.vllm_config.use_v2_model_runner + ) + assert ( + executor.vllm_config.max_concurrent_batches == expected_concurrent_batches + ) + + executor.check_health() + assert not executor.is_failed + + ranks = sorted(h.rank for h in executor.ray_worker_handles) + assert ranks == list(range(world_size)) + + for handle in executor.ray_worker_handles: + assert handle.node_id is not None + + +def test_select_tcpstore_port_seeds_disjoint_windows(monkeypatch): + """Co-located DP engines scan distinct, adjacent port windows, so two + engines on a node cannot pick the same TCPStore port.""" + requested = [] + + def fake_get_open_port(start_port, max_attempts): + requested.append((start_port, max_attempts)) + return start_port + + monkeypatch.setattr(ray_executor_v2, "_get_open_port", fake_get_open_port) + + ports = [ + RayExecutorV2._select_tcpstore_port(rank, master_port=29500) + for rank in range(4) + ] + + assert requested == [(29600, 32), (29632, 32), (29664, 32), (29696, 32)] + assert len(set(ports)) == 4 + + +def test_select_tcpstore_port_non_dp_uses_random(monkeypatch): + """A non-DP engine has no local rank and uses a random port.""" + monkeypatch.setattr(ray_executor_v2, "get_open_port", lambda: 54321) + assert RayExecutorV2._select_tcpstore_port(None, master_port=29500) == 54321 + + +def test_select_tcpstore_port_full_window_uses_random(monkeypatch): + """A fully occupied window falls back to a random port.""" + + def raise_full(start_port, max_attempts): + raise RuntimeError("no open port") + + monkeypatch.setattr(ray_executor_v2, "_get_open_port", raise_full) + monkeypatch.setattr(ray_executor_v2, "get_open_port", lambda: 54321) + assert RayExecutorV2._select_tcpstore_port(0, master_port=29500) == 54321 + + +@pytest.mark.parametrize("tp_size, pp_size", [(1, 1), (2, 1), (4, 1), (2, 2)]) +def test_ray_v2_executor(tp_size, pp_size): + """Validate RayExecutorV2 with various TP/PP configs.""" + vllm_config = create_vllm_config( + tensor_parallel_size=tp_size, + pipeline_parallel_size=pp_size, + ) + executor = RayExecutorV2(vllm_config=vllm_config) + try: + assert_executor(executor, tp_size, pp_size) + finally: + executor.shutdown() + + +@pytest.mark.parametrize( + "tp_size, pp_size, create_placement_group", + [(2, 1, 2), (4, 1, 4), (2, 2, 4)], + indirect=["create_placement_group"], +) +def test_ray_v2_executor_pg(tp_size, pp_size, create_placement_group): + """Validate RayExecutorV2 with various TP/PP configs using external PG.""" + vllm_config = create_vllm_config( + tensor_parallel_size=tp_size, + pipeline_parallel_size=pp_size, + placement_group=create_placement_group, + ) + executor = RayExecutorV2(vllm_config=vllm_config) + try: + assert_executor(executor, tp_size, pp_size) + finally: + executor.shutdown() + + +@pytest.mark.parametrize( + "executor", + [create_vllm_config(tensor_parallel_size=2)], + indirect=True, +) +def test_ray_v2_executor_failure_callback(executor): + """Validate failure callback registration.""" + callback_invoked = False + + def test_callback(): + nonlocal callback_invoked + callback_invoked = True + + executor.register_failure_callback(test_callback) + assert not callback_invoked + + executor.is_failed = True + executor.register_failure_callback(test_callback) + assert callback_invoked + + +@pytest.mark.parametrize( + "executor", + [create_vllm_config(tensor_parallel_size=2)], + indirect=True, +) +def test_ray_v2_executor_collective_rpc(executor): + """Validate collective RPC calls through MessageQueue.""" + executor.check_health() + assert not executor.is_failed + assert executor.rpc_broadcast_mq is not None + + +@pytest.mark.parametrize( + "executor", + [create_vllm_config(tensor_parallel_size=2)], + indirect=True, +) +def test_ray_v2_executor_driver_node_rank_0(executor): + """Validate that driver node workers get the lowest ranks.""" + driver_node = ray.get_runtime_context().get_node_id() + + for handle in executor.ray_worker_handles: + assert handle.node_id == driver_node + + rank0_handle = next(h for h in executor.ray_worker_handles if h.rank == 0) + assert rank0_handle.node_id == driver_node + + +@pytest.mark.parametrize( + "executor", + [create_vllm_config(tensor_parallel_size=2)], + indirect=True, +) +def test_ray_v2_executor_worker_death(executor): + """Validate executor detects worker death via ray.wait().""" + callback_event = threading.Event() + + def on_failure(): + callback_event.set() + + executor.register_failure_callback(on_failure) + assert not executor.is_failed + + # Kill one worker actor externally + victim = executor.ray_worker_handles[1].actor + ray.kill(victim, no_restart=True) + + # Monitor thread should detect the death and invoke callback + assert callback_event.wait(timeout=30) + assert executor.is_failed + assert executor.shutting_down + + +def test_ray_v2_executor_shutdown(): + """Validate graceful shutdown: ray.kill() terminates all worker actors.""" + executor = RayExecutorV2(vllm_config=create_vllm_config(tensor_parallel_size=2)) + assert executor.rpc_broadcast_mq is not None + assert len(executor.response_mqs) == executor.world_size + + actors = [h.actor for h in executor.ray_worker_handles] + executor.shutdown() + + for actor in actors: + with pytest.raises(ray.exceptions.RayActorError): + ray.get(actor.wait_for_init.remote(), timeout=5) + + assert executor.rpc_broadcast_mq is None + assert len(executor.response_mqs) == 0 + + +@pytest.mark.parametrize( + "executor", + [create_vllm_config(tensor_parallel_size=2)], + indirect=True, +) +def test_ray_v2_run_refs_stored_for_monitoring(executor): + """Validate worker handles store run_ref for monitoring.""" + for handle in executor.ray_worker_handles: + assert handle.run_ref is not None + ready, _ = ray.wait([handle.run_ref], timeout=0) + assert len(ready) == 0, "run_ref should be pending" + + +@pytest.mark.parametrize("tp_size, pp_size", [(2, 1), (2, 2)]) +def test_ray_v2_single_node_generation(tp_size, pp_size): + """End-to-end LLM generation with RayExecutorV2.""" + + llm = LLM( + model=MODEL, + tensor_parallel_size=tp_size, + pipeline_parallel_size=pp_size, + distributed_executor_backend="ray", + enforce_eager=True, + max_model_len=256, + gpu_memory_utilization=0.3, + ) + try: + prompts = [ + "Hello, my name is", + "The capital of France is", + "The future of AI is", + ] + outputs = llm.generate(prompts) + + assert len(outputs) == len(prompts) + for output in outputs: + assert len(output.outputs) > 0 + assert len(output.outputs[0].text) > 0 + finally: + llm.llm_engine.model_executor.shutdown() + del llm + gc.collect() + + +@pytest.mark.parametrize( + "bundle_indices, expected_bundle_ids, create_placement_group", + [("2,3", [2, 3], 4), ("3,2", [3, 2], 4)], + indirect=["create_placement_group"], +) +def test_ray_v2_bundle_indices_env( + bundle_indices, expected_bundle_ids, create_placement_group, monkeypatch +): + """Validate explicit VLLM_RAY_BUNDLE_INDICES bundle placement.""" + monkeypatch.setenv("VLLM_RAY_BUNDLE_INDICES", bundle_indices) + vllm_config = create_vllm_config( + tensor_parallel_size=2, + placement_group=create_placement_group, + ) + executor = RayExecutorV2(vllm_config=vllm_config) + try: + actual = [ + h.bundle_id_idx + for h in sorted(executor.ray_worker_handles, key=lambda h: h.rank) + ] + assert actual == expected_bundle_ids + assert_executor(executor, tp_size=2, pp_size=1) + finally: + executor.shutdown() + + +@pytest.mark.parametrize( + "bundle_indices, expected_error, create_placement_group", + [ + ("1,1", "cannot have duplicate values,", 4), + ("0,1,2", "must have the same size", 4), + ], + indirect=["create_placement_group"], +) +def test_ray_v2_invalid_bundle_indices( + bundle_indices, expected_error, create_placement_group, monkeypatch +): + """Validate invalid bundle indices are rejected.""" + monkeypatch.setenv("VLLM_RAY_BUNDLE_INDICES", bundle_indices) + vllm_config = create_vllm_config( + tensor_parallel_size=2, placement_group=create_placement_group + ) + with pytest.raises(AssertionError, match=expected_error): + RayExecutorV2(vllm_config=vllm_config) + + +@pytest.mark.parametrize("tp_size, pp_size", [(2, 1), (2, 2)]) +def test_ray_v2_single_node_generation_with_pg(tp_size, pp_size): + """E2E LLM generation with a user-provided placement group.""" + ensure_ray_initialized() + bundles = [{"GPU": 1, "CPU": 1} for _ in range(tp_size * pp_size)] + pg = ray.util.placement_group(bundles, strategy="PACK") + ray.get(pg.ready()) + + try: + with patch.object(ray.util, "get_current_placement_group", return_value=pg): + llm = LLM( + model=MODEL, + tensor_parallel_size=tp_size, + pipeline_parallel_size=pp_size, + distributed_executor_backend="ray", + enforce_eager=True, + max_model_len=256, + gpu_memory_utilization=0.3, + ) + prompts = [ + "Hello, my name is", + "The capital of France is", + "The future of AI is", + ] + outputs = llm.generate(prompts) + + assert len(outputs) == len(prompts) + for output in outputs: + assert len(output.outputs) > 0 + assert len(output.outputs[0].text) > 0 + finally: + llm.llm_engine.model_executor.shutdown() + del llm + gc.collect() diff --git a/asian_ai/performance/vLLM/tests/distributed/test_ray_v2_executor_e2e.py b/asian_ai/performance/vLLM/tests/distributed/test_ray_v2_executor_e2e.py new file mode 100644 index 0000000000000000000000000000000000000000..a114176e4d5a70b426a776e7074b155e4c4f09fa --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_ray_v2_executor_e2e.py @@ -0,0 +1,209 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Orchestration-level integration tests for RayExecutorV2. +""" + +import gc +import os +import pathlib + +import pytest +import ray + +pytestmark = pytest.mark.usefixtures("enable_ray_v2_backend") + +MODEL = "facebook/opt-125m" + + +def _get_env_var(worker, name): + return os.environ.get(name) + + +def _ray_init(): + """Start Ray with the project root on workers' PYTHONPATH. + + Without this, workers cannot unpickle actor classes defined in the + ``tests`` package, causing FunctionActorManager to fall back to + TemporaryActor which drops async method signatures.""" + project_root = str(pathlib.Path(__file__).resolve().parents[2]) + ray.init( + ignore_reinit_error=True, + runtime_env={"env_vars": {"PYTHONPATH": project_root}}, + ) + + +@pytest.fixture +def ray_init(): + _ray_init() + + +class _AsyncLLMActor: + def start(self, pg, bundle_indices=None, ray_runtime_env=None): + os.environ["VLLM_USE_RAY_V2_EXECUTOR_BACKEND"] = "1" + # Needed so collective_rpc can pickle _get_env_var over the + # AsyncLLM -> EngineCore ZMQ boundary. + os.environ["VLLM_ALLOW_INSECURE_SERIALIZATION"] = "1" + if bundle_indices is not None: + os.environ["VLLM_RAY_BUNDLE_INDICES"] = bundle_indices + else: + os.environ.pop("VLLM_RAY_BUNDLE_INDICES", None) + + from vllm.engine.arg_utils import AsyncEngineArgs + from vllm.v1.engine.async_llm import AsyncLLM + from vllm.v1.executor.abstract import Executor + + engine_args = AsyncEngineArgs( + model=MODEL, + tensor_parallel_size=2, + distributed_executor_backend="ray", + enforce_eager=True, + max_model_len=256, + gpu_memory_utilization=0.8, + ) + vllm_config = engine_args.create_engine_config() + vllm_config.parallel_config.placement_group = pg + if ray_runtime_env is not None: + vllm_config.parallel_config.ray_runtime_env = ray_runtime_env + + executor_class = Executor.get_class(vllm_config) + self.engine = AsyncLLM( + vllm_config=vllm_config, + executor_class=executor_class, + log_stats=False, + log_requests=False, + ) + + async def generate(self, prompt): + from vllm.sampling_params import SamplingParams + + params = SamplingParams(max_tokens=16) + result = None + async for output in self.engine.generate( + prompt, params, request_id="test_request_id" + ): + result = output + assert result is not None + return result.outputs[0].text + + async def generate_and_get_worker_envs(self, prompt, env_names): + from vllm.sampling_params import SamplingParams + + params = SamplingParams(max_tokens=16) + result = None + async for output in self.engine.generate( + prompt, params, request_id="test_request_id" + ): + result = output + assert result is not None + text = result.outputs[0].text + + env_results = {} + for name in env_names: + vals = await self.engine.collective_rpc( + _get_env_var, timeout=10, args=(name,) + ) + env_results[name] = vals + return text, env_results + + def shutdown(self): + if engine := getattr(self, "engine", None): + engine.shutdown() + del self.engine + gc.collect() + + +AsyncLLMActor = ray.remote(num_cpus=0, max_concurrency=1)(_AsyncLLMActor) + + +def test_multi_replicas(ray_init): + pg1 = ray.util.placement_group([{"GPU": 1, "CPU": 1}] * 2, strategy="PACK") + pg2 = ray.util.placement_group([{"GPU": 1, "CPU": 1}] * 2, strategy="PACK") + ray.get([pg1.ready(), pg2.ready()]) + + actor1 = AsyncLLMActor.remote() + actor2 = AsyncLLMActor.remote() + + ray.get(actor1.start.remote(pg1)) + ray.get(actor2.start.remote(pg2)) + + out1, out2 = ray.get( + [ + actor1.generate.remote("Hello world"), + actor2.generate.remote("Hello world"), + ] + ) + assert len(out1) > 0 + assert len(out2) > 0 + + +def test_multi_replicas_with_bundle_indices(ray_init): + pg = ray.util.placement_group([{"GPU": 1, "CPU": 1}] * 4, strategy="PACK") + ray.get(pg.ready()) + + actor1 = AsyncLLMActor.remote() + actor2 = AsyncLLMActor.remote() + + ray.get(actor1.start.remote(pg, bundle_indices="2,1")) + ray.get(actor2.start.remote(pg, bundle_indices="0,3")) + + out1, out2 = ray.get( + [ + actor1.generate.remote("Hello world"), + actor2.generate.remote("Hello world"), + ] + ) + assert len(out1) > 0 + assert len(out2) > 0 + + +def test_env_var_and_runtime_env_propagation(): + """ + Verify env vars (NCCL_, HF_) and parallel_config.ray_runtime_env + propagate to RayWorkerProc actors. + """ + sentinel_vars = { + "NCCL_DEBUG": "INFO", + "HF_TOKEN": "test_sentinel_token", + } + for k, v in sentinel_vars.items(): + os.environ[k] = v + + try: + # Called directly (not via the ray_init fixture) because sentinel + # env vars must be in os.environ before ray.init() so that Ray + # worker processes inherit them. + _ray_init() + + pg = ray.util.placement_group([{"GPU": 1, "CPU": 1}] * 2, strategy="PACK") + ray.get(pg.ready()) + + # Include the project root so that RayWorkerProc actors can + # unpickle _get_env_var. + project_root = str(pathlib.Path(__file__).resolve().parents[2]) + ray_runtime_env = { + "env_vars": { + "RAY_RUNTIME_ENV_TEST": "ray_runtime_env", + "PYTHONPATH": project_root, + }, + } + + actor = AsyncLLMActor.remote() + ray.get(actor.start.remote(pg, ray_runtime_env=ray_runtime_env)) + + all_env_names = list(sentinel_vars) + ["RAY_RUNTIME_ENV_TEST"] + text, env_results = ray.get( + actor.generate_and_get_worker_envs.remote("Hello world", all_env_names) + ) + assert len(text) > 0 + + for name, expected in sentinel_vars.items(): + for val in env_results[name]: + assert val == expected + + for val in env_results["RAY_RUNTIME_ENV_TEST"]: + assert val == "ray_runtime_env" + + finally: + for k in sentinel_vars: + os.environ.pop(k, None) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_rocm_quick_reduce.py b/asian_ai/performance/vLLM/tests/distributed/test_rocm_quick_reduce.py new file mode 100644 index 0000000000000000000000000000000000000000..1991cd6605ea18c3bf67c3faf576a671e23e3906 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_rocm_quick_reduce.py @@ -0,0 +1,750 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import contextlib +import importlib +import multiprocessing as mp +import os +import queue +import traceback +from functools import lru_cache +from types import SimpleNamespace +from typing import Literal +from unittest.mock import patch + +import pytest +import torch +import torch.distributed as dist +from huggingface_hub import snapshot_download + +import vllm.envs as envs +from vllm import LLM, SamplingParams +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.platforms import current_platform +from vllm.utils.network_utils import get_open_port + +pytestmark = pytest.mark.skipif( + not current_platform.is_rocm(), + reason="ROCm-only quick-reduce tests", +) + +MB = 1024 * 1024 +WORLD_SIZE = 2 +QUANT_LEVELS = ["FP", "INT8", "INT6", "INT4"] + + +def _log(message: str) -> None: + print(f"[rocm_quick_reduce] {message}", flush=True) + + +def _reload_envs(): + return importlib.reload(envs) + + +def _make_quick_allreduce( + *, + disabled: bool = False, + world_size: int = 2, + quant_level: str = "FP", + use_fp16_kernels: bool = False, + qr_max_size: int = 64 * MB, +): + from vllm.distributed.device_communicators.quick_all_reduce import ( + QuickAllReduce, + QuickReduceRegime, + ) + + qar = QuickAllReduce.__new__(QuickAllReduce) + qar.disabled = disabled + qar.world_size = world_size + qar.use_fp16_kernels = use_fp16_kernels + qar.qr_quant_level = QuickReduceRegime[quant_level] + qar.qr_max_size = qr_max_size + return qar + + +def _quick_allreduce_worker( + rank: int, + port: int, + quant_level: str, + dtype_name: str, + cast_bf16: bool, +): + os.environ["VLLM_ROCM_QUICK_REDUCE_QUANTIZATION"] = quant_level + os.environ["VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16"] = "1" if cast_bf16 else "0" + _log( + f"worker start: rank={rank} quant={quant_level} " + f"dtype={dtype_name} cast_bf16={cast_bf16}" + ) + + device = torch.device(f"cuda:{rank}") + torch.accelerator.set_device_index(device) + dist.init_process_group( + backend="gloo", + init_method=f"tcp://127.0.0.1:{port}", + rank=rank, + world_size=WORLD_SIZE, + ) + + qar = None + try: + from vllm.distributed.device_communicators.quick_all_reduce import ( + QuickAllReduce, + ) + + qar = QuickAllReduce(group=dist.GroupMember.WORLD, device=rank) + assert not qar.disabled + + num_elements = 8 * MB if dtype_name == "float16" else 4 * MB + + dtype = getattr(torch, dtype_name) + inp = torch.ones(num_elements, dtype=dtype, device=device) + + assert qar.should_quick_allreduce(inp) + if cast_bf16: + assert qar.use_fp16_kernels + + out = qar.quick_all_reduce(inp) + assert torch.allclose(out, inp * WORLD_SIZE, atol=2.5, rtol=0.1) + _log( + f"worker complete: rank={rank} quant={quant_level} " + f"dtype={dtype_name} num_elements={num_elements} " + f"use_fp16_kernels={qar.use_fp16_kernels}" + ) + finally: + if qar is not None: + qar.close() + if dist.is_initialized(): + dist.destroy_process_group() + + +def _run_two_gpu_quick_allreduce_test( + *, + quant_level: str, + dtype_name: str, + cast_bf16: bool, +): + _log( + f"launch 2-GPU case: quant={quant_level} " + f"dtype={dtype_name} cast_bf16={cast_bf16}" + ) + ctx = mp.get_context("spawn") + port = get_open_port() + procs = [] + + for rank in range(WORLD_SIZE): + proc = ctx.Process( + target=_quick_allreduce_worker, + args=(rank, port, quant_level, dtype_name, cast_bf16), + ) + proc.start() + procs.append(proc) + + for proc in procs: + proc.join(timeout=60) + assert proc.exitcode == 0, f"worker exited with code {proc.exitcode}" + _log( + f"finished 2-GPU case: quant={quant_level} " + f"dtype={dtype_name} cast_bf16={cast_bf16}" + ) + + +MODEL_NAME = "Qwen/Qwen2.5-0.5B-Instruct" +E2E_PREFILL_TOKENS = 1024 +E2E_MAX_MODEL_LEN = 1536 +E2E_GPU_MEMORY_UTILIZATION = 0.3 +E2E_KV_CACHE_MEMORY_BYTES = 2 << 30 + +_BACKGROUND_LINE = ( + "Background filler: this archived operations memo repeats a routine status " + "line so the distributed test uses a realistically long prefill." +) +_BACKGROUND_BLOCK = " ".join([_BACKGROUND_LINE] * 48) + + +def _build_prompt(*, fact_block: str, question: str) -> str: + return ( + "Read the archived operations memo below. Most of the memo is filler. " + "Use only the fact block near the end when answering.\n" + f"{_BACKGROUND_BLOCK}\n" + "Fact block:\n" + f"{fact_block}\n" + f"Question: {question}\n" + "Answer in one short sentence." + ) + + +E2E_PROMPTS = [ + _build_prompt( + fact_block=( + "- Festival city: Oslo\n- Mascot animal: otter\n- Welcome drink: tea" + ), + question="Which city hosts the festival, and what animal is the mascot?", + ), + _build_prompt( + fact_block=( + "- Meeting day: Tuesday\n" + "- Planned snack: apricot cake\n" + "- Backup room: Cedar" + ), + question="What day is the meeting, and what snack is planned?", + ), +] +RECORDED_RESPONSE_TEXTS = ( + " The city hosting the festival is Oslo, and the mascot is an otter.", + " The meeting is on Tuesday and the snack planned is apricot cake.", +) +REQUIRED_WORDS = (("oslo", "otter"), ("tuesday", "apricot")) + + +def _log_prompt_summaries() -> None: + for i, prompt in enumerate(E2E_PROMPTS): + prompt_lines = prompt.splitlines() + fact_block = [line for line in prompt_lines if line.startswith("- ")] + fact_summary = "; ".join(line.removeprefix("- ") for line in fact_block) + _log(f"prompt {i} facts: {fact_summary}") + + +@lru_cache(maxsize=1) +def _get_model_path() -> str: + try: + path = snapshot_download(repo_id=MODEL_NAME, local_files_only=True) + _log(f"using cached model snapshot: {path}") + return path + except Exception: + path = snapshot_download(repo_id=MODEL_NAME) + _log(f"downloaded model snapshot: {path}") + return path + + +def _get_hidden_size(model_config) -> int: + hidden_size = getattr(model_config, "hidden_size", None) + if hidden_size is None and hasattr(model_config, "text_config"): + hidden_size = getattr(model_config.text_config, "hidden_size", None) + assert isinstance(hidden_size, int) + return hidden_size + + +def _check_tp_allreduce_uses_quick_reduce( + self, + num_tokens: int, + dtype_name: str = "float16", +) -> dict[str, int | bool]: + from vllm.distributed.communication_op import tensor_model_parallel_all_reduce + from vllm.distributed.parallel_state import get_tp_group + + assert self.device is not None + qr_comm = get_tp_group().device_communicator.qr_comm + assert qr_comm is not None + assert not qr_comm.disabled + + hidden_size = _get_hidden_size(self.model_runner.model.config) + dtype = getattr(torch, dtype_name) + sample = torch.full( + (num_tokens, hidden_size), + fill_value=float(self.rank + 1), + dtype=dtype, + device=self.device, + ) + + assert qr_comm.should_quick_allreduce(sample) + + expected = sample.clone() + reduced = tensor_model_parallel_all_reduce(sample) + dist.all_reduce(expected, group=get_tp_group().device_group) + torch.testing.assert_close(reduced, expected, atol=2.5, rtol=0.1) + + stats = { + "rank": self.rank, + "hidden_size": hidden_size, + "num_tokens": num_tokens, + "use_fp16_kernels": qr_comm.use_fp16_kernels, + } + _log( + "worker quick-reduce check: " + f"rank={self.rank} hidden_size={hidden_size} " + f"num_tokens={num_tokens} use_fp16_kernels={qr_comm.use_fp16_kernels}" + ) + return stats + + +def _check_quick_reduce_disabled(self) -> int: + from vllm.distributed.parallel_state import get_tp_group + + qr_comm = get_tp_group().device_communicator.qr_comm + assert qr_comm is not None + assert qr_comm.disabled + _log(f"worker confirmed quick reduce is disabled: rank={self.rank}") + return self.rank + + +def _collect_generations(outputs) -> list[tuple[tuple[int, ...], str]]: + return [ + (tuple(output.outputs[0].token_ids), output.outputs[0].text) + for output in outputs + ] + + +def _shutdown_llm(llm: LLM | None) -> None: + if llm is None: + cleanup_dist_env_and_memory() + return + + with contextlib.suppress(Exception): + llm.llm_engine.engine_core.shutdown() + + del llm + cleanup_dist_env_and_memory() + + +def _log_generations( + label: str, + generations: list[tuple[tuple[int, ...], str]], +) -> None: + for i, (token_ids, text) in enumerate(generations): + _log(f"{label} prompt {i} token ids: {list(token_ids)}") + _log(f"{label} prompt {i} text: {text!r}") + + +def _assert_required_words( + label: str, + generations: list[tuple[tuple[int, ...], str]], +) -> None: + for i, (_, text) in enumerate(generations): + lowered = text.lower() + missing = [word for word in REQUIRED_WORDS[i] if word not in lowered] + assert not missing, ( + f"{label} prompt {i} is missing required words {missing}. " + f"Observed text: {text!r}" + ) + + +def _collect_soft_mismatches( + baseline_generations: list[tuple[tuple[int, ...], str]], + quick_reduce_generations: list[tuple[tuple[int, ...], str]], +) -> list[str]: + mismatches = [] + + for i, (_, text) in enumerate(baseline_generations): + expected = RECORDED_RESPONSE_TEXTS[i] + if text != expected: + mismatches.append( + f"baseline prompt {i} drifted from the recorded response.\n" + f"expected={expected!r}\nactual={text!r}" + ) + + for i, (_, text) in enumerate(quick_reduce_generations): + expected = RECORDED_RESPONSE_TEXTS[i] + if text != expected: + mismatches.append( + f"quick-reduce prompt {i} drifted from the recorded response.\n" + f"expected={expected!r}\nactual={text!r}" + ) + + for i, ((_, baseline_text), (_, quick_reduce_text)) in enumerate( + zip(baseline_generations, quick_reduce_generations) + ): + if baseline_text != quick_reduce_text: + mismatches.append( + f"baseline and quick-reduce responses differ for prompt {i}.\n" + f"baseline={baseline_text!r}\nquick_reduce={quick_reduce_text!r}" + ) + + return mismatches + + +def _run_generation( + *, + backend: Literal["mp", "ray"], + quant_mode: str, + expect_quick_reduce: bool, +) -> list[tuple[tuple[int, ...], str]]: + llm = None + monkeypatch = pytest.MonkeyPatch() + with monkeypatch.context() as m: + m.setenv("VLLM_ALLOW_INSECURE_SERIALIZATION", "1") + m.setenv("VLLM_ROCM_QUICK_REDUCE_QUANTIZATION", quant_mode) + model_path = _get_model_path() + _log( + f"starting generation: backend={backend} quant={quant_mode} " + f"gpu_memory_utilization={E2E_GPU_MEMORY_UTILIZATION} " + f"kv_cache_bytes={E2E_KV_CACHE_MEMORY_BYTES} model={model_path}" + ) + + try: + llm = LLM( + model=model_path, + tokenizer=model_path, + tensor_parallel_size=2, + distributed_executor_backend=backend, + dtype="half", + enforce_eager=True, + max_model_len=E2E_MAX_MODEL_LEN, + max_num_seqs=len(E2E_PROMPTS), + gpu_memory_utilization=E2E_GPU_MEMORY_UTILIZATION, + kv_cache_memory_bytes=E2E_KV_CACHE_MEMORY_BYTES, + seed=0, + ) + + if not expect_quick_reduce: + assert llm.collective_rpc(_check_quick_reduce_disabled) == [0, 1] + + if expect_quick_reduce: + worker_stats = llm.collective_rpc( + _check_tp_allreduce_uses_quick_reduce, + args=(E2E_PREFILL_TOKENS,), + ) + assert [stat["rank"] for stat in worker_stats] == [0, 1] + worker_summary = "; ".join( + "rank={rank} hidden_size={hidden_size} num_tokens={num_tokens} " + "use_fp16_kernels={use_fp16_kernels}".format(**stat) + for stat in worker_stats + ) + _log(f"{backend} quick-reduce worker checks: {worker_summary}") + + outputs = llm.generate( + E2E_PROMPTS, + SamplingParams( + temperature=0.0, + max_tokens=20, + stop=["\nAnswer:", " Answer:"], + ), + use_tqdm=False, + ) + generations = _collect_generations(outputs) + assert all(text.strip() for _, text in generations) + _log_generations(f"{backend} {quant_mode}", generations) + return generations + finally: + _shutdown_llm(llm) + + +def _run_quick_reduce_llm_e2e_in_subprocess( + *, + backend: Literal["mp", "ray"], +) -> str | None: + _log(f"running LLM e2e: backend={backend}") + _log_prompt_summaries() + baseline_outputs = _run_generation( + backend=backend, + quant_mode="NONE", + expect_quick_reduce=False, + ) + quick_reduce_outputs = _run_generation( + backend=backend, + quant_mode="FP", + expect_quick_reduce=True, + ) + + _assert_required_words("baseline", baseline_outputs) + _assert_required_words("quick-reduce", quick_reduce_outputs) + + mismatches = _collect_soft_mismatches(baseline_outputs, quick_reduce_outputs) + if mismatches: + details = "\n\n".join(mismatches) + _log(f"soft response mismatch:\n{details}") + return details + + _log(f"LLM e2e backend={backend} matched the recorded responses exactly") + return None + + +def _quick_reduce_llm_e2e_worker( + result_queue: mp.Queue, + backend: Literal["mp", "ray"], +) -> None: + try: + xfail_reason = _run_quick_reduce_llm_e2e_in_subprocess(backend=backend) + except Exception: + result_queue.put({"status": "error", "reason": traceback.format_exc()}) + raise + else: + if xfail_reason is not None: + result_queue.put({"status": "xfail", "reason": xfail_reason}) + else: + result_queue.put({"status": "ok"}) + + +def run_quick_reduce_llm_e2e( + *, + backend: Literal["mp", "ray"], +) -> None: + ctx = mp.get_context("spawn") + result_queue = ctx.Queue() + proc = ctx.Process( + target=_quick_reduce_llm_e2e_worker, + args=(result_queue, backend), + ) + proc.start() + proc.join(timeout=600) + + try: + result = result_queue.get(timeout=5) + except queue.Empty as exc: + if proc.exitcode != 0: + raise AssertionError( + f"quick-reduce llm e2e subprocess failed for backend={backend} " + f"with exit code {proc.exitcode} and produced no result" + ) from exc + raise AssertionError( + f"quick-reduce llm e2e subprocess produced no result for backend={backend}" + ) from exc + + if result["status"] == "xfail": + pytest.xfail(result["reason"]) + if result["status"] == "error": + raise AssertionError( + f"quick-reduce llm e2e subprocess failed for backend={backend}:\n" + f"{result['reason']}" + ) + + assert proc.exitcode == 0, ( + f"quick-reduce llm e2e subprocess failed for backend={backend} " + f"with exit code {proc.exitcode}" + ) + + +def test_quick_reduce_regime_values(): + from vllm.distributed.device_communicators.quick_all_reduce import QuickReduceRegime + + assert QuickReduceRegime.FP.value == 0 + assert QuickReduceRegime.INT8.value == 1 + assert QuickReduceRegime.INT6.value == 2 + assert QuickReduceRegime.INT4.value == 3 + assert QuickReduceRegime.NONE.value == 4 + + +def test_quick_reduce_regime_names(): + from vllm.distributed.device_communicators.quick_all_reduce import QuickReduceRegime + + assert set(QuickReduceRegime.__members__) == {"FP", "INT8", "INT6", "INT4", "NONE"} + + +@pytest.mark.parametrize("quant_level", QUANT_LEVELS + ["NONE"]) +def test_quick_reduce_quantization_env_var(monkeypatch, quant_level): + monkeypatch.setenv("VLLM_ROCM_QUICK_REDUCE_QUANTIZATION", quant_level) + + reloaded_envs = _reload_envs() + assert quant_level == reloaded_envs.VLLM_ROCM_QUICK_REDUCE_QUANTIZATION + + +def test_quick_reduce_quantization_default(monkeypatch): + monkeypatch.delenv("VLLM_ROCM_QUICK_REDUCE_QUANTIZATION", raising=False) + + reloaded_envs = _reload_envs() + assert reloaded_envs.VLLM_ROCM_QUICK_REDUCE_QUANTIZATION == "NONE" + + +@pytest.mark.parametrize("cast_bf16", [True, False]) +def test_quick_reduce_cast_bf16_to_fp16_env_var(monkeypatch, cast_bf16): + monkeypatch.setenv( + "VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16", "1" if cast_bf16 else "0" + ) + + reloaded_envs = _reload_envs() + assert reloaded_envs.VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16 is cast_bf16 + + +def test_quick_reduce_cast_bf16_to_fp16_default(monkeypatch): + monkeypatch.delenv("VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16", raising=False) + + reloaded_envs = _reload_envs() + assert reloaded_envs.VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16 is True + + +@pytest.mark.parametrize("max_mb", [128, 512, 2048, None]) +def test_quick_reduce_max_size_env_var(monkeypatch, max_mb): + if max_mb is None: + monkeypatch.delenv("VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB", raising=False) + else: + monkeypatch.setenv("VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB", str(max_mb)) + + reloaded_envs = _reload_envs() + assert max_mb == reloaded_envs.VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB + + +def test_quick_reduce_max_size_default(monkeypatch): + monkeypatch.delenv("VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB", raising=False) + + reloaded_envs = _reload_envs() + assert reloaded_envs.VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB is None + + +@pytest.mark.parametrize( + ("gcn_arch_name", "expected"), + [ + ("gfx942", True), + ("gfx950", True), + ("gfx90a", False), + ("", False), + ], +) +def test_quick_allreduce_rocm_arch_available(gcn_arch_name, expected): + from vllm.distributed.device_communicators.quick_all_reduce import QuickAllReduce + + qar = QuickAllReduce.__new__(QuickAllReduce) + qar.disabled = True + + with ( + patch( + "vllm.distributed.device_communicators.quick_all_reduce.current_platform." + "is_rocm", + return_value=True, + ), + patch( + "torch.cuda.get_device_properties", + return_value=SimpleNamespace(gcnArchName=gcn_arch_name), + ), + ): + assert qar._rocm_arch_available() is expected + + +def test_quick_allreduce_rocm_arch_available_handles_probe_failure(): + from vllm.distributed.device_communicators.quick_all_reduce import QuickAllReduce + + qar = QuickAllReduce.__new__(QuickAllReduce) + qar.disabled = True + + with ( + patch( + "vllm.distributed.device_communicators.quick_all_reduce.current_platform." + "is_rocm", + return_value=True, + ), + patch("torch.cuda.get_device_properties", side_effect=RuntimeError), + ): + assert qar._rocm_arch_available() is False + + +def test_quick_allreduce_rejects_disabled(): + qar = _make_quick_allreduce(disabled=True) + + inp = torch.zeros(1024, dtype=torch.float16) + assert qar.should_quick_allreduce(inp) is False + + +def test_quick_allreduce_rejects_unsupported_dtype(): + qar = _make_quick_allreduce() + + inp = torch.zeros(1024 * 1024, dtype=torch.float32) + assert qar.should_quick_allreduce(inp) is False + + +def test_quick_allreduce_rejects_non_aligned_input(): + qar = _make_quick_allreduce() + + inp = torch.zeros(5, dtype=torch.float16) + assert qar.should_quick_allreduce(inp) is False + + +def test_quick_allreduce_rejects_non_contiguous_input(): + qar = _make_quick_allreduce() + + inp = torch.zeros((1024, 1024), dtype=torch.float16)[:, ::2] + assert qar.should_quick_allreduce(inp) is False + + +def test_quick_allreduce_rejects_input_smaller_than_threshold(): + qar = _make_quick_allreduce() + + inp = torch.zeros((MB // 2) - 8, dtype=torch.float16) + assert qar.should_quick_allreduce(inp) is False + + +def test_quick_allreduce_accepts_input_at_threshold(): + qar = _make_quick_allreduce() + + inp = torch.zeros(MB // 2, dtype=torch.float16) + assert qar.should_quick_allreduce(inp) is True + + +def test_quick_allreduce_rejects_input_larger_than_max_size(): + qar = _make_quick_allreduce(qr_max_size=1 * MB) + + inp = torch.zeros(MB, dtype=torch.float16) + assert qar.should_quick_allreduce(inp) is False + + +def test_quick_allreduce_bf16_uses_fp16_threshold_when_cast_enabled(): + inp = torch.zeros(MB // 2, dtype=torch.bfloat16) + + without_cast = _make_quick_allreduce(use_fp16_kernels=False) + with_cast = _make_quick_allreduce(use_fp16_kernels=True) + + assert without_cast.should_quick_allreduce(inp) is False + assert with_cast.should_quick_allreduce(inp) is True + + +def test_quick_allreduce_supported_world_sizes(): + from vllm.distributed.device_communicators.quick_all_reduce import QuickAllReduce + + assert QuickAllReduce._SUPPORTED_WORLD_SIZES == [2, 4, 8] + + +def test_quick_allreduce_supported_dtypes(): + from vllm.distributed.device_communicators.quick_all_reduce import QuickAllReduce + + assert [torch.float16, torch.bfloat16] == QuickAllReduce._SUPPORTED_DTYPES + + +def test_quick_allreduce_min_size_table(): + from vllm.distributed.device_communicators.quick_all_reduce import QuickAllReduce + + for dtype in [torch.float16, torch.bfloat16]: + for world_size in QuickAllReduce._SUPPORTED_WORLD_SIZES: + min_sizes = QuickAllReduce._QR_MIN_SIZE[(dtype, world_size)] + assert len(min_sizes) == 4 + assert all(size > 0 for size in min_sizes) + + +def test_qr_max_size(): + from vllm import _custom_ops as ops + + max_size = ops.qr_max_size() + assert isinstance(max_size, int) + assert max_size > 0 + + +@pytest.mark.skipif( + current_platform.device_count() < WORLD_SIZE, + reason="requires 2 ROCm GPUs", +) +@pytest.mark.parametrize("quant_level", QUANT_LEVELS) +def test_quick_allreduce_two_gpu_correctness(quant_level): + _log(f"two-GPU correctness case: quant={quant_level}") + _run_two_gpu_quick_allreduce_test( + quant_level=quant_level, + dtype_name="float16", + cast_bf16=False, + ) + + +@pytest.mark.skipif( + current_platform.device_count() < WORLD_SIZE, + reason="requires 2 ROCm GPUs", +) +def test_quick_allreduce_bf16_cast_mode(): + _log("BF16 cast case") + _run_two_gpu_quick_allreduce_test( + quant_level="FP", + dtype_name="bfloat16", + cast_bf16=True, + ) + + +@pytest.mark.skipif( + current_platform.device_count() < WORLD_SIZE, + reason="requires 2 ROCm GPUs", +) +def test_quick_allreduce_llm_e2e(): + _log("LLM e2e case: backend=mp") + run_quick_reduce_llm_e2e(backend="mp") + + +@pytest.mark.skipif( + current_platform.device_count() < WORLD_SIZE, + reason="requires 2 ROCm GPUs", +) +def test_quick_allreduce_llm_e2e_ray(): + _log("LLM e2e case: backend=ray") + run_quick_reduce_llm_e2e(backend="ray") diff --git a/asian_ai/performance/vLLM/tests/distributed/test_same_node.py b/asian_ai/performance/vLLM/tests/distributed/test_same_node.py new file mode 100644 index 0000000000000000000000000000000000000000..1370da842b4e57b34019edb2d45c865713059a37 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_same_node.py @@ -0,0 +1,49 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import os + +import torch +import torch.distributed as dist + +from vllm.distributed.parallel_state import in_the_same_node_as +from vllm.distributed.utils import StatelessProcessGroup +from vllm.utils.network_utils import get_ip, get_open_port + + +def _run_test(pg): + test_result = all(in_the_same_node_as(pg, source_rank=0)) + + expected = os.environ.get("VLLM_TEST_SAME_HOST", "1") == "1" + assert test_result == expected, f"Expected {expected}, got {test_result}" + if pg == dist.group.WORLD: + print("Same node test passed! when using torch distributed!") + else: + print("Same node test passed! when using StatelessProcessGroup!") + + +if __name__ == "__main__": + dist.init_process_group(backend="gloo") + + rank = dist.get_rank() + if rank == 0: + port = get_open_port() + ip = get_ip() + dist.broadcast_object_list([ip, port], src=0) + else: + recv = [None, None] + dist.broadcast_object_list(recv, src=0) + ip, port = recv + + stateless_pg = StatelessProcessGroup.create(ip, port, rank, dist.get_world_size()) + + for pg in [dist.group.WORLD, stateless_pg]: + if os.environ.get("VLLM_TEST_WITH_DEFAULT_DEVICE_SET", "0") == "1": + default_devices = ["cpu"] + if torch.cuda.is_available(): + default_devices.append("cuda") + for device in default_devices: + torch.set_default_device(device) + _run_test(pg) + else: + _run_test(pg) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_shm_broadcast.py b/asian_ai/performance/vLLM/tests/distributed/test_shm_broadcast.py new file mode 100644 index 0000000000000000000000000000000000000000..93100b3a480e009680de7eed15b7a627778c6386 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_shm_broadcast.py @@ -0,0 +1,394 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import random +import threading +import time +from unittest import mock + +import multiprocess as mp +import numpy as np +import pytest +import torch.distributed as dist + +from vllm.distributed.device_communicators.shm_broadcast import MessageQueue +from vllm.distributed.utils import StatelessProcessGroup +from vllm.utils.network_utils import get_open_port +from vllm.utils.system_utils import update_environment_variables + + +def get_arrays(n: int, seed: int = 0) -> list[np.ndarray]: + np.random.seed(seed) + sizes = np.random.randint(1, 10_000, n) + # on average, each array will have 5k elements + # with int64, each array will have 40kb + return [np.random.randint(1, 100, i) for i in sizes] + + +def distributed_run(fn, world_size, timeout=60): + """Run a function in multiple processes with proper error handling. + + Args: + fn: Function to run in each process + world_size: Number of processes to spawn + timeout: Maximum time in seconds to wait for processes (default: 60) + """ + number_of_processes = world_size + processes = [] + for i in range(number_of_processes): + env = {} + env["RANK"] = str(i) + env["LOCAL_RANK"] = str(i) + env["WORLD_SIZE"] = str(number_of_processes) + env["LOCAL_WORLD_SIZE"] = str(number_of_processes) + env["MASTER_ADDR"] = "localhost" + env["MASTER_PORT"] = "12345" + p = mp.Process(target=fn, args=(env,)) + processes.append(p) + p.start() + + # Monitor processes and fail fast if any process fails + start_time = time.time() + failed_processes = [] + + # Wait for all processes, checking for failures + while time.time() - start_time < timeout: + all_done = True + for i, p in enumerate(processes): + if p.is_alive(): + all_done = False + elif p.exitcode != 0: + # Process failed + failed_processes.append((i, p.exitcode)) + break + + if failed_processes or all_done: + break + time.sleep(0.1) # Check every 100ms + + # Check for timeout if no failures detected yet + for i, p in enumerate(processes): + if p.is_alive(): + p.kill() + p.join() + + # Report failures + if failed_processes: + error_msg = "Distributed test failed:\n" + for rank, status in failed_processes: + error_msg += f" Rank {rank}: Exit code {status}\n" + raise AssertionError(error_msg) + + +def worker_fn_wrapper(fn): + # `mp.Process` cannot accept environment variables directly + # so we need to pass the environment variables as arguments + # and update the environment variables in the function + def wrapped_fn(env): + update_environment_variables(env) + dist.init_process_group(backend="gloo") + fn() + + return wrapped_fn + + +@worker_fn_wrapper +def worker_fn(): + rank = dist.get_rank() + if rank == 0: + port = get_open_port() + ip = "127.0.0.1" + dist.broadcast_object_list([ip, port], src=0) + else: + recv = [None, None] + dist.broadcast_object_list(recv, src=0) + ip, port = recv # type: ignore + + stateless_pg = StatelessProcessGroup.create(ip, port, rank, dist.get_world_size()) + + for pg in [dist.group.WORLD, stateless_pg]: + writer_rank = 2 + broadcaster = MessageQueue.create_from_process_group( + pg, 40 * 1024, 2, writer_rank + ) + if rank == writer_rank: + seed = random.randint(0, 1000) + dist.broadcast_object_list([seed], writer_rank) + else: + recv = [None] + dist.broadcast_object_list(recv, writer_rank) + seed = recv[0] # type: ignore + + if pg == dist.group.WORLD: + dist.barrier() + else: + pg.barrier() + + # in case we find a race condition + # print the seed so that we can reproduce the error + print(f"Rank {rank} got seed {seed}") + # test broadcasting with about 400MB of data + N = 10_000 + if rank == writer_rank: + arrs = get_arrays(N, seed) + for x in arrs: + broadcaster.broadcast_object(x) + time.sleep(random.random() / 1000) + else: + arrs = get_arrays(N, seed) + for x in arrs: + y = broadcaster.broadcast_object(None) + assert np.array_equal(x, y) + time.sleep(random.random() / 1000) + + if pg == dist.group.WORLD: + dist.barrier() + print(f"torch distributed passed the test! Rank {rank}") + else: + pg.barrier() + print(f"StatelessProcessGroup passed the test! Rank {rank}") + + +def test_shm_broadcast(): + distributed_run(worker_fn, 4) + + +@worker_fn_wrapper +def worker_fn_test_shutdown_busy(): + rank = dist.get_rank() + writer_rank = 2 + message_queue = MessageQueue.create_from_process_group( + dist.group.WORLD, 40 * 1024, 2, writer_rank + ) + + if not message_queue._is_writer: + # Put into busy mode + message_queue._spin_condition.busy_loop_s = 9999 + + shutdown_event = threading.Event() + + def shutdown_thread(mq, shutdown_event): + shutdown_event.wait() + mq.shutdown() + + threading.Thread( + target=shutdown_thread, args=(message_queue, shutdown_event) + ).start() + + with pytest.raises(TimeoutError): + message_queue.dequeue(timeout=0.01) + + shutdown_event.set() + + with pytest.raises(RuntimeError, match="cancelled"): + message_queue.dequeue(timeout=1) + + assert message_queue.shutting_down + + print(f"torch distributed passed the test! Rank {rank}") + dist.barrier() + + +def test_message_queue_shutdown_busy(caplog_vllm): + distributed_run(worker_fn_test_shutdown_busy, 4) + print(caplog_vllm.text) + + +@worker_fn_wrapper +def worker_fn_test_shutdown_idle(): + rank = dist.get_rank() + writer_rank = 2 + message_queue = MessageQueue.create_from_process_group( + dist.group.WORLD, 40 * 1024, 2, writer_rank + ) + + if not message_queue._is_writer: + # Put into idle mode + message_queue._spin_condition.last_read = 0 + + shutdown_event = threading.Event() + + def shutdown_thread(mq, shutdown_event): + shutdown_event.wait() + mq.shutdown() + + threading.Thread( + target=shutdown_thread, args=(message_queue, shutdown_event) + ).start() + + with pytest.raises(TimeoutError): + message_queue.dequeue(timeout=0.01) + + shutdown_event.set() + + with pytest.raises(RuntimeError, match="cancelled"): + message_queue.dequeue(timeout=1) + + assert message_queue.shutting_down + + print(f"torch distributed passed the test! Rank {rank}") + dist.barrier() + + +def test_message_queue_shutdown_idle(): + distributed_run(worker_fn_test_shutdown_idle, 4) + + +@worker_fn_wrapper +def worker_fn_test_idle_to_busy(): + rank = dist.get_rank() + writer_rank = 2 + message_queue = MessageQueue.create_from_process_group( + dist.group.WORLD, 40 * 1024, 2, writer_rank + ) + + message1 = "hello world" + message2 = np.random.randint(1, 100, 100) + with mock.patch.object( + message_queue._spin_condition, "wait", wraps=message_queue._spin_condition.wait + ) as wrapped_wait: + if not message_queue._is_writer: + # Put into idle mode + message_queue._spin_condition.last_read = 0 + + # no messages, so expect a TimeoutError + with pytest.raises(TimeoutError): + message_queue.dequeue(timeout=0.01) + # wait should only be called once while idle + assert wrapped_wait.call_count == 1 + + # sync with the writer and wait for message1 + dist.barrier() + recv_message = message_queue.dequeue(timeout=5) + assert recv_message == message1 + # second call to wait, with a message read, this puts in a busy spin + assert wrapped_wait.call_count == 2 + + # sync with the writer and wait for message2 + dist.barrier() + recv_message = message_queue.dequeue(timeout=1) + assert np.array_equal(recv_message, message2) + # in busy mode, we expect wait to have been called multiple times + assert wrapped_wait.call_count > 3 + else: + # writer writes two messages in sync with the reader + dist.barrier() + # sleep delays the send to ensure reader enters the read loop + time.sleep(0.1) + message_queue.enqueue(message1) + + dist.barrier() + time.sleep(0.1) + message_queue.enqueue(message2) + + message_queue.shutdown() + assert message_queue.shutting_down + print(f"torch distributed passed the test! Rank {rank}") + + +def test_message_queue_idle_wake(): + distributed_run(worker_fn_test_idle_to_busy, 4) + + +@worker_fn_wrapper +def worker_fn_test_busy_to_idle(): + rank = dist.get_rank() + writer_rank = 2 + message_queue = MessageQueue.create_from_process_group( + dist.group.WORLD, 40 * 1024, 2, writer_rank + ) + + message1 = 12345 + message2 = list(range(3)) + with mock.patch.object( + message_queue._spin_condition, "wait", wraps=message_queue._spin_condition.wait + ) as wrapped_wait: + if not message_queue._is_writer: + # Put into busy mode + message_queue._spin_condition.busy_loop_s = 9999 + + # sync with the writer and wait for message1 + dist.barrier() + recv_message = message_queue.dequeue(timeout=1) + assert recv_message == message1 + # in busy mode, we expect wait to have been called many times + assert wrapped_wait.call_count > 1 + + # simulate busy loop ending + message_queue._spin_condition.busy_loop_s = 0 + # ensure we enter idle mode, then record call count + with pytest.raises(TimeoutError): + message_queue.dequeue(timeout=0.01) + call_count = wrapped_wait.call_count + + # sync with the writer and wait for message2 + dist.barrier() + recv_message = message_queue.dequeue(timeout=1) + assert recv_message == message2 + + # call to wait after idle should only happen once + assert wrapped_wait.call_count == call_count + 1 + else: + # writer writes two messages in sync with the reader + dist.barrier() + # sleep delays the send to ensure reader enters the read loop + time.sleep(0.1) + message_queue.enqueue(message1) + + dist.barrier() + time.sleep(0.1) + message_queue.enqueue(message2) + + message_queue.shutdown() + assert message_queue.shutting_down + print(f"torch distributed passed the test! Rank {rank}") + + +def test_message_queue_busy_to_idle(): + distributed_run(worker_fn_test_busy_to_idle, 4) + + +def test_warning_logs(caplog_vllm): + """ + Test that warning logs are emitted at VLLM_RINGBUFFER_WARNING_INTERVAL intervals + when indefinite=False, and are not emitted when indefinite=True. + """ + + # Patch the warning log interval to every 1 ms during reads + with mock.patch( + "vllm.distributed.device_communicators.shm_broadcast.VLLM_RINGBUFFER_WARNING_INTERVAL", + new=0.001, # 1 ms + ): + writer = MessageQueue( + n_reader=1, + n_local_reader=1, + max_chunk_bytes=1024 * 1024, # 1MB chunks + max_chunks=10, + ) + reader = MessageQueue.create_from_handle(writer.export_handle(), rank=0) + writer.wait_until_ready() + reader.wait_until_ready() + + # We should have at least one warning log here + # "0 seconds" expected due to rounding of 1ms test interval + with pytest.raises(TimeoutError): + reader.dequeue(timeout=0.01, indefinite=False) + assert any( + "No available shared memory broadcast block found in 0 seconds" + in record.message + for record in caplog_vllm.records + ) + caplog_vllm.clear() + + # We should have no warnings this time + with pytest.raises(TimeoutError): + reader.dequeue(timeout=0.01, indefinite=True) + assert all( + "No available shared memory broadcast block found in 0 seconds" + not in record.message + for record in caplog_vllm.records + ) + + # Clean up when done + writer.shutdown() + reader.shutdown() diff --git a/asian_ai/performance/vLLM/tests/distributed/test_shm_buffer.py b/asian_ai/performance/vLLM/tests/distributed/test_shm_buffer.py new file mode 100644 index 0000000000000000000000000000000000000000..17fabe213599d143bb3e36ef9aff4c5c1f601224 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_shm_buffer.py @@ -0,0 +1,243 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import traceback +import unittest + +import numpy as np + +from vllm.distributed.device_communicators.shm_object_storage import ( + SingleWriterShmRingBuffer, +) + + +class TestSingleWriterShmRingBuffer(unittest.TestCase): + """Test suite for the ring buffer implementation""" + + def setUp(self): + """Set up test fixtures""" + self.buffer_size = 4096 + self.ring_buffer = None + + def tearDown(self): + """Clean up after tests""" + if self.ring_buffer: + self.ring_buffer.close() + + def test_buffer_opening(self): + """Test opening an existing buffer""" + # First create a buffer + self.ring_buffer = SingleWriterShmRingBuffer( + data_buffer_size=self.buffer_size, create=True + ) + + # Then open it with another instance + reader_buffer = SingleWriterShmRingBuffer(*self.ring_buffer.handle()) + self.assertFalse(reader_buffer.is_writer) + self.assertEqual( + reader_buffer.shared_memory.name, self.ring_buffer.shared_memory.name + ) + + def test_buffer_access(self): + """Test accessing allocated buffers""" + self.ring_buffer = SingleWriterShmRingBuffer( + data_buffer_size=self.buffer_size, create=True + ) + + size = 100 + address, monotonic_id = self.ring_buffer.allocate_buf(size) + + # Write some test data + test_data = b"Hello, World!" * 7 # 91 bytes + with self.ring_buffer.access_buf(address) as (data_buf, metadata): + data_buf[0 : len(test_data)] = test_data + + # Read it back + with self.ring_buffer.access_buf(address) as (data_buf2, metadata2): + read_data = bytes(data_buf2[0 : len(test_data)]) + read_id = metadata2[0] + + self.assertEqual(read_data, test_data) + self.assertEqual(read_id, monotonic_id) + + def test_memory_error_on_full_buffer(self): + """Test that MemoryError is raised when buffer is full""" + small_buffer_size = 200 + self.ring_buffer = SingleWriterShmRingBuffer( + data_buffer_size=small_buffer_size, create=True + ) + + # Fill up the buffer + self.ring_buffer.allocate_buf(100) + self.ring_buffer.allocate_buf(80) # Total: 196 bytes used + + # This should fail + with self.assertRaises(MemoryError): + self.ring_buffer.allocate_buf(1) # Would exceed buffer capacity + + def test_allocation_and_free(self): + """Test allocation and freeing of buffers""" + small_buffer_size = 200 + self.ring_buffer = SingleWriterShmRingBuffer( + data_buffer_size=small_buffer_size, create=True + ) + + size = 80 + # Write some data + test_data = b"Repeated test data" + for i in range(5): + address, monotonic_id = self.ring_buffer.allocate_buf(size) + with self.ring_buffer.access_buf(address) as (data_buf, metadata): + data_buf[0:4] = (0).to_bytes(4, "little") # 0 for not in-use + data_buf[4 : len(test_data) + 4] = test_data + print(self.ring_buffer.metadata) + freed_ids = self.ring_buffer.free_buf(lambda *args: True) + print(f" Freed IDs: {freed_ids}") + self.assertEqual(freed_ids[0], i) + + def test_clear_buffer(self): + """Test clearing the buffer""" + self.ring_buffer = SingleWriterShmRingBuffer( + data_buffer_size=self.buffer_size, create=True + ) + + # Allocate some buffers + for _ in range(3): + self.ring_buffer.allocate_buf(100) + + # Clear the buffer + self.ring_buffer.clear() + + # Check that metadata is empty and IDs reset + self.assertEqual(len(self.ring_buffer.metadata), 0) + self.assertEqual(self.ring_buffer.monotonic_id_start, 0) + self.assertEqual(self.ring_buffer.monotonic_id_end, 0) + self.assertEqual(self.ring_buffer.data_buffer_start, 0) + self.assertEqual(self.ring_buffer.data_buffer_end, 0) + + def test_allocation_cycles(self): + buffer_size = 100 + ring = SingleWriterShmRingBuffer(data_buffer_size=buffer_size, create=True) + + # tracking allocations for assertions + allocated_bitmap = np.zeros( + (buffer_size,), dtype=np.bool_ + ) # addr -> is_allocated + allocation_map = dict() # monotonic_id -> (addr, size) + + def count_allocated(bitmap) -> int: + return np.sum(bitmap).item() + + def is_free_fn(a, b) -> bool: + return True + + def mark_allocated_with_assertion(id, addr, size): + addr = addr % buffer_size + self.assertEqual(count_allocated(allocated_bitmap[addr : addr + size]), 0) + + allocated_bitmap[addr : addr + size] = True + allocation_map[id] = (addr, size) + + def mark_freed_with_assertion(id): + self.assertTrue(id in allocation_map) + + addr, size = allocation_map.pop(id) + addr = addr % buffer_size + self.assertEqual( + count_allocated(allocated_bitmap[addr : addr + size]), size + ) + + allocated_bitmap[addr : addr + size] = False + + def ring_free(free_size=None): + freed_ids = ring.free_buf(is_free_fn, free_size) + for freed_id in freed_ids: + mark_freed_with_assertion(freed_id) + + def ring_allocate(allocate_size): + allocate_size_with_md = allocate_size + ring.MD_SIZE + try: + addr, monotonic_id = ring.allocate_buf(allocate_size) + mark_allocated_with_assertion(monotonic_id, addr, allocate_size_with_md) + except MemoryError: + # free 2x size for enough space if wrapping happened + ring_free(allocate_size_with_md * 2) + + # retry allocating + addr, monotonic_id = ring.allocate_buf(allocate_size) + mark_allocated_with_assertion(monotonic_id, addr, allocate_size_with_md) + + # 1. allocation & free cycles + for _ in range(33): + # will consume 2 + 8 = 10 bytes per allocation + ring_allocate(2) + + # 2. free all allocations + ring_free() + + # 3. try allocate the largest possible buffer + ring_allocate(buffer_size - ring.MD_SIZE) + + +def main(): + """Main function demonstrating usage and running tests""" + print("=== SingleWriterShmRingBuffer Test Suite ===\n") + + # Run unit tests + print("Running unit tests...") + unittest.main(argv=[""], exit=False, verbosity=2) + + print("\n" + "=" * 50) + print("=== Manual Demo ===\n") + + # Manual demonstration + try: + print("Creating ring buffer...") + writer_buffer = SingleWriterShmRingBuffer(data_buffer_size=2048, create=True) + reader_buffer = SingleWriterShmRingBuffer(*writer_buffer.handle()) + + print(f"Buffer created with name: {writer_buffer.shared_memory.name}") + + # Allocate some buffers + print("\nAllocating buffers...") + address_array = [] + for i in range(3): + size = 100 + i * 50 + try: + writer_buffer.free_buf(lambda *args: True) + address, monotonic_id = writer_buffer.allocate_buf(size) + address_array.append((address, size, monotonic_id)) + + # Write some test data + with writer_buffer.access_buf(address) as (data_buf, metadata): + test_message = f"Test message {i}".encode() + data_buf[0 : len(test_message)] = test_message + + except MemoryError as e: + print(f" Failed to allocate {size} bytes: {e}") + + print("\nBuffer state:") + print(f" Data buffer start: {writer_buffer.data_buffer_start}") + print(f" Data buffer end: {writer_buffer.data_buffer_end}") + print(f" Monotonic ID start: {writer_buffer.monotonic_id_start}") + print(f" Monotonic ID end: {writer_buffer.monotonic_id_end}") + print(f" Metadata entries: {len(writer_buffer.metadata)}") + + # Try to read back the data + print("\nReading back data...") + for address, size, monotonic_id in address_array: + with reader_buffer.access_buf(address) as (data_buf, metadata): + # Find null terminator or read first 50 chars + data_bytes = bytes(data_buf[0:size]) + message = data_bytes.decode() + print(f" ID {monotonic_id}: '{message}'") + + except Exception as e: + print(f"Demo error: {e}") + traceback.print_exc() + + print("\n=== Demo Complete ===") + + +if __name__ == "__main__": + main() diff --git a/asian_ai/performance/vLLM/tests/distributed/test_shm_storage.py b/asian_ai/performance/vLLM/tests/distributed/test_shm_storage.py new file mode 100644 index 0000000000000000000000000000000000000000..e66225243196fca98543bb227193f9b66ce93898 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_shm_storage.py @@ -0,0 +1,325 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import multiprocessing +import random +import time +import traceback +import unittest +from multiprocessing import Lock + +import torch + +# Assuming these are imported from your module +from vllm.distributed.device_communicators.shm_object_storage import ( + MsgpackSerde, + SingleWriterShmObjectStorage, + SingleWriterShmRingBuffer, +) +from vllm.multimodal.inputs import ( + MultiModalFieldElem, + MultiModalKwargsItem, + MultiModalSharedField, +) + + +def _dummy_elem(size: int): + return MultiModalFieldElem( + data=torch.empty((size,), dtype=torch.int8), + field=MultiModalSharedField(batch_size=1), + ) + + +def _dummy_item(size_by_key: dict[str, int]): + return MultiModalKwargsItem( + {key: _dummy_elem(size) for key, size in size_by_key.items()} + ) + + +class TestSingleWriterShmObjectStorage(unittest.TestCase): + def setUp(self): + """Set up test fixtures before each test method.""" + ring_buffer = SingleWriterShmRingBuffer( + data_buffer_size=1024 * 100, + create=True, # 10 MB buffer + ) + self.storage = SingleWriterShmObjectStorage( + max_object_size=1024 * 10, # 10KB max object + n_readers=2, + ring_buffer=ring_buffer, + serde_class=MsgpackSerde, + reader_lock=Lock(), + ) + + def tearDown(self): + """Clean up after each test.""" + if self.storage: + self.storage.close() + + def test_minimal_put_get_cycle(self): + """Test basic put and get operations.""" + key = "test_key" + value = _dummy_item({"field1": 10, "field2": 20}) + + # Put operation + address, monotonic_id = self.storage.put(key, value) + + # Verify key is in index + self.assertIn(key, self.storage.key_index) + self.assertEqual(self.storage.key_index[key], (address, monotonic_id)) + self.assertEqual(self.storage.id_index[monotonic_id], key) + + # Get operation + result = self.storage.get(address, monotonic_id) + + # Verify result + self.assertEqual(result, value) + + def test_put_same_key_twice(self): + """Test behavior when putting the same key multiple times.""" + key = "duplicate_key" + value1 = "first value" + value2 = "second value" + + # First put + address1, id1 = self.storage.put(key, value1) + retrieved1 = self.storage.get(address1, id1) + self.assertEqual(retrieved1, value1) + + # should raise an error on second put + with self.assertRaises(ValueError) as context: + self.storage.put(key, value2) + + self.assertIn("already exists in the storage", str(context.exception)) + + def test_large_object_rejection(self): + """Test that objects exceeding max_object_size are rejected.""" + # Create an object larger than max_object_size + large_data = "x" * (self.storage.max_object_size + 100) + + with self.assertRaises(ValueError) as context: + self.storage.put("large_key", large_data) + + self.assertIn("exceeds max object size", str(context.exception)) + + def test_buffer_overflow_and_cleanup(self): + """Test behavior when buffer fills up and needs cleanup.""" + # Fill up the buffer with many small objects + stored_items = [] + + try: + for i in range(1000): # Try to store many items + key = f"item_{i}" + value = f"data_{i}" * 100 # Make it reasonably sized + address, monotonic_id = self.storage.put(key, value) + stored_items.append((key, value, address, monotonic_id)) + except MemoryError: + print(f"Buffer filled after {len(stored_items)} items") + + # Verify that some items are still accessible + accessible_count = 0 + for key, original_value, address, monotonic_id in stored_items: + for i in range(self.storage.n_readers): + retrieved = self.storage.get(address, monotonic_id) + if retrieved == original_value: + accessible_count += 1 + + self.assertEqual(accessible_count, len(stored_items)) + + try: + for i in range(len(stored_items), 1000): # Try to store many items + key = f"item_{i}" + value = f"data_{i}" * 100 # Make it reasonably sized + address, monotonic_id = self.storage.put(key, value) + stored_items.append((key, value, address, monotonic_id)) + except MemoryError: + print(f"Buffer filled after {len(stored_items)} items") + + # Verify that some items are still accessibles + for key, original_value, address, monotonic_id in stored_items: + try: + for i in range(self.storage.n_readers): + retrieved = self.storage.get(address, monotonic_id) + if retrieved == original_value: + accessible_count += 1 + except ValueError as e: + print(f"Error retrieving {key}: {e}") + + # some items from the first batch may still be accessible + self.assertGreaterEqual(accessible_count, len(stored_items)) + + def test_blocking_unread_object(self): + """Test behavior when buffer fills up and needs cleanup.""" + # Fill up the buffer with many small objects + stored_items = [] + + try: + for i in range(1000): # Try to store many items + key = f"item_{i}" + value = f"data_{i}" * 100 # Make it reasonably sized + address, monotonic_id = self.storage.put(key, value) + stored_items.append((key, value, address, monotonic_id)) + except MemoryError: + print(f"Buffer filled after {len(stored_items)} items") + + # read all items except the first one + # to simulate a blocking situation + accessible_count = 0 + for key, original_value, address, monotonic_id in stored_items[1:]: + for i in range(self.storage.n_readers): + retrieved = self.storage.get(address, monotonic_id) + if retrieved == original_value: + accessible_count += 1 + + self.assertEqual(accessible_count, len(stored_items) - 1) + + try: + key = f"item_{len(stored_items)}" + value = f"data_{len(stored_items)}" * 100 + address, monotonic_id = self.storage.put(key, value) + except MemoryError: + print(f"Buffer filled after {len(stored_items)} items") + + # read the first item + for i in range(self.storage.n_readers): + key, original_value, address, monotonic_id = stored_items[0] + retrieved = self.storage.get(address, monotonic_id) + self.assertEqual(retrieved, original_value) + + try: + for i in range(len(stored_items), 1000): # Try to store many items + key = f"item_{i}" + value = f"data_{i}" * 100 # Make it reasonably sized + address, monotonic_id = self.storage.put(key, value) + stored_items.append((key, value, address, monotonic_id)) + except MemoryError: + print(f"Buffer filled after {len(stored_items)} items") + + # some items from the first batch may still be accessible + self.assertGreaterEqual(len(stored_items), accessible_count + 10) + + def test_invalid_get_operations(self): + """Test various invalid get operations.""" + # Test with non-existent address + with self.assertRaises(ValueError): # Could be various exceptions + self.storage.get(99999, 1) + + # Store something first + address, monotonic_id = self.storage.put("test", "value") + + # Test with wrong monotonic_id + with self.assertRaises(ValueError) as context: + self.storage.get(address, monotonic_id + 100) + + self.assertIn("has been modified or is invalid", str(context.exception)) + + def test_clear_storage(self): + """Test clearing the storage.""" + # Store some items + for i in range(5): + self.storage.put(f"item_{i}", f"value_{i}") + + # Clear the storage + self.storage.clear() + + # Verify that all indices are empty + self.assertEqual(len(self.storage.key_index), 0) + self.assertEqual(len(self.storage.id_index), 0) + self.assertEqual(len(self.storage.ring_buffer.metadata), 0) + + # Verify that new items can be added after clearing + address, monotonic_id = self.storage.put("new_item", "new_value") + self.assertIn("new_item", self.storage.key_index) + self.assertEqual((address, monotonic_id), (0, 0)) + + +# Reader process function +def reader_process(process_id, storage_handle, items_to_read): + """Reader process that connects to existing shared memory and reads data.""" + reader_storage = SingleWriterShmObjectStorage.create_from_handle(storage_handle) + + print(f"Reader {process_id} started") + + errors = [] + + for key, original_value, address, monotonic_id in items_to_read: + time.sleep(random.random() / 100) + try: + # Read data from shared memory + retrieved_value = reader_storage.get(address, monotonic_id) + + # Verify data integrity + assert retrieved_value == original_value + print(f"Reader {process_id} retrieved {key}: {retrieved_value}") + except Exception as e: + errors.append((key, str(e), type(e).__name__)) + + +def run_multiprocess_example(): + """Run a minimal working example with real shared memory.""" + print("=== Minimal Object Storage Example ===") + + try: + # Create storage instance + ring_buffer = SingleWriterShmRingBuffer( + data_buffer_size=1024 * 100, + create=True, # 10 MB buffer + ) + storage = SingleWriterShmObjectStorage( + max_object_size=1024, + n_readers=3, + ring_buffer=ring_buffer, + serde_class=MsgpackSerde, + reader_lock=Lock(), + ) + + print(f"Created storage (writer: {storage.is_writer})") + + # Test basic data types + test_data = [ + ("user_data", {"name": "Alice", "age": 30, "scores": [95, 87, 92]}), + ("simple_string", "Hello, World!"), + ("number", 42), + ("list_data", [1, 2, 3, "four", 5.0]), + ] + + stored_items = [] + + # Store all data + for key, value in test_data: + print(f"Storing {key}: {value}") + address, monotonic_id = storage.put(key, value) + stored_items.append((key, value, address, monotonic_id)) + print(f" -> Stored at address {address}, ID {monotonic_id}") + + print("\n--- Retrieving Data ---") + processes = [] + handle = storage.handle() + # initialize lock for reader processes + handle.reader_lock = Lock() + for i in range(storage.n_readers): + p = multiprocessing.Process( + target=reader_process, args=(i, handle, stored_items) + ) + processes.append(p) + p.start() + + for p in processes: + p.join(timeout=10) + if p.is_alive(): + p.terminate() + p.join() + + except Exception as e: + print(f"Error in minimal example: {e}") + traceback.print_exc() + + +if __name__ == "__main__": + # Run the minimal example first + run_multiprocess_example() + print("\n" + "=" * 50 + "\n") + + # Run the test suite + print("Running comprehensive test suite...") + unittest.main(verbosity=2, exit=False) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_split_group.py b/asian_ai/performance/vLLM/tests/distributed/test_split_group.py new file mode 100644 index 0000000000000000000000000000000000000000..f7c91d633a04f1c1a50dc99a73fe75ada2299aaf --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_split_group.py @@ -0,0 +1,233 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for split_group in GroupCoordinator. + +These tests verify that: +1. split_group is used for both device and CPU group creation. +2. Multiple subgroups work correctly with split_group. +3. Both GPU and CPU all-reduce work on split groups. +""" + +import os +from typing import Any + +import multiprocess as mp +import pytest +import torch +import torch.distributed + +import vllm.envs as envs +from vllm.distributed.parallel_state import ( + GroupCoordinator, + init_distributed_environment, +) +from vllm.utils.system_utils import update_environment_variables + +# The whole module exercises the split_group code path, which is opt-in +# behind VLLM_DISTRIBUTED_USE_SPLIT_GROUP=1. +pytestmark = pytest.mark.skipif( + not envs.VLLM_DISTRIBUTED_USE_SPLIT_GROUP, + reason=("VLLM_DISTRIBUTED_USE_SPLIT_GROUP=1 not set; split_group path is opt-in."), +) + +mp.set_start_method("spawn", force=True) + + +def distributed_run(fn, world_size): + number_of_processes = world_size + processes: list[mp.Process] = [] + for i in range(number_of_processes): + env: dict[str, str] = {} + env["RANK"] = str(i) + env["LOCAL_RANK"] = str(i) + env["WORLD_SIZE"] = str(number_of_processes) + env["LOCAL_WORLD_SIZE"] = str(number_of_processes) + env["MASTER_ADDR"] = "localhost" + env["MASTER_PORT"] = "12346" + # propagate the opt-in flag to the spawned child workers + env["VLLM_DISTRIBUTED_USE_SPLIT_GROUP"] = "1" + p = mp.Process(target=fn, args=(env,)) + processes.append(p) + p.start() + + for p in processes: + p.join() + + for p in processes: + assert p.exitcode == 0 + + +def worker_fn_wrapper(fn): + def wrapped_fn(env): + update_environment_variables(env) + local_rank = os.environ["LOCAL_RANK"] + device = torch.device(f"cuda:{local_rank}") + torch.accelerator.set_device_index(device) + init_distributed_environment() + fn() + + return wrapped_fn + + +def _verify_device_group(coordinator: GroupCoordinator): + """Verify device group works via all-reduce.""" + local_rank = torch.distributed.get_rank() + device = torch.device(f"cuda:{local_rank}") + tensor = torch.ones(16, 16, dtype=torch.float32, device=device) + torch.distributed.all_reduce(tensor, group=coordinator.device_group) + torch.accelerator.synchronize() + expected = coordinator.world_size + assert torch.all(tensor == expected).cpu().item(), ( + f"Device group all-reduce failed: expected {expected}, " + f"got {tensor.flatten()[0].item()}" + ) + + +def _verify_cpu_group(coordinator: GroupCoordinator): + """Verify CPU group works via all-reduce.""" + tensor = torch.ones(16, dtype=torch.float32) + torch.distributed.all_reduce(tensor, group=coordinator.cpu_group) + expected = coordinator.world_size + assert torch.all(tensor == expected).cpu().item(), ( + f"CPU group all-reduce failed: expected {expected}, " + f"got {tensor.flatten()[0].item()}" + ) + + +# --------------------------------------------------------------------------- +# Test 1: Basic split_group path with 2 GPUs +# --------------------------------------------------------------------------- +@worker_fn_wrapper +def split_group_basic_worker(): + rank = torch.distributed.get_rank() + world_size = torch.distributed.get_world_size() + group_ranks = [list(range(world_size))] + + coordinator = GroupCoordinator( + group_ranks=group_ranks, + local_rank=rank, + torch_distributed_backend="nccl", + use_device_communicator=False, + group_name="test_split_basic", + ) + + _verify_device_group(coordinator) + _verify_cpu_group(coordinator) + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 2, + reason="Need at least 2 GPUs to run the test.", +) +def test_split_group_basic(): + """Test basic GroupCoordinator creation with split_group.""" + distributed_run(split_group_basic_worker, 2) + + +# --------------------------------------------------------------------------- +# Test 2: Multiple subgroups with split_group (4 GPUs) +# --------------------------------------------------------------------------- +@worker_fn_wrapper +def split_group_multiple_subgroups_worker(): + rank = torch.distributed.get_rank() + group_ranks = [[0, 1], [2, 3]] + + coordinator = GroupCoordinator( + group_ranks=group_ranks, + local_rank=rank, + torch_distributed_backend="nccl", + use_device_communicator=False, + group_name="test_split_multi", + ) + + assert coordinator.world_size == 2 + + _verify_device_group(coordinator) + _verify_cpu_group(coordinator) + + if rank in [0, 1]: + assert coordinator.ranks == [0, 1] + else: + assert coordinator.ranks == [2, 3] + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 4, + reason="Need at least 4 GPUs to run the test.", +) +def test_split_group_multiple_subgroups(): + """Test GroupCoordinator with multiple independent subgroups.""" + distributed_run(split_group_multiple_subgroups_worker, 4) + + +# --------------------------------------------------------------------------- +# Test 3: split_group contract — every parent rank must enter with the same +# ``split_ranks``. NCCL happens to produce +# correct subgroups for disjoint partitions because the wrapper hashes +# ``my_group`` to derive the comm-split color, but the contract violation is +# real and would break under non-partition / non-NCCL backends. This test +# captures the actual ``split_ranks`` argument passed on every rank and +# asserts they match. +# --------------------------------------------------------------------------- +@worker_fn_wrapper +def split_group_contract_worker(): + rank = torch.distributed.get_rank() + group_ranks = [[0, 1], [2, 3]] + + captured: list[list[list[int]]] = [] + original_split_group = torch.distributed.split_group + + def capturing_split_group(*args, split_ranks=None, **kwargs): + captured.append([list(g) for g in split_ranks]) + return original_split_group(*args, split_ranks=split_ranks, **kwargs) + + torch.distributed.split_group = capturing_split_group + try: + GroupCoordinator( + group_ranks=group_ranks, + local_rank=rank, + torch_distributed_backend="nccl", + use_device_communicator=False, + group_name="test_split_contract", + ) + finally: + torch.distributed.split_group = original_split_group + + # GroupCoordinator builds two subgroups (device + cpu) per coordinator, + # so every rank must have made exactly two split_group calls. + if len(captured) != 2: + raise AssertionError( + f"rank {rank} expected 2 split_group calls (device + cpu), " + f"got {len(captured)}: {captured}" + ) + + world_size = torch.distributed.get_world_size() + for call_idx in range(2): + gathered: list[Any] = [None] * world_size + torch.distributed.all_gather_object(gathered, captured[call_idx]) + # Normalize for stable comparison (sort each subgroup and the outer list). + norm = [ + sorted([sorted(sg) for sg in per_rank_args]) for per_rank_args in gathered + ] + reference = norm[0] + for r, args in enumerate(norm): + if args != reference: + raise AssertionError( + f"split_group contract violation on call #{call_idx}: " + f"rank {r} passed split_ranks={gathered[r]}, but rank 0 " + f"passed split_ranks={gathered[0]}. PyTorch requires every " + "parent rank to enter split_group with the same split_ranks." + ) + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 4, + reason="Need at least 4 GPUs to run the test.", +) +def test_split_group_contract_same_split_ranks_on_all_ranks(): + """All parent ranks must call torch.distributed.split_group with the same + ``split_ranks`` argument. This catches the bug where each rank passed + only its own subgroup (``split_ranks=[ranks]``), which NCCL forgives for + disjoint partitions but is a documented contract violation. + """ + distributed_run(split_group_contract_worker, 4) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_symm_mem_allreduce.py b/asian_ai/performance/vLLM/tests/distributed/test_symm_mem_allreduce.py new file mode 100644 index 0000000000000000000000000000000000000000..fd51ba28b1b155055e845ea3d1ca13186719a7b2 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_symm_mem_allreduce.py @@ -0,0 +1,140 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import queue +import random +import typing + +import pytest +import torch +import torch.distributed as dist +import torch.multiprocessing as mp + +import vllm.envs as envs +from vllm.config import ParallelConfig, VllmConfig, set_current_vllm_config +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.distributed.communication_op import tensor_model_parallel_all_reduce +from vllm.distributed.device_communicators.cuda_communicator import CudaCommunicator +from vllm.distributed.parallel_state import ( + get_tp_group, + init_distributed_environment, + initialize_model_parallel, +) +from vllm.engine.arg_utils import EngineArgs +from vllm.engine.llm_engine import LLMEngine +from vllm.platforms import current_platform +from vllm.utils.system_utils import update_environment_variables + +torch.manual_seed(42) +random.seed(44) + +test_size_elements = 1024 * 1024 + + +def symm_mem_allreduce_worker(local_rank: int, world_size: int, q: mp.Queue): + monkeypatch = pytest.MonkeyPatch() + config = VllmConfig(parallel_config=ParallelConfig(tensor_parallel_size=world_size)) + + with monkeypatch.context() as m, set_current_vllm_config(config): + m.delenv("CUDA_VISIBLE_DEVICES", raising=False) + dtype = torch.bfloat16 + device = torch.device(f"cuda:{local_rank}") + torch.accelerator.set_device_index(device) + torch.set_default_device(device) + torch.set_default_dtype(dtype) + update_environment_variables( + { + "RANK": str(local_rank), + "LOCAL_RANK": str(local_rank), + "WORLD_SIZE": str(world_size), + "MASTER_ADDR": "localhost", + "MASTER_PORT": "12345", + } + ) + + init_distributed_environment() + initialize_model_parallel(tensor_model_parallel_size=world_size) + + cuda_communicator = typing.cast( + CudaCommunicator, get_tp_group().device_communicator + ) + symm_mem_comm = cuda_communicator.symm_mem_comm + if symm_mem_comm is None or symm_mem_comm.disabled: + # can't use skip under multiprocessing + q.put("SymmMemCommunicator is not available or disabled.") + return + + inp_direct_symm_mem = torch.randint( + 1, 23, (test_size_elements,), dtype=dtype, device=device + ) + if not symm_mem_comm.should_use_symm_mem(inp_direct_symm_mem): + # can't use skip under multiprocessing + q.put("SymmMemCommunicator isn't used for this world and input size.") + return + + original_inp_direct_symm_mem = inp_direct_symm_mem.clone() + out_direct_symm_mem = symm_mem_comm.all_reduce(inp_direct_symm_mem) + assert out_direct_symm_mem is not None + + group = get_tp_group().device_group + dist.all_reduce(original_inp_direct_symm_mem, group=group) + torch.testing.assert_close( + out_direct_symm_mem, original_inp_direct_symm_mem, atol=2.5, rtol=0.1 + ) + + # Test tensor_model_parallel_all_reduce which should use symm_mem + inp_tensor_parallel = torch.randint( + -23, 1, (test_size_elements,), dtype=dtype, device=device + ) + original_inp_tensor_parallel = inp_tensor_parallel.clone() + out_tensor_parallel = tensor_model_parallel_all_reduce(inp_tensor_parallel) + dist.all_reduce(original_inp_tensor_parallel, group=group) + torch.testing.assert_close( + out_tensor_parallel, original_inp_tensor_parallel, atol=2.5, rtol=0.1 + ) + + +@pytest.mark.skipif( + not current_platform.is_cuda(), + reason="SymmMemAllreduce is only available for CUDA platforms.", +) +@pytest.mark.parametrize("tp_size", [2]) +@pytest.mark.parametrize("pipeline_parallel_size", [1]) +@pytest.mark.skipif(envs.VLLM_TARGET_DEVICE not in ["cuda"], reason="Only test on CUDA") +def test_symm_mem_allreduce( + monkeypatch: pytest.MonkeyPatch, tp_size, pipeline_parallel_size +): + world_size = tp_size * pipeline_parallel_size + if world_size > torch.accelerator.device_count(): + pytest.skip("Not enough GPUs to run the test.") + q = mp.get_context("spawn").Queue() + mp.spawn(symm_mem_allreduce_worker, args=(world_size, q), nprocs=world_size) + try: + val = q.get(timeout=1) + except queue.Empty: + val = None + finally: + cleanup_dist_env_and_memory() + if val is not None: + pytest.skip(val) + + +@pytest.mark.skipif( + not current_platform.is_cuda(), + reason="SymmMemAllreduce is only available for CUDA platforms.", +) +@pytest.mark.skipif(envs.VLLM_TARGET_DEVICE not in ["cuda"], reason="Only test on CUDA") +def test_dp_with_symm_mem_allreduce(monkeypatch: pytest.MonkeyPatch): + world_size = 4 + if world_size > torch.accelerator.device_count(): + pytest.skip("Not enough GPUs to run the test.") + # Verify that the DataParallel runs without error + engine_args = EngineArgs( + model="distilbert/distilgpt2", + enforce_eager=True, + enable_prefix_caching=True, + data_parallel_size=2, + tensor_parallel_size=2, + data_parallel_backend="mp", + ) + LLMEngine.from_engine_args(engine_args) diff --git a/asian_ai/performance/vLLM/tests/distributed/test_torchrun_example.py b/asian_ai/performance/vLLM/tests/distributed/test_torchrun_example.py new file mode 100644 index 0000000000000000000000000000000000000000..5859307e439ec31f15becf22da4275a7a8ab995a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_torchrun_example.py @@ -0,0 +1,82 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +# unit test for `examples/features/torchrun/torchrun_example_offline.py` +import os +import random + +import torch +import torch.distributed as dist + +import vllm.envs as envs +from vllm import LLM, SamplingParams +from vllm.distributed.parallel_state import get_world_group + +# By default, let PyTorch choose the WORLD backend for the current device +# type (legacy lazy-init path). When VLLM_DISTRIBUTED_USE_SPLIT_GROUP=1, +# use the explicit eager-init pattern required by `split_group` (mixed +# cpu:gloo,cuda:nccl backend + device_id binding). +if envs.VLLM_DISTRIBUTED_USE_SPLIT_GROUP: + local_rank = int(os.environ["LOCAL_RANK"]) + torch.accelerator.set_device_index(local_rank) + dist.init_process_group( + backend="cpu:gloo,cuda:nccl", + device_id=torch.device(f"cuda:{local_rank}"), + ) +else: + dist.init_process_group() + +# Create prompts +prompts = [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + "The future of AI is", +] + +sampling_params = SamplingParams(temperature=0.8, top_p=0.95) + +# set different `gpu_memory_utilization` for different ranks, +# to test if all ranks agree on the same kv cache configuration. +llm = LLM( + model="facebook/opt-125m", + tensor_parallel_size=2, + pipeline_parallel_size=int(os.getenv("PP_SIZE", 1)), + distributed_executor_backend="external_launcher", + gpu_memory_utilization=random.uniform(0.8, 0.92), + seed=0, +) + +outputs = llm.generate(prompts, sampling_params) + +cpu_group = get_world_group().cpu_group + +torch_rank = dist.get_rank(group=cpu_group) + + +def test_consistent_across_ranks(obj): + if torch_rank == 0: + dist.broadcast_object_list([obj], src=0, group=cpu_group) + else: + container = [None] + dist.broadcast_object_list(container, src=0, group=cpu_group) + assert container[0] == obj + + +test_consistent_across_ranks(llm.llm_engine.vllm_config.cache_config.num_cpu_blocks) +test_consistent_across_ranks(llm.llm_engine.vllm_config.cache_config.num_gpu_blocks) + +# make sure we can access the model parameters from the calling process +# of the `LLM` instance. +params = list( + llm.llm_engine.model_executor.driver_worker.worker.model_runner.model.parameters() +) +test_consistent_across_ranks(len(params)) + +# all ranks should have the same outputs +for output in outputs: + prompt = output.prompt + generated_text = output.outputs[0].text + test_consistent_across_ranks(prompt) + test_consistent_across_ranks(generated_text) + print(f"Rank {torch_rank}, Prompt: {prompt!r}, Generated text: {generated_text!r}") diff --git a/asian_ai/performance/vLLM/tests/distributed/test_torchrun_example_moe.py b/asian_ai/performance/vLLM/tests/distributed/test_torchrun_example_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..6983f4de7a02b6ad1d33ac293e4cbc57da9c9438 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_torchrun_example_moe.py @@ -0,0 +1,91 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +# unit test for `examples/features/torchrun/torchrun_example_offline.py` +import os +import random + +import torch +import torch.distributed as dist + +import vllm.envs as envs +from vllm import LLM, SamplingParams +from vllm.distributed.parallel_state import get_tp_group, get_world_group + +# By default, let PyTorch choose the WORLD backend for the current device +# type (legacy lazy-init path). When VLLM_DISTRIBUTED_USE_SPLIT_GROUP=1, +# use the explicit eager-init pattern required by `split_group` (mixed +# cpu:gloo,cuda:nccl backend + device_id binding). +if envs.VLLM_DISTRIBUTED_USE_SPLIT_GROUP: + local_rank = int(os.environ["LOCAL_RANK"]) + torch.accelerator.set_device_index(local_rank) + dist.init_process_group( + backend="cpu:gloo,cuda:nccl", + device_id=torch.device(f"cuda:{local_rank}"), + ) +else: + dist.init_process_group() + +# Create prompts +prompts = [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + "The future of AI is", +] * 10 +dp_size = int(os.getenv("DP_SIZE", "1")) +dp_rank = int(os.getenv("DP_RANK", "0")) + +if dp_size > 1: + # distribute the prompts across the data parallel ranks + prompts = [prompt for idx, prompt in enumerate(prompts) if idx % dp_size == dp_rank] + +sampling_params = SamplingParams(temperature=0.8, top_p=0.95) + +# set different `gpu_memory_utilization` for different ranks, +# to test if all ranks agree on the same kv cache configuration. +llm = LLM( + model="microsoft/Phi-mini-MoE-instruct", + tensor_parallel_size=int(os.getenv("TP_SIZE", "1")), + pipeline_parallel_size=int(os.getenv("PP_SIZE", "1")), + enable_expert_parallel=int(os.getenv("ENABLE_EP", "0")) == 1, + distributed_executor_backend="external_launcher", + gpu_memory_utilization=random.uniform(0.8, 0.92), + seed=0, + max_model_len=1024, + max_num_seqs=16, +) + +outputs = llm.generate(prompts, sampling_params) + +group = get_world_group() if dp_size == 1 else get_tp_group() +cpu_group = group.cpu_group +group_rank = dist.get_rank(group=cpu_group) + + +def test_consistent_across_ranks(obj): + if group_rank == 0: + dist.broadcast_object_list([obj], src=group.ranks[0], group=cpu_group) + else: + container = [None] + dist.broadcast_object_list(container, src=group.ranks[0], group=cpu_group) + assert container[0] == obj + + +test_consistent_across_ranks(llm.llm_engine.vllm_config.cache_config.num_cpu_blocks) +test_consistent_across_ranks(llm.llm_engine.vllm_config.cache_config.num_gpu_blocks) + +# make sure we can access the model parameters from the calling process +# of the `LLM` instance. +params = list( + llm.llm_engine.model_executor.driver_worker.worker.model_runner.model.parameters() +) +test_consistent_across_ranks(len(params)) + +# all ranks should have the same outputs +for output in outputs: + prompt = output.prompt + generated_text = output.outputs[0].text + test_consistent_across_ranks(prompt) + test_consistent_across_ranks(generated_text) + print(f"Rank {group_rank}, Prompt: {prompt!r}, Generated text: {generated_text!r}") diff --git a/asian_ai/performance/vLLM/tests/distributed/test_utils.py b/asian_ai/performance/vLLM/tests/distributed/test_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..3047fb338e5a6ab0c3023a99fae20433b6b817fc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_utils.py @@ -0,0 +1,141 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import socket + +import pytest +import ray +import torch + +import vllm.envs as envs +from vllm.distributed.device_communicators.pynccl import PyNcclCommunicator +from vllm.distributed.utils import StatelessProcessGroup +from vllm.platforms import current_platform +from vllm.utils.network_utils import get_open_port +from vllm.utils.system_utils import update_environment_variables + +from ..utils import multi_gpu_test + + +@ray.remote +class _CUDADeviceCountStatelessTestActor: + def get_count(self): + return current_platform.device_count() + + def set_cuda_visible_devices(self, cuda_visible_devices: str): + update_environment_variables({"CUDA_VISIBLE_DEVICES": cuda_visible_devices}) + + def get_cuda_visible_devices(self): + return envs.CUDA_VISIBLE_DEVICES + + +def test_cuda_device_count_stateless(): + """Test that cuda_device_count_stateless changes return value if + CUDA_VISIBLE_DEVICES is changed.""" + if current_platform.is_rocm(): + pytest.skip("Skip for ROCm because Ray uses HIP_VISIBLE_DEVICES.") + actor = _CUDADeviceCountStatelessTestActor.options( # type: ignore + num_gpus=2 + ).remote() + assert len(sorted(ray.get(actor.get_cuda_visible_devices.remote()).split(","))) == 2 + assert ray.get(actor.get_count.remote()) == 2 + ray.get(actor.set_cuda_visible_devices.remote("0")) + assert ray.get(actor.get_count.remote()) == 1 + ray.get(actor.set_cuda_visible_devices.remote("")) + assert ray.get(actor.get_count.remote()) == 0 + + +def cpu_worker(rank, WORLD_SIZE, port1, port2): + pg1 = StatelessProcessGroup.create( + host="127.0.0.1", port=port1, rank=rank, world_size=WORLD_SIZE + ) + if rank <= 2: + pg2 = StatelessProcessGroup.create( + host="127.0.0.1", port=port2, rank=rank, world_size=3 + ) + data = torch.tensor([rank]) + data = pg1.broadcast_obj(data, src=2) + assert data.item() == 2 + if rank <= 2: + data = torch.tensor([rank + 1]) + data = pg2.broadcast_obj(data, src=2) + assert data.item() == 3 + pg2.barrier() + pg1.barrier() + + +def gpu_worker(rank, WORLD_SIZE, port1, port2): + torch.accelerator.set_device_index(rank) + pg1 = StatelessProcessGroup.create( + host="127.0.0.1", port=port1, rank=rank, world_size=WORLD_SIZE + ) + pynccl1 = PyNcclCommunicator(pg1, device=rank) + if rank <= 2: + pg2 = StatelessProcessGroup.create( + host="127.0.0.1", port=port2, rank=rank, world_size=3 + ) + pynccl2 = PyNcclCommunicator(pg2, device=rank) + data = torch.tensor([rank]).cuda() + pynccl1.all_reduce(data) + pg1.barrier() + torch.accelerator.synchronize() + if rank <= 2: + pynccl2.all_reduce(data) + pg2.barrier() + torch.accelerator.synchronize() + item = data[0].item() + print(f"rank: {rank}, item: {item}") + if rank == 3: + assert item == 6 + else: + assert item == 18 + + +def broadcast_worker(rank, WORLD_SIZE, port1, port2): + pg1 = StatelessProcessGroup.create( + host="127.0.0.1", port=port1, rank=rank, world_size=WORLD_SIZE + ) + if rank == 2: + pg1.broadcast_obj("secret", src=2) + else: + obj = pg1.broadcast_obj(None, src=2) + assert obj == "secret" + pg1.barrier() + + +def allgather_worker(rank, WORLD_SIZE, port1, port2): + pg1 = StatelessProcessGroup.create( + host="127.0.0.1", port=port1, rank=rank, world_size=WORLD_SIZE + ) + data = pg1.all_gather_obj(rank) + assert data == list(range(WORLD_SIZE)) + pg1.barrier() + + +@pytest.mark.skip(reason="This test is flaky and prone to hang.") +@multi_gpu_test(num_gpus=4) +@pytest.mark.parametrize( + "worker", [cpu_worker, gpu_worker, broadcast_worker, allgather_worker] +) +def test_stateless_process_group(worker): + port1 = get_open_port() + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.bind(("", port1)) + port2 = get_open_port() + WORLD_SIZE = 4 + from multiprocessing import get_context + + ctx = get_context("fork") + processes = [] + for i in range(WORLD_SIZE): + rank = i + processes.append( + ctx.Process(target=worker, args=(rank, WORLD_SIZE, port1, port2)) + ) + for p in processes: + p.start() + for p in processes: + p.join() + for p in processes: + assert not p.exitcode + print("All processes finished.") diff --git a/asian_ai/performance/vLLM/tests/distributed/test_weight_transfer.py b/asian_ai/performance/vLLM/tests/distributed/test_weight_transfer.py new file mode 100644 index 0000000000000000000000000000000000000000..83bd5b92b4408a23f6aa2a1f9479a1d876048db3 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/distributed/test_weight_transfer.py @@ -0,0 +1,1192 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for weight transfer engine backends. + +Unit tests for engine classes (parsing, validation, registry). +Integration tests for NCCL and IPC weight transfer between processes using Ray. +""" + +import pickle +from unittest.mock import MagicMock + +import pybase64 as base64 +import pytest +import ray +import torch +from torch.multiprocessing.reductions import reduce_tensor + +from vllm.config.parallel import ParallelConfig +from vllm.config.weight_transfer import WeightTransferConfig +from vllm.distributed.weight_transfer import WeightTransferEngineFactory +from vllm.distributed.weight_transfer.ipc_engine import ( + IPCWeightTransferEngine, + IPCWeightTransferInitInfo, + IPCWeightTransferUpdateInfo, +) +from vllm.distributed.weight_transfer.nccl_engine import ( + NCCLWeightTransferEngine, + NCCLWeightTransferInitInfo, + NCCLWeightTransferUpdateInfo, +) +from vllm.distributed.weight_transfer.sparse_nccl_engine import ( + SparseNCCLWeightTransferEngine, + SparseNCCLWeightTransferUpdateInfo, + SparseWeightPatch, +) +from vllm.platforms import current_platform +from vllm.utils.network_utils import get_open_port + + +def _init_ray_for_weight_transfer() -> None: + if ray.is_initialized(): + return + ray.init( + ignore_reinit_error=True, + runtime_env={ + "env_vars": { + "RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES": "1", + "RAY_EXPERIMENTAL_NOSET_HIP_VISIBLE_DEVICES": "1", + "RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES": "1", + } + }, + ) + + +def _get_ray_assigned_device() -> torch.device: + gpu_ids = ray.get_gpu_ids() + if not gpu_ids: + return torch.device("cuda:0") + return torch.device(f"cuda:{int(gpu_ids[0])}") + + +def _set_ray_assigned_device() -> torch.device: + device = _get_ray_assigned_device() + current_platform.set_device(device) + return device + + +def create_mock_parallel_config( + rank: int = 0, + world_size: int = 1, + dp_rank: int = 0, +) -> ParallelConfig: + """Create a mock ParallelConfig for testing.""" + config = MagicMock(spec=ParallelConfig) + config.rank = rank + config.world_size = world_size + config.data_parallel_rank = dp_rank + config.data_parallel_index = dp_rank + return config + + +def create_mock_vllm_config( + rank: int = 0, + world_size: int = 1, + dp_rank: int = 0, +) -> MagicMock: + """Create a mock VllmConfig exposing parallel_config and model_config.""" + vllm_config = MagicMock() + vllm_config.parallel_config = create_mock_parallel_config(rank, world_size, dp_rank) + vllm_config.model_config = MagicMock() + return vllm_config + + +# --- Unit Tests: NCCLWeightTransferUpdateInfo Validation --- + + +class TestNCCLWeightTransferUpdateInfoValidation: + """Test NCCLWeightTransferUpdateInfo dataclass validation.""" + + def test_valid_update_info(self): + info = NCCLWeightTransferUpdateInfo( + names=["layer.weight", "layer.bias"], + dtype_names=["float32", "float32"], + shapes=[[10, 10], [10]], + ) + assert info.names == ["layer.weight", "layer.bias"] + assert info.dtype_names == ["float32", "float32"] + assert info.shapes == [[10, 10], [10]] + + def test_mismatched_dtype_names_raises(self): + with pytest.raises(ValueError, match="dtype_names"): + NCCLWeightTransferUpdateInfo( + names=["layer.weight", "layer.bias"], + dtype_names=["float32"], # Only one dtype + shapes=[[10, 10], [10]], + ) + + def test_mismatched_shapes_raises(self): + with pytest.raises(ValueError, match="shapes"): + NCCLWeightTransferUpdateInfo( + names=["layer.weight", "layer.bias"], + dtype_names=["float32", "float32"], + shapes=[[10, 10]], # Only one shape + ) + + def test_empty_lists_valid(self): + info = NCCLWeightTransferUpdateInfo(names=[], dtype_names=[], shapes=[]) + assert len(info.names) == 0 + + +# --- Unit Tests: SparseNCCLWeightTransferUpdateInfo Validation --- + + +class TestSparseNCCLWeightTransferUpdateInfoValidation: + """Test SparseNCCLWeightTransferUpdateInfo dataclass validation.""" + + def test_valid_sparse_update_info(self): + info = SparseNCCLWeightTransferUpdateInfo( + names=["layer.weight", "layer.bias"], + dtype_names=["float32", "bfloat16"], + shapes=[[10, 10], [10]], + num_updates_list=[4, 2], + ) + assert info.num_updates_list == [4, 2] + + def test_mismatched_dtype_names_raises(self): + with pytest.raises(ValueError, match="dtype_names"): + SparseNCCLWeightTransferUpdateInfo( + names=["layer.weight", "layer.bias"], + dtype_names=["float32"], + shapes=[[10, 10], [10]], + num_updates_list=[4, 2], + ) + + def test_rejects_empty_num_updates_list(self): + with pytest.raises(ValueError, match="cannot be empty"): + SparseNCCLWeightTransferUpdateInfo( + names=[], + dtype_names=[], + shapes=[], + num_updates_list=[], + ) + + def test_rejects_mismatched_num_updates(self): + with pytest.raises(ValueError, match="`num_updates_list`"): + SparseNCCLWeightTransferUpdateInfo( + names=["layer.weight", "layer.bias"], + dtype_names=["float32", "float32"], + shapes=[[10, 10], [10]], + num_updates_list=[3], + ) + + def test_rejects_negative_num_updates(self): + with pytest.raises(ValueError, match="non-negative"): + SparseNCCLWeightTransferUpdateInfo( + names=["layer.weight"], + dtype_names=["float32"], + shapes=[[10, 10]], + num_updates_list=[-1], + ) + + +# --- Unit Tests: Engine Parsing --- + + +class TestNCCLEngineParsing: + """Test NCCLWeightTransferEngine parsing methods.""" + + def _make_engine(self): + config = WeightTransferConfig(backend="nccl") + return NCCLWeightTransferEngine( + config, + create_mock_vllm_config(), + "cuda", + MagicMock(spec=torch.nn.Module), + ) + + def test_parse_init_info_valid(self): + engine = self._make_engine() + init_info = engine.parse_init_info( + { + "master_address": "127.0.0.1", + "master_port": 12345, + "rank_offset": 1, + "world_size": 3, + } + ) + assert isinstance(init_info, NCCLWeightTransferInitInfo) + assert init_info.master_address == "127.0.0.1" + assert init_info.master_port == 12345 + assert init_info.rank_offset == 1 + assert init_info.world_size == 3 + + def test_parse_init_info_missing_field_raises(self): + engine = self._make_engine() + with pytest.raises(ValueError, match="Invalid init_info"): + engine.parse_init_info({"master_address": "127.0.0.1"}) + + def test_parse_update_info_valid(self): + engine = self._make_engine() + update_info = engine.parse_update_info( + { + "names": ["w1", "w2"], + "dtype_names": ["float32", "bfloat16"], + "shapes": [[100, 100], [50]], + } + ) + assert isinstance(update_info, NCCLWeightTransferUpdateInfo) + assert update_info.names == ["w1", "w2"] + assert update_info.dtype_names == ["float32", "bfloat16"] + assert update_info.shapes == [[100, 100], [50]] + + +# --- Unit Tests: Engine Registry --- + + +class TestEngineRegistry: + """Test weight transfer engine registry.""" + + def test_create_engine_nccl(self): + config = WeightTransferConfig(backend="nccl") + engine = WeightTransferEngineFactory.create_engine( + config, create_mock_vllm_config(), "cuda", MagicMock(spec=torch.nn.Module) + ) + assert isinstance(engine, NCCLWeightTransferEngine) + + def test_create_engine_ipc(self): + config = WeightTransferConfig(backend="ipc") + engine = WeightTransferEngineFactory.create_engine( + config, create_mock_vllm_config(), "cuda", MagicMock(spec=torch.nn.Module) + ) + assert isinstance(engine, IPCWeightTransferEngine) + + def test_create_engine_sparse_nccl(self): + config = WeightTransferConfig(backend="sparse_nccl") + engine = WeightTransferEngineFactory.create_engine( + config, create_mock_vllm_config(), "cuda", MagicMock(spec=torch.nn.Module) + ) + assert isinstance(engine, SparseNCCLWeightTransferEngine) + + def test_create_engine_invalid_backend(self): + config = WeightTransferConfig(backend="invalid") + with pytest.raises(ValueError, match="Invalid weight transfer backend"): + WeightTransferEngineFactory.create_engine( + config, + create_mock_vllm_config(), + "cuda", + MagicMock(spec=torch.nn.Module), + ) + + def test_register_duplicate_raises(self): + with pytest.raises(ValueError, match="already registered"): + WeightTransferEngineFactory.register_engine( + "nccl", NCCLWeightTransferEngine + ) + + +# --- Unit Tests: Sparse patch application (CPU) --- + + +class TestSparseNCCLPatchApplication: + """Test SparseNCCLWeightTransferEngine._apply_patch on a real param.""" + + def _make_engine(self, model): + config = WeightTransferConfig(backend="sparse_nccl") + return SparseNCCLWeightTransferEngine( + config, create_mock_vllm_config(), "cpu", model + ) + + def _make_model(self, numel: int = 8): + model = torch.nn.Module() + model.register_parameter( + "w", torch.nn.Parameter(torch.zeros(numel), requires_grad=False) + ) + + def get_parameter(name): + assert name == "w" + return model.w + + model.get_parameter = get_parameter + return model + + def test_apply_patch_updates_only_selected_entries(self): + model = self._make_model(8) + engine = self._make_engine(model) + engine._apply_patch( + SparseWeightPatch( + name="w", + indices=torch.tensor([1, 3], dtype=torch.int32), + values=torch.tensor([5.0, 7.0], dtype=torch.float32), + ) + ) + expected = torch.zeros(8) + expected[1] = 5.0 + expected[3] = 7.0 + assert torch.equal(model.w.data, expected) + + def test_apply_patch_rejects_mismatched_lengths(self): + model = self._make_model(8) + engine = self._make_engine(model) + with pytest.raises(ValueError, match="matching lengths"): + engine._apply_patch( + SparseWeightPatch( + name="w", + indices=torch.tensor([1, 3], dtype=torch.int32), + values=torch.tensor([5.0], dtype=torch.float32), + ) + ) + + def test_apply_patch_rejects_non_int32_indices(self): + model = self._make_model(8) + engine = self._make_engine(model) + with pytest.raises(ValueError, match="int32 indices"): + engine._apply_patch( + SparseWeightPatch( + name="w", + indices=torch.tensor([1], dtype=torch.int64), + values=torch.tensor([5.0], dtype=torch.float32), + ) + ) + + def test_apply_patch_rejects_dtype_mismatch(self): + model = self._make_model(8) + engine = self._make_engine(model) + with pytest.raises(ValueError, match="does not match"): + engine._apply_patch( + SparseWeightPatch( + name="w", + indices=torch.tensor([1], dtype=torch.int32), + values=torch.tensor([5.0], dtype=torch.bfloat16), + ) + ) + + def test_apply_patch_rejects_non_contiguous_param(self): + model = torch.nn.Module() + model.register_parameter( + "w", + torch.nn.Parameter( + torch.arange(12, dtype=torch.float32).view(3, 4).t(), + requires_grad=False, + ), + ) + model.get_parameter = lambda name: model.w + engine = self._make_engine(model) + with pytest.raises(NotImplementedError, match="contiguous params"): + engine._apply_patch( + SparseWeightPatch( + name="w", + indices=torch.tensor([1], dtype=torch.int32), + values=torch.tensor([1.0], dtype=torch.float32), + ) + ) + + +# --- Test receive_weights without init raises --- + + +def test_nccl_receive_weights_without_init_raises(): + """Test that receive_weights raises if init_transfer_engine wasn't called.""" + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + config = WeightTransferConfig(backend="nccl") + engine = NCCLWeightTransferEngine( + config, create_mock_vllm_config(), "cuda", MagicMock(spec=torch.nn.Module) + ) + + update_info = NCCLWeightTransferUpdateInfo( + names=["w"], dtype_names=["float32"], shapes=[[10]] + ) + + with pytest.raises(RuntimeError, match="not initialized"): + engine.receive_weights(update_info) + + +def test_sparse_nccl_receive_weights_without_init_raises(): + """Test that sparse receive raises if init_transfer_engine wasn't called.""" + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + config = WeightTransferConfig(backend="sparse_nccl") + engine = SparseNCCLWeightTransferEngine( + config, create_mock_vllm_config(), "cuda", MagicMock(spec=torch.nn.Module) + ) + + update_info = SparseNCCLWeightTransferUpdateInfo( + names=["w"], + dtype_names=["float32"], + shapes=[[10]], + num_updates_list=[2], + ) + + with pytest.raises(RuntimeError, match="not initialized"): + engine.receive_weights(update_info) + + +# --- Integration Test: NCCL Weight Transfer Between Ray Tasks --- + + +@ray.remote(num_gpus=1) +def trainer_broadcast_tensor( + master_address: str, + master_port: int, + world_size: int, + tensor_shape: list[int], + tensor_dtype: str, +) -> bool: + """Trainer task that broadcasts a tensor via NCCL.""" + import torch + + device = _set_ray_assigned_device() + + from vllm.distributed.device_communicators.pynccl import PyNcclCommunicator + from vllm.distributed.utils import StatelessProcessGroup + + # Create process group as rank 0 (trainer) + pg = StatelessProcessGroup.create( + host=master_address, + port=master_port, + rank=0, + world_size=world_size, + ) + comm = PyNcclCommunicator(pg, device=device.index) + + # Create and broadcast the tensor + dtype = getattr(torch, tensor_dtype) + tensor_to_send = torch.ones(tensor_shape, dtype=dtype, device=device) + comm.broadcast(tensor_to_send, src=0, stream=torch.cuda.current_stream()) + torch.accelerator.synchronize() + + return True + + +@ray.remote(num_gpus=1) +def inference_receive_tensor( + master_address: str, + master_port: int, + world_size: int, + tensor_shape: list[int], + tensor_dtype: str, +) -> dict: + """Inference task that receives tensor via NCCLWeightTransferEngine.""" + import contextlib + from unittest.mock import MagicMock + + import torch + + _set_ray_assigned_device() + + from vllm.config.parallel import ParallelConfig + from vllm.config.weight_transfer import WeightTransferConfig + from vllm.distributed.weight_transfer.nccl_engine import ( + NCCLWeightTransferEngine, + NCCLWeightTransferInitInfo, + NCCLWeightTransferUpdateInfo, + ) + + class Recorder(torch.nn.Module): + def __init__(self): + super().__init__() + self.received = [] + + def load_weights(self, weights): + for name, tensor in weights: + self.received.append((name, tensor.clone())) + + config = WeightTransferConfig(backend="nccl") + vllm_config = MagicMock() + parallel_config = MagicMock(spec=ParallelConfig) + parallel_config.rank = 0 + parallel_config.world_size = 1 + parallel_config.data_parallel_rank = 0 + parallel_config.data_parallel_index = 0 + vllm_config.parallel_config = parallel_config + vllm_config.model_config = MagicMock() + + recorder = Recorder() + engine = NCCLWeightTransferEngine(config, vllm_config, "cuda", recorder) + # Transport-only test: bypass the set_current_vllm_config context that + # receive_weights enters, since vllm_config here is a mock. + import vllm.config as _vllm_config_mod + + _vllm_config_mod.set_current_vllm_config = lambda cfg: contextlib.nullcontext() + + # Initialize the engine (joins as rank 1) + init_info = NCCLWeightTransferInitInfo( + master_address=master_address, + master_port=master_port, + rank_offset=1, # Trainer is rank 0, we become rank 1 + world_size=world_size, + ) + engine.init_transfer_engine(init_info) + + update_info = NCCLWeightTransferUpdateInfo( + names=["test.weight"], + dtype_names=[tensor_dtype], + shapes=[tensor_shape], + ) + engine.receive_weights(update_info) + torch.accelerator.synchronize() + + # Verify we received the tensor + success = False + received_shape = None + received_sum = None + + if len(recorder.received) == 1: + name, tensor = recorder.received[0] + received_shape = list(tensor.shape) + received_sum = tensor.sum().item() + if received_shape == tensor_shape: + expected_sum = 1.0 * torch.tensor(tensor_shape).prod().item() + if abs(received_sum - expected_sum) < 0.01: + success = True + + engine.shutdown() + + return { + "success": success, + "received_shape": received_shape, + "received_sum": received_sum, + } + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 2, + reason="Need at least 2 GPUs to run NCCL weight transfer test.", +) +def test_nccl_weight_transfer_between_processes(): + """Test NCCL weight transfer from trainer to inference process using Ray. + + This test verifies that the NCCLWeightTransferEngine can receive + tensors broadcast by a trainer process via NCCL. + """ + _init_ray_for_weight_transfer() + + master_address = "127.0.0.1" + master_port = get_open_port() + world_size = 2 # 1 trainer + 1 inference worker + + tensor_shape = [100, 100] + tensor_dtype = "float32" + + inference_future = inference_receive_tensor.remote( + master_address, master_port, world_size, tensor_shape, tensor_dtype + ) + trainer_future = trainer_broadcast_tensor.remote( + master_address, master_port, world_size, tensor_shape, tensor_dtype + ) + + trainer_result, result = ray.get([trainer_future, inference_future]) + + assert trainer_result, "Trainer should complete successfully" + assert result["success"], ( + f"Weight transfer failed. " + f"Received shape: {result['received_shape']}, " + f"Received sum: {result['received_sum']}" + ) + + +@ray.remote(num_gpus=1) +def trainer_broadcast_sparse_tensor( + master_address: str, + master_port: int, + world_size: int, +) -> bool: + """Trainer task that broadcasts sparse patches via NCCL.""" + import torch + + device = _set_ray_assigned_device() + + from vllm.distributed.device_communicators.pynccl import PyNcclCommunicator + from vllm.distributed.utils import StatelessProcessGroup + from vllm.distributed.weight_transfer.nccl_engine import ( + NCCLTrainerSendWeightsArgs, + ) + from vllm.distributed.weight_transfer.sparse_nccl_engine import ( + SparseNCCLWeightTransferEngine, + SparseWeightPatch, + ) + + pg = StatelessProcessGroup.create( + host=master_address, + port=master_port, + rank=0, + world_size=world_size, + ) + comm = PyNcclCommunicator(pg, device=device.index) + + patch = SparseWeightPatch( + name="test.weight", + indices=torch.tensor([1, 7, 25], dtype=torch.int32, device=device), + values=torch.tensor([10.0, 20.0, 30.0], dtype=torch.float32, device=device), + ) + SparseNCCLWeightTransferEngine.trainer_send_weights( + iter([patch]), + NCCLTrainerSendWeightsArgs(group=comm), + ) + torch.accelerator.synchronize() + return True + + +@ray.remote(num_gpus=1) +def inference_receive_sparse_tensor( + master_address: str, + master_port: int, + world_size: int, +) -> dict: + """Inference task that receives sparse patches via the sparse engine.""" + from unittest.mock import MagicMock + + import torch + + device = _set_ray_assigned_device() + + from vllm.config.parallel import ParallelConfig + from vllm.config.weight_transfer import WeightTransferConfig + from vllm.distributed.weight_transfer.sparse_nccl_engine import ( + SparseNCCLWeightTransferEngine, + SparseNCCLWeightTransferUpdateInfo, + ) + + config = WeightTransferConfig(backend="sparse_nccl") + vllm_config = MagicMock() + parallel_config = MagicMock(spec=ParallelConfig) + parallel_config.rank = 0 + parallel_config.world_size = 1 + parallel_config.data_parallel_rank = 0 + parallel_config.data_parallel_index = 0 + vllm_config.parallel_config = parallel_config + vllm_config.model_config = MagicMock() + + # Real module holding the target parameter the patch will modify. + model = torch.nn.Module() + model.register_parameter( + "w", torch.nn.Parameter(torch.zeros(30, device="cuda"), requires_grad=False) + ) + model.get_parameter = lambda name: model.w + + update_info = SparseNCCLWeightTransferUpdateInfo( + names=["w"], + dtype_names=["float32"], + shapes=[[30]], + num_updates_list=[3], + ) + + engine = SparseNCCLWeightTransferEngine(config, vllm_config, "cuda", model) + from vllm.distributed.weight_transfer.nccl_common import ( + NCCLWeightTransferInitInfo, + ) + + engine.init_transfer_engine( + NCCLWeightTransferInitInfo( + master_address=master_address, + master_port=master_port, + rank_offset=1, + world_size=world_size, + ) + ) + engine.receive_weights(update_info) + torch.accelerator.synchronize() + + expected = torch.zeros(30, dtype=torch.float32, device=device) + expected[[1, 7, 25]] = torch.tensor( + [10.0, 20.0, 30.0], dtype=torch.float32, device=device + ) + success = torch.equal(model.w.data, expected) + engine.shutdown() + return { + "success": success, + "selected_values": model.w.data[[1, 7, 25]].cpu().tolist(), + } + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 2, + reason="Need at least 2 GPUs to run NCCL sparse weight transfer test.", +) +def test_nccl_sparse_weight_transfer_between_processes(): + """Test NCCL sparse weight transfer from trainer to inference process.""" + _init_ray_for_weight_transfer() + + master_address = "127.0.0.1" + master_port = get_open_port() + world_size = 2 + + inference_future = inference_receive_sparse_tensor.remote( + master_address, master_port, world_size + ) + trainer_future = trainer_broadcast_sparse_tensor.remote( + master_address, master_port, world_size + ) + + trainer_result, result = ray.get([trainer_future, inference_future]) + + assert trainer_result, "Trainer should complete successfully" + assert result["success"], ( + "Sparse weight transfer failed. " + f"Received selected values: {result['selected_values']}" + ) + + +# --- Unit Tests: IPCWeightTransferUpdateInfo Validation --- + + +class TestIPCWeightTransferUpdateInfoValidation: + """Test IPCWeightTransferUpdateInfo dataclass validation.""" + + def test_valid_update_info(self): + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + dummy_tensor = torch.ones(10, 10, device="cuda:0") + _, ipc_handle = reduce_tensor(dummy_tensor) + gpu_uuid = str(torch.cuda.get_device_properties(0).uuid) + ipc_handles = [{gpu_uuid: ipc_handle}] + + info = IPCWeightTransferUpdateInfo( + names=["layer.weight"], + dtype_names=["float32"], + shapes=[[10, 10]], + ipc_handles=ipc_handles, + ) + assert info.names == ["layer.weight"] + assert info.dtype_names == ["float32"] + assert info.shapes == [[10, 10]] + assert len(info.ipc_handles) == 1 + + def test_mismatched_dtype_names_raises(self): + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + dummy_tensor = torch.ones(10, 10, device="cuda:0") + _, ipc_handle = reduce_tensor(dummy_tensor) + gpu_uuid = str(torch.cuda.get_device_properties(0).uuid) + ipc_handles = [{gpu_uuid: ipc_handle}, {gpu_uuid: ipc_handle}] + + with pytest.raises(ValueError, match="dtype_names"): + IPCWeightTransferUpdateInfo( + names=["layer.weight", "layer.bias"], + dtype_names=["float32"], # Only one dtype + shapes=[[10, 10], [10]], + ipc_handles=ipc_handles, + ) + + def test_mismatched_shapes_raises(self): + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + dummy_tensor = torch.ones(10, 10, device="cuda:0") + _, ipc_handle = reduce_tensor(dummy_tensor) + gpu_uuid = str(torch.cuda.get_device_properties(0).uuid) + ipc_handles = [{gpu_uuid: ipc_handle}, {gpu_uuid: ipc_handle}] + + with pytest.raises(ValueError, match="shapes"): + IPCWeightTransferUpdateInfo( + names=["layer.weight", "layer.bias"], + dtype_names=["float32", "float32"], + shapes=[[10, 10]], # Only one shape + ipc_handles=ipc_handles, + ) + + def test_mismatched_ipc_handles_raises(self): + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + dummy_tensor = torch.ones(10, 10, device="cuda:0") + _, ipc_handle = reduce_tensor(dummy_tensor) + gpu_uuid = str(torch.cuda.get_device_properties(0).uuid) + ipc_handles = [{gpu_uuid: ipc_handle}] # Only one handle + + with pytest.raises(ValueError, match="ipc_handles"): + IPCWeightTransferUpdateInfo( + names=["layer.weight", "layer.bias"], + dtype_names=["float32", "float32"], + shapes=[[10, 10], [10]], + ipc_handles=ipc_handles, + ) + + def test_valid_update_info_from_pickled(self, monkeypatch): + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + monkeypatch.setenv("VLLM_ALLOW_INSECURE_SERIALIZATION", "1") + + dummy_tensor = torch.ones(10, 10, device="cuda:0") + ipc_handle = reduce_tensor(dummy_tensor) + gpu_uuid = str(torch.cuda.get_device_properties(0).uuid) + ipc_handles = [{gpu_uuid: ipc_handle}] + + pickled = base64.b64encode(pickle.dumps(ipc_handles)).decode("utf-8") + + info = IPCWeightTransferUpdateInfo( + names=["layer.weight"], + dtype_names=["float32"], + shapes=[[10, 10]], + ipc_handles_pickled=pickled, + ) + assert info.ipc_handles == ipc_handles + assert info.ipc_handles_pickled is None + + def test_pickled_requires_insecure_serialization_flag(self, monkeypatch): + monkeypatch.setenv("VLLM_ALLOW_INSECURE_SERIALIZATION", "0") + + with pytest.raises(ValueError, match="VLLM_ALLOW_INSECURE_SERIALIZATION=1"): + IPCWeightTransferUpdateInfo( + names=[], + dtype_names=[], + shapes=[], + ipc_handles_pickled=base64.b64encode(pickle.dumps([])).decode("utf-8"), + ) + + def test_both_handles_and_pickled_raises(self): + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + dummy_tensor = torch.ones(10, 10, device="cuda:0") + ipc_handle = reduce_tensor(dummy_tensor) + gpu_uuid = str(torch.cuda.get_device_properties(0).uuid) + ipc_handles = [{gpu_uuid: ipc_handle}] + + pickled = base64.b64encode(pickle.dumps(ipc_handles)).decode("utf-8") + + with pytest.raises(ValueError, match="Cannot specify both"): + IPCWeightTransferUpdateInfo( + names=["layer.weight"], + dtype_names=["float32"], + shapes=[[10, 10]], + ipc_handles=ipc_handles, + ipc_handles_pickled=pickled, + ) + + def test_neither_handles_nor_pickled_raises(self): + with pytest.raises(ValueError, match="must be provided"): + IPCWeightTransferUpdateInfo( + names=["layer.weight"], + dtype_names=["float32"], + shapes=[[10, 10]], + ) + + def test_empty_lists_valid(self): + info = IPCWeightTransferUpdateInfo( + names=[], + dtype_names=[], + shapes=[], + ipc_handles=[], + ) + assert len(info.names) == 0 + + +# --- Unit Tests: IPC Engine Parsing --- + + +class TestIPCEngineParsing: + """Test IPCWeightTransferEngine parsing methods.""" + + def _make_engine(self): + config = WeightTransferConfig(backend="ipc") + return IPCWeightTransferEngine( + config, + create_mock_vllm_config(), + "cuda", + MagicMock(spec=torch.nn.Module), + ) + + def test_parse_update_info_valid(self): + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + engine = self._make_engine() + + dummy_tensor1 = torch.ones(100, 100, device="cuda:0") + dummy_tensor2 = torch.ones(50, device="cuda:0") + _, ipc_args1 = reduce_tensor(dummy_tensor1) + _, ipc_args2 = reduce_tensor(dummy_tensor2) + gpu_uuid = str(torch.cuda.get_device_properties(0).uuid) + ipc_handles = [{gpu_uuid: ipc_args1}, {gpu_uuid: ipc_args2}] + + update_info = engine.parse_update_info( + { + "names": ["w1", "w2"], + "dtype_names": ["float32", "bfloat16"], + "shapes": [[100, 100], [50]], + "ipc_handles": ipc_handles, + } + ) + + assert isinstance(update_info, IPCWeightTransferUpdateInfo) + assert update_info.names == ["w1", "w2"] + assert update_info.dtype_names == ["float32", "bfloat16"] + assert update_info.shapes == [[100, 100], [50]] + assert len(update_info.ipc_handles) == 2 + + def test_parse_update_info_pickled(self, monkeypatch): + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + monkeypatch.setenv("VLLM_ALLOW_INSECURE_SERIALIZATION", "1") + + engine = self._make_engine() + + dummy_tensor1 = torch.ones(100, 100, device="cuda:0") + dummy_tensor2 = torch.ones(50, device="cuda:0") + _, ipc_args1 = reduce_tensor(dummy_tensor1) + _, ipc_args2 = reduce_tensor(dummy_tensor2) + gpu_uuid = str(torch.cuda.get_device_properties(0).uuid) + ipc_handles = [{gpu_uuid: ipc_args1}, {gpu_uuid: ipc_args2}] + + pickled = base64.b64encode(pickle.dumps(ipc_handles)).decode("utf-8") + + update_info = engine.parse_update_info( + { + "names": ["w1", "w2"], + "dtype_names": ["float32", "bfloat16"], + "shapes": [[100, 100], [50]], + "ipc_handles_pickled": pickled, + } + ) + + assert isinstance(update_info, IPCWeightTransferUpdateInfo) + assert update_info.names == ["w1", "w2"] + assert len(update_info.ipc_handles) == 2 + assert gpu_uuid in update_info.ipc_handles[0] + assert gpu_uuid in update_info.ipc_handles[1] + + def test_parse_update_info_ignores_none_pickled_handles(self): + engine = self._make_engine() + ipc_handles = [{"gpu-uuid": ("ipc-args",)}] + + update_info = engine.parse_update_info( + { + "names": ["w1"], + "dtype_names": ["float32"], + "shapes": [[1]], + "ipc_handles": ipc_handles, + "ipc_handles_pickled": None, + } + ) + + assert isinstance(update_info, IPCWeightTransferUpdateInfo) + assert update_info.ipc_handles == ipc_handles + + def test_parse_update_info_both_handles_and_pickled_raises(self): + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + engine = self._make_engine() + + dummy_tensor = torch.ones(10, 10, device="cuda:0") + _, ipc_handle = reduce_tensor(dummy_tensor) + gpu_uuid = str(torch.cuda.get_device_properties(0).uuid) + ipc_handles = [{gpu_uuid: ipc_handle}] + + pickled = base64.b64encode(pickle.dumps(ipc_handles)).decode("utf-8") + + with pytest.raises(ValueError, match="Cannot specify both"): + engine.parse_update_info( + { + "names": ["layer.weight"], + "dtype_names": ["float32"], + "shapes": [[10, 10]], + "ipc_handles": ipc_handles, + "ipc_handles_pickled": pickled, + } + ) + + +# --- Integration Test: IPC Weight Transfer Between Ray Tasks --- + + +def get_physical_gpu_id(device_index: int = 0) -> str: + """Get physical GPU UUID for a device.""" + props = torch.cuda.get_device_properties(device_index) + return str(props.uuid) + + +@ray.remote(num_gpus=0.5) +class TrainerActor: + """Trainer actor that creates and holds CUDA IPC handles.""" + + def __init__(self, tensor_shape: list[int], tensor_dtype: str): + device = _set_ray_assigned_device() + + # Create tensor on GPU and keep it alive + dtype = getattr(torch, tensor_dtype) + self.tensor = torch.ones(tensor_shape, dtype=dtype, device=device) + self.tensor.fill_(42.0) # Fill with 42 to verify correct transfer + + _, ipc_args = reduce_tensor(self.tensor) + gpu_uuid = get_physical_gpu_id(device.index) + + torch.accelerator.synchronize() + + self.ipc_handle_dict = { + "ipc_handle": ipc_args, + "gpu_uuid": gpu_uuid, + "shape": tensor_shape, + "dtype": tensor_dtype, + } + + def get_ipc_handle_dict(self) -> dict: + """Return IPC handle dict. Tensor stays alive in this actor.""" + return self.ipc_handle_dict + + +@ray.remote(num_gpus=0.5) +def inference_receive_ipc_tensor( + ipc_handle_dict: dict, + mode: str = "ray", +) -> dict: + """Inference task that receives tensor via IPCWeightTransferEngine.""" + import contextlib + import os + + # Worker-side: ipc_handles_pickled is deserialized via pickle. + if mode == "http": + os.environ["VLLM_ALLOW_INSECURE_SERIALIZATION"] = "1" + + from unittest.mock import MagicMock + + import torch + + _set_ray_assigned_device() + + from vllm.config.parallel import ParallelConfig + from vllm.config.weight_transfer import WeightTransferConfig + from vllm.distributed.weight_transfer.ipc_engine import ( + IPCWeightTransferEngine, + ) + + class Recorder(torch.nn.Module): + def __init__(self): + super().__init__() + self.received = [] + + def load_weights(self, weights): + for name, tensor in weights: + self.received.append((name, tensor.clone())) + + config = WeightTransferConfig(backend="ipc") + vllm_config = MagicMock() + parallel_config = MagicMock(spec=ParallelConfig) + parallel_config.rank = 0 + parallel_config.world_size = 1 + parallel_config.data_parallel_rank = 0 + parallel_config.data_parallel_index = 0 + vllm_config.parallel_config = parallel_config + vllm_config.model_config = MagicMock() + + recorder = Recorder() + engine = IPCWeightTransferEngine(config, vllm_config, "cuda", recorder) + # Transport-only test: bypass the set_current_vllm_config context that + # receive_weights enters, since vllm_config here is a mock. + import vllm.config as _vllm_config_mod + + _vllm_config_mod.set_current_vllm_config = lambda cfg: contextlib.nullcontext() + + init_info = IPCWeightTransferInitInfo() + engine.init_transfer_engine(init_info) + + ipc_handles = [{ipc_handle_dict["gpu_uuid"]: ipc_handle_dict["ipc_handle"]}] + + if mode == "ray": + update_dict: dict = { + "names": ["test.weight"], + "dtype_names": [ipc_handle_dict["dtype"]], + "shapes": [ipc_handle_dict["shape"]], + "ipc_handles": ipc_handles, + } + elif mode == "http": + pickled = base64.b64encode(pickle.dumps(ipc_handles)).decode("utf-8") + update_dict = { + "names": ["test.weight"], + "dtype_names": [ipc_handle_dict["dtype"]], + "shapes": [ipc_handle_dict["shape"]], + "ipc_handles_pickled": pickled, + } + else: + raise ValueError(f"Unknown mode: {mode}") + + update_info = engine.parse_update_info(update_dict) + engine.receive_weights(update_info) + torch.accelerator.synchronize() + + success = False + received_shape = None + received_sum = None + + if len(recorder.received) == 1: + name, tensor = recorder.received[0] + received_shape = list(tensor.shape) + received_sum = tensor.sum().item() + if received_shape == ipc_handle_dict["shape"]: + expected_sum = 42.0 * torch.tensor(ipc_handle_dict["shape"]).prod().item() + if abs(received_sum - expected_sum) < 0.01: + success = True + + engine.shutdown() + + return { + "success": success, + "received_shape": received_shape, + "received_sum": received_sum, + } + + +@pytest.mark.skipif( + torch.accelerator.device_count() < 1, + reason="Need at least 1 GPU to run IPC weight transfer test.", +) +@pytest.mark.parametrize("mode", ["ray", "http"]) +def test_ipc_weight_transfer_between_processes(mode: str): + """Test IPC weight transfer from trainer to inference process using Ray.""" + from ray.util.placement_group import placement_group + from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy + + _init_ray_for_weight_transfer() + + pg = placement_group([{"GPU": 1, "CPU": 2}]) + ray.get(pg.ready()) + + scheduling_strategy = PlacementGroupSchedulingStrategy( + placement_group=pg, + placement_group_capture_child_tasks=True, + ) + + tensor_shape = [100, 100] + tensor_dtype = "float32" + + trainer_actor = TrainerActor.options( # type: ignore[attr-defined] + scheduling_strategy=scheduling_strategy + ).remote(tensor_shape, tensor_dtype) + + ipc_handle_dict = ray.get(trainer_actor.get_ipc_handle_dict.remote()) + + inference_result = ray.get( + inference_receive_ipc_tensor.options( + scheduling_strategy=scheduling_strategy + ).remote(ipc_handle_dict, mode=mode) + ) + + assert inference_result["success"], ( + f"IPC weight transfer failed (mode={mode}). " + f"Received shape: {inference_result['received_shape']}, " + f"Received sum: {inference_result['received_sum']}" + ) + + +def test_ipc_receive_weights_missing_gpu_uuid_raises(): + """Test that receive_weights raises if GPU UUID not found in IPC handles.""" + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + config = WeightTransferConfig(backend="ipc") + engine = IPCWeightTransferEngine( + config, create_mock_vllm_config(), "cuda", MagicMock(spec=torch.nn.Module) + ) + + dummy_tensor = torch.ones(10, 10, device="cuda:0") + _, ipc_handle = reduce_tensor(dummy_tensor) + wrong_uuid = "wrong-uuid-12345" + ipc_handles = [{wrong_uuid: ipc_handle}] + + update_info = IPCWeightTransferUpdateInfo( + names=["w"], + dtype_names=["float32"], + shapes=[[10, 10]], + ipc_handles=ipc_handles, + ) + + with pytest.raises(ValueError, match="IPC handle not found"): + engine.receive_weights(update_info) diff --git a/asian_ai/performance/vLLM/tests/engine/__init__.py b/asian_ai/performance/vLLM/tests/engine/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/engine/test_arg_utils.py b/asian_ai/performance/vLLM/tests/engine/test_arg_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..a0b6a1d93215b68d62c18b16cc0324f01204baa9 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/engine/test_arg_utils.py @@ -0,0 +1,855 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json +from argparse import ArgumentError +from contextlib import AbstractContextManager, nullcontext +from typing import Annotated, Literal + +import pytest +from pydantic import Field + +from vllm.config import AttentionConfig, CompilationConfig, ModelConfig, config +from vllm.engine.arg_utils import ( + EngineArgs, + _expand_json_human_readable_numbers, + contains_type, + get_kwargs, + get_type, + get_type_hints, + is_not_builtin, + is_type, + literal_to_kwargs, + optional_type, + parse_type, +) +from vllm.utils.argparse_utils import FlexibleArgumentParser + + +@pytest.mark.parametrize( + ("type", "value", "expected"), + [ + (int, "42", 42), + (float, "3.14", 3.14), + (str, "Hello World!", "Hello World!"), + (json.loads, '{"foo":1,"bar":2}', {"foo": 1, "bar": 2}), + ], +) +def test_parse_type(type, value, expected): + parse_type_func = parse_type(type) + assert parse_type_func(value) == expected + + +def test_optional_type(): + optional_type_func = optional_type(int) + assert optional_type_func("None") is None + assert optional_type_func("42") == 42 + + +@pytest.mark.parametrize( + ("type_hint", "type", "expected"), + [ + (int, int, True), + (int, float, False), + (list[int], list, True), + (list[int], tuple, False), + (Literal[0, 1], Literal, True), + ], +) +def test_is_type(type_hint, type, expected): + assert is_type(type_hint, type) == expected + + +@pytest.mark.parametrize( + ("type_hints", "type", "expected"), + [ + ({float, int}, int, True), + ({int, tuple}, int, True), + ({int, tuple[int]}, int, True), + ({int, tuple[int, ...]}, int, True), + ({int, tuple[int]}, float, False), + ({int, tuple[int, ...]}, float, False), + ({str, Literal["x", "y"]}, Literal, True), + ], +) +def test_contains_type(type_hints, type, expected): + assert contains_type(type_hints, type) == expected + + +@pytest.mark.parametrize( + ("type_hints", "type", "expected"), + [ + ({int, float}, int, int), + ({int, float}, str, None), + ({str, Literal["x", "y"]}, Literal, Literal["x", "y"]), + ], +) +def test_get_type(type_hints, type, expected): + assert get_type(type_hints, type) == expected + + +@pytest.mark.parametrize( + ("type_hints", "expected"), + [ + ({Literal[1, 2]}, {"type": int, "choices": [1, 2]}), + ({str, Literal["x", "y"]}, {"type": str, "metavar": ["x", "y"]}), + ({Literal[1, "a"]}, Exception), + ], +) +def test_literal_to_kwargs(type_hints, expected): + context: AbstractContextManager[object] = nullcontext() + if expected is Exception: + context = pytest.raises(expected) + with context: + assert literal_to_kwargs(type_hints) == expected + + +@config +class NestedConfig: + field: int = 1 + """field""" + + +@config +class DummyConfig: + regular_bool: bool = True + """Regular bool with default True""" + optional_bool: bool | None = None + """Optional bool with default None""" + + optional_bool_or_str: bool | str | None = None + """Optional bool-or-str with default None""" + + optional_literal: Literal["x", "y"] | None = None + """Optional literal with default None""" + tuple_n: tuple[int, ...] = Field(default_factory=lambda: (1, 2, 3)) + """Tuple with variable length""" + tuple_2: tuple[int, int] = Field(default_factory=lambda: (1, 2)) + """Tuple with fixed length""" + list_n: list[int] = Field(default_factory=lambda: [1, 2, 3]) + """List with variable length""" + list_literal: list[Literal[1, 2]] = Field(default_factory=list) + """List with literal choices""" + list_union: list[str | type[object]] = Field(default_factory=list) + """List with union type""" + set_n: set[int] = Field(default_factory=lambda: {1, 2, 3}) + """Set with variable length""" + literal_literal: Literal[Literal[1], Literal[2]] = 1 + """Literal of literals with default 1""" + json_tip: dict = Field(default_factory=dict) + """Dict which will be JSON in CLI""" + nested_config: NestedConfig = Field(default_factory=NestedConfig) + """Nested config""" + + +@pytest.mark.parametrize( + ("type_hint", "expected"), + [ + (int, False), + (DummyConfig, True), + ], +) +def test_is_not_builtin(type_hint, expected): + assert is_not_builtin(type_hint) == expected + + +@pytest.mark.parametrize( + ("type_hint", "expected"), + [ + (Annotated[int, "annotation"], {int}), + (int | None, {int, type(None)}), + (Annotated[int | None, "annotation"], {int, type(None)}), + (Annotated[int, "annotation"] | None, {int, type(None)}), + ], + ids=["Annotated", "or_None", "Annotated_or_None", "or_None_Annotated"], +) +def test_get_type_hints(type_hint, expected): + assert get_type_hints(type_hint) == expected + + +def test_get_kwargs(): + kwargs = get_kwargs(DummyConfig) + print(kwargs) + + # bools should not have their type set + assert kwargs["regular_bool"].get("type") is None + assert kwargs["optional_bool"].get("type") is None + # optional bool-or-str should accept an optional string value + assert kwargs["optional_bool_or_str"]["type"] is str + assert kwargs["optional_bool_or_str"]["nargs"] == "?" + assert kwargs["optional_bool_or_str"]["const"] is True + assert "action" not in kwargs["optional_bool_or_str"] + # optional literals should have None as a choice + assert kwargs["optional_literal"]["choices"] == ["x", "y", "None"] + # tuples should have the correct nargs + assert kwargs["tuple_n"]["nargs"] == "+" + assert kwargs["tuple_2"]["nargs"] == 2 + # lists should work + assert kwargs["list_n"]["type"] is int + assert kwargs["list_n"]["nargs"] == "+" + # lists with literals should have the correct choices + assert kwargs["list_literal"]["type"] is int + assert kwargs["list_literal"]["nargs"] == "+" + assert kwargs["list_literal"]["choices"] == [1, 2] + # lists with unions should become str type. + # If not, we cannot know which type to use for parsing + assert kwargs["list_union"]["type"] is str + # sets should work like lists + assert kwargs["set_n"]["type"] is int + assert kwargs["set_n"]["nargs"] == "+" + # literals of literals should have merged choices + assert kwargs["literal_literal"]["choices"] == [1, 2] + # dict should have json tip in help + json_tip = "Should either be a valid JSON string or JSON keys" + assert json_tip in kwargs["json_tip"]["help"] + # nested config should construct the nested config + assert kwargs["nested_config"]["type"]('{"field": 2}') == NestedConfig(2) # type: ignore[call-arg] + + +def test_jit_monitor_verbose_arg(): + parser = EngineArgs.add_cli_args(FlexibleArgumentParser()) + args = parser.parse_args(["--jit-monitor-verbose"]) + + assert args.jit_monitor_verbose + assert EngineArgs(model="test", jit_monitor_verbose=True).jit_monitor_verbose + + +@pytest.mark.parametrize("mode", ["warn", "error"]) +def test_jit_monitor_mode_arg(mode): + parser = EngineArgs.add_cli_args(FlexibleArgumentParser()) + args = parser.parse_args(["--jit-monitor-mode", mode]) + + assert args.jit_monitor_mode == mode + engine_args = EngineArgs(model="test", jit_monitor_mode=mode) + assert engine_args.jit_monitor_mode == mode + assert engine_args.create_observability_config().jit_monitor_mode == mode + + +def test_hf_token_get_kwargs(): + kwargs = get_kwargs(ModelConfig)["hf_token"] + + assert kwargs["type"] is str + assert kwargs["nargs"] == "?" + assert kwargs["const"] is True + assert "action" not in kwargs + + +@pytest.mark.parametrize( + ("cli_args", "expected"), + [ + ([], None), + (["--hf-token"], True), + (["--hf-token", "hf_secret"], "hf_secret"), + (["--hf-token", "None"], "None"), + ], +) +def test_hf_token_cli_arg(cli_args, expected): + parser = EngineArgs.add_cli_args(FlexibleArgumentParser()) + + args = parser.parse_args(cli_args) + + assert args.hf_token == expected + + +@pytest.mark.parametrize( + ("arg", "expected"), + [ + (None, dict()), + ('{"video": {"num_frames": 123} }', {"video": {"num_frames": 123}}), + ( + '{"video": {"num_frames": 123, "fps": 1.0, "foo": "bar"}, "image": {"foo": "bar"} }', # noqa + { + "video": {"num_frames": 123, "fps": 1.0, "foo": "bar"}, + "image": {"foo": "bar"}, + }, + ), + ], +) +def test_media_io_kwargs_parser(arg, expected): + parser = EngineArgs.add_cli_args(FlexibleArgumentParser()) + if arg is None: + args = parser.parse_args([]) + else: + args = parser.parse_args(["--media-io-kwargs", arg]) + + assert args.media_io_kwargs == expected + + +@pytest.mark.parametrize( + ("args", "expected"), + [ + (["-O", "1"], "1"), + (["-O", "2"], "2"), + (["-O", "3"], "3"), + (["-O0"], "0"), + (["-O1"], "1"), + (["-O2"], "2"), + (["-O3"], "3"), + ], +) +def test_optimization_level(args, expected): + """ + Test space-separated optimization levels (-O 1, -O 2, -O 3) map to + optimization_level. + """ + parser = EngineArgs.add_cli_args(FlexibleArgumentParser()) + parsed_args = parser.parse_args(args) + assert parsed_args.optimization_level == expected + assert parsed_args.compilation_config.mode is None + + +@pytest.mark.parametrize( + ("args", "expected"), + [ + (["-cc.mode=0"], 0), + (["-cc.mode=1"], 1), + (["-cc.mode=2"], 2), + (["-cc.mode=3"], 3), + ], +) +def test_mode_parser(args, expected): + """ + Test compilation config modes (-cc.mode=int) map to compilation_config. + """ + parser = EngineArgs.add_cli_args(FlexibleArgumentParser()) + parsed_args = parser.parse_args(args) + assert parsed_args.compilation_config.mode == expected + + +def test_compilation_config(): + parser = EngineArgs.add_cli_args(FlexibleArgumentParser()) + + # default value + args = parser.parse_args([]) + assert args.compilation_config == CompilationConfig() + + # set to string form of a dict + args = parser.parse_args( + [ + "-cc", + '{"mode": 3, "cudagraph_capture_sizes": [1, 2, 4, 8], "backend": "eager"}', + ] + ) + assert ( + args.compilation_config.mode == 3 + and args.compilation_config.cudagraph_capture_sizes == [1, 2, 4, 8] + and args.compilation_config.backend == "eager" + ) + + # set to string form of a dict + args = parser.parse_args( + [ + "--compilation-config=" + '{"mode": 3, "cudagraph_capture_sizes": [1, 2, 4, 8], ' + '"backend": "inductor"}', + ] + ) + assert ( + args.compilation_config.mode == 3 + and args.compilation_config.cudagraph_capture_sizes == [1, 2, 4, 8] + and args.compilation_config.backend == "inductor" + ) + + +def test_attention_config(): + from vllm.v1.attention.backends.registry import AttentionBackendEnum + + parser = EngineArgs.add_cli_args(FlexibleArgumentParser()) + + # default value + args = parser.parse_args([]) + assert args is not None + engine_args = EngineArgs.from_cli_args(args) + assert engine_args.attention_config == AttentionConfig() + + # set backend via dot notation + args = parser.parse_args(["--attention-config.backend", "FLASH_ATTN"]) + assert args is not None + engine_args = EngineArgs.from_cli_args(args) + assert engine_args.attention_config.backend is not None + assert engine_args.attention_config.backend.name == "FLASH_ATTN" + + # set backend via --attention-backend shorthand + args = parser.parse_args(["--attention-backend", "FLASHINFER"]) + assert args is not None + engine_args = EngineArgs.from_cli_args(args) + assert engine_args.attention_backend is not None + assert engine_args.attention_backend == "FLASHINFER" + + # set all fields via dot notation + args = parser.parse_args( + [ + "--attention-config.backend", + "FLASH_ATTN", + "--attention-config.flash_attn_version", + "3", + "--attention-config.use_prefill_decode_attention", + "true", + "--attention-config.flash_attn_max_num_splits_for_cuda_graph", + "16", + "--attention-config.use_trtllm_attention", + "true", + "--attention-config.disable_flashinfer_q_quantization", + "true", + ] + ) + assert args is not None + engine_args = EngineArgs.from_cli_args(args) + assert engine_args.attention_config.backend is not None + assert engine_args.attention_config.backend.name == "FLASH_ATTN" + assert engine_args.attention_config.flash_attn_version == 3 + assert engine_args.attention_config.use_prefill_decode_attention is True + assert engine_args.attention_config.flash_attn_max_num_splits_for_cuda_graph == 16 + assert engine_args.attention_config.use_trtllm_attention is True + assert engine_args.attention_config.disable_flashinfer_q_quantization is True + + # set to string form of a dict with all fields + args = parser.parse_args( + [ + "--attention-config=" + '{"backend": "FLASHINFER", "flash_attn_version": 2, ' + '"use_prefill_decode_attention": false, ' + '"flash_attn_max_num_splits_for_cuda_graph": 8, ' + '"use_trtllm_attention": false, ' + '"disable_flashinfer_q_quantization": false}', + ] + ) + assert args is not None + engine_args = EngineArgs.from_cli_args(args) + assert engine_args.attention_config.backend is not None + assert engine_args.attention_config.backend.name == "FLASHINFER" + assert engine_args.attention_config.flash_attn_version == 2 + assert engine_args.attention_config.use_prefill_decode_attention is False + assert engine_args.attention_config.flash_attn_max_num_splits_for_cuda_graph == 8 + assert engine_args.attention_config.use_trtllm_attention is False + assert engine_args.attention_config.disable_flashinfer_q_quantization is False + + # test --attention-backend flows into VllmConfig.attention_config + args = parser.parse_args( + [ + "--model", + "facebook/opt-125m", + "--attention-backend", + "FLASH_ATTN", + ] + ) + assert args is not None + engine_args = EngineArgs.from_cli_args(args) + vllm_config = engine_args.create_engine_config() + assert vllm_config.attention_config.backend == AttentionBackendEnum.FLASH_ATTN + + # test --attention-config.backend flows into VllmConfig.attention_config + args = parser.parse_args( + [ + "--model", + "facebook/opt-125m", + "--attention-config.backend", + "FLASHINFER", + ] + ) + assert args is not None + engine_args = EngineArgs.from_cli_args(args) + vllm_config = engine_args.create_engine_config() + assert vllm_config.attention_config.backend == AttentionBackendEnum.FLASHINFER + + # test --attention-backend and --attention-config.backend are mutually exclusive + args = parser.parse_args( + [ + "--model", + "facebook/opt-125m", + "--attention-backend", + "FLASH_ATTN", + "--attention-config.backend", + "FLASHINFER", + ] + ) + assert args is not None + engine_args = EngineArgs.from_cli_args(args) + with pytest.raises(ValueError, match="mutually exclusive"): + engine_args.create_engine_config() + + +def test_prefix_cache_default(): + parser = EngineArgs.add_cli_args(FlexibleArgumentParser()) + args = parser.parse_args([]) + + # should be None by default (depends on model). + engine_args = EngineArgs.from_cli_args(args=args) + assert engine_args.enable_prefix_caching is None + + # with flag to turn it on. + args = parser.parse_args(["--enable-prefix-caching"]) + engine_args = EngineArgs.from_cli_args(args=args) + assert engine_args.enable_prefix_caching + + # with disable flag to turn it off. + args = parser.parse_args(["--no-enable-prefix-caching"]) + engine_args = EngineArgs.from_cli_args(args=args) + assert not engine_args.enable_prefix_caching + + +@pytest.mark.parametrize( + ("arg", "expected", "option"), + [ + (None, None, "mm-processor-kwargs"), + ("{}", {}, "mm-processor-kwargs"), + ('{"num_crops": 4}', {"num_crops": 4}, "mm-processor-kwargs"), + ('{"foo": {"bar": "baz"}}', {"foo": {"bar": "baz"}}, "mm-processor-kwargs"), + ], +) +def test_composite_arg_parser(arg, expected, option): + parser = EngineArgs.add_cli_args(FlexibleArgumentParser()) + if arg is None: + args = parser.parse_args([]) + else: + args = parser.parse_args([f"--{option}", arg]) + assert getattr(args, option.replace("-", "_")) == expected + + +def test_human_readable_model_len(): + # `exit_on_error` disabled to test invalid values below + parser = EngineArgs.add_cli_args(FlexibleArgumentParser(exit_on_error=False)) + + args = parser.parse_args([]) + assert args.max_model_len is None + + args = parser.parse_args(["--max-model-len", "1024"]) + assert args.max_model_len == 1024 + + # Lower + args = parser.parse_args(["--max-model-len", "1m"]) + assert args.max_model_len == 1_000_000 + args = parser.parse_args(["--max-model-len", "10k"]) + assert args.max_model_len == 10_000 + args = parser.parse_args(["--max-model-len", "2g"]) + assert args.max_model_len == 2_000_000_000 + args = parser.parse_args(["--max-model-len", "2t"]) + assert args.max_model_len == 2_000_000_000_000 + + # Capital + args = parser.parse_args(["--max-model-len", "3K"]) + assert args.max_model_len == 2**10 * 3 + args = parser.parse_args(["--max-model-len", "10M"]) + assert args.max_model_len == 2**20 * 10 + args = parser.parse_args(["--max-model-len", "4G"]) + assert args.max_model_len == 2**30 * 4 + args = parser.parse_args(["--max-model-len", "4T"]) + assert args.max_model_len == 2**40 * 4 + + # Decimal values + args = parser.parse_args(["--max-model-len", "10.2k"]) + assert args.max_model_len == 10200 + # ..truncated to the nearest int + args = parser.parse_args(["--max-model-len", "10.2123451234567k"]) + assert args.max_model_len == 10212 + args = parser.parse_args(["--max-model-len", "10.2123451234567m"]) + assert args.max_model_len == 10212345 + args = parser.parse_args(["--max-model-len", "10.2123451234567g"]) + assert args.max_model_len == 10212345123 + args = parser.parse_args(["--max-model-len", "10.2123451234567t"]) + assert args.max_model_len == 10212345123456 + + # Special value -1 for auto-fit to GPU memory + args = parser.parse_args(["--max-model-len", "-1"]) + assert args.max_model_len == -1 + + # 'auto' is an alias for -1 + args = parser.parse_args(["--max-model-len", "auto"]) + assert args.max_model_len == -1 + args = parser.parse_args(["--max-model-len", "AUTO"]) + assert args.max_model_len == -1 + + # Invalid (do not allow decimals with binary multipliers) + for invalid in ["1a", "pwd", "10.24", "1.23M", "1.22T"]: + with pytest.raises(ArgumentError): + parser.parse_args(["--max-model-len", invalid]) + + +def test_numa_bind_args(): + parser = EngineArgs.add_cli_args(FlexibleArgumentParser()) + args = parser.parse_args( + [ + "--numa-bind", + "--numa-bind-nodes", + "0", + "0", + "1", + "1", + "--numa-bind-cpus", + "0-3", + "4-7", + "8-11", + "12-15", + ] + ) + engine_args = EngineArgs.from_cli_args(args=args) + assert engine_args.numa_bind is True + assert engine_args.numa_bind_nodes == [0, 0, 1, 1] + assert engine_args.numa_bind_cpus == ["0-3", "4-7", "8-11", "12-15"] + + +def test_ir_op_priority(): + from vllm.config.kernel import IrOpPriorityConfig, KernelConfig + + ir_op_priority = IrOpPriorityConfig(rms_norm=["vllm_c"]) + cfg1 = EngineArgs(ir_op_priority=ir_op_priority).create_engine_config() + cfg2 = EngineArgs( + kernel_config=KernelConfig(ir_op_priority=ir_op_priority) + ).create_engine_config() + assert cfg1.kernel_config.ir_op_priority == cfg2.kernel_config.ir_op_priority + + with pytest.raises(ValueError, match="rms_norm"): + _ = EngineArgs( + ir_op_priority=ir_op_priority, + kernel_config=KernelConfig(ir_op_priority=ir_op_priority), + ).create_engine_config() + + +@pytest.mark.parametrize( + ("input_json", "expected_json"), + [ + # Decimal suffixes (lowercase) + ('{"x": 80g}', '{"x": 80000000000}'), + ('{"x": 1k}', '{"x": 1000}'), + ('{"x": 5m}', '{"x": 5000000}'), + ('{"x": 2t}', '{"x": 2000000000000}'), + # Binary suffixes (uppercase) + ('{"x": 1K}', f'{{"x": {2**10}}}'), + ('{"x": 1G}', f'{{"x": {2**30}}}'), + # Decimal values + ('{"x": 1.5g}', '{"x": 1500000000}'), + # Quoted strings must NOT be modified + ('{"my_key": 80g}', '{"my_key": 80000000000}'), + ('{"name": "80g"}', '{"name": "80g"}'), + ('{"model_name": "foo_bar"}', '{"model_name": "foo_bar"}'), + # Multiple values + ('{"a": 1k, "b": 2m}', '{"a": 1000, "b": 2000000}'), + # Plain numbers are untouched + ('{"x": 42}', '{"x": 42}'), + # Nested JSON + ('{"outer": {"inner": 10g}}', '{"outer": {"inner": 10000000000}}'), + ], +) +def test_expand_json_human_readable_numbers(input_json, expected_json): + assert _expand_json_human_readable_numbers(input_json) == expected_json + + +@pytest.mark.parametrize( + "uri", + ["s3://bucket/model", "gs://bucket/model", "az://container/model"], +) +def test_cloud_storage_uri_skips_get_model_path(uri, monkeypatch): + """Cloud storage URIs should not be passed to get_model_path() + when HF_HUB_OFFLINE=1, as they are not valid HF repo IDs.""" + import huggingface_hub + + monkeypatch.setattr(huggingface_hub.constants, "HF_HUB_OFFLINE", True) + + args = EngineArgs(model=uri) + # model should remain the original cloud URI, not raise + assert args.model == uri + + +def test_cloud_storage_tokenizer_skips_get_model_path(monkeypatch): + """Cloud storage tokenizer URI should not be passed to + get_model_path() when HF_HUB_OFFLINE=1.""" + import huggingface_hub + + monkeypatch.setattr(huggingface_hub.constants, "HF_HUB_OFFLINE", True) + + args = EngineArgs(model="s3://bucket/model", tokenizer="s3://bucket/tokenizer") + assert args.model == "s3://bucket/model" + assert args.tokenizer == "s3://bucket/tokenizer" + + +class TestDeviceIds: + def test_device_ids_with_cvd_out_of_range(self, monkeypatch): + """--device-ids index beyond the CVD set raises ValueError.""" + from vllm.platforms import current_platform + + key = current_platform.device_control_env_var + monkeypatch.setenv(key, "4,5") + args = EngineArgs(model="m", device_ids=[0, 2]) + with pytest.raises(ValueError, match="out of range"): + args._resolve_device_ids() + + def test_device_ids_with_cvd_resolve_to_physical_ids(self, monkeypatch): + """--device-ids are CVD-local indices resolved to physical ids.""" + from vllm.platforms import current_platform + + key = current_platform.device_control_env_var + monkeypatch.setenv(key, "4,5") + args = EngineArgs(model="m", device_ids=[0, 1]) + assert args._resolve_device_ids() == [4, 5] + + def test_device_ids_with_uuid_cvd_resolve_to_physical_ids(self, monkeypatch): + """--device-ids support UUID CVD values resolved by the platform.""" + from vllm.platforms import current_platform + + key = current_platform.device_control_env_var + monkeypatch.setenv(key, "GPU-abcd1234,GPU-ef567890") + monkeypatch.setattr( + type(current_platform), + "device_control_id_to_physical_device_id", + classmethod( + lambda cls, device_id: {"GPU-abcd1234": 4, "GPU-ef567890": 5}[device_id] + ), + ) + + args = EngineArgs(model="m", device_ids=[0, 1]) + assert args._resolve_device_ids() == [4, 5] + + def test_device_ids_with_uuid_args_resolve_to_physical_ids(self, monkeypatch): + """UUID --device-ids are resolved to physical IDs immediately.""" + from vllm.platforms import current_platform + + monkeypatch.setattr( + type(current_platform), + "device_control_id_to_physical_device_id", + classmethod(lambda cls, device_id: {"GPU-abcd1234": 4}[device_id]), + ) + + args = EngineArgs(model="m", device_ids=["GPU-abcd1234"]) + assert args._resolve_device_ids() == [4] + + def test_device_ids_reject_mixed_integer_and_uuid_args(self): + """--device-ids must not mix CVD indices and UUIDs.""" + args = EngineArgs(model="m", device_ids=[0, "GPU-abcd1234"]) + with pytest.raises(ValueError, match="must not mix"): + args._resolve_device_ids() + + def test_no_device_ids(self): + """No --device-ids returns None.""" + args = EngineArgs(model="m") + assert args._resolve_device_ids() is None + + def test_cli_parsing(self): + """--device-ids parses comma-separated string from CLI.""" + parser = FlexibleArgumentParser() + EngineArgs.add_cli_args(parser) + parsed = parser.parse_args(["--model", "m", "--device-ids", "0,2,4"]) + assert parsed.device_ids == [0, 2, 4] + + def test_cli_parsing_uuid(self): + """--device-ids parses comma-separated UUID strings from CLI.""" + parser = FlexibleArgumentParser() + EngineArgs.add_cli_args(parser) + parsed = parser.parse_args( + ["--model", "m", "--device-ids", "GPU-abcd1234,GPU-ef567890"] + ) + assert parsed.device_ids == ["GPU-abcd1234", "GPU-ef567890"] + + def test_assigned_physical_gpu_ids_are_physical_with_cvd(self, monkeypatch): + """assigned_physical_gpu_ids are already physical and not composed with CVD.""" + import vllm.platforms.interface as platform_interface + from vllm.platforms import current_platform + + monkeypatch.setattr(platform_interface, "_assigned_physical_gpu_ids", [4, 5]) + monkeypatch.setenv(current_platform.device_control_env_var, "4,5") + + assert current_platform.device_id_to_physical_device_id(0) == 4 + assert current_platform.device_id_to_physical_device_id(1) == 5 + assert current_platform.logical_device_id_to_visible_device_id(0) == 0 + assert current_platform.logical_device_id_to_visible_device_id(1) == 1 + + def test_assigned_physical_gpu_ids_map_to_visible_uuid_cvd(self, monkeypatch): + """Physical IDs map back to visible ordinals when CVD uses UUIDs.""" + import vllm.platforms.interface as platform_interface + from vllm.platforms import current_platform + + monkeypatch.setattr(platform_interface, "_assigned_physical_gpu_ids", [5]) + monkeypatch.setenv( + current_platform.device_control_env_var, + "GPU-abcd1234,GPU-ef567890", + ) + monkeypatch.setattr( + type(current_platform), + "device_control_id_to_physical_device_id", + classmethod( + lambda cls, device_id: {"GPU-abcd1234": 4, "GPU-ef567890": 5}[device_id] + ), + ) + + assert current_platform.logical_device_id_to_visible_device_id(0) == 1 + + def test_device_ids_reject_duplicates(self): + """--device-ids must not contain duplicate entries.""" + args = EngineArgs(model="m", device_ids=[2, 2]) + with pytest.raises(ValueError, match="duplicates"): + args._resolve_device_ids() + + def test_cli_parsing_strips_whitespace(self): + """--device-ids tolerates whitespace around commas.""" + parser = FlexibleArgumentParser() + EngineArgs.add_cli_args(parser) + parsed = parser.parse_args(["--model", "m", "--device-ids", "0, 2, 4"]) + assert parsed.device_ids == [0, 2, 4] + + def test_visible_ordinal_to_physical_ignores_assigned_ids(self, monkeypatch): + """visible_device_id_to_physical_device_id maps torch device ordinals, + independent of the logical-to-physical mapping. + + Regression test: CustomAllreduce passes device.index (a visible + ordinal) and must not index into assigned_physical_gpu_ids, which + raised IndexError for non-identity --device-ids like [2, 3]. + """ + import vllm.platforms.interface as platform_interface + from vllm.platforms import current_platform + + monkeypatch.setattr(platform_interface, "_assigned_physical_gpu_ids", [2, 3]) + monkeypatch.delenv(current_platform.device_control_env_var, raising=False) + + # CVD unset: visible ordinal == physical ID, even beyond the + # assigned list's length. + assert current_platform.visible_device_id_to_physical_device_id(2) == 2 + assert current_platform.visible_device_id_to_physical_device_id(3) == 3 + + monkeypatch.setenv(current_platform.device_control_env_var, "4,5") + assert current_platform.visible_device_id_to_physical_device_id(1) == 5 + with pytest.raises(IndexError, match="out of range"): + current_platform.visible_device_id_to_physical_device_id(2) + + +class TestDpDeviceIdSharding: + def test_dp_supervisor_device_ids_stay_env_relative(self): + """Regression test: the DP supervisor must pass env-relative indices, + not physical IDs, because each child re-resolves --device-ids + against its inherited device-control env var.""" + import argparse + + from vllm.entrypoints.openai.dp_supervisor import _build_device_ids + + args = argparse.Namespace( + tensor_parallel_size=2, pipeline_parallel_size=1, device_ids=None + ) + assert _build_device_ids(args, local_rank=0) == [0, 1] + assert _build_device_ids(args, local_rank=1) == [2, 3] + + def test_dp_supervisor_shards_user_device_ids(self): + """User-provided --device-ids are sharded across DP children.""" + import argparse + + from vllm.entrypoints.openai.dp_supervisor import _build_device_ids + + args = argparse.Namespace( + tensor_parallel_size=2, pipeline_parallel_size=1, device_ids=[4, 5, 6, 7] + ) + assert _build_device_ids(args, local_rank=0) == [4, 5] + assert _build_device_ids(args, local_rank=1) == [6, 7] + with pytest.raises(ValueError, match="needs devices"): + _build_device_ids(args, local_rank=2) + + def test_dp_rank_shards_user_assigned_gpu_ids(self): + """get_physical_gpu_ids_for_local_dp_rank slices the user-provided + --device-ids list instead of recomputing from the env var.""" + from vllm.platforms import current_platform + from vllm.v1.engine.utils import get_physical_gpu_ids_for_local_dp_rank + + evar = current_platform.device_control_env_var + assert get_physical_gpu_ids_for_local_dp_rank( + evar, local_dp_rank=1, world_size=2, user_assigned_gpu_ids=[4, 5, 6, 7] + ) == [6, 7] + with pytest.raises(ValueError, match="needs devices"): + get_physical_gpu_ids_for_local_dp_rank( + evar, local_dp_rank=2, world_size=2, user_assigned_gpu_ids=[4, 5, 6, 7] + ) diff --git a/asian_ai/performance/vLLM/tests/engine/test_short_mm_context.py b/asian_ai/performance/vLLM/tests/engine/test_short_mm_context.py new file mode 100644 index 0000000000000000000000000000000000000000..2761b5f98d5f0c1531efa14c0760d908a396eb2c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/engine/test_short_mm_context.py @@ -0,0 +1,37 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from ..conftest import IMAGE_ASSETS + +HF_IMAGE_PROMPTS = IMAGE_ASSETS.prompts( + { + "stop_sign": "USER: \nWhat's the content of the image?\nASSISTANT:", + "cherry_blossom": "USER: \nWhat is the season?\nASSISTANT:", + } +) + +models = ["llava-hf/llava-1.5-7b-hf"] + + +@pytest.mark.parametrize("model", models) +def test_context_length_too_short(vllm_runner, image_assets, model): + images = [asset.pil_image for asset in image_assets] + + with pytest.raises(ValueError, match="longer than the maximum model length"): + vllm_model = vllm_runner( + model, + # LLaVA has a feature size of 576 + # For the HF processor to execute successfully but still + # failing the overall context length check, we need the + # max_model_len to at least contain all image tokens + max_model_len=579, + enforce_eager=True, + load_format="dummy", + ) + + with vllm_model: + vllm_model.generate_greedy( + [HF_IMAGE_PROMPTS[0]], max_tokens=1, images=[images[0]] + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/anthropic/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/anthropic/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/anthropic/test_anthropic_messages_conversion.py b/asian_ai/performance/vLLM/tests/entrypoints/anthropic/test_anthropic_messages_conversion.py new file mode 100644 index 0000000000000000000000000000000000000000..5584a74184cf00d3ec5bcd3637f5d9b19e7e3341 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/anthropic/test_anthropic_messages_conversion.py @@ -0,0 +1,1422 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for Anthropic-to-OpenAI request conversion. + +Tests the image source handling and tool_result content parsing in +AnthropicServingMessages._convert_anthropic_to_openai_request(). + +Also covers extended-thinking edge cases such as ``redacted_thinking`` +blocks echoed back by Anthropic clients, and streaming conversion in +``message_stream_converter``. + +Also covers cache usage computation in ``_build_anthropic_usage``. +""" + +import json +from unittest.mock import MagicMock + +import pytest + +from vllm.entrypoints.anthropic.protocol import ( + AnthropicMessagesRequest, +) +from vllm.entrypoints.anthropic.serving import ( + AnthropicServingMessages, + _build_anthropic_usage, + _get_cached_tokens, +) +from vllm.entrypoints.openai.chat_completion.protocol import ( + ChatCompletionResponse, + ChatCompletionResponseChoice, + ChatCompletionResponseStreamChoice, + ChatCompletionStreamResponse, + ChatMessage, +) +from vllm.entrypoints.openai.engine.protocol import ( + DeltaFunctionCall, + DeltaMessage, + DeltaToolCall, + PromptTokenUsageInfo, + UsageInfo, +) + +_convert = AnthropicServingMessages._convert_anthropic_to_openai_request +_img_url = AnthropicServingMessages._convert_image_source_to_url + + +def _make_request( + messages: list[dict], + **kwargs, +) -> AnthropicMessagesRequest: + return AnthropicMessagesRequest( + model="test-model", + max_tokens=128, + messages=messages, + **kwargs, + ) + + +# ====================================================================== +# _convert_image_source_to_url +# ====================================================================== + + +class TestConvertImageSourceToUrl: + def test_base64_source(self): + source = { + "type": "base64", + "media_type": "image/jpeg", + "data": "iVBORw0KGgo=", + } + assert _img_url(source) == "data:image/jpeg;base64,iVBORw0KGgo=" + + def test_base64_png(self): + source = { + "type": "base64", + "media_type": "image/png", + "data": "AAAA", + } + assert _img_url(source) == "data:image/png;base64,AAAA" + + def test_url_source(self): + source = { + "type": "url", + "url": "https://example.com/image.jpg", + } + assert _img_url(source) == "https://example.com/image.jpg" + + def test_missing_type_defaults_to_base64(self): + """When 'type' is absent, treat as base64.""" + source = { + "media_type": "image/webp", + "data": "UklGR", + } + assert _img_url(source) == "data:image/webp;base64,UklGR" + + def test_missing_media_type_defaults_to_jpeg(self): + source = {"type": "base64", "data": "abc123"} + assert _img_url(source) == "data:image/jpeg;base64,abc123" + + def test_url_source_missing_url_returns_empty(self): + source = {"type": "url"} + assert _img_url(source) == "" + + def test_empty_source_returns_data_uri_shell(self): + source: dict = {} + assert _img_url(source) == "data:image/jpeg;base64," + + +# ====================================================================== +# Image blocks inside user messages +# ====================================================================== + + +class TestImageContentBlocks: + def test_base64_image_in_user_message(self): + request = _make_request( + [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Describe this image"}, + { + "type": "image", + "source": { + "type": "base64", + "media_type": "image/jpeg", + "data": "iVBORw0KGgo=", + }, + }, + ], + } + ] + ) + + result = _convert(request) + user_msg = result.messages[0] + assert user_msg["role"] == "user" + + parts = user_msg["content"] + assert len(parts) == 2 + assert parts[0] == {"type": "text", "text": "Describe this image"} + assert parts[1] == { + "type": "image_url", + "image_url": {"url": "data:image/jpeg;base64,iVBORw0KGgo="}, + } + + def test_url_image_in_user_message(self): + request = _make_request( + [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What is this?"}, + { + "type": "image", + "source": { + "type": "url", + "url": "https://example.com/cat.png", + }, + }, + ], + } + ] + ) + + result = _convert(request) + parts = result.messages[0]["content"] + assert parts[1] == { + "type": "image_url", + "image_url": {"url": "https://example.com/cat.png"}, + } + + +# ====================================================================== +# tool_result content handling +# ====================================================================== + + +class TestToolResultContent: + def _make_tool_result_request( + self, tool_result_content + ) -> AnthropicMessagesRequest: + """Build a request with assistant tool_use followed by user + tool_result.""" + return _make_request( + [ + { + "role": "assistant", + "content": [ + { + "type": "tool_use", + "id": "call_001", + "name": "read_file", + "input": {"path": "/tmp/img.png"}, + } + ], + }, + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": "call_001", + "content": tool_result_content, + } + ], + }, + ] + ) + + def test_tool_result_string_content(self): + request = self._make_tool_result_request("file contents here") + result = _convert(request) + + tool_msg = [m for m in result.messages if m["role"] == "tool"] + assert len(tool_msg) == 1 + assert tool_msg[0]["content"] == "file contents here" + assert tool_msg[0]["tool_call_id"] == "call_001" + + def test_tool_result_text_blocks(self): + request = self._make_tool_result_request( + [ + {"type": "text", "text": "line 1"}, + {"type": "text", "text": "line 2"}, + ] + ) + result = _convert(request) + + tool_msg = [m for m in result.messages if m["role"] == "tool"] + assert len(tool_msg) == 1 + assert tool_msg[0]["content"] == "line 1\nline 2" + + def test_tool_result_with_image(self): + """Image in tool_result should produce a follow-up user message.""" + request = self._make_tool_result_request( + [ + { + "type": "image", + "source": { + "type": "base64", + "media_type": "image/png", + "data": "AAAA", + }, + } + ] + ) + result = _convert(request) + + tool_msg = [m for m in result.messages if m["role"] == "tool"] + assert len(tool_msg) == 1 + assert tool_msg[0]["content"] == "" + + # The image should be injected as a follow-up user message + follow_up = [ + m + for m in result.messages + if m["role"] == "user" and isinstance(m.get("content"), list) + ] + assert len(follow_up) == 1 + img_parts = follow_up[0]["content"] + assert len(img_parts) == 1 + assert img_parts[0] == { + "type": "image_url", + "image_url": {"url": "data:image/png;base64,AAAA"}, + } + + def test_tool_result_with_text_and_image(self): + """Mixed text+image tool_result: text in tool msg, image in user + msg.""" + request = self._make_tool_result_request( + [ + {"type": "text", "text": "Here is the screenshot"}, + { + "type": "image", + "source": { + "type": "base64", + "media_type": "image/jpeg", + "data": "QUFB", + }, + }, + ] + ) + result = _convert(request) + + tool_msg = [m for m in result.messages if m["role"] == "tool"] + assert len(tool_msg) == 1 + assert tool_msg[0]["content"] == "Here is the screenshot" + + follow_up = [ + m + for m in result.messages + if m["role"] == "user" and isinstance(m.get("content"), list) + ] + assert len(follow_up) == 1 + assert follow_up[0]["content"][0]["image_url"]["url"] == ( + "data:image/jpeg;base64,QUFB" + ) + + def test_tool_result_with_multiple_images(self): + request = self._make_tool_result_request( + [ + { + "type": "image", + "source": { + "type": "base64", + "media_type": "image/png", + "data": "IMG1", + }, + }, + { + "type": "image", + "source": { + "type": "url", + "url": "https://example.com/img2.jpg", + }, + }, + ] + ) + result = _convert(request) + + follow_up = [ + m + for m in result.messages + if m["role"] == "user" and isinstance(m.get("content"), list) + ] + assert len(follow_up) == 1 + urls = [p["image_url"]["url"] for p in follow_up[0]["content"]] + assert urls == [ + "data:image/png;base64,IMG1", + "https://example.com/img2.jpg", + ] + + def test_tool_result_none_content(self): + request = self._make_tool_result_request(None) + result = _convert(request) + + tool_msg = [m for m in result.messages if m["role"] == "tool"] + assert len(tool_msg) == 1 + assert tool_msg[0]["content"] == "" + + def test_tool_result_no_follow_up_when_no_images(self): + """Ensure no extra user message is added when there are no images.""" + request = self._make_tool_result_request( + [ + {"type": "text", "text": "just text"}, + ] + ) + result = _convert(request) + + user_follow_ups = [ + m + for m in result.messages + if m["role"] == "user" and isinstance(m.get("content"), list) + ] + assert len(user_follow_ups) == 0 + + +# ====================================================================== +# Attribution header stripping +# ====================================================================== + + +class TestAttributionHeaderStripping: + def test_billing_header_stripped_from_system(self): + """Claude Code's x-anthropic-billing-header block should be + stripped to preserve prefix caching.""" + request = _make_request( + [{"role": "user", "content": "Hello"}], + system=[ + {"type": "text", "text": "You are a helpful assistant."}, + { + "type": "text", + "text": "x-anthropic-billing-header: " + "cc_version=2.1.37.abc; cc_entrypoint=cli;", + }, + ], + ) + result = _convert(request) + system_msg = result.messages[0] + assert system_msg["role"] == "system" + assert system_msg["content"] == "You are a helpful assistant." + + def test_system_without_billing_header_unchanged(self): + """Normal system blocks should pass through unchanged.""" + request = _make_request( + [{"role": "user", "content": "Hello"}], + system=[ + {"type": "text", "text": "You are a helpful assistant."}, + {"type": "text", "text": " Be concise."}, + ], + ) + result = _convert(request) + system_msg = result.messages[0] + assert system_msg["content"] == "You are a helpful assistant. Be concise." + + def test_system_string_unchanged(self): + """String system prompts should pass through unchanged.""" + request = _make_request( + [{"role": "user", "content": "Hello"}], + system="You are a helpful assistant.", + ) + result = _convert(request) + system_msg = result.messages[0] + assert system_msg["content"] == "You are a helpful assistant." + + +# ====================================================================== +# Thinking block conversion (Anthropic → OpenAI) +# ====================================================================== + + +class TestThinkingBlockConversion: + """Verify that thinking blocks in assistant messages are correctly + moved to the ``reasoning`` field and stripped from ``content`` during + the Anthropic→OpenAI conversion. + + This is the Anthropic-endpoint path: the client echoes back the full + assistant message (including thinking blocks emitted by vllm) in + subsequent requests. + """ + + def test_thinking_plus_text_in_assistant_message(self): + """thinking + text → reasoning field + plain-string content.""" + request = _make_request( + [ + {"role": "user", "content": "Write me some code."}, + { + "role": "assistant", + "content": [ + { + "type": "thinking", + "thinking": "I should write a simple example.", + "signature": "sig_abc123", + }, + {"type": "text", "text": "Sure! Here is the code."}, + ], + }, + {"role": "user", "content": "Can you fix the bug?"}, + ] + ) + result = _convert(request) + + # Find the assistant message in the converted output. + asst_msgs = [m for m in result.messages if m.get("role") == "assistant"] + assert len(asst_msgs) == 1 + asst = asst_msgs[0] + + # Thinking content must be in reasoning, NOT in content. + assert asst.get("reasoning") == "I should write a simple example." + assert asst.get("content") == "Sure! Here is the code." + + def test_thinking_only_in_assistant_message(self): + """Assistant message with only a thinking block (no visible text). + + This can happen when the model emits reasoning but no final answer + yet (e.g. a mid-turn reasoning step). Content should be None. + """ + request = _make_request( + [ + {"role": "user", "content": "Hello"}, + { + "role": "assistant", + "content": [ + { + "type": "thinking", + "thinking": "Just thinking...", + "signature": "sig_xyz", + } + ], + }, + {"role": "user", "content": "Go on."}, + ] + ) + result = _convert(request) + + asst_msgs = [m for m in result.messages if m.get("role") == "assistant"] + assert len(asst_msgs) == 1 + asst = asst_msgs[0] + + assert asst.get("reasoning") == "Just thinking..." + # No visible text → content should be absent or None. + assert asst.get("content") is None + + def test_thinking_plus_tool_use_in_assistant_message(self): + """thinking + tool_use: reasoning field set, tool_calls populated.""" + request = _make_request( + [ + {"role": "user", "content": "What is 2+2?"}, + { + "role": "assistant", + "content": [ + { + "type": "thinking", + "thinking": "I need to call the calculator.", + "signature": "sig_tool", + }, + { + "type": "tool_use", + "id": "call_001", + "name": "calculator", + "input": {"expression": "2+2"}, + }, + ], + }, + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": "call_001", + "content": "4", + } + ], + }, + ] + ) + result = _convert(request) + + asst_msgs = [m for m in result.messages if m.get("role") == "assistant"] + assert len(asst_msgs) == 1 + asst = asst_msgs[0] + + assert asst.get("reasoning") == "I need to call the calculator." + tool_calls = list(asst.get("tool_calls", [])) + assert len(tool_calls) == 1 + assert tool_calls[0]["function"]["name"] == "calculator" + # No text content alongside reasoning + tool_use. + assert asst.get("content") is None + + def test_multiple_thinking_blocks_concatenated(self): + """Multiple thinking blocks should be joined in order.""" + request = _make_request( + [ + {"role": "user", "content": "Think hard."}, + { + "role": "assistant", + "content": [ + { + "type": "thinking", + "thinking": "First thought. ", + "signature": "s1", + }, + { + "type": "thinking", + "thinking": "Second thought.", + "signature": "s2", + }, + {"type": "text", "text": "Done."}, + ], + }, + ] + ) + result = _convert(request) + + asst_msgs = [m for m in result.messages if m.get("role") == "assistant"] + assert len(asst_msgs) == 1 + asst = asst_msgs[0] + + assert asst.get("reasoning") == "First thought. Second thought." + assert asst.get("content") == "Done." + + def test_no_thinking_blocks_unchanged(self): + """Messages without thinking blocks must not be modified.""" + request = _make_request( + [ + {"role": "user", "content": "Hi"}, + {"role": "assistant", "content": "Hello!"}, + ] + ) + result = _convert(request) + + asst_msgs = [m for m in result.messages if m.get("role") == "assistant"] + assert len(asst_msgs) == 1 + asst = asst_msgs[0] + + assert asst.get("content") == "Hello!" + assert "reasoning" not in asst + + def test_multi_turn_with_thinking_blocks(self): + """Full multi-turn conversation: previous assistant messages that + include thinking blocks must all be converted without a 400 error. + + This is the primary regression scenario from the bug report: + upgrading vllm from v0.15.1 → v0.17.0 introduced thinking-block + support in responses, but echoing those responses back in subsequent + requests caused a Pydantic validation failure. + """ + request = _make_request( + [ + {"role": "user", "content": "Turn 1 question"}, + { + "role": "assistant", + "content": [ + { + "type": "thinking", + "thinking": "Reasoning for turn 1.", + "signature": "s_t1", + }, + {"type": "text", "text": "Answer for turn 1."}, + ], + }, + {"role": "user", "content": "Turn 2 question"}, + { + "role": "assistant", + "content": [ + { + "type": "thinking", + "thinking": "Reasoning for turn 2.", + "signature": "s_t2", + }, + {"type": "text", "text": "Answer for turn 2."}, + ], + }, + {"role": "user", "content": "Turn 3 question"}, + ] + ) + # Must not raise a ValidationError / 400. + result = _convert(request) + + asst_msgs = [m for m in result.messages if m.get("role") == "assistant"] + assert len(asst_msgs) == 2 + + assert asst_msgs[0].get("reasoning") == "Reasoning for turn 1." + assert asst_msgs[0].get("content") == "Answer for turn 1." + assert asst_msgs[1].get("reasoning") == "Reasoning for turn 2." + assert asst_msgs[1].get("content") == "Answer for turn 2." + + def test_redacted_thinking_block_is_accepted(self): + """Anthropic clients may echo back redacted thinking blocks. + + vLLM should accept these blocks (to avoid 400 validation errors) + and ignore them when constructing the OpenAI-format prompt. + """ + request = _make_request( + [ + {"role": "user", "content": "Hello"}, + { + "role": "assistant", + "content": [ + { + "type": "thinking", + "thinking": "Thinking...", + "signature": "sig_think", + }, + { + "type": "redacted_thinking", + "data": "BASE64_OR_OTHER_OPAQUE_DATA", + }, + {"type": "text", "text": "Hi!"}, + ], + }, + {"role": "user", "content": "Continue"}, + ] + ) + result = _convert(request) + + asst_msgs = [m for m in result.messages if m.get("role") == "assistant"] + assert len(asst_msgs) == 1 + asst = asst_msgs[0] + + # Redacted thinking is ignored, normal thinking still becomes reasoning. + assert asst.get("reasoning") == "Thinking..." + assert asst.get("content") == "Hi!" + + +# ====================================================================== +# Cache usage computation +# ====================================================================== + + +class TestGetCachedTokens: + """Tests for _get_cached_tokens helper.""" + + def test_none_usage(self): + assert _get_cached_tokens(None) is None + + def test_no_prompt_tokens_details(self): + usage = UsageInfo(prompt_tokens=100, completion_tokens=10) + assert _get_cached_tokens(usage) is None + + def test_cached_tokens_present(self): + usage = UsageInfo( + prompt_tokens=100, + completion_tokens=10, + prompt_tokens_details=PromptTokenUsageInfo(cached_tokens=80), + ) + assert _get_cached_tokens(usage) == 80 + + def test_cached_tokens_zero(self): + """Zero cached tokens should return 0, not None.""" + usage = UsageInfo( + prompt_tokens=100, + completion_tokens=10, + prompt_tokens_details=PromptTokenUsageInfo(cached_tokens=0), + ) + assert _get_cached_tokens(usage) == 0 + + def test_cached_tokens_none_in_details(self): + usage = UsageInfo( + prompt_tokens=100, + completion_tokens=10, + prompt_tokens_details=PromptTokenUsageInfo(cached_tokens=None), + ) + assert _get_cached_tokens(usage) is None + + +class TestBuildAnthropicUsage: + """Tests for _build_anthropic_usage helper. + + Anthropic defines: total_input = input_tokens + cache_read + cache_creation + vLLM's prompt_tokens is the total. + """ + + def test_no_cache_info(self): + """When cache info is unavailable, return raw prompt_tokens.""" + result = _build_anthropic_usage(100, 10, None) + assert result.input_tokens == 100 + assert result.output_tokens == 10 + assert result.cache_read_input_tokens is None + assert result.cache_creation_input_tokens is None + + def test_cache_hit(self): + """When cache is hit, input_tokens excludes cached tokens.""" + usage = UsageInfo( + prompt_tokens=100, + completion_tokens=10, + prompt_tokens_details=PromptTokenUsageInfo(cached_tokens=80), + ) + result = _build_anthropic_usage(100, 10, usage) + assert result.input_tokens == 20 # 100 - 80 + assert result.output_tokens == 10 + assert result.cache_read_input_tokens == 80 + assert result.cache_creation_input_tokens == 0 + + def test_zero_cached_tokens(self): + """Zero cached tokens should still set cache_creation to 0.""" + usage = UsageInfo( + prompt_tokens=100, + completion_tokens=10, + prompt_tokens_details=PromptTokenUsageInfo(cached_tokens=0), + ) + result = _build_anthropic_usage(100, 10, usage) + assert result.input_tokens == 100 # 100 - 0 + assert result.cache_read_input_tokens == 0 + assert result.cache_creation_input_tokens == 0 + + def test_all_tokens_cached(self): + """When all tokens are cached, input_tokens should be 0.""" + usage = UsageInfo( + prompt_tokens=100, + completion_tokens=10, + prompt_tokens_details=PromptTokenUsageInfo(cached_tokens=100), + ) + result = _build_anthropic_usage(100, 10, usage) + assert result.input_tokens == 0 + assert result.cache_read_input_tokens == 100 + assert result.cache_creation_input_tokens == 0 + + def test_no_prompt_tokens_details(self): + """UsageInfo without prompt_tokens_details returns no cache info.""" + usage = UsageInfo(prompt_tokens=100, completion_tokens=10) + result = _build_anthropic_usage(100, 10, usage) + assert result.input_tokens == 100 + assert result.cache_read_input_tokens is None + assert result.cache_creation_input_tokens is None + + +class TestInlineSystemMessageInMessagesArray: + """Verify that ``role: system`` messages embedded inside the ``messages`` + array are preserved in their original position. + + Unlike the previous approach that merged all system messages into a single + leading system message (breaking prefix caching), this preserves the + conversation structure so KV-cache hits remain intact. + """ + + def test_inline_system_merged_with_top_level_system(self): + """Full integration: inline system + top-level system + user message.""" + request = _make_request( + [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "\n.....\n\n\n", + }, + { + "type": "text", + "text": "help?", + "cache_control": {"type": "ephemeral"}, + }, + ], + }, + { + "role": "system", + "content": ".....", + }, + ], + system=[ + { + "type": "text", + "text": "x-anthropic-billing-header: " + "cc_version=2.1.160.bca; cc_entrypoint=cli; cch=d1d48;", + }, + { + "type": "text", + "text": "You are Claude Code, Anthropic's official CLI for Claude.", + "cache_control": {"type": "ephemeral"}, + }, + { + "type": "text", + "text": "....", + "cache_control": {"type": "ephemeral"}, + }, + ], + tools=[], + ) + + result = _convert(request) + + # First message: top-level system prompt (billing header stripped). + assert result.messages[0]["role"] == "system" + assert ( + result.messages[0]["content"] + == "You are Claude Code, Anthropic's official CLI for Claude." + "...." + ) + + # Second message: user message, content preserved at original position. + assert result.messages[1]["role"] == "user" + user_content = result.messages[1]["content"] + assert len(user_content) == 2 + assert user_content[0] == { + "type": "text", + "text": "\n.....\n\n\n", + } + assert user_content[1] == { + "type": "text", + "text": "help?", + } + + # Third message: inline system stays in original position + # (after user, not merged into leading system). + assert result.messages[2]["role"] == "system" + assert result.messages[2]["content"] == "....." + + def test_inline_system_string_only(self): + """Only an inline system string, no top-level system.""" + request = _make_request( + [ + {"role": "user", "content": "Hello"}, + {"role": "system", "content": "Be concise."}, + ] + ) + result = _convert(request) + + # Inline system stays in its original position. + assert result.messages[0]["role"] == "user" + assert result.messages[0]["content"] == "Hello" + assert result.messages[1]["role"] == "system" + assert result.messages[1]["content"] == "Be concise." + + def test_inline_system_list_content(self): + """Inline system with list content blocks.""" + request = _make_request( + [ + {"role": "user", "content": "Hi"}, + { + "role": "system", + "content": [ + {"type": "text", "text": "Part one. "}, + {"type": "text", "text": "Part two."}, + ], + }, + ] + ) + result = _convert(request) + + # Inline system stays in its original position; + # text blocks are concatenated (same as top-level system). + assert result.messages[0]["role"] == "user" + assert result.messages[0]["content"] == "Hi" + assert result.messages[1]["role"] == "system" + assert result.messages[1]["content"] == "Part one. Part two." + + def test_multiple_inline_system_messages(self): + """Multiple inline system messages each stay in their position.""" + request = _make_request( + [ + {"role": "system", "content": "First system."}, + {"role": "user", "content": "Hello"}, + {"role": "system", "content": "Second system."}, + ] + ) + result = _convert(request) + + # Each system message stays in its original position. + assert result.messages[0]["role"] == "system" + assert result.messages[0]["content"] == "First system." + assert result.messages[1]["role"] == "user" + assert result.messages[1]["content"] == "Hello" + assert result.messages[2]["role"] == "system" + assert result.messages[2]["content"] == "Second system." + + def test_inline_system_with_top_level_string(self): + """Top-level system is a string, inline system is also present.""" + request = _make_request( + [ + {"role": "user", "content": "Hello"}, + {"role": "system", "content": "Inline hint."}, + ], + system="Top-level prompt.", + ) + result = _convert(request) + + # Top-level system goes first; inline system stays in position. + assert result.messages[0]["role"] == "system" + assert result.messages[0]["content"] == "Top-level prompt." + assert result.messages[1]["role"] == "user" + assert result.messages[1]["content"] == "Hello" + assert result.messages[2]["role"] == "system" + assert result.messages[2]["content"] == "Inline hint." + + def test_inline_system_billing_header_stripped(self): + """Inline system that is only a billing header is omitted.""" + request = _make_request( + [ + {"role": "user", "content": "Hello"}, + { + "role": "system", + "content": "x-anthropic-billing-header: cc_version=2.1.160", + }, + {"role": "assistant", "content": "Hi there"}, + ] + ) + result = _convert(request) + + # Billing-header-only system message should be dropped entirely. + assert len(result.messages) == 2 + assert result.messages[0]["role"] == "user" + assert result.messages[1]["role"] == "assistant" + + def test_inline_system_billing_header_mixed_with_content(self): + """Inline system with billing header block + real content.""" + request = _make_request( + [ + {"role": "user", "content": "Hello"}, + { + "role": "system", + "content": [ + { + "type": "text", + "text": "x-anthropic-billing-header: " + "cc_version=2.1.160.bca; cch=d1d48;", + }, + {"type": "text", "text": "Real system content."}, + ], + }, + ] + ) + result = _convert(request) + + # Billing header stripped, real content preserved in position. + assert len(result.messages) == 2 + assert result.messages[0]["role"] == "user" + assert result.messages[0]["content"] == "Hello" + assert result.messages[1]["role"] == "system" + assert result.messages[1]["content"] == "Real system content." + + +# ====================================================================== +# Streaming conversion: message_stream_converter +# ====================================================================== + + +def _make_stream_converter(): + obj = MagicMock(spec=AnthropicServingMessages) + obj.stop_reason_map = { + "stop": "end_turn", + "length": "max_tokens", + "tool_calls": "tool_use", + } + obj.message_stream_converter = ( + AnthropicServingMessages.message_stream_converter.__get__(obj) + ) + return obj + + +def _parse_sse_events(raw_events: list[str]) -> list[tuple[str, dict]]: + results = [] + for raw in raw_events: + headers = dict( + line.split(": ", 1) for line in raw.strip().split("\n") if ": " in line + ) + if "event" in headers and "data" in headers: + results.append((headers["event"], json.loads(headers["data"]))) + return results + + +def _make_stream_chunk( + *, + delta: DeltaMessage | None = None, + finish_reason: str | None = None, + choices: list[ChatCompletionResponseStreamChoice] | None = None, + usage: UsageInfo | None = None, +) -> str: + if choices is None: + choices = [ + ChatCompletionResponseStreamChoice( + index=0, + delta=delta or DeltaMessage(), + finish_reason=finish_reason, + ) + ] + chunk = ChatCompletionStreamResponse( + id="chatcmpl-test", + created=0, + model="test-model", + choices=choices, + usage=usage, + ) + return f"data: {chunk.model_dump_json()}" + + +def _tc(*, args, id=None, name=None): + return DeltaToolCall( + index=0, + id=id, + function=DeltaFunctionCall(name=name, arguments=args), + ) + + +class TestMessageStreamConverterToolUseContentBuffering: + """Regression test for tool_use arguments being silently dropped. + + With speculative decoding or multi-token prediction, a single delta + can carry both the final tool_call argument fragment and trailing + content. + """ + + @pytest.mark.asyncio + async def test_tool_use_args_not_dropped_when_content_in_same_chunk( + self, + ): + async def sse_input(): + yield _make_stream_chunk( + delta=DeltaMessage(role="assistant"), + usage=UsageInfo(prompt_tokens=10, total_tokens=10), + ) + yield _make_stream_chunk( + delta=DeltaMessage( + tool_calls=[ + _tc(id="call_abc123", name="read_file", args=""), + ] + ) + ) + yield _make_stream_chunk( + delta=DeltaMessage( + tool_calls=[ + _tc(args='{"path":"/tmp/f"'), + ] + ) + ) + # BUG TRIGGER: final tool_call args and trailing content in + # one delta, as happens with spec decoding / multi-token + # prediction where multiple tokens land in a single chunk. + yield _make_stream_chunk( + delta=DeltaMessage( + content="\nOkay", + tool_calls=[_tc(args="}")], + ) + ) + yield _make_stream_chunk(finish_reason="tool_calls") + yield _make_stream_chunk( + choices=[], + usage=UsageInfo( + prompt_tokens=10, + total_tokens=30, + completion_tokens=20, + ), + ) + yield "data: [DONE]" + + converter = _make_stream_converter() + output = [] + async for event in converter.message_stream_converter(sse_input()): + output.append(event) + + events = _parse_sse_events(output) + + assert events[0][0] == "message_start" + + arg_fragments = [ + data["delta"]["partial_json"] + for _, data in events + if data.get("delta", {}).get("type") == "input_json_delta" + ] + full_args = "".join(arg_fragments) + assert full_args == '{"path":"/tmp/f"}' + + text_deltas = [ + data["delta"]["text"] + for _, data in events + if data.get("delta", {}).get("type") == "text_delta" + ] + assert text_deltas == ["\nOkay"] + + block_starts = [ + (data["content_block"]["type"], data.get("index")) + for ev_type, data in events + if ev_type == "content_block_start" + ] + assert block_starts[0] == ("tool_use", 0) + assert block_starts[1] == ("text", 1) + + msg_deltas = [data for ev_type, data in events if ev_type == "message_delta"] + assert msg_deltas[0]["delta"]["stop_reason"] == "tool_use" + + assert events[-1][0] == "message_stop" + + @pytest.mark.asyncio + async def test_buffered_content_flushed_on_done_without_usage_chunk(self): + """Content buffered during tool_use must be emitted even if the + stream jumps straight from finish_reason to [DONE], skipping the + empty-choices usage chunk.""" + + async def sse_input(): + yield _make_stream_chunk( + delta=DeltaMessage(role="assistant"), + usage=UsageInfo(prompt_tokens=10, total_tokens=10), + ) + yield _make_stream_chunk( + delta=DeltaMessage( + tool_calls=[ + _tc(id="call_xyz", name="get_weather", args=""), + ] + ) + ) + yield _make_stream_chunk( + delta=DeltaMessage( + tool_calls=[_tc(args='{"city":"NYC"}')], + ) + ) + yield _make_stream_chunk( + delta=DeltaMessage(content="\nDone"), + finish_reason="tool_calls", + ) + # No empty-choices usage chunk — go straight to [DONE]. + yield "data: [DONE]" + + converter = _make_stream_converter() + output = [] + async for event in converter.message_stream_converter(sse_input()): + output.append(event) + + events = _parse_sse_events(output) + + text_deltas = [ + data["delta"]["text"] + for _, data in events + if data.get("delta", {}).get("type") == "text_delta" + ] + assert text_deltas == ["\nDone"] + + block_starts = [ + data["content_block"]["type"] + for ev_type, data in events + if ev_type == "content_block_start" + ] + assert "tool_use" in block_starts + assert "text" in block_starts + + assert events[-1][0] == "message_stop" + + +class TestMessageStartIncludesTypeAndRole: + """Regression test for issue #45367: the streaming message_start event is + serialized with exclude_unset=True, which silently dropped the + default-valued ``type``/``role`` fields of the nested message object. + Strict Anthropic SDK clients (e.g. Claude Code) validate + ``message_start.message.type``/``role`` and reject the whole stream when + they are missing. + """ + + @pytest.mark.asyncio + async def test_message_start_contains_message_type_and_role(self): + async def sse_input(): + yield _make_stream_chunk( + delta=DeltaMessage(content="Hello"), + usage=UsageInfo( + prompt_tokens=20, + total_tokens=20, + completion_tokens=0, + ), + ) + yield _make_stream_chunk(finish_reason="stop") + yield "data: [DONE]" + + converter = _make_stream_converter() + output = [] + async for event in converter.message_stream_converter(sse_input()): + output.append(event) + + events = _parse_sse_events(output) + + assert events[0][0] == "message_start" + message = events[0][1]["message"] + assert message["type"] == "message" + assert message["role"] == "assistant" + + +class TestStreamingCacheUsageSemantics: + """Locks in the documented streaming behavior of cache usage fields. + + vLLM's OpenAI chat completion streaming only attaches + ``prompt_tokens_details`` to the terminal usage chunk. The Anthropic layer + mirrors that contract: cache fields are omitted on ``message_start`` (key + absence signals "unknown") and populated on ``message_delta`` (the final + cumulative count). This is intentionally consistent with vLLM's OpenAI + behavior, even though Anthropic's upstream API populates cache fields on + ``message_start``; closing that gap requires plumbing cache info into the + first chunk at the OpenAI layer, which is out of scope here. + """ + + @pytest.mark.asyncio + async def test_streaming_cache_fields_absent_then_populated(self): + """First chunk lacks prompt_tokens_details (vLLM contract); + message_start omits cache fields. The final chunk carries + prompt_tokens_details, so message_delta carries resolved values.""" + + async def sse_input(): + yield _make_stream_chunk( + delta=DeltaMessage(role="assistant", content="hi"), + usage=UsageInfo(prompt_tokens=100, total_tokens=100), + ) + yield _make_stream_chunk(finish_reason="stop") + yield _make_stream_chunk( + choices=[], + usage=UsageInfo( + prompt_tokens=100, + completion_tokens=5, + total_tokens=105, + prompt_tokens_details=PromptTokenUsageInfo(cached_tokens=80), + ), + ) + yield "data: [DONE]" + + converter = _make_stream_converter() + output = [] + async for event in converter.message_stream_converter(sse_input()): + output.append(event) + events = _parse_sse_events(output) + + # message_start: cache fields unknown → omitted from JSON entirely. + start_usage = events[0][1]["message"]["usage"] + assert events[0][0] == "message_start" + assert start_usage["input_tokens"] == 100 + assert "cache_read_input_tokens" not in start_usage + assert "cache_creation_input_tokens" not in start_usage + + # message_delta: authoritative usage with cache fields populated. + delta_usage = next( + data["usage"] for ev, data in events if ev == "message_delta" + ) + assert delta_usage["input_tokens"] == 20 # 100 - 80 + assert delta_usage["cache_read_input_tokens"] == 80 + assert delta_usage["cache_creation_input_tokens"] == 0 + + @pytest.mark.asyncio + async def test_streaming_no_cache_hit(self): + """When the final chunk reports cached_tokens=0, message_delta carries + cache fields = 0 (cache miss); message_start still omits them.""" + + async def sse_input(): + yield _make_stream_chunk( + delta=DeltaMessage(role="assistant"), + usage=UsageInfo(prompt_tokens=50, total_tokens=50), + ) + yield _make_stream_chunk(finish_reason="stop") + yield _make_stream_chunk( + choices=[], + usage=UsageInfo( + prompt_tokens=50, + completion_tokens=5, + total_tokens=55, + prompt_tokens_details=PromptTokenUsageInfo(cached_tokens=0), + ), + ) + yield "data: [DONE]" + + converter = _make_stream_converter() + output = [] + async for event in converter.message_stream_converter(sse_input()): + output.append(event) + events = _parse_sse_events(output) + + start_usage = events[0][1]["message"]["usage"] + delta_usage = next( + data["usage"] for ev, data in events if ev == "message_delta" + ) + assert start_usage["input_tokens"] == 50 + assert "cache_read_input_tokens" not in start_usage + assert "cache_creation_input_tokens" not in start_usage + assert delta_usage["input_tokens"] == 50 # 50 - 0 + assert delta_usage["cache_read_input_tokens"] == 0 + assert delta_usage["cache_creation_input_tokens"] == 0 + + @pytest.mark.asyncio + async def test_streaming_no_prompt_tokens_details_at_all(self): + """If --enable-prompt-tokens-details is off, no chunk carries cache + info; both message_start and message_delta omit cache fields.""" + + async def sse_input(): + yield _make_stream_chunk( + delta=DeltaMessage(role="assistant"), + usage=UsageInfo(prompt_tokens=30, total_tokens=30), + ) + yield _make_stream_chunk(finish_reason="stop") + yield _make_stream_chunk( + choices=[], + usage=UsageInfo(prompt_tokens=30, completion_tokens=2, total_tokens=32), + ) + yield "data: [DONE]" + + converter = _make_stream_converter() + output = [] + async for event in converter.message_stream_converter(sse_input()): + output.append(event) + events = _parse_sse_events(output) + + start_usage = events[0][1]["message"]["usage"] + delta_usage = next( + data["usage"] for ev, data in events if ev == "message_delta" + ) + assert "cache_read_input_tokens" not in start_usage + assert "cache_creation_input_tokens" not in start_usage + assert "cache_read_input_tokens" not in delta_usage + assert "cache_creation_input_tokens" not in delta_usage + + +# ====================================================================== +# Auto-detection of system-first template requirement +# ====================================================================== + + +Q35_TEMPLATE = ( + "{%- for message in messages %}" + "{%- if message.role == 'system' %}" + "{%- if not loop.first %}" + "{{- raise_exception('System message must be at the beginning.') }}" + "{%- endif %}" + "{%- endif %}" + "{%- endfor %}" +) + + +class TestDetectMergeInlineSystem: + """Verify _detect_merge_inline_system auto-detection. + + Tests three scenarios: + 1. Template with system-first guard (e.g. Qwen) → merge needed + 2. Template without restrictions → no merge, cache-friendly + 3. No template provided → safe default: merge + """ + + def test_qwen_template_requires_merge(self): + """Template with loop.first guard rejects mid-conversation system.""" + assert ( + AnthropicServingMessages._detect_merge_inline_system(Q35_TEMPLATE) is True + ) + + def test_no_restriction_no_merge(self): + """Template without restriction accepts mid-conversation system.""" + assert ( + AnthropicServingMessages._detect_merge_inline_system( + "{%- for message in messages %}" + "{{- message.role }}: {{ message.content }}\n" + "{%- endfor %}" + ) + is False + ) + + def test_no_template_defaults_merge(self): + """No chat_template → conservative default: merge.""" + assert AnthropicServingMessages._detect_merge_inline_system(None) is True + + +# ====================================================================== +# Full (non-streaming) response conversion: messages_full_converter +# ====================================================================== + + +def _make_full_converter(): + obj = MagicMock(spec=AnthropicServingMessages) + obj.messages_full_converter = ( + AnthropicServingMessages.messages_full_converter.__get__(obj) + ) + return obj + + +class TestMessagesFullConverter: + def test_empty_completion_emits_one_text_block(self): + """An empty completion still yields exactly one (empty) text block.""" + generator = ChatCompletionResponse( + id="chatcmpl-empty", + model="test-model", + choices=[ + ChatCompletionResponseChoice( + index=0, + message=ChatMessage(role="assistant", content=None), + finish_reason="stop", + ) + ], + usage=UsageInfo(prompt_tokens=10, completion_tokens=0, total_tokens=10), + ) + + result = _make_full_converter().messages_full_converter(generator) + + assert len(result.content) == 1 + assert result.content[0].type == "text" + assert result.content[0].text == "" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/anthropic/test_messages.py b/asian_ai/performance/vLLM/tests/entrypoints/anthropic/test_messages.py new file mode 100644 index 0000000000000000000000000000000000000000..0d6749b753676f4c73e487147fd7f7250e6b6405 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/anthropic/test_messages.py @@ -0,0 +1,193 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import anthropic +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer + +MODEL_NAME = "Qwen/Qwen3-0.6B" + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--max-model-len", + "2048", + "--enforce-eager", + "--enable-auto-tool-choice", + "--tool-call-parser", + "hermes", + "--served-model-name", + "claude-3-7-sonnet-latest", + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client_anthropic() as async_client: + yield async_client + + +@pytest.mark.asyncio +async def test_simple_messages(client: anthropic.AsyncAnthropic): + resp = await client.messages.create( + model="claude-3-7-sonnet-latest", + max_tokens=1024, + messages=[{"role": "user", "content": "how are you!"}], + ) + assert resp.stop_reason == "end_turn" + assert resp.role == "assistant" + + print(f"Anthropic response: {resp.model_dump_json()}") + + +@pytest.mark.asyncio +async def test_system_message(client: anthropic.AsyncAnthropic): + resp = await client.messages.create( + model="claude-3-7-sonnet-latest", + max_tokens=1024, + system="you are a helpful assistant", + messages=[{"role": "user", "content": "how are you!"}], + ) + assert resp.stop_reason == "end_turn" + assert resp.role == "assistant" + + print(f"Anthropic response: {resp.model_dump_json()}") + + +@pytest.mark.asyncio +async def test_anthropic_streaming(client: anthropic.AsyncAnthropic): + resp = await client.messages.create( + model="claude-3-7-sonnet-latest", + max_tokens=1024, + messages=[{"role": "user", "content": "how are you!"}], + stream=True, + ) + + first_chunk = None + chunk_count = 0 + async for chunk in resp: + chunk_count += 1 + if first_chunk is None and chunk.type == "message_start": + first_chunk = chunk + print(chunk.model_dump_json()) + + assert chunk_count > 0 + assert first_chunk is not None, "message_start chunk was never observed" + assert first_chunk.message is not None, "first chunk should include message" + assert first_chunk.message.usage is not None, ( + "first chunk should include usage stats" + ) + assert first_chunk.message.usage.output_tokens == 0 + assert first_chunk.message.usage.input_tokens > 5 + + +@pytest.mark.asyncio +async def test_anthropic_tool_call(client: anthropic.AsyncAnthropic): + resp = await client.messages.create( + model="claude-3-7-sonnet-latest", + max_tokens=1024, + messages=[ + {"role": "user", "content": "What's the weather like in New York today?"} + ], + tools=[ + { + "name": "get_current_weather", + "description": "Useful for querying the weather in a specified city.", + "input_schema": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "City or region, for example: " + "New York, London, Tokyo, etc.", + } + }, + "required": ["location"], + }, + } + ], + stream=False, + ) + assert resp.stop_reason == "tool_use" + assert resp.role == "assistant" + + print(f"Anthropic response: {resp.model_dump_json()}") + + +@pytest.mark.asyncio +async def test_anthropic_tool_call_streaming(client: anthropic.AsyncAnthropic): + resp = await client.messages.create( + model="claude-3-7-sonnet-latest", + max_tokens=1024, + messages=[ + { + "role": "user", + "content": "What's the weather like in New York today?", + } + ], + tools=[ + { + "name": "get_current_weather", + "description": "Useful for querying the weather in a specified city.", + "input_schema": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "City or region, for example: " + "New York, London, Tokyo, etc.", + } + }, + "required": ["location"], + }, + } + ], + stream=True, + ) + + async for chunk in resp: + print(chunk.model_dump_json()) + + +@pytest.mark.asyncio +async def test_anthropic_structured_output(client: anthropic.AsyncAnthropic): + response = await client.messages.create( + model="claude-3-7-sonnet-latest", + max_tokens=1024, + messages=[ + { + "role": "user", + "content": "Extract the key information from this email:" + "John Smith (john@example.com) is interested in our " + "Enterprise plan and wants to schedule a demo for next Tuesday at 2pm.", + } + ], + output_config={ + "format": { + "type": "json_schema", + "schema": { + "type": "object", + "properties": { + "name": {"type": "string"}, + "email": {"type": "string"}, + "plan_interest": {"type": "string"}, + "demo_requested": {"type": "boolean"}, + }, + "required": ["name", "email", "plan_interest", "demo_requested"], + "additionalProperties": False, + }, + } + }, + ) + print(response.content[0].text) + json_obj = json.loads(response.content[0].text) + for key in ["name", "email", "plan_interest", "demo_requested"]: + assert key in json_obj, f"Missing key in output: {key}" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/anthropic/test_protocol_exports.py b/asian_ai/performance/vLLM/tests/entrypoints/anthropic/test_protocol_exports.py new file mode 100644 index 0000000000000000000000000000000000000000..c391d6985346f4cbd654778311d6621bf0cb14d4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/anthropic/test_protocol_exports.py @@ -0,0 +1,50 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Regression tests for Anthropic protocol exports used by serving. + +Guards against Docker/nightly images shipping a stale protocol module that is +missing symbols imported by ``vllm.entrypoints.anthropic.serving`` (issue #44759). +""" + +import pytest + +from vllm.entrypoints.anthropic.protocol import ( + AnthropicContentBlock, + AnthropicContextManagement, + AnthropicCountTokensRequest, + AnthropicCountTokensResponse, + AnthropicDelta, + AnthropicError, + AnthropicMessagesRequest, + AnthropicMessagesResponse, + AnthropicOutputConfig, + AnthropicStreamEvent, + AnthropicUsage, +) + +pytestmark = pytest.mark.skip_global_cleanup + +SERVING_PROTOCOL_EXPORTS = ( + AnthropicContentBlock, + AnthropicContextManagement, + AnthropicCountTokensRequest, + AnthropicCountTokensResponse, + AnthropicDelta, + AnthropicError, + AnthropicMessagesRequest, + AnthropicMessagesResponse, + AnthropicOutputConfig, + AnthropicStreamEvent, + AnthropicUsage, +) + + +def test_serving_protocol_exports_are_importable(): + for export in SERVING_PROTOCOL_EXPORTS: + assert export is not None + + +def test_anthropic_output_config_instantiation(): + config = AnthropicOutputConfig() + assert config.effort is None + assert config.format is None diff --git a/asian_ai/performance/vLLM/tests/entrypoints/conftest.py b/asian_ai/performance/vLLM/tests/entrypoints/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..54f67f621726a6f10bcd9787a405925bac8d6490 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/conftest.py @@ -0,0 +1,219 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + + +@pytest.fixture +def sample_prompts(): + return [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + "The future of AI is", + ] + + +@pytest.fixture +def sample_token_ids(): + return [ + [0], + [0, 1], + [0, 2, 1], + [0, 3, 1, 2], + ] + + +@pytest.fixture +def sample_regex(): + return ( + r"((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)\.){3}" + r"(25[0-5]|(2[0-4]|1\d|[1-9]|)\d)" + ) + + +@pytest.fixture +def sample_json_schema(): + return { + "type": "object", + "properties": { + "name": {"type": "string"}, + "age": {"type": "integer"}, + "skills": { + "type": "array", + "items": {"type": "string", "maxLength": 10}, + "minItems": 3, + }, + "work_history": { + "type": "array", + "items": { + "type": "object", + "properties": { + "company": {"type": "string"}, + "duration": {"type": "number"}, + "position": {"type": "string"}, + }, + "required": ["company", "position"], + }, + }, + }, + "required": ["name", "age", "skills", "work_history"], + } + + +@pytest.fixture +def sample_complex_json_schema(): + return { + "type": "object", + "properties": { + "score": { + "type": "integer", + "minimum": 0, + "maximum": 100, # Numeric range + }, + "grade": { + "type": "string", + "pattern": "^[A-D]$", # Regex pattern + }, + "email": { + "type": "string", + "pattern": "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$", + }, + "tags": { + "type": "array", + "items": { + "type": "string", + # Combining length and pattern restrictions + "pattern": "^[a-z]{1,10}$", + }, + }, + }, + "required": ["score", "grade", "email", "tags"], + } + + +@pytest.fixture +def sample_definition_json_schema(): + return { + "$defs": { + "Step": { + "properties": { + "explanation": {"title": "Explanation", "type": "string"}, + "output": {"title": "Output", "type": "string"}, + }, + "required": ["explanation", "output"], + "title": "Step", + "type": "object", + } + }, + "properties": { + "steps": { + "items": {"$ref": "#/$defs/Step"}, + "title": "Steps", + "type": "array", + }, + "final_answer": {"title": "Final Answer", "type": "string"}, + }, + "required": ["steps", "final_answer"], + "title": "MathReasoning", + "type": "object", + } + + +@pytest.fixture +def sample_enum_json_schema(): + return { + "type": "object", + "properties": { + "status": { + "type": "string", + "enum": ["active", "inactive", "pending"], # Literal values using enum + }, + "priority": { + "type": "string", + "enum": ["low", "medium", "high", "critical"], + }, + "category": { + "type": "object", + "properties": { + "type": { + "type": "string", + "enum": ["bug", "feature", "improvement"], + }, + "severity": { + "type": "integer", + "enum": [1, 2, 3, 4, 5], # Enum can also contain numbers + }, + }, + "required": ["type", "severity"], + }, + "flags": { + "type": "array", + "items": { + "type": "string", + "enum": ["urgent", "blocked", "needs_review", "approved"], + }, + }, + }, + "required": ["status", "priority", "category", "flags"], + } + + +@pytest.fixture +def sample_structured_outputs_choices(): + return [ + "Python", + "Java", + "JavaScript", + "C++", + "C#", + "PHP", + "TypeScript", + "Ruby", + "Swift", + "Kotlin", + ] + + +@pytest.fixture +def sample_sql_statements(): + return """ +start: select_statement +select_statement: "SELECT" column "from" table "where" condition +column: "col_1" | "col_2" +table: "table_1" | "table_2" +condition: column "=" number +number: "1" | "2" +""" + + +@pytest.fixture(scope="session") +def qwen3_lora_files(): + """Download Qwen3 LoRA files once per test session.""" + from huggingface_hub import snapshot_download + + return snapshot_download(repo_id="charent/self_cognition_Alice") + + +@pytest.fixture(scope="session") +def qwen3_meowing_lora_files(): + """Download Qwen3 LoRA files once per test session.""" + from huggingface_hub import snapshot_download + + return snapshot_download(repo_id="Jackmin108/Qwen3-0.6B-Meow-LoRA") + + +@pytest.fixture(scope="session") +def qwen3_woofing_lora_files(): + """Download Qwen3 LoRA files once per test session.""" + from huggingface_hub import snapshot_download + + return snapshot_download(repo_id="Jackmin108/Qwen3-0.6B-Woof-LoRA") + + +@pytest.fixture(scope="session") +def opt125_lora_files() -> str: + """Download opt-125m LoRA files once per test session.""" + from huggingface_hub import snapshot_download + + return snapshot_download(repo_id="peft-internal-testing/opt-125m-dummy-lora") diff --git a/asian_ai/performance/vLLM/tests/entrypoints/generate/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/generate/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/generate/generative_scoring/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/generate/generative_scoring/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..0643de4834f45beb52db0a52ff79be718f6ec2dc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/generate/generative_scoring/__init__.py @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project diff --git a/asian_ai/performance/vLLM/tests/entrypoints/generate/generative_scoring/test_generative_scoring.py b/asian_ai/performance/vLLM/tests/entrypoints/generate/generative_scoring/test_generative_scoring.py new file mode 100644 index 0000000000000000000000000000000000000000..7e047c67901ed112f217479f77d6e833663789c4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/generate/generative_scoring/test_generative_scoring.py @@ -0,0 +1,323 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the Generative Scoring API. + +Tests cover: +1. Protocol models (request/response construction) +2. Probability computation (softmax normalization) +3. Input validation +4. Score formula: P(token[0]) / (P(token[0]) + P(token[1])) +5. Prompt building and item ordering +""" + +import math +from dataclasses import dataclass, field +from typing import Any +from unittest.mock import MagicMock + +import pytest + +from vllm.config.multimodal import MultiModalConfig +from vllm.entrypoints.generate.generative_scoring.serving import ( + GenerativeScoringItemResult, + GenerativeScoringRequest, + GenerativeScoringResponse, + ServingGenerativeScoring, +) +from vllm.entrypoints.openai.engine.protocol import ErrorResponse +from vllm.entrypoints.openai.models.protocol import BaseModelPath +from vllm.entrypoints.openai.models.serving import OpenAIServingModels +from vllm.logprobs import Logprob +from vllm.outputs import CompletionOutput, RequestOutput +from vllm.tokenizers import get_tokenizer +from vllm.v1.engine.async_llm import AsyncLLM + +MODEL_NAME = "Qwen/Qwen3-0.6B" +BASE_MODEL_PATHS = [BaseModelPath(name=MODEL_NAME, model_path=MODEL_NAME)] + + +@dataclass +class MockHFConfig: + model_type: str = "any" + + +@dataclass +class MockModelConfig: + task = "generate" + runner_type = "generate" + tokenizer = MODEL_NAME + trust_remote_code = False + tokenizer_mode = "auto" + max_model_len = 100 + tokenizer_revision = None + multimodal_config = MultiModalConfig() + hf_config = MockHFConfig() + logits_processor_pattern = None + logits_processors: list[str] | None = None + diff_sampling_param: dict | None = None + allowed_local_media_path: str = "" + allowed_media_domains: list[str] | None = None + encoder_config = None + generation_config: str = "auto" + media_io_kwargs: dict[str, dict[str, Any]] = field(default_factory=dict) + skip_tokenizer_init = False + vocab_size = 151936 + + def get_diff_sampling_param(self): + return self.diff_sampling_param or {} + + def get_vocab_size(self): + return self.vocab_size + + +def _create_mock_engine(): + """Create a mock AsyncLLM engine.""" + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.get_tokenizer.return_value = get_tokenizer(MODEL_NAME) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + + mock_renderer = MagicMock() + mock_renderer.tokenizer = get_tokenizer(MODEL_NAME) + mock_engine.renderer = mock_renderer + + return mock_engine + + +def _create_serving(mock_engine) -> ServingGenerativeScoring: + """Create an ServingGenerativeScoring instance with mocks.""" + models = OpenAIServingModels( + engine_client=mock_engine, + base_model_paths=BASE_MODEL_PATHS, + ) + return ServingGenerativeScoring(mock_engine, models, request_logger=None) + + +def _create_mock_request_output(logprobs_dict: dict[int, float]) -> RequestOutput: + """Create a mock RequestOutput with specified logprobs.""" + logprobs_with_objs = { + tid: Logprob(logprob=lp, rank=i + 1) + for i, (tid, lp) in enumerate(logprobs_dict.items()) + } + completion_output = CompletionOutput( + index=0, + text="", + token_ids=[100], + cumulative_logprob=-1.0, + logprobs=[logprobs_with_objs], + finish_reason="length", + ) + return RequestOutput( + request_id="test-request", + prompt="test prompt", + prompt_token_ids=[1, 2, 3], + prompt_logprobs=None, + outputs=[completion_output], + finished=True, + ) + + +class TestProtocolModels: + """Tests for GenerativeScoringRequest and GenerativeScoringResponse.""" + + def test_request_and_response_all_fields(self): + """Test request construction with all field types and response structure.""" + # Test request with string inputs + req_str = GenerativeScoringRequest( + query="Is this the capital?", + items=["Paris", "London"], + label_token_ids=[9454, 2753], + ) + assert req_str.query == "Is this the capital?" + assert req_str.items == ["Paris", "London"] + assert req_str.label_token_ids == [9454, 2753] + assert req_str.apply_softmax is True # default + assert req_str.item_first is False # default + assert req_str.add_special_tokens is True # default + + # Test request with pre-tokenized inputs and custom options + req_tok = GenerativeScoringRequest( + query=[100, 200, 300], + items=[[400, 500], [600, 700]], + label_token_ids=[1234, 5678], + apply_softmax=False, + item_first=True, + add_special_tokens=False, + ) + assert req_tok.query == [100, 200, 300] + assert req_tok.items == [[400, 500], [600, 700]] + assert req_tok.apply_softmax is False + assert req_tok.item_first is True + assert req_tok.add_special_tokens is False + + # Test response structure + response = GenerativeScoringResponse( + model="test-model", + data=[ + GenerativeScoringItemResult(index=0, score=0.7), + GenerativeScoringItemResult(index=1, score=0.4), + ], + usage={"prompt_tokens": 10, "total_tokens": 12, "completion_tokens": 2}, + ) + assert response.object == "list" + assert response.model == "test-model" + assert len(response.data) == 2 + assert response.data[0].score == 0.7 + assert response.data[0].object == "score" + assert response.data[1].score == 0.4 + assert response.usage.prompt_tokens == 10 + + +class TestProbabilityComputation: + """Tests for _compute_probabilities with both softmax modes.""" + + @pytest.mark.parametrize( + "label_logprobs,apply_softmax,should_sum_to_one", + [ + ({100: -1.0, 200: -2.0}, True, True), + ({100: -100.0, 200: -100.5}, True, True), # numerical stability + ({100: -1.0, 200: -2.0}, False, False), + ], + ids=["softmax_basic", "softmax_extreme_values", "true_probs"], + ) + def test_compute_probabilities( + self, label_logprobs, apply_softmax, should_sum_to_one + ): + """Test probability computation for softmax and true probability modes.""" + serving = ServingGenerativeScoring.__new__(ServingGenerativeScoring) + probs = serving._compute_probabilities( + label_logprobs, apply_softmax=apply_softmax + ) + + # Verify sum behavior + total = sum(probs.values()) + if should_sum_to_one: + assert abs(total - 1.0) < 1e-6 + else: + assert total < 1.0 + + # Verify math + if apply_softmax: + max_lp = max(label_logprobs.values()) + exp_vals = {k: math.exp(v - max_lp) for k, v in label_logprobs.items()} + sum_exp = sum(exp_vals.values()) + for tid, lp in label_logprobs.items(): + assert abs(probs[tid] - exp_vals[tid] / sum_exp) < 1e-9 + else: + for tid, lp in label_logprobs.items(): + assert abs(probs[tid] - math.exp(lp)) < 1e-9 + + def test_score_formula(self): + """Test the score formula: P(token[0]) / (P(token[0]) + P(token[1])).""" + serving = ServingGenerativeScoring.__new__(ServingGenerativeScoring) + + # With logprobs -0.5 and -2.0, softmax gives higher prob to first token + logprobs = {9454: -0.5, 2753: -2.0} + probs = serving._compute_probabilities(logprobs, apply_softmax=True) + + # Score = P(9454) / (P(9454) + P(2753)) = P(9454) since they sum to 1 + score = probs[9454] + + # Manual calculation + exp_0 = math.exp(-0.5) + exp_1 = math.exp(-2.0) + expected_score = exp_0 / (exp_0 + exp_1) + + assert abs(score - expected_score) < 1e-9 + assert score > 0.5 # First token has higher logprob, so higher probability + + +class TestValidation: + """Tests for input validation errors.""" + + @pytest.mark.asyncio + @pytest.mark.parametrize( + "request_kwargs,expected_error", + [ + ( + {"query": "q", "items": ["i"], "label_token_ids": [999999, 999998]}, + "out of vocabulary", + ), + ( + {"query": "q", "items": [], "label_token_ids": [100, 200]}, + "at least one item", + ), + ], + ids=["invalid_token_id", "empty_items"], + ) + async def test_validation_errors(self, request_kwargs, expected_error): + """Test that invalid inputs return appropriate errors.""" + mock_engine = _create_mock_engine() + serving = _create_serving(mock_engine) + request = GenerativeScoringRequest(model=MODEL_NAME, **request_kwargs) + result = await serving.create_generative_scoring(request, None) + + assert isinstance(result, ErrorResponse) + assert expected_error in result.error.message.lower() + + +class TestPromptBuilding: + """Tests for prompt construction and item ordering.""" + + @pytest.mark.asyncio + @pytest.mark.parametrize( + "item_first,expected", + [ + (False, [[100, 101, 200, 201], [100, 101, 300, 301]]), # query + item + (True, [[200, 201, 100, 101], [300, 301, 100, 101]]), # item + query + ], + ids=["query_first", "item_first"], + ) + async def test_item_ordering(self, item_first, expected): + """Test that item_first flag controls prompt concatenation order.""" + mock_engine = _create_mock_engine() + serving = _create_serving(mock_engine) + + request = GenerativeScoringRequest( + query=[100, 101], + items=[[200, 201], [300, 301]], + label_token_ids=[500, 501], + item_first=item_first, + ) + engine_inputs, _ = await serving._build_prompts( + request, MagicMock(), max_model_len=4096 + ) + + for i, exp in enumerate(expected): + assert engine_inputs[i]["prompt_token_ids"] == exp + + +class TestGeneration: + """Tests for the full generation flow with mocked engine.""" + + @pytest.mark.asyncio + async def test_successful_generation(self): + """Test successful score generation returns valid response.""" + mock_engine = _create_mock_engine() + serving = _create_serving(mock_engine) + + mock_logprobs = {1234: -0.5, 5678: -2.0, 100: -3.0} + mock_output = _create_mock_request_output(mock_logprobs) + + async def mock_generate(*args, **kwargs): + yield mock_output + + mock_engine.generate = mock_generate + + request = GenerativeScoringRequest( + model=MODEL_NAME, + query="Is Paris the capital?", + items=["Yes", "No"], + label_token_ids=[1234, 5678], + ) + result = await serving.create_generative_scoring(request, None) + + assert isinstance(result, GenerativeScoringResponse) + assert len(result.data) == 2 + for item_result in result.data: + assert 0.0 <= item_result.score <= 1.0 + + +if __name__ == "__main__": + pytest.main([__file__, "-v"]) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/generate/generative_scoring/test_generative_scoring_e2e.py b/asian_ai/performance/vLLM/tests/entrypoints/generate/generative_scoring/test_generative_scoring_e2e.py new file mode 100644 index 0000000000000000000000000000000000000000..bcd5ecdb506a09fb74b4805a04ee9975268611dc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/generate/generative_scoring/test_generative_scoring_e2e.py @@ -0,0 +1,157 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""End-to-end tests for the Generative Scoring API. + +Tests verify the full HTTP request/response flow using RemoteOpenAIServer. +""" + +import pytest +import requests + +from tests.utils import RemoteOpenAIServer + +MODEL_NAME = "Qwen/Qwen3-0.6B" + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "512", + "--enforce-eager", + "--max-num-seqs", + "32", + ] + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +class TestGenerativeScoringAPI: + """End-to-end tests for the Generative Scoring API.""" + + @pytest.mark.asyncio + async def test_basic_score_and_response_structure(self, server: RemoteOpenAIServer): + """Test basic generative scoring request and verify response structure.""" + response = requests.post( + server.url_for("generative_scoring"), + json={ + "model": MODEL_NAME, + "query": "Is Paris the capital of France? Answer Yes or No: ", + "items": ["Paris is beautiful.", "London is rainy."], + "label_token_ids": [9454, 2753], + }, + ) + assert response.status_code == 200, f"Response: {response.text}" + data = response.json() + + # Verify response structure + assert data["id"].startswith("generative-scoring-") + assert data["object"] == "list" + assert "model" in data + assert "usage" in data + assert len(data["data"]) == 2 + + # Verify each result + for i, result in enumerate(data["data"]): + assert result["index"] == i + assert result["object"] == "score" + assert 0.0 <= result["score"] <= 1.0 + + # Verify usage tracking + usage = data["usage"] + assert usage["prompt_tokens"] > 0 + assert usage["completion_tokens"] > 0 + assert ( + usage["total_tokens"] == usage["prompt_tokens"] + usage["completion_tokens"] + ) + + @pytest.mark.asyncio + async def test_multiple_items(self, server: RemoteOpenAIServer): + """Test generative scoring request with multiple items.""" + response = requests.post( + server.url_for("generative_scoring"), + json={ + "model": MODEL_NAME, + "query": "Is this city a capital? ", + "items": ["Paris", "London", "Berlin", "New York", "Tokyo"], + "label_token_ids": [9454, 2753], + }, + ) + assert response.status_code == 200 + data = response.json() + assert len(data["data"]) == 5 + + @pytest.mark.asyncio + async def test_validation_missing_label_token_ids(self, server: RemoteOpenAIServer): + """Test that missing label_token_ids returns a validation error.""" + response = requests.post( + server.url_for("generative_scoring"), + json={ + "model": MODEL_NAME, + "query": "Test query", + "items": ["item1", "item2"], + }, + ) + # Missing required field returns 400 (manual JSON parsing) + assert response.status_code == 400 + + @pytest.mark.asyncio + async def test_validation_empty_items(self, server: RemoteOpenAIServer): + """Test that empty items returns an error.""" + response = requests.post( + server.url_for("generative_scoring"), + json={ + "model": MODEL_NAME, + "query": "Test query", + "items": [], + "label_token_ids": [100, 200], + }, + ) + assert response.status_code == 400 + + @pytest.mark.asyncio + @pytest.mark.parametrize( + "label_token_ids,expected_status", + [ + ([9999999999, 9999999998], 400), # Out of vocab range + ], + ids=["invalid_token_ids"], + ) + async def test_validation_errors( + self, server: RemoteOpenAIServer, label_token_ids, expected_status + ): + """Test validation errors for various invalid inputs.""" + response = requests.post( + server.url_for("generative_scoring"), + json={ + "model": MODEL_NAME, + "query": "Test query", + "items": ["item1"], + "label_token_ids": label_token_ids, + }, + ) + assert response.status_code == expected_status + + @pytest.mark.asyncio + async def test_score_consistency(self, server: RemoteOpenAIServer): + """Test that scores are deterministic across identical requests.""" + request_body = { + "model": MODEL_NAME, + "query": "Is this consistent? ", + "items": ["Yes it is."], + "label_token_ids": [100, 200], + } + + r1 = requests.post(server.url_for("generative_scoring"), json=request_body) + r2 = requests.post(server.url_for("generative_scoring"), json=request_body) + + assert r1.status_code == 200 and r2.status_code == 200 + r1_score = r1.json()["data"][0]["score"] + r2_score = r2.json()["data"][0]["score"] + assert abs(r1_score - r2_score) < 1e-6 + + +if __name__ == "__main__": + pytest.main([__file__, "-v"]) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/llm/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/llm/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/llm/offline_mode/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/llm/offline_mode/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/llm/offline_mode/test_offline_mode.py b/asian_ai/performance/vLLM/tests/entrypoints/llm/offline_mode/test_offline_mode.py new file mode 100644 index 0000000000000000000000000000000000000000..13665ef216f1dfac5e7bf7865b8f879e33cd5f88 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/llm/offline_mode/test_offline_mode.py @@ -0,0 +1,166 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for HF_HUB_OFFLINE mode""" + +import importlib +import sys + +import pytest +import regex as re +import urllib3 + +from vllm import LLM +from vllm.distributed import cleanup_dist_env_and_memory + +MODEL_CONFIGS = [ + { + "model": "facebook/opt-125m", + "enforce_eager": True, + "gpu_memory_utilization": 0.20, + "max_model_len": 64, + "max_num_batched_tokens": 64, + "max_num_seqs": 64, + "tensor_parallel_size": 1, + }, + { + "model": "Qwen/Qwen3-0.6B", + "enforce_eager": True, + "gpu_memory_utilization": 0.50, + "max_model_len": 64, + "max_num_batched_tokens": 64, + "max_num_seqs": 64, + "tensor_parallel_size": 1, + "tokenizer": "Qwen/Qwen3-4B", + }, + { + "model": "mistralai/Mistral-7B-Instruct-v0.1", + "enforce_eager": True, + "gpu_memory_utilization": 0.95, + "max_model_len": 64, + "max_num_batched_tokens": 64, + "max_num_seqs": 64, + "tensor_parallel_size": 1, + "tokenizer_mode": "mistral", + }, + # TODO: re-enable once these tests are run with V1 + # { + # "model": "sentence-transformers/all-MiniLM-L12-v2", + # "enforce_eager": True, + # "gpu_memory_utilization": 0.20, + # "max_model_len": 64, + # "max_num_batched_tokens": 64, + # "max_num_seqs": 64, + # "tensor_parallel_size": 1, + # }, +] + + +@pytest.fixture(scope="module") +def cache_models(): + # Cache model files first + for model_config in MODEL_CONFIGS: + LLM(**model_config) + cleanup_dist_env_and_memory() + + yield + + +@pytest.mark.skip_global_cleanup +@pytest.mark.usefixtures("cache_models") +def test_offline_mode(monkeypatch: pytest.MonkeyPatch): + # Set HF to offline mode and ensure we can still construct an LLM + with monkeypatch.context() as m: + try: + m.setenv("HF_HUB_OFFLINE", "1") + m.setenv("VLLM_NO_USAGE_STATS", "1") + + def disable_connect(*args, **kwargs): + raise RuntimeError("No http calls allowed") + + m.setattr( + urllib3.connection.HTTPConnection, + "connect", + disable_connect, + ) + m.setattr( + urllib3.connection.HTTPSConnection, + "connect", + disable_connect, + ) + + # Need to re-import huggingface_hub + # and friends to set up offline mode + _re_import_modules() + # Cached model files should be used in offline mode + for model_config in MODEL_CONFIGS: + LLM(**model_config) + finally: + # Reset the environment after the test + # NB: Assuming tests are run in online mode + _re_import_modules() + + +def _re_import_modules(): + hf_hub_module_names = [k for k in sys.modules if k.startswith("huggingface_hub")] + transformers_module_names = [ + k + for k in sys.modules + if k.startswith("transformers") and not k.startswith("transformers_modules") + ] + + # These modules are aliased in Transformers v5 and so cannot be reloaded directly + aliased_module_patterns = [ + r".+\.tokenization_utils$", + r".+\.tokenization_utils_fast$", + r".+\.image_processing_utils_fast$", + r".+\.models\..+\.image_processing_.+_fast$", + ] + + reload_exception = None + for module_name in hf_hub_module_names + transformers_module_names: + if any(re.match(pattern, module_name) for pattern in aliased_module_patterns): + # Remove from sys.modules so they are re-aliased on next import + del sys.modules[module_name] + continue + try: + importlib.reload(sys.modules[module_name]) + except Exception as e: + reload_exception = e + # Try to continue clean up so that other tests are less likely to + # be affected + + # Error this test if reloading a module failed + if reload_exception is not None: + raise reload_exception + + +@pytest.mark.skip_global_cleanup +@pytest.mark.usefixtures("cache_models") +def test_model_from_huggingface_offline(monkeypatch: pytest.MonkeyPatch): + # Set HF to offline mode and ensure we can still construct an LLM + with monkeypatch.context() as m: + try: + m.setenv("HF_HUB_OFFLINE", "1") + m.setenv("VLLM_NO_USAGE_STATS", "1") + + def disable_connect(*args, **kwargs): + raise RuntimeError("No http calls allowed") + + m.setattr( + urllib3.connection.HTTPConnection, + "connect", + disable_connect, + ) + m.setattr( + urllib3.connection.HTTPSConnection, + "connect", + disable_connect, + ) + # Need to re-import huggingface_hub + # and friends to set up offline mode + _re_import_modules() + LLM(model="facebook/opt-125m") + finally: + # Reset the environment after the test + # NB: Assuming tests are run in online mode + _re_import_modules() diff --git a/asian_ai/performance/vLLM/tests/entrypoints/llm/test_accuracy.py b/asian_ai/performance/vLLM/tests/entrypoints/llm/test_accuracy.py new file mode 100644 index 0000000000000000000000000000000000000000..cea8026357f46ecc1f252225fc5802548e0155dc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/llm/test_accuracy.py @@ -0,0 +1,94 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +This file test accuracy of the vLLM server via LMEval. +It uses local-completions, which interacts with vLLM +through the OAI API with N concurrent connections. +This simulates real work usage of the API and makes +sure that the zmq frontend mp RPC message passing and +AsyncLLMEngine are working correctly. +""" + +import lm_eval +import pytest + +from vllm.platforms import current_platform + +MODEL_NAMES = [ + "Qwen/Qwen3-1.7B", + "google/gemma-3-1b-it", +] +FP8_KV_MODEL_NAMES = [ + "Qwen/Qwen3-1.7B", +] +NUM_CONCURRENT = 500 +TASK = "gsm8k" +FILTER = "exact_match,strict-match" +RTOL = 0.03 +EXPECTED_VALUES = { + "Qwen/Qwen3-1.7B": 0.68, + "google/gemma-3-1b-it": 0.25, +} + + +def run_test(model_name, more_args=None): + """Run the end to end accuracy test.""" + + model_args = f"pretrained={model_name},max_model_len=4096" + + if more_args is not None: + model_args = "{},{}".format(model_args, more_args) + + results = lm_eval.simple_evaluate( + model="vllm", + model_args=model_args, + tasks="gsm8k", + batch_size="auto", + ) + + measured_value = results["results"][TASK][FILTER] + assert model_name in EXPECTED_VALUES, ( + f"Cannot find the expected value for the model {model_name=}" + ) + expected_value = EXPECTED_VALUES[model_name] + assert ( + measured_value - RTOL < expected_value + and measured_value + RTOL > expected_value + ), f"Expected: {expected_value} | Measured: {measured_value}" + + +# TODO: [AlexM] Fix it with new CI/CD tests +TPU_TP_TEST_STR = "" # "tensor_parallel_size=4" + + +@pytest.mark.parametrize("model", MODEL_NAMES) +def test_lm_eval_accuracy_v1_engine(model): + """Run with the V1 Engine.""" + + more_args = None + if current_platform.is_tpu(): + # Limit compilation time for TPU V1 + + more_args = "max_model_len=2048,max_num_seqs=64" + + # Add TP test (if provided) + if TPU_TP_TEST_STR: + more_args += ",{}".format(TPU_TP_TEST_STR) + + run_test(model, more_args) + + +@pytest.mark.parametrize("model", FP8_KV_MODEL_NAMES) +def test_lm_eval_accuracy_v1_engine_fp8_kv_cache(model): + """Run with the V1 Engine.""" + + more_args = None + if current_platform.is_tpu(): + # Limit compilation time for TPU V1 + more_args = "max_model_len=2048,max_num_seqs=128,kv_cache_dtype=fp8" + + # Add TP test (if provided) + if TPU_TP_TEST_STR: + more_args += ",{}".format(TPU_TP_TEST_STR) + + run_test(model, more_args) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/llm/test_chat.py b/asian_ai/performance/vLLM/tests/entrypoints/llm/test_chat.py new file mode 100644 index 0000000000000000000000000000000000000000..3fb6e63d157a871a2e52edb0d5abf08b0d73be1b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/llm/test_chat.py @@ -0,0 +1,166 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import weakref + +import pytest + +from vllm import LLM +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.sampling_params import SamplingParams + + +@pytest.fixture(scope="function") +def text_llm(): + # pytest caches the fixture so we use weakref.proxy to + # enable garbage collection + llm = LLM(model="meta-llama/Llama-3.2-1B-Instruct", enforce_eager=True, seed=0) + + yield weakref.proxy(llm) + + del llm + + cleanup_dist_env_and_memory() + + +@pytest.fixture(scope="function") +def llm_for_failure_test(): + """ + Fixture for testing issue #26081. + Uses a small max_model_len to easily trigger length errors. + """ + # pytest caches the fixture so we use weakref.proxy to + # enable garbage collection + llm = LLM( + model="meta-llama/Llama-3.2-1B-Instruct", + enforce_eager=True, + seed=0, + max_model_len=128, + disable_log_stats=True, + ) + + yield weakref.proxy(llm) + + del llm + + cleanup_dist_env_and_memory() + + +def test_chat(text_llm): + prompt1 = "Explain the concept of entropy." + messages = [ + {"role": "system", "content": "You are a helpful assistant"}, + {"role": "user", "content": prompt1}, + ] + outputs = text_llm.chat(messages) + assert len(outputs) == 1 + + +def test_multi_chat(text_llm): + prompt1 = "Explain the concept of entropy." + prompt2 = "Explain what among us is." + + conversation1 = [ + {"role": "system", "content": "You are a helpful assistant"}, + {"role": "user", "content": prompt1}, + ] + + conversation2 = [ + {"role": "system", "content": "You are a helpful assistant"}, + {"role": "user", "content": prompt2}, + ] + + messages = [conversation1, conversation2] + + outputs = text_llm.chat(messages) + assert len(outputs) == 2 + + +def test_llm_chat_tokenization_no_double_bos(text_llm): + """ + LLM.chat() should not add special tokens when using chat templates. + Check we get a single BOS token for llama chat. + """ + messages = [ + {"role": "system", "content": "You are a helpful assistant"}, + {"role": "user", "content": "Hello!"}, + ] + outputs = text_llm.chat(messages) + assert len(outputs) == 1 + + prompt_token_ids = outputs[0].prompt_token_ids + assert prompt_token_ids is not None + + bos_token = text_llm.get_tokenizer().bos_token_id + + # Ensure we have a single BOS + assert prompt_token_ids[0] == bos_token + assert prompt_token_ids[1] != bos_token, "Double BOS" + + +@pytest.fixture(scope="function") +def thinking_llm(): + # pytest caches the fixture so we use weakref.proxy to + # enable garbage collection + llm = LLM( + model="Qwen/Qwen3-0.6B", + max_model_len=4096, + enforce_eager=True, + seed=0, + ) + + yield weakref.proxy(llm) + + del llm + + cleanup_dist_env_and_memory() + + +@pytest.mark.parametrize("enable_thinking", [True, False]) +def test_chat_extra_kwargs(thinking_llm, enable_thinking): + messages = [ + {"role": "system", "content": "You are a helpful assistant"}, + {"role": "user", "content": "What is 1+1?"}, + ] + + outputs = thinking_llm.chat( + messages, + chat_template_kwargs={"enable_thinking": enable_thinking}, + ) + assert len(outputs) == 1 + + prompt_token_ids = outputs[0].prompt_token_ids + assert prompt_token_ids is not None + + think_id = thinking_llm.get_tokenizer().get_vocab()[""] + + if enable_thinking: + assert think_id not in prompt_token_ids + else: + # The chat template includes dummy thinking process + assert think_id in prompt_token_ids + + +def test_chat_batch_failure_cleanup(llm_for_failure_test): + """ + Tests that if a batch call to llm.chat() fails mid-way + (e.g., due to one invalid prompt), the requests that + were already enqueued are properly aborted and do not + pollute the queue for subsequent calls. + (Fixes Issue #26081) + """ + llm = llm_for_failure_test + valid_msg = [{"role": "user", "content": "Hello"}] + long_text = "This is a very long text to test the error " * 50 + invalid_msg = [{"role": "user", "content": long_text}] + + batch_1 = [valid_msg, valid_msg, invalid_msg] + batch_2 = [valid_msg, valid_msg] + sampling_params = SamplingParams(temperature=0, max_tokens=10) + + with pytest.raises(ValueError, match="maximum context length is"): + llm.chat(batch_1, sampling_params=sampling_params) + assert llm.llm_engine.get_num_unfinished_requests() == 0 + + outputs_2 = llm.chat(batch_2, sampling_params=sampling_params) + assert len(outputs_2) == len(batch_2) + assert llm.llm_engine.get_num_unfinished_requests() == 0 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/llm/test_collective_rpc.py b/asian_ai/performance/vLLM/tests/entrypoints/llm/test_collective_rpc.py new file mode 100644 index 0000000000000000000000000000000000000000..73f15cf53f164b13b04ecfe3fe736ab74383b45c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/llm/test_collective_rpc.py @@ -0,0 +1,36 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm import LLM + +from ...utils import create_new_process_for_each_test + + +@pytest.mark.parametrize("tp_size", [1, 2]) +@pytest.mark.parametrize("backend", ["mp", "ray"]) +@create_new_process_for_each_test() +def test_collective_rpc(tp_size, backend, monkeypatch): + if torch.accelerator.device_count() < tp_size: + pytest.skip(f"Not enough GPUs for tensor parallelism {tp_size}") + if tp_size == 1 and backend == "ray": + pytest.skip("Skip duplicate test case") + if tp_size == 1: + backend = None + + # intentionally define the method and class in the test function, + # to test if they can be serialized and sent to the workers + def echo_rank(self): + return self.rank + + monkeypatch.setenv("VLLM_ALLOW_INSECURE_SERIALIZATION", "1") + llm = LLM( + model="hmellor/tiny-random-LlamaForCausalLM", + enforce_eager=True, + load_format="dummy", + tensor_parallel_size=tp_size, + distributed_executor_backend=backend, + ) + assert llm.collective_rpc(echo_rank) == list(range(tp_size)) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/llm/test_generate.py b/asian_ai/performance/vLLM/tests/entrypoints/llm/test_generate.py new file mode 100644 index 0000000000000000000000000000000000000000..21b6af6d28e369e978b2d3b37d4a154fa0044400 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/llm/test_generate.py @@ -0,0 +1,130 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import weakref + +import pytest + +from vllm import LLM, SamplingParams +from vllm.distributed import cleanup_dist_env_and_memory + +MODEL_NAME = "distilbert/distilgpt2" + +PROMPTS = [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + "The future of AI is", +] + +TOKEN_IDS = [ + [0], + [0, 1], + [0, 2, 1], + [0, 3, 1, 2], +] + + +@pytest.fixture(scope="module") +def llm(): + # pytest caches the fixture so we use weakref.proxy to + # enable garbage collection + llm = LLM( + model=MODEL_NAME, + max_num_batched_tokens=4096, + tensor_parallel_size=1, + gpu_memory_utilization=0.10, + enforce_eager=True, + ) + + yield weakref.proxy(llm) + + del llm + + cleanup_dist_env_and_memory() + + +@pytest.mark.skip_global_cleanup +def test_multiple_sampling_params(llm: LLM): + sampling_params = [ + SamplingParams(temperature=0.01, top_p=0.95), + SamplingParams(temperature=0.3, top_p=0.95), + SamplingParams(temperature=0.7, top_p=0.95), + SamplingParams(temperature=0.99, top_p=0.95), + ] + + # Multiple SamplingParams should be matched with each prompt + outputs = llm.generate(PROMPTS, sampling_params=sampling_params) + assert len(PROMPTS) == len(outputs) + + # Exception raised, if the size of params does not match the size of prompts + with pytest.raises(ValueError): + outputs = llm.generate(PROMPTS, sampling_params=sampling_params[:3]) + + # Single SamplingParams should be applied to every prompt + single_sampling_params = SamplingParams(temperature=0.3, top_p=0.95) + outputs = llm.generate(PROMPTS, sampling_params=single_sampling_params) + assert len(PROMPTS) == len(outputs) + + # sampling_params is None, default params should be applied + outputs = llm.generate(PROMPTS, sampling_params=None) + assert len(PROMPTS) == len(outputs) + + +def test_multiple_priority(llm: LLM): + # Generate works when priority is None + outputs = llm.generate(PROMPTS, sampling_params=None, priority=None) + assert len(PROMPTS) == len(outputs) + + # Generate works when length of priority is same as the len(PROMPTS) + outputs = llm.generate(PROMPTS, sampling_params=None, priority=[0] * len(PROMPTS)) + assert len(PROMPTS) == len(outputs) + + # Exception raised, if the length of priority does not match the length of prompts + with pytest.raises(ValueError): + outputs = llm.generate( + PROMPTS, sampling_params=None, priority=[0] * (len(PROMPTS) - 1) + ) + + # Exception raised, if the priority list is empty + with pytest.raises(ValueError): + outputs = llm.generate(PROMPTS, sampling_params=None, priority=[]) + + +def test_single_prompt_priority(llm: LLM): + # Single string prompts should be normalized to one request. + outputs = llm.generate(PROMPTS[0], sampling_params=None, priority=[0]) + assert len(outputs) == 1 + + +def test_max_model_len(): + max_model_len = 20 + llm = LLM( + model=MODEL_NAME, + max_model_len=max_model_len, + gpu_memory_utilization=0.10, + enforce_eager=True, # reduce test time + ) + sampling_params = SamplingParams(max_tokens=max_model_len + 10) + outputs = llm.generate(PROMPTS, sampling_params) + for output in outputs: + num_total_tokens = len(output.prompt_token_ids) + len( + output.outputs[0].token_ids + ) + # Total tokens must not exceed max_model_len. + # It can be less if generation finishes due to other reasons (e.g., EOS) + # before reaching the absolute model length limit. + assert num_total_tokens <= max_model_len + + +def test_log_stats(): + llm = LLM( + model=MODEL_NAME, + disable_log_stats=False, + gpu_memory_utilization=0.10, + enforce_eager=True, # reduce test time + ) + outputs = llm.generate(PROMPTS, sampling_params=None) + + # disable_log_stats is False, every output should have metrics + assert all(output.metrics is not None for output in outputs) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/llm/test_gpu_utilization.py b/asian_ai/performance/vLLM/tests/entrypoints/llm/test_gpu_utilization.py new file mode 100644 index 0000000000000000000000000000000000000000..6f1d08ae25f2dd43aa0750c1f050904910e42583 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/llm/test_gpu_utilization.py @@ -0,0 +1,27 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from vllm import LLM, SamplingParams + + +def test_gpu_memory_utilization(): + prompts = [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + "The future of AI is", + ] + sampling_params = SamplingParams(temperature=0.8, top_p=0.95) + + # makes sure gpu_memory_utilization is per-instance limit, + # not a global limit + llms = [ + LLM(model="facebook/opt-125m", gpu_memory_utilization=0.3, enforce_eager=True) + for i in range(3) + ] + for llm in llms: + outputs = llm.generate(prompts, sampling_params) + for output in outputs: + prompt = output.prompt + generated_text = output.outputs[0].text + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") diff --git a/asian_ai/performance/vLLM/tests/entrypoints/llm/test_prompt_validation.py b/asian_ai/performance/vLLM/tests/entrypoints/llm/test_prompt_validation.py new file mode 100644 index 0000000000000000000000000000000000000000..60e1591c45af1ae33c3e8105cba5d4a683561d6e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/llm/test_prompt_validation.py @@ -0,0 +1,34 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm import LLM + + +def test_empty_prompt(): + llm = LLM(model="openai-community/gpt2", enforce_eager=True) + with pytest.raises(ValueError, match="decoder prompt cannot be empty"): + llm.generate([""]) + + +def test_out_of_vocab_token(): + llm = LLM(model="openai-community/gpt2", enforce_eager=True) + with pytest.raises(ValueError, match="out of vocabulary"): + llm.generate({"prompt_token_ids": [999999]}) + + +def test_require_mm_embeds(): + llm = LLM( + model="llava-hf/llava-1.5-7b-hf", + enforce_eager=True, + enable_mm_embeds=False, + ) + with pytest.raises(ValueError, match="--enable-mm-embeds"): + llm.generate( + { + "prompt": "", + "multi_modal_data": {"image": torch.empty(1, 1, 1)}, + } + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/llm/test_struct_output_generate.py b/asian_ai/performance/vLLM/tests/entrypoints/llm/test_struct_output_generate.py new file mode 100644 index 0000000000000000000000000000000000000000..e63909608998850a44390203680e8b3a5fb790d1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/llm/test_struct_output_generate.py @@ -0,0 +1,1027 @@ +# ruff: noqa: E501 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json +from enum import Enum +from typing import Any + +import jsonschema +import pytest +import regex as re +import torch +from pydantic import BaseModel + +from tests.reasoning.utils import run_reasoning_extraction +from vllm.config import StructuredOutputsConfig +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.entrypoints.llm import LLM +from vllm.outputs import RequestOutput +from vllm.platforms import current_platform +from vllm.reasoning.abs_reasoning_parsers import ReasoningParserManager +from vllm.sampling_params import ( + SamplingParams, + StructuredOutputsParams, +) + +SAMPLE_REGEX = ( + r"((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)\.){3}" + r"(25[0-5]|(2[0-4]|1\d|[1-9]|)\d)" +) + +# Note: Ensure this only uses attributes compatible with xgrammar +SAMPLE_JSON_SCHEMA = { + "type": "object", + "properties": { + "name": {"type": "string"}, + "age": {"type": "integer"}, + "skills": { + "type": "array", + "items": { + "type": "string", + }, + }, + "grade": { + "type": "string", + "pattern": "^[A-D]$", # Regex pattern + }, + "email": { + "type": "string", + "pattern": "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$", + }, + "work_history": { + "type": "array", + "items": { + "type": "object", + "properties": { + "company": {"type": "string"}, + "duration": { + "type": "number", + "minimum": 0.0, + "maximum": 100.0, # Numeric range + }, + "position": {"type": "string"}, + }, + "required": ["company", "duration", "position"], + "additionalProperties": False, + }, + "minItems": 0, + "maxItems": 3, + }, + }, + "required": ["name", "age", "skills", "grade", "email", "work_history"], + "additionalProperties": False, + "minProperties": 1, + "maxProperties": 10, +} + +# A schema unsupported by xgrammar +UNSUPPORTED_JSON_SCHEMA = { + "type": "object", + "properties": { + "score": { + "type": "integer", + "multipleOf": 5, # Numeric multiple + }, + "tags": { + "type": "array", + "items": {"type": "string", "minLength": 10, "maxLength": 20}, + }, + }, + "required": ["score", "tags"], + "additionalProperties": False, + "patternProperties": { + "^score$": {"type": "integer"}, + }, +} + +SAMPLE_STRUCTURED_OUTPUTS_CHOICES = [ + "Python", + "Java", + "JavaScript", + "C++", + "C#", + "PHP", + "TypeScript", + "Ruby", + "Swift", + "Kotlin", +] + +SAMPLE_SQL_EBNF = """ +root ::= select_statement +select_statement ::= "SELECT" column "from" table "where" condition +column ::= "col_1" | "col_2" +table ::= "table_1" | "table_2" +condition ::= column "=" number +number ::= "1" | "2" +""" + +SAMPLE_SQL_LARK = """ +start: select_statement +select_statement: "SELECT" column "from" table "where" condition +column: "col_1" | "col_2" +table: "table_1" | "table_2" +condition: column "=" number +number: "1" | "2" +""" + +NGRAM_SPEC_CONFIG = { + "model": "[ngram]", + "num_speculative_tokens": 5, + "prompt_lookup_max": 5, + "prompt_lookup_min": 1, +} + +EAGLE_SPEC_CONFIG = { + "method": "eagle", + "model": "yuhuili/EAGLE-LLaMA3.1-Instruct-8B", + "num_speculative_tokens": 5, +} + +PARAMS_MODELS_BACKENDS_TOKENIZER_MODE = [ + ("mistralai/Ministral-8B-Instruct-2410", "xgrammar", "auto", None), + # FIXME: Since "auto" will use Mistral tokenizer and these backends do not support + # it, we skip these tests for now. + # ("mistralai/Ministral-8B-Instruct-2410", "guidance", "auto", None), + # ("mistralai/Ministral-8B-Instruct-2410", "lm-format-enforcer", "auto", None), + ("mistralai/Ministral-8B-Instruct-2410", "guidance", "hf", None), + pytest.param( + "mistralai/Ministral-8B-Instruct-2410", + "lm-format-enforcer", + "hf", + None, + marks=pytest.mark.skip( + reason=( + "Flaky: lm-format-enforcer intermittently returns" + "incomplete JSON." + "See https://github.com/noamgat/lm-format-enforcer/issues/169" + ) + ), + ), + ("mistralai/Ministral-8B-Instruct-2410", "xgrammar", "mistral", None), + ("Qwen/Qwen2.5-1.5B-Instruct", "xgrammar", "auto", None), + pytest.param( + "Qwen/Qwen2.5-1.5B-Instruct", + "lm-format-enforcer", + "auto", + None, + marks=pytest.mark.skip( + reason=( + "Flaky: lm-format-enforcer intermittently returns" + "incomplete JSON." + "See https://github.com/noamgat/lm-format-enforcer/issues/169" + ) + ), + ), + # FIXME: This tests are flaky on CI thus disabled. Tracking in Issue #24402 + # ("mistralai/Ministral-8B-Instruct-2410", "outlines", "auto", None), + # ("mistralai/Ministral-8B-Instruct-2410", "outlines", "mistral", None), + # ("Qwen/Qwen2.5-1.5B-Instruct", "guidance", "auto"), + ("mistralai/Ministral-8B-Instruct-2410", "outlines", "auto", NGRAM_SPEC_CONFIG), + ("mistralai/Ministral-8B-Instruct-2410", "guidance", "hf", NGRAM_SPEC_CONFIG), + ("Qwen/Qwen2.5-1.5B-Instruct", "xgrammar", "auto", NGRAM_SPEC_CONFIG), + ("meta-llama/Meta-Llama-3.1-8B-Instruct", "xgrammar", "auto", EAGLE_SPEC_CONFIG), +] + +PARAMS_MODELS_TOKENIZER_MODE = [ + ("mistralai/Ministral-8B-Instruct-2410", "auto"), + ("Qwen/Qwen2.5-1.5B-Instruct", "auto"), +] + +platform_args = {} +if current_platform.is_rocm(): + platform_args["async_scheduling"] = False + + +class CarType(str, Enum): + sedan = "sedan" + suv = "SUV" + truck = "Truck" + coupe = "Coupe" + + +class CarDescription(BaseModel): + brand: str + model: str + car_type: CarType + + +@pytest.mark.parametrize( + "model_name, backend, tokenizer_mode, speculative_config", + PARAMS_MODELS_BACKENDS_TOKENIZER_MODE, +) +def test_structured_output( + backend: str, + tokenizer_mode: str, + model_name: str, + speculative_config: dict[str, Any], +): + sample_json_schema = SAMPLE_JSON_SCHEMA + unsupported_json_schema = UNSUPPORTED_JSON_SCHEMA + sample_sql_ebnf = SAMPLE_SQL_EBNF + sample_sql_lark = SAMPLE_SQL_LARK + sample_regex = SAMPLE_REGEX + sample_structured_outputs_choices = SAMPLE_STRUCTURED_OUTPUTS_CHOICES + if current_platform.is_tpu() and speculative_config: + pytest.skip("TPU does not support speculative decoding") + + # Use a single LLM instance for several scenarios to + # speed up the test suite. + llm = LLM( + model=model_name, + enforce_eager=True, + max_model_len=1024, + structured_outputs_config=dict( + backend=backend, disable_any_whitespace=backend in {"xgrammar", "guidance"} + ), + seed=120, + tokenizer_mode=tokenizer_mode, + load_format="auto" if not model_name.startswith("mistralai/") else "hf", + config_format="auto" if not model_name.startswith("mistralai/") else "hf", + speculative_config=speculative_config, + **platform_args, + ) + + # + # Test 1: Generate JSON output based on a provided schema + # + sampling_params = SamplingParams( + temperature=1.0, + max_tokens=4096, + structured_outputs=StructuredOutputsParams(json=sample_json_schema), + ) + + prompt = ( + "Give an example JSON for an employee profile that fits this " + "schema. Make the response as short as possible. Schema: " + f"{sample_json_schema}" + ) + outputs = llm.generate( + [prompt] * 2, + sampling_params=sampling_params, + use_tqdm=True, + ) + + assert outputs is not None + + for output in outputs: + assert output is not None + assert isinstance(output, RequestOutput) + prompt = output.prompt + + generated_text = output.outputs[0].text + assert generated_text is not None + if backend != "lm-format-enforcer": + assert "\n" not in generated_text + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") + try: + output_json = json.loads(generated_text) + except json.JSONDecodeError as e: + pytest.fail( + f"Invalid JSON from backend={backend}: {generated_text!r}\n" + f"Schema: {sample_json_schema}\nError: {e}" + ) + jsonschema.validate(instance=output_json, schema=sample_json_schema) + + # + # Test 2: Generate JSON object without a schema + # + if backend != "outlines": + sampling_params = SamplingParams( + temperature=1.0, + max_tokens=4096, + n=2, + structured_outputs=StructuredOutputsParams(json_object=True), + ) + + outputs = llm.generate( + prompts=( + "Generate a JSON object with curly braces for a person with " + "name and age fields for John Smith who is 31 years old. " + "Make the response as short as possible." + ), + sampling_params=sampling_params, + use_tqdm=True, + ) + + assert outputs is not None + for output in outputs: + assert output is not None + assert isinstance(output, RequestOutput) + + for i in range(2): + generated_text = output.outputs[i].text + print(generated_text) + assert generated_text is not None + + # Parse to verify it is a valid JSON object + parsed_json = json.loads(generated_text) + assert isinstance(parsed_json, dict) + + # + # Test 3: test a jsonschema incompatible with xgrammar + # + sampling_params = SamplingParams( + temperature=1.0, + max_tokens=4096, + structured_outputs=StructuredOutputsParams(json=unsupported_json_schema), + ) + if backend.startswith("xgrammar"): + with pytest.raises( + ValueError, + match="The provided JSON schema contains features " + "not supported by xgrammar.", + ): + prompt = ( + f"Give an example JSON for an employee profile that " + f"fits this schema: {unsupported_json_schema}. " + f"Make the response as short as possible." + ) + llm.generate( + [prompt] * 2, + sampling_params=sampling_params, + use_tqdm=True, + ) + else: + prompt = ( + f"Give an example JSON object for a grade that " + f"fits this schema: {unsupported_json_schema}. " + f"Make the response as short as possible." + ) + outputs = llm.generate( + prompt, + sampling_params=sampling_params, + use_tqdm=True, + ) + assert outputs is not None + for output in outputs: + assert output is not None + assert isinstance(output, RequestOutput) + generated_text = output.outputs[0].text + assert generated_text is not None + print(generated_text) + + # Parse to verify it is valid JSON + parsed_json = json.loads(generated_text) + assert isinstance(parsed_json, dict) + + if backend not in ["outlines", "lm-format-enforcer"]: + # + # Test 4: Generate SQL statement using EBNF grammar + # + sampling_params = SamplingParams( + temperature=0.8, + top_p=0.95, + max_tokens=1000, + structured_outputs=StructuredOutputsParams(grammar=sample_sql_ebnf), + ) + outputs = llm.generate( + ( + "Generate a sql statement that selects col_1 from " + "table_1 where it is equal to 1. Make the response as short as " + "possible." + ), + sampling_params=sampling_params, + use_tqdm=True, + ) + + assert outputs is not None + for output in outputs: + assert output is not None + assert isinstance(output, RequestOutput) + prompt = output.prompt + + generated_text = output.outputs[0].text + assert generated_text is not None + + # remove spaces for comparison b/c we removed them in the grammar + ground_truth = "SELECT col_1 from table_1 where col_1 = 1".replace(" ", "") + + assert generated_text.strip() == ground_truth + + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") + + # + # Test 5: Generate SQL statement using Lark grammar + # + sampling_params = SamplingParams( + temperature=0.8, + top_p=0.95, + max_tokens=1000, + structured_outputs=StructuredOutputsParams(grammar=sample_sql_lark), + ) + outputs = llm.generate( + ( + "Generate a sql statement that selects col_1 from " + "table_1 where it is equal to 1. Make the response as short as " + "possible." + ), + sampling_params=sampling_params, + use_tqdm=True, + ) + + assert outputs is not None + for output in outputs: + assert output is not None + assert isinstance(output, RequestOutput) + prompt = output.prompt + + generated_text = output.outputs[0].text + assert generated_text is not None + + # use Lark to parse the output, and make sure it's a valid parse tree + from lark import Lark + + parser = Lark(sample_sql_lark) + parser.parse(generated_text) + + # remove spaces for comparison b/c we removed them in the grammar + ground_truth = "SELECT col_1 from table_1 where col_1 = 1".replace(" ", "") + + assert generated_text.strip() == ground_truth + + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") + + # + # Test 6: Test invalid grammar input + # + sampling_params = SamplingParams( + temperature=0.8, + top_p=0.95, + max_tokens=1000, + structured_outputs=StructuredOutputsParams(grammar="not a grammar"), + ) + with pytest.raises(ValueError, match="Failed to convert the grammar "): + llm.generate( + ( + "Generate a sql statement that selects col_1 from " + "table_1 where it is equal to 1. Make the response as short " + "as possible." + ), + sampling_params=sampling_params, + use_tqdm=True, + ) + + # + # Test 7: Generate text based on a regex pattern + # + sampling_params = SamplingParams( + temperature=0.8, + top_p=0.95, + structured_outputs=StructuredOutputsParams(regex=sample_regex), + ) + + prompt = ( + f"Give an example IPv4 address with this regex: {sample_regex}. " + f"Make the response as short as possible." + ) + outputs = llm.generate( + [prompt] * 2, + sampling_params=sampling_params, + use_tqdm=True, + ) + + assert outputs is not None + for output in outputs: + assert output is not None + assert isinstance(output, RequestOutput) + prompt = output.prompt + generated_text = output.outputs[0].text + print(generated_text) + assert generated_text is not None + assert re.fullmatch(sample_regex, generated_text) is not None + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") + + # + # Test 8: Generate text based on a choices + # + sampling_params = SamplingParams( + temperature=0.8, + top_p=0.95, + structured_outputs=StructuredOutputsParams( + choice=sample_structured_outputs_choices + ), + ) + + outputs = llm.generate( + ( + "The best language for type-safe systems programming is " + "(Make the response as short as possible.) " + ), + sampling_params=sampling_params, + use_tqdm=True, + ) + assert outputs is not None + for output in outputs: + assert output is not None + assert isinstance(output, RequestOutput) + prompt = output.prompt + generated_text = output.outputs[0].text + print(generated_text) + assert generated_text is not None + assert generated_text in sample_structured_outputs_choices + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") + + # + # Test 9: Generate structured output using a Pydantic model with an enum + # + json_schema = CarDescription.model_json_schema() + sampling_params = SamplingParams( + temperature=1.0, + max_tokens=1000, + structured_outputs=StructuredOutputsParams(json=json_schema), + ) + + outputs = llm.generate( + ( + "Generate a JSON with the brand, model and car_type of the most " + "iconic car from the 90's. Make the response as short as " + "possible." + ), + sampling_params=sampling_params, + use_tqdm=True, + ) + + assert outputs is not None + + for output in outputs: + assert output is not None + assert isinstance(output, RequestOutput) + prompt = output.prompt + + generated_text = output.outputs[0].text + assert generated_text is not None + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") + try: + output_json = json.loads(generated_text) + except json.JSONDecodeError as e: + pytest.fail( + f"Invalid JSON from backend={backend}: {generated_text!r}\n" + f"Schema: {json_schema}\nError: {e}" + ) + jsonschema.validate(instance=output_json, schema=json_schema) + + # + # Test 10: Generate structured with minLength and maxLength + # + min_length = 50 + max_length = 50 + json_schema = { + "type": "object", + "properties": { + "description": { + "type": "string", + "maxLength": max_length, + "minLength": min_length, + } + }, + "required": ["description"], + "additionalProperties": False, + } + + sampling_params = SamplingParams( + temperature=1.0, + max_tokens=4096, + structured_outputs=StructuredOutputsParams(json=json_schema), + ) + + outputs = llm.generate( + ( + "Generate a description of a frog using 50 characters. " + "Make the response as short as possible." + ), + sampling_params=sampling_params, + use_tqdm=True, + ) + + assert outputs is not None + + for output in outputs: + assert output is not None + assert isinstance(output, RequestOutput) + prompt = output.prompt + + generated_text = output.outputs[0].text + assert generated_text is not None + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") + try: + output_json = json.loads(generated_text) + except json.JSONDecodeError as e: + pytest.fail( + f"Invalid JSON from backend={backend}: {generated_text!r}\n" + f"Schema: {json_schema}\nError: {e}" + ) + jsonschema.validate(instance=output_json, schema=json_schema) + + if backend not in ["outlines", "lm-format-enforcer"]: + # + # Test 11: Generate structured output using structural_tag format + # + structural_tag_config = { + "type": "structural_tag", + "structures": [ + { + "begin": "", + "schema": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "additionalProperties": False, + }, + "end": "", + } + ], + "triggers": ["{parameters}{end_tag} +where + +start_tag => ` a JSON dict with the function argument name + as key and function argument value as value. +end_tag => `` + +Here is an example, +{"example_name": "example_value"} + +Reminder: +- Function calls MUST follow the specified format +- Required parameters MUST be specified +- Only call one function at a time +- Put the entire function call reply on one line +- Always add your sources when using search results to answer the user query + +You are a helpful assistant. + +Given the previous instructions, what is the weather in New York City? \ +Make the response as short as possible. +""" + + # Change this once other backends support structural_tag + outputs = llm.generate(prompt, sampling_params=sampling_params, use_tqdm=True) + assert outputs is not None + + for output in outputs: + assert output is not None + assert isinstance(output, RequestOutput) + generated_text = output.outputs[0].text + assert generated_text is not None + + # Search for function call pattern in the response + function_call_pattern = r"(.*?)" + matches = re.findall(function_call_pattern, generated_text) + + if not matches: + print( + f"Warning: No function calls found in response: {generated_text!r}" + ) + continue + + # Take the first function call if multiple are found + json_str = matches[0] + try: + json_content = json.loads(json_str) + assert "city" in json_content + assert isinstance(json_content["city"], str) + print(f"Found valid function call: {generated_text!r}") + except (json.JSONDecodeError, AssertionError) as e: + pytest.fail( + f"Invalid function call format: {generated_text!r}\nError: {str(e)}" + ) + + +@pytest.mark.parametrize( + "model_name, backend, tokenizer_mode, reasoning_parser, speculative_config, async_scheduling", # noqa: E501 + [ + ( + "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B", + "xgrammar", + "auto", + "deepseek_r1", + NGRAM_SPEC_CONFIG, + False, + ), + ("Qwen/Qwen3-1.7B", "xgrammar", "auto", "deepseek_r1", None, False), + ("Qwen/Qwen3-1.7B", "xgrammar", "auto", "deepseek_r1", None, True), + ], +) +def test_structured_output_with_reasoning_matrices( + backend: str, + tokenizer_mode: str, + reasoning_parser: str, + model_name: str, + speculative_config: dict[str, Any] | None, + async_scheduling: bool, +): + if current_platform.is_tpu() and speculative_config: + pytest.skip("TPU does not support speculative decoding") + + # Use a single LLM instance for several scenarios to + # speed up the test suite. + llm = LLM( + model=model_name, + # Don't use eager execution on TPUs because we want to test for no + # recompilation at runtime + enforce_eager=bool(not current_platform.is_tpu()), + max_model_len=1024, + max_num_seqs=16, + structured_outputs_config=dict( + backend=backend, + disable_any_whitespace=backend in {"xgrammar", "guidance"}, + reasoning_parser=reasoning_parser, + ), + tokenizer_mode=tokenizer_mode, + speculative_config=speculative_config, + async_scheduling=async_scheduling, + ) + tokenizer = llm.get_tokenizer() + reasoner = ReasoningParserManager.get_reasoning_parser(reasoning_parser)( + tokenizer=tokenizer + ) + + reasoning_prompt = "Solve the following math problem step-by-step, then provide the final answer as JSON object with a single key 'result'. Make sure to correct your reasoning if there are any issue should it arise.\nProblem: What is 5 * 8 + 2?" # noqa: E501 + reasoning_schema = { + "type": "object", + "properties": {"result": {"type": "integer"}}, + "required": ["result"], + "additionalProperties": False, + } + if "Qwen3" in model_name: + reasoning_prompt += "\n" + + sampling_params = SamplingParams( + temperature=0.1, + max_tokens=8192, + structured_outputs=StructuredOutputsParams(json=reasoning_schema), + ) + outputs = llm.generate( + [reasoning_prompt], + sampling_params=sampling_params, + use_tqdm=True, + ) + + assert outputs is not None + output = outputs[0] + assert output is not None and isinstance(output, RequestOutput) + prompt = output.prompt + generated_text = output.outputs[0].text + reasoning, content = run_reasoning_extraction(reasoner, [generated_text]) + print(f"Prompt: {prompt!r}\nReasoning: {reasoning!r}\nContent: {content!r}") + + if "Qwen3" in model_name: + assert content is not None + + assert reasoning is not None + + if content is not None: + output_json = json.loads(content) + jsonschema.validate(instance=output_json, schema=reasoning_schema) + + +@pytest.mark.parametrize("model_name, tokenizer_mode", PARAMS_MODELS_TOKENIZER_MODE) +def test_structured_output_auto_mode( + model_name: str, + tokenizer_mode: str, +): + unsupported_json_schema = UNSUPPORTED_JSON_SCHEMA + llm = LLM( + model=model_name, + max_model_len=1024, + structured_outputs_config=dict(backend="auto"), + tokenizer_mode=tokenizer_mode, + load_format="auto", + config_format="auto", + ) + + sampling_params = SamplingParams( + temperature=1.0, + max_tokens=1000, + structured_outputs=StructuredOutputsParams(json=unsupported_json_schema), + ) + + prompts = ( + "Give an example JSON object for a grade " + "that fits this schema: " + f"{unsupported_json_schema}. Make the response as short as possible." + ) + # This would fail with the default of "xgrammar", but in "auto" + # we will handle fallback automatically. + outputs = llm.generate(prompts, sampling_params=sampling_params, use_tqdm=True) + # Make sure `auto` backend handling doesn't mess up sampling_params + # and that we can reuse it without error. + outputs.extend( + llm.generate(prompts, sampling_params=sampling_params, use_tqdm=True) + ) + + assert outputs is not None + for output in outputs: + assert output is not None + assert isinstance(output, RequestOutput) + generated_text = output.outputs[0].text + assert generated_text is not None + print(generated_text) + + # Parse to verify it is valid JSON + parsed_json = json.loads(generated_text) + assert isinstance(parsed_json, dict) + + +def test_guidance_no_additional_properties(): + llm = LLM( + model="Qwen/Qwen2.5-1.5B-Instruct", + max_model_len=1024, + structured_outputs_config=dict( + backend="guidance", + disable_any_whitespace=True, + disable_additional_properties=True, + ), + ) + + schema = { + "type": "object", + "properties": { + "a1": {"type": "string"}, + "a2": {"type": "string"}, + "a3": {"type": "string"}, + }, + "required": ["a1", "a2", "a3"], + } + + prompt = ( + "<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a " + "helpful assistant.<|im_end|>\n<|im_start|>user\nPlease generate a " + "large JSON object with key-value pairs a1=b1, a2=b2, ..., a20=b20. " + "Make the response as short as possible." + "<|im_end|>\n<|im_start|>assistant\n" + ) + + def generate_with_backend(backend): + structured_outputs_params = StructuredOutputsParams( + json=schema, + backend=backend, + disable_any_whitespace=True, + disable_additional_properties=True, + ) + sampling_params = SamplingParams( + temperature=0, max_tokens=256, structured_outputs=structured_outputs_params + ) + + outputs = llm.generate(prompt, sampling_params=sampling_params) + assert outputs is not None + generated_text = outputs[0].outputs[0].text + assert generated_text is not None + parsed_json = json.loads(generated_text) + assert isinstance(parsed_json, dict) + jsonschema.validate(instance=parsed_json, schema=schema) + return parsed_json + + generated = generate_with_backend("guidance") + assert "a1" in generated + assert "a2" in generated + assert "a3" in generated + assert "a4" not in generated + assert "a5" not in generated + assert "a6" not in generated + + +@pytest.mark.parametrize("backend", ["guidance", "xgrammar", "outlines"]) +def test_structured_output_batched_with_non_structured_outputs_requests( + backend: str, +): + sample_json_schema = SAMPLE_JSON_SCHEMA + # Don't use eager execution on TPUs because we want to test for no + # recompilation at runtime + enforce_eager = bool(not current_platform.is_tpu()) + + llm = LLM( + model="meta-llama/Meta-Llama-3.1-8B-Instruct", + enforce_eager=enforce_eager, + max_model_len=1024, + structured_outputs_config=StructuredOutputsConfig( + backend=backend, + disable_any_whitespace=backend in {"xgrammar", "guidance"}, + ), + ) + + structured_outputs_prompt = ( + "Give an example JSON for an employee profile that fits this " + "schema. Make the response as short as possible. Schema: " + f"{sample_json_schema}" + ) + + non_structured_outputs_prompt = "The diameter of the Earth in kilometers is " + + prompts = [structured_outputs_prompt, non_structured_outputs_prompt] + sampling_params = [ + SamplingParams( + temperature=1.0, + max_tokens=400, + structured_outputs=StructuredOutputsParams(json=sample_json_schema), + ), + # No max tokens, temp=0 to assert on contents + SamplingParams( + seed=42, + temperature=0, + top_p=1.0, + ), + ] + + outputs = llm.generate( + prompts=prompts, sampling_params=sampling_params, use_tqdm=True + ) + + assert outputs is not None + + # Free memory as soon as possible as failed assertions + # will short circuit and not free up memory + del llm + torch.accelerator.empty_cache() + cleanup_dist_env_and_memory() + + for index, output in enumerate(outputs): + assert output is not None + assert isinstance(output, RequestOutput) + prompt = output.prompt + + generated_text = output.outputs[0].text + assert generated_text is not None + print(f"Prompt:\n{prompt!r}\nGenerated text:\n{generated_text!r}") + + if index == 0: + # First prompt is structured outputs, expect valid JSON + assert "\n" not in generated_text + output_json = json.loads(generated_text) + jsonschema.validate(instance=output_json, schema=sample_json_schema) + else: + # Second prompt is not structured outputs, expect valid output + # Cannot assert on exact output, but we can expect it to be factual + assert "12,742" in generated_text + + # non-structured outputs requests should not return a valid JSON here + with pytest.raises(ValueError): + output_json = json.loads(generated_text) + + +@pytest.mark.parametrize("backend", ["xgrammar"]) +def test_structured_output_with_structural_tag(backend: str): + llm = LLM( + model="Qwen/Qwen2.5-1.5B-Instruct", + structured_outputs_config=StructuredOutputsConfig(backend=backend), + ) + + structural_tag_config = { + "type": "structural_tag", + "format": { + "type": "triggered_tags", + "tags": [ + {"begin": "hello_flag", "content": {"type": "any_text"}, "end": "hello"} + ], + "triggers": ["hello"], + "stop_after_first": False, + }, + } + + sampling_params = SamplingParams( + temperature=0.0, + max_tokens=500, + structured_outputs=StructuredOutputsParams( + structural_tag=json.dumps(structural_tag_config) + ), + ) + + prompt = "Hello and repeat hello 10 times, do not say anything else. Only say hello hello hello, now start" + outputs = llm.generate(prompt, sampling_params=sampling_params, use_tqdm=True) + assert outputs is not None + for output in outputs: + assert output is not None + assert isinstance(output, RequestOutput) + prompt = output.prompt + generated_text = output.outputs[0].text + assert generated_text is not None + assert "hello_flag" in generated_text, ( + f"Expected 'hello_flag' to be in generated text, but got: {generated_text}" + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/conftest.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..10ca1baccc3162a1fedb14e5f30bc0a4055fa7f2 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/conftest.py @@ -0,0 +1,82 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from collections.abc import Callable, Iterator +from contextlib import contextmanager +from typing import Any + +import pytest + +# Test different image extensions (JPG/PNG) and formats (gray/RGB/RGBA) +TEST_IMAGE_ASSETS = [ + "2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg", # "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" + "Grayscale_8bits_palette_sample_image.png", # "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/Grayscale_8bits_palette_sample_image.png", + "1280px-Venn_diagram_rgb.svg.png", # "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/1280px-Venn_diagram_rgb.svg.png", + "RGBA_comp.png", # "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/RGBA_comp.png", +] + + +def _shutdown_llm(llm: Any, gpu_memory_utilization: float) -> None: + from vllm.distributed import cleanup_dist_env_and_memory + from vllm.platforms import current_platform + + try: + shutdown_timeout = 60.0 if current_platform.is_rocm() else None + llm.llm_engine.engine_core.shutdown(timeout=shutdown_timeout) + except Exception: + pass + + del llm + + try: + import torch + + torch._dynamo.reset() + except Exception: + pass + + cleanup_dist_env_and_memory() + + if current_platform.is_rocm(): + from tests.utils import wait_for_rocm_memory_to_settle + + wait_for_rocm_memory_to_settle(threshold_ratio=1.0 - gpu_memory_utilization) + + +@contextmanager +def managed_llm(*args: Any, **kwargs: Any) -> Iterator[Any]: + from vllm import LLM + + llm = LLM(*args, **kwargs) + gpu_memory_utilization = ( + llm.llm_engine.vllm_config.cache_config.gpu_memory_utilization + ) + try: + yield llm + finally: + _shutdown_llm(llm, gpu_memory_utilization) + + +def _make_managed_llm_factory() -> Iterator[Callable[..., Any]]: + from vllm import LLM + + llms: list[tuple[Any, float]] = [] + + def make_llm(*args: Any, **kwargs: Any) -> Any: + llm = LLM(*args, **kwargs) + gpu_memory_utilization = ( + llm.llm_engine.vllm_config.cache_config.gpu_memory_utilization + ) + llms.append((llm, gpu_memory_utilization)) + return llm + + try: + yield make_llm + finally: + while llms: + llm, gpu_memory_utilization = llms.pop() + _shutdown_llm(llm, gpu_memory_utilization) + + +@pytest.fixture +def multimodal_llm_factory() -> Iterator[Callable[..., Any]]: + yield from _make_managed_llm_factory() diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_chat.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_chat.py new file mode 100644 index 0000000000000000000000000000000000000000..3c9829121859d66b5d0ac11b8d49fae5d7c88985 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_chat.py @@ -0,0 +1,38 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest + +from tests.entrypoints.multimodal.conftest import TEST_IMAGE_ASSETS + + +@pytest.fixture(scope="function") +def vision_llm(multimodal_llm_factory): + return multimodal_llm_factory( + model="microsoft/Phi-3.5-vision-instruct", + max_model_len=4096, + max_num_seqs=5, + enforce_eager=True, + trust_remote_code=True, + limit_mm_per_prompt={"image": 2}, + seed=0, + ) + + +@pytest.mark.parametrize( + "image_urls", [[TEST_IMAGE_ASSETS[0], TEST_IMAGE_ASSETS[1]]], indirect=True +) +def test_chat_multi_image(vision_llm, image_urls: list[str]): + messages = [ + { + "role": "user", + "content": [ + *( + {"type": "image_url", "image_url": {"url": image_url}} + for image_url in image_urls + ), + {"type": "text", "text": "What's in this image?"}, + ], + } + ] + outputs = vision_llm.chat(messages) + assert len(outputs) >= 0 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_mm_cache_external_injection.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_mm_cache_external_injection.py new file mode 100644 index 0000000000000000000000000000000000000000..43ac7db96bb13f96d4fda3ed839b86603ac7894a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_mm_cache_external_injection.py @@ -0,0 +1,195 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Test that ``InputProcessor.inject_into_mm_cache()`` correctly injects +pre-processed mm_kwargs into the processor cache and reports MM cache +hit rate metrics accurately. + +This is used by frameworks like Dynamo that run the HF processor on a +frontend and transfer pre-processed mm_kwargs to the backend, avoiding +redundant processing. +""" + +import logging + +import pytest +import regex as re + +from tests.entrypoints.multimodal.conftest import TEST_IMAGE_ASSETS +from vllm import LLM, SamplingParams +from vllm.renderers.params import ChatParams +from vllm.v1.metrics import loggers as stat_loggers +from vllm.v1.metrics.reader import Counter, Metric + + +def _make_messages(image_url: str): + return [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": {"url": image_url}, + }, + ], + } + ] + + +def _get_counter_value(metrics: list[Metric], name: str): + metric = next(m for m in metrics if m.name == name) + assert isinstance(metric, Counter) + return metric.value + + +def _get_mm_cache_stats(metrics: list[Metric]): + mm_cache_queries = _get_counter_value(metrics, "vllm:mm_cache_queries") + mm_cache_hits = _get_counter_value(metrics, "vllm:mm_cache_hits") + return mm_cache_queries, mm_cache_hits + + +def _get_mm_cache_log(llm: LLM, caplog_vllm: pytest.LogCaptureFixture) -> float: + caplog_vllm.clear() + with caplog_vllm.at_level(logging.INFO, logger=stat_loggers.__name__): + llm.llm_engine.do_log_stats() + + assert len(caplog_vllm.records) == 1 + msg = caplog_vllm.records[0].getMessage() + + assert "MM cache hit rate" in msg + match = re.search(r"MM cache hit rate: ([0-9.]+)%", msg) + assert match is not None + return float(match.group(1)) + + +@pytest.mark.parametrize("image_urls", [TEST_IMAGE_ASSETS[:2]], indirect=True) +@pytest.mark.parametrize("mm_processor_cache_type", ["lru", "shm"]) +def test_inject_into_mm_cache( + num_gpus_available, + image_urls, + mm_processor_cache_type, + caplog_vllm, + multimodal_llm_factory, +): + """Test that inject_into_mm_cache() injects pre-processed mm_kwargs into + the processor cache and MM cache hit metrics are updated correctly. + + Steps: + 1. Two normal requests (same image) -> cache miss then hit (baseline) + 2. Extract cached kwargs, call inject_into_mm_cache with a new hash, + then generate with a pre-rendered input -> verifies injection works + """ + llm = multimodal_llm_factory( + model="llava-hf/llava-1.5-7b-hf", + max_model_len=4096, + max_num_seqs=5, + enforce_eager=True, + disable_log_stats=False, + limit_mm_per_prompt={"image": 2}, + mm_processor_cache_type=mm_processor_cache_type, + ) + + # Step 1: Normal requests to populate the cache + llm.chat(_make_messages(image_urls[0])) + assert _get_mm_cache_stats(llm.get_metrics()) == (1, 0) + + llm.chat(_make_messages(image_urls[0])) + assert _get_mm_cache_stats(llm.get_metrics()) == (2, 1) + assert _get_mm_cache_log(llm, caplog_vllm) == pytest.approx(50.0) + + # Step 2: Use a second image to get valid expanded tokens and + # placeholder positions via the renderer. + llm.chat(_make_messages(image_urls[1])) + queries_before = _get_mm_cache_stats(llm.get_metrics())[0] # 3 + + renderer = llm.llm_engine.renderer + cache = renderer.mm_processor_cache + assert cache is not None, "Processor cache should be enabled" + + _, eng_prompts = renderer.render_chat( + [_make_messages(image_urls[1])], + ChatParams(), + ) + eng_input = eng_prompts[0] + + # Inject pre-processed mm_kwargs with a NEW hash via public API + new_mm_hash = "deadbeef" * 8 + mm_hashes = {"image": [new_mm_hash]} + mm_kwargs = eng_input["mm_kwargs"] + + llm.llm_engine.input_processor.inject_into_mm_cache(mm_hashes, mm_kwargs) + + # Build pre-rendered input (no externally_processed flag needed) + pre_rendered_input = { + "type": "multimodal", + "prompt_token_ids": eng_input["prompt_token_ids"], + "mm_kwargs": mm_kwargs, + "mm_hashes": mm_hashes, + "mm_placeholders": eng_input["mm_placeholders"], + } + + llm.generate( + pre_rendered_input, + sampling_params=SamplingParams(max_tokens=1), + ) + + # Verify cache was queried and injection happened + queries_after = _get_mm_cache_stats(llm.get_metrics())[0] + assert queries_after > queries_before, ( + "Cache should have been queried for the injected item" + ) + mm_rate = _get_mm_cache_log(llm, caplog_vllm) + assert mm_rate >= 0.0, "MM cache hit rate should be reported" + + +@pytest.mark.parametrize("image_urls", [TEST_IMAGE_ASSETS[:1]], indirect=True) +def test_inject_into_mm_cache_without_cache( + num_gpus_available, + image_urls, + multimodal_llm_factory, +): + """Test that inject_into_mm_cache works gracefully when processor cache + is disabled (mm_processor_cache_gb=0). Should not crash. + """ + llm = multimodal_llm_factory( + model="llava-hf/llava-1.5-7b-hf", + max_model_len=4096, + max_num_seqs=5, + enforce_eager=True, + disable_log_stats=False, + limit_mm_per_prompt={"image": 2}, + mm_processor_cache_gb=0, + ) + + # Run a normal chat request first to warm up the model. + llm.chat(_make_messages(image_urls[0])) + + # Use the renderer to get a proper EngineInput with expanded tokens + renderer = llm.llm_engine.renderer + _, eng_prompts = renderer.render_chat( + [_make_messages(image_urls[0])], + ChatParams(), + ) + eng_input = eng_prompts[0] + + mm_hashes = {"image": ["abcd1234" * 8]} + mm_kwargs = eng_input["mm_kwargs"] + + # inject_into_mm_cache should not crash even without cache + llm.llm_engine.input_processor.inject_into_mm_cache(mm_hashes, mm_kwargs) + + # Build and generate with pre-rendered input + pre_rendered_input = { + "type": "multimodal", + "prompt_token_ids": eng_input["prompt_token_ids"], + "mm_kwargs": mm_kwargs, + "mm_hashes": mm_hashes, + "mm_placeholders": eng_input["mm_placeholders"], + } + + result = llm.generate( + pre_rendered_input, + sampling_params=SamplingParams(max_tokens=1), + ) + assert len(result) == 1, "Should produce one output" + assert len(result[0].outputs) >= 1, "Should have at least one output sequence" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_mm_cache_stats.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_mm_cache_stats.py new file mode 100644 index 0000000000000000000000000000000000000000..42dd9bcb740e3526eca3e2956b4899d5cb7de26f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_mm_cache_stats.py @@ -0,0 +1,98 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import logging + +import pytest +import regex as re + +from tests.entrypoints.multimodal.conftest import TEST_IMAGE_ASSETS +from vllm import LLM +from vllm.entrypoints.chat_utils import ChatCompletionMessageParam +from vllm.v1.metrics import loggers as stat_loggers +from vllm.v1.metrics.reader import Counter, Metric + + +def _make_messages(image_url: str) -> list[ChatCompletionMessageParam]: + return [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": {"url": image_url}, + }, + ], + } + ] + + +def _get_counter_value(metrics: list[Metric], name: str): + metric = next(m for m in metrics if m.name == name) + assert isinstance(metric, Counter) + return metric.value + + +def _get_mm_cache_stats(metrics: list[Metric]): + mm_cache_queries = _get_counter_value(metrics, "vllm:mm_cache_queries") + mm_cache_hits = _get_counter_value(metrics, "vllm:mm_cache_hits") + + return mm_cache_queries, mm_cache_hits + + +def _get_mm_cache_log(llm: LLM, caplog_vllm: pytest.LogCaptureFixture) -> float: + caplog_vllm.clear() + with caplog_vllm.at_level(logging.INFO, logger=stat_loggers.__name__): + llm.llm_engine.do_log_stats() + + assert len(caplog_vllm.records) == 1 + msg = caplog_vllm.records[0].getMessage() + + assert "MM cache hit rate" in msg + match = re.search(r"MM cache hit rate: ([0-9.]+)%", msg) + assert match is not None + return float(match.group(1)) + + +@pytest.mark.parametrize("image_urls", [TEST_IMAGE_ASSETS[:2]], indirect=True) +@pytest.mark.parametrize("mm_processor_cache_type", ["lru", "shm"]) +def test_mm_cache_stats( + num_gpus_available, + image_urls, + mm_processor_cache_type, + caplog_vllm, + multimodal_llm_factory, +): + llm = multimodal_llm_factory( + model="llava-hf/llava-1.5-7b-hf", + max_model_len=4096, + max_num_seqs=5, + enforce_eager=True, + mm_processor_cache_type=mm_processor_cache_type, + disable_log_stats=False, + limit_mm_per_prompt={"image": 2}, + ) + + llm.chat(_make_messages(image_urls[0])) + assert _get_mm_cache_stats(llm.get_metrics()) == (1, 0) + assert _get_mm_cache_log(llm, caplog_vllm) == pytest.approx(0.0) + + llm.chat(_make_messages(image_urls[1])) + assert _get_mm_cache_stats(llm.get_metrics()) == (2, 0) + assert _get_mm_cache_log(llm, caplog_vllm) == pytest.approx(0.0) + + llm.chat(_make_messages(image_urls[0])) + assert _get_mm_cache_stats(llm.get_metrics()) == (3, 1) + assert _get_mm_cache_log(llm, caplog_vllm) == pytest.approx(33.3) + + # NOTE: This only resets hit rate stats in CachingMetrics + # The raw queries and hits counts remain unaffected + llm.reset_mm_cache() + + llm.chat(_make_messages(image_urls[0])) + assert _get_mm_cache_stats(llm.get_metrics()) == (4, 1) + assert _get_mm_cache_log(llm, caplog_vllm) == pytest.approx(0.0) + + llm.chat(_make_messages(image_urls[1])) + assert _get_mm_cache_stats(llm.get_metrics()) == (5, 1) + assert _get_mm_cache_log(llm, caplog_vllm) == pytest.approx(0.0) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_mm_embeds_only.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_mm_embeds_only.py new file mode 100644 index 0000000000000000000000000000000000000000..3ac31b7984fc7e343653fb012f8ce38469f42da1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_mm_embeds_only.py @@ -0,0 +1,60 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from tests.entrypoints.multimodal.conftest import managed_llm +from vllm import LLM, SamplingParams +from vllm.assets.image import ImageAsset + +MODEL = "llava-hf/llava-1.5-7b-hf" +PROMPT = "USER: \nDescribe this image briefly.\nASSISTANT:" +TEXT_ONLY_PROMPT = "USER: What is 2 + 2?\nASSISTANT:" + + +@pytest.fixture(scope="module") +def llm(): + """LLM with enable_mm_embeds=True and all modality limits zeroed out.""" + with managed_llm( + model=MODEL, + max_model_len=2048, + enforce_eager=True, + gpu_memory_utilization=0.8, + enable_mm_embeds=True, + limit_mm_per_prompt={"image": 0}, + ) as llm: + yield llm + + +@pytest.mark.skip_global_cleanup +def test_generate_with_embedding(llm: LLM): + """Pre-computed embedding produces tokens without hanging.""" + embedding = ImageAsset("stop_sign").image_embeds + outputs = llm.generate( + {"prompt": PROMPT, "multi_modal_data": {"image": embedding}}, + sampling_params=SamplingParams(max_tokens=32, temperature=0.0), + ) + assert len(outputs) == 1 + assert len(outputs[0].outputs[0].text) > 0 + + +@pytest.mark.skip_global_cleanup +def test_raw_image_rejected(llm: LLM): + """Raw image input is still rejected when limit=0.""" + raw_image = ImageAsset("stop_sign").pil_image + with pytest.raises(ValueError, match=r"At most 0 image\(s\)"): + llm.generate( + {"prompt": PROMPT, "multi_modal_data": {"image": raw_image}}, + sampling_params=SamplingParams(max_tokens=16), + ) + + +@pytest.mark.skip_global_cleanup +def test_text_only_prompt(llm: LLM): + """Text-only prompts still work under this config.""" + outputs = llm.generate( + TEXT_ONLY_PROMPT, + sampling_params=SamplingParams(max_tokens=16, temperature=0.0), + ) + assert len(outputs) == 1 + assert len(outputs[0].outputs[0].text) > 0 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_mm_processor_kwargs.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_mm_processor_kwargs.py new file mode 100644 index 0000000000000000000000000000000000000000..b9dc6f0d690c95467506a3f6c6950c50e3a10fe7 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/llm/test_mm_processor_kwargs.py @@ -0,0 +1,288 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from vllm import LLM, SamplingParams + + +def _make_mock_llm() -> LLM: + llm = object.__new__(LLM) + llm.model_config = SimpleNamespace( + runner_type="generate", enable_prompt_embeds=False + ) + return llm + + +def test_generate_forwards_mm_processor_kwargs() -> None: + llm = _make_mock_llm() + mm_processor_kwargs = {"num_crops": 4} + sampling_params = SamplingParams(max_tokens=1) + + llm._run_completion = Mock(return_value=["ok"]) + + outputs = llm.generate( + "prompt", + sampling_params=sampling_params, + mm_processor_kwargs=mm_processor_kwargs, + ) + + assert outputs == ["ok"] + assert llm._run_completion.call_args.kwargs["mm_processor_kwargs"] == ( + mm_processor_kwargs + ) + + +def test_enqueue_forwards_mm_processor_kwargs() -> None: + llm = _make_mock_llm() + mm_processor_kwargs = {"do_resize": False} + sampling_params = SamplingParams(max_tokens=1) + + llm._add_completion_requests = Mock(return_value=["req-0"]) + + request_ids = llm.enqueue( + "prompt", + sampling_params=sampling_params, + use_tqdm=False, + mm_processor_kwargs=mm_processor_kwargs, + ) + + assert request_ids == ["req-0"] + assert llm._add_completion_requests.call_args.kwargs["mm_processor_kwargs"] == ( + mm_processor_kwargs + ) + + +def test_chat_forwards_mm_processor_kwargs() -> None: + llm = _make_mock_llm() + mm_processor_kwargs = {"do_pan_and_scan": True} + sampling_params = SamplingParams(max_tokens=1) + messages = [{"role": "user", "content": "hello"}] + + llm._run_chat = Mock(return_value=["ok"]) + + outputs = llm.chat( + messages, + sampling_params=sampling_params, + mm_processor_kwargs=mm_processor_kwargs, + ) + + assert outputs == ["ok"] + assert llm._run_chat.call_args.kwargs["mm_processor_kwargs"] == ( + mm_processor_kwargs + ) + + +def test_enqueue_chat_forwards_mm_processor_kwargs() -> None: + llm = _make_mock_llm() + mm_processor_kwargs = {"do_pan_and_scan": True} + sampling_params = SamplingParams(max_tokens=1) + messages = [{"role": "user", "content": "hello"}] + + llm._add_chat_requests = Mock(return_value=["req-0"]) + + request_ids = llm.enqueue_chat( + messages, + sampling_params=sampling_params, + use_tqdm=False, + mm_processor_kwargs=mm_processor_kwargs, + ) + + assert request_ids == ["req-0"] + assert llm._add_chat_requests.call_args.kwargs["mm_processor_kwargs"] == ( + mm_processor_kwargs + ) + + +def test_run_chat_forwards_mm_processor_kwargs() -> None: + llm = _make_mock_llm() + mm_processor_kwargs = {"num_crops": 8} + sampling_params = SamplingParams(max_tokens=1) + messages = [{"role": "user", "content": "hello"}] + sentinel_output = ["done"] + + llm._add_chat_requests = Mock() + llm._run_engine = Mock(return_value=sentinel_output) + + outputs = llm._run_chat( + messages=messages, + params=sampling_params, + output_type=object, + use_tqdm=False, + mm_processor_kwargs=mm_processor_kwargs, + ) + + assert outputs == sentinel_output + assert llm._add_chat_requests.call_args.kwargs["mm_processor_kwargs"] == ( + mm_processor_kwargs + ) + + +def test_run_completion_forwards_mm_processor_kwargs() -> None: + llm = _make_mock_llm() + mm_processor_kwargs = {"min_pixels": 4 * 28 * 28} + sampling_params = SamplingParams(max_tokens=1) + sentinel_output = ["done"] + + llm._add_completion_requests = Mock() + llm._run_engine = Mock(return_value=sentinel_output) + + outputs = llm._run_completion( + prompts=["prompt"], + params=sampling_params, + output_type=object, + use_tqdm=False, + mm_processor_kwargs=mm_processor_kwargs, + ) + + assert outputs == sentinel_output + assert llm._add_completion_requests.call_args.kwargs["mm_processor_kwargs"] == ( + mm_processor_kwargs + ) + + +def test_add_completion_requests_forwards_mm_processor_kwargs() -> None: + llm = _make_mock_llm() + mm_processor_kwargs = {"max_dynamic_patch": 4} + sampling_params = SamplingParams(max_tokens=1) + + llm._params_to_seq = Mock(return_value=[sampling_params]) + llm._lora_request_to_seq = Mock(return_value=[None]) + llm._priority_to_seq = Mock(return_value=[0]) + llm._preprocess_cmpl_one = Mock(return_value={"prompt_token_ids": [1]}) + + captured_prompts = [] + + def fake_render_and_add_requests(*, prompts, **_kwargs): + captured_prompts.extend(prompts) + return ["req-0"] + + llm._render_and_add_requests = Mock(side_effect=fake_render_and_add_requests) + + request_ids = llm._add_completion_requests( + prompts=["prompt"], + params=sampling_params, + use_tqdm=False, + mm_processor_kwargs=mm_processor_kwargs, + ) + + assert request_ids == ["req-0"] + llm._preprocess_cmpl_one.assert_called_once_with( + "prompt", + None, + mm_processor_kwargs=mm_processor_kwargs, + ) + assert captured_prompts == [{"prompt_token_ids": [1]}] + + +def test_preprocess_cmpl_applies_mm_processor_kwargs_to_renderer( + monkeypatch: pytest.MonkeyPatch, +) -> None: + llm = _make_mock_llm() + mm_processor_kwargs = {"num_crops": 8} + prompt = {"prompt": "", "multi_modal_data": {"image": object()}} + + renderer = Mock() + renderer.default_cmpl_tok_params = Mock() + renderer.default_cmpl_tok_params.with_kwargs.return_value = "tok-params" + renderer.render_cmpl.return_value = ["engine-input"] + llm.renderer = renderer + + monkeypatch.setattr( + "vllm.entrypoints.offline_utils.parse_model_prompt", + lambda _model_config, parsed_prompt: parsed_prompt, + ) + + outputs = llm._preprocess_cmpl( + [prompt], + mm_processor_kwargs=mm_processor_kwargs, + ) + + assert outputs == ["engine-input"] + renderer.render_cmpl.assert_called_once_with( + [prompt], + "tok-params", + prompt_extras={"mm_processor_kwargs": mm_processor_kwargs}, + ) + + +def test_preprocess_cmpl_keeps_prompt_mm_processor_kwargs_when_no_override( + monkeypatch: pytest.MonkeyPatch, +) -> None: + llm = _make_mock_llm() + prompt = { + "prompt": "", + "multi_modal_data": {"image": object()}, + "mm_processor_kwargs": {"num_crops": 2}, + } + + renderer = Mock() + renderer.default_cmpl_tok_params = Mock() + renderer.default_cmpl_tok_params.with_kwargs.return_value = "tok-params" + renderer.render_cmpl.return_value = ["engine-input"] + llm.renderer = renderer + + monkeypatch.setattr( + "vllm.entrypoints.offline_utils.parse_model_prompt", + lambda _model_config, parsed_prompt: parsed_prompt, + ) + + outputs = llm._preprocess_cmpl([prompt]) + + assert outputs == ["engine-input"] + renderer.render_cmpl.assert_called_once_with( + [prompt], + "tok-params", + prompt_extras=None, + ) + + +def test_preprocess_chat_applies_mm_processor_kwargs_to_renderer() -> None: + llm = _make_mock_llm() + mm_processor_kwargs = {"num_crops": 8} + messages = [[{"role": "user", "content": "Describe this image."}]] + + renderer = Mock() + renderer.tokenizer = object() + renderer.default_chat_tok_params = Mock() + renderer.default_chat_tok_params.with_kwargs.return_value = "tok-params" + renderer.render_chat.return_value = (messages, ["engine-input"]) + llm.renderer = renderer + + outputs = llm._preprocess_chat( + messages, + mm_processor_kwargs=mm_processor_kwargs, + ) + + assert outputs == ["engine-input"] + call_args = renderer.render_chat.call_args + assert call_args.args[0] == messages + assert call_args.args[1].mm_processor_kwargs == mm_processor_kwargs + assert call_args.args[2] == "tok-params" + assert call_args.kwargs["prompt_extras"] == { + "mm_processor_kwargs": mm_processor_kwargs + } + + +def test_preprocess_chat_omits_mm_processor_kwargs_when_no_override() -> None: + llm = _make_mock_llm() + messages = [[{"role": "user", "content": "Describe this image."}]] + + renderer = Mock() + renderer.tokenizer = object() + renderer.default_chat_tok_params = Mock() + renderer.default_chat_tok_params.with_kwargs.return_value = "tok-params" + renderer.render_chat.return_value = (messages, ["engine-input"]) + llm.renderer = renderer + + outputs = llm._preprocess_chat(messages) + + assert outputs == ["engine-input"] + call_args = renderer.render_chat.call_args + assert call_args.args[0] == messages + assert call_args.args[1].mm_processor_kwargs is None + assert call_args.args[2] == "tok-params" + assert call_args.kwargs["prompt_extras"] is None diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_audio.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_audio.py new file mode 100644 index 0000000000000000000000000000000000000000..4ac506bb9d7402ca1e4f80859d418e18bb9e63b8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_audio.py @@ -0,0 +1,397 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import openai +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer +from vllm.assets.audio import AudioAsset +from vllm.multimodal.utils import encode_audio_base64, encode_audio_url, fetch_audio + +MODEL_NAME = "fixie-ai/ultravox-v0_5-llama-3_2-1b" +TEST_AUDIO_URLS = [ + AudioAsset("winning_call").url, + AudioAsset("mary_had_lamb").url, +] +MAXIMUM_AUDIOS = 2 + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--dtype", + "float32", + "--max-model-len", + "2048", + "--max-num-seqs", + "5", + "--enforce-eager", + "--trust-remote-code", + "--limit-mm-per-prompt", + json.dumps({"audio": MAXIMUM_AUDIOS}), + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.fixture(scope="session") +def base64_encoded_audio() -> dict[str, str]: + return { + audio_url: encode_audio_base64(*fetch_audio(audio_url)) + for audio_url in TEST_AUDIO_URLS + } + + +@pytest.fixture(scope="session") +def url_encoded_audio() -> dict[str, str]: + return { + audio_url: encode_audio_url(*fetch_audio(audio_url)) + for audio_url in TEST_AUDIO_URLS + } + + +def dummy_messages_from_audio_url( + audio_urls: str | list[str], + content_text: str = "What's happening in this audio?", +): + if isinstance(audio_urls, str): + audio_urls = [audio_urls] + + return [ + { + "role": "user", + "content": [ + *( + {"type": "audio_url", "audio_url": {"url": audio_url}} + for audio_url in audio_urls + ), + {"type": "text", "text": content_text}, + ], + } + ] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("audio_url", [TEST_AUDIO_URLS[0]]) +async def test_single_chat_session_audio( + client: openai.AsyncOpenAI, model_name: str, audio_url: str +): + messages = dummy_messages_from_audio_url(audio_url) + + # test single completion + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + logprobs=True, + temperature=0.0, + top_logprobs=5, + ) + assert len(chat_completion.choices) == 1 + + choice = chat_completion.choices[0] + assert choice.finish_reason == "length" + assert chat_completion.usage == openai.types.CompletionUsage( + completion_tokens=10, prompt_tokens=202, total_tokens=212 + ) + + message = choice.message + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 10 + assert message.role == "assistant" + messages.append({"role": "assistant", "content": message.content}) + + # test multi-turn dialogue + messages.append({"role": "user", "content": "express your result in json"}) + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + ) + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("audio_url", [TEST_AUDIO_URLS[0]]) +async def test_error_on_invalid_audio_url_type( + client: openai.AsyncOpenAI, model_name: str, audio_url: str +): + messages = [ + { + "role": "user", + "content": [ + {"type": "audio_url", "audio_url": audio_url}, + {"type": "text", "text": "What's happening in this audio?"}, + ], + } + ] + + # audio_url should be a dict {"url": "some url"}, not directly a string + with pytest.raises(openai.BadRequestError): + _ = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("audio_url", [TEST_AUDIO_URLS[0]]) +async def test_single_chat_session_audio_base64encoded( + client: openai.AsyncOpenAI, + model_name: str, + audio_url: str, + url_encoded_audio: dict[str, str], +): + messages = dummy_messages_from_audio_url(url_encoded_audio[audio_url]) + + # test single completion + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + logprobs=True, + temperature=0.0, + top_logprobs=5, + ) + assert len(chat_completion.choices) == 1 + + choice = chat_completion.choices[0] + assert choice.finish_reason == "length" + assert chat_completion.usage == openai.types.CompletionUsage( + completion_tokens=10, prompt_tokens=202, total_tokens=212 + ) + + message = choice.message + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 10 + assert message.role == "assistant" + messages.append({"role": "assistant", "content": message.content}) + + # test multi-turn dialogue + messages.append({"role": "user", "content": "express your result in json"}) + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + ) + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("audio_url", [TEST_AUDIO_URLS[0]]) +async def test_single_chat_session_input_audio( + client: openai.AsyncOpenAI, + model_name: str, + audio_url: str, + base64_encoded_audio: dict[str, str], +): + messages = [ + { + "role": "user", + "content": [ + { + "type": "input_audio", + "input_audio": { + "data": base64_encoded_audio[audio_url], + "format": "wav", + }, + }, + {"type": "text", "text": "What's happening in this audio?"}, + ], + } + ] + + # test single completion + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + logprobs=True, + top_logprobs=5, + ) + assert len(chat_completion.choices) == 1 + + choice = chat_completion.choices[0] + assert choice.finish_reason == "length" + assert chat_completion.usage == openai.types.CompletionUsage( + completion_tokens=10, prompt_tokens=202, total_tokens=212 + ) + + message = choice.message + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 10 + assert message.role == "assistant" + messages.append({"role": "assistant", "content": message.content}) + + # test multi-turn dialogue + messages.append({"role": "user", "content": "express your result in json"}) + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + ) + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("audio_url", TEST_AUDIO_URLS) +async def test_chat_streaming_audio( + client: openai.AsyncOpenAI, model_name: str, audio_url: str +): + messages = dummy_messages_from_audio_url( + audio_url, "What's a short title for this audio?" + ) + + # test single completion + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=8, + temperature=0.0, + ) + output = chat_completion.choices[0].message.content + stop_reason = chat_completion.choices[0].finish_reason + + # test streaming + stream = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=8, + temperature=0.0, + stream=True, + ) + chunks: list[str] = [] + finish_reason_count = 0 + async for chunk in stream: + delta = chunk.choices[0].delta + if delta.role: + assert delta.role == "assistant" + if delta.content: + chunks.append(delta.content) + if chunk.choices[0].finish_reason is not None: + finish_reason_count += 1 + # finish reason should only return in last block + assert finish_reason_count == 1 + assert chunk.choices[0].finish_reason == stop_reason + assert delta.content + assert "".join(chunks) == output + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("audio_url", TEST_AUDIO_URLS) +async def test_chat_streaming_input_audio( + client: openai.AsyncOpenAI, + model_name: str, + audio_url: str, + base64_encoded_audio: dict[str, str], +): + messages = [ + { + "role": "user", + "content": [ + { + "type": "input_audio", + "input_audio": { + "data": base64_encoded_audio[audio_url], + "format": "wav", + }, + }, + {"type": "text", "text": "What's a short title for this audio?"}, + ], + } + ] + + # test single completion + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=8, + temperature=0.0, + ) + output = chat_completion.choices[0].message.content + stop_reason = chat_completion.choices[0].finish_reason + + # test streaming + stream = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=8, + temperature=0.0, + stream=True, + ) + chunks: list[str] = [] + finish_reason_count = 0 + async for chunk in stream: + delta = chunk.choices[0].delta + if delta.role: + assert delta.role == "assistant" + if delta.content: + chunks.append(delta.content) + if chunk.choices[0].finish_reason is not None: + finish_reason_count += 1 + # finish reason should only return in last block + assert finish_reason_count == 1 + assert chunk.choices[0].finish_reason == stop_reason + assert delta.content + assert "".join(chunks) == output + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize( + "audio_urls", [TEST_AUDIO_URLS, TEST_AUDIO_URLS + [TEST_AUDIO_URLS[0]]] +) +async def test_multi_audio_input( + client: openai.AsyncOpenAI, model_name: str, audio_urls: list[str] +): + messages = dummy_messages_from_audio_url(audio_urls) + + if len(audio_urls) > MAXIMUM_AUDIOS: + with pytest.raises(openai.BadRequestError): # test multi-audio input + await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + ) + + # the server should still work afterwards + completion = await client.completions.create( + model=model_name, + prompt=[0, 0, 0, 0, 0], + max_tokens=5, + temperature=0.0, + ) + completion = completion.choices[0].text + assert completion is not None and len(completion) >= 0 + else: + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + ) + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 0 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_audio_in_video.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_audio_in_video.py new file mode 100644 index 0000000000000000000000000000000000000000..c2da38d996439585c631da85de38dacc304643d0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_audio_in_video.py @@ -0,0 +1,194 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import openai +import pybase64 as base64 +import pytest +import pytest_asyncio + +from tests.conftest import VideoTestAssets +from tests.utils import ROCM_EXTRA_ARGS, RemoteOpenAIServer + +MODEL_NAME = "Qwen/Qwen2.5-Omni-3B" + + +@pytest.fixture(scope="module") +def server(): + # Use module scope so the server is started once and shared across all + # tests in this file. Starting a new vLLM server per test on XPU can + # cause the second server startup to hang silently and exceed the + # wait-for-server timeout, resulting in RuntimeError. + args = [ + "--max-model-len", + "16384", + "--enforce-eager", + "--limit-mm-per-prompt", + json.dumps({"audio": 3, "video": 3}), + *ROCM_EXTRA_ARGS, + ] + + with RemoteOpenAIServer( + MODEL_NAME, + args, + ) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.core_model +@pytest.mark.asyncio +async def test_online_audio_in_video( + client: openai.AsyncOpenAI, video_assets: VideoTestAssets +): + """Test video input with `audio_in_video=True`""" + + # we don't use video_urls above because they missed audio stream. + video_path = video_assets[0].video_path + with open(video_path, "rb") as f: + video_base64 = base64.b64encode(f.read()).decode("utf-8") + + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What's in this video?"}, + { + "type": "video_url", + "video_url": {"url": f"data:video/mp4;base64,{video_base64}"}, + }, + ], + } + ] + + # multi-turn to test mm processor cache as well + for turn in range(2): + chat_completion = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_tokens=8, + temperature=0.0, + extra_body={ + "mm_processor_kwargs": { + "use_audio_in_video": True, + } + }, + ) + + assert len(chat_completion.choices) == 1 + choice = chat_completion.choices[0] + print( + f"[DEBUG][single-video] turn={turn} " + f"finish_reason={choice.finish_reason!r} " + f"content={choice.message.content!r} " + f"usage={chat_completion.usage}" + ) + assert choice.finish_reason == "length" + + +@pytest.mark.core_model +@pytest.mark.asyncio +async def test_online_audio_in_video_multi_videos( + client: openai.AsyncOpenAI, video_assets: VideoTestAssets +): + """Test multi-video input with `audio_in_video=True`""" + + # we don't use video_urls above because they missed audio stream. + video_path = video_assets[0].video_path + with open(video_path, "rb") as f: + video_base64 = base64.b64encode(f.read()).decode("utf-8") + + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What's in these two videos?"}, + { + "type": "video_url", + "video_url": {"url": f"data:video/mp4;base64,{video_base64}"}, + }, + { + "type": "video_url", + "video_url": {"url": f"data:video/mp4;base64,{video_base64}"}, + }, + ], + } + ] + + # multi-turn to test mm processor cache as well + for turn in range(2): + chat_completion = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_tokens=8, + temperature=0.0, + extra_body={ + "mm_processor_kwargs": { + "use_audio_in_video": True, + } + }, + ) + + assert len(chat_completion.choices) == 1 + choice = chat_completion.choices[0] + print( + f"[DEBUG][multi-video] turn={turn} " + f"finish_reason={choice.finish_reason!r} " + f"content={choice.message.content!r} " + f"usage={chat_completion.usage}" + ) + assert choice.finish_reason == "length" + + +@pytest.mark.core_model +@pytest.mark.asyncio +async def test_online_audio_in_video_interleaved( + client: openai.AsyncOpenAI, video_assets: VideoTestAssets +): + """Test interleaved video/audio input with `audio_in_video=True`""" + + # we don't use video_urls above because they missed audio stream. + video_path = video_assets[0].video_path + with open(video_path, "rb") as f: + video_base64 = base64.b64encode(f.read()).decode("utf-8") + + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What's in these two videos?"}, + { + "type": "video_url", + "video_url": {"url": f"data:video/mp4;base64,{video_base64}"}, + }, + { + "type": "audio_url", + "audio_url": {"url": f"data:audio/mp4;base64,{video_base64}"}, + }, + { + "type": "video_url", + "video_url": {"url": f"data:video/mp4;base64,{video_base64}"}, + }, + ], + } + ] + with pytest.raises( + openai.BadRequestError, + match="use_audio_in_video requires equal number of audio and video items", + ): + await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_tokens=16, + extra_body={ + "mm_processor_kwargs": { + "use_audio_in_video": True, + } + }, + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_chat_completion_with_image_embeds.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_chat_completion_with_image_embeds.py new file mode 100644 index 0000000000000000000000000000000000000000..27f5db9e3a6fc79ff22eb3f2c7e4ee31c95b0e5c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_chat_completion_with_image_embeds.py @@ -0,0 +1,86 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import openai # use the official client for correctness check +import pytest +import pytest_asyncio +import torch +from transformers import AutoConfig + +from tests.conftest import ImageTestAssets +from tests.utils import RemoteOpenAIServer +from vllm.utils.serial_utils import tensor2base64 + +# any model with a chat template should work here +MODEL_NAME = "llava-hf/llava-1.5-7b-hf" +CONFIG = AutoConfig.from_pretrained(MODEL_NAME) +MAXIMUM_IMAGES = 2 + + +@pytest.fixture(scope="module") +def default_image_embeds_server_args() -> list[str]: + return [ + "--dtype", + "bfloat16", + "--max-model-len", + "2048", + "--max-num-seqs", + "4", + "--enforce-eager", + "--limit-mm-per-prompt", + json.dumps({"image": MAXIMUM_IMAGES}), + "--enable-mm-embeds", + ] + + +@pytest.fixture(scope="module") +def server_with_image_embeds(default_image_embeds_server_args): + with RemoteOpenAIServer( + MODEL_NAME, default_image_embeds_server_args, max_wait_seconds=600 + ) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client_with_image_embeds(server_with_image_embeds): + async with server_with_image_embeds.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("dtype", [torch.half, torch.float16, torch.float32]) +async def test_chat_completions_with_image_embeds( + client_with_image_embeds: openai.AsyncOpenAI, + model_name: str, + image_assets: ImageTestAssets, + dtype: torch.dtype, +): + # Test case: Single image embeds input + image_embeds = image_assets[0].image_embeds.to(dtype=dtype) + base64_image_embedding = tensor2base64(image_embeds) + chat_completion = await client_with_image_embeds.chat.completions.create( + messages=[ + {"role": "system", "content": "You are a helpful assistant."}, + { + "role": "user", + "content": [ + { + "type": "text", + "text": "Describe these images separately. For each image," + "reply with a short sentence (no more than 10 words).", + }, + { + "type": "image_embeds", + "image_embeds": base64_image_embedding, + }, + ], + }, + ], + model=model_name, + ) + assert chat_completion.choices[0].message.content is not None + assert isinstance(chat_completion.choices[0].message.content, str) + assert len(chat_completion.choices[0].message.content) > 0 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_chat_completion_with_mixed_audio_embeds.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_chat_completion_with_mixed_audio_embeds.py new file mode 100644 index 0000000000000000000000000000000000000000..8fc93e1901662191d751a915d706e910b7704111 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_chat_completion_with_mixed_audio_embeds.py @@ -0,0 +1,190 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""E2E test for mixing `prompt_embeds` with `audio_embeds` in a single +Chat Completions request.""" + +import json + +import openai +import pytest +import pytest_asyncio +import safetensors +import torch +import torch.nn as nn +from huggingface_hub import hf_hub_download +from transformers import AutoConfig, AutoTokenizer + +from tests.utils import RemoteOpenAIServer +from vllm.utils.serial_utils import tensor2base64 + +QWEN2AUDIO_MODEL = "Qwen/Qwen2-Audio-7B-Instruct" + +# Use the model's native dtype to avoid an implicit cast inside +# `safe_load_prompt_embeds` (mismatched floating-point dtypes are cast to the +# model's dtype automatically, matching here just skips the conversion). +QWEN2AUDIO_DTYPE = torch.bfloat16 + + +@pytest.fixture(scope="module") +def qwen2audio_server_args() -> list[str]: + return [ + "--dtype", + "bfloat16", + "--max-model-len", + "2048", + "--max-num-seqs", + "4", + "--enforce-eager", + "--trust-remote-code", + "--gpu-memory-utilization", + "0.85", + "--limit-mm-per-prompt", + json.dumps({"audio": 1}), + "--enable-prompt-embeds", + "--enable-mm-embeds", + ] + + +@pytest.fixture(scope="module") +def qwen2audio_server(qwen2audio_server_args): + with RemoteOpenAIServer( + QWEN2AUDIO_MODEL, + qwen2audio_server_args, + max_wait_seconds=600, + ) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def qwen2audio_client(qwen2audio_server): + async with qwen2audio_server.get_async_client() as async_client: + yield async_client + + +@pytest.fixture(scope="module") +def qwen2audio_hidden_size() -> int: + config = AutoConfig.from_pretrained(QWEN2AUDIO_MODEL, trust_remote_code=True) + return config.text_config.hidden_size + + +@pytest.fixture(scope="module") +def qwen2audio_prompt_embeds_b64(qwen2audio_hidden_size: int) -> str: + tensor = torch.randn(4, qwen2audio_hidden_size, dtype=QWEN2AUDIO_DTYPE) + return tensor2base64(tensor) + + +@pytest.fixture(scope="module") +def qwen2audio_audio_embeds_b64(qwen2audio_hidden_size: int) -> str: + # Shape matches the `audio_embeds` unit-test fixture. + torch.manual_seed(0) + tensor = torch.randn(1, 128, qwen2audio_hidden_size, dtype=QWEN2AUDIO_DTYPE) + return tensor2base64(tensor) + + +@pytest.mark.asyncio +async def test_prompt_embeds_plus_audio_embeds( + qwen2audio_client: openai.AsyncOpenAI, + qwen2audio_prompt_embeds_b64: str, + qwen2audio_audio_embeds_b64: str, +): + """Single user message carrying both prompt_embeds and audio_embeds parts.""" + chat = await qwen2audio_client.chat.completions.create( + model=QWEN2AUDIO_MODEL, + max_tokens=5, + temperature=0.0, + messages=[ + { + "role": "user", + "content": [ + { + "type": "prompt_embeds", + "data": qwen2audio_prompt_embeds_b64, + }, + { + "type": "audio_embeds", + "audio_embeds": qwen2audio_audio_embeds_b64, + }, + {"type": "text", "text": "Continue."}, + ], + } + ], + ) + assert chat.choices[0].message.content is not None + assert len(chat.choices[0].message.content) > 0 + + +@pytest.fixture(scope="module") +def qwen2audio_aligned_content_and_embeds_b64() -> tuple[str, str]: + """Return `(content, base64_embeds)` where the embeddings are the model's + embedding of `content` tokenized WITHOUT special tokens. + + Loads only the `embed_tokens` shard from disk on CPU (~1.1 GB of host + RAM) instead of the full 7B model on GPU. + """ + content = "Describe this audio." + tokenizer = AutoTokenizer.from_pretrained(QWEN2AUDIO_MODEL, trust_remote_code=True) + + index_path = hf_hub_download(QWEN2AUDIO_MODEL, "model.safetensors.index.json") + with open(index_path) as f: + weight_map = json.load(f)["weight_map"] + embed_key = next(k for k in weight_map if k.endswith("embed_tokens.weight")) + shard_path = hf_hub_download(QWEN2AUDIO_MODEL, weight_map[embed_key]) + with safetensors.safe_open(shard_path, framework="pt", device="cpu") as f: + embed_weight = f.get_tensor(embed_key) + embed_layer = nn.Embedding.from_pretrained(embed_weight.to(QWEN2AUDIO_DTYPE)) + + ids = tokenizer(content, add_special_tokens=False, return_tensors="pt").input_ids + embeds = embed_layer(ids).squeeze(0) + return content, tensor2base64(embeds) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "audio_first", + [True, False], + ids=["audio_embeds-then-text", "text-then-audio_embeds"], +) +async def test_text_content_and_prompt_embeds_match_with_audio_embeds( + qwen2audio_client: openai.AsyncOpenAI, + qwen2audio_audio_embeds_b64: str, + qwen2audio_aligned_content_and_embeds_b64: tuple[str, str], + audio_first: bool, +): + """Same content as text vs `prompt_embeds` should yield identical Chat + Completions output when mixed with `audio_embeds` in the same message. + """ + content, encoded_text_embeds = qwen2audio_aligned_content_and_embeds_b64 + + audio_part = { + "type": "audio_embeds", + "audio_embeds": qwen2audio_audio_embeds_b64, + } + text_part = {"type": "text", "text": content} + embeds_part = {"type": "prompt_embeds", "data": encoded_text_embeds} + + if audio_first: + text_content = [audio_part, text_part] + embeds_content = [audio_part, embeds_part] + else: + text_content = [text_part, audio_part] + embeds_content = [embeds_part, audio_part] + + text_resp = await qwen2audio_client.chat.completions.create( + model=QWEN2AUDIO_MODEL, + max_tokens=10, + temperature=0.0, + messages=[{"role": "user", "content": text_content}], + ) + embeds_resp = await qwen2audio_client.chat.completions.create( + model=QWEN2AUDIO_MODEL, + max_tokens=10, + temperature=0.0, + messages=[{"role": "user", "content": embeds_content}], + ) + + text_out = text_resp.choices[0].message.content + embeds_out = embeds_resp.choices[0].message.content + assert text_out is not None and len(text_out) > 0 + assert embeds_out is not None and len(embeds_out) > 0 + assert text_out == embeds_out diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_chat_completion_with_mixed_image_embeds.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_chat_completion_with_mixed_image_embeds.py new file mode 100644 index 0000000000000000000000000000000000000000..26f2315b4b41fdbee336a72f3a017e4b21a8c9cf --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_chat_completion_with_mixed_image_embeds.py @@ -0,0 +1,212 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""E2E tests for mixing `prompt_embeds` with image content parts in a single +Chat Completions request. +""" + +import json + +import openai +import pytest +import pytest_asyncio +import safetensors +import torch +import torch.nn as nn +from huggingface_hub import hf_hub_download +from transformers import AutoTokenizer + +from tests.utils import RemoteOpenAIServer +from vllm.assets.image import ImageAsset +from vllm.multimodal.utils import encode_image_url +from vllm.utils.serial_utils import tensor2base64 + +MODEL_NAME = "Qwen/Qwen2-VL-2B-Instruct" + +# Use the model's native dtype to skip the implicit cast inside +# `safe_load_prompt_embeds` (mismatched floating-point dtypes are cast to the +# model's dtype automatically). +MODEL_DTYPE = torch.bfloat16 + + +@pytest.fixture(scope="module") +def server_args() -> list[str]: + return [ + "--dtype", + "bfloat16", + "--max-model-len", + "2048", + "--max-num-seqs", + "4", + "--enforce-eager", + "--gpu-memory-utilization", + "0.4", + "--limit-mm-per-prompt", + json.dumps({"image": 1}), + "--enable-prompt-embeds", + "--enable-mm-embeds", + ] + + +@pytest.fixture(scope="module") +def server(server_args): + with RemoteOpenAIServer( + MODEL_NAME, + server_args, + max_wait_seconds=600, + ) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.fixture(scope="module") +def image_url() -> str: + """Stable real image as a data URL, kept identical across both the + text and prompt_embeds requests so any output difference must come from + how the text content is delivered.""" + return encode_image_url(ImageAsset("stop_sign").pil_image) + + +@pytest.fixture(scope="module") +def aligned_content_and_embeds_b64() -> tuple[str, str]: + """`(content, base64_embeds)` where the embeddings are the model's + embedding of `content` tokenized WITHOUT special tokens. + + Loads only the `embed_tokens` shard from disk on CPU instead of the full + model on GPU, so the fixture has zero VRAM footprint and won't contend + with the running vLLM server. + """ + content = "Describe this image." + tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True) + + index_path = hf_hub_download(MODEL_NAME, "model.safetensors.index.json") + with open(index_path) as f: + weight_map = json.load(f)["weight_map"] + embed_key = next(k for k in weight_map if k.endswith("embed_tokens.weight")) + shard_path = hf_hub_download(MODEL_NAME, weight_map[embed_key]) + with safetensors.safe_open(shard_path, framework="pt", device="cpu") as f: + embed_weight = f.get_tensor(embed_key) + embed_layer = nn.Embedding.from_pretrained(embed_weight.to(MODEL_DTYPE)) + + ids = tokenizer(content, add_special_tokens=False, return_tensors="pt").input_ids + embeds = embed_layer(ids).squeeze(0) + return content, tensor2base64(embeds) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "image_first", + [True, False], + ids=["image_url-then-text", "text-then-image_url"], +) +async def test_text_content_and_prompt_embeds_match_with_image_url( + client: openai.AsyncOpenAI, + image_url: str, + aligned_content_and_embeds_b64: tuple[str, str], + image_first: bool, +): + """Same content as text vs `prompt_embeds` should yield identical Chat + Completions output when mixed with an `image_url` part in the same + message under greedy decoding. + """ + content, encoded_text_embeds = aligned_content_and_embeds_b64 + + image_part = {"type": "image_url", "image_url": {"url": image_url}} + text_part = {"type": "text", "text": content} + embeds_part = {"type": "prompt_embeds", "data": encoded_text_embeds} + + if image_first: + text_content = [image_part, text_part] + embeds_content = [image_part, embeds_part] + else: + text_content = [text_part, image_part] + embeds_content = [embeds_part, image_part] + + text_resp = await client.chat.completions.create( + model=MODEL_NAME, + max_tokens=10, + temperature=0.0, + messages=[{"role": "user", "content": text_content}], + ) + embeds_resp = await client.chat.completions.create( + model=MODEL_NAME, + max_tokens=10, + temperature=0.0, + messages=[{"role": "user", "content": embeds_content}], + ) + + text_out = text_resp.choices[0].message.content + embeds_out = embeds_resp.choices[0].message.content + assert text_out is not None and len(text_out) > 0 + assert embeds_out is not None and len(embeds_out) > 0 + assert text_out == embeds_out + + +@pytest.fixture(scope="module") +def image_embeds_b64() -> dict[str, str]: + """Synthetic but stable `image_embeds` for Qwen2-VL.""" + grid = (1, 4, 4) + spatial_merge_size = 2 + num_patches = (grid[1] // spatial_merge_size) * (grid[2] // spatial_merge_size) + text_hidden_size = 1536 # Qwen2-VL-2B + torch.manual_seed(0) + return { + "image_embeds": tensor2base64( + torch.randn(num_patches, text_hidden_size, dtype=MODEL_DTYPE) + ), + "image_grid_thw": tensor2base64(torch.tensor(grid)), + } + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "image_first", + [True, False], + ids=["image_embeds-then-text", "text-then-image_embeds"], +) +async def test_text_content_and_prompt_embeds_match_with_image_embeds( + client: openai.AsyncOpenAI, + image_embeds_b64: dict[str, str], + aligned_content_and_embeds_b64: tuple[str, str], + image_first: bool, +): + """Same content as text vs `prompt_embeds` should yield identical Chat + Completions output when mixed with a precomputed `image_embeds` part in + the same message under greedy decoding. + """ + content, encoded_text_embeds = aligned_content_and_embeds_b64 + + image_part = {"type": "image_embeds", "image_embeds": image_embeds_b64} + text_part = {"type": "text", "text": content} + embeds_part = {"type": "prompt_embeds", "data": encoded_text_embeds} + + if image_first: + text_content = [image_part, text_part] + embeds_content = [image_part, embeds_part] + else: + text_content = [text_part, image_part] + embeds_content = [embeds_part, image_part] + + text_resp = await client.chat.completions.create( + model=MODEL_NAME, + max_tokens=10, + temperature=0.0, + messages=[{"role": "user", "content": text_content}], + ) + embeds_resp = await client.chat.completions.create( + model=MODEL_NAME, + max_tokens=10, + temperature=0.0, + messages=[{"role": "user", "content": embeds_content}], + ) + + text_out = text_resp.choices[0].message.content + embeds_out = embeds_resp.choices[0].message.content + assert text_out is not None and len(text_out) > 0 + assert embeds_out is not None and len(embeds_out) > 0 + assert text_out == embeds_out diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_default_mm_loras.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_default_mm_loras.py new file mode 100644 index 0000000000000000000000000000000000000000..b66802bcc46db679ca581649086e1a80cca54260 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_default_mm_loras.py @@ -0,0 +1,96 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import os + +import openai # use the official client for correctness check +import pytest +import pytest_asyncio +from huggingface_hub import snapshot_download + +from tests.conftest import AudioTestAssets +from tests.utils import RemoteOpenAIServer + +# NOTE - the tests in this module are currently analogous to test_chat, but are +# separated to avoid OOM killing due to module-scoped servers, since we +# need a multimodal model for these tests. + +# Contains a modality specific lora alongside the base model +MULTIMODAL_MODEL_NAME = snapshot_download("microsoft/Phi-4-multimodal-instruct") +AUDIO_LORA_PATH = os.path.join(MULTIMODAL_MODEL_NAME, "speech-lora") + +ACTIVE_MM_LORA_RESPONSE = "Spoken text: The first words I spoke in the original chronograph, a little piece of practical poetry. Mary had a little lamb, it slept with quite a snow, and everywhere that Mary went, the lamb was sure to go." # noqa: E501 + + +@pytest.fixture(scope="module") +def multimodal_server(): + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "half", + "--max-model-len", + "4096", + "--enforce-eager", + # lora config below + "--enable-lora", + "--lora-modules", + f"speech={AUDIO_LORA_PATH}", + "--max-lora-rank", + "320", + "--max-num-seqs", + "2", + "--trust-remote-code", + "--gpu-memory-utilization", + "0.8", + "--default-mm-loras", + f'{{"audio": "{AUDIO_LORA_PATH}"}}', + ] + + with RemoteOpenAIServer( + MULTIMODAL_MODEL_NAME, args, max_wait_seconds=480 + ) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def multi_modal_client(multimodal_server): + async with multimodal_server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + # base model with default lora should give the same response as lora model + "model_name", + [MULTIMODAL_MODEL_NAME, "speech"], +) +async def test_default_mm_lora_chat_completions( + model_name: str, + multi_modal_client: openai.AsyncOpenAI, + audio_assets: AudioTestAssets, +): + messages = [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "Can you transcribe this audio?", + }, + { + "type": "audio_url", + "audio_url": {"url": audio_assets[0].url}, + }, + ], + } + ] + + chat_completion = await multi_modal_client.chat.completions.create( + model=model_name, messages=messages, max_completion_tokens=128, temperature=0.0 + ) + + assert len(chat_completion.choices) > 0 + + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 0 + assert message.content == ACTIVE_MM_LORA_RESPONSE diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_video.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_video.py new file mode 100644 index 0000000000000000000000000000000000000000..ff6c69c68cc15a4462fe76fa2ac333f94e8e8a5c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_video.py @@ -0,0 +1,403 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import openai +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer +from vllm.multimodal.utils import encode_video_url, fetch_video +from vllm.platforms import current_platform + +MODEL_NAME = "llava-hf/llava-onevision-qwen2-0.5b-ov-hf" +MAXIMUM_VIDEOS = 3 + +TEST_VIDEO_URLS = [ + "https://www.bogotobogo.com/python/OpenCV_Python/images/mean_shift_tracking/slow_traffic_small.mp4", + "https://github.com/opencv/opencv/raw/refs/tags/4.12.0/samples/data/vtest.avi", + "https://github.com/opencv/opencv/raw/refs/tags/4.12.0/samples/data/Megamind.avi", +] + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--runner", + "generate", + "--max-model-len", + "32768", + "--max-num-seqs", + "2", + "--enforce-eager", + "--trust-remote-code", + "--limit-mm-per-prompt", + json.dumps({"video": MAXIMUM_VIDEOS}), + "--media-io-kwargs", + json.dumps({"video": {"num_frames": 32}}), + ] + + # ROCm: Increase timeouts to handle potential network delays and slower + # video processing when downloading multiple videos from external sources + env_overrides = {} + if current_platform.is_rocm(): + env_overrides = { + "VLLM_VIDEO_FETCH_TIMEOUT": "120", + "VLLM_ENGINE_ITERATION_TIMEOUT_S": "300", + } + + with RemoteOpenAIServer(MODEL_NAME, args, env_dict=env_overrides) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.fixture(scope="session") +def url_encoded_video() -> dict[str, str]: + return { + video_url: encode_video_url(fetch_video(video_url)[0]) + for video_url in TEST_VIDEO_URLS + } + + +def dummy_messages_from_video_url( + video_urls: str | list[str], + content_text: str = "What's in this video?", +): + if isinstance(video_urls, str): + video_urls = [video_urls] + + return [ + { + "role": "user", + "content": [ + *( + {"type": "video_url", "video_url": {"url": video_url}} + for video_url in video_urls + ), + {"type": "text", "text": content_text}, + ], + } + ] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("video_url", TEST_VIDEO_URLS) +async def test_single_chat_session_video( + client: openai.AsyncOpenAI, model_name: str, video_url: str +): + messages = dummy_messages_from_video_url(video_url) + + # test single completion + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + logprobs=True, + temperature=0.0, + top_logprobs=5, + ) + assert len(chat_completion.choices) == 1 + + choice = chat_completion.choices[0] + assert choice.finish_reason == "length" + assert chat_completion.usage == openai.types.CompletionUsage( + completion_tokens=10, prompt_tokens=6287, total_tokens=6297 + ) + + message = choice.message + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 10 + assert message.role == "assistant" + messages.append({"role": "assistant", "content": message.content}) + + # test multi-turn dialogue + messages.append({"role": "user", "content": "express your result in json"}) + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + ) + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("video_url", [TEST_VIDEO_URLS[0]]) +async def test_request_media_io_kwargs_override_uses_fewer_video_frames( + client: openai.AsyncOpenAI, model_name: str, video_url: str +): + messages = dummy_messages_from_video_url(video_url) + + default_resp = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=1, + temperature=0.0, + ) + override_resp = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=1, + temperature=0.0, + extra_body={ + "media_io_kwargs": { + "video": { + "num_frames": 4, + } + } + }, + ) + + assert default_resp.usage is not None + assert override_resp.usage is not None + assert override_resp.usage.prompt_tokens < default_resp.usage.prompt_tokens + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("video_url", [TEST_VIDEO_URLS[0]]) +async def test_invalid_num_frames_request_recoverable( + client: openai.AsyncOpenAI, model_name: str, video_url: str +): + messages = dummy_messages_from_video_url(video_url) + + with pytest.raises((openai.BadRequestError, openai.APIStatusError)): + await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=1, + temperature=0.0, + extra_body={ + "media_io_kwargs": { + "video": { + "num_frames": "invalid", + } + } + }, + ) + + # Server should still handle subsequent requests after the failed one. + recovery_resp = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=1, + temperature=0.0, + ) + recovery_msg = recovery_resp.choices[0].message + assert recovery_msg.content is not None and len(recovery_msg.content) >= 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("video_url", TEST_VIDEO_URLS) +async def test_error_on_invalid_video_url_type( + client: openai.AsyncOpenAI, model_name: str, video_url: str +): + messages = [ + { + "role": "user", + "content": [ + {"type": "video_url", "video_url": video_url}, + {"type": "text", "text": "What's in this video?"}, + ], + } + ] + + # video_url should be a dict {"url": "some url"}, not directly a string + with pytest.raises(openai.BadRequestError): + _ = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("video_url", TEST_VIDEO_URLS) +async def test_single_chat_session_video_beamsearch( + client: openai.AsyncOpenAI, model_name: str, video_url: str +): + messages = dummy_messages_from_video_url(video_url) + + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + n=2, + max_completion_tokens=10, + logprobs=True, + top_logprobs=5, + extra_body=dict(use_beam_search=True), + ) + assert len(chat_completion.choices) == 2 + assert ( + chat_completion.choices[0].message.content + != chat_completion.choices[1].message.content + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("video_url", TEST_VIDEO_URLS) +async def test_single_chat_session_video_base64encoded( + client: openai.AsyncOpenAI, + model_name: str, + video_url: str, + url_encoded_video: dict[str, str], +): + messages = dummy_messages_from_video_url(url_encoded_video[video_url]) + + # test single completion + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + logprobs=True, + temperature=0.0, + top_logprobs=5, + ) + assert len(chat_completion.choices) == 1 + + choice = chat_completion.choices[0] + assert choice.finish_reason == "length" + assert chat_completion.usage == openai.types.CompletionUsage( + completion_tokens=10, prompt_tokens=6287, total_tokens=6297 + ) + + message = choice.message + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 10 + assert message.role == "assistant" + messages.append({"role": "assistant", "content": message.content}) + + # test multi-turn dialogue + messages.append({"role": "user", "content": "express your result in json"}) + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + ) + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("video_url", TEST_VIDEO_URLS) +async def test_single_chat_session_video_base64encoded_beamsearch( + client: openai.AsyncOpenAI, + model_name: str, + video_url: str, + url_encoded_video: dict[str, str], +): + messages = dummy_messages_from_video_url(url_encoded_video[video_url]) + + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + n=2, + max_completion_tokens=10, + extra_body=dict(use_beam_search=True), + ) + assert len(chat_completion.choices) == 2 + assert ( + chat_completion.choices[0].message.content + != chat_completion.choices[1].message.content + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("video_url", TEST_VIDEO_URLS) +async def test_chat_streaming_video( + client: openai.AsyncOpenAI, model_name: str, video_url: str +): + messages = dummy_messages_from_video_url(video_url) + + # test single completion + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + ) + output = chat_completion.choices[0].message.content + stop_reason = chat_completion.choices[0].finish_reason + + # test streaming + stream = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + stream=True, + ) + chunks: list[str] = [] + finish_reason_count = 0 + async for chunk in stream: + delta = chunk.choices[0].delta + if delta.role: + assert delta.role == "assistant" + if delta.content: + chunks.append(delta.content) + if chunk.choices[0].finish_reason is not None: + finish_reason_count += 1 + # finish reason should only return in last block + assert finish_reason_count == 1 + assert chunk.choices[0].finish_reason == stop_reason + assert delta.content + assert "".join(chunks) == output + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize( + "video_urls", [TEST_VIDEO_URLS[:i] for i in range(2, len(TEST_VIDEO_URLS))] +) +@pytest.mark.flaky( + reruns=2, + reruns_delay=5, + condition=current_platform.is_rocm(), +) +async def test_multi_video_input( + client: openai.AsyncOpenAI, model_name: str, video_urls: list[str] +): + messages = dummy_messages_from_video_url(video_urls) + + if len(video_urls) > MAXIMUM_VIDEOS: + with pytest.raises(openai.BadRequestError): # test multi-video input + await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + ) + + # the server should still work afterwards + completion = await client.completions.create( + model=model_name, + prompt=[0, 0, 0, 0, 0], + max_tokens=5, + temperature=0.0, + ) + completion = completion.choices[0].text + assert completion is not None and len(completion) >= 0 + else: + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + ) + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 0 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_vision.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_vision.py new file mode 100644 index 0000000000000000000000000000000000000000..c32e19e3a2f006b4f5f2141dcd6f55370b648e93 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_vision.py @@ -0,0 +1,680 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import openai +import pytest +import pytest_asyncio +from transformers import AutoProcessor + +from tests.entrypoints.multimodal.conftest import TEST_IMAGE_ASSETS +from tests.utils import ROCM_ENV_OVERRIDES, ROCM_EXTRA_ARGS, RemoteOpenAIServer +from vllm.multimodal.media import MediaWithBytes +from vllm.multimodal.utils import encode_image_url, fetch_image +from vllm.platforms import current_platform + +MODEL_NAME = "microsoft/Phi-3.5-vision-instruct" +MAXIMUM_IMAGES = 2 + +# Required terms for beam search validation +# Each entry is a list of term groups - ALL groups must match +# Each group is a list of alternatives - at least ONE term in the group must appear +# This provides semantic validation while allowing wording variation +REQUIRED_BEAM_SEARCH_TERMS = [ + # Boardwalk image: must have "boardwalk" AND ("wooden" or "wood") + [["boardwalk"], ["wooden", "wood"]], + # Parrots image: must have ("parrot" or "bird") AND "two" + [["parrot", "bird"], ["two"]], + # Venn diagram: must have "venn" AND "diagram" + [["venn"], ["diagram"]], + # Gradient image: must have "gradient" AND ("color" or "spectrum") + [["gradient"], ["color", "spectrum"]], +] + + +def check_output_matches_terms(content: str, term_groups: list[list[str]]) -> bool: + """ + Check if content matches all required term groups. + Each term group requires at least one of its terms to be present. + All term groups must be satisfied. + """ + content_lower = content.lower() + return all( + any(term.lower() in content_lower for term in group) for group in term_groups + ) + + +def assert_non_empty_content(chat_completion, *, context: str = "") -> str: + """Assert the first choice has non-empty string content; return it. + + Provides a detailed failure message including the full ChatCompletion + response so flaky / model-quality issues are easy to diagnose. + """ + prefix = f"[{context}] " if context else "" + choice = chat_completion.choices[0] + content = choice.message.content + + assert content is not None, ( + f"{prefix}Expected non-None content but got None. " + f"finish_reason={choice.finish_reason!r}, " + f"full message={choice.message!r}, " + f"usage={chat_completion.usage!r}" + ) + assert isinstance(content, str), ( + f"{prefix}Expected str content, got {type(content).__name__}: {content!r}" + ) + assert len(content) > 0, ( + f"{prefix}Expected non-empty content but got empty string. " + f"finish_reason={choice.finish_reason!r}, " + f"full message={choice.message!r}, " + f"usage={chat_completion.usage!r}" + ) + return content + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--runner", + "generate", + "--max-model-len", + "2048", + "--max-num-seqs", + "5", + "--enforce-eager", + "--trust-remote-code", + "--limit-mm-per-prompt", + json.dumps({"image": MAXIMUM_IMAGES}), + *ROCM_EXTRA_ARGS, + ] + + # ROCm: Increase timeouts to handle potential network delays and slower + # video processing when downloading multiple videos from external sources + env_overrides = { + **ROCM_ENV_OVERRIDES, + **( + { + "VLLM_VIDEO_FETCH_TIMEOUT": "120", + "VLLM_ENGINE_ITERATION_TIMEOUT_S": "300", + } + if current_platform.is_rocm() + else {} + ), + } + + with RemoteOpenAIServer(MODEL_NAME, args, env_dict=env_overrides) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.fixture(scope="session") +def url_encoded_image(local_asset_server) -> dict[str, str]: + return { + image_asset: encode_image_url(local_asset_server.get_image_asset(image_asset)) + for image_asset in TEST_IMAGE_ASSETS + } + + +def dummy_messages_from_image_url( + image_urls: str | list[str], + content_text: str = "What's in this image?", +): + if isinstance(image_urls, str): + image_urls = [image_urls] + + return [ + { + "role": "user", + "content": [ + *( + {"type": "image_url", "image_url": {"url": image_url}} + for image_url in image_urls + ), + {"type": "text", "text": content_text}, + ], + } + ] + + +def describe_image_messages( + image_url: str, *, extra_image_fields: dict | None = None +) -> list[dict]: + """Build the system + user messages used by the completions-with-image + family of tests. *extra_image_fields* is merged into the top-level + image content block (for uuid / bad-key tests).""" + image_block: dict = { + "type": "image_url", + "image_url": {"url": image_url}, + } + if extra_image_fields: + image_block.update(extra_image_fields) + + return [ + {"role": "system", "content": "You are a helpful assistant."}, + { + "role": "user", + "content": [ + {"type": "text", "text": "Describe this image."}, + image_block, + ], + }, + ] + + +async def complete_and_check( + client: openai.AsyncOpenAI, + model_name: str, + messages: list[dict], + *, + context: str, + max_completion_tokens: int = 50, + temperature: float = 0.0, +) -> str: + """Run a chat completion and assert the output is non-empty. + Returns the content string.""" + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=max_completion_tokens, + temperature=temperature, + ) + return assert_non_empty_content(chat_completion, context=context) + + +def get_hf_prompt_tokens(model_name, content, image_url): + processor = AutoProcessor.from_pretrained( + model_name, trust_remote_code=True, num_crops=4 + ) + + placeholder = "<|image_1|>\n" + messages = [ + { + "role": "user", + "content": f"{placeholder}{content}", + } + ] + image = fetch_image(image_url) + # Unwrap MediaWithBytes if present + if isinstance(image, MediaWithBytes): + image = image.media + images = [image] + + prompt = processor.tokenizer.apply_chat_template( + messages, tokenize=False, add_generation_prompt=True + ) + inputs = processor(prompt, images, return_tensors="pt") + + return inputs.input_ids.shape[1] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("image_url", TEST_IMAGE_ASSETS, indirect=True) +async def test_single_chat_session_image( + client: openai.AsyncOpenAI, model_name: str, image_url: str +): + content_text = "What's in this image?" + messages = dummy_messages_from_image_url(image_url, content_text) + + max_completion_tokens = 10 + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=max_completion_tokens, + logprobs=True, + temperature=0.0, + top_logprobs=5, + ) + assert len(chat_completion.choices) == 1, ( + f"Expected 1 choice, got {len(chat_completion.choices)}" + ) + + choice = chat_completion.choices[0] + assert choice.finish_reason == "length", ( + f"Expected finish_reason='length' (capped at {max_completion_tokens} " + f"tokens), got {choice.finish_reason!r}. " + f"content={choice.message.content!r}" + ) + + hf_prompt_tokens = get_hf_prompt_tokens(model_name, content_text, image_url) + expected_usage = openai.types.CompletionUsage( + completion_tokens=max_completion_tokens, + prompt_tokens=hf_prompt_tokens, + total_tokens=hf_prompt_tokens + max_completion_tokens, + ) + assert chat_completion.usage == expected_usage, ( + f"Usage mismatch: got {chat_completion.usage!r}, expected {expected_usage!r}" + ) + + message = choice.message + assert message.content is not None and len(message.content) >= 10, ( + f"Expected content with >=10 chars, got {message.content!r}" + ) + assert message.role == "assistant", ( + f"Expected role='assistant', got {message.role!r}" + ) + + messages.append({"role": "assistant", "content": message.content}) + + # test multi-turn dialogue + messages.append({"role": "user", "content": "express your result in json"}) + await complete_and_check( + client, + model_name, + messages, + context=f"multi-turn follow-up for {image_url}", + max_completion_tokens=10, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("image_url", TEST_IMAGE_ASSETS, indirect=True) +async def test_error_on_invalid_image_url_type( + client: openai.AsyncOpenAI, model_name: str, image_url: str +): + content_text = "What's in this image?" + messages = [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": image_url}, + {"type": "text", "text": content_text}, + ], + } + ] + + # image_url should be a dict {"url": "some url"}, not directly a string + with pytest.raises(openai.BadRequestError): + await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("image_url", TEST_IMAGE_ASSETS, indirect=True) +async def test_single_chat_session_image_beamsearch( + client: openai.AsyncOpenAI, model_name: str, image_url: str +): + content_text = "What's in this image?" + messages = dummy_messages_from_image_url(image_url, content_text) + + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + n=2, + max_completion_tokens=10, + logprobs=True, + top_logprobs=5, + extra_body=dict(use_beam_search=True), + ) + assert len(chat_completion.choices) == 2, ( + f"Expected 2 beam search choices, got {len(chat_completion.choices)}" + ) + + content_0 = chat_completion.choices[0].message.content + content_1 = chat_completion.choices[1].message.content + assert content_0 != content_1, ( + f"Beam search should produce different outputs for {image_url}, " + f"but both returned: {content_0!r}" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("raw_image_url", TEST_IMAGE_ASSETS) +@pytest.mark.parametrize("image_url", TEST_IMAGE_ASSETS, indirect=True) +async def test_single_chat_session_image_base64encoded( + client: openai.AsyncOpenAI, + model_name: str, + raw_image_url: str, + image_url: str, + url_encoded_image: dict[str, str], +): + content_text = "What's in this image?" + messages = dummy_messages_from_image_url( + url_encoded_image[raw_image_url], + content_text, + ) + + max_completion_tokens = 10 + # test single completion + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=max_completion_tokens, + logprobs=True, + temperature=0.0, + top_logprobs=5, + ) + assert len(chat_completion.choices) == 1, ( + f"Expected 1 choice, got {len(chat_completion.choices)}" + ) + + choice = chat_completion.choices[0] + assert choice.finish_reason == "length", ( + f"Expected finish_reason='length', got {choice.finish_reason!r}. " + f"content={choice.message.content!r}" + ) + + hf_prompt_tokens = get_hf_prompt_tokens(model_name, content_text, image_url) + expected_usage = openai.types.CompletionUsage( + completion_tokens=max_completion_tokens, + prompt_tokens=hf_prompt_tokens, + total_tokens=hf_prompt_tokens + max_completion_tokens, + ) + assert chat_completion.usage == expected_usage, ( + f"Usage mismatch: got {chat_completion.usage!r}, expected {expected_usage!r}" + ) + + message = choice.message + assert message.content is not None and len(message.content) >= 10, ( + f"Expected content with >=10 chars, got {message.content!r}" + ) + assert message.role == "assistant", ( + f"Expected role='assistant', got {message.role!r}" + ) + + messages.append({"role": "assistant", "content": message.content}) + + # test multi-turn dialogue + messages.append({"role": "user", "content": "express your result in json"}) + await complete_and_check( + client, + model_name, + messages, + context=f"multi-turn base64 follow-up for {raw_image_url}", + max_completion_tokens=10, + temperature=0.0, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("image_idx", list(range(len(TEST_IMAGE_ASSETS)))) +async def test_single_chat_session_image_base64encoded_beamsearch( + client: openai.AsyncOpenAI, + model_name: str, + image_idx: int, + url_encoded_image: dict[str, str], +): + # NOTE: This test validates that we pass MM data through beam search + raw_image_url = TEST_IMAGE_ASSETS[image_idx] + required_terms = REQUIRED_BEAM_SEARCH_TERMS[image_idx] + + messages = dummy_messages_from_image_url(url_encoded_image[raw_image_url]) + + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + n=2, + max_completion_tokens=10, + temperature=0.0, + extra_body=dict(use_beam_search=True), + ) + assert len(chat_completion.choices) == 2, ( + f"Expected 2 beam search choices for image {image_idx} " + f"({raw_image_url}), got {len(chat_completion.choices)}" + ) + + # Verify beam search produces two different non-empty outputs + content_0 = chat_completion.choices[0].message.content + content_1 = chat_completion.choices[1].message.content + + # Emit beam search outputs for debugging + print( + f"Beam search outputs for image {image_idx} ({raw_image_url}): " + f"Output 0: {content_0!r}, Output 1: {content_1!r}" + ) + + assert content_0, ( + f"First beam output is empty for image {image_idx} ({raw_image_url}). " + f"finish_reason={chat_completion.choices[0].finish_reason!r}" + ) + assert content_1, ( + f"Second beam output is empty for image {image_idx} " + f"({raw_image_url}). " + f"finish_reason={chat_completion.choices[1].finish_reason!r}" + ) + assert content_0 != content_1, ( + f"Beam search produced identical outputs for image {image_idx} " + f"({raw_image_url}): {content_0!r}" + ) + + # Verify each output contains the required terms for this image + for i, content in enumerate([content_0, content_1]): + assert check_output_matches_terms(content, required_terms), ( + f"Beam output {i} for image {image_idx} ({raw_image_url}) " + f"doesn't match required terms.\n" + f" content: {content!r}\n" + f" required (all groups, >=1 per group): {required_terms}" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("image_url", TEST_IMAGE_ASSETS, indirect=True) +async def test_chat_streaming_image( + client: openai.AsyncOpenAI, model_name: str, image_url: str +): + messages = dummy_messages_from_image_url(image_url) + + # test single completion + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + ) + output = chat_completion.choices[0].message.content + stop_reason = chat_completion.choices[0].finish_reason + + # test streaming + stream = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + stream=True, + ) + chunks: list[str] = [] + finish_reason_count = 0 + async for chunk in stream: + delta = chunk.choices[0].delta + if delta.role: + assert delta.role == "assistant", ( + f"Expected role='assistant' in stream delta, got {delta.role!r}" + ) + if delta.content: + chunks.append(delta.content) + if chunk.choices[0].finish_reason is not None: + finish_reason_count += 1 + # finish reason should only return in last block + assert finish_reason_count == 1, ( + f"Expected exactly 1 finish_reason across stream chunks, " + f"got {finish_reason_count}" + ) + assert chunk.choices[0].finish_reason == stop_reason, ( + f"Stream finish_reason={chunk.choices[0].finish_reason!r} " + f"doesn't match non-stream finish_reason={stop_reason!r}" + ) + + streamed_text = "".join(chunks) + assert streamed_text == output, ( + f"Streamed output doesn't match non-streamed for {image_url}.\n" + f" streamed: {streamed_text!r}\n" + f" non-streamed: {output!r}" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize( + "image_urls", + [TEST_IMAGE_ASSETS[:i] for i in range(2, len(TEST_IMAGE_ASSETS))], + indirect=True, +) +async def test_multi_image_input( + client: openai.AsyncOpenAI, model_name: str, image_urls: list[str] +): + messages = dummy_messages_from_image_url(image_urls) + + if len(image_urls) > MAXIMUM_IMAGES: + with pytest.raises(openai.BadRequestError): # test multi-image input + await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + ) + + # the server should still work afterwards + completion = await client.completions.create( + model=model_name, + prompt=[0, 0, 0, 0, 0], + max_tokens=5, + temperature=0.0, + ) + assert completion.choices[0].text is not None, ( + "Server failed to produce output after rejecting over-limit " + "multi-image request" + ) + else: + await complete_and_check( + client, + model_name, + messages, + context=f"multi-image input ({len(image_urls)} images)", + max_completion_tokens=10, + temperature=0.0, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize( + "image_urls", + [TEST_IMAGE_ASSETS[:i] for i in range(2, len(TEST_IMAGE_ASSETS))], + indirect=True, +) +async def test_completions_with_image( + client: openai.AsyncOpenAI, + model_name: str, + image_urls: list[str], +): + for image_url in image_urls: + messages = describe_image_messages(image_url) + await complete_and_check( + client, + model_name, + messages, + context=f"completions_with_image url={image_url}", + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize( + "image_urls", + [TEST_IMAGE_ASSETS[:i] for i in range(2, len(TEST_IMAGE_ASSETS))], + indirect=True, +) +async def test_completions_with_image_with_uuid( + client: openai.AsyncOpenAI, + model_name: str, + image_urls: list[str], +): + for image_url in image_urls: + messages = describe_image_messages( + image_url, + extra_image_fields={"uuid": image_url}, + ) + await complete_and_check( + client, + model_name, + messages, + context=f"uuid first request url={image_url}", + ) + + cached_messages: list[dict] = [ + {"role": "system", "content": "You are a helpful assistant."}, + { + "role": "user", + "content": [ + {"type": "text", "text": "Describe this image."}, + {"type": "image_url", "image_url": {}, "uuid": image_url}, + ], + }, + ] + await complete_and_check( + client, + model_name, + cached_messages, + context=f"uuid cached (empty image) uuid={image_url}", + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_completions_with_empty_image_with_uuid_without_cache_hit( + client: openai.AsyncOpenAI, + model_name: str, +): + with pytest.raises(openai.BadRequestError): + await client.chat.completions.create( + messages=[ + {"role": "system", "content": "You are a helpful assistant."}, + { + "role": "user", + "content": [ + {"type": "text", "text": "Describe this image."}, + { + "type": "image_url", + "image_url": {}, + "uuid": "uuid_not_previously_seen", + }, + ], + }, + ], + model=model_name, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize( + "image_urls", + [TEST_IMAGE_ASSETS[:i] for i in range(2, len(TEST_IMAGE_ASSETS))], + indirect=True, +) +async def test_completions_with_image_with_incorrect_uuid_format( + client: openai.AsyncOpenAI, + model_name: str, + image_urls: list[str], +): + for image_url in image_urls: + messages = describe_image_messages( + image_url, + extra_image_fields={ + "also_incorrect_uuid_key": image_url, + }, + ) + # Inject the bad key inside image_url dict too + messages[1]["content"][1]["image_url"]["incorrect_uuid_key"] = image_url + + await complete_and_check( + client, + model_name, + messages, + context=f"incorrect uuid format url={image_url}", + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_vision_embeds.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_vision_embeds.py new file mode 100644 index 0000000000000000000000000000000000000000..96d6f6398962caa6e6bc975b9817abe7db810c01 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/chat_completion/test_vision_embeds.py @@ -0,0 +1,160 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import importlib.util + +import numpy as np +import pybase64 as base64 +import pytest +import requests +import torch + +from tests.utils import RemoteOpenAIServer +from vllm.utils.serial_utils import tensor2base64 + +# Prithvi requires terratorch, which is temporarily unavailable while PyPI has +# `lightning` quarantined (#41376). Skip just the Prithvi case; leave the +# Qwen3-VL case in the same file untouched. +_TERRATORCH_AVAILABLE = importlib.util.find_spec("terratorch") is not None + + +@pytest.mark.skipif( + not _TERRATORCH_AVAILABLE, + reason="terratorch unavailable while PyPI has `lightning` quarantined; see #41376", +) +@pytest.mark.parametrize( + "model_name", ["ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11"] +) +def test_single_content(model_name: str): + args = [ + "--runner", + "pooling", + # use half precision for speed and memory savings in CI environment + "--dtype", + "float16", + "--enforce-eager", + "--trust-remote-code", + "--max-num-seqs", + "32", + "--model-impl", + "terratorch", + "--skip-tokenizer-init", + "--enable-mm-embeds", + ] + + with RemoteOpenAIServer(model_name, args) as server: + response = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "messages": [ + { + "role": "user", + "content": [ + { + "type": "image_embeds", + "image_embeds": { + "pixel_values": tensor2base64( + torch.ones((6, 512, 512), dtype=torch.float16) + ), + "location_coords": tensor2base64( + torch.ones((1, 2), dtype=torch.float16) + ), + }, + }, + ], + } + ], + "encoding_format": "base64", + }, + ) + response.raise_for_status() + + output = response.json()["data"][0]["data"] + + np_response = np.frombuffer(base64.b64decode(output), dtype=np.float32) + assert len(np_response) == 524288 + + +@pytest.mark.parametrize("model_name", ["Qwen/Qwen3-VL-2B-Instruct"]) +def test_multi_content(model_name: str): + args = [ + "--enforce-eager", + "--max-num-seqs", + "32", + "--max-model-len", + "8192", + "--enable-mm-embeds", + ] + + with RemoteOpenAIServer(model_name, args) as server: + client = server.get_client() + + # Image only + chat_completion = client.chat.completions.create( + model=model_name, + messages=[ + { + "role": "user", + "content": [ + { + "type": "image_embeds", + "image_embeds": { + "image_embeds": tensor2base64(torch.zeros(220, 8192)), + "image_grid_thw": tensor2base64( + torch.tensor([1, 22, 40]) + ), + }, + }, + { + "type": "image_embeds", + "image_embeds": { + "image_embeds": tensor2base64(torch.zeros(220, 8192)), + "image_grid_thw": tensor2base64( + torch.tensor([1, 22, 40]) + ), + }, + }, + ], + } + ], + max_tokens=5, + ) + + assert chat_completion.id is not None + assert len(chat_completion.choices) == 1 + + # Interleaved text and image + chat_completion = client.chat.completions.create( + model=model_name, + messages=[ + { + "role": "user", + "content": [ + { + "type": "image_embeds", + "image_embeds": { + "image_embeds": tensor2base64(torch.zeros(220, 8192)), + "image_grid_thw": tensor2base64( + torch.tensor([1, 22, 40]) + ), + }, + }, + {"type": "text", "text": "OCR:"}, + { + "type": "image_embeds", + "image_embeds": { + "image_embeds": tensor2base64(torch.zeros(220, 8192)), + "image_grid_thw": tensor2base64( + torch.tensor([1, 22, 40]) + ), + }, + }, + ], + } + ], + max_tokens=5, + ) + + assert chat_completion.id is not None + assert len(chat_completion.choices) == 1 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/responses/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/responses/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/responses/test_image.py b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/responses/test_image.py new file mode 100644 index 0000000000000000000000000000000000000000..7bf0324af4cee57d4a10324a18215a585f67fec7 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/multimodal/openai/responses/test_image.py @@ -0,0 +1,165 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import openai +import pytest +import pytest_asyncio + +from tests.entrypoints.multimodal.conftest import TEST_IMAGE_ASSETS +from tests.utils import RemoteOpenAIServer +from vllm.multimodal.utils import encode_image_url + +# Use a small vision model for testing +MODEL_NAME = "Qwen/Qwen2.5-VL-3B-Instruct" +MAXIMUM_IMAGES = 2 + + +@pytest.fixture(scope="module") +def default_image_server_args(): + return [ + "--enforce-eager", + "--max-model-len", + "6000", + "--max-num-seqs", + "128", + "--limit-mm-per-prompt", + json.dumps({"image": MAXIMUM_IMAGES}), + ] + + +@pytest.fixture(scope="module") +def image_server(default_image_server_args): + with RemoteOpenAIServer( + MODEL_NAME, + default_image_server_args, + env_dict={"VLLM_ENABLE_RESPONSES_API_STORE": "1"}, + ) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(image_server): + async with image_server.get_async_client() as async_client: + yield async_client + + +@pytest.fixture(scope="session") +def url_encoded_image(local_asset_server) -> dict[str, str]: + return { + image_url: encode_image_url(local_asset_server.get_image_asset(image_url)) + for image_url in TEST_IMAGE_ASSETS + } + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("image_url", TEST_IMAGE_ASSETS, indirect=True) +async def test_single_chat_session_image( + client: openai.AsyncOpenAI, model_name: str, image_url: str +): + content_text = "What's in this image?" + messages = [ + { + "role": "user", + "content": [ + { + "type": "input_image", + "image_url": image_url, + "detail": "auto", + }, + {"type": "input_text", "text": content_text}, + ], + } + ] + + # test image url + response = await client.responses.create( + model=model_name, + input=messages, + ) + assert len(response.output_text) > 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("raw_image_url", TEST_IMAGE_ASSETS) +async def test_single_chat_session_image_base64encoded( + client: openai.AsyncOpenAI, + model_name: str, + raw_image_url: str, + url_encoded_image: dict[str, str], +): + content_text = "What's in this image?" + messages = [ + { + "role": "user", + "content": [ + { + "type": "input_image", + "image_url": url_encoded_image[raw_image_url], + "detail": "auto", + }, + {"type": "input_text", "text": content_text}, + ], + } + ] + # test image base64 + response = await client.responses.create( + model=model_name, + input=messages, + ) + assert len(response.output_text) > 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize( + "image_urls", + [TEST_IMAGE_ASSETS[:i] for i in range(2, len(TEST_IMAGE_ASSETS))], + indirect=True, +) +async def test_multi_image_input( + client: openai.AsyncOpenAI, model_name: str, image_urls: list[str] +): + messages = [ + { + "role": "user", + "content": [ + *( + { + "type": "input_image", + "image_url": image_url, + "detail": "auto", + } + for image_url in image_urls + ), + {"type": "input_text", "text": "What's in this image?"}, + ], + } + ] + + if len(image_urls) > MAXIMUM_IMAGES: + with pytest.raises(openai.BadRequestError): # test multi-image input + await client.responses.create( + model=model_name, + input=messages, + ) + # the server should still work afterwards + response = await client.responses.create( + model=model_name, + input=[ + { + "role": "user", + "content": "What's the weather like in Paris today?", + } + ], + ) + assert len(response.output_text) > 0 + else: + response = await client.responses.create( + model=model_name, + input=messages, + ) + assert len(response.output_text) > 0 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_batched_chat_completions.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_batched_chat_completions.py new file mode 100644 index 0000000000000000000000000000000000000000..e8fec5b9fb5cce2e911af231b902348fd3e05b7c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_batched_chat_completions.py @@ -0,0 +1,189 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import httpx +import pytest + +from tests.utils import RemoteOpenAIServer + +# any model with a chat template defined in tokenizer_config should work here +MODEL_NAME = "Qwen/Qwen2.5-1.5B-Instruct" + + +@pytest.fixture(scope="module") +def default_server_args(): + return [ + # use half precision for speed and memory savings in CI environment + "--max-model-len", + "2048", + "--max-num-seqs", + "128", + "--enforce-eager", + ] + + +@pytest.fixture(scope="module") +def server(default_server_args): + with RemoteOpenAIServer(MODEL_NAME, default_server_args) as remote_server: + yield remote_server + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_batched_chat_completions( + server: RemoteOpenAIServer, model_name: str +) -> None: + conversations = [ + [{"role": "user", "content": "Reply with exactly the word: alpha"}], + [{"role": "user", "content": "Reply with exactly the word: beta"}], + ] + + async with httpx.AsyncClient() as http_client: + response = await http_client.post( + f"{server.url_for('v1/chat/completions/batch')}", + json={ + "model": model_name, + "messages": conversations, + }, + timeout=60, + ) + + assert response.status_code == 200, response.text + data = response.json() + + choices = data["choices"] + assert len(choices) == 2 + + indices = {choice["index"] for choice in choices} + assert indices == {0, 1} + + # Each conversation should produce a non-empty text response. + for choice in choices: + assert choice["message"]["content"] + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_batched_chat_completions_with_json_schema( + server: RemoteOpenAIServer, model_name: str +) -> None: + schema = { + "type": "object", + "properties": { + "answer": {"type": "string", "enum": ["yes", "no"]}, + }, + "required": ["answer"], + } + conversations = [ + [{"role": "user", "content": "Is the sky blue? Answer in JSON."}], + [{"role": "user", "content": "Is fire cold? Answer in JSON."}], + ] + + async with httpx.AsyncClient() as http_client: + response = await http_client.post( + f"{server.url_for('v1/chat/completions/batch')}", + json={ + "model": model_name, + "messages": conversations, + "response_format": { + "type": "json_schema", + "json_schema": {"name": "answer", "schema": schema, "strict": True}, + }, + }, + timeout=60, + ) + + assert response.status_code == 200, response.text + data = response.json() + + choices = data["choices"] + assert len(choices) == 2 + + for choice in choices: + parsed = json.loads(choice["message"]["content"]) + assert "answer" in parsed + assert parsed["answer"] in ("yes", "no") + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_batched_chat_completions_logprobs_not_token_id_placeholders( + server: RemoteOpenAIServer, model_name: str +) -> None: + # Regression test: requesting `return_token_ids` alongside logprobs must not + # corrupt the logprob `token` fields into "token_id:{id}" placeholders. That + # placeholder rendering is controlled by `return_tokens_as_token_ids`, which + # this request leaves unset. + conversations = [ + [{"role": "user", "content": "Reply with exactly the word: alpha"}], + ] + + async with httpx.AsyncClient() as http_client: + response = await http_client.post( + f"{server.url_for('v1/chat/completions/batch')}", + json={ + "model": model_name, + "messages": conversations, + "logprobs": True, + "top_logprobs": 1, + "return_token_ids": True, + }, + timeout=60, + ) + + assert response.status_code == 200, response.text + data = response.json() + + content = data["choices"][0]["logprobs"]["content"] + assert content + for entry in content: + assert not entry["token"].startswith("token_id:") + for top in entry["top_logprobs"]: + assert not top["token"].startswith("token_id:") + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_batched_chat_completions_return_tokens_as_token_ids( + server: RemoteOpenAIServer, model_name: str +) -> None: + # Complementary check: when `return_tokens_as_token_ids` is explicitly set, + # the logprob tokens *should* be rendered as "token_id:{id}" placeholders, + # proving the new field is actually wired through. + conversations = [ + [{"role": "user", "content": "Reply with exactly the word: alpha"}], + ] + + async with httpx.AsyncClient() as http_client: + response = await http_client.post( + f"{server.url_for('v1/chat/completions/batch')}", + json={ + "model": model_name, + "messages": conversations, + "logprobs": True, + "top_logprobs": 1, + "return_tokens_as_token_ids": True, + }, + timeout=60, + ) + + assert response.status_code == 200, response.text + data = response.json() + + content = data["choices"][0]["logprobs"]["content"] + assert content + assert all(entry["token"].startswith("token_id:") for entry in content) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat.py new file mode 100644 index 0000000000000000000000000000000000000000..ac0a0ab1016458f6909613dd3dd81ac2af4a33e7 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat.py @@ -0,0 +1,1167 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +# imports for structured outputs tests +import json +from collections import defaultdict + +import jsonschema +import openai # use the official client for correctness check +import pytest +import pytest_asyncio +import regex as re +import requests +import torch +from openai import BadRequestError + +from tests.utils import RemoteOpenAIServer +from vllm.entrypoints.openai.chat_completion.protocol import ( + ChatCompletionRequest, +) +from vllm.sampling_params import SamplingParams + +# any model with a chat template should work here +MODEL_NAME = "HuggingFaceH4/zephyr-7b-beta" + + +@pytest.fixture(scope="module") +def zephyr_lora_files(): + """Download zephyr LoRA files once per test session.""" + from huggingface_hub import snapshot_download + + return snapshot_download(repo_id="typeof/zephyr-7b-beta-lora") + + +@pytest.fixture(scope="module") +def server(zephyr_lora_files): + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-model-len", + "8192", + "--enforce-eager", + # lora config below + "--enable-lora", + "--lora-modules", + f"zephyr-lora={zephyr_lora_files}", + "--max-lora-rank", + "64", + "--max-cpu-loras", + "2", + "--max-num-seqs", + "128", + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + # first test base model, then test loras + "model_name", + [MODEL_NAME, "zephyr-lora"], +) +async def test_no_logprobs_chat(client: openai.AsyncOpenAI, model_name: str): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + {"role": "user", "content": "what is 1+1?"}, + ] + + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=5, + temperature=0.0, + logprobs=False, + ) + + choice = chat_completion.choices[0] + assert choice.logprobs is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + # just test 1 lora hereafter + "model_name", + [MODEL_NAME, "zephyr-lora"], +) +async def test_zero_logprobs_chat(client: openai.AsyncOpenAI, model_name: str): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + {"role": "user", "content": "what is 1+1?"}, + ] + + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=5, + temperature=0.0, + logprobs=True, + top_logprobs=0, + ) + + choice = chat_completion.choices[0] + assert choice.logprobs is not None + assert choice.logprobs.content is not None + assert len(choice.logprobs.content[0].top_logprobs) == 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME, "zephyr-lora"], +) +async def test_some_logprobs_chat(client: openai.AsyncOpenAI, model_name: str): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + {"role": "user", "content": "what is 1+1?"}, + ] + + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=5, + temperature=0.0, + logprobs=True, + top_logprobs=5, + ) + + choice = chat_completion.choices[0] + assert choice.logprobs is not None + assert choice.logprobs.content is not None + assert len(choice.logprobs.content[0].top_logprobs) == 5 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME, "zephyr-lora"], +) +async def test_too_many_chat_logprobs(client: openai.AsyncOpenAI, model_name: str): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + {"role": "user", "content": "what is 1+1?"}, + ] + + # Default max_logprobs is 20, so this should raise an error + with pytest.raises((openai.BadRequestError, openai.APIError)): + stream = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + logprobs=True, + top_logprobs=21, + stream=True, + ) + async for chunk in stream: + ... + + with pytest.raises(openai.BadRequestError): + await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + logprobs=True, + top_logprobs=30, + stream=False, + ) + + # the server should still work afterwards + chat_completion = await client.chat.completions.create( + model=model_name, messages=messages, max_completion_tokens=10, stream=False + ) + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name, prompt_logprobs", + [(MODEL_NAME, 1), (MODEL_NAME, 0), (MODEL_NAME, -1), (MODEL_NAME, None)], +) +async def test_prompt_logprobs_chat( + client: openai.AsyncOpenAI, model_name: str, prompt_logprobs: int | None +): + params: dict = { + "messages": [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Who won the world series in 2020?"}, + { + "role": "assistant", + "content": "The Los Angeles Dodgers won the World Series in 2020.", + }, + {"role": "user", "content": "Where was it played?"}, + ], + "model": model_name, + } + + if prompt_logprobs is not None: + params["extra_body"] = {"prompt_logprobs": prompt_logprobs} + + if prompt_logprobs is not None and prompt_logprobs < 0: + with pytest.raises(BadRequestError): + await client.chat.completions.create(**params) + else: + completion = await client.chat.completions.create(**params) + if prompt_logprobs is not None: + assert completion.prompt_logprobs is not None + assert len(completion.prompt_logprobs) > 0 + else: + assert completion.prompt_logprobs is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_more_than_one_prompt_logprobs_chat( + client: openai.AsyncOpenAI, model_name: str +): + params: dict = { + "messages": [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Who won the world series in 2020?"}, + { + "role": "assistant", + "content": "The Los Angeles Dodgers won the World Series in 2020.", + }, + {"role": "user", "content": "Where was it played?"}, + ], + "model": model_name, + "extra_body": {"prompt_logprobs": 1}, + } + + completion_1 = await client.chat.completions.create(**params) + + params["extra_body"] = {"prompt_logprobs": 2} + completion_2 = await client.chat.completions.create(**params) + + assert len(completion_1.prompt_logprobs[3]) == 1 + assert len(completion_2.prompt_logprobs[3]) == 2 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME, "zephyr-lora"], +) +async def test_single_chat_session(client: openai.AsyncOpenAI, model_name: str): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + {"role": "user", "content": "what is 1+1?"}, + ] + # test single completion + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=5, + logprobs=True, + top_logprobs=5, + ) + assert chat_completion.id is not None + assert len(chat_completion.choices) == 1 + + choice = chat_completion.choices[0] + + assert choice.finish_reason == "length" + assert chat_completion.usage == openai.types.CompletionUsage( + completion_tokens=5, prompt_tokens=37, total_tokens=42 + ) + + message = choice.message + assert message.content is not None and len(message.content) >= 5 + assert message.role == "assistant" + messages.append({"role": "assistant", "content": message.content}) + + # test multi-turn dialogue + messages.append({"role": "user", "content": "express your result in json"}) + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=5, + ) + message = chat_completion.choices[0].message + assert message.content is not None and len(message.content) >= 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + # just test 1 lora hereafter + "model_name", + [MODEL_NAME, "zephyr-lora"], +) +async def test_chat_streaming(client: openai.AsyncOpenAI, model_name: str): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + {"role": "user", "content": "what is 1+1?"}, + ] + + # test single completion + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + ) + output = chat_completion.choices[0].message.content + stop_reason = chat_completion.choices[0].finish_reason + + # test streaming + stream = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + stream=True, + ) + chunks: list[str] = [] + finish_reason_count = 0 + async for chunk in stream: + delta = chunk.choices[0].delta + if delta.role: + assert delta.role == "assistant" + if delta.content: + chunks.append(delta.content) + if chunk.choices[0].finish_reason is not None: + finish_reason_count += 1 + # finish reason should only return in last block + assert finish_reason_count == 1 + assert chunk.choices[0].finish_reason == stop_reason + assert delta.content + assert "".join(chunks) == output + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + ["HuggingFaceH4/zephyr-7b-beta", "zephyr-lora"], +) +async def test_chat_completion_stream_options( + client: openai.AsyncOpenAI, model_name: str +): + messages = [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "What is the capital of France?"}, + ] + + # Test stream=True, stream_options={"include_usage": False} + stream = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + stream=True, + stream_options={"include_usage": False}, + ) + async for chunk in stream: + assert chunk.usage is None + + # Test stream=True, stream_options={"include_usage": True, + # "continuous_usage_stats": False}} + stream = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + stream=True, + stream_options={"include_usage": True, "continuous_usage_stats": False}, + ) + + async for chunk in stream: + if chunk.choices[0].finish_reason is None: + assert chunk.usage is None + else: + assert chunk.usage is None + final_chunk = await anext(stream) + assert final_chunk.usage is not None + assert final_chunk.usage.prompt_tokens > 0 + assert final_chunk.usage.completion_tokens > 0 + assert final_chunk.usage.total_tokens == ( + final_chunk.usage.prompt_tokens + final_chunk.usage.completion_tokens + ) + assert final_chunk.choices == [] + + # Test stream=False, stream_options={"include_usage": None} + with pytest.raises(BadRequestError): + await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + stream=False, + stream_options={"include_usage": None}, + ) + + # Test stream=False, stream_options={"include_usage": True} + with pytest.raises(BadRequestError): + await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.0, + stream=False, + stream_options={"include_usage": True}, + ) + + # Test stream=True, stream_options={"include_usage": True, + # "continuous_usage_stats": True} + stream = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + extra_body=dict(min_tokens=10), + temperature=0.0, + stream=True, + stream_options={ + "include_usage": True, + "continuous_usage_stats": True, + }, + ) + last_completion_tokens = 0 + async for chunk in stream: + assert chunk.usage.prompt_tokens >= 0 + assert ( + last_completion_tokens == 0 + or chunk.usage.completion_tokens > last_completion_tokens + or ( + not chunk.choices + and chunk.usage.completion_tokens == last_completion_tokens + ) + ) + assert chunk.usage.total_tokens == ( + chunk.usage.prompt_tokens + chunk.usage.completion_tokens + ) + last_completion_tokens = chunk.usage.completion_tokens + + assert last_completion_tokens == 10 + + +@pytest.mark.asyncio +async def test_structured_outputs_choice_chat( + client: openai.AsyncOpenAI, + sample_structured_outputs_choices, +): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + { + "role": "user", + "content": "The best language for type-safe systems programming is ", + }, + ] + chat_completion = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_completion_tokens=10, + temperature=0.7, + extra_body=dict( + structured_outputs={"choice": sample_structured_outputs_choices} + ), + ) + choice1 = chat_completion.choices[0].message.content + assert choice1 in sample_structured_outputs_choices + + messages.append({"role": "assistant", "content": choice1}) + messages.append({"role": "user", "content": "I disagree, pick another one"}) + chat_completion = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_completion_tokens=10, + temperature=0.7, + extra_body=dict( + structured_outputs={"choice": sample_structured_outputs_choices} + ), + ) + choice2 = chat_completion.choices[0].message.content + assert choice2 in sample_structured_outputs_choices + assert choice1 != choice2 + + +@pytest.mark.asyncio +async def test_structured_outputs_json_chat( + client: openai.AsyncOpenAI, + sample_json_schema, +): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + { + "role": "user", + "content": f"Give an example JSON for an employee profile that " + f"fits this schema: {sample_json_schema}", + }, + ] + chat_completion = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_completion_tokens=1000, + extra_body=dict(structured_outputs={"json": sample_json_schema}), + ) + message = chat_completion.choices[0].message + assert message.content is not None + json1 = json.loads(message.content) + jsonschema.validate(instance=json1, schema=sample_json_schema) + + messages.append({"role": "assistant", "content": message.content}) + messages.append( + {"role": "user", "content": "Give me another one with a different name and age"} + ) + chat_completion = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_completion_tokens=1000, + extra_body=dict(structured_outputs={"json": sample_json_schema}), + ) + message = chat_completion.choices[0].message + assert message.content is not None + json2 = json.loads(message.content) + jsonschema.validate(instance=json2, schema=sample_json_schema) + assert json1["name"] != json2["name"] + assert json1["age"] != json2["age"] + + +@pytest.mark.asyncio +async def test_structured_outputs_regex_chat( + client: openai.AsyncOpenAI, + sample_regex, +): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + { + "role": "user", + "content": f"Give an example IP address with this regex: {sample_regex}", + }, + ] + chat_completion = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_completion_tokens=20, + extra_body=dict(structured_outputs={"regex": sample_regex}), + ) + ip1 = chat_completion.choices[0].message.content + assert ip1 is not None + assert re.fullmatch(sample_regex, ip1) is not None + + messages.append({"role": "assistant", "content": ip1}) + messages.append({"role": "user", "content": "Give me a different one"}) + chat_completion = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_completion_tokens=20, + extra_body=dict(structured_outputs={"regex": sample_regex}), + ) + ip2 = chat_completion.choices[0].message.content + assert ip2 is not None + assert re.fullmatch(sample_regex, ip2) is not None + assert ip1 != ip2 + + +@pytest.mark.asyncio +async def test_structured_outputs_type_error(client: openai.AsyncOpenAI): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + { + "role": "user", + "content": "The best language for type-safe systems programming is ", + }, + ] + + with pytest.raises(openai.BadRequestError): + _ = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + extra_body=dict(structured_outputs={"regex": {1: "Python", 2: "C++"}}), + ) + + +@pytest.mark.asyncio +async def test_structured_outputs_choice_chat_logprobs( + client: openai.AsyncOpenAI, sample_structured_outputs_choices +): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + { + "role": "user", + "content": "The best language for type-safe systems programming is ", + }, + ] + chat_completion = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_completion_tokens=10, + logprobs=True, + top_logprobs=5, + extra_body=dict( + structured_outputs={"choice": sample_structured_outputs_choices} + ), + ) + + assert chat_completion.choices[0].logprobs is not None + assert chat_completion.choices[0].logprobs.content is not None + top_logprobs = chat_completion.choices[0].logprobs.content[0].top_logprobs + + # -9999.0 is the minimum logprob returned by OpenAI + for item in top_logprobs: + assert item.logprob >= -9999.0, f"Failed (top_logprobs={top_logprobs})" + + +@pytest.mark.asyncio +async def test_response_format_json_object(client: openai.AsyncOpenAI): + for _ in range(2): + resp = await client.chat.completions.create( + model=MODEL_NAME, + messages=[ + { + "role": "user", + "content": ( + "what is 1+1? please respond with a JSON object, " + 'the format is {"result": 2}' + ), + } + ], + response_format={"type": "json_object"}, + ) + + content = resp.choices[0].message.content + assert content is not None + + loaded = json.loads(content) + assert loaded == {"result": 2}, loaded + + +@pytest.mark.asyncio +async def test_response_format_json_schema(client: openai.AsyncOpenAI): + prompt = 'what is 1+1? The format is "result": 2' + # Check that this prompt cannot lead to a valid JSON without json_schema + for _ in range(2): + resp = await client.chat.completions.create( + model=MODEL_NAME, + messages=[{"role": "user", "content": prompt}], + ) + content = resp.choices[0].message.content + assert content is not None + with pytest.raises((json.JSONDecodeError, AssertionError)): + loaded = json.loads(content) + assert loaded == {"result": 2}, loaded + + for _ in range(2): + resp = await client.chat.completions.create( + model=MODEL_NAME, + messages=[{"role": "user", "content": prompt}], + response_format={ + "type": "json_schema", + "json_schema": { + "name": "foo_test", + "schema": { + "type": "object", + "properties": { + "result": {"type": "integer"}, + }, + }, + }, + }, + ) + + content = resp.choices[0].message.content + assert content is not None + + loaded = json.loads(content) + assert loaded == {"result": 2}, loaded + + +@pytest.mark.asyncio +async def test_response_format_text(client: openai.AsyncOpenAI): + for _ in range(2): + resp = await client.chat.completions.create( + model=MODEL_NAME, + messages=[ + { + "role": "user", + "content": "what is 1+1?", + } + ], + max_completion_tokens=10, + response_format={"type": "text"}, + ) + + content = resp.choices[0].message.content + assert content is not None + + +@pytest.mark.asyncio +async def test_extra_fields_allowed(client: openai.AsyncOpenAI): + resp = await client.chat.completions.create( + model=MODEL_NAME, + messages=[ + { + "role": "user", + "content": "what is 1+1?", + "extra_field": "0", + } + ], # type: ignore + temperature=0, + seed=0, + ) + + content = resp.choices[0].message.content + assert content is not None + + +@pytest.mark.asyncio +async def test_complex_message_content(client: openai.AsyncOpenAI): + content = [ + { + "type": "text", + "text": "what is 1+1? please provide the result without any other text.", + } + ] + resp = await client.chat.completions.create( + model=MODEL_NAME, + messages=[ + { + "role": "user", + "content": content, + } + ], + temperature=0, + seed=0, + ) + content = resp.choices[0].message.content + assert content == "2" + + +@pytest.mark.asyncio +async def test_custom_role(client: openai.AsyncOpenAI): + # Not sure how the model handles custom roles so we just check that + # both string and complex message content are handled in the same way + + resp1 = await client.chat.completions.create( + model=MODEL_NAME, + messages=[ + { + "role": "my-custom-role", + "content": "what is 1+1?", + } + ], # type: ignore + temperature=0, + seed=0, + ) + + resp2 = await client.chat.completions.create( + model=MODEL_NAME, + messages=[ + { + "role": "my-custom-role", + "content": [{"type": "text", "text": "what is 1+1?"}], + } + ], # type: ignore + temperature=0, + seed=0, + ) + + content1 = resp1.choices[0].message.content + content2 = resp2.choices[0].message.content + assert content1 == content2 + + +@pytest.mark.asyncio +async def test_long_seed(client: openai.AsyncOpenAI): + for seed in [torch.iinfo(torch.long).min - 1, torch.iinfo(torch.long).max + 1]: + with pytest.raises(BadRequestError) as exc_info: + await client.chat.completions.create( + model=MODEL_NAME, + messages=[ + { + "role": "system", + "content": "You are a helpful assistant.", + } + ], + temperature=0, + seed=seed, + ) + + assert ( + "greater_than_equal" in exc_info.value.message + or "less_than_equal" in exc_info.value.message + ) + + +@pytest.mark.asyncio +async def test_invocations(server: RemoteOpenAIServer, client: openai.AsyncOpenAI): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + {"role": "user", "content": "what is 1+1?"}, + ] + + request_args = { + "model": MODEL_NAME, + "messages": messages, + "max_completion_tokens": 5, + "temperature": 0.0, + "logprobs": False, + } + + # Use raw HTTP for both endpoints so we compare server responses + # directly, without the openai SDK injecting extra fields + # (e.g. `moderation` added in newer SDK versions). + chat_response = requests.post( + server.url_for("v1/chat/completions"), json=request_args + ) + chat_response.raise_for_status() + + invocation_response = requests.post( + server.url_for("invocations"), json=request_args + ) + invocation_response.raise_for_status() + + chat_output = chat_response.json() + invocation_output = invocation_response.json() + + extra_keys = invocation_output.keys() - chat_output.keys() + missing_keys = chat_output.keys() - invocation_output.keys() + assert missing_keys == set() + assert extra_keys <= {"moderation"} + assert chat_output["choices"] == invocation_output["choices"] + + +# Test n parameter for chat completions +# Tests that the n parameter works correctly for regular sampling +# (non-beam search) in chat completions, addressing issue #34305. + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_chat_completion_n_parameter_non_streaming( + client: openai.AsyncOpenAI, model_name: str +): + """Test that n parameter returns multiple choices for non-streaming requests.""" + messages = [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "What is the opposite of big?"}, + ] + + # Test with n=3 + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=50, + temperature=1.0, + n=3, + seed=42, + stream=False, + ) + + assert len(chat_completion.choices) == 3 + + # Verify each choice has content and correct index + for i, choice in enumerate(chat_completion.choices): + assert choice.index == i + assert choice.message.content is not None + assert len(choice.message.content) > 0 + + contents = [choice.message.content for choice in chat_completion.choices] + assert len(set(contents)) > 1, "Expected different responses with n=3" + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_chat_completion_n_parameter_streaming( + client: openai.AsyncOpenAI, model_name: str +): + """Test that n parameter returns multiple choices for streaming requests.""" + messages = [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "What is the capital of France?"}, + ] + + stream = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=15, + temperature=0.7, + n=2, + stream=True, + ) + + # Collect all chunks using defaultdict for dynamic handling + chunks_by_index = defaultdict(list) + async for chunk in stream: + for choice in chunk.choices: + if choice.delta.content: + chunks_by_index[choice.index].append(choice.delta.content) + + # Verify both choices received content + assert len(chunks_by_index[0]) > 0, "Choice 0 received no content chunks" + assert len(chunks_by_index[1]) > 0, "Choice 1 received no content chunks" + + # Reconstruct full responses + response_0 = "".join(chunks_by_index[0]) + response_1 = "".join(chunks_by_index[1]) + + assert len(response_0) > 0, "Choice 0 has empty response" + assert len(response_1) > 0, "Choice 1 has empty response" + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_chat_completion_n_with_seed(client: openai.AsyncOpenAI, model_name: str): + """Test that n parameter works correctly with seed parameter.""" + messages = [ + {"role": "user", "content": "Say hello."}, + ] + + # Test that seed parameter is accepted and works with n > 1 + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.8, + n=2, + seed=42, + stream=False, + ) + + # Verify we get n=2 choices + assert len(chat_completion.choices) == 2 + + # Verify both choices have valid content + for i, choice in enumerate(chat_completion.choices): + assert choice.index == i + assert choice.message.content is not None + assert len(choice.message.content) > 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_chat_completion_n_equals_1(client: openai.AsyncOpenAI, model_name: str): + """Test that n=1 (default) still works correctly.""" + messages = [ + {"role": "user", "content": "Hello!"}, + ] + + chat_completion = await client.chat.completions.create( + model=model_name, + messages=messages, + max_completion_tokens=10, + temperature=0.7, + n=1, + stream=False, + ) + + assert len(chat_completion.choices) == 1 + assert chat_completion.choices[0].index == 0 + assert chat_completion.choices[0].message.content is not None + + +# Unit tests for n parameter in ChatCompletionRequest.to_sampling_params() +def test_chat_completion_request_n_parameter_to_sampling_params(): + """Test that n parameter is correctly passed to SamplingParams.""" + # Test with n=3 + request = ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "Hello"}], + n=3, + max_tokens=10, + ) + + sampling_params = request.to_sampling_params( + max_tokens=10, + default_sampling_params={}, + ) + + assert isinstance(sampling_params, SamplingParams) + assert sampling_params.n == 3, f"Expected n=3, got n={sampling_params.n}" + + +def test_chat_completion_request_n_parameter_default(): + """Test that n parameter defaults to 1.""" + request = ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "Hello"}], + # n not specified, should default to 1 + max_tokens=10, + ) + + assert request.n == 1, "n should default to 1" + sampling_params = request.to_sampling_params( + max_tokens=10, + default_sampling_params={}, + ) + + # SamplingParams.from_optional converts None to 1 + assert sampling_params.n == 1, f"Expected n=1 (default), got n={sampling_params.n}" + + +def test_chat_completion_request_accepts_model_specific_reasoning_effort(): + request = ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "Hello"}], + reasoning_effort="max", + ) + + chat_params = request.build_chat_params( + default_template=None, + default_template_content_format="auto", + ) + + assert request.reasoning_effort == "max" + assert chat_params.chat_template_kwargs["reasoning_effort"] == "max" + + +def test_chat_completion_request_rejects_unknown_reasoning_effort(): + with pytest.raises(ValueError, match="Input should be"): + ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "Hello"}], + reasoning_effort="extra_high", + ) + + +def test_chat_completion_request_n_parameter_various_values(): + """Test n parameter with various values.""" + for n_value in [1, 2, 5, 10]: + request = ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "Test"}], + n=n_value, + max_tokens=10, + ) + + sampling_params = request.to_sampling_params( + max_tokens=10, + default_sampling_params={}, + ) + + assert sampling_params.n == n_value, ( + f"Expected n={n_value}, got n={sampling_params.n}" + ) + + +def test_chat_completion_request_n_parameter_exceeds_default_limit( + monkeypatch: pytest.MonkeyPatch, +): + """Test that n values exceeding the default limit are rejected.""" + import vllm.envs as envs + + monkeypatch.delenv("VLLM_MAX_N_SEQUENCES", raising=False) + if hasattr(envs.__getattr__, "cache_clear"): + envs.__getattr__.cache_clear() + + max_n = envs.VLLM_MAX_N_SEQUENCES + request = ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "Test"}], + n=max_n + 1, + max_tokens=10, + ) + + with pytest.raises(ValueError, match="n must be at most"): + request.to_sampling_params( + max_tokens=10, + default_sampling_params={}, + ) + + +def test_chat_completion_request_n_parameter_at_limit( + monkeypatch: pytest.MonkeyPatch, +): + """Test that n at exactly the limit is accepted.""" + import vllm.envs as envs + + monkeypatch.delenv("VLLM_MAX_N_SEQUENCES", raising=False) + if hasattr(envs.__getattr__, "cache_clear"): + envs.__getattr__.cache_clear() + + max_n = envs.VLLM_MAX_N_SEQUENCES + request = ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "Test"}], + n=max_n, + max_tokens=10, + ) + + sampling_params = request.to_sampling_params( + max_tokens=10, + default_sampling_params={}, + ) + assert sampling_params.n == max_n + + +def test_chat_completion_request_n_parameter_custom_limit( + monkeypatch: pytest.MonkeyPatch, +): + """Test that VLLM_MAX_N_SEQUENCES env var overrides the default limit.""" + import vllm.envs as envs + + monkeypatch.setenv("VLLM_MAX_N_SEQUENCES", "128") + if hasattr(envs.__getattr__, "cache_clear"): + envs.__getattr__.cache_clear() + + request = ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "Test"}], + n=128, + max_tokens=10, + ) + + sampling_params = request.to_sampling_params( + max_tokens=10, + default_sampling_params={}, + ) + assert sampling_params.n == 128 + + request_over = ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "Test"}], + n=129, + max_tokens=10, + ) + + with pytest.raises(ValueError, match="n must be at most 128"): + request_over.to_sampling_params( + max_tokens=10, + default_sampling_params={}, + ) + + +def test_chat_completion_request_n_parameter_massive_value( + monkeypatch: pytest.MonkeyPatch, +): + """Test that astronomically large n values are rejected (CVE fix).""" + import vllm.envs as envs + + monkeypatch.delenv("VLLM_MAX_N_SEQUENCES", raising=False) + if hasattr(envs.__getattr__, "cache_clear"): + envs.__getattr__.cache_clear() + + request = ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "Test"}], + n=100_000_000, + max_tokens=1, + ) + + with pytest.raises(ValueError, match="n must be at most"): + request.to_sampling_params( + max_tokens=1, + default_sampling_params={}, + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_completion.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_completion.py new file mode 100644 index 0000000000000000000000000000000000000000..02792c327685cb3622168cce9e547a2fcc843592 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_completion.py @@ -0,0 +1,160 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import openai # use the official client for correctness check +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer + +# any model with a chat template defined in tokenizer_config should work here +MODEL_NAME = "Qwen/Qwen2.5-1.5B-Instruct" + + +@pytest.fixture(scope="module") +def default_server_args(): + return [ + # use half precision for speed and memory savings in CI environment + "--max-model-len", + "2048", + "--max-num-seqs", + "128", + "--enforce-eager", + ] + + +@pytest.fixture(scope="module") +def server(default_server_args): + with RemoteOpenAIServer(MODEL_NAME, default_server_args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_invalid_json_schema(client: openai.AsyncOpenAI, model_name: str) -> None: + invalid_json_schema = { + "$defs": { + "CarType": { + "enum": ["sedan", "SUV", "Truck", "Coupe"], + "title": "CarType", + "type": "string", + } + }, + "properties": { + "brand": {"title": "Brand", "type": "string"}, + "model": {"title": "Model", "type": "string"}, + "car_type": {"$ref": "#/$defs/CarType"}, + "foo": "bar", + }, + "required": ["brand", "model", "car_type"], + "title": "CarDescription", + "type": "object", + } + prompt = ( + "Generate a JSON with the brand, model and car_type of" + "the most iconic car from the 90's" + ) + with pytest.raises((openai.BadRequestError, openai.APIError)): + await client.chat.completions.create( + model=model_name, + messages=[ + { + "role": "user", + "content": prompt, + } + ], + extra_body={"structured_outputs": {"json": invalid_json_schema}}, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_invalid_regex(client: openai.AsyncOpenAI, model_name: str): + prompt = ( + "Generate an email address for Alan Turing, who works in Enigma." + "End in .com and new line. Example result:" + "alan.turing@enigma.com\n" + ) + + with pytest.raises((openai.BadRequestError, openai.APIError)): + await client.chat.completions.create( + model=model_name, + messages=[ + { + "role": "user", + "content": prompt, + } + ], + extra_body={"structured_outputs": {"regex": r"[.*"}, "stop": ["\n"]}, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_invalid_grammar(client: openai.AsyncOpenAI, model_name: str): + invalid_simplified_sql_grammar = """ + root ::= select_statementinvalidsyntax + + select_statement ::= "SELECT " column " from " table " where " condition + + column ::= "col_1 " | "col_2 " + + table ::= "table_1 " | "table_2 " + + condition ::= column "= " number + + number ::= "1 " | "2 " + """ + + prompt = ( + "Generate an SQL query to show the 'username' and 'email'" + "from the 'users' table." + ) + with pytest.raises((openai.BadRequestError, openai.APIError)): + await client.chat.completions.create( + model=model_name, + messages=[ + { + "role": "user", + "content": prompt, + } + ], + extra_body={ + "structured_outputs": {"grammar": invalid_simplified_sql_grammar} + }, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_empty_grammar(client: openai.AsyncOpenAI, model_name: str) -> None: + prompt = "Say hello" + with pytest.raises((openai.BadRequestError, openai.APIError)): + await client.chat.completions.create( + model=model_name, + messages=[ + { + "role": "user", + "content": prompt, + } + ], + extra_body={"structured_outputs": {"grammar": ""}}, + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_completion_with_prompt_embeds.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_completion_with_prompt_embeds.py new file mode 100644 index 0000000000000000000000000000000000000000..a8d7ead415e8bd5b1b79999b2b8b2780cddaf56f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_completion_with_prompt_embeds.py @@ -0,0 +1,298 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""E2E tests for `prompt_embeds` content parts in the Chat Completions API.""" + +import asyncio +import io + +import openai +import pybase64 as base64 +import pytest +import pytest_asyncio +import torch +from openai import BadRequestError + +from tests.utils import VLLM_PATH, RemoteOpenAIServer +from vllm.platforms import current_platform + +MODEL_NAME = "facebook/opt-125m" +CHAT_TEMPLATE = VLLM_PATH / "examples/template_chatml.jinja" +# Matches `--dtype` in `server_args` to avoid an implicit cast in +# `safe_load_prompt_embeds` (mismatched floating-point dtypes are cast to the +# model's dtype automatically, we match here just to skip the conversion). +SERVER_DTYPE: torch.dtype = torch.bfloat16 + + +@pytest.fixture(scope="module") +def server_args() -> list[str]: + return [ + "--dtype", + "bfloat16", + "--max-model-len", + "2048", + "--max-num-seqs", + "128", + "--enforce-eager", + "--chat-template", + str(CHAT_TEMPLATE), + # Prompt Embeds server args + "--enable-prompt-embeds", + ] + + +@pytest.fixture(scope="module") +def server(server_args, request): + if current_platform.is_rocm(): + # Materialize HF embeddings before the server reserves ROCm VRAM. + request.getfixturevalue("prompt_embeds_b64") + request.getfixturevalue("aligned_content_and_embeds_b64") + with RemoteOpenAIServer(MODEL_NAME, server_args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +def _encode_embeds(embeds: torch.Tensor) -> str: + buf = io.BytesIO() + torch.save(embeds, buf) + return base64.b64encode(buf.getvalue()).decode("utf-8") + + +@pytest.fixture(scope="module") +def prompt_embeds_b64(hf_runner) -> list[str]: + """Pre-compute embeddings for two short prompts and return as base64.""" + prompts = ["Hello, my name is", "What is an LLM?"] + with hf_runner(MODEL_NAME) as hf_model: + embeddings = hf_model.get_prompt_embeddings(prompts) + # Cast to the server's dtype so `safe_load_prompt_embeds` doesn't need to + # convert on its own, the function accepts any floating-point dtype and + # will cast to the model's dtype, but matching up front skips the work. + return [_encode_embeds(e.to(SERVER_DTYPE)) for e in embeddings] + + +@pytest.mark.asyncio +async def test_single_prompt_embeds_part( + client: openai.AsyncOpenAI, + prompt_embeds_b64: list[str], +): + """A user message with one prompt_embeds part + text.""" + b64 = prompt_embeds_b64[0] + chat = await client.chat.completions.create( + model=MODEL_NAME, + max_tokens=5, + temperature=0.0, + messages=[ + { + "role": "user", + "content": [ + {"type": "prompt_embeds", "data": b64}, + {"type": "text", "text": "Continue:"}, + ], + } + ], + ) + assert chat.choices[0].message.content is not None + assert len(chat.choices[0].message.content) > 0 + + +@pytest.mark.asyncio +async def test_multiple_prompt_embeds_parts( + client: openai.AsyncOpenAI, + prompt_embeds_b64: list[str], +): + """Multiple prompt_embeds parts in a single message.""" + b64_a, b64_b = prompt_embeds_b64 + chat = await client.chat.completions.create( + model=MODEL_NAME, + max_tokens=5, + temperature=0.0, + messages=[ + { + "role": "user", + "content": [ + {"type": "prompt_embeds", "data": b64_a}, + {"type": "text", "text": " and "}, + {"type": "prompt_embeds", "data": b64_b}, + ], + } + ], + ) + assert chat.choices[0].message.content is not None + assert len(chat.choices[0].message.content) > 0 + + +@pytest.mark.asyncio +async def test_multi_message_conversation( + client: openai.AsyncOpenAI, + prompt_embeds_b64: list[str], +): + """prompt_embeds in both system and user messages.""" + b64_sys, b64_usr = prompt_embeds_b64 + chat = await client.chat.completions.create( + model=MODEL_NAME, + max_tokens=5, + temperature=0.0, + messages=[ + { + "role": "system", + "content": [ + {"type": "text", "text": "You are helpful."}, + {"type": "prompt_embeds", "data": b64_sys}, + ], + }, + { + "role": "user", + "content": [ + {"type": "prompt_embeds", "data": b64_usr}, + {"type": "text", "text": "Summarize."}, + ], + }, + ], + ) + assert chat.choices[0].message.content is not None + assert len(chat.choices[0].message.content) > 0 + + +@pytest.mark.asyncio +async def test_streaming( + client: openai.AsyncOpenAI, + prompt_embeds_b64: list[str], +): + """Streaming chat completion with prompt_embeds.""" + b64 = prompt_embeds_b64[0] + + # Non-streaming baseline. + baseline = await client.chat.completions.create( + model=MODEL_NAME, + max_tokens=5, + temperature=0.0, + messages=[ + { + "role": "user", + "content": [ + {"type": "prompt_embeds", "data": b64}, + {"type": "text", "text": "Continue:"}, + ], + } + ], + ) + expected = baseline.choices[0].message.content + + # Streaming. + stream = await client.chat.completions.create( + model=MODEL_NAME, + max_tokens=5, + temperature=0.0, + stream=True, + messages=[ + { + "role": "user", + "content": [ + {"type": "prompt_embeds", "data": b64}, + {"type": "text", "text": "Continue:"}, + ], + } + ], + ) + chunks: list[str] = [] + async for chunk in stream: + delta = chunk.choices[0].delta.content + if delta: + chunks.append(delta) + assert "".join(chunks) == expected + + +@pytest.fixture(scope="module") +def aligned_content_and_embeds_b64(hf_runner) -> tuple[str, str]: + """Return `(content, base64_embeds)` where the embeddings are the model's + embedding of `content` tokenized WITHOUT special tokens. + """ + content = "Hello, my name is" + with hf_runner(MODEL_NAME) as hf_model: + ids = hf_model.tokenizer( + content, add_special_tokens=False, return_tensors="pt" + ).input_ids + ids = hf_model.wrap_device({"input_ids": ids})["input_ids"] + embed_layer = hf_model.model.get_input_embeddings() + embeds = embed_layer(ids).squeeze(0).to(SERVER_DTYPE).cpu() + return content, _encode_embeds(embeds) + + +@pytest.mark.asyncio +async def test_text_content_and_prompt_embeds_match( + client: openai.AsyncOpenAI, + aligned_content_and_embeds_b64: tuple[str, str], +): + """Equal content in text and `prompt_embeds` should yield identical + Chat Completions output under greedy decoding. + """ + content, encoded_embeds = aligned_content_and_embeds_b64 + + text_resp, embeds_resp = await asyncio.gather( + client.chat.completions.create( + model=MODEL_NAME, + max_tokens=10, + temperature=0.0, + messages=[{"role": "user", "content": content}], + ), + client.chat.completions.create( + model=MODEL_NAME, + max_tokens=10, + temperature=0.0, + messages=[ + { + "role": "user", + "content": [{"type": "prompt_embeds", "data": encoded_embeds}], + } + ], + ), + ) + + text_out = text_resp.choices[0].message.content + embeds_out = embeds_resp.choices[0].message.content + assert text_out is not None and len(text_out) > 0 + assert embeds_out is not None and len(embeds_out) > 0 + assert text_out == embeds_out + + +@pytest.mark.asyncio +async def test_missing_data_field( + client: openai.AsyncOpenAI, +): + """A prompt_embeds part without `data` should return a clear error.""" + with pytest.raises(BadRequestError): + await client.chat.completions.create( + model=MODEL_NAME, + max_tokens=5, + messages=[ + { + "role": "user", + "content": [{"type": "prompt_embeds"}], + } + ], + ) + + +@pytest.mark.asyncio +async def test_invalid_base64( + client: openai.AsyncOpenAI, +): + """Invalid base64 in the `data` field should return a clear error.""" + with pytest.raises(BadRequestError): + await client.chat.completions.create( + model=MODEL_NAME, + max_tokens=5, + messages=[ + { + "role": "user", + "content": [ + {"type": "prompt_embeds", "data": "not_valid_base64!!"}, + ], + } + ], + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_echo.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_echo.py new file mode 100644 index 0000000000000000000000000000000000000000..3d0927d653c966d665f84fe2e981bf878a463816 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_echo.py @@ -0,0 +1,131 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from typing import NamedTuple + +import openai # use the official client for correctness check +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer +from vllm.config import ModelConfig + +# # any model with a chat template should work here +MODEL_NAME = "Qwen/Qwen2-1.5B-Instruct" + + +def get_vocab_size(model_name): + config = ModelConfig( + model=model_name, + seed=0, + dtype="float16", + ) + return config.get_vocab_size() + + +@pytest.fixture(scope="module") +def server(): + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "float16", + "--enforce-eager", + "--max-model-len", + "4080", + "--max-logprobs", # test prompt_logprobs equal to -1 + "151936", + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +class TestCase(NamedTuple): + model_name: str + echo: bool + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "test_case", + [ + TestCase(model_name=MODEL_NAME, echo=True), + TestCase(model_name=MODEL_NAME, echo=False), + ], +) +async def test_chat_session_with_echo_and_continue_final_message( + client: openai.AsyncOpenAI, test_case: TestCase +): + saying: str = "Here is a common saying about apple. An apple a day, keeps" + # test echo with continue_final_message parameter + chat_completion = await client.chat.completions.create( + model=test_case.model_name, + messages=[ + {"role": "user", "content": "tell me a common saying"}, + {"role": "assistant", "content": saying}, + ], + extra_body={ + "echo": test_case.echo, + "continue_final_message": True, + "add_generation_prompt": False, + }, + ) + assert chat_completion.id is not None + assert len(chat_completion.choices) == 1 + + choice = chat_completion.choices[0] + assert choice.finish_reason == "stop" + + message = choice.message + if test_case.echo: + assert message.content is not None and saying in message.content + else: + assert message.content is not None and saying not in message.content + assert message.role == "assistant" + + +@pytest.mark.asyncio +async def test_prompt_logprobs(client: openai.AsyncOpenAI): + messages = [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Beijing is the capital of which country?"}, + ] + + completion = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + extra_body={"prompt_logprobs": -1}, + ) + + assert completion.prompt_logprobs is not None + assert len(completion.prompt_logprobs) > 0 + + +@pytest.mark.asyncio +async def test_top_logprobs(client: openai.AsyncOpenAI): + messages = [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Beijing is the capital of which country?"}, + ] + + completion = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_tokens=1, + extra_body={ + "top_logprobs": -1, + "logprobs": "true", + }, + ) + assert completion.choices[0].logprobs is not None + assert completion.choices[0].logprobs.content is not None + assert len(completion.choices[0].logprobs.content) > 0 + assert len( + completion.choices[0].logprobs.content[0].top_logprobs + ) == get_vocab_size(MODEL_NAME) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_error.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_error.py new file mode 100644 index 0000000000000000000000000000000000000000..808778541ed3a5d169f4b4280cd7d45ede8204a9 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_error.py @@ -0,0 +1,517 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from dataclasses import dataclass, field +from typing import Any +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest +from pydantic import ValidationError + +from vllm.config.multimodal import MultiModalConfig +from vllm.entrypoints.openai.chat_completion.protocol import ( + BatchChatCompletionRequest, + ChatCompletionRequest, +) +from vllm.entrypoints.openai.chat_completion.serving import OpenAIServingChat +from vllm.entrypoints.openai.engine.protocol import GenerationError +from vllm.entrypoints.openai.models.protocol import BaseModelPath +from vllm.entrypoints.openai.models.serving import OpenAIServingModels +from vllm.entrypoints.scale_out.render.serving import ServingRender +from vllm.outputs import CompletionOutput, RequestOutput +from vllm.renderers.hf import HfRenderer +from vllm.renderers.online_renderer import OnlineRenderer +from vllm.tokenizers.registry import cached_tokenizer_from_config +from vllm.v1.engine.async_llm import AsyncLLM + +MODEL_NAME = "openai-community/gpt2" +MODEL_NAME_SHORT = "gpt2" +BASE_MODEL_PATHS = [ + BaseModelPath(name=MODEL_NAME, model_path=MODEL_NAME), + BaseModelPath(name=MODEL_NAME_SHORT, model_path=MODEL_NAME_SHORT), +] + + +@dataclass +class MockHFConfig: + model_type: str = "any" + + +@dataclass +class MockModelConfig: + task = "generate" + runner_type = "generate" + model = MODEL_NAME + tokenizer = MODEL_NAME + trust_remote_code = False + tokenizer_mode = "auto" + max_model_len = 100 + tokenizer_revision = None + multimodal_config = MultiModalConfig() + hf_config = MockHFConfig() + hf_text_config = MockHFConfig() + logits_processors: list[str] | None = None + diff_sampling_param: dict | None = None + allowed_local_media_path: str = "" + allowed_media_domains: list[str] | None = None + encoder_config = None + generation_config: str = "auto" + media_io_kwargs: dict[str, dict[str, Any]] = field(default_factory=dict) + skip_tokenizer_init = False + is_encoder_decoder: bool = False + is_multimodal_model: bool = False + renderer_num_workers: int = 1 + + def get_diff_sampling_param(self): + return self.diff_sampling_param or {} + + +@dataclass +class MockParallelConfig: + _api_process_rank: int = 0 + + +@dataclass +class MockVllmConfig: + model_config: MockModelConfig + parallel_config: MockParallelConfig + + +def _build_renderer(model_config: MockModelConfig): + return HfRenderer( + MockVllmConfig(model_config, parallel_config=MockParallelConfig()), + cached_tokenizer_from_config(model_config), + ) + + +def _build_serving_chat(engine: AsyncLLM) -> OpenAIServingChat: + models = OpenAIServingModels( + engine_client=engine, + base_model_paths=BASE_MODEL_PATHS, + ) + online_renderer = OnlineRenderer( + model_config=engine.model_config, + renderer=engine.renderer, + request_logger=None, + chat_template=None, + chat_template_content_format="auto", + ) + + serving_chat = OpenAIServingChat( + engine, + models, + response_role="assistant", + online_renderer=online_renderer, + request_logger=None, + chat_template=None, + chat_template_content_format="auto", + ) + + async def _fake_preprocess_chat(*args, **kwargs): + # return conversation, engine_inputs + return ( + [{"role": "user", "content": "Test"}], + [{"prompt_token_ids": [1, 2, 3]}], + ) + + serving_chat.online_renderer.preprocess_chat = AsyncMock( + side_effect=_fake_preprocess_chat + ) + return serving_chat + + +@pytest.mark.asyncio +async def test_chat_error_non_stream(): + """test finish_reason='error' returns 500 InternalServerError (non-streaming)""" + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + serving_chat = _build_serving_chat(mock_engine) + + completion_output = CompletionOutput( + index=0, + text="", + token_ids=[], + cumulative_logprob=None, + logprobs=None, + finish_reason="error", + ) + + request_output = RequestOutput( + request_id="test-id", + prompt="Test prompt", + prompt_token_ids=[1, 2, 3], + prompt_logprobs=None, + outputs=[completion_output], + finished=True, + metrics=None, + lora_request=None, + encoder_prompt=None, + encoder_prompt_token_ids=None, + ) + + async def mock_generate(*args, **kwargs): + yield request_output + + mock_engine.generate = MagicMock(side_effect=mock_generate) + + request = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "Test prompt"}], + max_tokens=10, + stream=False, + ) + + with pytest.raises(GenerationError): + await serving_chat.create_chat_completion(request) + + +@pytest.mark.asyncio +async def test_openai_chat_keeps_mm_cache_for_engine_execution(): + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + serving_chat = _build_serving_chat(mock_engine) + + request = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "Test prompt"}], + ) + + result = await serving_chat.render_chat_request(request) + + assert isinstance(result, tuple) + assert ( + serving_chat.online_renderer.preprocess_chat.call_args.kwargs["skip_mm_cache"] + is False + ) + + +def _build_serving_render(engine: AsyncLLM) -> ServingRender: + models = OpenAIServingModels( + engine_client=engine, + base_model_paths=BASE_MODEL_PATHS, + ) + online_renderer = OnlineRenderer( + model_config=engine.model_config, + renderer=engine.renderer, + request_logger=None, + chat_template=None, + chat_template_content_format="auto", + ) + + serving_render = ServingRender(models, online_renderer) + + async def _fake_preprocess_chat(*args, **kwargs): + # return conversation, engine_inputs + return ( + [{"role": "user", "content": "Test"}], + [{"prompt_token_ids": [1, 2, 3]}], + ) + + serving_render.online_renderer.preprocess_chat = AsyncMock( + side_effect=_fake_preprocess_chat + ) + return serving_render + + +@pytest.mark.asyncio +async def test_renderer_only_chat_request_skips_mm_cache(): + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + serving_render = _build_serving_render(mock_engine) + + request = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "Test prompt"}], + ) + + result = await serving_render.render_chat_request(request) + + assert result.token_ids == [1, 2, 3] + assert ( + serving_render.online_renderer.preprocess_chat.call_args.kwargs["skip_mm_cache"] + is True + ) + + +@pytest.mark.asyncio +async def test_chat_error_stream(): + """test finish_reason='error' returns 500 InternalServerError (streaming)""" + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + serving_chat = _build_serving_chat(mock_engine) + + completion_output_1 = CompletionOutput( + index=0, + text="Hello", + token_ids=[100], + cumulative_logprob=None, + logprobs=None, + finish_reason=None, + ) + + request_output_1 = RequestOutput( + request_id="test-id", + prompt="Test prompt", + prompt_token_ids=[1, 2, 3], + prompt_logprobs=None, + outputs=[completion_output_1], + finished=False, + metrics=None, + lora_request=None, + encoder_prompt=None, + encoder_prompt_token_ids=None, + ) + + completion_output_2 = CompletionOutput( + index=0, + text="Hello", + token_ids=[100], + cumulative_logprob=None, + logprobs=None, + finish_reason="error", + ) + + request_output_2 = RequestOutput( + request_id="test-id", + prompt="Test prompt", + prompt_token_ids=[1, 2, 3], + prompt_logprobs=None, + outputs=[completion_output_2], + finished=True, + metrics=None, + lora_request=None, + encoder_prompt=None, + encoder_prompt_token_ids=None, + ) + + async def mock_generate(*args, **kwargs): + yield request_output_1 + yield request_output_2 + + mock_engine.generate = MagicMock(side_effect=mock_generate) + + request = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "Test prompt"}], + max_tokens=10, + stream=True, + ) + + response = await serving_chat.create_chat_completion(request) + + chunks = [] + async for chunk in response: + chunks.append(chunk) + + assert len(chunks) >= 2 + assert any("Internal server error" in chunk for chunk in chunks), ( + f"Expected error message in chunks: {chunks}" + ) + assert chunks[-1] == "data: [DONE]\n\n" + + +@pytest.mark.parametrize( + "image_content", + [ + [{"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}], + [{"image_url": {"url": "https://example.com/image.jpg"}}], + ], +) +def test_system_message_warns_on_image(image_content): + """Test that system messages with image content trigger a warning.""" + with patch( + "vllm.entrypoints.openai.chat_completion.protocol.logger" + ) as mock_logger: + ChatCompletionRequest( + model=MODEL_NAME, + messages=[ + { + "role": "system", + "content": image_content, + } + ], + ) + + mock_logger.warning_once.assert_called() + call_args = str(mock_logger.warning_once.call_args) + assert "System messages should only contain text" in call_args + assert "image_url" in call_args + + +def test_system_message_accepts_text(): + """Test that system messages can contain text content.""" + # Should not raise an exception + request = ChatCompletionRequest( + model=MODEL_NAME, + messages=[ + {"role": "system", "content": "You are a helpful assistant."}, + ], + ) + assert request.messages[0]["role"] == "system" + + +def test_system_message_accepts_text_array(): + """Test that system messages can contain an array with text content.""" + # Should not raise an exception + request = ChatCompletionRequest( + model=MODEL_NAME, + messages=[ + { + "role": "system", + "content": [{"type": "text", "text": "You are a helpful assistant."}], + }, + ], + ) + assert request.messages[0]["role"] == "system" + + +def test_user_message_accepts_image(): + """Test that user messages can still contain image content.""" + # Should not raise an exception + request = ChatCompletionRequest( + model=MODEL_NAME, + messages=[ + { + "role": "user", + "content": [ + {"type": "text", "text": "What's in this image?"}, + { + "type": "image_url", + "image_url": {"url": "https://example.com/image.jpg"}, + }, + ], + }, + ], + ) + assert request.messages[0]["role"] == "user" + + +@pytest.mark.parametrize( + "audio_content", + [ + [ + { + "type": "input_audio", + "input_audio": {"data": "base64data", "format": "wav"}, + } + ], + [{"input_audio": {"data": "base64data", "format": "wav"}}], + ], +) +def test_system_message_warns_on_audio(audio_content): + """Test that system messages with audio content trigger a warning.""" + with patch( + "vllm.entrypoints.openai.chat_completion.protocol.logger" + ) as mock_logger: + ChatCompletionRequest( + model=MODEL_NAME, + messages=[ + { + "role": "system", + "content": audio_content, + } + ], + ) + + mock_logger.warning_once.assert_called() + call_args = str(mock_logger.warning_once.call_args) + assert "System messages should only contain text" in call_args + assert "input_audio" in call_args + + +@pytest.mark.parametrize( + "video_content", + [ + [{"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}}], + [{"video_url": {"url": "https://example.com/video.mp4"}}], + ], +) +def test_system_message_warns_on_video(video_content): + """Test that system messages with video content trigger a warning.""" + with patch( + "vllm.entrypoints.openai.chat_completion.protocol.logger" + ) as mock_logger: + ChatCompletionRequest( + model=MODEL_NAME, + messages=[ + { + "role": "system", + "content": video_content, + } + ], + ) + + mock_logger.warning_once.assert_called() + call_args = str(mock_logger.warning_once.call_args) + assert "System messages should only contain text" in call_args + assert "video_url" in call_args + + +def test_json_schema_response_format_missing_schema(): + """When response_format type is 'json_schema' but the json_schema field + is not provided, request construction should raise a validation error + so the API returns 400 instead of 500.""" + with pytest.raises(Exception, match="json_schema.*must be provided"): + ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "hello"}], + response_format={"type": "json_schema"}, + ) + + +@pytest.mark.parametrize("format_value", [None, {}]) +def test_structural_tag_response_format_invalid(format_value): + """Malformed structural tags should be rejected during request validation.""" + with pytest.raises( + ValidationError, + match="Invalid response_format structural_tag", + ): + ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "hello"}], + response_format={"type": "structural_tag", "format": format_value}, + ) + + +@pytest.mark.parametrize("format_value", [None, {}]) +def test_batch_structural_tag_response_format_invalid(format_value): + """Batch chat should reject malformed structural tags at request parsing.""" + with pytest.raises( + ValidationError, + match="Invalid response_format structural_tag", + ): + BatchChatCompletionRequest( + model=MODEL_NAME, + messages=[[{"role": "user", "content": "hello"}]], + response_format={"type": "structural_tag", "format": format_value}, + ) + + +@pytest.mark.parametrize("structural_tag", ["not json", ""]) +def test_structured_outputs_structural_tag_invalid(structural_tag): + """Malformed direct structured_outputs structural tags should be rejected.""" + with pytest.raises( + ValidationError, + match="Invalid structured_outputs structural_tag", + ): + ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "hello"}], + structured_outputs={"structural_tag": structural_tag}, + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_logit_bias_validation.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_logit_bias_validation.py new file mode 100644 index 0000000000000000000000000000000000000000..b68093eed4fb3c9ce10309cd675f2d353bb8a5be --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_chat_logit_bias_validation.py @@ -0,0 +1,135 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import openai +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer +from vllm.config import ModelConfig + +MODEL_NAME = "Qwen/Qwen2.5-1.5B-Instruct" + + +def get_vocab_size(model_name): + config = ModelConfig( + model=model_name, + seed=0, + dtype="bfloat16", + ) + return config.get_vocab_size() + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "1024", + "--enforce-eager", + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +async def test_chat_logit_bias_valid(client): + """Test that valid logit_bias values are accepted in chat completions.""" + vocab_size = get_vocab_size(MODEL_NAME) + valid_token_id = vocab_size - 1 + + completion = await client.chat.completions.create( + model=MODEL_NAME, + messages=[{"role": "user", "content": "Testing valid logit bias"}], + max_tokens=5, + logit_bias={str(valid_token_id): 1.0}, + ) + + assert completion.choices[0].message.content is not None + + +@pytest.mark.asyncio +async def test_chat_logit_bias_invalid(client): + """Test that invalid logit_bias values are rejected in chat completions.""" + vocab_size = get_vocab_size(MODEL_NAME) + invalid_token_id = vocab_size + 1 + + with pytest.raises(openai.BadRequestError) as excinfo: + await client.chat.completions.create( + model=MODEL_NAME, + messages=[{"role": "user", "content": "Testing invalid logit bias"}], + max_tokens=5, + logit_bias={str(invalid_token_id): 1.0}, + ) + + error = excinfo.value + error_message = str(error) + + assert error.status_code == 400 + assert str(invalid_token_id) in error_message + assert str(vocab_size) in error_message + + +@pytest.mark.asyncio +async def test_chat_logit_bias_non_integer_key(client): + """Test that a non-integer logit_bias key is rejected with a clean, + informative error instead of a raw 'invalid literal for int()' message.""" + with pytest.raises(openai.BadRequestError) as excinfo: + await client.chat.completions.create( + model=MODEL_NAME, + messages=[{"role": "user", "content": "Testing invalid logit bias key"}], + max_tokens=5, + logit_bias={"not_a_token_id": 50}, + ) + + error = excinfo.value + error_message = str(error) + + assert error.status_code == 400 + assert "not_a_token_id" in error_message + assert "logit_bias" in error_message + + +@pytest.mark.asyncio +async def test_chat_logit_bias_non_numeric_value(client): + """Test that a non-numeric logit_bias value is rejected with a message + that names the specific offending token, not just a generic TypeError.""" + with pytest.raises(openai.BadRequestError) as excinfo: + await client.chat.completions.create( + model=MODEL_NAME, + messages=[{"role": "user", "content": "Testing invalid logit bias value"}], + max_tokens=5, + logit_bias={"1": "not_a_number"}, + ) + + error = excinfo.value + error_message = str(error) + + assert error.status_code == 400 + assert "logit_bias" in error_message + + +@pytest.mark.asyncio +async def test_chat_logit_bias_multiple_non_integer_keys(client): + """Test that ALL invalid logit_bias keys are reported together, + not just the first one encountered.""" + with pytest.raises(openai.BadRequestError) as excinfo: + await client.chat.completions.create( + model=MODEL_NAME, + messages=[{"role": "user", "content": "Testing multiple bad keys"}], + max_tokens=5, + logit_bias={"bad1": 50.0, "bad2": 20.0}, + ) + + error_message = str(excinfo.value) + assert excinfo.value.status_code == 400 + assert "bad1" in error_message + assert "bad2" in error_message diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_completion_with_function_calling.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_completion_with_function_calling.py new file mode 100644 index 0000000000000000000000000000000000000000..bda96fc587f5bae16851a8ca8f300456b233b55d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_completion_with_function_calling.py @@ -0,0 +1,474 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import datetime +import json + +import jsonschema +import openai # use the official client for correctness check +import pytest +import pytest_asyncio + +# downloading lora to test lora requests +from tests.utils import ROCM_ENV_OVERRIDES, ROCM_EXTRA_ARGS, RemoteOpenAIServer + +# any model with a chat template should work here +MODEL_NAME = "Qwen/Qwen3-0.6B" + + +tools = [ + { + "type": "function", + "function": { + "name": "get_current_weather", + "description": "Get the current weather in a given location", + "parameters": { + "type": "object", + "strict": True, + "properties": { + "city": { + "type": "string", + "description": "The city to find the weather for, e.g. " + "'Vienna'", + "default": "Vienna", + }, + "country": { + "type": "string", + "description": "The country that the city is in, e.g. " + "'Austria'", + }, + "unit": { + "type": "string", + "description": "The unit to fetch the temperature in", + "enum": ["celsius", "fahrenheit"], + }, + "options": { + "$ref": "#/$defs/WeatherOptions", + "description": "Optional parameters for weather query", + }, + }, + "required": ["country", "unit"], + "$defs": { + "WeatherOptions": { + "title": "WeatherOptions", + "type": "object", + "additionalProperties": False, + "properties": { + "unit": { + "type": "string", + "enum": ["celsius", "fahrenheit"], + "default": "celsius", + "description": "Temperature unit", + "title": "Temperature Unit", + }, + "include_forecast": { + "type": "boolean", + "default": False, + "description": "Whether to include a 24-hour forecast", + "title": "Include Forecast", + }, + "language": { + "type": "string", + "default": "zh-CN", + "description": "Language of the response", + "title": "Language", + "enum": ["zh-CN", "en-US", "ja-JP"], + }, + }, + }, + }, + }, + }, + }, + { + "type": "function", + "function": { + "name": "get_forecast", + "description": "Get the weather forecast for a given location", + "parameters": { + "type": "object", + "properties": { + "city": { + "type": "string", + "description": "The city to get the forecast for, e.g. " + "'Vienna'", + "default": "Vienna", + }, + "country": { + "type": "string", + "description": "The country that the city is in, e.g. " + "'Austria'", + }, + "days": { + "type": "integer", + "description": "Number of days to get the forecast for (1-7)", + }, + "unit": { + "type": "string", + "description": "The unit to fetch the temperature in", + "enum": ["celsius", "fahrenheit"], + }, + }, + "required": ["country", "days", "unit"], + }, + }, + }, +] + +messages = [ + {"role": "user", "content": "Hi! How are you doing today?"}, + {"role": "assistant", "content": "I'm doing well! How can I help you?"}, + { + "role": "user", + "content": "Can you tell me what the current weather is in Berlin and the " + "forecast for the next 5 days, in fahrenheit?", + }, +] + + +@pytest.fixture(scope="module") +def server(): + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "half", + "--enable-auto-tool-choice", + "--structured-outputs-config.backend", + "xgrammar", + "--tool-call-parser", + "hermes", + "--reasoning-parser", + "qwen3", + "--gpu-memory-utilization", + "0.4", + "--enforce-eager", + ] + ROCM_EXTRA_ARGS + + with RemoteOpenAIServer( + MODEL_NAME, args, env_dict=ROCM_ENV_OVERRIDES + ) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("stream", [True, False]) +@pytest.mark.parametrize( + "tool_choice", + [ + "auto", + "required", + {"type": "function", "function": {"name": "get_current_weather"}}, + ], +) +@pytest.mark.parametrize("enable_thinking", [True, False]) +async def test_function_tool_use( + client: openai.AsyncOpenAI, + model_name: str, + stream: bool, + tool_choice: str | dict, + enable_thinking: bool, +): + if not stream: + # Non-streaming test + chat_completion = await client.chat.completions.create( + messages=messages, + model=model_name, + tools=tools, + tool_choice=tool_choice, + extra_body={"chat_template_kwargs": {"enable_thinking": enable_thinking}}, + ) + if enable_thinking: + assert chat_completion.choices[0].message.reasoning is not None + assert chat_completion.choices[0].message.reasoning != "" + assert chat_completion.choices[0].message.tool_calls is not None + assert len(chat_completion.choices[0].message.tool_calls) > 0 + else: + # Streaming test + output_stream = await client.chat.completions.create( + messages=messages, + model=model_name, + tools=tools, + tool_choice=tool_choice, + stream=True, + extra_body={"chat_template_kwargs": {"enable_thinking": enable_thinking}}, + ) + + output = [] + reasoning = [] + async for chunk in output_stream: + if chunk.choices: + if enable_thinking and getattr( + chunk.choices[0].delta, "reasoning", None + ): + reasoning.append(chunk.choices[0].delta.reasoning) + if chunk.choices[0].delta.tool_calls: + output.extend(chunk.choices[0].delta.tool_calls) + + assert len(output) > 0 + if enable_thinking: + assert len(reasoning) > 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("arguments", ["{}", ""]) +async def test_no_args_tool_call( + client: openai.AsyncOpenAI, model_name: str, arguments: str +): + # Step 1: Define a tool that requires no parameters + tools = [ + { + "type": "function", + "function": { + "name": "get_current_time", + "description": ( + "Get the current date and time. Call this when the user " + "asks what time or date it is. No parameters needed." + ), + "parameters": { + "type": "object", + "properties": {}, # No parameters + "required": [], # No required fields + }, + }, + } + ] + messages = [ + { + "role": "system", + "content": ( + "You are a helpful assistant. Always use the available tools " + "when relevant, and reply with a short sentence after " + "receiving a tool result." + ), + }, + {"role": "user", "content": "What time is it now?"}, + ] + + shared_kwargs = dict( + model=model_name, + temperature=0.0, + seed=42, + extra_body={"chat_template_kwargs": {"enable_thinking": False}}, + ) + + # Step 2: Send user message and let model decide whether to call the tool + response = await client.chat.completions.create( + **shared_kwargs, + messages=messages, + tools=tools, + tool_choice="auto", # Let model choose automatically + ) + + # Step 3: Check if model wants to call a tool + message = response.choices[0].message + if message.tool_calls: + # Get the first tool call + tool_call = message.tool_calls[0] + tool_name = tool_call.function.name + # Step 4: Execute the tool locally (no parameters) + if tool_name == "get_current_time": + # Test both empty string and "{}" for no-arg tool calls + tool_call.function.arguments = arguments + messages.append(message) + current_time = datetime.datetime.now() + result = current_time.isoformat() + messages.append( + { + "role": "tool", + "tool_call_id": tool_call.id, + "content": result, + } + ) + # Step 5: Send tool result back to model to continue conversation + final_response = await client.chat.completions.create( + **shared_kwargs, + messages=messages, + max_completion_tokens=128, + ) + # Output final natural language response + assert ( + final_response.choices[0].message.content is not None + and final_response.choices[0].message.content.strip() != "" + ) + + else: + # No tool called — just print model's direct reply + assert message.content is not None + + +@pytest.mark.asyncio +async def test_named_tool_use( + client: openai.AsyncOpenAI, + sample_json_schema, +): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + { + "role": "user", + "content": ( + "Give an example JSON for an employee profile using the specified tool." + ), + }, + ] + tools = [ + { + "type": "function", + "function": { + "name": "dummy_function_name", + "description": "This is a dummy function", + "parameters": sample_json_schema, + }, + } + ] + tool_choice = {"type": "function", "function": {"name": "dummy_function_name"}} + + # non-streaming + + chat_completion = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_completion_tokens=1000, + tools=tools, + temperature=0.0, + tool_choice=tool_choice, + ) + message = chat_completion.choices[0].message + assert len(message.content) == 0 + json_string = message.tool_calls[0].function.arguments + json1 = json.loads(json_string) + jsonschema.validate(instance=json1, schema=sample_json_schema) + + messages.append({"role": "assistant", "content": json_string}) + messages.append( + {"role": "user", "content": "Give me another one with a different name and age"} + ) + + # streaming + + stream = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_completion_tokens=1000, + tools=tools, + tool_choice=tool_choice, + temperature=0.0, + stream=True, + ) + + output = [] + finish_reason_count = 0 + async for chunk in stream: + delta = chunk.choices[0].delta + if delta.role: + assert delta.role == "assistant" + assert delta.content is None or len(delta.content) == 0 + if delta.tool_calls and delta.tool_calls[0].function.arguments: + output.append(delta.tool_calls[0].function.arguments) + if chunk.choices[0].finish_reason is not None: + finish_reason_count += 1 + # finish reason should only return in last block + assert finish_reason_count == 1 + json2 = json.loads("".join(output)) + jsonschema.validate(instance=json2, schema=sample_json_schema) + assert json1["name"] != json2["name"] + assert json1["age"] != json2["age"] + + +@pytest.mark.asyncio +async def test_inconsistent_tool_choice_and_tools( + client: openai.AsyncOpenAI, sample_json_schema +): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + { + "role": "user", + "content": f"Give an example JSON for an employee profile that " + f"fits this schema: {sample_json_schema}", + }, + ] + + with pytest.raises(openai.BadRequestError): + await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_completion_tokens=1000, + tool_choice={ + "type": "function", + "function": {"name": "dummy_function_name"}, + }, + ) + + with pytest.raises(openai.BadRequestError): + await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_completion_tokens=1000, + tools=[ + { + "type": "function", + "function": { + "name": "dummy_function_name", + "description": "This is a dummy function", + "parameters": sample_json_schema, + }, + } + ], + tool_choice={ + "type": "function", + "function": {"name": "nondefined_function_name"}, + }, + ) + with pytest.raises(openai.BadRequestError): + await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_completion_tokens=1000, + tools=[ + { + "type": "function", + "function": { + "name": "dummy_function_name", + "description": "This is a dummy function", + "parameters": sample_json_schema, + }, + } + ], + tool_choice={}, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "tool_choice", + ["required", {"type": "function", "function": {"name": "get_current_weather"}}], +) +async def test_max_tokens_with_tool_choice_required( + client: openai.AsyncOpenAI, tool_choice +): + """ """ + models = await client.models.list() + model_name: str = models.data[0].id + + # This combination previously crashed the engine + chat_completion = await client.chat.completions.create( + messages=messages, + temperature=0, + max_completion_tokens=1, + model=model_name, + tools=tools, + tool_choice=tool_choice, + ) + # When `tool_choice="required"` and the tokens of `tools` exceed `max_tokens`, + # `tool_calls` should be absent and `content` should be empty. + # This behavior should be consistent with OpenAI. + choice = chat_completion.choices[0] + assert choice.finish_reason == "length" + assert choice.message.tool_calls is None diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_enable_force_include_usage.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_enable_force_include_usage.py new file mode 100644 index 0000000000000000000000000000000000000000..1b169de6f2062a236fab92b0159236d9a4c2efdb --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_enable_force_include_usage.py @@ -0,0 +1,69 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import openai +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer + + +@pytest.fixture(scope="module") +def chat_server_with_force_include_usage(request): + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-model-len", + "128", + "--enforce-eager", + "--max-num-seqs", + "4", + "--enable-force-include-usage", + "--port", + "55857", + "--gpu-memory-utilization", + "0.2", + ] + + with RemoteOpenAIServer("Qwen/Qwen3-0.6B", args, auto_port=False) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def chat_client_with_force_include_usage(chat_server_with_force_include_usage): + async with chat_server_with_force_include_usage.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +async def test_chat_with_enable_force_include_usage( + chat_client_with_force_include_usage: openai.AsyncOpenAI, +): + messages = [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "What is the capital of France?"}, + ] + + stream = await chat_client_with_force_include_usage.chat.completions.create( + model="Qwen/Qwen3-0.6B", + messages=messages, + max_completion_tokens=10, + extra_body=dict(min_tokens=10), + temperature=0.0, + stream=True, + ) + last_completion_tokens = 0 + async for chunk in stream: + assert chunk.usage.prompt_tokens >= 0 + assert ( + last_completion_tokens == 0 + or chunk.usage.completion_tokens > last_completion_tokens + or ( + not chunk.choices + and chunk.usage.completion_tokens == last_completion_tokens + ) + ) + assert chunk.usage.total_tokens == ( + chunk.usage.prompt_tokens + chunk.usage.completion_tokens + ) + last_completion_tokens = chunk.usage.completion_tokens diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_root_path.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_root_path.py new file mode 100644 index 0000000000000000000000000000000000000000..215cdac3b41cab775ed78517beaef601aea333c5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_root_path.py @@ -0,0 +1,104 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import contextlib +import os +from typing import Any, NamedTuple + +import openai # use the official client for correctness check +import pytest + +from tests.utils import RemoteOpenAIServer + +# # any model with a chat template should work here +MODEL_NAME = "Qwen/Qwen2-1.5B-Instruct" +API_KEY = "abc-123" +ERROR_API_KEY = "abc" +ROOT_PATH = "llm" + + +@pytest.fixture(scope="module") +def server(): + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "float16", + "--enforce-eager", + "--max-model-len", + "4080", + "--root-path", # use --root-path=/llm for testing + "/" + ROOT_PATH, + ] + envs = os.environ.copy() + + envs["VLLM_API_KEY"] = API_KEY + with RemoteOpenAIServer(MODEL_NAME, args, env_dict=envs) as remote_server: + yield remote_server + + +class TestCase(NamedTuple): + model_name: str + base_url: list[str] + api_key: str + expected_error: Any + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "test_case", + [ + TestCase( + model_name=MODEL_NAME, + base_url=["v1"], # http://localhost:8000/v1 + api_key=ERROR_API_KEY, + expected_error=openai.AuthenticationError, + ), + TestCase( + model_name=MODEL_NAME, + base_url=[ROOT_PATH, "v1"], # http://localhost:8000/llm/v1 + api_key=ERROR_API_KEY, + expected_error=openai.AuthenticationError, + ), + TestCase( + model_name=MODEL_NAME, + base_url=["v1"], # http://localhost:8000/v1 + api_key=API_KEY, + expected_error=None, + ), + TestCase( + model_name=MODEL_NAME, + base_url=[ROOT_PATH, "v1"], # http://localhost:8000/llm/v1 + api_key=API_KEY, + expected_error=None, + ), + ], +) +async def test_chat_session_root_path_with_api_key( + server: RemoteOpenAIServer, test_case: TestCase +): + saying: str = "Here is a common saying about apple. An apple a day, keeps" + ctx = contextlib.nullcontext() + if test_case.expected_error is not None: + ctx = pytest.raises(test_case.expected_error) + with ctx: + client = openai.AsyncOpenAI( + api_key=test_case.api_key, + base_url=server.url_for(*test_case.base_url), + max_retries=0, + ) + chat_completion = await client.chat.completions.create( + model=test_case.model_name, + messages=[ + {"role": "user", "content": "tell me a common saying"}, + {"role": "assistant", "content": saying}, + ], + extra_body={"continue_final_message": True, "add_generation_prompt": False}, + ) + + assert chat_completion.id is not None + assert len(chat_completion.choices) == 1 + choice = chat_completion.choices[0] + assert choice.finish_reason == "stop" + message = choice.message + assert len(message.content) > 0 + assert message.role == "assistant" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_serving_chat.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_serving_chat.py new file mode 100644 index 0000000000000000000000000000000000000000..041bf0638ec8769d29e96d89f83ed79b2c657a52 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_serving_chat.py @@ -0,0 +1,2129 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import asyncio +import json +from contextlib import suppress +from dataclasses import dataclass, field +from typing import Any +from unittest.mock import AsyncMock, MagicMock + +import pytest +import pytest_asyncio +from openai import OpenAI + +from tests.entrypoints.openai.utils import ( + accumulate_streaming_response, + verify_chat_response, + verify_harmony_messages, +) +from tests.utils import RemoteOpenAIServer +from vllm._aiter_ops import is_aiter_found_and_supported +from vllm.config import MultiModalConfig +from vllm.entrypoints.openai.chat_completion.protocol import ( + ChatCompletionRequest, + ChatCompletionResponse, +) +from vllm.entrypoints.openai.chat_completion.serving import ( + OpenAIServingChat, + _get_mm_token_counts, + _make_prompt_tokens_details, +) +from vllm.entrypoints.openai.engine.protocol import ( + ErrorResponse, + RequestResponseMetadata, +) +from vllm.entrypoints.openai.models.serving import ( + BaseModelPath, + OpenAIModelRegistry, + OpenAIServingModels, +) +from vllm.entrypoints.openai.parser.harmony_utils import get_encoding +from vllm.exceptions import VLLMValidationError +from vllm.inputs import TokensPrompt +from vllm.multimodal.inputs import PlaceholderRange +from vllm.outputs import CompletionOutput, RequestOutput +from vllm.parser import HarmonyParser +from vllm.renderers.hf import HfRenderer +from vllm.renderers.mistral import MistralRenderer +from vllm.renderers.online_renderer import OnlineRenderer +from vllm.tokenizers import get_tokenizer +from vllm.tokenizers.mistral import MistralTokenizer +from vllm.tokenizers.registry import cached_tokenizer_from_config +from vllm.v1.engine.async_llm import AsyncLLM + +GPT_OSS_MODEL_NAME = "openai/gpt-oss-20b" +GPT_OSS_SPECULATOR_NAME = "RedHatAI/gpt-oss-20b-speculator.eagle3" + + +@pytest.fixture(scope="module") +def monkeypatch_module(): + from _pytest.monkeypatch import MonkeyPatch + + mpatch = MonkeyPatch() + yield mpatch + mpatch.undo() + + +@pytest.fixture( + scope="module", + params=[True, False], + ids=["with_tool_parser", "without_tool_parser"], +) +def with_tool_parser(request) -> bool: + return request.param + + +@pytest.fixture( + scope="module", + params=[True], + ids=["exclude_tools_when_tool_choice_none"], +) +def exclude_tools_when_tool_choice_none(request) -> bool: + return request.param + + +@pytest.fixture(scope="module") +def default_server_args( + with_tool_parser: bool, + exclude_tools_when_tool_choice_none: bool, +): + args = [ + # use half precision for speed and memory savings in CI environment + "--enforce-eager", + "--max-model-len", + "4096", + "--reasoning-parser", + "openai_gptoss", + "--gpu-memory-utilization", + "0.85", + ] + if with_tool_parser: + args.extend( + [ + "--tool-call-parser", + "openai", + "--enable-auto-tool-choice", + ] + ) + if exclude_tools_when_tool_choice_none: + args.append("--exclude-tools-when-tool-choice-none") + return args + + +@pytest.fixture(scope="class") +def gptoss_server(default_server_args: list[str]): + server_args = default_server_args + ["--attention-backend=TRITON_ATTN"] + with RemoteOpenAIServer(GPT_OSS_MODEL_NAME, server_args) as remote_server: + yield remote_server + + +@pytest.fixture(scope="class") +def gptoss_speculative_server(default_server_args: list[str]): + attention_backend = ( + "TRITON_ATTN" + if not is_aiter_found_and_supported() + else "ROCM_AITER_UNIFIED_ATTN" + ) + server_args = default_server_args + [ + "--speculative-config", + f'{{"model": "{GPT_OSS_SPECULATOR_NAME}", ' + f'"method": "eagle3", "num_speculative_tokens": 3}}', + f"--attention-backend={attention_backend}", + ] + # gpt-oss requires AITER unified attention on ROCm + # TODO: Remove after fixing TRITON_ATTN issue on ROCm + # https://github.com/vllm-project/vllm/issues/32434 + env_dict = None + if is_aiter_found_and_supported(): + env_dict = {"VLLM_ROCM_USE_AITER": "1"} + with RemoteOpenAIServer( + GPT_OSS_MODEL_NAME, server_args, env_dict=env_dict, max_wait_seconds=480 + ) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def gptoss_client(gptoss_server): + async with gptoss_server.get_async_client() as async_client: + yield async_client + + +@pytest_asyncio.fixture +async def gptoss_speculative_client(gptoss_speculative_server): + async with gptoss_speculative_server.get_async_client() as async_client: + yield async_client + + +class TestGPTOSSChat: + @pytest.mark.asyncio + async def test_gpt_oss_chat_tool_call_streaming( + self, gptoss_client: OpenAI, with_tool_parser: bool + ): + tools = [ + { + "type": "function", + "function": { + "name": "get_current_weather", + "description": "Get the current weather in a given location", + "parameters": { + "type": "object", + "properties": { + "city": {"type": "string"}, + "state": {"type": "string"}, + "unit": { + "type": "string", + "enum": ["celsius", "fahrenheit"], + }, + }, + "required": ["city", "state", "unit"], + }, + }, + } + ] + + messages = [ + {"role": "user", "content": "What is the weather in Dallas, TX?"}, + ] + + stream = await gptoss_client.chat.completions.create( + model=GPT_OSS_MODEL_NAME, + messages=messages, + tools=tools if with_tool_parser else None, + stream=True, + ) + + name = None + args_buf = "" + content_buf = "" + async for chunk in stream: + delta = chunk.choices[0].delta + if delta.tool_calls: + tc = delta.tool_calls[0] + if tc.function and tc.function.name: + name = tc.function.name + if tc.function and tc.function.arguments: + args_buf += tc.function.arguments + if getattr(delta, "content", None): + content_buf += delta.content + if with_tool_parser: + assert name is not None + assert len(args_buf) > 0 + else: + assert name is None + assert len(args_buf) == 0 + assert len(content_buf) > 0 + + @pytest.mark.asyncio + async def test_gpt_oss_multi_turn_chat( + self, gptoss_client: OpenAI, with_tool_parser: bool + ): + if not with_tool_parser: + pytest.skip("skip non-tool for multi-turn tests") + tools = [ + { + "type": "function", + "function": { + "name": "get_current_weather", + "description": "Get the current weather in a given location", + "parameters": { + "type": "object", + "properties": { + "city": {"type": "string"}, + "state": {"type": "string"}, + "unit": { + "type": "string", + "enum": ["celsius", "fahrenheit"], + }, + }, + "required": ["city", "state", "unit"], + }, + }, + } + ] + + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + { + "role": "user", + "content": "What is the weather in Dallas, TX with celsius?", + }, + ] + + first = await gptoss_client.chat.completions.create( + model=GPT_OSS_MODEL_NAME, + messages=messages, + tools=tools, + temperature=0.0, + ) + first_msg = first.choices[0].message + assert first_msg.tool_calls is not None and len(first_msg.tool_calls) > 0 + tc = first_msg.tool_calls[0] + assert tc.function is not None and tc.function.name == "get_current_weather" + args1 = tc.function.arguments + assert args1 is not None and len(args1) > 0 + assert not first_msg.content + + messages.append({"role": "assistant", "content": args1}) + messages.append( + {"role": "user", "content": "Now convert to celsius and return JSON only"} + ) + + second = await gptoss_client.chat.completions.create( + model=GPT_OSS_MODEL_NAME, + messages=messages, + tools=tools, + temperature=0.0, + ) + second_msg = second.choices[0].message + assert (second_msg.content is not None and len(second_msg.content) > 0) or ( + second_msg.tool_calls is not None and len(second_msg.tool_calls) > 0 + ) + + @pytest.mark.asyncio + async def test_gpt_oss_tool_message_array_content( + self, gptoss_client: OpenAI, with_tool_parser: bool + ): + """Test that tool messages support both string and array content formats.""" + if not with_tool_parser: + pytest.skip("skip non-tool for array content tests") + + tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather in a given location", + "parameters": { + "type": "object", + "properties": { + "city": {"type": "string"}, + "state": {"type": "string"}, + }, + "required": ["city", "state"], + }, + }, + } + ] + + # Test 1: Tool message with string content + messages_string = [ + {"role": "user", "content": "What's the weather in Paris?"}, + { + "role": "assistant", + "tool_calls": [ + { + "id": "call_123", + "type": "function", + "function": { + "name": "get_weather", + "arguments": '{"city": "Paris", "state": "TX"}', + }, + } + ], + }, + {"role": "tool", "content": "The weather in Paris, TX is sunny, 22°C"}, + ] + + response_string = await gptoss_client.chat.completions.create( + model=GPT_OSS_MODEL_NAME, + messages=messages_string, + tools=tools, + temperature=0.0, + ) + + assert response_string is not None + assert response_string.choices[0].message is not None + + # Test 2: Tool message with array content + messages_array = [ + {"role": "user", "content": "What's the weather in Dallas?"}, + { + "role": "assistant", + "tool_calls": [ + { + "id": "call_456", + "type": "function", + "function": { + "name": "get_weather", + "arguments": '{"city": "Dallas", "state": "TX"}', + }, + } + ], + }, + { + "role": "tool", + "content": [ + {"type": "text", "text": "f2e897a7-2705-4337-8193-2a8f57b81618"} + ], + }, + ] + + response_array = await gptoss_client.chat.completions.create( + model=GPT_OSS_MODEL_NAME, + messages=messages_array, + tools=tools, + temperature=0.0, + ) + + assert response_array is not None + assert response_array.choices[0].message is not None + + # Test 3: Tool message with multiple array content items + messages_multi_array = [ + {"role": "user", "content": "Search for information"}, + { + "role": "assistant", + "tool_calls": [ + { + "id": "call_789", + "type": "function", + "function": { + "name": "get_weather", + "arguments": '{"city": "Austin", "state": "TX"}', + }, + } + ], + }, + { + "role": "tool", + "content": [ + {"type": "text", "text": "Weather data: "}, + {"type": "text", "text": "Austin, TX - Partly cloudy, 25°C"}, + {"type": "text", "text": " with 60% humidity"}, + ], + }, + ] + + response_multi_array = await gptoss_client.chat.completions.create( + model=GPT_OSS_MODEL_NAME, + messages=messages_multi_array, + tools=tools, + temperature=0.0, + ) + + assert response_multi_array is not None + assert response_multi_array.choices[0].message is not None + + @pytest.mark.asyncio + async def test_gpt_oss_tool_choice_none( + self, + gptoss_client: OpenAI, + with_tool_parser: bool, + exclude_tools_when_tool_choice_none: bool, + ): + if not (with_tool_parser and exclude_tools_when_tool_choice_none): + pytest.skip( + "skip tool_choice tests when non-tool or " + "--exclude-tools-when-tool-choice-none not set" + ) + + tools = [ + { + "type": "function", + "function": { + "name": "get_current_weather", + "description": "Get the current weather in a given location", + "parameters": { + "type": "object", + "properties": { + "city": {"type": "string"}, + "state": {"type": "string"}, + "unit": { + "type": "string", + "enum": ["celsius", "fahrenheit"], + }, + }, + "required": ["city", "state", "unit"], + }, + }, + } + ] + + messages = [ + { + "role": "user", + "content": "What's the temperature(in degrees Celsius) in Dallas?", + }, + ] + + tool_choice_auto = await gptoss_client.chat.completions.create( + model=GPT_OSS_MODEL_NAME, + messages=messages, + tools=tools, + tool_choice="auto", + temperature=0.0, + ) + msg = tool_choice_auto.choices[0].message + assert len(msg.tool_calls) == 1 + + tool_choice_none = await gptoss_client.chat.completions.create( + model=GPT_OSS_MODEL_NAME, + messages=messages, + tools=tools, + tool_choice="none", + temperature=0.0, + ) + + msg = tool_choice_none.choices[0].message + assert msg.tool_calls is None + + +class TestGPTOSSSpeculativeChat: + @pytest.mark.asyncio + async def test_gpt_oss_speculative_reasoning_leakage( + self, + gptoss_speculative_client: OpenAI, + with_tool_parser: bool, + ): + if not with_tool_parser: + pytest.skip("skip non-tool for array content tests") + + messages = [ + {"role": "user", "content": "Calculate 2+2. Return the answer 4 only."}, + ] + + stream = await gptoss_speculative_client.chat.completions.create( + model=GPT_OSS_MODEL_NAME, + messages=messages, + stream=True, + temperature=0.0, + ) + + content = "" + reasoning = "" + async for chunk in stream: + delta = chunk.choices[0].delta + if delta.content: + content += delta.content + + chunk_reasoning = getattr(delta, "reasoning", None) + if chunk_reasoning: + reasoning += delta.reasoning + + assert len(reasoning) > 0, "No reasoning was generated." + assert content.strip() == "4" + + +MODEL_NAME = "openai-community/gpt2" +MODEL_NAME_SHORT = "gpt2" +CHAT_TEMPLATE = "Dummy chat template for testing {}" +BASE_MODEL_PATHS = [ + BaseModelPath(name=MODEL_NAME, model_path=MODEL_NAME), + BaseModelPath(name=MODEL_NAME_SHORT, model_path=MODEL_NAME_SHORT), +] + + +@dataclass +class MockHFConfig: + model_type: str = "any" + + +@dataclass +class MockModelConfig: + task = "generate" + runner_type = "generate" + model = MODEL_NAME + tokenizer = MODEL_NAME + trust_remote_code = False + tokenizer_mode = "auto" + max_model_len = 100 + tokenizer_revision = None + multimodal_config = MultiModalConfig() + hf_config = MockHFConfig() + hf_text_config = MockHFConfig() + logits_processors: list[str] | None = None + diff_sampling_param: dict | None = None + allowed_local_media_path: str = "" + allowed_media_domains: list[str] | None = None + encoder_config = None + generation_config: str = "auto" + override_generation_config: dict[str, Any] = field(default_factory=dict) + media_io_kwargs: dict[str, dict[str, Any]] = field(default_factory=dict) + skip_tokenizer_init: bool = False + is_encoder_decoder: bool = False + is_multimodal_model: bool = False + renderer_num_workers: int = 1 + enable_prompt_embeds: bool = False + + def get_diff_sampling_param(self): + return self.diff_sampling_param or {} + + +@dataclass +class MockParallelConfig: + _api_process_rank: int = 0 + + +@dataclass +class MockVllmConfig: + model_config: MockModelConfig + parallel_config: MockParallelConfig + + +def _build_renderer(model_config: MockModelConfig): + return HfRenderer( + MockVllmConfig(model_config, parallel_config=MockParallelConfig()), + cached_tokenizer_from_config(model_config), + ) + + +def _build_online_renderer( + engine, model_registry: OpenAIModelRegistry +) -> OnlineRenderer: + return OnlineRenderer( + model_config=engine.model_config, + renderer=engine.renderer, + request_logger=None, + chat_template=CHAT_TEMPLATE, + chat_template_content_format="auto", + ) + + +def _build_serving_chat( + engine: AsyncLLM, + *, + reasoning_parser: str = "", + tool_parser: str | None = None, + enable_auto_tools: bool = False, +) -> OpenAIServingChat: + models = OpenAIServingModels( + engine_client=engine, + base_model_paths=BASE_MODEL_PATHS, + ) + online_renderer = _build_online_renderer(engine, models.registry) + + serving_chat = OpenAIServingChat( + engine, + models, + response_role="assistant", + online_renderer=online_renderer, + chat_template=CHAT_TEMPLATE, + chat_template_content_format="auto", + request_logger=None, + reasoning_parser=reasoning_parser, + tool_parser=tool_parser, + enable_auto_tools=enable_auto_tools, + ) + + return serving_chat + + +@dataclass +class MockEngine: + model_config: MockModelConfig = field(default_factory=MockModelConfig) + input_processor: MagicMock = field(default_factory=MagicMock) + renderer: MagicMock = field(default_factory=MagicMock) + errored: bool = False + + +async def _async_serving_chat_init(): + engine = MockEngine() + + models = OpenAIServingModels(engine, BASE_MODEL_PATHS) + online_renderer = _build_online_renderer(engine, models.registry) + + serving_completion = OpenAIServingChat( + engine, + models, + response_role="assistant", + online_renderer=online_renderer, + chat_template=CHAT_TEMPLATE, + chat_template_content_format="auto", + request_logger=None, + ) + return serving_completion + + +def test_async_serving_chat_init(): + serving_completion = asyncio.run(_async_serving_chat_init()) + assert serving_completion.chat_template == CHAT_TEMPLATE + + +def test_mm_prompt_tokens_details(): + # Text-only input has no multimodal placeholders. + assert _get_mm_token_counts({"type": "tokens"}) == {} + + # Per-modality counts sum each modality's placeholder ranges. + counts = _get_mm_token_counts( + { + "mm_placeholders": { + "image": [ + PlaceholderRange(offset=0, length=576), + PlaceholderRange(offset=600, length=24), + ], + "video": [PlaceholderRange(offset=700, length=1200)], + } + } + ) + assert counts == {"image": 600, "video": 1200} + + # Gated off, or nothing to report -> no details. + assert _make_prompt_tokens_details(False, 5, counts) is None + assert _make_prompt_tokens_details(True, None, None) is None + + # Zero cached_tokens is still reported (not None), matching the cached-only + # behavior; multimodal counts ride alongside even when cached_tokens is None. + assert _make_prompt_tokens_details(True, 0, None).cached_tokens == 0 + details = _make_prompt_tokens_details(True, None, counts) + assert details.cached_tokens is None + assert details.multimodal_tokens == {"image": 600, "video": 1200} + assert _make_prompt_tokens_details(True, 3, counts).cached_tokens == 3 + + +@pytest.mark.asyncio +async def test_serving_chat_returns_correct_model_name(): + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + serving_chat = _build_serving_chat(mock_engine) + messages = [{"role": "user", "content": "what is 1+1?"}] + + async def return_model_name(*args, **kwargs): + return args[3] + + serving_chat.chat_completion_full_generator = return_model_name + + # Test that full name is returned when short name is requested + req = ChatCompletionRequest(model=MODEL_NAME_SHORT, messages=messages) + assert await serving_chat.create_chat_completion(req) == MODEL_NAME + + # Test that full name is returned when empty string is specified + req = ChatCompletionRequest(model="", messages=messages) + assert await serving_chat.create_chat_completion(req) == MODEL_NAME + + # Test that full name is returned when no model is specified + req = ChatCompletionRequest(messages=messages) + assert await serving_chat.create_chat_completion(req) == MODEL_NAME + + +@pytest.mark.asyncio +async def test_serving_chat_should_set_correct_max_tokens(): + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + serving_chat = _build_serving_chat(mock_engine) + + req = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "what is 1+1?"}], + ) + + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert mock_engine.generate.call_args.args[1].max_tokens == 93 + + req.max_tokens = 10 + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert mock_engine.generate.call_args.args[1].max_tokens == 10 + + # Model author's generation_config.json sets max_tokens (auto, no override) + # — should act as fallback only, not ceiling + mock_model_config = MockModelConfig() + mock_model_config.diff_sampling_param = {"max_tokens": 10} + + # Reinitialize the engine with new settings + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = mock_model_config + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + # Initialize the serving chat + serving_chat = _build_serving_chat(mock_engine) + + # Test Case 1: No max_tokens specified in request + req = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "what is 1+1?"}], + ) + + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert mock_engine.generate.call_args.args[1].max_tokens == 10 + + # Test Case 2: Request's max_tokens set higher than generation_config + # default so request-provided max_tokens takes precedence + req.max_tokens = 15 + + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert mock_engine.generate.call_args.args[1].max_tokens == 15 + + # Test Case 3: Request's max_tokens set lower than server accepts + req.max_tokens = 5 + + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert mock_engine.generate.call_args.args[1].max_tokens == 5 + + # User explicitly sets max_tokens via --override-generation-config + # — should act as a ceiling + mock_model_config = MockModelConfig() + mock_model_config.diff_sampling_param = {"max_tokens": 10} + mock_model_config.override_generation_config = {"max_new_tokens": 10} + + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = mock_model_config + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + serving_chat = _build_serving_chat(mock_engine) + + # Test Case 3.1: No max_tokens — uses override as default + req = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "what is 1+1?"}], + ) + + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert mock_engine.generate.call_args.args[1].max_tokens == 10 + + # Test Case 3.2: Request max_tokens higher — capped by user ceiling from override + req.max_tokens = 15 + + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert mock_engine.generate.call_args.args[1].max_tokens == 10 + + # Test Case 3.3: Request max_tokens lower — respected + req.max_tokens = 5 + + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert mock_engine.generate.call_args.args[1].max_tokens == 5 + + # Setting server's max_tokens in the generation_config.json + # higher than context_window - prompt_tokens + mock_model_config = MockModelConfig() + mock_model_config.diff_sampling_param = {"max_tokens": 200} + + # Reinitialize the engine with new settings + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = mock_model_config + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + # Initialize the serving chat + serving_chat = _build_serving_chat(mock_engine) + + # Test case 1: No max_tokens specified, defaults to context_window + req = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "what is 1+1?"}], + ) + + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert mock_engine.generate.call_args.args[1].max_tokens == 93 + + # Test Case 2: Request's max_tokens set higher than server accepts + req.max_tokens = 100 + + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert mock_engine.generate.call_args.args[1].max_tokens == 93 + + # Test Case 3: Request's max_tokens set lower than server accepts + req.max_tokens = 5 + + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert mock_engine.generate.call_args.args[1].max_tokens == 5 + + +@pytest.mark.asyncio +async def test_serving_chat_truncate_prompt_tokens_max_token_accounting(): + """When truncate_prompt_tokens is set, max_tokens must be calculated using + the truncated prompt length, not the original prompt length. + + Regression: without the fix, get_max_tokens received the untruncated prompt + length, causing the output budget to be underestimated. + """ + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + serving_chat = _build_serving_chat(mock_engine) + + # "what is 1+1?" tokenizes to 7 tokens with the test chat template + # (max_model_len=100 -> max_tokens = 93 without truncation, confirmed by + # test_serving_chat_should_set_correct_max_tokens above). + messages = [{"role": "user", "content": "what is 1+1?"}] + + # Baseline: no truncation -> max_tokens = 100 - 7 = 93. + req = ChatCompletionRequest(model=MODEL_NAME, messages=messages) + with suppress(Exception): + await serving_chat.create_chat_completion(req) + assert mock_engine.generate.call_args.args[1].max_tokens == 93 + + # With truncate_prompt_tokens=5 (less than 7): the effective prompt length + # is 5, so max_tokens should be 100 - 5 = 95, not 93. + req = ChatCompletionRequest( + model=MODEL_NAME, + messages=messages, + truncate_prompt_tokens=5, + ) + with suppress(Exception): + await serving_chat.create_chat_completion(req) + assert mock_engine.generate.call_args.args[1].max_tokens == 95 + + # With truncate_prompt_tokens=-1 (meaning use full max_model_len as the + # truncation limit, i.e., no practical truncation vs the window): effective + # length = min(7, 100) = 7 -> max_tokens = 93 again. + req = ChatCompletionRequest( + model=MODEL_NAME, + messages=messages, + truncate_prompt_tokens=-1, + ) + with suppress(Exception): + await serving_chat.create_chat_completion(req) + assert mock_engine.generate.call_args.args[1].max_tokens == 93 + + +@pytest.mark.asyncio +async def test_serving_chat_truncation_side_controls_prompt_truncation(): + model_config = MockModelConfig() + model_config.model = MODEL_NAME_SHORT + model_config.tokenizer = MODEL_NAME_SHORT + mock_engine = MockEngine( + model_config=model_config, + renderer=_build_renderer(model_config), + ) + + serving_chat = _build_serving_chat(mock_engine) + messages = [ + { + "role": "user", + "content": "Summarize how prompt truncation works in one sentence.", + } + ] + + full_token_ids = await _render_chat_prompt_token_ids( + serving_chat, + messages, + model_name=MODEL_NAME_SHORT, + ) + assert len(full_token_ids) > 4 + + right_token_ids = await _render_chat_prompt_token_ids( + serving_chat, + messages, + model_name=MODEL_NAME_SHORT, + truncate_prompt_tokens=4, + truncation_side="right", + ) + assert right_token_ids == full_token_ids[:4] + + left_token_ids = await _render_chat_prompt_token_ids( + serving_chat, + messages, + model_name=MODEL_NAME_SHORT, + truncate_prompt_tokens=4, + truncation_side="left", + ) + assert left_token_ids == full_token_ids[-4:] + + +@pytest.mark.asyncio +async def test_serving_chat_mistral_token_ids_prompt_is_validated(): + """Regression test: when the Mistral tokenizer path returns token IDs + directly, we must still apply input length + max_tokens validation. + """ + + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig(skip_tokenizer_init=True) + mock_engine.input_processor = MagicMock() + + mock_tokenizer = MagicMock(spec=MistralTokenizer) + mock_renderer = MistralRenderer( + MockVllmConfig(mock_engine.model_config, parallel_config=MockParallelConfig()), + tokenizer=mock_tokenizer, + ) + # Force the Mistral chat template renderer to return token IDs. + # Choose a prompt length that is < max_model_len, but large enough that + # adding max_tokens should exceed the model context window. + mock_renderer.render_messages_async = AsyncMock( + return_value=( + [], + TokensPrompt(prompt_token_ids=list(range(95))), + ) + ) + mock_engine.renderer = mock_renderer + + serving_chat = _build_serving_chat(mock_engine) + + req = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "what is 1+1?"}], + max_tokens=10, + ) + + with pytest.raises(VLLMValidationError): + await serving_chat.create_chat_completion(req) + + +@pytest.mark.asyncio +async def test_serving_chat_mistral_token_ids_prompt_too_long_is_rejected(): + """Regression test: MistralTokenizer token-id prompts must still enforce + the max context length for the input itself (token_num >= max_model_len). + """ + + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig(skip_tokenizer_init=True) + mock_engine.input_processor = MagicMock() + + mock_tokenizer = MagicMock(spec=MistralTokenizer) + mock_renderer = MistralRenderer( + MockVllmConfig(mock_engine.model_config, parallel_config=MockParallelConfig()), + tokenizer=mock_tokenizer, + ) + # prompt_token_ids length == max_model_len should be rejected for + # completion-like requests (ChatCompletionRequest). + mock_renderer.render_messages_async = AsyncMock( + return_value=( + [], + TokensPrompt( + prompt_token_ids=list(range(mock_engine.model_config.max_model_len)) + ), + ) + ) + mock_engine.renderer = mock_renderer + + serving_chat = _build_serving_chat(mock_engine) + + req = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "what is 1+1?"}], + max_tokens=1, + ) + + with pytest.raises(VLLMValidationError): + await serving_chat.create_chat_completion(req) + + +@pytest.mark.asyncio +async def test_serving_chat_could_load_correct_generation_config(): + mock_model_config = MockModelConfig() + mock_model_config.diff_sampling_param = { + "temperature": 0.5, + "repetition_penalty": 1.05, + } + + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = mock_model_config + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + # Initialize the serving chat + serving_chat = _build_serving_chat(mock_engine) + + req = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "what is 1+1?"}], + ) + + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert mock_engine.generate.call_args.args[1].temperature == 0.5 + assert mock_engine.generate.call_args.args[1].repetition_penalty == 1.05 + + # Test the param when user set it + req.temperature = 0.1 + + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert mock_engine.generate.call_args.args[1].temperature == 0.1 + assert mock_engine.generate.call_args.args[1].repetition_penalty == 1.05 + + # Test When temperature==0.0 + req.temperature = 0.0 + + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert mock_engine.generate.call_args.args[1].temperature == 0.0 + assert mock_engine.generate.call_args.args[1].repetition_penalty == 1.05 + + +@pytest.mark.parametrize("model_type", ["gpt_oss", "any"]) +@pytest.mark.asyncio +async def test_serving_chat_did_set_correct_cache_salt(model_type): + mock_model_config = MockModelConfig() + mock_model_config.hf_config.model_type = model_type + + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = mock_model_config + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + serving_chat = _build_serving_chat(mock_engine) + + orig_render_chat_request = serving_chat.render_chat_request + captured_inputs = [] + + async def render_chat_request(request): + result = await orig_render_chat_request(request) + + assert isinstance(result, tuple) + conversation, engine_inputs = result + captured_inputs.extend(engine_inputs) + + return result + + serving_chat.render_chat_request = render_chat_request + + # Test cache_salt + req = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "what is 1+1?"}], + ) + + # By default, cache_salt in the engine prompt is not set + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert len(captured_inputs) == 1 + assert "cache_salt" not in captured_inputs[0] + + captured_inputs.clear() + + # Test with certain cache_salt + req.cache_salt = "test_salt" + with suppress(Exception): + await serving_chat.create_chat_completion(req) + + assert len(captured_inputs) == 1 + assert captured_inputs[0]["cache_salt"] == "test_salt" + + +@pytest.mark.asyncio +async def test_serving_chat_data_parallel_rank_extraction(): + """Test that data_parallel_rank is properly extracted from header and + passed to engine.""" + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + # Mock the generate method to return an async generator + async def mock_generate(*args, **kwargs): + # Yield a fake RequestOutput + from vllm.outputs import CompletionOutput, RequestOutput + + yield RequestOutput( + request_id="test-request", + prompt="test prompt", + prompt_token_ids=[1, 2, 3], + prompt_logprobs=None, + outputs=[ + CompletionOutput( + index=0, + text="test response", + token_ids=[4, 5, 6], + cumulative_logprob=0.0, + logprobs=None, + finish_reason="stop", + stop_reason=None, + ) + ], + finished=True, + ) + + mock_engine.generate = AsyncMock(side_effect=mock_generate) + + serving_chat = _build_serving_chat(mock_engine) + + # Test when data_parallel_rank is present in header + req = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "what is 1+1?"}], + ) + + # Mock request with X-data-parallel-rank header + mock_raw_request = MagicMock() + mock_raw_request.headers = {"X-data-parallel-rank": "2"} + mock_raw_request.state = MagicMock() + + with suppress(Exception): + await serving_chat.create_chat_completion(req, mock_raw_request) + + # Verify that data_parallel_rank was passed to engine.generate + assert "data_parallel_rank" in mock_engine.generate.call_args.kwargs + assert mock_engine.generate.call_args.kwargs["data_parallel_rank"] == 2 + + # Test when data_parallel_rank is not present (defaults to None) + req_no_dp = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "what is 2+2?"}], + ) + + # Mock request with no header + mock_raw_request_no_dp = MagicMock() + mock_raw_request_no_dp.headers = {} + mock_raw_request_no_dp.state = MagicMock() + + with suppress(Exception): + await serving_chat.create_chat_completion(req_no_dp, mock_raw_request_no_dp) + + # Verify that data_parallel_rank defaults to None + assert "data_parallel_rank" in mock_engine.generate.call_args.kwargs + assert mock_engine.generate.call_args.kwargs["data_parallel_rank"] is None + + +async def _render_chat_prompt_token_ids( + serving_chat: OpenAIServingChat, + messages: list[dict[str, str]], + *, + model_name: str = MODEL_NAME, + truncate_prompt_tokens: int | None = None, + truncation_side: str | None = None, +) -> list[int]: + request = ChatCompletionRequest( + model=model_name, + messages=messages, + max_tokens=1, + temperature=0.0, + return_token_ids=True, + truncate_prompt_tokens=truncate_prompt_tokens, + truncation_side=truncation_side, + ) + + result = await serving_chat.render_chat_request(request) + assert not isinstance(result, ErrorResponse) + + _, engine_inputs = result + assert len(engine_inputs) == 1 + + prompt_token_ids = serving_chat._extract_prompt_components( + engine_inputs[0] + ).token_ids + assert prompt_token_ids is not None + return prompt_token_ids + + +class TestServingChatWithHarmony: + """ + These tests ensure Chat Completion requests are being properly converted into + Harmony messages and Harmony response messages back into Chat Completion responses. + These tests are not exhaustive, but each one was created to cover a specific case + that we got wrong but is now fixed. + + Any changes to the tests and their expectations may result in changes to the + accuracy of model prompting and responses generated. It is suggested to run + an evaluation or benchmarking suite (such as bfcl multi_turn) to understand + any impact of changes in how we prompt Harmony models. + """ + + @pytest.fixture(params=[False, True], ids=["non_streaming", "streaming"]) + def stream(self, request) -> bool: + """Parameterize tests to run in both non-streaming and streaming modes.""" + return request.param + + @pytest.fixture() + def mock_engine(self) -> AsyncLLM: + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.model_config.hf_config = MockHFConfig(model_type="gpt_oss") + mock_engine.model_config.hf_text_config = MockHFConfig(model_type="gpt_oss") + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + return mock_engine + + @pytest.fixture() + def serving_chat(self, mock_engine) -> OpenAIServingChat: + chat = _build_serving_chat( + mock_engine, + reasoning_parser="openai_gptoss", + tool_parser="openai", + enable_auto_tools=True, + ) + assert chat.parser_cls is HarmonyParser + return chat + + def mock_request_output_from_req_and_token_ids( + self, req: ChatCompletionRequest, token_ids: list[int], finished: bool = False + ) -> RequestOutput: + # Our tests don't use most fields, so just get the token ids correct + completion_output = CompletionOutput( + index=0, + text="", + token_ids=token_ids, + cumulative_logprob=0.0, + logprobs=None, + ) + return RequestOutput( + request_id=req.request_id, + prompt=[], + prompt_token_ids=[], + prompt_logprobs=None, + outputs=[completion_output], + finished=finished, + ) + + @pytest.fixture + def weather_tools(self) -> list[dict[str, Any]]: + return [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the weather in a given location", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"}, + }, + "required": ["location"], + }, + }, + }, + ] + + @pytest.fixture + def weather_messages_start(self) -> list[dict[str, Any]]: + return [ + { + "role": "user", + "content": "What's the weather like in Paris today?", + }, + ] + + async def generate_response_from_harmony_str( + self, + serving_chat: OpenAIServingChat, + req: ChatCompletionRequest, + harmony_str: str, + stream: bool = False, + ) -> ChatCompletionResponse: + harmony_token_ids = get_encoding().encode(harmony_str, allowed_special="all") + tokenizer = get_tokenizer(GPT_OSS_MODEL_NAME) + + async def result_generator(): + if stream: + for token_id in harmony_token_ids: + yield self.mock_request_output_from_req_and_token_ids( + req, [token_id] + ) + yield self.mock_request_output_from_req_and_token_ids( + req, [], finished=True + ) + else: + yield self.mock_request_output_from_req_and_token_ids( + req, harmony_token_ids, finished=True + ) + + generator_func = ( + serving_chat.chat_completion_stream_generator + if stream + else serving_chat.chat_completion_full_generator + ) + + chat_template_kwargs = serving_chat._effective_chat_template_kwargs(req) + if stream: + extra_kwargs: dict[str, Any] = { + "chat_template_kwargs": chat_template_kwargs, + } + else: + parser = None + if serving_chat.parser_cls is not None: + parser = serving_chat.parser_cls( + tokenizer, + req.tools, + chat_template_kwargs=chat_template_kwargs, + ) + extra_kwargs = {"parser": parser} + + result = generator_func( + request=req, + result_generator=result_generator(), + request_id=req.request_id, + model_name=req.model, + conversation=[], + tokenizer=tokenizer, + request_metadata=RequestResponseMetadata( + request_id=req.request_id, + model_name=req.model, + ), + **extra_kwargs, + ) + + if stream: + return await accumulate_streaming_response(result) + return await result + + @pytest.mark.asyncio + @pytest.mark.parametrize( + "include_reasoning", [True, False], ids=["with_reasoning", "no_reasoning"] + ) + async def test_simple_chat(self, serving_chat, stream, include_reasoning): + messages = [{"role": "user", "content": "what is 1+1?"}] + + # Test the Harmony messages for the first turn's input + req = ChatCompletionRequest( + model=MODEL_NAME, + messages=messages, + include_reasoning=include_reasoning, + ) + input_messages, _ = serving_chat.online_renderer._make_request_with_harmony(req) + verify_harmony_messages( + input_messages, + [ + {"role": "system"}, + {"role": "user", "content": messages[0]["content"]}, + ], + ) + + # Test the Chat Completion response for the first turn's output + reasoning_str = "We need to think really hard about this." + final_str = "The answer is 2." + response_str = ( + f"<|channel|>analysis<|message|>{reasoning_str}<|end|>" + f"<|start|>assistant<|channel|>final<|message|>{final_str}<|end|>" + ) + response = await self.generate_response_from_harmony_str( + serving_chat, req, response_str, stream=stream + ) + verify_chat_response( + response, + content=final_str, + reasoning=reasoning_str if include_reasoning else None, + ) + + # Add the output messages from the first turn as input to the second turn + for choice in response.choices: + messages.append(choice.message.model_dump(exclude_none=True)) + + # Test the Harmony messages for the second turn's input + req_2 = ChatCompletionRequest(model=MODEL_NAME, messages=messages) + input_messages_2, _ = serving_chat.online_renderer._make_request_with_harmony( + req_2 + ) + expected_input_messages_2 = [ + {"role": "system"}, + {"role": "user"}, + ] + if include_reasoning: + expected_input_messages_2.append( + { + "role": "assistant", + "channel": "analysis", + } + ) + expected_input_messages_2.append( + {"role": "assistant", "channel": "final", "content": final_str} + ) + verify_harmony_messages( + input_messages_2, + expected_input_messages_2, + ) + + @pytest.mark.asyncio + async def test_system_message_without_tools(self, serving_chat, stream): + """Leading system message produces a developer message with + DeveloperContent (# Instructions header).""" + messages = [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Hello"}, + ] + req = ChatCompletionRequest(model=MODEL_NAME, messages=messages) + input_messages, _ = serving_chat.online_renderer._make_request_with_harmony(req) + verify_harmony_messages( + input_messages, + [ + {"role": "system"}, + { + "role": "developer", + "instructions": "You are a helpful assistant.", + }, + {"role": "user", "content": "Hello"}, + ], + ) + + @pytest.mark.asyncio + async def test_system_message_with_tools(self, serving_chat, stream, weather_tools): + """Leading system message is folded into the developer message + alongside tool definitions.""" + messages = [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "What's the weather?"}, + ] + req = ChatCompletionRequest( + model=MODEL_NAME, messages=messages, tools=weather_tools + ) + input_messages, _ = serving_chat.online_renderer._make_request_with_harmony(req) + verify_harmony_messages( + input_messages, + [ + {"role": "system"}, + { + "role": "developer", + "instructions": "You are a helpful assistant.", + "tool_definitions": ["get_weather"], + }, + {"role": "user", "content": "What's the weather?"}, + ], + ) + + @pytest.mark.asyncio + async def test_tool_call_response_with_content( + self, serving_chat, stream, weather_tools, weather_messages_start + ): + tools = weather_tools + messages = list(weather_messages_start) + + # Test the Harmony messages for the first turn's input + req = ChatCompletionRequest(model=MODEL_NAME, messages=messages, tools=tools) + input_messages, _ = serving_chat.online_renderer._make_request_with_harmony(req) + verify_harmony_messages( + input_messages, + [ + {"role": "system"}, + {"role": "developer", "tool_definitions": ["get_weather"]}, + {"role": "user", "content": messages[0]["content"]}, + ], + ) + + # Test the Chat Completion response for the first turn's output + commentary_str = "We'll call get_weather." + tool_args_str = '{"location": "Paris"}' + response_str = ( + f"<|channel|>commentary<|message|>{commentary_str}<|end|>" + "<|start|>assistant to=functions.get_weather<|channel|>commentary" + f"<|constrain|>json<|message|>{tool_args_str}<|call|>" + ) + response = await self.generate_response_from_harmony_str( + serving_chat, req, response_str, stream=stream + ) + verify_chat_response( + response, + content=commentary_str, + tool_calls=[("get_weather", tool_args_str)], + ) + + tool_call = response.choices[0].message.tool_calls[0] + + # Add the output messages from the first turn as input to the second turn + for choice in response.choices: + messages.append(choice.message.model_dump(exclude_none=True)) + + # Add our tool output message + messages.append( + { + "role": "tool", + "tool_call_id": tool_call.id, + "content": "20 degrees Celsius", + }, + ) + + # Test the Harmony messages for the second turn's input + req_2 = ChatCompletionRequest(model=MODEL_NAME, messages=messages, tools=tools) + input_messages_2, _ = serving_chat.online_renderer._make_request_with_harmony( + req_2 + ) + verify_harmony_messages( + input_messages_2, + [ + {"role": "system"}, + {"role": "developer"}, + {"role": "user"}, + { + "role": "assistant", + "channel": "commentary", + "content": commentary_str, + }, + { + "role": "assistant", + "channel": "commentary", + "recipient": "functions.get_weather", + "content": tool_args_str, + }, + { + "role": "tool", + "author_name": "functions.get_weather", + "channel": "commentary", + "recipient": "assistant", + "content": "20 degrees Celsius", + }, + ], + ) + + @pytest.mark.asyncio + async def test_multi_turn_tools_and_reasoning( + self, serving_chat, stream, weather_tools, weather_messages_start + ): + tools = weather_tools + messages = list(weather_messages_start) + + # Test the Harmony messages for the first turn's input + req = ChatCompletionRequest(model=MODEL_NAME, messages=messages, tools=tools) + input_messages, _ = serving_chat.online_renderer._make_request_with_harmony(req) + verify_harmony_messages( + input_messages, + [ + {"role": "system"}, + {"role": "developer", "tool_definitions": ["get_weather"]}, + {"role": "user", "content": messages[0]["content"]}, + ], + ) + + # Test the Chat Completion response for the first turn's output + reasoning_str = "I'll call get_weather." + paris_tool_args_str = '{"location": "Paris"}' + response_str = ( + f"<|channel|>analysis<|message|>{reasoning_str}<|end|>" + "<|start|>assistant to=functions.get_weather<|channel|>commentary" + f"<|constrain|>json<|message|>{paris_tool_args_str}<|call|>" + ) + response = await self.generate_response_from_harmony_str( + serving_chat, req, response_str, stream=stream + ) + verify_chat_response( + response, + reasoning=reasoning_str, + tool_calls=[("get_weather", paris_tool_args_str)], + ) + + tool_call = response.choices[0].message.tool_calls[0] + + # Add the output messages from the first turn as input to the second turn + for choice in response.choices: + messages.append(choice.message.model_dump(exclude_none=True)) + + # Add our tool output message + messages.append( + { + "role": "tool", + "tool_call_id": tool_call.id, + "content": "20 degrees Celsius", + }, + ) + + # Test the Harmony messages for the second turn's input + req_2 = ChatCompletionRequest(model=MODEL_NAME, messages=messages, tools=tools) + input_messages_2, _ = serving_chat.online_renderer._make_request_with_harmony( + req_2 + ) + verify_harmony_messages( + input_messages_2, + [ + {"role": "system"}, + {"role": "developer"}, + {"role": "user"}, + { + "role": "assistant", + "channel": "analysis", + }, + { + "role": "assistant", + "channel": "commentary", + "recipient": "functions.get_weather", + "content": paris_tool_args_str, + }, + { + "role": "tool", + "author_name": "functions.get_weather", + "channel": "commentary", + "recipient": "assistant", + "content": "20 degrees Celsius", + }, + ], + ) + + # Test the Chat Completion response for the second turn's output + paris_weather_str = "The weather in Paris today is 20 degrees Celsius." + response_str = f"<|channel|>final<|message|>{paris_weather_str}<|end|>" + response_2 = await self.generate_response_from_harmony_str( + serving_chat, req_2, response_str, stream=stream + ) + verify_chat_response(response_2, content=paris_weather_str) + + # Add the output messages from the second turn as input to the third turn + for choice in response_2.choices: + messages.append(choice.message.model_dump(exclude_none=True)) + + # Add a new user message for the third turn + messages.append( + { + "role": "user", + "content": "What's the weather like in Boston today?", + }, + ) + + # Test the Harmony messages for the third turn's input + req_3 = ChatCompletionRequest(model=MODEL_NAME, messages=messages, tools=tools) + input_messages_3, _ = serving_chat.online_renderer._make_request_with_harmony( + req_3 + ) + verify_harmony_messages( + input_messages_3, + [ + {"role": "system"}, + {"role": "developer"}, + {"role": "user"}, + { + "role": "assistant", + "channel": "analysis", + "content": reasoning_str, + }, + { + "role": "assistant", + "channel": "commentary", + "recipient": "functions.get_weather", + "content": paris_tool_args_str, + }, + { + "role": "tool", + "author_name": "functions.get_weather", + "channel": "commentary", + "recipient": "assistant", + "content": "20 degrees Celsius", + }, + { + "role": "assistant", + "channel": "final", + "content": paris_weather_str, + }, + {"role": "user", "content": messages[-1]["content"]}, + ], + ) + + # Test the Chat Completion response for the third turn's output + reasoning_str = "I'll call get_weather." + boston_tool_args_str = '{"location": "Boston"}' + response_str = ( + f"<|channel|>analysis<|message|>{reasoning_str}<|end|>" + "<|start|>assistant to=functions.get_weather<|channel|>commentary" + f"<|constrain|>json<|message|>{boston_tool_args_str}<|call|>" + ) + response_3 = await self.generate_response_from_harmony_str( + serving_chat, req, response_str, stream=stream + ) + verify_chat_response( + response_3, + reasoning=reasoning_str, + tool_calls=[("get_weather", boston_tool_args_str)], + ) + + tool_call = response_3.choices[0].message.tool_calls[0] + + # Add the output messages from the third turn as input to the fourth turn + for choice in response_3.choices: + messages.append(choice.message.model_dump(exclude_none=True)) + + # Add our tool output message + messages.append( + { + "role": "tool", + "tool_call_id": tool_call.id, + "content": "10 degrees Celsius", + }, + ) + + # Test the Harmony messages for the fourth turn's input + req_4 = ChatCompletionRequest(model=MODEL_NAME, messages=messages, tools=tools) + input_messages_4, _ = serving_chat.online_renderer._make_request_with_harmony( + req_4 + ) + verify_harmony_messages( + input_messages_4, + [ + {"role": "system"}, + {"role": "developer"}, + {"role": "user"}, + { + "role": "assistant", + "channel": "analysis", + }, + {"role": "assistant"}, + {"role": "tool"}, + { + "role": "assistant", + "channel": "final", + }, + {"role": "user"}, + { + "role": "assistant", + "channel": "analysis", + "content": reasoning_str, + }, + { + "role": "assistant", + "channel": "commentary", + "recipient": "functions.get_weather", + "content": boston_tool_args_str, + }, + { + "role": "tool", + "author_name": "functions.get_weather", + "channel": "commentary", + "recipient": "assistant", + "content": "10 degrees Celsius", + }, + ], + ) + + @pytest.mark.asyncio + async def test_non_tool_reasoning(self, serving_chat): + messages: list[dict[str, Any]] = [ + { + "role": "user", + "content": "What's 2+2?", + }, + { + "role": "assistant", + "reasoning": "Adding 2 and 2 is easy. The result is 4.", + "content": "4", + }, + ] + req = ChatCompletionRequest(model=MODEL_NAME, messages=messages) + input_messages, _ = serving_chat.online_renderer._make_request_with_harmony(req) + + verify_harmony_messages( + input_messages, + [ + {"role": "system"}, + {"role": "user", "content": messages[0]["content"]}, + { + "role": "assistant", + "channel": "analysis", + }, + { + "role": "assistant", + "channel": "final", + "content": messages[1]["content"], + }, + ], + ) + + @pytest.mark.asyncio + async def test_non_tool_reasoning_empty_content(self, serving_chat): + messages: list[dict[str, Any]] = [ + { + "role": "user", + "content": "What's 2+2?", + }, + { + "role": "assistant", + "reasoning": "Adding 2 and 2 is easy. The result is 4.", + "content": "", + }, + ] + req = ChatCompletionRequest(model=MODEL_NAME, messages=messages) + input_messages, _ = serving_chat.online_renderer._make_request_with_harmony(req) + + verify_harmony_messages( + input_messages, + [ + {"role": "system"}, + {"role": "user", "content": messages[0]["content"]}, + { + "role": "assistant", + "channel": "analysis", + "content": messages[1]["reasoning"], + }, + ], + ) + + @pytest.mark.asyncio + async def test_non_tool_reasoning_empty_content_list(self, serving_chat): + messages: list[dict[str, Any]] = [ + { + "role": "user", + "content": "What's 2+2?", + }, + { + "role": "assistant", + "reasoning": "Adding 2 and 2 is easy. The result is 4.", + "content": [], + }, + ] + req = ChatCompletionRequest(model=MODEL_NAME, messages=messages) + input_messages, _ = serving_chat.online_renderer._make_request_with_harmony(req) + + verify_harmony_messages( + input_messages, + [ + {"role": "system"}, + {"role": "user", "content": messages[0]["content"]}, + { + "role": "assistant", + "channel": "analysis", + "content": messages[1]["reasoning"], + }, + ], + ) + + +@pytest.mark.asyncio +async def test_tool_choice_validation_without_parser(): + """Test that tool_choice='required' or named tool without tool_parser + returns an appropriate error message.""" + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + models = OpenAIServingModels( + engine_client=mock_engine, + base_model_paths=BASE_MODEL_PATHS, + ) + online_renderer = _build_online_renderer(mock_engine, models.registry) + + # Create serving_chat without tool_parser (enable_auto_tools=False) + serving_chat = OpenAIServingChat( + mock_engine, + models, + response_role="assistant", + online_renderer=online_renderer, + chat_template=CHAT_TEMPLATE, + chat_template_content_format="auto", + request_logger=None, + enable_auto_tools=False, # No tool parser + ) + + tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the weather in a given location", + "parameters": { + "type": "object", + "properties": {"location": {"type": "string"}}, + "required": ["location"], + }, + }, + } + ] + + # Test tool_choice="required" without tool_parser + req_required = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "What's the weather?"}], + tools=tools, + tool_choice="required", + ) + response_required = await serving_chat.create_chat_completion(req_required) + assert isinstance(response_required, ErrorResponse) + assert "tool_choice" in response_required.error.message + assert "--tool-call-parser" in response_required.error.message + + # Test named tool_choice without tool_parser + req_named = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "What's the weather?"}], + tools=tools, + tool_choice={"type": "function", "function": {"name": "get_weather"}}, + ) + response_named = await serving_chat.create_chat_completion(req_named) + assert isinstance(response_named, ErrorResponse) + assert "tool_choice" in response_named.error.message + assert "--tool-call-parser" in response_named.error.message + + +@pytest.mark.asyncio +async def test_streaming_n_gt1_independent_tool_parsers(): + """n>1 streaming must use independent parser instances + and token-id histories per choice. + """ + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + models = OpenAIServingModels( + engine_client=mock_engine, + base_model_paths=BASE_MODEL_PATHS, + ) + online_renderer = _build_online_renderer(mock_engine, models.registry) + + serving_chat = OpenAIServingChat( + mock_engine, + models, + response_role="assistant", + online_renderer=online_renderer, + chat_template=CHAT_TEMPLATE, + chat_template_content_format="auto", + request_logger=None, + enable_auto_tools=True, + tool_parser="hermes", + ) + + tokenizer = get_tokenizer(MODEL_NAME) + + tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get weather", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"], + }, + }, + } + ] + + num_choices = 2 + + request = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "test"}], + n=num_choices, + stream=True, + tools=tools, + tool_choice="auto", + ) + + tool_call_text = ( + "\n" + '{"name": "get_weather", "arguments": {"city": "Tokyo"}}\n' + "" + ) + all_token_ids = tokenizer.encode(tool_call_text, add_special_tokens=False) + + # Compute proper delta text for each token so that concatenated deltas + # reproduce the original string exactly. + steps: list[tuple[str, int]] = [] + prev_decoded = "" + for i, tid in enumerate(all_token_ids): + decoded_so_far = tokenizer.decode(all_token_ids[: i + 1]) + delta = decoded_so_far[len(prev_decoded) :] + steps.append((delta, tid)) + prev_decoded = decoded_so_far + + async def result_generator(): + for delta_text, token_id in steps: + yield RequestOutput( + request_id="test-req", + prompt="test", + prompt_token_ids=[1, 2, 3], + prompt_logprobs=None, + outputs=[ + CompletionOutput( + index=choice_idx, + text=delta_text, + token_ids=[token_id], + cumulative_logprob=0.0, + logprobs=None, + ) + for choice_idx in range(num_choices) + ], + finished=False, + ) + # Final output with finish_reason + yield RequestOutput( + request_id="test-req", + prompt="test", + prompt_token_ids=[1, 2, 3], + prompt_logprobs=None, + outputs=[ + CompletionOutput( + index=choice_idx, + text="", + token_ids=[], + cumulative_logprob=0.0, + logprobs=None, + finish_reason="stop", + ) + for choice_idx in range(num_choices) + ], + finished=True, + ) + + # Collect tool-call deltas and finish_reasons per choice from the SSE + # stream. + tc_deltas_by_choice: dict[int, list[dict]] = {i: [] for i in range(num_choices)} + finish_reasons_by_choice: dict[int, list[str]] = {i: [] for i in range(num_choices)} + async for chunk_str in serving_chat.chat_completion_stream_generator( + request=request, + result_generator=result_generator(), + request_id="test-req", + model_name=MODEL_NAME, + conversation=[], + tokenizer=tokenizer, + request_metadata=RequestResponseMetadata( + request_id="test-req", + model_name=MODEL_NAME, + ), + ): + if not chunk_str.strip() or "data: [DONE]" in chunk_str: + continue + if chunk_str.startswith("data: "): + data = json.loads(chunk_str[6:].strip()) + for choice in data.get("choices", []): + idx = choice["index"] + delta = choice.get("delta", {}) + if delta.get("tool_calls"): + for tc in delta["tool_calls"]: + tc_deltas_by_choice[idx].append(tc) + if choice.get("finish_reason") is not None: + finish_reasons_by_choice[idx].append(choice["finish_reason"]) + + # Both choices must independently produce the correct tool call. + for choice_idx in range(num_choices): + deltas = tc_deltas_by_choice[choice_idx] + assert len(deltas) > 0, ( + f"Choice {choice_idx}: expected tool-call deltas but got none" + ) + + name = None + args_buf = "" + for tc in deltas: + fn = tc.get("function", {}) + if fn.get("name"): + name = fn["name"] + if fn.get("arguments"): + args_buf += fn["arguments"] + + assert name == "get_weather", ( + f"Choice {choice_idx}: expected 'get_weather', got {name!r}" + ) + parsed_args = json.loads(args_buf) + assert parsed_args == {"city": "Tokyo"}, ( + f"Choice {choice_idx}: expected {{'city': 'Tokyo'}}, got {parsed_args}" + ) + + reasons = finish_reasons_by_choice[choice_idx] + assert len(reasons) == 1, ( + f"Choice {choice_idx}: expected exactly 1 finish_reason, got {reasons}" + ) + assert reasons[0] == "tool_calls", ( + f"Choice {choice_idx}: expected finish_reason='tool_calls', " + f"got '{reasons[0]}'" + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_thinking_token_budget.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_thinking_token_budget.py new file mode 100644 index 0000000000000000000000000000000000000000..234bd7d8823677bdc17f4b550c3ea51c24779725 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_thinking_token_budget.py @@ -0,0 +1,338 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""E2E tests for ``thinking_token_budget`` with reasoning models. + +Covers Qwen3-0.6B and Qwen3.5 FP8 + MTP. +""" + +import asyncio +import json +from typing import Literal + +import openai +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer, multi_gpu_only, requires_fp8 +from vllm.platforms import current_platform +from vllm.tokenizers import get_tokenizer + +MODEL_NAME = "Qwen/Qwen3-0.6B" +QWEN35_FP8_MTP_MODEL = "Qwen/Qwen3.5-35B-A3B-FP8" +MESSAGES = [{"role": "user", "content": "What is 1+1? Be concise."}] +THINK_BUDGET = 5 + +REASONING_START_STR = "" +REASONING_END_STR = "" + + +def _count_reasoning_decode_token_ids_between_markers( + full_token_ids: list[int], + reasoning_start_ids: list[int], + reasoning_end_ids: list[int], +) -> int | None: + """Count decode tokens in the thinking span (after last start, before first end).""" + + if not reasoning_start_ids or not reasoning_end_ids: + raise ValueError("reasoning marker token id lists must be non-empty") + + def _last_subseq_index(haystack: list[int], needle: list[int]) -> int: + n = len(needle) + if n > len(haystack): + return -1 + for i in range(len(haystack) - n, -1, -1): + if haystack[i : i + n] == needle: + return i + return -1 + + last_start = _last_subseq_index(full_token_ids, reasoning_start_ids) + if last_start < 0: + return None + + pos_after_start = last_start + len(reasoning_start_ids) + end_n = len(reasoning_end_ids) + for j in range(pos_after_start, len(full_token_ids) - end_n + 1): + if full_token_ids[j : j + end_n] == reasoning_end_ids: + return j - pos_after_start + return len(full_token_ids) - pos_after_start + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--reasoning-parser", + "qwen3", + "--reasoning-config", + '{"reasoning_start_str": "", "reasoning_end_str": ""}', + "--max-model-len", + "2048", + "--enforce-eager", + "--gpu-memory-utilization", + "0.4", + "--no-async-scheduling", + ] + # thinking_token_budget is not yet supported by the V2 model runner. + env_dict = {"VLLM_USE_V2_MODEL_RUNNER": "0"} + with RemoteOpenAIServer(MODEL_NAME, args, env_dict=env_dict) as remote_server: + yield remote_server + + +@pytest.fixture(scope="module") +def server_with_auto_reasoning_config(): + args = [ + "--reasoning-parser", + "qwen3", + "--max-model-len", + "2048", + "--enforce-eager", + "--gpu-memory-utilization", + "0.4", + "--no-async-scheduling", + ] + # thinking_token_budget is not yet supported by the V2 model runner. + env_dict = {"VLLM_USE_V2_MODEL_RUNNER": "0"} + with RemoteOpenAIServer(MODEL_NAME, args, env_dict=env_dict) as remote_server: + yield remote_server + + +@pytest.fixture(scope="module") +def server_qwen35_fp8_mtp_tp2(): + """Qwen3.5-35B FP8 with MTP speculative decoding and tensor parallel size 2.""" + if current_platform.device_count() < 2: + pytest.skip("Need at least 2 GPUs for --tensor-parallel-size 2") + if not current_platform.supports_fp8(): + pytest.skip("FP8 is not supported on this platform") + + spec_cfg = { + "method": "mtp", + "num_speculative_tokens": 2, + "max_model_len": 32768, + } + args = [ + "--tensor-parallel-size", + "2", + "--max-model-len", + "32768", + "--speculative-config", + json.dumps(spec_cfg), + "--reasoning-parser", + "qwen3", + "--reasoning-config", + json.dumps( + { + "reasoning_start_str": REASONING_START_STR, + "reasoning_end_str": REASONING_END_STR, + } + ), + ] + # thinking_token_budget is not yet supported by the V2 model runner. + env_dict: dict[str, str] = {"VLLM_USE_V2_MODEL_RUNNER": "0"} + # With 4+ GPUs, run TP=2 on physical devices 2,3 so module-scoped 0.6B servers + # on 0,1 do not exhaust memory on the same devices as this worker. + if current_platform.device_count() >= 4: + env_dict["CUDA_VISIBLE_DEVICES"] = "2,3" + + with RemoteOpenAIServer( + QWEN35_FP8_MTP_MODEL, + args, + max_wait_seconds=3000, + env_dict=env_dict, + ) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(request, server, server_with_auto_reasoning_config): + server_map = { + "default": server, + "auto_config": server_with_auto_reasoning_config, + } + target_server = server_map[request.param] + async with target_server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize("client", ["default", "auto_config"], indirect=True) +async def test_thinking_token_budget_mixed_requests(client: openai.AsyncOpenAI): + """Test that mixed requests (some with thinking_token_budget, some without) + complete successfully without errors.""" + + response_with_budget = await client.chat.completions.create( + model=MODEL_NAME, + messages=MESSAGES, + max_tokens=100, + extra_body={"thinking_token_budget": THINK_BUDGET}, + ) + response_without_budget = await client.chat.completions.create( + model=MODEL_NAME, + messages=MESSAGES, + max_tokens=100, + ) + + msg_with = response_with_budget.choices[0].message + msg_without = response_without_budget.choices[0].message + + assert msg_with.content or getattr(msg_with, "reasoning", None) + assert msg_without.content or getattr(msg_without, "reasoning", None) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("client", ["default", "auto_config"], indirect=True) +async def test_thinking_token_budget_limits_reasoning(client: openai.AsyncOpenAI): + """Test that thinking_token_budget limits the number of reasoning tokens. + + Counts non-empty streaming ``delta.reasoning`` chunks (coarse proxy; each + chunk may represent multiple decode tokens — see + ``_count_reasoning_decode_token_ids_between_markers`` and the Qwen3.5 MTP + test for id-based checks). + """ + + reasoning_token_count = 0 + stream = await client.chat.completions.create( + model=MODEL_NAME, + messages=MESSAGES, + max_tokens=100, + stream=True, + extra_body={"thinking_token_budget": THINK_BUDGET}, + ) + async for chunk in stream: + delta = chunk.choices[0].delta + if getattr(delta, "reasoning", None): + reasoning_token_count += 1 + + assert reasoning_token_count == THINK_BUDGET, ( + f"reasoning tokens ({reasoning_token_count}) exceeded " + f"thinking_token_budget ({THINK_BUDGET})" + ) + + +@pytest.mark.asyncio +@multi_gpu_only(num_gpus=2) +@requires_fp8 +async def test_thinking_token_budget_qwen35_fp8_mtp_concurrent_mixed_budget_and_plain( + server_qwen35_fp8_mtp_tp2, +): + """Concurrent chat requests: some with ``thinking_token_budget``, some without. + + Exercises the scheduler / input processor under a mixed batch on the same + Qwen3.5 FP8 + MTP (TP=2) server. Budgeted calls are checked with + ``_count_reasoning_decode_token_ids_between_markers`` on full token ids. + """ + + _batch_spec: list[tuple[Literal["budget"], int] | tuple[Literal["plain"], None]] = [ + ("budget", 1), + ("budget", 12), + ("plain", None), + ("budget", 20), + ("budget", 14), + ("plain", None), + ("plain", None), + ("budget", 12), + ("plain", None), + ] + + tokenizer = get_tokenizer(tokenizer_name=QWEN35_FP8_MTP_MODEL) + start_ids = list(tokenizer.encode(REASONING_START_STR, add_special_tokens=False)) + end_ids = list(tokenizer.encode(REASONING_END_STR, add_special_tokens=False)) + + async with server_qwen35_fp8_mtp_tp2.get_async_client() as client: + + async def budgeted_call(expected_budget: int): + return await client.chat.completions.create( + model=QWEN35_FP8_MTP_MODEL, + messages=MESSAGES, + max_tokens=256, + stream=False, + extra_body={ + "thinking_token_budget": expected_budget, + "return_token_ids": True, + }, + ) + + async def plain_call(): + return await client.chat.completions.create( + model=QWEN35_FP8_MTP_MODEL, + messages=MESSAGES, + max_tokens=256, + stream=False, + ) + + coros = [] + for row in _batch_spec: + if row[0] == "budget": + b = row[1] + assert isinstance(b, int) + coros.append(budgeted_call(b)) + else: + coros.append(plain_call()) + results = await asyncio.gather(*coros) + + for i, (response, (kind, expected_budget)) in enumerate( + zip(results, _batch_spec, strict=True) + ): + msg = response.choices[0].message + assert msg.content or getattr(msg, "reasoning", None), ( + f"index {i} ({kind}): empty message" + ) + + if kind == "budget": + assert expected_budget is not None + assert response.prompt_token_ids is not None + assert response.choices[0].token_ids is not None + full_ids = list(response.prompt_token_ids) + list( + response.choices[0].token_ids + ) + n_reason = _count_reasoning_decode_token_ids_between_markers( + full_ids, start_ids, end_ids + ) + assert n_reason is not None, f"index {i}: missing reasoning start in ids" + assert n_reason == expected_budget, ( + f"index {i}: reasoning decode token ids ({n_reason}) != " + f"thinking_token_budget ({expected_budget})" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("client", ["default", "auto_config"], indirect=True) +async def test_streaming_with_thinking_disabled_stays_in_content( + client: openai.AsyncOpenAI, +): + request_kwargs = { + "model": MODEL_NAME, + "messages": [ + { + "role": "user", + "content": "Which is larger, 4 or 12?" + " Output exactly one token: 4 or 12.", + } + ], + "max_tokens": 16, + "temperature": 0.0, + "extra_body": {"chat_template_kwargs": {"enable_thinking": False}}, + } + + response = await client.chat.completions.create(**request_kwargs) + message = response.choices[0].message + assert message.content is not None and message.content.strip() != "" + assert getattr(message, "reasoning", None) in (None, "") + + stream = await client.chat.completions.create( + **request_kwargs, + stream=True, + ) + + content_chunks = [] + reasoning_chunks = [] + async for chunk in stream: + if not chunk.choices: + continue + delta = chunk.choices[0].delta + if getattr(delta, "content", None): + content_chunks.append(delta.content) + if getattr(delta, "reasoning", None): + reasoning_chunks.append(delta.reasoning) + + assert "".join(content_chunks).strip() != "" + assert reasoning_chunks == [] diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_thinking_token_budget_validation.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_thinking_token_budget_validation.py new file mode 100644 index 0000000000000000000000000000000000000000..a4c706a820e684ccad62e2920c058a4417484645 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/chat_completion/test_thinking_token_budget_validation.py @@ -0,0 +1,76 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +from pydantic import ValidationError + +from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest +from vllm.entrypoints.openai.completion.protocol import CompletionRequest + + +@pytest.mark.parametrize("raw_value", [-2, 0.6, 10.5]) +def test_chat_completion_request_rejects_invalid_thinking_token_budget(raw_value): + with pytest.raises(ValidationError, match="thinking_token_budget"): + ChatCompletionRequest.model_validate( + { + "model": "qwen", + "messages": [{"role": "user", "content": "hello"}], + "thinking_token_budget": raw_value, + } + ) + + +def test_chat_completion_request_accepts_valid_thinking_token_budget(): + request = ChatCompletionRequest.model_validate( + { + "model": "qwen", + "messages": [{"role": "user", "content": "hello"}], + "thinking_token_budget": 10, + } + ) + assert request.thinking_token_budget == 10 + + +def test_chat_completion_request_accepts_minus_one_as_unlimited(): + request = ChatCompletionRequest.model_validate( + { + "model": "qwen", + "messages": [{"role": "user", "content": "hello"}], + "thinking_token_budget": -1, + } + ) + assert request.thinking_token_budget is None + + +@pytest.mark.parametrize("raw_value", [0.6, 3.14, -2]) +def test_completion_request_rejects_invalid_thinking_token_budget(raw_value): + with pytest.raises(ValidationError, match="thinking_token_budget"): + CompletionRequest.model_validate( + { + "model": "qwen", + "prompt": "hello", + "thinking_token_budget": raw_value, + } + ) + + +def test_completion_request_accepts_valid_thinking_token_budget(): + request = CompletionRequest.model_validate( + { + "model": "qwen", + "prompt": "hello", + "thinking_token_budget": 5, + } + ) + assert request.thinking_token_budget == 5 + + +def test_completion_request_accepts_minus_one_as_unlimited(): + request = CompletionRequest.model_validate( + { + "model": "qwen", + "prompt": "hello", + "thinking_token_budget": -1, + } + ) + assert request.thinking_token_budget is None diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_completion.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_completion.py new file mode 100644 index 0000000000000000000000000000000000000000..fc9f408962547f7eb8b67a1008009319c2b7a975 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_completion.py @@ -0,0 +1,732 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import openai # use the official client for correctness check +import pytest +import pytest_asyncio +import regex as re +from openai import BadRequestError + +from tests.utils import RemoteOpenAIServer +from vllm.tokenizers import get_tokenizer + +# any model with a chat template should work here +MODEL_NAME = "facebook/opt-125m" + + +@pytest.fixture(scope="module") +def default_server_args(): + return [ + "--dtype", + "float32", + "--max-model-len", + "2048", + "--max-num-seqs", + "128", + "--enforce-eager", + "--enable-prompt-tokens-details", + "--no-enable-prefix-caching", + ] + + +@pytest.fixture(scope="module") +def server(default_server_args): + with RemoteOpenAIServer(MODEL_NAME, default_server_args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_single_completion(client: openai.AsyncOpenAI, model_name: str) -> None: + completion = await client.completions.create( + model=model_name, prompt="Hello, my name is", max_tokens=5, temperature=0.0 + ) + + assert completion.id is not None + assert completion.choices is not None and len(completion.choices) == 1 + + choice = completion.choices[0] + assert len(choice.text) >= 5 + assert choice.finish_reason == "length" + assert completion.usage is not None + assert completion.usage.completion_tokens == 5 + assert completion.usage.prompt_tokens == 6 + assert completion.usage.total_tokens == 11 + assert completion.usage.prompt_tokens_details is not None + assert completion.usage.prompt_tokens_details.cached_tokens == 0 + + # test using token IDs + completion = await client.completions.create( + model=model_name, + prompt=[0, 0, 0, 0, 0], + max_tokens=5, + temperature=0.0, + ) + assert len(completion.choices[0].text) >= 1 + assert completion.choices[0].prompt_logprobs is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_completion_truncation_side_controls_prompt_truncation( + client: openai.AsyncOpenAI, model_name: str +) -> None: + prompt_token_ids = list(range(8)) + + right_completion = await client.completions.create( + model=model_name, + prompt=prompt_token_ids, + max_tokens=1, + temperature=0.0, + extra_body={ + "return_token_ids": True, + "truncate_prompt_tokens": 4, + "truncation_side": "right", + }, + ) + assert right_completion.choices[0].prompt_token_ids == prompt_token_ids[:4] + + left_completion = await client.completions.create( + model=model_name, + prompt=prompt_token_ids, + max_tokens=1, + temperature=0.0, + extra_body={ + "return_token_ids": True, + "truncate_prompt_tokens": 4, + "truncation_side": "left", + }, + ) + assert left_completion.choices[0].prompt_token_ids == prompt_token_ids[-4:] + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_no_logprobs(client: openai.AsyncOpenAI, model_name: str): + # test using token IDs + completion = await client.completions.create( + model=model_name, + prompt=[0, 0, 0, 0, 0], + max_tokens=5, + temperature=0.0, + logprobs=None, + ) + choice = completion.choices[0] + assert choice.logprobs is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_zero_logprobs(client: openai.AsyncOpenAI, model_name: str): + # test using token IDs + completion = await client.completions.create( + model=model_name, + prompt=[0, 0, 0, 0, 0], + max_tokens=5, + temperature=0.0, + logprobs=0, + ) + choice = completion.choices[0] + assert choice.logprobs is not None + assert choice.logprobs.token_logprobs is not None + assert choice.logprobs.top_logprobs is not None + assert len(choice.logprobs.top_logprobs[0]) == 1 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_some_logprobs(client: openai.AsyncOpenAI, model_name: str): + # test using token IDs + completion = await client.completions.create( + model=model_name, + prompt=[0, 0, 0, 0, 0], + max_tokens=5, + temperature=0.0, + logprobs=5, + ) + choice = completion.choices[0] + assert choice.logprobs is not None + assert choice.logprobs.token_logprobs is not None + assert choice.logprobs.top_logprobs is not None + assert 5 <= len(choice.logprobs.top_logprobs[0]) <= 6 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_too_many_completion_logprobs( + client: openai.AsyncOpenAI, model_name: str +) -> None: + with pytest.raises( + (openai.BadRequestError, openai.APIError) + ): # test using token IDs + await client.completions.create( + model=model_name, + prompt=[0, 0, 0, 0, 0], + max_tokens=5, + temperature=0.0, + # vLLM has higher default max_logprobs (20 instead of 5) to support + # both Completion API and Chat Completion API + logprobs=21, + ) + ... + with pytest.raises( + (openai.BadRequestError, openai.APIError) + ): # test using token IDs + stream = await client.completions.create( + model=model_name, + prompt=[0, 0, 0, 0, 0], + max_tokens=5, + temperature=0.0, + # vLLM has higher default max_logprobs (20 instead of 5) to support + # both Completion API and Chat Completion API + logprobs=30, + stream=True, + ) + async for chunk in stream: + ... + + # the server should still work afterwards + completion = await client.completions.create( + model=model_name, + prompt=[0, 0, 0, 0, 0], + max_tokens=5, + temperature=0.0, + ) + assert len(completion.choices[0].text) >= 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name, prompt_logprobs", + [(MODEL_NAME, -1), (MODEL_NAME, 0), (MODEL_NAME, 1), (MODEL_NAME, None)], +) +async def test_prompt_logprobs_completion( + client: openai.AsyncOpenAI, model_name: str, prompt_logprobs: int | None +): + params: dict = { + "prompt": ["A robot may not injure another robot", "My name is"], + "model": model_name, + } + if prompt_logprobs is not None: + params["extra_body"] = {"prompt_logprobs": prompt_logprobs} + + if prompt_logprobs is not None and prompt_logprobs < 0: + with pytest.raises(BadRequestError): + await client.completions.create(**params) + else: + completion = await client.completions.create(**params) + if prompt_logprobs is not None: + assert completion.choices[0].prompt_logprobs is not None + assert len(completion.choices[0].prompt_logprobs) > 0 + + assert completion.choices[1].prompt_logprobs is not None + assert len(completion.choices[1].prompt_logprobs) > 0 + + else: + assert completion.choices[0].prompt_logprobs is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_completion_streaming( + client: openai.AsyncOpenAI, model_name: str +) -> None: + prompt = "What is an LLM?" + + single_completion = await client.completions.create( + model=model_name, + prompt=prompt, + max_tokens=5, + temperature=0.0, + ) + single_output = single_completion.choices[0].text + stream = await client.completions.create( + model=model_name, prompt=prompt, max_tokens=5, temperature=0.0, stream=True + ) + chunks: list[str] = [] + finish_reason_count = 0 + async for chunk in stream: + chunks.append(chunk.choices[0].text) + if chunk.choices[0].finish_reason is not None: + finish_reason_count += 1 + # finish reason should only return in last block + assert finish_reason_count == 1 + assert chunk.choices[0].finish_reason == "length" + assert chunk.choices[0].text + assert "".join(chunks) == single_output + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_parallel_no_streaming(client: openai.AsyncOpenAI, model_name: str): + """Parallel sampling without streaming. + A single request output contains a list of completions. + """ + + prompt = "What is an LLM?" + n = 3 + max_tokens = 50 # we want some to finish earlier than others + + # High temperature to maximize chance of unique completions. + completion = await client.completions.create( + model=model_name, + prompt=prompt, + max_tokens=max_tokens, + n=n, + temperature=1.0, + stream=False, + logprobs=0, + seed=42, + ) + + # Assert `n` completions + num_completions = len(completion.choices) + assert num_completions == n, f"Num completions {num_completions} but expected {n}." + completion_repeats: dict[str, int] = {} + output_token_lengths = set() + for idx, choice in enumerate(completion.choices): + # Assert correct completion index & some finish reason. + assert choice.index == idx, f"Index {choice.index} but expected {idx}." + assert choice.finish_reason is not None, "None finish_reason is invalid." + text = choice.text + completion_repeats[text] = completion_repeats.get(text, 0) + 1 + output_token_lengths.add(len(choice.logprobs.tokens)) + # Assert subrequests finished at different times + assert len(output_token_lengths) > 1 + # Assert `n` unique completions + num_unique = len(completion_repeats) + if num_unique != n: + repeats = {txt: num for (txt, num) in completion_repeats.items() if num > 1} + raise AssertionError( + f"Expected {n} unique completions, got {num_unique}; repeats: {repeats}." + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_parallel_streaming(client: openai.AsyncOpenAI, model_name: str): + """Streaming for parallel sampling. + The tokens from multiple samples, are flattened into a single stream, + with an index to indicate which sample the token belongs to. + """ + + prompt = "What is an LLM?" + n = 3 + max_tokens = 50 # we want some to finish earlier than others + + stream = await client.completions.create( + model=model_name, + prompt=prompt, + max_tokens=max_tokens, + n=n, + temperature=1.0, + stream=True, + seed=42, + ) + chunks: list[list[str]] = [[] for _ in range(n)] + finish_reason_count = 0 + async for chunk in stream: + index = chunk.choices[0].index + text = chunk.choices[0].text + chunks[index].append(text) + if chunk.choices[0].finish_reason is not None: + finish_reason_count += 1 + # Assert `n` completions with correct finish reasons + assert finish_reason_count == n, ( + f"Expected {n} completions with valid indices and finish_reason." + ) + completion_repeats: dict[str, int] = {} + chunk_lengths = set() + for chunk in chunks: + chunk_len = len(chunk) + # Assert correct number of completion tokens + chunk_lengths.add(chunk_len) + assert chunk_len <= max_tokens, ( + f"max_tokens={max_tokens} but chunk len is {chunk_len}." + ) + text = "".join(chunk) + completion_repeats[text] = completion_repeats.get(text, 0) + 1 + print(text) + # Assert subrequests finished at different times + assert len(chunk_lengths) > 1 + # Assert `n` unique completions + num_unique = len(completion_repeats) + if num_unique != n: + repeats = {txt: num for (txt, num) in completion_repeats.items() if num > 1} + raise AssertionError( + f"{num_unique} unique completions, expected {n}; repeats: {repeats}" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_completion_stream_options(client: openai.AsyncOpenAI, model_name: str): + prompt = "What is the capital of France?" + + # Test stream=True, stream_options= + # {"include_usage": False, "continuous_usage_stats": False} + stream = await client.completions.create( + model=model_name, + prompt=prompt, + max_tokens=5, + temperature=0.0, + stream=True, + stream_options={ + "include_usage": False, + "continuous_usage_stats": False, + }, + ) + + async for chunk in stream: + assert chunk.usage is None + + # Test stream=True, stream_options= + # {"include_usage": False, "continuous_usage_stats": True} + stream = await client.completions.create( + model=model_name, + prompt=prompt, + max_tokens=5, + temperature=0.0, + stream=True, + stream_options={ + "include_usage": False, + "continuous_usage_stats": True, + }, + ) + async for chunk in stream: + assert chunk.usage is None + + # Test stream=True, stream_options= + # {"include_usage": True, "continuous_usage_stats": False} + stream = await client.completions.create( + model=model_name, + prompt=prompt, + max_tokens=5, + temperature=0.0, + stream=True, + stream_options={ + "include_usage": True, + "continuous_usage_stats": False, + }, + ) + async for chunk in stream: + if chunk.choices[0].finish_reason is None: + assert chunk.usage is None + else: + assert chunk.usage is None + final_chunk = await anext(stream) + assert final_chunk.usage is not None + assert final_chunk.usage.prompt_tokens > 0 + assert final_chunk.usage.completion_tokens > 0 + assert final_chunk.usage.total_tokens == ( + final_chunk.usage.prompt_tokens + final_chunk.usage.completion_tokens + ) + assert final_chunk.choices == [] + + # Test stream=True, stream_options= + # {"include_usage": True, "continuous_usage_stats": True} + stream = await client.completions.create( + model=model_name, + prompt=prompt, + max_tokens=5, + temperature=0.0, + stream=True, + stream_options={ + "include_usage": True, + "continuous_usage_stats": True, + }, + ) + async for chunk in stream: + assert chunk.usage is not None + assert chunk.usage.prompt_tokens > 0 + assert chunk.usage.completion_tokens > 0 + assert chunk.usage.total_tokens == ( + chunk.usage.prompt_tokens + chunk.usage.completion_tokens + ) + if chunk.choices[0].finish_reason is not None: + final_chunk = await anext(stream) + assert final_chunk.usage is not None + assert final_chunk.usage.prompt_tokens > 0 + assert final_chunk.usage.completion_tokens > 0 + assert final_chunk.usage.total_tokens == ( + final_chunk.usage.prompt_tokens + final_chunk.usage.completion_tokens + ) + assert final_chunk.choices == [] + + # Test stream=True, stream_options={} + stream = await client.completions.create( + model=model_name, + prompt=prompt, + max_tokens=5, + temperature=0.0, + stream=True, + stream_options={}, + ) + async for chunk in stream: + assert chunk.usage is None + + # Test stream=False, stream_options= + # {"include_usage": None} + with pytest.raises(BadRequestError): + await client.completions.create( + model=model_name, + prompt=prompt, + max_tokens=5, + temperature=0.0, + stream=False, + stream_options={"include_usage": None}, + ) + + # Test stream=False, stream_options= + # {"include_usage": True} + with pytest.raises(BadRequestError): + await client.completions.create( + model=model_name, + prompt=prompt, + max_tokens=5, + temperature=0.0, + stream=False, + stream_options={"include_usage": True}, + ) + + # Test stream=False, stream_options= + # {"continuous_usage_stats": None} + with pytest.raises(BadRequestError): + await client.completions.create( + model=model_name, + prompt=prompt, + max_tokens=5, + temperature=0.0, + stream=False, + stream_options={"continuous_usage_stats": None}, + ) + + # Test stream=False, stream_options= + # {"continuous_usage_stats": True} + with pytest.raises(BadRequestError): + await client.completions.create( + model=model_name, + prompt=prompt, + max_tokens=5, + temperature=0.0, + stream=False, + stream_options={"continuous_usage_stats": True}, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_batch_completions(client: openai.AsyncOpenAI, model_name: str): + # test both text and token IDs + for prompts in (["Hello, my name is"] * 2, [[0, 0, 0, 0, 0]] * 2): + # test simple list + batch = await client.completions.create( + model=model_name, + prompt=prompts, + max_tokens=5, + temperature=0.0, + ) + assert len(batch.choices) == 2 + assert batch.choices[0].text == batch.choices[1].text + + # test n = 2 + batch = await client.completions.create( + model=model_name, + prompt=prompts, + n=2, + max_tokens=5, + temperature=0.0, + extra_body=dict( + # NOTE: this has to be true for n > 1 in vLLM, but + # not necessary for official client. + use_beam_search=True + ), + ) + assert len(batch.choices) == 4 + assert batch.choices[0].text != batch.choices[1].text, ( + "beam search should be different" + ) + assert batch.choices[0].text == batch.choices[2].text, ( + "two copies of the same prompt should be the same" + ) + assert batch.choices[1].text == batch.choices[3].text, ( + "two copies of the same prompt should be the same" + ) + + # test streaming + batch = await client.completions.create( + model=model_name, + prompt=prompts, + max_tokens=5, + temperature=0.0, + stream=True, + ) + texts = [""] * 2 + async for chunk in batch: + assert len(chunk.choices) == 1 + choice = chunk.choices[0] + texts[choice.index] += choice.text + assert texts[0] == texts[1] + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +@pytest.mark.parametrize("logprobs_arg", [1, 0]) +async def test_echo_logprob_completion( + client: openai.AsyncOpenAI, model_name: str, logprobs_arg: int +): + tokenizer = get_tokenizer(tokenizer_name=MODEL_NAME) + # test using text and token IDs + for prompt in ("Hello, my name is", [0, 0, 0, 0, 0]): + completion = await client.completions.create( + model=model_name, + prompt=prompt, + max_tokens=5, + temperature=0.0, + echo=True, + logprobs=logprobs_arg, + ) + + prompt_text = tokenizer.decode(prompt) if isinstance(prompt, list) else prompt + assert re.search(r"^" + prompt_text, completion.choices[0].text) + logprobs = completion.choices[0].logprobs + assert logprobs is not None + assert len(logprobs.text_offset) > 5 + assert len(logprobs.token_logprobs) > 5 and logprobs.token_logprobs[0] is None + assert len(logprobs.top_logprobs) > 5 and logprobs.top_logprobs[0] is None + for top_logprobs in logprobs.top_logprobs[1:]: + assert max(logprobs_arg, 1) <= len(top_logprobs) <= logprobs_arg + 1 + assert len(logprobs.tokens) > 5 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_invalid_json_schema(client: openai.AsyncOpenAI, model_name: str) -> None: + invalid_json_schema = { + "$defs": { + "CarType": { + "enum": ["sedan", "SUV", "Truck", "Coupe"], + "title": "CarType", + "type": "string", + } + }, + "properties": { + "brand": {"title": "Brand", "type": "string"}, + "model": {"title": "Model", "type": "string"}, + "car_type": {"$ref": "#/$defs/CarType"}, + "foo": "bar", + }, + "required": ["brand", "model", "car_type"], + "title": "CarDescription", + "type": "object", + } + prompt = ( + "Generate a JSON with the brand, model and car_type of" + "the most iconic car from the 90's" + ) + with pytest.raises((openai.BadRequestError, openai.APIError)): + await client.completions.create( + model=model_name, + prompt=prompt, + extra_body={"structured_outputs": {"json": invalid_json_schema}}, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_invalid_regex(client: openai.AsyncOpenAI, model_name: str): + prompt = ( + "Generate an email address for Alan Turing, who works in Enigma." + "End in .com and new line. Example result:" + "alan.turing@enigma.com\n" + ) + + with pytest.raises((openai.BadRequestError, openai.APIError)): + await client.completions.create( + model=model_name, + prompt=prompt, + extra_body={"structured_outputs": {"regex": r"[.*"}, "stop": ["\n"]}, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_invalid_grammar(client: openai.AsyncOpenAI, model_name: str): + invalid_simplified_sql_grammar = """ + root ::= select_statementinvalidsyntax + + select_statement ::= "SELECT " column " from " table " where " condition + + column ::= "col_1 " | "col_2 " + + table ::= "table_1 " | "table_2 " + + condition ::= column "= " number + + number ::= "1 " | "2 " + """ + + prompt = ( + "Generate an SQL query to show the 'username' and 'email'" + "from the 'users' table." + ) + with pytest.raises((openai.BadRequestError, openai.APIError)): + await client.completions.create( + model=model_name, + prompt=prompt, + extra_body={ + "structured_outputs": {"grammar": invalid_simplified_sql_grammar} + }, + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_completion_error.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_completion_error.py new file mode 100644 index 0000000000000000000000000000000000000000..194326895e87cfc07ac446207c4c17b3bbbe383a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_completion_error.py @@ -0,0 +1,380 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from dataclasses import dataclass, field +from typing import Any +from unittest.mock import AsyncMock, MagicMock + +import pytest +from pydantic import ValidationError + +from vllm.config.multimodal import MultiModalConfig +from vllm.entrypoints.openai.completion.protocol import CompletionRequest +from vllm.entrypoints.openai.completion.serving import OpenAIServingCompletion +from vllm.entrypoints.openai.engine.protocol import GenerationError +from vllm.entrypoints.openai.models.protocol import BaseModelPath +from vllm.entrypoints.openai.models.serving import OpenAIServingModels +from vllm.entrypoints.scale_out.render.serving import ServingRender +from vllm.outputs import CompletionOutput, RequestOutput +from vllm.renderers.hf import HfRenderer +from vllm.renderers.online_renderer import OnlineRenderer +from vllm.tokenizers.registry import cached_tokenizer_from_config +from vllm.v1.engine.async_llm import AsyncLLM + +MODEL_NAME = "openai-community/gpt2" +MODEL_NAME_SHORT = "gpt2" +BASE_MODEL_PATHS = [ + BaseModelPath(name=MODEL_NAME, model_path=MODEL_NAME), + BaseModelPath(name=MODEL_NAME_SHORT, model_path=MODEL_NAME_SHORT), +] + + +@dataclass +class MockHFConfig: + model_type: str = "any" + + +@dataclass +class MockModelConfig: + task = "generate" + runner_type = "generate" + model = MODEL_NAME + tokenizer = MODEL_NAME + trust_remote_code = False + tokenizer_mode = "auto" + max_model_len = 100 + tokenizer_revision = None + multimodal_config = MultiModalConfig() + hf_config = MockHFConfig() + logits_processors: list[str] | None = None + diff_sampling_param: dict | None = None + allowed_local_media_path: str = "" + allowed_media_domains: list[str] | None = None + encoder_config = None + generation_config: str = "auto" + media_io_kwargs: dict[str, dict[str, Any]] = field(default_factory=dict) + skip_tokenizer_init = False + is_encoder_decoder: bool = False + is_multimodal_model: bool = False + renderer_num_workers: int = 1 + + def get_diff_sampling_param(self): + return self.diff_sampling_param or {} + + +@dataclass +class MockParallelConfig: + _api_process_rank: int = 0 + + +@dataclass +class MockVllmConfig: + model_config: MockModelConfig + parallel_config: MockParallelConfig + + +def _build_serving_completion(engine: AsyncLLM) -> OpenAIServingCompletion: + models = OpenAIServingModels( + engine_client=engine, + base_model_paths=BASE_MODEL_PATHS, + ) + online_renderer = OnlineRenderer( + model_config=engine.model_config, + renderer=engine.renderer, + request_logger=None, + chat_template=None, + chat_template_content_format="auto", + ) + return OpenAIServingCompletion( + engine, + models, + online_renderer=online_renderer, + request_logger=None, + ) + + +def _build_renderer(model_config: MockModelConfig): + return HfRenderer( + MockVllmConfig(model_config, parallel_config=MockParallelConfig()), + cached_tokenizer_from_config(model_config), + ) + + +@pytest.mark.asyncio +async def test_completion_error_non_stream(): + """test finish_reason='error' returns 500 InternalServerError (non-streaming)""" + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + serving_completion = _build_serving_completion(mock_engine) + + completion_output = CompletionOutput( + index=0, + text="", + token_ids=[], + cumulative_logprob=None, + logprobs=None, + finish_reason="error", + ) + + request_output = RequestOutput( + request_id="test-id", + prompt="Test prompt", + prompt_token_ids=[1, 2, 3], + prompt_logprobs=None, + outputs=[completion_output], + finished=True, + metrics=None, + lora_request=None, + encoder_prompt=None, + encoder_prompt_token_ids=None, + ) + + async def mock_generate(*args, **kwargs): + yield request_output + + mock_engine.generate = MagicMock(side_effect=mock_generate) + + request = CompletionRequest( + model=MODEL_NAME, + prompt="Test prompt", + max_tokens=10, + stream=False, + ) + + with pytest.raises(GenerationError): + await serving_completion.create_completion(request) + + +@pytest.mark.asyncio +async def test_openai_completion_keeps_mm_cache_for_engine_execution(): + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + serving_completion = _build_serving_completion(mock_engine) + serving_completion.online_renderer.preprocess_completion = AsyncMock( + return_value=[{"prompt_token_ids": [1, 2, 3]}] + ) + + request = CompletionRequest( + model=MODEL_NAME, + prompt="Test prompt", + ) + + result = await serving_completion.render_completion_request(request) + + assert isinstance(result, list) + assert ( + serving_completion.online_renderer.preprocess_completion.call_args.kwargs[ + "skip_mm_cache" + ] + is False + ) + + +def _build_serving_render(engine: AsyncLLM) -> ServingRender: + models = OpenAIServingModels( + engine_client=engine, + base_model_paths=BASE_MODEL_PATHS, + ) + online_renderer = OnlineRenderer( + model_config=engine.model_config, + renderer=engine.renderer, + request_logger=None, + chat_template=None, + chat_template_content_format="auto", + ) + + serving_render = ServingRender(models, online_renderer) + + async def _fake_preprocess_chat(*args, **kwargs): + # return conversation, engine_inputs + return ( + [{"role": "user", "content": "Test"}], + [{"prompt_token_ids": [1, 2, 3]}], + ) + + serving_render.online_renderer.preprocess_chat = AsyncMock( + side_effect=_fake_preprocess_chat + ) + return serving_render + + +@pytest.mark.asyncio +async def test_renderer_only_completion_request_skips_mm_cache(): + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + serving_render = _build_serving_render(mock_engine) + + serving_render.online_renderer.preprocess_completion = AsyncMock( + return_value=[{"prompt_token_ids": [1, 2, 3]}] + ) + + request = CompletionRequest( + model=MODEL_NAME, + prompt="Test prompt", + ) + + result = await serving_render.render_completion_request(request) + + assert isinstance(result, list) + assert ( + serving_render.online_renderer.preprocess_completion.call_args.kwargs[ + "skip_mm_cache" + ] + is True + ) + + +@pytest.mark.asyncio +async def test_completion_error_stream(): + """test finish_reason='error' returns 500 InternalServerError (streaming)""" + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + serving_completion = _build_serving_completion(mock_engine) + + completion_output_1 = CompletionOutput( + index=0, + text="Hello", + token_ids=[100], + cumulative_logprob=None, + logprobs=None, + finish_reason=None, + ) + + request_output_1 = RequestOutput( + request_id="test-id", + prompt="Test prompt", + prompt_token_ids=[1, 2, 3], + prompt_logprobs=None, + outputs=[completion_output_1], + finished=False, + metrics=None, + lora_request=None, + encoder_prompt=None, + encoder_prompt_token_ids=None, + ) + + completion_output_2 = CompletionOutput( + index=0, + text="Hello", + token_ids=[100], + cumulative_logprob=None, + logprobs=None, + finish_reason="error", + ) + + request_output_2 = RequestOutput( + request_id="test-id", + prompt="Test prompt", + prompt_token_ids=[1, 2, 3], + prompt_logprobs=None, + outputs=[completion_output_2], + finished=True, + metrics=None, + lora_request=None, + encoder_prompt=None, + encoder_prompt_token_ids=None, + ) + + async def mock_generate(*args, **kwargs): + yield request_output_1 + yield request_output_2 + + mock_engine.generate = MagicMock(side_effect=mock_generate) + + request = CompletionRequest( + model=MODEL_NAME, + prompt="Test prompt", + max_tokens=10, + stream=True, + ) + + response = await serving_completion.create_completion(request) + + chunks = [] + async for chunk in response: + chunks.append(chunk) + + assert len(chunks) >= 2 + assert any("Internal server error" in chunk for chunk in chunks), ( + f"Expected error message in chunks: {chunks}" + ) + assert chunks[-1] == "data: [DONE]\n\n" + + +def test_json_schema_response_format_missing_schema(): + """When response_format type is 'json_schema' but the json_schema field + is not provided, request construction should raise a validation error + so the API returns 400 instead of 500.""" + with pytest.raises(Exception, match="json_schema.*must be provided"): + CompletionRequest( + model=MODEL_NAME, + prompt="Test prompt", + max_tokens=10, + response_format={"type": "json_schema"}, + ) + + +@pytest.mark.parametrize("format_value", [None, {}]) +def test_structural_tag_response_format_invalid(format_value): + """Malformed structural tags should be rejected during request validation.""" + with pytest.raises( + ValidationError, + match="Invalid response_format structural_tag", + ): + CompletionRequest( + model=MODEL_NAME, + prompt="Test prompt", + max_tokens=10, + response_format={"type": "structural_tag", "format": format_value}, + ) + + +@pytest.mark.parametrize("structural_tag", ["not json", ""]) +def test_structured_outputs_structural_tag_invalid(structural_tag): + """Malformed direct structured_outputs structural tags should be rejected.""" + with pytest.raises( + ValidationError, + match="Invalid structured_outputs structural_tag", + ): + CompletionRequest( + model=MODEL_NAME, + prompt="Test prompt", + max_tokens=10, + structured_outputs={"structural_tag": structural_tag}, + ) + + +def test_negative_prompt_token_ids_nested(): + """Negative token IDs in prompt (nested list) should raise validation error.""" + with pytest.raises(Exception, match="greater than or equal to 0"): + CompletionRequest( + model=MODEL_NAME, + prompt=[[-1]], + max_tokens=10, + ) + + +def test_negative_prompt_token_ids_flat(): + """Negative token IDs in prompt (flat list) should raise validation error.""" + with pytest.raises(Exception, match="greater than or equal to 0"): + CompletionRequest( + model=MODEL_NAME, + prompt=[-1], + max_tokens=10, + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_completion_with_prompt_embeds.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_completion_with_prompt_embeds.py new file mode 100644 index 0000000000000000000000000000000000000000..8beb6e5e81f3de19f81d99b981e77dfc62965f9e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_completion_with_prompt_embeds.py @@ -0,0 +1,304 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import io +import json + +import openai # use the official client for correctness check +import pybase64 as base64 +import pytest +import pytest_asyncio +import torch + +# downloading lora to test lora requests +from openai import BadRequestError +from transformers import AutoConfig + +from tests.utils import RemoteOpenAIServer + +# any model with a chat template should work here +MODEL_NAME = "facebook/opt-125m" +LORA_SERVING_MODEL_NAME = "opt125m-lora" + +CONFIG = AutoConfig.from_pretrained(MODEL_NAME) + + +@pytest.fixture(scope="module", params=["use-lora"]) +def default_server_args( + request: pytest.FixtureRequest, opt125_lora_files: str +) -> list[str]: + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-model-len", + "2048", + "--max-num-seqs", + "128", + "--enforce-eager", + # Prompt Embeds server args + "--enable-prompt-embeds", + ] + + if request.param == "use-lora": + lora_module_1 = { + "name": LORA_SERVING_MODEL_NAME, + "path": opt125_lora_files, + "base_model_name": MODEL_NAME, + } + + args.extend( + [ + "--enable-lora", + "--lora-module", + json.dumps(lora_module_1), + "--max-lora-rank", + "64", + "--max-cpu-loras", + "2", + ] + ) + + return args + + +EXAMPLE_PROMPTS = [ + "Hello, my name is", + "What is an LLM?", +] + + +def _encode_embeds(embeds: torch.Tensor): + buffer = io.BytesIO() + torch.save(embeds, buffer) + return base64.b64encode(buffer.getvalue()).decode("utf-8") + + +@pytest.fixture(scope="module") +def example_prompt_embeds(hf_runner): + """Create example embeddings and return them as base64 encoded string.""" + with hf_runner(MODEL_NAME) as hf_model: + example_embeddings = hf_model.get_prompt_embeddings(EXAMPLE_PROMPTS) + + return [_encode_embeds(item) for item in example_embeddings] + + +@pytest.fixture(scope="module") +def server_with_prompt_embeds(default_server_args): + with RemoteOpenAIServer(MODEL_NAME, default_server_args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client_with_prompt_embeds(server_with_prompt_embeds): + async with server_with_prompt_embeds.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME, LORA_SERVING_MODEL_NAME]) +async def test_completions_with_prompt_embeds( + example_prompt_embeds, + client_with_prompt_embeds: openai.AsyncOpenAI, + model_name: str, +): + encoded_embeds, encoded_embeds2 = example_prompt_embeds + + # Test case: Single prompt embeds input + completion = await client_with_prompt_embeds.completions.create( + model=model_name, + prompt=None, + max_tokens=5, + temperature=0.0, + extra_body={"prompt_embeds": encoded_embeds}, + ) + assert len(completion.choices[0].text) >= 1 + assert completion.choices[0].prompt_logprobs is None + + # Test case: batch completion with prompt_embeds + completion = await client_with_prompt_embeds.completions.create( + model=model_name, + prompt=None, + max_tokens=5, + temperature=0.0, + extra_body={"prompt_embeds": [encoded_embeds, encoded_embeds2]}, + ) + assert len(completion.choices) == 2 + assert len(completion.choices[0].text) >= 1 + assert len(completion.choices[1].text) >= 1 + + # Test case: streaming with prompt_embeds + single_completion = await client_with_prompt_embeds.completions.create( + model=model_name, + prompt=None, + max_tokens=5, + temperature=0.0, + extra_body={"prompt_embeds": encoded_embeds}, + ) + single_output = single_completion.choices[0].text + + stream = await client_with_prompt_embeds.completions.create( + model=model_name, + prompt=None, + max_tokens=5, + temperature=0.0, + stream=True, + extra_body={"prompt_embeds": encoded_embeds}, + ) + chunks = [] + finish_reason_count = 0 + async for chunk in stream: + chunks.append(chunk.choices[0].text) + if chunk.choices[0].finish_reason is not None: + finish_reason_count += 1 + assert finish_reason_count == 1 + assert chunk.choices[0].finish_reason == "length" + assert chunk.choices[0].text + assert "".join(chunks) == single_output + + # Test case: batch streaming with prompt_embeds + stream = await client_with_prompt_embeds.completions.create( + model=model_name, + prompt=None, + max_tokens=5, + temperature=0.0, + stream=True, + extra_body={"prompt_embeds": [encoded_embeds, encoded_embeds2]}, + ) + chunks_stream_embeds: list[list[str]] = [[], []] + finish_reason_count = 0 + async for chunk in stream: + chunks_stream_embeds[chunk.choices[0].index].append(chunk.choices[0].text) + if chunk.choices[0].finish_reason is not None: + finish_reason_count += 1 + assert finish_reason_count == 2 + assert chunk.choices[0].finish_reason == "length" + assert chunk.choices[0].text + assert len(chunks_stream_embeds[0]) > 0 + assert len(chunks_stream_embeds[1]) > 0 + + # Test case: mixed text and prompt_embeds + completion_mixed = await client_with_prompt_embeds.completions.create( + model=model_name, + prompt="This is a prompt", + max_tokens=5, + temperature=0.0, + extra_body={"prompt_embeds": encoded_embeds}, + ) + assert len(completion.choices) == 2 + completion_text_only = await client_with_prompt_embeds.completions.create( + model=model_name, + prompt="This is a prompt", + max_tokens=5, + temperature=0.0, + ) + completion_embeds_only = await client_with_prompt_embeds.completions.create( + model=model_name, + prompt=None, + max_tokens=5, + temperature=0.0, + extra_body={"prompt_embeds": encoded_embeds}, + ) + # Embeddings responses should be handled first + assert completion_mixed.choices[0].text == completion_embeds_only.choices[0].text + assert completion_mixed.choices[1].text == completion_text_only.choices[0].text + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME, LORA_SERVING_MODEL_NAME]) +async def test_completions_errors_with_prompt_embeds( + client_with_prompt_embeds: openai.AsyncOpenAI, model_name: str +): + # Test error case: invalid prompt_embeds + with pytest.raises(BadRequestError): + await client_with_prompt_embeds.completions.create( + prompt=None, + model=model_name, + max_tokens=5, + temperature=0.0, + extra_body={"prompt_embeds": "invalid_base64"}, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("logprobs_arg", [1, 0]) +@pytest.mark.parametrize("model_name", [MODEL_NAME, LORA_SERVING_MODEL_NAME]) +async def test_completions_with_logprobs_and_prompt_embeds( + example_prompt_embeds, + client_with_prompt_embeds: openai.AsyncOpenAI, + logprobs_arg: int, + model_name: str, +): + encoded_embeds, encoded_embeds2 = example_prompt_embeds + + # Test case: Logprobs using prompt_embeds + completion = await client_with_prompt_embeds.completions.create( + model=model_name, + prompt=None, + max_tokens=5, + temperature=0.0, + echo=False, + logprobs=logprobs_arg, + extra_body={"prompt_embeds": encoded_embeds}, + ) + + logprobs = completion.choices[0].logprobs + assert logprobs is not None + assert len(logprobs.text_offset) == 5 + assert len(logprobs.token_logprobs) == 5 + assert len(logprobs.top_logprobs) == 5 + for top_logprobs in logprobs.top_logprobs[1:]: + assert max(logprobs_arg, 1) <= len(top_logprobs) <= logprobs_arg + 1 + assert len(logprobs.tokens) == 5 + + # Test case: Log probs with batch completion and prompt_embeds + completion = await client_with_prompt_embeds.completions.create( + model=model_name, + prompt=None, + max_tokens=5, + temperature=0.0, + echo=False, + logprobs=logprobs_arg, + extra_body={"prompt_embeds": [encoded_embeds, encoded_embeds2]}, + ) + + assert len(completion.choices) == 2 + for choice in completion.choices: + logprobs = choice.logprobs + assert logprobs is not None + assert len(logprobs.text_offset) == 5 + assert len(logprobs.token_logprobs) == 5 + assert len(logprobs.top_logprobs) == 5 + for top_logprobs in logprobs.top_logprobs[1:]: + assert max(logprobs_arg, 1) <= len(top_logprobs) <= logprobs_arg + 1 + assert len(logprobs.tokens) == 5 + + +@pytest.mark.asyncio +async def test_prompt_logprobs_raises_error( + example_prompt_embeds, + client_with_prompt_embeds: openai.AsyncOpenAI, +): + encoded_embeds, _ = example_prompt_embeds + + with pytest.raises(BadRequestError, match="not compatible"): + await client_with_prompt_embeds.completions.create( + model=MODEL_NAME, + prompt=None, + max_tokens=5, + temperature=0.0, + extra_body={"prompt_embeds": encoded_embeds, "prompt_logprobs": True}, + ) + + +@pytest.mark.asyncio +async def test_empty_prompt_embeds( + client_with_prompt_embeds: openai.AsyncOpenAI, +) -> None: + await client_with_prompt_embeds.completions.create( + model=MODEL_NAME, + prompt="Hello", + max_tokens=5, + temperature=0.0, + extra_body={"prompt_embeds": []}, + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_lora_resolvers.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_lora_resolvers.py new file mode 100644 index 0000000000000000000000000000000000000000..d21c9d6675d3b210821a09d1993b11c36d358f6c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_lora_resolvers.py @@ -0,0 +1,255 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from contextlib import suppress +from dataclasses import dataclass, field +from http import HTTPStatus +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from vllm.config.multimodal import MultiModalConfig +from vllm.entrypoints.openai.completion.protocol import CompletionRequest +from vllm.entrypoints.openai.completion.serving import OpenAIServingCompletion +from vllm.entrypoints.openai.engine.protocol import ErrorResponse +from vllm.entrypoints.openai.models.protocol import BaseModelPath +from vllm.entrypoints.openai.models.serving import OpenAIServingModels +from vllm.lora.request import LoRARequest +from vllm.lora.resolver import LoRAResolver, LoRAResolverRegistry +from vllm.renderers.hf import HfRenderer +from vllm.renderers.online_renderer import OnlineRenderer +from vllm.tokenizers.registry import cached_tokenizer_from_config +from vllm.v1.engine.async_llm import AsyncLLM + +MODEL_NAME = "openai-community/gpt2" +BASE_MODEL_PATHS = [BaseModelPath(name=MODEL_NAME, model_path=MODEL_NAME)] + +MOCK_RESOLVER_NAME = "mock_test_resolver" + + +@dataclass +class MockHFConfig: + model_type: str = "any" + + +@dataclass +class MockModelConfig: + """Minimal mock ModelConfig for testing.""" + + model: str = MODEL_NAME + runner_type = "generate" + tokenizer: str = MODEL_NAME + trust_remote_code: bool = False + tokenizer_mode: str = "auto" + max_model_len: int = 100 + tokenizer_revision: str | None = None + multimodal_config: MultiModalConfig = field(default_factory=MultiModalConfig) + hf_config: MockHFConfig = field(default_factory=MockHFConfig) + logits_processors: list[str] | None = None + diff_sampling_param: dict | None = None + allowed_local_media_path: str = "" + allowed_media_domains: list[str] | None = None + encoder_config = None + generation_config: str = "auto" + skip_tokenizer_init: bool = False + is_encoder_decoder: bool = False + is_multimodal_model: bool = False + renderer_num_workers: int = 1 + + def get_diff_sampling_param(self): + return self.diff_sampling_param or {} + + +@dataclass +class MockParallelConfig: + _api_process_rank: int = 0 + + +@dataclass +class MockVllmConfig: + model_config: MockModelConfig + parallel_config: MockParallelConfig + + +class MockLoRAResolver(LoRAResolver): + async def resolve_lora( + self, base_model_name: str, lora_name: str + ) -> LoRARequest | None: + if lora_name == "test-lora": + return LoRARequest( + lora_name="test-lora", + lora_int_id=1, + lora_path="/fake/path/test-lora", + ) + elif lora_name == "invalid-lora": + return LoRARequest( + lora_name="invalid-lora", + lora_int_id=2, + lora_path="/fake/path/invalid-lora", + ) + return None + + +@pytest.fixture(autouse=True) +def register_mock_resolver(): + """Fixture to register and unregister the mock LoRA resolver.""" + resolver = MockLoRAResolver() + LoRAResolverRegistry.register_resolver(MOCK_RESOLVER_NAME, resolver) + yield + # Cleanup: remove the resolver after the test runs + if MOCK_RESOLVER_NAME in LoRAResolverRegistry.resolvers: + del LoRAResolverRegistry.resolvers[MOCK_RESOLVER_NAME] + + +def _build_renderer(model_config: MockModelConfig): + return HfRenderer( + MockVllmConfig(model_config, parallel_config=MockParallelConfig()), + cached_tokenizer_from_config(model_config), + ) + + +@pytest.fixture +def mock_serving_setup(): + """Provides a mocked engine and serving completion instance.""" + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + + async def mock_add_lora_side_effect(lora_request: LoRARequest): + """Simulate engine behavior when adding LoRAs.""" + if lora_request.lora_name == "test-lora": + # Simulate successful addition + return True + if lora_request.lora_name == "invalid-lora": + # Simulate failure during addition (e.g. invalid format) + raise ValueError(f"Simulated failure adding LoRA: {lora_request.lora_name}") + return True + + mock_engine.add_lora = AsyncMock(side_effect=mock_add_lora_side_effect) + + async def mock_generate(*args, **kwargs): + for _ in []: + yield _ + + mock_engine.generate = MagicMock(spec=AsyncLLM.generate, side_effect=mock_generate) + + mock_engine.generate.reset_mock() + mock_engine.add_lora.reset_mock() + + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = _build_renderer(mock_engine.model_config) + + models = OpenAIServingModels( + engine_client=mock_engine, + base_model_paths=BASE_MODEL_PATHS, + ) + + online_renderer = OnlineRenderer( + model_config=mock_engine.model_config, + renderer=mock_engine.renderer, + request_logger=None, + chat_template=None, + chat_template_content_format="auto", + ) + serving_completion = OpenAIServingCompletion( + mock_engine, models, online_renderer=online_renderer, request_logger=None + ) + + return mock_engine, serving_completion + + +@pytest.mark.asyncio +async def test_serving_completion_with_lora_resolver(mock_serving_setup, monkeypatch): + monkeypatch.setenv("VLLM_ALLOW_RUNTIME_LORA_UPDATING", "true") + + mock_engine, serving_completion = mock_serving_setup + + lora_model_name = "test-lora" + req_found = CompletionRequest( + model=lora_model_name, + prompt="Generate with LoRA", + ) + + # Suppress potential errors during the mocked generate call, + # as we are primarily checking for add_lora and generate calls + with suppress(Exception): + await serving_completion.create_completion(req_found) + + mock_engine.add_lora.assert_awaited_once() + called_lora_request = mock_engine.add_lora.call_args[0][0] + assert isinstance(called_lora_request, LoRARequest) + assert called_lora_request.lora_name == lora_model_name + + mock_engine.generate.assert_called_once() + called_lora_request = mock_engine.generate.call_args[1]["lora_request"] + assert isinstance(called_lora_request, LoRARequest) + assert called_lora_request.lora_name == lora_model_name + + +@pytest.mark.asyncio +async def test_serving_completion_resolver_not_found(mock_serving_setup, monkeypatch): + monkeypatch.setenv("VLLM_ALLOW_RUNTIME_LORA_UPDATING", "true") + + mock_engine, serving_completion = mock_serving_setup + + non_existent_model = "non-existent-lora-adapter" + req = CompletionRequest( + model=non_existent_model, + prompt="what is 1+1?", + ) + + response = await serving_completion.create_completion(req) + + mock_engine.add_lora.assert_not_awaited() + mock_engine.generate.assert_not_called() + + assert isinstance(response, ErrorResponse) + assert response.error.code == HTTPStatus.NOT_FOUND.value + assert non_existent_model in response.error.message + + +@pytest.mark.asyncio +async def test_serving_completion_resolver_add_lora_fails( + mock_serving_setup, monkeypatch +): + monkeypatch.setenv("VLLM_ALLOW_RUNTIME_LORA_UPDATING", "true") + + mock_engine, serving_completion = mock_serving_setup + + invalid_model = "invalid-lora" + req = CompletionRequest( + model=invalid_model, + prompt="what is 1+1?", + ) + + response = await serving_completion.create_completion(req) + + # Assert add_lora was called before the failure + mock_engine.add_lora.assert_awaited_once() + called_lora_request = mock_engine.add_lora.call_args[0][0] + assert isinstance(called_lora_request, LoRARequest) + assert called_lora_request.lora_name == invalid_model + + # Assert generate was *not* called due to the failure + mock_engine.generate.assert_not_called() + + # Assert the correct error response + assert isinstance(response, ErrorResponse) + assert response.error.code == HTTPStatus.BAD_REQUEST.value + assert invalid_model in response.error.message + + +@pytest.mark.asyncio +async def test_serving_completion_flag_not_set(mock_serving_setup): + mock_engine, serving_completion = mock_serving_setup + + lora_model_name = "test-lora" + req_found = CompletionRequest( + model=lora_model_name, + prompt="Generate with LoRA", + ) + + await serving_completion.create_completion(req_found) + + mock_engine.add_lora.assert_not_called() + mock_engine.generate.assert_not_called() diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_prompt_validation.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_prompt_validation.py new file mode 100644 index 0000000000000000000000000000000000000000..ac461fad6f323b83735c1933806d8c7016d0ed39 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_prompt_validation.py @@ -0,0 +1,115 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import io +from unittest.mock import Mock + +# imports for structured outputs tests +import openai +import pybase64 +import pytest +import regex as re +import torch + +from tests.utils import RemoteOpenAIServer +from vllm.config import ModelConfig +from vllm.renderers.embed_utils import safe_load_prompt_embeds + + +@pytest.mark.asyncio +async def test_empty_prompt(): + model_name = "gpt2" + server_args = ["--enforce-eager"] + with RemoteOpenAIServer(model_name, server_args) as remote_server: + client = remote_server.get_async_client() + + with pytest.raises( + openai.BadRequestError, + match="Either prompt or prompt_embeds must be provided and non-empty.", + ): + await client.completions.create( + model=model_name, + prompt=None, + max_tokens=5, + temperature=0.0, + extra_body={"prompt_embeds": []}, + ) + + +@pytest.mark.asyncio +async def test_out_of_vocab_token_ids(): + model_name = "gpt2" + server_args = ["--enforce-eager"] + with RemoteOpenAIServer(model_name, server_args) as remote_server: + client = remote_server.get_async_client() + + with pytest.raises( + openai.BadRequestError, match=re.compile(".*out of vocabulary.*").pattern + ): + await client.completions.create( + model=model_name, prompt=[999999], max_tokens=5, temperature=0.0 + ) + + +@pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16, torch.float16]) +@pytest.mark.parametrize( + "layout", [torch.strided, torch.sparse_coo, torch.sparse_csc, torch.sparse_csr] +) +@pytest.mark.parametrize("seq_len", [2, 10]) +@pytest.mark.parametrize("hidden_size", [2, 10]) +def test_load_prompt_embeds( + dtype: torch.dtype, layout: torch.layout, seq_len: int, hidden_size: int +): + model_config = Mock(spec=ModelConfig) + model_config.enable_prompt_embeds = True + model_config.get_hidden_size.return_value = hidden_size + model_config.dtype = dtype + + # construct arbitrary tensors of various dtypes, layouts, and sizes. + # We need to check against different layouts to make sure that if a user + # uses sparse tensors to reduce the transmission size of prompt embeddings, + # we must cast them to dense/strided before passing them into the engine. + # We don't use non-CPU tensors in this test to avoid preemptively + # initializing cuda and break other tests in the suite that fork processes. + # We also need to make sure that we only use devices that are actually + # available in the environment the test is running on. For simplicity, + # we just test against CPU. + tensor = torch.randn((seq_len, hidden_size), dtype=dtype) + if layout == torch.strided: + tensor = tensor.contiguous() + elif layout == torch.sparse_coo: + tensor = tensor.to_sparse_coo() + elif layout == torch.sparse_csc: + tensor = tensor.to_sparse_csc() + elif layout == torch.sparse_csr: + tensor = tensor.to_sparse_csr() + + buffer = io.BytesIO() + torch.save(tensor, buffer) + buffer.seek(0) + encoded_tensor = pybase64.b64encode(buffer.getvalue()) + + loaded_tensor = safe_load_prompt_embeds(model_config, encoded_tensor) + assert loaded_tensor.device.type == "cpu" + assert loaded_tensor.layout == torch.strided + torch.testing.assert_close( + loaded_tensor, tensor.to("cpu").to_dense(), equal_nan=True + ) + + +@pytest.mark.parametrize("dtype", [torch.float32]) +@pytest.mark.parametrize("seq_len", [2]) +@pytest.mark.parametrize("hidden_size", [2]) +def test_disable_prompt_embeds(dtype: torch.dtype, seq_len: int, hidden_size: int): + model_config = Mock(spec=ModelConfig) + model_config.enable_prompt_embeds = False + + tensor = torch.randn((seq_len, hidden_size), dtype=dtype) + + buffer = io.BytesIO() + torch.save(tensor, buffer) + buffer.seek(0) + encoded_tensor = pybase64.b64encode(buffer.getvalue()) + + with pytest.raises(ValueError, match="--enable-prompt-embeds"): + safe_load_prompt_embeds(model_config, encoded_tensor) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_shutdown.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_shutdown.py new file mode 100644 index 0000000000000000000000000000000000000000..02d3fc568715d271f4f1877617b203d928e7a6f8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_shutdown.py @@ -0,0 +1,570 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Integration tests for shutdown behavior, timeout, and signal handling.""" + +import asyncio +import signal +import subprocess +import sys +import time +from dataclasses import dataclass, field + +import httpx +import openai +import psutil +import pytest + +from tests.utils import RemoteOpenAIServer +from vllm.platforms import current_platform +from vllm.utils.network_utils import get_open_port + +MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM" + +# GPU initialization might take take longer +_IS_ROCM = current_platform.is_rocm() +_SERVER_STARTUP_TIMEOUT = 120 +_PROCESS_EXIT_TIMEOUT = 15 +_SHUTDOWN_DETECTION_TIMEOUT = 10 +_CHILD_CLEANUP_TIMEOUT = 10 +_INFLIGHT_REQUEST_START_TIMEOUT = 5 +_INFLIGHT_REQUEST_POLL_INTERVAL = 0.1 +_ABORT_CLIENT_TIMEOUT = 3 + + +def _get_child_pids(parent_pid: int) -> list[int]: + try: + parent = psutil.Process(parent_pid) + return [c.pid for c in parent.children(recursive=True)] + except psutil.NoSuchProcess: + return [] + + +async def _assert_children_cleaned_up( + child_pids: list[int], + timeout: float = _CHILD_CLEANUP_TIMEOUT, +): + """Wait for child processes to exit and fail if any remain.""" + if not child_pids: + return + + deadline = time.time() + timeout + while time.time() < deadline: + still_alive = [] + for pid in child_pids: + try: + p = psutil.Process(pid) + if p.is_running() and p.status() != psutil.STATUS_ZOMBIE: + still_alive.append(pid) + except psutil.NoSuchProcess: + pass + if not still_alive: + return + await asyncio.sleep(0.5) + + pytest.fail( + f"Child processes {still_alive} still alive after {timeout}s. " + f"Process cleanup may not be working correctly." + ) + + +@dataclass +class ShutdownState: + got_503: bool = False + got_500: bool = False + requests_after_sigterm: int = 0 + aborted_requests: int = 0 + connection_errors: int = 0 + inflight_requests: int = 0 + stop_requesting: bool = False + errors: list[str] = field(default_factory=list) + + +async def _concurrent_request_loop( + client: openai.AsyncOpenAI, + state: ShutdownState, + sigterm_sent: asyncio.Event | None = None, + concurrency: int = 10, +): + """Run multiple concurrent requests to keep the server busy.""" + + async def single_request(): + while not state.stop_requesting: + try: + state.inflight_requests += 1 + response = await client.completions.create( + model=MODEL_NAME, + prompt="Write a story: ", + max_tokens=200, + ) + if sigterm_sent is not None and sigterm_sent.is_set(): + state.requests_after_sigterm += 1 + # Check if any choice has finish_reason='abort' + if any(choice.finish_reason == "abort" for choice in response.choices): + state.aborted_requests += 1 + except openai.APIStatusError as e: + if e.status_code == 503: + state.got_503 = True + elif e.status_code == 500: + state.got_500 = True + else: + state.errors.append(f"API error: {e}") + except (openai.APIConnectionError, httpx.RemoteProtocolError): + state.connection_errors += 1 + if sigterm_sent is not None and sigterm_sent.is_set(): + break + except Exception as e: + state.errors.append(f"Unexpected error: {e}") + break + finally: + state.inflight_requests -= 1 + await asyncio.sleep(0.01) + + tasks = [asyncio.create_task(single_request()) for _ in range(concurrency)] + try: + await asyncio.gather(*tasks, return_exceptions=True) + finally: + for t in tasks: + if not t.done(): + t.cancel() + + +@pytest.mark.asyncio +async def test_shutdown_on_engine_failure(): + """Verify that API returns connection error when server process is killed. + + Starts a vLLM server, kills it to simulate a crash, then verifies that + subsequent API calls fail appropriately. + """ + + port = get_open_port() + + proc = subprocess.Popen( + [ + # dtype, max-len etc set so that this can run in CI + sys.executable, + "-m", + "vllm.entrypoints.openai.api_server", + "--model", + MODEL_NAME, + "--dtype", + "bfloat16", + "--max-model-len", + "128", + "--enforce-eager", + "--port", + str(port), + "--gpu-memory-utilization", + "0.05", + "--max-num-seqs", + "2", + ], + # ROCm: Disable stdout/stderr pipe capture. Subprocess hangs when + # stdout/stderr pipes are enabled during ROCm GPU initialization. + stdout=None if _IS_ROCM else subprocess.PIPE, + stderr=None if _IS_ROCM else subprocess.PIPE, + text=None if _IS_ROCM else True, + preexec_fn=lambda: signal.signal(signal.SIGINT, signal.SIG_IGN), + ) + + # Wait for server startup + start_time = time.time() + client = openai.AsyncOpenAI( + base_url=f"http://localhost:{port}/v1", + api_key="dummy", + max_retries=0, + timeout=10, + ) + + # Poll until server is ready + while time.time() - start_time < _SERVER_STARTUP_TIMEOUT: + try: + await client.completions.create( + model=MODEL_NAME, prompt="Hello", max_tokens=1 + ) + break + except Exception: + time.sleep(0.5) + if proc.poll() is not None: + if _IS_ROCM: + pytest.fail(f"Server died during startup: {proc.returncode}") + else: + stdout, stderr = proc.communicate(timeout=1) + pytest.fail( + f"Server died during startup. " + f"stdout: {stdout}, stderr: {stderr}" + ) + else: + proc.terminate() + proc.wait(timeout=_PROCESS_EXIT_TIMEOUT) + pytest.fail(f"Server failed to start in {_SERVER_STARTUP_TIMEOUT} seconds") + + # Kill server to simulate crash + proc.terminate() + time.sleep(1) + + # Verify API calls now fail + with pytest.raises((openai.APIConnectionError, openai.APIStatusError)): + await client.completions.create( + model=MODEL_NAME, prompt="This should fail", max_tokens=1 + ) + + return_code = proc.wait(timeout=_PROCESS_EXIT_TIMEOUT) + assert return_code is not None + + +@pytest.mark.asyncio +async def test_wait_timeout_completes_requests(): + """Verify wait timeout: new requests rejected, in-flight requests complete.""" + server_args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "256", + "--enforce-eager", + "--gpu-memory-utilization", + "0.05", + "--max-num-seqs", + "4", + "--shutdown-timeout", + "30", + ] + + with RemoteOpenAIServer(MODEL_NAME, server_args) as remote_server: + client = remote_server.get_async_client() + proc = remote_server.proc + child_pids = _get_child_pids(proc.pid) + + state = ShutdownState() + sigterm_sent = asyncio.Event() + + request_task = asyncio.create_task( + _concurrent_request_loop(client, state, sigterm_sent, concurrency=10) + ) + + await asyncio.sleep(0.5) + proc.send_signal(signal.SIGTERM) + sigterm_sent.set() + + try: + await asyncio.wait_for(request_task, timeout=_SHUTDOWN_DETECTION_TIMEOUT) + except asyncio.TimeoutError: + pass + finally: + state.stop_requesting = True + if not request_task.done(): + request_task.cancel() + await asyncio.gather(request_task, return_exceptions=True) + + # wait timeout should complete in-flight requests + assert state.requests_after_sigterm > 0, ( + f"Wait timeout should complete in-flight requests. " + f"503: {state.got_503}, 500: {state.got_500}, " + f"conn_errors: {state.connection_errors}, errors: {state.errors}" + ) + # server must stop accepting new requests (503, 500, or connection close) + assert state.got_503 or state.got_500 or state.connection_errors > 0, ( + f"Server should stop accepting requests. " + f"completed: {state.requests_after_sigterm}, errors: {state.errors}" + ) + + await _assert_children_cleaned_up(child_pids) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("wait_for_engine_idle", [0.0, 2.0]) +async def test_abort_timeout_exits_quickly(wait_for_engine_idle: float): + server_args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "256", + "--enforce-eager", + "--gpu-memory-utilization", + "0.05", + "--max-num-seqs", + "4", + "--shutdown-timeout", + "0", + ] + + with RemoteOpenAIServer(MODEL_NAME, server_args) as remote_server: + proc = remote_server.proc + child_pids = _get_child_pids(proc.pid) + + if wait_for_engine_idle > 0: + client = remote_server.get_async_client() + # Send requests to ensure engine is fully initialized + for _ in range(2): + await client.completions.create( + model=MODEL_NAME, + prompt="Test request: ", + max_tokens=10, + ) + # Wait for engine to become idle + await asyncio.sleep(wait_for_engine_idle) + + start_time = time.time() + proc.send_signal(signal.SIGTERM) + + # abort timeout (0) should stop the server promptly. + try: + proc.wait(timeout=4.0) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait(timeout=5) + pytest.fail("Process did not exit after SIGTERM with abort timeout") + + exit_time = time.time() - start_time + assert exit_time < 4.1, f"Default shutdown took too long: {exit_time:.1f}s" + assert proc.returncode in (0, -15, None), f"Unexpected: {proc.returncode}" + + await _assert_children_cleaned_up(child_pids) + + +@pytest.mark.asyncio +async def test_wait_timeout_with_short_duration(): + """Verify server exits cleanly with a short wait timeout.""" + wait_timeout = 3 + server_args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "256", + "--enforce-eager", + "--gpu-memory-utilization", + "0.05", + "--max-num-seqs", + "4", + "--shutdown-timeout", + str(wait_timeout), + ] + + with RemoteOpenAIServer(MODEL_NAME, server_args) as remote_server: + client = remote_server.get_async_client() + proc = remote_server.proc + child_pids = _get_child_pids(proc.pid) + + state = ShutdownState() + request_task = asyncio.create_task( + _concurrent_request_loop(client, state, concurrency=3) + ) + + await asyncio.sleep(0.5) + + start_time = time.time() + proc.send_signal(signal.SIGTERM) + + # server should exit within wait_timeout + buffer + max_wait = wait_timeout + 15 + for _ in range(int(max_wait * 10)): + if proc.poll() is not None: + break + time.sleep(0.1) + + exit_time = time.time() - start_time + + state.stop_requesting = True + if not request_task.done(): + request_task.cancel() + await asyncio.gather(request_task, return_exceptions=True) + + if proc.poll() is None: + proc.kill() + proc.wait(timeout=5) + pytest.fail(f"Process did not exit within {max_wait}s after SIGTERM") + + assert exit_time < wait_timeout + 10, ( + f"Took too long to exit ({exit_time:.1f}s), expected <{wait_timeout + 10}s" + ) + assert proc.returncode in (0, -15, None), f"Unexpected: {proc.returncode}" + + await _assert_children_cleaned_up(child_pids) + + +@pytest.mark.asyncio +async def test_abort_timeout_fails_inflight_requests(): + """Verify abort timeout (0) immediately aborts in-flight requests.""" + server_args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "256", + "--enforce-eager", + "--gpu-memory-utilization", + "0.05", + "--max-num-seqs", + "4", + "--shutdown-timeout", + "0", + ] + + with RemoteOpenAIServer(MODEL_NAME, server_args) as remote_server: + client = remote_server.get_async_client(timeout=_ABORT_CLIENT_TIMEOUT) + proc = remote_server.proc + child_pids = _get_child_pids(proc.pid) + + state = ShutdownState() + sigterm_sent = asyncio.Event() + + request_task = asyncio.create_task( + _concurrent_request_loop(client, state, sigterm_sent, concurrency=10) + ) + + deadline = time.time() + _INFLIGHT_REQUEST_START_TIMEOUT + while state.inflight_requests == 0 and time.time() < deadline: + await asyncio.sleep(_INFLIGHT_REQUEST_POLL_INTERVAL) + assert state.inflight_requests > 0 + + proc.send_signal(signal.SIGTERM) + sigterm_sent.set() + + try: + await asyncio.wait_for(request_task, timeout=5) + except asyncio.TimeoutError: + pass + finally: + state.stop_requesting = True + if not request_task.done(): + request_task.cancel() + await asyncio.gather(request_task, return_exceptions=True) + + # With abort timeout (0), requests should be aborted (finish_reason='abort') + # or rejected (connection errors or API errors) + assert ( + state.aborted_requests > 0 + or state.connection_errors > 0 + or state.got_500 + or state.got_503 + ), ( + f"Abort timeout should cause request aborts or failures. " + f"aborted: {state.aborted_requests}, " + f"503: {state.got_503}, 500: {state.got_500}, " + f"conn_errors: {state.connection_errors}, " + f"completed: {state.requests_after_sigterm}" + ) + + # Verify fast shutdown + start_time = time.time() + for _ in range(100): + if proc.poll() is not None: + break + time.sleep(0.1) + + exit_time = time.time() - start_time + assert exit_time < 10, f"Abort timeout shutdown took too long: {exit_time:.1f}s" + + await _assert_children_cleaned_up(child_pids) + + +@pytest.mark.asyncio +async def test_request_rejection_during_shutdown(): + """Verify new requests are rejected with error during shutdown.""" + server_args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "256", + "--enforce-eager", + "--gpu-memory-utilization", + "0.05", + "--max-num-seqs", + "4", + "--shutdown-timeout", + "30", + ] + + with RemoteOpenAIServer(MODEL_NAME, server_args) as remote_server: + client = remote_server.get_async_client() + proc = remote_server.proc + child_pids = _get_child_pids(proc.pid) + + proc.send_signal(signal.SIGTERM) + + await asyncio.sleep(1.0) + + # Try to send new requests - they should be rejected + rejected_count = 0 + for _ in range(10): + try: + await client.completions.create( + model=MODEL_NAME, prompt="Hello", max_tokens=10 + ) + except ( + openai.APIStatusError, + openai.APIConnectionError, + httpx.RemoteProtocolError, + ): + rejected_count += 1 + await asyncio.sleep(0.1) + + assert rejected_count > 0, ( + f"Expected requests to be rejected during shutdown, " + f"but {rejected_count} were rejected out of 10" + ) + + await _assert_children_cleaned_up(child_pids) + + +@pytest.mark.asyncio +async def test_multi_api_server_shutdown(): + """Verify shutdown works with multiple API servers.""" + server_args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "256", + "--enforce-eager", + "--gpu-memory-utilization", + "0.05", + "--max-num-seqs", + "4", + "--shutdown-timeout", + "30", + "--api-server-count", + "2", + ] + + with RemoteOpenAIServer(MODEL_NAME, server_args, auto_port=True) as remote_server: + client = remote_server.get_async_client() + proc = remote_server.proc + child_pids = _get_child_pids(proc.pid) + + assert len(child_pids) >= 2, ( + f"Expected at least 2 child processes, got {len(child_pids)}" + ) + + state = ShutdownState() + sigterm_sent = asyncio.Event() + + # Start concurrent requests across both API servers + request_task = asyncio.create_task( + _concurrent_request_loop(client, state, sigterm_sent, concurrency=8) + ) + + await asyncio.sleep(0.5) + + # Send SIGTERM to parent - should propagate to all children + proc.send_signal(signal.SIGTERM) + sigterm_sent.set() + + try: + await asyncio.wait_for(request_task, timeout=_SHUTDOWN_DETECTION_TIMEOUT) + except asyncio.TimeoutError: + pass + finally: + state.stop_requesting = True + if not request_task.done(): + request_task.cancel() + await asyncio.gather(request_task, return_exceptions=True) + + for _ in range(300): # up to 30 seconds + if proc.poll() is not None: + break + time.sleep(0.1) + + if proc.poll() is None: + proc.kill() + proc.wait(timeout=5) + pytest.fail("Process did not exit after SIGTERM") + + await _assert_children_cleaned_up(child_pids) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_tensorizer_entrypoint.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_tensorizer_entrypoint.py new file mode 100644 index 0000000000000000000000000000000000000000..e40bd41a07ef53c7bf56260434c12a458f75205b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_tensorizer_entrypoint.py @@ -0,0 +1,107 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import gc +import os +import tempfile + +import openai +import pytest +import pytest_asyncio +import torch.cuda + +from tests.utils import RemoteOpenAIServer +from vllm.engine.arg_utils import EngineArgs +from vllm.model_executor.model_loader.tensorizer import ( + TensorizerConfig, + tensorize_lora_adapter, + tensorize_vllm_model, +) +from vllm.platforms import current_platform + +MODEL_NAME = "unsloth/llama-3.2-1b-Instruct" +LORA_PATH = "davzoku/finqa_adapter_1b" + + +def _cleanup(): + gc.collect() + torch.accelerator.empty_cache() + + +@pytest.fixture(autouse=True) +def cleanup(): + _cleanup() + + +@pytest.fixture(scope="module") +def tmp_dir(): + with tempfile.TemporaryDirectory() as path: + yield path + + +@pytest.fixture(scope="module") +def model_uri(tmp_dir): + yield f"{tmp_dir}/model.tensors" + + +@pytest.fixture(scope="module") +def tensorize_model_and_lora(tmp_dir, model_uri): + tensorizer_config = TensorizerConfig(tensorizer_uri=model_uri, lora_dir=tmp_dir) + args = EngineArgs(model=MODEL_NAME) + + tensorize_lora_adapter(LORA_PATH, tensorizer_config) + tensorize_vllm_model(args, tensorizer_config) + + # Manually invoke a _cleanup() here, as the cleanup() + # fixture won't be guaranteed to be called after this + # when this fixture is used for a test + _cleanup() + yield + + +@pytest.fixture(scope="module") +def server(model_uri, tensorize_model_and_lora): + # In this case, model_uri is a directory with a model.tensors + # file and all necessary model artifacts, particularly a + # HF `config.json` file. In this case, Tensorizer can infer the + # `TensorizerConfig` so --model-loader-extra-config can be completely + # omitted. + + ## Start OpenAI API server + args = [ + "--load-format", + "tensorizer", + "--served-model-name", + MODEL_NAME, + "--enable-lora", + ] + if current_platform.is_rocm(): + args += ["--attention-backend", "TRITON_ATTN"] + + model_dir = os.path.dirname(model_uri) + with RemoteOpenAIServer(model_dir, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_single_completion(client: openai.AsyncOpenAI, model_name: str): + _cleanup() + completion = await client.completions.create( + model=model_name, prompt="Hello, my name is", max_tokens=5, temperature=0.0 + ) + + assert completion.id is not None + assert completion.choices is not None and len(completion.choices) == 1 + assert completion.model == MODEL_NAME + assert len(completion.choices) == 1 + assert len(completion.choices[0].text) >= 5 + assert completion.choices[0].finish_reason == "length" + assert completion.usage == openai.types.CompletionUsage( + completion_tokens=5, prompt_tokens=6, total_tokens=11 + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_token_in_token_out.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_token_in_token_out.py new file mode 100644 index 0000000000000000000000000000000000000000..b24426e96032537ea6f5fcaf349aa785ac26044f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/completion/test_token_in_token_out.py @@ -0,0 +1,73 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import os +import tempfile + +import pytest + +from tests.utils import RemoteOpenAIServer +from vllm.model_executor.model_loader.weight_utils import download_weights_from_hf +from vllm.tokenizers import get_tokenizer + +MODEL_NAME = "Qwen/Qwen3-0.6B" +MODEL_PATH = os.path.join(tempfile.gettempdir(), "qwen3_06b") + + +@pytest.fixture(scope="module") +def server(): + global MODEL_PATH + MODEL_PATH = download_weights_from_hf( + MODEL_NAME, + allow_patterns=["*"], + cache_dir=MODEL_PATH, + ignore_patterns=["tokenizer*", "vocab*", "*.safetensors"], + ) + args = [ + "--max-model-len", + "2048", + "--max-num-seqs", + "128", + "--enforce-eager", + "--skip-tokenizer-init", + "--load-format", + "dummy", + ] + with RemoteOpenAIServer(MODEL_PATH, args) as remote_server: + yield remote_server + + +@pytest.mark.asyncio +async def test_token_in_token_out_and_logprobs(server): + """ + Test token-in-token-out and token_ids align with prompt_logprobs + & logprobs when return_tokens_as_token_ids is enabled. + """ + tokenizer = get_tokenizer(tokenizer_name=MODEL_NAME) + text = "Hello, world! How are you today?" + token_ids = tokenizer.encode(text) + async with server.get_async_client() as client: + # Test with both return_token_ids and return_tokens_as_token_ids enabled + completion = await client.completions.create( + model=MODEL_PATH, + prompt=token_ids, + max_tokens=20, + temperature=0, + echo=True, + extra_body={ + "return_token_ids": True, + }, + ) + + # Verify all fields are present + assert ( + completion.choices[0].token_ids is not None + and 0 < len(completion.choices[0].token_ids) <= 20 + ) + assert completion.choices[0].prompt_token_ids is not None + + # Decode prompt tokens + if completion.choices[0].prompt_token_ids: + prompt_text = tokenizer.decode(completion.choices[0].prompt_token_ids) + # The decoded prompt should match or close to original prompt + assert prompt_text == text diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/correctness/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/correctness/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/correctness/test_lmeval.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/correctness/test_lmeval.py new file mode 100644 index 0000000000000000000000000000000000000000..aa8e08b2cdf6e737edb10b705411c2081192d94a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/correctness/test_lmeval.py @@ -0,0 +1,78 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +This file test accuracy of the vLLM server via LMEval. +It uses local-completions, which interacts with vLLM +through the OAI API with N concurrent connections. +This simulates real work usage of the API and makes +sure that the zmq frontend mp RPC message passing and +AsyncLLMEngine are working correctly. +""" + +import lm_eval + +from vllm.platforms import current_platform + +from ....utils import RemoteOpenAIServer + +MODEL_NAME = "Qwen/Qwen2-1.5B-Instruct" +NUM_CONCURRENT = 500 +TASK = "gsm8k" +FILTER = "exact_match,strict-match" +RTOL = 0.03 +EXPECTED_VALUE = 0.54 +DEFAULT_ARGS = ["--max-model-len", "4096"] +MORE_ARGS_LIST = [ + [], # Default + ["--enable-chunked-prefill"], # Chunked +] +MAX_WAIT_SECONDS = None + +if current_platform.is_tpu(): + MORE_ARGS_LIST = [ + [], # Default + ] + MAX_WAIT_SECONDS = 600 + + +def run_test(more_args): + """Run the end to end accuracy test.""" + + args = list(DEFAULT_ARGS) + args.extend(more_args) + print(f"Running with: {args}") + + with RemoteOpenAIServer( + MODEL_NAME, args, max_wait_seconds=MAX_WAIT_SECONDS + ) as remote_server: + url = f"{remote_server.url_for('v1')}/completions" + + model_args = ( + f"model={MODEL_NAME}," + f"base_url={url}," + f"num_concurrent={NUM_CONCURRENT},tokenized_requests=False" + ) + + results = lm_eval.simple_evaluate( + model="local-completions", + model_args=model_args, + tasks=TASK, + ) + + measured_value = results["results"][TASK][FILTER] + assert ( + measured_value - RTOL < EXPECTED_VALUE + and measured_value + RTOL > EXPECTED_VALUE + ), f"Expected: {EXPECTED_VALUE} | Measured: {measured_value}" + + +def test_lm_eval_accuracy_v1_engine(): + """Run with the V1 Engine.""" + + more_args = [] + + # Limit compilation time for V1 + if current_platform.is_tpu(): + more_args = ["--max-num-seqs", "64"] + + run_test(more_args) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/models/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/models/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/models/test_models.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/models/test_models.py new file mode 100644 index 0000000000000000000000000000000000000000..1d11c20df1c4e005379178fc2103b7940b2af8df --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/models/test_models.py @@ -0,0 +1,56 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import openai # use the official client for correctness check +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer + +# any model with a chat template should work here +MODEL_NAME = "Qwen/Qwen3-0.6B" +# technically this needs Mistral-7B-v0.1 as base, but we're not testing +# generation quality here + + +@pytest.fixture(scope="module") +def server(qwen3_lora_files): + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-model-len", + "8192", + "--enforce-eager", + # lora config below + "--enable-lora", + "--lora-modules", + f"qwen3-lora={qwen3_lora_files}", + "--max-lora-rank", + "64", + "--max-cpu-loras", + "2", + "--max-num-seqs", + "128", + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +async def test_check_models(client: openai.AsyncOpenAI, qwen3_lora_files): + models = await client.models.list() + models = models.data + served_model = models[0] + lora_models = models[1:] + assert served_model.id == MODEL_NAME + assert served_model.root == MODEL_NAME + assert all(lora_model.root == qwen3_lora_files for lora_model in lora_models) + assert lora_models[0].id == "qwen3-lora" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/parser/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/parser/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/parser/test_harmony_render_parity.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/parser/test_harmony_render_parity.py new file mode 100644 index 0000000000000000000000000000000000000000..2964fc179ef808ba4e85683ee670dac60213f660 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/parser/test_harmony_render_parity.py @@ -0,0 +1,611 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Cross-API render parity tests. + +Verifies that the chat completion input path (parse_chat_input_to_harmony_message) +and the responses API input path (response_input_to_harmony) produce identical +Harmony messages and identical rendered token sequences when given equivalent +conversation representations. + +The chat completion API encodes reasoning and tool calls as fields on a single +assistant message dict; the responses API encodes them as separate typed items +in request.input. Both paths must converge on the same Harmony message list and +therefore the same rendered prompt. + +Each test: + 1. Builds Harmony messages from each path for a single message or sequence. + 2. Asserts message-level properties (role, channel, recipient, content) + using verify_harmony_messages. + 3. Asserts that render_for_completion produces identical token sequences. +""" + +from openai.types.responses import ResponseFunctionToolCall + +from tests.entrypoints.openai.utils import verify_harmony_messages +from vllm.entrypoints.openai.parser.harmony_utils import ( + get_encoding, + get_system_message, + parse_chat_input_to_harmony_message, + render_for_completion, +) +from vllm.entrypoints.openai.responses.harmony import ( + response_input_to_harmony, + response_previous_input_to_harmony, +) + +# Use a fixed date so the system message is deterministic across both paths. +_DATE = "2025-01-01" + + +def _system(): + return get_system_message(start_date=_DATE) + + +class TestResponseInputToHarmonyRenderParity: + """Each test drives the same conversation through both APIs and asserts + identical Harmony messages and rendered token sequences.""" + + # ----------------------------------------------------------------------- + # Single-message cases + # ----------------------------------------------------------------------- + + def test_developer_message(self): + """Both APIs must render developer messages identically using + DeveloperContent (with the '# Instructions' header).""" + chat_msgs = parse_chat_input_to_harmony_message( + {"role": "developer", "content": "Be concise."} + ) + resp_msgs = [ + response_input_to_harmony( + { + "type": "message", + "role": "developer", + "content": "Be concise.", + }, + prev_responses=[], + ) + ] + + expected = [{"role": "developer", "instructions": "Be concise."}] + verify_harmony_messages(chat_msgs, expected) + verify_harmony_messages(resp_msgs, expected) + + assert render_for_completion([_system()] + chat_msgs) == render_for_completion( + [_system()] + resp_msgs + ) + + def test_user_message(self): + chat_msgs = parse_chat_input_to_harmony_message( + {"role": "user", "content": "What's the weather in Paris?"} + ) + resp_msgs = [ + response_input_to_harmony( + { + "type": "message", + "role": "user", + "content": "What's the weather in Paris?", + }, + prev_responses=[], + ) + ] + + expected = [{"role": "user", "content": "What's the weather in Paris?"}] + verify_harmony_messages(chat_msgs, expected) + verify_harmony_messages(resp_msgs, expected) + + assert render_for_completion([_system()] + chat_msgs) == render_for_completion( + [_system()] + resp_msgs + ) + + def test_assistant_final_message(self): + chat_msgs = parse_chat_input_to_harmony_message( + {"role": "assistant", "content": "It is 18°C in Paris."} + ) + resp_msgs = [ + response_input_to_harmony( + { + "type": "message", + "role": "assistant", + "content": "It is 18°C in Paris.", + }, + prev_responses=[], + ) + ] + + expected = [ + {"role": "assistant", "channel": "final", "content": "It is 18°C in Paris."} + ] + verify_harmony_messages(chat_msgs, expected) + verify_harmony_messages(resp_msgs, expected) + + assert render_for_completion([_system()] + chat_msgs) == render_for_completion( + [_system()] + resp_msgs + ) + + def test_reasoning_item(self): + # Chat path: assistant message with only a reasoning field and no content. + chat_msgs = parse_chat_input_to_harmony_message( + { + "role": "assistant", + "reasoning": "I should call get_weather.", + "content": "", + } + ) + resp_msgs = [ + response_input_to_harmony( + { + "type": "reasoning", + "content": [ + {"type": "reasoning_text", "text": "I should call get_weather."} + ], + }, + prev_responses=[], + ) + ] + + expected = [ + { + "role": "assistant", + "channel": "analysis", + "content": "I should call get_weather.", + } + ] + verify_harmony_messages(chat_msgs, expected) + verify_harmony_messages(resp_msgs, expected) + + assert render_for_completion([_system()] + chat_msgs) == render_for_completion( + [_system()] + resp_msgs + ) + + def test_function_call(self): + chat_msgs = parse_chat_input_to_harmony_message( + { + "role": "assistant", + "tool_calls": [ + { + "id": "call_1", + "function": { + "name": "get_weather", + "arguments": '{"location": "Paris"}', + }, + } + ], + } + ) + resp_msgs = [ + response_input_to_harmony( + { + "type": "function_call", + "name": "get_weather", + "arguments": '{"location": "Paris"}', + }, + prev_responses=[], + ) + ] + + expected = [ + { + "role": "assistant", + "channel": "commentary", + "recipient": "functions.get_weather", + "content": '{"location": "Paris"}', + "content_type": "json", + } + ] + verify_harmony_messages(chat_msgs, expected) + verify_harmony_messages(resp_msgs, expected) + + assert render_for_completion([_system()] + chat_msgs) == render_for_completion( + [_system()] + resp_msgs + ) + + def test_tool_output(self): + prev_call = ResponseFunctionToolCall( + id="fc_1", + call_id="call_1", + name="get_weather", + arguments='{"location": "Paris"}', + type="function_call", + ) + + chat_msgs = parse_chat_input_to_harmony_message( + {"role": "tool", "tool_call_id": "call_1", "content": "18°C, clear skies."}, + tool_id_names={"call_1": "get_weather"}, + ) + resp_msgs = [ + response_input_to_harmony( + { + "type": "function_call_output", + "call_id": "call_1", + "output": "18°C, clear skies.", + }, + prev_responses=[prev_call], + ) + ] + + expected = [ + { + "role": "tool", + "author_name": "functions.get_weather", + "channel": "commentary", + "recipient": "assistant", + "content": "18°C, clear skies.", + } + ] + verify_harmony_messages(chat_msgs, expected) + verify_harmony_messages(resp_msgs, expected) + + assert render_for_completion([_system()] + chat_msgs) == render_for_completion( + [_system()] + resp_msgs + ) + + # ----------------------------------------------------------------------- + # Combined and multi-turn cases + # ----------------------------------------------------------------------- + + def test_reasoning_combined_with_function_call(self): + """Chat API packs reasoning + tool_calls into one dict; responses API + represents them as two separate items. Both must produce the same two + Harmony messages in the same order: analysis then commentary.""" + chat_msgs = parse_chat_input_to_harmony_message( + { + "role": "assistant", + "reasoning": "I should get the weather for Paris.", + "tool_calls": [ + { + "id": "call_1", + "function": { + "name": "get_weather", + "arguments": '{"location": "Paris"}', + }, + } + ], + } + ) + resp_msgs = [ + response_input_to_harmony( + { + "type": "reasoning", + "content": [ + { + "type": "reasoning_text", + "text": "I should get the weather for Paris.", + } + ], + }, + prev_responses=[], + ), + response_input_to_harmony( + { + "type": "function_call", + "name": "get_weather", + "arguments": '{"location": "Paris"}', + }, + prev_responses=[], + ), + ] + + expected = [ + { + "role": "assistant", + "channel": "analysis", + "content": "I should get the weather for Paris.", + }, + { + "role": "assistant", + "channel": "commentary", + "recipient": "functions.get_weather", + "content": '{"location": "Paris"}', + "content_type": "json", + }, + ] + verify_harmony_messages(chat_msgs, expected) + verify_harmony_messages(resp_msgs, expected) + + assert render_for_completion([_system()] + chat_msgs) == render_for_completion( + [_system()] + resp_msgs + ) + + def test_full_multi_turn_tool_call_conversation(self): + """Full conversation: user -> reasoning + tool_call -> tool_output -> final. + + Both APIs must render the complete conversation to identical token sequences. + This exercises the entire input pipeline including all message types and + the Rust harmony encoder. + """ + prev_call = ResponseFunctionToolCall( + id="fc_1", + call_id="call_1", + name="get_weather", + arguments='{"location": "Paris"}', + type="function_call", + ) + + # --- Chat completion API path --- + tool_id_names = {"call_1": "get_weather"} + chat_msgs = [] + chat_msgs += parse_chat_input_to_harmony_message( + {"role": "user", "content": "What's the weather in Paris?"} + ) + chat_msgs += parse_chat_input_to_harmony_message( + { + "role": "assistant", + "reasoning": "I should call get_weather for Paris.", + "tool_calls": [ + { + "id": "call_1", + "function": { + "name": "get_weather", + "arguments": '{"location": "Paris"}', + }, + } + ], + } + ) + chat_msgs += parse_chat_input_to_harmony_message( + {"role": "tool", "tool_call_id": "call_1", "content": "18°C, clear skies."}, + tool_id_names=tool_id_names, + ) + chat_msgs += parse_chat_input_to_harmony_message( + { + "role": "assistant", + "content": "It is currently 18°C in Paris with clear skies.", + } + ) + + # --- Responses API path --- + resp_input = [ + { + "type": "message", + "role": "user", + "content": "What's the weather in Paris?", + }, + { + "type": "reasoning", + "content": [ + { + "type": "reasoning_text", + "text": "I should call get_weather for Paris.", + } + ], + }, + { + "type": "function_call", + "name": "get_weather", + "arguments": '{"location": "Paris"}', + }, + { + "type": "function_call_output", + "call_id": "call_1", + "output": "18°C, clear skies.", + }, + { + "type": "message", + "role": "assistant", + "content": "It is currently 18°C in Paris with clear skies.", + }, + ] + resp_msgs = [ + response_input_to_harmony(item, prev_responses=[prev_call]) + for item in resp_input + ] + + assert render_for_completion([_system()] + chat_msgs) == render_for_completion( + [_system()] + resp_msgs + ) + + def test_multi_turn_two_tool_calls_with_reasoning_between(self): + """Validates parity for a chain of two tool calls, each with its own + reasoning trace. Reasoning traces in between commentary-channel tool + calls must survive as analysis-channel messages in both paths. + """ + first_reasoning = "I need current weather first." + second_reasoning = "Now I need the weekly forecast." + + prev_call_1 = ResponseFunctionToolCall( + id="fc_1", + call_id="call_1", + name="get_weather", + arguments='{"location": "Paris"}', + type="function_call", + ) + prev_call_2 = ResponseFunctionToolCall( + id="fc_2", + call_id="call_2", + name="get_forecast", + arguments='{"location": "Paris", "days": 7}', + type="function_call", + ) + + # --- Chat completion API path --- + tool_id_names = {"call_1": "get_weather", "call_2": "get_forecast"} + chat_msgs = [] + chat_msgs += parse_chat_input_to_harmony_message( + {"role": "user", "content": "What's the weather and forecast for Paris?"} + ) + # First reasoning + tool call + chat_msgs += parse_chat_input_to_harmony_message( + { + "role": "assistant", + "reasoning": first_reasoning, + "tool_calls": [ + { + "id": "call_1", + "function": { + "name": "get_weather", + "arguments": '{"location": "Paris"}', + }, + } + ], + } + ) + chat_msgs += parse_chat_input_to_harmony_message( + {"role": "tool", "tool_call_id": "call_1", "content": "18°C, clear skies."}, + tool_id_names=tool_id_names, + ) + # Second reasoning + tool call + chat_msgs += parse_chat_input_to_harmony_message( + { + "role": "assistant", + "reasoning": second_reasoning, + "tool_calls": [ + { + "id": "call_2", + "function": { + "name": "get_forecast", + "arguments": '{"location": "Paris", "days": 7}', + }, + } + ], + } + ) + chat_msgs += parse_chat_input_to_harmony_message( + { + "role": "tool", + "tool_call_id": "call_2", + "content": "Mon 17°C, Tue 19°C, Wed 16°C", + }, + tool_id_names=tool_id_names, + ) + + # --- Responses API path --- + prev_responses = [prev_call_1, prev_call_2] + resp_input = [ + { + "type": "message", + "role": "user", + "content": "What's the weather and forecast for Paris?", + }, + # First reasoning + tool call + { + "type": "reasoning", + "content": [{"type": "reasoning_text", "text": first_reasoning}], + }, + { + "type": "function_call", + "name": "get_weather", + "arguments": '{"location": "Paris"}', + }, + { + "type": "function_call_output", + "call_id": "call_1", + "output": "18°C, clear skies.", + }, + # Second reasoning + tool call + { + "type": "reasoning", + "content": [ + { + "type": "reasoning_text", + "text": second_reasoning, + } + ], + }, + { + "type": "function_call", + "name": "get_forecast", + "arguments": '{"location": "Paris", "days": 7}', + }, + { + "type": "function_call_output", + "call_id": "call_2", + "output": "Mon 17°C, Tue 19°C, Wed 16°C", + }, + ] + resp_msgs = [ + response_input_to_harmony(item, prev_responses=prev_responses) + for item in resp_input + ] + + chat_completion_tokens = render_for_completion([_system()] + chat_msgs) + responses_tokens = render_for_completion([_system()] + resp_msgs) + + assert chat_completion_tokens == responses_tokens + + rendered_prompt = get_encoding().decode(chat_completion_tokens) + assert first_reasoning in rendered_prompt + assert second_reasoning in rendered_prompt + + def test_completed_turns_drop_reasoning(self): + """Validates that reasoning from completed turns is dropped, while + reasoning from the current in-progress tool-call turn is preserved + in both chat completions and responses previous_input_messages.""" + first_turn_reasoning = "FIRST_TURN_REASONING" + second_turn_reasoning = "SECOND_TURN_REASONING" + + chat_completion_msgs = [] + for chat_message in [ + {"role": "user", "content": "What is 2+2?"}, + { + "role": "assistant", + "reasoning": first_turn_reasoning, + "content": "The answer is 4.", + }, + {"role": "user", "content": "Now what is 3+3?"}, + { + "role": "assistant", + "reasoning": second_turn_reasoning, + "tool_calls": [ + { + "id": "call_1", + "function": { + "name": "calc", + "arguments": '{"a":3,"b":3}', + }, + } + ], + }, + ]: + chat_completion_msgs.extend( + parse_chat_input_to_harmony_message(chat_message) + ) + + responses_prev_input_msgs = [] + for responses_message in [ + { + "author": {"role": "user"}, + "content": [{"type": "text", "text": "What is 2+2?"}], + }, + { + "author": {"role": "assistant"}, + "channel": "analysis", + "content": [{"type": "text", "text": first_turn_reasoning}], + }, + { + "author": {"role": "assistant"}, + "channel": "final", + "content": [{"type": "text", "text": "The answer is 4."}], + }, + { + "author": {"role": "user"}, + "content": [{"type": "text", "text": "Now what is 3+3?"}], + }, + { + "author": {"role": "assistant"}, + "channel": "analysis", + "content": [{"type": "text", "text": second_turn_reasoning}], + }, + { + "author": {"role": "assistant"}, + "channel": "commentary", + "recipient": "functions.calc", + "content_type": "json", + "content": [{"type": "text", "text": '{"a":3,"b":3}'}], + }, + ]: + responses_prev_input_msgs.extend( + response_previous_input_to_harmony(responses_message) + ) + + chat_completion_tokens = render_for_completion( + [_system()] + chat_completion_msgs + ) + responses_tokens = render_for_completion( + [_system()] + responses_prev_input_msgs + ) + + assert chat_completion_tokens == responses_tokens + + rendered_prompt = get_encoding().decode(responses_tokens) + assert first_turn_reasoning not in rendered_prompt + assert second_turn_reasoning in rendered_prompt diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/parser/test_harmony_utils.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/parser/test_harmony_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..0f6b76c2478d1590c3cbce5d0d87da1890351574 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/parser/test_harmony_utils.py @@ -0,0 +1,1067 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +from openai.types.responses import FunctionTool +from openai_harmony import DeveloperContent, Message, Role + +from tests.entrypoints.openai.utils import verify_harmony_messages +from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionToolsParam +from vllm.entrypoints.openai.parser.harmony_utils import ( + auto_drop_analysis_messages, + create_tool_definition, + extract_function_from_recipient, + get_system_message, + has_custom_tools, + is_function_recipient, + parse_chat_input_to_harmony_message, +) +from vllm.entrypoints.openai.responses.harmony import ( + response_input_to_harmony, + response_previous_input_to_harmony, +) + +_TOOL_PARAMETERS = { + "type": "object", + "properties": {"status": {"type": "string"}}, + "required": ["status"], + "additionalProperties": False, +} + + +class TestCreateToolDefinition: + def test_chat_completion_omitted_description_defaults_to_empty_string(self): + tool = ChatCompletionToolsParam( + function={ + "name": "report_status", + "parameters": _TOOL_PARAMETERS, + } + ) + + tool_definition = create_tool_definition(tool) + + assert tool_definition.name == "report_status" + assert tool_definition.description == "" + assert tool_definition.parameters == _TOOL_PARAMETERS + + def test_chat_completion_none_description_defaults_to_empty_string(self): + tool = ChatCompletionToolsParam( + function={ + "name": "report_status", + "description": None, + "parameters": _TOOL_PARAMETERS, + } + ) + + tool_definition = create_tool_definition(tool) + + assert tool_definition.name == "report_status" + assert tool_definition.description == "" + assert tool_definition.parameters == _TOOL_PARAMETERS + + def test_response_tool_none_description_defaults_to_empty_string(self): + tool = FunctionTool( + name="report_status", + description=None, + parameters=_TOOL_PARAMETERS, + type="function", + ) + + tool_definition = create_tool_definition(tool) + + assert tool_definition.name == "report_status" + assert tool_definition.description == "" + assert tool_definition.parameters == _TOOL_PARAMETERS + + +class TestIsFunctionRecipient: + @pytest.mark.parametrize( + "recipient", + [ + "functions.get_weather", + "functions.search_web", + "functions.math.sum", + ], + ) + def test_functions_prefix_accepted(self, recipient): + assert is_function_recipient(recipient) is True + + @pytest.mark.parametrize( + "recipient", + [ + "get_weather", + "search_web", + "calculator", + "my-tool", + ], + ) + def test_bare_function_name_accepted(self, recipient): + assert is_function_recipient(recipient) is True + + @pytest.mark.parametrize( + "recipient", + [ + "assistant", + ], + ) + def test_assistant_rejected(self, recipient): + assert is_function_recipient(recipient) is False + + @pytest.mark.parametrize( + "recipient", + [ + "math.sum", + "code.run", + "namespace.tool_name", + "my.deeply.nested.tool", + ], + ) + def test_dotted_function_names_accepted(self, recipient): + assert is_function_recipient(recipient) is True + + @pytest.mark.parametrize( + "recipient", + [ + "python", + "browser", + "container", + ], + ) + def test_builtin_tool_names_rejected(self, recipient): + assert is_function_recipient(recipient) is False + + @pytest.mark.parametrize( + "recipient", + [ + "python.run", + "python.execute", + "browser.search", + "browser.open", + "container.exec", + ], + ) + def test_builtin_dotted_variants_rejected(self, recipient): + assert is_function_recipient(recipient) is False + + @pytest.mark.parametrize( + "recipient", + [ + "", + "functions.", + ], + ) + def test_empty_recipients_rejected(self, recipient): + assert is_function_recipient(recipient) is False + + @pytest.mark.parametrize( + "recipient", + [ + "<|start|>", + "<|end|>", + "<|channel|>", + ], + ) + def test_harmony_tokens_rejected(self, recipient): + assert is_function_recipient(recipient) is False + + +class TestIsFunctionRecipientWithAllowedNames: + """Tests for is_function_recipient with allowed_function_tool_names.""" + + def test_prefixed_always_accepted(self): + """functions. prefix is always accepted regardless of allowed names.""" + fn_names = frozenset({"other_tool"}) + assert is_function_recipient("functions.get_weather", fn_names) is True + + def test_bare_name_accepted_when_in_allowed_names(self): + fn_names = frozenset({"get_weather", "search_web"}) + assert is_function_recipient("get_weather", fn_names) is True + assert is_function_recipient("search_web", fn_names) is True + + def test_bare_name_rejected_when_not_in_allowed_names(self): + fn_names = frozenset({"get_weather"}) + assert is_function_recipient("unknown_tool", fn_names) is False + + def test_dotted_name_accepted_when_in_allowed_names(self): + fn_names = frozenset({"math.sum", "namespace.tool_name"}) + assert is_function_recipient("math.sum", fn_names) is True + assert is_function_recipient("namespace.tool_name", fn_names) is True + + def test_dotted_name_rejected_when_not_in_allowed_names(self): + fn_names = frozenset({"get_weather"}) + assert is_function_recipient("custom_server.search", fn_names) is False + + def test_empty_allowed_names_rejects_bare_names(self): + """Empty frozenset means no function tools — bare names are not functions.""" + fn_names: frozenset[str] = frozenset() + assert is_function_recipient("get_weather", fn_names) is False + assert is_function_recipient("math.sum", fn_names) is False + + def test_builtin_tools_always_rejected(self): + fn_names = frozenset({"python", "browser", "container"}) + assert is_function_recipient("python", fn_names) is False + assert is_function_recipient("browser", fn_names) is False + assert is_function_recipient("container", fn_names) is False + + def test_builtin_dotted_always_rejected(self): + fn_names = frozenset({"python.run", "browser.search"}) + assert is_function_recipient("python.run", fn_names) is False + assert is_function_recipient("browser.search", fn_names) is False + + def test_none_allowed_names_uses_heuristic(self): + """When allowed names is None (Chat Completions), use heuristic.""" + assert is_function_recipient("get_weather", None) is True + assert is_function_recipient("math.sum", None) is True + assert is_function_recipient("python", None) is False + + +class TestExtractFunctionFromRecipient: + @pytest.mark.parametrize( + "recipient,expected", + [ + ("functions.get_weather", "get_weather"), + ("functions.search_web", "search_web"), + ("functions.", ""), + ], + ) + def test_strips_functions_prefix(self, recipient, expected): + assert extract_function_from_recipient(recipient) == expected + + @pytest.mark.parametrize( + "recipient", + [ + "get_weather", + "calculator", + "my-tool", + ], + ) + def test_bare_name_returned_as_is(self, recipient): + assert extract_function_from_recipient(recipient) == recipient + + @pytest.mark.parametrize( + "recipient,expected", + [ + ("functions.math.sum", "math.sum"), + ("math.sum", "math.sum"), + ("namespace.tool_name", "namespace.tool_name"), + ], + ) + def test_dotted_function_name_extraction(self, recipient, expected): + assert extract_function_from_recipient(recipient) == expected + + +class TestCommonParseInputToHarmonyMessage: + """ + Tests for scenarios that are common to both Chat Completion + parse_chat_input_to_harmony_message and Responses API + response_previous_input_to_harmony functions. + """ + + @pytest.fixture( + params=[parse_chat_input_to_harmony_message, response_previous_input_to_harmony] + ) + def parse_function(self, request): + return request.param + + def test_assistant_message_with_tool_calls(self, parse_function): + """Test parsing assistant message with tool calls.""" + chat_msg = { + "role": "assistant", + "tool_calls": [ + { + "function": { + "name": "get_weather", + "arguments": '{"location": "San Francisco"}', + } + }, + { + "function": { + "name": "search_web", + "arguments": '{"query": "latest news"}', + } + }, + ], + } + + messages = parse_function(chat_msg) + + assert len(messages) == 2 + + # First tool call + assert messages[0].author.role == Role.ASSISTANT + assert messages[0].content[0].text == '{"location": "San Francisco"}' + assert messages[0].channel == "commentary" + assert messages[0].recipient == "functions.get_weather" + assert messages[0].content_type == "json" + + # Second tool call + assert messages[1].author.role == Role.ASSISTANT + assert messages[1].content[0].text == '{"query": "latest news"}' + assert messages[1].channel == "commentary" + assert messages[1].recipient == "functions.search_web" + assert messages[1].content_type == "json" + + def test_assistant_message_with_empty_tool_call_arguments(self, parse_function): + """Test parsing assistant message with tool call having None arguments.""" + chat_msg = { + "role": "assistant", + "tool_calls": [ + { + "function": { + "name": "get_current_time", + "arguments": None, + } + } + ], + } + + messages = parse_function(chat_msg) + + assert len(messages) == 1 + assert messages[0].content[0].text == "" + assert messages[0].recipient == "functions.get_current_time" + + def test_system_message(self, parse_function): + """Test parsing system messages, which are parsed into developer messages + with DeveloperContent.""" + chat_msg = { + "role": "system", + "content": "You are a helpful assistant", + } + + messages = parse_function(chat_msg) + + assert len(messages) == 1 + assert messages[0].author.role == Role.DEVELOPER + assert isinstance(messages[0].content[0], DeveloperContent) + assert messages[0].content[0].instructions == "You are a helpful assistant" + + def test_developer_message(self, parse_function): + """Test parsing developer message.""" + chat_msg = { + "role": "developer", + "content": "Use concise language", + } + + messages = parse_function(chat_msg) + + assert len(messages) == 1 + assert messages[0].author.role == Role.DEVELOPER + assert isinstance(messages[0].content[0], DeveloperContent) + assert messages[0].content[0].instructions == "Use concise language" + + def test_user_message_with_string_content(self, parse_function): + """Test parsing user message with string content.""" + chat_msg = { + "role": "user", + "content": "What's the weather in San Francisco?", + } + + messages = parse_function(chat_msg) + + assert len(messages) == 1 + assert messages[0].author.role == Role.USER + assert messages[0].content[0].text == "What's the weather in San Francisco?" + + def test_user_message_with_array_content(self, parse_function): + """Test parsing user message with array content.""" + chat_msg = { + "role": "user", + "content": [ + {"text": "What's in this image? "}, + {"text": "Please describe it."}, + ], + } + + messages = parse_function(chat_msg) + + assert len(messages) == 1 + assert messages[0].author.role == Role.USER + assert len(messages[0].content) == 2 + assert messages[0].content[0].text == "What's in this image? " + assert messages[0].content[1].text == "Please describe it." + + def test_assistant_message_with_string_content(self, parse_function): + """Test parsing assistant message with string content (no tool calls).""" + chat_msg = { + "role": "assistant", + "content": "Hello! How can I help you today?", + } + + messages = parse_function(chat_msg) + + assert len(messages) == 1 + assert messages[0].author.role == Role.ASSISTANT + assert messages[0].content[0].text == "Hello! How can I help you today?" + + def test_pydantic_model_input(self, parse_function): + """Test parsing Pydantic model input (has model_dump method).""" + + class MockPydanticModel: + def model_dump(self, exclude_none=True): + return { + "role": "user", + "content": "Test message", + } + + chat_msg = MockPydanticModel() + messages = parse_function(chat_msg) + + assert len(messages) == 1 + assert messages[0].author.role == Role.USER + assert messages[0].content[0].text == "Test message" + + def test_tool_call_with_missing_function_fields(self, parse_function): + """Test parsing tool call with missing name or arguments.""" + chat_msg = { + "role": "assistant", + "tool_calls": [ + { + "function": {} # Missing both name and arguments + } + ], + } + + messages = parse_function(chat_msg) + + assert len(messages) == 1 + assert messages[0].recipient == "functions." + assert messages[0].content[0].text == "" + + def test_array_content_with_missing_text(self, parse_function): + """Test parsing array content where text field is missing.""" + chat_msg = { + "role": "user", + "content": [ + {}, # Missing text field + {"text": "actual text"}, + ], + } + + messages = parse_function(chat_msg) + + assert len(messages) == 1 + assert len(messages[0].content) == 2 + assert messages[0].content[0].text == "" + assert messages[0].content[1].text == "actual text" + + +class TestParseChatInputToHarmonyMessage: + """ + Tests for scenarios that are specific to the Chat Completion API + parse_chat_input_to_harmony_message function. + """ + + def test_user_message_with_empty_content(self): + chat_msg = { + "role": "user", + "content": "", + } + + messages = parse_chat_input_to_harmony_message(chat_msg) + + verify_harmony_messages( + messages, + [ + { + "role": "user", + "content": "", + }, + ], + ) + + def test_user_message_with_none_content(self): + chat_msg = { + "role": "user", + "content": None, + } + + messages = parse_chat_input_to_harmony_message(chat_msg) + + verify_harmony_messages( + messages, + [ + { + "role": "user", + "content": "", + }, + ], + ) + + def test_assistant_message_with_empty_content(self): + chat_msg = { + "role": "assistant", + "content": "", + } + + messages = parse_chat_input_to_harmony_message(chat_msg) + + assert len(messages) == 0 + + def test_assistant_message_with_none_content(self): + chat_msg = { + "role": "assistant", + "content": None, + } + + messages = parse_chat_input_to_harmony_message(chat_msg) + + assert len(messages) == 0 + + def test_assistant_message_with_content_but_empty_reasoning(self): + chat_msg = { + "role": "assistant", + "content": "The answer is 4.", + "reasoning": "", + } + + messages = parse_chat_input_to_harmony_message(chat_msg) + + verify_harmony_messages( + messages, + [ + { + "role": "assistant", + "channel": "final", + "content": "The answer is 4.", + }, + ], + ) + + def test_assistant_message_with_reasoning_but_empty_content(self): + chat_msg = { + "role": "assistant", + "reasoning": "I'm thinking about the user's question.", + "content": "", + } + + messages = parse_chat_input_to_harmony_message(chat_msg) + + verify_harmony_messages( + messages, + [ + { + "role": "assistant", + "channel": "analysis", + "content": "I'm thinking about the user's question.", + }, + ], + ) + + def test_assistant_message_with_reasoning_but_none_content(self): + chat_msg = { + "role": "assistant", + "reasoning": "I'm thinking about the user's question.", + "content": None, + } + + messages = parse_chat_input_to_harmony_message(chat_msg) + + verify_harmony_messages( + messages, + [ + { + "role": "assistant", + "channel": "analysis", + "content": "I'm thinking about the user's question.", + }, + ], + ) + + def test_assistant_message_with_tool_calls_but_no_content(self): + chat_msg = { + "role": "assistant", + "tool_calls": [ + { + "function": { + "name": "get_weather", + "arguments": '{"location": "San Francisco"}', + } + } + ], + } + + messages = parse_chat_input_to_harmony_message(chat_msg) + + verify_harmony_messages( + messages, + [ + { + "role": "assistant", + "channel": "commentary", + "recipient": "functions.get_weather", + "content": '{"location": "San Francisco"}', + "content_type": "json", + }, + ], + ) + + def test_assistant_message_with_tool_calls_and_content(self): + chat_msg = { + "role": "assistant", + "tool_calls": [ + { + "function": { + "name": "get_weather", + "arguments": '{"location": "San Francisco"}', + } + } + ], + "content": "I'll call the tool.", + } + + messages = parse_chat_input_to_harmony_message(chat_msg) + + verify_harmony_messages( + messages, + [ + { + "role": "assistant", + "channel": "commentary", + "content": "I'll call the tool.", + }, + { + "role": "assistant", + "channel": "commentary", + "recipient": "functions.get_weather", + "content": '{"location": "San Francisco"}', + "content_type": "json", + }, + ], + ) + + def test_assistant_message_with_tool_calls_and_reasoning(self): + chat_msg = { + "role": "assistant", + "tool_calls": [ + { + "function": { + "name": "get_weather", + "arguments": '{"location": "San Francisco"}', + } + } + ], + "reasoning": "I should use the get_weather tool.", + } + + messages = parse_chat_input_to_harmony_message(chat_msg) + + verify_harmony_messages( + messages, + [ + { + "role": "assistant", + "channel": "analysis", + "content": "I should use the get_weather tool.", + }, + { + "role": "assistant", + "channel": "commentary", + "recipient": "functions.get_weather", + "content": '{"location": "San Francisco"}', + "content_type": "json", + }, + ], + ) + + def test_assistant_message_with_tool_calls_and_reasoning_and_content(self): + chat_msg = { + "role": "assistant", + "tool_calls": [ + { + "function": { + "name": "get_weather", + "arguments": '{"location": "San Francisco"}', + } + } + ], + "reasoning": "I should use the get_weather tool.", + "content": "I'll call the tool.", + } + + messages = parse_chat_input_to_harmony_message(chat_msg) + + verify_harmony_messages( + messages, + [ + { + "role": "assistant", + "channel": "commentary", + "content": "I'll call the tool.", + }, + { + "role": "assistant", + "channel": "analysis", + "content": "I should use the get_weather tool.", + }, + { + "role": "assistant", + "channel": "commentary", + "recipient": "functions.get_weather", + "content": '{"location": "San Francisco"}', + "content_type": "json", + }, + ], + ) + + def test_tool_message_with_string_content(self): + tool_id_names = { + "call_123": "get_weather", + } + chat_msg = { + "role": "tool", + "tool_call_id": "call_123", + "content": "The weather in San Francisco is sunny, 72°F", + } + + messages = parse_chat_input_to_harmony_message( + chat_msg, tool_id_names=tool_id_names + ) + + verify_harmony_messages( + messages, + [ + { + "role": "tool", + "name": "functions.get_weather", + "content": "The weather in San Francisco is sunny, 72°F", + "channel": "commentary", + }, + ], + ) + + def test_tool_message_with_array_content(self): + tool_id_names = { + "call_123": "search_results", + } + chat_msg = { + "role": "tool", + "tool_call_id": "call_123", + "content": [ + {"type": "text", "text": "Result 1: "}, + {"type": "text", "text": "Result 2: "}, + { + "type": "image", + "url": "http://example.com/img.png", + }, # Should be ignored + {"type": "text", "text": "Result 3"}, + ], + } + + messages = parse_chat_input_to_harmony_message( + chat_msg, tool_id_names=tool_id_names + ) + + verify_harmony_messages( + messages, + [ + { + "role": "tool", + "name": "functions.search_results", + "content": "Result 1: Result 2: Result 3", + "channel": "commentary", + }, + ], + ) + + def test_tool_message_with_empty_content(self): + tool_id_names = { + "call_123": "empty_tool", + } + chat_msg = { + "role": "tool", + "tool_call_id": "call_123", + "content": "", + } + + messages = parse_chat_input_to_harmony_message( + chat_msg, tool_id_names=tool_id_names + ) + + verify_harmony_messages( + messages, + [ + { + "role": "tool", + "name": "functions.empty_tool", + "content": "", + "channel": "commentary", + }, + ], + ) + + def test_tool_message_with_none_content(self): + tool_id_names = { + "call_123": "empty_tool", + } + chat_msg = { + "role": "tool", + "tool_call_id": "call_123", + "content": None, + } + + messages = parse_chat_input_to_harmony_message( + chat_msg, tool_id_names=tool_id_names + ) + + verify_harmony_messages( + messages, + [ + { + "role": "tool", + "name": "functions.empty_tool", + "content": "", + "channel": "commentary", + }, + ], + ) + + +class TestAutoDropAnalysisMessages: + def test_no_analysis_messages(self) -> None: + messages = [ + Message.from_role_and_content( + Role.ASSISTANT, "The answer is 4." + ).with_channel("final"), + ] + cleaned_messages = auto_drop_analysis_messages(messages) + assert cleaned_messages == messages + + def test_only_analysis_message(self) -> None: + messages = [ + Message.from_role_and_content( + Role.ASSISTANT, "I'm thinking about the user's question." + ).with_channel("analysis"), + ] + cleaned_messages = auto_drop_analysis_messages(messages) + assert cleaned_messages == messages + + def test_multiple_analysis_messages_without_final_message(self) -> None: + messages = [ + Message.from_role_and_content( + Role.ASSISTANT, "I'm thinking about the user's question." + ).with_channel("analysis"), + Message.from_role_and_content( + Role.ASSISTANT, "I'm thinking more." + ).with_channel("analysis"), + Message.from_role_and_content( + Role.ASSISTANT, "I'm thinking even more." + ).with_channel("analysis"), + ] + cleaned_messages = auto_drop_analysis_messages(messages) + assert cleaned_messages == messages + + def test_only_final_message(self) -> None: + messages = [ + Message.from_role_and_content( + Role.ASSISTANT, "The answer is 4." + ).with_channel("final"), + ] + cleaned_messages = auto_drop_analysis_messages(messages) + assert cleaned_messages == messages + + def test_drops_one_analysis_messages_before_final_message(self) -> None: + messages = [ + Message.from_role_and_content( + Role.ASSISTANT, "I'm thinking about the user's question." + ).with_channel("analysis"), + Message.from_role_and_content( + Role.ASSISTANT, "The answer is 4." + ).with_channel("final"), + Message.from_role_and_content( + Role.ASSISTANT, "I should think harder." + ).with_channel("analysis"), + ] + cleaned_messages = auto_drop_analysis_messages(messages) + # Should have dropped the first analysis message + assert cleaned_messages == messages[1:] + + def test_drops_all_analysis_messages_before_final_message(self) -> None: + messages = [ + Message.from_role_and_content( + Role.ASSISTANT, "I'm thinking about the user's question." + ).with_channel("analysis"), + Message.from_role_and_content( + Role.ASSISTANT, "I'm thinking more." + ).with_channel("analysis"), + Message.from_role_and_content( + Role.ASSISTANT, "I'm thinking even more." + ).with_channel("analysis"), + Message.from_role_and_content( + Role.ASSISTANT, "The answer is 4." + ).with_channel("final"), + Message.from_role_and_content( + Role.ASSISTANT, "I should think harder." + ).with_channel("analysis"), + ] + cleaned_messages = auto_drop_analysis_messages(messages) + # Should have dropped the first 3 analysis messages + assert cleaned_messages == messages[3:] + + def test_multiple_analysis_messages_with_multiple_final_messages(self) -> None: + messages = [ + Message.from_role_and_content( + Role.ASSISTANT, "I'm thinking about the user's question." + ).with_channel("analysis"), + Message.from_role_and_content( + Role.ASSISTANT, "I'm thinking more." + ).with_channel("analysis"), + Message.from_role_and_content( + Role.ASSISTANT, "I'm thinking even more." + ).with_channel("analysis"), + Message.from_role_and_content( + Role.ASSISTANT, "The answer is 4." + ).with_channel("final"), + Message.from_role_and_content( + Role.ASSISTANT, "I should think harder." + ).with_channel("analysis"), + Message.from_role_and_content( + Role.ASSISTANT, "The answer is 5." + ).with_channel("final"), + ] + cleaned_messages = auto_drop_analysis_messages(messages) + # Should have dropped all those analysis messages + assert len(cleaned_messages) == 2 + assert cleaned_messages[0].content[0].text == "The answer is 4." + assert cleaned_messages[1].content[0].text == "The answer is 5." + + def test_drops_non_assistant_analysis_messages(self) -> None: + messages = [ + Message.from_role_and_content( + Role.TOOL, "The tool thinks we should think harder." + ).with_channel("analysis"), + Message.from_role_and_content( + Role.ASSISTANT, "The answer is 4." + ).with_channel("final"), + ] + cleaned_messages = auto_drop_analysis_messages(messages) + # Should have dropped the analysis message + assert cleaned_messages == messages[1:] + + +def test_has_custom_tools() -> None: + assert not has_custom_tools(set()) + assert not has_custom_tools({"web_search_preview", "code_interpreter", "container"}) + assert has_custom_tools({"others"}) + assert has_custom_tools( + {"web_search_preview", "code_interpreter", "container", "others"} + ) + + +class TestGetSystemMessage: + """Tests for get_system_message channel configuration.""" + + def test_commentary_channel_present_without_custom_tools(self) -> None: + """Commentary channel must be valid even without custom tools.""" + sys_msg = get_system_message(with_custom_tools=False) + valid_channels = sys_msg.content[0].channel_config.valid_channels + assert "commentary" in valid_channels + + def test_commentary_channel_present_with_custom_tools(self) -> None: + """Commentary channel present when custom tools are enabled.""" + sys_msg = get_system_message(with_custom_tools=True) + valid_channels = sys_msg.content[0].channel_config.valid_channels + assert "commentary" in valid_channels + + def test_all_standard_channels_present(self) -> None: + """All three standard Harmony channels should always be valid.""" + for with_tools in (True, False): + sys_msg = get_system_message(with_custom_tools=with_tools) + valid_channels = sys_msg.content[0].channel_config.valid_channels + for channel in ("analysis", "commentary", "final"): + assert channel in valid_channels, ( + f"{channel} missing when with_custom_tools={with_tools}" + ) + + def test_unsupported_reasoning_effort_raises_clear_error(self) -> None: + with pytest.raises( + ValueError, + match="reasoning_effort='max' is not supported by Harmony", + ): + get_system_message(reasoning_effort="max") + + +class TestResponseInputToHarmonyReasoningItem: + """Tests for response_input_to_harmony handling of reasoning input items. + + Per the OpenAI spec, ResponseReasoningItem.content is + Optional[List[Content]] = None. Clients like langchain-openai may omit + this field when constructing multi-turn input from previous responses. + + Reasoning items with content are converted to Harmony messages on the + 'analysis' channel. All content items are concatenated. Items without + content return None (skipped by the caller). + """ + + def test_reasoning_with_single_content(self): + """Test reasoning item with a single content entry.""" + item = { + "type": "reasoning", + "id": "rs_123", + "content": [{"type": "reasoning_text", "text": "Thinking step by step"}], + } + + msg = response_input_to_harmony(item, prev_responses=[]) + + assert msg is not None + assert msg.author.role == Role.ASSISTANT + assert msg.content[0].text == "Thinking step by step" + assert msg.channel == "analysis" + + def test_reasoning_with_multiple_content_items(self): + """Test reasoning item with multiple content entries concatenated.""" + item = { + "type": "reasoning", + "id": "rs_123", + "content": [ + {"type": "reasoning_text", "text": "First, let me analyze"}, + {"type": "reasoning_text", "text": "Second, I should consider"}, + {"type": "reasoning_text", "text": "Finally, the answer is"}, + ], + } + + msg = response_input_to_harmony(item, prev_responses=[]) + + assert msg is not None + assert msg.author.role == Role.ASSISTANT + assert msg.content[0].text == ( + "First, let me analyze\nSecond, I should consider\nFinally, the answer is" + ) + assert msg.channel == "analysis" + + def test_reasoning_without_content_returns_none(self): + """Test reasoning item without content field returns None.""" + item = { + "type": "reasoning", + "id": "rs_123", + "summary": [{"type": "summary_text", "text": "Thinking about math"}], + } + + msg = response_input_to_harmony(item, prev_responses=[]) + + assert msg is None + + def test_reasoning_with_none_content_returns_none(self): + """Test reasoning item with content=None returns None.""" + item = { + "type": "reasoning", + "id": "rs_123", + "content": None, + "summary": [{"type": "summary_text", "text": "Thinking about math"}], + } + + msg = response_input_to_harmony(item, prev_responses=[]) + + assert msg is None + + def test_reasoning_with_empty_content_returns_none(self): + """Test reasoning item with empty content list returns None.""" + item = { + "type": "reasoning", + "id": "rs_123", + "content": [], + } + + msg = response_input_to_harmony(item, prev_responses=[]) + + assert msg is None diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/conftest.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..e5858a197a5d3ee74acbf84d47fd65b6d7d2550a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/conftest.py @@ -0,0 +1,407 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from __future__ import annotations + +import json +import logging +from collections.abc import Callable +from typing import Any + +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer + +logger = logging.getLogger(__name__) + +BASE_TEST_ENV = { + # The day vLLM said "hello world" on arxiv 🚀 + "VLLM_SYSTEM_START_DATE": "2023-09-12", +} +DEFAULT_MAX_RETRIES = 3 + + +@pytest.fixture +def pairs_of_event_types() -> dict[str, str]: + """Links the 'done' event type with the corresponding 'start' event type. + + This mapping should link all done <-> start events; if tests mean to + restrict the allowed events, they should filter this fixture to avoid + copy + paste errors in the mappings or unexpected KeyErrors due to missing + events. + """ + # fmt: off + event_pairs = { + "response.completed": "response.created", + "response.output_item.done": "response.output_item.added", + "response.content_part.done": "response.content_part.added", + "response.output_text.done": "response.output_text.delta", + "response.reasoning_text.done": "response.reasoning_text.delta", + "response.reasoning_part.done": "response.reasoning_part.added", + "response.mcp_call_arguments.done": "response.mcp_call_arguments.delta", + "response.mcp_call.completed": "response.mcp_call.in_progress", + "response.function_call_arguments.done": "response.function_call_arguments.delta", # noqa: E501 + "response.code_interpreter_call_code.done": "response.code_interpreter_call_code.delta", # noqa: E501 + "response.code_interpreter_call.completed": "response.code_interpreter_call.in_progress", # noqa: E501 + "response.web_search_call.completed": "response.web_search_call.in_progress", + } + # fmt: on + return event_pairs + + +async def retry_for_tool_call( + client, + *, + model: str, + expected_tool_type: str, + max_retries: int = DEFAULT_MAX_RETRIES, + **create_kwargs: Any, +): + """Call ``client.responses.create`` up to *max_retries* times, returning + the first response that contains an output item of *expected_tool_type*. + + Returns the **last** response if none match so the caller's assertions + fire with a clear diagnostic. + """ + last_response = None + for attempt in range(max_retries): + response = await client.responses.create(model=model, **create_kwargs) + last_response = response + if any( + getattr(item, "type", None) == expected_tool_type + for item in response.output + ): + return response + assert last_response is not None + return last_response + + +async def retry_streaming_for( + client, + *, + model: str, + validate_events: Callable[[list], bool], + max_retries: int = DEFAULT_MAX_RETRIES, + **create_kwargs: Any, +) -> list: + """Call ``client.responses.create(stream=True)`` up to *max_retries* + times, returning the first event list where *validate_events* returns + ``True``. + """ + last_events: list = [] + for attempt in range(max_retries): + stream = await client.responses.create( + model=model, stream=True, **create_kwargs + ) + events: list = [] + async for event in stream: + events.append(event) + last_events = events + if validate_events(events): + return events + return last_events + + +def has_output_type(response, type_name: str) -> bool: + """Return True if *response* has at least one output item of *type_name*.""" + return any(getattr(item, "type", None) == type_name for item in response.output) + + +def events_contain_type(events: list, type_substring: str) -> bool: + """Return True if any event's type contains *type_substring*.""" + return any(type_substring in getattr(e, "type", "") for e in events) + + +def _validate_event_pairing(events: list, pairs_of_event_types: dict[str, str]) -> None: + """Validate that streaming events are properly nested/paired. + + Derives push/pop sets from *pairs_of_event_types* so that every + start/end pair in the dict is handled automatically. + """ + start_events = set(pairs_of_event_types.values()) + end_events = set(pairs_of_event_types.keys()) + + stack: list[str] = [] + for event in events: + etype = event.type + if etype in end_events: + expected_start = pairs_of_event_types[etype] + assert stack and stack[-1] == expected_start, ( + f"Stack mismatch for {etype}: " + f"expected {expected_start}, " + f"got {stack[-1] if stack else ''}" + ) + stack.pop() + elif etype in start_events: + # Consecutive deltas of the same type share a single stack slot. + if etype.endswith("delta") and stack and stack[-1] == etype: + continue + stack.append(etype) + # else: passthrough event (e.g. response.in_progress, + # web_search_call.searching, code_interpreter_call.interpreting) + assert len(stack) == 0, f"Unclosed events on stack: {stack}" + + +def _validate_event_ordering(events: list) -> None: + """Validate that envelope events appear in the correct positions.""" + assert len(events) >= 2, f"Expected at least 2 events, got {len(events)}" + + # First event must be response.created + assert events[0].type == "response.created", ( + f"First event must be response.created, got {events[0].type}" + ) + # Last event must be response.completed + assert events[-1].type == "response.completed", ( + f"Last event must be response.completed, got {events[-1].type}" + ) + + # response.in_progress, if present, must be the second event + in_progress_indices = [ + i for i, e in enumerate(events) if e.type == "response.in_progress" + ] + if in_progress_indices: + assert in_progress_indices == [1], ( + f"response.in_progress must be the second event, " + f"found at indices {in_progress_indices}" + ) + + # Exactly one created and one completed + created_count = sum(1 for e in events if e.type == "response.created") + completed_count = sum(1 for e in events if e.type == "response.completed") + assert created_count == 1, ( + f"Expected exactly 1 response.created, got {created_count}" + ) + assert completed_count == 1, ( + f"Expected exactly 1 response.completed, got {completed_count}" + ) + + +def _validate_field_consistency(events: list) -> None: + """Validate item_id, output_index, and content_index consistency. + + Tracks the active output item established by ``output_item.added`` + and verifies that all subsequent events for that item carry matching + identifiers until ``output_item.done`` closes it. + """ + _SESSION_EVENTS = { + "response.created", + "response.in_progress", + "response.completed", + } + + active_item_id: str | None = None + active_output_index: int | None = None + last_output_index: int = -1 + active_content_index: int | None = None + + for event in events: + etype = event.type + + if etype in _SESSION_EVENTS: + continue + + # --- output_item.added: opens a new item ------------------ + if etype == "response.output_item.added": + item = getattr(event, "item", None) + output_index = getattr(event, "output_index", None) + + assert item is not None, "output_item.added must have an item" + item_id = getattr(item, "id", None) + assert item_id, "output_item.added item must have an id" + + # output_index must be non-decreasing across items + if output_index is not None: + assert output_index >= last_output_index, ( + f"output_index went backwards: {output_index} < {last_output_index}" + ) + last_output_index = output_index + + active_item_id = item_id + active_output_index = output_index + active_content_index = None + continue + + # --- output_item.done: closes the active item ------------- + if etype == "response.output_item.done": + item = getattr(event, "item", None) + output_index = getattr(event, "output_index", None) + + assert item is not None, "output_item.done must have an item" + done_item_id = getattr(item, "id", None) + + if active_item_id is not None and done_item_id: + assert done_item_id == active_item_id, ( + f"output_item.done item.id mismatch: " + f"expected {active_item_id}, got {done_item_id}" + ) + if active_output_index is not None and output_index is not None: + assert output_index == active_output_index, ( + f"output_item.done output_index mismatch: " + f"expected {active_output_index}, got {output_index}" + ) + + active_item_id = None + active_output_index = None + active_content_index = None + continue + + # --- content_part / reasoning_part added: sets content_index + if etype in ( + "response.content_part.added", + "response.reasoning_part.added", + ): + _assert_item_fields(event, etype, active_item_id, active_output_index) + content_index = getattr(event, "content_index", None) + if active_content_index is None: + assert content_index == 0, ( + f"{etype} for a new item must start at content_index 0, " + f"got {content_index}" + ) + active_content_index = content_index + continue + + # --- all other item-level events -------------------------- + _assert_item_fields(event, etype, active_item_id, active_output_index) + + # content_index (only meaningful on events that carry it) + content_index = getattr(event, "content_index", None) + if content_index is not None and active_content_index is not None: + assert content_index == active_content_index, ( + f"{etype} content_index mismatch: " + f"expected {active_content_index}, got {content_index}" + ) + + +def _assert_item_fields( + event, + etype: str, + active_item_id: str | None, + active_output_index: int | None, +) -> None: + """Check that *event*'s item_id and output_index match the active item.""" + event_item_id = getattr(event, "item_id", None) + output_index = getattr(event, "output_index", None) + + if active_item_id is not None and event_item_id is not None: + assert event_item_id == active_item_id, ( + f"{etype} item_id mismatch: expected {active_item_id}, got {event_item_id}" + ) + if active_output_index is not None and output_index is not None: + assert output_index == active_output_index, ( + f"{etype} output_index mismatch: " + f"expected {active_output_index}, got {output_index}" + ) + + +def validate_streaming_event_stack( + events: list, pairs_of_event_types: dict[str, str] +) -> None: + """Validate streaming events: pairing, ordering, and field consistency. + + Checks three aspects: + 1. **Event pairing** — start/end events are properly nested + (stack-based matching derived from *pairs_of_event_types*). + 2. **Event ordering** — envelope events (``created``, + ``in_progress``, ``completed``) appear at the correct positions. + 3. **Field consistency** — ``item_id``, ``output_index``, and + ``content_index`` are consistent across related events within + each output item's lifecycle. + """ + _validate_event_pairing(events, pairs_of_event_types) + _validate_event_ordering(events) + _validate_field_consistency(events) + + +def log_response_diagnostics( + response, + *, + label: str = "Response Diagnostics", +) -> dict[str, Any]: + """Extract and log diagnostic info from a Responses API response. + + Logs reasoning, tool-call attempts, MCP items, and output types so + that CI output (``pytest -s`` or ``--log-cli-level=INFO``) gives + full visibility into model behaviour even on passing runs. + + Returns the extracted data so callers can make additional assertions + if needed. + """ + reasoning_texts = [ + text + for item in response.output + if getattr(item, "type", None) == "reasoning" + for content in getattr(item, "content", []) + if (text := getattr(content, "text", None)) + ] + + tool_call_attempts = [ + { + "recipient": msg.get("recipient"), + "channel": msg.get("channel"), + } + for msg in response.output_messages + if (msg.get("recipient") or "").startswith("python") + ] + + mcp_items = [ + { + "name": getattr(item, "name", None), + "status": getattr(item, "status", None), + } + for item in response.output + if getattr(item, "type", None) == "mcp_call" + ] + + output_types = [getattr(o, "type", None) for o in response.output] + + diagnostics = { + "model_attempted_tool_calls": bool(tool_call_attempts), + "tool_call_attempts": tool_call_attempts, + "mcp_items": mcp_items, + "reasoning": reasoning_texts, + "output_text": response.output_text, + "output_types": output_types, + } + + logger.info( + "\n====== %s ======\n%s\n==============================", + label, + json.dumps(diagnostics, indent=2, default=str), + ) + + return diagnostics + + +@pytest.fixture(scope="module") +def default_server_args(): + return [ + "--max-model-len", + "18192", + "--enforce-eager", # For faster startup. + "--enable-auto-tool-choice", + "--structured-outputs-config.backend", + "xgrammar", + "--tool-call-parser", + "hermes", + "--reasoning-parser", + "qwen3", + ] + + +@pytest.fixture(scope="module") +def server_with_store(default_server_args): + with RemoteOpenAIServer( + "Qwen/Qwen3-1.7B", + default_server_args, + env_dict={ + "VLLM_ENABLE_RESPONSES_API_STORE": "1", + "VLLM_SERVER_DEV_MODE": "1", + }, + ) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server_with_store): + async with server_with_store.get_async_client() as async_client: + yield async_client diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_basic.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_basic.py new file mode 100644 index 0000000000000000000000000000000000000000..45989b6297f931864b193f2ea1c740475f424469 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_basic.py @@ -0,0 +1,93 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import openai # use the official client for correctness check +import openai.types.responses as openai_responses_types +import pytest + + +@pytest.mark.asyncio +async def test_simple_input(client: openai.AsyncOpenAI): + response = await client.responses.create(input="What is 13 * 24?") + print(response) + + outputs = response.output + # Whether the output contains the answer. + assert outputs[-1].type == "message" + assert "312" in outputs[-1].content[0].text + + # Whether the output contains the reasoning. + assert outputs[0].type == "reasoning" + assert outputs[0].content[0].text != "" + + +@pytest.mark.asyncio +async def test_instructions(client: openai.AsyncOpenAI): + response = await client.responses.create( + instructions="Finish the answer with QED.", + input="What is 13 * 24?", + ) + print(response) + + output_text = response.output[-1].content[0].text + assert "312" in output_text + assert "QED" in output_text + + +@pytest.mark.asyncio +async def test_chat(client: openai.AsyncOpenAI): + response = await client.responses.create( + input=[ + {"role": "system", "content": "Finish the answer with QED."}, + {"role": "user", "content": "What is 5 * 3?"}, + {"role": "assistant", "content": "15. QED."}, + {"role": "user", "content": "Multiply the result by 2."}, + ], + ) + print(response) + + output_text = response.output[-1].content[0].text + assert "30" in output_text + assert "QED" in output_text + + +@pytest.mark.asyncio +async def test_chat_with_input_type(client: openai.AsyncOpenAI): + response = await client.responses.create( + input=[ + { + "role": "user", + "content": [{"type": "input_text", "text": "Hello!"}], + }, + ], + ) + print(response) + assert response.status == "completed" + + +@pytest.mark.asyncio +async def test_logprobs(client: openai.AsyncOpenAI): + response = await client.responses.create( + include=["message.output_text.logprobs"], + input="What is 13 * 24?", + top_logprobs=5, + ) + print(response) + outputs = response.output + assert outputs[-1].content[-1].logprobs + assert len(outputs[-1].content[-1].logprobs[0].top_logprobs) == 5 + + +@pytest.mark.asyncio +async def test_streaming(client: openai.AsyncOpenAI): + stream = await client.responses.create( + input="What is 13 * 24?", + stream=True, + ) + events = [event async for event in stream] + assert isinstance(events[0], openai_responses_types.ResponseCreatedEvent) + assert any( + isinstance(event, openai_responses_types.ResponseTextDeltaEvent) + for event in events + ) + assert isinstance(events[-1], openai_responses_types.ResponseCompletedEvent) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_errors.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_errors.py new file mode 100644 index 0000000000000000000000000000000000000000..12e7e2deaeedca9743ad95df3c10a92231985ff7 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_errors.py @@ -0,0 +1,96 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from http import HTTPStatus +from unittest.mock import MagicMock + +import pytest + +import vllm.envs as envs +from vllm.entrypoints.generate.base.serving import GenerateBaseServing, GenerationError +from vllm.envs import disable_envs_cache + + +@pytest.mark.asyncio +async def test_raise_if_error_raises_generation_error(): + """test _raise_if_error raises GenerationError""" + # create a minimal GenerateBaseServing instance + mock_engine = MagicMock() + mock_engine.model_config = MagicMock() + mock_engine.model_config.max_model_len = 100 + mock_models = MagicMock() + + serving = GenerateBaseServing( + engine_client=mock_engine, + models=mock_models, + request_logger=None, + ) + + # test that error finish_reason raises GenerationError + with pytest.raises(GenerationError) as exc_info: + serving._raise_if_error("error", "test-request-id") + + assert str(exc_info.value) == "Internal server error" + assert exc_info.value.status_code == HTTPStatus.INTERNAL_SERVER_ERROR + + # test that other finish_reasons don't raise + serving._raise_if_error("stop", "test-request-id") # should not raise + serving._raise_if_error("length", "test-request-id") # should not raise + serving._raise_if_error(None, "test-request-id") # should not raise + + +@pytest.mark.asyncio +async def test_convert_generation_error_to_streaming_response(): + """test _convert_generation_error_to_streaming_response output""" + mock_engine = MagicMock() + mock_engine.model_config = MagicMock() + mock_engine.model_config.max_model_len = 100 + mock_models = MagicMock() + + serving = GenerateBaseServing( + engine_client=mock_engine, + models=mock_models, + request_logger=None, + ) + + # create a GenerationError + gen_error = GenerationError("Internal server error") + + # convert to streaming error response + error_json = serving._convert_generation_error_to_streaming_response(gen_error) + + assert isinstance(error_json, str) + assert "Internal server error" in error_json + assert "InternalServerError" in error_json + + +def test_is_model_supported_skip_name_validation_env( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """When VLLM_SKIP_MODEL_NAME_VALIDATION is set, accept any model id.""" + disable_envs_cache() + monkeypatch.delenv("VLLM_SKIP_MODEL_NAME_VALIDATION", raising=False) + + mock_engine = MagicMock() + mock_engine.model_config = MagicMock() + mock_engine.model_config.max_model_len = 100 + mock_models = MagicMock() + mock_models.is_base_model.return_value = False + + serving = GenerateBaseServing( + engine_client=mock_engine, + models=mock_models, + request_logger=None, + ) + + assert serving._is_model_supported("not-a-registered-model") is False + + monkeypatch.setenv("VLLM_SKIP_MODEL_NAME_VALIDATION", "1") + disable_envs_cache() + assert envs.VLLM_SKIP_MODEL_NAME_VALIDATION is True + assert serving._is_model_supported("not-a-registered-model") is True + + monkeypatch.setenv("VLLM_SKIP_MODEL_NAME_VALIDATION", "true") + disable_envs_cache() + assert envs.VLLM_SKIP_MODEL_NAME_VALIDATION is True + assert serving._is_model_supported("another-alias") is True diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_function_call.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_function_call.py new file mode 100644 index 0000000000000000000000000000000000000000..9f60dab62faac030265243f051329730c45e2bb6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_function_call.py @@ -0,0 +1,529 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import openai # use the official client for correctness check +import pytest + +MODEL_NAME = "Qwen/Qwen3-1.7B" +tools = [ + { + "type": "function", + "name": "get_current_weather", + "description": "Get the current weather in a given location", + "parameters": { + "type": "object", + "properties": { + "city": { + "type": "string", + "description": "The city to find the weather for, e.g. 'Vienna'", + "default": "Vienna", + }, + "country": { + "type": "string", + "description": "The country that the city is in, e.g. 'Austria'", + }, + "unit": { + "type": "string", + "description": "The unit to fetch the temperature in", + "enum": ["celsius", "fahrenheit"], + }, + "options": { + "$ref": "#/$defs/WeatherOptions", + "description": "Optional parameters for weather query", + }, + }, + "required": ["country", "unit"], + "$defs": { + "WeatherOptions": { + "title": "WeatherOptions", + "type": "object", + "additionalProperties": False, + "properties": { + "unit": { + "type": "string", + "enum": ["celsius", "fahrenheit"], + "default": "celsius", + "description": "Temperature unit", + "title": "Temperature Unit", + }, + "include_forecast": { + "type": "boolean", + "default": False, + "description": "Whether to include a 24-hour forecast", + "title": "Include Forecast", + }, + "language": { + "type": "string", + "default": "zh-CN", + "description": "Language of the response", + "title": "Language", + "enum": ["zh-CN", "en-US", "ja-JP"], + }, + }, + }, + }, + }, + }, + { + "type": "function", + "name": "get_forecast", + "description": "Get the weather forecast for a given location", + "parameters": { + "type": "object", + "properties": { + "city": { + "type": "string", + "description": "The city to get the forecast for, e.g. 'Vienna'", + "default": "Vienna", + }, + "country": { + "type": "string", + "description": "The country that the city is in, e.g. 'Austria'", + }, + "days": { + "type": "integer", + "description": "Number of days to get the forecast for (1-7)", + }, + "unit": { + "type": "string", + "description": "The unit to fetch the temperature in", + "enum": ["celsius", "fahrenheit"], + }, + }, + "required": ["country", "days", "unit"], + }, + }, +] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("tool_choice", ["auto", "required"]) +async def test_function_tool_use( + client: openai.AsyncOpenAI, model_name: str, tool_choice: str +): + prompt = [ + { + "role": "user", + "content": "Can you tell me what the current weather is in Berlin and the " + "forecast for the next 5 days, in fahrenheit?", + }, + ] + response = await client.responses.create( + model=model_name, + input=prompt, + tools=tools, + tool_choice=tool_choice, + temperature=0.0, + ) + assert len(response.output) >= 1 + tool_call = None + reasoning = None + for out in response.output: + if out.type == "function_call": + tool_call = out + if out.type == "reasoning": + reasoning = out + if response.incomplete_details is None: + assert tool_call is not None + assert tool_call.type == "function_call" + assert json.loads(tool_call.arguments) is not None + assert reasoning is not None + assert reasoning.type == "reasoning" + else: + print(response.model_dump_json(indent=2)) + assert response.incomplete_details.reason == "max_output_tokens" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_max_tokens_with_tool_choice_required( + client: openai.AsyncOpenAI, model_name: str +): + prompt = [ + { + "role": "user", + "content": "Can you tell me what the current weather is in Berlin and the " + "forecast for the next 5 days, in fahrenheit?", + }, + ] + response = await client.responses.create( + model=model_name, + input=prompt, + tools=tools, + tool_choice="required", + max_output_tokens=10, + ) + assert len(response.output) >= 1 + for out in response.output: + # When `tool_choice="required"` and the tokens of `tools` + # exceed `max_output_tokens`,`function_call` should be empty. + # This behavior should be consistent with OpenAI + assert out.type != "function_call" + assert response.incomplete_details.reason == "max_output_tokens" + + +@pytest.mark.asyncio +async def test_named_tool_use(client: openai.AsyncOpenAI): + def get_weather(latitude: float, longitude: float) -> str: + """ + Mock function to simulate getting weather data. + In a real application, this would call an external weather API. + """ + return f"Current temperature at ({latitude}, {longitude}) is 20°C." + + tools = [ + { + "type": "function", + "name": "get_weather", + "description": ( + "Get current temperature for provided coordinates in celsius." + ), + "parameters": { + "type": "object", + "properties": { + "latitude": {"type": "number"}, + "longitude": {"type": "number"}, + }, + "required": ["latitude", "longitude"], + "additionalProperties": False, + }, + "strict": True, + } + ] + + input_messages = [ + {"role": "user", "content": "What's the weather like in Paris today?"} + ] + + response = await client.responses.create( + model=MODEL_NAME, + input=input_messages, + tools=tools, + tool_choice={"type": "function", "name": "get_weather"}, + ) + assert len(response.output) >= 1 + for out in response.output: + if out.type == "function_call": + tool_call = out + assert tool_call is not None + assert tool_call.type == "function_call" + assert tool_call.name == "get_weather" + args = json.loads(tool_call.arguments) + assert args["latitude"] is not None + assert args["longitude"] is not None + # call the tool + result = get_weather(args["latitude"], args["longitude"]) + input_messages.append(tool_call) # append model's function call message + input_messages.append( + { # append result message + "type": "function_call_output", + "call_id": tool_call.call_id, + "output": str(result), + } + ) + # create a new response with the tool call result + response_2 = await client.responses.create(model=MODEL_NAME, input=input_messages) + # check the output + assert len(response_2.output_text) > 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_function_calling_with_streaming_expected_arguments( + client: openai.AsyncOpenAI, model_name: str +): + tools = [ + { + "type": "function", + "name": "get_weather", + "description": "Get current temperature for provided location in celsius.", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"}, + }, + "required": ["location"], + "additionalProperties": False, + }, + "strict": True, + }, + { + "type": "function", + "name": "get_time", + "description": "Get current local time for provided location.", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"}, + }, + "required": ["location"], + "additionalProperties": False, + }, + "strict": True, + }, + ] + + stream_response = await client.responses.create( + model=model_name, + input=( + "Use tools only. Call get_weather for Berlin and get_time for Tokyo. " + "Do not answer directly." + ), + tools=tools, + stream=True, + ) + + tool_call_items = {} + arguments_done_events = {} + completed_events = {} + async for event in stream_response: + if ( + event.type == "response.output_item.added" + and event.item.type == "function_call" + ): + tool_call_items[event.output_index] = event.item + elif event.type == "response.function_call_arguments.delta": + tool_call_item = tool_call_items[event.output_index] + tool_call_item.arguments += event.delta + elif event.type == "response.function_call_arguments.done": + arguments_done_events[event.output_index] = event + elif ( + event.type == "response.output_item.done" + and event.item.type == "function_call" + ): + completed_events[event.output_index] = event + assert len(tool_call_items) >= 2 + assert len(arguments_done_events) >= 2 + assert len(completed_events) >= 2 + + tool_calls_by_name = { + event.item.name: ( + tool_call_items[output_index], + arguments_done_events[output_index], + event.item, + ) + for output_index, event in completed_events.items() + } + assert {"get_weather", "get_time"}.issubset(tool_calls_by_name) + for added_item, arguments_done_event, completed_item in tool_calls_by_name.values(): + assert added_item.type == "function_call" + assert added_item.arguments == arguments_done_event.arguments + assert added_item.arguments == completed_item.arguments + assert added_item.name == arguments_done_event.name + assert added_item.name == completed_item.name + args = json.loads(added_item.arguments) + assert "location" in args + assert args["location"] is not None + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize( + "tool_choice", + ["auto", "required", {"type": "function", "name": "get_current_weather"}], +) +@pytest.mark.parametrize( + "enable_thinking", + [True, False], +) +async def test_function_calling_with_streaming_types( + client: openai.AsyncOpenAI, model_name: str, tool_choice, enable_thinking: bool +): + # this links the "done" type with the "start" type + # so every "done" type should have a corresponding "start" type + # and every open block should be closed by the end of the stream + # + # stream of events for a response with function call could look like this: + # option1: reasoning -> content(option) -> function_call + # response.created + # -> response.in_progress + # -> response.output_item.added + # -> response.reasoning_part.added + # -> response.reasoning_text.delta + # .... + # -> response.reasoning_text.delta + # -> response.reasoning_text.done + # -> response.reasoning_part.done + # -> response.output_item.done + # -> response.output_item.added + # -> response.content_part.added + # -> response.output_text.delta + # ... + # -> response.output_text.delta + # -> response.output_text.done + # -> response.content_part.done + # -> response.output_item.done + # -> response.output_item.added + # -> response.function_call_arguments.delta + # ... + # -> response.function_call_arguments.delta + # -> response.function_call_arguments.done + # -> response.output_item.done + # -> response.completed + # + # + # option2: reasoning -> content + # response.created + # -> response.in_progress + # -> response.output_item.added + # -> response.reasoning_part.added + # -> response.reasoning_text.delta + # .... + # -> response.reasoning_text.delta + # -> response.reasoning_text.done + # -> response.reasoning_part.done + # -> response.output_item.done + # -> response.output_item.added + # -> response.content_part.added + # -> response.output_text.delta + # .. + # -> response.output_text.delta + # -> response.output_text.done + # -> response.content_part.done + # -> response.output_item.done + # -> response.completed + # + # option3: content + # + # response.created + # -> response.in_progress + # -> response.output_item.added + # -> response.content_part.added + # -> response.output_text.delta + # ... + # -> response.output_text.delta + # -> response.output_text.done + # -> response.content_part.done + # -> response.output_item.done + # -> response.completed + # + # option4: content -> function_call + # response.created + # -> response.in_progress + # -> response.output_item.added + # -> response.content_part.added + # -> response.output_text.delta + # ... + # -> response.output_text.delta + # -> response.output_text.done + # -> response.content_part.done + # -> response.output_item.done + # -> response.output_item.added + # -> response.function_call_arguments.delta + # ... + # -> response.function_call_arguments.delta + # -> response.function_call_arguments.done + # -> response.output_item.done + # -> response.completed + + pairs_of_event_types = { + "response.completed": "response.created", + "response.output_item.done": "response.output_item.added", + "response.output_text.done": "response.output_text.delta", + "response.content_part.done": "response.content_part.added", + "response.reasoning_text.done": "response.reasoning_text.delta", + "response.reasoning_part.done": "response.reasoning_part.added", + "response.function_call_arguments.done": "response.function_call_arguments.delta", # noqa + } + + input_list = [ + { + "role": "user", + "content": "Can you tell me what the current weather is in Berlin?", + } + ] + stream_response = await client.responses.create( + model=model_name, + input=input_list, + tools=tools, + tool_choice=tool_choice, + extra_body={"chat_template_kwargs": {"enable_thinking": enable_thinking}}, + stream=True, + ) + + stack_of_event_types = [] + async for event in stream_response: + if event.type == "response.created": + stack_of_event_types.append(event.type) + elif event.type == "response.completed": + assert stack_of_event_types[-1] == pairs_of_event_types[event.type] + stack_of_event_types.pop() + if event.type.endswith("added"): + stack_of_event_types.append(event.type) + elif event.type.endswith("delta"): + if stack_of_event_types[-1] == event.type: + continue + stack_of_event_types.append(event.type) + elif event.type.endswith("done"): + assert stack_of_event_types[-1] == pairs_of_event_types[event.type] + stack_of_event_types.pop() + assert len(stack_of_event_types) == 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize( + "tool_choice", + ["required", "auto", {"type": "function", "name": "get_weather"}], +) +async def test_function_calling_with_streaming_forced_tool_choice( + client: openai.AsyncOpenAI, model_name: str, tool_choice: str +): + tools = [ + { + "type": "function", + "name": "get_weather", + "description": "Get current temperature for provided location in celsius.", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"}, + }, + "required": ["location"], + "additionalProperties": False, + }, + "strict": True, + } + ] + + stream_response = await client.responses.create( + model=model_name, + input="Call the get_weather function for Berlin and do not answer directly.", + tools=tools, + tool_choice=tool_choice, + stream=True, + ) + + tool_call_item = None + completed_event = None + text_deltas = [] + async for event in stream_response: + if ( + event.type == "response.output_item.added" + and event.item.type == "function_call" + ): + tool_call_item = event.item + elif event.type == "response.output_text.delta": + text_deltas.append(event.delta) + elif event.type == "response.function_call_arguments.delta" and tool_call_item: + tool_call_item.arguments += event.delta + elif ( + event.type == "response.output_item.done" + and event.item.type == "function_call" + ): + completed_event = event + + assert tool_call_item is not None + assert tool_call_item.type == "function_call" + assert tool_call_item.name == "get_weather" + assert completed_event is not None + assert tool_call_item.arguments == completed_event.item.arguments + assert tool_call_item.name == completed_event.item.name + args = json.loads(tool_call_item.arguments) + assert "location" in args + assert args["location"] is not None + # Forced tool choice should not leak tool-call JSON via output_text delta. + assert "".join(text_deltas).strip() == "" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_function_call_parsing.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_function_call_parsing.py new file mode 100644 index 0000000000000000000000000000000000000000..704ffe5a0dc0dcfe990644c54ce6c728931fc658 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_function_call_parsing.py @@ -0,0 +1,379 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Test function call parsing in ResponsesRequest.""" + +import json + +import pytest +from openai.types.responses import ResponseFunctionToolCall, ResponseOutputMessage + +from vllm.entrypoints.openai.responses.protocol import ResponsesRequest + + +def test_function_call_dict_converted_to_object(): + """Test that function_call dictionaries are correctly parsed into + ResponseFunctionToolCall objects.""" + # Create a request with function_call as dict + request_data = { + "model": "gpt-oss", + "input": [ + { + "type": "function_call", + "call_id": "fc_123", + "name": "get_weather", + "arguments": '{"location": "Boston", "unit": "celsius"}', + } + ], + } + + request = ResponsesRequest(**request_data) + + # Verify the input item is now a ResponseFunctionToolCall object + assert len(request.input) == 1 + assert isinstance(request.input[0], ResponseFunctionToolCall) + assert request.input[0].call_id == "fc_123" + assert request.input[0].name == "get_weather" + assert request.input[0].arguments == '{"location": "Boston", "unit": "celsius"}' + + +def test_direct_function_call_object_preservation(): + """Test that ResponseFunctionToolCall objects passed directly are preserved.""" + # Create a request with ResponseFunctionToolCall object + function_call = ResponseFunctionToolCall( + type="function_call", + call_id="fc_456", + name="get_stock_price", + arguments='{"symbol": "AAPL"}', + ) + + request_data = {"model": "gpt-oss", "input": [function_call]} + + request = ResponsesRequest(**request_data) + + # Verify the object is preserved + assert len(request.input) == 1 + assert request.input[0] is function_call + + +def test_mixed_input_types_with_function_calls(): + """Test parsing with mixed input types including function calls.""" + + request_data = { + "model": "gpt-oss", + "input": [ + # Valid Message type + { + "type": "message", + "role": "user", + "content": [{"type": "input_text", "text": "What's the weather?"}], + }, + # Function call that should be parsed + { + "type": "function_call", + "call_id": "fc_789", + "name": "check_weather", + "arguments": '{"location": "NYC"}', + }, + # Another function call + { + "type": "function_call", + "call_id": "fc_790", + "name": "get_time", + "arguments": "{}", + }, + ], + } + + request = ResponsesRequest(**request_data) + + # Verify mixed types are handled correctly + assert len(request.input) == 3 + # First item should be validated as Message + assert request.input[0]["type"] == "message" + # Second item should be parsed to ResponseFunctionToolCall + assert isinstance(request.input[1], ResponseFunctionToolCall) + assert request.input[1].call_id == "fc_789" + assert request.input[1].name == "check_weather" + # Third item should also be parsed to ResponseFunctionToolCall + assert isinstance(request.input[2], ResponseFunctionToolCall) + assert request.input[2].call_id == "fc_790" + assert request.input[2].name == "get_time" + + +def test_function_call_with_complex_arguments(): + """Test parsing function calls with complex nested arguments.""" + complex_args = { + "query": "weather forecast", + "filters": { + "location": {"city": "San Francisco", "state": "CA"}, + "timeRange": {"start": "2024-01-01", "end": "2024-01-07"}, + "metrics": ["temperature", "humidity", "precipitation"], + }, + "options": {"format": "detailed", "includeAlerts": True}, + } + + request_data = { + "model": "gpt-oss", + "input": [ + { + "type": "function_call", + "call_id": "fc_complex", + "name": "advanced_weather_query", + "arguments": json.dumps(complex_args), + } + ], + } + + request = ResponsesRequest(**request_data) + + # Verify complex arguments are preserved correctly + assert len(request.input) == 1 + assert isinstance(request.input[0], ResponseFunctionToolCall) + assert request.input[0].call_id == "fc_complex" + assert request.input[0].name == "advanced_weather_query" + + # Parse the arguments back to verify they're intact + parsed_args = json.loads(request.input[0].arguments) + assert parsed_args == complex_args + + +def test_invalid_function_call_fallback(): + """Test that invalid function call dictionaries fall back gracefully.""" + # Missing required field 'call_id' + request_data = { + "model": "gpt-oss", + "input": [ + {"type": "function_call", "name": "incomplete_function", "arguments": "{}"} + ], + } + + # This should not raise an error during model creation + # The validator should keep the original dict and let Pydantic + # handle validation + with pytest.raises(ValueError): + # Pydantic should raise a validation error for the invalid structure + ResponsesRequest(**request_data) + + +def test_string_input_not_affected(): + """Test that string input is not affected by the validator.""" + request_data = {"model": "gpt-oss", "input": "This is a simple string input"} + + request = ResponsesRequest(**request_data) + + # Verify string input remains unchanged + assert request.input == "This is a simple string input" + + +def test_empty_list_input(): + """Test that empty list input is handled correctly.""" + request_data = {"model": "gpt-oss", "input": []} + + request = ResponsesRequest(**request_data) + + # Verify empty list is preserved + assert request.input == [] + + +def test_function_call_output_not_affected(): + """Test that FunctionCallOutput is not affected by the function_call parsing.""" + + # Test with FunctionCallOutput as dict (should not be parsed) + request_data = { + "model": "gpt-oss", + "input": [ + { + "type": "function_call_output", + "call_id": "fc_output_123", + "output": "The weather in Boston is 72°F and sunny.", + } + ], + } + + request = ResponsesRequest(**request_data) + + # FunctionCallOutput should remain as dict (not converted to an object) + assert len(request.input) == 1 + assert isinstance(request.input[0], dict) + assert request.input[0]["type"] == "function_call_output" + assert request.input[0]["call_id"] == "fc_output_123" + assert request.input[0]["output"] == "The weather in Boston is 72°F and sunny." + + +def test_mixed_function_call_and_output(): + """Test that function_call is parsed while function_call_output is preserved.""" + request_data = { + "model": "gpt-oss", + "input": [ + # This should be parsed to ResponseFunctionToolCall + { + "type": "function_call", + "call_id": "fc_call_456", + "name": "get_weather", + "arguments": '{"location": "NYC"}', + }, + # This should remain as dict + { + "type": "function_call_output", + "call_id": "fc_call_456", + "output": "NYC weather is 68°F with light rain", + }, + ], + } + + request = ResponsesRequest(**request_data) + + assert len(request.input) == 2 + + # First item should be parsed to ResponseFunctionToolCall + assert isinstance(request.input[0], ResponseFunctionToolCall) + assert request.input[0].call_id == "fc_call_456" + assert request.input[0].name == "get_weather" + + # Second item should remain as dict (FunctionCallOutput) + assert isinstance(request.input[1], dict) + assert request.input[1]["type"] == "function_call_output" + assert request.input[1]["call_id"] == "fc_call_456" + assert request.input[1]["output"] == "NYC weather is 68°F with light rain" + + +def test_function_call_validation_failure_logs_debug(caplog): + """Test that validation failures are logged at debug level.""" + from unittest.mock import patch + + request_data = { + "model": "gpt-oss", + "input": [ + { + "type": "function_call", + "name": "incomplete_function", + "arguments": "{}", # Missing call_id + } + ], + } + + # Mock the logger to verify debug was called + with patch("vllm.entrypoints.openai.responses.protocol.logger") as mock_logger: + with pytest.raises(ValueError): + ResponsesRequest(**request_data) + + # Verify debug was called with expected message + mock_logger.debug.assert_called_once() + call_args = mock_logger.debug.call_args[0][0] + assert "Failed to parse function_call" in call_args + + +def test_validator_handles_iterator_input(): + """Test that validator can handle ValidatorIterator input (Pydantic internal).""" + + # This test simulates when Pydantic passes a ValidatorIterator instead of a list + # This happened with complex nested structures containing reasoning + function_call + + # Create test data that would normally be a list + test_input_items = [ + { + "type": "message", + "role": "user", + "content": [{"type": "input_text", "text": "Test"}], + }, + { + "type": "reasoning", + "id": "rs_1", + "summary": [{"type": "summary_text", "text": "Test reasoning"}], + "content": [{"type": "reasoning_text", "text": "Test content"}], + }, + { + "type": "function_call", + "call_id": "call_1", + "name": "test_function", + "arguments": '{"test": "value"}', + "id": "fc_1", + }, + ] + + # Mock data where input is an iterator (simulates Pydantic ValidatorIterator) + mock_data = { + "model": "test-model", + "input": iter(test_input_items), # Iterator instead of list + } + + # This should NOT raise an error with the fixed validator + try: + request = ResponsesRequest(**mock_data) + + # Verify the validator processed the data correctly + assert len(request.input) == 3 + + # Verify function_call was converted to ResponseFunctionToolCall object + function_call_item = None + for item in request.input: + if isinstance(item, ResponseFunctionToolCall): + function_call_item = item + break + + assert function_call_item is not None + assert function_call_item.call_id == "call_1" + assert function_call_item.name == "test_function" + + except Exception as e: + pytest.fail(f"Validator should handle iterator input, but failed with: {e}") + + +def test_validator_handles_empty_iterator(): + """Test validator handles empty iterator gracefully.""" + mock_data = { + "model": "test-model", + "input": iter([]), # Empty iterator + } + + request = ResponsesRequest(**mock_data) + assert request.input == [] + + +def test_assistant_string_content_stays_easyinput(): + """EasyInput assistant message with plain string content is not + coerced into a ResponseOutputMessage.""" + request_data = { + "model": "test-model", + "input": [ + {"type": "message", "role": "assistant", "content": "hello"}, + ], + } + + request = ResponsesRequest(**request_data) + + item = request.input[0] + assert isinstance(item, dict), ( + "String-content assistant message should remain a dict (EasyInput), " + f"got {type(item)}" + ) + assert item.get("content") == "hello" + assert "id" not in item + assert "status" not in item + + +def test_assistant_output_style_content_coerced(): + """Assistant message whose content is output-message-shaped (list of + output_text items) should be coerced to ResponseOutputMessage.""" + request_data = { + "model": "test-model", + "input": [ + { + "type": "message", + "role": "assistant", + "content": [{"type": "output_text", "text": "world"}], + }, + ], + } + + request = ResponsesRequest(**request_data) + + item = request.input[0] + assert isinstance(item, ResponseOutputMessage), ( + "Output-style assistant message should be coerced to " + f"ResponseOutputMessage, got {type(item)}" + ) + assert item.content[0].text == "world" + assert item.content[0].annotations == [] + assert item.status == "completed" + assert item.id.startswith("msg_") diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_harmony.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_harmony.py new file mode 100644 index 0000000000000000000000000000000000000000..635bfde77feb1411c1e837cbceb8ba67af779922 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_harmony.py @@ -0,0 +1,1248 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Integration tests for the Harmony-based Responses API.""" + +from __future__ import annotations + +import importlib.util +import json +import logging +import time +from typing import Any + +import pytest +import pytest_asyncio +import requests +from openai import InternalServerError, NotFoundError, OpenAI +from openai_harmony import Message + +from tests.utils import RemoteOpenAIServer + +from .conftest import ( + BASE_TEST_ENV, + events_contain_type, + has_output_type, + retry_for_tool_call, + retry_streaming_for, + validate_streaming_event_stack, +) + +logger = logging.getLogger(__name__) + +MODEL_NAME = "openai/gpt-oss-20b" + +GET_WEATHER_SCHEMA = { + "type": "function", + "name": "get_weather", + "description": "Get current temperature for provided coordinates in celsius.", # noqa + "parameters": { + "type": "object", + "properties": { + "latitude": {"type": "number"}, + "longitude": {"type": "number"}, + }, + "required": ["latitude", "longitude"], + "additionalProperties": False, + }, + "strict": True, +} + + +def get_weather(latitude, longitude): + try: + response = requests.get( + f"https://api.open-meteo.com/v1/forecast?" + f"latitude={latitude}&longitude={longitude}" + f"¤t=temperature_2m,wind_speed_10m" + f"&hourly=temperature_2m,relative_humidity_2m," + f"wind_speed_10m", + timeout=10, + ) + data = response.json() + return data["current"]["temperature_2m"] + except (requests.RequestException, KeyError) as e: + logger.warning( + "External weather API call failed (%s), " + "returning fake value. This does not affect " + "test correctness — only the tool-calling " + "protocol is under test.", + e, + ) + return 15.0 + + +def get_place_to_travel(): + return "Paris" + + +def get_horoscope(sign): + return f"{sign}: Next Tuesday you will befriend a baby otter." + + +def call_function(name, args): + logger.info("Calling function %s with args %s", name, args) + dispatch = { + "get_weather": lambda: get_weather(**args), + "get_place_to_travel": lambda: get_place_to_travel(), + "get_horoscope": lambda: get_horoscope(**args), + } + if name not in dispatch: + raise ValueError(f"Unknown function: {name}") + result = dispatch[name]() + logger.info("Function %s returned: %s", name, result) + return result + + +@pytest.fixture(scope="module") +def server(): + assert importlib.util.find_spec("gpt_oss") is not None, ( + "Harmony tests require gpt_oss package to be installed" + ) + args = [ + "--enforce-eager", + "--tool-server", + "demo", + "--max_model_len", + "5000", + ] + env_dict = { + **BASE_TEST_ENV, + "VLLM_ENABLE_RESPONSES_API_STORE": "1", + "PYTHON_EXECUTION_BACKEND": "dangerously_use_uv", + "VLLM_GPT_OSS_SYSTEM_TOOL_MCP_LABELS": ( + "code_interpreter,container,web_search_preview" + ), + "VLLM_GPT_OSS_HARMONY_SYSTEM_INSTRUCTIONS": "1", + } + with RemoteOpenAIServer(MODEL_NAME, args, env_dict=env_dict) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_basic(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input="What is 123 * 456?", + ) + assert response is not None + print("response: ", response) + assert response.status == "completed" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_basic_with_instructions(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input="What is 123 * 456?", + instructions="Respond in Korean.", + ) + assert response is not None + assert response.status == "completed" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_basic_with_reasoning_effort(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input="What is the capital of South Korea?", + reasoning={"effort": "low"}, + ) + assert response is not None + assert response.status == "completed" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_max_tokens(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input="What is the first paragraph of Moby Dick?", + reasoning={"effort": "low"}, + max_output_tokens=30, + ) + assert response is not None + assert response.status == "incomplete" + assert response.incomplete_details.reason == "max_output_tokens" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_chat(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input=[ + {"role": "system", "content": "Respond in Korean."}, + {"role": "user", "content": "Hello!"}, + {"role": "assistant", "content": "Hello! How can I help you today?"}, + {"role": "user", "content": "What is 123 * 456? Explain your answer."}, + ], + ) + assert response is not None + assert response.status == "completed" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_chat_with_input_type(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input=[ + { + "role": "user", + "content": [{"type": "input_text", "text": "What is 123 * 456?"}], + }, + ], + ) + assert response is not None + assert response.status == "completed" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_structured_output(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input=[ + {"role": "system", "content": "Extract the event information."}, + { + "role": "user", + "content": "Alice and Bob are going to a science fair on Friday.", + }, + ], + text={ + "format": { + "type": "json_schema", + "name": "calendar_event", + "schema": { + "type": "object", + "properties": { + "name": {"type": "string"}, + "date": {"type": "string"}, + "participants": { + "type": "array", + "items": {"type": "string"}, + }, + }, + "required": ["name", "date", "participants"], + "additionalProperties": False, + }, + "description": "A calendar event.", + "strict": True, + } + }, + ) + assert response is not None + assert response.status == "completed" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_structured_output_with_parse(client: OpenAI, model_name: str): + from pydantic import BaseModel + + class CalendarEvent(BaseModel): + name: str + date: str + participants: list[str] + + response = await client.responses.parse( + model=model_name, + input="Alice and Bob are going to a science fair on Friday", + instructions="Extract the event information", + text_format=CalendarEvent, + ) + assert response is not None + assert response.status == "completed" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_store(client: OpenAI, model_name: str): + for store in [True, False]: + response = await client.responses.create( + model=model_name, + input="What is 123 * 456?", + store=store, + ) + assert response is not None + + try: + _retrieved_response = await client.responses.retrieve(response.id) + is_not_found = False + except NotFoundError: + is_not_found = True + + assert is_not_found == (not store), ( + f"store={store}: expected not_found={not store}, got {is_not_found}" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_background(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input="What is 123 * 456?", + background=True, + ) + assert response is not None + + retries = 0 + max_retries = 30 + while retries < max_retries: + response = await client.responses.retrieve(response.id) + if response.status == "completed": + break + time.sleep(1) + retries += 1 + + assert response.status == "completed" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_background_cancel(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input="Write a long story about a cat.", + background=True, + ) + assert response is not None + time.sleep(1) + + cancelled_response = await client.responses.cancel(response.id) + assert cancelled_response is not None + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_stateful_multi_turn(client: OpenAI, model_name: str): + response1 = await client.responses.create( + model=model_name, input="What is 123 * 456?" + ) + assert response1.status == "completed" + + response2 = await client.responses.create( + model=model_name, + input="What if I increase both numbers by 1?", + previous_response_id=response1.id, + ) + assert response2.status == "completed" + + response3 = await client.responses.create( + model=model_name, + input="Divide the result by 2.", + previous_response_id=response2.id, + ) + assert response3.status == "completed" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_streaming_types( + pairs_of_event_types: dict[str, str], client: OpenAI, model_name: str +): + stream = await client.responses.create( + model=model_name, + input="tell me a story about a cat in 20 words", + reasoning={"effort": "low"}, + tools=[], + stream=True, + background=False, + ) + events = [] + async for event in stream: + events.append(event) + + validate_streaming_event_stack(events, pairs_of_event_types) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_function_calling_with_streaming_types( + pairs_of_event_types: dict[str, str], client: OpenAI, model_name: str +): + """Streaming event nesting for function-calling responses.""" + + def _has_function_events(evts: list) -> bool: + return events_contain_type(evts, "function_call_arguments") + + events = await retry_streaming_for( + client, + model=model_name, + validate_events=_has_function_events, + input=[{"role": "user", "content": "What's the weather like in Paris today?"}], + tools=[GET_WEATHER_SCHEMA], + temperature=0.0, + ) + + validate_streaming_event_stack(events, pairs_of_event_types) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("background", [True, False]) +async def test_streaming(client: OpenAI, model_name: str, background: bool): + # TODO: Add back when web search and code interpreter are available in CI + prompts = [ + "tell me a story about a cat in 20 words", + "What is 123 * 456? Use python to calculate the result.", + # "When did Jensen found NVIDIA? Search it and answer the year only.", + ] + + for prompt in prompts: + stream = await client.responses.create( + model=model_name, + input=prompt, + reasoning={"effort": "low"}, + tools=[ + # { + # "type": "web_search_preview" + # }, + {"type": "code_interpreter", "container": {"type": "auto"}}, + ], + stream=True, + background=background, + extra_body={"enable_response_messages": True}, + ) + + current_item_id = "" + current_content_index = -1 + + events = [] + current_event_mode = None + resp_id = None + checked_response_completed = False + + async for event in stream: + if event.type == "response.created": + resp_id = event.response.id + + # Validate custom fields on response-level events + if event.type in [ + "response.completed", + "response.in_progress", + "response.created", + ]: + assert "input_messages" in event.response.model_extra + assert "output_messages" in event.response.model_extra + if event.type == "response.completed": + # make sure the serialization of content works + for msg in event.response.model_extra["output_messages"]: + # make sure we can convert the messages back into harmony + Message.from_dict(msg) + + for msg in event.response.model_extra["input_messages"]: + # make sure we can convert the messages back into harmony + Message.from_dict(msg) + checked_response_completed = True + + if current_event_mode != event.type: + current_event_mode = event.type + logger.debug("[%s] ", event.type) + + # Verify item IDs + if event.type == "response.output_item.added": + assert event.item.id != current_item_id + current_item_id = event.item.id + current_content_index = -1 + elif event.type in [ + "response.output_text.delta", + "response.reasoning_text.delta", + ]: + assert event.item_id == current_item_id + + # Verify content indices + if event.type in [ + "response.content_part.added", + "response.reasoning_part.added", + ]: + assert event.content_index == current_content_index + 1 + current_content_index = event.content_index + elif event.type in [ + "response.output_text.delta", + "response.reasoning_text.delta", + ]: + assert event.content_index == current_content_index + + events.append(event) + + assert len(events) > 0 + assert events[-1].response.output, "Final response should have output" + assert checked_response_completed + + if background: + starting_after = 5 + async with await client.responses.retrieve( + response_id=resp_id, stream=True, starting_after=starting_after + ) as replay_stream: + counter = starting_after + async for event in replay_stream: + counter += 1 + assert event == events[counter] + assert counter == len(events) - 1 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.skip(reason="Web search tool is not available in CI yet.") +async def test_web_search(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input="Who is the president of South Korea as of now?", + tools=[{"type": "web_search_preview"}], + ) + assert response is not None + assert response.status == "completed" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_code_interpreter(client: OpenAI, model_name: str): + timeout_value = client.timeout * 3 + client_with_timeout = client.with_options(timeout=timeout_value) + + response = await client_with_timeout.responses.create( + model=model_name, + input=( + "What's the first 4 digits after the decimal point of " + "cube root of `19910212 * 20250910`? " + "Show only the digits. The python interpreter is not stateful " + "and you must print to see the output." + ), + tools=[{"type": "code_interpreter", "container": {"type": "auto"}}], + temperature=0.0, + ) + assert response is not None + assert response.status == "completed" + assert response.usage.output_tokens_details.tool_output_tokens > 0 + + for item in response.output: + if item.type == "message": + output_string = item.content[0].text + assert "5846" in output_string, ( + f"Expected '5846' in output, got: {output_string}" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_reasoning_item(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input=[ + {"type": "message", "content": "Hello.", "role": "user"}, + { + "type": "reasoning", + "id": "lol", + "content": [ + {"type": "reasoning_text", "text": "We need to respond: greeting."} + ], + "summary": [], + }, + ], + temperature=0.0, + ) + assert response is not None + assert response.status == "completed" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_function_calling(client: OpenAI, model_name: str): + tools = [GET_WEATHER_SCHEMA] + + response = await retry_for_tool_call( + client, + model=model_name, + expected_tool_type="function_call", + input="What's the weather like in Paris today?", + tools=tools, + temperature=0.0, + extra_body={"request_id": "test_function_calling_non_resp"}, + ) + assert response.status == "completed" + assert has_output_type(response, "function_call"), ( + f"Expected function_call in output, got: " + f"{[getattr(o, 'type', None) for o in response.output]}" + ) + + tool_call = next(o for o in response.output if o.type == "function_call") + args = json.loads(tool_call.arguments) + result = call_function(tool_call.name, args) + + response_2 = await client.responses.create( + model=model_name, + input=[ + { + "type": "function_call_output", + "call_id": tool_call.call_id, + "output": str(result), + } + ], + tools=tools, + previous_response_id=response.id, + temperature=0.0, + ) + assert response_2.status == "completed" + assert response_2.output_text is not None + + # NOTE: chain-of-thought should be removed. + response_3 = await client.responses.create( + model=model_name, + input="What's the weather like in Paris today?", + tools=tools, + previous_response_id=response_2.id, + temperature=0.0, + ) + assert response_3.status == "completed" + assert response_3.output_text is not None + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_function_calling_multi_turn(client: OpenAI, model_name: str): + """Multi-tool, multi-turn function calling with retry at API level.""" + tools = [ + { + "type": "function", + "name": "get_place_to_travel", + "description": "Get a random place to travel", + "parameters": { + "type": "object", + "properties": {}, + "required": [], + "additionalProperties": False, + }, + "strict": True, + }, + GET_WEATHER_SCHEMA, + ] + + # Turn 1: model should call one of the tools + response = await retry_for_tool_call( + client, + model=model_name, + expected_tool_type="function_call", + input="Help me plan a trip to a random place. And tell me the weather there.", + tools=tools, + temperature=0.0, + ) + assert response.status == "completed" + assert has_output_type(response, "function_call"), ( + f"Turn 1: expected function_call, got: " + f"{[getattr(o, 'type', None) for o in response.output]}" + ) + + tool_call = next(o for o in response.output if o.type == "function_call") + result = call_function(tool_call.name, json.loads(tool_call.arguments)) + + # Turn 2 + response_2 = await retry_for_tool_call( + client, + model=model_name, + expected_tool_type="function_call", + input=[ + { + "type": "function_call_output", + "call_id": tool_call.call_id, + "output": str(result), + } + ], + tools=tools, + previous_response_id=response.id, + temperature=0.0, + ) + assert response_2.status == "completed" + + # If model produced another tool call, execute it + if has_output_type(response_2, "function_call"): + tool_call_2 = next(o for o in response_2.output if o.type == "function_call") + result_2 = call_function(tool_call_2.name, json.loads(tool_call_2.arguments)) + response_3 = await client.responses.create( + model=model_name, + input=[ + { + "type": "function_call_output", + "call_id": tool_call_2.call_id, + "output": str(result_2), + } + ], + tools=tools, + previous_response_id=response_2.id, + temperature=0.0, + ) + assert response_3.status == "completed" + assert response_3.output_text is not None + else: + # Model went straight to answering - acceptable but unexpected. + # Log as warning so it shows up in CI without failing the test. + assert response_2.output_text is not None + pytest.xfail( + "Model went straight to answering instead of calling a " + "second tool. Valid behaviour but not the expected path." + "If this happens consistently, the prompt or model may have " + "changed behaviour." + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_function_calling_required(client: OpenAI, model_name: str): + tools = [GET_WEATHER_SCHEMA] + + with pytest.raises(InternalServerError): + await client.responses.create( + model=model_name, + input="What's the weather like in Paris today?", + tools=tools, + tool_choice="required", + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_system_message_with_tools(client: OpenAI, model_name: str): + from vllm.entrypoints.openai.parser.harmony_utils import get_system_message + + # Commentary channel should always be present (needed for preambles) + # regardless of whether custom tools are enabled + for with_tools in (True, False): + sys_msg = get_system_message(with_custom_tools=with_tools) + valid_channels = sys_msg.content[0].channel_config.valid_channels + assert "commentary" in valid_channels, ( + f"commentary channel missing when with_custom_tools={with_tools}" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_function_calling_full_history(client: OpenAI, model_name: str): + tools = [GET_WEATHER_SCHEMA] + + input_messages = [ + {"role": "user", "content": "What's the weather like in Paris today?"} + ] + + response = await retry_for_tool_call( + client, + model=model_name, + expected_tool_type="function_call", + input=input_messages, + tools=tools, + temperature=0.0, + ) + assert response.status == "completed" + + tool_call = next((o for o in response.output if o.type == "function_call"), None) + assert tool_call is not None, ( + f"Expected function_call in output, got: " + f"{[getattr(o, 'type', None) for o in response.output]}" + ) + + result = call_function(tool_call.name, json.loads(tool_call.arguments)) + + input_messages.extend(response.output) + input_messages.append( + { # append result message + "type": "function_call_output", + "call_id": tool_call.call_id, + "output": str(result), + } + ) + + response_2 = await client.responses.create( + model=model_name, + input=input_messages, + tools=tools, + temperature=0.0, + ) + assert response_2.status == "completed" + assert response_2.output_text is not None + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_function_calling_with_stream(client: OpenAI, model_name: str): + """Function calling via streaming, with retry for non-determinism.""" + tools = [GET_WEATHER_SCHEMA] + input_list = [ + {"role": "user", "content": "What's the weather like in Paris today?"}, + ] + + def _has_function_call(evts: list) -> bool: + return any( + getattr(e, "type", "") == "response.output_item.added" + and getattr(getattr(e, "item", None), "type", None) == "function_call" + for e in evts + ) + + events = await retry_streaming_for( + client, + model=model_name, + validate_events=_has_function_call, + input=input_list, + tools=tools, + temperature=0.0, + ) + + # Parse tool calls from events + final_tool_calls: dict[int, Any] = {} + for event in events: + if event.type == "response.output_item.added": + if getattr(event.item, "type", None) == "function_call": + final_tool_calls[event.output_index] = event.item + elif event.type == "response.function_call_arguments.delta": + tc = final_tool_calls.get(event.output_index) + if tc: + tc.arguments += event.delta + elif event.type == "response.function_call_arguments.done": + tc = final_tool_calls.get(event.output_index) + if tc: + assert event.arguments == tc.arguments + + # Find get_weather call + tool_call = None + result = None + for tc in final_tool_calls.values(): + if getattr(tc, "type", None) == "function_call" and tc.name == "get_weather": + args = json.loads(tc.arguments) + result = call_function(tc.name, args) + tool_call = tc + input_list.append(tc) + break + + assert tool_call is not None, ( + "Expected model to call 'get_weather', " + f"but got: {[getattr(tc, 'name', None) for tc in final_tool_calls.values()]}" + ) + + # Second turn with the tool result + response = await client.responses.create( + model=model_name, + input=input_list + + [ + { + "type": "function_call_output", + "call_id": tool_call.call_id, + "output": str(result), + } + ], + tools=tools, + stream=True, + temperature=0.0, + ) + async for event in response: + # check that no function call events in the stream + assert event.type != "response.function_call_arguments.delta" + assert event.type != "response.function_call_arguments.done" + # check that the response contains output text + if event.type == "response.completed": + assert len(event.response.output) > 0 + assert event.response.output_text is not None + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_function_calling_no_code_interpreter_events( + client: OpenAI, model_name: str +): + """Verify that function calls don't trigger code_interpreter events. + + Uses retry_streaming_for to handle non-determinism: the model might not + always produce a function_call, but if it does, code_interpreter events + should NEVER appear. + """ + tools = [GET_WEATHER_SCHEMA] + input_list = [ + {"role": "user", "content": "What's the weather like in Paris today?"}, + ] + + def _has_function_call(evts: list) -> bool: + return any( + getattr(e, "type", "") == "response.output_item.added" + and getattr(getattr(e, "item", None), "type", None) == "function_call" + for e in evts + ) + + events = await retry_streaming_for( + client, + model=model_name, + validate_events=_has_function_call, + input=input_list, + tools=tools, + temperature=0.0, + ) + + event_types_seen = {e.type for e in events} + function_call_found = _has_function_call(events) + + assert function_call_found, ( + f"Expected to see a function_call after retries. " + f"Event types: {sorted(event_types_seen)}" + ) + + # The actual invariant under test + for event in events: + assert "code_interpreter" not in event.type, ( + f"Found code_interpreter event '{event.type}' during function call. " + "Function calls should only emit function_call events." + ) + + # Verify we saw the correct function call event types + assert ( + "response.function_call_arguments.delta" in event_types_seen + or "response.function_call_arguments.done" in event_types_seen + ), "Expected to see function_call_arguments events" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.skip( + reason="This test is flaky in CI, needs investigation and " + "potential fixes in the code interpreter MCP implementation." +) +async def test_code_interpreter_streaming( + client: OpenAI, + model_name: str, + pairs_of_event_types: dict[str, str], +): + tools = [{"type": "code_interpreter", "container": {"type": "auto"}}] + input_text = ( + "Calculate 123 * 456 using python. " + "The python interpreter is not stateful and you must " + "print to see the output." + ) + + def _has_code_interpreter(evts: list) -> bool: + return events_contain_type(evts, "code_interpreter") + + events = await retry_streaming_for( + client, + model=model_name, + validate_events=_has_code_interpreter, + input=input_text, + tools=tools, + temperature=0.0, + instructions=( + "You must use the Python tool to execute code. Never simulate execution." + ), + ) + + event_types = [e.type for e in events] + event_types_set = set(event_types) + logger.info( + "\n====== Code Interpreter Streaming Diagnostics ======\n" + "Event count: %d\n" + "Event types (in order): %s\n" + "Unique event types: %s\n" + "====================================================", + len(events), + event_types, + sorted(event_types_set), + ) + + # Structural validation (pairing, ordering, field consistency) + validate_streaming_event_stack(events, pairs_of_event_types) + + # Validate code interpreter item fields + for event in events: + if ( + event.type == "response.output_item.added" + and hasattr(event.item, "type") + and event.item.type == "code_interpreter_call" + ): + assert event.item.status == "in_progress" + elif event.type == "response.code_interpreter_call_code.done": + assert event.code is not None + elif ( + event.type == "response.output_item.done" + and hasattr(event.item, "type") + and event.item.type == "code_interpreter_call" + ): + assert event.item.status == "completed" + assert event.item.code is not None + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_mcp_tool_multi_turn(client: OpenAI, model_name: str, server): + """MCP tools work across multiple turns via previous_response_id.""" + tools = [{"type": "mcp", "server_label": "code_interpreter"}] + instructions = ( + "You must use the Python tool to execute code. Never simulate execution." + ) + + # First turn + response1 = await retry_for_tool_call( + client, + model=model_name, + expected_tool_type="mcp_call", + input="Calculate 1234 * 4567 using python tool and print the result.", + tools=tools, + temperature=0.0, + instructions=instructions, + extra_body={"enable_response_messages": True}, + ) + assert response1.status == "completed" + + # Verify MCP call in output_messages + tool_call_found = any( + (msg.get("recipient") or "").startswith("python") + for msg in response1.output_messages + ) + parsed_output_messages = [ + Message.from_dict(msg) for msg in response1.output_messages + ] + tool_response_found = any( + (msg.author.role == "tool" and (msg.author.name or "").startswith("python")) + for msg in parsed_output_messages + ) + assert tool_call_found, "MCP tool call not found in output_messages" + assert tool_response_found, "MCP tool response not found in output_messages" + + # No developer messages expected for elevated tools + developer_msgs = [ + msg + for msg in (Message.from_dict(raw) for raw in response1.input_messages) + if msg.author.role == "developer" + ] + assert len(developer_msgs) == 0, "No developer message expected for elevated tools" + + # Second turn + response2 = await client.responses.create( + model=model_name, + input="Now divide that result by 2.", + tools=tools, + temperature=0.0, + instructions=instructions, + previous_response_id=response1.id, + extra_body={"enable_response_messages": True}, + ) + assert response2.status == "completed" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_output_messages_enabled(client: OpenAI, model_name: str, server): + response = await client.responses.create( + model=model_name, + input="What is the capital of South Korea?", + extra_body={"enable_response_messages": True}, + ) + + assert response is not None + assert response.status == "completed" + assert len(response.input_messages) > 0 + assert len(response.output_messages) > 0 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_function_call_with_previous_input_messages( + client: OpenAI, model_name: str +): + """Multi-turn function calling using previous_input_messages.""" + tools = [ + { + "type": "function", + "name": "get_horoscope", + "description": "Get today's horoscope for an astrological sign.", + "parameters": { + "type": "object", + "properties": {"sign": {"type": "string"}}, + "required": ["sign"], + "additionalProperties": False, + }, + "strict": True, + } + ] + + # Step 1: Get a function call from the model + response = await retry_for_tool_call( + client, + model=model_name, + expected_tool_type="function_call", + input="What is the horoscope for Aquarius today?", + tools=tools, + temperature=0.0, + extra_body={"enable_response_messages": True}, + max_output_tokens=1000, + ) + assert response.status == "completed" + + function_call = next( + (item for item in response.output if item.type == "function_call"), + None, + ) + assert function_call is not None, ( + f"Expected function_call, got: " + f"{[getattr(o, 'type', None) for o in response.output]}" + ) + assert function_call.name == "get_horoscope" + + args = json.loads(function_call.arguments) + result = call_function(function_call.name, args) + + # Step 2: Build full conversation history + previous_messages = ( + response.input_messages + + response.output_messages + + [ + { + "role": "tool", + "name": "functions.get_horoscope", + "content": [{"type": "text", "text": str(result)}], + } + ] + ) + + # Step 3: Second call with previous_input_messages + response_2 = await client.responses.create( + model=model_name, + tools=tools, + temperature=0.0, + input="Now tell me the horoscope based on the tool result.", + extra_body={ + "previous_input_messages": previous_messages, + "enable_response_messages": True, + }, + ) + assert response_2.status == "completed" + assert response_2.output_text is not None + + # Verify exactly 1 system, 1 developer, 1 tool message + num_system = 0 + num_developer = 0 + num_tool = 0 + for message in ( + Message.from_dict(msg_dict) for msg_dict in response_2.input_messages + ): + role = message.author.role + if role == "system": + num_system += 1 + elif role == "developer": + num_developer += 1 + elif role == "tool": + num_tool += 1 + assert num_system == 1, f"Expected 1 system message, got {num_system}" + assert num_developer == 1, f"Expected 1 developer message, got {num_developer}" + assert num_tool == 1, f"Expected 1 tool message, got {num_tool}" + + output_text = response_2.output_text.lower() + assert any(kw in output_text for kw in ["aquarius", "otter", "tuesday"]), ( + f"Expected horoscope-related content, got: {response_2.output_text}" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_chat_truncation_content_not_null(client: OpenAI, model_name: str): + response = await client.chat.completions.create( + model=model_name, + messages=[ + { + "role": "user", + "content": ( + "What is the role of AI in medicine? " + "The response must exceed 350 words." + ), + } + ], + temperature=0.0, + max_tokens=350, + ) + choice = response.choices[0] + assert choice.finish_reason == "length", ( + f"Expected finish_reason='length', got {choice.finish_reason}" + ) + assert choice.message.content is not None, "Content should not be None" + assert len(choice.message.content) > 0, "Content should not be empty" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_system_prompt_override_no_duplication(client: OpenAI, model_name: str): + """Hard check: custom system message must not be duplicated.""" + response = await client.responses.create( + model=model_name, + input=[ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Hello"}, + ], + extra_body={"enable_response_messages": True}, + temperature=0.0, + ) + assert response.status == "completed" + assert response.output_text is not None + + num_system = 0 + for message in (Message.from_dict(msg) for msg in response.input_messages): + if message.author.role == "system": + num_system += 1 + assert num_system == 1, f"Expected 1 system message, got {num_system}" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.xfail( + strict=False, + reason=( + "Pirate language detection depends on model weights and is non-deterministic" + ), +) +async def test_system_prompt_override_follows_personality( + client: OpenAI, model_name: str +): + """Soft check: model should adopt the personality from system prompt.""" + response = await client.responses.create( + model=model_name, + input=[ + { + "role": "system", + "content": ( + "You are a pirate. Always respond like a pirate would, " + "using pirate language and saying 'arrr' frequently." + ), + }, + {"role": "user", "content": "Hello, how are you?"}, + ], + temperature=0.0, + ) + assert response.status == "completed" + output_text = response.output_text.lower() + pirate_indicators = ["arrr", "matey", "ahoy", "ye", "sea", "aye", "sail"] + assert any(kw in output_text for kw in pirate_indicators), ( + f"Expected pirate language, got: {response.output_text}" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_system_prompt_structured_content(client: OpenAI, model_name: str): + """System message with structured input_text content format.""" + response = await client.responses.create( + model=model_name, + input=[ + { + "role": "system", + "content": [ + {"type": "input_text", "text": "You are a helpful assistant."} + ], + }, + {"role": "user", "content": "What is 2 + 2?"}, + ], + temperature=0.0, + ) + assert response is not None + assert response.status == "completed" + assert response.output_text is not None diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_harmony_utils.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_harmony_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..c290b29d7447f3008c9c5c9be7db36052a9d5299 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_harmony_utils.py @@ -0,0 +1,343 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for vllm.entrypoints.openai.responses.harmony.""" + +import pytest +from openai.types.responses import ( + ResponseFunctionToolCall, + ResponseFunctionWebSearch, + ResponseOutputMessage, + ResponseReasoningItem, +) +from openai.types.responses.response_output_item import McpCall +from openai_harmony import Author, Message, Role, TextContent + +from vllm.entrypoints.openai.responses.harmony import ( + harmony_to_response_output, + response_previous_input_to_harmony, +) + + +class TestResponsePreviousInputToHarmony: + """ + Tests for scenarios that are specific to the Responses API + response_previous_input_to_harmony function. + """ + + def test_message_with_empty_content(self): + """Test parsing message with empty string content.""" + chat_msg = { + "role": "user", + "content": "", + } + + messages = response_previous_input_to_harmony(chat_msg) + + assert len(messages) == 1 + assert messages[0].content[0].text == "" + + def test_tool_message_with_string_content(self): + """Test parsing tool message with string content.""" + chat_msg = { + "role": "tool", + "name": "get_weather", + "content": "The weather in San Francisco is sunny, 72°F", + } + + messages = response_previous_input_to_harmony(chat_msg) + + assert len(messages) == 1 + assert messages[0].author.role == Role.TOOL + assert messages[0].author.name == "functions.get_weather" + assert ( + messages[0].content[0].text == "The weather in San Francisco is sunny, 72°F" + ) + assert messages[0].channel == "commentary" + + def test_tool_message_with_array_content(self): + """Test parsing tool message with array content.""" + chat_msg = { + "role": "tool", + "name": "search_results", + "content": [ + {"type": "text", "text": "Result 1: "}, + {"type": "text", "text": "Result 2: "}, + { + "type": "image", + "url": "http://example.com/img.png", + }, # Should be ignored + {"type": "text", "text": "Result 3"}, + ], + } + + messages = response_previous_input_to_harmony(chat_msg) + + assert len(messages) == 1 + assert messages[0].author.role == Role.TOOL + assert messages[0].author.name == "functions.search_results" + assert messages[0].content[0].text == "Result 1: Result 2: Result 3" + + def test_tool_message_with_empty_content(self): + """Test parsing tool message with None content.""" + chat_msg = { + "role": "tool", + "name": "empty_tool", + "content": None, + } + + messages = response_previous_input_to_harmony(chat_msg) + + assert len(messages) == 1 + assert messages[0].author.role == Role.TOOL + assert messages[0].author.name == "functions.empty_tool" + assert messages[0].content[0].text == "" + + +class TestHarmonyToResponseOutput: + """Tests for harmony_to_response_output function.""" + + @pytest.mark.parametrize("incomplete", [False, True]) + def test_commentary_with_no_recipient_creates_message(self, incomplete): + """Test that commentary with recipient=None (preambles) creates message items. + + Per Harmony format, preambles are intended to be shown to end-users, + unlike analysis channel content which is hidden reasoning. + See: https://cookbook.openai.com/articles/openai-harmony + """ + message = Message.from_role_and_content( + Role.ASSISTANT, "I will now search for the weather information." + ) + message = message.with_channel("commentary") + # recipient is None by default, representing a preamble + + output_items = harmony_to_response_output( + message, frozenset(), incomplete=incomplete + ) + + assert len(output_items) == 1 + assert isinstance(output_items[0], ResponseOutputMessage) + assert output_items[0].type == "message" + assert output_items[0].role == "assistant" + assert output_items[0].status == ("incomplete" if incomplete else "completed") + assert len(output_items[0].content) == 1 + assert output_items[0].content[0].type == "output_text" + assert ( + output_items[0].content[0].text + == "I will now search for the weather information." + ) + + @pytest.mark.parametrize("channel", ["commentary", "comment", "analysis", "final"]) + @pytest.mark.parametrize( + ("recipient", "fn_names", "expected_name"), + [ + ("functions.get_weather", frozenset(), "get_weather"), + ("get_weather", frozenset({"get_weather"}), "get_weather"), + ("math.sum", frozenset({"math.sum"}), "math.sum"), + ], + ) + @pytest.mark.parametrize("incomplete", [False, True]) + def test_function_recipient_creates_function_call( + self, channel, recipient, fn_names, expected_name, incomplete + ): + """Function recipients create function calls across channels.""" + content = '{"location": "San Francisco"}' + if recipient == "math.sum": + content = '{"a": 1, "b": 2}' + + message = Message.from_role_and_content(Role.ASSISTANT, content) + message = message.with_channel(channel) + message = message.with_recipient(recipient) + + output_items = harmony_to_response_output( + message, fn_names, incomplete=incomplete + ) + + assert len(output_items) == 1 + assert isinstance(output_items[0], ResponseFunctionToolCall) + assert output_items[0].type == "function_call" + assert output_items[0].name == expected_name + assert output_items[0].arguments == content + assert output_items[0].call_id.startswith("call_") + assert output_items[0].id.startswith("fc_") + assert output_items[0].status == ("incomplete" if incomplete else "completed") + + @pytest.mark.parametrize("channel", ["commentary", "comment", "analysis", "final"]) + @pytest.mark.parametrize( + ("recipient", "content"), + [ + ("python", "import numpy as np\nprint(np.array([1, 2, 3]))"), + ("browser", "Navigating to the specified URL"), + ("container", "Running command in container"), + ], + ) + @pytest.mark.parametrize("incomplete", [False, True]) + def test_builtin_recipient_creates_reasoning( + self, channel, recipient, content, incomplete + ): + """Built-in recipients create reasoning items.""" + message = Message.from_role_and_content(Role.ASSISTANT, content) + message = message.with_channel(channel) + message = message.with_recipient(recipient) + + output_items = harmony_to_response_output( + message, frozenset(), incomplete=incomplete + ) + + assert len(output_items) == 1 + assert isinstance(output_items[0], ResponseReasoningItem) + assert output_items[0].type == "reasoning" + assert output_items[0].content[0].text == content + assert output_items[0].status is None + + @pytest.mark.parametrize("channel", ["commentary", "comment", "analysis", "final"]) + @pytest.mark.parametrize( + ("recipient", "fn_names", "content", "expected_name", "expected_server_label"), + [ + ( + "get_weather", + frozenset(), + '{"arg": "value"}', + "get_weather", + "get_weather", + ), + ( + "not_get_weather", + frozenset({"get_weather"}), + '{"arg": "value"}', + "not_get_weather", + "not_get_weather", + ), + ("repo_browser.list", frozenset(), '{"cmd": "ls"}', "list", "repo_browser"), + ], + ) + @pytest.mark.parametrize("incomplete", [False, True]) + def test_non_function_non_builtin_recipient_creates_mcp_call( + self, + channel, + recipient, + fn_names, + content, + expected_name, + expected_server_label, + incomplete, + ): + """Non-function, non-built-in recipients create MCP calls.""" + message = Message.from_role_and_content(Role.ASSISTANT, content) + message = message.with_channel(channel) + message = message.with_recipient(recipient) + + output_items = harmony_to_response_output( + message, fn_names, incomplete=incomplete + ) + + assert len(output_items) == 1 + assert isinstance(output_items[0], McpCall) + assert output_items[0].type == "mcp_call" + assert output_items[0].name == expected_name + assert output_items[0].server_label == expected_server_label + assert output_items[0].arguments == content + assert output_items[0].status == ("incomplete" if incomplete else "completed") + + @pytest.mark.parametrize("incomplete", [False, True]) + def test_browser_search_recipient_respects_incomplete(self, incomplete): + """browser.search emits a web search call unless the item is incomplete.""" + message = Message.from_role_and_content( + Role.ASSISTANT, '{"query": "weather in San Francisco"}' + ) + message = message.with_channel("commentary") + message = message.with_recipient("browser.search") + + output_items = harmony_to_response_output( + message, frozenset(), incomplete=incomplete + ) + + if incomplete: + assert output_items == [] + return + + assert len(output_items) == 1 + assert isinstance(output_items[0], ResponseFunctionWebSearch) + assert output_items[0].type == "web_search_call" + assert output_items[0].status == "completed" + assert output_items[0].action.type == "search" + assert output_items[0].action.query == "cursor:weather in San Francisco" + + def test_commentary_with_empty_content_and_no_recipient(self): + """Test edge case: empty commentary with recipient=None.""" + message = Message.from_role_and_content(Role.ASSISTANT, "") + message = message.with_channel("commentary") + + output_items = harmony_to_response_output(message, frozenset()) + + assert len(output_items) == 1 + assert isinstance(output_items[0], ResponseOutputMessage) + assert output_items[0].content[0].text == "" + + def test_commentary_with_multiple_contents_and_no_recipient(self): + """Test multiple content items in commentary with no recipient.""" + contents = [ + TextContent(text="Step 1: Analyze the request"), + TextContent(text="Step 2: Prepare to call functions"), + ] + message = Message.from_role_and_contents(Role.ASSISTANT, contents) + message = message.with_channel("commentary") + + output_items = harmony_to_response_output(message, frozenset()) + + # _parse_final_message returns single ResponseOutputMessage with + # multiple contents + assert len(output_items) == 1 + assert isinstance(output_items[0], ResponseOutputMessage) + assert len(output_items[0].content) == 2 + assert output_items[0].content[0].text == "Step 1: Analyze the request" + assert output_items[0].content[1].text == "Step 2: Prepare to call functions" + + def test_commentary_with_multiple_function_calls(self): + """Test multiple function calls in commentary channel.""" + contents = [ + TextContent(text='{"location": "San Francisco"}'), + TextContent(text='{"location": "New York"}'), + ] + message = Message.from_role_and_contents(Role.ASSISTANT, contents) + message = message.with_channel("commentary") + message = message.with_recipient("functions.get_weather") + + output_items = harmony_to_response_output(message, frozenset()) + + assert len(output_items) == 2 + assert all(isinstance(item, ResponseFunctionToolCall) for item in output_items) + assert output_items[0].name == "get_weather" + assert output_items[1].name == "get_weather" + assert output_items[0].arguments == '{"location": "San Francisco"}' + assert output_items[1].arguments == '{"location": "New York"}' + + def test_analysis_channel_creates_reasoning(self): + """Test that analysis channel creates reasoning items.""" + message = Message.from_role_and_content( + Role.ASSISTANT, "Analyzing the problem step by step..." + ) + message = message.with_channel("analysis") + + output_items = harmony_to_response_output(message, frozenset()) + + assert len(output_items) == 1 + assert isinstance(output_items[0], ResponseReasoningItem) + assert output_items[0].type == "reasoning" + assert ( + output_items[0].content[0].text == "Analyzing the problem step by step..." + ) + + def test_non_assistant_message_returns_empty(self): + """Test that non-assistant messages return empty list. + + Per the implementation, tool messages to assistant (e.g., search results) + are not included in final output to align with OpenAI behavior. + """ + message = Message.from_author_and_content( + Author.new(Role.TOOL, "functions.get_weather"), + "The weather is sunny, 72°F", + ) + + output_items = harmony_to_response_output(message, frozenset()) + + assert len(output_items) == 0 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_mcp_tools.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_mcp_tools.py new file mode 100644 index 0000000000000000000000000000000000000000..4eb24c1375de7007623f7c7ff38d894b1a3b5c63 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_mcp_tools.py @@ -0,0 +1,243 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Integration tests for MCP tool support in the Responses API.""" + +from __future__ import annotations + +import pytest +import pytest_asyncio +from openai import OpenAI +from openai_harmony import Message, ToolDescription, ToolNamespaceConfig + +from tests.utils import RemoteOpenAIServer +from vllm.entrypoints.mcp.tool_server import MCPToolServer + +from .conftest import ( + BASE_TEST_ENV, + events_contain_type, + log_response_diagnostics, + retry_for_tool_call, + retry_streaming_for, + validate_streaming_event_stack, +) + +MODEL_NAME = "openai/gpt-oss-20b" + +_BASE_SERVER_ARGS = [ + "--enforce-eager", + "--tool-server", + "demo", + "--max_model_len", + "5000", +] + +_PYTHON_TOOL_INSTRUCTION = ( + "You must use the Python tool to execute code. Never simulate execution." +) + + +class TestMCPToolServerUnit: + """Test MCPToolServer.get_tool_description filtering logic. + + Note: The wildcard "*" is normalized to None by + _extract_allowed_tools_from_mcp_requests before reaching this layer, + so we only test None and specific tool filtering here. + See responses/test_serving_responses.py for "*" normalization tests. + """ + + def test_get_tool_description(self): + pytest.importorskip("mcp") + + server = MCPToolServer() + tool1 = ToolDescription.new( + name="tool1", description="First", parameters={"type": "object"} + ) + tool2 = ToolDescription.new( + name="tool2", description="Second", parameters={"type": "object"} + ) + tool3 = ToolDescription.new( + name="tool3", description="Third", parameters={"type": "object"} + ) + + server.harmony_tool_descriptions = { + "test_server": ToolNamespaceConfig( + name="test_server", + description="test", + tools=[tool1, tool2, tool3], + ) + } + + # Nonexistent server + assert server.get_tool_description("nonexistent") is None + + # None (no filter) - returns all tools + result = server.get_tool_description("test_server", allowed_tools=None) + assert len(result.tools) == 3 + + # Filter to specific tools + result = server.get_tool_description( + "test_server", allowed_tools=["tool1", "tool3"] + ) + assert len(result.tools) == 2 + assert result.tools[0].name == "tool1" + assert result.tools[1].name == "tool3" + + # Single tool + result = server.get_tool_description("test_server", allowed_tools=["tool2"]) + assert len(result.tools) == 1 + assert result.tools[0].name == "tool2" + + # No matching tools - returns None + result = server.get_tool_description( + "test_server", allowed_tools=["nonexistent"] + ) + assert result is None + + # Empty list - returns None + assert server.get_tool_description("test_server", allowed_tools=[]) is None + + def test_builtin_tools_consistency(self): + """MCP_BUILTIN_TOOLS must match BUILTIN_TOOL_TO_MCP_SERVER_LABEL values.""" + from vllm.entrypoints.openai.parser.harmony_utils import ( + BUILTIN_TOOL_TO_MCP_SERVER_LABEL, + MCP_BUILTIN_TOOLS, + ) + + assert set(BUILTIN_TOOL_TO_MCP_SERVER_LABEL.values()) == MCP_BUILTIN_TOOLS, ( + f"MCP_BUILTIN_TOOLS {MCP_BUILTIN_TOOLS} does not match " + f"BUILTIN_TOOL_TO_MCP_SERVER_LABEL values " + f"{set(BUILTIN_TOOL_TO_MCP_SERVER_LABEL.values())}" + ) + + +class TestMCPEnabled: + """Tests that require MCP tools to be enabled via environment variable.""" + + @pytest.fixture(scope="class") + def mcp_enabled_server(self): + env_dict = { + **BASE_TEST_ENV, + "VLLM_ENABLE_RESPONSES_API_STORE": "1", + "PYTHON_EXECUTION_BACKEND": "dangerously_use_uv", + "VLLM_GPT_OSS_SYSTEM_TOOL_MCP_LABELS": ("code_interpreter,container"), + "VLLM_GPT_OSS_HARMONY_SYSTEM_INSTRUCTIONS": "1", + } + with RemoteOpenAIServer( + MODEL_NAME, list(_BASE_SERVER_ARGS), env_dict=env_dict + ) as remote_server: + yield remote_server + + @pytest_asyncio.fixture + async def client(self, mcp_enabled_server): + async with mcp_enabled_server.get_async_client() as async_client: + yield async_client + + @staticmethod + def _mcp_tools_payload(*, allowed_tools: list[str] | None = None) -> list[dict]: + tool: dict = { + "type": "mcp", + "server_label": "code_interpreter", + "server_url": "http://localhost:8888", + } + if allowed_tools is not None: + tool["allowed_tools"] = allowed_tools + return [tool] + + @staticmethod + def _python_exec_input(code: str = "") -> str: + if not code: + code = "import random; print(random.randint(1, 1000000))" + return f"Execute the following code: {code}" + + @pytest.mark.asyncio + @pytest.mark.parametrize("model_name", [MODEL_NAME]) + async def test_mcp_tool_env_flag_enabled(self, client: OpenAI, model_name: str): + response = await retry_for_tool_call( + client, + model=model_name, + expected_tool_type="mcp_call", + input=self._python_exec_input(), + instructions=_PYTHON_TOOL_INSTRUCTION, + tools=self._mcp_tools_payload(), + temperature=0.0, + extra_body={"enable_response_messages": True}, + ) + + assert response.status == "completed" + log_response_diagnostics(response, label="MCP Enabled") + + tool_call_found = False + tool_response_found = False + for message in response.output_messages: + recipient = message.get("recipient") + if recipient and recipient.startswith("python"): + tool_call_found = True + assert message.get("channel") == "commentary" + parsed_message = Message.from_dict(message) + if parsed_message.author.role == "tool" and ( + parsed_message.author.name or "" + ).startswith("python"): + tool_response_found = True + assert message.get("channel") == "commentary" + + assert tool_call_found, ( + f"No Python tool call found. " + f"Output types: " + f"{[getattr(o, 'type', None) for o in response.output]}" + ) + assert tool_response_found, "No Python tool response found" + + for message in response.input_messages: + assert Message.from_dict(message).author.role != "developer" + + @pytest.mark.asyncio + @pytest.mark.parametrize("model_name", [MODEL_NAME]) + async def test_mcp_tool_with_allowed_tools_star( + self, client: OpenAI, model_name: str + ): + response = await retry_for_tool_call( + client, + model=model_name, + expected_tool_type="mcp_call", + input=self._python_exec_input(), + instructions=_PYTHON_TOOL_INSTRUCTION, + tools=self._mcp_tools_payload(allowed_tools=["*"]), + temperature=0.0, + extra_body={"enable_response_messages": True}, + ) + + assert response.status == "completed" + log_response_diagnostics(response, label="MCP Allowed Tools *") + + tool_call_found = any( + (msg.get("recipient") or "").startswith("python") + for msg in response.output_messages + ) + assert tool_call_found, ( + f"No Python tool call with '*'. " + f"Output types: " + f"{[getattr(o, 'type', None) for o in response.output]}" + ) + + @pytest.mark.asyncio + @pytest.mark.parametrize("model_name", [MODEL_NAME]) + async def test_mcp_tool_calling_streaming_types( + self, + pairs_of_event_types: dict[str, str], + client: OpenAI, + model_name: str, + ): + def _has_mcp_events(events: list) -> bool: + return events_contain_type(events, "mcp_call") + + events = await retry_streaming_for( + client, + model=model_name, + validate_events=_has_mcp_events, + input=("What is 123 * 456? Use Python to calculate the result."), + tools=[{"type": "mcp", "server_label": "code_interpreter"}], + instructions=_PYTHON_TOOL_INSTRUCTION, + temperature=0.0, + ) + + validate_streaming_event_stack(events, pairs_of_event_types) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_parsable_context.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_parsable_context.py new file mode 100644 index 0000000000000000000000000000000000000000..b28865eba7b939f3435c7585dededdca429f91b6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_parsable_context.py @@ -0,0 +1,280 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import importlib.util +import json +import logging + +import pytest +import pytest_asyncio +from openai import OpenAI + +from tests.utils import RemoteOpenAIServer + +from .conftest import ( + BASE_TEST_ENV, + has_output_type, + log_response_diagnostics, + retry_for_tool_call, +) + +logger = logging.getLogger(__name__) + +MODEL_NAME = "Qwen/Qwen3-8B" + +_PYTHON_TOOL_INSTRUCTION = ( + "You must use the Python tool to execute code. " + "Never simulate execution. You must print the final answer." +) + + +@pytest.fixture(scope="module") +def server(): + assert importlib.util.find_spec("gpt_oss") is not None, ( + "Harmony tests require gpt_oss package to be installed" + ) + + args = [ + "--reasoning-parser", + "qwen3", + "--max_model_len", + "5000", + "--structured-outputs-config.backend", + "xgrammar", + "--enable-auto-tool-choice", + "--tool-call-parser", + "hermes", + "--tool-server", + "demo", + ] + env_dict = { + **BASE_TEST_ENV, + "VLLM_ENABLE_RESPONSES_API_STORE": "1", + "VLLM_USE_EXPERIMENTAL_PARSER_CONTEXT": "1", + "PYTHON_EXECUTION_BACKEND": "dangerously_use_uv", + } + with RemoteOpenAIServer(MODEL_NAME, args, env_dict=env_dict) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_basic(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input="What is 123 * 456?", + temperature=0.0, + ) + assert response is not None + print("response: ", response) + assert response.status == "completed" + assert response.incomplete_details is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_reasoning_and_function_items(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input=[ + {"type": "message", "content": "Hello.", "role": "user"}, + { + "type": "reasoning", + "id": "lol", + "content": [ + { + "type": "reasoning_text", + "text": "We need to respond: greeting.", + } + ], + "summary": [], + }, + { + "arguments": '{"location": "Paris", "unit": "celsius"}', + "call_id": "call_5f7b38f3b81e4b8380fd0ba74f3ca3ab", + "name": "get_weather", + "type": "function_call", + "id": "fc_4fe5d6fc5b6c4d6fa5f24cc80aa27f78", + "status": "completed", + }, + { + "call_id": "call_5f7b38f3b81e4b8380fd0ba74f3ca3ab", + "id": "fc_4fe5d6fc5b6c4d6fa5f24cc80aa27f78", + "output": "The weather in Paris is 20 Celsius", + "status": "completed", + "type": "function_call_output", + }, + ], + temperature=0.0, + ) + assert response is not None + assert response.status == "completed" + + output_types = [getattr(o, "type", None) for o in response.output] + assert "reasoning" in output_types, ( + f"Expected reasoning in output, got: {output_types}" + ) + assert "message" in output_types, f"Expected message in output, got: {output_types}" + + msg = next(o for o in response.output if o.type == "message") + assert type(msg.content[0].text) is str + + +def get_horoscope(sign): + return f"{sign}: Next Tuesday you will befriend a baby otter." + + +def call_function(name, args): + logger.info("Calling function %s with args %s", name, args) + if name == "get_horoscope": + return get_horoscope(**args) + raise ValueError(f"Unknown function: {name}") + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_function_call_first_turn(client: OpenAI, model_name: str): + tools = [ + { + "type": "function", + "name": "get_horoscope", + "description": "Get today's horoscope for an astrological sign.", + "parameters": { + "type": "object", + "properties": { + "sign": {"type": "string"}, + }, + "required": ["sign"], + "additionalProperties": False, + }, + "strict": True, + } + ] + + response = await retry_for_tool_call( + client, + model=model_name, + expected_tool_type="function_call", + input="What is the horoscope for Aquarius today?", + tools=tools, + temperature=0.0, + ) + assert response is not None + assert response.status == "completed" + + output_types = [getattr(o, "type", None) for o in response.output] + assert "reasoning" in output_types, ( + f"Expected reasoning in output, got: {output_types}" + ) + assert has_output_type(response, "function_call"), ( + f"Expected function_call in output, got: {output_types}" + ) + + function_call = next(o for o in response.output if o.type == "function_call") + assert function_call.name == "get_horoscope" + assert function_call.call_id is not None + + args = json.loads(function_call.arguments) + assert "sign" in args + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_mcp_tool_call(client: OpenAI, model_name: str): + """MCP tool calling with code_interpreter. + + The model may make one or more tool calls before producing a final + message. We validate server invariants (mcp_call items have correct + fields) with hard assertions. Output indices are never hardcoded + since the model can produce multiple tool-call rounds. + """ + # MCP + container init + code execution can be slow + client_with_timeout = client.with_options(timeout=client.timeout * 3) + + response = await retry_for_tool_call( + client_with_timeout, + model=model_name, + expected_tool_type="mcp_call", + input=( + "What is 123 * 456? Use python to calculate the result. " + "Print the result with print()." + ), + tools=[{"type": "code_interpreter", "container": {"type": "auto"}}], + instructions=_PYTHON_TOOL_INSTRUCTION, + temperature=0.0, + extra_body={"enable_response_messages": True}, + ) + + assert response is not None + + output_types = [getattr(o, "type", None) for o in response.output] + log_response_diagnostics(response, label="test_mcp_tool_call") + + assert response.status == "completed", ( + f"Response status={response.status} " + f"(details={getattr(response, 'incomplete_details', None)}). " + f"Output types: {output_types}." + ) + + assert "reasoning" in output_types, ( + f"Expected reasoning in output, got: {output_types}" + ) + assert "mcp_call" in output_types, ( + f"Expected mcp_call in output, got: {output_types}" + ) + + # Every mcp_call item must have well-typed fields + for item in response.output: + if getattr(item, "type", None) == "mcp_call": + assert type(item.arguments) is str, ( + f"mcp_call.arguments should be str, got {type(item.arguments)}" + ) + assert type(item.output) is str, ( + f"mcp_call.output should be str, got {type(item.output)}" + ) + + # The model may make 1+ tool-call rounds but must still produce + # a final message for a trivial calculation like 123 * 456. + message_outputs = [ + o for o in response.output if getattr(o, "type", None) == "message" + ] + assert message_outputs, ( + f"Model did not produce a final message. Output types: {output_types}" + ) + + final_message = message_outputs[-1] + assert any(s in final_message.content[0].text for s in ("56088", "56,088")), ( + f"Expected 56088 in final message, got: {final_message.content[0].text!r}" + ) + + # Validate raw input_messages / output_messages + assert len(response.input_messages) >= 1, "Expected at least 1 input message" + assert len(response.output_messages) >= 1, "Expected at least 1 output message" + assert any( + any(s in str(msg) for s in ("56088", "56,088")) + for msg in response.output_messages + ), ( + f"Expected 56088 in at least one output_message, " + f"got {len(response.output_messages)} messages" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_max_tokens(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input="What is the first paragraph of Moby Dick?", + reasoning={"effort": "low"}, + max_output_tokens=30, + temperature=0.0, + ) + assert response is not None + assert response.status == "incomplete" + assert response.incomplete_details.reason == "max_output_tokens" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_parsable_context_unit.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_parsable_context_unit.py new file mode 100644 index 0000000000000000000000000000000000000000..f7379ab24d858ee71bb313f165ca5886f017b118 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_parsable_context_unit.py @@ -0,0 +1,363 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for ParsableContext's parsing behavior. + +These tests verify that ParsableContext correctly delegates to the unified +Parser (via parse) and properly builds response output items. +""" + +from collections.abc import Sequence +from unittest.mock import MagicMock + +import pytest + +from vllm.entrypoints.openai.engine.protocol import ( + DeltaMessage, + ExtractedToolCallInformation, + FunctionCall, + ToolCall, +) +from vllm.entrypoints.openai.responses.context import ParsableContext +from vllm.entrypoints.openai.responses.protocol import ResponsesRequest +from vllm.outputs import CompletionOutput, RequestOutput +from vllm.parser.abstract_parser import DelegatingParser + +pytestmark = pytest.mark.skip_global_cleanup + + +# --------------------------------------------------------------------------- +# Test parser stubs +# --------------------------------------------------------------------------- + + +class _NoOpParser(DelegatingParser): + """Parser that extracts no reasoning and no tool calls.""" + + def is_reasoning_end(self, input_ids: list[int]) -> bool: + return False + + def extract_content_ids(self, input_ids: list[int]) -> list[int]: + return input_ids + + def extract_reasoning(self, model_output, request): + return None, model_output + + def extract_reasoning_streaming(self, *args, **kwargs): + return None + + def extract_tool_calls(self, model_output, request): + return ExtractedToolCallInformation( + tools_called=False, tool_calls=[], content=model_output + ) + + def extract_tool_calls_streaming(self, *args, **kwargs): + return None + + def parse_delta(self, *args, **kwargs) -> DeltaMessage | None: + return None + + +class _ReasoningOnlyParser(DelegatingParser): + """Parser that extracts reasoning but no tool calls.""" + + def is_reasoning_end(self, input_ids: list[int]) -> bool: + return False + + def extract_content_ids(self, input_ids: list[int]) -> list[int]: + return input_ids + + def extract_reasoning(self, model_output, request): + if "" in model_output and "" in model_output: + start = model_output.index("") + len("") + end = model_output.index("") + reasoning = model_output[start:end] + content = model_output[end + len("") :] + return reasoning, content.strip() or None + return None, model_output + + def extract_reasoning_streaming(self, *args, **kwargs): + return None + + def extract_tool_calls(self, model_output, request): + return ExtractedToolCallInformation( + tools_called=False, tool_calls=[], content=model_output + ) + + def extract_tool_calls_streaming(self, *args, **kwargs): + return None + + def parse_delta(self, *args, **kwargs) -> DeltaMessage | None: + return None + + +class _StubToolParser: + """Minimal tool parser stub that always returns a hardcoded tool call.""" + + supports_required_and_named = False + + def __init__(self, tokenizer=None, tools=None): + pass + + def extract_tool_calls(self, model_output, request): + return ExtractedToolCallInformation( + tools_called=True, + tool_calls=[ + ToolCall( + id="call_123", + type="function", + function=FunctionCall( + name="get_weather", + arguments='{"location": "Paris"}', + ), + ) + ], + content=None, + ) + + def extract_tool_calls_streaming(self, *args, **kwargs): + return None + + def adjust_request(self, request): + return request + + +class _ToolCallingParser(DelegatingParser): + """Parser that extracts a hardcoded tool call from any input.""" + + def __init__(self, tokenizer, *args, **kwargs): + super().__init__(tokenizer) + self._tool_parser = _StubToolParser() + + def is_reasoning_end(self, input_ids: list[int]) -> bool: + return False + + def extract_content_ids(self, input_ids: list[int]) -> list[int]: + return input_ids + + def extract_reasoning(self, model_output, request): + return None, model_output + + def extract_reasoning_streaming(self, *args, **kwargs): + return None + + def extract_tool_calls_streaming(self, *args, **kwargs): + return None + + def parse_delta(self, *args, **kwargs) -> DeltaMessage | None: + return None + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _make_request(**overrides) -> ResponsesRequest: + defaults = {"model": "test-model", "input": "test"} + defaults.update(overrides) + return ResponsesRequest.model_validate(defaults) + + +def _make_request_output( + text: str = "Hello, world!", + token_ids: Sequence[int] = (1, 2, 3), + finish_reason: str = "stop", +) -> RequestOutput: + return RequestOutput( + request_id="test", + prompt=None, + prompt_token_ids=[], + prompt_logprobs=None, + outputs=[ + CompletionOutput( + index=0, + text=text, + token_ids=list(token_ids), + cumulative_logprob=None, + logprobs=None, + finish_reason=finish_reason, + ) + ], + finished=True, + ) + + +def _make_context(parser_cls, **overrides): + # ParsableContext no longer lazily builds a parser from ``parser_cls``; + # the caller (here, the serving layer in production) must supply one. + request = overrides.get("request", _make_request()) + response_parser = overrides.pop("response_parser", None) + if response_parser is None and parser_cls is not None: + response_parser = parser_cls(MagicMock(), request.tools) + + defaults = dict( + tokenizer=MagicMock(), + parser_cls=parser_cls, + response_parser=response_parser, + response_messages=[], + request=request, + available_tools=None, + chat_template=None, + chat_template_content_format="auto", + ) + defaults.update(overrides) + return ParsableContext(**defaults) + + +# --------------------------------------------------------------------------- +# Tests: basic text passthrough +# --------------------------------------------------------------------------- + + +def test_process_text_with_parser(): + """Parser with no reasoning/tools returns a single message item.""" + ctx = _make_context(_NoOpParser) + ctx.append_output(_make_request_output(text="Hello!")) + + assert len(ctx.response_messages) == 1 + msg = ctx.response_messages[0] + assert msg.type == "message" + assert msg.content[0].text == "Hello!" + + +def test_process_text_without_parser(): + """parser_cls=None falls back to plain text wrapping.""" + ctx = _make_context(None) + ctx.append_output(_make_request_output(text="Hello!")) + + assert len(ctx.response_messages) == 1 + msg = ctx.response_messages[0] + assert msg.type == "message" + assert msg.content[0].text == "Hello!" + + +# --------------------------------------------------------------------------- +# Tests: empty / whitespace output +# --------------------------------------------------------------------------- + + +def test_process_empty_text_without_parser(): + """Empty text with no parser produces no output items.""" + ctx = _make_context(None) + ctx.append_output(_make_request_output(text="")) + + assert len(ctx.response_messages) == 0 + + +def test_process_empty_text_with_parser(): + """Empty text with parser produces no output items.""" + ctx = _make_context(_NoOpParser) + ctx.append_output(_make_request_output(text="")) + + assert len(ctx.response_messages) == 0 + + +# --------------------------------------------------------------------------- +# Tests: reasoning extraction +# --------------------------------------------------------------------------- + + +def test_process_extracts_reasoning(): + """Parser that finds reasoning produces both reasoning and message items.""" + ctx = _make_context(_ReasoningOnlyParser) + ctx.append_output( + _make_request_output(text="Let me checkThe answer is 42") + ) + + types = [m.type for m in ctx.response_messages] + assert "reasoning" in types + assert "message" in types + + reasoning_item = next(m for m in ctx.response_messages if m.type == "reasoning") + assert reasoning_item.content[0].text == "Let me check" + + message_item = next(m for m in ctx.response_messages if m.type == "message") + assert message_item.content[0].text == "The answer is 42" + + +def test_process_reasoning_only_no_content(): + """When reasoning consumes all text, only a reasoning item is produced.""" + ctx = _make_context(_ReasoningOnlyParser) + ctx.append_output(_make_request_output(text="Just thinking")) + + types = [m.type for m in ctx.response_messages] + assert "reasoning" in types + assert "message" not in types + + +# --------------------------------------------------------------------------- +# Tests: tool call extraction +# --------------------------------------------------------------------------- + + +def test_process_extracts_tool_calls(): + """Parser that finds tool calls produces function_call items.""" + request = _make_request( + tool_choice="auto", + tools=[ + { + "type": "function", + "name": "get_weather", + "parameters": {"type": "object", "properties": {}}, + } + ], + ) + ctx = _make_context(_ToolCallingParser, request=request, enable_auto_tools=True) + ctx.append_output(_make_request_output(text="calling tool")) + + types = [m.type for m in ctx.response_messages] + assert "function_call" in types + + tool_item = next(m for m in ctx.response_messages if m.type == "function_call") + assert tool_item.name == "get_weather" + assert tool_item.arguments == '{"location": "Paris"}' + assert tool_item.status == "completed" + + +# --------------------------------------------------------------------------- +# Tests: finish_reason tracking +# --------------------------------------------------------------------------- + + +def test_finish_reason_tracked(): + """finish_reason from CompletionOutput is stored on the context.""" + ctx = _make_context(_NoOpParser) + assert ctx.finish_reason is None + + ctx.append_output(_make_request_output(finish_reason="stop")) + assert ctx.finish_reason == "stop" + + ctx.append_output(_make_request_output(finish_reason="length")) + assert ctx.finish_reason == "length" + + +# --------------------------------------------------------------------------- +# Tests: multi-turn accumulation +# --------------------------------------------------------------------------- + + +def test_multi_turn_accumulation(): + """Multiple append_output() calls accumulate response_messages.""" + ctx = _make_context(_NoOpParser) + + ctx.append_output(_make_request_output(text="First turn")) + ctx.append_output(_make_request_output(text="Second turn")) + + assert len(ctx.response_messages) == 2 + texts = [m.content[0].text for m in ctx.response_messages] + assert texts == ["First turn", "Second turn"] + + +def test_num_init_messages_offset(): + """Initial messages are preserved and offset works correctly.""" + init_messages = [MagicMock(type="message")] + ctx = _make_context(_NoOpParser, response_messages=init_messages) + + assert ctx.num_init_messages == 1 + + ctx.append_output(_make_request_output(text="New output")) + + assert len(ctx.response_messages) == 2 + items = ctx.make_response_output_items() + assert len(items) == 1 + assert items[0].type == "message" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_protocol.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_protocol.py new file mode 100644 index 0000000000000000000000000000000000000000..38a4cacabd40766a5efbbfa28928271ce50d516b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_protocol.py @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from openai_harmony import ( + Message, +) + +from vllm.entrypoints.openai.responses.protocol import ( + serialize_message, + serialize_messages, +) + + +def test_serialize_message() -> None: + dict_value = {"a": 1, "b": "2"} + assert serialize_message(dict_value) == dict_value + + msg_value = { + "role": "assistant", + "name": None, + "content": [{"type": "text", "text": "Test 1"}], + "channel": "analysis", + } + msg = Message.from_dict(msg_value) + assert serialize_message(msg) == msg_value + + +def test_serialize_messages() -> None: + assert serialize_messages(None) is None + assert serialize_messages([]) is None + + dict_value = {"a": 3, "b": "4"} + msg_value = { + "role": "assistant", + "name": None, + "content": [{"type": "text", "text": "Test 2"}], + "channel": "analysis", + } + msg = Message.from_dict(msg_value) + assert serialize_messages([msg, dict_value]) == [msg_value, dict_value] diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_response_input_to_harmony.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_response_input_to_harmony.py new file mode 100644 index 0000000000000000000000000000000000000000..2ca6490a5b30729b391605dce57a8b919c3a03db --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_response_input_to_harmony.py @@ -0,0 +1,281 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for response_input_to_harmony. + +Covers every type branch in the function and verifies that each produced +Harmony Message has the correct role, channel, recipient, content_type, +author name, and text content. +""" + +import pytest +from openai.types.responses import ResponseFunctionToolCall, ResponseReasoningItem +from openai.types.responses.response_reasoning_item import ( + Content as ReasoningTextContent, +) +from openai_harmony import DeveloperContent, Role + +from vllm.entrypoints.openai.responses.harmony import response_input_to_harmony + +# --------------------------------------------------------------------------- +# Shared fixtures +# --------------------------------------------------------------------------- + +_PREV_CALL = ResponseFunctionToolCall( + id="fc_test", + call_id="call_test", + name="get_weather", + arguments='{"location": "Paris"}', + type="function_call", +) + +_REASONING_ITEM = ResponseReasoningItem( + id="rs_test", + type="reasoning", + content=[ReasoningTextContent(type="reasoning_text", text="Thinking hard.")], + summary=[], + status=None, +) + + +class TestResponseInputToHarmonyMessage: + """Unit tests for every message type handled by response_input_to_harmony.""" + + # ----------------------------------------------------------------------- + # type="message" (or no type key) + # ----------------------------------------------------------------------- + + def test_user_message_string_content(self): + msg = response_input_to_harmony( + {"type": "message", "role": "user", "content": "Hello"}, + prev_responses=[], + ) + + assert msg.author.role == Role.USER + assert msg.content[0].text == "Hello" + assert msg.channel is None + + def test_no_type_key_defaults_to_message_branch(self): + """Omitting 'type' should fall through to the message branch.""" + msg = response_input_to_harmony( + {"role": "user", "content": "Hello"}, + prev_responses=[], + ) + + assert msg.author.role == Role.USER + assert msg.content[0].text == "Hello" + + def test_system_message(self): + """System messages carry developer instructions and must be rendered + as developer messages with DeveloperContent.""" + msg = response_input_to_harmony( + {"type": "message", "role": "system", "content": "Be helpful."}, + prev_responses=[], + ) + + assert msg.author.role == Role.DEVELOPER + assert isinstance(msg.content[0], DeveloperContent) + assert msg.content[0].instructions == "Be helpful." + + def test_assistant_message_gets_final_channel(self): + msg = response_input_to_harmony( + {"type": "message", "role": "assistant", "content": "The answer is 42."}, + prev_responses=[], + ) + + assert msg.author.role == Role.ASSISTANT + assert msg.channel == "final" + assert msg.content[0].text == "The answer is 42." + + def test_developer_message_gets_instructions_prefix(self): + """Developer messages must use DeveloperContent which adds the + '# Instructions' header the model was trained on.""" + msg = response_input_to_harmony( + {"type": "message", "role": "developer", "content": "Be concise."}, + prev_responses=[], + ) + + assert msg.author.role == Role.DEVELOPER + assert isinstance(msg.content[0], DeveloperContent) + assert msg.content[0].instructions == "Be concise." + + def test_message_with_array_content(self): + msg = response_input_to_harmony( + { + "type": "message", + "role": "user", + "content": [ + {"type": "text", "text": "Part one. "}, + {"type": "text", "text": "Part two."}, + ], + }, + prev_responses=[], + ) + + assert msg.author.role == Role.USER + assert len(msg.content) == 2 + assert msg.content[0].text == "Part one. " + assert msg.content[1].text == "Part two." + + def test_developer_message_array_content_concatenated(self): + """Array content in developer messages is flattened and rendered + via DeveloperContent with the '# Instructions' header.""" + msg = response_input_to_harmony( + { + "type": "message", + "role": "developer", + "content": [ + {"type": "text", "text": "Rule 1."}, + {"type": "text", "text": "Rule 2."}, + ], + }, + prev_responses=[], + ) + + assert msg.author.role == Role.DEVELOPER + assert isinstance(msg.content[0], DeveloperContent) + assert msg.content[0].instructions == "Rule 1.Rule 2." + + # ----------------------------------------------------------------------- + # type="reasoning" + # ----------------------------------------------------------------------- + + def test_reasoning_gets_analysis_channel(self): + msg = response_input_to_harmony( + { + "type": "reasoning", + "content": [ + {"type": "reasoning_text", "text": "I should call get_weather."} + ], + }, + prev_responses=[], + ) + + assert msg.author.role == Role.ASSISTANT + assert msg.channel == "analysis" + assert msg.content[0].text == "I should call get_weather." + + def test_reasoning_pydantic_model_input(self): + """A Pydantic ResponseReasoningItem should be model_dump()'d before parsing.""" + msg = response_input_to_harmony(_REASONING_ITEM, prev_responses=[]) + + assert msg.author.role == Role.ASSISTANT + assert msg.channel == "analysis" + assert msg.content[0].text == "Thinking hard." + + # ----------------------------------------------------------------------- + # type="function_call" + # ----------------------------------------------------------------------- + + def test_function_call_channel_recipient_and_content_type(self): + msg = response_input_to_harmony( + { + "type": "function_call", + "name": "get_weather", + "arguments": '{"location": "Paris"}', + }, + prev_responses=[], + ) + + assert msg.author.role == Role.ASSISTANT + assert msg.channel == "commentary" + assert msg.recipient == "functions.get_weather" + assert msg.content_type == "json" + assert msg.content[0].text == '{"location": "Paris"}' + + def test_function_call_empty_arguments(self): + msg = response_input_to_harmony( + {"type": "function_call", "name": "ping", "arguments": ""}, + prev_responses=[], + ) + + assert msg.recipient == "functions.ping" + assert msg.content[0].text == "" + + # ----------------------------------------------------------------------- + # type="function_call_output" + # ----------------------------------------------------------------------- + + def test_function_call_output_channel_recipient_and_author_name(self): + msg = response_input_to_harmony( + {"type": "function_call_output", "call_id": "call_test", "output": "18°C"}, + prev_responses=[_PREV_CALL], + ) + + assert msg.author.role == Role.TOOL + assert msg.author.name == "functions.get_weather" + assert msg.channel == "commentary" + assert msg.recipient == "assistant" + assert msg.content[0].text == "18°C" + + def test_function_call_output_uses_most_recent_matching_call(self): + """When multiple prev_responses share a call_id, the last one wins + because the search is reversed.""" + earlier = ResponseFunctionToolCall( + id="fc_old", + call_id="call_test", + name="old_func", + arguments="{}", + type="function_call", + ) + later = ResponseFunctionToolCall( + id="fc_new", + call_id="call_test", + name="get_weather", + arguments="{}", + type="function_call", + ) + + msg = response_input_to_harmony( + { + "type": "function_call_output", + "call_id": "call_test", + "output": "result", + }, + prev_responses=[earlier, later], + ) + + assert msg.author.name == "functions.get_weather" + + def test_function_call_output_skips_non_function_call_items_in_prev_responses( + self, + ): + """ResponseReasoningItem entries in prev_responses should be ignored.""" + msg = response_input_to_harmony( + { + "type": "function_call_output", + "call_id": "call_test", + "output": "18°C", + }, + prev_responses=[_REASONING_ITEM, _PREV_CALL], + ) + + assert msg.author.name == "functions.get_weather" + + def test_function_call_output_raises_if_no_matching_call(self): + with pytest.raises(ValueError, match="No call message found for"): + response_input_to_harmony( + { + "type": "function_call_output", + "call_id": "no_such_id", + "output": "x", + }, + prev_responses=[_PREV_CALL], + ) + + def test_function_call_output_raises_on_empty_prev_responses(self): + with pytest.raises(ValueError, match="No call message found for"): + response_input_to_harmony( + {"type": "function_call_output", "call_id": "call_test", "output": "x"}, + prev_responses=[], + ) + + # ----------------------------------------------------------------------- + # Error cases + # ----------------------------------------------------------------------- + + def test_unknown_type_raises_value_error(self): + with pytest.raises(ValueError, match="Unknown input type"): + response_input_to_harmony( + {"type": "image_url", "url": "https://example.com/img.png"}, + prev_responses=[], + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_responses_utils.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_responses_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..6c14bedbfea30588880d529d8f5002322172723e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_responses_utils.py @@ -0,0 +1,923 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from unittest.mock import patch + +import pytest +from openai.types.responses.response_function_tool_call import ResponseFunctionToolCall +from openai.types.responses.response_function_tool_call_output_item import ( + ResponseFunctionToolCallOutputItem, +) +from openai.types.responses.response_output_message import ResponseOutputMessage +from openai.types.responses.response_output_text import ResponseOutputText +from openai.types.responses.response_reasoning_item import ( + Content, + ResponseReasoningItem, + Summary, +) + +from vllm.entrypoints.openai.responses.utils import ( + _construct_message_from_response_item, + construct_chat_messages_with_tool_call, + construct_input_messages, + convert_tool_responses_to_completions_format, + should_continue_final_message, +) + + +def _single_chat_message(item): + message = _construct_message_from_response_item(item) + assert message is not None + return message + + +def make_output_message( + text: str, + *, + id: str = "msg_1", + status: str = "completed", +) -> ResponseOutputMessage: + return ResponseOutputMessage( + id=id, + content=[ + ResponseOutputText( + annotations=[], + text=text, + type="output_text", + logprobs=None, + ) + ], + role="assistant", + status=status, + type="message", + ) + + +def make_reasoning_item( + *, + content_text: str | None = None, + summary_text: str | None = None, + content: list[Content] | None = None, + summary: list[Summary] | None = None, + encrypted_content: str | None = None, + id: str = "reasoning_1", + status: str | None = None, +) -> ResponseReasoningItem: + if content is None and content_text is not None: + content = [Content(text=content_text, type="reasoning_text")] + if summary is None and summary_text is not None: + summary = [Summary(text=summary_text, type="summary_text")] + + return ResponseReasoningItem( + id=id, + summary=[] if summary is None else summary, + type="reasoning", + content=content, + encrypted_content=encrypted_content, + status=status, + ) + + +def make_function_call( + *, + call_id: str, + name: str = "test_function", + arguments: str = "{}", + id: str = "tool_id", + status: str | None = None, +) -> ResponseFunctionToolCall: + kwargs = { + "type": "function_call", + "id": id, + "call_id": call_id, + "name": name, + "arguments": arguments, + } + if status is not None: + kwargs["status"] = status + + return ResponseFunctionToolCall(**kwargs) + + +def make_function_call_output( + *, + call_id: str, + output: str = "42", + id: str = "output_1", + status: str = "completed", +) -> ResponseFunctionToolCallOutputItem: + return ResponseFunctionToolCallOutputItem( + id=id, + type="function_call_output", + call_id=call_id, + output=output, + status=status, + ) + + +class TestResponsesUtils: + """Tests for convert_tool_responses_to_completions_format function.""" + + def test_convert_tool_responses_to_completions_format(self): + """Test basic conversion of a flat tool schema to nested format.""" + input_tool = { + "type": "function", + "name": "get_weather", + "description": "Get the current weather in a given location", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"}, + "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, + }, + "required": ["location", "unit"], + }, + } + + result = convert_tool_responses_to_completions_format(input_tool) + + assert result == {"type": "function", "function": input_tool} + + def test_construct_chat_messages_with_tool_call(self): + """Test construction of chat messages with tool calls.""" + reasoning_item = ResponseReasoningItem( + id="lol", + summary=[], + type="reasoning", + content=[ + Content( + text="Leroy Jenkins", + type="reasoning_text", + ) + ], + encrypted_content=None, + status=None, + ) + mcp_tool_item = ResponseFunctionToolCall( + id="mcp_123", + call_id="call_123", + type="function_call", + status="completed", + name="python", + arguments='{"code": "123+456"}', + ) + input_items = [reasoning_item, mcp_tool_item] + messages = construct_chat_messages_with_tool_call(input_items) + + assert len(messages) == 1 + message = messages[0] + assert message["role"] == "assistant" + assert message["reasoning"] == "Leroy Jenkins" + assert message["tool_calls"][0]["id"] == "call_123" + assert message["tool_calls"][0]["function"]["name"] == "python" + assert ( + message["tool_calls"][0]["function"]["arguments"] == '{"code": "123+456"}' + ) + + def test_construct_chat_messages_preserves_single_item_conversions(self): + item = ResponseReasoningItem( + id="lol", + summary=[], + type="reasoning", + content=[ + Content( + text="Leroy Jenkins", + type="reasoning_text", + ) + ], + encrypted_content=None, + status=None, + ) + formatted_item = _single_chat_message(item) + assert formatted_item["role"] == "assistant" + assert formatted_item["reasoning"] == "Leroy Jenkins" + + item = ResponseReasoningItem( + id="lol", + summary=[ + Summary( + text='Hmm, the user has just started with a simple "Hello,"', + type="summary_text", + ) + ], + type="reasoning", + content=None, + encrypted_content=None, + status=None, + ) + + formatted_item = _single_chat_message(item) + assert formatted_item["role"] == "assistant" + assert ( + formatted_item["reasoning"] + == 'Hmm, the user has just started with a simple "Hello,"' + ) + + tool_call_output = ResponseFunctionToolCallOutputItem( + id="temp_id", + type="function_call_output", + call_id="temp", + output="1234", + status="completed", + ) + formatted_item = _single_chat_message(tool_call_output) + assert formatted_item["role"] == "tool" + assert formatted_item["content"] == "1234" + assert formatted_item["tool_call_id"] == "temp" + + formatted_item = _single_chat_message( + { + "type": "function_call_output", + "call_id": "temp_dict", + "output": "5678", + } + ) + assert formatted_item["role"] == "tool" + assert formatted_item["content"] == "5678" + assert formatted_item["tool_call_id"] == "temp_dict" + + item = ResponseReasoningItem( + id="lol", + summary=[], + type="reasoning", + content=None, + encrypted_content="TOP_SECRET_MESSAGE", + status=None, + ) + with pytest.raises(ValueError): + construct_chat_messages_with_tool_call([item]) + + output_item = ResponseOutputMessage( + id="msg_bf585bbbe3d500e0", + content=[ + ResponseOutputText( + annotations=[], + text="dongyi", + type="output_text", + logprobs=None, + ) + ], + role="assistant", + status="completed", + type="message", + ) + + formatted_item = _single_chat_message(output_item) + assert formatted_item["role"] == "assistant" + assert formatted_item["content"] == "dongyi" + + +class TestReasoningItemContentPriority: + """Tests that content is prioritized over summary for reasoning items.""" + + def test_content_preferred_over_summary(self): + """When both content and summary are present, content should win.""" + item = ResponseReasoningItem( + id="reasoning_1", + summary=[ + Summary( + text="This is a summary", + type="summary_text", + ) + ], + type="reasoning", + content=[ + Content( + text="This is the actual content", + type="reasoning_text", + ) + ], + encrypted_content=None, + status=None, + ) + formatted = _single_chat_message(item) + assert formatted["reasoning"] == "This is the actual content" + + def test_content_only(self): + """When only content is present (no summary), content is used.""" + item = ResponseReasoningItem( + id="reasoning_2", + summary=[], + type="reasoning", + content=[ + Content( + text="Content without summary", + type="reasoning_text", + ) + ], + encrypted_content=None, + status=None, + ) + formatted = _single_chat_message(item) + assert formatted["reasoning"] == "Content without summary" + + @patch("vllm.entrypoints.openai.responses.utils.logger") + def test_summary_fallback_when_no_content(self, mock_logger): + """When content is absent, summary is used as fallback with warning.""" + item = ResponseReasoningItem( + id="reasoning_3", + summary=[ + Summary( + text="Fallback summary text", + type="summary_text", + ) + ], + type="reasoning", + content=None, + encrypted_content=None, + status=None, + ) + formatted = _single_chat_message(item) + assert formatted["reasoning"] == "Fallback summary text" + mock_logger.warning.assert_called_once() + assert ( + "summary text as reasoning content" in mock_logger.warning.call_args[0][0] + ) + + @patch("vllm.entrypoints.openai.responses.utils.logger") + def test_summary_fallback_when_content_empty(self, mock_logger): + """When content is an empty list, summary is used as fallback.""" + item = ResponseReasoningItem( + id="reasoning_4", + summary=[ + Summary( + text="Summary when content empty", + type="summary_text", + ) + ], + type="reasoning", + content=[], + encrypted_content=None, + status=None, + ) + formatted = _single_chat_message(item) + assert formatted["reasoning"] == "Summary when content empty" + mock_logger.warning.assert_called_once() + assert ( + "summary text as reasoning content" in mock_logger.warning.call_args[0][0] + ) + + def test_neither_content_nor_summary(self): + """When neither content nor summary is present, reasoning is empty.""" + item = ResponseReasoningItem( + id="reasoning_5", + summary=[], + type="reasoning", + content=None, + encrypted_content=None, + status=None, + ) + formatted = _single_chat_message(item) + assert formatted["reasoning"] == "" + + def test_encrypted_content_raises(self): + """Encrypted content should still raise ValueError.""" + item = ResponseReasoningItem( + id="reasoning_6", + summary=[ + Summary( + text="Some summary", + type="summary_text", + ) + ], + type="reasoning", + content=[ + Content( + text="Some content", + type="reasoning_text", + ) + ], + encrypted_content="ENCRYPTED", + status=None, + ) + with pytest.raises(ValueError): + construct_chat_messages_with_tool_call([item]) + + @patch("vllm.entrypoints.openai.responses.utils.logger") + def test_summary_with_multiple_entries_uses_first(self, mock_logger): + """When multiple summary entries exist, the first one is used.""" + item = ResponseReasoningItem( + id="reasoning_7", + summary=[ + Summary( + text="First summary", + type="summary_text", + ), + Summary( + text="Second summary", + type="summary_text", + ), + ], + type="reasoning", + content=None, + encrypted_content=None, + status=None, + ) + formatted = _single_chat_message(item) + assert formatted["reasoning"] == "First summary" + mock_logger.warning.assert_called_once() + assert ( + "summary text as reasoning content" in mock_logger.warning.call_args[0][0] + ) + + @patch("vllm.entrypoints.openai.responses.utils.logger") + def test_no_warning_when_content_used(self, mock_logger): + """No warning should be emitted when content is available.""" + item = ResponseReasoningItem( + id="reasoning_8", + summary=[ + Summary( + text="Summary text", + type="summary_text", + ) + ], + type="reasoning", + content=[ + Content( + text="Content text", + type="reasoning_text", + ) + ], + encrypted_content=None, + status=None, + ) + construct_chat_messages_with_tool_call([item]) + mock_logger.warning.assert_not_called() + + +class TestShouldContinueFinalMessage: + """Tests for should_continue_final_message function. + + This function enables Anthropic-style partial message completion, where + users can provide an incomplete assistant message and have the model + continue from where it left off. + """ + + def test_string_input_returns_false(self): + """String input is always a user message, so should not continue.""" + assert should_continue_final_message("Hello, world!") is False + + def test_empty_list_returns_false(self): + """Empty list should not continue.""" + assert should_continue_final_message([]) is False + + def test_completed_message_returns_false(self): + """Completed message should not be continued.""" + output_item = ResponseOutputMessage( + id="msg_123", + content=[ + ResponseOutputText( + annotations=[], + text="The answer is 42.", + type="output_text", + logprobs=None, + ) + ], + role="assistant", + status="completed", + type="message", + ) + assert should_continue_final_message([output_item]) is False + + def test_in_progress_message_returns_true(self): + """In-progress message should be continued. + + This is the key use case for partial message completion. + Example: The user provides "The best answer is (" and wants + the model to continue from there. + """ + output_item = ResponseOutputMessage( + id="msg_123", + content=[ + ResponseOutputText( + annotations=[], + text="The best answer is (", + type="output_text", + logprobs=None, + ) + ], + role="assistant", + status="in_progress", + type="message", + ) + assert should_continue_final_message([output_item]) is True + + def test_incomplete_message_returns_true(self): + """Incomplete message should be continued.""" + output_item = ResponseOutputMessage( + id="msg_123", + content=[ + ResponseOutputText( + annotations=[], + text="The answer", + type="output_text", + logprobs=None, + ) + ], + role="assistant", + status="incomplete", + type="message", + ) + assert should_continue_final_message([output_item]) is True + + def test_in_progress_reasoning_returns_true(self): + """In-progress reasoning should be continued.""" + reasoning_item = ResponseReasoningItem( + id="reasoning_123", + summary=[], + type="reasoning", + content=[ + Content( + text="Let me think about this...", + type="reasoning_text", + ) + ], + encrypted_content=None, + status="in_progress", + ) + assert should_continue_final_message([reasoning_item]) is True + + def test_incomplete_reasoning_returns_true(self): + """Incomplete reasoning should be continued.""" + reasoning_item = ResponseReasoningItem( + id="reasoning_123", + summary=[], + type="reasoning", + content=[ + Content( + text="Let me think", + type="reasoning_text", + ) + ], + encrypted_content=None, + status="incomplete", + ) + assert should_continue_final_message([reasoning_item]) is True + + reasoning_item = { + "id": "reasoning_123", + "summary": [], + "type": "reasoning", + "content": [], + "status": "incomplete", + } + assert should_continue_final_message([reasoning_item]) is True + + def test_completed_reasoning_returns_false(self): + """Completed reasoning should not be continued.""" + reasoning_item = ResponseReasoningItem( + id="reasoning_123", + summary=[], + type="reasoning", + content=[ + Content( + text="I have thought about this.", + type="reasoning_text", + ) + ], + encrypted_content=None, + status="completed", + ) + assert should_continue_final_message([reasoning_item]) is False + + def test_reasoning_with_none_status_returns_false(self): + """Reasoning with None status should not be continued.""" + reasoning_item = ResponseReasoningItem( + id="reasoning_123", + summary=[], + type="reasoning", + content=[ + Content( + text="Some reasoning", + type="reasoning_text", + ) + ], + encrypted_content=None, + status=None, + ) + assert should_continue_final_message([reasoning_item]) is False + + def test_only_last_item_matters(self): + """Only the last item in the list determines continuation.""" + completed_item = ResponseOutputMessage( + id="msg_1", + content=[ + ResponseOutputText( + annotations=[], + text="Complete message.", + type="output_text", + logprobs=None, + ) + ], + role="assistant", + status="completed", + type="message", + ) + in_progress_item = ResponseOutputMessage( + id="msg_2", + content=[ + ResponseOutputText( + annotations=[], + text="Partial message...", + type="output_text", + logprobs=None, + ) + ], + role="assistant", + status="in_progress", + type="message", + ) + + # In-progress as last item -> should continue + assert should_continue_final_message([completed_item, in_progress_item]) is True + + # Completed as last item -> should not continue + assert ( + should_continue_final_message([in_progress_item, completed_item]) is False + ) + + def test_tool_call_returns_false(self): + """Tool calls should not trigger continuation.""" + tool_call = ResponseFunctionToolCall( + id="fc_123", + call_id="call_123", + type="function_call", + status="in_progress", + name="get_weather", + arguments='{"location": "NYC"}', + ) + assert should_continue_final_message([tool_call]) is False + + tool_call = { + "id": "msg_123", + "call_id": "call_123", + "type": "function_call", + "status": "in_progress", + "name": "get_weather", + "arguments": '{"location": "NYC"}', + } + assert should_continue_final_message([tool_call]) is False + + # Tests for dict inputs (e.g., from curl requests) + def test_dict_in_progress_message_returns_true(self): + """Dict with in_progress status should be continued (curl input).""" + dict_item = { + "id": "msg_123", + "type": "message", + "role": "assistant", + "status": "in_progress", + "content": [{"type": "output_text", "text": "The answer is ("}], + } + assert should_continue_final_message([dict_item]) is True + + def test_dict_incomplete_message_returns_true(self): + """Dict with incomplete status should be continued (curl input).""" + dict_item = { + "id": "msg_123", + "type": "message", + "role": "assistant", + "status": "incomplete", + "content": [{"type": "output_text", "text": "Partial answer"}], + } + assert should_continue_final_message([dict_item]) is True + + def test_dict_completed_message_returns_false(self): + """Dict with completed status should not be continued (curl input).""" + dict_item = { + "id": "msg_123", + "type": "message", + "role": "assistant", + "status": "completed", + "content": [{"type": "output_text", "text": "Complete answer."}], + } + assert should_continue_final_message([dict_item]) is False + + def test_dict_reasoning_in_progress_returns_true(self): + """Dict reasoning item with in_progress status should be continued.""" + dict_item = { + "id": "reasoning_123", + "type": "reasoning", + "status": "in_progress", + "content": [{"type": "reasoning_text", "text": "Let me think..."}], + } + assert should_continue_final_message([dict_item]) is True + + def test_dict_without_status_returns_false(self): + """Dict without status field should not be continued.""" + dict_item = { + "id": "msg_123", + "type": "message", + "role": "assistant", + "content": [{"type": "output_text", "text": "Some text"}], + } + assert should_continue_final_message([dict_item]) is False + + def test_dict_with_none_status_returns_false(self): + """Dict with None status should not be continued.""" + dict_item = { + "id": "msg_123", + "type": "message", + "role": "assistant", + "status": None, + "content": [{"type": "output_text", "text": "Some text"}], + } + assert should_continue_final_message([dict_item]) is False + + +class TestConstructChatMessagesCombinePolicy: + """Tests for contiguous assistant-side merging.""" + + @pytest.mark.parametrize( + ("items", "expected_content", "expected_reasoning", "expected_tool_call_ids"), + [ + pytest.param( + [ + make_reasoning_item(content_text="Let me think"), + make_output_message("Hello"), + ], + "Hello", + "Let me think", + None, + id="reasoning-output-messages", + ), + pytest.param( + [ + make_function_call(call_id="call_123"), + make_function_call(call_id="call_456"), + ], + None, + None, + ["call_123", "call_456"], + id="consecutive-tool-calls", + ), + pytest.param( + [ + make_reasoning_item(content_text="Let me think"), + make_function_call(call_id="call_123"), + ], + None, + "Let me think", + ["call_123"], + id="reasoning-tool-call", + ), + pytest.param( + [ + make_output_message("Hello"), + make_function_call(call_id="call_123"), + ], + "Hello", + None, + ["call_123"], + id="output-tool-call", + ), + pytest.param( + [ + make_reasoning_item(content_text="Thinking"), + make_output_message("Hello"), + make_function_call(call_id="call_123"), + make_function_call(call_id="call_456"), + ], + "Hello", + "Thinking", + ["call_123", "call_456"], + id="reasoning-output-tool-call", + ), + pytest.param( + [ + make_reasoning_item(content_text="Let me think"), + {"type": "message", "role": "assistant", "content": "Hello"}, + ], + "Hello", + "Let me think", + None, + id="reasoning-easyinput-assistant", + ), + ], + ) + def test_assistant_side_items_merge_until_tool_output( + self, + items, + expected_content, + expected_reasoning, + expected_tool_call_ids, + ): + messages = construct_chat_messages_with_tool_call(items) + + assert len(messages) == 1 + assert messages[0]["role"] == "assistant" + if expected_content is None: + assert "content" not in messages[0] + else: + assert messages[0]["content"] == expected_content + if expected_reasoning is None: + assert "reasoning" not in messages[0] + else: + assert messages[0]["reasoning"] == expected_reasoning + if expected_tool_call_ids is None: + assert "tool_calls" not in messages[0] + else: + assert [tool_call["id"] for tool_call in messages[0]["tool_calls"]] == ( + expected_tool_call_ids + ) + + @pytest.mark.parametrize( + ("items", "num_expected_messages"), + [ + pytest.param( + [ + make_output_message("Hello"), + make_output_message("World"), + ], + 2, + id="consecutive-output-messages", + ), + pytest.param( + [ + make_reasoning_item(content_text="Let me think"), + make_reasoning_item(content_text="Let me think more"), + ], + 2, + id="consecutive-reasoning-messages", + ), + pytest.param( + [ + make_function_call(call_id="call_123"), + make_function_call_output(call_id="call_123", output="42"), + make_function_call(call_id="call_456"), + ], + 3, + id="interrupted-by-non-assistant-item", + ), + ], + ) + def test_merge_chain_breaks(self, items, num_expected_messages): + messages = construct_chat_messages_with_tool_call(items) + assert len(messages) == num_expected_messages + + +class TestConstructInputMessagesInstructionsLeak: + """Regression tests for #37697: instructions from a prior response + should NOT leak through previous_response_id.""" + + def test_old_instructions_stripped_from_prev_msg(self): + """System message in prev_msg must be dropped so the new request's + instructions are the only system message in the conversation.""" + prev = [ + {"role": "system", "content": "old instructions"}, + {"role": "user", "content": "What is 2+2?"}, + {"role": "assistant", "content": "4"}, + ] + msgs = construct_input_messages( + request_instructions="new instructions", + request_input="What is 3+3?", + prev_msg=prev, + ) + system_msgs = [m for m in msgs if m.get("role") == "system"] + assert len(system_msgs) == 1 + assert system_msgs[0]["content"] == "new instructions" + + def test_no_instructions_in_new_request(self): + """If the new request has no instructions, old ones should still + be stripped -- they must not carry over.""" + prev = [ + {"role": "system", "content": "old instructions"}, + {"role": "user", "content": "Hi"}, + {"role": "assistant", "content": "Hello"}, + ] + msgs = construct_input_messages( + request_instructions=None, + request_input="What is 3+3?", + prev_msg=prev, + ) + system_msgs = [m for m in msgs if m.get("role") == "system"] + assert len(system_msgs) == 0 + + def test_non_system_messages_preserved(self): + """User/assistant messages from prev_msg must remain intact.""" + prev = [ + {"role": "system", "content": "old instructions"}, + {"role": "user", "content": "Hi"}, + {"role": "assistant", "content": "Hello"}, + ] + msgs = construct_input_messages( + request_instructions="new instructions", + request_input="Follow up", + prev_msg=prev, + ) + roles = [m["role"] for m in msgs] + assert roles == ["system", "user", "assistant", "user"] + assert msgs[0]["content"] == "new instructions" + assert msgs[1]["content"] == "Hi" + assert msgs[2]["content"] == "Hello" + assert msgs[3]["content"] == "Follow up" + + def test_no_prev_msg(self): + """Baseline: when there's no prev_msg, instructions work normally.""" + msgs = construct_input_messages( + request_instructions="be helpful", + request_input="hello", + prev_msg=None, + ) + assert len(msgs) == 2 + assert msgs[0] == {"role": "system", "content": "be helpful"} + assert msgs[1] == {"role": "user", "content": "hello"} diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_sampling_params.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_sampling_params.py new file mode 100644 index 0000000000000000000000000000000000000000..42506c4b68dcd83114d9942c96fc3cb19d1acf9c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_sampling_params.py @@ -0,0 +1,156 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Unit tests for ResponsesRequest.to_sampling_params() parameter mapping.""" + +import pytest +import torch +from openai.types.responses.response_format_text_json_schema_config import ( + ResponseFormatTextJSONSchemaConfig, +) +from pydantic import ValidationError + +from vllm.entrypoints.openai.responses.protocol import ( + ResponsesRequest, + ResponseTextConfig, +) +from vllm.sampling_params import StructuredOutputsParams + + +class TestResponsesRequestSamplingParams: + """Test that ResponsesRequest correctly maps parameters to SamplingParams.""" + + def test_basic_sampling_params(self): + """Test basic sampling parameters are correctly mapped.""" + request = ResponsesRequest( + model="test-model", + input="test input", + temperature=0.8, + top_p=0.95, + top_k=50, + max_output_tokens=100, + ) + + sampling_params = request.to_sampling_params(default_max_tokens=1000) + + assert sampling_params.temperature == 0.8 + assert sampling_params.top_p == 0.95 + assert sampling_params.top_k == 50 + assert sampling_params.max_tokens == 100 + + def test_extra_sampling_params(self): + """Test extra sampling parameters are correctly mapped.""" + request = ResponsesRequest( + model="test-model", + input="test input", + repetition_penalty=1.2, + seed=42, + stop=["END", "STOP"], + ignore_eos=True, + vllm_xargs={"custom": "value"}, + ) + + sampling_params = request.to_sampling_params(default_max_tokens=1000) + + assert sampling_params.repetition_penalty == 1.2 + assert sampling_params.seed == 42 + assert sampling_params.stop == ["END", "STOP"] + assert sampling_params.ignore_eos is True + assert sampling_params.extra_args == {"custom": "value"} + + def test_stop_string_conversion(self): + """Test that single stop string is converted to list.""" + request = ResponsesRequest( + model="test-model", + input="test input", + stop="STOP", + ) + + sampling_params = request.to_sampling_params(default_max_tokens=1000) + + assert sampling_params.stop == ["STOP"] + + def test_default_values(self): + """Test default values for optional parameters.""" + request = ResponsesRequest( + model="test-model", + input="test input", + ) + + sampling_params = request.to_sampling_params(default_max_tokens=1000) + + assert sampling_params.repetition_penalty == 1.0 # None → 1.0 + assert sampling_params.stop == [] # Empty list + assert sampling_params.extra_args == {} # Empty dict + + def test_seed_bounds_validation(self): + """Test that seed values outside torch.long bounds are rejected.""" + # Test seed below minimum + with pytest.raises(ValidationError) as exc_info: + ResponsesRequest( + model="test-model", + input="test input", + seed=torch.iinfo(torch.long).min - 1, + ) + assert "greater_than_equal" in str(exc_info.value).lower() + + # Test seed above maximum + with pytest.raises(ValidationError) as exc_info: + ResponsesRequest( + model="test-model", + input="test input", + seed=torch.iinfo(torch.long).max + 1, + ) + assert "less_than_equal" in str(exc_info.value).lower() + + # Test valid seed at boundaries + request_min = ResponsesRequest( + model="test-model", + input="test input", + seed=torch.iinfo(torch.long).min, + ) + assert request_min.seed == torch.iinfo(torch.long).min + + request_max = ResponsesRequest( + model="test-model", + input="test input", + seed=torch.iinfo(torch.long).max, + ) + assert request_max.seed == torch.iinfo(torch.long).max + + def test_structured_outputs_passed_through(self): + """Test that structured_outputs field is passed to SamplingParams.""" + structured_outputs = StructuredOutputsParams(grammar="root ::= 'hello'") + request = ResponsesRequest( + model="test-model", + input="test input", + structured_outputs=structured_outputs, + ) + + sampling_params = request.to_sampling_params(default_max_tokens=1000) + + assert sampling_params.structured_outputs is not None + assert sampling_params.structured_outputs.grammar == "root ::= 'hello'" + + def test_structured_outputs_and_json_schema_conflict(self): + """Test that specifying both structured_outputs and json_schema raises.""" + structured_outputs = StructuredOutputsParams(grammar="root ::= 'hello'") + text_config = ResponseTextConfig() + text_config.format = ResponseFormatTextJSONSchemaConfig( + type="json_schema", + name="test", + schema={"type": "object"}, + ) + request = ResponsesRequest( + model="test-model", + input="test input", + structured_outputs=structured_outputs, + text=text_config, + ) + + with pytest.raises(ValueError) as exc_info: + request.to_sampling_params(default_max_tokens=1000) + + assert "Cannot specify both structured_outputs and text.format" in str( + exc_info.value + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_serving_responses.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_serving_responses.py new file mode 100644 index 0000000000000000000000000000000000000000..f480a487ae4d48e6dfc2e703cc17aa61207b1ca9 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_serving_responses.py @@ -0,0 +1,1234 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from contextlib import AsyncExitStack +from unittest.mock import MagicMock + +import pytest +import pytest_asyncio +from openai.types.responses import ( + ResponseOutputItemDoneEvent, + ResponseReasoningItem, + ResponseReasoningTextDeltaEvent, + ResponseReasoningTextDoneEvent, + ResponseTextConfig, + ResponseTextDeltaEvent, +) +from openai.types.responses.response_format_text_json_schema_config import ( + ResponseFormatTextJSONSchemaConfig, +) +from openai.types.responses.tool import ( + CodeInterpreterContainerCodeInterpreterToolAuto, + LocalShell, + Mcp, + Tool, +) + +import vllm.envs as envs +from vllm.entrypoints.mcp.tool_server import ToolServer +from vllm.entrypoints.openai.engine.protocol import ( + DeltaFunctionCall, + DeltaMessage, + DeltaToolCall, + ErrorResponse, + RequestResponseMetadata, +) +from vllm.entrypoints.openai.responses.context import ConversationContext, SimpleContext +from vllm.entrypoints.openai.responses.protocol import ( + ResponseCreatedEvent, + ResponseRawMessageAndToken, + ResponsesRequest, + ResponsesResponse, + serialize_message, +) +from vllm.entrypoints.openai.responses.serving import ( + OpenAIServingResponses, + _extract_allowed_tools_from_mcp_requests, + extract_tool_types, +) +from vllm.entrypoints.openai.responses.streaming_events import ( + StreamingState, +) +from vllm.inputs import tokens_input +from vllm.outputs import CompletionOutput, RequestOutput +from vllm.parser.harmony import Segment +from vllm.sampling_params import SamplingParams + + +class MockConversationContext(ConversationContext): + """Mock conversation context for testing""" + + def __init__(self): + self.init_tool_sessions_called = False + self.init_tool_sessions_args = None + self.init_tool_sessions_kwargs = None + + def append_output(self, output) -> None: + pass + + def append_tool_output(self, output) -> None: + pass + + async def call_tool(self): + return [] + + def need_builtin_tool_call(self) -> bool: + return False + + def render_for_completion(self): + return [] + + async def init_tool_sessions(self, tool_server, exit_stack, request_id, mcp_tools): + self.init_tool_sessions_called = True + self.init_tool_sessions_args = (tool_server, exit_stack, request_id, mcp_tools) + + async def cleanup_session(self) -> None: + pass + + +def test_serialize_message_pydantic_model_returns_dict() -> None: + msg = ResponseRawMessageAndToken(message="hello", tokens=[1, 2, 3]) + + serialized = serialize_message(msg) + + assert isinstance(serialized, dict) + assert serialized["type"] == "raw_message_tokens" + assert serialized["message"] == "hello" + + +@pytest.fixture +def mock_serving_responses(): + """Create a mock OpenAIServingResponses instance""" + serving_responses = MagicMock(spec=OpenAIServingResponses) + serving_responses.tool_server = MagicMock(spec=ToolServer) + return serving_responses + + +@pytest.fixture +def mock_context(): + """Create a mock conversation context""" + return MockConversationContext() + + +@pytest.fixture +def mock_exit_stack(): + """Create a mock async exit stack""" + return MagicMock(spec=AsyncExitStack) + + +def test_extract_tool_types(monkeypatch: pytest.MonkeyPatch) -> None: + tools: list[Tool] = [] + assert extract_tool_types(tools) == set() + + tools.append(LocalShell(type="local_shell")) + assert extract_tool_types(tools) == {"local_shell"} + + tools.append(CodeInterpreterContainerCodeInterpreterToolAuto(type="auto")) + assert extract_tool_types(tools) == {"local_shell", "auto"} + + tools.extend( + [ + Mcp(type="mcp", server_label="random", server_url=""), + Mcp(type="mcp", server_label="container", server_url=""), + Mcp(type="mcp", server_label="code_interpreter", server_url=""), + Mcp(type="mcp", server_label="web_search_preview", server_url=""), + ] + ) + # When envs.VLLM_GPT_OSS_SYSTEM_TOOL_MCP_LABELS is not set, + # mcp tool types are all ignored. + assert extract_tool_types(tools) == {"local_shell", "auto"} + + # container is allowed, it would be extracted + monkeypatch.setenv("VLLM_GPT_OSS_SYSTEM_TOOL_MCP_LABELS", "container") + assert extract_tool_types(tools) == {"local_shell", "auto", "container"} + + # code_interpreter and web_search_preview are allowed, + # they would be extracted + monkeypatch.setenv( + "VLLM_GPT_OSS_SYSTEM_TOOL_MCP_LABELS", "code_interpreter,web_search_preview" + ) + assert extract_tool_types(tools) == { + "local_shell", + "auto", + "code_interpreter", + "web_search_preview", + } + + +@pytest.mark.skip_global_cleanup +def test_response_created_event_uses_public_json_schema_alias() -> None: + schema = { + "type": "object", + "properties": { + "event_name": {"type": "string"}, + "date": {"type": "string"}, + "participants": {"type": "array", "items": {"type": "string"}}, + }, + "required": ["event_name", "date", "participants"], + "additionalProperties": False, + } + text = ResponseTextConfig() + text.format = ResponseFormatTextJSONSchemaConfig( + type="json_schema", + name="calendar_event", + schema=schema, + description="A calendar event.", + strict=True, + ) + request = ResponsesRequest( + model="test-model", + input="Alice and Bob are going to a science fair on Friday.", + text=text, + ) + sampling_params = request.to_sampling_params(default_max_tokens=64) + initial_response = ResponsesResponse.from_request( + request=request, + sampling_params=sampling_params, + model_name="test-model", + created_time=0, + output=[], + status="in_progress", + usage=None, + ).model_dump(mode="json", by_alias=True) + + fmt = initial_response["text"]["format"] + assert fmt["schema"] == schema + assert "schema_" not in fmt + + event = ResponseCreatedEvent( + type="response.created", + sequence_number=0, + response=initial_response, + ) + assert event.response.text is not None + assert event.response.text.format is not None + assert event.response.text.format.model_dump(by_alias=True)["schema"] == schema + + +class TestInitializeToolSessions: + """Test class for _initialize_tool_sessions method""" + + @pytest_asyncio.fixture + async def serving_responses_instance(self): + """Create a real OpenAIServingResponses instance for testing""" + # Create minimal mocks for required dependencies + engine_client = MagicMock() + + model_config = MagicMock() + model_config.max_model_len = 100 + model_config.hf_config.model_type = "test" + model_config.get_diff_sampling_param.return_value = {} + engine_client.model_config = model_config + + engine_client.input_processor = MagicMock() + engine_client.renderer = MagicMock() + + models = MagicMock() + + tool_server = MagicMock(spec=ToolServer) + + # Create the actual instance + instance = OpenAIServingResponses( + engine_client=engine_client, + models=models, + online_renderer=MagicMock(), + request_logger=None, + chat_template=None, + chat_template_content_format="auto", + tool_server=tool_server, + ) + + return instance + + @pytest.mark.asyncio + async def test_initialize_tool_sessions( + self, serving_responses_instance, mock_context, mock_exit_stack + ): + """Test that method works correctly with only MCP tools""" + + request = ResponsesRequest(input="test input", tools=[]) + + # Call the method + await serving_responses_instance._initialize_tool_sessions( + request, mock_context, mock_exit_stack + ) + assert mock_context.init_tool_sessions_called is False + + # Create only MCP tools + tools = [ + {"type": "web_search_preview"}, + {"type": "code_interpreter", "container": {"type": "auto"}}, + ] + + request = ResponsesRequest(input="test input", tools=tools) + + # Call the method + await serving_responses_instance._initialize_tool_sessions( + request, mock_context, mock_exit_stack + ) + + # Verify that init_tool_sessions was called + assert mock_context.init_tool_sessions_called + + def test_validate_create_responses_input( + self, serving_responses_instance, mock_context, mock_exit_stack + ): + request = ResponsesRequest( + input="test input", + previous_input_messages=[ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "What is my horoscope? I am an Aquarius.", + } + ], + } + ], + previous_response_id="lol", + ) + error = serving_responses_instance._validate_create_responses_input(request) + assert error is not None + assert error.error.type == "invalid_request_error" + + +class TestValidateGeneratorInput: + """Test class for _validate_generator_input method""" + + @pytest_asyncio.fixture + async def serving_responses_instance(self): + """Create a real OpenAIServingResponses instance for testing""" + # Create minimal mocks for required dependencies + engine_client = MagicMock() + + model_config = MagicMock() + model_config.max_model_len = 100 + model_config.hf_config.model_type = "test" + model_config.get_diff_sampling_param.return_value = {} + engine_client.model_config = model_config + + engine_client.input_processor = MagicMock() + engine_client.renderer = MagicMock() + + models = MagicMock() + + # Create the actual instance + instance = OpenAIServingResponses( + engine_client=engine_client, + models=models, + online_renderer=MagicMock(), + request_logger=None, + chat_template=None, + chat_template_content_format="auto", + ) + + return instance + + def test_validate_generator_input(self, serving_responses_instance): + """Test _validate_generator_input with valid prompt length""" + # Create an engine prompt with valid length (less than max_model_len) + valid_prompt_token_ids = list(range(5)) # 5 tokens < 100 max_model_len + engine_input = tokens_input(valid_prompt_token_ids) + + # Call the method + result = serving_responses_instance._validate_generator_input(engine_input) + + # Should return None for valid input + assert result is None + + # create an invalid engine prompt + invalid_prompt_token_ids = list(range(200)) # 100 tokens >= 100 max_model_len + engine_input = tokens_input(invalid_prompt_token_ids) + + # Call the method + result = serving_responses_instance._validate_generator_input(engine_input) + + # Should return an ErrorResponse + assert result is not None + assert isinstance(result, ErrorResponse) + + +@pytest.mark.asyncio +async def test_reasoning_tokens_counted_for_text_reasoning_model(monkeypatch): + """Ensure reasoning_tokens usage is derived from thinking token spans.""" + + class FakeTokenizer: + def __init__(self): + self._vocab = {"": 1, "": 2, "reason": 3, "final": 4} + + def get_vocab(self): + return self._vocab + + # Force non-harmony, SimpleContext path + monkeypatch.setattr(envs, "VLLM_USE_EXPERIMENTAL_PARSER_CONTEXT", False) + + engine_client = MagicMock() + model_config = MagicMock() + model_config.hf_config.model_type = "test" + model_config.hf_text_config = MagicMock() + model_config.get_diff_sampling_param.return_value = {} + engine_client.model_config = model_config + engine_client.input_processor = MagicMock() + engine_client.renderer = MagicMock() + + tokenizer = FakeTokenizer() + engine_client.renderer.get_tokenizer.return_value = tokenizer + + models = MagicMock() + + serving = OpenAIServingResponses( + engine_client=engine_client, + models=models, + online_renderer=MagicMock(), + request_logger=None, + chat_template=None, + chat_template_content_format="auto", + reasoning_parser="qwen3", + ) + + request = ResponsesRequest(input="hi", tools=[], stream=False) + response_parser = serving._make_response_parser( + request, + tokenizer, + serving._effective_chat_template_kwargs(request), + ) + + # Build a SimpleContext with thinking tokens in the output. + context = SimpleContext(response_parser=response_parser) + token_ids = [1, 10, 2, 20] # 10 20 -> reasoning token count = 1 + completion = CompletionOutput( + index=0, + text="reasonfinal", + token_ids=token_ids, + cumulative_logprob=0.0, + logprobs=None, + finish_reason="stop", + stop_reason=None, + ) + req_output = RequestOutput( + request_id="req", + prompt="hi", + prompt_token_ids=[7, 8], + prompt_logprobs=None, + outputs=[completion], + finished=True, + num_cached_tokens=0, + ) + context.append_output(req_output) + + async def dummy_result_generator(): + yield None + + sampling_params = SamplingParams(max_tokens=16) + metadata = RequestResponseMetadata(request_id="req") + + response = await serving.responses_full_generator( + request=request, + sampling_params=sampling_params, + result_generator=dummy_result_generator(), + context=context, + model_name="test-model", + tokenizer=tokenizer, + request_metadata=metadata, + ) + + assert response.usage.output_tokens_details.reasoning_tokens == 1 + + +class TestExtractAllowedToolsFromMcpRequests: + """Test class for _extract_allowed_tools_from_mcp_requests function""" + + def test_extract_allowed_tools_basic_formats(self): + """Test extraction with list format, object format, and None.""" + from openai.types.responses.tool import McpAllowedToolsMcpToolFilter + + tools = [ + # List format + Mcp( + type="mcp", + server_label="server1", + allowed_tools=["tool1", "tool2"], + ), + # Object format + Mcp( + type="mcp", + server_label="server2", + allowed_tools=McpAllowedToolsMcpToolFilter( + tool_names=["tool3", "tool4"] + ), + ), + # None (no filter) + Mcp( + type="mcp", + server_label="server3", + allowed_tools=None, + ), + ] + result = _extract_allowed_tools_from_mcp_requests(tools) + assert result == { + "server1": ["tool1", "tool2"], + "server2": ["tool3", "tool4"], + "server3": None, + } + + def test_extract_allowed_tools_star_normalization(self): + """Test that '*' wildcard is normalized to None (select all tools). + + This is the key test requested by reviewers to explicitly demonstrate + that the "*" select-all scenario is handled correctly. + """ + from openai.types.responses.tool import McpAllowedToolsMcpToolFilter + + tools = [ + # Star in list format + Mcp( + type="mcp", + server_label="server1", + allowed_tools=["*"], + ), + # Star mixed with other tools in list + Mcp( + type="mcp", + server_label="server2", + allowed_tools=["tool1", "*"], + ), + # Star in object format + Mcp( + type="mcp", + server_label="server3", + allowed_tools=McpAllowedToolsMcpToolFilter(tool_names=["*"]), + ), + ] + result = _extract_allowed_tools_from_mcp_requests(tools) + # All should be normalized to None (allows all tools) + assert result == { + "server1": None, + "server2": None, + "server3": None, + } + + def test_extract_allowed_tools_filters_non_mcp(self): + """Test that non-MCP tools are ignored during extraction.""" + tools = [ + Mcp( + type="mcp", + server_label="server1", + allowed_tools=["tool1"], + ), + LocalShell(type="local_shell"), # Non-MCP tool should be ignored + Mcp( + type="mcp", + server_label="server2", + allowed_tools=["tool2"], + ), + ] + result = _extract_allowed_tools_from_mcp_requests(tools) + # Non-MCP tools should be ignored + assert result == { + "server1": ["tool1"], + "server2": ["tool2"], + } + + +class TestHarmonyPreambleStreaming: + """Tests for preamble (commentary with no recipient) streaming events.""" + + @staticmethod + def _make_segment(*, channel, recipient, delta="hello"): + """Build a lightweight segment for Harmony streaming tests.""" + return Segment(channel=channel, recipient=recipient, delta=delta) + + @staticmethod + def _make_previous_item(*, channel, recipient, text="preamble text"): + """Build a lightweight mock previous_item (openai_harmony Message).""" + content_part = MagicMock() + content_part.text = text + item = MagicMock() + item.channel = channel + item.recipient = recipient + item.content = [content_part] + return item + + def test_preamble_delta_emits_text_events(self) -> None: + """commentary + recipient=None should emit output_text.delta events.""" + from vllm.entrypoints.openai.responses.streaming_events import ( + emit_content_delta_events, + ) + + segment = self._make_segment(channel="commentary", recipient=None) + state = StreamingState() + + events = emit_content_delta_events(segment, state) + + type_names = [e.type for e in events] + assert "response.output_text.delta" in type_names + assert "response.output_item.added" in type_names + + def test_preamble_delta_second_token_no_added(self) -> None: + """Second preamble token should emit delta only, not added again.""" + from vllm.entrypoints.openai.responses.streaming_events import ( + emit_content_delta_events, + ) + + segment = self._make_segment(channel="commentary", recipient=None, delta="w") + state = StreamingState() + state.sent_output_item_added = True + state.current_item_id = "msg_test" + state.current_content_index = 0 + + events = emit_content_delta_events(segment, state) + + type_names = [e.type for e in events] + assert "response.output_text.delta" in type_names + assert "response.output_item.added" not in type_names + + def test_commentary_with_function_recipient_not_preamble(self) -> None: + """commentary + recipient='functions.X' must NOT use preamble path.""" + from vllm.entrypoints.openai.responses.streaming_events import ( + emit_content_delta_events, + ) + + segment = self._make_segment( + channel="commentary", + recipient="functions.get_weather", + ) + state = StreamingState() + + events = emit_content_delta_events(segment, state) + + type_names = [e.type for e in events] + assert "response.output_text.delta" not in type_names + + def test_preamble_done_emits_text_done_events(self) -> None: + """Completed preamble should emit text done + content_part done + + output_item done, same shape as final channel.""" + from vllm.entrypoints.openai.responses.streaming_events import ( + emit_previous_item_done_events, + ) + + previous = self._make_previous_item(channel="commentary", recipient=None) + state = StreamingState() + state.sent_output_item_added = True + state.current_item_id = "msg_test" + state.current_output_index = 0 + state.current_content_index = 0 + + events = emit_previous_item_done_events(previous, state) + + type_names = [e.type for e in events] + assert "response.output_text.done" in type_names + assert "response.content_part.done" in type_names + assert "response.output_item.done" in type_names + + def test_commentary_with_recipient_no_preamble_done(self) -> None: + """commentary + recipient='functions.X' should route to function call + done, not preamble done.""" + from vllm.entrypoints.openai.responses.streaming_events import ( + emit_previous_item_done_events, + ) + + previous = self._make_previous_item( + channel="commentary", recipient="functions.get_weather" + ) + state = StreamingState() + state.is_first_function_call_delta = True + state.current_item_id = "fc_test" + state.current_call_id = "call_test" + + events = emit_previous_item_done_events(previous, state) + + type_names = [e.type for e in events] + assert "response.output_text.done" not in type_names + + @pytest.mark.xfail( + reason=( + "TODO: Ensure added/in-progress events are emitted for zero-delta items." + "So we can safely emit done events for zero-delta items." + ), + strict=True, + ) + def test_zero_delta_items_should_preserve_streaming_lifecycle( + self, + ) -> None: + """Zero-delta Harmony items should still produce a coherent lifecycle.""" + from vllm.entrypoints.openai.responses.streaming_events import ( + emit_previous_item_done_events, + ) + + cases: list[tuple[str, str | None, str]] = [ + ("commentary", None, "msg_stale"), + ("analysis", None, "msg_stale"), + ("commentary", "functions.get_weather", "fc_stale"), + ("commentary", "python", "tool_stale"), + ("commentary", "repo_browser.list", "mcp_stale"), + ] + + for channel, recipient, current_item_id in cases: + previous = self._make_previous_item(channel=channel, recipient=recipient) + state = StreamingState() + state.current_item_id = current_item_id + state.current_call_id = "call_stale" + state.current_content_index = 0 + + events = emit_previous_item_done_events( + previous, state, function_tool_names=None + ) + + type_names = [e.type for e in events] + assert "response.output_item.added" in type_names + assert "response.output_item.done" in type_names + + +def _make_simple_context_with_output(text, token_ids, response_parser=None): + """Create a SimpleContext with a RequestOutput containing the given text.""" + ctx = SimpleContext(response_parser=response_parser) + completion = CompletionOutput( + index=0, + text=text, + token_ids=token_ids, + cumulative_logprob=0.0, + logprobs=None, + finish_reason=None, + stop_reason=None, + ) + req_output = RequestOutput( + request_id="req", + prompt="hi", + prompt_token_ids=[7, 8], + prompt_logprobs=None, + outputs=[completion], + finished=False, + num_cached_tokens=0, + ) + ctx.append_output(req_output) + return ctx + + +def _make_serving_instance_with_reasoning(): + """Create an OpenAIServingResponses with a mocked reasoning parser.""" + engine_client = MagicMock() + model_config = MagicMock() + model_config.max_model_len = 100 + model_config.hf_config.model_type = "test" + model_config.hf_text_config = MagicMock() + model_config.get_diff_sampling_param.return_value = {} + engine_client.model_config = model_config + engine_client.input_processor = MagicMock() + engine_client.renderer = MagicMock() + + models = MagicMock() + + serving = OpenAIServingResponses( + engine_client=engine_client, + models=models, + online_renderer=MagicMock(), + request_logger=None, + chat_template=None, + chat_template_content_format="auto", + reasoning_parser="qwen3", + ) + return serving + + +def _identity_increment(event): + """Simple identity callable for _increment_sequence_number_and_return.""" + seq = getattr(_identity_increment, "_counter", 0) + if hasattr(event, "sequence_number"): + event.sequence_number = seq + _identity_increment._counter = seq + 1 # type: ignore + return event + + +def _mock_parser_with_reasoning(serving, delta_sequence: list[DeltaMessage]): + """Set up serving.parser so that it returns a mock parser instance + with a reasoning parser that returns the given delta_sequence. + + The mock has reasoning_parser set (truthy) but tool_parser as None, + so the parser's parse_delta enters the reasoning-only branch. + """ + call_count = 0 + + def mock_parse_delta(**kwargs): + nonlocal call_count + if call_count >= len(delta_sequence): + return None + result = delta_sequence[call_count] + call_count += 1 + return result + + mock_parser_instance = MagicMock() + mock_parser_instance.reasoning_parser = MagicMock() # truthy + mock_parser_instance.tool_parser = None + mock_parser_instance.parse_delta = mock_parse_delta + mock_parser_instance.is_reasoning_end = MagicMock(return_value=False) + serving.parser = MagicMock(return_value=mock_parser_instance) + return mock_parser_instance + + +class TestStreamingReasoningToContentTransition: + """Tests for _process_simple_streaming_events reasoning-to-content + transition, specifically the fix for mixed deltas that carry both + reasoning and content simultaneously.""" + + @pytest.mark.asyncio + async def test_mixed_delta_reasoning_and_content_emits_reasoning_delta( + self, monkeypatch + ): + """When the reasoning parser produces a delta with both reasoning + and content set (e.g. reasoning end and content start in the same + chunk), the trailing reasoning text must be emitted as a + ResponseReasoningTextDeltaEvent and included in the + ResponseReasoningTextDoneEvent text.""" + + monkeypatch.setattr(envs, "VLLM_USE_EXPERIMENTAL_PARSER_CONTEXT", False) + serving = _make_serving_instance_with_reasoning() + + # Sequence of DeltaMessages the mock orchestrator will return + delta_sequence = [ + DeltaMessage(reasoning="thinking..."), + DeltaMessage(reasoning=" end", content="hello"), # mixed delta + DeltaMessage(content=" world"), + ] + response_parser = _mock_parser_with_reasoning(serving, delta_sequence) + # Create contexts for each streaming chunk + contexts = [ + _make_simple_context_with_output("chunk1", [10], response_parser), + _make_simple_context_with_output("chunk2", [20], response_parser), + _make_simple_context_with_output("chunk3", [30], response_parser), + ] + + async def result_generator(): + for ctx in contexts: + yield ctx + + request = ResponsesRequest(input="hi", tools=[], stream=True) + sampling_params = SamplingParams(max_tokens=64) + metadata = RequestResponseMetadata(request_id="req") + _identity_increment._counter = 0 # type: ignore + + events = [] + async for event in serving._process_simple_streaming_events( + request=request, + sampling_params=sampling_params, + result_generator=result_generator(), + context=SimpleContext(response_parser=response_parser), + model_name="test-model", + tokenizer=MagicMock(), + request_metadata=metadata, + created_time=0, + _increment_sequence_number_and_return=_identity_increment, + ): + events.append(event) + + # The first reasoning delta should be emitted + reasoning_deltas = [ + e for e in events if isinstance(e, ResponseReasoningTextDeltaEvent) + ] + assert len(reasoning_deltas) == 2 + assert reasoning_deltas[0].delta == "thinking..." + # The trailing reasoning from the mixed delta must also be emitted + assert reasoning_deltas[1].delta == " end" + + # The done event must include both reasoning parts + reasoning_done = [ + e for e in events if isinstance(e, ResponseReasoningTextDoneEvent) + ] + assert len(reasoning_done) == 1 + assert reasoning_done[0].text == "thinking... end" + + # Content deltas should be emitted for both the mixed delta's + # content and the pure content delta + text_deltas = [e for e in events if isinstance(e, ResponseTextDeltaEvent)] + assert len(text_deltas) == 2 + assert text_deltas[0].delta == "hello" + assert text_deltas[1].delta == " world" + + @pytest.mark.asyncio + async def test_transition_without_mixed_delta_no_extra_reasoning_event( + self, monkeypatch + ): + """When the transition from reasoning to content is clean (no mixed + delta), no extra reasoning delta event should be emitted.""" + + monkeypatch.setattr(envs, "VLLM_USE_EXPERIMENTAL_PARSER_CONTEXT", False) + serving = _make_serving_instance_with_reasoning() + + delta_sequence = [ + DeltaMessage(reasoning="thinking"), + DeltaMessage(content="answer"), + ] + response_parser = _mock_parser_with_reasoning(serving, delta_sequence) + + contexts = [ + _make_simple_context_with_output("chunk1", [10], response_parser), + _make_simple_context_with_output("chunk2", [20], response_parser), + ] + + async def result_generator(): + for ctx in contexts: + yield ctx + + request = ResponsesRequest(input="hi", tools=[], stream=True) + sampling_params = SamplingParams(max_tokens=64) + metadata = RequestResponseMetadata(request_id="req") + _identity_increment._counter = 0 # type: ignore + + events = [] + async for event in serving._process_simple_streaming_events( + request=request, + sampling_params=sampling_params, + result_generator=result_generator(), + context=SimpleContext(response_parser=response_parser), + model_name="test-model", + tokenizer=MagicMock(), + request_metadata=metadata, + created_time=0, + _increment_sequence_number_and_return=_identity_increment, + ): + events.append(event) + + # Exactly one reasoning delta + reasoning_deltas = [ + e for e in events if isinstance(e, ResponseReasoningTextDeltaEvent) + ] + assert len(reasoning_deltas) == 1 + assert reasoning_deltas[0].delta == "thinking" + + # Done event has just "thinking" + reasoning_done = [ + e for e in events if isinstance(e, ResponseReasoningTextDoneEvent) + ] + assert len(reasoning_done) == 1 + assert reasoning_done[0].text == "thinking" + + # One content delta + text_deltas = [e for e in events if isinstance(e, ResponseTextDeltaEvent)] + assert len(text_deltas) == 1 + assert text_deltas[0].delta == "answer" + + @pytest.mark.asyncio + async def test_reasoning_only_stream_no_content(self, monkeypatch): + """When the stream has only reasoning deltas and no content, the + reasoning done event should be emitted at finalization with the + full accumulated text, and no text delta events should appear.""" + + monkeypatch.setattr(envs, "VLLM_USE_EXPERIMENTAL_PARSER_CONTEXT", False) + serving = _make_serving_instance_with_reasoning() + + delta_sequence = [ + DeltaMessage(reasoning="step 1"), + DeltaMessage(reasoning=" step 2"), + ] + response_parser = _mock_parser_with_reasoning(serving, delta_sequence) + + contexts = [ + _make_simple_context_with_output("chunk1", [10], response_parser), + _make_simple_context_with_output("chunk2", [20], response_parser), + ] + + async def result_generator(): + for ctx in contexts: + yield ctx + + request = ResponsesRequest(input="hi", tools=[], stream=True) + sampling_params = SamplingParams(max_tokens=64) + metadata = RequestResponseMetadata(request_id="req") + _identity_increment._counter = 0 # type: ignore + + events = [] + async for event in serving._process_simple_streaming_events( + request=request, + sampling_params=sampling_params, + result_generator=result_generator(), + context=SimpleContext(response_parser=response_parser), + model_name="test-model", + tokenizer=MagicMock(), + request_metadata=metadata, + created_time=0, + _increment_sequence_number_and_return=_identity_increment, + ): + events.append(event) + + # Two reasoning deltas + reasoning_deltas = [ + e for e in events if isinstance(e, ResponseReasoningTextDeltaEvent) + ] + assert len(reasoning_deltas) == 2 + assert reasoning_deltas[0].delta == "step 1" + assert reasoning_deltas[1].delta == " step 2" + + # Done event at finalization with accumulated text + reasoning_done = [ + e for e in events if isinstance(e, ResponseReasoningTextDoneEvent) + ] + assert len(reasoning_done) == 1 + assert reasoning_done[0].text == "step 1 step 2" + + # No content text deltas + text_deltas = [e for e in events if isinstance(e, ResponseTextDeltaEvent)] + assert len(text_deltas) == 0 + + # Final item should be a reasoning item + item_done_events = [ + e for e in events if isinstance(e, ResponseOutputItemDoneEvent) + ] + assert len(item_done_events) == 1 + assert isinstance(item_done_events[0].item, ResponseReasoningItem) + + +class TestAutoToolStreaming: + @staticmethod + async def _collect_events(delta_sequence: list[DeltaMessage]): + serving = _make_serving_instance_with_reasoning() + response_parser = _mock_parser_with_reasoning(serving, delta_sequence) + + contexts = [ + _make_simple_context_with_output("chunk", [i], response_parser) + for i in range(len(delta_sequence)) + ] + + async def result_generator(): + for ctx in contexts: + yield ctx + + request = ResponsesRequest( + input="hi", + tools=[ + { + "type": "function", + "name": "get_weather", + "description": "Get weather.", + "parameters": { + "type": "object", + "properties": {"location": {"type": "string"}}, + "required": ["location"], + "additionalProperties": False, + }, + } + ], + tool_choice="auto", + stream=True, + ) + sampling_params = SamplingParams(max_tokens=64) + metadata = RequestResponseMetadata(request_id="req") + _identity_increment._counter = 0 # type: ignore + + events = [] + async for event in serving._process_simple_streaming_events( + request=request, + sampling_params=sampling_params, + result_generator=result_generator(), + context=SimpleContext(response_parser=response_parser), + model_name="test-model", + tokenizer=MagicMock(), + request_metadata=metadata, + created_time=0, + _increment_sequence_number_and_return=_identity_increment, + ): + events.append(event) + return events + + @pytest.mark.skip_global_cleanup + @pytest.mark.asyncio + async def test_auto_multi_tool_streaming_opens_one_item_per_tool(self, monkeypatch): + monkeypatch.setattr(envs, "VLLM_USE_EXPERIMENTAL_PARSER_CONTEXT", False) + + delta_sequence = [ + DeltaMessage( + tool_calls=[ + DeltaToolCall( + id="call_vienna", + type="function", + index=0, + function=DeltaFunctionCall( + name="get_weather", + arguments="", + ), + ) + ] + ), + DeltaMessage( + tool_calls=[ + DeltaToolCall( + index=0, + function=DeltaFunctionCall( + arguments='{"location":"Vienna"}', + ), + ) + ] + ), + DeltaMessage( + tool_calls=[ + DeltaToolCall( + id="call_berlin", + type="function", + index=1, + function=DeltaFunctionCall( + name="get_weather", + arguments='{"location":"Berlin"}', + ), + ) + ] + ), + ] + events = await self._collect_events(delta_sequence) + + function_items = [ + event + for event in events + if event.type == "response.output_item.added" + and getattr(event.item, "type", None) == "function_call" + ] + assert len(function_items) == 2 + assert [event.item.name for event in function_items] == [ + "get_weather", + "get_weather", + ] + assert [event.output_index for event in function_items] == [0, 1] + + argument_deltas = [ + event.delta + for event in events + if event.type == "response.function_call_arguments.delta" + ] + assert argument_deltas == [ + '{"location":"Vienna"}', + '{"location":"Berlin"}', + ] + + argument_done = [ + event + for event in events + if event.type == "response.function_call_arguments.done" + ] + assert [event.arguments for event in argument_done] == [ + '{"location":"Vienna"}', + '{"location":"Berlin"}', + ] + assert [event.output_index for event in argument_done] == [0, 1] + + function_done = [ + event + for event in events + if event.type == "response.output_item.done" + and getattr(event.item, "type", None) == "function_call" + ] + assert [event.item.arguments for event in function_done] == [ + '{"location":"Vienna"}', + '{"location":"Berlin"}', + ] + assert [event.output_index for event in function_done] == [0, 1] + + @pytest.mark.skip_global_cleanup + @pytest.mark.asyncio + async def test_auto_tool_choice_first_delta_tool_call_does_not_duplicate_item( + self, monkeypatch + ): + monkeypatch.setattr(envs, "VLLM_USE_EXPERIMENTAL_PARSER_CONTEXT", False) + + delta_sequence = [ + DeltaMessage( + tool_calls=[ + DeltaToolCall( + id="call_test", + type="function", + index=0, + function=DeltaFunctionCall( + name="get_weather", + arguments="", + ), + ) + ] + ), + DeltaMessage( + tool_calls=[ + DeltaToolCall( + index=0, + function=DeltaFunctionCall( + arguments='{"location":"Berlin"}', + ), + ) + ] + ), + ] + events = await self._collect_events(delta_sequence) + + function_items = [ + event + for event in events + if event.type == "response.output_item.added" + and getattr(event.item, "type", None) == "function_call" + ] + assert len(function_items) == 1 + assert function_items[0].item.name == "get_weather" + + argument_deltas = [ + event.delta + for event in events + if event.type == "response.function_call_arguments.delta" + ] + assert "".join(argument_deltas) == '{"location":"Berlin"}' + + @pytest.mark.skip_global_cleanup + @pytest.mark.asyncio + async def test_compound_content_and_tool_name_args_same_delta(self, monkeypatch): + monkeypatch.setattr(envs, "VLLM_USE_EXPERIMENTAL_PARSER_CONTEXT", False) + + tool_args = '{"location":"Berlin"}' + + delta_sequence = [ + DeltaMessage( + content="Let me check.", + tool_calls=[ + DeltaToolCall( + id="call_weather", + type="function", + index=0, + function=DeltaFunctionCall(name="get_weather"), + ), + DeltaToolCall( + index=0, + function=DeltaFunctionCall(arguments=tool_args), + ), + ], + ) + ] + + events = await self._collect_events(delta_sequence) + + text_deltas = [ + event.delta + for event in events + if event.type == "response.output_text.delta" + ] + assert text_deltas == ["Let me check."] + + argument_deltas = [ + event.delta + for event in events + if event.type == "response.function_call_arguments.delta" + ] + assert argument_deltas == [tool_args] + + types = [event.type for event in events] + assert types.index("response.output_text.delta") < types.index( + "response.function_call_arguments.delta" + ) + + argument_done = [ + event + for event in events + if event.type == "response.function_call_arguments.done" + ] + assert [event.arguments for event in argument_done] == [tool_args] + + function_done = [ + event + for event in events + if event.type == "response.output_item.done" + and getattr(event.item, "type", None) == "function_call" + ] + assert len(function_done) == 1 + assert function_done[0].item.name == "get_weather" + assert function_done[0].item.arguments == tool_args diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_simple.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_simple.py new file mode 100644 index 0000000000000000000000000000000000000000..55f3110c126f2c2634bb584977a487079380ebe0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_simple.py @@ -0,0 +1,296 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import pytest_asyncio +from openai import OpenAI + +from tests.utils import RemoteOpenAIServer + +from .conftest import validate_streaming_event_stack + +MODEL_NAME = "Qwen/Qwen3-8B" + + +@pytest.fixture(scope="module") +def server(): + from .conftest import BASE_TEST_ENV + + args = ["--reasoning-parser", "qwen3", "--max_model_len", "5000"] + env_dict = { + **BASE_TEST_ENV, + "VLLM_ENABLE_RESPONSES_API_STORE": "1", + # uncomment for tool calling + # PYTHON_EXECUTION_BACKEND: "dangerously_use_uv", + } + with RemoteOpenAIServer(MODEL_NAME, args, env_dict=env_dict) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_basic(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input="What is 123 * 456?", + ) + assert response is not None + print("response: ", response) + assert response.status == "completed" + assert response.incomplete_details is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_enable_response_messages(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input="Hello?", + extra_body={"enable_response_messages": True}, + ) + assert response.status == "completed" + assert response.input_messages[0]["type"] == "raw_message_tokens" + assert type(response.input_messages[0]["message"]) is str + assert len(response.input_messages[0]["message"]) > 10 + assert type(response.input_messages[0]["tokens"][0]) is int + assert type(response.output_messages[0]["message"]) is str + assert len(response.output_messages[0]["message"]) > 10 + assert type(response.output_messages[0]["tokens"][0]) is int + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_reasoning_item(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input=[ + {"type": "message", "content": "Hello.", "role": "user"}, + { + "type": "reasoning", + "id": "lol", + "content": [ + { + "type": "reasoning_text", + "text": "We need to respond: greeting.", + } + ], + "summary": [], + }, + ], + temperature=0.0, + ) + assert response is not None + assert response.status == "completed" + # make sure we get a reasoning and text output + assert response.output[0].type == "reasoning" + assert response.output[1].type == "message" + assert type(response.output[1].content[0].text) is str + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_streaming_output_consistency(client: OpenAI, model_name: str): + """Test that streaming delta text matches the final response output_text. + + This test verifies that when using streaming mode: + 1. The concatenated text from all 'response.output_text.delta' events + 2. Matches the 'output_text' in the final 'response.completed' event + """ + response = await client.responses.create( + model=model_name, + input="Say hello in one sentence.", + stream=True, + ) + + events = [] + async for event in response: + events.append(event) + + assert len(events) > 0 + + # Concatenate all delta text from streaming events + streaming_text = "".join( + event.delta for event in events if event.type == "response.output_text.delta" + ) + + # Get the final response from the last event + response_completed_event = events[-1] + assert response_completed_event.type == "response.completed" + assert response_completed_event.response.status == "completed" + + # Get output_text from the final response + final_output_text = response_completed_event.response.output_text + + # Verify final response has output + assert len(response_completed_event.response.output) > 0 + + # Verify streaming text matches final output_text + assert streaming_text == final_output_text, ( + f"Streaming text does not match final output_text.\n" + f"Streaming: {streaming_text!r}\n" + f"Final: {final_output_text!r}" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_streaming_logprobs(client: OpenAI, model_name: str): + """Test that streaming with logprobs returns valid logprob data on + output_text.delta events and that top_logprobs has the requested count.""" + response = await client.responses.create( + model=model_name, + input="Say hello.", + stream=True, + top_logprobs=3, + include=["message.output_text.logprobs"], + ) + + events = [] + async for event in response: + events.append(event) + + assert len(events) > 0 + + # Collect all output_text.delta events that carry logprobs + text_delta_events = [e for e in events if e.type == "response.output_text.delta"] + assert len(text_delta_events) > 0, "Expected at least one text delta event" + + for delta_event in text_delta_events: + logprobs = delta_event.logprobs + assert logprobs is not None, "logprobs should be present on text delta events" + assert len(logprobs) > 0, "logprobs list should not be empty" + for lp in logprobs: + # Each logprob entry must have a token and a logprob value + assert lp.token is not None + assert isinstance(lp.logprob, float) + assert lp.logprob <= 0.0, f"logprob should be <= 0, got {lp.logprob}" + # top_logprobs should have up to 3 entries + assert lp.top_logprobs is not None + assert len(lp.top_logprobs) <= 3 + for tl in lp.top_logprobs: + assert tl.token is not None + assert isinstance(tl.logprob, float) + + # Verify that top_logprobs are actually populated, not always empty + all_top_logprobs = [ + tl for e in text_delta_events for lp in e.logprobs for tl in lp.top_logprobs + ] + assert len(all_top_logprobs) > 0, ( + "Expected at least one top_logprobs entry across all delta events" + ) + + # Verify the completed event still has valid output + completed = events[-1] + assert completed.type == "response.completed" + assert completed.response.status == "completed" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_streaming_reasoning_tokens_e2e(client: OpenAI, model_name: str): + """Verify final usage includes reasoning_tokens in streaming mode.""" + response = await client.responses.create( + model=model_name, + input="Compute 17 * 19 and explain briefly.", + reasoning={"effort": "low"}, + temperature=0.0, + stream=True, + ) + + completed_event = None + async for event in response: + if event.type == "response.completed": + completed_event = event + + assert completed_event is not None + assert completed_event.response.status == "completed" + assert completed_event.response.usage is not None + assert completed_event.response.usage.output_tokens_details is not None + assert completed_event.response.usage.output_tokens_details.reasoning_tokens > 0, ( + "Expected reasoning_tokens > 0 for streamed Qwen3 response." + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_non_streaming_reasoning_tokens_e2e(client: OpenAI, model_name: str): + """Verify usage includes reasoning_tokens in non-streaming mode.""" + response = await client.responses.create( + model=model_name, + input="Compute 23 * 17 and explain briefly.", + reasoning={"effort": "low"}, + temperature=0.0, + stream=False, + ) + + assert response is not None + assert response.status == "completed" + assert response.usage is not None + assert response.usage.output_tokens_details is not None + assert response.usage.output_tokens_details.reasoning_tokens > 0, ( + "Expected reasoning_tokens > 0 for non-streamed Qwen3 response." + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_max_tokens(client: OpenAI, model_name: str): + response = await client.responses.create( + model=model_name, + input="What is the first paragraph of Moby Dick?", + reasoning={"effort": "low"}, + max_output_tokens=30, + ) + assert response is not None + assert response.status == "incomplete" + assert response.incomplete_details.reason == "max_output_tokens" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_extra_sampling_params(client: OpenAI, model_name: str): + """Test that extra sampling parameters are accepted and work.""" + # Test with multiple sampling parameters - just verify they're accepted + response = await client.responses.create( + model=model_name, + input="Write a short sentence", + max_output_tokens=50, + temperature=0.7, + top_p=0.9, + extra_body={ + "top_k": 40, + "repetition_penalty": 1.2, + "seed": 42, + }, + ) + + # Verify request succeeded and parameters were accepted + assert response.status in ["completed", "incomplete"] + assert len(response.output) > 0 + assert response.output[0].content[0].text # Has text output + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_streaming_types( + pairs_of_event_types: dict[str, str], client: OpenAI, model_name: str +): + stream = await client.responses.create( + model=model_name, + input="tell me a story about a cat in 20 words", + reasoning={"effort": "low"}, + tools=[], + stream=True, + background=False, + ) + events = [] + async for event in stream: + events.append(event) + + validate_streaming_event_stack(events, pairs_of_event_types) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_stateful.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_stateful.py new file mode 100644 index 0000000000000000000000000000000000000000..ac05aceaf12cd9310e8cce84e03697e9e7ada4d5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_stateful.py @@ -0,0 +1,152 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import asyncio + +import openai +import pytest + + +@pytest.mark.asyncio +async def test_store(client: openai.AsyncOpenAI): + # By default, store is True. + response = await client.responses.create(input="Hello!") + assert response.status == "completed" + + # Retrieve the response. + response = await client.responses.retrieve(response.id) + assert response.status == "completed" + + # Test store=False. + response = await client.responses.create( + input="Hello!", + store=False, + ) + assert response.status == "completed" + + # The response should not be found. + with pytest.raises(openai.NotFoundError, match="Response with id .* not found."): + await client.responses.retrieve(response.id) + + +@pytest.mark.asyncio +async def test_background(client: openai.AsyncOpenAI): + # NOTE: This query should be easy enough for the model to answer + # within the 10 seconds. + response = await client.responses.create( + input="Hello!", + background=True, + ) + assert response.status == "queued" + + max_retries = 10 + for _ in range(max_retries): + await asyncio.sleep(1) + response = await client.responses.retrieve(response.id) + if response.status != "queued": + break + print(response) + + assert response.status == "completed" + + +@pytest.mark.asyncio +async def test_background_error(client: openai.AsyncOpenAI): + with pytest.raises( + openai.BadRequestError, match="background can only be used when `store` is true" + ): + _ = await client.responses.create( + input="What is 13 * 24?", + background=True, + store=False, + ) + + +@pytest.mark.asyncio +async def test_background_cancel(client: openai.AsyncOpenAI): + response = await client.responses.create( + input="Write a long story about a cat.", + background=True, + ) + assert response.status == "queued" + + # Cancel the response before it is completed. + # Poll until the response is no longer queued (started processing) or timeout + loop = asyncio.get_running_loop() + start_time = loop.time() + max_wait_seconds = 5.0 + poll_interval = 0.1 + while loop.time() - start_time < max_wait_seconds: + response = await client.responses.retrieve(response.id) + if response.status != "queued": + # Started processing or completed - try to cancel + break + await asyncio.sleep(poll_interval) + + response = await client.responses.cancel(response.id) + assert response.status == "cancelled" + + # Make sure the response status remains unchanged after some time. + max_retries = 10 + for _ in range(max_retries): + await asyncio.sleep(0.5) + response = await client.responses.retrieve(response.id) + # Verify status is still cancelled + assert response.status == "cancelled" + + +@pytest.mark.asyncio +async def test_cancel_completed(client: openai.AsyncOpenAI): + response = await client.responses.create(input="Hello") + assert response.status == "completed" + + with pytest.raises( + openai.BadRequestError, match="Cannot cancel a synchronous response." + ): + await client.responses.cancel(response.id) + + +@pytest.mark.asyncio +async def test_previous_response_id(client: openai.AsyncOpenAI): + response1 = await client.responses.create( + instructions="You are tested on your ability to retrieve the correct " + "information from the previous response.", + input="Hello, my name is John.", + ) + + response2 = await client.responses.create( + input="Actually, my name is not John. My real name is Mark.", + previous_response_id=response1.id, + ) + + response3 = await client.responses.create( + input="What is my real name again? Answer in one word.", + previous_response_id=response2.id, + ) + print(response3) + assert "Mark" in response3.output[-1].content[0].text + assert "John" not in response3.output[-1].content[0].text + + +@pytest.mark.asyncio +async def test_two_responses_with_same_prev_id(client: openai.AsyncOpenAI): + response1 = await client.responses.create( + instructions="You are tested on your ability to retrieve the correct " + "information from the previous response.", + input="Hello, my name is John.", + ) + + # Both response 2 and 3 use response 1 as the previous response. + response2 = client.responses.create( + input="Actually, my name is not John. My name is Mark.", + previous_response_id=response1.id, + ) + response3 = client.responses.create( + input="What is my name again? Answer in one word.", + previous_response_id=response1.id, + ) + + _ = await response2 + response3_result = await response3 + print(response3_result) + assert "John" in response3_result.output[-1].content[0].text + assert "Mark" not in response3_result.output[-1].content[0].text diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_streaming_events.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_streaming_events.py new file mode 100644 index 0000000000000000000000000000000000000000..9aacd0e1465d225ef79a222d37cc58fd5fdac938 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_streaming_events.py @@ -0,0 +1,126 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from vllm.entrypoints.openai.engine.protocol import ( + DeltaFunctionCall, + DeltaMessage, + DeltaToolCall, +) +from vllm.entrypoints.openai.responses.streaming_events import ( + SimpleStreamingEventProcessor, + _StateType, + split_delta, +) + + +def _make_tool_call( + index: int, name: str | None = None, arguments: str | None = None +) -> DeltaToolCall: + fn = DeltaFunctionCall(name=name, arguments=arguments) + return DeltaToolCall(index=index, function=fn) + + +class TestSplitDelta: + def test_all_three_fields(self): + tc = _make_tool_call(0, name="f") + delta = DeltaMessage(reasoning="r", content="c", tool_calls=[tc]) + result = split_delta(delta) + + assert len(result) == 3 + assert result[0].reasoning == "r" and result[0].content is None + assert result[1].content == "c" and result[1].reasoning is None + assert len(result[2].tool_calls) == 1 and result[2].content is None + + def test_tool_calls_grouped_by_index(self): + tc0 = _make_tool_call(0, name="f1") + tc1 = _make_tool_call(1, name="f2") + tc0b = _make_tool_call(0, arguments='{"a":1}') + + # Different indices → split + result = split_delta(DeltaMessage(tool_calls=[tc0, tc1])) + assert len(result) == 2 + assert result[0].tool_calls == [tc0] + assert result[1].tool_calls == [tc1] + + # Same index → stays together + delta = DeltaMessage(tool_calls=[tc0, tc0b]) + result = split_delta(delta) + assert len(result) == 1 + assert result[0] is delta + + +def _run_through_processor( + processor: SimpleStreamingEventProcessor, + delta_message: DeltaMessage, +) -> list: + """Simulate the streaming loop from serving.py for a single delta.""" + events = [] + for dm in split_delta(delta_message): + target_state, tool_call = processor.resolve_target_state(dm) + if target_state == _StateType.NONE: + continue + if processor.needs_transition(target_state, tool_call): + events.extend(processor.close_current()) + events.extend(processor.open(target_state, tool_call)) + events.extend(processor.emit_delta(dm, None)) + return events + + +class TestProcessorCompoundDeltas: + def test_all_three_states(self): + tc = _make_tool_call(0, name="f", arguments="{}") + delta = DeltaMessage(reasoning="r", content="c", tool_calls=[tc]) + + processor = SimpleStreamingEventProcessor() + events = _run_through_processor(processor, delta) + + types = [e.type for e in events] + r_idx = types.index("response.reasoning_text.delta") + c_idx = types.index("response.output_text.delta") + fc_idx = types.index("response.function_call_arguments.delta") + assert r_idx < c_idx < fc_idx + + def test_parallel_tool_calls(self): + tc0 = _make_tool_call(0, name="f1", arguments='{"a":1}') + tc1 = _make_tool_call(1, name="f2", arguments='{"b":2}') + delta = DeltaMessage(tool_calls=[tc0, tc1]) + + processor = SimpleStreamingEventProcessor() + events = _run_through_processor(processor, delta) + + added = [e for e in events if e.type == "response.output_item.added"] + deltas = [ + e for e in events if e.type == "response.function_call_arguments.delta" + ] + assert len(added) == 2 + assert len(deltas) == 2 + + def test_split_name_and_args_same_index(self): + """Regression: parsers like KimiK2 emit name and args as separate + DeltaToolCalls at the same index within one DeltaMessage.""" + tc_name = _make_tool_call(0, name="get_weather") + tc_args = _make_tool_call(0, arguments='{"city":"SF"}') + delta = DeltaMessage(tool_calls=[tc_name, tc_args]) + + processor = SimpleStreamingEventProcessor() + events = _run_through_processor(processor, delta) + + deltas = [ + e for e in events if e.type == "response.function_call_arguments.delta" + ] + assert len(deltas) == 1 + assert deltas[0].delta == '{"city":"SF"}' + + def test_reasoning_to_content_transition(self): + """Regression: the old special case in emit_delta handled this; + now split_delta handles it generically.""" + processor = SimpleStreamingEventProcessor() + _run_through_processor(processor, DeltaMessage(reasoning="think")) + assert processor.state.current_state == _StateType.REASONING + + events = _run_through_processor( + processor, DeltaMessage(reasoning="more", content="answer") + ) + types = [e.type for e in events] + assert "response.reasoning_text.delta" in types + assert "response.output_text.delta" in types diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_structured_output.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_structured_output.py new file mode 100644 index 0000000000000000000000000000000000000000..bb2b622f8cf6ef792593772844f5bdea568e72a1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/responses/test_structured_output.py @@ -0,0 +1,78 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import json + +import openai +import pytest +from pydantic import BaseModel + + +@pytest.mark.asyncio +async def test_structured_output(client: openai.AsyncOpenAI): + response = await client.responses.create( + input=[ + {"role": "system", "content": "Extract the event information."}, + { + "role": "user", + "content": "Alice and Bob are going to a science fair on Friday.", + }, + ], + text={ + "format": { + "type": "json_schema", + "name": "calendar_event", + "schema": { + "type": "object", + "properties": { + "event_name": {"type": "string"}, + "date": {"type": "string"}, + "participants": {"type": "array", "items": {"type": "string"}}, + }, + "required": ["event_name", "date", "participants"], + "additionalProperties": False, + }, + "description": "A calendar event.", + "strict": True, + } + }, + ) + print(response) + + # NOTE: The JSON schema is applied to the output text, not reasoning. + output_text = response.output[-1].content[0].text + event = json.loads(output_text) + + assert event["event_name"].lower() == "science fair" + assert event["date"] == "Friday" + participants = event["participants"] + assert len(participants) == 2 + assert participants[0] == "Alice" + assert participants[1] == "Bob" + + +@pytest.mark.asyncio +async def test_structured_output_with_parse(client: openai.AsyncOpenAI): + class CalendarEvent(BaseModel): + event_name: str + date: str + participants: list[str] + + response = await client.responses.parse( + model=None, + instructions="Extract the event information.", + input="Alice and Bob are going to a science fair on Friday.", + text_format=CalendarEvent, + ) + print(response) + + # The output is successfully parsed. + event = response.output_parsed + assert event is not None + + # The output is correct. + assert event.event_name.lower() == "science fair" + assert event.date == "Friday" + participants = event.participants + assert len(participants) == 2 + assert participants[0] == "Alice" + assert participants[1] == "Bob" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_async_tokenization.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_async_tokenization.py new file mode 100644 index 0000000000000000000000000000000000000000..850eadca221b9b3b29a0009bd996460977215f7e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_async_tokenization.py @@ -0,0 +1,82 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import asyncio +import random +from collections.abc import Callable + +import openai +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer + +MODEL_NAME = "Qwen/Qwen2.5-1.5B-Instruct" + + +@pytest.fixture(scope="module") +def server(): + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-model-len", + "8192", + "--enforce-eager", + "--max-num-seqs", + "128", + "--load-format", + "dummy", + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ids=["completion", "chat"], + argnames=["create_func_gen", "content_body"], + argvalues=[ + (lambda x: x.completions.create, {"prompt": " ".join(["A"] * 10_000)}), + ( + lambda x: x.chat.completions.create, + {"messages": [{"role": "user", "content": " ".join(["A"] * 10_000)}]}, + ), + ], +) +async def test_with_and_without_truncate( + server: RemoteOpenAIServer, + client: openai.AsyncOpenAI, + create_func_gen: Callable, + content_body: dict, +): + create_func = create_func_gen(client) + body = {"model": MODEL_NAME, **content_body, "max_tokens": 10} + + num_requests = 10 + truncate_prompt_tokens = [1000] * (num_requests // 2) + [None] * ( + num_requests - num_requests // 2 + ) + random.shuffle(truncate_prompt_tokens) + + bodies = [ + {**body, "extra_body": {"truncate_prompt_tokens": t}} + for t in truncate_prompt_tokens + ] + + async def get_status_code(**kwargs): + try: + await create_func(**kwargs) + return 200 + except openai.APIStatusError as e: + return e.status_code + + responses = await asyncio.gather(*[get_status_code(**b) for b in bodies]) + assert 500 not in responses diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_chunked_prompt.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_chunked_prompt.py new file mode 100644 index 0000000000000000000000000000000000000000..ddc3ddd8959ddc8a86010e26899fceea39325600 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_chunked_prompt.py @@ -0,0 +1,133 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import openai # use the official client for correctness check +import pytest +import pytest_asyncio + +from ...utils import RemoteOpenAIServer + +# any model with a chat template should work here +MODEL_NAME = "Qwen/Qwen3-0.6B" + + +@pytest.fixture(scope="module") +def server(): + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-model-len", + "8192", + "--enforce-eager", + "--max-num-seqs", + "128", + "--enable-chunked-prefill", + "--max-num-batched-tokens", + "1000", + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +async def test_completion_stream_options_and_logprobs_with_long_prompts( + client: openai.AsyncOpenAI, +): + # Test stream with long prompt + prompt = "What is the capital of France?" * 400 + + stream = await client.completions.create( + model=MODEL_NAME, + prompt=prompt, + max_tokens=5, + temperature=0.0, + stream=True, + stream_options={ + "include_usage": True, + "continuous_usage_stats": True, + }, + logprobs=5, + ) + + tokens_received = 0 + finished = False + async for chunk in stream: + assert chunk.usage.prompt_tokens >= 0 + assert chunk.usage.completion_tokens >= 0 + assert chunk.usage.total_tokens == ( + chunk.usage.prompt_tokens + chunk.usage.completion_tokens + ) + if not finished: + assert chunk.choices[0].text + # Count actual tokens from logprobs since multiple tokens + # can be batched into a single chunk + assert chunk.choices[0].logprobs and chunk.choices[0].logprobs.tokens + tokens_received += len(chunk.choices[0].logprobs.tokens) + + if chunk.choices[0].finish_reason is not None: + finished = True + + if finished: + assert chunk.usage.completion_tokens == tokens_received + + +@pytest.mark.asyncio +async def test_chat_completion_stream_options_and_logprobs_with_long_prompts( + client: openai.AsyncOpenAI, +): + # Test stream with long prompt + messages = [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "What is the capital of France?" * 400}, + ] + stream = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_tokens=5, + temperature=0.0, + stream=True, + stream_options={ + "include_usage": True, + "continuous_usage_stats": True, + }, + logprobs=True, + top_logprobs=5, + ) + + tokens_received = 0 + empty_chunks_received = 0 + finished = False + async for chunk in stream: + assert chunk.usage.prompt_tokens >= 0 + assert chunk.usage.completion_tokens >= 0 + assert chunk.usage.total_tokens == ( + chunk.usage.prompt_tokens + chunk.usage.completion_tokens + ) + + if not finished: + if chunk.choices[0].delta.content == "": + # when there is no tokens generated + assert chunk.usage.completion_tokens == 0 + assert chunk.choices[0].logprobs is None + empty_chunks_received += 1 + else: + # Count actual tokens from logprobs since multiple tokens + # can be batched into a single chunk + assert chunk.choices[0].logprobs and chunk.choices[0].logprobs.content + tokens_received += len(chunk.choices[0].logprobs.content) + + if chunk.choices[0].finish_reason is not None: + finished = True + + if finished: + assert chunk.usage.completion_tokens == tokens_received + + assert empty_chunks_received <= 1 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_cli_args.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_cli_args.py new file mode 100644 index 0000000000000000000000000000000000000000..a0b4ff6c082deb74697d79b06db90d62c6ac4c72 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_cli_args.py @@ -0,0 +1,322 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import pytest + +from vllm.entrypoints.openai.cli_args import make_arg_parser, validate_parsed_serve_args +from vllm.entrypoints.openai.models.protocol import LoRAModulePath +from vllm.utils.argparse_utils import FlexibleArgumentParser + +from ...utils import VLLM_PATH + +LORA_MODULE = { + "name": "module2", + "path": "/path/to/module2", + "base_model_name": "llama", +} +CHATML_JINJA_PATH = VLLM_PATH / "examples/template_chatml.jinja" +assert CHATML_JINJA_PATH.exists() + + +def _build_vllm_parsers(): + vllm_parser = FlexibleArgumentParser() + subparsers = vllm_parser.add_subparsers() + serve_parser = subparsers.add_parser("serve") + make_arg_parser(serve_parser) + return {"vllm": vllm_parser, "vllm serve": serve_parser} + + +@pytest.fixture +def vllm_parser(): + return _build_vllm_parsers()["vllm"] + + +@pytest.fixture +def serve_parser(): + return _build_vllm_parsers()["vllm serve"] + + +### Test config parsing +def test_config_arg_parsing(serve_parser, cli_config_file): + args = serve_parser.parse_args([]) + assert args.port == 8000 + args = serve_parser.parse_args(["--config", cli_config_file]) + assert args.port == 12312 + args = serve_parser.parse_args( + [ + "--config", + cli_config_file, + "--port", + "9000", + ] + ) + assert args.port == 9000 + args = serve_parser.parse_args( + [ + "--port", + "9000", + "--config", + cli_config_file, + ] + ) + assert args.port == 9000 + + +### Tests for LoRA module parsing +def test_valid_key_value_format(serve_parser): + # Test old format: name=path + args = serve_parser.parse_args( + [ + "--lora-modules", + "module1=/path/to/module1", + ] + ) + expected = [LoRAModulePath(name="module1", path="/path/to/module1")] + assert args.lora_modules == expected + + +def test_valid_json_format(serve_parser): + # Test valid JSON format input + args = serve_parser.parse_args( + [ + "--lora-modules", + json.dumps(LORA_MODULE), + ] + ) + expected = [ + LoRAModulePath(name="module2", path="/path/to/module2", base_model_name="llama") + ] + assert args.lora_modules == expected + + +def test_invalid_json_format(serve_parser): + # Test invalid JSON format input, missing closing brace + with pytest.raises(SystemExit): + serve_parser.parse_args( + ["--lora-modules", '{"name": "module3", "path": "/path/to/module3"'] + ) + + +def test_invalid_type_error(serve_parser): + # Test type error when values are not JSON or key=value + with pytest.raises(SystemExit): + serve_parser.parse_args( + [ + "--lora-modules", + "invalid_format", # This is not JSON or key=value format + ] + ) + + +def test_invalid_json_field(serve_parser): + # Test valid JSON format but missing required fields + with pytest.raises(SystemExit): + serve_parser.parse_args( + [ + "--lora-modules", + '{"name": "module4"}', # Missing required 'path' field + ] + ) + + +def test_empty_values(serve_parser): + # Test when no LoRA modules are provided + args = serve_parser.parse_args(["--lora-modules", ""]) + assert args.lora_modules == [] + + +def test_multiple_valid_inputs(serve_parser): + # Test multiple valid inputs (both old and JSON format) + args = serve_parser.parse_args( + [ + "--lora-modules", + "module1=/path/to/module1", + json.dumps(LORA_MODULE), + ] + ) + expected = [ + LoRAModulePath(name="module1", path="/path/to/module1"), + LoRAModulePath( + name="module2", path="/path/to/module2", base_model_name="llama" + ), + ] + assert args.lora_modules == expected + + +### Tests for serve argument validation that run prior to loading +def test_enable_auto_choice_passes_without_tool_call_parser(serve_parser): + """Ensure validation fails if tool choice is enabled with no call parser""" + # If we enable-auto-tool-choice, explode with no tool-call-parser + args = serve_parser.parse_args(args=["--enable-auto-tool-choice"]) + with pytest.raises(TypeError): + validate_parsed_serve_args(args) + + +def test_enable_auto_choice_passes_with_tool_call_parser(serve_parser): + """Ensure validation passes with tool choice enabled with a call parser""" + args = serve_parser.parse_args( + args=[ + "--enable-auto-tool-choice", + "--tool-call-parser", + "mistral", + ] + ) + validate_parsed_serve_args(args) + + +def test_enable_auto_choice_fails_with_enable_reasoning(serve_parser): + """Ensure validation fails if reasoning is enabled with auto tool choice""" + args = serve_parser.parse_args( + args=[ + "--enable-auto-tool-choice", + "--reasoning-parser", + "deepseek_r1", + ] + ) + with pytest.raises(TypeError): + validate_parsed_serve_args(args) + + +def test_passes_with_reasoning_parser(serve_parser): + """Ensure validation passes if reasoning is enabled + with a reasoning parser""" + args = serve_parser.parse_args( + args=[ + "--reasoning-parser", + "deepseek_r1", + ] + ) + validate_parsed_serve_args(args) + + +def test_chat_template_validation_for_happy_paths(serve_parser): + """Ensure validation passes if the chat template exists""" + args = serve_parser.parse_args( + args=["--chat-template", CHATML_JINJA_PATH.absolute().as_posix()] + ) + validate_parsed_serve_args(args) + + +def test_chat_template_validation_for_sad_paths(serve_parser): + """Ensure validation fails if the chat template doesn't exist""" + args = serve_parser.parse_args(args=["--chat-template", "does/not/exist"]) + with pytest.raises(ValueError): + validate_parsed_serve_args(args) + + +@pytest.mark.parametrize( + "cli_args, expected_middleware", + [ + ( + ["--middleware", "middleware1", "--middleware", "middleware2"], + ["middleware1", "middleware2"], + ), + ([], []), + ], +) +def test_middleware(serve_parser, cli_args, expected_middleware): + """Ensure multiple middleware args are parsed properly""" + args = serve_parser.parse_args(args=cli_args) + assert args.middleware == expected_middleware + + +def test_default_chat_template_kwargs_parsing(serve_parser): + """Ensure default_chat_template_kwargs JSON is parsed correctly""" + args = serve_parser.parse_args( + args=["--default-chat-template-kwargs", '{"enable_thinking": false}'] + ) + assert args.default_chat_template_kwargs == {"enable_thinking": False} + + +def test_default_chat_template_kwargs_complex(serve_parser): + """Ensure complex default_chat_template_kwargs JSON is parsed correctly""" + kwargs_json = '{"enable_thinking": false, "custom_param": "value", "num": 42}' + args = serve_parser.parse_args(args=["--default-chat-template-kwargs", kwargs_json]) + assert args.default_chat_template_kwargs == { + "enable_thinking": False, + "custom_param": "value", + "num": 42, + } + + +def test_default_chat_template_kwargs_default_none(serve_parser): + """Ensure default_chat_template_kwargs defaults to None""" + args = serve_parser.parse_args(args=[]) + assert args.default_chat_template_kwargs is None + + +def test_default_chat_template_kwargs_invalid_json(serve_parser): + """Ensure invalid JSON raises an error""" + with pytest.raises(SystemExit): + serve_parser.parse_args( + args=["--default-chat-template-kwargs", "not valid json"] + ) + + +@pytest.mark.parametrize( + "args, raises", + [ + (["user/model"], None), + (["user/model", "--served-model-name", "model"], None), + (["--served-model-name", "model", "user/model"], ValueError), + (["--served-model-name", "model", "--config", "config.yaml"], None), + (["--served-model-name", "model", "--config", "config.yaml"], ValueError), + ], + ids=[ + "model_tag_only", + "model_tag_with_served_model_name", + "served_model_name_before_model_tag", + "served_model_name_with_model_in_config", + "served_model_name_with_no_model_in_config", + ], +) +def test_served_model_name_parsing(tmp_path, vllm_parser, args, raises): + """Ensure that users don't misuse --served-model-name and end up with the default + model tag instead of the one they intended to serve.""" + # Call the serve subparser + args.insert(0, "serve") + # Create a dummy config file if the test case includes it + if "config.yaml" in args: + # Create a dummy config file if the test case includes it + config_path = tmp_path / "config.yaml" + config_path.write_text("model: user/model" if raises is None else "port: 8000") + args[args.index("config.yaml")] = config_path.as_posix() + # Do the parsing and check for expected exceptions or values + if raises is None: + parsed_args = vllm_parser.parse_args(args=args) + expected = "user/model" + assert parsed_args.model_tag == expected or parsed_args.model == expected + else: + with pytest.raises(raises): + vllm_parser.parse_args(args=args) + + +### Tests for LoRA target modules parsing +def test_lora_target_modules_single(serve_parser): + """Test parsing single lora-target-modules argument""" + args = serve_parser.parse_args( + args=["--enable-lora", "--lora-target-modules", "o_proj"] + ) + assert args.lora_target_modules == ["o_proj"] + + +def test_lora_target_modules_multiple(serve_parser): + """Test parsing multiple lora-target-modules arguments""" + args = serve_parser.parse_args( + args=[ + "--enable-lora", + "--lora-target-modules", + "o_proj", + "qkv_proj", + "down_proj", + ] + ) + assert args.lora_target_modules == ["o_proj", "qkv_proj", "down_proj"] + + +def test_lora_target_modules_default_none(serve_parser): + """Test that lora-target-modules defaults to None""" + args = serve_parser.parse_args(args=[]) + assert args.lora_target_modules is None diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_dp_supervisor.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_dp_supervisor.py new file mode 100644 index 0000000000000000000000000000000000000000..6c0bd06893c538d0be61fc430ca284a152f7db8c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_dp_supervisor.py @@ -0,0 +1,760 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +""" +Tests for DPSupervisor: unit tests and lifecycle integration tests. + +Lifecycle integration tests replace child vLLM servers with lightweight +aiohttp "fake" servers controlled by the test, so the suite runs without GPUs. +_start_children is monkeypatched to install FakeProcess objects (with +controllable liveness/timing) alongside those fake HTTP servers. + +Port allocation (kept far from default vLLM ports to avoid conflicts): + Supervisor : 19256 + Children : 18000, 18001 +""" + +from __future__ import annotations + +import argparse +import asyncio +import contextlib +import os +import signal +import subprocess +import tempfile +import time +from pathlib import Path +from types import SimpleNamespace + +import aiohttp +import pytest +import uvicorn +from fastapi import FastAPI, Response + +import vllm.entrypoints.openai.dp_supervisor as dp_sup +from vllm.entrypoints.openai.dp_supervisor import ( + CHILD_EXIT_GRACE_S, + DPSupervisor, + _build_vllm_dp_server_args, + infer_multi_port_external_lb_start_rank, + validate_multi_port_external_lb_args, +) +from vllm.logger import init_logger + +logger = init_logger(__name__) + +_SUPERVISOR_PORT = 19256 +_CHILD_PORT_BASE = 18000 +_N_CHILDREN = 2 +_PROBE_INTERVAL = 1.0 +_POLL_INTERVAL = 1.0 + + +# --------------------------------------------------------------------------- +# Args factories +# --------------------------------------------------------------------------- + + +def _make_unit_args(**overrides) -> argparse.Namespace: + """Minimal args for unit tests (no real network activity).""" + base = { + "host": None, + "port": 8000, + "data_parallel_multi_port_external_lb": True, + "data_parallel_supervisor_port": 9256, + "dp_supervisor_probe_interval_s": 5.0, + "dp_supervisor_probe_timeout_s": 5.0, + "dp_supervisor_probe_failure_threshold": 3, + "data_parallel_size": 8, + "data_parallel_size_local": 4, + "data_parallel_start_rank": None, + "data_parallel_rank": None, + "data_parallel_external_lb": False, + "data_parallel_hybrid_lb": False, + "api_server_count": None, + "headless": False, + "grpc": False, + "uds": None, + "ssl_keyfile": None, + "ssl_certfile": None, + "ssl_ca_certs": None, + "ssl_cert_reqs": 0, + "ssl_ciphers": None, + "node_rank": 1, + "tensor_parallel_size": 1, + "pipeline_parallel_size": 1, + "uvicorn_log_level": "info", + "shutdown_timeout": 5.0, + } + base.update(overrides) + return argparse.Namespace(**base) + + +def _make_args(**overrides) -> argparse.Namespace: + """Args for lifecycle integration tests (real loopback servers).""" + base: dict = dict( + host="127.0.0.1", + port=_CHILD_PORT_BASE, + data_parallel_multi_port_external_lb=True, + data_parallel_supervisor_port=_SUPERVISOR_PORT, + dp_supervisor_probe_interval_s=_PROBE_INTERVAL, + dp_supervisor_probe_timeout_s=1.0, + dp_supervisor_probe_failure_threshold=3, + data_parallel_size=_N_CHILDREN, + data_parallel_size_local=_N_CHILDREN, + data_parallel_start_rank=0, + data_parallel_rank=None, + data_parallel_external_lb=False, + data_parallel_hybrid_lb=False, + api_server_count=None, + headless=False, + grpc=False, + uds=None, + ssl_keyfile=None, + ssl_certfile=None, + ssl_ca_certs=None, + ssl_cert_reqs=0, + ssl_ciphers=None, + node_rank=0, + tensor_parallel_size=1, + pipeline_parallel_size=1, + uvicorn_log_level="warning", + shutdown_timeout=0.0, + ) + base.update(overrides) + return argparse.Namespace(**base) + + +def _generate_self_signed_cert(cert_dir: Path) -> tuple[Path, Path]: + """Generate a self-signed certificate for HTTPS lifecycle tests.""" + cert_file = cert_dir / "cert.pem" + key_file = cert_dir / "key.pem" + subprocess.run( + [ + "openssl", + "req", + "-x509", + "-newkey", + "rsa:2048", + "-keyout", + str(key_file), + "-out", + str(cert_file), + "-days", + "1", + "-nodes", + "-subj", + "/CN=localhost", + ], + check=True, + capture_output=True, + ) + return cert_file, key_file + + +# --------------------------------------------------------------------------- +# Unit tests +# --------------------------------------------------------------------------- + + +def test_infer_multi_port_external_lb_start_rank_uses_node_rank(): + args = _make_unit_args() + assert infer_multi_port_external_lb_start_rank(args) == 4 + + +def test_build_multi_port_external_lb_child_args_sets_external_rank_server(): + args = _make_unit_args(data_parallel_start_rank=8, api_server_count=None) + child_args = _build_vllm_dp_server_args(args, local_rank=2) + + assert child_args.port == 8002 + assert child_args.data_parallel_rank == 10 + assert child_args.data_parallel_size_local == 1 + assert child_args.data_parallel_external_lb is True + assert child_args.data_parallel_hybrid_lb is False + assert child_args.data_parallel_multi_port_external_lb is False + assert child_args.api_server_count == 1 + + +def test_run_vllm_dp_server_uses_python_server_by_default(monkeypatch): + calls: list[str] = [] + + monkeypatch.setattr(dp_sup.os, "setpgrp", lambda: None) + monkeypatch.setattr(dp_sup, "set_process_title", lambda *_args: None) + monkeypatch.setattr(dp_sup, "decorate_logs", lambda *_args: None) + monkeypatch.setattr(dp_sup.envs, "VLLM_RUST_FRONTEND_PATH", None, raising=False) + monkeypatch.setattr( + dp_sup, "_run_python_vllm_dp_server", lambda _args: calls.append("python") + ) + monkeypatch.setattr( + dp_sup, "_run_rust_vllm_dp_server", lambda _args: calls.append("rust") + ) + + dp_sup._run_vllm_dp_server(_make_unit_args(data_parallel_rank=4)) + + assert calls == ["python"] + + +def test_run_vllm_dp_server_uses_rust_frontend_when_enabled(monkeypatch): + calls: list[str] = [] + + monkeypatch.setattr(dp_sup.os, "setpgrp", lambda: None) + monkeypatch.setattr(dp_sup, "set_process_title", lambda *_args: None) + monkeypatch.setattr(dp_sup, "decorate_logs", lambda *_args: None) + monkeypatch.setattr( + dp_sup.envs, + "VLLM_RUST_FRONTEND_PATH", + "/tmp/vllm-rs", + raising=False, + ) + monkeypatch.setattr( + dp_sup, "_run_python_vllm_dp_server", lambda _args: calls.append("python") + ) + monkeypatch.setattr( + dp_sup, "_run_rust_vllm_dp_server", lambda _args: calls.append("rust") + ) + + dp_sup._run_vllm_dp_server(_make_unit_args(data_parallel_rank=4)) + + assert calls == ["rust"] + + +def test_validate_multi_port_external_lb_args_allows_ssl(): + args = _make_unit_args( + ssl_keyfile="/tmp/server.key", + ssl_certfile="/tmp/server.crt", + ssl_ca_certs="/tmp/ca.crt", + ) + validate_multi_port_external_lb_args(args) + + +def test_aggregates_health(): + supervisor = DPSupervisor(_make_unit_args()) + supervisor._is_ready = True + assert supervisor.is_ready is True + + +def test_handles_shutdown_event(): + supervisor = DPSupervisor(_make_unit_args()) + supervisor._is_ready = True + supervisor._shutdown_event.set() + assert supervisor.is_ready is False + + +@pytest.mark.asyncio +async def test_handles_child_exit( + monkeypatch: pytest.MonkeyPatch, +): + supervisor = DPSupervisor(_make_unit_args()) + supervisor._processes = [ + SimpleNamespace( + name="APIServer_DPRank_4", exitcode=None, is_alive=lambda: True + ), + SimpleNamespace(name="APIServer_DPRank_5", exitcode=17, is_alive=lambda: False), + ] + + async def fake_probe(*_args, **_kwargs) -> bool: + return True + + monkeypatch.setattr(dp_sup, "_probe_endpoint", fake_probe) + + await supervisor._monitor_children() + assert supervisor._is_ready is False + + +@pytest.mark.asyncio +async def test_handles_probe_failure( + monkeypatch: pytest.MonkeyPatch, +): + supervisor = DPSupervisor(_make_unit_args(dp_supervisor_probe_interval_s=0.0)) + supervisor.child_ports = [8000] + probe_results = iter([True, False]) + + async def fake_probe(*_args, **_kwargs) -> bool: + return next(probe_results) + + monkeypatch.setattr(dp_sup, "_probe_endpoint", fake_probe) + + await supervisor._monitor_children() + assert supervisor._is_ready is False + + +@pytest.mark.asyncio +async def test_shutdown_if_supervisor_server_error_on_startup( + monkeypatch: pytest.MonkeyPatch, +): + class FakeLoop: + def add_signal_handler(self, *_args, **_kwargs): + pass + + def remove_signal_handler(self, *_args, **_kwargs): + pass + + class FakeServer: + def __init__(self, _config): + self.started = False + self.should_exit = False + + async def serve(self): + raise ValueError("supervisor boom") + + async def fake_shutdown_children(self): + return None + + monkeypatch.setattr(dp_sup.asyncio, "get_running_loop", lambda: FakeLoop()) + monkeypatch.setattr(dp_sup.uvicorn, "Server", FakeServer) + monkeypatch.setattr(DPSupervisor, "_shutdown_children", fake_shutdown_children) + + supervisor = DPSupervisor(_make_unit_args()) + + with pytest.raises(ValueError, match="supervisor boom"): + await supervisor.run() + + +# --------------------------------------------------------------------------- +# Lifecycle integration tests – MockVLLMServer +# --------------------------------------------------------------------------- + + +class MockVLLMServer: + """ + Minimal FastAPI server that mimics one vLLM replica. + GET /health returns 200 when healthy, 503 otherwise. + Health state is toggled by the test via set_healthy(). + """ + + def __init__( + self, + port: int, + drain_seconds: float = 0.0, + ssl_keyfile: str | None = None, + ssl_certfile: str | None = None, + ) -> None: + self.port = port + self._healthy = False + self._drain_seconds = drain_seconds + self._ssl_keyfile = ssl_keyfile + self._ssl_certfile = ssl_certfile + self._server: uvicorn.Server | None = None + self._serve_task: asyncio.Task | None = None + + async def start(self) -> None: + app = FastAPI() + + @app.get("/health") + async def health() -> Response: + print(f"MockServer {self.port}: /health: {self._healthy}") + return Response(status_code=200 if self._healthy else 503) + + @app.get("/set_healthy") + async def set_healthy() -> Response: + print(f"MockServer {self.port}: /set_healthy") + self._healthy = True + return Response(status_code=200) + + @app.get("/set_unhealthy") + async def set_unhealthy() -> Response: + print(f"MockServer {self.port}: /set_unhealthy") + self._healthy = False + return Response(status_code=200) + + @app.get("/kill") + async def kill() -> Response: + print(f"MockServer {self.port}: /kill") + os.kill(os.getpid(), signal.SIGKILL) + + config = uvicorn.Config( + app, + host="127.0.0.1", + port=self.port, + log_level="warning", + lifespan="off", + ssl_keyfile=self._ssl_keyfile, + ssl_certfile=self._ssl_certfile, + ) + self._server = uvicorn.Server(config) + + # Configure request draining if needed. + # Uvicorn's capture_signals() installs signal.signal(SIGTERM, self.handle_exit), + # which sets should_exit=True immediately. Override handle_exit on the instance + # so capture_signals() picks up our version that drains first. + if self._drain_seconds > 0: + self._shutdown_event = asyncio.Event() + loop = asyncio.get_running_loop() + + async def _drain_and_stop() -> None: + await self._shutdown_event.wait() + print(f"MockServer {self.port}: draining for {self._drain_seconds}s.") + await asyncio.sleep(self._drain_seconds) + print("Setting should_exit") + if self._server is not None: + self._server.should_exit = True + + self._drain_task = asyncio.create_task(_drain_and_stop()) + + def _custom_handle_exit(sig: int, frame: object) -> None: + print("Got SIGTERM, setting shutdown.") + if not self._shutdown_event.is_set(): + loop.call_soon_threadsafe(self._shutdown_event.set) + + self._server.handle_exit = _custom_handle_exit + + self._serve_task = asyncio.create_task(self._server.serve()) + while not self._server.started: + await asyncio.sleep(0.01) + print(f"Mock DP Server on port {self.port} started") + + await self._serve_task + + +def launch_mock_vllm(child_args: argparse.Namespace): + logger.info("Launching mock vLLM on port %s", child_args.port) + mock_vllm = MockVLLMServer( + port=child_args.port, + ssl_keyfile=child_args.ssl_keyfile, + ssl_certfile=child_args.ssl_certfile, + ) + asyncio.run(mock_vllm.start()) + + +def launch_mock_vllm_with_drain( + child_args: argparse.Namespace, +): + logger.info("Launching mock vLLM with 15s drain on port %s", child_args.port) + mock_vllm = MockVLLMServer( + port=child_args.port, + drain_seconds=10.0, + ssl_keyfile=child_args.ssl_keyfile, + ssl_certfile=child_args.ssl_certfile, + ) + asyncio.run(mock_vllm.start()) + + +# --------------------------------------------------------------------------- +# Lifecycle test helpers +# --------------------------------------------------------------------------- + + +async def _poll_supervisor_health(expected_status: int, use_ssl: bool = False) -> bool: + """ + Poll GET /health on the supervisor until expected_status is seen. + A connection error is treated as 503-equivalent when expected_status != 200. + """ + scheme = "https" if use_ssl else "http" + url = f"{scheme}://127.0.0.1:{_SUPERVISOR_PORT}/health" + async with aiohttp.ClientSession() as session: + try: + async with session.get(url, ssl=False if use_ssl else None) as resp: + if resp.status != expected_status: + print(f"expected: {expected_status=}, got: {resp.status=}") + return False + return True + except aiohttp.ClientError: + if expected_status != -1: + print(f"expected: {expected_status=}, got: aiohttp.ClientError") + return False + return True + + +async def _poll_until_api_server_running( + port: int, retries: int = 10, use_ssl: bool = False +) -> None: + scheme = "https" if use_ssl else "http" + url = f"{scheme}://127.0.0.1:{port}/health" + async with aiohttp.ClientSession() as session: + for _ in range(retries): + try: + async with session.get(url, ssl=False if use_ssl else None) as resp: + if resp.status != 200: + return + await asyncio.sleep(1.0) + except aiohttp.ClientError: + print("Test detected not started yet, sleeping for 1s") + await asyncio.sleep(1.0) + + +async def _set_healthy(port: int, use_ssl: bool = False) -> None: + scheme = "https" if use_ssl else "http" + url = f"{scheme}://127.0.0.1:{port}/set_healthy" + async with ( + aiohttp.ClientSession() as session, + session.get(url, ssl=False if use_ssl else None) as resp, + ): + assert resp.status == 200 + + +async def _set_unhealthy(port: int, use_ssl: bool = False) -> None: + scheme = "https" if use_ssl else "http" + url = f"{scheme}://127.0.0.1:{port}/set_unhealthy" + async with ( + aiohttp.ClientSession() as session, + session.get(url, ssl=False if use_ssl else None) as resp, + ): + assert resp.status == 200 + + +async def _kill_server(port: int, use_ssl: bool = False) -> None: + scheme = "https" if use_ssl else "http" + url = f"{scheme}://127.0.0.1:{port}/kill" + try: + async with ( + aiohttp.ClientSession() as session, + session.get(url, ssl=False if use_ssl else None) as resp, + ): + assert resp.status != 200 + except Exception as e: + assert isinstance(e, aiohttp.ClientConnectorError) + + +@contextlib.asynccontextmanager +async def _run_supervisor( + args: argparse.Namespace, + monkeypatch: pytest.MonkeyPatch, + launch_fn=None, +): + if launch_fn is None: + launch_fn = launch_mock_vllm + monkeypatch.setattr(dp_sup, "_run_vllm_dp_server", launch_fn) + supervisor = DPSupervisor(args) + task = asyncio.create_task(supervisor.run()) + await asyncio.sleep(1.0) + try: + yield supervisor, task + finally: + task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await task + + +# --------------------------------------------------------------------------- +# Lifecycle integration tests +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_basic_lifecycle(monkeypatch): + """ + A) Supervisor /health returns 503 while children are unhealthy. + B) /health returns 200 once every child reports healthy. + C) SIGTERM and shutdown + """ + args = _make_args() + + vllm_server_ports = [_CHILD_PORT_BASE + i for i in range(_N_CHILDREN)] + + async with _run_supervisor(args, monkeypatch) as (supervisor, _task): + assert await _poll_supervisor_health(503) + assert not supervisor.is_ready + + for port in vllm_server_ports: + assert await _poll_supervisor_health(503) + assert not supervisor.is_ready + await _poll_until_api_server_running(port) + + await _set_healthy(vllm_server_ports[0]) + await asyncio.sleep(1.0) + assert await _poll_supervisor_health(503) + assert not supervisor.is_ready + print("/health is 503 --- expected!") + + for port in vllm_server_ports: + await _set_healthy(port) + await asyncio.sleep(1.0) + assert await _poll_supervisor_health(200) + assert supervisor.is_ready + print("/health is 200 --- expected!") + + await asyncio.sleep(1.0) + assert await _poll_supervisor_health(200) + assert supervisor.is_ready + print("/health is 200 --- expected!") + + os.kill(os.getpid(), signal.SIGTERM) + + await asyncio.wait_for(_task, timeout=5.0) + for p in supervisor._processes: + assert not p.is_alive() + print("everything was cleaned up!") + + +@pytest.mark.asyncio +async def test_basic_lifecycle_with_ssl(monkeypatch): + with tempfile.TemporaryDirectory() as cert_dir: + cert_file, key_file = _generate_self_signed_cert(Path(cert_dir)) + args = _make_args( + ssl_keyfile=str(key_file), + ssl_certfile=str(cert_file), + ) + + vllm_server_ports = [_CHILD_PORT_BASE + i for i in range(_N_CHILDREN)] + + async with _run_supervisor(args, monkeypatch) as (supervisor, _task): + assert await _poll_supervisor_health(503, use_ssl=True) + assert not supervisor.is_ready + + for port in vllm_server_ports: + assert await _poll_supervisor_health(503, use_ssl=True) + assert not supervisor.is_ready + await _poll_until_api_server_running(port, use_ssl=True) + + for port in vllm_server_ports: + await _set_healthy(port, use_ssl=True) + await asyncio.sleep(1.0) + + assert await _poll_supervisor_health(200, use_ssl=True) + assert supervisor.is_ready + + +@pytest.mark.asyncio +async def test_failed_startup(monkeypatch): + """ + A) One of the vLLM servers crashes during startup. + B) DPSupervisor detects this, and cleans up resources. + """ + args = _make_args() + + vllm_server_ports = [_CHILD_PORT_BASE + i for i in range(_N_CHILDREN)] + + async with _run_supervisor(args, monkeypatch) as (supervisor, _task): + assert await _poll_supervisor_health(503) + assert not supervisor.is_ready + + for port in vllm_server_ports: + await _poll_until_api_server_running(port) + + await _kill_server(port) + + await asyncio.wait_for(_task, timeout=5.0) + for p in supervisor._processes: + assert not p.is_alive() + + +@pytest.mark.asyncio +async def test_becomes_unhealthy(monkeypatch): + """ + A) Supervisor /health returns 503 while children are unhealthy. + B) /health returns 200 once every child reports healthy. + C) Child process becomes unhealthy. + D) Detected and shutdown. + """ + args = _make_args() + + vllm_server_ports = [_CHILD_PORT_BASE + i for i in range(_N_CHILDREN)] + + async with _run_supervisor(args, monkeypatch) as (supervisor, _task): + assert await _poll_supervisor_health(503) + assert not supervisor.is_ready + + for port in vllm_server_ports: + assert await _poll_supervisor_health(503) + assert not supervisor.is_ready + await _poll_until_api_server_running(port) + + await _set_healthy(vllm_server_ports[0]) + await asyncio.sleep(1.0) + assert await _poll_supervisor_health(503) + assert not supervisor.is_ready + print("/health is 503 --- expected!") + + for port in vllm_server_ports: + await _set_healthy(port) + await asyncio.sleep(1.0) + assert await _poll_supervisor_health(200) + assert supervisor.is_ready + print("/health is 200 --- expected!") + + await _set_unhealthy(port) + + await asyncio.wait_for(_task, timeout=5.0) + for p in supervisor._processes: + assert not p.is_alive() + print("everything was cleaned up!") + + +@pytest.mark.asyncio +async def test_dp_server_fails(monkeypatch): + """ + A) Supervisor /health returns 503 while children are unhealthy. + B) /health returns 200 once every child reports healthy. + C) Child process fails. + D) Detected and shutdown. + """ + args = _make_args() + + vllm_server_ports = [_CHILD_PORT_BASE + i for i in range(_N_CHILDREN)] + + async with _run_supervisor(args, monkeypatch) as (supervisor, _task): + assert await _poll_supervisor_health(503) + assert not supervisor.is_ready + + for port in vllm_server_ports: + assert await _poll_supervisor_health(503) + assert not supervisor.is_ready + await _poll_until_api_server_running(port) + + await _set_healthy(vllm_server_ports[0]) + await asyncio.sleep(1.0) + assert await _poll_supervisor_health(503) + assert not supervisor.is_ready + print("/health is 503 --- expected!") + + for port in vllm_server_ports: + await _set_healthy(port) + await asyncio.sleep(1.0) + assert await _poll_supervisor_health(200) + assert supervisor.is_ready + print("/health is 200 --- expected!") + + dp_mock_server_process = supervisor._processes[0] + os.kill(dp_mock_server_process.pid, signal.SIGKILL) + await asyncio.sleep(1.0) + assert not dp_mock_server_process.is_alive() + + await asyncio.wait_for(_task, timeout=5.0) + for p in supervisor._processes: + assert not p.is_alive() + print("everything was cleaned up!") + + +@pytest.mark.asyncio +async def test_shutdown_timeout(monkeypatch: pytest.MonkeyPatch): + """ + Child mock servers delay shutdown by 10s on SIGTERM (simulating in-flight + request drain). The supervisor is configured with shutdown_timeout=10, + so its total wait budget is 10 + CHILD_EXIT_GRACE_S seconds. The + children exit naturally within that window, so no force-kill should occur + and the measured wall-clock time must be >= 10s. + """ + _DRAIN_SECONDS = 10.0 + _SHUTDOWN_TIMEOUT = 10.0 + + args = _make_args(shutdown_timeout=_SHUTDOWN_TIMEOUT) + vllm_server_ports = [_CHILD_PORT_BASE + i for i in range(_N_CHILDREN)] + + async with _run_supervisor( + args, monkeypatch, launch_fn=launch_mock_vllm_with_drain + ) as (supervisor, _task): + for port in vllm_server_ports: + await _poll_until_api_server_running(port) + + for port in vllm_server_ports: + await _set_healthy(port) + await asyncio.sleep(1.0) + assert await _poll_supervisor_health(200) + assert supervisor.is_ready + + start_t = time.perf_counter() + os.kill(os.getpid(), signal.SIGTERM) + + print(f"DRAINING FOR {_DRAIN_SECONDS}") + await asyncio.wait_for(_task, timeout=_DRAIN_SECONDS + CHILD_EXIT_GRACE_S + 5.0) + elapsed = time.perf_counter() - start_t + + assert elapsed >= _DRAIN_SECONDS, ( + f"Supervisor exited after only {elapsed:.1f}s; " + f"expected >= {_DRAIN_SECONDS}s for request draining" + ) + + for p in supervisor._processes: + assert not p.is_alive() + print(f"Supervisor waited {elapsed:.1f}s for children to drain — expected!") diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_multi_api_servers.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_multi_api_servers.py new file mode 100644 index 0000000000000000000000000000000000000000..f1a725ae2dbd2f7559ed2e33472d1379face80f0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_multi_api_servers.py @@ -0,0 +1,175 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import asyncio +import os + +import openai # use the official client for correctness check +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer +from tests.v1.utils import check_request_balancing + +MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM" + +DP_SIZE = os.getenv("DP_SIZE", "1") + + +@pytest.fixture(scope="module") +def default_server_args(): + return [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-model-len", + "2048", + "--max-num-seqs", + "128", + "--enforce-eager", + "--api-server-count", + "4", + "--data_parallel_size", + DP_SIZE, + ] + + +@pytest.fixture(scope="module") +def server(default_server_args): + with RemoteOpenAIServer(MODEL_NAME, default_server_args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_single_completion( + client: openai.AsyncOpenAI, server: RemoteOpenAIServer, model_name: str +) -> None: + async def make_request(): + completion = await client.completions.create( + model=model_name, prompt="Hello, my name is", max_tokens=10, temperature=1.0 + ) + + assert completion.id is not None + assert completion.choices is not None and len(completion.choices) == 1 + + choice = completion.choices[0] + # The exact number of tokens can vary slightly with temperature=1.0, + # so we check for a reasonable minimum length. + assert len(choice.text) >= 1 + # Finish reason might not always be 'length' if the model finishes early + # or due to other reasons, especially with high temperature. + # So, we'll accept 'length' or 'stop'. + assert choice.finish_reason in ("length", "stop") + + # Token counts can also vary, so we check they are positive. + assert completion.usage.completion_tokens > 0 + assert completion.usage.prompt_tokens > 0 + assert completion.usage.total_tokens > 0 + return completion + + # Test single request + result = await make_request() + assert result is not None + + await asyncio.sleep(0.5) + + # Send two bursts of requests + num_requests = 100 + tasks = [make_request() for _ in range(num_requests)] + results = await asyncio.gather(*tasks) + assert len(results) == num_requests + assert all(completion is not None for completion in results) + + await asyncio.sleep(0.5) + + tasks = [make_request() for _ in range(num_requests)] + results = await asyncio.gather(*tasks) + assert len(results) == num_requests + assert all(completion is not None for completion in results) + + # Check request balancing via Prometheus metrics if DP_SIZE > 1 + check_request_balancing(server, int(DP_SIZE)) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", + [MODEL_NAME], +) +async def test_completion_streaming( + client: openai.AsyncOpenAI, server: RemoteOpenAIServer, model_name: str +) -> None: + prompt = "What is an LLM?" + + async def make_streaming_request(): + # Perform a non-streaming request to get the expected full output + single_completion = await client.completions.create( + model=model_name, + prompt=prompt, + max_tokens=5, + temperature=0.0, + ) + single_output = single_completion.choices[0].text + + # Perform the streaming request + stream = await client.completions.create( + model=model_name, prompt=prompt, max_tokens=5, temperature=0.0, stream=True + ) + chunks: list[str] = [] + finish_reason_count = 0 + last_chunk = None + async for chunk in stream: + chunks.append(chunk.choices[0].text) + if chunk.choices[0].finish_reason is not None: + finish_reason_count += 1 + last_chunk = chunk # Keep track of the last chunk + + # finish reason should only return in the last block for OpenAI API + assert finish_reason_count == 1, "Finish reason should appear exactly once." + assert last_chunk is not None, "Stream should have yielded at least one chunk." + assert last_chunk.choices[0].finish_reason == "length", ( + "Finish reason should be 'length'." + ) + # Check that the combined text matches the non-streamed version. + assert "".join(chunks) == single_output, ( + "Streamed output should match non-streamed output." + ) + return True # Indicate success for this request + + # Test single request + result = await make_streaming_request() + assert result is not None + + await asyncio.sleep(0.5) + + # Send two bursts of requests + num_requests = 100 + tasks = [make_streaming_request() for _ in range(num_requests)] + results = await asyncio.gather(*tasks) + + assert len(results) == num_requests, ( + f"Expected {num_requests} results, got {len(results)}" + ) + assert all(results), "Not all streaming requests completed successfully." + + await asyncio.sleep(0.5) + + tasks = [make_streaming_request() for _ in range(num_requests)] + results = await asyncio.gather(*tasks) + + assert len(results) == num_requests, ( + f"Expected {num_requests} results, got {len(results)}" + ) + assert all(results), "Not all streaming requests completed successfully." + + # Check request balancing via Prometheus metrics if DP_SIZE > 1 + check_request_balancing(server, int(DP_SIZE)) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_openai_schema.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_openai_schema.py new file mode 100644 index 0000000000000000000000000000000000000000..39a2c09cd90615e3f460c6581e081e8cfa68b0da --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_openai_schema.py @@ -0,0 +1,167 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import json +from typing import Final + +import pytest +import schemathesis +from hypothesis import HealthCheck, settings +from schemathesis import GenerationMode +from schemathesis.config import ( + ChecksConfig, + CoveragePhaseConfig, + GenerationConfig, + PhasesConfig, + PositiveDataAcceptanceConfig, + ProjectConfig, + ProjectsConfig, + SchemathesisConfig, +) + +from vllm.platforms import current_platform + +from ...utils import RemoteOpenAIServer + +MODEL_NAME = "HuggingFaceTB/SmolVLM-256M-Instruct" +MAXIMUM_IMAGES = 2 +_ROCM_TIMEOUT_MULTIPLIER = 3 if current_platform.is_rocm() else 1 +DEFAULT_TIMEOUT_SECONDS: Final[int] = 10 * _ROCM_TIMEOUT_MULTIPLIER +LONG_TIMEOUT_SECONDS: Final[int] = 60 * _ROCM_TIMEOUT_MULTIPLIER + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--runner", + "generate", + "--max-model-len", + "2048", + "--max-num-seqs", + "5", + "--enforce-eager", + "--trust-remote-code", + "--limit-mm-per-prompt", + json.dumps({"image": MAXIMUM_IMAGES}), + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.fixture(scope="module") +def get_schema(server): + # avoid generating null (\x00) bytes in strings during test case generation + return schemathesis.openapi.from_url( + f"{server.url_root}/openapi.json", + config=SchemathesisConfig( + projects=ProjectsConfig( + default=ProjectConfig( + generation=GenerationConfig( + allow_x00=False, + modes=[GenerationMode.POSITIVE], + ), + checks=ChecksConfig( + positive_data_acceptance=PositiveDataAcceptanceConfig( + enabled=False, + ), + ), + phases=PhasesConfig( + coverage=CoveragePhaseConfig(enabled=False), + ), + ), + ), + ), + ) + + +schema = schemathesis.pytest.from_fixture("get_schema") + + +@schemathesis.hook +def before_generate_case(context: schemathesis.HookContext, strategy): + op = context.operation + assert op is not None + + def no_invalid_types(case: schemathesis.Case): + """ + Skips tool_calls with `"type": "custom"` which schemathesis incorrectly + generates instead of the valid `"type": "function"`. + + Example test case that is skipped: + curl -X POST -H 'Content-Type: application/json' \ + -d '{"messages": [{"role": "assistant", "tool_calls": [{"custom": {"input": "", "name": ""}, "id": "", "type": "custom"}]}]}' \ + http://localhost:8000/v1/chat/completions + """ # noqa: E501 + if ( + hasattr(case, "body") + and isinstance(case.body, dict) + and "messages" in case.body + and isinstance(case.body["messages"], list) + and len(case.body["messages"]) > 0 + ): + for message in case.body["messages"]: + if not isinstance(message, dict): + continue + + tool_calls = message.get("tool_calls", []) + if isinstance(tool_calls, list): + for tool_call in tool_calls: + if isinstance(tool_call, dict): + if tool_call.get("type") != "function": + return False + if "custom" in tool_call: + return False + + return True + + return strategy.filter(no_invalid_types) + + +@schema.parametrize() +@settings( + deadline=LONG_TIMEOUT_SECONDS * 1000, + max_examples=50, + # Under CI's derandomized hypothesis seed, the schemathesis strategy + # for /v1/chat/completions/batch's nested-message body, combined with + # the no_invalid_types filter (notably the grammar=="" rule), exceeds + # the default filtered-vs-good ratio. The filter is intentional, so + # suppress the health check rather than drop the filter — dropping it + # exposes pre-existing server bugs out of scope here. + # The same nested schema can also trip Hypothesis' entropy budget while + # generating large-but-valid request bodies before vLLM is called. + suppress_health_check=[HealthCheck.filter_too_much, HealthCheck.data_too_large], +) +def test_openapi_stateless(case: schemathesis.Case): + key = ( + case.operation.method.upper(), + case.operation.path, + ) + if case.operation.path.startswith("/v1/responses"): + # Skip responses API as it is meant to be stateful. + return + + # Skip weight transfer endpoints as they require special setup + # (weight_transfer_config) and are meant to be stateful. + if case.operation.path in ( + "/init_weight_transfer_engine", + "/start_weight_update", + "/update_weights", + "/finish_weight_update", + ): + return + + timeout = { + # requires a longer timeout + ("POST", "/v1/chat/completions"): LONG_TIMEOUT_SECONDS, + ("POST", "/v1/chat/completions/batch"): LONG_TIMEOUT_SECONDS, + ("POST", "/v1/completions"): LONG_TIMEOUT_SECONDS, + ("POST", "/v1/messages"): LONG_TIMEOUT_SECONDS, + ("POST", "/inference/v1/generate"): LONG_TIMEOUT_SECONDS, + }.get(key, DEFAULT_TIMEOUT_SECONDS) + + # No need to verify SSL certificate for localhost + case.call_and_validate( + verify=False, + timeout=timeout, + headers={"Content-Type": "application/json"}, + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_reasoning_enable_thinking.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_reasoning_enable_thinking.py new file mode 100644 index 0000000000000000000000000000000000000000..c4904b989b01ccd9bb0d136be64ecb502ca85d24 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_reasoning_enable_thinking.py @@ -0,0 +1,107 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for reasoning_effort -> enable_thinking mapping. + +Models like Gemma4 require enable_thinking=True in chat_template_kwargs to +activate thinking mode. This mapping ensures that when a user requests +reasoning (via reasoning_effort or reasoning.effort), the template kwarg +is injected automatically. +""" + +import pytest +from openai.types.shared import Reasoning + +from vllm.entrypoints.openai.chat_completion.protocol import ( + ChatCompletionRequest, +) +from vllm.entrypoints.openai.responses.protocol import ResponsesRequest + + +def _build_chat_request(**kwargs) -> ChatCompletionRequest: + defaults = dict( + model="test-model", + messages=[{"role": "user", "content": "Hello"}], + ) + defaults.update(kwargs) + return ChatCompletionRequest(**defaults) + + +def _build_responses_request(**kwargs) -> ResponsesRequest: + defaults = dict( + model="test-model", + input=[{"role": "user", "content": "Hello"}], + ) + defaults.update(kwargs) + return ResponsesRequest(**defaults) + + +class TestChatCompletionReasoningEffort: + """Chat Completions: reasoning_effort -> enable_thinking.""" + + @pytest.mark.parametrize("effort", ["low", "medium", "high"]) + def test_non_none_effort_injects_enable_thinking_true(self, effort): + request = _build_chat_request(reasoning_effort=effort) + params = request.build_chat_params(None, "auto") + assert params.chat_template_kwargs["enable_thinking"] is True + + def test_none_effort_injects_enable_thinking_false(self): + request = _build_chat_request(reasoning_effort="none") + params = request.build_chat_params(None, "auto") + assert params.chat_template_kwargs["enable_thinking"] is False + + def test_no_effort_does_not_inject(self): + request = _build_chat_request() + params = request.build_chat_params(None, "auto") + assert "enable_thinking" not in params.chat_template_kwargs + + def test_explicit_user_kwarg_not_overridden(self): + request = _build_chat_request( + reasoning_effort="high", + chat_template_kwargs={"enable_thinking": False}, + ) + params = request.build_chat_params(None, "auto") + assert params.chat_template_kwargs["enable_thinking"] is False + + def test_reasoning_effort_still_in_kwargs(self): + request = _build_chat_request(reasoning_effort="high") + params = request.build_chat_params(None, "auto") + assert params.chat_template_kwargs["reasoning_effort"] == "high" + + +class TestResponsesReasoningEffort: + """Responses API: reasoning.effort -> enable_thinking.""" + + @pytest.mark.parametrize("effort", ["low", "medium", "high"]) + def test_non_none_effort_injects_enable_thinking_true(self, effort): + request = _build_responses_request( + reasoning=Reasoning(effort=effort), + ) + params = request.build_chat_params(None, "auto") + assert params.chat_template_kwargs["enable_thinking"] is True + + def test_none_effort_injects_enable_thinking_false(self): + request = _build_responses_request( + reasoning=Reasoning(effort="none"), + ) + params = request.build_chat_params(None, "auto") + assert params.chat_template_kwargs["enable_thinking"] is False + + def test_no_reasoning_does_not_inject(self): + request = _build_responses_request() + params = request.build_chat_params(None, "auto") + assert "enable_thinking" not in params.chat_template_kwargs + + def test_explicit_user_kwarg_not_overridden(self): + request = _build_responses_request( + reasoning=Reasoning(effort="high"), + chat_template_kwargs={"enable_thinking": False}, + ) + params = request.build_chat_params(None, "auto") + assert params.chat_template_kwargs["enable_thinking"] is False + + def test_reasoning_effort_still_in_kwargs(self): + request = _build_responses_request( + reasoning=Reasoning(effort="high"), + ) + params = request.build_chat_params(None, "auto") + assert params.chat_template_kwargs["reasoning_effort"] == "high" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_render_token_offsets.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_render_token_offsets.py new file mode 100644 index 0000000000000000000000000000000000000000..e1598291d3e77f6d44cb0c1d98660654d38e5c3b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_render_token_offsets.py @@ -0,0 +1,80 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for the token-offsets request/response protocol wiring: +the request flag flowing into ``TokenizeParams`` and the ``GenerateRequest`` +serialization boundary. End-to-end behavior is covered by +``tests/entrypoints/scale_out/render/test_render.py``; plain Pydantic field +storage is not retested here. +""" + +from unittest.mock import Mock + +from vllm.config import ModelConfig +from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest +from vllm.entrypoints.openai.completion.protocol import CompletionRequest +from vllm.entrypoints.scale_out.token_in_token_out.protocol import GenerateRequest +from vllm.sampling_params import SamplingParams + + +def _model_config() -> Mock: + model_config = Mock(spec=ModelConfig) + model_config.max_model_len = 128 + return model_config + + +def test_completion_flag_forwarded_to_tok_params(): + """build_tok_params must forward return_token_offsets, defaulting to + False (zero behavioral change for existing callers) and coercing JSON + null to False via the bool() guard.""" + cfg = _model_config() + + default = CompletionRequest(model="m", prompt="hi") + assert default.build_tok_params(cfg).return_token_offsets is False + + on = CompletionRequest(model="m", prompt="hi", return_token_offsets=True) + assert on.build_tok_params(cfg).return_token_offsets is True + + null = CompletionRequest(model="m", prompt="hi", return_token_offsets=None) + assert null.build_tok_params(cfg).return_token_offsets is False + + +def test_chat_flag_forwarded_to_tok_params(): + """Chat build_tok_params has its own (max_completion_tokens) branch, so + its return_token_offsets forwarding is verified independently.""" + cfg = _model_config() + messages = [{"role": "user", "content": "hi"}] + + default = ChatCompletionRequest(model="m", messages=messages) + assert default.build_tok_params(cfg).return_token_offsets is False + + on = ChatCompletionRequest(model="m", messages=messages, return_token_offsets=True) + assert on.build_tok_params(cfg).return_token_offsets is True + + null = ChatCompletionRequest( + model="m", messages=messages, return_token_offsets=None + ) + assert null.build_tok_params(cfg).return_token_offsets is False + + +def test_generate_request_token_offsets_default_none(): + """Defaults to None so existing /v1/.../render responses are unchanged.""" + req = GenerateRequest(token_ids=[1, 2, 3], sampling_params=SamplingParams()) + assert req.token_offsets is None + + +def test_generate_request_token_offsets_survive_json_round_trip(): + """GenerateRequest crosses the disagg serialization boundary; the + tuple[int, int] offsets must survive model_dump and re-validate.""" + req = GenerateRequest( + token_ids=[10, 20], + sampling_params=SamplingParams(), + token_offsets=[(0, 1), (1, 3)], + ) + dumped = req.model_dump() + assert dumped["token_offsets"] == [(0, 1), (1, 3)] + # Re-validate from the dumped dict (sampling_params doesn't round-trip + # cleanly via dump, so re-inject a fresh instance). + again = GenerateRequest.model_validate( + {**dumped, "sampling_params": SamplingParams()} + ) + assert again.token_offsets == [(0, 1), (1, 3)] diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_return_routed_experts.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_return_routed_experts.py new file mode 100644 index 0000000000000000000000000000000000000000..4b9c61e938f3cc120e587760a635910b5d7fc2da --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_return_routed_experts.py @@ -0,0 +1,64 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import io + +import numpy as np +import pybase64 as base64 +import pytest + +from ...utils import RemoteOpenAIServer + +MODEL_NAME = "TitanML/tiny-mixtral" + +# tiny-mixtral config: 8 local experts, top-2 routing, 2 hidden layers. +# The published config has sliding_window=4096, which produces +# SlidingWindowSpec kv-cache groups; RoutedExpertsManager requires a +# FullAttentionSpec group, so we override sliding_window=null below. +NUM_LOCAL_EXPERTS = 8 +NUM_EXPERTS_PER_TOK = 2 +NUM_HIDDEN_LAYERS = 2 + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--max-model-len", + "256", + "--max-num-seqs", + "32", + "--enforce-eager", + "--enable-return-routed-experts", + "--hf-overrides", + '{"sliding_window": null}', + ] + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.mark.asyncio +async def test_routed_experts(server): + """Test that /v1/completions returns routed_experts when enabled.""" + async with server.get_async_client() as client: + result = await client.completions.create( + model=MODEL_NAME, + prompt="Hello, world", + max_tokens=10, + temperature=0, + extra_body={"return_token_ids": True}, + ) + + choice = result.model_dump()["choices"][0] + + assert choice["routed_experts"] is not None + assert choice["token_ids"] is not None + + # routed_experts is base64-encoded .npy bytes; decode to ndarray. + routed_experts = np.load(io.BytesIO(base64.b64decode(choice["routed_experts"]))) + assert routed_experts.ndim == 3 + num_tokens, num_layers, topk = routed_experts.shape + assert num_tokens > 0 + assert num_layers == NUM_HIDDEN_LAYERS + assert topk == NUM_EXPERTS_PER_TOK + assert (routed_experts >= 0).all() + assert (routed_experts < NUM_LOCAL_EXPERTS).all() diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_return_token_ids.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_return_token_ids.py new file mode 100644 index 0000000000000000000000000000000000000000..4b3169e7c29f27ff31cec373fdb989cff1af1ffe --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_return_token_ids.py @@ -0,0 +1,369 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from vllm.tokenizers import get_tokenizer + +from ...utils import RemoteOpenAIServer + +MODEL_NAME = "Qwen/Qwen2.5-1.5B-Instruct" + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--max-model-len", + "2048", + "--max-num-seqs", + "128", + "--enable-auto-tool-choice", + "--tool-call-parser", + "hermes", + "--enforce-eager", + ] + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.mark.asyncio +@pytest.mark.parametrize("return_token_ids", [True, False, None]) +async def test_basic_completion_with_emoji(server, return_token_ids: bool | None): + """Test basic completion with emoji to verify token_ids field.""" + extra_body = None + if return_token_ids is not None: + extra_body = {"return_token_ids": return_token_ids} + async with server.get_async_client() as client: + # Test with return_token_ids enabled + completion = await client.completions.create( + model=MODEL_NAME, + prompt="Complete this sentence with emojis: I love coding 🚀", + max_tokens=10, + temperature=0, + logprobs=1, + extra_body=extra_body, + ) + + # Check the raw response to see the structure + completion_dict = completion.model_dump() + + # Verify prompt_token_ids field is present in the completion response + assert "prompt_token_ids" in completion_dict["choices"][0] + if not return_token_ids: + # If return_token_ids is False, token_ids should not be present + assert completion_dict["choices"][0].get("token_ids") is None + assert completion_dict["choices"][0].get("prompt_token_ids") is None + # Skip further checks + return + assert isinstance(completion.choices[0].prompt_token_ids, list) + + # Check against the expected prompt token IDs + tokenizer = get_tokenizer(tokenizer_name=MODEL_NAME) + encoded_tokens = tokenizer.encode( + "Complete this sentence with emojis: I love coding 🚀" + ) + # Check that encoded_tokens is a subsequence of prompt_token_ids + assert any( + completion.choices[0].prompt_token_ids[i : i + len(encoded_tokens)] + == encoded_tokens + for i in range( + len(completion.choices[0].prompt_token_ids) - len(encoded_tokens) + 1 + ) + ) + + # Verify token_ids field is present in the choice + assert completion.choices[0].token_ids is not None + assert isinstance(completion.choices[0].token_ids, list) + assert len(completion.choices[0].token_ids) > 0 + + # Verify decoding works correctly + decoded_text = tokenizer.decode(completion.choices[0].token_ids) + # The decoded text should contain a <|im_end|> at the end + assert decoded_text.startswith(completion.choices[0].text) + + # Test without return_token_ids (should be None) + completion_without = await client.completions.create( + model=MODEL_NAME, + prompt="Complete this sentence with emojis: I love coding 🚀", + max_tokens=10, + temperature=0, + logprobs=1, + extra_body={"return_token_ids": False}, + ) + + completion_without_dict = completion_without.model_dump() + assert completion_without_dict["choices"][0].get("token_ids") is None + assert completion_without_dict.get("prompt_token_ids") is None + + +@pytest.mark.asyncio +async def test_chat_completion_with_tool_use(server): + """Test chat completion with tool use (get_weather function).""" + tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather in a given location", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA", + }, + "unit": { + "type": "string", + "enum": ["celsius", "fahrenheit"], + "description": "The unit of temperature", + }, + }, + "required": ["location"], + }, + }, + } + ] + + async with server.get_async_client() as client: + # Test with return_token_ids enabled + response = await client.chat.completions.create( + model=MODEL_NAME, + messages=[ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "What's the weather like in Paris?"}, + ], + tools=tools, + tool_choice="auto", + max_tokens=100, + temperature=0, + logprobs=True, + extra_body={"return_token_ids": True}, + ) + + # Verify token_ids field is present in choices + assert response.choices[0].token_ids is not None + assert isinstance(response.choices[0].token_ids, list) + + # Verify prompt_token_ids field is present + assert response.prompt_token_ids is not None + assert isinstance(response.prompt_token_ids, list) + + # Verify the prompt texts and response texts + tokenizer = get_tokenizer(tokenizer_name=MODEL_NAME) + prompt_text = tokenizer.decode(response.prompt_token_ids) + assert prompt_text.startswith( + "<|im_start|>system\nYou are a helpful assistant." + ) + assert prompt_text.endswith( + "What's the weather like in Paris?<|im_end|>\n<|im_start|>assistant\n" + ) + + response_text = tokenizer.decode(response.choices[0].token_ids) + assert response_text.startswith('\n{"name": "get_weather"') + assert response_text.endswith("<|im_end|>") + + # If tool call was made, verify the response structure + if response.choices[0].message.tool_calls: + assert len(response.choices[0].message.tool_calls) > 0 + tool_call = response.choices[0].message.tool_calls[0] + assert tool_call.function.name == "get_weather" + + # Test without return_token_ids + response_without = await client.chat.completions.create( + model=MODEL_NAME, + messages=[ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "What's the weather like in Paris?"}, + ], + tools=tools, + tool_choice="auto", + max_tokens=100, + temperature=0, + logprobs=True, + extra_body={"return_token_ids": False}, + ) + + assert response_without.choices[0].token_ids is None + assert response_without.prompt_token_ids is None + + +@pytest.mark.asyncio +async def test_comparison_with_prompt_logprobs_and_logprobs(server): + """ + Test that token_ids align with prompt_logprobs and + logprobs when return_tokens_as_token_ids is enabled. + """ + async with server.get_async_client() as client: + # Test with both return_token_ids and return_tokens_as_token_ids enabled + completion = await client.completions.create( + model=MODEL_NAME, + prompt="Hello, world! How are you today?", + max_tokens=20, + temperature=0, + echo=True, + logprobs=1, + extra_body={ + "return_token_ids": True, + "return_tokens_as_token_ids": True, + "prompt_logprobs": 1, + }, + ) + + # Verify all fields are present + assert completion.choices[0].token_ids is not None + assert completion.choices[0].prompt_token_ids is not None + assert completion.choices[0].prompt_logprobs is not None + assert completion.choices[0].logprobs is not None + + # Extract token IDs from logprobs + # (when return_tokens_as_token_ids is True) + logprobs_token_ids = [] + for token_str in completion.choices[0].logprobs.tokens: + # Token format is "token_id:12345" when + # return_tokens_as_token_ids is True + if token_str.startswith("token_id:"): + token_id = int(token_str.removeprefix("token_id:")) + logprobs_token_ids.append(token_id) + + # When echo=True, the logprobs include both prompt and response tokens + # The token_ids field should match the suffix of response portion + # The prompt_token_ids should match the prompt portion + assert len(completion.choices[0].token_ids) < len(logprobs_token_ids) + response_token_ids_length = len(completion.choices[0].token_ids) + assert ( + logprobs_token_ids[-response_token_ids_length:] + == completion.choices[0].token_ids + ) + + # Verify tokenizer consistency + tokenizer = get_tokenizer(tokenizer_name=MODEL_NAME) + + # Decode prompt tokens + if completion.choices[0].prompt_token_ids: + prompt_text = tokenizer.decode(completion.choices[0].prompt_token_ids) + # The decoded prompt should match or close to original prompt + assert "Hello, world" in prompt_text + + # Decode response tokens + if completion.choices[0].token_ids: + response_text = tokenizer.decode(completion.choices[0].token_ids) + assert completion.choices[0].text.endswith(response_text) + + # Test streaming mode + stream = await client.completions.create( + model=MODEL_NAME, + prompt="Tell me a short fact about Python:", + max_tokens=30, + temperature=0, + stream=True, + echo=False, + logprobs=1, + extra_body={"return_token_ids": True, "return_tokens_as_token_ids": True}, + ) + + # Collect streamed tokens + streamed_prompt_token_ids = [] + streamed_token_ids = [] + streamed_logprob_token_ids = [] + first_chunk = True + async for chunk in stream: + for token_str in chunk.choices[0].logprobs.tokens: + # Token format is "token_id:12345" when + # return_tokens_as_token_ids is True + if token_str.startswith("token_id:"): + token_id = int(token_str.removeprefix("token_id:")) + streamed_logprob_token_ids.append(token_id) + if first_chunk: + streamed_prompt_token_ids = chunk.choices[0].prompt_token_ids + first_chunk = False + streamed_token_ids += chunk.choices[0].token_ids + + # Verify we collected some tokens and first chunk had prompt_token_ids + assert len(streamed_prompt_token_ids) > 0 + assert streamed_token_ids == streamed_logprob_token_ids + + +@pytest.mark.asyncio +async def test_chat_completion_with_emoji_and_token_ids(server): + """Test chat completion with emojis to verify token_ids handling.""" + chat_messages = [ + {"role": "system", "content": "You like to use emojis in your responses."}, + {"role": "user", "content": "Repeat after me: I love cats 🐱"}, + ] + async with server.get_async_client() as client: + response = await client.chat.completions.create( + model=MODEL_NAME, + messages=chat_messages, + max_tokens=50, + temperature=0, + logprobs=True, + extra_body={"return_token_ids": True}, + ) + + # Verify token_ids are present + response_dict = response.model_dump() + assert response.choices[0].token_ids is not None + assert "prompt_token_ids" in response_dict + + # Verify the response contains the expected fields + assert response.choices[0].message.content is not None + + # Decode token_ids and verify consistency + tokenizer = get_tokenizer(tokenizer_name=MODEL_NAME) + + decoded_prompt = tokenizer.decode(response.prompt_token_ids) + assert decoded_prompt.startswith( + "<|im_start|>system\nYou like to use emojis in your responses." + ) + assert decoded_prompt.endswith( + "I love cats 🐱<|im_end|>\n<|im_start|>assistant\n" + ) + + decoded_response = tokenizer.decode(response.choices[0].token_ids) + # The content should match the response text + # except the ending <|im_end|> + assert decoded_response == response.choices[0].message.content + "<|im_end|>" + + # Test with streaming + stream = await client.chat.completions.create( + model=MODEL_NAME, + messages=chat_messages, + max_tokens=50, + temperature=0, + stream=True, + extra_body={"return_token_ids": True}, + ) + + collected_content = "" + collected_token_ids = [] + first_chunk = True + + async for chunk in stream: + if first_chunk: + assert chunk.prompt_token_ids is not None + assert isinstance(chunk.prompt_token_ids, list) + # Check the prompt_token_ids match the initial prompt + decoded_prompt_stream = tokenizer.decode(chunk.prompt_token_ids) + assert decoded_prompt_stream == decoded_prompt + first_chunk = False + else: + chunk_dump = chunk.model_dump() + assert "prompt_token_ids" not in chunk_dump, ( + "Subsequent chunks should not have prompt_token_ids" + ) + + if chunk.choices: + if chunk.choices[0].delta.content: + collected_content += chunk.choices[0].delta.content + # token_ids may not present in all chunks + choice_dump = chunk.choices[0].model_dump() + if "token_ids" in choice_dump: + collected_token_ids.extend(chunk.choices[0].token_ids) + + # Verify we got response and token_ids + assert len(collected_content) > 0 + assert len(collected_token_ids) > 0 + + # Verify token_ids decode properly + decoded_response = tokenizer.decode(collected_token_ids) + assert decoded_response == collected_content + "<|im_end|>" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_return_tokens_as_ids.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_return_tokens_as_ids.py new file mode 100644 index 0000000000000000000000000000000000000000..bdbfb5c180114c279678c68bd79fb43c35f26966 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_return_tokens_as_ids.py @@ -0,0 +1,162 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +# Separate these tests out from test_completion and test_chat, because they +# require launching a second server with a different flag. Running both servers +# at the same time on a single node will OOM. + +import pytest + +from vllm.tokenizers import get_tokenizer + +from ...utils import RemoteOpenAIServer + +MODEL_NAME = "Qwen/Qwen3-0.6B" + + +@pytest.fixture(scope="module") +def default_server_args(qwen3_lora_files): + return [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-model-len", + "8192", + "--max-num-seqs", + "128", + "--enforce-eager", + # lora config + "--enable-lora", + "--lora-modules", + f"qwen3-lora={qwen3_lora_files}", + "--max-lora-rank", + "64", + "--max-cpu-loras", + "2", + ] + + +@pytest.fixture(scope="module") +def server_fixture(request, default_server_args): + use_server_flag = request.param + if use_server_flag: + args_with_flag = default_server_args + ["--return-tokens-as-token-ids"] + with RemoteOpenAIServer(MODEL_NAME, args_with_flag) as remote_server: + yield (remote_server, True) + else: + with RemoteOpenAIServer(MODEL_NAME, default_server_args) as remote_server: + yield (remote_server, False) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("server_fixture", [True, False], indirect=True) +async def test_completion_return_tokens_as_token_ids_completion(server_fixture): + server, use_server_flag = server_fixture + request_args = {} + if not use_server_flag: + request_args["return_tokens_as_token_ids"] = True + + async with server.get_async_client() as client: + completion = await client.completions.create( + model=MODEL_NAME, + # Include Unicode characters to test for dividing a single + # character across multiple tokens: 🎉 is [28705, 31862] for the + # Zephyr tokenizer + prompt="Say 'Hello, world! 🎉'", + echo=True, + temperature=0, + max_tokens=10, + logprobs=1, + extra_body=request_args, + ) + + text = completion.choices[0].text + token_strs = completion.choices[0].logprobs.tokens + tokenizer = get_tokenizer(tokenizer_name=MODEL_NAME) + # Check that the token representations are consistent between raw + # tokens and top_logprobs + # Slice off the first one, because there's no scoring associated + # with BOS + top_logprobs = completion.choices[0].logprobs.top_logprobs[1:] + top_logprob_keys = [ + next(iter(logprob_by_tokens)) for logprob_by_tokens in top_logprobs + ] + assert token_strs[1:] == top_logprob_keys + + # Check that decoding the tokens gives the expected text + tokens = [int(token.removeprefix("token_id:")) for token in token_strs] + assert text == tokenizer.decode(tokens, skip_special_tokens=True) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("server_fixture", [True, False], indirect=True) +async def test_chat_return_tokens_as_token_ids_completion(server_fixture): + server, use_server_flag = server_fixture + request_args = {} + if not use_server_flag: + request_args["return_tokens_as_token_ids"] = True + + async with server.get_async_client() as client: + response = await client.chat.completions.create( + model=MODEL_NAME, + # Include Unicode characters to test for dividing a single + # character across multiple tokens: 🎉 is [28705, 31862] for the + # Zephyr tokenizer + messages=[ + { + "role": "system", + "content": "You like to respond in only emojis, like 🎉", + }, + {"role": "user", "content": "Please write some emojis: 🐱🐶🎉"}, + ], + temperature=0, + max_tokens=8, + logprobs=True, + extra_body=request_args, + ) + + text = response.choices[0].message.content + tokenizer = get_tokenizer(tokenizer_name=MODEL_NAME) + token_ids = [] + for logprob_content in response.choices[0].logprobs.content: + token_ids.append(int(logprob_content.token.removeprefix("token_id:"))) + assert tokenizer.decode(token_ids, skip_special_tokens=True) == text + + +def test_responses_api_logprobs_with_return_tokens_as_token_ids(): + """Test that return_tokens_as_token_ids works in Responses API logprobs.""" + from unittest.mock import MagicMock + + from vllm.entrypoints.generate.base.serving import GenerateBaseServing + from vllm.entrypoints.openai.responses.serving import OpenAIServingResponses + from vllm.logprobs import Logprob as SampleLogprob + + serving = MagicMock(spec=OpenAIServingResponses) + serving.return_tokens_as_token_ids = True + serving._get_decoded_token = GenerateBaseServing._get_decoded_token + + tokenizer = MagicMock() + tokenizer.decode = lambda token_id: "decoded" + + token_ids = [100, 200, 300] + sample_logprobs = [ + {100: SampleLogprob(logprob=-0.5, decoded_token="hello")}, + {200: SampleLogprob(logprob=-1.2, decoded_token="world")}, + {300: SampleLogprob(logprob=-0.8, decoded_token="!")}, + ] + + result = OpenAIServingResponses._create_response_logprobs( + serving, + token_ids=token_ids, + logprobs=sample_logprobs, + tokenizer=tokenizer, + top_logprobs=1, + ) + + assert len(result) == 3 + assert result[0].token == "token_id:100" + assert result[1].token == "token_id:200" + assert result[2].token == "token_id:300" + assert result[0].logprob == -0.5 + assert result[1].logprob == -1.2 + assert result[2].logprob == -0.8 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_run_batch.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_run_batch.py new file mode 100644 index 0000000000000000000000000000000000000000..8c99ffe7da81c856c1e1a589a3838d41d485528d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_run_batch.py @@ -0,0 +1,881 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json +import subprocess +import tempfile +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from vllm.assets.audio import AudioAsset +from vllm.entrypoints.openai.run_batch import ( + BatchRequestOutput, + download_bytes_from_url, +) + +CHAT_MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM" +EMBEDDING_MODEL_NAME = "intfloat/multilingual-e5-small" +RERANKER_MODEL_NAME = "BAAI/bge-reranker-v2-m3" +REASONING_MODEL_NAME = "Qwen/Qwen3-0.6B" +SPEECH_LARGE_MODEL_NAME = "openai/whisper-large-v3" +SPEECH_SMALL_MODEL_NAME = "openai/whisper-small" + +INPUT_BATCH = "\n".join( + json.dumps(req) + for req in [ + { + "custom_id": "request-1", + "method": "POST", + "url": "/v1/chat/completions", + "body": { + "model": CHAT_MODEL_NAME, + "messages": [ + { + "role": "system", + "content": "You are a helpful assistant.", + }, + {"role": "user", "content": "Hello world!"}, + ], + "max_tokens": 1000, + }, + }, + { + "custom_id": "request-2", + "method": "POST", + "url": "/v1/chat/completions", + "body": { + "model": CHAT_MODEL_NAME, + "messages": [ + { + "role": "system", + "content": "You are an unhelpful assistant.", + }, + {"role": "user", "content": "Hello world!"}, + ], + "max_tokens": 1000, + }, + }, + { + "custom_id": "request-3", + "method": "POST", + "url": "/v1/chat/completions", + "body": { + "model": "NonExistModel", + "messages": [ + { + "role": "system", + "content": "You are an unhelpful assistant.", + }, + {"role": "user", "content": "Hello world!"}, + ], + "max_tokens": 1000, + }, + }, + { + "custom_id": "request-4", + "method": "POST", + "url": "/bad_url", + "body": { + "model": CHAT_MODEL_NAME, + "messages": [ + { + "role": "system", + "content": "You are an unhelpful assistant.", + }, + {"role": "user", "content": "Hello world!"}, + ], + "max_tokens": 1000, + }, + }, + { + "custom_id": "request-5", + "method": "POST", + "url": "/v1/chat/completions", + "body": { + "stream": "True", + "model": CHAT_MODEL_NAME, + "messages": [ + { + "role": "system", + "content": "You are an unhelpful assistant.", + }, + {"role": "user", "content": "Hello world!"}, + ], + "max_tokens": 1000, + }, + }, + ] +) + +INVALID_INPUT_BATCH = "\n".join( + json.dumps(req) + for req in [ + { + "invalid_field": "request-1", + "method": "POST", + "url": "/v1/chat/completions", + "body": { + "model": CHAT_MODEL_NAME, + "messages": [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Hello world!"}, + ], + "max_tokens": 1000, + }, + }, + { + "custom_id": "request-2", + "method": "POST", + "url": "/v1/chat/completions", + "body": { + "model": CHAT_MODEL_NAME, + "messages": [ + {"role": "system", "content": "You are an unhelpful assistant."}, + {"role": "user", "content": "Hello world!"}, + ], + "max_tokens": 1000, + }, + }, + ] +) + +INPUT_EMBEDDING_BATCH = "\n".join( + json.dumps(req) + for req in [ + { + "custom_id": "request-1", + "method": "POST", + "url": "/v1/embeddings", + "body": { + "model": EMBEDDING_MODEL_NAME, + "input": "You are a helpful assistant.", + }, + }, + { + "custom_id": "request-2", + "method": "POST", + "url": "/v1/embeddings", + "body": { + "model": EMBEDDING_MODEL_NAME, + "input": "You are an unhelpful assistant.", + }, + }, + { + "custom_id": "request-3", + "method": "POST", + "url": "/v1/embeddings", + "body": { + "model": EMBEDDING_MODEL_NAME, + "input": "Hello world!", + }, + }, + { + "custom_id": "request-4", + "method": "POST", + "url": "/v1/embeddings", + "body": { + "model": "NonExistModel", + "input": "Hello world!", + }, + }, + ] +) + +_SCORE_RERANK_DOCUMENTS = [ + "The capital of Brazil is Brasilia.", + "The capital of France is Paris.", +] + +INPUT_SCORE_BATCH = "\n".join( + json.dumps(req) + for req in [ + { + "custom_id": "request-1", + "method": "POST", + "url": "/score", + "body": { + "model": RERANKER_MODEL_NAME, + "queries": "What is the capital of France?", + "documents": _SCORE_RERANK_DOCUMENTS, + }, + }, + { + "custom_id": "request-2", + "method": "POST", + "url": "/v1/score", + "body": { + "model": RERANKER_MODEL_NAME, + "queries": "What is the capital of France?", + "documents": _SCORE_RERANK_DOCUMENTS, + }, + }, + ] +) + +INPUT_RERANK_BATCH = "\n".join( + json.dumps(req) + for req in [ + { + "custom_id": "request-1", + "method": "POST", + "url": "/rerank", + "body": { + "model": RERANKER_MODEL_NAME, + "query": "What is the capital of France?", + "documents": _SCORE_RERANK_DOCUMENTS, + }, + }, + { + "custom_id": "request-2", + "method": "POST", + "url": "/v1/rerank", + "body": { + "model": RERANKER_MODEL_NAME, + "query": "What is the capital of France?", + "documents": _SCORE_RERANK_DOCUMENTS, + }, + }, + { + "custom_id": "request-2", + "method": "POST", + "url": "/v2/rerank", + "body": { + "model": RERANKER_MODEL_NAME, + "query": "What is the capital of France?", + "documents": _SCORE_RERANK_DOCUMENTS, + }, + }, + ] +) + +INPUT_REASONING_BATCH = "\n".join( + json.dumps(req) + for req in [ + { + "custom_id": "request-1", + "method": "POST", + "url": "/v1/chat/completions", + "body": { + "model": REASONING_MODEL_NAME, + "messages": [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Solve this math problem: 2+2=?"}, + ], + }, + }, + { + "custom_id": "request-2", + "method": "POST", + "url": "/v1/chat/completions", + "body": { + "model": REASONING_MODEL_NAME, + "messages": [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "What is the capital of France?"}, + ], + }, + }, + ] +) + +MINIMAL_WAV_BASE64 = "UklGRigAAABXQVZFZm10IBAAAAABAAEAgD4AAAB9AAACABAAZGF0YQQAAAAAAP9/" +INPUT_TRANSCRIPTION_BATCH = ( + json.dumps( + { + "custom_id": "request-1", + "method": "POST", + "url": "/v1/audio/transcriptions", + "body": { + "model": SPEECH_LARGE_MODEL_NAME, + "file_url": f"data:audio/wav;base64,{MINIMAL_WAV_BASE64}", + "response_format": "json", + }, + } + ) + + "\n" +) + +INPUT_TRANSCRIPTION_HTTP_BATCH = ( + json.dumps( + { + "custom_id": "request-1", + "method": "POST", + "url": "/v1/audio/transcriptions", + "body": { + "model": SPEECH_LARGE_MODEL_NAME, + "file_url": AudioAsset("mary_had_lamb").url, + "language": "en", + "response_format": "json", + }, + } + ) + + "\n" +) + +INPUT_TRANSLATION_BATCH = ( + json.dumps( + { + "custom_id": "request-1", + "method": "POST", + "url": "/v1/audio/translations", + "body": { + "model": SPEECH_SMALL_MODEL_NAME, + "file_url": AudioAsset("mary_had_lamb").url, + "response_format": "text", + "language": "it", + "to_language": "en", + "temperature": 0.0, + }, + } + ) + + "\n" +) + +WEATHER_TOOL = { + "type": "function", + "function": { + "name": "get_current_weather", + "description": "Get the current weather in a given location", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA", + }, + "unit": { + "type": "string", + "enum": ["celsius", "fahrenheit"], + }, + }, + "required": ["location"], + }, + }, +} + +INPUT_TOOL_CALLING_BATCH = json.dumps( + { + "custom_id": "request-1", + "method": "POST", + "url": "/v1/chat/completions", + "body": { + "model": REASONING_MODEL_NAME, + "messages": [ + {"role": "user", "content": "What is the weather in San Francisco?"}, + ], + "tools": [WEATHER_TOOL], + "tool_choice": "required", + "max_tokens": 1000, + }, + } +) + + +def test_empty_file(): + with ( + tempfile.NamedTemporaryFile("w") as input_file, + tempfile.NamedTemporaryFile("r") as output_file, + ): + input_file.write("") + input_file.flush() + proc = subprocess.Popen( + [ + "vllm", + "run-batch", + "-i", + input_file.name, + "-o", + output_file.name, + "--model", + EMBEDDING_MODEL_NAME, + ], + ) + proc.communicate() + proc.wait() + assert proc.returncode == 0, f"{proc=}" + + contents = output_file.read() + assert contents.strip() == "" + + +def test_completions(): + with ( + tempfile.NamedTemporaryFile("w") as input_file, + tempfile.NamedTemporaryFile("r") as output_file, + ): + input_file.write(INPUT_BATCH) + input_file.flush() + proc = subprocess.Popen( + [ + "vllm", + "run-batch", + "-i", + input_file.name, + "-o", + output_file.name, + "--model", + CHAT_MODEL_NAME, + ], + ) + proc.communicate() + proc.wait() + assert proc.returncode == 0, f"{proc=}" + + contents = output_file.read() + for line in contents.strip().split("\n"): + # Ensure that the output format conforms to the openai api. + # Validation should throw if the schema is wrong. + BatchRequestOutput.model_validate_json(line) + + +def test_completions_invalid_input(): + """ + Ensure that we fail when the input doesn't conform to the openai api. + """ + with ( + tempfile.NamedTemporaryFile("w") as input_file, + tempfile.NamedTemporaryFile("r") as output_file, + ): + input_file.write(INVALID_INPUT_BATCH) + input_file.flush() + proc = subprocess.Popen( + [ + "vllm", + "run-batch", + "-i", + input_file.name, + "-o", + output_file.name, + "--model", + CHAT_MODEL_NAME, + ], + ) + proc.communicate() + proc.wait() + assert proc.returncode != 0, f"{proc=}" + + +def test_embeddings(): + with ( + tempfile.NamedTemporaryFile("w") as input_file, + tempfile.NamedTemporaryFile("r") as output_file, + ): + input_file.write(INPUT_EMBEDDING_BATCH) + input_file.flush() + proc = subprocess.Popen( + [ + "vllm", + "run-batch", + "-i", + input_file.name, + "-o", + output_file.name, + "--model", + EMBEDDING_MODEL_NAME, + ], + ) + proc.communicate() + proc.wait() + assert proc.returncode == 0, f"{proc=}" + + contents = output_file.read() + for line in contents.strip().split("\n"): + # Ensure that the output format conforms to the openai api. + # Validation should throw if the schema is wrong. + BatchRequestOutput.model_validate_json(line) + + +@pytest.mark.parametrize("input_batch", [INPUT_SCORE_BATCH, INPUT_RERANK_BATCH]) +def test_score(input_batch): + with ( + tempfile.NamedTemporaryFile("w") as input_file, + tempfile.NamedTemporaryFile("r") as output_file, + ): + input_file.write(input_batch) + input_file.flush() + proc = subprocess.Popen( + [ + "vllm", + "run-batch", + "-i", + input_file.name, + "-o", + output_file.name, + "--model", + RERANKER_MODEL_NAME, + ], + ) + proc.communicate() + proc.wait() + assert proc.returncode == 0, f"{proc=}" + + contents = output_file.read() + for line in contents.strip().split("\n"): + # Ensure that the output format conforms to the openai api. + # Validation should throw if the schema is wrong. + BatchRequestOutput.model_validate_json(line) + + # Ensure that there is no error in the response. + line_dict = json.loads(line) + assert isinstance(line_dict, dict) + assert line_dict["error"] is None + + +def test_reasoning_parser(): + """ + Test that reasoning_parser parameter works correctly in run_batch. + """ + with ( + tempfile.NamedTemporaryFile("w") as input_file, + tempfile.NamedTemporaryFile("r") as output_file, + ): + input_file.write(INPUT_REASONING_BATCH) + input_file.flush() + proc = subprocess.Popen( + [ + "vllm", + "run-batch", + "-i", + input_file.name, + "-o", + output_file.name, + "--model", + REASONING_MODEL_NAME, + "--reasoning-parser", + "qwen3", + ], + ) + proc.communicate() + proc.wait() + assert proc.returncode == 0, f"{proc=}" + + contents = output_file.read() + for line in contents.strip().split("\n"): + # Ensure that the output format conforms to the openai api. + # Validation should throw if the schema is wrong. + BatchRequestOutput.model_validate_json(line) + + # Ensure that there is no error in the response. + line_dict = json.loads(line) + assert isinstance(line_dict, dict) + assert line_dict["error"] is None + + # Check that reasoning is present and not empty + reasoning = line_dict["response"]["body"]["choices"][0]["message"][ + "reasoning" + ] + assert reasoning is not None + assert len(reasoning) > 0 + + +def test_transcription(): + with ( + tempfile.NamedTemporaryFile("w") as input_file, + tempfile.NamedTemporaryFile("r") as output_file, + ): + input_file.write(INPUT_TRANSCRIPTION_BATCH) + input_file.flush() + proc = subprocess.Popen( + [ + "vllm", + "run-batch", + "-i", + input_file.name, + "-o", + output_file.name, + "--model", + SPEECH_LARGE_MODEL_NAME, + ], + ) + proc.communicate() + proc.wait() + assert proc.returncode == 0, f"{proc=}" + + contents = output_file.read() + print(f"\n\ncontents: {contents}\n\n") + for line in contents.strip().split("\n"): + BatchRequestOutput.model_validate_json(line) + + line_dict = json.loads(line) + assert isinstance(line_dict, dict) + assert line_dict["error"] is None + + response_body = line_dict["response"]["body"] + assert response_body is not None + assert "text" in response_body + assert "usage" in response_body + + +def test_transcription_http_url(): + with ( + tempfile.NamedTemporaryFile("w") as input_file, + tempfile.NamedTemporaryFile("r") as output_file, + ): + input_file.write(INPUT_TRANSCRIPTION_HTTP_BATCH) + input_file.flush() + proc = subprocess.Popen( + [ + "vllm", + "run-batch", + "-i", + input_file.name, + "-o", + output_file.name, + "--model", + SPEECH_LARGE_MODEL_NAME, + ], + ) + proc.communicate() + proc.wait() + assert proc.returncode == 0, f"{proc=}" + + contents = output_file.read() + for line in contents.strip().split("\n"): + BatchRequestOutput.model_validate_json(line) + + line_dict = json.loads(line) + assert isinstance(line_dict, dict) + assert line_dict["error"] is None + + response_body = line_dict["response"]["body"] + assert response_body is not None + assert "text" in response_body + assert "usage" in response_body + + transcription_text = response_body["text"] + assert "Mary had a little lamb" in transcription_text + + +def test_translation(): + with ( + tempfile.NamedTemporaryFile("w") as input_file, + tempfile.NamedTemporaryFile("r") as output_file, + ): + input_file.write(INPUT_TRANSLATION_BATCH) + input_file.flush() + proc = subprocess.Popen( + [ + "vllm", + "run-batch", + "-i", + input_file.name, + "-o", + output_file.name, + "--model", + SPEECH_SMALL_MODEL_NAME, + ], + ) + proc.communicate() + proc.wait() + assert proc.returncode == 0, f"{proc=}" + + contents = output_file.read() + for line in contents.strip().split("\n"): + BatchRequestOutput.model_validate_json(line) + + line_dict = json.loads(line) + assert isinstance(line_dict, dict) + assert line_dict["error"] is None + + response_body = line_dict["response"]["body"] + assert response_body is not None + assert "text" in response_body + + translation_text = response_body["text"] + translation_text_lower = str(translation_text).strip().lower() + assert "mary" in translation_text_lower or "lamb" in translation_text_lower + + +def test_tool_calling(): + """ + Test that tool calling works correctly in run_batch. + Verifies that requests with tools return tool_calls in the response. + """ + with ( + tempfile.NamedTemporaryFile("w") as input_file, + tempfile.NamedTemporaryFile("r") as output_file, + ): + input_file.write(INPUT_TOOL_CALLING_BATCH) + input_file.flush() + proc = subprocess.Popen( + [ + "vllm", + "run-batch", + "-i", + input_file.name, + "-o", + output_file.name, + "--model", + REASONING_MODEL_NAME, + "--enable-auto-tool-choice", + "--tool-call-parser", + "hermes", + ], + ) + proc.communicate() + proc.wait() + assert proc.returncode == 0, f"{proc=}" + + contents = output_file.read() + for line in contents.strip().split("\n"): + if not line.strip(): # Skip empty lines + continue + # Ensure that the output format conforms to the openai api. + # Validation should throw if the schema is wrong. + BatchRequestOutput.model_validate_json(line) + + # Ensure that there is no error in the response. + line_dict = json.loads(line) + assert isinstance(line_dict, dict) + assert line_dict["error"] is None + + # Check that tool_calls are present in the response + # With tool_choice="required", the model must call a tool + response_body = line_dict["response"]["body"] + assert response_body is not None + message = response_body["choices"][0]["message"] + assert "tool_calls" in message + tool_calls = message.get("tool_calls") + # With tool_choice="required", tool_calls must be present and non-empty + assert tool_calls is not None + assert isinstance(tool_calls, list) + assert len(tool_calls) > 0 + # Verify tool_calls have the expected structure + for tool_call in tool_calls: + assert "id" in tool_call + assert "type" in tool_call + assert tool_call["type"] == "function" + assert "function" in tool_call + assert "name" in tool_call["function"] + assert "arguments" in tool_call["function"] + # Verify the tool name matches our tool definition + assert tool_call["function"]["name"] == "get_current_weather" + + +# --------------------------------------------------------------------------- +# Unit tests for download_bytes_from_url SSRF protection +# --------------------------------------------------------------------------- + + +def _make_aiohttp_mocks(response_data: bytes = b"fake-data", status: int = 200): + """Create mock objects that simulate aiohttp.ClientSession context managers.""" + mock_resp = MagicMock() + mock_resp.status = status + mock_resp.read = AsyncMock(return_value=response_data) + mock_resp.__aenter__ = AsyncMock(return_value=mock_resp) + mock_resp.__aexit__ = AsyncMock(return_value=False) + + mock_session = MagicMock() + mock_session.get = MagicMock(return_value=mock_resp) + mock_session.__aenter__ = AsyncMock(return_value=mock_session) + mock_session.__aexit__ = AsyncMock(return_value=False) + return mock_session + + +@pytest.mark.asyncio +async def test_download_bytes_data_url_bypasses_domain_check(): + """data: URLs must work regardless of the domain allowlist.""" + data_url = f"data:audio/wav;base64,{MINIMAL_WAV_BASE64}" + result = await download_bytes_from_url( + data_url, allowed_media_domains=["example.com"] + ) + assert isinstance(result, bytes) + assert len(result) > 0 + + +@pytest.mark.asyncio +async def test_download_bytes_rejects_disallowed_domain(): + """HTTP URLs whose hostname is not in the allowlist must be rejected.""" + url = "https://evil.internal/secret" + with pytest.raises(ValueError, match="allowed domains"): + await download_bytes_from_url(url, allowed_media_domains=["example.com"]) + + +@pytest.mark.asyncio +async def test_download_bytes_rejects_cloud_metadata_ip(): + """Cloud metadata endpoints must be blocked when an allowlist is set.""" + url = "http://169.254.169.254/latest/meta-data/" + with pytest.raises(ValueError, match="allowed domains"): + await download_bytes_from_url(url, allowed_media_domains=["example.com"]) + + +@pytest.mark.asyncio +async def test_download_bytes_rejects_internal_ip(): + """Private-range IPs must be blocked when an allowlist is set.""" + for internal_url in [ + "http://10.0.0.1/secret", + "http://192.168.1.1/admin", + "http://127.0.0.1:8080/internal", + ]: + with pytest.raises(ValueError, match="allowed domains"): + await download_bytes_from_url( + internal_url, allowed_media_domains=["example.com"] + ) + + +@pytest.mark.asyncio +async def test_download_bytes_allows_permitted_domain(): + """HTTP URLs whose hostname IS in the allowlist must be fetched.""" + url = "https://example.com/audio.wav" + expected = b"audio-bytes" + mock_session = _make_aiohttp_mocks(expected) + + with patch( + "vllm.entrypoints.openai.run_batch.aiohttp.ClientSession", + return_value=mock_session, + ): + result = await download_bytes_from_url( + url, allowed_media_domains=["example.com"] + ) + assert result == expected + + +@pytest.mark.asyncio +async def test_download_bytes_no_allowlist_permits_any_domain(): + """Without an allowlist all HTTP URLs must be attempted (backward compat).""" + url = "https://any-domain.example.org/file.wav" + expected = b"some-data" + mock_session = _make_aiohttp_mocks(expected) + + with patch( + "vllm.entrypoints.openai.run_batch.aiohttp.ClientSession", + return_value=mock_session, + ): + result = await download_bytes_from_url(url, allowed_media_domains=None) + assert result == expected + + +@pytest.mark.asyncio +async def test_download_bytes_empty_allowlist_denies_all(): + """An empty allowlist must deny all HTTP URLs (least privilege).""" + url = "https://any-domain.example.org/file.wav" + with pytest.raises(ValueError, match="allowed domains"): + await download_bytes_from_url(url, allowed_media_domains=[]) + + +@pytest.mark.asyncio +async def test_download_bytes_unsupported_scheme(): + """Unsupported URL schemes must be rejected regardless of allowlist.""" + with pytest.raises(ValueError, match="Unsupported URL scheme"): + await download_bytes_from_url("ftp://example.com/file.wav") + + with pytest.raises(ValueError, match="Unsupported URL scheme"): + await download_bytes_from_url( + "ftp://example.com/file.wav", + allowed_media_domains=["example.com"], + ) + + +@pytest.mark.asyncio +async def test_download_bytes_backslash_bypass(): + """Backslash-@ URL confusion must not bypass the allowlist. + + urllib3.parse_url() and aiohttp/yarl disagree on backslash-before-@. + The fix normalizes through urllib3 before handing to aiohttp. + """ + bypass_url = "http://allowed.example.com\\@evil.internal/secret" + with pytest.raises(ValueError, match="allowed domains"): + await download_bytes_from_url( + bypass_url, allowed_media_domains=["evil.internal"] + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_stop_token_ids.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_stop_token_ids.py new file mode 100644 index 0000000000000000000000000000000000000000..3957956eba7c4408aaa1478e1cf376cda4a122da --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_stop_token_ids.py @@ -0,0 +1,160 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +""" +Unit tests for stop_token_ids propagation from default_sampling_params +to SamplingParams in ChatCompletionRequest and CompletionRequest. + +Regression test for https://github.com/vllm-project/vllm/issues/22519 +where gpt-oss model stop tokens (e.g., = 200012) were loaded into +default_sampling_params at server startup but silently discarded on every +request because to_sampling_params() never fell back to defaults. +""" + +import pytest + +from vllm.entrypoints.openai.chat_completion.protocol import ( + ChatCompletionRequest, +) +from vllm.entrypoints.openai.completion.protocol import ( + CompletionRequest, +) + + +class TestChatCompletionStopTokenIds: + """Test stop_token_ids merging in ChatCompletionRequest.to_sampling_params().""" + + @pytest.fixture + def minimal_chat_request(self): + return ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "hello"}], + ) + + def test_default_stop_token_ids_applied(self, minimal_chat_request): + """Server-default stop_token_ids are applied when client sends none.""" + default_sampling_params = { + "stop_token_ids": [200012, 200002], + } + + sampling_params = minimal_chat_request.to_sampling_params( + max_tokens=100, + default_sampling_params=default_sampling_params, + ) + + assert set(sampling_params.stop_token_ids) == {200012, 200002} + + def test_client_stop_token_ids_merged_with_defaults(self): + """Client-specified stop_token_ids are merged with server defaults.""" + request = ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "hello"}], + stop_token_ids=[99999], + ) + default_sampling_params = { + "stop_token_ids": [200012, 200002], + } + + sampling_params = request.to_sampling_params( + max_tokens=100, + default_sampling_params=default_sampling_params, + ) + + assert set(sampling_params.stop_token_ids) == {200012, 200002, 99999} + assert sampling_params.stop_token_ids == [99999, 200012, 200002] + + def test_no_stop_token_ids_anywhere(self, minimal_chat_request): + """When neither client nor server specifies stop_token_ids, result is empty.""" + sampling_params = minimal_chat_request.to_sampling_params( + max_tokens=100, + default_sampling_params={}, + ) + + assert not sampling_params.stop_token_ids + + def test_only_client_stop_token_ids(self): + """Client stop_token_ids work when no server defaults exist.""" + request = ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "hello"}], + stop_token_ids=[42, 43], + ) + + sampling_params = request.to_sampling_params( + max_tokens=100, + default_sampling_params={}, + ) + + assert set(sampling_params.stop_token_ids) == {42, 43} + + def test_duplicate_stop_token_ids_deduplicated(self): + """Overlapping stop_token_ids between client and server are deduplicated.""" + request = ChatCompletionRequest( + model="test-model", + messages=[{"role": "user", "content": "hello"}], + stop_token_ids=[200012, 55555], + ) + default_sampling_params = { + "stop_token_ids": [200012, 200002], + } + + sampling_params = request.to_sampling_params( + max_tokens=100, + default_sampling_params=default_sampling_params, + ) + + assert set(sampling_params.stop_token_ids) == {200012, 200002, 55555} + assert sampling_params.stop_token_ids == [200012, 55555, 200002] + assert len(sampling_params.stop_token_ids) == 3 + + +class TestCompletionStopTokenIds: + """Test stop_token_ids merging in CompletionRequest.to_sampling_params().""" + + @pytest.fixture + def minimal_completion_request(self): + return CompletionRequest( + model="test-model", + prompt="hello", + ) + + def test_default_stop_token_ids_applied(self, minimal_completion_request): + """Server-default stop_token_ids are applied when client sends none.""" + default_sampling_params = { + "stop_token_ids": [200012, 200002], + } + + sampling_params = minimal_completion_request.to_sampling_params( + max_tokens=100, + default_sampling_params=default_sampling_params, + ) + + assert set(sampling_params.stop_token_ids) == {200012, 200002} + + def test_client_stop_token_ids_merged_with_defaults(self): + """Client-specified stop_token_ids are merged with server defaults.""" + request = CompletionRequest( + model="test-model", + prompt="hello", + stop_token_ids=[99999], + ) + default_sampling_params = { + "stop_token_ids": [200012, 200002], + } + + sampling_params = request.to_sampling_params( + max_tokens=100, + default_sampling_params=default_sampling_params, + ) + + assert set(sampling_params.stop_token_ids) == {200012, 200002, 99999} + assert sampling_params.stop_token_ids == [99999, 200012, 200002] + + def test_no_stop_token_ids_anywhere(self, minimal_completion_request): + """When neither client nor server specifies stop_token_ids, result is empty.""" + sampling_params = minimal_completion_request.to_sampling_params( + max_tokens=100, + default_sampling_params={}, + ) + + assert not sampling_params.stop_token_ids diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_tool_calls_serialization.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_tool_calls_serialization.py new file mode 100644 index 0000000000000000000000000000000000000000..cde2189731234dba44f48b06435e21fdc57567b5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_tool_calls_serialization.py @@ -0,0 +1,150 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for tool_calls Iterable → list materialisation. + +Regression tests for https://github.com/vllm-project/vllm/issues/34792. + +Setting VLLM_LOGGING_LEVEL=debug caused tool calling to break for Mistral +models because: + 1. The OpenAI Python SDK types tool_calls as Iterable[...] in + ChatCompletionAssistantMessageParam. + 2. Pydantic v2, when validating from Python objects (not from raw JSON), + wraps Iterable fields in a one-shot lazy iterator. + 3. Debug logging called model_dump_json() which consumed that iterator. + 4. The Mistral tokenizer then saw empty tool_calls and raised + "ValueError: Unexpected tool call id ...". +""" + +import pytest + +from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest + + +def _make_tool_call(tc_id: str, name: str, args: str) -> dict: + return { + "id": tc_id, + "type": "function", + "function": {"name": name, "arguments": args}, + } + + +def _make_request(messages: list) -> ChatCompletionRequest: + return ChatCompletionRequest( + model="test-model", + messages=messages, + ) + + +def test_tool_calls_list_preserved_after_model_dump(): + """tool_calls in assistant messages must be readable after model_dump_json. + + When the request is built from Python dicts (as in the Anthropic → OpenAI + conversion path), Pydantic v2 previously wrapped the Iterable tool_calls + in a one-shot iterator. model_dump_json() consumed it, leaving subsequent + readers (e.g. the Mistral tokenizer) with an empty sequence. + """ + tool_call = _make_tool_call("call_abc123", "get_weather", '{"city": "Paris"}') + messages = [ + {"role": "user", "content": "What is the weather in Paris?"}, + {"role": "assistant", "content": None, "tool_calls": [tool_call]}, + { + "role": "tool", + "tool_call_id": "call_abc123", + "content": '{"temperature": 20}', + }, + ] + + req = _make_request(messages) + + # Simulate debug logging: serialize the model (this was the trigger) + _ = req.model_dump_json() + + # The assistant message must still have accessible tool_calls afterwards + assistant_msg = req.messages[1] + assert isinstance(assistant_msg, dict) + tool_calls = assistant_msg.get("tool_calls") + assert tool_calls is not None, "tool_calls must not be None after model_dump_json" + assert isinstance(tool_calls, list), "tool_calls must be a list" + assert len(tool_calls) > 0, "tool_calls must not be empty after model_dump_json" + + +def test_tool_calls_from_generator_are_materialised(): + """tool_calls passed as a generator must be converted to list on validation.""" + tool_call = _make_tool_call("call_gen1", "search", '{"query": "vllm"}') + + def tool_calls_gen(): + yield tool_call + + messages = [ + {"role": "user", "content": "Search for vllm"}, + { + "role": "assistant", + "content": None, + "tool_calls": tool_calls_gen(), # one-shot generator + }, + ] + + req = _make_request(messages) + assistant_msg = req.messages[1] + assert isinstance(assistant_msg, dict) + + # Iterate twice — must not raise or return empty on second pass + tool_calls_first = list(assistant_msg.get("tool_calls", [])) + tool_calls_second = list(assistant_msg.get("tool_calls", [])) + + assert len(tool_calls_first) == 1, "First read must return the tool call" + assert len(tool_calls_second) == 1, "Second read must also return the tool call" + + +def test_tool_calls_list_passthrough(): + """tool_calls already provided as a list must remain a list.""" + tool_call = _make_tool_call("call_list1", "calculate", '{"expr": "2+2"}') + messages = [ + {"role": "user", "content": "Calculate 2+2"}, + {"role": "assistant", "content": None, "tool_calls": [tool_call]}, + ] + + req = _make_request(messages) + assistant_msg = req.messages[1] + assert isinstance(assistant_msg, dict) + assert isinstance(assistant_msg.get("tool_calls"), list) + + +def test_messages_without_tool_calls_unaffected(): + """Messages without tool_calls must be handled correctly.""" + messages = [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "Hello!"}, + {"role": "assistant", "content": "Hi there!"}, + ] + + req = _make_request(messages) + # None of the messages should have tool_calls injected + for msg in req.messages: + assert isinstance(msg, dict) + assert msg.get("tool_calls") is None or msg.get("tool_calls") == [] + + +@pytest.mark.parametrize("num_tool_calls", [1, 3]) +def test_multiple_tool_calls_materialised(num_tool_calls: int): + """Multiple tool calls in a single message are all preserved.""" + tool_calls = [ + _make_tool_call(f"call_{i}", f"func_{i}", f'{{"arg": {i}}}') + for i in range(num_tool_calls) + ] + messages = [ + {"role": "user", "content": "Do things"}, + {"role": "assistant", "content": None, "tool_calls": iter(tool_calls)}, + ] + + req = _make_request(messages) + assistant_msg = req.messages[1] + assert isinstance(assistant_msg, dict) + + result_tool_calls = assistant_msg.get("tool_calls") + assert isinstance(result_tool_calls, list) + assert len(result_tool_calls) == num_tool_calls + + # Verify after model_dump_json too + _ = req.model_dump_json() + assert len(assistant_msg.get("tool_calls", [])) == num_tool_calls diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_tool_choice_content_none.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_tool_choice_content_none.py new file mode 100644 index 0000000000000000000000000000000000000000..ed9886d6898beb25f5fbe4f8551cb816af9f39e3 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_tool_choice_content_none.py @@ -0,0 +1,203 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +from openai.types.chat.chat_completion import ChatCompletion as OpenAIChatCompletion +from openai.types.chat.chat_completion_chunk import ChatCompletionChunk + +from vllm.entrypoints.openai.chat_completion.protocol import ( + ChatCompletionRequest, + ChatCompletionResponse, + ChatCompletionResponseChoice, + ChatCompletionResponseStreamChoice, + ChatCompletionStreamResponse, + ChatMessage, +) +from vllm.entrypoints.openai.engine.protocol import ( + DeltaFunctionCall, + DeltaMessage, + DeltaToolCall, + FunctionCall, + ToolCall, + UsageInfo, +) +from vllm.entrypoints.openai.responses.protocol import ResponsesRequest +from vllm.parser.abstract_parser import DelegatingParser + +pytestmark = pytest.mark.skip_global_cleanup + + +class _DummyDelegatingParser(DelegatingParser): + def is_reasoning_end(self, input_ids: list[int]) -> bool: + return False + + def extract_content_ids(self, input_ids: list[int]) -> list[int]: + return input_ids + + def extract_reasoning(self, model_output: str, request): + return None, model_output + + def extract_reasoning_streaming( + self, + previous_text: str, + current_text: str, + delta_text: str, + previous_token_ids: list[int], + current_token_ids: list[int], + delta_token_ids: list[int], + ): + return None + + +def test_chat_completion_named_tool_choice_with_none_content(): + request = ChatCompletionRequest.model_validate( + { + "model": "test-model", + "messages": [{"role": "user", "content": "test"}], + "tools": [ + { + "type": "function", + "function": { + "name": "get_weather", + "parameters": {"type": "object", "properties": {}}, + }, + } + ], + "tool_choice": {"type": "function", "function": {"name": "get_weather"}}, + } + ) + parser = _DummyDelegatingParser(tokenizer=None) + + tool_calls, content = parser._extract_tool_calls( + content=None, + request=request, + enable_auto_tools=True, + ) + + assert content is None + assert tool_calls == [] + + +def test_responses_parser_allows_named_tool_choice_with_none_content(): + request = ResponsesRequest.model_validate( + { + "model": "test-model", + "input": "test", + "tools": [ + { + "type": "function", + "name": "get_weather", + "parameters": {"type": "object", "properties": {}}, + } + ], + "tool_choice": {"type": "function", "name": "get_weather"}, + } + ) + parser = _DummyDelegatingParser(tokenizer=None) + + tool_calls, content = parser._extract_tool_calls( + content=None, + request=request, + enable_auto_tools=False, + ) + + assert content is None + assert tool_calls == [] + + +def _chat_response(message: ChatMessage) -> ChatCompletionResponse: + return ChatCompletionResponse( + model="test-model", + choices=[ + ChatCompletionResponseChoice( + index=0, + message=message, + finish_reason="stop", + ) + ], + usage=UsageInfo(prompt_tokens=1, completion_tokens=1, total_tokens=2), + ) + + +def test_chat_completion_response_omits_empty_tool_calls_payload(): + response = _chat_response(ChatMessage(role="assistant", content="done")) + + payload = response.model_dump() + payload_exclude_unset = response.model_dump(exclude_unset=True) + + assert "tool_calls" not in payload["choices"][0]["message"] + assert "tool_calls" not in payload_exclude_unset["choices"][0]["message"] + parsed = OpenAIChatCompletion.model_validate(payload) + assert parsed.choices[0].message.tool_calls is None + + +def test_chat_completion_response_keeps_non_empty_tool_calls_payload(): + response = _chat_response( + ChatMessage( + role="assistant", + content="", + tool_calls=[ + ToolCall( + function=FunctionCall( + name="get_weather", + arguments='{"city": "Beijing"}', + ) + ) + ], + ) + ) + + message = response.model_dump()["choices"][0]["message"] + + assert len(message["tool_calls"]) == 1 + assert message["tool_calls"][0]["function"]["name"] == "get_weather" + + +def _stream_response(delta: DeltaMessage) -> ChatCompletionStreamResponse: + return ChatCompletionStreamResponse( + id="chatcmpl-test", + object="chat.completion.chunk", + created=1, + model="test-model", + choices=[ + ChatCompletionResponseStreamChoice( + index=0, + delta=delta, + finish_reason=None, + ) + ], + ) + + +def test_chat_completion_stream_response_omits_empty_tool_calls_payload(): + response = _stream_response(DeltaMessage(content="done")) + + payload = response.model_dump(exclude_unset=True) + payload_json = response.model_dump_json(exclude_unset=True) + + assert "tool_calls" not in payload["choices"][0]["delta"] + parsed = ChatCompletionChunk.model_validate_json(payload_json) + assert parsed.choices[0].delta.tool_calls is None + + +def test_chat_completion_stream_response_keeps_non_empty_tool_calls_payload(): + response = _stream_response( + DeltaMessage( + tool_calls=[ + DeltaToolCall( + index=0, + id="call-test", + type="function", + function=DeltaFunctionCall( + name="get_weather", + arguments='{"city": "Beijing"}', + ), + ) + ] + ) + ) + + delta = response.model_dump(exclude_unset=True)["choices"][0]["delta"] + + assert len(delta["tool_calls"]) == 1 + assert delta["tool_calls"][0]["function"]["name"] == "get_weather" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/test_uds.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_uds.py new file mode 100644 index 0000000000000000000000000000000000000000..0f4b21ad6e5f846c55fd6b405577c96327e3393b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/test_uds.py @@ -0,0 +1,43 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from tempfile import TemporaryDirectory + +import httpx +import pytest + +from vllm.version import __version__ as VLLM_VERSION + +from ...utils import RemoteOpenAIServer + +MODEL_NAME = "Qwen/Qwen3-0.6B" + + +@pytest.fixture(scope="module") +def server(): + with TemporaryDirectory() as tmpdir: + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-model-len", + "8192", + "--enforce-eager", + "--max-num-seqs", + "128", + "--uds", + f"{tmpdir}/vllm.sock", + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.mark.asyncio +async def test_show_version(server: RemoteOpenAIServer): + transport = httpx.HTTPTransport(uds=server.uds) + client = httpx.Client(transport=transport) + response = client.get(server.url_for("version")) + response.raise_for_status() + + assert response.json() == {"version": VLLM_VERSION} diff --git a/asian_ai/performance/vLLM/tests/entrypoints/openai/utils.py b/asian_ai/performance/vLLM/tests/entrypoints/openai/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..62ab1f4b5e7d94f5aaedef90414050885b27354f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/openai/utils.py @@ -0,0 +1,192 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import json +from collections.abc import AsyncGenerator +from typing import Any + +from vllm.entrypoints.openai.chat_completion.protocol import ( + ChatCompletionResponse, + ChatCompletionResponseChoice, + ChatCompletionStreamResponse, + ChatMessage, +) +from vllm.entrypoints.openai.engine.protocol import UsageInfo + + +async def accumulate_streaming_response( + stream_generator: AsyncGenerator[str, None], +) -> ChatCompletionResponse: + """ + Accumulate streaming SSE chunks into a complete ChatCompletionResponse. + + This helper parses the SSE format and builds up the complete response + by combining all the delta chunks. + """ + accumulated_content = "" + accumulated_reasoning = None + accumulated_tool_calls: list[dict[str, Any]] = [] + role = None + finish_reason = None + response_id = None + created = None + model = None + index = 0 + + async for chunk_str in stream_generator: + # Skip empty lines and [DONE] marker + if not chunk_str.strip() or chunk_str.strip() == "data: [DONE]": + continue + + # Parse SSE format: "data: {json}\n\n" + if chunk_str.startswith("data: "): + json_str = chunk_str[6:].strip() + try: + chunk_data = json.loads(json_str) + # print(f"DEBUG: Parsed chunk_data: {chunk_data}") + chunk = ChatCompletionStreamResponse(**chunk_data) + + # Store metadata from first chunk + if response_id is None: + response_id = chunk.id + created = chunk.created + model = chunk.model + + # Process each choice in the chunk + for choice in chunk.choices: + if choice.delta.role: + role = choice.delta.role + if choice.delta.content: + accumulated_content += choice.delta.content + if choice.delta.reasoning: + if accumulated_reasoning is None: + accumulated_reasoning = "" + accumulated_reasoning += choice.delta.reasoning + if choice.delta.tool_calls: + # Accumulate tool calls + for tool_call_delta in choice.delta.tool_calls: + # Find or create the tool call at this index + while len(accumulated_tool_calls) <= tool_call_delta.index: + accumulated_tool_calls.append( + { + "id": None, + "type": "function", + "function": {"name": "", "arguments": ""}, + } + ) + + if tool_call_delta.id: + accumulated_tool_calls[tool_call_delta.index]["id"] = ( + tool_call_delta.id + ) + if tool_call_delta.function: + if tool_call_delta.function.name: + accumulated_tool_calls[tool_call_delta.index][ + "function" + ]["name"] += tool_call_delta.function.name + if tool_call_delta.function.arguments: + accumulated_tool_calls[tool_call_delta.index][ + "function" + ]["arguments"] += tool_call_delta.function.arguments + + if choice.finish_reason: + finish_reason = choice.finish_reason + if choice.index is not None: + index = choice.index + + except json.JSONDecodeError: + continue + + # Build the final message + message_kwargs = { + "role": role or "assistant", + "content": accumulated_content if accumulated_content else None, + "reasoning": accumulated_reasoning, + } + + # Only include tool_calls if there are any + if accumulated_tool_calls: + message_kwargs["tool_calls"] = [ + {"id": tc["id"], "type": tc["type"], "function": tc["function"]} + for tc in accumulated_tool_calls + ] + + message = ChatMessage(**message_kwargs) + + # Build the final response + choice = ChatCompletionResponseChoice( + index=index, + message=message, + finish_reason=finish_reason or "stop", + ) + + # Create usage info (with dummy values for tests) + usage = UsageInfo( + prompt_tokens=0, + completion_tokens=0, + total_tokens=0, + ) + + response = ChatCompletionResponse( + id=response_id or "chatcmpl-test", + object="chat.completion", + created=created or 0, + model=model or "test-model", + choices=[choice], + usage=usage, + ) + + return response + + +def verify_harmony_messages( + messages: list[Any], expected_messages: list[dict[str, Any]] +): + assert len(messages) == len(expected_messages) + for msg, expected in zip(messages, expected_messages): + if "role" in expected: + assert msg.author.role == expected["role"] + if "author_name" in expected: + assert msg.author.name == expected["author_name"] + if "channel" in expected: + assert msg.channel == expected["channel"] + if "recipient" in expected: + assert msg.recipient == expected["recipient"] + if "content" in expected: + assert msg.content[0].text == expected["content"] + if "content_type" in expected: + assert msg.content_type == expected["content_type"] + if "instructions" in expected: + assert msg.content[0].instructions == expected["instructions"] + if "tool_definitions" in expected: + # Check that the tool definitions match the expected list of tool names + actual_tools = [t.name for t in msg.content[0].tools["functions"].tools] + assert actual_tools == expected["tool_definitions"] + + +def verify_chat_response( + response: ChatCompletionResponse, + content: str | None = None, + reasoning: str | None = None, + tool_calls: list[tuple[str, str]] | None = None, +): + assert len(response.choices) == 1 + message = response.choices[0].message + + if content is not None: + assert message.content == content + else: + assert not message.content + + if reasoning is not None: + assert message.reasoning == reasoning + else: + assert not message.reasoning + + if tool_calls: + assert message.tool_calls is not None + assert len(message.tool_calls) == len(tool_calls) + for tc, (expected_name, expected_args) in zip(message.tool_calls, tool_calls): + assert tc.function.name == expected_name + assert tc.function.arguments == expected_args + else: + assert not message.tool_calls diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/basic/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/basic/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/basic/test_encode.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/basic/test_encode.py new file mode 100644 index 0000000000000000000000000000000000000000..b3149a5628ff1b0e17d90470d7be9b12a827f260 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/basic/test_encode.py @@ -0,0 +1,92 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import weakref + +import pytest + +from vllm import LLM, PoolingParams +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.platforms import current_platform + +MODEL_NAME = "intfloat/multilingual-e5-small" + +PROMPTS = [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + "The future of AI is", +] + +TOKEN_IDS = [ + # Using ID={0, 1, 2, 3} results in NaN values, + # so we add this offset of 1000 + [1000], + [1000, 1001], + [1000, 1002, 1001], + [1000, 1003, 1001, 1002], +] + + +@pytest.fixture(scope="module") +def llm(): + # ROCm: Use FLEX_ATTENTION backend as it's the only attention backend + # that supports encoder-only models on ROCm. + attention_config = None + if current_platform.is_rocm(): + attention_config = {"backend": "FLEX_ATTENTION"} + + # pytest caches the fixture so we use weakref.proxy to + # enable garbage collection + llm = LLM( + model=MODEL_NAME, + max_num_batched_tokens=32768, + tensor_parallel_size=1, + gpu_memory_utilization=0.75, + enforce_eager=True, + seed=0, + attention_config=attention_config, + ) + + yield weakref.proxy(llm) + + del llm + + cleanup_dist_env_and_memory() + + +@pytest.mark.skip_global_cleanup +def test_multiple_pooling_params(llm: LLM): + pooling_params = [ + PoolingParams(), + PoolingParams(), + PoolingParams(), + PoolingParams(), + ] + + # Multiple PoolingParams should be matched with each prompt + outputs = llm.encode(PROMPTS, pooling_params=pooling_params, pooling_task="embed") + assert len(PROMPTS) == len(outputs) + + # Exception raised, if the size of params does not match the size of prompts + with pytest.raises(ValueError): + outputs = llm.encode( + PROMPTS, pooling_params=pooling_params[:3], pooling_task="embed" + ) + + # Single PoolingParams should be applied to every prompt + single_pooling_params = PoolingParams() + outputs = llm.encode( + PROMPTS, pooling_params=single_pooling_params, pooling_task="embed" + ) + assert len(PROMPTS) == len(outputs) + + # pooling_params is None, default params should be applied + outputs = llm.encode(PROMPTS, pooling_params=None, pooling_task="embed") + assert len(PROMPTS) == len(outputs) + + +def test_right_side_truncation(llm: LLM): + # Embeddings models should truncate the end of the prompt + tokenizer = llm.get_tokenizer() + assert tokenizer.truncation_side == "right" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/basic/test_truncation.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/basic/test_truncation.py new file mode 100644 index 0000000000000000000000000000000000000000..5f9dbd758bf2511f5c54601bba5d3f05829a101a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/basic/test_truncation.py @@ -0,0 +1,104 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from typing import Any + +import openai +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer +from vllm.platforms import current_platform + +MODEL_NAME = "sentence-transformers/all-MiniLM-L12-v2" +max_model_len = 128 + +input = """Immerse yourself in the enchanting chronicle of calculus, a + mathematical domain that has radically transformed our comprehension of + change and motion. Despite its roots in ancient civilizations, the + formal birth of calculus predominantly occurred in the 17th century, + primarily under the influential guidance of Sir Isaac Newton and Gottfried + Wilhelm Leibniz. The earliest traces of calculus concepts are found in + ancient Greek mathematics,most notably in the works of Eudoxus and + Archimedes, around 300 BCE. They utilized the 'method of exhaustion'—a + technique for computing areas and volumes through the use of finite sums. + This methodology laid crucial foundational work for integral calculus. + In the 17th century, both Newton and Leibniz independently pioneered + calculus, each contributing unique perspectives that would shape this new + field.""" + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--runner", + "pooling", + "--dtype", + "bfloat16", + "--enforce-eager", + "--max-model-len", + str(max_model_len), + ] + + # ROCm: Use Flex Attention to support encoder-only self-attention. + if current_platform.is_rocm(): + args.extend(["--attention-backend", "FLEX_ATTENTION"]) + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +async def test_smaller_truncation_size(client: openai.AsyncOpenAI): + truncation_size = 10 + kwargs: dict[str, Any] = { + "model": MODEL_NAME, + "input": input, + "truncate_prompt_tokens": truncation_size, + } + + response = await client.post(path="embeddings", cast_to=object, body={**kwargs}) + + assert response["usage"]["prompt_tokens"] == truncation_size + + +@pytest.mark.asyncio +async def test_bigger_truncation_size(client: openai.AsyncOpenAI): + truncation_size = max_model_len + 1 + kwargs: dict[str, Any] = { + "model": MODEL_NAME, + "input": input, + "truncate_prompt_tokens": truncation_size, + } + + with pytest.raises(openai.BadRequestError) as err: + await client.post(path="embeddings", cast_to=object, body={**kwargs}) + + assert err.value.status_code == 400 + error_details = err.value.response.json()["error"] + assert error_details["type"] == "BadRequestError" + expected_message = ( + "truncate_prompt_tokens value is " + "greater than max_model_len." + " Please request a smaller truncation size." + ) + assert error_details["message"] == expected_message + + +@pytest.mark.asyncio +async def test_max_truncation_size(client: openai.AsyncOpenAI): + truncation_size = -1 + kwargs: dict[str, Any] = { + "model": MODEL_NAME, + "input": input, + "truncate_prompt_tokens": truncation_size, + } + + response = await client.post(path="embeddings", cast_to=object, body={**kwargs}) + + assert response["usage"]["prompt_tokens"] == max_model_len diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/classify/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/classify/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/classify/test_offline.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/classify/test_offline.py new file mode 100644 index 0000000000000000000000000000000000000000..0b30443a7a0afc86df204fd3661da1c600803d5b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/classify/test_offline.py @@ -0,0 +1,109 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import weakref + +import pytest +import torch + +from tests.models.utils import softmax +from vllm import LLM, ClassificationRequestOutput, PoolingParams +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.tasks import PoolingTask + +MODEL_NAME = "jason9693/Qwen2.5-1.5B-apeach" + +prompt = "The chef prepared a delicious meal." +prompt_token_ids = [785, 29706, 10030, 264, 17923, 15145, 13] +num_labels = 2 + + +@pytest.fixture(scope="module") +def llm(): + # pytest caches the fixture so we use weakref.proxy to + # enable garbage collection + llm = LLM( + model=MODEL_NAME, + max_num_batched_tokens=32768, + tensor_parallel_size=1, + gpu_memory_utilization=0.75, + enforce_eager=True, + seed=0, + ) + + yield weakref.proxy(llm) + + del llm + + cleanup_dist_env_and_memory() + + +@pytest.mark.skip_global_cleanup +def test_str_prompts(llm: LLM): + outputs = llm.classify(prompt, use_tqdm=False) + assert len(outputs) == 1 + assert isinstance(outputs[0], ClassificationRequestOutput) + assert outputs[0].prompt_token_ids == prompt_token_ids + assert len(outputs[0].outputs.probs) == num_labels + + +@pytest.mark.skip_global_cleanup +def test_token_ids_prompts(llm: LLM): + outputs = llm.classify([prompt_token_ids], use_tqdm=False) + assert len(outputs) == 1 + assert isinstance(outputs[0], ClassificationRequestOutput) + assert outputs[0].prompt_token_ids == prompt_token_ids + assert len(outputs[0].outputs.probs) == num_labels + + +@pytest.mark.skip_global_cleanup +def test_list_prompts(llm: LLM): + outputs = llm.classify([prompt, prompt_token_ids], use_tqdm=False) + assert len(outputs) == 2 + for i in range(len(outputs)): + assert isinstance(outputs[i], ClassificationRequestOutput) + assert outputs[i].prompt_token_ids == prompt_token_ids + assert len(outputs[i].outputs.probs) == num_labels + + +@pytest.mark.skip_global_cleanup +def test_pooling_params(llm: LLM): + def get_outputs(use_activation): + outputs = llm.classify( + prompt, + pooling_params=PoolingParams(use_activation=use_activation), + use_tqdm=False, + ) + return torch.tensor([x.outputs.probs for x in outputs]) + + default = get_outputs(use_activation=None) + w_activation = get_outputs(use_activation=True) + wo_activation = get_outputs(use_activation=False) + + assert torch.allclose(default, w_activation, atol=1e-2), ( + "Default should use activation." + ) + assert not torch.allclose(w_activation, wo_activation, atol=1e-2), ( + "wo_activation should not use activation." + ) + assert torch.allclose(softmax(wo_activation), w_activation, atol=1e-2), ( + "w_activation should be close to activation(wo_activation)." + ) + + +@pytest.mark.skip_global_cleanup +def test_score_api(llm: LLM): + err_msg = "Scoring API is only enabled for num_labels == 1." + with pytest.raises(ValueError, match=err_msg): + llm.score("ping", "pong", use_tqdm=False) + + +@pytest.mark.parametrize("task", ["embed", "token_embed", "token_classify", "plugin"]) +def test_unsupported_tasks(llm: LLM, task: PoolingTask): + if task == "plugin": + err_msg = "No IOProcessor plugin installed." + elif task == "token_classify": + err_msg = "Try switching the model's pooling_task via.+" + else: + err_msg = "Embedding API is not supported by this model.+" + with pytest.raises(ValueError, match=err_msg): + llm.encode(prompt, pooling_task=task, use_tqdm=False) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/classify/test_online.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/classify/test_online.py new file mode 100644 index 0000000000000000000000000000000000000000..2aeb03cdb4ec13435b5c9285126e93e6b599ae93 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/classify/test_online.py @@ -0,0 +1,460 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import requests +import torch +import torch.nn.functional as F + +from tests.utils import RemoteOpenAIServer +from vllm.entrypoints.pooling.classify.protocol import ClassificationResponse +from vllm.entrypoints.pooling.pooling.protocol import PoolingResponse + +MODEL_NAME = "jason9693/Qwen2.5-1.5B-apeach" +DTYPE = "float32" # Use float32 to avoid NaN issue +input_text = "This product was excellent and exceeded my expectations" +input_tokens = [1986, 1985, 572, 9073, 323, 33808, 847, 16665] + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--enforce-eager", + "--max-model-len", + "512", + "--dtype", + DTYPE, + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_basic(server: RemoteOpenAIServer, model_name: str): + # test /v1/models + response = requests.get(server.url_for("/v1/models")) + served_model = response.json()["data"][0]["id"] + assert served_model == MODEL_NAME + + # test /tokenize + response = requests.post( + server.url_for("/tokenize"), + json={"model": model_name, "prompt": input_text}, + ) + assert response.json()["tokens"] == input_tokens + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_completion_request(server: RemoteOpenAIServer, model_name: str): + # test input: str + classification_response = requests.post( + server.url_for("classify"), + json={"model": model_name, "input": input_text}, + ) + + classification_response.raise_for_status() + output = ClassificationResponse.model_validate(classification_response.json()) + + assert output.object == "list" + assert output.model == MODEL_NAME + assert len(output.data) == 1 + assert hasattr(output.data[0], "label") + assert hasattr(output.data[0], "probs") + + # test input: list[int] + classification_response = requests.post( + server.url_for("classify"), + json={"model": model_name, "input": input_tokens}, + ) + + classification_response.raise_for_status() + output = ClassificationResponse.model_validate(classification_response.json()) + + assert output.object == "list" + assert output.model == MODEL_NAME + assert len(output.data) == 1 + assert hasattr(output.data[0], "label") + assert hasattr(output.data[0], "probs") + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_completion_request_batched(server: RemoteOpenAIServer, model_name: str): + N = 10 + + # test input: list[str] + classification_response = requests.post( + server.url_for("classify"), + json={"model": model_name, "input": [input_text] * N}, + ) + output = ClassificationResponse.model_validate(classification_response.json()) + + assert len(output.data) == N + for i, item in enumerate(output.data): + assert item.index == i + assert hasattr(item, "label") + assert hasattr(item, "probs") + assert len(item.probs) == item.num_classes + assert item.label in ["Default", "Spoiled"] + + # test input: list[list[int]] + classification_response = requests.post( + server.url_for("classify"), + json={"model": model_name, "input": [input_tokens] * N}, + ) + output = ClassificationResponse.model_validate(classification_response.json()) + + assert len(output.data) == N + for i, item in enumerate(output.data): + assert item.index == i + assert hasattr(item, "label") + assert hasattr(item, "probs") + assert len(item.probs) == item.num_classes + assert item.label in ["Default", "Spoiled"] + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_empty_input_error(server: RemoteOpenAIServer, model_name: str): + classification_response = requests.post( + server.url_for("classify"), + json={"model": model_name, "input": ""}, + ) + + error = classification_response.json() + assert classification_response.status_code == 400 + assert "error" in error + + classification_response = requests.post( + server.url_for("classify"), + json={"model": model_name, "input": []}, + ) + + error = classification_response.json() + assert classification_response.status_code == 400 + assert "error" in error + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_truncate_prompt_tokens(server: RemoteOpenAIServer, model_name: str): + long_text = "hello " * 600 + + classification_response = requests.post( + server.url_for("classify"), + json={"model": model_name, "input": long_text, "truncate_prompt_tokens": 5}, + ) + + classification_response.raise_for_status() + output = ClassificationResponse.model_validate(classification_response.json()) + + assert len(output.data) == 1 + assert output.data[0].index == 0 + assert hasattr(output.data[0], "probs") + assert output.usage.prompt_tokens == 5 + assert output.usage.total_tokens == 5 + + # invalid_truncate_prompt_tokens + classification_response = requests.post( + server.url_for("classify"), + json={"model": model_name, "input": "test", "truncate_prompt_tokens": 513}, + ) + + error = classification_response.json() + assert classification_response.status_code == 400 + assert "truncate_prompt_tokens" in error["error"]["message"] + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_add_special_tokens(server: RemoteOpenAIServer, model_name: str): + # The add_special_tokens parameter doesn't seem to be working with this model. + # working with papluca/xlm-roberta-base-language-detection + response = requests.post( + server.url_for("classify"), + json={"model": model_name, "input": input_text, "add_special_tokens": False}, + ) + response.raise_for_status() + ClassificationResponse.model_validate(response.json()) + + response = requests.post( + server.url_for("classify"), + json={"model": model_name, "input": input_text, "add_special_tokens": True}, + ) + response.raise_for_status() + ClassificationResponse.model_validate(response.json()) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_chat_request(server: RemoteOpenAIServer, model_name: str): + messages = [ + { + "role": "user", + "content": "The cat sat on the mat.", + }, + { + "role": "assistant", + "content": "A feline was resting on a rug.", + }, + { + "role": "user", + "content": "Stars twinkle brightly in the night sky.", + }, + ] + + # test chat request basic usage + response = requests.post( + server.url_for("classify"), + json={"model": model_name, "messages": messages}, + ) + + response.raise_for_status() + output = ClassificationResponse.model_validate(response.json()) + + assert output.object == "list" + assert output.model == MODEL_NAME + assert len(output.data) == 1 + assert hasattr(output.data[0], "label") + assert hasattr(output.data[0], "probs") + assert output.usage.prompt_tokens == 51 + + # test add_generation_prompt + response = requests.post( + server.url_for("classify"), + json={"model": model_name, "messages": messages, "add_generation_prompt": True}, + ) + + response.raise_for_status() + output = ClassificationResponse.model_validate(response.json()) + + assert output.object == "list" + assert output.model == MODEL_NAME + assert len(output.data) == 1 + assert hasattr(output.data[0], "label") + assert hasattr(output.data[0], "probs") + assert output.usage.prompt_tokens == 54 + + # test continue_final_message + response = requests.post( + server.url_for("classify"), + json={ + "model": model_name, + "messages": messages, + "continue_final_message": True, + }, + ) + + response.raise_for_status() + output = ClassificationResponse.model_validate(response.json()) + + assert output.object == "list" + assert output.model == MODEL_NAME + assert len(output.data) == 1 + assert hasattr(output.data[0], "label") + assert hasattr(output.data[0], "probs") + assert output.usage.prompt_tokens == 49 + + # test add_special_tokens + # The add_special_tokens parameter doesn't seem to be working with this model. + response = requests.post( + server.url_for("classify"), + json={"model": model_name, "messages": messages, "add_special_tokens": True}, + ) + + response.raise_for_status() + output = ClassificationResponse.model_validate(response.json()) + + assert output.object == "list" + assert output.model == MODEL_NAME + assert len(output.data) == 1 + assert hasattr(output.data[0], "label") + assert hasattr(output.data[0], "probs") + assert output.usage.prompt_tokens == 51 + + # test continue_final_message with add_generation_prompt + response = requests.post( + server.url_for("classify"), + json={ + "model": model_name, + "messages": messages, + "continue_final_message": True, + "add_generation_prompt": True, + }, + ) + assert ( + "Cannot set both `continue_final_message` and `add_generation_prompt` to True." + in response.json()["error"]["message"] + ) + + +@pytest.mark.asyncio +async def test_invocations_completion_request(server: RemoteOpenAIServer): + request_args = { + "model": MODEL_NAME, + "input": input_text, + } + + classification_response = requests.post( + server.url_for("classify"), json=request_args + ) + classification_response.raise_for_status() + + invocation_response = requests.post( + server.url_for("invocations"), json=request_args + ) + invocation_response.raise_for_status() + + classification_output = classification_response.json() + invocation_output = invocation_response.json() + + assert classification_output.keys() == invocation_output.keys() + for classification_data, invocation_data in zip( + classification_output["data"], invocation_output["data"] + ): + assert classification_data.keys() == invocation_data.keys() + assert classification_data["probs"] == pytest.approx( + invocation_data["probs"], rel=0.01 + ) + + +@pytest.mark.asyncio +async def test_invocations_chat_request(server: RemoteOpenAIServer): + messages = [ + { + "role": "user", + "content": "The cat sat on the mat.", + }, + { + "role": "assistant", + "content": "A feline was resting on a rug.", + }, + { + "role": "user", + "content": "Stars twinkle brightly in the night sky.", + }, + ] + + request_args = {"model": MODEL_NAME, "messages": messages} + + classification_response = requests.post( + server.url_for("classify"), json=request_args + ) + classification_response.raise_for_status() + + invocation_response = requests.post( + server.url_for("invocations"), json=request_args + ) + invocation_response.raise_for_status() + + classification_output = classification_response.json() + invocation_output = invocation_response.json() + + assert classification_output.keys() == invocation_output.keys() + for classification_data, invocation_data in zip( + classification_output["data"], invocation_output["data"] + ): + assert classification_data.keys() == invocation_data.keys() + assert classification_data["probs"] == pytest.approx( + invocation_data["probs"], rel=0.01 + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_use_activation(server: RemoteOpenAIServer, model_name: str): + async def get_outputs(use_activation): + response = requests.post( + server.url_for("classify"), + json={ + "model": model_name, + "input": input_text, + "use_activation": use_activation, + }, + ) + outputs = response.json() + return torch.tensor([x["probs"] for x in outputs["data"]]) + + default = await get_outputs(use_activation=None) + w_activation = await get_outputs(use_activation=True) + wo_activation = await get_outputs(use_activation=False) + + assert torch.allclose(default, w_activation, atol=1e-2), ( + "Default should use activation." + ) + assert not torch.allclose(w_activation, wo_activation, atol=1e-2), ( + "wo_activation should not use activation." + ) + assert torch.allclose(F.softmax(wo_activation, dim=-1), w_activation, atol=1e-2), ( + "w_activation should be close to activation(wo_activation)." + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_score(server: RemoteOpenAIServer, model_name: str): + # Scoring API is only enabled for num_labels == 1. + response = requests.post( + server.url_for("score"), + json={ + "model": model_name, + "queries": "ping", + "documents": "pong", + }, + ) + assert response.json()["detail"] == "Not Found" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_rerank(server: RemoteOpenAIServer, model_name: str): + # Scoring API is only enabled for num_labels == 1. + response = requests.post( + server.url_for("rerank"), + json={ + "model": model_name, + "query": "ping", + "documents": ["pong"], + }, + ) + assert response.json()["detail"] == "Not Found" + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_pooling_classify(server: RemoteOpenAIServer, model_name: str): + response = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "input": input_text, + "encoding_format": "float", + "task": "classify", + }, + ) + poolings = PoolingResponse.model_validate(response.json()) + assert len(poolings.data) == 1 + assert len(poolings.data[0].data) == 2 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("task", ["embed", "token_embed", "token_classify", "plugin"]) +async def test_pooling_not_supported( + server: RemoteOpenAIServer, model_name: str, task: str +): + response = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "input": input_text, + "encoding_format": "float", + "task": task, + }, + ) + assert response.json()["error"]["type"] == "BadRequestError" + + if task == "plugin": + err_msg = "No IOProcessor plugin installed." + elif task == "token_classify": + err_msg = "Try switching the model's pooling_task via" + else: + err_msg = f"Unsupported task: {task!r}" + assert response.json()["error"]["message"].startswith(err_msg) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/classify/test_online_vision.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/classify/test_online_vision.py new file mode 100644 index 0000000000000000000000000000000000000000..dc559fcad8bc93bcdd64da014d881691d74c6084 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/classify/test_online_vision.py @@ -0,0 +1,146 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import json + +import pytest +import requests + +from tests.utils import RemoteOpenAIServer +from vllm.entrypoints.pooling.classify.protocol import ClassificationResponse +from vllm.multimodal.utils import encode_image_url, fetch_image + +MODEL_NAME = "muziyongshixin/Qwen2.5-VL-7B-for-VideoCls" +MAXIMUM_VIDEOS = 1 + +HF_OVERRIDES = {"architectures": ["Qwen2_5_VLForSequenceClassification"]} +input_text = "This product was excellent and exceeded my expectations" +image_url = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/cat_snow.jpg" +image_base64 = {"url": encode_image_url(fetch_image(image_url))} +video_url = "https://www.bogotobogo.com/python/OpenCV_Python/images/mean_shift_tracking/slow_traffic_small.mp4" + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--runner", + "pooling", + "--max-model-len", + "16384", + "--enforce-eager", + "--limit-mm-per-prompt", + json.dumps({"video": MAXIMUM_VIDEOS}), + ] + + with RemoteOpenAIServer( + MODEL_NAME, args, override_hf_configs=HF_OVERRIDES + ) as remote_server: + yield remote_server + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_chat_text_request(server: RemoteOpenAIServer, model_name: str): + messages = [ + { + "role": "assistant", + "content": "Please classify this text request.", + }, + { + "role": "user", + "content": input_text, + }, + ] + + response = requests.post( + server.url_for("classify"), + json={"model": model_name, "messages": messages}, + ) + response.raise_for_status() + + output = ClassificationResponse.model_validate(response.json()) + + assert output.object == "list" + assert output.model == model_name + assert len(output.data) == 1 + assert len(output.data[0].probs) == 2 + assert output.usage.prompt_tokens == 35 + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_chat_image_url_request(server: RemoteOpenAIServer, model_name: str): + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Please classify this image."}, + {"type": "image_url", "image_url": {"url": image_url}}, + ], + } + ] + + response = requests.post( + server.url_for("classify"), + json={"model": model_name, "messages": messages}, + ) + response.raise_for_status() + + output = ClassificationResponse.model_validate(response.json()) + + assert output.object == "list" + assert output.model == model_name + assert len(output.data) == 1 + assert len(output.data[0].probs) == 2 + assert output.usage.prompt_tokens == 47 + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_chat_image_base64_request(server: RemoteOpenAIServer, model_name: str): + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Please classify this image."}, + {"type": "image_url", "image_url": image_base64}, + ], + } + ] + + response = requests.post( + server.url_for("classify"), + json={"model": model_name, "messages": messages}, + ) + response.raise_for_status() + + output = ClassificationResponse.model_validate(response.json()) + + assert output.object == "list" + assert output.model == model_name + assert len(output.data) == 1 + assert len(output.data[0].probs) == 2 + assert output.usage.prompt_tokens == 47 + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_chat_video_url_request(server: RemoteOpenAIServer, model_name: str): + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Please classify this video."}, + {"type": "video_url", "video_url": {"url": video_url}}, + ], + } + ] + + response = requests.post( + server.url_for("classify"), + json={"model": model_name, "messages": messages}, + ) + response.raise_for_status() + + output = ClassificationResponse.model_validate(response.json()) + + assert output.object == "list" + assert output.model == model_name + assert len(output.data) == 1 + assert len(output.data[0].probs) == 2 + assert output.usage.prompt_tokens == 8993 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/conftest.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..47e0943b5693620a78b1839c6c6d48d1f9251838 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/conftest.py @@ -0,0 +1,28 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Pytest configuration for vLLM pooling embed tests.""" + +import warnings + +import torch + +from vllm.platforms import current_platform + + +def pytest_collection_modifyitems(config, items): + """Configure ROCm-specific settings based on collected tests.""" + if not current_platform.is_rocm(): + return + + # Disable Flash/MemEfficient SDP on ROCm to avoid HF Transformers + # accuracy issues: https://github.com/vllm-project/vllm/issues/30167 + # TODO: Remove once ROCm SDP accuracy issues are resolved on HuggingFace + torch.backends.cuda.enable_flash_sdp(False) + torch.backends.cuda.enable_mem_efficient_sdp(False) + torch.backends.cuda.enable_math_sdp(True) + warnings.warn( + "ROCm: Disabled flash_sdp and mem_efficient_sdp, enabled math_sdp " + "to avoid HuggingFace Transformers accuracy issues", + UserWarning, + stacklevel=1, + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_cohere_online.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_cohere_online.py new file mode 100644 index 0000000000000000000000000000000000000000..f3ba851a775ab9fa7cabe53927ccded7c6d26477 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_cohere_online.py @@ -0,0 +1,310 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the Cohere /v2/embed API with generic (non-Cohere) models. + +Validates that the Cohere v2 embed endpoint works correctly with standard +embedding models, covering text embedding, embedding type conversions, +response structure, batching, normalisation, and semantic similarity. +""" + +import struct + +import numpy as np +import pybase64 as base64 +import pytest +import requests + +from tests.utils import RemoteOpenAIServer + +DTYPE = "bfloat16" + +MODELS: list[tuple[str, list[str]]] = [ + ("intfloat/multilingual-e5-small", []), + ( + "Snowflake/snowflake-arctic-embed-m-v1.5", + [ + "--trust_remote_code", + "--hf_overrides", + '{"matryoshka_dimensions":[256]}', + ], + ), +] + + +@pytest.fixture(scope="module", params=MODELS, ids=lambda m: m[0]) +def model_config(request): + return request.param + + +@pytest.fixture(scope="module") +def model_name(model_config): + return model_config[0] + + +@pytest.fixture(scope="module") +def server(model_config): + name, extra_args = model_config + args = [ + "--runner", + "pooling", + "--dtype", + DTYPE, + "--enforce-eager", + "--max-model-len", + "512", + "--gpu-memory-utilization", + "0.02", + ] + extra_args + with RemoteOpenAIServer(name, args) as remote_server: + yield remote_server + + +def _cohere_embed( + server: RemoteOpenAIServer, + model_name: str, + texts: list[str] | None = None, + images: list[str] | None = None, + input_type: str | None = None, + embedding_types: list[str] | None = None, +) -> dict: + body: dict = {"model": model_name} + if input_type is not None: + body["input_type"] = input_type + if texts is not None: + body["texts"] = texts + if images is not None: + body["images"] = images + if embedding_types is not None: + body["embedding_types"] = embedding_types + resp = requests.post(server.url_for("/v2/embed"), json=body) + resp.raise_for_status() + return resp.json() + + +def _openai_embed( + server: RemoteOpenAIServer, model_name: str, texts: list[str] +) -> dict: + body = {"model": model_name, "input": texts, "encoding_format": "float"} + resp = requests.post(server.url_for("/v1/embeddings"), json=body) + resp.raise_for_status() + return resp.json() + + +def _cosine_sim(a: list[float], b: list[float]) -> float: + va, vb = np.array(a), np.array(b) + return float(np.dot(va, vb) / (np.linalg.norm(va) * np.linalg.norm(vb))) + + +# ----------------------------------------------------------- +# Text embedding tests +# ----------------------------------------------------------- + + +def test_basic_embed(server: RemoteOpenAIServer, model_name: str): + r = _cohere_embed( + server, model_name, texts=["hello world"], embedding_types=["float"] + ) + assert "embeddings" in r + assert len(r["embeddings"]["float"]) == 1 + assert len(r["embeddings"]["float"][0]) > 0 + + +def test_unsupported_input_type_rejected(server: RemoteOpenAIServer, model_name: str): + """An input_type not defined in the model's prompt config should be + rejected with a 400 error.""" + body = { + "model": model_name, + "input_type": "nonexistent_type", + "texts": ["hello world"], + "embedding_types": ["float"], + } + resp = requests.post(server.url_for("/v2/embed"), json=body) + assert resp.status_code == 400 + assert "Unsupported input_type" in resp.json()["error"]["message"] + + +def test_omitted_input_type_accepted(server: RemoteOpenAIServer, model_name: str): + """Omitting input_type should always work (no prompt prefix applied).""" + body = { + "model": model_name, + "texts": ["hello world"], + "embedding_types": ["float"], + } + resp = requests.post(server.url_for("/v2/embed"), json=body) + assert resp.status_code == 200 + data = resp.json() + assert len(data["embeddings"]["float"]) == 1 + + +def test_v1_v2_parity(server: RemoteOpenAIServer, model_name: str): + """v1 (OpenAI) and v2 (Cohere) endpoints should produce the same + float embeddings for a generic model.""" + texts = ["hello world"] + v2 = _cohere_embed(server, model_name, texts=texts, embedding_types=["float"]) + v1 = _openai_embed(server, model_name, texts) + cos = _cosine_sim(v2["embeddings"]["float"][0], v1["data"][0]["embedding"]) + assert cos > 0.9999, f"v1/v2 parity failed, cosine={cos}" + + +def test_embedding_types(server: RemoteOpenAIServer, model_name: str): + r = _cohere_embed( + server, + model_name, + texts=["test"], + embedding_types=["float", "binary", "ubinary"], + ) + dim = len(r["embeddings"]["float"][0]) + assert len(r["embeddings"]["binary"][0]) == dim // 8 + assert len(r["embeddings"]["ubinary"][0]) == dim // 8 + + +def test_response_structure(server: RemoteOpenAIServer, model_name: str): + r = _cohere_embed(server, model_name, texts=["test"], embedding_types=["float"]) + assert "id" in r + assert "embeddings" in r + assert "texts" in r + assert r["texts"] == ["test"] + assert "meta" in r + assert r["meta"]["api_version"]["version"] == "2" + assert "billed_units" in r["meta"] + assert r["meta"]["billed_units"]["input_tokens"] > 0 + assert r["meta"]["billed_units"]["image_tokens"] == 0 + + +def test_batch(server: RemoteOpenAIServer, model_name: str): + texts = ["apple", "banana", "cherry"] + r = _cohere_embed(server, model_name, texts=texts, embedding_types=["float"]) + assert len(r["embeddings"]["float"]) == 3 + dim = len(r["embeddings"]["float"][0]) + for emb in r["embeddings"]["float"]: + assert len(emb) == dim + + +def test_l2_normalized(server: RemoteOpenAIServer, model_name: str): + r = _cohere_embed( + server, model_name, texts=["hello world"], embedding_types=["float"] + ) + emb = np.array(r["embeddings"]["float"][0]) + assert abs(float(np.linalg.norm(emb)) - 1.0) < 0.01 + + +def test_semantic_similarity(server: RemoteOpenAIServer, model_name: str): + r = _cohere_embed( + server, + model_name, + texts=["machine learning", "deep learning", "chocolate cake recipe"], + embedding_types=["float"], + ) + embs = r["embeddings"]["float"] + cos_related = _cosine_sim(embs[0], embs[1]) + cos_unrelated = _cosine_sim(embs[0], embs[2]) + assert cos_related > cos_unrelated + + +def test_missing_input_returns_error(server: RemoteOpenAIServer, model_name: str): + body = {"model": model_name} + resp = requests.post(server.url_for("/v2/embed"), json=body) + assert resp.status_code == 400 + + +def test_base64_embedding_type(server: RemoteOpenAIServer, model_name: str): + r = _cohere_embed( + server, + model_name, + texts=["test encoding"], + embedding_types=["float", "base64"], + ) + float_emb = r["embeddings"]["float"][0] + b64_str = r["embeddings"]["base64"][0] + decoded = struct.unpack(f"<{len(float_emb)}f", base64.b64decode(b64_str)) + np.testing.assert_allclose(float_emb, decoded, rtol=1e-5) + + +# ----------------------------------------------------------- +# Truncation tests +# ----------------------------------------------------------- + + +def _cohere_embed_raw( + server: RemoteOpenAIServer, + body: dict, +) -> requests.Response: + return requests.post(server.url_for("/v2/embed"), json=body) + + +def test_truncate_end_succeeds(server: RemoteOpenAIServer, model_name: str): + """truncate=END should silently truncate long input.""" + long_text = " ".join(["word"] * 2000) + body = { + "model": model_name, + "texts": [long_text], + "embedding_types": ["float"], + "truncate": "END", + } + resp = _cohere_embed_raw(server, body) + assert resp.status_code == 200 + data = resp.json() + assert len(data["embeddings"]["float"]) == 1 + + +def test_truncate_start_succeeds(server: RemoteOpenAIServer, model_name: str): + """truncate=START should silently truncate long input from the start.""" + long_text = " ".join(["word"] * 2000) + body = { + "model": model_name, + "texts": [long_text], + "embedding_types": ["float"], + "truncate": "START", + } + resp = _cohere_embed_raw(server, body) + assert resp.status_code == 200 + data = resp.json() + assert len(data["embeddings"]["float"]) == 1 + + +def test_truncate_none_rejects_long_input(server: RemoteOpenAIServer, model_name: str): + """truncate=NONE should error when input exceeds model context.""" + long_text = " ".join(["word"] * 2000) + body = { + "model": model_name, + "texts": [long_text], + "embedding_types": ["float"], + "truncate": "NONE", + } + resp = _cohere_embed_raw(server, body) + assert resp.status_code == 400 + + +def test_truncate_start_vs_end_differ(server: RemoteOpenAIServer, model_name: str): + """START and END truncation should produce different embeddings + when the input is long enough to actually be truncated. + + We construct input with distinct tokens at the start vs end + so that keeping different halves produces different embeddings. + """ + start_words = " ".join([f"alpha{i}" for i in range(300)]) + end_words = " ".join([f"omega{i}" for i in range(300)]) + long_text = start_words + " " + end_words + + body_end = { + "model": model_name, + "texts": [long_text], + "embedding_types": ["float"], + "truncate": "END", + } + body_start = { + "model": model_name, + "texts": [long_text], + "embedding_types": ["float"], + "truncate": "START", + } + r_end = _cohere_embed_raw(server, body_end).json() + r_start = _cohere_embed_raw(server, body_start).json() + + emb_end = r_end["embeddings"]["float"][0] + emb_start = r_start["embeddings"]["float"][0] + cos = _cosine_sim(emb_end, emb_start) + assert cos < 0.99, ( + f"START and END truncation should produce different embeddings " + f"for long input, but cosine similarity was {cos}" + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_cohere_online_vision.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_cohere_online_vision.py new file mode 100644 index 0000000000000000000000000000000000000000..e584c3c18a535e1dce5b8c24f95f60d9a53782cc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_cohere_online_vision.py @@ -0,0 +1,135 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the Cohere /v2/embed API with a multimodal model (SigLIP). + +Validates image embedding, batching, normalisation, and embedding type +conversions through the /v2/embed endpoint. +""" + +import struct +import zlib + +import numpy as np +import pybase64 as base64 +import pytest +import requests + +from tests.utils import RemoteOpenAIServer + +MODEL_NAME = "google/siglip-so400m-patch14-384" +DTYPE = "bfloat16" + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--runner", + "pooling", + "--dtype", + DTYPE, + "--enforce-eager", + "--max-model-len", + "64", + "--gpu-memory-utilization", + "0.3", + ] + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +def _make_tiny_png(r: int, g: int, b: int, w: int = 2, h: int = 2) -> str: + raw = b"" + for _ in range(h): + raw += b"\x00" + bytes([r, g, b]) * w + compressed = zlib.compress(raw) + + def chunk(ctype: bytes, cdata: bytes) -> bytes: + c = ctype + cdata + return ( + struct.pack(">I", len(cdata)) + + c + + struct.pack(">I", zlib.crc32(c) & 0xFFFFFFFF) + ) + + ihdr = struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0) + png = ( + b"\x89PNG\r\n\x1a\n" + + chunk(b"IHDR", ihdr) + + chunk(b"IDAT", compressed) + + chunk(b"IEND", b"") + ) + return "data:image/png;base64," + base64.b64encode(png).decode() + + +def _cohere_embed( + server: RemoteOpenAIServer, + texts: list[str] | None = None, + images: list[str] | None = None, + embedding_types: list[str] | None = None, +) -> dict: + body: dict = {"model": MODEL_NAME} + if texts is not None: + body["texts"] = texts + if images is not None: + body["images"] = images + if embedding_types is not None: + body["embedding_types"] = embedding_types + resp = requests.post(server.url_for("/v2/embed"), json=body) + resp.raise_for_status() + return resp.json() + + +def test_image_embed(server: RemoteOpenAIServer): + img_uri = _make_tiny_png(255, 0, 0) + r = _cohere_embed( + server, + images=[img_uri], + embedding_types=["float"], + ) + assert "embeddings" in r + assert len(r["embeddings"]["float"]) == 1 + assert len(r["embeddings"]["float"][0]) > 0 + assert r["meta"]["billed_units"]["image_tokens"] > 0 + assert r["meta"]["billed_units"]["input_tokens"] == 0 + + +def test_image_batch(server: RemoteOpenAIServer): + red = _make_tiny_png(255, 0, 0) + blue = _make_tiny_png(0, 0, 255) + r = _cohere_embed( + server, + images=[red, blue], + embedding_types=["float"], + ) + assert len(r["embeddings"]["float"]) == 2 + + +def test_image_l2_normalized(server: RemoteOpenAIServer): + img_uri = _make_tiny_png(0, 255, 0) + r = _cohere_embed( + server, + images=[img_uri], + embedding_types=["float"], + ) + emb = np.array(r["embeddings"]["float"][0]) + assert abs(float(np.linalg.norm(emb)) - 1.0) < 0.01 + + +def test_image_embedding_types(server: RemoteOpenAIServer): + img_uri = _make_tiny_png(128, 128, 128) + r = _cohere_embed( + server, + images=[img_uri], + embedding_types=["float", "binary", "ubinary"], + ) + dim = len(r["embeddings"]["float"][0]) + assert len(r["embeddings"]["binary"][0]) == dim // 8 + assert len(r["embeddings"]["ubinary"][0]) == dim // 8 + + +def test_text_embed_on_multimodal(server: RemoteOpenAIServer): + """SigLIP also supports text-only embedding via /v2/embed.""" + r = _cohere_embed(server, texts=["hello world"], embedding_types=["float"]) + assert "embeddings" in r + assert len(r["embeddings"]["float"]) == 1 + assert len(r["embeddings"]["float"][0]) > 0 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_cohere_openai_parity.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_cohere_openai_parity.py new file mode 100644 index 0000000000000000000000000000000000000000..04bb3e1ace918552e9215378887d5ae16ac66779 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_cohere_openai_parity.py @@ -0,0 +1,121 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Parity test between Cohere /v2/embed and OpenAI /v1/embeddings. + +Verifies that both endpoints produce identical float embeddings when +no prompt prefix is applied (input_type omitted for Cohere /v2/embed). +""" + +import numpy as np +import pytest +import requests + +from tests.utils import ROCM_EXTRA_ARGS, RemoteOpenAIServer + +MODEL_NAME = "BAAI/bge-base-en-v1.5" +DTYPE = "bfloat16" + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--runner", + "pooling", + "--dtype", + DTYPE, + "--enforce-eager", + "--max-model-len", + "512", + "--gpu-memory-utilization", + "0.02", + ] + ROCM_EXTRA_ARGS + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +def _cohere_embed( + server: RemoteOpenAIServer, + texts: list[str], +) -> list[list[float]]: + body = { + "model": MODEL_NAME, + "texts": texts, + "embedding_types": ["float"], + } + resp = requests.post(server.url_for("/v2/embed"), json=body) + resp.raise_for_status() + return resp.json()["embeddings"]["float"] + + +def _openai_embed( + server: RemoteOpenAIServer, + texts: list[str], +) -> list[list[float]]: + body = {"model": MODEL_NAME, "input": texts, "encoding_format": "float"} + resp = requests.post(server.url_for("/v1/embeddings"), json=body) + resp.raise_for_status() + return [item["embedding"] for item in resp.json()["data"]] + + +def _cosine_sim(a: list[float], b: list[float]) -> float: + va, vb = np.array(a), np.array(b) + return float(np.dot(va, vb) / (np.linalg.norm(va) * np.linalg.norm(vb))) + + +def test_single_text_parity(server: RemoteOpenAIServer): + """A single text should produce equivalent embeddings via both APIs.""" + texts = ["the quick brown fox jumps over the lazy dog"] + v2 = _cohere_embed(server, texts) + v1 = _openai_embed(server, texts) + # Full-suite BF16 runs can introduce tiny numerical drift even when both + # endpoints are functionally equivalent, so compare semantic equivalence + # instead of exact elementwise equality. + cos = _cosine_sim(v2[0], v1[0]) + assert cos > 0.9999, f"single-text parity failed, cosine={cos}" + + +def test_batch_parity(server: RemoteOpenAIServer): + """A batch of texts should produce equivalent embeddings via both APIs, + in the same order.""" + texts = [ + "machine learning", + "deep learning", + "natural language processing", + ] + v2 = _cohere_embed(server, texts) + v1 = _openai_embed(server, texts) + assert len(v2) == len(v1) == 3 + + similarities = np.array( + [[_cosine_sim(v2_emb, v1_emb) for v1_emb in v1] for v2_emb in v2] + ) + for i in range(3): + assert int(np.argmax(similarities[i])) == i, ( + f"batch parity order mismatch at index {i}: " + f"similarities={similarities[i].tolist()}" + ) + assert similarities[i, i] > 0.9999, ( + f"batch parity failed at index {i}, cosine={similarities[i, i]}" + ) + + +def test_token_count_parity(server: RemoteOpenAIServer): + """Both APIs should report the same prompt token count.""" + texts = ["hello world"] + v2_resp = requests.post( + server.url_for("/v2/embed"), + json={ + "model": MODEL_NAME, + "texts": texts, + "embedding_types": ["float"], + }, + ) + v1_resp = requests.post( + server.url_for("/v1/embeddings"), + json={"model": MODEL_NAME, "input": texts, "encoding_format": "float"}, + ) + v2_resp.raise_for_status() + v1_resp.raise_for_status() + v2_tokens = v2_resp.json()["meta"]["billed_units"]["input_tokens"] + v1_tokens = v1_resp.json()["usage"]["prompt_tokens"] + assert v2_tokens == v1_tokens diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_correctness_mteb.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_correctness_mteb.py new file mode 100644 index 0000000000000000000000000000000000000000..c4254a43b93c554c8d734a6628fa72e67c1ae7b8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_correctness_mteb.py @@ -0,0 +1,46 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import os + +import pytest + +from tests.models.language.pooling_mteb_test.mteb_embed_utils import ( + MTEB_EMBED_TASKS, + MTEB_EMBED_TOL, + OpenAIClientMtebEncoder, + run_mteb_embed_task, +) +from tests.utils import RemoteOpenAIServer +from vllm.platforms import current_platform + +os.environ["VLLM_LOGGING_LEVEL"] = "WARNING" + +MODEL_NAME = "intfloat/e5-small" +MAIN_SCORE = 0.7422994752439667 + + +@pytest.fixture(scope="module") +def server(): + args = ["--runner", "pooling", "--enforce-eager", "--disable-uvicorn-access-log"] + + # ROCm: Use Flex Attention to support encoder-only self-attention. + if current_platform.is_rocm(): + args.extend(["--attention-backend", "FLEX_ATTENTION"]) + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +def test_mteb_embed(server): + client = server.get_client() + encoder = OpenAIClientMtebEncoder(MODEL_NAME, client) + vllm_main_score = run_mteb_embed_task(encoder, MTEB_EMBED_TASKS) + st_main_score = MAIN_SCORE + + print("VLLM main score: ", vllm_main_score) + print("SentenceTransformer main score: ", st_main_score) + print("Difference: ", st_main_score - vllm_main_score) + + # We are not concerned that the vllm mteb results are better + # than SentenceTransformers, so we only perform one-sided testing. + assert st_main_score - vllm_main_score < MTEB_EMBED_TOL diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_io_processor.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_io_processor.py new file mode 100644 index 0000000000000000000000000000000000000000..e8fb54f08fae673e8ab851ac01cd60e8c65e03b9 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_io_processor.py @@ -0,0 +1,714 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for EmbedIOProcessor.""" + +import pytest +import torch +from pydantic import TypeAdapter, ValidationError + +from vllm import PoolingParams +from vllm.entrypoints.pooling.embed.io_processor import EmbedIOProcessor +from vllm.entrypoints.pooling.embed.protocol import ( + CohereEmbedContent, + CohereEmbedInput, + CohereEmbedRequest, + EmbeddingBatchChatInputRequest, + EmbeddingBatchChatRequest, + EmbeddingChatInputRequest, + EmbeddingChatRequest, + EmbeddingCompletionRequest, + EmbeddingRequest, +) +from vllm.entrypoints.pooling.typing import PoolingServeContext +from vllm.outputs import PoolingOutput, PoolingRequestOutput + + +class TestEmbeddingRequestParsing: + """Unit tests for OpenAI embedding request parsing.""" + + def test_input_messages_parses_as_chat_request(self): + request = TypeAdapter(EmbeddingRequest).validate_python( + { + "model": "test", + "input": [{"role": "user", "content": "hello"}], + "chat_template_kwargs": {"instruction": "Represent the query: "}, + } + ) + + assert isinstance(request, EmbeddingChatInputRequest) + assert request.input == [{"role": "user", "content": "hello"}] + assert request.messages == [{"role": "user", "content": "hello"}] + assert request.chat_template_kwargs == {"instruction": "Represent the query: "} + + def test_batched_input_messages_parses_as_batch_chat_input_request(self): + request = TypeAdapter(EmbeddingRequest).validate_python( + { + "model": "test", + "input": [ + [{"role": "user", "content": "hello"}], + [{"role": "user", "content": "goodbye"}], + ], + "chat_template_kwargs": {"instruction": "Represent the query: "}, + } + ) + + assert isinstance(request, EmbeddingBatchChatInputRequest) + assert request.input == [ + [{"role": "user", "content": "hello"}], + [{"role": "user", "content": "goodbye"}], + ] + assert request.messages == [ + [{"role": "user", "content": "hello"}], + [{"role": "user", "content": "goodbye"}], + ] + assert request.chat_template_kwargs == {"instruction": "Represent the query: "} + + def test_token_ids_still_parse_as_completion_request(self): + request = TypeAdapter(EmbeddingRequest).validate_python( + { + "model": "test", + "input": [[1, 2, 3], [4, 5]], + } + ) + + assert isinstance(request, EmbeddingCompletionRequest) + assert request.input == [[1, 2, 3], [4, 5]] + + def test_messages_still_parses_as_chat_request(self): + request = TypeAdapter(EmbeddingRequest).validate_python( + { + "model": "test", + "messages": [{"role": "user", "content": "hello"}], + "chat_template_kwargs": {"instruction": "Represent the query: "}, + } + ) + + assert isinstance(request, EmbeddingChatRequest) + assert request.messages == [{"role": "user", "content": "hello"}] + assert request.chat_template_kwargs == {"instruction": "Represent the query: "} + + def test_batched_messages_parses_as_batch_chat_request(self): + request = TypeAdapter(EmbeddingRequest).validate_python( + { + "model": "test", + "messages": [ + [{"role": "user", "content": "hello"}], + [{"role": "user", "content": "goodbye"}], + ], + "chat_template_kwargs": {"instruction": "Represent the query: "}, + } + ) + + assert isinstance(request, EmbeddingBatchChatRequest) + assert request.messages == [ + [{"role": "user", "content": "hello"}], + [{"role": "user", "content": "goodbye"}], + ] + assert request.chat_template_kwargs == {"instruction": "Represent the query: "} + + +class TestCohereEmbedRequestParsing: + """Unit tests for Cohere embed request parsing.""" + + @pytest.mark.parametrize( + "request_body", + [ + {"model": "test"}, + {"model": "test", "texts": ["hello"], "images": ["image-uri"]}, + { + "model": "test", + "texts": ["hello"], + "inputs": [ + {"content": [{"type": "text", "text": "hello"}]}, + ], + }, + { + "model": "test", + "images": ["image-uri"], + "inputs": [ + {"content": [{"type": "text", "text": "hello"}]}, + ], + }, + {"model": "test", "texts": []}, + {"model": "test", "images": []}, + {"model": "test", "inputs": []}, + ], + ) + def test_rejects_invalid_input_field_combinations(self, request_body): + with pytest.raises( + ValidationError, + match="Exactly one of texts, images, or inputs must be provided", + ): + CohereEmbedRequest(**request_body) + + @pytest.mark.parametrize( + "request_body", + [ + {"model": "test", "texts": ["hello"]}, + {"model": "test", "images": ["image-uri"]}, + { + "model": "test", + "inputs": [ + {"content": [{"type": "text", "text": "hello"}]}, + ], + }, + { + "model": "test", + "inputs": [ + { + "content": [ + {"type": "image_url", "image_url": {"url": "image-uri"}} + ] + }, + ], + }, + ], + ) + def test_accepts_exactly_one_non_empty_input_field(self, request_body): + request = CohereEmbedRequest(**request_body) + + assert request.model == "test" + + @pytest.mark.parametrize( + ("content", "error"), + [ + ( + {"type": "text"}, + "CohereEmbedContent with type='text' requires text", + ), + ( + {"type": "image_url"}, + "CohereEmbedContent with type='image_url' requires image_url.url", + ), + ( + {"type": "image_url", "image_url": {}}, + "CohereEmbedContent with type='image_url' requires image_url.url", + ), + ( + {"type": "image_url", "image_url": {"url": ""}}, + "CohereEmbedContent with type='image_url' requires image_url.url", + ), + ], + ) + def test_rejects_invalid_mixed_content_payloads(self, content, error): + with pytest.raises(ValidationError, match=error): + CohereEmbedRequest( + model="test", + inputs=[ + { + "content": [content], + }, + ], + ) + + +class TestResolveTruncation: + """Unit tests for EmbedIOProcessor._resolve_cohere_truncation.""" + + @staticmethod + def _make_request(**kwargs) -> CohereEmbedRequest: + defaults = { + "model": "test", + "input_type": "search_document", + "texts": ["hello"], + } + return CohereEmbedRequest(**(defaults | kwargs)) + + def test_truncate_end_default(self): + req = self._make_request() + tokens, side = EmbedIOProcessor._resolve_cohere_truncation(req) + assert tokens == -1 + assert side is None + + def test_truncate_end_explicit(self): + req = self._make_request(truncate="END") + tokens, side = EmbedIOProcessor._resolve_cohere_truncation(req) + assert tokens == -1 + assert side is None + + def test_truncate_end_with_max_tokens(self): + req = self._make_request(truncate="END", max_tokens=128) + tokens, side = EmbedIOProcessor._resolve_cohere_truncation(req) + assert tokens == 128 + assert side is None + + def test_truncate_none(self): + req = self._make_request(truncate="NONE") + tokens, side = EmbedIOProcessor._resolve_cohere_truncation(req) + assert tokens is None + assert side is None + + def test_truncate_none_with_max_tokens(self): + """truncate=NONE should NOT set truncate_prompt_tokens; the + max_tokens limit is enforced separately via _check_max_tokens.""" + req = self._make_request(truncate="NONE", max_tokens=10) + tokens, side = EmbedIOProcessor._resolve_cohere_truncation(req) + assert tokens is None + assert side is None + + def test_truncate_start(self): + req = self._make_request(truncate="START") + tokens, side = EmbedIOProcessor._resolve_cohere_truncation(req) + assert tokens == -1 + assert side == "left" + + def test_truncate_start_with_max_tokens(self): + req = self._make_request(truncate="START", max_tokens=64) + tokens, side = EmbedIOProcessor._resolve_cohere_truncation(req) + assert tokens == 64 + assert side == "left" + + +class TestApplyStPrompt: + """Unit tests for EmbedIOProcessor._apply_task_instruction.""" + + @staticmethod + def _make_handler(task_instructions: dict[str, str] | None): + handler = object.__new__(EmbedIOProcessor) + handler.task_instructions = task_instructions + return handler + + def test_no_prompts_configured(self): + handler = self._make_handler(None) + texts = ["hello", "world"] + assert handler._apply_task_instruction(texts, "query") is texts + + def test_matching_input_type(self): + handler = self._make_handler({"query": "search_query: "}) + result = handler._apply_task_instruction(["hello"], "query") + assert result == ["search_query: hello"] + + def test_non_matching_input_type(self): + handler = self._make_handler({"query": "search_query: "}) + texts = ["hello"] + assert handler._apply_task_instruction(texts, "document") is texts + + def test_multiple_texts(self): + handler = self._make_handler( + {"query": "Represent this sentence for searching: "} + ) + result = handler._apply_task_instruction(["a", "b", "c"], "query") + assert result == [ + "Represent this sentence for searching: a", + "Represent this sentence for searching: b", + "Represent this sentence for searching: c", + ] + + def test_empty_prefix_returns_unchanged(self): + handler = self._make_handler({"passage": ""}) + texts = ["hello"] + assert handler._apply_task_instruction(texts, "passage") is texts + + +class TestLoadTaskInstructions: + """Unit tests for EmbedIOProcessor._load_task_instructions.""" + + def test_no_attribute(self): + class FakeConfig: + pass + + assert EmbedIOProcessor._load_task_instructions(FakeConfig()) is None + + def test_with_task_instructions(self): + class FakeConfig: + task_instructions = { + "retrieval.query": "Represent the query: ", + "retrieval.passage": "", + } + + result = EmbedIOProcessor._load_task_instructions(FakeConfig()) + assert result == { + "retrieval.query": "Represent the query: ", + "retrieval.passage": "", + } + + def test_empty_dict(self): + class FakeConfig: + task_instructions = {} + + assert EmbedIOProcessor._load_task_instructions(FakeConfig()) is None + + def test_non_dict(self): + class FakeConfig: + task_instructions = "not a dict" + + assert EmbedIOProcessor._load_task_instructions(FakeConfig()) is None + + +class TestCheckMaxTokens: + """Unit tests for EmbedIOProcessor._check_cohere_max_tokens.""" + + @staticmethod + def _fake_output(n_tokens: int): + class _Out: + def __init__(self, n: int): + self.prompt_token_ids = list(range(n)) + + return _Out(n_tokens) + + def test_none_check_is_noop(self): + outs = [self._fake_output(100)] + EmbedIOProcessor._check_cohere_max_tokens(outs, None) + + def test_within_limit(self): + outs = [self._fake_output(5), self._fake_output(3)] + EmbedIOProcessor._check_cohere_max_tokens(outs, 5) + + def test_exceeds_limit(self): + outs = [self._fake_output(3), self._fake_output(10)] + with pytest.raises(ValueError, match="exceeds max_tokens=5"): + EmbedIOProcessor._check_cohere_max_tokens(outs, 5) + + def test_exact_limit(self): + outs = [self._fake_output(5)] + EmbedIOProcessor._check_cohere_max_tokens(outs, 5) + + +class TestValidateInputType: + """Unit tests for EmbedIOProcessor._validate_input_type.""" + + @staticmethod + def _make_handler(task_instructions: dict[str, str] | None): + handler = object.__new__(EmbedIOProcessor) + handler.task_instructions = task_instructions + return handler + + def test_none_input_type_always_accepted(self): + handler = self._make_handler(None) + handler._validate_input_type(None) + handler_with = self._make_handler({"query": "q: "}) + handler_with._validate_input_type(None) + + def test_no_prompts_rejects(self): + handler = self._make_handler(None) + with pytest.raises(ValueError, match="does not define any input_type"): + handler._validate_input_type("anything") + + def test_known_type_accepted(self): + handler = self._make_handler({"query": "q: ", "document": "d: "}) + handler._validate_input_type("query") + handler._validate_input_type("document") + + def test_unknown_type_rejected(self): + handler = self._make_handler({"query": "q: ", "document": "d: "}) + with pytest.raises(ValueError, match="Unsupported input_type 'other'"): + handler._validate_input_type("other") + + def test_error_lists_supported(self): + handler = self._make_handler({"a": "", "b": ""}) + with pytest.raises(ValueError, match="Supported values: a, b"): + handler._validate_input_type("z") + + +class TestChunkedEmbeddingProcessing: + """Unit tests for chunked embedding aggregation.""" + + class _FakeModelConfig: + max_model_len = 3 + + @classmethod + def _make_handler(cls): + handler = object.__new__(EmbedIOProcessor) + handler.model_config = cls._FakeModelConfig() + return handler + + @staticmethod + def _make_context() -> PoolingServeContext[EmbeddingCompletionRequest]: + request = TypeAdapter(EmbeddingRequest).validate_python( + { + "model": "test", + "input": [[0, 1, 2, 3, 4], [10, 11]], + } + ) + assert isinstance(request, EmbeddingCompletionRequest) + return PoolingServeContext( + request=request, + pooling_params=PoolingParams(), + model_name="test", + request_id="embd-client-prompt-999-chunk-888", + engine_inputs=[ + {"prompt_token_ids": [0, 1, 2, 3, 4]}, + {"prompt_token_ids": [10, 11]}, + ], + ) + + @staticmethod + def _make_output( + request_id: str, + prompt_token_ids: list[int], + embedding: list[float], + ) -> PoolingRequestOutput: + return PoolingRequestOutput( + request_id=request_id, + outputs=PoolingOutput(data=torch.tensor(embedding)), + prompt_token_ids=prompt_token_ids, + num_cached_tokens=0, + finished=True, + ) + + def test_aggregation_uses_metadata_not_request_id_parsing(self): + handler = self._make_handler() + ctx = self._make_context() + + handler._pre_process_chunked(ctx) + + assert ctx.prompt_request_ids == [ + "embd-client-prompt-999-chunk-888-prompt-0-chunk-0", + "embd-client-prompt-999-chunk-888-prompt-0-chunk-1", + "embd-client-prompt-999-chunk-888-prompt-1-chunk-0", + ] + assert ctx.chunked_embedding_metadata is not None + assert [ + (item.prompt_index, item.chunk_index) + for item in ctx.chunked_embedding_metadata + ] == [(0, 0), (0, 1), (1, 0)] + + ctx.final_res_batch = [ + self._make_output(ctx.prompt_request_ids[0], [0, 1, 2], [1.0, 1.0]), + self._make_output(ctx.prompt_request_ids[1], [3, 4], [4.0, 7.0]), + self._make_output(ctx.prompt_request_ids[2], [10, 11], [9.0, 9.0]), + ] + + handler._post_process_chunked(ctx) + + assert len(ctx.final_res_batch) == 2 + assert ctx.final_res_batch[0].request_id == ( + "embd-client-prompt-999-chunk-888-prompt-0" + ) + assert ctx.final_res_batch[0].prompt_token_ids == [0, 1, 2, 3, 4] + assert torch.allclose( + ctx.final_res_batch[0].outputs.data, + torch.tensor([2.2, 3.4]), + ) + assert ctx.final_res_batch[1].request_id == ( + "embd-client-prompt-999-chunk-888-prompt-1" + ) + assert ctx.final_res_batch[1].prompt_token_ids == [10, 11] + assert torch.allclose( + ctx.final_res_batch[1].outputs.data, + torch.tensor([9.0, 9.0]), + ) + + +class TestPreProcessCohereOnline: + """Unit tests for EmbedIOProcessor._pre_process_cohere_online.""" + + @staticmethod + def _make_context(**request_kwargs) -> PoolingServeContext[CohereEmbedRequest]: + return PoolingServeContext( + request=CohereEmbedRequest(model="test", **request_kwargs), + pooling_params=PoolingParams(), + model_name="test", + request_id="embd-test", + ) + + @staticmethod + def _make_handler(): + handler = object.__new__(EmbedIOProcessor) + handler._validate_input_type = lambda _input_type: None + return handler + + def test_text_only_without_task_prefix_uses_completion_path(self): + handler = self._make_handler() + ctx = self._make_context(texts=["hello"]) + calls: list[tuple[str, object]] = [] + + def preprocess_cmpl_online(request, prompt_input, prompt_embeds): + calls.append(("completion", prompt_input)) + return ["completion"] + + handler._get_task_instruction_prefix = lambda _input_type: None + handler._has_chat_template = lambda: False + handler._preprocess_cmpl_online = preprocess_cmpl_online + handler._batch_render_chat = lambda *_args, **_kwargs: pytest.fail( + "text-only request should not require chat rendering" + ) + + handler._pre_process_cohere_online(ctx) + + assert ctx.engine_inputs == ["completion"] + assert calls == [("completion", ["hello"])] + + def test_text_only_falls_back_to_prefixed_completion_without_template(self): + handler = self._make_handler() + ctx = self._make_context(texts=["hello"], input_type="query") + calls: list[tuple[str, object]] = [] + + def preprocess_cmpl(request, prompt_input, prompt_embeds): + calls.append(("completion", prompt_input)) + return ["fallback"] + + handler._get_task_instruction_prefix = lambda _input_type: "query: " + handler._has_chat_template = lambda: False + handler._batch_render_chat = lambda *_args, **_kwargs: pytest.fail( + "chat rendering should be skipped without a template" + ) + handler._preprocess_cmpl_online = preprocess_cmpl + + handler._pre_process_cohere_online(ctx) + + assert ctx.engine_inputs == ["fallback"] + assert calls == [("completion", ["query: hello"])] + + def test_text_only_with_template_uses_chat_path(self): + handler = self._make_handler() + ctx = self._make_context(texts=["hello"], input_type="query") + calls: list[tuple[str, object]] = [] + + def batch_render_chat( + request, + all_messages, + truncate_prompt_tokens, + truncation_side, + ): + calls.append( + ( + "chat", + { + "request": request, + "all_messages": all_messages, + "truncate_prompt_tokens": truncate_prompt_tokens, + "truncation_side": truncation_side, + }, + ) + ) + return ["chat"] + + handler._get_task_instruction_prefix = lambda _input_type: "query: " + handler._has_chat_template = lambda: True + handler._batch_render_chat = batch_render_chat + handler._preprocess_cmpl_online = lambda *_args, **_kwargs: pytest.fail( + "completion path should be skipped when a template exists" + ) + + handler._pre_process_cohere_online(ctx) + + assert ctx.engine_inputs == ["chat"] + assert calls == [ + ( + "chat", + { + "request": ctx.request, + "all_messages": [ + handler._mixed_input_to_messages( + CohereEmbedInput( + content=[CohereEmbedContent(type="text", text="hello")] + ), + task_prefix="query: ", + ) + ], + "truncate_prompt_tokens": -1, + "truncation_side": None, + }, + ) + ] + + +class TestPreProcessOpenAIEmbeddingChatOnline: + """Unit tests for OpenAI embedding chat preprocessing.""" + + class _FakeModelConfig: + max_model_len = 128 + encoder_config: dict[str, object] = {} + pooler_config = None + multimodal_config = None + is_encoder_decoder = False + + class _FakeRenderer: + tokenizer = object() + + def __init__(self): + self.calls = [] + + def render_chat( + self, + all_messages, + chat_params, + tok_params, + prompt_extras=None, + ): + self.calls.append( + { + "all_messages": all_messages, + "chat_params": chat_params, + "tok_params": tok_params, + "prompt_extras": prompt_extras, + } + ) + return all_messages, [ + {"prompt_token_ids": [index]} for index, _ in enumerate(all_messages) + ] + + @classmethod + def _make_handler(cls, renderer): + handler = object.__new__(EmbedIOProcessor) + handler.renderer = renderer + handler.model_config = cls._FakeModelConfig() + handler.chat_template = "template" + handler.chat_template_content_format = "auto" + handler.trust_request_chat_template = False + handler.enable_chunked_processing = False + return handler + + @staticmethod + def _make_context( + request: ( + EmbeddingChatRequest + | EmbeddingBatchChatRequest + | EmbeddingChatInputRequest + | EmbeddingBatchChatInputRequest + ), + ) -> PoolingServeContext[ + EmbeddingChatRequest + | EmbeddingBatchChatRequest + | EmbeddingChatInputRequest + | EmbeddingBatchChatInputRequest + ]: + return PoolingServeContext( + request=request, + pooling_params=PoolingParams(), + model_name="test", + request_id="embd-test", + ) + + def test_chat_template_kwargs_forwarded_for_batched_input_messages(self): + request = TypeAdapter(EmbeddingRequest).validate_python( + { + "model": "test", + "input": [ + [{"role": "user", "content": "hello"}], + [{"role": "user", "content": "goodbye"}], + ], + "add_generation_prompt": True, + "chat_template_kwargs": {"instruction": "Represent the query: "}, + "mm_processor_kwargs": {"max_pixels": 1}, + "cache_salt": "salt", + } + ) + assert isinstance(request, EmbeddingBatchChatInputRequest) + + renderer = self._FakeRenderer() + handler = self._make_handler(renderer) + ctx = self._make_context(request) + + handler.pre_process_online(ctx) + + assert ctx.engine_inputs == [ + {"prompt_token_ids": [0]}, + {"prompt_token_ids": [1]}, + ] + assert len(renderer.calls) == 1 + + call = renderer.calls[0] + assert call["all_messages"] == request.messages + assert call["prompt_extras"] == { + "mm_processor_kwargs": {"max_pixels": 1}, + "cache_salt": "salt", + } + + chat_template_kwargs = call["chat_params"].chat_template_kwargs + assert chat_template_kwargs["instruction"] == "Represent the query: " + assert chat_template_kwargs["add_generation_prompt"] is True + assert chat_template_kwargs["continue_final_message"] is False + assert "tools" not in chat_template_kwargs + assert chat_template_kwargs["tokenize"] is False diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_offline.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_offline.py new file mode 100644 index 0000000000000000000000000000000000000000..9ef22bd16d6dd250ec5be69fa6932761adb99c8b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_offline.py @@ -0,0 +1,110 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import weakref + +import pytest +import torch +import torch.nn.functional as F + +from vllm import LLM, EmbeddingRequestOutput, PoolingParams +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.platforms import current_platform +from vllm.tasks import PoolingTask + +MODEL_NAME = "intfloat/multilingual-e5-small" + +prompt = "The chef prepared a delicious meal." +prompt_token_ids = [0, 581, 21861, 133888, 10, 8, 150, 60744, 109911, 5, 2] +embedding_size = 384 + + +@pytest.fixture(scope="module") +def llm(): + # ROCm: Use FLEX_ATTENTION backend as it's the only attention backend + # that supports encoder-only models on ROCm. + attention_config = None + if current_platform.is_rocm(): + attention_config = {"backend": "FLEX_ATTENTION"} + + # pytest caches the fixture so we use weakref.proxy to + # enable garbage collection + llm = LLM( + model=MODEL_NAME, + max_num_batched_tokens=32768, + tensor_parallel_size=1, + gpu_memory_utilization=0.75, + enforce_eager=True, + seed=0, + attention_config=attention_config, + ) + assert embedding_size == llm.model_config.embedding_size + + yield weakref.proxy(llm) + + del llm + cleanup_dist_env_and_memory() + + +@pytest.mark.skip_global_cleanup +def test_str_prompts(llm: LLM): + outputs = llm.embed(prompt, use_tqdm=False) + assert len(outputs) == 1 + assert isinstance(outputs[0], EmbeddingRequestOutput) + assert outputs[0].prompt_token_ids == prompt_token_ids + assert len(outputs[0].outputs.embedding) == embedding_size + + +@pytest.mark.skip_global_cleanup +def test_token_ids_prompts(llm: LLM): + outputs = llm.embed([prompt_token_ids], use_tqdm=False) + assert len(outputs) == 1 + assert isinstance(outputs[0], EmbeddingRequestOutput) + assert outputs[0].prompt_token_ids == prompt_token_ids + assert len(outputs[0].outputs.embedding) == embedding_size + + +@pytest.mark.skip_global_cleanup +def test_list_prompts(llm: LLM): + outputs = llm.embed([prompt, prompt_token_ids], use_tqdm=False) + assert len(outputs) == 2 + for i in range(len(outputs)): + assert isinstance(outputs[i], EmbeddingRequestOutput) + assert outputs[i].prompt_token_ids == prompt_token_ids + assert len(outputs[i].outputs.embedding) == embedding_size + + +@pytest.mark.skip_global_cleanup +def test_pooling_params(llm: LLM): + def get_outputs(normalize): + outputs = llm.embed( + [prompt], + pooling_params=PoolingParams(use_activation=normalize), + use_tqdm=False, + ) + return torch.tensor([x.outputs.embedding for x in outputs]) + + default = get_outputs(normalize=None) + w_normal = get_outputs(normalize=True) + wo_normal = get_outputs(normalize=False) + + assert torch.allclose(default, w_normal, atol=1e-2), "Default should use normal." + assert not torch.allclose(w_normal, wo_normal, atol=1e-2), ( + "wo_normal should not use normal." + ) + assert torch.allclose(w_normal, F.normalize(wo_normal, p=2, dim=-1), atol=1e-2), ( + "w_normal should be close to normal(wo_normal)." + ) + + +@pytest.mark.parametrize( + "task", ["token_classify", "classify", "token_embed", "plugin"] +) +def test_unsupported_tasks(llm: LLM, task: PoolingTask): + if task == "plugin": + err_msg = "No IOProcessor plugin installed." + elif task == "token_embed": + err_msg = "Try switching the model's pooling_task via.+" + else: + err_msg = "Classification API is not supported by this model.+" + with pytest.raises(ValueError, match=err_msg): + llm.encode(prompt, pooling_task=task, use_tqdm=False) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_online.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_online.py new file mode 100644 index 0000000000000000000000000000000000000000..61102d20ba78d176d661b0af11f1e2a9b3eff818 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_online.py @@ -0,0 +1,757 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import numpy as np +import openai +import pybase64 as base64 +import pytest +import pytest_asyncio +import requests +import torch +import torch.nn.functional as F + +from tests.models.language.pooling.embed_utils import run_embedding_correctness_test +from tests.models.utils import check_embeddings_close +from tests.utils import RemoteOpenAIServer +from vllm.entrypoints.pooling.embed.protocol import EmbeddingResponse +from vllm.entrypoints.pooling.pooling.protocol import PoolingResponse +from vllm.entrypoints.pooling.utils import ( + MetadataItem, + build_metadata_items, + decode_pooling_output, +) +from vllm.platforms import current_platform +from vllm.tokenizers import get_tokenizer +from vllm.utils.serial_utils import EMBED_DTYPES, ENDIANNESS, binary2tensor + +MODEL_NAME = "intfloat/multilingual-e5-small" +DUMMY_CHAT_TEMPLATE = """{% for message in messages %}{{message['role'] + ': ' + message['content'] + '\\n'}}{% endfor %}""" # noqa: E501 +DTYPE = "bfloat16" +input_text = "The best thing about vLLM is that it supports many different models" +input_tokens = [ + 0, + 581, + 2965, + 13580, + 1672, + 81, + 23708, + 594, + 83, + 450, + 442, + 8060, + 7, + 5941, + 12921, + 115774, + 2, +] + +if current_platform.is_rocm(): + # Disable Flash/MemEfficient SDP on ROCm to avoid HF Transformers + # accuracy issues: https://github.com/vllm-project/vllm/issues/30167 + # TODO: Remove once ROCm SDP accuracy issues are resolved on HuggingFace + torch.backends.cuda.enable_flash_sdp(False) + torch.backends.cuda.enable_mem_efficient_sdp(False) + torch.backends.cuda.enable_math_sdp(True) + +# On ROCm, floating-point reductions in attention and GEMM kernels are +# non-associative and sensitive to batch geometry. Force LLM instances +# into an identical, deterministic execution mode: +ROCM_DETERMINISM_ARGS: list[str] = ( + ["--max-num-seqs", "1"] if current_platform.is_rocm() else [] +) + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--runner", + "pooling", + "--dtype", + DTYPE, + "--enforce-eager", + "--max-model-len", + "512", + "--chat-template", + DUMMY_CHAT_TEMPLATE, + *ROCM_DETERMINISM_ARGS, + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.fixture(scope="module") +def hf_model(hf_runner): + with hf_runner(MODEL_NAME, dtype=DTYPE, is_sentence_transformer=True) as hf_model: + yield hf_model + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_basic( + server: RemoteOpenAIServer, client: openai.AsyncOpenAI, model_name: str +): + # test /v1/models + response = requests.get(server.url_for("/v1/models")) + model = response.json()["data"][0]["id"] + assert model == MODEL_NAME + + models = await client.models.list() + models = models.data + served_model = models[0] + assert served_model.id == MODEL_NAME + + # test /tokenize + response = requests.post( + server.url_for("/tokenize"), + json={"model": model_name, "prompt": input_text}, + ) + assert response.json()["tokens"] == input_tokens + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_completion_request( + client: openai.AsyncOpenAI, model_name: str, hf_model +): + # test input: str + embedding_response = await client.embeddings.create( + model=model_name, + input=input_text, + encoding_format="float", + ) + embeddings = EmbeddingResponse.model_validate( + embedding_response.model_dump(mode="json") + ) + + assert embeddings.id is not None + assert len(embeddings.data) == 1 + assert len(embeddings.data[0].embedding) == 384 + assert embeddings.usage.completion_tokens == 0 + assert embeddings.usage.prompt_tokens == len(input_tokens) + assert embeddings.usage.total_tokens == len(input_tokens) + + vllm_outputs = [d.embedding for d in embeddings.data] + run_embedding_correctness_test(hf_model, [input_text], vllm_outputs) + + # test input: list[int] + embedding_response = await client.embeddings.create( + model=model_name, + input=input_tokens, + encoding_format="float", + ) + embeddings = EmbeddingResponse.model_validate( + embedding_response.model_dump(mode="json") + ) + + assert embeddings.id is not None + assert len(embeddings.data) == 1 + assert len(embeddings.data[0].embedding) == 384 + assert embeddings.usage.completion_tokens == 0 + assert embeddings.usage.prompt_tokens == len(input_tokens) + assert embeddings.usage.total_tokens == len(input_tokens) + + vllm_outputs = [d.embedding for d in embeddings.data] + run_embedding_correctness_test(hf_model, [input_text], vllm_outputs) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_completion_request_batched( + client: openai.AsyncOpenAI, model_name: str, hf_model +): + N = 10 + input_texts = [input_text] * N + + # test input: list[str] + embedding_response = await client.embeddings.create( + model=model_name, + input=input_texts, + encoding_format="float", + ) + embeddings = EmbeddingResponse.model_validate( + embedding_response.model_dump(mode="json") + ) + + assert embeddings.id is not None + assert len(embeddings.data) == N + assert len(embeddings.data[0].embedding) == 384 + assert embeddings.usage.completion_tokens == 0 + assert embeddings.usage.prompt_tokens == len(input_tokens) * N + assert embeddings.usage.total_tokens == len(input_tokens) * N + + vllm_outputs = [d.embedding for d in embeddings.data] + run_embedding_correctness_test(hf_model, input_texts, vllm_outputs) + + # test list[list[int]] + embedding_response = await client.embeddings.create( + model=model_name, + input=[input_tokens] * N, + encoding_format="float", + ) + embeddings = EmbeddingResponse.model_validate( + embedding_response.model_dump(mode="json") + ) + + assert embeddings.id is not None + assert len(embeddings.data) == N + assert len(embeddings.data[0].embedding) == 384 + assert embeddings.usage.completion_tokens == 0 + assert embeddings.usage.prompt_tokens == len(input_tokens) * N + assert embeddings.usage.total_tokens == len(input_tokens) * N + + vllm_outputs = [d.embedding for d in embeddings.data] + run_embedding_correctness_test(hf_model, input_texts, vllm_outputs) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_truncate_prompt_tokens(client: openai.AsyncOpenAI, model_name: str): + input_texts = [ + "Como o Brasil pode fomentar o desenvolvimento de modelos de IA?", + ] + + # test single embedding + embedding_response = await client.embeddings.create( + model=model_name, input=input_texts, extra_body={"truncate_prompt_tokens": 10} + ) + embeddings = EmbeddingResponse.model_validate( + embedding_response.model_dump(mode="json") + ) + + assert embeddings.id is not None + assert len(embeddings.data) == 1 + assert len(embeddings.data[0].embedding) == 384 + assert embeddings.usage.completion_tokens == 0 + assert embeddings.usage.prompt_tokens == 10 + assert embeddings.usage.total_tokens == 10 + + input_tokens = [ + 1, + 24428, + 289, + 18341, + 26165, + 285, + 19323, + 283, + 289, + 26789, + 3871, + 28728, + 9901, + 340, + 2229, + 385, + 340, + 315, + 28741, + 28804, + 2, + ] + embedding_response = await client.embeddings.create( + model=model_name, input=input_tokens, extra_body={"truncate_prompt_tokens": 10} + ) + embeddings = EmbeddingResponse.model_validate( + embedding_response.model_dump(mode="json") + ) + + assert embeddings.id is not None + assert len(embeddings.data) == 1 + assert len(embeddings.data[0].embedding) == 384 + assert embeddings.usage.completion_tokens == 0 + assert embeddings.usage.prompt_tokens == 10 + assert embeddings.usage.total_tokens == 10 + + # invalid_truncate_prompt_tokens + input_texts = [ + "Como o Brasil pode fomentar o desenvolvimento de modelos de IA?", + ] + + with pytest.raises(openai.BadRequestError): + response = await client.embeddings.create( + model=model_name, + input=input_texts, + extra_body={"truncate_prompt_tokens": 8193}, + ) + assert "error" in response.object + assert ( + "truncate_prompt_tokens value is greater than max_model_len. " + "Please request a smaller truncation size." in response.message + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_chat_request( + server: RemoteOpenAIServer, client: openai.AsyncOpenAI, model_name: str +): + messages = [ + { + "role": "user", + "content": "The cat sat on the mat.", + }, + { + "role": "assistant", + "content": "A feline was resting on a rug.", + }, + { + "role": "user", + "content": "Stars twinkle brightly in the night sky.", + }, + ] + + # test chat request basic usage + chat_response = requests.post( + server.url_for("v1/embeddings"), + json={ + "model": model_name, + "messages": messages, + "encoding_format": "float", + }, + ) + chat_response.raise_for_status() + chat_embeddings = EmbeddingResponse.model_validate(chat_response.json()) + + tokenizer = get_tokenizer(tokenizer_name=model_name) + prompt = tokenizer.apply_chat_template( + messages, + chat_template=DUMMY_CHAT_TEMPLATE, + add_generation_prompt=True, + continue_final_message=False, + tokenize=False, + ) + completion_response = await client.embeddings.create( + model=model_name, + input=prompt, + encoding_format="float", + # To be consistent with chat + extra_body={"add_special_tokens": False}, + ) + completion_embeddings = EmbeddingResponse.model_validate( + completion_response.model_dump(mode="json") + ) + + assert chat_embeddings.id is not None + assert completion_embeddings.id is not None + assert chat_embeddings.created <= completion_embeddings.created + # Use tolerance-based comparison for embeddings + check_embeddings_close( + embeddings_0_lst=[d.embedding for d in chat_embeddings.data], + embeddings_1_lst=[d.embedding for d in completion_embeddings.data], + name_0="chat", + name_1="completion", + ) + assert chat_embeddings.model_dump(exclude={"id", "created", "data"}) == ( + completion_embeddings.model_dump(exclude={"id", "created", "data"}) + ) + + # test add_generation_prompt + response = requests.post( + server.url_for("v1/embeddings"), + json={"model": model_name, "messages": messages, "add_generation_prompt": True}, + ) + + response.raise_for_status() + output = EmbeddingResponse.model_validate(response.json()) + + assert output.object == "list" + assert len(output.data) == 1 + assert output.model == MODEL_NAME + assert output.usage.prompt_tokens == 34 + + # test continue_final_message + response = requests.post( + server.url_for("v1/embeddings"), + json={ + "model": model_name, + "messages": messages, + "continue_final_message": True, + }, + ) + + response.raise_for_status() + output = EmbeddingResponse.model_validate(response.json()) + + assert output.object == "list" + assert len(output.data) == 1 + assert output.model == MODEL_NAME + assert output.usage.prompt_tokens == 33 + + # test add_special_tokens + response = requests.post( + server.url_for("v1/embeddings"), + json={"model": model_name, "messages": messages, "add_special_tokens": True}, + ) + + response.raise_for_status() + output = EmbeddingResponse.model_validate(response.json()) + + assert output.object == "list" + assert len(output.data) == 1 + assert output.model == MODEL_NAME + assert output.usage.prompt_tokens == 36 + + # test continue_final_message with add_generation_prompt + response = requests.post( + server.url_for("v1/embeddings"), + json={ + "model": model_name, + "messages": messages, + "continue_final_message": True, + "add_generation_prompt": True, + }, + ) + assert ( + "Cannot set both `continue_final_message` and `add_generation_prompt` to True." + in response.json()["error"]["message"] + ) + + +@pytest.mark.asyncio +async def test_invocations_completion_request( + server: RemoteOpenAIServer, client: openai.AsyncOpenAI +): + request_args = { + "model": MODEL_NAME, + "input": input_text, + "encoding_format": "float", + } + + completion_response = await client.embeddings.create(**request_args) + + invocation_response = requests.post( + server.url_for("invocations"), json=request_args + ) + invocation_response.raise_for_status() + + completion_output = completion_response.model_dump() + invocation_output = invocation_response.json() + + assert completion_output.keys() == invocation_output.keys() + for completion_data, invocation_data in zip( + completion_output["data"], invocation_output["data"] + ): + assert completion_data.keys() == invocation_data.keys() + check_embeddings_close( + embeddings_0_lst=[completion_data["embedding"]], + embeddings_1_lst=[invocation_data["embedding"]], + name_0="completion", + name_1="invocation", + ) + + +@pytest.mark.asyncio +async def test_invocations_chat_request(server: RemoteOpenAIServer): + messages = [ + { + "role": "user", + "content": "The cat sat on the mat.", + }, + { + "role": "assistant", + "content": "A feline was resting on a rug.", + }, + { + "role": "user", + "content": "Stars twinkle brightly in the night sky.", + }, + ] + + request_args = { + "model": MODEL_NAME, + "messages": messages, + "encoding_format": "float", + } + + chat_response = requests.post(server.url_for("v1/embeddings"), json=request_args) + chat_response.raise_for_status() + + invocation_response = requests.post( + server.url_for("invocations"), json=request_args + ) + invocation_response.raise_for_status() + + chat_output = chat_response.json() + invocation_output = invocation_response.json() + + assert chat_output.keys() == invocation_output.keys() + for chat_data, invocation_data in zip( + chat_output["data"], invocation_output["data"] + ): + assert chat_data.keys() == invocation_data.keys() + check_embeddings_close( + embeddings_0_lst=[chat_data["embedding"]], + embeddings_1_lst=[invocation_data["embedding"]], + name_0="chat", + name_1="invocation", + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_base64_embedding(hf_model, client: openai.AsyncOpenAI, model_name: str): + input_texts = [ + "Hello my name is", + "The best thing about vLLM is that it supports many different models", + ] + + responses_float = await client.embeddings.create( + input=input_texts, model=model_name, encoding_format="float" + ) + float_data = [d.embedding for d in responses_float.data] + run_embedding_correctness_test(hf_model, input_texts, float_data) + + responses_base64 = await client.embeddings.create( + input=input_texts, model=model_name, encoding_format="base64" + ) + base64_data = [] + for data in responses_base64.data: + base64_data.append( + np.frombuffer(base64.b64decode(data.embedding), dtype="float32").tolist() + ) + + run_embedding_correctness_test(hf_model, input_texts, base64_data) + + # Default response is float32 decoded from base64 by OpenAI Client + responses_default = await client.embeddings.create( + input=input_texts, model=model_name + ) + default_data = [d.embedding for d in responses_default.data] + run_embedding_correctness_test(hf_model, input_texts, default_data) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_base64_embed_dtype_and_endianness( + server: RemoteOpenAIServer, client: openai.AsyncOpenAI, model_name: str +): + input_texts = [input_text] * 3 + responses_float = await client.embeddings.create( + input=input_texts, model=model_name, encoding_format="float" + ) + float_data = [d.embedding for d in responses_float.data] + + for embed_dtype in EMBED_DTYPES: + for endianness in ENDIANNESS: + responses_base64 = requests.post( + server.url_for("/v1/embeddings"), + json={ + "model": model_name, + "input": input_texts, + "encoding_format": "base64", + "embed_dtype": embed_dtype, + "endianness": endianness, + }, + ) + + base64_data = [] + for data in responses_base64.json()["data"]: + binary = base64.b64decode(data["embedding"]) + tensor = binary2tensor(binary, (-1,), embed_dtype, endianness) + base64_data.append(tensor.to(torch.float32).tolist()) + + check_embeddings_close( + embeddings_0_lst=float_data, + embeddings_1_lst=base64_data, + name_0="float_data", + name_1="base64_data", + tol=1e-2, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_bytes_embed_dtype_and_endianness( + server: RemoteOpenAIServer, client: openai.AsyncOpenAI, model_name: str +): + input_texts = [input_text] * 3 + responses_float = await client.embeddings.create( + input=input_texts, model=model_name, encoding_format="float" + ) + float_data = [d.embedding for d in responses_float.data] + + for embed_dtype in EMBED_DTYPES: + for endianness in ENDIANNESS: + responses_bytes = requests.post( + server.url_for("/v1/embeddings"), + json={ + "model": model_name, + "input": input_texts, + "encoding_format": "bytes", + "embed_dtype": embed_dtype, + "endianness": endianness, + }, + ) + + metadata = json.loads(responses_bytes.headers["metadata"]) + body = responses_bytes.content + items = [MetadataItem(**x) for x in metadata["data"]] + + bytes_data = decode_pooling_output(items=items, body=body) + bytes_data = [x.to(torch.float32).tolist() for x in bytes_data] + + check_embeddings_close( + embeddings_0_lst=float_data, + embeddings_1_lst=bytes_data, + name_0="float_data", + name_1="bytes_data", + tol=1e-2, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_bytes_only_embed_dtype_and_endianness( + server: RemoteOpenAIServer, client: openai.AsyncOpenAI, model_name: str +): + input_texts = [ + "The best thing about vLLM is that it supports many different models", + ] * 2 + + responses_float = await client.embeddings.create( + input=input_texts, model=model_name, encoding_format="float" + ) + float_data = [d.embedding for d in responses_float.data] + embedding_size = len(float_data[0]) + + for embed_dtype in EMBED_DTYPES: + for endianness in ENDIANNESS: + responses_bytes = requests.post( + server.url_for("/v1/embeddings"), + json={ + "model": model_name, + "input": input_texts, + "encoding_format": "bytes_only", + "embed_dtype": embed_dtype, + "endianness": endianness, + }, + ) + + assert "metadata" not in responses_bytes.headers + body = responses_bytes.content + items = build_metadata_items( + embed_dtype=embed_dtype, + endianness=endianness, + shape=(embedding_size,), + n_request=len(input_texts), + ) + + bytes_data = decode_pooling_output(items=items, body=body) + bytes_data = [x.to(torch.float32).tolist() for x in bytes_data] + + check_embeddings_close( + embeddings_0_lst=float_data, + embeddings_1_lst=bytes_data, + name_0="float_data", + name_1="bytes_data", + tol=1e-2, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("param_name", ["encoding_format", "embed_dtype", "endianness"]) +async def test_params_not_supported( + server: RemoteOpenAIServer, model_name: str, param_name: str +): + responses_base64 = requests.post( + server.url_for("/v1/embeddings"), + json={ + "model": model_name, + "input": input_text, + "encoding_format": "base64", + param_name: f"bad_{param_name}", + }, + ) + + assert responses_base64.status_code == 400 + assert "literal_error" in responses_base64.json()["error"]["message"] + assert f"bad_{param_name}" in responses_base64.json()["error"]["message"] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_use_activation(server: RemoteOpenAIServer, model_name: str): + async def get_outputs(use_activation): + request_args = { + "model": MODEL_NAME, + "input": input_text, + "encoding_format": "float", + "use_activation": use_activation, + } + + response = requests.post(server.url_for("v1/embeddings"), json=request_args) + outputs = response.json() + + return torch.tensor([x["embedding"] for x in outputs["data"]]) + + default = await get_outputs(use_activation=None) + w_normal = await get_outputs(use_activation=True) + wo_normal = await get_outputs(use_activation=False) + + assert torch.allclose(default, w_normal, atol=1e-2), "Default should use normal." + assert not torch.allclose(w_normal, wo_normal, atol=1e-2), ( + "wo_normal should not use normal." + ) + assert torch.allclose(w_normal, F.normalize(wo_normal, p=2, dim=-1), atol=1e-2), ( + "w_normal should be close to normal(wo_normal)." + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_pooling_embed(server: RemoteOpenAIServer, model_name: str): + task = "embed" + response = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "input": input_text, + "encoding_format": "float", + "task": task, + }, + ) + + poolings = PoolingResponse.model_validate(response.json()) + + assert len(poolings.data) == 1 + assert len(poolings.data[0].data) == 384 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize( + "task", ["classify", "token_classify", "token_embed", "plugin"] +) +async def test_pooling_not_supported( + server: RemoteOpenAIServer, model_name: str, task: str +): + response = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "input": "test", + "encoding_format": "float", + "task": task, + }, + ) + assert response.json()["error"]["type"] == "BadRequestError" + if task == "plugin": + err_msg = "No IOProcessor plugin installed." + elif task == "token_embed": + err_msg = "Try switching the model's pooling_task via" + else: + err_msg = f"Unsupported task: {task!r}" + assert response.json()["error"]["message"].startswith(err_msg) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_online_dimensions.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_online_dimensions.py new file mode 100644 index 0000000000000000000000000000000000000000..5daa4126bb184cb753113e3b31419583675f0e03 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_online_dimensions.py @@ -0,0 +1,131 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Run `pytest tests/entrypoints/openai/test_embedding_dimensions.py`. +""" + +import openai +import pytest + +from tests.conftest import HfRunner +from tests.models.language.pooling.embed_utils import run_embedding_correctness_test +from tests.models.utils import EmbedModelInfo +from tests.utils import ROCM_EXTRA_ARGS, RemoteOpenAIServer +from vllm.entrypoints.pooling.embed.protocol import EmbeddingResponse +from vllm.platforms import current_platform + +MODELS = [ + EmbedModelInfo("intfloat/multilingual-e5-small", is_matryoshka=False), + EmbedModelInfo( + "Snowflake/snowflake-arctic-embed-m-v1.5", + is_matryoshka=True, + matryoshka_dimensions=[256], + ), +] + +input_texts = [ + "The chef prepared a delicious meal.", +] + + +@pytest.fixture(scope="module", params=MODELS) +def model_info(request): + return request.param + + +@pytest.fixture(scope="module", params=["bfloat16"]) +def dtype(request): + return request.param + + +@pytest.fixture(scope="module") +def server(model_info, dtype: str): + args = [ + "--runner", + "pooling", + # use half precision for speed and memory savings in CI environment + "--dtype", + dtype, + "--enforce-eager", + "--max-model-len", + "512", + ] + ROCM_EXTRA_ARGS + + if model_info.name == "Snowflake/snowflake-arctic-embed-m-v1.5": + # Manually enable Matryoshka Embeddings + args.extend( + ["--trust_remote_code", "--hf_overrides", '{"matryoshka_dimensions":[256]}'] + ) + + # ROCm: Use Flex Attention to support encoder-only self-attention. + if current_platform.is_rocm(): + args.extend(["--attention-backend", "FLEX_ATTENTION"]) + + with RemoteOpenAIServer(model_info.name, args) as remote_server: + yield remote_server + + +@pytest.fixture(scope="module") +def hf_model(hf_runner, model_info, dtype: str): + with hf_runner( + model_info.name, dtype=dtype, is_sentence_transformer=True + ) as hf_model: + yield hf_model + + +@pytest.mark.asyncio +async def test_matryoshka( + model_info: EmbedModelInfo, server: RemoteOpenAIServer, hf_model: HfRunner +): + client = server.get_async_client() + + async def make_request_and_correctness_test(dimensions): + prompts = input_texts * 3 + + embedding_response = await client.embeddings.create( + model=model_info.name, + input=prompts, + dimensions=dimensions, + encoding_format="float", + ) + embeddings = EmbeddingResponse.model_validate( + embedding_response.model_dump(mode="json") + ) + + assert embeddings.id is not None + assert len(embeddings.data) == 3 + assert len(embeddings.data[0].embedding) > 0 + assert embeddings.usage.completion_tokens == 0 + assert embeddings.usage.prompt_tokens > 0 + assert embeddings.usage.total_tokens > 0 + + if dimensions is not None: + assert len(embeddings.data[0].embedding) == dimensions + + vllm_outputs = [d.embedding for d in embeddings.data] + run_embedding_correctness_test(hf_model, prompts, vllm_outputs, dimensions) + + if model_info.is_matryoshka: + valid_dimensions: list[int | None] = [None] + if model_info.matryoshka_dimensions is not None: + valid_dimensions += model_info.matryoshka_dimensions[:2] + + for dimensions in valid_dimensions: + await make_request_and_correctness_test(dimensions) + + invalid_dimensions: list[int | None] = [-1] + if model_info.matryoshka_dimensions is not None: + assert 5 not in model_info.matryoshka_dimensions + invalid_dimensions.append(5) + + for dimensions in invalid_dimensions: + with pytest.raises(openai.BadRequestError): + await make_request_and_correctness_test(dimensions) + + else: + for dimensions in [None]: + await make_request_and_correctness_test(dimensions) + + for dimensions in [-1, 16]: + with pytest.raises(openai.BadRequestError): + await make_request_and_correctness_test(dimensions) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_online_long_text.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_online_long_text.py new file mode 100644 index 0000000000000000000000000000000000000000..915a1f414b5bb588e22b8958c1c635e3ce929e86 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_online_long_text.py @@ -0,0 +1,457 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Test cases for long text embedding with automatic chunking mechanism. + +This test suite validates vLLM's automatic chunking functionality for handling +text inputs that exceed the model's maximum token length, specifically targeting +the intfloat/multilingual-e5-small model (max token length: 512). +""" + +import random + +import openai +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer +from vllm.entrypoints.pooling.embed.protocol import EmbeddingResponse +from vllm.platforms import current_platform + + +def _generate_random_text(word_count: int) -> str: + """Generate random text with approximately the specified word count.""" + # Common English words with focus on verbs and nouns for realistic text + common_words = [ + # Essential articles and pronouns (minimal) + "the", + "and", + "you", + "they", + "this", + "that", + "these", + "those", + # Action verbs + "create", + "build", + "develop", + "design", + "implement", + "execute", + "analyze", + "process", + "generate", + "calculate", + "evaluate", + "optimize", + "transform", + "integrate", + "configure", + "deploy", + "monitor", + "manage", + "discover", + "explore", + "investigate", + "research", + "study", + "examine", + "improve", + "enhance", + "upgrade", + "modify", + "update", + "maintain", + "solve", + "resolve", + "handle", + "address", + "tackle", + "overcome", + "communicate", + "collaborate", + "coordinate", + "organize", + "plan", + "achieve", + "accomplish", + "complete", + "finish", + "deliver", + "provide", + # Technology and science nouns + "system", + "application", + "software", + "hardware", + "network", + "database", + "algorithm", + "model", + "framework", + "platform", + "interface", + "protocol", + "architecture", + "infrastructure", + "component", + "module", + "service", + "technology", + "innovation", + "solution", + "methodology", + "approach", + "artificial", + "intelligence", + "machine", + "learning", + "neural", + "network", + "computer", + "processor", + "memory", + "storage", + "computation", + "data", + "information", + "knowledge", + "insight", + "pattern", + "trend", + "analysis", + "research", + "development", + "engineering", + "science", + "mathematics", + "statistics", + "probability", + "optimization", + "performance", + "efficiency", + # General nouns + "project", + "team", + "organization", + "company", + "business", + "industry", + "market", + "customer", + "user", + "client", + "product", + "feature", + "function", + "requirement", + "specification", + "documentation", + "report", + "result", + "outcome", + "impact", + "benefit", + "advantage", + "challenge", + "problem", + "opportunity", + "strategy", + "goal", + "objective", + "target", + "milestone", + "process", + "procedure", + "workflow", + "pipeline", + "operation", + "task", + "activity", + "event", + "session", + "meeting", + "discussion", + "decision", + ] + + words = [] + for _ in range(word_count): + words.append(random.choice(common_words)) + + # Add some punctuation for more realistic text + text = " ".join(words) + # Add periods every 10-20 words + words_list = text.split() + result = [] + for i, word in enumerate(words_list): + result.append(word) + if (i + 1) % random.randint(10, 20) == 0 and i < len(words_list) - 1: + result[-1] += "." + + return " ".join(result) + + +MODEL_NAME = "intfloat/multilingual-e5-small" +DTYPE = "bfloat16" + +# Test text: Generate text with approximately 1500 words to exceed 1024 tokens +LONG_TEXT_1500_WORDS = _generate_random_text(1500) + +# Test text: Generate text with approximately 2500 words to exceed 2048 tokens +LONG_TEXT_2500_WORDS = _generate_random_text(2500) + + +@pytest.fixture(scope="module") +def server_with_chunked_processing(): + """Start server with automatic chunking processing enabled.""" + args = [ + "--runner", + "pooling", + "--dtype", + DTYPE, + "--enforce-eager", + "--max-model-len", + "512", # Set smaller max_model_len to trigger chunking mechanism + "--pooler-config", + ( + '{"pooling_type": "MEAN", "use_activation": true, ' + '"enable_chunked_processing": true, "max_embed_len": 10000}' + ), + "--gpu-memory-utilization", + "0.8", + ] + + # ROCm: Use Flex Attention to support encoder-only self-attention. + if current_platform.is_rocm(): + args.extend(["--attention-backend", "FLEX_ATTENTION"]) + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client_with_chunked_processing(server_with_chunked_processing): + """Create async client with chunking processing support.""" + async with server_with_chunked_processing.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_long_text_embedding_1500_chars( + client_with_chunked_processing: openai.AsyncOpenAI, model_name: str +): + """Test embedding processing for ~1500 character long text + (~1028 tokens, exceeding 512 token limit).""" + + # Verify text length + # Verify text has sufficient word count (approximately 1500 words) + word_count = len(LONG_TEXT_1500_WORDS.split()) + assert word_count >= 1400, f"Test text word count insufficient: {word_count} words" + + # Send embedding request + embedding_response = await client_with_chunked_processing.embeddings.create( + model=model_name, + input=[LONG_TEXT_1500_WORDS], + encoding_format="float", + ) + + # Verify response structure + embeddings = EmbeddingResponse.model_validate( + embedding_response.model_dump(mode="json") + ) + + assert embeddings.id is not None + assert len(embeddings.data) == 1 + assert ( + len(embeddings.data[0].embedding) == 384 + ) # multilingual-e5-small embedding dimension + assert embeddings.usage.completion_tokens == 0 + # Due to chunked processing, token count should + # reflect actual processed tokens + # With ~1500 words, we expect roughly + # 1024+ tokens (exceeding 512 token limit) + # Should exceed single chunk limit of 512 + assert embeddings.usage.prompt_tokens > 800 + assert embeddings.usage.total_tokens == embeddings.usage.prompt_tokens + + # Verify embedding vector validity + embedding_vector = embeddings.data[0].embedding + assert all(isinstance(x, float) for x in embedding_vector), ( + "Embedding vector should contain floats" + ) + assert not all(x == 0 for x in embedding_vector), ( + "Embedding vector should not be all zeros" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_long_text_embedding_2500_chars( + client_with_chunked_processing: openai.AsyncOpenAI, model_name: str +): + """Test embedding processing for ~2500 character long text + (~2048 tokens, requiring multiple chunks).""" + + # Verify text length + # Verify text has sufficient word count (approximately 2500 words) + word_count = len(LONG_TEXT_2500_WORDS.split()) + assert word_count >= 2300, f"Test text word count insufficient: {word_count} words" + + # Send embedding request + embedding_response = await client_with_chunked_processing.embeddings.create( + model=model_name, + input=[LONG_TEXT_2500_WORDS], + encoding_format="float", + ) + + # Verify response structure + embeddings = EmbeddingResponse.model_validate( + embedding_response.model_dump(mode="json") + ) + + assert embeddings.id is not None + assert len(embeddings.data) == 1 + assert ( + len(embeddings.data[0].embedding) == 384 + ) # multilingual-e5-small embedding dimension + assert embeddings.usage.completion_tokens == 0 + # Due to chunked processing, token count should + # reflect actual processed tokens + # With ~2500 words, we expect + # roughly 2048+ tokens (requiring multiple chunks) + # Should require multiple chunks for processing + assert embeddings.usage.prompt_tokens > 1500 + assert embeddings.usage.total_tokens == embeddings.usage.prompt_tokens + + # Verify embedding vector validity + embedding_vector = embeddings.data[0].embedding + assert all(isinstance(x, float) for x in embedding_vector), ( + "Embedding vector should contain floats" + ) + assert not all(x == 0 for x in embedding_vector), ( + "Embedding vector should not be all zeros" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_batch_long_text_embedding( + client_with_chunked_processing: openai.AsyncOpenAI, model_name: str +): + """Test batch long text embedding processing.""" + + input_texts = [ + LONG_TEXT_1500_WORDS, + LONG_TEXT_2500_WORDS, + "This is a short text test.", # Short text for comparison + ] + + # Send batch embedding request + embedding_response = await client_with_chunked_processing.embeddings.create( + model=model_name, + input=input_texts, + encoding_format="float", + ) + + # Verify response structure + embeddings = EmbeddingResponse.model_validate( + embedding_response.model_dump(mode="json") + ) + + assert embeddings.id is not None + assert len(embeddings.data) == 3 # Three input texts + + # Verify each embedding dimension + for i, embedding_data in enumerate(embeddings.data): + assert len(embedding_data.embedding) == 384 + assert embedding_data.index == i + + # Verify embedding vector validity + embedding_vector = embedding_data.embedding + assert all(isinstance(x, float) for x in embedding_vector) + assert not all(x == 0 for x in embedding_vector) + + # Verify token usage + assert embeddings.usage.completion_tokens == 0 + # Total token count should be very substantial + assert embeddings.usage.prompt_tokens > 1000 + assert embeddings.usage.total_tokens == embeddings.usage.prompt_tokens + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_chunked_vs_normal_consistency( + client_with_chunked_processing: openai.AsyncOpenAI, model_name: str +): + """Test consistency between chunked and + normal processing (using short text).""" + + # Use a short text within the 512 token limit + short_text = ( + "Artificial intelligence technology is changing our world, " + "bringing unprecedented opportunities and challenges." + ) + + # Send embedding request + embedding_response = await client_with_chunked_processing.embeddings.create( + model=model_name, + input=[short_text], + encoding_format="float", + ) + + # Verify response structure + embeddings = EmbeddingResponse.model_validate( + embedding_response.model_dump(mode="json") + ) + + assert embeddings.id is not None + assert len(embeddings.data) == 1 + assert len(embeddings.data[0].embedding) == 384 + assert embeddings.usage.completion_tokens == 0 + # Short text should not require chunked processing + assert embeddings.usage.prompt_tokens < 512 + assert embeddings.usage.total_tokens == embeddings.usage.prompt_tokens + + # 验证embedding向量的有效性 + embedding_vector = embeddings.data[0].embedding + assert all(isinstance(x, float) for x in embedding_vector) + assert not all(x == 0 for x in embedding_vector) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_chunked_processing_response_format( + client_with_chunked_processing: openai.AsyncOpenAI, model_name: str +): + """Test response format and structure during chunked processing.""" + + # Test with long text to trigger chunking + embedding_response = await client_with_chunked_processing.embeddings.create( + model=model_name, + input=[LONG_TEXT_1500_WORDS], + encoding_format="float", + ) + + # Verify response structure + embeddings = EmbeddingResponse.model_validate( + embedding_response.model_dump(mode="json") + ) + + assert embeddings.id is not None + assert len(embeddings.data) == 1 + assert embeddings.data[0].object == "embedding" + assert embeddings.data[0].index == 0 + + # Verify embedding vector properties + embedding_vector = embeddings.data[0].embedding + import math + + vector_norm = math.sqrt(sum(x * x for x in embedding_vector)) + # Check that the vector is normalized + # (default behavior for most embedding models) + assert 0.8 < vector_norm < 1.2, ( + f"Vector norm should be reasonable, actual: {vector_norm}" + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_online_vision.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_online_vision.py new file mode 100644 index 0000000000000000000000000000000000000000..02c3554496c456ccdcd92e1ed0c3de25980bee75 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_online_vision.py @@ -0,0 +1,210 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import pytest +import requests +from transformers import AutoProcessor + +from tests.utils import VLLM_PATH, RemoteOpenAIServer +from vllm.entrypoints.pooling.embed.protocol import EmbeddingResponse +from vllm.multimodal.media import MediaWithBytes +from vllm.multimodal.utils import encode_image_url, fetch_image + +MODEL_NAME = "TIGER-Lab/VLM2Vec-Full" +MAXIMUM_IMAGES = 2 + +vlm2vec_jinja_path = VLLM_PATH / "examples/pooling/embed/template/vlm2vec_phi3v.jinja" +assert vlm2vec_jinja_path.exists() + +# Test different image extensions (JPG/PNG) and formats (gray/RGB/RGBA) +TEST_IMAGE_ASSETS = [ + "2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg", # "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" + "Grayscale_8bits_palette_sample_image.png", # "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/Grayscale_8bits_palette_sample_image.png", + "1280px-Venn_diagram_rgb.svg.png", # "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/1280px-Venn_diagram_rgb.svg.png", + "RGBA_comp.png", # "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/RGBA_comp.png", +] + +input_text = "The best thing about vLLM is that it supports many different models" +image_url = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/cat_snow.jpg" +image_base64 = {"url": encode_image_url(fetch_image(image_url))} + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--runner", + "pooling", + "--max-model-len", + "2048", + "--max-num-seqs", + "5", + "--enforce-eager", + "--trust-remote-code", + "--limit-mm-per-prompt", + json.dumps({"image": MAXIMUM_IMAGES}), + "--chat-template", + str(vlm2vec_jinja_path), + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_chat_text_request(server: RemoteOpenAIServer, model_name: str): + messages = [ + { + "role": "user", + "content": input_text, + }, + ] + + # note: vlm2vec_phi3v.jinja + # Embedding models should only embed one message at a time. + + response = requests.post( + server.url_for("v1/embeddings"), + json={"model": model_name, "messages": messages}, + ) + response.raise_for_status() + + output = EmbeddingResponse.model_validate(response.json()) + assert len(output.data) == 1 + assert output.model == MODEL_NAME + assert len(output.data[0].embedding) == 3072 + assert output.usage.prompt_tokens == 14 + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_chat_image_url_request(server: RemoteOpenAIServer, model_name: str): + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Represent the user's input."}, + {"type": "image_url", "image_url": {"url": image_url}}, + ], + } + ] + + response = requests.post( + server.url_for("v1/embeddings"), + json={"model": model_name, "messages": messages}, + ) + response.raise_for_status() + + output = EmbeddingResponse.model_validate(response.json()) + assert len(output.data) == 1 + assert output.model == MODEL_NAME + assert len(output.data[0].embedding) == 3072 + assert output.usage.prompt_tokens == 767 + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_chat_image_base64_request(server: RemoteOpenAIServer, model_name: str): + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Represent the user's input."}, + {"type": "image_url", "image_url": image_base64}, + ], + } + ] + + response = requests.post( + server.url_for("v1/embeddings"), + json={"model": model_name, "messages": messages}, + ) + response.raise_for_status() + + output = EmbeddingResponse.model_validate(response.json()) + assert len(output.data) == 1 + assert output.model == MODEL_NAME + assert len(output.data[0].embedding) == 3072 + assert output.usage.prompt_tokens == 767 + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_chat_image_with_media_io_kwargs(server: RemoteOpenAIServer, model_name: str): + rgba_image_url = ( + "https://vllm-public-assets.s3.us-west-2.amazonaws.com" + "/vision_model_images/RGBA_comp.png" + ) + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Represent the user's input."}, + {"type": "image_url", "image_url": {"url": rgba_image_url}}, + ], + } + ] + + response = requests.post( + server.url_for("v1/embeddings"), + json={ + "model": model_name, + "messages": messages, + "media_io_kwargs": { + "image": {"rgba_background_color": [0, 0, 0]}, + }, + }, + ) + response.raise_for_status() + + output = EmbeddingResponse.model_validate(response.json()) + assert len(output.data) == 1 + assert len(output.data[0].embedding) == 3072 + + +def get_hf_prompt_tokens(model_name, content, image_url): + processor = AutoProcessor.from_pretrained( + model_name, trust_remote_code=True, num_crops=4 + ) + + placeholder = "<|image_1|> " + prompt = f"{placeholder}{content}" + image = fetch_image(image_url) + # Unwrap MediaWithBytes if present + if isinstance(image, MediaWithBytes): + image = image.media + images = [image] + inputs = processor(prompt, images, return_tensors="pt") + return inputs.input_ids.shape[1] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("image_url", TEST_IMAGE_ASSETS, indirect=True) +async def test_image_embedding( + server: RemoteOpenAIServer, model_name: str, image_url: str +): + content_text = "Represent the given image." + messages = [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": content_text}, + ], + } + ] + + response = requests.post( + server.url_for("v1/embeddings"), + json={"model": model_name, "messages": messages, "encoding_format": "float"}, + ) + response.raise_for_status() + embeddings = EmbeddingResponse.model_validate(response.json()) + + hf_prompt_tokens = get_hf_prompt_tokens(model_name, content_text, image_url) + + assert embeddings.id is not None + assert len(embeddings.data) == 1 + assert len(embeddings.data[0].embedding) == 3072 + assert embeddings.usage.completion_tokens == 0 + assert embeddings.usage.prompt_tokens == hf_prompt_tokens + assert embeddings.usage.total_tokens == hf_prompt_tokens diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_protocol.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_protocol.py new file mode 100644 index 0000000000000000000000000000000000000000..c8f9dd3abd97697086e2c6619924bd7c27b6496d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/embed/test_protocol.py @@ -0,0 +1,129 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for Cohere embed protocol: build_typed_embeddings and its +underlying packing helpers, plus Cohere-specific serving helpers.""" + +import struct + +import numpy as np +import pybase64 as base64 +import pytest + +from vllm.entrypoints.pooling.embed.protocol import ( + build_typed_embeddings, +) + + +@pytest.fixture +def sample_embeddings() -> list[list[float]]: + return [ + [0.1, -0.2, 0.3, -0.4, 0.5, -0.6, 0.7, -0.8], + [-0.05, 0.15, -0.25, 0.35, -0.45, 0.55, -0.65, 0.75], + ] + + +class TestBuildTypedEmbeddingsFloat: + def test_float_passthrough(self, sample_embeddings: list[list[float]]): + result = build_typed_embeddings(sample_embeddings, ["float"]) + assert result.float == sample_embeddings + assert result.binary is None + + def test_empty_input(self): + result = build_typed_embeddings([], ["float"]) + assert result.float == [] + + +class TestBuildTypedEmbeddingsBinary: + def test_binary_packing(self): + # 8 values: positive->1, negative->0 => bits: 10101010 = 0xAA = 170 + # signed: 170 - 128 = 42 + embs = [[1.0, -1.0, 1.0, -1.0, 1.0, -1.0, 1.0, -1.0]] + result = build_typed_embeddings(embs, ["binary"]) + assert result.binary is not None + assert result.binary[0] == [42] + + def test_ubinary_packing(self): + embs = [[1.0, -1.0, 1.0, -1.0, 1.0, -1.0, 1.0, -1.0]] + result = build_typed_embeddings(embs, ["ubinary"]) + assert result.ubinary is not None + assert result.ubinary[0] == [170] # 0b10101010 + + def test_binary_all_positive(self): + embs = [[0.1] * 8] + result = build_typed_embeddings(embs, ["binary"]) + assert result.binary is not None + # all bits = 1 => 0xFF = 255, signed: 255 - 128 = 127 + assert result.binary[0] == [127] + + def test_binary_all_negative(self): + embs = [[-0.1] * 8] + result = build_typed_embeddings(embs, ["binary"]) + assert result.binary is not None + # all bits = 0, signed: 0 - 128 = -128 + assert result.binary[0] == [-128] + + def test_binary_dimension_is_eighth(self, sample_embeddings: list[list[float]]): + result = build_typed_embeddings(sample_embeddings, ["binary"]) + assert result.binary is not None + for orig, packed in zip(sample_embeddings, result.binary): + assert len(packed) == len(orig) // 8 + + def test_zero_treated_as_positive(self): + embs = [[0.0] * 8] + result = build_typed_embeddings(embs, ["binary"]) + assert result.binary is not None + # 0.0 >= 0 is True, so bit=1 for all => 127 (signed) + assert result.binary[0] == [127] + + def test_non_multiple_of_8_raises(self): + embs = [[0.1] * 7] + with pytest.raises(ValueError, match="multiple of 8"): + build_typed_embeddings(embs, ["binary"]) + + def test_ubinary_non_multiple_of_8_raises(self): + embs = [[0.1] * 10] + with pytest.raises(ValueError, match="multiple of 8"): + build_typed_embeddings(embs, ["ubinary"]) + + +class TestBuildTypedEmbeddingsBase64: + def test_base64_roundtrip(self, sample_embeddings: list[list[float]]): + result = build_typed_embeddings(sample_embeddings, ["base64"]) + assert result.base64 is not None + assert len(result.base64) == 2 + + for orig, b64_str in zip(sample_embeddings, result.base64): + decoded = base64.b64decode(b64_str) + n = len(orig) + values = struct.unpack(f"<{n}f", decoded) + np.testing.assert_allclose(orig, values, rtol=1e-5) + + def test_base64_byte_length(self): + embs = [[0.1, 0.2, 0.3]] + result = build_typed_embeddings(embs, ["base64"]) + assert result.base64 is not None + raw = base64.b64decode(result.base64[0]) + assert len(raw) == 3 * 4 # 3 floats * 4 bytes each + + +class TestBuildTypedEmbeddingsMultiple: + def test_all_types_at_once(self, sample_embeddings: list[list[float]]): + result = build_typed_embeddings( + sample_embeddings, + ["float", "binary", "ubinary", "base64"], + ) + assert result.float is not None + assert result.binary is not None + assert result.ubinary is not None + assert result.base64 is not None + + def test_subset_types(self, sample_embeddings: list[list[float]]): + result = build_typed_embeddings(sample_embeddings, ["float", "binary"]) + assert result.float is not None + assert result.binary is not None + assert result.ubinary is None + assert result.base64 is None + + def test_unknown_type_ignored(self, sample_embeddings: list[list[float]]): + result = build_typed_embeddings(sample_embeddings, ["float", "unknown_type"]) + assert result.float is not None diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/reward/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/reward/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/reward/test_token_reward_offline.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/reward/test_token_reward_offline.py new file mode 100644 index 0000000000000000000000000000000000000000..b43c607f4b816181a86033b7e12eb3ccc5667036 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/reward/test_token_reward_offline.py @@ -0,0 +1,68 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import weakref + +import pytest +import torch + +from tests.models.utils import softmax +from vllm import LLM, PoolingParams +from vllm.distributed import cleanup_dist_env_and_memory + +MODEL_NAME = "internlm/internlm2-1_8b-reward" + +prompts = ["The chef prepared a delicious meal."] + + +@pytest.fixture(scope="module") +def llm(): + # pytest caches the fixture so we use weakref.proxy to + # enable garbage collection + llm = LLM( + model=MODEL_NAME, + max_num_batched_tokens=32768, + tensor_parallel_size=1, + gpu_memory_utilization=0.75, + enforce_eager=True, + trust_remote_code=True, + seed=0, + ) + + yield weakref.proxy(llm) + + del llm + + cleanup_dist_env_and_memory() + + +@pytest.mark.skip_global_cleanup +def test_config(llm: LLM): + vllm_config = llm.llm_engine.vllm_config + assert vllm_config.cache_config.enable_prefix_caching + assert vllm_config.scheduler_config.enable_chunked_prefill + + +def test_pooling_params(llm: LLM): + def get_outputs(use_activation): + outputs = llm.encode( + prompts, + pooling_params=PoolingParams(use_activation=use_activation), + pooling_task="token_classify", + use_tqdm=False, + ) + return torch.cat([x.outputs.data for x in outputs]) + + default = get_outputs(use_activation=None) + w_activation = get_outputs(use_activation=True) + wo_activation = get_outputs(use_activation=False) + + assert torch.allclose(default, w_activation, atol=1e-2), ( + "Default should use activation." + ) + assert not torch.allclose(w_activation, wo_activation, atol=1e-2), ( + "wo_activation should not use activation." + ) + assert torch.allclose(softmax(wo_activation), w_activation, atol=1e-2), ( + "w_activation should be close to activation(wo_activation)." + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/reward/test_token_reward_online.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/reward/test_token_reward_online.py new file mode 100644 index 0000000000000000000000000000000000000000..e7cdb3967abc459582078c49c693402c4622d842 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/reward/test_token_reward_online.py @@ -0,0 +1,569 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import numpy as np +import pybase64 as base64 +import pytest +import requests +import torch + +from tests.models.utils import check_embeddings_close +from tests.utils import RemoteOpenAIServer +from vllm.entrypoints.pooling.pooling.protocol import PoolingResponse +from vllm.entrypoints.pooling.utils import ( + MetadataItem, + build_metadata_items, + decode_pooling_output, +) +from vllm.tokenizers import get_tokenizer +from vllm.utils.serial_utils import EMBED_DTYPES, ENDIANNESS, binary2tensor + +MODEL_NAME = "internlm/internlm2-1_8b-reward" +DUMMY_CHAT_TEMPLATE = """{% for message in messages %}{{message['role'] + ': ' + message['content'] + '\\n'}}{% endfor %}""" # noqa: E501 +input_text = "The chef prepared a delicious meal." +input_tokens = [1, 918, 29981, 10166, 395, 18067, 15265, 281] + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--runner", + "pooling", + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--enforce-eager", + "--max-model-len", + "512", + "--chat-template", + DUMMY_CHAT_TEMPLATE, + "--trust-remote-code", + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_basic(server: RemoteOpenAIServer, model_name: str): + # test /v1/models + response = requests.get(server.url_for("/v1/models")) + served_model = response.json()["data"][0]["id"] + assert served_model == MODEL_NAME + + # test /tokenize + response = requests.post( + server.url_for("/tokenize"), + json={"model": model_name, "prompt": input_text}, + ) + assert response.json()["tokens"] == input_tokens + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_completion_request(server: RemoteOpenAIServer, model_name: str): + # test input: str + response = requests.post( + server.url_for("pooling"), + json={"model": model_name, "input": input_text, "encoding_format": "float"}, + ) + response.raise_for_status() + poolings = PoolingResponse.model_validate(response.json()) + + assert poolings.id is not None + assert len(poolings.data) == 1 + assert len(poolings.data[0].data) == len(input_tokens) + assert poolings.usage.completion_tokens == 0 + assert poolings.usage.prompt_tokens == len(input_tokens) + assert poolings.usage.total_tokens == len(input_tokens) + + # test input: list[int] + response = requests.post( + server.url_for("pooling"), + json={"model": model_name, "input": input_tokens, "encoding_format": "float"}, + ) + response.raise_for_status() + poolings = PoolingResponse.model_validate(response.json()) + + assert poolings.id is not None + assert len(poolings.data) == 1 + assert len(poolings.data[0].data) == len(input_tokens) + assert poolings.usage.completion_tokens == 0 + assert poolings.usage.prompt_tokens == len(input_tokens) + assert poolings.usage.total_tokens == len(input_tokens) + + +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +def test_completion_request_batched(server: RemoteOpenAIServer, model_name: str): + N = 10 + input_texts = [input_text] * N + + response = requests.post( + server.url_for("pooling"), + json={"model": model_name, "input": input_texts, "encoding_format": "float"}, + ) + response.raise_for_status() + poolings = PoolingResponse.model_validate(response.json()) + + assert poolings.id is not None + assert len(poolings.data) == N + assert len(poolings.data[0].data) == len(input_tokens) + assert poolings.usage.completion_tokens == 0 + assert poolings.usage.prompt_tokens == len(input_tokens) * N + assert poolings.usage.total_tokens == len(input_tokens) * N + + # test list[list[int]] + response = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "input": [input_tokens] * N, + "encoding_format": "float", + }, + ) + response.raise_for_status() + poolings = PoolingResponse.model_validate(response.json()) + + assert poolings.id is not None + assert len(poolings.data) == N + assert len(poolings.data[0].data) == len(input_tokens) + assert poolings.usage.completion_tokens == 0 + assert poolings.usage.prompt_tokens == len(input_tokens) * N + assert poolings.usage.total_tokens == len(input_tokens) * N + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_chat_request(server: RemoteOpenAIServer, model_name: str): + messages = [ + { + "role": "user", + "content": "The cat sat on the mat.", + }, + { + "role": "assistant", + "content": "A feline was resting on a rug.", + }, + { + "role": "user", + "content": "Stars twinkle brightly in the night sky.", + }, + ] + + # test chat request basic usage + chat_response = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "messages": messages, + "encoding_format": "float", + }, + ) + chat_response.raise_for_status() + chat_poolings = PoolingResponse.model_validate(chat_response.json()) + + tokenizer = get_tokenizer(tokenizer_name=model_name, trust_remote_code=True) + prompt = tokenizer.apply_chat_template( + messages, + chat_template=DUMMY_CHAT_TEMPLATE, + add_generation_prompt=True, + continue_final_message=False, + tokenize=False, + ) + completions_response = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "input": prompt, + "encoding_format": "float", + # To be consistent with chat + "add_special_tokens": False, + }, + ) + completions_response.raise_for_status() + completion_poolings = PoolingResponse.model_validate(completions_response.json()) + + assert chat_poolings.id is not None + assert completion_poolings.id is not None + assert chat_poolings.created <= completion_poolings.created + assert chat_poolings.model_dump(exclude={"id", "created"}) == ( + completion_poolings.model_dump(exclude={"id", "created"}) + ) + + # test add_generation_prompt + response = requests.post( + server.url_for("pooling"), + json={"model": model_name, "messages": messages, "add_generation_prompt": True}, + ) + + response.raise_for_status() + output = PoolingResponse.model_validate(response.json()) + + assert output.object == "list" + assert len(output.data) == 1 + assert output.model == MODEL_NAME + assert output.usage.prompt_tokens == 33 + + # test continue_final_message + # The continue_final_message parameter doesn't seem to be working with this model. + response = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "messages": messages, + "continue_final_message": True, + }, + ) + + response.raise_for_status() + output = PoolingResponse.model_validate(response.json()) + + assert output.object == "list" + assert len(output.data) == 1 + assert output.model == MODEL_NAME + assert output.usage.prompt_tokens == 33 + + # test add_special_tokens + response = requests.post( + server.url_for("pooling"), + json={"model": model_name, "messages": messages, "add_special_tokens": True}, + ) + + response.raise_for_status() + output = PoolingResponse.model_validate(response.json()) + + assert output.object == "list" + assert len(output.data) == 1 + assert output.model == MODEL_NAME + assert output.usage.prompt_tokens == 34 + + # test continue_final_message with add_generation_prompt + response = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "messages": messages, + "continue_final_message": True, + "add_generation_prompt": True, + }, + ) + assert ( + "Cannot set both `continue_final_message` and `add_generation_prompt` to True." + in response.json()["error"]["message"] + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_batch_base64_pooling(server: RemoteOpenAIServer, model_name: str): + input_texts = [ + "Hello my name is", + "The best thing about vLLM is that it supports many different models", + ] + + float_response = requests.post( + server.url_for("pooling"), + json={ + "input": input_texts, + "model": model_name, + "encoding_format": "float", + }, + ) + float_response.raise_for_status() + responses_float = PoolingResponse.model_validate(float_response.json()) + float_data = [np.array(d.data).squeeze(-1).tolist() for d in responses_float.data] + + base64_response = requests.post( + server.url_for("pooling"), + json={ + "input": input_texts, + "model": model_name, + "encoding_format": "base64", + }, + ) + base64_response.raise_for_status() + responses_base64 = PoolingResponse.model_validate(base64_response.json()) + + decoded_responses_base64_data = [] + for data in responses_base64.data: + decoded_responses_base64_data.append( + np.frombuffer(base64.b64decode(data.data), dtype="float32").tolist() + ) + + check_embeddings_close( + embeddings_0_lst=float_data, + embeddings_1_lst=decoded_responses_base64_data, + name_0="float32", + name_1="base64", + ) + + # Default response is float32 decoded from base64 by OpenAI Client + default_response = requests.post( + server.url_for("pooling"), + json={ + "input": input_texts, + "model": model_name, + }, + ) + default_response.raise_for_status() + responses_default = PoolingResponse.model_validate(default_response.json()) + default_data = [ + np.array(d.data).squeeze(-1).tolist() for d in responses_default.data + ] + + check_embeddings_close( + embeddings_0_lst=float_data, + embeddings_1_lst=default_data, + name_0="float32", + name_1="default", + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_base64_embed_dtype_and_endianness( + server: RemoteOpenAIServer, model_name: str +): + input_texts = [input_text] * 3 + + url = server.url_for("pooling") + float_response = requests.post( + url, + json={ + "model": model_name, + "input": input_texts, + "encoding_format": "float", + }, + ) + responses_float = PoolingResponse.model_validate(float_response.json()) + float_data = [np.array(d.data).squeeze(-1).tolist() for d in responses_float.data] + + for embed_dtype in EMBED_DTYPES: + for endianness in ENDIANNESS: + responses_base64 = requests.post( + url, + json={ + "model": model_name, + "input": input_texts, + "encoding_format": "base64", + "embed_dtype": embed_dtype, + "endianness": endianness, + }, + ) + + base64_data = [] + for data in responses_base64.json()["data"]: + binary = base64.b64decode(data["data"]) + tensor = binary2tensor(binary, (-1,), embed_dtype, endianness) + base64_data.append(tensor.to(torch.float32).tolist()) + + check_embeddings_close( + embeddings_0_lst=float_data, + embeddings_1_lst=base64_data, + name_0="float_data", + name_1="base64_data", + tol=1e-2, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_bytes_embed_dtype_and_endianness( + server: RemoteOpenAIServer, model_name: str +): + input_texts = [input_text] * 3 + + url = server.url_for("pooling") + float_response = requests.post( + url, + json={ + "model": model_name, + "input": input_texts, + "encoding_format": "float", + }, + ) + responses_float = PoolingResponse.model_validate(float_response.json()) + float_data = [np.array(d.data).squeeze(-1).tolist() for d in responses_float.data] + + for embed_dtype in EMBED_DTYPES: + for endianness in ENDIANNESS: + responses_bytes = requests.post( + url, + json={ + "model": model_name, + "input": input_texts, + "encoding_format": "bytes", + "embed_dtype": embed_dtype, + "endianness": endianness, + }, + ) + + metadata = json.loads(responses_bytes.headers["metadata"]) + body = responses_bytes.content + items = [MetadataItem(**x) for x in metadata["data"]] + + bytes_data = decode_pooling_output(items=items, body=body) + bytes_data = [x.to(torch.float32).view(-1).tolist() for x in bytes_data] + + check_embeddings_close( + embeddings_0_lst=float_data, + embeddings_1_lst=bytes_data, + name_0="float_data", + name_1="bytes_data", + tol=1e-2, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_bytes_only_embed_dtype_and_endianness( + server: RemoteOpenAIServer, model_name: str +): + input_texts = [input_text] * 3 + + url = server.url_for("pooling") + float_response = requests.post( + url, + json={ + "model": model_name, + "input": input_texts, + "encoding_format": "float", + }, + ) + responses_float = PoolingResponse.model_validate(float_response.json()) + float_data = [np.array(d.data).squeeze(-1).tolist() for d in responses_float.data] + n_tokens = responses_float.usage.prompt_tokens // len(input_texts) + + for embed_dtype in EMBED_DTYPES: + for endianness in ENDIANNESS: + responses_bytes = requests.post( + url, + json={ + "model": model_name, + "input": input_texts, + "encoding_format": "bytes_only", + "embed_dtype": embed_dtype, + "endianness": endianness, + }, + ) + + assert "metadata" not in responses_bytes.headers + body = responses_bytes.content + items = build_metadata_items( + embed_dtype=embed_dtype, + endianness=endianness, + shape=(n_tokens, 1), + n_request=len(input_texts), + ) + bytes_data = decode_pooling_output(items=items, body=body) + bytes_data = [x.to(torch.float32).view(-1).tolist() for x in bytes_data] + + check_embeddings_close( + embeddings_0_lst=float_data, + embeddings_1_lst=bytes_data, + name_0="float_data", + name_1="bytes_data", + tol=1e-2, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("param_name", ["encoding_format", "embed_dtype", "endianness"]) +async def test_params_not_supported( + server: RemoteOpenAIServer, model_name: str, param_name: str +): + responses_base64 = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "input": input_text, + "encoding_format": "base64", + param_name: f"bad_{param_name}", + }, + ) + + assert responses_base64.status_code == 400 + assert "literal_error" in responses_base64.json()["error"]["message"] + assert f"bad_{param_name}" in responses_base64.json()["error"]["message"] + + +@pytest.mark.asyncio +async def test_invocations_chat_request(server: RemoteOpenAIServer): + request_args = { + "model": MODEL_NAME, + "input": input_text, + "encoding_format": "float", + } + + completion_response = requests.post(server.url_for("pooling"), json=request_args) + completion_response.raise_for_status() + + invocation_response = requests.post( + server.url_for("invocations"), json=request_args + ) + invocation_response.raise_for_status() + + completion_output = completion_response.json() + invocation_output = invocation_response.json() + + assert completion_output.keys() == invocation_output.keys() + for completion_data, invocation_data in zip( + completion_output["data"], invocation_output["data"] + ): + assert completion_data.keys() == invocation_data.keys() + check_embeddings_close( + embeddings_0_lst=completion_data["data"], + embeddings_1_lst=invocation_data["data"], + name_0="completion", + name_1="invocation", + ) + + +@pytest.mark.asyncio +async def test_invocations_conversation_chat_request(server: RemoteOpenAIServer): + messages = [ + { + "role": "user", + "content": "The cat sat on the mat.", + }, + { + "role": "assistant", + "content": "A feline was resting on a rug.", + }, + { + "role": "user", + "content": "Stars twinkle brightly in the night sky.", + }, + ] + + request_args = { + "model": MODEL_NAME, + "messages": messages, + "encoding_format": "float", + } + + chat_response = requests.post(server.url_for("pooling"), json=request_args) + chat_response.raise_for_status() + + invocation_response = requests.post( + server.url_for("invocations"), json=request_args + ) + invocation_response.raise_for_status() + + chat_output = chat_response.json() + invocation_output = invocation_response.json() + + assert chat_output.keys() == invocation_output.keys() + for chat_data, invocation_data in zip( + chat_output["data"], invocation_output["data"] + ): + assert chat_data.keys() == invocation_data.keys() + check_embeddings_close( + embeddings_0_lst=chat_data["data"], + embeddings_1_lst=invocation_data["data"], + name_0="chat", + name_1="invocation", + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_bi_encoder_offline.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_bi_encoder_offline.py new file mode 100644 index 0000000000000000000000000000000000000000..f7ed7eddb5d8da1e3306807ee278971336b0811b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_bi_encoder_offline.py @@ -0,0 +1,114 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import weakref + +import pytest + +from tests.entrypoints.pooling.scoring.util import EncoderScoringHfRunner +from vllm import LLM +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.platforms import current_platform + +MODEL_NAME = "intfloat/multilingual-e5-small" +PROMPT = "The chef prepared a delicious meal." +EMBEDDING_SIZE = 384 + +TEXTS_1 = [ + "What is the capital of France?", + "What is the capital of Germany?", +] + +TEXTS_2 = [ + "The capital of France is Paris.", + "The capital of Germany is Berlin.", +] + +DTYPE = "half" + + +@pytest.fixture(scope="module") +def llm(): + # ROCm: Use FLEX_ATTENTION backend as it's the only attention backend + # that supports encoder-only models on ROCm. + attention_config = None + if current_platform.is_rocm(): + attention_config = {"backend": "FLEX_ATTENTION"} + + # pytest caches the fixture so we use weakref.proxy to + # enable garbage collection + llm = LLM( + model=MODEL_NAME, + max_num_batched_tokens=32768, + tensor_parallel_size=1, + gpu_memory_utilization=0.75, + enforce_eager=True, + seed=0, + attention_config=attention_config, + ) + + yield weakref.proxy(llm) + + del llm + + cleanup_dist_env_and_memory() + + +@pytest.fixture(scope="module") +def hf_model(): + return EncoderScoringHfRunner(MODEL_NAME) + + +@pytest.mark.skip_global_cleanup +def test_1_to_1(llm, hf_model): + text_pair = [TEXTS_1[0], TEXTS_2[0]] + + hf_outputs = hf_model.predict([text_pair]).tolist() + vllm_outputs = [ + output.outputs.score for output in llm.score(text_pair[0], text_pair[1]) + ] + + assert len(vllm_outputs) == 1 + assert len(hf_outputs) == 1 + + assert hf_outputs[0] == pytest.approx(vllm_outputs[0], rel=0.01) + + +@pytest.mark.skip_global_cleanup +def test_1_to_n(llm, hf_model): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[0], TEXTS_2[1]], + ] + + hf_outputs = hf_model.predict(text_pairs).tolist() + vllm_outputs = [output.outputs.score for output in llm.score(TEXTS_1[0], TEXTS_2)] + + assert len(vllm_outputs) == 2 + assert len(hf_outputs) == 2 + + assert hf_outputs[0] == pytest.approx(vllm_outputs[0], rel=0.01) + assert hf_outputs[1] == pytest.approx(vllm_outputs[1], rel=0.01) + + +@pytest.mark.skip_global_cleanup +def test_n_to_n(llm, hf_model): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[1], TEXTS_2[1]], + ] + + hf_outputs = hf_model.predict(text_pairs).tolist() + vllm_outputs = [output.outputs.score for output in llm.score(TEXTS_1, TEXTS_2)] + + assert len(vllm_outputs) == 2 + assert len(hf_outputs) == 2 + + assert hf_outputs[0] == pytest.approx(vllm_outputs[0], rel=0.01) + assert hf_outputs[1] == pytest.approx(vllm_outputs[1], rel=0.01) + + +def test_embed(llm): + outputs = llm.encode(PROMPT, pooling_task="embed", use_tqdm=False) + assert len(outputs) == 1 + assert len(outputs[0].outputs.data) == EMBEDDING_SIZE diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_bi_encoder_online.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_bi_encoder_online.py new file mode 100644 index 0000000000000000000000000000000000000000..50af2d83abc20b1f53c9539a2e8e56dadc2cd20e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_bi_encoder_online.py @@ -0,0 +1,418 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import requests + +from tests.entrypoints.pooling.scoring.util import EncoderScoringHfRunner +from tests.utils import RemoteOpenAIServer +from vllm.entrypoints.pooling.pooling.protocol import PoolingResponse +from vllm.entrypoints.pooling.scoring.protocol import RerankResponse, ScoreResponse +from vllm.platforms import current_platform + +MODEL_NAME = "BAAI/bge-base-en-v1.5" +input_text = "This product was excellent and exceeded my expectations" +DTYPE = "half" +EMBEDDING_SIZE = 768 + +TEXTS_1 = [ + "What is the capital of France?", + "What is the capital of Germany?", +] + +TEXTS_2 = [ + "The capital of France is Paris.", + "The capital of Germany is Berlin.", +] + + +@pytest.fixture(scope="module") +def server(): + args = ["--enforce-eager", "--max-model-len", "100", "--dtype", DTYPE] + + # ROCm: Use Flex Attention to support encoder-only self-attention. + if current_platform.is_rocm(): + args.extend(["--attention-backend", "FLEX_ATTENTION"]) + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.fixture(scope="module") +def hf_model(): + return EncoderScoringHfRunner(MODEL_NAME) + + +@pytest.mark.asyncio +async def test_score_api_queries_str_1_documents_str_1( + hf_model, server: RemoteOpenAIServer +): + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": TEXTS_1[0], + "documents": TEXTS_2[0], + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 1 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict([[TEXTS_1[0], TEXTS_2[0]]]).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_score_api_queries_str_1_documents_str_n( + hf_model, server: RemoteOpenAIServer +): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[0], TEXTS_2[1]], + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": TEXTS_1[0], + "documents": TEXTS_2, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 2 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict(text_pairs).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_score_api_queries_str_n_documents_str_n( + hf_model, server: RemoteOpenAIServer +): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[1], TEXTS_2[1]], + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": TEXTS_1, + "documents": TEXTS_2, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 2 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict(text_pairs).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_score_api_queries_vs_documents(hf_model, server: RemoteOpenAIServer): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[1], TEXTS_2[1]], + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": TEXTS_1, + "documents": TEXTS_2, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 2 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict(text_pairs).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_score_api_queries_vs_items(hf_model, server: RemoteOpenAIServer): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[1], TEXTS_2[1]], + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": TEXTS_1, + "items": TEXTS_2, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 2 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict(text_pairs).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_score_api_text_1_vs_text_2(hf_model, server: RemoteOpenAIServer): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[1], TEXTS_2[1]], + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "text_1": TEXTS_1, + "text_2": TEXTS_2, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 2 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict(text_pairs).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_score_api_data_1_vs_data_2(hf_model, server: RemoteOpenAIServer): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[1], TEXTS_2[1]], + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "data_1": TEXTS_1, + "data_2": TEXTS_2, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 2 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict(text_pairs).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_rerank_api_texts(server: RemoteOpenAIServer): + query = "What is the capital of France?" + documents = [ + "The capital of Brazil is Brasilia.", + "The capital of France is Paris.", + ] + + rerank_response = requests.post( + server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": documents, + }, + ) + rerank_response.raise_for_status() + rerank = RerankResponse.model_validate(rerank_response.json()) + + assert rerank.id is not None + assert rerank.results is not None + assert len(rerank.results) == 2 + paris_result = next(r for r in rerank.results if r.index == 1) + brazil_result = next(r for r in rerank.results if r.index == 0) + assert paris_result.relevance_score > brazil_result.relevance_score + + +@pytest.mark.asyncio +async def test_rerank_api_top_n(server: RemoteOpenAIServer): + query = "What is the capital of France?" + documents = [ + "The capital of Brazil is Brasilia.", + "The capital of France is Paris.", + "Cross-encoder models are neat", + ] + + rerank_response = requests.post( + server.url_for("rerank"), + json={"model": MODEL_NAME, "query": query, "documents": documents, "top_n": 2}, + ) + rerank_response.raise_for_status() + rerank = RerankResponse.model_validate(rerank_response.json()) + + assert rerank.id is not None + assert rerank.results is not None + assert len(rerank.results) == 2 + assert rerank.results[0].index == 1 + + +@pytest.mark.asyncio +async def test_rerank_api_max_model_len(server: RemoteOpenAIServer): + query = "What is the capital of France?" * 100 + documents = [ + "The capital of Brazil is Brasilia.", + "The capital of France is Paris.", + ] + + rerank_response = requests.post( + server.url_for("rerank"), + json={"model": MODEL_NAME, "query": query, "documents": documents}, + ) + assert rerank_response.status_code == 400 + # Assert just a small fragments of the response + assert "Please reduce the length of the input prompt" in rerank_response.text + + +@pytest.mark.asyncio +async def test_score_api_max_model_len(server: RemoteOpenAIServer): + queries = "What is the capital of France?" * 20 + documents = [ + "The capital of Brazil is Brasilia.", + "The capital of France is Paris.", + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": queries, + "documents": documents, + }, + ) + assert score_response.status_code == 400 + # Assert just a small fragments of the response + assert "Please reduce the length of the input prompt" in score_response.text + + # Test truncation + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": queries, + "documents": documents, + "truncate_prompt_tokens": 101, + }, + ) + assert score_response.status_code == 400 + assert "Please request a smaller truncation size." in score_response.text + + +@pytest.mark.asyncio +async def test_invocations(server: RemoteOpenAIServer): + query = "What is the capital of France?" + documents = [ + "The capital of Brazil is Brasilia.", + "The capital of France is Paris.", + ] + + request_args = { + "model": MODEL_NAME, + "query": query, + "documents": documents, + } + + rerank_response = requests.post(server.url_for("rerank"), json=request_args) + rerank_response.raise_for_status() + + invocation_response = requests.post( + server.url_for("invocations"), json=request_args + ) + invocation_response.raise_for_status() + + rerank_output = rerank_response.json() + invocation_output = invocation_response.json() + + assert rerank_output.keys() == invocation_output.keys() + for rerank_result, invocations_result in zip( + rerank_output["results"], invocation_output["results"] + ): + assert rerank_result.keys() == invocations_result.keys() + assert rerank_result["relevance_score"] == pytest.approx( + invocations_result["relevance_score"], rel=0.01 + ) + + +@pytest.mark.asyncio +async def test_pooling_embed(server: RemoteOpenAIServer): + response = requests.post( + server.url_for("pooling"), + json={ + "model": MODEL_NAME, + "input": input_text, + "encoding_format": "float", + "task": "embed", + }, + ) + poolings = PoolingResponse.model_validate(response.json()) + assert len(poolings.data) == 1 + assert len(poolings.data[0].data) == EMBEDDING_SIZE + + +@pytest.mark.asyncio +@pytest.mark.parametrize("task", ["classify", "token_classify", "plugin"]) +async def test_pooling_not_supported(server: RemoteOpenAIServer, task: str): + response = requests.post( + server.url_for("pooling"), + json={ + "model": MODEL_NAME, + "input": input_text, + "encoding_format": "float", + "task": task, + }, + ) + assert response.json()["error"]["type"] == "BadRequestError" + if task == "plugin": + err_msg = "No IOProcessor plugin installed." + else: + err_msg = f"Unsupported task: {task!r}" + assert response.json()["error"]["message"].startswith(err_msg) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_cross_encoder_correctness_mteb.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_cross_encoder_correctness_mteb.py new file mode 100644 index 0000000000000000000000000000000000000000..3aa6d87c57d100dfda40c26fd33b6ebd8407024f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_cross_encoder_correctness_mteb.py @@ -0,0 +1,61 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import os + +import pytest + +from tests.models.language.pooling_mteb_test.mteb_score_utils import ( + MTEB_RERANK_LANGS, + MTEB_RERANK_TASKS, + MTEB_RERANK_TOL, + RerankClientMtebEncoder, + ScoreClientMtebEncoder, + run_mteb_rerank, +) +from tests.utils import RemoteOpenAIServer +from vllm.platforms import current_platform + +os.environ["VLLM_LOGGING_LEVEL"] = "WARNING" + +MODEL_NAME = "cross-encoder/ms-marco-MiniLM-L-6-v2" +st_main_score = 0.33457 + + +@pytest.fixture(scope="module") +def server(): + args = ["--runner", "pooling", "--enforce-eager", "--disable-uvicorn-access-log"] + + # ROCm: Use Flex Attention to support encoder-only self-attention. + if current_platform.is_rocm(): + args.extend(["--attention-backend", "FLEX_ATTENTION"]) + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +def test_mteb_score(server): + url = server.url_for("score") + encoder = ScoreClientMtebEncoder(MODEL_NAME, url) + vllm_main_score = run_mteb_rerank(encoder, MTEB_RERANK_TASKS, MTEB_RERANK_LANGS) + + print("VLLM main score: ", vllm_main_score) + print("SentenceTransformer main score: ", st_main_score) + print("Difference: ", st_main_score - vllm_main_score) + + # We are not concerned that the vllm mteb results are better + # than SentenceTransformers, so we only perform one-sided testing. + assert st_main_score - vllm_main_score < MTEB_RERANK_TOL + + +def test_mteb_rerank(server): + url = server.url_for("rerank") + encoder = RerankClientMtebEncoder(MODEL_NAME, url) + vllm_main_score = run_mteb_rerank(encoder, MTEB_RERANK_TASKS, MTEB_RERANK_LANGS) + + print("VLLM main score: ", vllm_main_score) + print("SentenceTransformer main score: ", st_main_score) + print("Difference: ", st_main_score - vllm_main_score) + + # We are not concerned that the vllm mteb results are better + # than SentenceTransformers, so we only perform one-sided testing. + assert st_main_score - vllm_main_score < MTEB_RERANK_TOL diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_cross_encoder_offline.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_cross_encoder_offline.py new file mode 100644 index 0000000000000000000000000000000000000000..e36c0076fd1a79dc7f96b75b3806ffb606bfee2d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_cross_encoder_offline.py @@ -0,0 +1,166 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import weakref + +import pytest +import torch + +from tests.models.utils import softmax +from vllm import LLM, PoolingParams +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.platforms import current_platform + +MODEL_NAME = "tomaarsen/Qwen3-Reranker-0.6B-seq-cls" +PROMPT = "The chef prepared a delicious meal." +TEXTS_1 = [ + "What is the capital of France?", + "What is the capital of Germany?", +] + +TEXTS_2 = [ + "The capital of France is Paris.", + "The capital of Germany is Berlin.", +] + + +@pytest.fixture(scope="module") +def llm(): + # ROCm: Use FLEX_ATTENTION backend as it's the only attention backend + # that supports encoder-only models on ROCm. + attention_config = None + if current_platform.is_rocm(): + attention_config = {"backend": "FLEX_ATTENTION"} + + # pytest caches the fixture so we use weakref.proxy to + # enable garbage collection + llm = LLM( + model=MODEL_NAME, + max_num_batched_tokens=32768, + tensor_parallel_size=1, + gpu_memory_utilization=0.75, + enforce_eager=True, + seed=0, + attention_config=attention_config, + ) + + yield weakref.proxy(llm) + + del llm + + cleanup_dist_env_and_memory() + + +@pytest.fixture(scope="module") +def hf_model(hf_runner): + return hf_runner(MODEL_NAME, is_cross_encoder=True) + + +@pytest.mark.skip_global_cleanup +def test_1_to_1(llm, hf_model): + text_pair = [TEXTS_1[0], TEXTS_2[0]] + + hf_outputs = hf_model.predict([text_pair]).tolist() + vllm_outputs = [ + output.outputs.score for output in llm.score(text_pair[0], text_pair[1]) + ] + + assert len(vllm_outputs) == 1 + assert len(hf_outputs) == 1 + + assert hf_outputs[0] == pytest.approx(vllm_outputs[0], rel=0.01) + + +@pytest.mark.skip_global_cleanup +def test_1_to_n(llm, hf_model): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[0], TEXTS_2[1]], + ] + + vllm_outputs = [output.outputs.score for output in llm.score(TEXTS_1[0], TEXTS_2)] + hf_outputs = hf_model.predict(text_pairs).tolist() + + assert len(vllm_outputs) == 2 + assert len(hf_outputs) == 2 + + assert hf_outputs[0] == pytest.approx(vllm_outputs[0], rel=0.01) + assert hf_outputs[1] == pytest.approx(vllm_outputs[1], rel=0.01) + + +@pytest.mark.skip_global_cleanup +def test_n_to_n(llm, hf_model): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[1], TEXTS_2[1]], + ] + + vllm_outputs = [output.outputs.score for output in llm.score(TEXTS_1, TEXTS_2)] + hf_outputs = hf_model.predict(text_pairs).tolist() + + assert len(vllm_outputs) == 2 + assert len(hf_outputs) == 2 + + assert hf_outputs[0] == pytest.approx(vllm_outputs[0], rel=0.01) + assert hf_outputs[1] == pytest.approx(vllm_outputs[1], rel=0.01) + + +@pytest.mark.skip_global_cleanup +def test_classify(llm): + outputs = llm.encode(PROMPT, pooling_task="classify", use_tqdm=False) + assert len(outputs) == 1 + assert len(outputs[0].outputs.data) == 1 + + +@pytest.mark.skip_global_cleanup +def test_max_tokens_per_doc(llm: LLM): + """Test max_tokens_per_doc via PoolingParams.extra_kwargs (offline).""" + long_doc = "The capital of France is Paris. " * 20 + + # Without truncation + outputs_no_limit = llm.score( + TEXTS_1[0], + long_doc, + use_tqdm=False, + ) + + # With truncation via extra_kwargs + outputs_with_limit = llm.score( + TEXTS_1[0], + long_doc, + pooling_params=PoolingParams(extra_kwargs={"max_tokens_per_doc": 10}), + use_tqdm=False, + ) + + assert len(outputs_no_limit) == 1 + assert len(outputs_with_limit) == 1 + + # Truncated version should have fewer prompt tokens + no_limit_tokens = len(outputs_no_limit[0].prompt_token_ids) + with_limit_tokens = len(outputs_with_limit[0].prompt_token_ids) + assert with_limit_tokens < no_limit_tokens + + +def test_pooling_params(llm: LLM): + def get_outputs(use_activation): + outputs = llm.score( + TEXTS_1[0], + TEXTS_2[0], + pooling_params=PoolingParams(use_activation=use_activation), + use_tqdm=False, + ) + return torch.tensor([x.outputs.score for x in outputs]) + + default = get_outputs(use_activation=None) + w_activation = get_outputs(use_activation=True) + wo_activation = get_outputs(use_activation=False) + + assert torch.allclose(default, w_activation, atol=1e-2), ( + "Default should use activation." + ) + assert not torch.allclose(w_activation, wo_activation, atol=1e-2), ( + "wo_activation should not use activation." + ) + assert torch.allclose(softmax(wo_activation), w_activation, atol=1e-2), ( + "w_activation should be close to activation(wo_activation)." + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_cross_encoder_online.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_cross_encoder_online.py new file mode 100644 index 0000000000000000000000000000000000000000..04949846d05fb2cd7b1031048122f35a103e8452 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_cross_encoder_online.py @@ -0,0 +1,546 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import requests +import torch +import torch.nn.functional as F + +from tests.utils import RemoteOpenAIServer +from vllm.entrypoints.pooling.pooling.protocol import PoolingResponse +from vllm.entrypoints.pooling.scoring.protocol import RerankResponse, ScoreResponse +from vllm.platforms import current_platform + +MODEL_NAME = "BAAI/bge-reranker-base" +DTYPE = "half" +input_text = "This product was excellent and exceeded my expectations" +input_tokens = [0, 3293, 12996, 509, 40881, 136, 204839, 297, 759, 202702, 2] + + +TEXTS_1 = [ + "What is the capital of France?", + "What is the capital of Germany?", +] + +TEXTS_2 = [ + "The capital of France is Paris.", + "The capital of Germany is Berlin.", +] + + +@pytest.fixture(scope="module") +def server(): + args = ["--enforce-eager", "--max-model-len", "100", "--dtype", DTYPE] + + # ROCm: Use Flex Attention to support encoder-only self-attention. + if current_platform.is_rocm(): + args.extend(["--attention-backend", "FLEX_ATTENTION"]) + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.fixture(scope="module") +def hf_model(hf_runner): + return hf_runner(MODEL_NAME, is_cross_encoder=True) + + +@pytest.mark.asyncio +async def test_basic(server: RemoteOpenAIServer): + # test /v1/models + response = requests.get(server.url_for("/v1/models")) + served_model = response.json()["data"][0]["id"] + assert served_model == MODEL_NAME + + # test /tokenize + response = requests.post( + server.url_for("/tokenize"), + json={"model": MODEL_NAME, "prompt": input_text}, + ) + assert response.json()["tokens"] == input_tokens + + +@pytest.mark.asyncio +async def test_score_api_queries_str_1_documents_str_1( + hf_model, server: RemoteOpenAIServer +): + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": TEXTS_1[0], + "documents": TEXTS_2[0], + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 1 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict([[TEXTS_1[0], TEXTS_2[0]]]).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_score_api_queries_str_1_documents_str_n( + hf_model, server: RemoteOpenAIServer +): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[0], TEXTS_2[1]], + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": TEXTS_1[0], + "documents": TEXTS_2, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 2 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict(text_pairs).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_score_api_queries_str_n_documents_str_n( + hf_model, server: RemoteOpenAIServer +): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[1], TEXTS_2[1]], + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": TEXTS_1, + "documents": TEXTS_2, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 2 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict(text_pairs).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_score_api_queries_vs_documents(hf_model, server: RemoteOpenAIServer): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[1], TEXTS_2[1]], + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": TEXTS_1, + "documents": TEXTS_2, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 2 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict(text_pairs).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_score_api_queries_vs_items(hf_model, server: RemoteOpenAIServer): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[1], TEXTS_2[1]], + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": TEXTS_1, + "items": TEXTS_2, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 2 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict(text_pairs).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_score_api_text_1_vs_text_2(hf_model, server: RemoteOpenAIServer): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[1], TEXTS_2[1]], + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "text_1": TEXTS_1, + "text_2": TEXTS_2, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 2 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict(text_pairs).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_score_api_data_1_vs_data_2(hf_model, server: RemoteOpenAIServer): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[1], TEXTS_2[1]], + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "data_1": TEXTS_1, + "data_2": TEXTS_2, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 2 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict(text_pairs).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_rerank_api_texts(server: RemoteOpenAIServer): + query = "What is the capital of France?" + documents = [ + "The capital of Brazil is Brasilia.", + "The capital of France is Paris.", + ] + + rerank_response = requests.post( + server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": documents, + }, + ) + rerank_response.raise_for_status() + rerank = RerankResponse.model_validate(rerank_response.json()) + + assert rerank.id is not None + assert rerank.results is not None + assert len(rerank.results) == 2 + assert rerank.results[0].relevance_score >= 0.9 + assert rerank.results[1].relevance_score <= 0.01 + + +@pytest.mark.asyncio +async def test_rerank_api_top_n(server: RemoteOpenAIServer): + query = "What is the capital of France?" + documents = [ + "The capital of Brazil is Brasilia.", + "The capital of France is Paris.", + "Cross-encoder models are neat", + ] + + rerank_response = requests.post( + server.url_for("rerank"), + json={"model": MODEL_NAME, "query": query, "documents": documents, "top_n": 2}, + ) + rerank_response.raise_for_status() + rerank = RerankResponse.model_validate(rerank_response.json()) + + assert rerank.id is not None + assert rerank.results is not None + assert len(rerank.results) == 2 + assert rerank.results[0].relevance_score >= 0.9 + assert rerank.results[1].relevance_score <= 0.01 + + +@pytest.mark.asyncio +async def test_rerank_api_max_model_len(server: RemoteOpenAIServer): + query = "What is the capital of France?" * 100 + documents = [ + "The capital of Brazil is Brasilia.", + "The capital of France is Paris.", + ] + + rerank_response = requests.post( + server.url_for("rerank"), + json={"model": MODEL_NAME, "query": query, "documents": documents}, + ) + assert rerank_response.status_code == 400 + # Assert just a small fragments of the response + assert "Please reduce the length of the input prompt" in rerank_response.text + + +@pytest.mark.asyncio +async def test_score_api_max_model_len(server: RemoteOpenAIServer): + queries = "What is the capital of France?" * 20 + documents = [ + "The capital of Brazil is Brasilia.", + "The capital of France is Paris.", + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": queries, + "documents": documents, + }, + ) + assert score_response.status_code == 400 + # Assert just a small fragments of the response + assert "Please reduce the length of the input prompt" in score_response.text + + # Test truncation + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": queries, + "documents": documents, + "truncate_prompt_tokens": 101, + }, + ) + assert score_response.status_code == 400 + assert "Please request a smaller truncation size." in score_response.text + + +@pytest.mark.asyncio +async def test_invocations(server: RemoteOpenAIServer): + query = "What is the capital of France?" + documents = [ + "The capital of Brazil is Brasilia.", + "The capital of France is Paris.", + ] + + request_args = { + "model": MODEL_NAME, + "query": query, + "documents": documents, + } + + rerank_response = requests.post(server.url_for("rerank"), json=request_args) + rerank_response.raise_for_status() + + invocation_response = requests.post( + server.url_for("invocations"), json=request_args + ) + invocation_response.raise_for_status() + + rerank_output = rerank_response.json() + invocation_output = invocation_response.json() + + assert rerank_output.keys() == invocation_output.keys() + for rerank_result, invocations_result in zip( + rerank_output["results"], invocation_output["results"] + ): + assert rerank_result.keys() == invocations_result.keys() + assert rerank_result["relevance_score"] == pytest.approx( + invocations_result["relevance_score"], rel=0.01 + ) + + +@pytest.mark.asyncio +async def test_use_activation(server: RemoteOpenAIServer): + async def get_outputs(use_activation): + query = "What is the capital of France?" + documents = [ + "The capital of Brazil is Brasilia.", + "The capital of France is Paris.", + ] + + response = requests.post( + server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": documents, + "use_activation": use_activation, + }, + ) + outputs = response.json() + + return torch.tensor([x["relevance_score"] for x in outputs["results"]]) + + default = await get_outputs(use_activation=None) + w_activation = await get_outputs(use_activation=True) + wo_activation = await get_outputs(use_activation=False) + + assert torch.allclose(default, w_activation, atol=1e-2), ( + "Default should use activation." + ) + assert not torch.allclose(w_activation, wo_activation, atol=1e-2), ( + "wo_activation should not use activation." + ) + assert torch.allclose(F.sigmoid(wo_activation), w_activation, atol=1e-2), ( + "w_activation should be close to activation(wo_activation)." + ) + + +@pytest.mark.asyncio +async def test_pooling_classify(server: RemoteOpenAIServer): + response = requests.post( + server.url_for("pooling"), + json={ + "model": MODEL_NAME, + "input": input_text, + "encoding_format": "float", + "task": "classify", + }, + ) + poolings = PoolingResponse.model_validate(response.json()) + assert len(poolings.data) == 1 + assert len(poolings.data[0].data) == 1 + + +@pytest.mark.asyncio +async def test_rerank_max_tokens_per_doc( + server: RemoteOpenAIServer, +): + """Test that max_tokens_per_doc actually reduces the token count.""" + query = "What is the capital of France?" + # Use a doc that fits within max_model_len=100 (query ~8 tokens + 4 special) + long_doc = "The capital of France is Paris. " * 10 # ~70 tokens + + # Without max_tokens_per_doc + response_no_limit = requests.post( + server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": [long_doc], + "truncate_prompt_tokens": 99, + }, + ) + response_no_limit.raise_for_status() + rerank_no_limit = RerankResponse.model_validate(response_no_limit.json()) + + # With max_tokens_per_doc + response_with_limit = requests.post( + server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": [long_doc], + "max_tokens_per_doc": 10, + }, + ) + response_with_limit.raise_for_status() + rerank_with_limit = RerankResponse.model_validate(response_with_limit.json()) + + assert rerank_with_limit.usage.prompt_tokens < rerank_no_limit.usage.prompt_tokens + + +@pytest.mark.asyncio +async def test_rerank_max_tokens_per_doc_validation( + server: RemoteOpenAIServer, +): + """Test that max_tokens_per_doc validation works correctly.""" + query = "What is the capital of France?" + documents = ["The capital of France is Paris."] + + # Test with max_tokens_per_doc=0 (should succeed — means no truncation) + response = requests.post( + server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": documents, + "max_tokens_per_doc": 0, + }, + ) + response.raise_for_status() + + # Test with invalid max_tokens_per_doc (negative) + response = requests.post( + server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": documents, + "max_tokens_per_doc": -5, + }, + ) + assert response.status_code == 400 + assert "max_tokens_per_doc must be a non-negative integer" in response.text + + +@pytest.mark.asyncio +@pytest.mark.parametrize("task", ["embed", "token_embed", "token_classify", "plugin"]) +async def test_pooling_not_supported(server: RemoteOpenAIServer, task: str): + response = requests.post( + server.url_for("pooling"), + json={ + "model": MODEL_NAME, + "input": input_text, + "encoding_format": "float", + "task": task, + }, + ) + assert response.json()["error"]["type"] == "BadRequestError" + if task == "plugin": + err_msg = "No IOProcessor plugin installed." + elif task == "token_classify": + err_msg = "Try switching the model's pooling_task via" + else: + err_msg = f"Unsupported task: {task!r}" + assert response.json()["error"]["message"].startswith(err_msg) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_cross_encoder_online_vision.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_cross_encoder_online_vision.py new file mode 100644 index 0000000000000000000000000000000000000000..bd414ac56a2117260ba4884683705cb7e38fc8cf --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_cross_encoder_online_vision.py @@ -0,0 +1,516 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import pytest +import requests + +from tests.utils import VLLM_PATH, RemoteOpenAIServer +from vllm.entrypoints.pooling.scoring.protocol import RerankResponse, ScoreResponse +from vllm.multimodal.utils import encode_image_url, fetch_image +from vllm.platforms import current_platform + +MODEL_NAME = "Qwen/Qwen3-VL-Reranker-2B" +HF_OVERRIDES = { + "architectures": ["Qwen3VLForSequenceClassification"], + "classifier_from_token": ["no", "yes"], + "is_original_qwen3_reranker": True, +} + +ROCM_ATTN_BACKENDS = [ + "ROCM_ATTN", + "ROCM_AITER_FA", + "TRITON_ATTN", + "FLEX_ATTENTION", +] + +ATTN_BACKENDS = ROCM_ATTN_BACKENDS if current_platform.is_rocm() else ["auto"] + +# Per-backend tolerance with explicit entries; "default" is the fallback +BACKEND_TOL: dict[str, float] = { + "default": 0.05, # 5% tolerance for other backends (e.g. FLASH_ATTN) + # Relaxed tolerances for ROCm attn + # See: https://github.com/vllm-project/vllm/issues/35569 + "ROCM_ATTN": 0.09, # gfx950:~8.45%, gfx942:~3.70% + "ROCM_AITER_FA": 0.045, # gfx950:~2.00%, gfx942:~0.80% + "TRITON_ATTN": 0.045, # gfx950:~3.00%, gfx942:~2.20% + "FLEX_ATTENTION": 0.045, # gfx950:~3.25%, gfx942:~1.10% +} + +# Some ROCm attention backends show small absolute drift on the low +# text-vs-text probability even though larger scores remain well inside the +# relative tolerance. The absolute drift is uniform across score magnitudes +# (~0.005-0.010), so it only exceeds the relative tolerance for the small +# ~0.10 text-vs-text value. Keep the relative tolerances tight and add only a +# small absolute floor for the affected backends. +# TRITON_ATTN: gfx942/ROCm 7.2 drifts ~0.008 abs on text-vs-text (~7.9% rel). +BACKEND_ABS_TOL: dict[str, float] = { + "default": 0.0, + "ROCM_AITER_FA": 0.005, + "TRITON_ATTN": 0.009, + "FLEX_ATTENTION": 0.006, +} + +# ROCm: disable skinny GEMM to avoid non-deterministic results from +# atomic reductions in wvSplitKrc kernel. +# See: https://github.com/vllm-project/vllm/pull/33493#issuecomment-3906083975 +ROCM_ENV_OVERRIDES = ( + {"VLLM_ROCM_USE_SKINNY_GEMM": "0"} if current_platform.is_rocm() else {} +) +# ROCm: disable prefix caching and eliminate batch variance to reduce +# test flakiness. +ROCM_EXTRA_ARGS = ( + ["--no-enable-prefix-caching", "--max-num-seqs", "1"] + if current_platform.is_rocm() + else [] +) + + +def get_tol(backend: str) -> float: + return BACKEND_TOL.get(backend, BACKEND_TOL["default"]) + + +def get_abs_tol(backend: str) -> float: + return BACKEND_ABS_TOL.get(backend, BACKEND_ABS_TOL["default"]) + + +def assert_score(actual: float, expected: float, backend: str, label: str): + tol = get_tol(backend) + abs_tol = get_abs_tol(backend) + diff = abs(actual - expected) + rel_diff = diff / abs(expected) if expected != 0 else diff + print( + f"[{backend}] {label}: actual={actual:.6f} expected={expected:.6f} " + f"diff={diff:.6f} rel_diff={rel_diff:.4f} tol={tol} abs_tol={abs_tol}" + ) + assert actual == pytest.approx(expected, rel=tol, abs=abs_tol), ( + f"[{backend}] {label}: score mismatch — " + f"actual={actual:.6f}, expected={expected:.6f}, " + f"rel_diff={rel_diff:.4f}, tol={tol}, abs_tol={abs_tol}" + ) + + +query = "A cat standing in the snow." +document = "This product was excellent and exceeded my expectations." +image_url = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/cat_snow.jpg" +documents = [ + { + "type": "text", + "text": document, + }, + { + "type": "image_url", + "image_url": {"url": image_url}, + }, + { + "type": "image_url", + "image_url": {"url": encode_image_url(fetch_image(image_url))}, + }, +] + +TEXT_VS_TEXT = 0.10040374100208282 +TEXT_VS_IMAGE = 0.7423753142356873 +TEXT_VS_TEXT_PLUS_IMAGE = 0.5298863053321838 + + +@pytest.fixture(scope="module", params=ATTN_BACKENDS) +def server(request): + backend = request.param + print(f"\n=== Starting server with attention backend: {backend} ===") + args = [ + "--enforce-eager", + "--max-model-len", + "8192", + "--chat-template", + str(VLLM_PATH / "examples/pooling/score/template/qwen3_vl_reranker.jinja"), + ] + + env = dict() + if backend != "auto": + args += ["--attention-config", json.dumps({"backend": backend})] + args += ROCM_EXTRA_ARGS + + env = dict(ROCM_ENV_OVERRIDES) + if backend != "ROCM_AITER_FA": + env["VLLM_ROCM_USE_AITER"] = "0" + + with RemoteOpenAIServer( + MODEL_NAME, args, override_hf_configs=HF_OVERRIDES, env_dict=env + ) as remote_server: + print(f"=== Server ready with backend: {backend} ===") + yield remote_server, backend + + +@pytest.mark.asyncio +async def test_score_api_queries_str_documents_str( + server: tuple[RemoteOpenAIServer, str], +): + remote_server, backend = server + score_response = requests.post( + remote_server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": query, + "documents": document, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 1 + assert score.usage.prompt_tokens == 81 + assert_score(score.data[0].score, TEXT_VS_TEXT, backend, "text_vs_text") + + +@pytest.mark.asyncio +async def test_score_api_queries_str_documents_text_content( + server: tuple[RemoteOpenAIServer, str], +): + remote_server, backend = server + score_response = requests.post( + remote_server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": query, + "documents": {"content": [documents[0]]}, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 1 + assert score.usage.prompt_tokens == 81 + assert_score(score.data[0].score, TEXT_VS_TEXT, backend, "text_vs_text") + + +@pytest.mark.asyncio +async def test_score_api_queries_str_documents_image_url_content( + server: tuple[RemoteOpenAIServer, str], +): + remote_server, backend = server + score_response = requests.post( + remote_server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": query, + "documents": {"content": [documents[1]]}, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 1 + assert score.usage.prompt_tokens == 98 + assert_score(score.data[0].score, TEXT_VS_IMAGE, backend, "text_vs_image") + + +@pytest.mark.asyncio +async def test_score_api_queries_str_documents_image_base64_content( + server: tuple[RemoteOpenAIServer, str], +): + remote_server, backend = server + score_response = requests.post( + remote_server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": query, + "documents": {"content": [documents[2]]}, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 1 + assert score.usage.prompt_tokens == 98 + assert_score(score.data[0].score, TEXT_VS_IMAGE, backend, "text_vs_image_base64") + + +@pytest.mark.asyncio +async def test_score_api_queries_str_documents_image_url_plus_text_content( + server: tuple[RemoteOpenAIServer, str], +): + remote_server, backend = server + score_response = requests.post( + remote_server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": query, + "documents": {"content": [documents[0], documents[1]]}, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 1 + assert score.usage.prompt_tokens == 107 + assert_score( + score.data[0].score, TEXT_VS_TEXT_PLUS_IMAGE, backend, "text_vs_text_plus_image" + ) + + +@pytest.mark.asyncio +async def test_score_api_queries_str_documents_list( + server: tuple[RemoteOpenAIServer, str], +): + remote_server, backend = server + score_response = requests.post( + remote_server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": query, + "documents": [ + document, + {"content": [documents[0]]}, + {"content": [documents[1]]}, + {"content": [documents[0], documents[1]]}, + ], + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 4 + assert score.usage.prompt_tokens == 367 + assert_score(score.data[0].score, TEXT_VS_TEXT, backend, "list[0]_text_vs_text") + assert_score(score.data[1].score, TEXT_VS_TEXT, backend, "list[1]_text_vs_text") + assert_score(score.data[2].score, TEXT_VS_IMAGE, backend, "list[2]_text_vs_image") + assert_score( + score.data[3].score, + TEXT_VS_TEXT_PLUS_IMAGE, + backend, + "list[3]_text_vs_text_plus_image", + ) + + +@pytest.mark.asyncio +async def test_rerank_api_queries_str_documents_list( + server: tuple[RemoteOpenAIServer, str], +): + remote_server, backend = server + rerank_response = requests.post( + remote_server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": [ + document, + {"content": [documents[0]]}, + {"content": [documents[1]]}, + {"content": [documents[0], documents[1]]}, + ], + }, + ) + rerank_response.raise_for_status() + rerank = RerankResponse.model_validate(rerank_response.json()) + + assert rerank.id is not None + assert rerank.model is not None + assert rerank.usage is not None + assert len(rerank.results) == 4 + + rerank.results.sort(key=lambda x: x.index) + assert_score( + rerank.results[0].relevance_score, + TEXT_VS_TEXT, + backend, + "rerank[0]_text_vs_text", + ) + assert_score( + rerank.results[1].relevance_score, + TEXT_VS_TEXT, + backend, + "rerank[1]_text_vs_text", + ) + assert_score( + rerank.results[2].relevance_score, + TEXT_VS_IMAGE, + backend, + "rerank[2]_text_vs_image", + ) + assert_score( + rerank.results[3].relevance_score, + TEXT_VS_TEXT_PLUS_IMAGE, + backend, + "rerank[3]_text_vs_text_plus_image", + ) + + +@pytest.mark.asyncio +async def test_score_api_queries_list_documents_list( + server: tuple[RemoteOpenAIServer, str], +): + remote_server, backend = server + score_response = requests.post( + remote_server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": [query] * 4, + "documents": [ + document, + {"content": [documents[0]]}, + {"content": [documents[1]]}, + {"content": [documents[0], documents[1]]}, + ], + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 4 + assert score.usage.prompt_tokens == 367 + assert_score(score.data[0].score, TEXT_VS_TEXT, backend, "paired[0]_text_vs_text") + assert_score(score.data[1].score, TEXT_VS_TEXT, backend, "paired[1]_text_vs_text") + assert_score(score.data[2].score, TEXT_VS_IMAGE, backend, "paired[2]_text_vs_image") + assert_score( + score.data[3].score, + TEXT_VS_TEXT_PLUS_IMAGE, + backend, + "paired[3]_text_vs_text_plus_image", + ) + + +INSTRUCTION = ( + "Given a multimodal retrieval query, retrieve candidates that " + "visually or textually match the requested scene, object, or action." +) + + +@pytest.mark.asyncio +async def test_score_api_instruction_field( + server: tuple[RemoteOpenAIServer, str], +): + remote_server, _ = server + + default_response = requests.post( + remote_server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": query, + "documents": document, + }, + ) + default_response.raise_for_status() + default_score = ScoreResponse.model_validate(default_response.json()) + + instruction_response = requests.post( + remote_server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": query, + "documents": document, + "instruction": INSTRUCTION, + }, + ) + instruction_response.raise_for_status() + instruction_score = ScoreResponse.model_validate(instruction_response.json()) + + assert instruction_score.id is not None + assert instruction_score.data is not None + assert len(instruction_score.data) == 1 + assert instruction_score.usage.prompt_tokens > default_score.usage.prompt_tokens + + +@pytest.mark.asyncio +async def test_rerank_api_instruction_field( + server: tuple[RemoteOpenAIServer, str], +): + remote_server, _ = server + + doc_list = [ + document, + {"content": [documents[0]]}, + {"content": [documents[1]]}, + {"content": [documents[0], documents[1]]}, + ] + + default_response = requests.post( + remote_server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": doc_list, + }, + ) + default_response.raise_for_status() + default_rerank = RerankResponse.model_validate(default_response.json()) + + instruction_response = requests.post( + remote_server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": doc_list, + "instruction": INSTRUCTION, + }, + ) + instruction_response.raise_for_status() + instruction_rerank = RerankResponse.model_validate(instruction_response.json()) + + assert instruction_rerank.id is not None + assert instruction_rerank.model is not None + assert instruction_rerank.usage is not None + assert len(instruction_rerank.results) == len(default_rerank.results) + assert instruction_rerank.usage.prompt_tokens > default_rerank.usage.prompt_tokens + + +@pytest.mark.asyncio +async def test_rerank_api_instruction_field_matches_chat_template_kwargs( + server: tuple[RemoteOpenAIServer, str], +): + remote_server, _ = server + + doc_list = [ + document, + {"content": [documents[0]]}, + {"content": [documents[1]]}, + {"content": [documents[0], documents[1]]}, + ] + + field_response = requests.post( + remote_server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": doc_list, + "instruction": INSTRUCTION, + }, + ) + field_response.raise_for_status() + field_rerank = RerankResponse.model_validate(field_response.json()) + + kwargs_response = requests.post( + remote_server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": doc_list, + "chat_template_kwargs": {"instruction": INSTRUCTION}, + }, + ) + kwargs_response.raise_for_status() + kwargs_rerank = RerankResponse.model_validate(kwargs_response.json()) + + assert kwargs_rerank.usage.prompt_tokens == field_rerank.usage.prompt_tokens + + field_scores = [ + r.relevance_score for r in sorted(field_rerank.results, key=lambda x: x.index) + ] + kwargs_scores = [ + r.relevance_score for r in sorted(kwargs_rerank.results, key=lambda x: x.index) + ] + assert field_scores == pytest.approx(kwargs_scores) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_late_interaction_offline.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_late_interaction_offline.py new file mode 100644 index 0000000000000000000000000000000000000000..cf1a5fcbb2574e7d133d8797fd65e1bd07e07223 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_late_interaction_offline.py @@ -0,0 +1,119 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import weakref + +import pytest + +from vllm import LLM +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.platforms import current_platform + +from .util import ColBERTScoringHfRunner + +MODEL_NAME = "answerdotai/answerai-colbert-small-v1" +COLBERT_DIM = 96 + +LINEAR_WEIGHTS_KEY = "linear.weight" +PROMPT = "The chef prepared a delicious meal." + +TEXTS_1 = [ + "What is the capital of France?", + "What is the capital of Germany?", +] + +TEXTS_2 = [ + "The capital of France is Paris.", + "The capital of Germany is Berlin.", +] + +DTYPE = "half" + + +@pytest.fixture(scope="module") +def llm(): + # ROCm: Use FLEX_ATTENTION backend as it's the only attention backend + # that supports encoder-only models on ROCm. + attention_config = None + if current_platform.is_rocm(): + attention_config = {"backend": "FLEX_ATTENTION"} + + # pytest caches the fixture so we use weakref.proxy to + # enable garbage collection + llm = LLM( + model=MODEL_NAME, + max_num_batched_tokens=32768, + tensor_parallel_size=1, + gpu_memory_utilization=0.75, + enforce_eager=True, + seed=0, + attention_config=attention_config, + ) + + yield weakref.proxy(llm) + + del llm + + cleanup_dist_env_and_memory() + + +@pytest.fixture(scope="module") +def hf_model(): + return ColBERTScoringHfRunner( + model_name=MODEL_NAME, linear_weights_key=LINEAR_WEIGHTS_KEY + ) + + +@pytest.mark.skip_global_cleanup +def test_1_to_1(llm, hf_model): + text_pair = [TEXTS_1[0], TEXTS_2[0]] + + hf_outputs = hf_model.predict([text_pair]).tolist() + vllm_outputs = [ + output.outputs.score for output in llm.score(text_pair[0], text_pair[1]) + ] + + assert len(vllm_outputs) == 1 + assert len(hf_outputs) == 1 + + assert hf_outputs[0] == pytest.approx(vllm_outputs[0], rel=0.01) + + +@pytest.mark.skip_global_cleanup +def test_1_to_n(llm, hf_model): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[0], TEXTS_2[1]], + ] + + hf_outputs = hf_model.predict(text_pairs).tolist() + vllm_outputs = [output.outputs.score for output in llm.score(TEXTS_1[0], TEXTS_2)] + + assert len(vllm_outputs) == 2 + assert len(hf_outputs) == 2 + + assert hf_outputs[0] == pytest.approx(vllm_outputs[0], rel=0.01) + assert hf_outputs[1] == pytest.approx(vllm_outputs[1], rel=0.01) + + +@pytest.mark.skip_global_cleanup +def test_n_to_n(llm, hf_model): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[1], TEXTS_2[1]], + ] + + hf_outputs = hf_model.predict(text_pairs).tolist() + vllm_outputs = [output.outputs.score for output in llm.score(TEXTS_1, TEXTS_2)] + + assert len(vllm_outputs) == 2 + assert len(hf_outputs) == 2 + + assert hf_outputs[0] == pytest.approx(vllm_outputs[0], rel=0.01) + assert hf_outputs[1] == pytest.approx(vllm_outputs[1], rel=0.01) + + +def test_token_embed(llm): + outputs = llm.encode(PROMPT, pooling_task="token_embed", use_tqdm=False) + assert len(outputs) == 1 + assert outputs[0].outputs.data.shape == (9, COLBERT_DIM) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_late_interaction_offline_vision.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_late_interaction_offline_vision.py new file mode 100644 index 0000000000000000000000000000000000000000..550c3e4ad34edd0ce958652f6cc4e4716f1471a6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_late_interaction_offline_vision.py @@ -0,0 +1,93 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import weakref + +import pytest + +from vllm import LLM +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.platforms import current_platform + +from .util import make_base64_image, make_image_mm_param + +MODEL_NAME = "vidore/colpali-v1.3-hf" + + +@pytest.fixture(scope="module") +def llm(): + # ROCm: Use FLEX_ATTENTION backend as it's the only attention backend + # that supports encoder-only models on ROCm. + attention_config = None + if current_platform.is_rocm(): + attention_config = {"backend": "FLEX_ATTENTION"} + + # pytest caches the fixture so we use weakref.proxy to + # enable garbage collection + llm = LLM( + model=MODEL_NAME, + max_num_batched_tokens=32768, + tensor_parallel_size=1, + gpu_memory_utilization=0.75, + enforce_eager=True, + seed=0, + attention_config=attention_config, + ) + + yield weakref.proxy(llm) + + del llm + + cleanup_dist_env_and_memory() + + +@pytest.mark.skip_global_cleanup +def test_query_text_vs_docs_image(llm): + """Score a text query against image documents via the multimodal path.""" + red_image = make_base64_image(64, 64, color=(255, 0, 0)) + blue_image = make_base64_image(64, 64, color=(0, 0, 255)) + + query = "Describe the red object" + image_docs = [ + make_image_mm_param(red_image), + make_image_mm_param(blue_image), + ] + + scores = llm.score(query, image_docs) + + assert len(scores) == 2 + assert scores[0].outputs.score > scores[1].outputs.score + + +@pytest.mark.skip_global_cleanup +def test_query_text_vs_docs_mix(llm) -> None: + """Score a text query against a mix of text and image documents.""" + red_image = make_base64_image(64, 64, color=(255, 0, 0)) + + query = "What is the capital of France?" + documents: list = [ + "The capital of France is Paris.", + make_image_mm_param(red_image), + ] + + scores = llm.score(query, documents) + + assert len(scores) == 2 + assert scores[0].outputs.score > scores[1].outputs.score + + +@pytest.mark.skip_global_cleanup +def test_query_image_vs_docs_text(llm) -> None: + """Score an image query against text documents.""" + red_image = make_base64_image(64, 64, color=(255, 0, 0)) + image_query = make_image_mm_param(red_image, text="red color") + + documents = [ + "Describe the red object.", + "The capital of France is Paris.", + ] + + scores = llm.score(image_query, documents) + + assert len(scores) == 2 + assert scores[0].outputs.score > scores[1].outputs.score diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_late_interaction_online.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_late_interaction_online.py new file mode 100644 index 0000000000000000000000000000000000000000..b5e91671ec046db89b4402c9c3cf458e81137d46 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_late_interaction_online.py @@ -0,0 +1,237 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Online API tests for ColBERT late interaction scoring.""" + +import pytest +import requests + +from tests.utils import RemoteOpenAIServer +from vllm.entrypoints.pooling.scoring.protocol import RerankResponse, ScoreResponse + +from .util import ColBERTScoringHfRunner + +MODEL_NAME = "answerdotai/answerai-colbert-small-v1" +COLBERT_DIM = 96 +MAX_MODEL_LEN = 512 +LINEAR_WEIGHTS_KEY = "linear.weight" + +TEXTS_1 = [ + "What is the capital of France?", + "What is the capital of Germany?", +] + +TEXTS_2 = [ + "The capital of France is Paris.", + "The capital of Germany is Berlin.", +] + + +@pytest.fixture(scope="module", params=[True, False]) +def server(request): + args = [ + "--max-model-len", + str(MAX_MODEL_LEN), + ] + + # Test run pooling score MaxSim on worker side (GPU) + # aka flash-late-interaction + if not request.param: + args += ["--no-enable-flash-late-interaction"] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.fixture(scope="module") +def hf_model(): + return ColBERTScoringHfRunner( + model_name=MODEL_NAME, linear_weights_key=LINEAR_WEIGHTS_KEY + ) + + +@pytest.mark.asyncio +async def test_score_api_queries_str_1_documents_str_1( + hf_model, server: RemoteOpenAIServer +): + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": TEXTS_1[0], + "documents": TEXTS_2[0], + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 1 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict([[TEXTS_1[0], TEXTS_2[0]]]).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_score_api_queries_str_1_documents_str_n( + hf_model, server: RemoteOpenAIServer +): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[0], TEXTS_2[1]], + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": TEXTS_1[0], + "documents": TEXTS_2, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 2 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict(text_pairs).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_score_api_queries_str_n_documents_str_n( + hf_model, server: RemoteOpenAIServer +): + text_pairs = [ + [TEXTS_1[0], TEXTS_2[0]], + [TEXTS_1[1], TEXTS_2[1]], + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": TEXTS_1, + "documents": TEXTS_2, + }, + ) + score_response.raise_for_status() + score = ScoreResponse.model_validate(score_response.json()) + + assert score.id is not None + assert score.data is not None + assert len(score.data) == 2 + + vllm_outputs = [d.score for d in score.data] + hf_outputs = hf_model.predict(text_pairs).tolist() + + for i in range(len(vllm_outputs)): + assert hf_outputs[i] == pytest.approx(vllm_outputs[i], rel=0.01) + + +@pytest.mark.asyncio +async def test_rerank_api_texts(server: RemoteOpenAIServer): + """Test ColBERT rerank endpoint.""" + query = "What is the capital of France?" + documents = [ + "The capital of Brazil is Brasilia.", + "The capital of France is Paris.", + ] + + rerank_response = requests.post( + server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": documents, + }, + ) + rerank_response.raise_for_status() + rerank = RerankResponse.model_validate(rerank_response.json()) + + assert rerank.id is not None + assert rerank.results is not None + assert len(rerank.results) == 2 + + paris_result = next(r for r in rerank.results if r.index == 1) + brazil_result = next(r for r in rerank.results if r.index == 0) + + assert paris_result.relevance_score > brazil_result.relevance_score + + +@pytest.mark.asyncio +async def test_rerank_api_top_n(server: RemoteOpenAIServer): + """Test ColBERT rerank with top_n parameter.""" + query = "What is the capital of France?" + documents = [ + "The capital of Brazil is Brasilia.", + "The capital of France is Paris.", + "Machine learning is a field of AI.", + ] + + rerank_response = requests.post( + server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": documents, + "top_n": 2, + }, + ) + rerank_response.raise_for_status() + rerank = RerankResponse.model_validate(rerank_response.json()) + + assert len(rerank.results) == 2 + assert rerank.results[0].index == 1 + + +@pytest.mark.asyncio +async def test_token_embed(server: RemoteOpenAIServer): + """Test ColBERT token_embed task via pooling endpoint.""" + text = "What is the capital of France?" + + pooling_response = requests.post( + server.url_for("pooling"), + json={ + "model": MODEL_NAME, + "input": text, + "task": "token_embed", + }, + ) + pooling_response.raise_for_status() + pooling = pooling_response.json() + + assert "data" in pooling + assert len(pooling["data"]) == 1 + + embeddings = pooling["data"][0]["data"] + assert isinstance(embeddings, list) + assert len(embeddings) > 0 + assert len(embeddings[0]) == COLBERT_DIM + + +@pytest.mark.asyncio +async def test_embed_not_supported(server: RemoteOpenAIServer): + """Test that ColBERT model does not support 'embed' task.""" + task = "embed" + text = "What is the capital of France?" + + response = requests.post( + server.url_for("pooling"), + json={ + "model": MODEL_NAME, + "input": text, + "task": task, + }, + ) + + assert response.json()["error"]["type"] == "BadRequestError" + assert response.json()["error"]["message"].startswith(f"Unsupported task: {task!r}") diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_late_interaction_online_vision.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_late_interaction_online_vision.py new file mode 100644 index 0000000000000000000000000000000000000000..749fa7b7b6ad3f107cc47897078e9eec1dfd35e5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/test_late_interaction_online_vision.py @@ -0,0 +1,193 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import requests + +from tests.entrypoints.pooling.scoring.util import ( + make_base64_image, + make_image_mm_param, +) +from tests.utils import RemoteOpenAIServer +from vllm.entrypoints.pooling.scoring.protocol import RerankResponse, ScoreResponse + +MODEL_NAME = "vidore/colpali-v1.3-hf" + + +@pytest.fixture(scope="module") +def server(): + with RemoteOpenAIServer(MODEL_NAME, []) as remote_server: + yield remote_server + + +@pytest.mark.asyncio +async def test_score_api_query_text_vs_docs_image(server: RemoteOpenAIServer): + query = "Describe the red object" + + red_image = make_base64_image(64, 64, color=(255, 0, 0)) + blue_image = make_base64_image(64, 64, color=(0, 0, 255)) + + documents = [ + make_image_mm_param(red_image), + make_image_mm_param(blue_image), + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": query, + "documents": documents, + }, + ) + score_response.raise_for_status() + scores = ScoreResponse.model_validate(score_response.json()) + + assert scores.id is not None + assert scores.data is not None + assert len(scores.data) == 2 + assert scores.data[0].score > scores.data[1].score + + +@pytest.mark.asyncio +async def test_score_api_query_text_vs_docs_mix(server: RemoteOpenAIServer): + red_image = make_base64_image(64, 64, color=(255, 0, 0)) + query = "What is the capital of France?" + documents: list = [ + "The capital of France is Paris.", + make_image_mm_param(red_image), + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": query, + "documents": documents, + }, + ) + score_response.raise_for_status() + scores = ScoreResponse.model_validate(score_response.json()) + + assert scores.id is not None + assert scores.data is not None + assert len(scores.data) == 2 + assert scores.data[0].score > scores.data[1].score + + +@pytest.mark.asyncio +async def test_score_api_query_image_vs_docs_text(server: RemoteOpenAIServer): + red_image = make_base64_image(64, 64, color=(255, 0, 0)) + image_query = make_image_mm_param(red_image, text="red color") + + documents = [ + "Describe the red object.", + "The capital of France is Paris.", + ] + + score_response = requests.post( + server.url_for("score"), + json={ + "model": MODEL_NAME, + "queries": image_query, + "documents": documents, + }, + ) + score_response.raise_for_status() + scores = ScoreResponse.model_validate(score_response.json()) + + assert scores.id is not None + assert scores.data is not None + assert len(scores.data) == 2 + assert scores.data[0].score > scores.data[1].score + + +@pytest.mark.asyncio +async def test_rerank_api_query_text_vs_docs_image(server: RemoteOpenAIServer): + query = "Describe the red object" + + red_image = make_base64_image(64, 64, color=(255, 0, 0)) + blue_image = make_base64_image(64, 64, color=(0, 0, 255)) + + documents = [ + make_image_mm_param(red_image), + make_image_mm_param(blue_image), + ] + + rerank_response = requests.post( + server.url_for("rerank"), + json={"model": MODEL_NAME, "query": query, "documents": documents}, + ) + + rerank_response.raise_for_status() + rerank = RerankResponse.model_validate(rerank_response.json()) + + assert rerank.id is not None + assert rerank.results is not None + assert len(rerank.results) == 2 + + red_result = next(r for r in rerank.results if r.index == 0) + blue_result = next(r for r in rerank.results if r.index == 1) + + assert red_result.relevance_score > blue_result.relevance_score + + +@pytest.mark.asyncio +async def test_rerank_api_query_text_vs_docs_mix(server: RemoteOpenAIServer): + red_image = make_base64_image(64, 64, color=(255, 0, 0)) + query = "What is the capital of France?" + documents: list = [ + "The capital of France is Paris.", + make_image_mm_param(red_image), + ] + + rerank_response = requests.post( + server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": query, + "documents": documents, + }, + ) + rerank_response.raise_for_status() + rerank = RerankResponse.model_validate(rerank_response.json()) + + assert rerank.id is not None + assert rerank.results is not None + assert len(rerank.results) == 2 + + result0 = next(r for r in rerank.results if r.index == 0) + result1 = next(r for r in rerank.results if r.index == 1) + + assert result0.relevance_score > result1.relevance_score + + +@pytest.mark.asyncio +async def test_rerank_api_query_image_vs_docs_text(server: RemoteOpenAIServer): + red_image = make_base64_image(64, 64, color=(255, 0, 0)) + image_query = make_image_mm_param(red_image, text="red color") + + documents = [ + "Describe the red object.", + "The capital of France is Paris.", + ] + + rerank_response = requests.post( + server.url_for("rerank"), + json={ + "model": MODEL_NAME, + "query": image_query, + "documents": documents, + }, + ) + rerank_response.raise_for_status() + rerank = RerankResponse.model_validate(rerank_response.json()) + + assert rerank.id is not None + assert rerank.results is not None + assert len(rerank.results) == 2 + + result0 = next(r for r in rerank.results if r.index == 0) + result1 = next(r for r in rerank.results if r.index == 1) + + assert result0.relevance_score > result1.relevance_score diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/util.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/util.py new file mode 100644 index 0000000000000000000000000000000000000000..5476329885d79f47b40642fe19827301afe21694 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/scoring/util.py @@ -0,0 +1,107 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from io import BytesIO + +import pybase64 as base64 +import torch +import torch.nn.functional as F +from huggingface_hub import hf_hub_download +from PIL import Image +from safetensors.torch import load_file +from transformers import AutoModel, AutoTokenizer + +from tests.conftest import HfRunner +from vllm.entrypoints.chat_utils import ( + ChatCompletionContentPartImageParam, + ChatCompletionContentPartTextParam, +) +from vllm.entrypoints.pooling.scoring.typing import ScoreMultiModalParam +from vllm.entrypoints.pooling.scoring.utils import compute_maxsim_score + + +class ColBERTScoringHfRunner(torch.nn.Module): + def __init__(self, model_name, linear_weights_key): + super().__init__() + + self.tokenizer = AutoTokenizer.from_pretrained(model_name) + + self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + + extra = {} + if self.device.type == "cpu": + extra["attn_implementation"] = "eager" + + self.model = AutoModel.from_pretrained( + model_name, + **extra, + ).to(self.device) + self.model.eval() + + path = hf_hub_download(model_name, filename="model.safetensors") + weights = load_file(path) + + self.linear_weight = weights[linear_weights_key].to(self.device).float() + + @torch.inference_mode() + def forward(self, texts): + embeddings = [] + for text in texts: + inputs = self.tokenizer(text, return_tensors="pt").to(self.device) + hidden = self.model(**inputs).last_hidden_state.float() + projected = F.linear(hidden, self.linear_weight.float()) + normalised = F.normalize(projected, p=2, dim=-1) + embeddings.append(normalised.squeeze(0).cpu()) + return embeddings + + @torch.inference_mode() + def predict(self, prompts: list[list[str]], *args, **kwargs): + hf_embeddings = [self(prompt) for prompt in prompts] + hf_outputs = [ + compute_maxsim_score(*map(torch.tensor, pair)).item() + for pair in hf_embeddings + ] + return torch.as_tensor(hf_outputs) + + +class EncoderScoringHfRunner(HfRunner): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs, is_sentence_transformer=True) + + @torch.inference_mode() + def predict(self, prompts: list[list[str]], *args, **kwargs): + hf_embeddings = [self.encode(prompt) for prompt in prompts] + hf_outputs = [ + F.cosine_similarity(*map(torch.tensor, pair), dim=0) + for pair in hf_embeddings + ] + return torch.as_tensor(hf_outputs) + + +def make_base64_image( + width: int = 64, height: int = 64, color: tuple[int, int, int] = (255, 0, 0) +) -> str: + """Create a small solid-color PNG image and return its base64 data URI.""" + img = Image.new("RGB", (width, height), color) + buf = BytesIO() + img.save(buf, format="PNG") + b64 = base64.b64encode(buf.getvalue()).decode() + return f"data:image/png;base64,{b64}" + + +def make_image_mm_param( + image_uri: str, + text: str | None = None, +) -> ScoreMultiModalParam: + """Build a ScoreMultiModalParam containing an image (and optional text).""" + content: list = [ + ChatCompletionContentPartImageParam( + type="image_url", + image_url={"url": image_uri}, + ), + ] + if text is not None: + content.append( + ChatCompletionContentPartTextParam(type="text", text=text), + ) + return ScoreMultiModalParam(content=content) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/test_utils.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/test_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..a3431458e686b9796ed1cfe9f6c9b09bd2ea277b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/test_utils.py @@ -0,0 +1,59 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import importlib +import importlib.util +import json +import warnings +from types import SimpleNamespace + +import numpy as np +import pytest +import torch + +from vllm.entrypoints.pooling.utils import encode_pooling_output_float_or_ndarray + + +def _pooling_output(data): + return SimpleNamespace(outputs=SimpleNamespace(data=data)) + + +def test_encode_pooling_output_float_or_ndarray_returns_numpy_array(): + output = _pooling_output(torch.tensor([1.0, 2.0, 3.0], dtype=torch.float32)) + + encoded = encode_pooling_output_float_or_ndarray(output) + + assert isinstance(encoded, np.ndarray) + np.testing.assert_allclose(encoded, [1.0, 2.0, 3.0]) + + +@pytest.mark.skipif( + importlib.util.find_spec("orjson") is None, + reason="orjson is not installed", +) +def test_orjson_serializes_numpy_array(): + from fastapi.responses import ORJSONResponse + + output = _pooling_output(torch.tensor([1.0, 2.0, 3.0], dtype=torch.float32)) + encoded = encode_pooling_output_float_or_ndarray(output) + + with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + response = ORJSONResponse(content={"embedding": encoded}) + assert json.loads(response.body)["embedding"] == pytest.approx([1.0, 2.0, 3.0]) + + +def test_encode_pooling_output_float_or_ndarray_falls_back_to_list(): + class DataWithUnsupportedNumpy: + def is_contiguous(self): + return True + + def numpy(self): + raise TypeError("unsupported dtype") + + def tolist(self): + return [1.0, 2.0, 3.0] + + output = _pooling_output(DataWithUnsupportedNumpy()) + + assert encode_pooling_output_float_or_ndarray(output) == [1.0, 2.0, 3.0] diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_classify/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_classify/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_classify/test_offline.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_classify/test_offline.py new file mode 100644 index 0000000000000000000000000000000000000000..1af9428a45a33c18ce61d89ecb0f4a71765dc7fd --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_classify/test_offline.py @@ -0,0 +1,77 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import weakref + +import pytest + +from vllm import LLM, PoolingRequestOutput +from vllm.config import PoolerConfig +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.tasks import PoolingTask + +MODEL_NAME = "jason9693/Qwen2.5-1.5B-apeach" + +prompt = "The chef prepared a delicious meal." +prompt_token_ids = [785, 29706, 10030, 264, 17923, 15145, 13] +num_labels = 2 + + +@pytest.fixture(scope="module") +def llm(): + # pytest caches the fixture so we use weakref.proxy to + # enable garbage collection + llm = LLM( + model=MODEL_NAME, + pooler_config=PoolerConfig(task="token_classify"), + max_num_batched_tokens=32768, + tensor_parallel_size=1, + gpu_memory_utilization=0.75, + enforce_eager=True, + seed=0, + ) + + yield weakref.proxy(llm) + + del llm + + cleanup_dist_env_and_memory() + + +@pytest.mark.skip_global_cleanup +def test_str_prompts(llm: LLM): + outputs = llm.encode(prompt, pooling_task="token_classify", use_tqdm=False) + assert len(outputs) == 1 + assert isinstance(outputs[0], PoolingRequestOutput) + assert outputs[0].prompt_token_ids == prompt_token_ids + assert outputs[0].outputs.data.shape == (len(prompt_token_ids), num_labels) + + +@pytest.mark.skip_global_cleanup +def test_token_ids_prompts(llm: LLM): + outputs = llm.encode( + [prompt_token_ids], pooling_task="token_classify", use_tqdm=False + ) + assert len(outputs) == 1 + assert isinstance(outputs[0], PoolingRequestOutput) + assert outputs[0].prompt_token_ids == prompt_token_ids + assert outputs[0].outputs.data.shape == (len(prompt_token_ids), num_labels) + + +@pytest.mark.skip_global_cleanup +def test_score_api(llm: LLM): + err_msg = "This model does not support the Scoring API." + with pytest.raises(ValueError, match=err_msg): + llm.score("ping", "pong", use_tqdm=False) + + +@pytest.mark.parametrize("task", ["classify", "embed", "token_embed", "plugin"]) +def test_unsupported_tasks(llm: LLM, task: PoolingTask, caplog_vllm): + if task == "plugin": + err_msg = "No IOProcessor plugin installed." + elif task == "classify": + err_msg = "Try switching the model's pooling_task via.+" + else: + err_msg = "Embedding API is not supported by this model.+" + + with pytest.raises(ValueError, match=err_msg): + llm.encode(prompt, pooling_task=task, use_tqdm=False) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_classify/test_online.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_classify/test_online.py new file mode 100644 index 0000000000000000000000000000000000000000..ef4adc00b9aaf0c3631c36794c8563223e1376e6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_classify/test_online.py @@ -0,0 +1,74 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import requests + +from tests.utils import RemoteOpenAIServer +from vllm.entrypoints.pooling.pooling.protocol import PoolingResponse + +MODEL_NAME = "jason9693/Qwen2.5-1.5B-apeach" +DTYPE = "float32" # Use float32 to avoid NaN issue +input_text = "This product was excellent and exceeded my expectations" +input_tokens = [1986, 1985, 572, 9073, 323, 33808, 847, 16665] + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--enforce-eager", + "--max-model-len", + "512", + "--dtype", + DTYPE, + "--pooler-config.task", + "token_classify", + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_pooling_token_classify(server: RemoteOpenAIServer, model_name: str): + task = "token_classify" + response = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "input": input_text, + "encoding_format": "float", + "task": task, + }, + ) + poolings = PoolingResponse.model_validate(response.json()) + assert len(poolings.data) == 1 + assert len(poolings.data[0].data) == 8 + assert len(poolings.data[0].data[0]) == 2 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("task", ["classify", "embed", "token_embed", "plugin"]) +async def test_pooling_not_supported( + server: RemoteOpenAIServer, model_name: str, task: str +): + response = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "input": input_text, + "encoding_format": "float", + "task": task, + }, + ) + assert response.json()["error"]["type"] == "BadRequestError" + + if task == "plugin": + err_msg = "No IOProcessor plugin installed." + elif task == "classify": + err_msg = "Try switching the model's pooling_task via" + else: + err_msg = f"Unsupported task: {task!r}" + assert response.json()["error"]["message"].startswith(err_msg) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_embed/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_embed/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_embed/test_offline.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_embed/test_offline.py new file mode 100644 index 0000000000000000000000000000000000000000..42f483ec970fd985767431fccdc6a36e47fdb0db --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_embed/test_offline.py @@ -0,0 +1,74 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import weakref + +import pytest + +from vllm import LLM, PoolingRequestOutput +from vllm.config import PoolerConfig +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.platforms import current_platform +from vllm.tasks import PoolingTask + +MODEL_NAME = "intfloat/multilingual-e5-small" + +prompt = "The chef prepared a delicious meal." +prompt_token_ids = [0, 581, 21861, 133888, 10, 8, 150, 60744, 109911, 5, 2] +embedding_size = 384 + + +@pytest.fixture(scope="module") +def llm(): + # ROCm: Use FLEX_ATTENTION backend as it's the only attention backend + # that supports encoder-only models on ROCm. + attention_config = None + if current_platform.is_rocm(): + attention_config = {"backend": "FLEX_ATTENTION"} + + # pytest caches the fixture so we use weakref.proxy to + # enable garbage collection + llm = LLM( + model=MODEL_NAME, + pooler_config=PoolerConfig(task="token_embed"), + max_num_batched_tokens=32768, + tensor_parallel_size=1, + gpu_memory_utilization=0.75, + enforce_eager=True, + seed=0, + attention_config=attention_config, + ) + assert embedding_size == llm.model_config.embedding_size + + yield weakref.proxy(llm) + + del llm + cleanup_dist_env_and_memory() + + +@pytest.mark.skip_global_cleanup +def test_str_prompts(llm: LLM): + outputs = llm.encode(prompt, pooling_task="token_embed", use_tqdm=False) + assert len(outputs) == 1 + assert isinstance(outputs[0], PoolingRequestOutput) + assert outputs[0].outputs.data.shape == (11, 384) + + +@pytest.mark.skip_global_cleanup +def test_token_ids_prompts(llm: LLM): + outputs = llm.encode([prompt_token_ids], pooling_task="token_embed", use_tqdm=False) + assert len(outputs) == 1 + assert isinstance(outputs[0], PoolingRequestOutput) + assert outputs[0].outputs.data.shape == (11, 384) + + +@pytest.mark.parametrize("task", ["embed", "classify", "token_classify", "plugin"]) +def test_unsupported_tasks(llm: LLM, task: PoolingTask, caplog_vllm): + if task == "plugin": + err_msg = "No IOProcessor plugin installed." + elif task == "embed": + err_msg = "Try switching the model's pooling_task via.+" + else: + err_msg = "Classification API is not supported by this model.+" + + with pytest.raises(ValueError, match=err_msg): + llm.encode(prompt, pooling_task=task, use_tqdm=False) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_embed/test_online.py b/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_embed/test_online.py new file mode 100644 index 0000000000000000000000000000000000000000..72b971165982e24c8a20eef745001e02474abfba --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/pooling/token_embed/test_online.py @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import pytest +import requests + +from tests.utils import RemoteOpenAIServer +from vllm.entrypoints.pooling.pooling.protocol import PoolingResponse + +MODEL_NAME = "intfloat/multilingual-e5-small" +DTYPE = "bfloat16" +input_text = "The best thing about vLLM is that it supports many different models" +input_tokens = [ + 0, + 581, + 2965, + 13580, + 1672, + 81, + 23708, + 594, + 83, + 450, + 442, + 8060, + 7, + 5941, + 12921, + 115774, + 2, +] + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--runner", + "pooling", + "--dtype", + DTYPE, + "--enforce-eager", + "--max-model-len", + "512", + "--pooler-config.task", + "token_embed", + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_pooling_token_embed(server: RemoteOpenAIServer, model_name: str): + task = "token_embed" + response = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "input": input_text, + "encoding_format": "float", + "task": task, + }, + ) + + poolings = PoolingResponse.model_validate(response.json()) + + assert len(poolings.data) == 1 + assert len(poolings.data[0].data) == len(input_tokens) + assert len(poolings.data[0].data[0]) == 384 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +@pytest.mark.parametrize("task", ["embed", "classify", "token_classify", "plugin"]) +async def test_pooling_not_supported( + server: RemoteOpenAIServer, model_name: str, task: str +): + response = requests.post( + server.url_for("pooling"), + json={ + "model": model_name, + "input": "test", + "encoding_format": "float", + "task": task, + }, + ) + assert response.json()["error"]["type"] == "BadRequestError" + + if task == "plugin": + err_msg = "No IOProcessor plugin installed." + elif task == "embed": + err_msg = "Try switching the model's pooling_task via" + else: + err_msg = f"Unsupported task: {task!r}" + assert response.json()["error"]["message"].startswith(err_msg) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/derender/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/derender/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/derender/test_derender.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/derender/test_derender.py new file mode 100644 index 0000000000000000000000000000000000000000..668e7c9ed975d536b66e4bbc32d0d36de09ee3f5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/derender/test_derender.py @@ -0,0 +1,924 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Tests for the /derender endpoints (postprocessing counterpart to /render).""" + +import httpx +import pytest +import pytest_asyncio + +from tests.utils import RemoteLaunchRenderServer +from vllm.tokenizers import get_tokenizer + +MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM" + + +@pytest.fixture(scope="module") +def server(): + with RemoteLaunchRenderServer(MODEL_NAME, []) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with httpx.AsyncClient( + base_url=server.url_for(""), timeout=30.0 + ) as http_client: + yield http_client + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +async def _render_chat(client: httpx.AsyncClient) -> dict: + """Render a minimal chat request and return the GenerateRequest dict.""" + resp = await client.post( + "/v1/chat/completions/render", + json={ + "model": MODEL_NAME, + "messages": [{"role": "user", "content": "Hello"}], + }, + ) + assert resp.status_code == 200 + return resp.json() + + +def _make_generate_response( + token_ids: list[int] | None, + request_id: str = "chatcmpl-test-id", + finish_reason: str = "stop", + logprobs: dict | None = None, + prompt_logprobs: list | None = None, + kv_transfer_params: dict | None = None, +) -> dict: + choice: dict = { + "index": 0, + "token_ids": token_ids, + "finish_reason": finish_reason, + "logprobs": logprobs, + } + return { + "request_id": request_id, + "choices": [choice], + "prompt_logprobs": prompt_logprobs, + "kv_transfer_params": kv_transfer_params, + } + + +def _make_logprobs_with_placeholders(token_id: int = 1234) -> dict: + entry = { + "token": f"token_id:{token_id}", + "logprob": -1.0, + "bytes": None, + "top_logprobs": [ + {"token": f"token_id:{token_id + 1}", "logprob": -2.0, "bytes": None} + ], + } + return {"content": [entry]} + + +# --------------------------------------------------------------------------- +# Chat derender tests +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_derender_chat_roundtrip(client): + """Render then derender: decoded content should be a non-empty string.""" + gen_req = await _render_chat(client) + # Use the first 5 rendered token IDs as synthetic "generated" tokens. + synthetic_ids = gen_req["token_ids"][:5] + + response = await client.post( + "/v1/chat/completions/derender", + json={ + "model": MODEL_NAME, + "generate_response": _make_generate_response(synthetic_ids), + }, + ) + assert response.status_code == 200 + data = response.json() + assert data["object"] == "chat.completion" + assert len(data["choices"]) == 1 + assert data["choices"][0]["message"]["content"] + assert data["choices"][0]["message"]["role"] == "assistant" + + +@pytest.mark.asyncio +async def test_derender_chat_usage(client): + """Supplied prompt_tokens flows through into usage correctly.""" + gen_req = await _render_chat(client) + synthetic_ids = gen_req["token_ids"][:3] + + response = await client.post( + "/v1/chat/completions/derender", + json={ + "model": MODEL_NAME, + "generate_response": _make_generate_response(synthetic_ids), + "prompt_tokens": 10, + }, + ) + assert response.status_code == 200 + usage = response.json()["usage"] + assert usage["prompt_tokens"] == 10 + assert usage["completion_tokens"] == len(synthetic_ids) + assert usage["total_tokens"] == 10 + len(synthetic_ids) + + +@pytest.mark.asyncio +async def test_derender_chat_usage_default(client): + """Omitting prompt_tokens gives usage.prompt_tokens == 0.""" + gen_req = await _render_chat(client) + synthetic_ids = gen_req["token_ids"][:3] + + response = await client.post( + "/v1/chat/completions/derender", + json={ + "model": MODEL_NAME, + "generate_response": _make_generate_response(synthetic_ids), + }, + ) + assert response.status_code == 200 + usage = response.json()["usage"] + assert usage["prompt_tokens"] == 0 + + +@pytest.mark.asyncio +async def test_derender_chat_logprobs(client): + """token_id:N placeholders in content.token are resolved to real strings.""" + gen_req = await _render_chat(client) + synthetic_ids = gen_req["token_ids"][:3] + token_id = synthetic_ids[0] + + response = await client.post( + "/v1/chat/completions/derender", + json={ + "model": MODEL_NAME, + "generate_response": _make_generate_response( + synthetic_ids, + logprobs=_make_logprobs_with_placeholders(token_id), + ), + }, + ) + assert response.status_code == 200 + data = response.json() + logprobs = data["choices"][0]["logprobs"] + assert logprobs is not None + content = logprobs["content"] + assert content is not None and len(content) == 1 + token_str = content[0]["token"] + assert not token_str.startswith("token_id:"), ( + f"Placeholder was not resolved: {token_str!r}" + ) + + +@pytest.mark.asyncio +async def test_derender_chat_logprobs_bytes(client): + """Resolved logprob entries have bytes populated as list[int].""" + gen_req = await _render_chat(client) + synthetic_ids = gen_req["token_ids"][:3] + token_id = synthetic_ids[0] + + response = await client.post( + "/v1/chat/completions/derender", + json={ + "model": MODEL_NAME, + "generate_response": _make_generate_response( + synthetic_ids, + logprobs=_make_logprobs_with_placeholders(token_id), + ), + }, + ) + assert response.status_code == 200 + content = response.json()["choices"][0]["logprobs"]["content"] + bytes_field = content[0]["bytes"] + assert isinstance(bytes_field, list) + assert len(bytes_field) > 0 + assert all(isinstance(b, int) for b in bytes_field) + + +@pytest.mark.asyncio +async def test_derender_chat_top_logprobs(client): + """top_logprobs entries also have their placeholders resolved.""" + gen_req = await _render_chat(client) + synthetic_ids = gen_req["token_ids"][:3] + token_id = synthetic_ids[0] + + response = await client.post( + "/v1/chat/completions/derender", + json={ + "model": MODEL_NAME, + "generate_response": _make_generate_response( + synthetic_ids, + logprobs=_make_logprobs_with_placeholders(token_id), + ), + }, + ) + assert response.status_code == 200 + content = response.json()["choices"][0]["logprobs"]["content"] + top = content[0]["top_logprobs"] + assert len(top) == 1 + assert not top[0]["token"].startswith("token_id:"), ( + f"top_logprobs placeholder not resolved: {top[0]['token']!r}" + ) + + +@pytest.mark.asyncio +async def test_derender_chat_prompt_logprobs_passthrough(client): + """prompt_logprobs on GenerateResponse passes through unchanged.""" + gen_req = await _render_chat(client) + synthetic_ids = gen_req["token_ids"][:3] + # prompt_logprobs is a list[dict[int, Logprob] | None]; use None entries. + prompt_logprobs = [None, None] + + response = await client.post( + "/v1/chat/completions/derender", + json={ + "model": MODEL_NAME, + "generate_response": _make_generate_response( + synthetic_ids, prompt_logprobs=prompt_logprobs + ), + }, + ) + assert response.status_code == 200 + assert response.json()["prompt_logprobs"] == prompt_logprobs + + +@pytest.mark.asyncio +async def test_derender_chat_kv_transfer_params_passthrough(client): + """kv_transfer_params passes through to the ChatCompletionResponse.""" + gen_req = await _render_chat(client) + synthetic_ids = gen_req["token_ids"][:3] + kv = {"key": "value"} + + response = await client.post( + "/v1/chat/completions/derender", + json={ + "model": MODEL_NAME, + "generate_response": _make_generate_response( + synthetic_ids, kv_transfer_params=kv + ), + }, + ) + assert response.status_code == 200 + assert response.json()["kv_transfer_params"] == kv + + +@pytest.mark.asyncio +async def test_derender_chat_empty_token_ids(client): + """Empty token_ids list returns 400.""" + response = await client.post( + "/v1/chat/completions/derender", + json={ + "model": MODEL_NAME, + "generate_response": _make_generate_response([]), + }, + ) + assert response.status_code == 400 + + +@pytest.mark.asyncio +async def test_derender_chat_null_token_ids(client): + """Null token_ids returns 400.""" + response = await client.post( + "/v1/chat/completions/derender", + json={ + "model": MODEL_NAME, + "generate_response": _make_generate_response(None), + }, + ) + assert response.status_code == 400 + + +@pytest.mark.asyncio +async def test_derender_chat_unknown_model(client): + """Unknown model returns 404.""" + gen_req = await _render_chat(client) + synthetic_ids = gen_req["token_ids"][:3] + + response = await client.post( + "/v1/chat/completions/derender", + json={ + "model": "does-not-exist", + "generate_response": _make_generate_response(synthetic_ids), + }, + ) + assert response.status_code == 404 + + +# --------------------------------------------------------------------------- +# Completion derender tests +# --------------------------------------------------------------------------- + + +async def _render_completion(client: httpx.AsyncClient, prompt: str) -> dict: + """Render a completion prompt and return the first GenerateRequest dict.""" + resp = await client.post( + "/v1/completions/render", + json={"model": MODEL_NAME, "prompt": prompt}, + ) + assert resp.status_code == 200 + data = resp.json() + assert isinstance(data, list) and len(data) >= 1 + return data[0] + + +def _make_completion_generate_response( + token_ids: list[int], + request_id: str, + kv_transfer_params: dict | None = None, + logprobs: dict | None = None, +) -> dict: + return { + "request_id": request_id, + "choices": [ + { + "index": 0, + "token_ids": token_ids, + "finish_reason": "stop", + "logprobs": logprobs, + } + ], + "prompt_logprobs": None, + "kv_transfer_params": kv_transfer_params, + } + + +@pytest.mark.asyncio +async def test_derender_completion_roundtrip(client): + """Two prompts rendered, two GenerateResponses → two choices with indices 0, 1.""" + gr1 = await _render_completion(client, "Hello world") + gr2 = await _render_completion(client, "Goodbye world") + + ids1 = gr1["token_ids"][:4] + ids2 = gr2["token_ids"][:4] + + response = await client.post( + "/v1/completions/derender", + json={ + "model": MODEL_NAME, + "generate_responses": [ + _make_completion_generate_response(ids1, gr1["request_id"]), + _make_completion_generate_response(ids2, gr2["request_id"]), + ], + }, + ) + assert response.status_code == 200 + data = response.json() + assert data["object"] == "text_completion" + choices = data["choices"] + assert len(choices) == 2 + assert choices[0]["index"] == 0 + assert choices[1]["index"] == 1 + assert choices[0]["text"] + assert choices[1]["text"] + + +@pytest.mark.asyncio +async def test_derender_completion_usage_aggregation(client): + """prompt_tokens=[5, 10] is aggregated correctly into usage.""" + gr1 = await _render_completion(client, "Hello") + gr2 = await _render_completion(client, "World") + + ids1 = gr1["token_ids"][:3] + ids2 = gr2["token_ids"][:4] + + response = await client.post( + "/v1/completions/derender", + json={ + "model": MODEL_NAME, + "generate_responses": [ + _make_completion_generate_response(ids1, gr1["request_id"]), + _make_completion_generate_response(ids2, gr2["request_id"]), + ], + "prompt_tokens": [5, 10], + }, + ) + assert response.status_code == 200 + usage = response.json()["usage"] + assert usage["prompt_tokens"] == 15 + assert usage["completion_tokens"] == len(ids1) + len(ids2) + assert usage["total_tokens"] == 15 + len(ids1) + len(ids2) + + +@pytest.mark.asyncio +async def test_derender_completion_prompt_tokens_length_mismatch(client): + """len(prompt_tokens) != len(generate_responses) returns 400.""" + gr1 = await _render_completion(client, "Hello") + ids1 = gr1["token_ids"][:3] + + response = await client.post( + "/v1/completions/derender", + json={ + "model": MODEL_NAME, + "generate_responses": [ + _make_completion_generate_response(ids1, gr1["request_id"]), + ], + "prompt_tokens": [5, 10], + }, + ) + assert response.status_code == 400 + + +@pytest.mark.asyncio +async def test_derender_completion_empty_generate_responses(client): + """Empty generate_responses list returns 400.""" + response = await client.post( + "/v1/completions/derender", + json={"model": MODEL_NAME, "generate_responses": []}, + ) + assert response.status_code == 400 + + +@pytest.mark.asyncio +async def test_derender_completion_logprobs(client): + """token_id:N placeholders in logprobs are resolved; CompletionLogProbs + flat-list structure is returned with non-empty tokens and text_offsets.""" + gr1 = await _render_completion(client, "Hello world") + ids1 = gr1["token_ids"][:3] + token_id = ids1[0] + + response = await client.post( + "/v1/completions/derender", + json={ + "model": MODEL_NAME, + "generate_responses": [ + _make_completion_generate_response( + ids1, + gr1["request_id"], + logprobs=_make_logprobs_with_placeholders(token_id), + ), + ], + }, + ) + assert response.status_code == 200 + logprobs = response.json()["choices"][0]["logprobs"] + assert logprobs is not None + tokens = logprobs["tokens"] + assert len(tokens) == 1 + assert not tokens[0].startswith("token_id:"), ( + f"Placeholder was not resolved: {tokens[0]!r}" + ) + assert len(logprobs["token_logprobs"]) == 1 + assert isinstance(logprobs["token_logprobs"][0], float) + assert len(logprobs["text_offset"]) == 1 + assert logprobs["text_offset"][0] == 0 + + +@pytest.mark.asyncio +async def test_derender_completion_kv_transfer_params_passthrough(client): + """kv_transfer_params passes through to CompletionResponse.""" + gr1 = await _render_completion(client, "Hello") + ids1 = gr1["token_ids"][:3] + kv = {"node": "abc"} + + response = await client.post( + "/v1/completions/derender", + json={ + "model": MODEL_NAME, + "generate_responses": [ + _make_completion_generate_response( + ids1, gr1["request_id"], kv_transfer_params=kv + ), + ], + }, + ) + assert response.status_code == 200 + assert response.json()["kv_transfer_params"] == kv + + +# --------------------------------------------------------------------------- +# E2E: render -> derender roundtrip with parser (reasoning + tool calls) +# --------------------------------------------------------------------------- + +PARSER_MODEL = "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B" + +_E2E_TOOLS = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get weather for a city", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + }, + }, + } +] + + +@pytest.fixture(scope="module") +def parser_server(): + args = [ + "--enable-auto-tool-choice", + "--tool-call-parser", + "hermes", + "--reasoning-parser", + "deepseek_r1", + ] + with RemoteLaunchRenderServer(PARSER_MODEL, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def parser_client(parser_server): + async with httpx.AsyncClient( + base_url=parser_server.url_for(""), timeout=60.0 + ) as http_client: + yield http_client + + +@pytest.fixture(scope="module") +def parser_tokenizer(): + return get_tokenizer(PARSER_MODEL) + + +def _encode(tokenizer, text: str) -> list[int]: + return tokenizer.encode(text, add_special_tokens=False) + + +def _decoded(tokenizer, token_ids: list[int]) -> str: + return tokenizer.decode(token_ids, skip_special_tokens=True) + + +def _require_markers_survive(tokenizer, text: str, *markers: str) -> list[int]: + """Encode text and skip the test if any marker is lost in roundtrip.""" + ids = _encode(tokenizer, text) + decoded = tokenizer.decode(ids, skip_special_tokens=False) + for m in markers: + if m not in decoded: + pytest.skip(f"Marker {m!r} lost in encode->decode roundtrip") + return ids + + +async def _e2e_render_chat( + client: httpx.AsyncClient, + model: str, + messages: list[dict], +) -> dict: + resp = await client.post( + "/v1/chat/completions/render", + json={"model": model, "messages": messages}, + ) + assert resp.status_code == 200, resp.text + return resp.json() + + +def _e2e_generate_response( + token_ids: list[int], + request_id: str = "chatcmpl-e2e-test", +) -> dict: + return { + "request_id": request_id, + "choices": [ + { + "index": 0, + "token_ids": token_ids, + "finish_reason": "stop", + } + ], + } + + +@pytest.mark.asyncio +async def test_e2e_plain_roundtrip(parser_client, parser_tokenizer): + """Plain text without reasoning markers roundtrips correctly.""" + messages = [{"role": "user", "content": "What is 2+2?"}] + gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages) + + answer = "The answer is four." + output_ids = _encode(parser_tokenizer, answer) + expected = _decoded(parser_tokenizer, output_ids) + + resp = await parser_client.post( + "/v1/chat/completions/derender", + json={ + "model": PARSER_MODEL, + "generate_response": _e2e_generate_response(output_ids), + "prompt_tokens": len(gen_req["token_ids"]), + }, + ) + assert resp.status_code == 200, resp.text + content = resp.json()["choices"][0]["message"]["content"] + assert content == expected + + +@pytest.mark.asyncio +async def test_e2e_token_identity(parser_client, parser_tokenizer): + """encode(derender(token_ids)) == token_ids (RL invariant).""" + messages = [{"role": "user", "content": "Hi"}] + gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages) + + answer = "Hello! How can I help?" + output_ids = _encode(parser_tokenizer, answer) + + resp = await parser_client.post( + "/v1/chat/completions/derender", + json={ + "model": PARSER_MODEL, + "generate_response": _e2e_generate_response(output_ids), + "prompt_tokens": len(gen_req["token_ids"]), + }, + ) + assert resp.status_code == 200 + content = resp.json()["choices"][0]["message"]["content"] + re_encoded = _encode(parser_tokenizer, content) + assert output_ids == re_encoded + + +@pytest.mark.asyncio +async def test_e2e_non_ascii_roundtrip(parser_client, parser_tokenizer): + """CJK + emoji roundtrip without U+FFFD.""" + messages = [{"role": "user", "content": "Reply in Chinese"}] + gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages) + + answer = "你好世界 😀" + output_ids = _encode(parser_tokenizer, answer) + + resp = await parser_client.post( + "/v1/chat/completions/derender", + json={ + "model": PARSER_MODEL, + "generate_response": _e2e_generate_response(output_ids), + "prompt_tokens": len(gen_req["token_ids"]), + }, + ) + assert resp.status_code == 200 + content = resp.json()["choices"][0]["message"]["content"] + assert "�" not in content + + +@pytest.mark.asyncio +async def test_e2e_parsed_reasoning(parser_client, parser_tokenizer): + """... splits into reasoning + content.""" + messages = [{"role": "user", "content": "What is 2+3?"}] + gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages) + + reasoning_text = "The user wants 2 plus 3. That is 5." + answer_text = "The answer is 5." + output_text = f"{reasoning_text}{answer_text}" + output_ids = _require_markers_survive(parser_tokenizer, output_text, "") + + resp = await parser_client.post( + "/v1/chat/completions/derender", + json={ + "model": PARSER_MODEL, + "generate_response": _e2e_generate_response(output_ids), + "prompt_tokens": len(gen_req["token_ids"]), + "chat_request": { + "model": PARSER_MODEL, + "messages": messages, + "include_reasoning": True, + }, + }, + ) + assert resp.status_code == 200, resp.text + msg = resp.json()["choices"][0]["message"] + assert msg["reasoning"] is not None + assert reasoning_text in msg["reasoning"] + assert answer_text in msg["content"] + assert "" not in msg["content"] + + +@pytest.mark.asyncio +async def test_e2e_parsed_tool_call(parser_client, parser_tokenizer): + """ extracted into tool_calls field.""" + messages = [{"role": "user", "content": "Weather in Paris?"}] + gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages) + + output_text = ( + "Let me check the weather." + '\n{"name": "get_weather", ' + '"arguments": {"city": "Paris"}}\n' + ) + output_ids = _require_markers_survive( + parser_tokenizer, + output_text, + "", + "", + "", + ) + + resp = await parser_client.post( + "/v1/chat/completions/derender", + json={ + "model": PARSER_MODEL, + "generate_response": _e2e_generate_response(output_ids), + "prompt_tokens": len(gen_req["token_ids"]), + "chat_request": { + "model": PARSER_MODEL, + "messages": messages, + "tools": _E2E_TOOLS, + "tool_choice": "auto", + }, + }, + ) + assert resp.status_code == 200, resp.text + choice = resp.json()["choices"][0] + assert choice["message"]["tool_calls"] + assert choice["message"]["tool_calls"][0]["function"]["name"] == "get_weather" + + +@pytest.mark.asyncio +async def test_e2e_parsed_reasoning_and_tool_call(parser_client, parser_tokenizer): + """Reasoning + tool call in the same output.""" + messages = [{"role": "user", "content": "Weather in Paris?"}] + gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages) + + reasoning_text = "I should look up the weather." + tool_text = ( + '\n{"name": "get_weather", ' + '"arguments": {"city": "Paris"}}\n' + ) + output_text = f"{reasoning_text}{tool_text}" + output_ids = _require_markers_survive( + parser_tokenizer, output_text, "", "" + ) + + resp = await parser_client.post( + "/v1/chat/completions/derender", + json={ + "model": PARSER_MODEL, + "generate_response": _e2e_generate_response(output_ids), + "prompt_tokens": len(gen_req["token_ids"]), + "chat_request": { + "model": PARSER_MODEL, + "messages": messages, + "tools": _E2E_TOOLS, + "tool_choice": "auto", + "include_reasoning": True, + }, + }, + ) + assert resp.status_code == 200, resp.text + choice = resp.json()["choices"][0] + assert choice["message"]["reasoning"] is not None + assert reasoning_text in choice["message"]["reasoning"] + assert choice["message"]["tool_calls"] + + +@pytest.mark.asyncio +async def test_e2e_no_chat_request_fallback(parser_client, parser_tokenizer): + """Without chat_request, derender falls back to plain detokenization.""" + messages = [{"role": "user", "content": "Hello"}] + gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages) + + answer = "Hi there!" + output_ids = _encode(parser_tokenizer, answer) + + resp = await parser_client.post( + "/v1/chat/completions/derender", + json={ + "model": PARSER_MODEL, + "generate_response": _e2e_generate_response(output_ids), + "prompt_tokens": len(gen_req["token_ids"]), + }, + ) + assert resp.status_code == 200 + content = resp.json()["choices"][0]["message"]["content"] + assert "Hi" in content + + +# --------------------------------------------------------------------------- +# E2E: HarmonyParser + GPT-OSS +# --------------------------------------------------------------------------- + +HARMONY_MODEL = "openai/gpt-oss-20b" + + +def _ensure_harmony_vocab(): + """Pre-cache the o200k_base BPE file needed by openai-harmony. + + The Rust tiktoken-rs backend downloads from Azure Blob Storage, which + may be unreachable in some environments. When the cache is cold we + fetch the file ourselves and place it in ``/tmp/tiktoken-rs-cache/`` + using the SHA-1(URL) filename that tiktoken-rs expects. + """ + import hashlib + import urllib.request + from pathlib import Path + + url = "https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken" + cache_dir = Path("/tmp/tiktoken-rs-cache") + cache_key = hashlib.sha1(url.encode()).hexdigest() + cache_file = cache_dir / cache_key + if not cache_file.exists(): + cache_dir.mkdir(parents=True, exist_ok=True) + urllib.request.urlretrieve(url, cache_file) + + +@pytest.fixture(scope="module") +def harmony_server(): + _ensure_harmony_vocab() + args = [ + "--trust-remote-code", + "--enable-auto-tool-choice", + "--tool-call-parser", + "openai", + "--reasoning-parser", + "openai_gptoss", + ] + with RemoteLaunchRenderServer(HARMONY_MODEL, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def harmony_client(harmony_server): + async with httpx.AsyncClient( + base_url=harmony_server.url_for(""), timeout=60.0 + ) as http_client: + yield http_client + + +@pytest.fixture(scope="module") +def harmony_tokenizer(): + return get_tokenizer(HARMONY_MODEL, trust_remote_code=True) + + +def _harmony_extract_assistant_ids( + tokenizer, assistant_msg: dict, user_content: str = "test" +) -> list[int]: + """Extract assistant token IDs via apply_chat_template diff.""" + prompt = [{"role": "user", "content": user_content}] + full = prompt + [assistant_msg] + text_prompt = tokenizer.apply_chat_template( + prompt, add_generation_prompt=True, tokenize=False + ) + text_full = tokenizer.apply_chat_template( + full, add_generation_prompt=False, tokenize=False + ) + prompt_ids = tokenizer.encode(text_prompt) + full_ids = tokenizer.encode(text_full) + assistant_ids = list(full_ids[len(prompt_ids) :]) + if not assistant_ids: + pytest.skip("Could not extract assistant tokens for Harmony") + return assistant_ids + + +@pytest.mark.asyncio +async def test_e2e_harmony_plain_roundtrip(harmony_client, harmony_tokenizer): + """GPT-OSS content-only roundtrip.""" + messages = [{"role": "user", "content": "What is 2+2?"}] + gen_req = await _e2e_render_chat(harmony_client, HARMONY_MODEL, messages) + + assistant_msg = {"role": "assistant", "content": "Four."} + output_ids = _harmony_extract_assistant_ids(harmony_tokenizer, assistant_msg) + + resp = await harmony_client.post( + "/v1/chat/completions/derender", + json={ + "model": HARMONY_MODEL, + "generate_response": _e2e_generate_response(output_ids), + "prompt_tokens": len(gen_req["token_ids"]), + "chat_request": { + "model": HARMONY_MODEL, + "messages": messages, + }, + }, + ) + assert resp.status_code == 200, resp.text + content = resp.json()["choices"][0]["message"]["content"] + assert content is not None and len(content) > 0 + assert "Four" in content + + +@pytest.mark.asyncio +async def test_e2e_harmony_reasoning(harmony_client, harmony_tokenizer): + """GPT-OSS reasoning: analysis channel extracted.""" + messages = [{"role": "user", "content": "Add 2 and 3."}] + gen_req = await _e2e_render_chat(harmony_client, HARMONY_MODEL, messages) + + reasoning_text = "The user wants 2 plus 3." + answer_text = "The answer is 5." + assistant_msg = { + "role": "assistant", + "thinking": reasoning_text, + "content": answer_text, + } + output_ids = _harmony_extract_assistant_ids(harmony_tokenizer, assistant_msg) + + decoded = harmony_tokenizer.decode(output_ids) + if reasoning_text not in decoded: + pytest.skip("Harmony template did not render thinking") + + resp = await harmony_client.post( + "/v1/chat/completions/derender", + json={ + "model": HARMONY_MODEL, + "generate_response": _e2e_generate_response(output_ids), + "prompt_tokens": len(gen_req["token_ids"]), + "chat_request": { + "model": HARMONY_MODEL, + "messages": messages, + "include_reasoning": True, + }, + }, + ) + assert resp.status_code == 200, resp.text + msg = resp.json()["choices"][0]["message"] + assert msg["reasoning"] is not None + assert reasoning_text in msg["reasoning"] + assert answer_text in (msg["content"] or "") diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/render/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/render/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/render/test_launch_render.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/render/test_launch_render.py new file mode 100644 index 0000000000000000000000000000000000000000..9b807badddd5c2ba7c0a0f537cc323bbbecc5b8f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/render/test_launch_render.py @@ -0,0 +1,179 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""E2E tests for render endpoints via `vllm launch` (GPU-less serving).""" + +import httpx +import pytest +import pytest_asyncio + +from tests.utils import RemoteLaunchRenderServer + +MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM" + + +@pytest.fixture(scope="module") +def server(): + args: list[str] = [] + with RemoteLaunchRenderServer(MODEL_NAME, args, max_wait_seconds=120) as srv: + yield srv + + +@pytest_asyncio.fixture +async def client(server): + async with httpx.AsyncClient( + base_url=server.url_for(""), timeout=30.0 + ) as http_client: + yield http_client + + +# -- Chat Completion Render -- + + +@pytest.mark.asyncio +async def test_chat_render_basic(client): + response = await client.post( + "/v1/chat/completions/render", + json={ + "model": MODEL_NAME, + "messages": [{"role": "user", "content": "Hello, how are you?"}], + }, + ) + + assert response.status_code == 200 + data = response.json() + + # Response should be a GenerateRequest dict + assert isinstance(data, dict) + assert "token_ids" in data + assert isinstance(data["token_ids"], list) + assert len(data["token_ids"]) > 0 + assert all(isinstance(t, int) for t in data["token_ids"]) + + +@pytest.mark.asyncio +async def test_chat_render_multi_turn(client): + response = await client.post( + "/v1/chat/completions/render", + json={ + "model": MODEL_NAME, + "messages": [ + {"role": "user", "content": "Hello"}, + {"role": "assistant", "content": "Hi there!"}, + {"role": "user", "content": "How are you?"}, + ], + }, + ) + + assert response.status_code == 200 + data = response.json() + + assert isinstance(data, dict) + assert "token_ids" in data + assert isinstance(data["token_ids"], list) + assert len(data["token_ids"]) > 0 + + +# -- Completion Render -- + + +@pytest.mark.asyncio +async def test_completion_render_basic(client): + response = await client.post( + "/v1/completions/render", + json={ + "model": MODEL_NAME, + "prompt": "Once upon a time", + }, + ) + + assert response.status_code == 200 + data = response.json() + + assert isinstance(data, list) + assert len(data) > 0 + + first_prompt = data[0] + assert "token_ids" in first_prompt + assert "sampling_params" in first_prompt + assert "model" in first_prompt + assert "request_id" in first_prompt + assert isinstance(first_prompt["token_ids"], list) + assert len(first_prompt["token_ids"]) > 0 + assert first_prompt["request_id"].startswith("cmpl-") + + +@pytest.mark.asyncio +async def test_completion_render_multiple_prompts(client): + response = await client.post( + "/v1/completions/render", + json={ + "model": MODEL_NAME, + "prompt": ["Hello world", "Goodbye world"], + }, + ) + + assert response.status_code == 200 + data = response.json() + + assert isinstance(data, list) + assert len(data) == 2 + + for prompt in data: + assert "token_ids" in prompt + assert "sampling_params" in prompt + assert "model" in prompt + assert "request_id" in prompt + assert len(prompt["token_ids"]) > 0 + assert prompt["request_id"].startswith("cmpl-") + + +@pytest.mark.asyncio +async def test_completion_render_invalid_model(client): + response = await client.post( + "/v1/completions/render", + json={ + "model": "nonexistent-model", + "prompt": "Hello", + }, + ) + + assert response.status_code == 404 + assert "error" in response.json() + + +@pytest.mark.asyncio +async def test_render_is_fast(client): + """Render should complete quickly since there is no inference.""" + import time + + start = time.perf_counter() + response = await client.post( + "/v1/completions/render", + json={ + "model": MODEL_NAME, + "prompt": "Tell me a very long story about " * 10, + }, + ) + elapsed = time.perf_counter() - start + + assert response.status_code == 200 + assert elapsed < 2.0 + + +# -- Health & Models -- + + +@pytest.mark.asyncio +async def test_health_endpoint(client): + response = await client.get("/health") + assert response.status_code == 200 + + +@pytest.mark.asyncio +async def test_models_endpoint(client): + response = await client.get("/v1/models") + assert response.status_code == 200 + data = response.json() + assert "data" in data + model_ids = [m["id"] for m in data["data"]] + assert MODEL_NAME in model_ids diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/render/test_render.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/render/test_render.py new file mode 100644 index 0000000000000000000000000000000000000000..695ddcdafee89ac9080e5b1b133c73f8cfc774f8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/render/test_render.py @@ -0,0 +1,495 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Tests for the /render endpoints that expose prompt preprocessing.""" + +import httpx +import pytest +import pytest_asyncio + +from tests.utils import RemoteLaunchRenderServer + +MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM" + + +@pytest.fixture(scope="module") +def server(): + args: list[str] = ["--trust-request-chat-template"] + + with RemoteLaunchRenderServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with httpx.AsyncClient( + base_url=server.url_for(""), timeout=30.0 + ) as http_client: + yield http_client + + +@pytest.mark.asyncio +async def test_completion_render_basic(client): + """Test basic completion render endpoint.""" + # Make request to render endpoint + response = await client.post( + "/v1/completions/render", + json={ + "model": MODEL_NAME, + "prompt": "When should a chat-completions handler return an empty string?", + }, + ) + + assert response.status_code == 200 + data = response.json() + + # Verify response structure - list of GenerateRequest + assert isinstance(data, list) + assert len(data) > 0 + + # Verify first prompt is a GenerateRequest + first_prompt = data[0] + assert "token_ids" in first_prompt + assert "sampling_params" in first_prompt + assert "model" in first_prompt + assert "request_id" in first_prompt + assert isinstance(first_prompt["token_ids"], list) + assert len(first_prompt["token_ids"]) > 0 + assert first_prompt["model"] == MODEL_NAME + assert first_prompt["request_id"].startswith("cmpl-") + + +@pytest.mark.asyncio +async def test_chat_completion_render_basic(client): + """Test basic chat completion render endpoint.""" + # Make request to render endpoint + response = await client.post( + "/v1/chat/completions/render", + json={ + "model": MODEL_NAME, + "messages": [ + { + "role": "user", + "content": ( + "Returning an empty string for the prompt may be confusing." + ), + } + ], + }, + ) + + assert response.status_code == 200 + data = response.json() + + # Verify response structure - should be a GenerateRequest + assert isinstance(data, dict) + assert "token_ids" in data + assert isinstance(data["token_ids"], list) + assert len(data["token_ids"]) > 0 + + # Verify token IDs are integers and BOS token is present + token_ids = data["token_ids"] + assert all(isinstance(tid, int) for tid in token_ids) + assert token_ids[0] == 1 + + +@pytest.mark.asyncio +async def test_completion_render_multiple_prompts(client): + """Test completion render with multiple prompts.""" + response = await client.post( + "/v1/completions/render", + json={ + "model": MODEL_NAME, + "prompt": ["Hello world", "Goodbye world"], + }, + ) + + assert response.status_code == 200 + data = response.json() + + # Should return two GenerateRequest items + assert isinstance(data, list) + assert len(data) == 2 + + # Verify both prompts have GenerateRequest fields + for prompt in data: + assert "token_ids" in prompt + assert "sampling_params" in prompt + assert "model" in prompt + assert "request_id" in prompt + assert len(prompt["token_ids"]) > 0 + assert prompt["request_id"].startswith("cmpl-") + + +@pytest.mark.asyncio +async def test_chat_completion_render_multi_turn(client): + """Test chat completion render with multi-turn conversation.""" + response = await client.post( + "/v1/chat/completions/render", + json={ + "model": MODEL_NAME, + "messages": [ + {"role": "user", "content": "Hello"}, + {"role": "assistant", "content": "Hi there!"}, + {"role": "user", "content": "How are you?"}, + ], + }, + ) + + assert response.status_code == 200 + data = response.json() + + # Verify tokenization occurred + assert isinstance(data, dict) + assert "token_ids" in data + assert isinstance(data["token_ids"], list) + assert len(data["token_ids"]) > 0 + + +@pytest.mark.asyncio +async def test_chat_completion_render_with_stream_true(client): + """Render accepts stream params but still returns JSON (non-streamed).""" + + response = await client.post( + "/v1/chat/completions/render", + json={ + "model": MODEL_NAME, + "stream": True, + "stream_options": { + "include_usage": True, + "continuous_usage_stats": True, + }, + "messages": [ + { + "role": "user", + "content": "Stream options should be accepted by /render.", + } + ], + }, + ) + + assert response.status_code == 200 + assert response.headers.get("content-type", "").startswith("application/json") + + data = response.json() + assert isinstance(data, dict) + assert "token_ids" in data + assert isinstance(data["token_ids"], list) + assert len(data["token_ids"]) > 0 + + # /render should preserve stream fields on the returned token-in request. + assert data.get("stream") is True + assert isinstance(data.get("stream_options"), dict) + assert data["stream_options"].get("include_usage") is True + assert data["stream_options"].get("continuous_usage_stats") is True + + +@pytest.mark.asyncio +async def test_completion_render_error_invalid_model(client): + """Test completion render with invalid model returns error.""" + response = await client.post( + "/v1/completions/render", + json={ + "model": "invalid-model-name", + "prompt": "Hello", + }, + ) + + assert response.status_code == 404 + data = response.json() + assert "error" in data + + +@pytest.mark.asyncio +async def test_chat_completion_render_error_invalid_model(client): + """Test chat completion render with invalid model returns error.""" + response = await client.post( + "/v1/chat/completions/render", + json={ + "model": "invalid-model-name", + "messages": [{"role": "user", "content": "Hello"}], + }, + ) + + assert response.status_code == 404 + data = response.json() + assert "error" in data + + +@pytest.mark.asyncio +async def test_completion_render_no_generation(client): + """Verify render endpoint does not generate text.""" + # This test verifies that calling render is fast (no generation) + import time + + start = time.perf_counter() + response = await client.post( + "/v1/completions/render", + json={ + "model": MODEL_NAME, + "prompt": "Tell me a very long story about " * 10, + }, + ) + elapsed = time.perf_counter() - start + + assert response.status_code == 200 + # Render should be fast (< 1 second) since no generation + assert elapsed < 1.0 + + +@pytest.mark.asyncio +async def test_chat_completion_render_with_sampling_params(client): + """Verify sampling params are correctly returned by /render.""" + response = await client.post( + "/v1/chat/completions/render", + json={ + "model": MODEL_NAME, + "messages": [{"role": "user", "content": "Test sampling params"}], + "temperature": 0.123, + "top_p": 0.456, + "frequency_penalty": 1.1, + }, + ) + + assert response.status_code == 200 + data = response.json() + + assert "sampling_params" in data + sampling_params = data["sampling_params"] + + assert sampling_params.get("temperature") == 0.123 + assert sampling_params.get("top_p") == 0.456 + assert sampling_params.get("frequency_penalty") == 1.1 + + # Check that internal fields are not present + assert "_all_stop_token_ids" not in sampling_params + + +@pytest.mark.asyncio +async def test_completion_render_emits_token_offsets(client): + """With return_token_offsets, /v1/completions/render returns per-token + (start, end) char offsets aligned with token_ids.""" + prompt = "Hello, world." + response = await client.post( + "/v1/completions/render", + json={ + "model": MODEL_NAME, + "prompt": prompt, + "return_token_offsets": True, + }, + ) + + assert response.status_code == 200 + data = response.json() + assert isinstance(data, list) + offsets = data[0]["token_offsets"] + assert offsets is not None + assert len(offsets) == len(data[0]["token_ids"]) + for start, end in offsets: + assert isinstance(start, int) and isinstance(end, int) + assert 0 <= start <= end <= len(prompt) + + +@pytest.mark.asyncio +async def test_completion_render_default_no_token_offsets(client): + """Without the flag, token_offsets must be null (existing responses + unchanged).""" + response = await client.post( + "/v1/completions/render", + json={ + "model": MODEL_NAME, + "prompt": "Hello, world.", + }, + ) + + assert response.status_code == 200 + data = response.json() + assert data[0]["token_offsets"] is None + + +@pytest.mark.asyncio +async def test_chat_render_emits_token_offsets(client): + """With return_token_offsets, /v1/chat/completions/render returns + per-token offsets relative to the templated prompt string.""" + response = await client.post( + "/v1/chat/completions/render", + json={ + "model": MODEL_NAME, + "messages": [{"role": "user", "content": "Hello, world."}], + "return_token_offsets": True, + }, + ) + + assert response.status_code == 200 + data = response.json() + assert isinstance(data, dict) + offsets = data["token_offsets"] + assert offsets is not None + assert len(offsets) == len(data["token_ids"]) + for start, end in offsets: + assert isinstance(start, int) and isinstance(end, int) + assert 0 <= start <= end + + +@pytest.mark.asyncio +async def test_chat_render_default_no_token_offsets(client): + """Without the flag, chat render token_offsets must be null.""" + response = await client.post( + "/v1/chat/completions/render", + json={ + "model": MODEL_NAME, + "messages": [{"role": "user", "content": "Hello, world."}], + }, + ) + + assert response.status_code == 200 + data = response.json() + assert data["token_offsets"] is None + + +@pytest.mark.asyncio +async def test_completion_render_multiple_prompts_token_offsets(client): + """Each prompt in a batch gets its own offsets aligned with its tokens.""" + prompts = ["Hello, world.", "Goodbye, world."] + response = await client.post( + "/v1/completions/render", + json={ + "model": MODEL_NAME, + "prompt": prompts, + "return_token_offsets": True, + }, + ) + + assert response.status_code == 200 + data = response.json() + assert len(data) == len(prompts) + for item, prompt in zip(data, prompts): + offsets = item["token_offsets"] + assert offsets is not None + assert len(offsets) == len(item["token_ids"]) + for start, end in offsets: + assert 0 <= start <= end <= len(prompt) + + +@pytest.mark.asyncio +async def test_chat_completion_render_assistant_tokens_mask_default(client): + """Without return_assistant_tokens_mask, assistant_tokens_mask should be null.""" + response = await client.post( + "/v1/chat/completions/render", + json={ + "model": MODEL_NAME, + "messages": [ + {"role": "user", "content": "Hello"}, + {"role": "assistant", "content": "Hi!"}, + {"role": "user", "content": "How are you?"}, + ], + }, + ) + + assert response.status_code == 200 + data = response.json() + assert data.get("assistant_tokens_mask") is None + + +@pytest.mark.asyncio +async def test_chat_completion_render_assistant_tokens_mask_false(client): + """Explicitly setting return_assistant_tokens_mask=false gives null.""" + response = await client.post( + "/v1/chat/completions/render", + json={ + "model": MODEL_NAME, + "messages": [ + {"role": "user", "content": "Hello"}, + ], + "return_assistant_tokens_mask": False, + }, + ) + + assert response.status_code == 200 + data = response.json() + assert data.get("assistant_tokens_mask") is None + + +@pytest.mark.asyncio +async def test_chat_render_assistant_tokens_mask_null_without_gen_tags( + client, +): + """The tiny test model lacks ``{% generation %}`` tags, so the mask is null.""" + response = await client.post( + "/v1/chat/completions/render", + json={ + "model": MODEL_NAME, + "messages": [ + {"role": "user", "content": "Hello"}, + {"role": "assistant", "content": "Hi!"}, + ], + "return_assistant_tokens_mask": True, + }, + ) + + assert response.status_code == 200 + assert response.json().get("assistant_tokens_mask") is None + + +# A minimal chat template with {% generation %} tags so we can test that +# the mask correctly marks assistant tokens. +_TEMPLATE_WITH_GENERATION = ( + "{% for m in messages %}" + "{% if m['role'] == 'user' %}User: {{ m['content'] }}\n" + "{% elif m['role'] == 'assistant' %}" + "{% generation %}Assistant: {{ m['content'] }}\n{% endgeneration %}" + "{% endif %}" + "{% endfor %}" +) + + +@pytest.mark.asyncio +async def test_chat_completion_render_assistant_tokens_mask_with_generation_tags( + client, +): + """With a ``{% generation %}``-enabled template, the mask marks assistant + tokens and the masked tokens decode to the assistant content.""" + response = await client.post( + "/v1/chat/completions/render", + json={ + "model": MODEL_NAME, + "messages": [ + {"role": "user", "content": "Hello"}, + {"role": "assistant", "content": "Hi!"}, + {"role": "user", "content": "Bye"}, + ], + "chat_template": _TEMPLATE_WITH_GENERATION, + "return_assistant_tokens_mask": True, + }, + ) + + assert response.status_code == 200 + data = response.json() + + mask = data["assistant_tokens_mask"] + token_ids = data["token_ids"] + assert mask is not None + assert isinstance(mask, list) + assert len(mask) == len(token_ids) + assert all(v in (0, 1) for v in mask) + assert sum(mask) > 0, "mask should mark at least one assistant token" + + # Detokenize masked (assistant) and unmasked (non-assistant) tokens + # separately to verify the mask is correct, not just non-empty. + masked_ids = [t for t, m in zip(token_ids, mask, strict=True) if m] + unmasked_ids = [t for t, m in zip(token_ids, mask, strict=True) if not m] + + detok = await client.post( + "/detokenize", + json={"model": MODEL_NAME, "tokens": masked_ids}, + ) + assert detok.status_code == 200 + assert "Hi!" in detok.json()["prompt"] + + detok_rest = await client.post( + "/detokenize", + json={"model": MODEL_NAME, "tokens": unmasked_ids}, + ) + assert detok_rest.status_code == 200 + assert "Hi!" not in detok_rest.json()["prompt"] + assert "Bye" in detok_rest.json()["prompt"] diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/render/test_render_multimodal.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/render/test_render_multimodal.py new file mode 100644 index 0000000000000000000000000000000000000000..d6ae31e3975b4651c2ca928492ab7fab87c6586a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/render/test_render_multimodal.py @@ -0,0 +1,155 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Multimodal tests for the /render endpoints that expose prompt preprocessing.""" + +import httpx +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer +from vllm.multimodal.utils import encode_image_url + +VISION_MODEL_NAME = "Qwen/Qwen3-VL-2B-Instruct" + + +@pytest.fixture(scope="module") +def vision_server(): + """Vision-capable server used for multimodal /render tests.""" + + args = [ + "--enforce-eager", + "--max-model-len", + "100", + "--max-num-seqs", + "1", + "--limit-mm-per-prompt.image", + "1", + "--limit-mm-per-prompt.video", + "0", + ] + + env_overrides: dict[str, str] = {} + + with RemoteOpenAIServer( + VISION_MODEL_NAME, + args, + env_dict=env_overrides, + ) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def vision_client(vision_server): + async with httpx.AsyncClient( + base_url=vision_server.url_for(""), timeout=60.0 + ) as http_client: + yield http_client + + +@pytest.mark.asyncio +async def test_chat_completion_render_with_base64_image_url( + vision_client, + local_asset_server, +): + """Render a multimodal chat request and verify tokens are returned.""" + + image = local_asset_server.get_image_asset("RGBA_comp.png") + data_url = encode_image_url(image, format="PNG") + + assert data_url.startswith("data:image/") + assert ";base64," in data_url + + response = await vision_client.post( + "/v1/chat/completions/render", + json={ + "model": VISION_MODEL_NAME, + "messages": [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": data_url}}, + {"type": "text", "text": "What's in this image?"}, + ], + } + ], + }, + ) + + assert response.status_code == 200 + + data = response.json() + assert isinstance(data, dict) + assert "token_ids" in data + assert isinstance(data["token_ids"], list) + assert len(data["token_ids"]) > 0 + + # Verify multimodal features are populated + assert "features" in data + features = data["features"] + assert features is not None + + # mm_hashes: should have an "image" key with a list of hash strings + assert "mm_hashes" in features + assert "image" in features["mm_hashes"] + image_hashes = features["mm_hashes"]["image"] + assert isinstance(image_hashes, list) + assert len(image_hashes) > 0 + assert all(isinstance(h, str) for h in image_hashes) + + # mm_placeholders: should have an "image" key with offset/length dicts + assert "mm_placeholders" in features + assert "image" in features["mm_placeholders"] + image_placeholders = features["mm_placeholders"]["image"] + assert isinstance(image_placeholders, list) + assert len(image_placeholders) > 0 + for p in image_placeholders: + assert "offset" in p + assert "length" in p + assert isinstance(p["offset"], int) + assert isinstance(p["length"], int) + assert p["length"] > 0 + + +@pytest.mark.asyncio +async def test_tokenize_matches_render_for_multimodal_input( + vision_client, + local_asset_server, +): + """`/tokenize` should match `/v1/chat/completions/render` token output.""" + + image = local_asset_server.get_image_asset("RGBA_comp.png") + data_url = encode_image_url(image, format="PNG") + + messages = [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": data_url}}, + {"type": "text", "text": "What's in this image?"}, + ], + } + ] + + render_response = await vision_client.post( + "/v1/chat/completions/render", + json={ + "model": VISION_MODEL_NAME, + "messages": messages, + }, + ) + assert render_response.status_code == 200 + render_data = render_response.json() + + tokenize_response = await vision_client.post( + "/tokenize", + json={ + "model": VISION_MODEL_NAME, + "messages": messages, + }, + ) + assert tokenize_response.status_code == 200 + tokenize_data = tokenize_response.json() + + assert tokenize_data["tokens"] == render_data["token_ids"] + assert tokenize_data["count"] == len(render_data["token_ids"]) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_generate_stream.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_generate_stream.py new file mode 100644 index 0000000000000000000000000000000000000000..fca2ed4cc4678652a88337b1ce006cd4635bce94 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_generate_stream.py @@ -0,0 +1,554 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json +from dataclasses import dataclass, field +from typing import Any +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from vllm.config.multimodal import MultiModalConfig +from vllm.entrypoints.openai.engine.protocol import StreamOptions +from vllm.entrypoints.openai.models.protocol import BaseModelPath +from vllm.entrypoints.openai.models.serving import OpenAIServingModels +from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( + GenerateRequest, + GenerateResponse, +) +from vllm.entrypoints.scale_out.token_in_token_out.serving import ServingTokens +from vllm.logprobs import Logprob +from vllm.outputs import CompletionOutput, RequestOutput +from vllm.renderers import renderer_from_config +from vllm.renderers.online_renderer import OnlineRenderer +from vllm.sampling_params import SamplingParams +from vllm.v1.engine.async_llm import AsyncLLM + +MODEL_NAME = "openai-community/gpt2" +BASE_MODEL_PATHS = [ + BaseModelPath(name=MODEL_NAME, model_path=MODEL_NAME), +] + + +@dataclass +class MockHFConfig: + model_type: str = "any" + + +@dataclass +class MockModelConfig: + task = "generate" + runner_type = "generate" + model = MODEL_NAME + tokenizer = MODEL_NAME + trust_remote_code = False + tokenizer_mode = "auto" + max_model_len = 100 + tokenizer_revision = None + multimodal_config = MultiModalConfig() + hf_config = MockHFConfig() + hf_text_config = MockHFConfig() + logits_processors: list[str] | None = None + diff_sampling_param: dict | None = None + allowed_local_media_path: str = "" + allowed_media_domains: list[str] | None = None + encoder_config = None + generation_config: str = "auto" + media_io_kwargs: dict[str, dict[str, Any]] = field(default_factory=dict) + skip_tokenizer_init = False + is_encoder_decoder: bool = False + is_multimodal_model: bool = False + renderer_num_workers: int = 1 + + def get_diff_sampling_param(self): + return self.diff_sampling_param or {} + + +@dataclass +class MockParallelConfig: + _api_process_rank: int = 0 + + +@dataclass +class MockSchedulerConfig: + max_num_seqs: int = 128 + + +@dataclass +class MockVllmConfig: + model_config: MockModelConfig + parallel_config: MockParallelConfig + scheduler_config: MockSchedulerConfig = field(default_factory=MockSchedulerConfig) + + +def _build_renderer(model_config: MockModelConfig): + return renderer_from_config( + MockVllmConfig(model_config, parallel_config=MockParallelConfig()), + ) + + +def _build_serving_tokens(engine: AsyncLLM, **kwargs) -> ServingTokens: + models = OpenAIServingModels( + engine_client=engine, + base_model_paths=BASE_MODEL_PATHS, + ) + online_renderer = OnlineRenderer( + model_config=engine.model_config, + renderer=engine.renderer, + request_logger=None, + chat_template=None, + chat_template_content_format="auto", + ) + serving = ServingTokens( + engine, + models, + online_renderer=online_renderer, + request_logger=None, + **kwargs, + ) + + async def _fake_preprocess(*args, **kwargs): + return [{"prompt_token_ids": [1, 2, 3]}] + + serving.online_renderer.preprocess_completion = AsyncMock( + side_effect=_fake_preprocess + ) + return serving + + +def _make_request_output( + request_id: str, + token_ids: list[int], + finish_reason: str | None = None, + finished: bool = False, + prompt_token_ids: list[int] | None = None, + logprobs: list[dict[int, Any] | None] | None = None, + num_cached_tokens: int | None = None, + index: int = 0, +) -> RequestOutput: + return RequestOutput( + request_id=request_id, + prompt=None, + prompt_token_ids=prompt_token_ids or [1, 2, 3], + prompt_logprobs=None, + outputs=[ + CompletionOutput( + index=index, + text="", + token_ids=token_ids, + cumulative_logprob=None, + logprobs=logprobs, + finish_reason=finish_reason, + ) + ], + finished=finished, + metrics=None, + lora_request=None, + encoder_prompt=None, + encoder_prompt_token_ids=None, + num_cached_tokens=num_cached_tokens, + ) + + +def _mock_engine() -> MagicMock: + engine = MagicMock(spec=AsyncLLM) + engine.errored = False + engine.model_config = MockModelConfig() + engine.vllm_config = MockVllmConfig( + engine.model_config, parallel_config=MockParallelConfig() + ) + engine.input_processor = MagicMock() + engine.renderer = _build_renderer(engine.model_config) + return engine + + +def _parse_sse_chunks(chunks: list[str]) -> list[Any]: + """Parse SSE chunks into dicts (JSON) or raw strings ([DONE]).""" + parsed: list[Any] = [] + for chunk in chunks: + assert chunk.startswith("data: ") and chunk.endswith("\n\n") + payload = chunk[len("data: ") : -len("\n\n")] + if payload == "[DONE]": + parsed.append("[DONE]") + else: + parsed.append(json.loads(payload)) + return parsed + + +@pytest.mark.asyncio +async def test_serve_tokens_skips_mm_cache_for_remote_engine_execution(): + engine = _mock_engine() + + async def mock_generate(*args, **kwargs): + yield _make_request_output( + "req-1", token_ids=[10], finish_reason="stop", finished=True + ) + + engine.generate = MagicMock(side_effect=mock_generate) + serving = _build_serving_tokens(engine) + + request = GenerateRequest( + token_ids=[1, 2, 3], + sampling_params=SamplingParams(max_tokens=1), + model=MODEL_NAME, + stream=False, + ) + + response = await serving.serve_tokens(request) + + assert isinstance(response, GenerateResponse) + assert ( + serving.online_renderer.preprocess_completion.call_args.kwargs["skip_mm_cache"] + is True + ) + + +@pytest.mark.asyncio +async def test_stream_basic(): + """Streaming returns SSE chunks with correct token_ids and ends with [DONE].""" + engine = _mock_engine() + + async def mock_generate(*args, **kwargs): + yield _make_request_output("req-1", token_ids=[10]) + yield _make_request_output("req-1", token_ids=[20, 30]) + yield _make_request_output( + "req-1", token_ids=[40], finish_reason="stop", finished=True + ) + + engine.generate = MagicMock(side_effect=mock_generate) + serving = _build_serving_tokens(engine) + + request = GenerateRequest( + token_ids=[1, 2, 3], + sampling_params=SamplingParams(max_tokens=10), + model=MODEL_NAME, + stream=True, + ) + + response = await serving.serve_tokens(request) + chunks = [] + async for chunk in response: + chunks.append(chunk) + + parsed = _parse_sse_chunks(chunks) + + # 3 data chunks + [DONE] + assert parsed[-1] == "[DONE]" + data_chunks = [c for c in parsed if c != "[DONE]"] + assert len(data_chunks) == 3 + + assert data_chunks[0]["choices"][0]["token_ids"] == [10] + assert data_chunks[1]["choices"][0]["token_ids"] == [20, 30] + assert data_chunks[2]["choices"][0]["token_ids"] == [40] + assert data_chunks[2]["choices"][0]["finish_reason"] == "stop" + + +@pytest.mark.asyncio +async def test_stream_error_mid_generation(): + """finish_reason='error' mid-stream yields error chunk then [DONE].""" + engine = _mock_engine() + + async def mock_generate(*args, **kwargs): + yield _make_request_output("req-1", token_ids=[10]) + yield _make_request_output( + "req-1", token_ids=[20], finish_reason="error", finished=True + ) + + engine.generate = MagicMock(side_effect=mock_generate) + serving = _build_serving_tokens(engine) + + request = GenerateRequest( + token_ids=[1, 2, 3], + sampling_params=SamplingParams(max_tokens=10), + model=MODEL_NAME, + stream=True, + ) + + response = await serving.serve_tokens(request) + chunks = [] + async for chunk in response: + chunks.append(chunk) + + assert len(chunks) >= 2 + assert any("Internal server error" in chunk for chunk in chunks), ( + f"Expected error message in chunks: {chunks}" + ) + assert chunks[-1] == "data: [DONE]\n\n" + + +@pytest.mark.asyncio +async def test_stream_error_with_empty_delta(): + """finish_reason='error' with empty delta_token_ids still raises.""" + engine = _mock_engine() + + async def mock_generate(*args, **kwargs): + yield _make_request_output("req-1", token_ids=[10]) + yield _make_request_output( + "req-1", token_ids=[], finish_reason="error", finished=True + ) + + engine.generate = MagicMock(side_effect=mock_generate) + serving = _build_serving_tokens(engine) + + request = GenerateRequest( + token_ids=[1, 2, 3], + sampling_params=SamplingParams(max_tokens=10), + model=MODEL_NAME, + stream=True, + ) + + response = await serving.serve_tokens(request) + chunks = [] + async for chunk in response: + chunks.append(chunk) + + assert any("Internal server error" in chunk for chunk in chunks), ( + f"Expected error message in chunks: {chunks}" + ) + assert chunks[-1] == "data: [DONE]\n\n" + + +@pytest.mark.asyncio +async def test_stream_skips_empty_token_output(): + """Outputs with empty token_ids are skipped (no chunk emitted).""" + engine = _mock_engine() + + async def mock_generate(*args, **kwargs): + yield _make_request_output("req-1", token_ids=[10]) + yield _make_request_output("req-1", token_ids=[]) + yield _make_request_output( + "req-1", token_ids=[20], finish_reason="stop", finished=True + ) + + engine.generate = MagicMock(side_effect=mock_generate) + serving = _build_serving_tokens(engine) + + request = GenerateRequest( + token_ids=[1, 2, 3], + sampling_params=SamplingParams(max_tokens=10), + model=MODEL_NAME, + stream=True, + ) + + response = await serving.serve_tokens(request) + chunks = [] + async for chunk in response: + chunks.append(chunk) + + parsed = _parse_sse_chunks(chunks) + assert parsed[-1] == "[DONE]" + data_chunks = [c for c in parsed if c != "[DONE]"] + + # Only 2 data chunks — the empty one is skipped + assert len(data_chunks) == 2 + assert data_chunks[0]["choices"][0]["token_ids"] == [10] + assert data_chunks[1]["choices"][0]["token_ids"] == [20] + + +@pytest.mark.asyncio +async def test_stream_include_usage(): + """stream_options.include_usage emits a final usage-only chunk.""" + engine = _mock_engine() + + async def mock_generate(*args, **kwargs): + yield _make_request_output("req-1", token_ids=[10]) + yield _make_request_output( + "req-1", token_ids=[20], finish_reason="stop", finished=True + ) + + engine.generate = MagicMock(side_effect=mock_generate) + serving = _build_serving_tokens(engine) + + request = GenerateRequest( + token_ids=[1, 2, 3], + sampling_params=SamplingParams(max_tokens=10), + model=MODEL_NAME, + stream=True, + stream_options=StreamOptions(include_usage=True), + ) + + response = await serving.serve_tokens(request) + chunks = [] + async for chunk in response: + chunks.append(chunk) + + parsed = _parse_sse_chunks(chunks) + assert parsed[-1] == "[DONE]" + + # The chunk before [DONE] should be the usage-only chunk + usage_chunk = parsed[-2] + assert usage_chunk["choices"] == [] + assert usage_chunk["usage"]["prompt_tokens"] == 3 + assert usage_chunk["usage"]["completion_tokens"] == 2 + assert usage_chunk["usage"]["total_tokens"] == 5 + + +@pytest.mark.asyncio +async def test_stream_continuous_usage(): + """continuous_usage_stats adds usage to every data chunk.""" + engine = _mock_engine() + + async def mock_generate(*args, **kwargs): + yield _make_request_output("req-1", token_ids=[10]) + yield _make_request_output( + "req-1", token_ids=[20], finish_reason="stop", finished=True + ) + + engine.generate = MagicMock(side_effect=mock_generate) + serving = _build_serving_tokens(engine) + + request = GenerateRequest( + token_ids=[1, 2, 3], + sampling_params=SamplingParams(max_tokens=10), + model=MODEL_NAME, + stream=True, + stream_options=StreamOptions( + include_usage=True, + continuous_usage_stats=True, + ), + ) + + response = await serving.serve_tokens(request) + chunks = [] + async for chunk in response: + chunks.append(chunk) + + parsed = _parse_sse_chunks(chunks) + data_chunks = [c for c in parsed if isinstance(c, dict) and c.get("choices")] + + # Every data chunk should have usage + for i, dc in enumerate(data_chunks): + assert dc["usage"] is not None, f"chunk {i} missing usage" + assert dc["usage"]["prompt_tokens"] == 3 + + # First chunk: 1 completion token + assert data_chunks[0]["usage"]["completion_tokens"] == 1 + assert data_chunks[0]["usage"]["total_tokens"] == 4 + + # Second chunk: 2 completion tokens (cumulative) + assert data_chunks[1]["usage"]["completion_tokens"] == 2 + assert data_chunks[1]["usage"]["total_tokens"] == 5 + + +@pytest.mark.asyncio +async def test_stream_with_logprobs(): + """Streaming with logprobs includes logprob data in each chunk.""" + engine = _mock_engine() + + async def mock_generate(*args, **kwargs): + yield _make_request_output( + "req-1", + token_ids=[10], + logprobs=[{10: Logprob(logprob=-0.5)}], + ) + yield _make_request_output( + "req-1", + token_ids=[20], + logprobs=[{20: Logprob(logprob=-1.0)}], + finish_reason="stop", + finished=True, + ) + + engine.generate = MagicMock(side_effect=mock_generate) + serving = _build_serving_tokens(engine) + + request = GenerateRequest( + token_ids=[1, 2, 3], + sampling_params=SamplingParams(max_tokens=10, logprobs=1), + model=MODEL_NAME, + stream=True, + ) + + response = await serving.serve_tokens(request) + chunks = [] + async for chunk in response: + chunks.append(chunk) + + parsed = _parse_sse_chunks(chunks) + data_chunks = [c for c in parsed if isinstance(c, dict) and c.get("choices")] + + for dc in data_chunks: + lp = dc["choices"][0]["logprobs"] + assert lp is not None + assert len(lp["content"]) == 1 + assert lp["content"][0]["token"].startswith("token_id:") + + +@pytest.mark.asyncio +async def test_stream_prompt_tokens_details(): + """enable_prompt_tokens_details includes cached_tokens in final usage.""" + engine = _mock_engine() + + async def mock_generate(*args, **kwargs): + yield _make_request_output( + "req-1", + token_ids=[10], + finish_reason="stop", + finished=True, + num_cached_tokens=2, + ) + + engine.generate = MagicMock(side_effect=mock_generate) + serving = _build_serving_tokens(engine, enable_prompt_tokens_details=True) + + request = GenerateRequest( + token_ids=[1, 2, 3], + sampling_params=SamplingParams(max_tokens=10), + model=MODEL_NAME, + stream=True, + stream_options=StreamOptions(include_usage=True), + ) + + response = await serving.serve_tokens(request) + chunks = [] + async for chunk in response: + chunks.append(chunk) + + parsed = _parse_sse_chunks(chunks) + # Usage-only chunk (before [DONE]) + usage_chunk = parsed[-2] + assert usage_chunk["choices"] == [] + assert usage_chunk["usage"]["prompt_tokens_details"]["cached_tokens"] == 2 + + +@pytest.mark.asyncio +async def test_stream_prompt_tokens_details_zero_cached(): + """enable_prompt_tokens_details includes cached_tokens=0 in final usage. + + Regression test for https://github.com/vllm-project/vllm/issues/44377: + zero cached tokens must not be treated as falsy and omitted. + """ + engine = _mock_engine() + + async def mock_generate(*args, **kwargs): + yield _make_request_output( + "req-1", + token_ids=[10], + finish_reason="stop", + finished=True, + num_cached_tokens=0, + ) + + engine.generate = MagicMock(side_effect=mock_generate) + serving = _build_serving_tokens(engine, enable_prompt_tokens_details=True) + + request = GenerateRequest( + token_ids=[1, 2, 3], + sampling_params=SamplingParams(max_tokens=10), + model=MODEL_NAME, + stream=True, + stream_options=StreamOptions(include_usage=True), + ) + + response = await serving.serve_tokens(request) + chunks = [] + async for chunk in response: + chunks.append(chunk) + + parsed = _parse_sse_chunks(chunks) + # Usage-only chunk (before [DONE]) + usage_chunk = parsed[-2] + assert usage_chunk["choices"] == [] + # Zero cached tokens must be present, not omitted + assert usage_chunk["usage"]["prompt_tokens_details"] is not None + assert usage_chunk["usage"]["prompt_tokens_details"]["cached_tokens"] == 0 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_mm_serde.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_mm_serde.py new file mode 100644 index 0000000000000000000000000000000000000000..5c61bed3c5d32d5679b3dd2a9aa9c34b8f234e50 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_mm_serde.py @@ -0,0 +1,114 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Roundtrip tests for multimodal serde used by the +token_in_token_out generate endpoint. +""" + +import torch + +from vllm.entrypoints.scale_out.token_in_token_out.mm_serde import ( + decode_mm_kwargs_item, + encode_mm_kwargs_item, +) +from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( + MultiModalFeatures, + PlaceholderRangeInfo, +) +from vllm.multimodal.inputs import ( + MultiModalBatchedField, + MultiModalFieldElem, + MultiModalFlatField, + MultiModalKwargsItem, + MultiModalSharedField, +) + + +def test_mm_kwargs_item_roundtrip(): + """Full roundtrip test with all three field types and multiple dtypes.""" + e1 = MultiModalFieldElem( + data=torch.zeros(1000, dtype=torch.bfloat16), + field=MultiModalBatchedField(), + ) + e2 = MultiModalFieldElem( + data=torch.ones(100, dtype=torch.int32), + field=MultiModalSharedField(batch_size=4), + ) + e3 = MultiModalFieldElem( + data=torch.randn(20, dtype=torch.float32), + field=MultiModalFlatField(slices=[slice(0, 10), slice(10, 20)], dim=0), + ) + + item = MultiModalKwargsItem({"pixel_values": e1, "grid_thw": e2, "embeds": e3}) + encoded = encode_mm_kwargs_item(item) + + # Encoded result is a base64 string + assert isinstance(encoded, str) + + decoded = decode_mm_kwargs_item(encoded) + + assert set(decoded.keys()) == {"pixel_values", "grid_thw", "embeds"} + assert torch.equal(item["pixel_values"].data, decoded["pixel_values"].data) + assert torch.equal(item["grid_thw"].data, decoded["grid_thw"].data) + assert torch.equal(item["embeds"].data, decoded["embeds"].data) + assert isinstance(decoded["pixel_values"].field, MultiModalBatchedField) + assert isinstance(decoded["grid_thw"].field, MultiModalSharedField) + assert isinstance(decoded["embeds"].field, MultiModalFlatField) + + +def test_mm_kwargs_item_none_data(): + """Roundtrip with None data field.""" + elem = MultiModalFieldElem( + data=None, + field=MultiModalSharedField(batch_size=2), + ) + item = MultiModalKwargsItem({"empty": elem}) + encoded = encode_mm_kwargs_item(item) + decoded = decode_mm_kwargs_item(encoded) + + assert decoded["empty"].data is None + assert isinstance(decoded["empty"].field, MultiModalSharedField) + + +def test_mm_kwargs_item_nested_tensors(): + """Roundtrip with nested tensor data.""" + nested = [torch.randn(3, 4), torch.randn(5, 4)] + elem = MultiModalFieldElem( + data=nested, + field=MultiModalBatchedField(), + ) + item = MultiModalKwargsItem({"nested": elem}) + encoded = encode_mm_kwargs_item(item) + decoded = decode_mm_kwargs_item(encoded) + + decoded_data = decoded["nested"].data + assert len(decoded_data) == 2 + assert torch.equal(nested[0], decoded_data[0]) + assert torch.equal(nested[1], decoded_data[1]) + + +def test_mm_features_with_kwargs_data(): + """Test that MultiModalFeatures can carry serialized tensor data.""" + elem = MultiModalFieldElem( + data=torch.randn(5, 3, dtype=torch.float32), + field=MultiModalBatchedField(), + ) + item = MultiModalKwargsItem({"pixel_values": elem}) + encoded = encode_mm_kwargs_item(item) + + features = MultiModalFeatures( + mm_hashes={"image": ["abc123"]}, + mm_placeholders={"image": [PlaceholderRangeInfo(offset=0, length=10)]}, + kwargs_data={"image": [encoded]}, + ) + + # JSON roundtrip + json_str = features.model_dump_json() + features2 = MultiModalFeatures.model_validate_json(json_str) + + assert features2.mm_hashes == {"image": ["abc123"]} + assert features2.kwargs_data is not None + assert len(features2.kwargs_data["image"]) == 1 + + decoded = decode_mm_kwargs_item(features2.kwargs_data["image"][0]) + assert torch.equal(elem.data, decoded["pixel_values"].data) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_protocol.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_protocol.py new file mode 100644 index 0000000000000000000000000000000000000000..a77edc69e4641de1fc28ed44d218ca71cb8a7f5f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_protocol.py @@ -0,0 +1,70 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for the token_in_token_out request/response protocol. + +These tests intentionally avoid spinning up a server — they exercise the +pydantic validators on ``GenerateRequest`` directly so they run fast and +fail loudly if the validator semantics ever drift. +""" + +import json + +from vllm.entrypoints.scale_out.token_in_token_out.protocol import GenerateRequest +from vllm.sampling_params import SamplingParams + + +def _base_payload() -> dict: + return {"token_ids": [1, 2, 3], "sampling_params": {}} + + +def test_omitted_max_tokens_is_not_provided(): + """Body without ``max_tokens`` must surface as 'not provided' so the + server can apply its own default instead of the dataclass 16.""" + req = GenerateRequest.model_validate(_base_payload()) + # SamplingParams' dataclass default leaks through the parsed instance — + # this is exactly the bug the server-side defaulting works around. + assert req.sampling_params.max_tokens == 16 + assert not req.is_sampling_param_provided("max_tokens") + + +def test_explicit_max_tokens_is_provided(): + """Even when the client picks the same value as the dataclass default, + it must register as explicitly set so the server won't override it.""" + payload = _base_payload() + payload["sampling_params"] = {"max_tokens": 16} + req = GenerateRequest.model_validate(payload) + assert req.sampling_params.max_tokens == 16 + assert req.is_sampling_param_provided("max_tokens") + + payload["sampling_params"] = {"max_tokens": 256} + req = GenerateRequest.model_validate(payload) + assert req.sampling_params.max_tokens == 256 + assert req.is_sampling_param_provided("max_tokens") + + +def test_other_fields_tracked_independently(): + payload = _base_payload() + payload["sampling_params"] = {"temperature": 0.0} + req = GenerateRequest.model_validate(payload) + assert not req.is_sampling_param_provided("max_tokens") + assert req.is_sampling_param_provided("temperature") + + +def test_json_roundtrip_preserves_provided_keys(): + payload = _base_payload() + payload["sampling_params"] = {"temperature": 0.5} + req = GenerateRequest.model_validate_json(json.dumps(payload)) + assert not req.is_sampling_param_provided("max_tokens") + assert req.is_sampling_param_provided("temperature") + + +def test_internal_instance_construction_treats_all_as_provided(): + """When internal callers build ``GenerateRequest`` from a pre-resolved + ``SamplingParams`` instance, every field is considered explicitly set + so server-side defaulting can't clobber values resolved upstream.""" + sp = SamplingParams(max_tokens=500, temperature=0.0) + req = GenerateRequest(token_ids=[1, 2, 3], sampling_params=sp) + assert req.is_sampling_param_provided("max_tokens") + assert req.is_sampling_param_provided("temperature") + # And keys we never touched should also count as provided in this path. + assert req.is_sampling_param_provided("top_p") diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_return_routed_experts.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_return_routed_experts.py new file mode 100644 index 0000000000000000000000000000000000000000..5b02bd6257633a45b22380d44f053d020e106ce3 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_return_routed_experts.py @@ -0,0 +1,81 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import io + +import httpx +import numpy as np +import pybase64 as base64 +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer + +MODEL_NAME = "TitanML/tiny-mixtral" +GEN_ENDPOINT = "/inference/v1/generate" + +# tiny-mixtral config: 8 local experts, top-2 routing, 2 hidden layers. +# The published config has sliding_window=4096, which produces +# SlidingWindowSpec kv-cache groups; RoutedExpertsManager requires a +# FullAttentionSpec group, so we override sliding_window=null below. +NUM_LOCAL_EXPERTS = 8 +NUM_EXPERTS_PER_TOK = 2 +NUM_HIDDEN_LAYERS = 2 + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--max-model-len", + "256", + "--max-num-seqs", + "32", + "--enforce-eager", + "--enable-return-routed-experts", + "--hf-overrides", + '{"sliding_window": null}', + ] + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server: RemoteOpenAIServer): + transport = httpx.AsyncHTTPTransport(uds=server.uds) if server.uds else None + headers = {"Authorization": f"Bearer {server.DUMMY_API_KEY}"} + async with httpx.AsyncClient( + transport=transport, + base_url=server.url_root, + timeout=600, + headers=headers, + ) as c: + yield c + + +@pytest.mark.asyncio +async def test_generate_routed_experts(client): + """Test that /inference/v1/generate returns routed_experts when enabled.""" + payload = { + "model": MODEL_NAME, + "token_ids": [1, 2, 3], + "sampling_params": {"max_tokens": 10, "temperature": 0.0}, + "stream": False, + } + resp = await client.post(GEN_ENDPOINT, json=payload) + resp.raise_for_status() + data = resp.json() + + choice = data["choices"][0] + + assert choice["routed_experts"] is not None + assert choice["token_ids"] is not None + + # routed_experts is base64-encoded .npy bytes; decode to ndarray. + routed_experts = np.load(io.BytesIO(base64.b64decode(choice["routed_experts"]))) + assert routed_experts.ndim == 3 + num_tokens, num_layers, topk = routed_experts.shape + assert num_tokens > 0 + assert num_layers == NUM_HIDDEN_LAYERS + assert topk == NUM_EXPERTS_PER_TOK + assert (routed_experts >= 0).all() + assert (routed_experts < NUM_LOCAL_EXPERTS).all() diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_serving_multimodal_tokens.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_serving_multimodal_tokens.py new file mode 100644 index 0000000000000000000000000000000000000000..0961d58e5e5cd1b05de61905f641d1caf185ce44 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_serving_multimodal_tokens.py @@ -0,0 +1,158 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for multimodal features through the /inference/v1/generate endpoint. + +Mirrors test_serving_tokens.py but exercises the multimodal piping +using Qwen/Qwen3-VL-2B-Instruct end-to-end via the server's /render -> +/generate -> /detokenize path. Intentionally avoids running the HF +processor in the pytest parent process to keep os.fork() in sibling +tests (e.g. test_weight_transfer_llm.py) deadlock-free. +""" + +import os + +import httpx +import pytest +import pytest_asyncio +from PIL import Image + +from tests.utils import RemoteOpenAIServer +from vllm.multimodal.utils import encode_image_url + +MODEL_NAME = "Qwen/Qwen3-VL-2B-Instruct" +GEN_ENDPOINT = "/inference/v1/generate" +RENDER_ENDPOINT = "/v1/chat/completions/render" +DETOKENIZE_ENDPOINT = "/detokenize" + + +@pytest.fixture(scope="module") +def test_image(): + return Image.new("RGB", (224, 224), color=(255, 0, 0)) + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "4096", + "--enforce-eager", + "--no-enable-prefix-caching", + ] + + envs = os.environ.copy() + envs["VLLM_ROCM_USE_SKINNY_GEMM"] = "0" + + with RemoteOpenAIServer(MODEL_NAME, args, env_dict=envs) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server: RemoteOpenAIServer): + transport = httpx.AsyncHTTPTransport(uds=server.uds) if server.uds else None + headers = {"Authorization": f"Bearer {server.DUMMY_API_KEY}"} + async with httpx.AsyncClient( + transport=transport, + base_url=server.url_root, + timeout=600, + headers=headers, + ) as c: + yield c + + +@pytest.mark.asyncio +async def test_render_to_generate_roundtrip(client, test_image): + """End-to-end: render a multimodal chat -> feed into generate -> decode. + + All preprocessing and detokenization happens in the server subprocess; + the pytest parent never imports transformers or touches torch tensors. + """ + data_url = encode_image_url(test_image, format="PNG") + + render_payload = { + "model": MODEL_NAME, + "messages": [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": data_url}}, + { + "type": "text", + "text": "What color is this image? Answer in one word.", + }, + ], + } + ], + } + + render_resp = await client.post(RENDER_ENDPOINT, json=render_payload) + render_resp.raise_for_status() + render_data = render_resp.json() + + # Validate render output structure: keys exist and values are non-empty + # and well-typed. + assert "token_ids" in render_data + assert isinstance(render_data["token_ids"], list) + assert len(render_data["token_ids"]) > 0 + assert all(isinstance(t, int) for t in render_data["token_ids"]) + + assert "features" in render_data + features = render_data["features"] + assert features is not None + assert isinstance(features, dict) + + assert "mm_hashes" in features + assert "image" in features["mm_hashes"] + image_hashes = features["mm_hashes"]["image"] + assert isinstance(image_hashes, list) + assert len(image_hashes) > 0 + assert all(isinstance(h, str) and h for h in image_hashes) + + assert "mm_placeholders" in features + assert "image" in features["mm_placeholders"] + image_placeholders = features["mm_placeholders"]["image"] + assert isinstance(image_placeholders, list) + assert len(image_placeholders) > 0 + for p in image_placeholders: + assert isinstance(p.get("offset"), int) + assert isinstance(p.get("length"), int) + assert p["length"] > 0 + + assert "kwargs_data" in features + assert "image" in features["kwargs_data"] + assert len(features["kwargs_data"]["image"]) > 0 + + # Build generate request from render output + generate_payload = render_data + generate_payload["sampling_params"] = { + "max_tokens": 10, + "temperature": 0.0, + } + + gen_resp = await client.post(GEN_ENDPOINT, json=generate_payload) + gen_resp.raise_for_status() + gen_data = gen_resp.json() + + assert "choices" in gen_data + assert isinstance(gen_data["choices"], list) + assert len(gen_data["choices"]) >= 1 + choice = gen_data["choices"][0] + assert "token_ids" in choice + assert isinstance(choice["token_ids"], list) + assert len(choice["token_ids"]) > 0 + assert all(isinstance(t, int) for t in choice["token_ids"]) + + detok_resp = await client.post( + DETOKENIZE_ENDPOINT, + json={"model": MODEL_NAME, "tokens": choice["token_ids"]}, + ) + detok_resp.raise_for_status() + detok_data = detok_resp.json() + assert "prompt" in detok_data + text = detok_data["prompt"] + assert isinstance(text, str) + assert len(text) > 0 + assert "red" in text.lower(), ( + f"Expected model to identify the red image, got: {text!r}" + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py new file mode 100644 index 0000000000000000000000000000000000000000..d7e0ddc5451ddbba73086193fd213a783c82d47e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_serving_tokens.py @@ -0,0 +1,425 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json +import os + +import httpx +import pytest +import pytest_asyncio +from transformers import AutoTokenizer + +from tests.utils import RemoteOpenAIServer +from vllm.config import ModelConfig +from vllm.config.utils import getattr_iter +from vllm.v1.engine.detokenizer import check_stop_strings + +MODEL_NAME = "Qwen/Qwen3-0.6B" +GEN_ENDPOINT = "/inference/v1/generate" + + +def get_vocab_size(model_name): + config = ModelConfig( + model=model_name, + seed=0, + dtype="bfloat16", + ) + return config.get_vocab_size() + + +@pytest.fixture(scope="module") +def tokenizer(): + return AutoTokenizer.from_pretrained(MODEL_NAME) + + +@pytest.fixture(scope="module") +def messages(): + return [ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "How many countries are in the EU?"}, + ] + + +@pytest.fixture(scope="module") +def server(request): + args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "1024", + "--enforce-eager", + # On ROCm (e.g. MI355X/gfx950), bf16 GEMM results can differ by + # 1 ULP when the batch dimension (M) changes, because different M + # values cause the Tensile backend to select different tile + # configurations with different fp32 accumulation orders. With + # prefix caching, cache-miss prefills compute all tokens in one + # pass (large M) while cache-hit requests compute only the + # uncached suffix (small M), seeding a divergence that amplifies + # through the residual stream and flips argmax tokens. + # See: https://github.com/vllm-project/vllm/issues/33123 + # + # Either disable prefix caching entirely, or enable it with + # --deterministic-prefix-caching which forces cache-miss prefills + # to split at block boundaries so the suffix GEMM shape is always + # identical regardless of cache state. + # + # Option A: disable prefix caching + "--no-enable-prefix-caching", + # + # Option B: deterministic prefix caching + # "--enable-prefix-caching", + # "--deterministic-prefix-caching", + ] + + extra_args = getattr(request, "param", None) + if extra_args is not None: + args = args + ( + list(extra_args) + if isinstance(extra_args, (list, tuple)) + else [str(extra_args)] + ) + + envs = os.environ.copy() + # See: https://github.com/vllm-project/vllm/pull/33493#issuecomment-3888060787 + envs["VLLM_ROCM_USE_SKINNY_GEMM"] = "0" + + with RemoteOpenAIServer(MODEL_NAME, args, env_dict=envs) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server: RemoteOpenAIServer): + transport = httpx.AsyncHTTPTransport(uds=server.uds) if server.uds else None + headers = {"Authorization": f"Bearer {server.DUMMY_API_KEY}"} + async with httpx.AsyncClient( + transport=transport, + base_url=server.url_root, + timeout=600, + headers=headers, + ) as c: + yield c + + +@pytest.mark.asyncio +async def test_generate_endpoint(client): + payload = { + "model": MODEL_NAME, + "token_ids": [1, 2, 3], + "sampling_params": {"max_tokens": 5}, + "stream": False, + } + resp = await client.post(GEN_ENDPOINT, json=payload) + resp.raise_for_status() + data = resp.json() + assert "choices" in data + + +@pytest.mark.asyncio +async def test_generate_defaults_max_tokens_when_omitted(client): + """Regression: omitting ``max_tokens`` must not silently cap at 16. + + ``SamplingParams.max_tokens`` defaults to 16. Before the server-side + defaulting was wired up, every request that didn't set ``max_tokens`` + truncated mid-generation. The server should now fill it in from + ``max_model_len - prompt_len`` (matching ``/v1/chat/completions``). + """ + payload = { + "model": MODEL_NAME, + "token_ids": [1, 2, 3], + "sampling_params": { + "temperature": 0.0, + "ignore_eos": True, + }, + "stream": False, + } + resp = await client.post(GEN_ENDPOINT, json=payload) + resp.raise_for_status() + data = resp.json() + completion_tokens = len(data["choices"][0]["token_ids"]) + # max_model_len=1024 in the test fixture, prompt is 3 tokens, so we + # should get ~1021 tokens of output (capped at max_model_len boundary). + assert completion_tokens > 16, ( + f"expected server-side default to exceed the legacy 16-token cap, " + f"got {completion_tokens}" + ) + + +@pytest.mark.asyncio +async def test_generate_stream(client): + payload = { + "model": MODEL_NAME, + "token_ids": [1, 2, 3], + "sampling_params": {"max_tokens": 5}, + "stream": True, + } + async with client.stream("POST", GEN_ENDPOINT, json=payload) as resp: + resp.raise_for_status() + chunks = [] + async for line in resp.aiter_lines(): + if not line.startswith("data: "): + continue + payload_str = line[len("data: ") :] + if payload_str == "[DONE]": + break + chunks.append(json.loads(payload_str)) + + assert len(chunks) > 0 + # Every chunk has choices with token_ids + all_token_ids = [] + for chunk in chunks: + assert "choices" in chunk + assert len(chunk["choices"]) == 1 + choice = chunk["choices"][0] + assert "token_ids" in choice + assert len(choice["token_ids"]) > 0 + all_token_ids.extend(choice["token_ids"]) + + # Last chunk should have a finish_reason + assert chunks[-1]["choices"][0]["finish_reason"] is not None + + # Streaming should produce the same tokens as non-streaming + non_stream_resp = await client.post( + GEN_ENDPOINT, + json={ + "model": MODEL_NAME, + "token_ids": [1, 2, 3], + "sampling_params": {"max_tokens": 5, "temperature": 0.0}, + "stream": False, + }, + ) + non_stream_data = non_stream_resp.json() + # Just verify we got the right number of tokens + assert len(all_token_ids) == len(non_stream_data["choices"][0]["token_ids"]) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("logprobs_value", [0, 1, 5]) +async def test_generate_logprobs(client, logprobs_value): + payload = { + "model": MODEL_NAME, + "token_ids": [1, 2, 3], + "sampling_params": { + "max_tokens": 5, + "temperature": 0.0, + "logprobs": logprobs_value, + }, + "stream": False, + } + resp = await client.post(GEN_ENDPOINT, json=payload) + resp.raise_for_status() + data = resp.json() + choice = data["choices"][0] + assert choice["logprobs"] is not None + logprobs_content = choice["logprobs"]["content"] + assert len(logprobs_content) == len(choice["token_ids"]) + for entry in logprobs_content: + assert "logprob" in entry + assert len(entry["top_logprobs"]) >= 1 + assert len(entry["top_logprobs"]) == max(logprobs_value, 1) + + +@pytest.mark.asyncio +async def test_same_response_as_chat_completions(client, tokenizer, messages): + token_ids = tokenizer.apply_chat_template( + messages, + add_generation_prompt=True, + enable_thinking=False, # default with Qwen3 + return_dict=True, # default with Transformers v5 + ).input_ids + + for ignore_eos in [True, False]: + payload = { + "model": MODEL_NAME, + "token_ids": token_ids, + "sampling_params": { + "max_tokens": 24, + "temperature": 0.0, + # NOTE coordinator will set this to skip detokenization + "detokenize": False, + "ignore_eos": ignore_eos, + }, + "stream": False, + } + generate_resp = await client.post(GEN_ENDPOINT, json=payload) + generate_data = generate_resp.json() + gen_token_ids = generate_data["choices"][0]["token_ids"] + generate_res = tokenizer.decode(gen_token_ids, skip_special_tokens=True) + + payload = { + "model": MODEL_NAME, + "messages": messages, + "max_tokens": 24, + "temperature": 0.0, + "stream": False, + "ignore_eos": ignore_eos, + "chat_template_kwargs": {"enable_thinking": False}, + } + completions_resp = await client.post("/v1/chat/completions", json=payload) + completions_data = completions_resp.json() + completions_res = completions_data["choices"][0]["message"]["content"] + + if ignore_eos: + # When ignoring EOS, only compare up to the first EOS token + # Post-EOS generation is undefined and may differ + eos_tokens = { + tokenizer.eos_token_id, + *getattr_iter( + tokenizer, + [ + "extra_special_tokens_ids", # Transformers v5 + "additional_special_tokens_ids", # Transformers v4 + ], + [], + ), + } + # Find first EOS in generated tokens + eos_pos = None + for i, tid in enumerate(gen_token_ids): + if tid in eos_tokens: + eos_pos = i + break + if eos_pos is not None: + gen_token_ids_truncated = gen_token_ids[:eos_pos] + generate_res = tokenizer.decode( + gen_token_ids_truncated, skip_special_tokens=True + ) + # Truncate completions_res to same length for comparison + completions_res = completions_res[: len(generate_res)] + + assert generate_res == completions_res + + +@pytest.mark.asyncio +async def test_stop_string_workflow(client, tokenizer, messages): + token_ids = tokenizer.apply_chat_template( + messages, + add_generation_prompt=True, + enable_thinking=False, # default with Qwen3 + return_dict=True, # default with Transformers v5 + ).input_ids + payload = { + "model": MODEL_NAME, + "token_ids": token_ids, + "sampling_params": { + "max_tokens": 24, + "temperature": 0.0, + "detokenize": False, + # stop strings are only supported when detokenize is True. + "stop": ["27 member"], + }, + # TODO stream test is much more interesting + "stream": False, + } + with pytest.raises(httpx.HTTPStatusError): + generate_resp = await client.post(GEN_ENDPOINT, json=payload) + generate_resp.raise_for_status() + + payload["sampling_params"]["stop"] = None + generate_resp = await client.post( + GEN_ENDPOINT, json=payload, headers={"X-Request-Id": "42"} + ) + generate_data = generate_resp.json() + generate_res = tokenizer.decode( + generate_data["choices"][0]["token_ids"], skip_special_tokens=True + ) + + # NOTE This is under the responsibility of the coordinator + # stop_checker = StopChecker( + # max_model_len=1024, get_tokenizer_for_seq=lambda _: tokenizer + # ) + stop_str, truncate_to = check_stop_strings( + generate_res, len(generate_res), ["27 member"], False + ) + assert stop_str == "27 member" + # abort request that hit stop string (requires tokens-only mode) + # res = await client.post("/abort_requests", json={"request_ids": ["generate-tokens-42"]}) # noqa: E501 + # res.raise_for_status() + generate_res = generate_res[:truncate_to] + + # Get stop_str response from chat completions + payload = { + "model": MODEL_NAME, + "messages": messages, + "max_tokens": 24, + "temperature": 0.0, + "stream": False, + "stop": ["27 member"], + "chat_template_kwargs": dict(enable_thinking=False), + } + completions_resp = await client.post("/v1/chat/completions", json=payload) + completions_data = completions_resp.json() + completions_res = completions_data["choices"][0]["message"]["content"] + assert generate_res == completions_res + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "server", + [ + [ + "--enable-lora", + "--lora-modules", + "Alice=charent/self_cognition_Alice", + "Bob=charent/self_cognition_Bob", + "--max-lora-rank", + "64", + "--max-cpu-loras", + "2", + ] + ], + indirect=True, +) +async def test_generate_with_lora_adapter(client, tokenizer, messages): + # Verify adapters are listed + models_resp = await client.get("/v1/models") + models_resp.raise_for_status() + models = {m["id"] for m in models_resp.json().get("data", [])} + assert {"Alice", "Bob"}.issubset(models) + + # Generate using a LoRA adapter by specifying its name as the model + payload = { + "model": "Alice", + "token_ids": [1, 2, 3], + "sampling_params": {"max_tokens": 5}, + "stream": False, + } + resp = await client.post(GEN_ENDPOINT, json=payload) + resp.raise_for_status() + data = resp.json() + assert "choices" in data + + token_ids = tokenizer.apply_chat_template( + messages, + add_generation_prompt=True, + enable_thinking=False, # default with Qwen3 + return_dict=True, # default with Transformers v5 + ).input_ids + payload = { + "model": "Alice", + "token_ids": token_ids, + "sampling_params": { + "max_tokens": 24, + "temperature": 0.0, + "detokenize": False, + }, + "stream": False, + } + generate_resp = await client.post(GEN_ENDPOINT, json=payload) + generate_data = generate_resp.json() + generate_res = tokenizer.decode( + generate_data["choices"][0]["token_ids"], skip_special_tokens=True + ) + + payload = { + "model": "Alice", + "messages": messages, + "max_tokens": 24, + "temperature": 0.0, + "stream": False, + "chat_template_kwargs": dict(enable_thinking=False), + } + completions_resp = await client.post("/v1/chat/completions", json=payload) + completions_data = completions_resp.json() + completions_res = completions_data["choices"][0]["message"]["content"] + + assert generate_res == completions_res diff --git a/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_tokens_logprobs.py b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_tokens_logprobs.py new file mode 100644 index 0000000000000000000000000000000000000000..b6eaca724f0b54da192ebc5b85abca98927946a5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/scale_out/token_in_token_out/test_tokens_logprobs.py @@ -0,0 +1,28 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from vllm.entrypoints.scale_out.token_in_token_out.serving import ServingTokens +from vllm.logprobs import Logprob + + +def test_top_logprobs_alternatives_have_own_token_ids(): + """Each top_logprobs alternative must carry its own token_id placeholder.""" + result = ServingTokens._create_tokens_logprobs( + None, + token_ids=[262], + top_logprobs=[{262: Logprob(-0.1), 257: Logprob(-1.2), 428: Logprob(-2.3)}], + num_output_top_logprobs=2, + ) + tokens = {e.token for e in result.content[0].top_logprobs} + assert tokens == {"token_id:262", "token_id:257"}, f"got {tokens}" + + +def test_logprobs_zero_emits_sampled_token(): + """logprobs=0 must still emit 1 entry (the sampled token).""" + result = ServingTokens._create_tokens_logprobs( + None, + token_ids=[7], + top_logprobs=[{7: Logprob(-0.9), 8: Logprob(-1.1)}], + num_output_top_logprobs=0, + ) + assert len(result.content[0].top_logprobs) == 1 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/dev/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/dev/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/dev/rpc/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/dev/rpc/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/dev/rpc/test_collective_rpc.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/dev/rpc/test_collective_rpc.py new file mode 100644 index 0000000000000000000000000000000000000000..04e8d7d8715db682274c7bd70af4f3bffdcfdbcf --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/dev/rpc/test_collective_rpc.py @@ -0,0 +1,84 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from typing import Any + +import pytest +import requests + +from tests.utils import RemoteOpenAIServer + +MODEL_NAME = "Qwen/Qwen3-0.6B" + + +class TestWorkerExtension: + def get_model_name(self) -> str: + """Test non-pydantic return type.""" + return MODEL_NAME + + def echo_args_kwargs(self, *args, **kwargs) -> dict[str, Any]: + """Echo back both args and kwargs.""" + return dict( + args=list(args), + kwargs=kwargs, + total_items=len(args) + len(kwargs), + ) + + def return_none(self, *args, **kwargs) -> None: + """Test method that does not return anything""" + return + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--max-model-len", + "8192", + "--max-num-seqs", + "128", + "--worker-extension-cls", + "tests.entrypoints.serve.dev.rpc.test_collective_rpc.TestWorkerExtension", + ] + with RemoteOpenAIServer( + MODEL_NAME, + args, + env_dict={"VLLM_SERVER_DEV_MODE": "1", "CUDA_VISIBLE_DEVICES": "0"}, + ) as remote_server: + yield remote_server + + +def test_get_model_name(server): + """Test basic response""" + response = requests.post( + server.url_for("collective_rpc"), json={"method": "get_model_name"} + ) + assert response.status_code == 200 + results = response.json() + assert "results" in results + assert results["results"] == [MODEL_NAME] + + +def test_return_none(server): + """Test return none""" + response = requests.post( + server.url_for("collective_rpc"), json={"method": "return_none"} + ) + assert response.status_code == 200 + results = response.json() + assert results["results"] == [None] + + +def test_echo_args_kwargs(server): + """Test args, kwargs, and dict response""" + args = ["arg1", "arg2"] + kwargs = {"key1": "value1", "key2": "value2"} + response = requests.post( + server.url_for("collective_rpc"), + json={"method": "echo_args_kwargs", "args": args, "kwargs": kwargs}, + ) + assert response.status_code == 200 + results = response.json() + result = results["results"][0] + assert result["args"] == args + assert result["kwargs"] == kwargs + assert result["total_items"] == len(args) + len(kwargs) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/dev/test_sleep.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/dev/test_sleep.py new file mode 100644 index 0000000000000000000000000000000000000000..5204d793175636c124eecb1b66ac388880da8793 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/dev/test_sleep.py @@ -0,0 +1,110 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import requests +from prometheus_client.parser import text_string_to_metric_families + +from tests.utils import RemoteOpenAIServer + +MODEL_NAME = "meta-llama/Llama-3.2-1B" + + +def test_sleep_mode(): + # dtype, max-len etc set so that this can run in CI + args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "8192", + "--max-num-seqs", + "128", + "--enable-sleep-mode", + ] + + with RemoteOpenAIServer( + MODEL_NAME, + args, + env_dict={"VLLM_SERVER_DEV_MODE": "1", "CUDA_VISIBLE_DEVICES": "0"}, + ) as remote_server: + response = requests.post(remote_server.url_for("sleep"), params={"level": "1"}) + assert response.status_code == 200 + response = requests.get(remote_server.url_for("is_sleeping")) + assert response.status_code == 200 + assert response.json().get("is_sleeping") is True + + # check sleep metrics + response = requests.get(remote_server.url_for("metrics")) + assert response.status_code == 200 + awake, weights_offloaded, discard_all = _get_sleep_metrics_from_api(response) + assert awake == 0 + assert weights_offloaded == 1 + assert discard_all == 0 + + response = requests.post(remote_server.url_for("wake_up")) + assert response.status_code == 200 + response = requests.get(remote_server.url_for("is_sleeping")) + assert response.status_code == 200 + assert response.json().get("is_sleeping") is False + + # check sleep metrics + response = requests.get(remote_server.url_for("metrics")) + assert response.status_code == 200 + awake, weights_offloaded, discard_all = _get_sleep_metrics_from_api(response) + assert awake == 1 + assert weights_offloaded == 0 + assert discard_all == 0 + + # test wake up with tags + response = requests.post(remote_server.url_for("sleep"), params={"level": "1"}) + assert response.status_code == 200 + + response = requests.post( + remote_server.url_for("wake_up"), params={"tags": ["weights"]} + ) + assert response.status_code == 200 + + # is sleeping should be false after waking up any part of the engine + response = requests.get(remote_server.url_for("is_sleeping")) + assert response.status_code == 200 + assert response.json().get("is_sleeping") is True + + response = requests.post( + remote_server.url_for("wake_up"), params={"tags": ["kv_cache"]} + ) + assert response.status_code == 200 + + response = requests.get(remote_server.url_for("is_sleeping")) + assert response.status_code == 200 + assert response.json().get("is_sleeping") is False + + # check sleep metrics + response = requests.get(remote_server.url_for("metrics")) + assert response.status_code == 200 + awake, weights_offloaded, discard_all = _get_sleep_metrics_from_api(response) + assert awake == 1 + assert weights_offloaded == 0 + assert discard_all == 0 + + +def _get_sleep_metrics_from_api(response: requests.Response): + """Return (awake, weights_offloaded, discard_all)""" + + awake, weights_offloaded, discard_all = None, None, None + + for family in text_string_to_metric_families(response.text): + if family.name == "vllm:engine_sleep_state": + for sample in family.samples: + if sample.name == "vllm:engine_sleep_state": + for label_name, label_value in sample.labels.items(): + if label_value == "awake": + awake = sample.value + elif label_value == "weights_offloaded": + weights_offloaded = sample.value + elif label_value == "discard_all": + discard_all = sample.value + + assert awake is not None + assert weights_offloaded is not None + assert discard_all is not None + + return awake, weights_offloaded, discard_all diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/instrumentator/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/instrumentator/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/instrumentator/test_basic.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/instrumentator/test_basic.py new file mode 100644 index 0000000000000000000000000000000000000000..61cdbd9b021721771ef40a970605bccb489e47c2 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/instrumentator/test_basic.py @@ -0,0 +1,222 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import asyncio +from http import HTTPStatus +from unittest.mock import AsyncMock, Mock + +import openai +import pytest +import pytest_asyncio +import requests +from fastapi import Request + +from tests.utils import RemoteOpenAIServer +from vllm.v1.engine.exceptions import EngineDeadError +from vllm.version import __version__ as VLLM_VERSION + +MODEL_NAME = "Qwen/Qwen3-0.6B" + + +@pytest.fixture(scope="module") +def server_args(request: pytest.FixtureRequest) -> list[str]: + """Provide extra arguments to the server via indirect parametrization + + Usage: + + >>> @pytest.mark.parametrize( + >>> "server_args", + >>> [ + >>> ["--max-model-len", "10100"], + >>> [ + >>> "--model=NousResearch/Hermes-3-Llama-3.1-70B", + >>> "--enable-auto-tool-choice", + >>> ], + >>> ], + >>> indirect=True, + >>> ) + >>> def test_foo(server, client): + >>> ... + + This will run `test_foo` twice with servers with: + - `--max-model-len 10100` + - `--model=NousResearch/Hermes-3-Llama-3.1-70B --enable-auto-tool-choice`. + + """ + if not hasattr(request, "param"): + return [] + + val = request.param + + if isinstance(val, str): + return [val] + + return request.param + + +@pytest.fixture(scope="module") +def server(server_args): + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-model-len", + "8192", + "--enforce-eager", + "--max-num-seqs", + "128", + *server_args, + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +async def test_show_version(server: RemoteOpenAIServer): + response = requests.get(server.url_for("version")) + response.raise_for_status() + + assert response.json() == {"version": VLLM_VERSION} + + +@pytest.mark.asyncio +async def test_check_health(server: RemoteOpenAIServer): + response = requests.get(server.url_for("health")) + + assert response.status_code == HTTPStatus.OK + + +@pytest.mark.parametrize( + "server_args", + [ + pytest.param(["--max-model-len", "10100"]), + ], + indirect=True, +) +@pytest.mark.asyncio +async def test_request_cancellation(server: RemoteOpenAIServer): + # clunky test: send an ungodly amount of load in with short timeouts + # then ensure that it still responds quickly afterwards + + chat_input = [{"role": "user", "content": "Write a long story"}] + client = server.get_async_client(timeout=0.5) + tasks = [] + # Request about 2 million tokens + for _ in range(200): + task = asyncio.create_task( + client.chat.completions.create( + messages=chat_input, + model=MODEL_NAME, + max_tokens=10000, + extra_body={"min_tokens": 10000}, + temperature=0.0, + ) + ) + tasks.append(task) + + done, pending = await asyncio.wait(tasks, return_when=asyncio.ALL_COMPLETED) + + # Make sure all requests were sent to the server and timed out + # (We don't want to hide other errors like 400s that would invalidate this + # test) + assert len(pending) == 0 + for d in done: + with pytest.raises(openai.APITimeoutError): + d.result() + + # If the server had not cancelled all the other requests, then it would not + # be able to respond to this one within the timeout + client = server.get_async_client(timeout=5) + response = await client.chat.completions.create( + messages=chat_input, model=MODEL_NAME, max_tokens=10, temperature=0.0 + ) + + assert len(response.choices) == 1 + + +@pytest.mark.asyncio +async def test_request_wrong_content_type(server: RemoteOpenAIServer): + chat_input = [{"role": "user", "content": "Write a long story"}] + client = server.get_async_client() + + with pytest.raises(openai.APIStatusError): + await client.chat.completions.create( + messages=chat_input, + model=MODEL_NAME, + max_tokens=10000, + extra_headers={"Content-Type": "application/x-www-form-urlencoded"}, + ) + + +@pytest.mark.parametrize( + "server_args", + [pytest.param(["--enable-server-load-tracking"], id="enable-server-load-tracking")], + indirect=True, +) +@pytest.mark.asyncio +async def test_server_load(server: RemoteOpenAIServer): + # Check initial server load + response = requests.get(server.url_for("load")) + assert response.status_code == HTTPStatus.OK + assert response.json().get("server_load") == 0 + + def make_long_completion_request(): + return requests.post( + server.url_for("v1/completions"), + headers={"Content-Type": "application/json"}, + json={ + "prompt": "Give me a long story", + "max_tokens": 1000, + "temperature": 0, + }, + ) + + # Start the completion request in a background thread. + completion_future = asyncio.create_task( + asyncio.to_thread(make_long_completion_request) + ) + + # Give a short delay to ensure the request has started. + await asyncio.sleep(0.1) + + # Check server load while the completion request is running. + response = requests.get(server.url_for("load")) + assert response.status_code == HTTPStatus.OK + assert response.json().get("server_load") == 1 + + # Wait for the completion request to finish. + await completion_future + await asyncio.sleep(0.1) + + # Check server load after the completion request has finished. + response = requests.get(server.url_for("load")) + assert response.status_code == HTTPStatus.OK + assert response.json().get("server_load") == 0 + + +@pytest.mark.asyncio +async def test_health_check_engine_dead_error(): + # Import the health function directly to test it in isolation + from vllm.entrypoints.serve.instrumentator.health import health + + # Create a mock request that simulates what FastAPI would provide + mock_request = Mock(spec=Request) + mock_app_state = Mock() + mock_engine_client = AsyncMock() + mock_engine_client.check_health.side_effect = EngineDeadError() + mock_app_state.engine_client = mock_engine_client + mock_request.app.state = mock_app_state + + # Test the health function directly with our mocked request + # This simulates what would happen if the engine dies + response = await health(mock_request) + + # Assert that it returns 503 Service Unavailable + assert response.status_code == 503 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/instrumentator/test_metrics.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/instrumentator/test_metrics.py new file mode 100644 index 0000000000000000000000000000000000000000..01874b0141ad78a78d8206a8157daa2a6c11981c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/instrumentator/test_metrics.py @@ -0,0 +1,496 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import asyncio +import subprocess +import sys +import tempfile +import time +from http import HTTPStatus + +import openai +import pytest +import pytest_asyncio +import requests +from prometheus_client.parser import text_string_to_metric_families +from transformers import AutoTokenizer + +from tests.conftest import LocalAssetServer +from tests.utils import RemoteOpenAIServer +from vllm import version +from vllm.utils.network_utils import get_open_port + +MODELS = { + "text": "TinyLlama/TinyLlama-1.1B-Chat-v1.0", + "multimodal": "HuggingFaceTB/SmolVLM-256M-Instruct", +} +PREV_MINOR_VERSION = version._prev_minor_version() + + +@pytest.fixture(scope="module", params=list(MODELS.keys())) +def model_key(request): + yield request.param + + +@pytest.fixture(scope="module") +def default_server_args(): + return [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-model-len", + "1024", + "--enforce-eager", + "--max-num-seqs", + "128", + ] + + +@pytest.fixture( + scope="module", + params=[ + "", + "--enable-chunked-prefill", + f"--show-hidden-metrics-for-version={PREV_MINOR_VERSION}", + ], +) +def server(model_key, default_server_args, request): + if request.param: + default_server_args.append(request.param) + + model_name = MODELS[model_key] + with RemoteOpenAIServer(model_name, default_server_args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as cl: + yield cl + + +_PROMPT = "Hello my name is Robert and I love magic" + + +def _get_expected_values(num_requests: int, prompt_ids: list[int], max_tokens: int): + num_prompt_tokens = len(prompt_ids) + + # {metric_family: [(suffix, expected_value)]} + return { + "vllm:time_to_first_token_seconds": [("_count", num_requests)], + "vllm:inter_token_latency_seconds": [ + ("_count", num_requests * (max_tokens - 1)) + ], + "vllm:e2e_request_latency_seconds": [("_count", num_requests)], + "vllm:request_queue_time_seconds": [("_count", num_requests)], + "vllm:request_inference_time_seconds": [("_count", num_requests)], + "vllm:request_prefill_time_seconds": [("_count", num_requests)], + "vllm:request_decode_time_seconds": [("_count", num_requests)], + "vllm:request_prompt_tokens": [ + ("_sum", num_requests * num_prompt_tokens), + ("_count", num_requests), + ], + "vllm:request_generation_tokens": [ + ("_sum", num_requests * max_tokens), + ("_count", num_requests), + ], + "vllm:request_params_n": [("_count", num_requests)], + "vllm:request_params_max_tokens": [ + ("_sum", num_requests * max_tokens), + ("_count", num_requests), + ], + "vllm:iteration_tokens_total": [ + ( + "_sum", + num_requests * (num_prompt_tokens + max_tokens), + ), + ("_count", num_requests * max_tokens), + ], + "vllm:prompt_tokens": [("_total", num_requests * num_prompt_tokens)], + "vllm:generation_tokens": [("_total", num_requests * max_tokens)], + "vllm:request_success": [("_total", num_requests)], + } + + +@pytest.mark.asyncio +async def test_metrics_counts( + server: RemoteOpenAIServer, + client: openai.AsyncClient, + model_key: str, +): + if model_key == "multimodal": + pytest.skip("Unnecessary test") + + model_name = MODELS[model_key] + tokenizer = AutoTokenizer.from_pretrained(model_name) + prompt_ids = tokenizer.encode(_PROMPT) + num_requests = 10 + max_tokens = 10 + + for _ in range(num_requests): + # sending a request triggers the metrics to be logged. + await client.completions.create( + model=model_name, + prompt=prompt_ids, + max_tokens=max_tokens, + ) + + response = requests.get(server.url_for("metrics")) + print(response.text) + assert response.status_code == HTTPStatus.OK + + # Loop over all expected metric_families + expected_values = _get_expected_values(num_requests, prompt_ids, max_tokens) + for metric_family, suffix_values_list in expected_values.items(): + if metric_family not in EXPECTED_METRICS_V1 or ( + not server.show_hidden_metrics + and metric_family in HIDDEN_DEPRECATED_METRICS + ): + continue + + found_metric = False + + # Check to see if the metric_family is found in the prom endpoint. + for family in text_string_to_metric_families(response.text): + if family.name == metric_family: + found_metric = True + + # Check that each suffix is found in the prom endpoint. + for suffix, expected_value in suffix_values_list: + metric_name_w_suffix = f"{metric_family}{suffix}" + found_suffix = False + + for sample in family.samples: + if sample.name == metric_name_w_suffix: + found_suffix = True + + # For each suffix, value sure the value matches + # what we expect. + assert sample.value == expected_value, ( + f"{metric_name_w_suffix} expected value of " + f"{expected_value} did not match found value " + f"{sample.value}" + ) + break + assert found_suffix, ( + f"Did not find {metric_name_w_suffix} in prom endpoint" + ) + break + + assert found_metric, f"Did not find {metric_family} in prom endpoint" + + +EXPECTED_METRICS_V1 = [ + "vllm:num_requests_running", + "vllm:num_requests_waiting", + "vllm:num_requests_waiting_by_reason", + "vllm:kv_cache_usage_perc", + "vllm:prefix_cache_queries", + "vllm:prefix_cache_hits", + "vllm:num_preemptions_total", + "vllm:prompt_tokens_total", + "vllm:generation_tokens_total", + "vllm:iteration_tokens_total", + "vllm:cache_config_info", + "vllm:request_success_total", + "vllm:request_prompt_tokens_sum", + "vllm:request_prompt_tokens_bucket", + "vllm:request_prompt_tokens_count", + "vllm:request_generation_tokens_sum", + "vllm:request_generation_tokens_bucket", + "vllm:request_generation_tokens_count", + "vllm:request_params_n_sum", + "vllm:request_params_n_bucket", + "vllm:request_params_n_count", + "vllm:request_params_max_tokens_sum", + "vllm:request_params_max_tokens_bucket", + "vllm:request_params_max_tokens_count", + "vllm:time_to_first_token_seconds_sum", + "vllm:time_to_first_token_seconds_bucket", + "vllm:time_to_first_token_seconds_count", + "vllm:inter_token_latency_seconds_sum", + "vllm:inter_token_latency_seconds_bucket", + "vllm:inter_token_latency_seconds_count", + "vllm:e2e_request_latency_seconds_sum", + "vllm:e2e_request_latency_seconds_bucket", + "vllm:e2e_request_latency_seconds_count", + "vllm:request_queue_time_seconds_sum", + "vllm:request_queue_time_seconds_bucket", + "vllm:request_queue_time_seconds_count", + "vllm:request_inference_time_seconds_sum", + "vllm:request_inference_time_seconds_bucket", + "vllm:request_inference_time_seconds_count", + "vllm:request_prefill_time_seconds_sum", + "vllm:request_prefill_time_seconds_bucket", + "vllm:request_prefill_time_seconds_count", + "vllm:request_decode_time_seconds_sum", + "vllm:request_decode_time_seconds_bucket", + "vllm:request_decode_time_seconds_count", +] + +EXPECTED_METRICS_MM = [ + "vllm:mm_cache_queries", + "vllm:mm_cache_hits", +] + +HIDDEN_DEPRECATED_METRICS: list[str] = [] + + +@pytest.mark.asyncio +async def test_metrics_exist( + local_asset_server: LocalAssetServer, + server: RemoteOpenAIServer, + client: openai.AsyncClient, + model_key: str, +): + model_name = MODELS[model_key] + + # sending a request triggers the metrics to be logged. + if model_key == "text": + await client.completions.create( + model=model_name, + prompt="Hello, my name is", + max_tokens=5, + temperature=0.0, + ) + else: + # https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg + await client.chat.completions.create( + model=model_name, + messages=[ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": { + "url": local_asset_server.url_for( + "2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" + ), + }, + }, + {"type": "text", "text": "What's in this image?"}, + ], + } + ], + max_tokens=5, + temperature=0.0, + ) + + response = requests.get(server.url_for("metrics")) + assert response.status_code == HTTPStatus.OK + + expected_metrics = EXPECTED_METRICS_V1 + if model_key == "multimodal": + # NOTE: Don't use in-place assignment + expected_metrics = expected_metrics + EXPECTED_METRICS_MM + + for metric in expected_metrics: + if metric in HIDDEN_DEPRECATED_METRICS and not server.show_hidden_metrics: + continue + assert metric in response.text + + cache_config_samples = [ + sample + for family in text_string_to_metric_families(response.text) + if family.name == "vllm:cache_config_info" + for sample in family.samples + ] + assert cache_config_samples + for sample in cache_config_samples: + assert sample.labels.get("kv_cache_size_tokens") not in (None, "None", "") + assert sample.labels.get("kv_cache_max_concurrency") not in (None, "None", "") + + +@pytest.mark.asyncio +async def test_abort_metrics_reset( + server: RemoteOpenAIServer, + client: openai.AsyncClient, + model_key: str, +): + model_name = MODELS[model_key] + tokenizer = AutoTokenizer.from_pretrained(model_name) + prompt_ids = tokenizer.encode(_PROMPT) + + running_requests, waiting_requests, kv_cache_usage = _get_running_metrics_from_api( + server, + ) + + # Expect no running requests or kvcache usage + assert running_requests == 0 + assert waiting_requests == 0 + assert kv_cache_usage == 0.0 + + # Start some long-running requests that we can abort + tasks = [] + for _ in range(3): + task = asyncio.create_task( + client.completions.create( + model=model_name, + prompt=prompt_ids, + max_tokens=500, # Long generation to give time to abort + temperature=0.0, + ) + ) + tasks.append(task) + + # Poll until we see running requests rather than using a fixed sleep, + # since generation speed varies across hardware. + try: + await _poll_until( + lambda: _get_running_metrics_from_api(server)[0] > 0, + timeout=10.0, + interval=0.1, + description="running_requests > 0", + ) + except TimeoutError: + for task in tasks: + task.cancel() + await asyncio.gather(*tasks, return_exceptions=True) + pytest.fail("Requests never appeared as running in metrics") + + # Check that we have running requests + running_requests, waiting_requests, kv_cache_usage = _get_running_metrics_from_api( + server, + ) + + # Expect running requests and kvcache usage + assert running_requests > 0 + assert kv_cache_usage > 0 + + # Cancel all tasks to abort the requests + for task in tasks: + task.cancel() + await asyncio.gather(*tasks, return_exceptions=True) + + # Poll until metrics reset rather than using a fixed sleep + await _poll_until( + lambda: _get_running_metrics_from_api(server) == (0, 0, 0), + timeout=10.0, + interval=0.2, + description="gauge metrics back to zero", + ) + + # Verify running and waiting requests counts and KV cache usage are zero + running_requests_after, waiting_requests_after, kv_cache_usage_after = ( + _get_running_metrics_from_api(server) + ) + + assert running_requests_after == 0, ( + f"Expected 0 running requests after abort, got {running_requests_after}" + ) + assert waiting_requests_after == 0, ( + f"Expected 0 waiting requests after abort, got {waiting_requests_after}" + ) + assert kv_cache_usage_after == 0, ( + f"Expected 0% KV cache usage after abort, got {kv_cache_usage_after}" + ) + + +async def _poll_until( + predicate, *, timeout: float, interval: float = 0.5, description: str = "condition" +): + """Poll until predicate() returns True, or raise TimeoutError.""" + start = time.time() + while time.time() - start < timeout: + if predicate(): + return + await asyncio.sleep(interval) + raise TimeoutError(f"Timed out after {timeout}s waiting for: {description}") + + +def _get_running_metrics_from_api(server: RemoteOpenAIServer): + """Return (running_count, waiting_count, kv_cache_usage)""" + + response = requests.get(server.url_for("metrics")) + assert response.status_code == HTTPStatus.OK + + # Verify running and waiting requests counts and KV cache usage are zero + running_requests, waiting_requests, kv_cache_usage = None, None, None + + kv_cache_usage_metric = "vllm:kv_cache_usage_perc" + + for family in text_string_to_metric_families(response.text): + if family.name == "vllm:num_requests_running": + for sample in family.samples: + if sample.name == "vllm:num_requests_running": + running_requests = sample.value + break + elif family.name == "vllm:num_requests_waiting": + for sample in family.samples: + if sample.name == "vllm:num_requests_waiting": + waiting_requests = sample.value + break + elif family.name == kv_cache_usage_metric: + for sample in family.samples: + if sample.name == kv_cache_usage_metric: + kv_cache_usage = sample.value + break + + assert running_requests is not None + assert waiting_requests is not None + assert kv_cache_usage is not None + + return running_requests, waiting_requests, kv_cache_usage + + +def test_metrics_exist_run_batch(): + input_batch = """{"custom_id": "request-0", "method": "POST", "url": "/v1/embeddings", "body": {"model": "intfloat/multilingual-e5-small", "input": "You are a helpful assistant."}}""" # noqa: E501 + + base_url = "0.0.0.0" + port = str(get_open_port()) + server_url = f"http://{base_url}:{port}" + + with ( + tempfile.NamedTemporaryFile("w") as input_file, + tempfile.NamedTemporaryFile("r") as output_file, + ): + input_file.write(input_batch) + input_file.flush() + proc = subprocess.Popen( + [ + sys.executable, + "-m", + "vllm.entrypoints.openai.run_batch", + "-i", + input_file.name, + "-o", + output_file.name, + "--model", + "intfloat/multilingual-e5-small", + "--enable-metrics", + "--host", + base_url, + "--port", + port, + ], + ) + + try: + + def is_server_up(url): + try: + response = requests.get(url) + return response.status_code == 200 + except requests.ConnectionError: + return False + + start = time.time() + timeout = 120 + while not is_server_up(server_url): + if proc.poll() is not None: + pytest.fail( + f"Batch process exited early with returncode={proc.returncode}" + ) + if time.time() - start > timeout: + pytest.fail("Batch server did not start within timeout") + time.sleep(1) + + response = requests.get(server_url + "/metrics") + assert response.status_code == HTTPStatus.OK + finally: + proc.terminate() + try: + proc.wait(timeout=15) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait(timeout=5) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/instrumentator/test_optional_middleware.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/instrumentator/test_optional_middleware.py new file mode 100644 index 0000000000000000000000000000000000000000..58e4ce667310ed43ca772679e44d2f7708df2b36 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/instrumentator/test_optional_middleware.py @@ -0,0 +1,162 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Tests for middleware that's off by default and can be toggled through +server arguments, mainly --api-key and --enable-request-id-headers. +""" + +from http import HTTPStatus + +import pytest +import requests + +from tests.utils import RemoteOpenAIServer + +# Use a small embeddings model for faster startup and smaller memory footprint. +# Since we are not testing any chat functionality, +# using a chat capable model is overkill. +MODEL_NAME = "intfloat/multilingual-e5-small" + + +@pytest.fixture(scope="module") +def server(request: pytest.FixtureRequest): + passed_params = [] + if hasattr(request, "param"): + passed_params = request.param + if isinstance(passed_params, str): + passed_params = [passed_params] + + args = [ + "--runner", + "pooling", + # use half precision for speed and memory savings in CI environment + "--dtype", + "float16", + "--max-model-len", + "512", + "--enforce-eager", + "--max-num-seqs", + "2", + *passed_params, + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.mark.asyncio +async def test_no_api_token(server: RemoteOpenAIServer): + response = requests.get(server.url_for("v1/models")) + assert response.status_code == HTTPStatus.OK + + +@pytest.mark.asyncio +async def test_no_request_id_header(server: RemoteOpenAIServer): + response = requests.get(server.url_for("health")) + assert "X-Request-Id" not in response.headers + + +@pytest.mark.parametrize( + "server", + [["--api-key", "test"]], + indirect=True, +) +@pytest.mark.asyncio +async def test_missing_api_token(server: RemoteOpenAIServer): + response = requests.get(server.url_for("v1/models")) + assert response.status_code == HTTPStatus.UNAUTHORIZED + + +@pytest.mark.parametrize( + "server", + [["--api-key", "test"]], + indirect=True, +) +@pytest.mark.asyncio +async def test_passed_api_token(server: RemoteOpenAIServer): + response = requests.get( + server.url_for("v1/models"), headers={"Authorization": "Bearer test"} + ) + assert response.status_code == HTTPStatus.OK + + +@pytest.mark.parametrize( + "server", + [["--api-key", "test"]], + indirect=True, +) +@pytest.mark.asyncio +async def test_not_v1_or_v2_path_skips_auth(server: RemoteOpenAIServer): + # Authorization check is skipped for paths that + # don't start with /v1 or /v2 (e.g. /health, /metrics). + response = requests.get(server.url_for("health")) + assert response.status_code == HTTPStatus.OK + + +# --------------------------------------------------------------------------- +# /v2 path authentication tests +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "server", + [["--api-key", "test"]], + indirect=True, +) +@pytest.mark.asyncio +async def test_v2_endpoint_rejects_missing_api_token(server: RemoteOpenAIServer): + # /v2/embed should require authentication when --api-key is set. + body = { + "model": MODEL_NAME, + "texts": ["hello"], + "embedding_types": ["float"], + } + response = requests.post(server.url_for("/v2/embed"), json=body) + assert response.status_code == HTTPStatus.UNAUTHORIZED + + +@pytest.mark.parametrize( + "server", + [["--api-key", "test"]], + indirect=True, +) +@pytest.mark.asyncio +async def test_v2_endpoint_accepts_valid_api_token(server: RemoteOpenAIServer): + # /v2/embed should accept requests with a valid API key. + body = { + "model": MODEL_NAME, + "texts": ["hello"], + "embedding_types": ["float"], + } + response = requests.post( + server.url_for("/v2/embed"), + json=body, + headers={"Authorization": "Bearer test"}, + ) + assert response.status_code == HTTPStatus.OK + + +@pytest.mark.parametrize( + "server", + ["--enable-request-id-headers"], + indirect=True, +) +@pytest.mark.asyncio +async def test_enable_request_id_header(server: RemoteOpenAIServer): + response = requests.get(server.url_for("health")) + assert "X-Request-Id" in response.headers + assert len(response.headers.get("X-Request-Id", "")) == 32 + + +@pytest.mark.parametrize( + "server", + ["--enable-request-id-headers"], + indirect=True, +) +@pytest.mark.asyncio +async def test_custom_request_id_header(server: RemoteOpenAIServer): + response = requests.get( + server.url_for("health"), headers={"X-Request-Id": "Custom"} + ) + assert "X-Request-Id" in response.headers + assert response.headers.get("X-Request-Id") == "Custom" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/instrumentator/test_orca_metrics.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/instrumentator/test_orca_metrics.py new file mode 100644 index 0000000000000000000000000000000000000000..8a7b3b001bf0b4c63c06bbbda1bfb9f989bc25b1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/instrumentator/test_orca_metrics.py @@ -0,0 +1,126 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import openai +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer + +# any model with a chat template should work here +MODEL_NAME = "Qwen/Qwen3-0.6B" + + +@pytest.fixture(scope="module") +def monkeypatch_module(): + from _pytest.monkeypatch import MonkeyPatch + + mpatch = MonkeyPatch() + yield mpatch + mpatch.undo() + + +@pytest.fixture(scope="module", params=[True]) +def server(request, monkeypatch_module): + args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "8192", + "--enforce-eager", + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +async def test_chat_completion_with_orca_header(server: RemoteOpenAIServer): + messages = [ + {"role": "system", "content": "you are a helpful assistant"}, + {"role": "user", "content": "what is 1+1?"}, + ] + + client = openai.OpenAI( + api_key="EMPTY", + base_url=f"http://localhost:{server.port}/v1", + default_headers={"endpoint-load-metrics-format": "TEXT"}, + ) + + # 1. Use raw client to get response headers. + raw_client = client.with_raw_response + + # 2. Make the API call using the raw_client + response_with_raw = raw_client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + extra_headers={"endpoint-load-metrics-format": "TEXT"}, + ) + + # 3. Access the raw httpx.Response object + raw_http_response = response_with_raw.http_response + + # 4. Get the headers from the httpx.Response object + response_headers = raw_http_response.headers + + assert "endpoint-load-metrics" in response_headers + + +@pytest.mark.asyncio +async def test_completion_with_orca_header(client: openai.AsyncOpenAI): + # 1. Use raw client to get response headers. + raw_client = client.with_raw_response + + # 2. Make the API call using the raw_client + completion = await raw_client.completions.create( + model=MODEL_NAME, + prompt="Hello, my name is", + max_tokens=5, + extra_headers={"endpoint-load-metrics-format": "JSON"}, + ) + + # 3. Access the raw httpx.Response object + raw_http_response = completion.http_response + + # 4. Get the headers from the httpx.Response object + response_headers = raw_http_response.headers + + assert "endpoint-load-metrics" in response_headers + + +@pytest.mark.asyncio +async def test_single_completion(client: openai.AsyncOpenAI): + completion = await client.completions.create( + model=MODEL_NAME, + prompt="Hello, my name is", + max_tokens=5, + extra_headers={"endpoint-load-metrics-format": "JSON"}, + temperature=0.0, + ) + + assert completion.id is not None + assert completion.choices is not None and len(completion.choices) == 1 + + choice = completion.choices[0] + assert len(choice.text) >= 5 + assert choice.finish_reason == "length" + # When using Qwen3-0.6B, prompt tokens=[9707, 11, 847, 829, 374] + assert completion.usage == openai.types.CompletionUsage( + completion_tokens=5, prompt_tokens=5, total_tokens=10 + ) + + # test using token IDs + completion = await client.completions.create( + model=MODEL_NAME, + prompt=[0, 0, 0, 0, 0], + max_tokens=5, + temperature=0.0, + ) + assert len(completion.choices[0].text) >= 1 + assert completion.choices[0].prompt_logprobs is None diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/lora/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/lora/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/lora/test_lora_adapters.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/lora/test_lora_adapters.py new file mode 100644 index 0000000000000000000000000000000000000000..1cee90cbd202209a648cd9c4b376b14295cccc69 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/lora/test_lora_adapters.py @@ -0,0 +1,370 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import asyncio +import json +import shutil +from contextlib import suppress + +import openai # use the official client for correctness check +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer + +# any model with a chat template should work here +MODEL_NAME = "Qwen/Qwen3-0.6B" + + +BADREQUEST_CASES = [ + ( + "test_rank", + {"r": 1024}, + "is greater than max_lora_rank", + ), + ("test_dora", {"use_dora": True}, "does not yet support DoRA"), + ( + "test_modules_to_save", + {"modules_to_save": ["lm_head"]}, + "only supports modules_to_save being None", + ), +] + + +@pytest.fixture(scope="module", params=[True]) +def server_with_lora_modules_json(request, qwen3_lora_files): + # Define the json format LoRA module configurations + lora_module_1 = { + "name": "qwen3-lora", + "path": qwen3_lora_files, + "base_model_name": MODEL_NAME, + } + + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-model-len", + "8192", + "--enforce-eager", + # lora config below + "--enable-lora", + "--lora-modules", + json.dumps(lora_module_1), + "--max-lora-rank", + "64", + "--max-cpu-loras", + "2", + "--max-num-seqs", + "64", + ] + + # Enable the /v1/load_lora_adapter endpoint + envs = {"VLLM_ALLOW_RUNTIME_LORA_UPDATING": "True"} + + with RemoteOpenAIServer(MODEL_NAME, args, env_dict=envs) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server_with_lora_modules_json): + async with server_with_lora_modules_json.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +async def test_static_lora_lineage(client: openai.AsyncOpenAI, qwen3_lora_files): + models = await client.models.list() + models = models.data + served_model = models[0] + lora_models = models[1:] + assert served_model.id == MODEL_NAME + assert served_model.root == MODEL_NAME + assert served_model.parent is None + assert all(lora_model.root == qwen3_lora_files for lora_model in lora_models) + assert all(lora_model.parent == MODEL_NAME for lora_model in lora_models) + assert lora_models[0].id == "qwen3-lora" + + +@pytest.mark.asyncio +async def test_dynamic_lora_lineage(client: openai.AsyncOpenAI, qwen3_lora_files): + response = await client.post( + "load_lora_adapter", + cast_to=str, + body={"lora_name": "qwen3-lora-3", "lora_path": qwen3_lora_files}, + ) + # Ensure adapter loads before querying /models + assert "success" in response + + models = await client.models.list() + models = models.data + dynamic_lora_model = models[-1] + assert dynamic_lora_model.root == qwen3_lora_files + assert dynamic_lora_model.parent == MODEL_NAME + assert dynamic_lora_model.id == "qwen3-lora-3" + + +@pytest.mark.asyncio +async def test_load_lora_adapter_with_same_name_replaces_inplace( + client: openai.AsyncOpenAI, qwen3_meowing_lora_files, qwen3_woofing_lora_files +): + """Test that loading a LoRA adapter with the same name replaces it inplace.""" + adapter_name = "replaceable-adapter" + messages = [ + {"content": "Follow the instructions to make animal noises", "role": "system"}, + {"content": "Make your favorite animal noise.", "role": "user"}, + ] + + # Load LoRA that makes model meow + response = await client.post( + "load_lora_adapter", + cast_to=str, + body={"lora_name": adapter_name, "lora_path": qwen3_meowing_lora_files}, + ) + assert "success" in response.lower() + + completion = await client.chat.completions.create( + model=adapter_name, + messages=messages, + max_tokens=10, + ) + assert "Meow Meow Meow" in completion.choices[0].message.content + + # Load LoRA that makes model woof + response = await client.post( + "load_lora_adapter", + cast_to=str, + body={ + "lora_name": adapter_name, + "lora_path": qwen3_woofing_lora_files, + "load_inplace": True, + }, + ) + assert "success" in response.lower() + + completion = await client.chat.completions.create( + model=adapter_name, + messages=messages, + max_tokens=10, + ) + assert "Woof Woof Woof" in completion.choices[0].message.content + + +@pytest.mark.asyncio +async def test_load_lora_adapter_with_load_inplace_false_errors( + client: openai.AsyncOpenAI, qwen3_meowing_lora_files +): + """Test that load_inplace=False returns an error when adapter already exists.""" + adapter_name = "test-load-inplace-false" + + # Load LoRA adapter first time (should succeed) + response = await client.post( + "load_lora_adapter", + cast_to=str, + body={"lora_name": adapter_name, "lora_path": qwen3_meowing_lora_files}, + ) + assert "success" in response.lower() + + # Try to load the same adapter again with load_inplace=False (should fail) + with pytest.raises(openai.BadRequestError) as exc_info: + await client.post( + "load_lora_adapter", + cast_to=str, + body={ + "lora_name": adapter_name, + "lora_path": qwen3_meowing_lora_files, + }, + ) + + # Verify the error message + assert "already been loaded" in str(exc_info.value) + + +@pytest.mark.asyncio +async def test_dynamic_lora_not_found(client: openai.AsyncOpenAI): + with pytest.raises(openai.NotFoundError): + await client.post( + "load_lora_adapter", + cast_to=str, + body={"lora_name": "notfound", "lora_path": "/not/an/adapter"}, + ) + + +@pytest.mark.asyncio +async def test_dynamic_lora_invalid_files(client: openai.AsyncOpenAI, tmp_path): + invalid_files = tmp_path / "invalid_files" + invalid_files.mkdir() + (invalid_files / "adapter_config.json").write_text("this is not json") + + with pytest.raises(openai.InternalServerError): + await client.post( + "load_lora_adapter", + cast_to=str, + body={"lora_name": "invalid-json", "lora_path": str(invalid_files)}, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("test_name,config_change,expected_error", BADREQUEST_CASES) +async def test_dynamic_lora_badrequests( + client: openai.AsyncOpenAI, + tmp_path, + qwen3_lora_files, + test_name: str, + config_change: dict, + expected_error: str, +): + # Create test directory + test_dir = tmp_path / test_name + + # Copy adapter files + shutil.copytree(qwen3_lora_files, test_dir) + + # Load and modify configuration + config_path = test_dir / "adapter_config.json" + with open(config_path) as f: + adapter_config = json.load(f) + # Apply configuration changes + adapter_config.update(config_change) + + # Save modified configuration + with open(config_path, "w") as f: + json.dump(adapter_config, f) + + # Test loading the adapter + with pytest.raises(openai.InternalServerError, match=expected_error): + await client.post( + "load_lora_adapter", + cast_to=str, + body={"lora_name": test_name, "lora_path": str(test_dir)}, + ) + + +@pytest.mark.asyncio +async def test_multiple_lora_adapters( + client: openai.AsyncOpenAI, tmp_path, qwen3_lora_files +): + """Validate that many loras can be dynamically registered and inferenced + with concurrently""" + + # This test file configures the server with --max-cpu-loras=2 and this test + # will concurrently load 10 adapters, so it should flex the LRU cache + async def load_and_run_adapter(adapter_name: str): + await client.post( + "load_lora_adapter", + cast_to=str, + body={"lora_name": adapter_name, "lora_path": str(qwen3_lora_files)}, + ) + for _ in range(3): + await client.completions.create( + model=adapter_name, + prompt=["Hello there", "Foo bar bazz buzz"], + max_tokens=5, + ) + + lora_tasks = [] + for i in range(10): + lora_tasks.append(asyncio.create_task(load_and_run_adapter(f"adapter_{i}"))) + + results, _ = await asyncio.wait(lora_tasks) + + for r in results: + assert not isinstance(r, Exception), f"Got exception {r}" + + +@pytest.mark.asyncio +async def test_loading_invalid_adapters_does_not_break_others( + client: openai.AsyncOpenAI, tmp_path, qwen3_lora_files +): + invalid_files = tmp_path / "invalid_files" + invalid_files.mkdir() + (invalid_files / "adapter_config.json").write_text("this is not json") + + stop_good_requests_event = asyncio.Event() + + async def run_good_requests(client): + # Run chat completions requests until event set + + results = [] + + while not stop_good_requests_event.is_set(): + try: + batch = await client.completions.create( + model="qwen3-lora", + prompt=["Hello there", "Foo bar bazz buzz"], + max_tokens=5, + ) + results.append(batch) + except Exception as e: + results.append(e) + + return results + + # Create task to run good requests + good_task = asyncio.create_task(run_good_requests(client)) + + # Run a bunch of bad adapter loads + for _ in range(25): + with suppress(openai.NotFoundError): + await client.post( + "load_lora_adapter", + cast_to=str, + body={"lora_name": "notfound", "lora_path": "/not/an/adapter"}, + ) + for _ in range(25): + with suppress(openai.InternalServerError): + await client.post( + "load_lora_adapter", + cast_to=str, + body={"lora_name": "invalid", "lora_path": str(invalid_files)}, + ) + + # Ensure all the running requests with lora adapters succeeded + stop_good_requests_event.set() + results = await good_task + for r in results: + assert not isinstance(r, Exception), f"Got exception {r}" + + # Ensure we can load another adapter and run it + await client.post( + "load_lora_adapter", + cast_to=str, + body={"lora_name": "valid", "lora_path": qwen3_lora_files}, + ) + await client.completions.create( + model="valid", + prompt=["Hello there", "Foo bar bazz buzz"], + max_tokens=5, + ) + + +@pytest.mark.asyncio +async def test_beam_search_with_lora_adapters( + client: openai.AsyncOpenAI, + tmp_path, + qwen3_lora_files, +): + """Validate that async beam search can be used with lora.""" + + async def load_and_run_adapter(adapter_name: str): + await client.post( + "load_lora_adapter", + cast_to=str, + body={"lora_name": adapter_name, "lora_path": str(qwen3_lora_files)}, + ) + for _ in range(3): + await client.completions.create( + model=adapter_name, + prompt=["Hello there", "Foo bar bazz buzz"], + max_tokens=5, + extra_body=dict(use_beam_search=True), + ) + + lora_tasks = [] + for i in range(3): + lora_tasks.append(asyncio.create_task(load_and_run_adapter(f"adapter_{i}"))) + + results, _ = await asyncio.wait(lora_tasks) + + for r in results: + assert not isinstance(r, Exception), f"Got exception {r}" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/lora/test_serving_models.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/lora/test_serving_models.py new file mode 100644 index 0000000000000000000000000000000000000000..29ca6a69b4372fba9c35aa188f919f333c27ceae --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/lora/test_serving_models.py @@ -0,0 +1,193 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from http import HTTPStatus +from unittest.mock import MagicMock + +import pytest + +from vllm import PoolingParams +from vllm.config import ModelConfig +from vllm.engine.protocol import EngineClient +from vllm.entrypoints.openai.engine.protocol import ( + ErrorResponse, +) +from vllm.entrypoints.openai.models.protocol import BaseModelPath +from vllm.entrypoints.openai.models.serving import OpenAIServingModels +from vllm.entrypoints.pooling.base.serving import PoolingBaseServing +from vllm.entrypoints.pooling.typing import PoolingServeContext +from vllm.entrypoints.serve.lora.protocol import ( + LoadLoRAAdapterRequest, + UnloadLoRAAdapterRequest, +) +from vllm.exceptions import VLLMNotFoundError +from vllm.lora.request import LoRARequest + +MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM" +BASE_MODEL_PATHS = [BaseModelPath(name=MODEL_NAME, model_path=MODEL_NAME)] +LORA_LOADING_SUCCESS_MESSAGE = "Success: LoRA adapter '{lora_name}' added successfully." +LORA_UNLOADING_SUCCESS_MESSAGE = ( + "Success: LoRA adapter '{lora_name}' removed successfully." +) + + +async def _async_serving_models_init() -> OpenAIServingModels: + mock_engine_client = MagicMock(spec=EngineClient) + # Set the max_model_len attribute to avoid missing attribute + mock_model_config = MagicMock(spec=ModelConfig) + mock_model_config.max_model_len = 2048 + mock_engine_client.model_config = mock_model_config + mock_engine_client.input_processor = MagicMock() + mock_engine_client.renderer = MagicMock() + + serving_models = OpenAIServingModels( + engine_client=mock_engine_client, + base_model_paths=BASE_MODEL_PATHS, + lora_modules=None, + ) + await serving_models.init_static_loras() + + return serving_models + + +@pytest.mark.asyncio +async def test_serving_model_name(): + serving_models = await _async_serving_models_init() + assert serving_models.model_name(None) == MODEL_NAME + request = LoRARequest( + lora_name="adapter", lora_path="/path/to/adapter2", lora_int_id=1 + ) + assert serving_models.model_name(request) == request.lora_name + + +@pytest.mark.asyncio +async def test_load_lora_adapter_success(): + serving_models = await _async_serving_models_init() + request = LoadLoRAAdapterRequest(lora_name="adapter", lora_path="/path/to/adapter2") + response = await serving_models.load_lora_adapter(request) + assert response == LORA_LOADING_SUCCESS_MESSAGE.format(lora_name="adapter") + assert len(serving_models.lora_requests) == 1 + assert "adapter" in serving_models.lora_requests + assert serving_models.lora_requests["adapter"].lora_name == "adapter" + + +@pytest.mark.asyncio +async def test_load_lora_adapter_missing_fields(): + serving_models = await _async_serving_models_init() + request = LoadLoRAAdapterRequest(lora_name="", lora_path="") + response = await serving_models.load_lora_adapter(request) + assert isinstance(response, ErrorResponse) + assert response.error.type == "InvalidUserInput" + assert response.error.code == HTTPStatus.BAD_REQUEST + + +@pytest.mark.asyncio +async def test_load_lora_adapter_duplicate(): + serving_models = await _async_serving_models_init() + request = LoadLoRAAdapterRequest( + lora_name="adapter1", lora_path="/path/to/adapter1" + ) + response = await serving_models.load_lora_adapter(request) + assert response == LORA_LOADING_SUCCESS_MESSAGE.format(lora_name="adapter1") + assert len(serving_models.lora_requests) == 1 + + request = LoadLoRAAdapterRequest( + lora_name="adapter1", lora_path="/path/to/adapter1" + ) + response = await serving_models.load_lora_adapter(request) + assert isinstance(response, ErrorResponse) + assert response.error.type == "InvalidUserInput" + assert response.error.code == HTTPStatus.BAD_REQUEST + assert len(serving_models.lora_requests) == 1 + + +@pytest.mark.asyncio +async def test_unload_lora_adapter_success(): + serving_models = await _async_serving_models_init() + request = LoadLoRAAdapterRequest( + lora_name="adapter1", lora_path="/path/to/adapter1" + ) + response = await serving_models.load_lora_adapter(request) + assert len(serving_models.lora_requests) == 1 + + request = UnloadLoRAAdapterRequest(lora_name="adapter1") + response = await serving_models.unload_lora_adapter(request) + assert response == LORA_UNLOADING_SUCCESS_MESSAGE.format(lora_name="adapter1") + assert len(serving_models.lora_requests) == 0 + + +@pytest.mark.asyncio +async def test_unload_lora_adapter_missing_fields(): + serving_models = await _async_serving_models_init() + request = UnloadLoRAAdapterRequest(lora_name="", lora_int_id=None) + response = await serving_models.unload_lora_adapter(request) + assert isinstance(response, ErrorResponse) + assert response.error.type == "InvalidUserInput" + assert response.error.code == HTTPStatus.BAD_REQUEST + + +@pytest.mark.asyncio +async def test_unload_lora_adapter_not_found(): + serving_models = await _async_serving_models_init() + request = UnloadLoRAAdapterRequest(lora_name="nonexistent_adapter") + response = await serving_models.unload_lora_adapter(request) + assert isinstance(response, ErrorResponse) + assert response.error.type == "NotFoundError" + assert response.error.code == HTTPStatus.NOT_FOUND + + +class _ConcretePoolingServing(PoolingBaseServing): + """Minimal concrete subclass used only in these unit tests.""" + + request_id_prefix = "test" + + def get_io_processor(self, request): + raise NotImplementedError + + def _build_response(self, ctx): + raise NotImplementedError + + +def _make_pooling_serving(lora_name: str) -> _ConcretePoolingServing: + lora_request = LoRARequest( + lora_name=lora_name, lora_int_id=1, lora_path="/path/to/lora" + ) + mock_models = MagicMock() + mock_models.lora_requests = {lora_name: lora_request} + mock_models.is_base_model.side_effect = lambda name: name == MODEL_NAME + + serving = object.__new__(_ConcretePoolingServing) + serving.models = mock_models + return serving + + +def _make_pooling_ctx(model_name: str) -> PoolingServeContext: + mock_request = MagicMock() + mock_request.model = model_name + return PoolingServeContext( + request=mock_request, + model_name=MODEL_NAME, + request_id="test-id", + pooling_params=PoolingParams(), + ) + + +def test_pooling_maybe_get_adapters_lora_name_sets_lora_request(): + """LoRA adapter name must populate ctx.lora_request without raising.""" + lora_name = "bot-embed-lora" + serving = _make_pooling_serving(lora_name) + ctx = _make_pooling_ctx(lora_name) + + ctx.lora_request = serving._maybe_get_adapters(ctx.request) + + assert ctx.lora_request is not None + assert ctx.lora_request.lora_name == lora_name + + +def test_pooling_maybe_get_adapters_unknown_model_raises(): + """An unrecognised model name must still raise VLLMNotFoundError.""" + serving = _make_pooling_serving("some-lora") + ctx = _make_pooling_ctx("unknown-model") + + with pytest.raises(VLLMNotFoundError): + serving._maybe_get_adapters(ctx.request) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/conftest.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..9e296fb44f114c59cc10bdbbf5f4ca79c4cd70f8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/conftest.py @@ -0,0 +1,60 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Shared fixtures and utilities for SageMaker tests.""" + +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer + +# Model name constants used across tests +MODEL_NAME_SMOLLM = "HuggingFaceTB/SmolLM2-135M-Instruct" +LORA_ADAPTER_NAME_SMOLLM = "jekunz/smollm-135m-lora-fineweb-faroese" + +# SageMaker header constants +HEADER_SAGEMAKER_CLOSED_SESSION_ID = "X-Amzn-SageMaker-Closed-Session-Id" +HEADER_SAGEMAKER_SESSION_ID = "X-Amzn-SageMaker-Session-Id" +HEADER_SAGEMAKER_NEW_SESSION_ID = "X-Amzn-SageMaker-New-Session-Id" + + +@pytest.fixture(scope="session") +def smollm2_lora_files(): + """Download LoRA files once per test session.""" + from huggingface_hub import snapshot_download + + return snapshot_download(repo_id=LORA_ADAPTER_NAME_SMOLLM) + + +@pytest.fixture(scope="module") +def basic_server_with_lora(smollm2_lora_files): + """Basic server fixture with standard configuration.""" + args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "8192", + "--enforce-eager", + # lora config below + "--enable-lora", + "--max-lora-rank", + "256", + "--max-cpu-loras", + "2", + "--max-num-seqs", + "64", + ] + + envs = { + "VLLM_ALLOW_RUNTIME_LORA_UPDATING": "True", + "SAGEMAKER_ENABLE_STATEFUL_SESSIONS": "True", + } + with RemoteOpenAIServer(MODEL_NAME_SMOLLM, args, env_dict=envs) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def async_client(basic_server_with_lora: RemoteOpenAIServer): + """Async OpenAI client fixture for use with basic_server.""" + async with basic_server_with_lora.get_async_client() as async_client: + yield async_client diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/test_sagemaker_handler_overrides.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/test_sagemaker_handler_overrides.py new file mode 100644 index 0000000000000000000000000000000000000000..a8f10a6c3ff709fc880505a56caed251c57da349 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/test_sagemaker_handler_overrides.py @@ -0,0 +1,629 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Integration tests for handler override functionality. + +Tests real customer usage scenarios: +- Using @custom_ping_handler and @custom_invocation_handler decorators + to override handlers +- Setting environment variables for handler specifications +- Writing customer scripts with custom_sagemaker_ping_handler() and + custom_sagemaker_invocation_handler() functions +- Priority: env vars > decorators > customer script files > framework + defaults + +The handler-override scenarios exercise the real vLLM SageMaker router and +bootstrap path via an in-process FastAPI ``TestClient`` instead of launching a +model server. These scenarios fully replace the ``/ping`` and ``/invocations`` +endpoints with customer handlers, so no inference engine is required to +validate override behavior. Avoiding the model server also keeps the tests +fast and deterministic rather than depending on the FastAPI version resolved +into the test environment at runtime. +""" + +import os + +import pytest +import requests +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from tests.utils import RemoteOpenAIServer + +from .conftest import ( + MODEL_NAME_SMOLLM, +) + + +def _build_sagemaker_test_client() -> TestClient: + """Build a TestClient over the real SageMaker router and bootstrap path. + + ``attach_router`` is called with empty supported tasks because the override + tests replace the endpoints with customer handlers, so no framework + invocation handler (and therefore no engine) is exercised. + """ + from vllm.entrypoints.serve.sagemaker.api_router import ( + attach_router, + sagemaker_standards_bootstrap, + ) + + app = FastAPI() + attach_router(app, ()) + return TestClient(sagemaker_standards_bootstrap(app)) + + +class TestHandlerOverrideIntegration: + """Integration tests simulating real customer usage scenarios. + + Each test simulates a fresh server startup where customers: + - Use @custom_ping_handler and @custom_invocation_handler decorators + - Set environment variables (CUSTOM_FASTAPI_PING_HANDLER, etc.) + - Write customer scripts with custom_sagemaker_ping_handler() and + custom_sagemaker_invocation_handler() functions + """ + + def setup_method(self): + """Setup for each test - simulate fresh server startup.""" + self._clear_caches() + self._clear_env_vars() + + def teardown_method(self): + """Cleanup after each test.""" + self._clear_env_vars() + + def _clear_caches(self): + """Clear handler registry and function loader cache.""" + try: + from model_hosting_container_standards.common.handler import ( + handler_registry, + ) + from model_hosting_container_standards.sagemaker.sagemaker_loader import ( + SageMakerFunctionLoader, + ) + + handler_registry.clear() + SageMakerFunctionLoader._default_function_loader = None + except ImportError: + pytest.skip("model-hosting-container-standards not available") + + def _clear_env_vars(self): + """Clear SageMaker environment variables.""" + try: + from model_hosting_container_standards.common.fastapi.config import ( + FastAPIEnvVars, + ) + from model_hosting_container_standards.sagemaker.config import ( + SageMakerEnvVars, + ) + + # Clear SageMaker env vars + for var in [ + SageMakerEnvVars.SAGEMAKER_MODEL_PATH, + SageMakerEnvVars.CUSTOM_SCRIPT_FILENAME, + ]: + os.environ.pop(var, None) + + # Clear FastAPI env vars + for var in [ + FastAPIEnvVars.CUSTOM_FASTAPI_PING_HANDLER, + FastAPIEnvVars.CUSTOM_FASTAPI_INVOCATION_HANDLER, + ]: + os.environ.pop(var, None) + except ImportError: + pass + + def test_customer_script_functions_auto_loaded(self, monkeypatch, tmp_path): + """Test customer scenario: script functions automatically override + framework defaults.""" + try: + from model_hosting_container_standards.sagemaker.config import ( + SageMakerEnvVars, + ) + except ImportError: + pytest.skip("model-hosting-container-standards not available") + + # Customer writes a script file with ping() and invoke() functions + script_path = tmp_path / "model.py" + script_path.write_text( + """ +from fastapi import Request + +async def custom_sagemaker_ping_handler(): + return { + "status": "healthy", + "source": "customer_override", + "message": "Custom ping from customer script" + } + +async def custom_sagemaker_invocation_handler(request: Request): + return { + "predictions": ["Custom response from customer script"], + "source": "customer_override" + } +""" + ) + + # Customer sets SageMaker environment variables to point to their script + monkeypatch.setenv(SageMakerEnvVars.SAGEMAKER_MODEL_PATH, str(tmp_path)) + monkeypatch.setenv(SageMakerEnvVars.CUSTOM_SCRIPT_FILENAME, script_path.name) + + with _build_sagemaker_test_client() as client: + # Customer tests their server and sees their overrides work + # automatically + ping_response = client.get("/ping") + assert ping_response.status_code == 200 + ping_data = ping_response.json() + + invoke_response = client.post( + "/invocations", + json={ + "model": MODEL_NAME_SMOLLM, + "messages": [{"role": "user", "content": "Hello"}], + "max_tokens": 5, + }, + ) + assert invoke_response.status_code == 200 + invoke_data = invoke_response.json() + + # Customer sees their functions are used + assert ping_data["source"] == "customer_override" + assert ping_data["message"] == "Custom ping from customer script" + assert invoke_data["source"] == "customer_override" + assert invoke_data["predictions"] == [ + "Custom response from customer script" + ] + + def test_customer_decorator_usage(self, monkeypatch, tmp_path): + """Test customer scenario: using @custom_ping_handler and + @custom_invocation_handler decorators.""" + try: + from model_hosting_container_standards.sagemaker.config import ( + SageMakerEnvVars, + ) + except ImportError: + pytest.skip("model-hosting-container-standards not available") + + # Customer writes a script file with decorators + script_path = tmp_path / "model.py" + script_path.write_text( + """ +import model_hosting_container_standards.sagemaker as sagemaker_standards +from fastapi import Request + +@sagemaker_standards.custom_ping_handler +async def my_ping(): + return { + "type": "ping", + "source": "customer_decorator" + } + +@sagemaker_standards.custom_invocation_handler +async def my_invoke(request: Request): + return { + "type": "invoke", + "source": "customer_decorator" + } +""" + ) + + monkeypatch.setenv(SageMakerEnvVars.SAGEMAKER_MODEL_PATH, str(tmp_path)) + monkeypatch.setenv(SageMakerEnvVars.CUSTOM_SCRIPT_FILENAME, script_path.name) + + with _build_sagemaker_test_client() as client: + ping_response = client.get("/ping") + assert ping_response.status_code == 200 + ping_data = ping_response.json() + + invoke_response = client.post( + "/invocations", + json={ + "model": MODEL_NAME_SMOLLM, + "messages": [{"role": "user", "content": "Hello"}], + "max_tokens": 5, + }, + ) + assert invoke_response.status_code == 200 + invoke_data = invoke_response.json() + + # Customer sees their handlers are used by the server + assert ping_data["source"] == "customer_decorator" + assert invoke_data["source"] == "customer_decorator" + + def test_handler_priority_order(self, monkeypatch, tmp_path): + """Test priority: @custom_ping_handler/@custom_invocation_handler + decorators vs script functions.""" + try: + from model_hosting_container_standards.sagemaker.config import ( + SageMakerEnvVars, + ) + except ImportError: + pytest.skip("model-hosting-container-standards not available") + + # Customer writes a script with both decorator and regular functions + script_path = tmp_path / "model.py" + script_path.write_text( + """ +import model_hosting_container_standards.sagemaker as sagemaker_standards +from fastapi import Request + +# Customer uses @custom_ping_handler decorator (higher priority than script functions) +@sagemaker_standards.custom_ping_handler +async def decorated_ping(): + return { + "status": "healthy", + "source": "ping_decorator_in_script", + "priority": "decorator" + } + +# Customer also has a regular function (lower priority than +# @custom_ping_handler decorator) +async def custom_sagemaker_ping_handler(): + return { + "status": "healthy", + "source": "script_function", + "priority": "function" + } + +# Customer has a regular invoke function +async def custom_sagemaker_invocation_handler(request: Request): + return { + "predictions": ["Script function response"], + "source": "script_invoke_function", + "priority": "function" + } +""" + ) + + monkeypatch.setenv(SageMakerEnvVars.SAGEMAKER_MODEL_PATH, str(tmp_path)) + monkeypatch.setenv(SageMakerEnvVars.CUSTOM_SCRIPT_FILENAME, script_path.name) + + with _build_sagemaker_test_client() as client: + ping_response = client.get("/ping") + assert ping_response.status_code == 200 + ping_data = ping_response.json() + + invoke_response = client.post( + "/invocations", + json={ + "model": MODEL_NAME_SMOLLM, + "messages": [{"role": "user", "content": "Hello"}], + "max_tokens": 5, + }, + ) + assert invoke_response.status_code == 200 + invoke_data = invoke_response.json() + + # @custom_ping_handler decorator has higher priority than + # script function + assert ping_data["source"] == "ping_decorator_in_script" + assert ping_data["priority"] == "decorator" + + # Script function is used for invoke + assert invoke_data["source"] == "script_invoke_function" + assert invoke_data["priority"] == "function" + + def test_environment_variable_script_loading(self, monkeypatch, tmp_path): + """Test that environment variables correctly specify script location + and loading.""" + try: + from model_hosting_container_standards.sagemaker.config import ( + SageMakerEnvVars, + ) + except ImportError: + pytest.skip("model-hosting-container-standards not available") + + # Customer writes a script in a specific directory + script_path = tmp_path / "model.py" + script_path.write_text( + """ +from fastapi import Request + +async def custom_sagemaker_ping_handler(): + return { + "status": "healthy", + "source": "env_loaded_script", + "method": "environment_variable_loading" + } + +async def custom_sagemaker_invocation_handler(request: Request): + return { + "predictions": ["Loaded via environment variables"], + "source": "env_loaded_script", + "method": "environment_variable_loading" + } +""" + ) + + # Test environment variable script loading + monkeypatch.setenv(SageMakerEnvVars.SAGEMAKER_MODEL_PATH, str(tmp_path)) + monkeypatch.setenv(SageMakerEnvVars.CUSTOM_SCRIPT_FILENAME, script_path.name) + + with _build_sagemaker_test_client() as client: + ping_response = client.get("/ping") + assert ping_response.status_code == 200 + ping_data = ping_response.json() + + invoke_response = client.post( + "/invocations", + json={ + "model": MODEL_NAME_SMOLLM, + "messages": [{"role": "user", "content": "Hello"}], + "max_tokens": 5, + }, + ) + assert invoke_response.status_code == 200 + invoke_data = invoke_response.json() + + # Verify that the script was loaded via environment variables + assert ping_data["source"] == "env_loaded_script" + assert ping_data["method"] == "environment_variable_loading" + assert invoke_data["source"] == "env_loaded_script" + assert invoke_data["method"] == "environment_variable_loading" + + @pytest.mark.asyncio + async def test_framework_default_handlers(self): + """Test that framework default handlers work when no customer + overrides exist. + + This scenario exercises the real inference path (default + ``/invocations``), so it keeps using a live model server rather than + the in-process TestClient. + """ + args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "2048", + "--enforce-eager", + "--max-num-seqs", + "32", + ] + + # Explicitly pass empty env_dict to ensure no SageMaker env vars are set + # This prevents pollution from previous tests + try: + from model_hosting_container_standards.common.fastapi.config import ( + FastAPIEnvVars, + ) + from model_hosting_container_standards.sagemaker.config import ( + SageMakerEnvVars, + ) + + env_dict = { + SageMakerEnvVars.SAGEMAKER_MODEL_PATH: "", + SageMakerEnvVars.CUSTOM_SCRIPT_FILENAME: "", + FastAPIEnvVars.CUSTOM_FASTAPI_PING_HANDLER: "", + FastAPIEnvVars.CUSTOM_FASTAPI_INVOCATION_HANDLER: "", + } + except ImportError: + env_dict = {} + + with RemoteOpenAIServer(MODEL_NAME_SMOLLM, args, env_dict=env_dict) as server: + # Test that default ping works + ping_response = requests.get(server.url_for("ping")) + assert ping_response.status_code == 200 + + # Test that default invocations work + invoke_response = requests.post( + server.url_for("invocations"), + json={ + "model": MODEL_NAME_SMOLLM, + "messages": [{"role": "user", "content": "Hello"}], + "max_tokens": 5, + }, + ) + assert invoke_response.status_code == 200 + + def test_handler_env_var_override(self, monkeypatch, tmp_path): + """Test CUSTOM_FASTAPI_PING_HANDLER and CUSTOM_FASTAPI_INVOCATION_HANDLER + environment variable overrides.""" + try: + from model_hosting_container_standards.common.fastapi.config import ( + FastAPIEnvVars, + ) + from model_hosting_container_standards.sagemaker.config import ( + SageMakerEnvVars, + ) + except ImportError: + pytest.skip("model-hosting-container-standards not available") + + # Create a script with both env var handlers and script functions + script_path = tmp_path / "model.py" + script_path.write_text( + """ +from fastapi import Request, Response +import json + +async def env_var_ping_handler(raw_request: Request) -> Response: + return Response( + content=json.dumps({ + "status": "healthy", + "source": "env_var_ping", + "method": "environment_variable" + }), + media_type="application/json" + ) + +async def env_var_invoke_handler(raw_request: Request) -> Response: + return Response( + content=json.dumps({ + "predictions": ["Environment variable response"], + "source": "env_var_invoke", + "method": "environment_variable" + }), + media_type="application/json" + ) + +async def custom_sagemaker_ping_handler(): + return { + "status": "healthy", + "source": "script_ping", + "method": "script_function" + } + +async def custom_sagemaker_invocation_handler(request: Request): + return { + "predictions": ["Script function response"], + "source": "script_invoke", + "method": "script_function" + } +""" + ) + + # Set environment variables to override both handlers + monkeypatch.setenv(SageMakerEnvVars.SAGEMAKER_MODEL_PATH, str(tmp_path)) + monkeypatch.setenv(SageMakerEnvVars.CUSTOM_SCRIPT_FILENAME, script_path.name) + monkeypatch.setenv( + FastAPIEnvVars.CUSTOM_FASTAPI_PING_HANDLER, + f"{script_path.name}:env_var_ping_handler", + ) + monkeypatch.setenv( + FastAPIEnvVars.CUSTOM_FASTAPI_INVOCATION_HANDLER, + f"{script_path.name}:env_var_invoke_handler", + ) + + with _build_sagemaker_test_client() as client: + # Test ping handler override + ping_response = client.get("/ping") + assert ping_response.status_code == 200 + ping_data = ping_response.json() + + # Environment variable should override script function + assert ping_data["method"] == "environment_variable" + assert ping_data["source"] == "env_var_ping" + + # Test invocation handler override + invoke_response = client.post( + "/invocations", + json={ + "model": MODEL_NAME_SMOLLM, + "messages": [{"role": "user", "content": "Hello"}], + "max_tokens": 5, + }, + ) + assert invoke_response.status_code == 200 + invoke_data = invoke_response.json() + + # Environment variable should override script function + assert invoke_data["method"] == "environment_variable" + assert invoke_data["source"] == "env_var_invoke" + + def test_env_var_priority_over_decorator_and_script(self, monkeypatch, tmp_path): + """Test that environment variables have highest priority over decorators + and script functions for both ping and invocation handlers.""" + try: + from model_hosting_container_standards.common.fastapi.config import ( + FastAPIEnvVars, + ) + from model_hosting_container_standards.sagemaker.config import ( + SageMakerEnvVars, + ) + except ImportError: + pytest.skip("model-hosting-container-standards not available") + + # Create a script with all three handler types for both ping and invocation + script_path = tmp_path / "model.py" + script_path.write_text( + """ +import model_hosting_container_standards.sagemaker as sagemaker_standards +from fastapi import Request, Response +import json + +# Environment variable handlers (highest priority) +async def env_priority_ping(raw_request: Request) -> Response: + return Response( + content=json.dumps({ + "status": "healthy", + "source": "env_var", + "priority": "environment_variable" + }), + media_type="application/json" + ) + +async def env_priority_invoke(raw_request: Request) -> Response: + return Response( + content=json.dumps({ + "predictions": ["Environment variable response"], + "source": "env_var", + "priority": "environment_variable" + }), + media_type="application/json" + ) + +# Decorator handlers (medium priority) +@sagemaker_standards.custom_ping_handler +async def decorator_ping(raw_request: Request) -> Response: + return Response( + content=json.dumps({ + "status": "healthy", + "source": "decorator", + "priority": "decorator" + }), + media_type="application/json" + ) + +@sagemaker_standards.custom_invocation_handler +async def decorator_invoke(raw_request: Request) -> Response: + return Response( + content=json.dumps({ + "predictions": ["Decorator response"], + "source": "decorator", + "priority": "decorator" + }), + media_type="application/json" + ) + +# Script functions (lowest priority) +async def custom_sagemaker_ping_handler(): + return { + "status": "healthy", + "source": "script", + "priority": "script_function" + } + +async def custom_sagemaker_invocation_handler(request: Request): + return { + "predictions": ["Script function response"], + "source": "script", + "priority": "script_function" + } +""" + ) + + # Set environment variables to specify highest priority handlers + monkeypatch.setenv(SageMakerEnvVars.SAGEMAKER_MODEL_PATH, str(tmp_path)) + monkeypatch.setenv(SageMakerEnvVars.CUSTOM_SCRIPT_FILENAME, script_path.name) + monkeypatch.setenv( + FastAPIEnvVars.CUSTOM_FASTAPI_PING_HANDLER, + f"{script_path.name}:env_priority_ping", + ) + monkeypatch.setenv( + FastAPIEnvVars.CUSTOM_FASTAPI_INVOCATION_HANDLER, + f"{script_path.name}:env_priority_invoke", + ) + + with _build_sagemaker_test_client() as client: + # Test ping handler priority + ping_response = client.get("/ping") + assert ping_response.status_code == 200 + ping_data = ping_response.json() + + # Environment variable has highest priority and should be used + assert ping_data["priority"] == "environment_variable" + assert ping_data["source"] == "env_var" + + # Test invocation handler priority + invoke_response = client.post( + "/invocations", + json={ + "model": MODEL_NAME_SMOLLM, + "messages": [{"role": "user", "content": "Hello"}], + "max_tokens": 5, + }, + ) + assert invoke_response.status_code == 200 + invoke_data = invoke_response.json() + + # Environment variable has highest priority and should be used + assert invoke_data["priority"] == "environment_variable" + assert invoke_data["source"] == "env_var" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/test_sagemaker_lora_adapters.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/test_sagemaker_lora_adapters.py new file mode 100644 index 0000000000000000000000000000000000000000..b2e85cc9e2d73597400e9d218972586933132171 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/test_sagemaker_lora_adapters.py @@ -0,0 +1,172 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import openai # use the official async_client for correctness check +import pytest +import requests + +from tests.utils import RemoteOpenAIServer + +from .conftest import MODEL_NAME_SMOLLM + + +@pytest.mark.asyncio +async def test_sagemaker_load_adapter_happy_path( + async_client: openai.AsyncOpenAI, + basic_server_with_lora: RemoteOpenAIServer, + smollm2_lora_files, +): + # The SageMaker standards library creates a POST /adapters endpoint + # that maps to the load_lora_adapter handler with request shape: + # {"lora_name": "body.name", "lora_path": "body.src"} + load_response = requests.post( + basic_server_with_lora.url_for("adapters"), + json={"name": "smollm2-lora-sagemaker", "src": smollm2_lora_files}, + ) + load_response.raise_for_status() + + models = await async_client.models.list() + models = models.data + dynamic_lora_model = models[-1] + assert dynamic_lora_model.root == smollm2_lora_files + assert dynamic_lora_model.parent == MODEL_NAME_SMOLLM + assert dynamic_lora_model.id == "smollm2-lora-sagemaker" + + +@pytest.mark.asyncio +async def test_sagemaker_unload_adapter_happy_path( + async_client: openai.AsyncOpenAI, + basic_server_with_lora: RemoteOpenAIServer, + smollm2_lora_files, +): + # First, load an adapter + adapter_name = "smollm2-lora-sagemaker-unload" + load_response = requests.post( + basic_server_with_lora.url_for("adapters"), + json={"name": adapter_name, "src": smollm2_lora_files}, + ) + load_response.raise_for_status() + + # Verify it's in the models list + models = await async_client.models.list() + adapter_ids = [model.id for model in models.data] + assert adapter_name in adapter_ids + + # Now unload it using DELETE /adapters/{adapter_name} + # The SageMaker standards maps this to unload_lora_adapter with: + # {"lora_name": "path_params.adapter_name"} + unload_response = requests.delete( + basic_server_with_lora.url_for("adapters", adapter_name), + ) + unload_response.raise_for_status() + + # Verify it's no longer in the models list + models = await async_client.models.list() + adapter_ids = [model.id for model in models.data] + assert adapter_name not in adapter_ids + + +@pytest.mark.asyncio +async def test_sagemaker_load_adapter_not_found( + basic_server_with_lora: RemoteOpenAIServer, +): + load_response = requests.post( + basic_server_with_lora.url_for("adapters"), + json={"name": "nonexistent-adapter", "src": "/path/does/not/exist"}, + ) + assert load_response.status_code == 404 + + +@pytest.mark.asyncio +async def test_sagemaker_load_adapter_invalid_files( + basic_server_with_lora: RemoteOpenAIServer, + tmp_path, +): + invalid_files = tmp_path / "invalid_adapter" + invalid_files.mkdir() + (invalid_files / "adapter_config.json").write_text("not valid json") + + load_response = requests.post( + basic_server_with_lora.url_for("adapters"), + json={"name": "invalid-adapter", "src": str(invalid_files)}, + ) + assert load_response.status_code == 500 + + +@pytest.mark.asyncio +async def test_sagemaker_unload_nonexistent_adapter( + basic_server_with_lora: RemoteOpenAIServer, +): + # Attempt to unload an adapter that doesn't exist + unload_response = requests.delete( + basic_server_with_lora.url_for("adapters", "nonexistent-adapter-name"), + ) + assert unload_response.status_code in (400, 404) + + +@pytest.mark.asyncio +async def test_sagemaker_invocations_with_adapter( + basic_server_with_lora: RemoteOpenAIServer, + smollm2_lora_files, +): + # First, load an adapter via SageMaker endpoint + adapter_name = "smollm2-lora-invoke-test" + load_response = requests.post( + basic_server_with_lora.url_for("adapters"), + json={"name": adapter_name, "src": smollm2_lora_files}, + ) + load_response.raise_for_status() + + # Now test the /invocations endpoint with the adapter + invocation_response = requests.post( + basic_server_with_lora.url_for("invocations"), + headers={ + "X-Amzn-SageMaker-Adapter-Identifier": adapter_name, + }, + json={ + "prompt": "Hello, how are you?", + "max_tokens": 10, + }, + ) + invocation_response.raise_for_status() + invocation_output = invocation_response.json() + + # Verify we got a valid completion response + assert "choices" in invocation_output + assert len(invocation_output["choices"]) > 0 + assert "text" in invocation_output["choices"][0] + + +@pytest.mark.asyncio +async def test_sagemaker_multiple_adapters_load_unload( + async_client: openai.AsyncOpenAI, + basic_server_with_lora: RemoteOpenAIServer, + smollm2_lora_files, +): + adapter_names = [f"sagemaker-adapter-{i}" for i in range(5)] + + # Load all adapters + for adapter_name in adapter_names: + load_response = requests.post( + basic_server_with_lora.url_for("adapters"), + json={"name": adapter_name, "src": smollm2_lora_files}, + ) + load_response.raise_for_status() + + # Verify all are in the models list + models = await async_client.models.list() + adapter_ids = [model.id for model in models.data] + for adapter_name in adapter_names: + assert adapter_name in adapter_ids + + # Unload all adapters + for adapter_name in adapter_names: + unload_response = requests.delete( + basic_server_with_lora.url_for("adapters", adapter_name), + ) + unload_response.raise_for_status() + + # Verify all are removed from models list + models = await async_client.models.list() + adapter_ids = [model.id for model in models.data] + for adapter_name in adapter_names: + assert adapter_name not in adapter_ids diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/test_sagemaker_middleware_integration.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/test_sagemaker_middleware_integration.py new file mode 100644 index 0000000000000000000000000000000000000000..2bb8fc819add9fefb7b059d7de88e6964ca7bbde --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/test_sagemaker_middleware_integration.py @@ -0,0 +1,347 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Integration test for middleware loader functionality. + +Tests that customer middlewares get called correctly with a vLLM server. +""" + +import os +import tempfile + +import pytest +import requests + +from tests.utils import RemoteOpenAIServer + +from .conftest import ( + MODEL_NAME_SMOLLM, +) + + +class TestMiddlewareIntegration: + """Integration test for middleware with vLLM server.""" + + def setup_method(self): + """Setup for each test - simulate fresh server startup.""" + self._clear_caches() + + def _clear_caches(self): + """Clear middleware registry and function loader cache.""" + try: + from model_hosting_container_standards.common.fastapi.middleware import ( + middleware_registry, + ) + from model_hosting_container_standards.common.fastapi.middleware.source.decorator_loader import ( # noqa: E501 + decorator_loader, + ) + from model_hosting_container_standards.sagemaker.sagemaker_loader import ( + SageMakerFunctionLoader, + ) + + middleware_registry.clear_middlewares() + decorator_loader.clear() + SageMakerFunctionLoader._default_function_loader = None + except ImportError: + pytest.skip("model-hosting-container-standards not available") + + @pytest.mark.asyncio + async def test_customer_middleware_with_vllm_server(self): + """Test that customer middlewares work with actual vLLM server. + + Tests decorator-based middlewares (@custom_middleware, @input_formatter, + @output_formatter) + on multiple endpoints (chat/completions, invocations). + """ + try: + from model_hosting_container_standards.sagemaker.config import ( + SageMakerEnvVars, + ) + except ImportError: + pytest.skip("model-hosting-container-standards not available") + + # Customer writes a middleware script with multiple decorators + with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f: + f.write( + """ +from model_hosting_container_standards.common.fastapi.middleware import ( + custom_middleware, input_formatter, output_formatter +) + +# Global flag to track if input formatter was called +_input_formatter_called = False + +@input_formatter +async def customer_input_formatter(request): + # Process input - mark that input formatter was called + global _input_formatter_called + _input_formatter_called = True + return request + +@custom_middleware("throttle") +async def customer_throttle_middleware(request, call_next): + response = await call_next(request) + response.headers["X-Customer-Throttle"] = "applied" + order = response.headers.get("X-Middleware-Order", "") + response.headers["X-Middleware-Order"] = order + "throttle," + return response + +@output_formatter +async def customer_output_formatter(response): + global _input_formatter_called + response.headers["X-Customer-Processed"] = "true" + # Since input_formatter and output_formatter are combined into + # pre_post_process middleware, + # if output_formatter is called, input_formatter should have been called too + if _input_formatter_called: + response.headers["X-Input-Formatter-Called"] = "true" + order = response.headers.get("X-Middleware-Order", "") + response.headers["X-Middleware-Order"] = order + "output_formatter," + return response +""" + ) + script_path = f.name + + try: + script_dir = os.path.dirname(script_path) + script_name = os.path.basename(script_path) + + # Set environment variables to point to customer script + env_vars = { + SageMakerEnvVars.SAGEMAKER_MODEL_PATH: script_dir, + SageMakerEnvVars.CUSTOM_SCRIPT_FILENAME: script_name, + } + + args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "2048", + "--enforce-eager", + "--max-num-seqs", + "32", + ] + + with RemoteOpenAIServer( + MODEL_NAME_SMOLLM, args, env_dict=env_vars + ) as server: + # Test 1: Middlewares applied to chat/completions endpoint + chat_response = requests.post( + server.url_for("v1/chat/completions"), + json={ + "model": MODEL_NAME_SMOLLM, + "messages": [{"role": "user", "content": "Hello"}], + "max_tokens": 5, + "temperature": 0.0, + }, + ) + + assert chat_response.status_code == 200 + + # Verify all middlewares were executed + assert "X-Customer-Throttle" in chat_response.headers + assert chat_response.headers["X-Customer-Throttle"] == "applied" + assert "X-Customer-Processed" in chat_response.headers + assert chat_response.headers["X-Customer-Processed"] == "true" + + # Verify input formatter was called + assert "X-Input-Formatter-Called" in chat_response.headers + assert chat_response.headers["X-Input-Formatter-Called"] == "true" + + # Verify middleware execution order + execution_order = chat_response.headers.get( + "X-Middleware-Order", "" + ).rstrip(",") + order_parts = execution_order.split(",") if execution_order else [] + assert "throttle" in order_parts + assert "output_formatter" in order_parts + + # Test 2: Middlewares applied to invocations endpoint + invocations_response = requests.post( + server.url_for("invocations"), + json={ + "model": MODEL_NAME_SMOLLM, + "messages": [{"role": "user", "content": "Hello"}], + "max_tokens": 5, + "temperature": 0.0, + }, + ) + + assert invocations_response.status_code == 200 + + # Verify all middlewares were executed + assert "X-Customer-Throttle" in invocations_response.headers + assert invocations_response.headers["X-Customer-Throttle"] == "applied" + assert "X-Customer-Processed" in invocations_response.headers + assert invocations_response.headers["X-Customer-Processed"] == "true" + + # Verify input formatter was called + assert "X-Input-Formatter-Called" in invocations_response.headers + assert ( + invocations_response.headers["X-Input-Formatter-Called"] == "true" + ) + + finally: + os.unlink(script_path) + + @pytest.mark.asyncio + async def test_middleware_with_ping_endpoint(self): + """Test that middlewares work with SageMaker ping endpoint.""" + try: + from model_hosting_container_standards.sagemaker.config import ( + SageMakerEnvVars, + ) + except ImportError: + pytest.skip("model-hosting-container-standards not available") + + # Customer writes a middleware script + with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f: + f.write( + """ +from model_hosting_container_standards.common.fastapi.middleware import ( + custom_middleware +) + +@custom_middleware("pre_post_process") +async def ping_tracking_middleware(request, call_next): + response = await call_next(request) + if request.url.path == "/ping": + response.headers["X-Ping-Tracked"] = "true" + return response +""" + ) + script_path = f.name + + try: + script_dir = os.path.dirname(script_path) + script_name = os.path.basename(script_path) + + env_vars = { + SageMakerEnvVars.SAGEMAKER_MODEL_PATH: script_dir, + SageMakerEnvVars.CUSTOM_SCRIPT_FILENAME: script_name, + } + + args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "2048", + "--enforce-eager", + "--max-num-seqs", + "32", + ] + + with RemoteOpenAIServer( + MODEL_NAME_SMOLLM, args, env_dict=env_vars + ) as server: + # Test ping endpoint with middleware + response = requests.get(server.url_for("ping")) + + assert response.status_code == 200 + assert "X-Ping-Tracked" in response.headers + assert response.headers["X-Ping-Tracked"] == "true" + + finally: + os.unlink(script_path) + + @pytest.mark.asyncio + async def test_middleware_env_var_override(self): + """Test middleware environment variable overrides.""" + try: + from model_hosting_container_standards.common.fastapi.config import ( + FastAPIEnvVars, + ) + from model_hosting_container_standards.sagemaker.config import ( + SageMakerEnvVars, + ) + except ImportError: + pytest.skip("model-hosting-container-standards not available") + + # Create a script with middleware functions specified via env vars + with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f: + f.write( + """ +from fastapi import Request + +# Global flag to track if pre_process was called +_pre_process_called = False + +async def env_throttle_middleware(request, call_next): + response = await call_next(request) + response.headers["X-Env-Throttle"] = "applied" + return response + +async def env_pre_process(request: Request) -> Request: + # Mark that pre_process was called + global _pre_process_called + _pre_process_called = True + return request + +async def env_post_process(response): + global _pre_process_called + if hasattr(response, 'headers'): + response.headers["X-Env-Post-Process"] = "applied" + # Since pre_process and post_process are combined into + # pre_post_process middleware, + # if post_process is called, pre_process should have been called too + if _pre_process_called: + response.headers["X-Pre-Process-Called"] = "true" + return response +""" + ) + script_path = f.name + + try: + script_dir = os.path.dirname(script_path) + script_name = os.path.basename(script_path) + + # Set environment variables for middleware + # Use script_name with .py extension as per plugin example + env_vars = { + SageMakerEnvVars.SAGEMAKER_MODEL_PATH: script_dir, + SageMakerEnvVars.CUSTOM_SCRIPT_FILENAME: script_name, + FastAPIEnvVars.CUSTOM_FASTAPI_MIDDLEWARE_THROTTLE: ( + f"{script_name}:env_throttle_middleware" + ), + FastAPIEnvVars.CUSTOM_PRE_PROCESS: f"{script_name}:env_pre_process", + FastAPIEnvVars.CUSTOM_POST_PROCESS: f"{script_name}:env_post_process", + } + + args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "2048", + "--enforce-eager", + "--max-num-seqs", + "32", + ] + + with RemoteOpenAIServer( + MODEL_NAME_SMOLLM, args, env_dict=env_vars + ) as server: + response = requests.get(server.url_for("ping")) + assert response.status_code == 200 + + # Check if environment variable middleware was applied + headers = response.headers + + # Verify that env var middlewares were applied + assert "X-Env-Throttle" in headers, ( + "Throttle middleware should be applied via env var" + ) + assert headers["X-Env-Throttle"] == "applied" + + assert "X-Env-Post-Process" in headers, ( + "Post-process middleware should be applied via env var" + ) + assert headers["X-Env-Post-Process"] == "applied" + + # Verify that pre_process was called + assert "X-Pre-Process-Called" in headers, ( + "Pre-process should be called via env var" + ) + assert headers["X-Pre-Process-Called"] == "true" + + finally: + os.unlink(script_path) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/test_sagemaker_stateful_sessions.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/test_sagemaker_stateful_sessions.py new file mode 100644 index 0000000000000000000000000000000000000000..51fe9c3b24eda3237f7737a50a4b98430c43529a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/sagemaker/test_sagemaker_stateful_sessions.py @@ -0,0 +1,154 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import openai # use the official client for correctness check +import pytest +import requests + +from tests.utils import RemoteOpenAIServer + +from .conftest import ( + HEADER_SAGEMAKER_CLOSED_SESSION_ID, + HEADER_SAGEMAKER_NEW_SESSION_ID, + HEADER_SAGEMAKER_SESSION_ID, + MODEL_NAME_SMOLLM, +) + +CLOSE_BADREQUEST_CASES = [ + ( + "nonexistent_session_id", + {"session_id": "nonexistent-session-id"}, + {}, + "session not found", + ), + ("malformed_close_request", {}, {"extra-field": "extra-field-data"}, None), +] + + +@pytest.mark.asyncio +async def test_create_session_badrequest(basic_server_with_lora: RemoteOpenAIServer): + bad_response = requests.post( + basic_server_with_lora.url_for("invocations"), + json={"requestType": "NEW_SESSION", "extra-field": "extra-field-data"}, + ) + + assert bad_response.status_code == 400 + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "test_name,session_id_change,request_body_change,expected_error", + CLOSE_BADREQUEST_CASES, +) +async def test_close_session_badrequest( + basic_server_with_lora: RemoteOpenAIServer, + test_name: str, + session_id_change: dict[str, str], + request_body_change: dict[str, str], + expected_error: str | None, +): + # first attempt to create a session + url = basic_server_with_lora.url_for("invocations") + create_response = requests.post(url, json={"requestType": "NEW_SESSION"}) + create_response.raise_for_status() + valid_session_id, expiration = create_response.headers.get( + HEADER_SAGEMAKER_NEW_SESSION_ID, "" + ).split(";") + assert valid_session_id + + close_request_json = {"requestType": "CLOSE"} + if request_body_change: + close_request_json.update(request_body_change) + bad_session_id = session_id_change.get("session_id") + bad_close_response = requests.post( + url, + headers={HEADER_SAGEMAKER_SESSION_ID: bad_session_id or valid_session_id}, + json=close_request_json, + ) + + # clean up created session, should succeed + clean_up_response = requests.post( + url, + headers={HEADER_SAGEMAKER_SESSION_ID: valid_session_id}, + json={"requestType": "CLOSE"}, + ) + clean_up_response.raise_for_status() + + assert bad_close_response.status_code == 400 + if expected_error: + assert expected_error in bad_close_response.json()["error"]["message"] + + +@pytest.mark.asyncio +async def test_close_session_invalidrequest( + basic_server_with_lora: RemoteOpenAIServer, async_client: openai.AsyncOpenAI +): + # first attempt to create a session + url = basic_server_with_lora.url_for("invocations") + create_response = requests.post(url, json={"requestType": "NEW_SESSION"}) + create_response.raise_for_status() + valid_session_id, expiration = create_response.headers.get( + HEADER_SAGEMAKER_NEW_SESSION_ID, "" + ).split(";") + assert valid_session_id + + close_request_json = {"requestType": "CLOSE"} + invalid_close_response = requests.post( + url, + # no headers to specify session_id + json=close_request_json, + ) + + # clean up created session, should succeed + clean_up_response = requests.post( + url, + headers={HEADER_SAGEMAKER_SESSION_ID: valid_session_id}, + json={"requestType": "CLOSE"}, + ) + clean_up_response.raise_for_status() + + assert invalid_close_response.status_code == 424 + assert "invalid session_id" in invalid_close_response.json()["error"]["message"] + + +@pytest.mark.asyncio +async def test_session(basic_server_with_lora: RemoteOpenAIServer): + # first attempt to create a session + url = basic_server_with_lora.url_for("invocations") + create_response = requests.post(url, json={"requestType": "NEW_SESSION"}) + create_response.raise_for_status() + valid_session_id, expiration = create_response.headers.get( + HEADER_SAGEMAKER_NEW_SESSION_ID, "" + ).split(";") + assert valid_session_id + + # test invocation with session id + + request_args = { + "model": MODEL_NAME_SMOLLM, + "prompt": "what is 1+1?", + "max_completion_tokens": 5, + "temperature": 0.0, + "logprobs": False, + } + + invocation_response = requests.post( + basic_server_with_lora.url_for("invocations"), + headers={HEADER_SAGEMAKER_SESSION_ID: valid_session_id}, + json=request_args, + ) + invocation_response.raise_for_status() + + # close created session, should succeed + close_response = requests.post( + url, + headers={HEADER_SAGEMAKER_SESSION_ID: valid_session_id}, + json={"requestType": "CLOSE"}, + ) + close_response.raise_for_status() + + assert ( + close_response.headers.get(HEADER_SAGEMAKER_CLOSED_SESSION_ID) + == valid_session_id + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/tokenize/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/tokenize/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/tokenize/test_serving_tokenization.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/tokenize/test_serving_tokenization.py new file mode 100644 index 0000000000000000000000000000000000000000..8b05b634373a682a01cc6b9e19716071b6d40658 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/tokenize/test_serving_tokenization.py @@ -0,0 +1,135 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from dataclasses import dataclass, field +from typing import Any +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from vllm.config.multimodal import MultiModalConfig +from vllm.entrypoints.openai.models.protocol import BaseModelPath +from vllm.entrypoints.openai.models.serving import OpenAIServingModels +from vllm.entrypoints.serve.tokenize.protocol import ( + TokenizeChatRequest, + TokenizeCompletionRequest, +) +from vllm.entrypoints.serve.tokenize.serving import ServingTokenization +from vllm.renderers.online_renderer import OnlineRenderer +from vllm.v1.engine.async_llm import AsyncLLM + +MODEL_NAME = "openai-community/gpt2" +BASE_MODEL_PATHS = [ + BaseModelPath(name=MODEL_NAME, model_path=MODEL_NAME), +] + + +@dataclass +class MockHFConfig: + model_type: str = "any" + + +@dataclass +class MockModelConfig: + task = "generate" + runner_type = "generate" + model = MODEL_NAME + tokenizer = MODEL_NAME + trust_remote_code = False + tokenizer_mode = "auto" + max_model_len = 100 + tokenizer_revision = None + multimodal_config = MultiModalConfig() + hf_config = MockHFConfig() + hf_text_config = MockHFConfig() + logits_processors: list[str] | None = None + diff_sampling_param: dict | None = None + allowed_local_media_path: str = "" + allowed_media_domains: list[str] | None = None + encoder_config = None + generation_config: str = "auto" + media_io_kwargs: dict[str, dict[str, Any]] = field(default_factory=dict) + skip_tokenizer_init = False + is_encoder_decoder: bool = False + is_multimodal_model: bool = False + renderer_num_workers: int = 1 + + def get_diff_sampling_param(self): + return self.diff_sampling_param or {} + + +def _build_serving_tokenization(engine: AsyncLLM) -> ServingTokenization: + models = OpenAIServingModels( + engine_client=engine, + base_model_paths=BASE_MODEL_PATHS, + ) + online_renderer = OnlineRenderer( + model_config=engine.model_config, + renderer=engine.renderer, + request_logger=None, + chat_template=None, + chat_template_content_format="auto", + ) + return ServingTokenization( + models, + online_renderer=online_renderer, + chat_template=None, + chat_template_content_format="auto", + ) + + +@pytest.mark.asyncio +async def test_tokenize_chat_skips_mm_cache_for_renderer_only_path(): + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = MagicMock() + + serving = _build_serving_tokenization(mock_engine) + serving.online_renderer.preprocess_chat = AsyncMock( + return_value=( + [{"role": "user", "content": "Test"}], + [{"prompt_token_ids": [1, 2, 3]}], + ) + ) + + request = TokenizeChatRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "Test prompt"}], + ) + + response = await serving.create_tokenize(request, MagicMock(headers={})) + + assert response.tokens == [1, 2, 3] + assert ( + serving.online_renderer.preprocess_chat.call_args.kwargs["skip_mm_cache"] + is True + ) + + +@pytest.mark.asyncio +async def test_tokenize_completion_skips_mm_cache_for_renderer_only_path(): + mock_engine = MagicMock(spec=AsyncLLM) + mock_engine.errored = False + mock_engine.model_config = MockModelConfig() + mock_engine.input_processor = MagicMock() + mock_engine.renderer = MagicMock() + + serving = _build_serving_tokenization(mock_engine) + serving.online_renderer.preprocess_completion = AsyncMock( + return_value=[{"prompt_token_ids": [1, 2, 3]}] + ) + + request = TokenizeCompletionRequest( + model=MODEL_NAME, + prompt="Test prompt", + ) + + response = await serving.create_tokenize(request, MagicMock(headers={})) + + assert response.tokens == [1, 2, 3] + assert ( + serving.online_renderer.preprocess_completion.call_args.kwargs["skip_mm_cache"] + is True + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/tokenize/test_tokenization.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/tokenize/test_tokenization.py new file mode 100644 index 0000000000000000000000000000000000000000..3405ba7823b2bd4e280628f5f503c762f0c8020c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/tokenize/test_tokenization.py @@ -0,0 +1,334 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import pytest_asyncio +import requests + +from tests.utils import RemoteOpenAIServer +from vllm.tokenizers import get_tokenizer + +# any model with a chat template should work here +MODEL_NAME = "HuggingFaceTB/SmolLM2-135M-Instruct" + + +@pytest.fixture(scope="module") +def server(): + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-model-len", + "8192", + "--enforce-eager", + "--max-num-seqs", + "128", + "--enable-tokenizer-info-endpoint", + ] + + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest.fixture(scope="module") +def tokenizer_name(model_name: str): + return model_name + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name,tokenizer_name", + [(MODEL_NAME, MODEL_NAME)], + indirect=["tokenizer_name"], +) +async def test_tokenize_completions( + server: RemoteOpenAIServer, + model_name: str, + tokenizer_name: str, +): + tokenizer = get_tokenizer(tokenizer_name=tokenizer_name) + + for add_special in [False, True]: + prompt = "vllm1 This is a test prompt." + tokens = tokenizer.encode(prompt, add_special_tokens=add_special) + + response = requests.post( + server.url_for("tokenize"), + json={ + "add_special_tokens": add_special, + "model": model_name, + "prompt": prompt, + }, + ) + response.raise_for_status() + + result = response.json() + assert result["tokens"] == tokens + assert result["count"] == len(tokens) + assert result["max_model_len"] == 8192 + assert result["token_strs"] is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name,tokenizer_name", + [(MODEL_NAME, MODEL_NAME)], + indirect=["tokenizer_name"], +) +async def test_tokenize_chat( + server: RemoteOpenAIServer, + model_name: str, + tokenizer_name: str, +): + tokenizer = get_tokenizer(tokenizer_name=tokenizer_name) + + for add_generation in [False, True]: + for add_special in [False, True]: + conversation = [ + {"role": "user", "content": "Hi there!"}, + {"role": "assistant", "content": "Nice to meet you!"}, + {"role": "user", "content": "Can I ask a question? vllm1"}, + ] + for continue_final in [False, True]: + if add_generation and continue_final: + continue + if continue_final: + conversation.append({"role": "assistant", "content": "Sure,"}) + + prompt = tokenizer.apply_chat_template( + add_generation_prompt=add_generation, + continue_final_message=continue_final, + conversation=conversation, + tokenize=False, + ) + tokens = tokenizer.encode(prompt, add_special_tokens=add_special) + + response = requests.post( + server.url_for("tokenize"), + json={ + "add_generation_prompt": add_generation, + "continue_final_message": continue_final, + "add_special_tokens": add_special, + "messages": conversation, + "model": model_name, + }, + ) + response.raise_for_status() + + result = response.json() + assert result["tokens"] == tokens + assert result["count"] == len(tokens) + assert result["max_model_len"] == 8192 + assert result["token_strs"] is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name,tokenizer_name", + [(MODEL_NAME, MODEL_NAME)], + indirect=["tokenizer_name"], +) +async def test_tokenize_chat_with_tools( + server: RemoteOpenAIServer, + model_name: str, + tokenizer_name: str, +): + tokenizer = get_tokenizer(tokenizer_name=tokenizer_name) + + for add_generation in [False, True]: + for add_special in [False, True]: + conversation = [ + { + "role": "user", + "content": "What's the weather like in Paris today?", + } + ] + + tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "parameters": { + "type": "object", + "properties": {"location": {"type": "string"}}, + }, + }, + } + ] + + for continue_final in [False, True]: + if add_generation and continue_final: + continue + if continue_final: + conversation.append({"role": "assistant", "content": "Sure,"}) + + prompt = tokenizer.apply_chat_template( + add_generation_prompt=add_generation, + continue_final_message=continue_final, + conversation=conversation, + tools=tools, + tokenize=False, + ) + tokens = tokenizer.encode(prompt, add_special_tokens=add_special) + + response = requests.post( + server.url_for("tokenize"), + json={ + "add_generation_prompt": add_generation, + "continue_final_message": continue_final, + "add_special_tokens": add_special, + "messages": conversation, + "model": model_name, + "tools": tools, + }, + ) + response.raise_for_status() + + result = response.json() + assert result["tokens"] == tokens + assert result["count"] == len(tokens) + assert result["max_model_len"] == 8192 + assert result["token_strs"] is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name, tokenizer_name", + [(MODEL_NAME, MODEL_NAME)], + indirect=["tokenizer_name"], +) +async def test_tokenize_with_return_token_strs( + server: RemoteOpenAIServer, + model_name: str, + tokenizer_name: str, +): + tokenizer = get_tokenizer(tokenizer_name=tokenizer_name) + + prompt = "This is a token_strs test prompt! vllm1" + response = requests.post( + server.url_for("tokenize"), + json={"prompt": prompt, "model": model_name, "return_token_strs": True}, + ) + response.raise_for_status() + + tokens = tokenizer.encode(prompt, add_special_tokens=True) + tokens_str = tokenizer.convert_ids_to_tokens(tokens) + + result = response.json() + assert result["tokens"] == tokens + assert result["count"] == len(tokens) + assert result["max_model_len"] == 8192 + assert result["token_strs"] == tokens_str + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name,tokenizer_name", + [(MODEL_NAME, MODEL_NAME)], + indirect=["tokenizer_name"], +) +async def test_detokenize( + server: RemoteOpenAIServer, + model_name: str, + tokenizer_name: str, +): + tokenizer = get_tokenizer(tokenizer_name=tokenizer_name) + + prompt = "This is a test prompt. vllm1" + tokens = tokenizer.encode(prompt, add_special_tokens=False) + + response = requests.post( + server.url_for("detokenize"), json={"model": model_name, "tokens": tokens} + ) + response.raise_for_status() + + assert response.json() == {"prompt": prompt} + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name,tokenizer_name", + [(MODEL_NAME, MODEL_NAME)], + indirect=["tokenizer_name"], +) +async def test_tokenizer_info_basic( + server: RemoteOpenAIServer, + model_name: str, + tokenizer_name: str, +): + """Test basic tokenizer info endpoint functionality.""" + response = requests.get(server.url_for("tokenizer_info")) + response.raise_for_status() + result = response.json() + assert "tokenizer_class" in result + assert isinstance(result["tokenizer_class"], str) + assert result["tokenizer_class"] + + +@pytest.mark.asyncio +async def test_tokenizer_info_schema(server: RemoteOpenAIServer): + """Test that the response matches expected schema types.""" + response = requests.get(server.url_for("tokenizer_info")) + response.raise_for_status() + result = response.json() + field_types = { + "add_bos_token": bool, + "add_prefix_space": bool, + "clean_up_tokenization_spaces": bool, + "split_special_tokens": bool, + "bos_token": str, + "eos_token": str, + "pad_token": str, + "unk_token": str, + "chat_template": str, + "errors": str, + "model_max_length": int, + "additional_special_tokens": list, + "added_tokens_decoder": dict, + } + for field, expected_type in field_types.items(): + if field in result and result[field] is not None: + assert isinstance(result[field], expected_type), ( + f"{field} should be {expected_type.__name__}" + ) + + +@pytest.mark.asyncio +async def test_tokenizer_info_consistency_with_tokenize( + server: RemoteOpenAIServer, +): + """Test that tokenizer info is consistent with tokenization endpoint.""" + info_response = requests.get(server.url_for("tokenizer_info")) + info_response.raise_for_status() + info = info_response.json() + tokenize_response = requests.post( + server.url_for("tokenize"), + json={"model": MODEL_NAME, "prompt": "Hello world!"}, + ) + tokenize_response.raise_for_status() + tokenize_result = tokenize_response.json() + info_max_len = info.get("model_max_length") + tokenize_max_len = tokenize_result.get("max_model_len") + if info_max_len and tokenize_max_len: + assert info_max_len >= tokenize_max_len, ( + "Info max length should be >= tokenize max length" + ) + + +@pytest.mark.asyncio +async def test_tokenizer_info_chat_template(server: RemoteOpenAIServer): + """Test chat template is properly included.""" + response = requests.get(server.url_for("tokenizer_info")) + response.raise_for_status() + result = response.json() + chat_template = result.get("chat_template") + if chat_template: + assert isinstance(chat_template, str), "Chat template should be a string" + assert chat_template.strip(), "Chat template should not be empty" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/tokenize/test_tokenization_vlm.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/tokenize/test_tokenization_vlm.py new file mode 100644 index 0000000000000000000000000000000000000000..e2b752ec93b2c083d4d129085ca286e3a2dd7151 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/tokenize/test_tokenization_vlm.py @@ -0,0 +1,61 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Regression test: ``/tokenize`` must expand image placeholders for VLM models. + +Fixed by PR #34560 ("Move InputPreprocessor into Renderer (2/2)"). +Before that change, ``/tokenize`` returned ~26 tokens for a message with an +image instead of the expected 1451. Confirmed broken on 0.15.1 and 0.16.0. +""" + +import json + +import pytest +import requests + +from tests.utils import RemoteOpenAIServer + +MODEL_NAME = "Qwen/Qwen2.5-VL-3B-Instruct" + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "4096", + "--max-num-seqs", + "5", + "--enforce-eager", + "--limit-mm-per-prompt", + json.dumps({"image": 1}), + ] + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +def test_tokenize_chat_expands_image_placeholders( + server: RemoteOpenAIServer, + local_asset_server, +): + image_url = local_asset_server.url_for("stop_sign.jpg") + messages = [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "Describe this image."}, + ], + } + ] + + response = requests.post( + server.url_for("tokenize"), + json={"model": MODEL_NAME, "messages": messages}, + ) + response.raise_for_status() + + # stop_sign.jpg (1300x876) produces 1451 tokens after expansion. + # Without expansion the count would be ~26 (text + one placeholder). + assert response.json()["count"] == 1451 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/tokenize/test_tokenize_then_chat_vlm.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/tokenize/test_tokenize_then_chat_vlm.py new file mode 100644 index 0000000000000000000000000000000000000000..c2bc96b5f394428457a991fc5088d1167864708a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/tokenize/test_tokenize_then_chat_vlm.py @@ -0,0 +1,85 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Regression test: calling ``/tokenize`` with multimodal data followed by +``/v1/chat/completions`` with the same data must not cause an error. + +Ensures that the ``/tokenize`` endpoint does not pollute internal caches +(e.g. multimodal feature caches) and that a subsequent +``/v1/chat/completions`` request with the same multimodal payload +completes successfully. +""" + +import json + +import openai +import pytest +import pytest_asyncio +import requests + +from tests.utils import RemoteOpenAIServer + +MODEL_NAME = "Qwen/Qwen2.5-VL-3B-Instruct" + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "4096", + "--max-num-seqs", + "5", + "--enforce-eager", + "--limit-mm-per-prompt", + json.dumps({"image": 1}), + ] + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +async def test_tokenize_then_chat_completion_with_image( + client: openai.AsyncOpenAI, + server: RemoteOpenAIServer, + local_asset_server, +): + """Tokenize a multimodal message, then send the same message to chat + completions. The chat completion must succeed (not 500).""" + + image_url = local_asset_server.url_for("stop_sign.jpg") + messages = [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "Describe this image briefly."}, + ], + } + ] + + tok_resp = requests.post( + server.url_for("tokenize"), + json={"model": MODEL_NAME, "messages": messages}, + ) + tok_resp.raise_for_status() + tok_data = tok_resp.json() + assert tok_data["count"] > 0, "Tokenization must return tokens" + + chat_completion = await client.chat.completions.create( + model=MODEL_NAME, + messages=messages, + max_tokens=10, + temperature=0.0, + ) + + assert chat_completion.choices[0].message.content, ( + "Chat completion must produce non-empty content after tokenize" + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_api_utils.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_api_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..2f74cfcc0b637c1e534b478fc3b20233133e55f1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_api_utils.py @@ -0,0 +1,120 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from vllm.entrypoints.openai.engine.protocol import StreamOptions +from vllm.entrypoints.serve.utils.api_utils import ( + get_max_tokens, + sanitize_message, + should_include_usage, +) + + +def test_sanitize_message(): + assert ( + sanitize_message("<_io.BytesIO object at 0x7a95e299e750>") + == "<_io.BytesIO object>" + ) + + +@pytest.mark.parametrize( + ("stream_options", "expected"), + [ + (None, (True, True)), + (StreamOptions(include_usage=False), (True, True)), + ( + StreamOptions(include_usage=False, continuous_usage_stats=False), + (True, True), + ), + ( + StreamOptions(include_usage=True, continuous_usage_stats=False), + (True, True), + ), + ], +) +def test_should_include_usage_force_enables_continuous_usage(stream_options, expected): + assert should_include_usage(stream_options, True) == expected + + +class TestGetMaxTokens: + """Tests for get_max_tokens() to ensure generation_config's max_tokens + acts as a default when from model author, and as a ceiling when + explicitly set by the user.""" + + def test_default_sampling_params_used_when_no_request_max_tokens(self): + """When user doesn't specify max_tokens, generation_config default + should apply.""" + result = get_max_tokens( + max_model_len=24000, + max_tokens=None, + input_length=100, + default_sampling_params={"max_tokens": 2048}, + ) + assert result == 2048 + + def test_request_max_tokens_not_capped_by_default_sampling_params(self): + """When user specifies max_tokens in request, model author's + generation_config max_tokens must NOT cap it (fixes #34005).""" + result = get_max_tokens( + max_model_len=24000, + max_tokens=5000, + input_length=100, + default_sampling_params={"max_tokens": 2048}, + ) + assert result == 5000 + + def test_override_max_tokens_caps_request(self): + """When user explicitly sets max_tokens, it acts as a ceiling.""" + result = get_max_tokens( + max_model_len=24000, + max_tokens=5000, + input_length=100, + default_sampling_params={"max_tokens": 2048}, + override_max_tokens=2048, + ) + assert result == 2048 + + def test_override_max_tokens_used_as_default(self): + """When no request max_tokens, override still applies as default.""" + result = get_max_tokens( + max_model_len=24000, + max_tokens=None, + input_length=100, + default_sampling_params={"max_tokens": 2048}, + override_max_tokens=2048, + ) + assert result == 2048 + + def test_max_model_len_still_caps_output(self): + """max_model_len - input_length is always the hard ceiling.""" + result = get_max_tokens( + max_model_len=3000, + max_tokens=5000, + input_length=100, + default_sampling_params={"max_tokens": 2048}, + ) + assert result == 2900 # 3000 - 100 + + def test_request_max_tokens_smaller_than_default(self): + """When user explicitly requests fewer tokens than gen_config default, + that should be respected.""" + result = get_max_tokens( + max_model_len=24000, + max_tokens=512, + input_length=100, + default_sampling_params={"max_tokens": 2048}, + ) + assert result == 512 + + def test_input_length_exceeds_max_model_len(self): + with pytest.raises( + ValueError, + match="Input length .* exceeds model's maximum context length .*", + ): + get_max_tokens( + max_model_len=100, + max_tokens=50, + input_length=150, + default_sampling_params={"max_tokens": 2048}, + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_error_sanitization.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_error_sanitization.py new file mode 100644 index 0000000000000000000000000000000000000000..0f97626bc7a6ca1f71fe83e9ab14aa383f197606 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_error_sanitization.py @@ -0,0 +1,81 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests that error messages in Anthropic and speech-to-text entrypoints +are sanitized to prevent memory address leakage. + +Verifies the fix for the incomplete CVE-2026-22778 remediation where +PIL repr addresses leaked via the Anthropic API router and the +speech-to-text WebSocket paths. +""" + +import pytest + +from vllm.entrypoints.serve.utils.api_utils import sanitize_message + + +class TestSanitizeMessageCoversLeakPatterns: + """Ensure sanitize_message strips addresses from realistic exceptions.""" + + @pytest.mark.parametrize( + ("raw", "expected"), + [ + ( + "cannot identify image file <_io.BytesIO object at 0x7a95e299e750>", + "cannot identify image file <_io.BytesIO object>", + ), + ( + "cannot identify image file <_io.BytesIO object at 0x7f3c1a2b4d90>", + "cannot identify image file <_io.BytesIO object>", + ), + ( + "", + "", + ), + ( + "Error processing <_io.BytesIO object at 0xdeadbeef>: invalid header", + "Error processing <_io.BytesIO object>: invalid header", + ), + ], + ids=[ + "bytesio-standard", + "bytesio-different-addr", + "pil-image-repr", + "mid-string-repr", + ], + ) + def test_address_stripped(self, raw: str, expected: str): + assert sanitize_message(raw) == expected + + def test_safe_message_unchanged(self): + msg = "Invalid request: missing 'messages' field" + assert sanitize_message(msg) == msg + + def test_multiple_addresses_stripped(self): + raw = " and " + result = sanitize_message(raw) + assert "0x" not in result + + +class TestAffectedModulesUseSanitize: + """Verify that affected modules call sanitize_message (source-level).""" + + @pytest.mark.parametrize( + "module", + [ + "vllm.entrypoints.anthropic.api_router", + "vllm.entrypoints.anthropic.serving", + "vllm.entrypoints.speech_to_text.realtime.connection", + ], + ) + def test_module_calls_sanitize_message(self, module: str): + import importlib.util + from pathlib import Path + + spec = importlib.util.find_spec(module) + assert spec is not None and spec.origin is not None, ( + f"Cannot locate module {module}" + ) + source = Path(spec.origin).read_text() + assert "sanitize_message" in source, f"{module} does not call sanitize_message" + assert "import" in source and "sanitize_message" in source diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_fingerprint.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_fingerprint.py new file mode 100644 index 0000000000000000000000000000000000000000..6fc82b046a23deebc4996eb189635d4f13008455 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_fingerprint.py @@ -0,0 +1,76 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for ``system_fingerprint`` construction.""" + +from types import SimpleNamespace + +import pytest + +from vllm.entrypoints.serve.utils import fingerprint as fp + + +def _cfg(tp=1, pp=1, dp=1, ep=False, digest="a3b21f94deadbeef"): + c = SimpleNamespace( + parallel_config=SimpleNamespace( + tensor_parallel_size=tp, + pipeline_parallel_size=pp, + data_parallel_size=dp, + enable_expert_parallel=ep, + ) + ) + c.compute_hash = lambda: digest # type: ignore[attr-defined] + return c + + +@pytest.fixture(autouse=True) +def _reset(): + fp.set_default_fingerprint_mode("full") + yield + fp.set_default_fingerprint_mode("full") + + +def test_four_modes_produce_expected_shapes(): + from vllm import __version__ as v + + cfg = _cfg(tp=8, ep=True) + + assert fp.build_system_fingerprint(cfg, "full") == (f"vllm-{v}-tp8-ep-a3b21f94") + assert fp.build_system_fingerprint(cfg, "hash") == f"vllm-{v}-a3b21f94" + assert fp.build_system_fingerprint(cfg, "custom", "my-fp") == "my-fp" + assert fp.build_system_fingerprint(cfg, "none") is None + + +def test_full_mode_emits_only_non_trivial_parallelism(): + from vllm import __version__ as v + + # Single-GPU: nothing between version and hash. + assert fp.build_system_fingerprint(_cfg(), "full") == f"vllm-{v}-a3b21f94" + # All parallelism axes. + assert ( + fp.build_system_fingerprint(_cfg(tp=8, pp=2, dp=4, ep=True), "full") + == f"vllm-{v}-tp8-pp2-dp4-ep-a3b21f94" + ) + + +def test_get_respects_set_default(): + cfg = _cfg(tp=8) + full = fp.get_system_fingerprint(cfg) + assert full == fp.get_system_fingerprint(cfg) + + fp.set_default_fingerprint_mode("hash") + hashed = fp.get_system_fingerprint(cfg) + assert hashed != full + assert "tp8" not in hashed + + fp.set_default_fingerprint_mode("custom", "deploy-42") + assert fp.get_system_fingerprint(cfg) == "deploy-42" + + fp.set_default_fingerprint_mode("none") + assert fp.get_system_fingerprint(cfg) is None + + +def test_compute_hash_failure_does_not_raise(): + cfg = _cfg() + cfg.compute_hash = lambda: (_ for _ in ()).throw(RuntimeError("boom")) + assert fp.build_system_fingerprint(cfg, "full").endswith("-nohash") + assert fp.build_system_fingerprint(cfg, "hash").endswith("-nohash") diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_request_logger.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_request_logger.py new file mode 100644 index 0000000000000000000000000000000000000000..999c0ad3d492a2d0076110506d57c33db958a8fc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_request_logger.py @@ -0,0 +1,248 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from unittest.mock import MagicMock, patch + +from vllm.entrypoints.serve.utils.request_logger import RequestLogger + + +def test_request_logger_log_outputs(): + """Test the new log_outputs functionality.""" + # Create a mock logger to capture log calls + mock_logger = MagicMock() + + with patch("vllm.entrypoints.serve.utils.request_logger.logger", mock_logger): + request_logger = RequestLogger(max_log_len=None) + + # Test basic output logging + request_logger.log_outputs( + request_id="test-123", + outputs="Hello, world!", + output_token_ids=[1, 2, 3, 4], + finish_reason="stop", + is_streaming=False, + delta=False, + ) + + mock_logger.info.assert_called_once() + call_args = mock_logger.info.call_args.args + assert "Generated response %s%s" in call_args[0] + assert call_args[1] == "test-123" + assert call_args[3] == "Hello, world!" + assert call_args[4] == [1, 2, 3, 4] + assert call_args[5] == "stop" + + +def test_request_logger_log_outputs_streaming_delta(): + """Test log_outputs with streaming delta mode.""" + mock_logger = MagicMock() + + with patch("vllm.entrypoints.serve.utils.request_logger.logger", mock_logger): + request_logger = RequestLogger(max_log_len=None) + + # Test streaming delta logging + request_logger.log_outputs( + request_id="test-456", + outputs="Hello", + output_token_ids=[1], + finish_reason=None, + is_streaming=True, + delta=True, + ) + + mock_logger.info.assert_called_once() + call_args = mock_logger.info.call_args.args + assert "Generated response %s%s" in call_args[0] + assert call_args[1] == "test-456" + assert call_args[2] == " (streaming delta)" + assert call_args[3] == "Hello" + assert call_args[4] == [1] + assert call_args[5] is None + + +def test_request_logger_log_outputs_streaming_complete(): + """Test log_outputs with streaming complete mode.""" + mock_logger = MagicMock() + + with patch("vllm.entrypoints.serve.utils.request_logger.logger", mock_logger): + request_logger = RequestLogger(max_log_len=None) + + # Test streaming complete logging + request_logger.log_outputs( + request_id="test-789", + outputs="Complete response", + output_token_ids=[1, 2, 3], + finish_reason="length", + is_streaming=True, + delta=False, + ) + + mock_logger.info.assert_called_once() + call_args = mock_logger.info.call_args.args + assert "Generated response %s%s" in call_args[0] + assert call_args[1] == "test-789" + assert call_args[2] == " (streaming complete)" + assert call_args[3] == "Complete response" + assert call_args[4] == [1, 2, 3] + assert call_args[5] == "length" + + +def test_request_logger_log_outputs_with_truncation(): + """Test log_outputs respects max_log_len setting.""" + mock_logger = MagicMock() + + with patch("vllm.entrypoints.serve.utils.request_logger.logger", mock_logger): + # Set max_log_len to 10 + request_logger = RequestLogger(max_log_len=10) + + # Test output truncation + long_output = "This is a very long output that should be truncated" + long_token_ids = list(range(20)) # 20 tokens + + request_logger.log_outputs( + request_id="test-truncate", + outputs=long_output, + output_token_ids=long_token_ids, + finish_reason="stop", + is_streaming=False, + delta=False, + ) + + mock_logger.info.assert_called_once() + call_args = mock_logger.info.call_args + + # Check that output was truncated to first 10 characters + logged_output = call_args[0][3] + assert logged_output == "This is a " + assert len(logged_output) == 10 + + # Check that token IDs were truncated to first 10 tokens + logged_token_ids = call_args[0][4] + assert logged_token_ids == list(range(10)) + assert len(logged_token_ids) == 10 + + +def test_request_logger_log_outputs_none_values(): + """Test log_outputs handles None values correctly.""" + mock_logger = MagicMock() + + with patch("vllm.entrypoints.serve.utils.request_logger.logger", mock_logger): + request_logger = RequestLogger(max_log_len=None) + + # Test with None output_token_ids + request_logger.log_outputs( + request_id="test-none", + outputs="Test output", + output_token_ids=None, + finish_reason="stop", + is_streaming=False, + delta=False, + ) + + mock_logger.info.assert_called_once() + call_args = mock_logger.info.call_args.args + assert "Generated response %s%s" in call_args[0] + assert call_args[1] == "test-none" + assert call_args[3] == "Test output" + assert call_args[4] is None + assert call_args[5] == "stop" + + +def test_request_logger_log_outputs_empty_output(): + """Test log_outputs handles empty output correctly.""" + mock_logger = MagicMock() + + with patch("vllm.entrypoints.serve.utils.request_logger.logger", mock_logger): + request_logger = RequestLogger(max_log_len=5) + + # Test with empty output + request_logger.log_outputs( + request_id="test-empty", + outputs="", + output_token_ids=[], + finish_reason="stop", + is_streaming=False, + delta=False, + ) + + mock_logger.info.assert_called_once() + call_args = mock_logger.info.call_args.args + assert "Generated response %s%s" in call_args[0] + assert call_args[1] == "test-empty" + assert call_args[3] == "" + assert call_args[4] == [] + assert call_args[5] == "stop" + + +def test_request_logger_log_outputs_integration(): + """Test that log_outputs can be called alongside log_inputs.""" + mock_logger = MagicMock() + + with patch("vllm.entrypoints.serve.utils.request_logger.logger", mock_logger): + request_logger = RequestLogger(max_log_len=None) + + # Test that both methods can be called without interference + request_logger.log_inputs( + request_id="test-integration", + prompt="Test prompt", + prompt_token_ids=[1, 2, 3], + prompt_embeds=None, + params=None, + lora_request=None, + ) + + request_logger.log_outputs( + request_id="test-integration", + outputs="Test output", + output_token_ids=[4, 5, 6], + finish_reason="stop", + is_streaming=False, + delta=False, + ) + + # Should have been called twice - once for inputs, once for outputs + assert mock_logger.info.call_count == 2 + + # Check that the calls were made with correct patterns + input_call = mock_logger.info.call_args_list[0][0] + output_call = mock_logger.info.call_args_list[1][0] + + assert "Received request %s" in input_call[0] + assert input_call[1] == "test-integration" + + assert "Generated response %s%s" in output_call[0] + assert output_call[1] == "test-integration" + + +def test_streaming_complete_logs_full_text_content(): + """Test that streaming complete logging includes + full accumulated text, not just token count.""" + mock_logger = MagicMock() + + with patch("vllm.entrypoints.serve.utils.request_logger.logger", mock_logger): + request_logger = RequestLogger(max_log_len=None) + + # Test with actual content instead of token count format + full_response = "This is a complete response from streaming" + request_logger.log_outputs( + request_id="test-streaming-full-text", + outputs=full_response, + output_token_ids=None, + finish_reason="streaming_complete", + is_streaming=True, + delta=False, + ) + + mock_logger.info.assert_called_once() + call_args = mock_logger.info.call_args.args + + # Verify the logged output is the full text, not a token count format + logged_output = call_args[3] + assert logged_output == full_response + assert "tokens>" not in logged_output + assert "streaming_complete" not in logged_output + + # Verify other parameters + assert call_args[1] == "test-streaming-full-text" + assert call_args[2] == " (streaming complete)" + assert call_args[5] == "streaming_complete" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_server_utils.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_server_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..21561c42e04f2f6011e156c3fc774c1a90c69cbe --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_server_utils.py @@ -0,0 +1,102 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests that validation_exception_handler populates the `param` field +in its error response using the Pydantic error's `loc`, even when no +custom VLLMValidationError context is present. + +Previously, `param` was only populated for errors carrying a custom +VLLMValidationError in their Pydantic `ctx`. Plain validation failures +(missing fields, wrong types) left `param` as None, even though the +field name was readily available from `error['loc']`. +""" + +import json +from types import SimpleNamespace + +import pytest +from fastapi.exceptions import RequestValidationError + +from vllm.entrypoints.serve.utils.server_utils import ( + clean_loc_for_param, + validation_exception_handler, +) + + +def _fake_request(log_error_stack: bool = False) -> SimpleNamespace: + """Minimal stand-in for a FastAPI Request - just enough for the + handler to read req.app.state.args.log_error_stack.""" + return SimpleNamespace( + app=SimpleNamespace( + state=SimpleNamespace(args=SimpleNamespace(log_error_stack=log_error_stack)) + ), + state=SimpleNamespace(), # no request_metadata -> hasattr(...) is False + ) + + +class TestValidationErrorParamFallback: + """Ensure `param` falls back to the Pydantic error's `loc` when no + custom VLLMValidationError context is present.""" + + @pytest.mark.parametrize( + ("error_type", "msg"), + [ + ("missing", "Field required"), + ("list_type", "Input should be a valid list"), + ], + ids=["missing-field", "wrong-type"], + ) + @pytest.mark.asyncio + async def test_param_falls_back_to_loc(self, error_type: str, msg: str): + errors = [{"type": error_type, "loc": ("body", "messages"), "msg": msg}] + exc = RequestValidationError(errors) + + response = await validation_exception_handler(_fake_request(), exc) + body = json.loads(response.body) + + assert body["error"]["param"] == "body.messages" + + @pytest.mark.asyncio + async def test_param_fallback_does_not_crash_on_non_dict_error(self): + """Schemathesis fuzzing found that errors[0] isn't always a dict. + The fallback must not crash in that case - it should just leave + param as None instead of raising.""" + exc = RequestValidationError(["some unexpected non-dict error"]) + + response = await validation_exception_handler(_fake_request(), exc) + body = json.loads(response.body) + + assert body["error"]["param"] is None + + +class TestCleanLocForParam: + """Guards against PR #1's naive dot-joined `loc` fallback leaking + Pydantic-internal wrapper/union-branch markers into `param`, e.g. + 'body.function-wrap[__log_extra_fields__()].prompt' instead of the + clean 'body.prompt' an API consumer would recognize. + """ + + @pytest.mark.parametrize( + "loc,expected", + [ + (("body", "prompt"), "body.prompt"), + (("body", "messages", 2, "content"), "body.messages.2.content"), + ( + ("body", "function-wrap[__log_extra_fields__()]", "prompt"), + "body.prompt", + ), + (("body", "stop", "str"), "body.stop"), + (("body", "stop", "list[str]"), "body.stop"), + ( + ("body", "prompt", "list[constrained-int]"), + "body.prompt", + ), + ], + ) + def test_strips_internal_markers(self, loc, expected): + assert clean_loc_for_param(loc) == expected + + def test_all_internal_falls_back_to_raw_join(self): + """If every loc segment looks internal, fall back to the raw + dot-join rather than returning an empty string.""" + loc = ("function-wrap[__log_extra_fields__()]",) + assert clean_loc_for_param(loc) == "function-wrap[__log_extra_fields__()]" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_ssl_cert_refresher.py b/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_ssl_cert_refresher.py new file mode 100644 index 0000000000000000000000000000000000000000..4d2d4dfefdf1f781ec50ad7432ad9158e7156bda --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/serve/utils/test_ssl_cert_refresher.py @@ -0,0 +1,102 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import asyncio +import tempfile +from pathlib import Path +from ssl import SSLContext + +import pytest + +from vllm.entrypoints.serve.utils.ssl import SSLCertRefresher + + +class MockSSLContext(SSLContext): + def __init__(self): + self.load_cert_chain_count = 0 + self.load_ca_count = 0 + + def load_cert_chain( + self, + certfile, + keyfile=None, + password=None, + ): + self.load_cert_chain_count += 1 + + def load_verify_locations( + self, + cafile=None, + capath=None, + cadata=None, + ): + self.load_ca_count += 1 + + +def create_file() -> str: + with tempfile.NamedTemporaryFile(dir="/tmp", delete=False) as f: + return f.name + + +def touch_file(path: str) -> None: + Path(path).touch() + + +async def wait_for_counts( + ssl_context: MockSSLContext, + *, + cert_chain_count: int, + ca_count: int, + timeout: float = 5.0, +) -> None: + deadline = asyncio.get_running_loop().time() + timeout + while True: + if ( + ssl_context.load_cert_chain_count >= cert_chain_count + and ssl_context.load_ca_count >= ca_count + ): + return + + if asyncio.get_running_loop().time() >= deadline: + assert ssl_context.load_cert_chain_count >= cert_chain_count + assert ssl_context.load_ca_count >= ca_count + + await asyncio.sleep(0.05) + + +@pytest.mark.asyncio +async def test_ssl_refresher(): + ssl_context = MockSSLContext() + key_path = create_file() + cert_path = create_file() + ca_path = create_file() + ssl_refresher = SSLCertRefresher(ssl_context, key_path, cert_path, ca_path) + await asyncio.sleep(1) + assert ssl_context.load_cert_chain_count == 0 + assert ssl_context.load_ca_count == 0 + + touch_file(key_path) + await wait_for_counts( + ssl_context, + cert_chain_count=1, + ca_count=0, + ) + assert ssl_context.load_ca_count == 0 + + touch_file(cert_path) + touch_file(ca_path) + await wait_for_counts( + ssl_context, + cert_chain_count=2, + ca_count=1, + ) + + ssl_refresher.stop() + await asyncio.sleep(0) + cert_chain_count = ssl_context.load_cert_chain_count + ca_count = ssl_context.load_ca_count + + touch_file(cert_path) + touch_file(ca_path) + await asyncio.sleep(1) + assert ssl_context.load_cert_chain_count == cert_chain_count + assert ssl_context.load_ca_count == ca_count diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/conftest.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..fcb9310ad92afd33dabb5e344978dd333f4514b6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/conftest.py @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest + +from vllm.assets.audio import AudioAsset + + +def add_attention_backend(server_args, attention_config): + """Append attention backend CLI arg if specified. + + Args: + server_args: List of server arguments to extend in-place. + attention_config: Dict with 'backend' key, or None. + """ + if attention_config and "backend" in attention_config: + server_args.extend(["--attention-backend", attention_config["backend"]]) + + +@pytest.fixture(scope="module") +def rocm_aiter_fa_attention(): + """Return attention config for transcription/translation tests on ROCm. + + On ROCm, audio tests require ROCM_AITER_FA attention backend. + """ + from vllm.platforms import current_platform + + if current_platform.is_rocm(): + return {"backend": "ROCM_AITER_FA"} + return None + + +@pytest.fixture +def mary_had_lamb(): + path = AudioAsset("mary_had_lamb").get_local_path() + with open(str(path), "rb") as f: + yield f + + +@pytest.fixture +def winning_call(): + path = AudioAsset("winning_call").get_local_path() + with open(str(path), "rb") as f: + yield f + + +@pytest.fixture +def foscolo(): + # Test translation it->en + path = AudioAsset("azacinto_foscolo").get_local_path() + with open(str(path), "rb") as f: + yield f diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/correctness/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/correctness/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/correctness/test_transcription_api_correctness.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/correctness/test_transcription_api_correctness.py new file mode 100644 index 0000000000000000000000000000000000000000..bfe8f3600010bc6d1fa288fd51aa46ee7e8ac702 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/correctness/test_transcription_api_correctness.py @@ -0,0 +1,400 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Evaluate Transcription API correctness by computing Word Error Rate (WER) +on a given ASR dataset. When provided, it will also compare the WER against +a baseline. +This simulates real work usage of the API and makes sure that the frontend and +AsyncLLMEngine are working correctly. +""" + +import asyncio +import io +import time +from statistics import mean, median + +import pytest +import soundfile +import torch +from datasets import Audio, load_dataset +from evaluate import load +from transformers.models.whisper.english_normalizer import EnglishTextNormalizer + +from vllm.benchmarks.datasets.datasets import ASRDataset +from vllm.multimodal.audio import get_audio_duration +from vllm.tokenizers import get_tokenizer + +from ....models.registry import HF_EXAMPLE_MODELS +from ....utils import RemoteOpenAIServer + +# Tuned to prevent OOM on 18GB GPUs in transcription correctness tests. +MAX_SEQS_FOR_TRANSCRIPTION_TEST = 8 +GPU_UTIL_FOR_TRANSCRIPTION_TEST = 0.5 + + +def to_bytes(y, sr): + buffer = io.BytesIO() + soundfile.write(buffer, y, sr, format="WAV") + buffer.seek(0) + return buffer + + +def load_audio_sample(audio): + # Avoid torchcodec in CI by decoding dataset audio with soundfile. + if "array" in audio and "sampling_rate" in audio: + return audio["array"], audio["sampling_rate"] + + if audio.get("path"): + return soundfile.read(audio["path"], dtype="float32") + + if audio.get("bytes") is not None: + return soundfile.read(io.BytesIO(audio["bytes"]), dtype="float32") + + raise ValueError("Audio sample did not contain array, path, or bytes data") + + +# not all models have a normalizer so use the one from whisper as a standard option +normalizer_model_info = HF_EXAMPLE_MODELS.find_hf_info("openai/whisper-large-v3") +normalizer_tokenizer = get_tokenizer( + "openai/whisper-large-v3", + tokenizer_mode=normalizer_model_info.tokenizer_mode, + trust_remote_code=normalizer_model_info.trust_remote_code, +) +normalizer = EnglishTextNormalizer(normalizer_tokenizer.english_spelling_normalizer) + + +async def transcribe_audio(client, tokenizer, y, sr, extra_body=None): + # Send loaded audio directly instead of loading from disk, + # don't account for that time though + with to_bytes(y, sr) as f: + start_time = time.perf_counter() + transcription = await client.audio.transcriptions.create( + file=f, + model=tokenizer.name_or_path, + language="en", + temperature=0.0, + extra_body=extra_body, + ) + end_time = time.perf_counter() + # NOTE there's no streaming in transcriptions, can't measure ttft + latency = end_time - start_time + num_output_tokens = len( + tokenizer(transcription.text, add_special_tokens=False).input_ids + ) + return latency, num_output_tokens, transcription.text + + +async def bound_transcribe( + sem, client, tokenizer, audio, sr, reference, extra_body=None +): + # Use semaphore to limit concurrent requests. + async with sem: + result = await transcribe_audio( + client, tokenizer, audio, sr, extra_body=extra_body + ) + # Normalize *english* output/reference for evaluation. + out = normalizer(result[2]) + ref = normalizer(reference) + return result[:2] + (out, ref) + + +async def process_dataset(model, client, data, concurrent_request, extra_body=None): + sem = asyncio.Semaphore(concurrent_request) + + model_info = HF_EXAMPLE_MODELS.find_hf_info(model) + tokenizer = get_tokenizer( + model, + tokenizer_mode=model_info.tokenizer_mode, + trust_remote_code=model_info.trust_remote_code, + ) + + # Warmup call as the first `load_audio` server-side is quite slow. + audio, sr = load_audio_sample(data[0]["audio"]) + _ = await bound_transcribe(sem, client, tokenizer, audio, sr, "", extra_body) + + tasks: list[asyncio.Task] = [] + for sample in data: + audio, sr = load_audio_sample(sample["audio"]) + task = asyncio.create_task( + bound_transcribe( + sem, client, tokenizer, audio, sr, sample["text"], extra_body + ) + ) + tasks.append(task) + return await asyncio.gather(*tasks) + + +def print_performance_metrics(results, total_time): + latencies = [res[0] for res in results] + total_tokens = sum([res[1] for res in results]) + + total = len(results) + print(f"Total Requests: {total}") + print(f"Successful Requests: {len(latencies)}") + print(f"Average Latency: {mean(latencies):.4f} seconds") + print(f"Median Latency: {median(latencies):.4f} seconds") + perc = sorted(latencies)[int(len(latencies) * 0.95) - 1] + print(f"95th Percentile Latency: {perc:.4f} seconds") + # Throughput + req_throughput = len(latencies) / total_time + print(f"Estimated req_Throughput: {req_throughput:.2f} requests/s") + throughput = total_tokens / total_time + print(f"Estimated Throughput: {throughput:.2f} tok/s") + + +def add_duration(sample): + y, sr = load_audio_sample(sample["audio"]) + sample["duration_ms"] = get_audio_duration(y=y, sr=sr) * 1000 + return sample + + +def load_asr_dataset_rows(dataset_repo: str, split="validation", **hf_kwargs): + if dataset_repo in ASRDataset.SUPPORTED_DATASET_PATHS: + asr_dataset_kwargs = { + "dataset_path": dataset_repo, + "dataset_split": split, + "disable_shuffle": True, + "no_stream": True, + } + for key in ("dataset_subset", "hf_name", "trust_remote_code"): + if key in hf_kwargs: + asr_dataset_kwargs[key] = hf_kwargs[key] + return ASRDataset(**asr_dataset_kwargs).data + + return load_dataset(dataset_repo, split=split, **hf_kwargs) + + +def load_shortform_eval_dataset(dataset_repo: str, split="validation", **hf_kwargs): + ## Load and filter the dataset. + dataset = load_asr_dataset_rows(dataset_repo, split=split, **hf_kwargs) + dataset = dataset.cast_column("audio", Audio(decode=False)) + if "duration_ms" not in dataset.column_names: + # Compute duration to filter. + dataset = dataset.map(add_duration) + + # Whisper max supported duration. + dataset = dataset.filter(lambda example: example["duration_ms"] < 30000) + return dataset + + +def run_evaluation( + model: str, + client, + dataset, + max_concurrent_reqs: int, + n_examples: int = -1, + print_metrics: bool = True, + extra_body=None, +): + if n_examples > 0: + dataset = dataset.select(range(n_examples)) + start = time.perf_counter() + results = asyncio.run( + process_dataset( + model, client, dataset, max_concurrent_reqs, extra_body=extra_body + ) + ) + end = time.perf_counter() + total_time = end - start + print(f"Total Test Time: {total_time:.4f} seconds") + if print_metrics: + print_performance_metrics(results, total_time) + # Compute WER + predictions = [res[2] for res in results] + references = [res[3] for res in results] + wer = load("wer") + wer_score = 100 * wer.compute(references=references, predictions=predictions) + print("WER:", wer_score) + return wer_score + + +LONGFORM_DATASET_REPO = ASRDataset.EARNINGS22_CLEANED_DATASET +LONGFORM_DATASET_SPLIT = "test" +LONGFORM_NUM_SAMPLES = 6 + + +def load_longform_dataset(): + dataset = load_asr_dataset_rows( + LONGFORM_DATASET_REPO, + split=LONGFORM_DATASET_SPLIT, + ) + assert len(dataset) >= LONGFORM_NUM_SAMPLES + return dataset.select(range(LONGFORM_NUM_SAMPLES)) + + +async def transcribe_audio_path(client, tokenizer, audio_path: str, extra_body=None): + with open(audio_path, "rb") as f: + start_time = time.perf_counter() + transcription = await client.audio.transcriptions.create( + file=f, + model=tokenizer.name_or_path, + language="en", + temperature=0.0, + extra_body=extra_body, + ) + end_time = time.perf_counter() + + latency = end_time - start_time + num_output_tokens = len( + tokenizer(transcription.text, add_special_tokens=False).input_ids + ) + return latency, num_output_tokens, transcription.text + + +async def bound_transcribe_path( + sem, client, tokenizer, audio_path, reference, extra_body=None +): + async with sem: + result = await transcribe_audio_path( + client, tokenizer, audio_path, extra_body=extra_body + ) + out = normalizer(result[2]) + ref = normalizer(reference) + return result[:2] + (out, ref) + + +async def process_longform_dataset( + model, client, data, concurrent_request, extra_body=None +): + sem = asyncio.Semaphore(concurrent_request) + + model_info = HF_EXAMPLE_MODELS.find_hf_info(model) + tokenizer = get_tokenizer( + model, + tokenizer_mode=model_info.tokenizer_mode, + trust_remote_code=model_info.trust_remote_code, + ) + + warmup_path = data[0]["audio"]["path"] + _ = await bound_transcribe_path(sem, client, tokenizer, warmup_path, "", extra_body) + + tasks: list[asyncio.Task] = [] + for sample in data: + audio_path = sample["audio"]["path"] + task = asyncio.create_task( + bound_transcribe_path( + sem, client, tokenizer, audio_path, sample["text"], extra_body + ) + ) + tasks.append(task) + return await asyncio.gather(*tasks) + + +def run_longform_evaluation( + model: str, + client, + dataset, + max_concurrent_reqs: int, + print_metrics: bool = True, + extra_body=None, +): + start = time.perf_counter() + results = asyncio.run( + process_longform_dataset( + model, client, dataset, max_concurrent_reqs, extra_body=extra_body + ) + ) + end = time.perf_counter() + total_time = end - start + print(f"Total Test Time: {total_time:.4f} seconds") + if print_metrics: + print_performance_metrics(results, total_time) + + predictions = [res[2] for res in results] + references = [res[3] for res in results] + wer = load("wer") + wer_score = 100 * wer.compute(references=references, predictions=predictions) + print("WER:", wer_score) + return wer_score + + +# alternatives "openai/whisper-large-v2", "openai/whisper-large-v3-turbo".. +# NOTE: Expected WER measured with equivalent hf.transformers args: +# whisper-large-v3 + esb-datasets-earnings22-validation-tiny-filtered. +@pytest.mark.parametrize( + "model_config", + [ + ("openai/whisper-large-v3", 12.744980), + # CohereASR is used to test the variable encoder length code paths + ("CohereLabs/cohere-transcribe-03-2026", 11.92), + ], +) +# Original dataset is 20GB+ in size, hence we use a pre-filtered slice. +@pytest.mark.parametrize( + "dataset_repo", ["D4nt3/esb-datasets-earnings22-validation-tiny-filtered"] +) +def test_wer_correctness( + model_config, dataset_repo, n_examples=-1, max_concurrent_request=None +): + model_name, expected_wer = model_config + model_info = HF_EXAMPLE_MODELS.find_hf_info(model_name) + server_args = [ + "--enforce-eager", + f"--tokenizer_mode={model_info.tokenizer_mode}", + f"--max_num_seqs={MAX_SEQS_FOR_TRANSCRIPTION_TEST}", + f"--gpu_memory_utilization={GPU_UTIL_FOR_TRANSCRIPTION_TEST}", + ] + if model_info.trust_remote_code: + server_args.append("--trust-remote-code") + with RemoteOpenAIServer( + model_name, + server_args, + ) as remote_server: + dataset = load_shortform_eval_dataset(dataset_repo) + + if not max_concurrent_request: + # No max concurrency + max_concurrent_request = n_examples if n_examples > 0 else len(dataset) + + client = remote_server.get_async_client() + wer = run_evaluation( + model_name, + client, + dataset, + max_concurrent_request, + n_examples, + ) + + print(f"Expected WER: {expected_wer}, Actual WER: {wer}") + + if expected_wer: + torch.testing.assert_close(wer, expected_wer, atol=1e-1, rtol=1e-2) + + +# 14-22mins of 6 audio samples of total ~115 mins and just 37MB. +# checks for long audio transcription correctness and RMS split. +@pytest.mark.parametrize( + "model_config", + [("openai/whisper-large-v3", 9.5)], +) +def test_long_audio_wer_correctness(model_config): + model_name, expected_wer = model_config + model_info = HF_EXAMPLE_MODELS.find_hf_info(model_name) + server_args = [ + f"--tokenizer_mode={model_info.tokenizer_mode}", + ] + + if model_info.trust_remote_code: + server_args.append("--trust-remote-code") + + # 1800 seconds is 30 minutes + env_dict = { + "VLLM_MAX_AUDIO_DECODE_DURATION_S": "1800", + } + + with RemoteOpenAIServer( + model_name, + server_args, + env_dict=env_dict, + ) as remote_server: + dataset = load_longform_dataset() + client = remote_server.get_async_client() + wer = run_longform_evaluation( + model=model_name, + client=client, + dataset=dataset, + max_concurrent_reqs=LONGFORM_NUM_SAMPLES, + ) + + print(f"Expected WER: {expected_wer}, Actual WER: {wer}") + torch.testing.assert_close(wer, expected_wer, atol=1e-1, rtol=1e-2) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/realtime/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/realtime/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/realtime/test_realtime_validation.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/realtime/test_realtime_validation.py new file mode 100644 index 0000000000000000000000000000000000000000..9fe8cb89142ef5b477b2266d46f5aa05d9ee1ebc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/realtime/test_realtime_validation.py @@ -0,0 +1,336 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import asyncio +import json +import warnings + +import numpy as np +import pybase64 as base64 +import pytest +import websockets + +from tests.entrypoints.speech_to_text.conftest import add_attention_backend +from tests.utils import ROCM_ENV_OVERRIDES, ROCM_EXTRA_ARGS, RemoteOpenAIServer +from vllm.assets.audio import AudioAsset +from vllm.multimodal.media.audio import load_audio + +# Increase engine iteration timeout for ROCm where first-use JIT compilation +# can exceed the default 60s, causing a silent deadlock in feed_tokens. +REALTIME_ENV_OVERRIDES = { + **ROCM_ENV_OVERRIDES, + "VLLM_ENGINE_ITERATION_TIMEOUT_S": "600", +} + +MISTRAL_FORMAT_ARGS = [ + "--tokenizer_mode", + "mistral", + "--config_format", + "mistral", + "--load_format", + "mistral", +] + ROCM_EXTRA_ARGS + +MODEL_NAME = "mistralai/Voxtral-Mini-4B-Realtime-2602" + + +def _get_websocket_url(server: RemoteOpenAIServer) -> str: + """Convert HTTP URL to WebSocket URL for realtime endpoint.""" + http_url = server.url_root + ws_url = http_url.replace("http://", "ws://") + return f"{ws_url}/v1/realtime" + + +async def receive_event(ws, timeout: float = 60.0) -> dict: + """Receive and parse JSON event from WebSocket.""" + message = await asyncio.wait_for(ws.recv(), timeout=timeout) + return json.loads(message) + + +async def send_event(ws, event: dict) -> None: + """Send JSON event to WebSocket.""" + await ws.send(json.dumps(event)) + + +@pytest.fixture +def mary_had_lamb_audio_chunks() -> list[str]: + """Audio split into ~1 second chunks for streaming.""" + path = AudioAsset("mary_had_lamb").get_local_path() + audio, _ = load_audio(str(path), sr=16000, mono=True) + + # Split into ~0.1 second chunks (1600 samples at 16kHz) + chunk_size = 1600 + chunks = [] + for i in range(0, len(audio), chunk_size): + chunk = audio[i : i + chunk_size] + chunk_int16 = (chunk * 32767).astype(np.int16) + chunk_bytes = chunk_int16.tobytes() + chunks.append(base64.b64encode(chunk_bytes).decode("utf-8")) + + return chunks + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_multi_chunk_streaming( + model_name, mary_had_lamb_audio_chunks, rocm_aiter_fa_attention +): + """Test streaming multiple audio chunks before committing.""" + server_args = ["--enforce-eager", "--max-model-len", "2048"] + + if model_name.startswith("mistralai"): + server_args += MISTRAL_FORMAT_ARGS + + add_attention_backend(server_args, rocm_aiter_fa_attention) + + with RemoteOpenAIServer( + model_name, server_args, env_dict=REALTIME_ENV_OVERRIDES + ) as remote_server: + ws_url = _get_websocket_url(remote_server) + async with websockets.connect(ws_url) as ws: + # Receive session.created + event = await receive_event(ws, timeout=30.0) + assert event["type"] == "session.created" + + await send_event(ws, {"type": "session.update", "model": model_name}) + + # Wait for the server to acknowledge the session update. + try: + while True: + event = await receive_event(ws, timeout=5.0) + if event["type"] == "session.updated": + break + except TimeoutError: + warnings.warn( + f"session.updated not received within {5.0}s after " + "session.update. The server may not implement this event.", + stacklevel=2, + ) + + # (ROCm) Warm-up: send a non-final commit (required to start + # transcription) with a small audio chunk to trigger aiter + # compilation on first use. + await send_event(ws, {"type": "input_audio_buffer.commit"}) + await send_event( + ws, + { + "type": "input_audio_buffer.append", + "audio": mary_had_lamb_audio_chunks[0], + }, + ) + await send_event(ws, {"type": "input_audio_buffer.commit", "final": True}) + + # (ROCm) Drain all warm-up responses with generous timeout for + # JIT compilation + warmup_done = False + while not warmup_done: + event = await receive_event(ws, timeout=600.0) + if event["type"] in ("transcription.done", "error"): + warmup_done = True + + # Now send the real test audio + await send_event(ws, {"type": "input_audio_buffer.commit"}) + + # Send multiple audio chunks + for chunk in mary_had_lamb_audio_chunks: + await send_event( + ws, {"type": "input_audio_buffer.append", "audio": chunk} + ) + + # Send commit to end + await send_event(ws, {"type": "input_audio_buffer.commit", "final": True}) + + # Collect transcription deltas + full_text = "" + done_received = False + + while not done_received: + event = await receive_event(ws, timeout=60.0) + + if event["type"] == "transcription.delta": + full_text += event["delta"] + elif event["type"] == "transcription.done": + done_received = True + assert "text" in event + elif event["type"] == "error": + pytest.fail(f"Received error: {event}") + + # Verify transcription contains expected content + assert event["type"] == "transcription.done" + assert event["text"] == full_text + assert full_text == ( + " First words I spoke in the original phonograph." + " A little piece of practical poetry. Mary had a little lamb," + " it sleeps with quite a flow, and everywhere that Mary went," + " the lamb was sure to go." + ) or full_text == ( + " First words I spoke in the original phonograph." + " A little piece of practical poetry. Mary had a little lamb," + " it squeaked with quite a flow, and everywhere that Mary went," + " the lamb was sure to go." + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_empty_commit_does_not_crash_engine( + model_name, mary_had_lamb_audio_chunks, rocm_aiter_fa_attention +): + """Test that committing without audio does not crash the engine. + + Regression test for https://github.com/vllm-project/vllm/issues/34532. + An empty commit (no prior input_audio_buffer.append) used to trigger + ``AssertionError: For realtime you must provide a multimodal_embedding + at every step`` which killed the entire engine process, disconnecting + every connected client. + """ + server_args = ["--enforce-eager", "--max-model-len", "2048"] + + if model_name.startswith("mistralai"): + server_args += MISTRAL_FORMAT_ARGS + + add_attention_backend(server_args, rocm_aiter_fa_attention) + + with RemoteOpenAIServer( + model_name, server_args, env_dict=REALTIME_ENV_OVERRIDES + ) as remote_server: + ws_url = _get_websocket_url(remote_server) + + # --- First connection: empty commit (no audio appended) ---------- + async with websockets.connect(ws_url) as ws: + event = await receive_event(ws, timeout=30.0) + assert event["type"] == "session.created" + + await send_event(ws, {"type": "session.update", "model": model_name}) + + try: + while True: + event = await receive_event(ws, timeout=5.0) + if event["type"] == "session.updated": + break + except TimeoutError: + warnings.warn( + f"session.updated not received within {5.0}s after " + "session.update. The server may not implement this event.", + stacklevel=2, + ) + + # Start generation without sending any audio + await send_event(ws, {"type": "input_audio_buffer.commit"}) + + # Immediately signal end-of-audio + await send_event(ws, {"type": "input_audio_buffer.commit", "final": True}) + + # We should get *some* response (error or empty transcription), + # but the engine must NOT crash. + # (ROCm) Use generous timeout for first request (aiter JIT compilation) + event = await receive_event(ws, timeout=360.0) + assert event["type"] in ( + "error", + "transcription.done", + "transcription.delta", + ) + + # --- Second connection: normal transcription --------------------- + # Verifies the engine is still alive after the empty commit above. + async with websockets.connect(ws_url) as ws: + event = await receive_event(ws, timeout=30.0) + assert event["type"] == "session.created" + + await send_event(ws, {"type": "session.update", "model": model_name}) + + try: + while True: + event = await receive_event(ws, timeout=5.0) + if event["type"] == "session.updated": + break + except TimeoutError: + warnings.warn( + f"session.updated not received within {5.0}s after " + "session.update. The server may not implement this event.", + stacklevel=2, + ) + + # Start transcription + await send_event(ws, {"type": "input_audio_buffer.commit"}) + + for chunk in mary_had_lamb_audio_chunks: + await send_event( + ws, {"type": "input_audio_buffer.append", "audio": chunk} + ) + + await send_event(ws, {"type": "input_audio_buffer.commit", "final": True}) + + done_received = False + while not done_received: + event = await receive_event(ws, timeout=60.0) + if event["type"] == "transcription.done": + done_received = True + elif event["type"] == "error": + pytest.fail(f"Engine error after empty commit: {event}") + assert done_received + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_session_update_invalid_model_returns_error( + model_name, rocm_aiter_fa_attention +): + """Test that session.update with an invalid model returns an error.""" + server_args = ["--enforce-eager", "--max-model-len", "2048"] + + if model_name.startswith("mistralai"): + server_args += MISTRAL_FORMAT_ARGS + + add_attention_backend(server_args, rocm_aiter_fa_attention) + + with RemoteOpenAIServer( + model_name, server_args, env_dict=REALTIME_ENV_OVERRIDES + ) as remote_server: + ws_url = _get_websocket_url(remote_server) + async with websockets.connect(ws_url) as ws: + event = await receive_event(ws, timeout=30.0) + assert event["type"] == "session.created" + + # Send session.update with a model that doesn't exist + await send_event( + ws, + {"type": "session.update", "model": "nonexistent-model"}, + ) + + event = await receive_event(ws, timeout=10.0) + assert event["type"] == "error" + assert "nonexistent-model" in event["error"] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("model_name", [MODEL_NAME]) +async def test_commit_without_session_update_returns_error( + model_name, rocm_aiter_fa_attention +): + """Test that committing before validating the model returns an error + and does not fall through to processing.""" + server_args = ["--enforce-eager", "--max-model-len", "2048"] + + if model_name.startswith("mistralai"): + server_args += MISTRAL_FORMAT_ARGS + + add_attention_backend(server_args, rocm_aiter_fa_attention) + + with RemoteOpenAIServer( + model_name, server_args, env_dict=REALTIME_ENV_OVERRIDES + ) as remote_server: + ws_url = _get_websocket_url(remote_server) + async with websockets.connect(ws_url) as ws: + event = await receive_event(ws, timeout=30.0) + assert event["type"] == "session.created" + + # Send commit without sending session.update first + await send_event( + ws, + {"type": "input_audio_buffer.commit", "final": True}, + ) + + event = await receive_event(ws, timeout=10.0) + assert event["type"] == "error" + assert "model_not_validated" in event.get("code", "") diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/test_speech_to_text_cancellation.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/test_speech_to_text_cancellation.py new file mode 100644 index 0000000000000000000000000000000000000000..dc7c6db81c3ec3904c26c23246c4710ba10f852d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/test_speech_to_text_cancellation.py @@ -0,0 +1,191 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import asyncio +from types import SimpleNamespace +from unittest.mock import AsyncMock, Mock + +import pytest + +from vllm.entrypoints.speech_to_text.base.serving import SpeechToTextBaseServing +from vllm.entrypoints.speech_to_text.transcription.protocol import TranscriptionResponse + + +async def _never_finishes(): + await asyncio.Event().wait() + yield + + +async def _records_start_then_never_finishes(started_request_ids, request_id): + started_request_ids.append(request_id) + await asyncio.Event().wait() + yield + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("engine_inputs", "expected_request_ids"), + [ + ([{"prompt": "chunk"}], ["transcribe-outer-request"]), + ( + [{"prompt": "chunk-0"}, {"prompt": "chunk-1"}], + ["transcribe-outer-request-0", "transcribe-outer-request-1"], + ), + ], +) +async def test_non_streaming_cancel_aborts_engine_requests( + engine_inputs, expected_request_ids +): + engine_client = SimpleNamespace( + errored=False, + generate=Mock(side_effect=lambda *_args, **_kwargs: _never_finishes()), + abort=AsyncMock(), + is_tracing_enabled=AsyncMock(return_value=False), + ) + + server = SpeechToTextBaseServing.__new__(SpeechToTextBaseServing) + server.engine_client = engine_client + server.task_type = "transcribe" + server.models = SimpleNamespace(model_name=lambda: "audio") + server.model_config = SimpleNamespace(max_model_len=1024) + server.model_cls = SimpleNamespace(no_space_languages=set()) + server.default_sampling_params = {} + server.asr_config = SimpleNamespace(max_audio_clip_s=30) + server._check_model = AsyncMock(return_value=None) + server._maybe_get_adapters = Mock(return_value=None) + server._preprocess_speech_to_text = AsyncMock(return_value=(engine_inputs, 40.0)) + server._log_inputs = Mock() + + request = SimpleNamespace( + model="audio", + response_format="json", + stream=False, + use_beam_search=False, + max_completion_tokens=None, + language="en", + prompt="", + to_sampling_params=Mock(return_value=object()), + ) + raw_request = SimpleNamespace( + headers={"X-Request-Id": "outer-request"}, + state=SimpleNamespace(), + ) + + task = asyncio.create_task( + server._create_speech_to_text( + audio_data=b"audio", + request=request, + raw_request=raw_request, + response_class=TranscriptionResponse, + stream_generator_method=Mock(), + ) + ) + await asyncio.sleep(0) + + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + generated_request_ids = [ + call.args[2] for call in engine_client.generate.call_args_list + ] + assert generated_request_ids == expected_request_ids + engine_client.abort.assert_awaited_once_with(expected_request_ids) + + +@pytest.mark.asyncio +async def test_non_streaming_cancel_advances_all_chunk_generators(): + started_request_ids: list[str] = [] + engine_client = SimpleNamespace( + errored=False, + generate=Mock( + side_effect=lambda *_args, **_kwargs: _records_start_then_never_finishes( + started_request_ids, _args[2] + ) + ), + abort=AsyncMock(), + is_tracing_enabled=AsyncMock(return_value=False), + ) + + engine_inputs = [ + {"prompt": "chunk-0"}, + {"prompt": "chunk-1"}, + {"prompt": "chunk-2"}, + ] + server = SpeechToTextBaseServing.__new__(SpeechToTextBaseServing) + server.engine_client = engine_client + server.task_type = "transcribe" + server.models = SimpleNamespace(model_name=lambda: "audio") + server.model_config = SimpleNamespace(max_model_len=1024) + server.model_cls = SimpleNamespace(no_space_languages=set()) + server.default_sampling_params = {} + server.asr_config = SimpleNamespace(max_audio_clip_s=30) + server._check_model = AsyncMock(return_value=None) + server._maybe_get_adapters = Mock(return_value=None) + server._preprocess_speech_to_text = AsyncMock(return_value=(engine_inputs, 90.0)) + server._log_inputs = Mock() + + request = SimpleNamespace( + model="audio", + response_format="json", + stream=False, + use_beam_search=False, + max_completion_tokens=None, + language="en", + prompt="", + to_sampling_params=Mock(return_value=object()), + ) + raw_request = SimpleNamespace( + headers={"X-Request-Id": "outer-request"}, + state=SimpleNamespace(), + ) + + task = asyncio.create_task( + server._create_speech_to_text( + audio_data=b"audio", + request=request, + raw_request=raw_request, + response_class=TranscriptionResponse, + stream_generator_method=Mock(), + ) + ) + await asyncio.sleep(0.01) + + expected_request_ids = [ + "transcribe-outer-request-0", + "transcribe-outer-request-1", + "transcribe-outer-request-2", + ] + assert set(started_request_ids) == set(expected_request_ids) + + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + +@pytest.mark.asyncio +async def test_language_detection_cancel_aborts_engine_request(): + engine_client = SimpleNamespace( + generate=Mock(return_value=_never_finishes()), + abort=AsyncMock(), + ) + + server = SpeechToTextBaseServing.__new__(SpeechToTextBaseServing) + server.engine_client = engine_client + server.asr_config = SimpleNamespace() + server.tokenizer = Mock() + server.model_cls = SimpleNamespace( + get_language_detection_prompt=Mock(return_value={"prompt": "detect"}), + get_language_token_ids=Mock(return_value=[1]), + parse_language_detection_output=Mock(), + ) + + request_id = "transcribe-outer-request-lang_detect" + task = asyncio.create_task(server._detect_language(Mock(), request_id)) + await asyncio.sleep(0) + + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + + engine_client.abort.assert_awaited_once_with(request_id) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/test_upload_size_limit.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/test_upload_size_limit.py new file mode 100644 index 0000000000000000000000000000000000000000..4e4474f81d8f4e852efb503f22c6951a89695c26 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/test_upload_size_limit.py @@ -0,0 +1,142 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Regression tests for the speech-to-text upload size pre-check. + +These tests verify that over-limit audio uploads are rejected *before* +the full file is materialized into memory, closing the vulnerability +where vLLM would allocate memory proportional to an oversized upload +before enforcing the VLLM_MAX_AUDIO_CLIP_FILESIZE_MB limit. +""" + +from unittest.mock import AsyncMock, patch + +import pytest + +from vllm.entrypoints.speech_to_text.base.utils import read_upload_with_limit +from vllm.exceptions import VLLMValidationError + + +def _make_upload_file(data: bytes, *, size: int | None = None) -> AsyncMock: + """Create a mock UploadFile that yields data in chunks.""" + mock = AsyncMock() + mock.size = size + + offset = 0 + + async def _read(n: int = -1): + nonlocal offset + if n <= 0: + chunk = data[offset:] + offset = len(data) + return chunk + chunk = data[offset : offset + n] + offset += len(chunk) + return chunk + + mock.read = AsyncMock(side_effect=_read) + return mock + + +@pytest.mark.asyncio +async def test_rejects_oversized_upload_via_content_length(): + """File is rejected early when file.size exceeds the limit.""" + max_mb = 1 + oversized_bytes = max_mb * 1024 * 1024 + 1 + + upload = _make_upload_file(b"", size=oversized_bytes) + + with pytest.raises(VLLMValidationError, match="Maximum file size exceeded"): + await read_upload_with_limit(upload, max_size_mb=max_mb) + + upload.read.assert_not_called() + + +@pytest.mark.asyncio +async def test_rejects_oversized_upload_via_chunked_read(): + """File is rejected mid-read without materializing the full content.""" + max_mb = 1 + max_bytes = max_mb * 1024 * 1024 + oversized_data = b"\x00" * (max_bytes + 1024) + + upload = _make_upload_file(oversized_data, size=None) + + with pytest.raises(VLLMValidationError, match="Maximum file size exceeded"): + await read_upload_with_limit(upload, max_size_mb=max_mb) + + +@pytest.mark.asyncio +async def test_accepts_file_within_limit(): + """File within the limit is read successfully.""" + max_mb = 1 + data = b"\x00" * (512 * 1024) # 512 KiB, well under 1 MB + + upload = _make_upload_file(data, size=len(data)) + result = await read_upload_with_limit(upload, max_size_mb=max_mb) + + assert result == data + + +@pytest.mark.asyncio +async def test_accepts_file_at_exact_limit(): + """File exactly at the limit boundary is accepted.""" + max_mb = 1 + max_bytes = max_mb * 1024 * 1024 + data = b"\x00" * max_bytes + + upload = _make_upload_file(data, size=len(data)) + result = await read_upload_with_limit(upload, max_size_mb=max_mb) + + assert result == data + + +@pytest.mark.asyncio +async def test_rejects_at_one_byte_over_limit(): + """File one byte over the limit is rejected.""" + max_mb = 1 + max_bytes = max_mb * 1024 * 1024 + data = b"\x00" * (max_bytes + 1) + + upload = _make_upload_file(data, size=None) + + with pytest.raises(VLLMValidationError, match="Maximum file size exceeded"): + await read_upload_with_limit(upload, max_size_mb=max_mb) + + +@pytest.mark.asyncio +async def test_uses_env_default_when_no_limit_specified(): + """Uses VLLM_MAX_AUDIO_CLIP_FILESIZE_MB when max_size_mb is not given.""" + with patch("vllm.entrypoints.speech_to_text.base.utils.envs") as mock_envs: + mock_envs.VLLM_MAX_AUDIO_CLIP_FILESIZE_MB = 2 + max_bytes = 2 * 1024 * 1024 + oversized_data = b"\x00" * (max_bytes + 1) + + upload = _make_upload_file(oversized_data, size=None) + + with pytest.raises(VLLMValidationError, match="Maximum file size exceeded"): + await read_upload_with_limit(upload) + + +@pytest.mark.asyncio +async def test_chunked_read_does_not_fully_materialize(): + """Verify that for large oversized files, we stop reading early. + + The function reads in 64 KiB chunks and aborts once the accumulated + size exceeds the limit. We confirm that far fewer read calls were made + than would be required to fully materialize the file. + """ + max_mb = 1 + max_bytes = max_mb * 1024 * 1024 + large_size = max_bytes * 10 # 10x the limit + data = b"\x00" * large_size + + upload = _make_upload_file(data, size=None) + + with pytest.raises(VLLMValidationError): + await read_upload_with_limit(upload, max_size_mb=max_mb) + + chunk_size = 64 * 1024 + calls_for_full_read = large_size // chunk_size + 1 + calls_to_exceed_limit = max_bytes // chunk_size + 1 + actual_calls = upload.read.call_count + assert actual_calls <= calls_to_exceed_limit + 1 + assert actual_calls < calls_for_full_read diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_enable_force_include_usage.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_enable_force_include_usage.py new file mode 100644 index 0000000000000000000000000000000000000000..99d6e17391c5eedaf9802f9b52aeb7b62054bba6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_enable_force_include_usage.py @@ -0,0 +1,61 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import pytest_asyncio + +from tests.utils import RemoteOpenAIServer + + +@pytest.fixture(scope="module") +def transcription_server_with_force_include_usage(): + args = [ + # use half precision for speed and memory savings in CI environment + "--dtype", + "bfloat16", + "--max-num-seqs", + "4", + "--enforce-eager", + "--enable-force-include-usage", + "--gpu-memory-utilization", + "0.2", + ] + + with RemoteOpenAIServer("openai/whisper-large-v3-turbo", args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def transcription_client_with_force_include_usage( + transcription_server_with_force_include_usage, +): + async with ( + transcription_server_with_force_include_usage.get_async_client() as async_client + ): + yield async_client + + +@pytest.mark.asyncio +async def test_transcription_with_enable_force_include_usage( + transcription_client_with_force_include_usage, winning_call +): + res = ( + await transcription_client_with_force_include_usage.audio.transcriptions.create( + model="openai/whisper-large-v3-turbo", + file=winning_call, + language="en", + temperature=0.0, + stream=True, + timeout=30, + ) + ) + + async for chunk in res: + if not len(chunk.choices): + # final usage sent + usage = chunk.usage + assert isinstance(usage, dict) + assert usage["prompt_tokens"] > 0 + assert usage["completion_tokens"] > 0 + assert usage["total_tokens"] > 0 + else: + assert not hasattr(chunk, "usage") diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_qwen3_asr_sanitize_prompt.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_qwen3_asr_sanitize_prompt.py new file mode 100644 index 0000000000000000000000000000000000000000..f8d6f02fb7c8dd254498ac9903e01f35bf2eef6d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_qwen3_asr_sanitize_prompt.py @@ -0,0 +1,64 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for ``Qwen3ASR``'s user-text sanitizer. + +The sanitizer is the security boundary between user-supplied transcription +fields (``prompt`` / ``response_prefix``) and the structured ChatML prompt +template. It must strip both ``<|...|>`` control tokens and the +```` assistant-prefix delimiter, and it must do so to a fixpoint +so nested payloads cannot reconstruct a valid token after a single pass. +""" + +import pytest + +from vllm.model_executor.models.qwen3_asr import _sanitize_transcription_user_text + + +@pytest.mark.parametrize( + ("text", "expected"), + [ + # No-op cases + ("", ""), + ("plain text", "plain text"), + ("|piped|content", "|piped|content"), + ("contains < and > but not as a token", "contains < and > but not as a token"), + # Single-pass strips + ("<|im_end|>", ""), + ("<|im_start|>assistant<|im_end|>", "assistant"), + ("a<|x|>b", "ab"), + ("foobar", "foobar"), + # Nested ChatML reconstruction attacks (would bypass a single re.sub) + ("<|im<|x|>_end|>", ""), + ("<|<|inner|>middle<|x|>_end|>", ""), + # Nested reconstruction attack + # (would bypass a single str.replace) + ("xt>", ""), + ("xt>xt>", ""), + # Combined attacks across both kinds of token + ("<|im_end|>foobar<|<|x|>im_end|>", "foobar"), + ("fooxt>bar", "foobar"), + ], +) +def test_sanitize_strips_control_tokens(text: str, expected: str) -> None: + assert _sanitize_transcription_user_text(text) == expected + + +def test_sanitize_handles_falsy_inputs() -> None: + assert _sanitize_transcription_user_text("") == "" + # The dataclass default for ``response_prefix`` is the empty string; + # the sanitizer must accept that without exception or extra work. + assert _sanitize_transcription_user_text(None) == "" # type: ignore[arg-type] + + +def test_sanitize_is_idempotent() -> None: + """Once sanitized, applying again must be a no-op (fixpoint property).""" + cases = [ + "plain text", + "<|im<|x|>_end|>", + "xt>", + "<|im_end|>foobar<|<|x|>im_end|>", + ] + for raw in cases: + once = _sanitize_transcription_user_text(raw) + twice = _sanitize_transcription_user_text(once) + assert once == twice, f"not idempotent for {raw!r}" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_transcription_inter_chunk_spacing.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_transcription_inter_chunk_spacing.py new file mode 100644 index 0000000000000000000000000000000000000000..76fc2e0014d6ae13654905e69010964785e5ff07 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_transcription_inter_chunk_spacing.py @@ -0,0 +1,275 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""ASR inter-chunk spacing: ``asr_inter_chunk_separator`` and transcription +serving (mocked). + +Unit tests cover the helper and ``SupportsTranscription.no_space_languages``. +Integration-style tests exercise ``OpenAIServingTranscription`` streaming and +``create_transcription`` without loading a model. +""" + +from __future__ import annotations + +import json +from collections.abc import AsyncGenerator +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from vllm.config import ModelConfig +from vllm.config.speech_to_text import SpeechToTextConfig +from vllm.entrypoints.openai.engine.protocol import ( + ErrorResponse, + RequestResponseMetadata, +) +from vllm.entrypoints.openai.models.serving import OpenAIServingModels +from vllm.entrypoints.speech_to_text.base.serving import ( + SpeechToTextBaseServing, + asr_inter_chunk_separator, +) +from vllm.entrypoints.speech_to_text.transcription.protocol import TranscriptionRequest +from vllm.entrypoints.speech_to_text.transcription.serving import ( + OpenAIServingTranscription, +) +from vllm.model_executor.models.interfaces import SupportsTranscription +from vllm.outputs import CompletionOutput, RequestOutput + +# --- Unit: helper + protocol ------------------------------------------------- + + +def test_default_no_space_languages_includes_zh_and_ja(): + assert SupportsTranscription.no_space_languages == {"ja", "zh"} + + +@pytest.mark.parametrize( + ("language", "expected_sep"), + [ + ("en", " "), + ("EN", " "), + ("zh", ""), + ("ZH", ""), + ("ja", ""), + (None, " "), + ], +) +def test_asr_inter_chunk_separator_matches_protocol(language, expected_sep): + sep = asr_inter_chunk_separator(language, SupportsTranscription.no_space_languages) + assert sep == expected_sep + + +def test_joined_chunks_english_has_space_between(): + sep = asr_inter_chunk_separator("en", SupportsTranscription.no_space_languages) + assert sep.join(["hello", "world"]) == "hello world" + + +def test_joined_chunks_chinese_has_no_space_between(): + sep = asr_inter_chunk_separator("zh", SupportsTranscription.no_space_languages) + assert sep.join(["你好", "世界"]) == "你好世界" + + +# --- Integration: serving (no model) ----------------------------------------- + + +class _StubTranscriptionModel: + """Minimal stand-in for a SupportsTranscription implementation (no torch).""" + + no_space_languages: set[str] = {"ja", "zh"} + supports_segment_timestamp = False + + @classmethod + def get_speech_to_text_config( + cls, model_config: ModelConfig, task_type: str + ) -> SpeechToTextConfig: + return SpeechToTextConfig( + sample_rate=16000.0, + max_audio_clip_s=5.0, + ) + + @classmethod + def post_process_output(cls, text: str) -> str: + return text + + +def _request_output(text: str) -> RequestOutput: + return RequestOutput( + request_id="rid", + prompt=None, + prompt_token_ids=None, + prompt_logprobs=None, + outputs=[ + CompletionOutput( + index=0, + text=text, + token_ids=(1, 2, 3), + cumulative_logprob=None, + logprobs=None, + finish_reason="stop", + ) + ], + finished=True, + ) + + +def _sse_delta_contents(sse_body: str) -> list[str]: + """Extract ``choices[0].delta.content`` from each ``data:`` line (streaming API).""" + contents: list[str] = [] + for line in sse_body.splitlines(): + if not line.startswith("data: "): + continue + payload = line.removeprefix("data: ").strip() + if payload == "[DONE]": + continue + obj = json.loads(payload) + for choice in obj.get("choices") or []: + delta = choice.get("delta") or {} + if "content" in delta: + contents.append(delta["content"]) + return contents + + +@pytest.mark.asyncio +async def test_transcription_stream_generator_english_inserts_space_between_chunks(): + """Online streaming: first output per audio chunk is prefixed with *separator*.""" + + async def gen_hello() -> AsyncGenerator[RequestOutput, None]: + yield _request_output("hello") + + async def gen_world() -> AsyncGenerator[RequestOutput, None]: + yield _request_output("world") + + serving = OpenAIServingTranscription.__new__(OpenAIServingTranscription) + serving.enable_force_include_usage = False + serving.model_cls = _StubTranscriptionModel + serving.task_type = "transcribe" + request = SimpleNamespace( + model="stub-model", + stream_include_usage=False, + stream_continuous_usage_stats=False, + ) + sep = asr_inter_chunk_separator("en", _StubTranscriptionModel.no_space_languages) + assert sep == " " + + out_lines: list[str] = [] + agen = OpenAIServingTranscription.transcription_stream_generator( + serving, + request=request, + result_generator=[gen_hello(), gen_world()], + request_id="test-req", + request_metadata=RequestResponseMetadata(request_id="test-req"), + audio_duration_s=1.0, + separator=sep, + ) + async for line in agen: + out_lines.append(line) + sse = "".join(out_lines) + combined = "".join(_sse_delta_contents(sse)) + assert combined.strip() == "hello world" + + +@pytest.mark.asyncio +async def test_transcription_stream_generator_chinese_no_space_between_chunks(): + async def gen_a() -> AsyncGenerator[RequestOutput, None]: + yield _request_output("你好") + + async def gen_b() -> AsyncGenerator[RequestOutput, None]: + yield _request_output("世界") + + serving = OpenAIServingTranscription.__new__(OpenAIServingTranscription) + serving.enable_force_include_usage = False + serving.model_cls = _StubTranscriptionModel + serving.task_type = "transcribe" + request = SimpleNamespace( + model="stub-model", + stream_include_usage=False, + stream_continuous_usage_stats=False, + ) + sep = asr_inter_chunk_separator("zh", _StubTranscriptionModel.no_space_languages) + assert sep == "" + + out_lines: list[str] = [] + agen = OpenAIServingTranscription.transcription_stream_generator( + serving, + request=request, + result_generator=[gen_a(), gen_b()], + request_id="test-req-zh", + request_metadata=RequestResponseMetadata(request_id="test-req-zh"), + audio_duration_s=1.0, + separator=sep, + ) + async for line in agen: + out_lines.append(line) + combined = "".join(_sse_delta_contents("".join(out_lines))) + assert combined == "你好世界" + + +@pytest.mark.asyncio +async def test_create_transcription_non_streaming_joins_chunks_by_language(): + """``create_transcription`` uses the same separator logic as the helper.""" + + async def gen_hello() -> AsyncGenerator[RequestOutput, None]: + yield _request_output("hello") + + async def gen_world() -> AsyncGenerator[RequestOutput, None]: + yield _request_output("world") + + engine_client = MagicMock() + engine_client.model_config = MagicMock() + engine_client.model_config.get_diff_sampling_param.return_value = { + "max_tokens": 256, + "temperature": 0.0, + } + engine_client.model_config.max_model_len = 8192 + engine_client.errored = False + engine_client.generate.side_effect = [gen_hello(), gen_world()] + + models = MagicMock(spec=OpenAIServingModels) + models.lora_requests = {} + models.is_base_model.return_value = True + + preprocess_mock = AsyncMock(return_value=([MagicMock(), MagicMock()], 1.0)) + + with ( + patch( + "vllm.model_executor.model_loader.get_model_cls", + return_value=_StubTranscriptionModel, + ), + patch.object( + SpeechToTextBaseServing, "_preprocess_speech_to_text", preprocess_mock + ), + ): + serving = OpenAIServingTranscription(engine_client, models, request_logger=None) + + req_en = TranscriptionRequest.model_construct( + file=MagicMock(), + model="stub-model", + language="en", + stream=False, + response_format="json", + ) + out_en = await serving.create_transcription( + b"\x00\x00", req_en, raw_request=None + ) + assert not isinstance(out_en, ErrorResponse) + assert out_en.text == "hello world" + + async def gen_nihao() -> AsyncGenerator[RequestOutput, None]: + yield _request_output("你好") + + async def gen_shijie() -> AsyncGenerator[RequestOutput, None]: + yield _request_output("世界") + + engine_client.generate.side_effect = [gen_nihao(), gen_shijie()] + + req_zh = TranscriptionRequest.model_construct( + file=MagicMock(), + model="stub-model", + language="zh", + stream=False, + response_format="json", + ) + out_zh = await serving.create_transcription( + b"\x00\x00", req_zh, raw_request=None + ) + assert not isinstance(out_zh, ErrorResponse) + assert out_zh.text == "你好世界" diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_transcription_validation.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_transcription_validation.py new file mode 100644 index 0000000000000000000000000000000000000000..6716ff7145a2d7760b36b2f8ee4945ac3890b603 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_transcription_validation.py @@ -0,0 +1,156 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +# imports for structured outputs tests +import json + +import pytest + +from tests.entrypoints.speech_to_text.conftest import add_attention_backend +from tests.utils import ROCM_ENV_OVERRIDES, ROCM_EXTRA_ARGS, RemoteOpenAIServer + +MISTRAL_FORMAT_ARGS = [ + "--tokenizer_mode", + "mistral", + "--config_format", + "mistral", + "--load_format", + "mistral", +] + + +async def transcribe_and_check( + client, + model_name: str, + file, + *, + language: str, + expected_text: str, + expected_seconds: int | None = None, + case_sensitive: bool = False, +): + """Run a transcription request and assert the output contains + *expected_text* and optionally that usage reports *expected_seconds*. + + Provides detailed failure messages with the actual transcription output. + """ + transcription = await client.audio.transcriptions.create( + model=model_name, + file=file, + language=language, + response_format="text", + temperature=0.0, + ) + out = json.loads(transcription) + out_text = out["text"] + out_usage = out["usage"] + + if case_sensitive: + assert expected_text in out_text, ( + f"Expected {expected_text!r} in transcription output, got: {out_text!r}" + ) + else: + assert expected_text.lower() in out_text.lower(), ( + f"Expected {expected_text!r} (case-insensitive) in transcription " + f"output, got: {out_text!r}" + ) + + if expected_seconds is not None: + assert out_usage["seconds"] == expected_seconds, ( + f"Expected {expected_seconds}s of audio, " + f"got {out_usage['seconds']}s. Full usage: {out_usage!r}" + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", ["mistralai/Voxtral-Mini-3B-2507", "Qwen/Qwen3-ASR-0.6B"] +) +async def test_basic_audio(mary_had_lamb, model_name, rocm_aiter_fa_attention): + server_args = ["--enforce-eager", *ROCM_EXTRA_ARGS] + + if model_name.startswith("mistralai"): + server_args += MISTRAL_FORMAT_ARGS + + add_attention_backend(server_args, rocm_aiter_fa_attention) + + # Based on https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb. + with RemoteOpenAIServer( + model_name, server_args, env_dict=ROCM_ENV_OVERRIDES + ) as remote_server: + client = remote_server.get_async_client() + await transcribe_and_check( + client, + model_name, + mary_had_lamb, + language="en", + expected_text="Mary had a little lamb", + expected_seconds=16, + ) + + +@pytest.mark.asyncio +async def test_basic_audio_with_lora(mary_had_lamb, rocm_aiter_fa_attention): + """Ensure STT (transcribe) requests can pass LoRA through to generate.""" + # ROCm SPECIFIC CONFIGURATION: + # To ensure the test passes on ROCm, we modify the max model length to 512. + # We DO NOT apply this to other platforms to maintain strict upstream parity. + from vllm.platforms import current_platform + + model_name = "ibm-granite/granite-speech-3.3-2b" + lora_model_name = "speech" + server_args = [ + "--enforce-eager", + "--enable-lora", + "--max-lora-rank", + "64", + "--lora-modules", + f"{lora_model_name}={model_name}", + "--max-model-len", + "512" if current_platform.is_rocm() else "2048", + "--max-num-seqs", + "1", + ] + + add_attention_backend(server_args, rocm_aiter_fa_attention) + + # Based on https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb. + with RemoteOpenAIServer( + model_name, server_args, env_dict=ROCM_ENV_OVERRIDES + ) as remote_server: + client = remote_server.get_async_client() + await transcribe_and_check( + client, + lora_model_name, + mary_had_lamb, + language="en", + expected_text="mary had a little lamb", + expected_seconds=16, + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "model_name", ["google/gemma-3n-E2B-it", "Qwen/Qwen3-ASR-0.6B"] +) +async def test_basic_audio_foscolo(foscolo, rocm_aiter_fa_attention, model_name): + # Gemma accuracy on some of the audio samples we use is particularly bad, + # hence we use a different one here. WER is evaluated separately. + server_args = ["--enforce-eager", *ROCM_EXTRA_ARGS] + + add_attention_backend(server_args, rocm_aiter_fa_attention) + + with RemoteOpenAIServer( + model_name, + server_args, + max_wait_seconds=480, + env_dict=ROCM_ENV_OVERRIDES, + ) as remote_server: + client = remote_server.get_async_client() + await transcribe_and_check( + client, + model_name, + foscolo, + language="it", + expected_text="ove il mio corpo fanciulletto", + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_transcription_validation_whisper.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_transcription_validation_whisper.py new file mode 100644 index 0000000000000000000000000000000000000000..4f8104555af26b24fea45799f702d568ae162161 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/transcription/test_transcription_validation_whisper.py @@ -0,0 +1,438 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +# imports for structured outputs tests +import asyncio +import io +import json + +import numpy as np +import openai +import pytest +import pytest_asyncio +import soundfile as sf + +from tests.utils import RemoteOpenAIServer +from vllm.multimodal.media.audio import load_audio +from vllm.platforms import current_platform + +MODEL_NAME = "openai/whisper-large-v3-turbo" + +# Disable prefix caching on ROCm to reduce non-determinism in +# streaming-vs-non-streaming comparisons. +_ROCM_ARGS = ["--no-enable-prefix-caching"] if current_platform.is_rocm() else [] + + +def _get_attention_backend_params() -> list[str | None]: + """Return attention backends to parametrize the server fixture with. + + On ROCm, we test multiple backends explicitly: + - None: default auto-selection (ROCM_ATTN for decoder self-attention, + falls back to ROCM_AITER_UNIFIED_ATTN or TRITON_ATTN for + cross-attention since ROCM_ATTN doesn't support ENCODER_DECODER) + - TRITON_ATTN: always available on ROCm + - ROCM_AITER_UNIFIED_ATTN: only on gfx942/gfx950 + + On non-ROCm platforms, we just run with the default backend. + """ + try: + from vllm.platforms import current_platform + + if current_platform.is_rocm(): + backends: list[str | None] = [None, "TRITON_ATTN"] + from vllm.platforms.rocm import _ON_MI3XX + + if _ON_MI3XX: + backends.append("ROCM_AITER_UNIFIED_ATTN") + return backends + except Exception: + pass + return [None] + + +# Aiter backends need VLLM_ROCM_USE_AITER=1 (and MHA=1 for ROCM_AITER_FA) +# to be enabled in the server subprocess. +_AITER_ENV = { + "VLLM_ROCM_USE_AITER": "1", + "VLLM_ROCM_USE_AITER_MHA": "1", +} + +_ATTN_BACKENDS = _get_attention_backend_params() +_ATTN_IDS = [b or "default" for b in _ATTN_BACKENDS] + + +@pytest.fixture(scope="module", params=_ATTN_BACKENDS, ids=_ATTN_IDS) +def server(request): + args = [*_ROCM_ARGS] + env_dict = None + if request.param is not None: + args += ["--attention-backend", request.param] + if "AITER" in request.param: + # TODO: re-enable once AITER reenables fp16 unified attention. + pytest.skip("ROCM_AITER_UNIFIED_ATTN does not support fp16") + with RemoteOpenAIServer(MODEL_NAME, args, env_dict=env_dict) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def whisper_client(server): + async with server.get_async_client() as async_client: + yield async_client + + +@pytest.mark.asyncio +async def test_basic_audio(whisper_client, mary_had_lamb): + # Based on https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb. + transcription = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=mary_had_lamb, + language="en", + response_format="text", + temperature=0.0, + ) + out = json.loads(transcription) + out_text = out["text"] + out_usage = out["usage"] + assert "Mary had a little lamb," in out_text + assert out_usage["seconds"] == 16, out_usage["seconds"] + + +@pytest.mark.asyncio +async def test_basic_audio_batched(mary_had_lamb, winning_call, whisper_client): + transcription = whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=mary_had_lamb, + language="en", + response_format="text", + temperature=0.0, + ) + transcription2 = whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=winning_call, + language="en", + response_format="text", + temperature=0.0, + ) + # Await both transcriptions by scheduling coroutines together + transcription, transcription2 = await asyncio.gather(transcription, transcription2) + out = json.loads(transcription) + out_text = out["text"] + assert "Mary had a little lamb," in out_text + out2 = json.loads(transcription2) + out_text2 = out2["text"] + assert "Edgar Martinez" in out_text2 + + +@pytest.mark.asyncio +async def test_bad_requests(mary_had_lamb, whisper_client): + # invalid language + with pytest.raises(openai.BadRequestError): + await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, file=mary_had_lamb, language="hh", temperature=0.0 + ) + + +@pytest.mark.asyncio +async def test_long_audio_request(mary_had_lamb, whisper_client): + mary_had_lamb.seek(0) + audio, sr = load_audio(mary_had_lamb) + # Add small silence after each audio for repeatability in the split process + audio = np.pad(audio, (0, 1600)) + repeated_audio = np.tile(audio, 10) + # Repeated audio to buffer + buffer = io.BytesIO() + sf.write(buffer, repeated_audio, sr, format="WAV") + buffer.seek(0) + transcription = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=buffer, + language="en", + response_format="text", + temperature=0.0, + ) + out = json.loads(transcription) + out_text = out["text"] + out_usage = out["usage"] + counts = out_text.count("Mary had a little lamb") + assert counts == 10, counts + assert out_usage["seconds"] == 161, out_usage["seconds"] + + +@pytest.mark.asyncio +async def test_invalid_audio_file(whisper_client): + """Corrupted audio should surface as HTTP 400.""" + invalid_audio = io.BytesIO(b"not a valid audio file") + invalid_audio.name = "invalid.wav" + + with pytest.raises(openai.BadRequestError) as exc_info: + await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=invalid_audio, + language="en", + ) + + assert exc_info.value.status_code == 400 + assert "Invalid or unsupported audio file" in exc_info.value.message + + +@pytest.mark.asyncio +async def test_completion_endpoints(whisper_client): + # text to text model + with pytest.raises(openai.NotFoundError): + await whisper_client.chat.completions.create( + model=MODEL_NAME, + messages=[{"role": "system", "content": "You are a helpful assistant."}], + ) + + with pytest.raises(openai.NotFoundError): + await whisper_client.completions.create(model=MODEL_NAME, prompt="Hello") + + +@pytest.mark.asyncio +async def test_streaming_response(winning_call, whisper_client): + transcription = "" + res_no_stream = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=winning_call, + response_format="json", + language="en", + temperature=0.0, + ) + res = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=winning_call, + language="en", + temperature=0.0, + stream=True, + timeout=30, + ) + # Reconstruct from chunks and validate + async for chunk in res: + text = chunk.choices[0]["delta"]["content"] + transcription += text + + assert transcription == res_no_stream.text + + +@pytest.mark.asyncio +async def test_stream_options(winning_call, whisper_client): + res = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=winning_call, + language="en", + temperature=0.0, + stream=True, + extra_body=dict(stream_include_usage=True, stream_continuous_usage_stats=True), + timeout=30, + ) + final = False + continuous = True + async for chunk in res: + if not len(chunk.choices): + # final usage sent + final = True + else: + continuous = continuous and hasattr(chunk, "usage") + assert final and continuous + + +@pytest.mark.asyncio +async def test_sampling_params(mary_had_lamb, whisper_client): + """ + Compare sampling with params and greedy sampling to assert results + are different when extreme sampling parameters values are picked. + """ + transcription = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=mary_had_lamb, + language="en", + temperature=0.8, + extra_body=dict( + seed=42, + repetition_penalty=1.9, + top_k=12, + top_p=0.4, + min_p=0.5, + frequency_penalty=1.8, + presence_penalty=2.0, + ), + ) + + greedy_transcription = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=mary_had_lamb, + language="en", + temperature=0.0, + extra_body=dict(seed=42), + ) + + assert greedy_transcription.text != transcription.text + + +@pytest.mark.asyncio +async def test_audio_prompt(mary_had_lamb, whisper_client): + prompt = "This is a speech, recorded in a phonograph." + # Prompts should not omit the part of original prompt while transcribing. + prefix = "The first words I spoke in the original phonograph" + transcription = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=mary_had_lamb, + language="en", + response_format="text", + temperature=0.0, + ) + out = json.loads(transcription)["text"] + assert prefix in out + transcription_wprompt = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=mary_had_lamb, + language="en", + response_format="text", + prompt=prompt, + temperature=0.0, + ) + out_prompt = json.loads(transcription_wprompt)["text"] + assert prefix in out_prompt + + +@pytest.mark.asyncio +async def test_audio_with_timestamp(mary_had_lamb, whisper_client): + transcription = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=mary_had_lamb, + language="en", + response_format="verbose_json", + temperature=0.0, + ) + assert transcription.segments is not None + assert len(transcription.segments) > 0 + assert transcription.segments[0].avg_logprob is not None + assert transcription.segments[0].compression_ratio is not None + + +@pytest.mark.asyncio +async def test_audio_with_max_tokens(whisper_client, mary_had_lamb): + transcription = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=mary_had_lamb, + language="en", + response_format="text", + temperature=0.0, + extra_body={"max_completion_tokens": 1}, + ) + out = json.loads(transcription) + out_text = out["text"] + from transformers import AutoTokenizer + + tok = AutoTokenizer.from_pretrained(MODEL_NAME) + out_tokens = tok(out_text, add_special_tokens=False)["input_ids"] + assert len(out_tokens) == 1 + # max_completion_tokens > max_model_len + transcription = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=mary_had_lamb, + language="en", + response_format="text", + temperature=0.0, + extra_body={"max_completion_tokens": int(1e6)}, + ) + out = json.loads(transcription) + out_text = out["text"] + out_tokens = tok(out_text, add_special_tokens=False)["input_ids"] + assert len(out_tokens) < 450 # ~Whisper max output len + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("fixture_name", "expected_lang", "expected_text"), + [ + ("mary_had_lamb", "en", ["Mary had a little lamb"]), + ("foscolo", "it", ["zacinto", "sacre"]), + ], + ids=["english", "italian"], +) +async def test_language_auto_detect( + whisper_client, fixture_name, expected_lang, expected_text, request +): + """Auto-detect language when no language param is provided.""" + audio_file = request.getfixturevalue(fixture_name) + transcription = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=audio_file, + response_format="verbose_json", + temperature=0.0, + ) + assert transcription.language == expected_lang + text_lower = transcription.text.lower() + assert any(word.lower() in text_lower for word in expected_text), ( + f"Expected {expected_lang} text but got: {transcription.text}" + ) + + +@pytest.mark.asyncio +async def test_whisper_beam_search_single_beam(mary_had_lamb, whisper_client): + """Test beam search with encoder-decoder model (Whisper) on transcriptions with + one beam aligns with greedy decoding. + """ + beam_transcription = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=mary_had_lamb, + language="en", + response_format="text", + temperature=0.0, + extra_body=dict( + use_beam_search=True, + n=1, + ), + ) + + greedy_transcription = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=mary_had_lamb, + response_format="text", + temperature=0.0, + ) + + greedy_res = json.loads(greedy_transcription)["text"] + beam_res = json.loads(beam_transcription)["text"] + assert greedy_res == beam_res + + +@pytest.mark.asyncio +async def test_whisper_beam_search_multibeam(mary_had_lamb, whisper_client): + """Test n>1 for beam search returns one transcription (best beam).""" + transcription = await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=mary_had_lamb, + language="en", + response_format="text", + temperature=0.0, + extra_body=dict( + use_beam_search=True, + n=2, + ), + ) + + result = json.loads(transcription) + + text = result["text"] + + assert text is not None + assert len(text) > 0 + assert "mary had a little lamb" in text.lower() + + +@pytest.mark.asyncio +async def test_stream_with_beams_raises(winning_call, whisper_client): + """Test that stream=True + beam search raises bad request for now.""" + with pytest.raises(openai.BadRequestError): + await whisper_client.audio.transcriptions.create( + model=MODEL_NAME, + file=winning_call, + language="en", + stream=True, + extra_body=dict( + use_beam_search=True, + n=2, + ), + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/translation/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/translation/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/translation/test_translation_validation.py b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/translation/test_translation_validation.py new file mode 100644 index 0000000000000000000000000000000000000000..d315691e81a25a4fef14effeb53d9f8922b60469 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/speech_to_text/translation/test_translation_validation.py @@ -0,0 +1,318 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import io + +# imports for structured outputs tests +import json + +import httpx +import numpy as np +import openai +import pytest +import pytest_asyncio +import soundfile as sf + +from tests.entrypoints.speech_to_text.conftest import add_attention_backend +from tests.utils import RemoteOpenAIServer +from vllm.logger import init_logger +from vllm.multimodal.media.audio import load_audio + +logger = init_logger(__name__) + +SERVER_ARGS = ["--enforce-eager"] + + +def _get_rocm_attention_config(model_name): + """Return appropriate ROCm attention config for the given model. + + Whisper uses cross-attention (ENCODER_DECODER) which ROCM_AITER_FA does + not support. For Whisper we use ROCM_AITER_UNIFIED_ATTN (or TRITON_ATTN + as fallback); other models can use ROCM_AITER_FA. + """ + from vllm.platforms import current_platform + + if not current_platform.is_rocm(): + return None + + if "whisper" in model_name.lower(): + try: + from vllm.platforms.rocm import _ON_MI3XX + + if _ON_MI3XX: + return {"backend": "ROCM_AITER_UNIFIED_ATTN"} + except ImportError: + logger.warning( + "Could not import _ON_MI3XX from rocm platform, " + "falling back to TRITON_ATTN for Whisper." + ) + return {"backend": "TRITON_ATTN"} + + return {"backend": "ROCM_AITER_FA"} + + +def _get_server_args(attention_config): + """Get server args with attention backend if specified.""" + args = SERVER_ARGS.copy() + add_attention_backend(args, attention_config) + return args + + +@pytest.fixture( + scope="module", params=["openai/whisper-small", "google/gemma-3n-E2B-it"] +) +def server(request): + # Parametrize over model name + attention_config = _get_rocm_attention_config(request.param) + with RemoteOpenAIServer( + request.param, _get_server_args(attention_config) + ) as remote_server: + yield remote_server, request.param + + +@pytest_asyncio.fixture +async def client_and_model(server): + server, model_name = server + async with server.get_async_client() as async_client: + yield async_client, model_name + + +@pytest.mark.asyncio +async def test_non_asr_model(foscolo): + # text to text model + model_name = "JackFram/llama-68m" + attention_config = _get_rocm_attention_config(model_name) + with RemoteOpenAIServer( + model_name, _get_server_args(attention_config) + ) as remote_server: + client = remote_server.get_async_client() + + with pytest.raises(openai.NotFoundError): + await client.audio.translations.create( + model=model_name, file=foscolo, temperature=0.0 + ) + + +@pytest.mark.asyncio +async def test_basic_audio_with_lora(mary_had_lamb): + """Ensure STT (translate) requests can pass LoRA through to generate.""" + # ROCm SPECIFIC CONFIGURATION: + # To ensure the test passes on ROCm, we modify the max model length to 512. + # We DO NOT apply this to other platforms to maintain strict upstream parity. + from vllm.platforms import current_platform + + # NOTE - careful to call this test before the module scoped server + # fixture, otherwise it'll OOMkill the CI + model_name = "ibm-granite/granite-speech-3.3-2b" + lora_model_name = "speech" + server_args = [ + "--enforce-eager", + "--enable-lora", + "--max-lora-rank", + "64", + "--lora-modules", + f"{lora_model_name}={model_name}", + "--max-model-len", + "512" if current_platform.is_rocm() else "2048", + "--max-num-seqs", + "1", + ] + + add_attention_backend(server_args, _get_rocm_attention_config(model_name)) + + # Based on https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb. + with RemoteOpenAIServer(model_name, server_args) as remote_server: + client = remote_server.get_async_client() + translation = await client.audio.translations.create( + model=lora_model_name, + file=mary_had_lamb, + extra_body=dict(language="en", to_language="es"), + response_format="text", + temperature=0.0, + ) + out = json.loads(translation)["text"].strip().lower() + assert "pequeño" in out.split(" ") + + +# NOTE: (NickLucche) the large-v3-turbo model was not trained on translation! +@pytest.mark.asyncio +async def test_basic_audio(foscolo, client_and_model): + client, model_name = client_and_model + translation = await client.audio.translations.create( + model=model_name, + file=foscolo, + response_format="text", + # TODO remove `language="it"` once language detection is implemented + extra_body=dict(language="it", to_language="en"), + temperature=0.0, + ) + out = json.loads(translation)["text"].strip().lower() + assert "greek sea" in out + + +@pytest.mark.asyncio +async def test_audio_prompt(foscolo, client_and_model): + client, model_name = client_and_model + # Condition whisper on starting text + prompt = "Nor have I ever" + transcription = await client.audio.translations.create( + model=model_name, + file=foscolo, + prompt=prompt, + extra_body=dict(language="it", to_language="en"), + response_format="text", + temperature=0.0, + ) + out = json.loads(transcription)["text"] + assert "Nor will I ever touch the sacred" not in out + assert prompt not in out + + +@pytest.mark.asyncio +async def test_streaming_response(foscolo, client_and_model, server): + client, model_name = client_and_model + translation = "" + res_no_stream = await client.audio.translations.create( + model=model_name, + file=foscolo, + response_format="json", + extra_body=dict(language="it", to_language="en", seed=42), + temperature=0.0, + ) + + # Stream via HTTPX since OpenAI translation client doesn't expose streaming + server, model_name = server + url = server.url_for("v1/audio/translations") + headers = {"Authorization": f"Bearer {server.DUMMY_API_KEY}"} + data = { + "model": model_name, + "language": "it", + "to_language": "en", + "stream": True, + "temperature": 0.0, + "seed": 42, + } + foscolo.seek(0) + async with httpx.AsyncClient() as http_client: + files = {"file": foscolo} + async with http_client.stream( + "POST", url, headers=headers, data=data, files=files + ) as response: + async for line in response.aiter_lines(): + if not line: + continue + if line.startswith("data: "): + line = line[len("data: ") :] + if line.strip() == "[DONE]": + break + chunk = json.loads(line) + text = chunk["choices"][0].get("delta", {}).get("content") + translation += text or "" + + res_stream = translation.split() + # NOTE There's a small non-deterministic issue here, likely in the attn + # computation, which will cause a few tokens to be different, while still + # being very close semantically. + assert ( + sum([x == y for x, y in zip(res_stream, res_no_stream.text.split())]) + >= len(res_stream) * 0.87 + ) + + +@pytest.mark.asyncio +async def test_stream_options(foscolo, server): + server, model_name = server + url = server.url_for("v1/audio/translations") + headers = {"Authorization": f"Bearer {server.DUMMY_API_KEY}"} + data = { + "model": model_name, + "language": "it", + "to_language": "en", + "stream": True, + "stream_include_usage": True, + "stream_continuous_usage_stats": True, + "temperature": 0.0, + } + foscolo.seek(0) + final = False + continuous = True + async with httpx.AsyncClient() as http_client: + files = {"file": foscolo} + async with http_client.stream( + "POST", url, headers=headers, data=data, files=files + ) as response: + async for line in response.aiter_lines(): + if not line: + continue + if line.startswith("data: "): + line = line[len("data: ") :] + if line.strip() == "[DONE]": + break + chunk = json.loads(line) + choices = chunk.get("choices", []) + if not choices: + # final usage sent + final = True + else: + continuous = continuous and ("usage" in chunk) + assert final and continuous + + +@pytest.mark.asyncio +async def test_long_audio_request(foscolo, client_and_model): + client, model_name = client_and_model + if model_name == "google/gemma-3n-E2B-it": + pytest.skip("Gemma3n does not support long audio requests") + foscolo.seek(0) + audio, sr = load_audio(foscolo) + repeated_audio = np.tile(audio, 2) + # Repeated audio to buffer + buffer = io.BytesIO() + sf.write(buffer, repeated_audio, sr, format="WAV") + buffer.seek(0) + translation = await client.audio.translations.create( + model=model_name, + file=buffer, + extra_body=dict(language="it", to_language="en"), + response_format="text", + temperature=0.0, + ) + out = json.loads(translation)["text"].strip().lower() + assert out.count("greek sea") == 2 + + +@pytest.mark.asyncio +async def test_audio_with_max_tokens(mary_had_lamb, client_and_model): + client, model_name = client_and_model + transcription = await client.audio.translations.create( + model=model_name, + file=mary_had_lamb, + response_format="text", + temperature=0.0, + extra_body={"max_completion_tokens": 1}, + ) + out = json.loads(transcription) + out_text = out["text"] + print(out_text) + from transformers import AutoTokenizer + + tok = AutoTokenizer.from_pretrained(model_name) + out_tokens = tok(out_text, add_special_tokens=False)["input_ids"] + assert len(out_tokens) == 1 + # max_completion_tokens > max_model_len + # max_model_len=32768 for Gemma-3n-E2B-it + transcription = await client.audio.transcriptions.create( + model=model_name, + file=mary_had_lamb, + response_format="text", + temperature=0.0, + extra_body={ + "max_completion_tokens": int(1e6), + "repetition_penalty": 1.3, + }, + ) + out = json.loads(transcription) + out_text = out["text"] + print(out_text) + out_tokens = tok(out_text, add_special_tokens=False)["input_ids"] + assert len(out_tokens) < 450 # ~Whisper max output len diff --git a/asian_ai/performance/vLLM/tests/entrypoints/tool_parsers/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/tool_parsers/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/tool_parsers/test_granite4_tool_parser.py b/asian_ai/performance/vLLM/tests/entrypoints/tool_parsers/test_granite4_tool_parser.py new file mode 100644 index 0000000000000000000000000000000000000000..b5f2fbfe3fd06ab6b443af9adc7a73ebbae908c4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/tool_parsers/test_granite4_tool_parser.py @@ -0,0 +1,220 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import json + +import openai +import pytest + +from tests.utils import RemoteOpenAIServer + +MODEL = "ibm-granite/granite-4.0-h-tiny" + + +@pytest.fixture(scope="module") +def server(): + model = MODEL + args_for_model = [ + "--enforce-eager", + "--enable-auto-tool-choice", + "--tool-call-parser", + "granite4", + "--tokenizer", + "ibm-granite/granite-4.0-h-tiny", + "--max-model-len", + "4096", + "--max-num-seqs", + "2", + ] + with RemoteOpenAIServer(model, args_for_model, max_wait_seconds=480) as server: + yield server + + +tools = [ + { + "type": "function", + "function": { + "name": "get_acme_region_name_for_transaction_id", + "description": "Returns ACME transaction/transaction ID information" + " including ACME regions\n\nArgs:\n start_time " + "(str): Start date and time in datetime format " + '"%Y-%m-%dT%H:%M:%S.%f"\n end_time (str): End ' + "date and time in datetime format " + '"%Y-%m-%dT%H:%M:%S.%f"\n size (int, optional): ' + "Number of ACME Transaction IDs to return\n " + "order (str, optional): Sort by most run " + "transaction IDs. The value can be 'asc' for " + "ascending or 'desc' for descending\n " + "transaction_id (str, optional): ACME Transaction " + "ID to filter on\n acme_region (str, optional): " + "ACME Region to filter on\nReturns:\n - A " + "dictionary containing a list of ACME transaction " + "ids and the ACME regions they run in:\n {\n" + ' "Number of transaction IDs" : int,\n' + ' "Total transaction IDs available": int' + ',\n "ACME Transaction IDs": [\n ' + ' {\n "Transaction ID": ' + 'str,\n "Number of runs": int,\n' + ' "ACME Regions": [str],\n ' + " },\n ...\n ]," + '\n "Start time" : datetime,\n ' + ' "End time" : datetime,\n ' + ' "Order" : str\n }\n ' + " - If no ACME region found for transaction id, " + 'returns:\n {"Success": "No ACME region ' + 'found for transaction id."}\n - If an error ' + 'occurs, returns:\n {"Error": "{exception' + ' message}"}', + "parameters": { + "properties": { + "start_time": {}, + "end_time": {}, + "size": {"default": 500}, + "order": {"default": "desc"}, + "transaction_id": {"default": None}, + "acme_region": {"default": None}, + }, + "required": ["start_time", "end_time"], + "type": "object", + }, + }, + } +] + +tools2 = [ + { + "type": "function", + "function": { + "name": "get_current_weather", + "description": "Get the current weather", + "parameters": { + "type": "object", + "properties": { + "location": { + "description": "The city and state, e.g. San Francisco, CA", + "type": "string", + } + }, + "required": ["location"], + }, + }, + }, + { + "type": "function", + "function": { + "name": "get_stock_price", + "description": "Retrieves the current stock price for a given " + "ticker symbol. The ticker symbol must be a valid " + "symbol for a publicly traded company on a major US" + " stock exchange like NYSE or NASDAQ. The tool will" + " return the latest trade price in USD. It should " + "be used when the user asks about the current or " + "most recent price of a specific stock. It will not" + " provide any other information about the stock or" + " company.", + "parameters": { + "type": "object", + "properties": { + "ticker": { + "description": "The stock ticker symbol, e.g." + " AAPL for Apple Inc.", + "type": "string", + } + }, + }, + }, + }, +] + +messages = [ + { + "content": "\n\nSystem: You are a helpful, precise, and methodical AI" + " assistant that uses tool outputs provided inline.\nAlways" + " assume the current datetime is 2026-01-29T13:59:09.238901" + "+00:00.\n\nIf you receive a ToolMessage with `tool_call_id" + '` equal to "get_time_range" (or "time_range_tool"), you ' + "MUST:\n 1. Parse that JSON and use the values `start` and" + " `end` directly when calling other tools.\n 2. Do not " + "re-call or re-compute the time range.\n 3. Pass resolved " + "values (ISO strings) as arguments to any subsequent tool " + "(do not pass function metadata or placeholders).\n 4. If " + "a tool requires datetime objects rather than strings, " + "convert the ISO strings into language-native datetime " + "objects before invoking.\n\nAlways return fully resolved " + "arguments in correct types (e.g., ISO datetime strings or" + " datetime objects) and never include placeholders like " + '"".\n\n', + "role": "system", + }, + { + "content": "What are the transaction IDs that ran in the" + " ACME region A9345 over the last two months?", + "role": "user", + }, + { + "content": '["2026-01-26T09: 51: 55.467722Z", "2026-01-27T09: 51: 55.467722Z"]', + "role": "tool", + "tool_call_id": "time_range_tool", + }, +] +messages2 = [{"role": "user", "content": "What's stock price for IBM?"}] + +messages3 = [{"role": "user", "content": "What's the current weather in New York?"}] + + +def get_args(client: openai.OpenAI, _tools, _messages, _stop): + response = client.chat.completions.create( + model=MODEL, + messages=_messages, + temperature=0, + tools=_tools, + max_tokens=200, + stop=_stop, + tool_choice="auto", + ) + + return response.choices[0].message.tool_calls[0].function.arguments + + +async def get_args_streaming( + async_client: openai.AsyncOpenAI, _tools, _messages, _stop +): + stream = await async_client.chat.completions.create( + model=MODEL, + messages=_messages, + temperature=0, + tools=_tools, + max_tokens=200, + stop=_stop, + tool_choice="auto", + stream=True, + ) + full_call = [] + async for chunk in stream: + tc = chunk.choices[0].delta.tool_calls + if tc and tc[0].function.arguments: + full_call.append(tc[0].function.arguments) + return "".join(full_call) + + +async def run_scenario(server: RemoteOpenAIServer, _tools, _messages, _stop): + non_streaming = get_args(server.get_client(), _tools, _messages, _stop) + json.loads(non_streaming) # verify that it is json loadable + streaming = await get_args_streaming( + server.get_async_client(), _tools, _messages, _stop + ) + json.loads(streaming) + assert non_streaming == streaming, f"{non_streaming=}, {streaming=}" + + +@pytest.mark.asyncio +async def test_stop_sequence_interference(server: RemoteOpenAIServer): + print("Testing scenario 1") + await run_scenario(server, tools, messages, "veroniqueprattyushveroniqueprattyush") + + print("Testing scenario 2") + await run_scenario( + server, tools2, messages2, "veroniqueprattyushveroniqueprattyush" + ) + + print("Testing scenario 3") + await run_scenario(server, tools2, messages3, "prattyush") diff --git a/asian_ai/performance/vLLM/tests/entrypoints/tool_parsers/test_hermes_tool_parser.py b/asian_ai/performance/vLLM/tests/entrypoints/tool_parsers/test_hermes_tool_parser.py new file mode 100644 index 0000000000000000000000000000000000000000..b51feb6021787c7a5bde5d5abb797383e8b6f7ec --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/tool_parsers/test_hermes_tool_parser.py @@ -0,0 +1,324 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import openai +import pytest +import pytest_asyncio +from huggingface_hub import snapshot_download +from typing_extensions import TypedDict + +from tests.utils import RemoteOpenAIServer +from vllm.tool_parsers.abstract_tool_parser import ToolParser +from vllm.tool_parsers.granite4_tool_parser import Granite4ToolParser +from vllm.tool_parsers.hermes_tool_parser import Hermes2ProToolParser + +LORA_MODEL = "minpeter/LoRA-Llama-3.2-1B-tool-vllm-ci" + +TOOLS = [ + { + "type": "function", + "function": { + "name": "get_current_weather", + "description": "Get the current weather in a given location", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA", + }, + "unit": { + "type": "string", + "enum": ["celsius", "fahrenheit"], + }, + }, + "required": ["location"], + }, + }, + } +] + + +class ServerConfig(TypedDict, total=False): + model: str + arguments: list[str] + model_arg: str + tool_parser: ToolParser + + +CONFIGS: dict[str, ServerConfig] = { + "llama": { + "model": "meta-llama/Llama-3.2-1B-Instruct", + "arguments": [ + "--enforce-eager", + "--enable-auto-tool-choice", + "--tool-call-parser", + "hermes", + "--enable-lora", + "--lora-modules", + f"{LORA_MODEL}={LORA_MODEL}", + "--tokenizer", + f"{LORA_MODEL}", + ], + "model_arg": LORA_MODEL, + "tool_parser": Hermes2ProToolParser, + }, + "granite4": { + "model": "ibm-granite/granite-4.0-h-tiny", + "arguments": [ + "--enforce-eager", + "--enable-auto-tool-choice", + "--tool-call-parser", + "granite4", + "--tokenizer", + "ibm-granite/granite-4.0-h-tiny", + "--max-model-len", + "4096", + "--max-num-seqs", + "2", + ], + "model_arg": "ibm-granite/granite-4.0-h-tiny", + "tool_parser": Granite4ToolParser, + }, +} + + +# for each server config, download the model and return the config +@pytest.fixture(scope="session", params=CONFIGS.keys()) +def server_config(request): + config = CONFIGS[request.param] + + # download model and tokenizer using transformers + snapshot_download(config["model"]) + yield CONFIGS[request.param] + + +@pytest.fixture(scope="module") +def server(request, server_config: ServerConfig): + model = server_config["model"] + args_for_model = server_config["arguments"] + with RemoteOpenAIServer(model, args_for_model, max_wait_seconds=480) as server: + yield server + + +@pytest_asyncio.fixture +async def client(server: RemoteOpenAIServer): + async with server.get_async_client() as async_client: + yield async_client + + +PRODUCT_TOOLS = [ + { + "type": "function", + "function": { + "name": "get_product_info", + "description": "Get detailed information of a product based on its " + "product ID.", + "parameters": { + "type": "object", + "properties": { + "inserted": { + "type": "boolean", + "description": "inserted.", + }, + "product_id": { + "type": "integer", + "description": "The product ID of the product.", + }, + }, + "required": ["product_id", "inserted"], + }, + }, + } +] + +MESSAGES = [{"role": "user", "content": "What's the weather like in Boston?"}] + +PRODUCT_MESSAGES = [ + { + "role": "user", + "content": "Hi! Do you have any detailed information about the product id " + "7355608 and inserted true?", + } +] + + +@pytest.mark.asyncio +async def test_non_streaming_tool_call( + client: openai.AsyncOpenAI, server_config: ServerConfig +): + """Test tool call in non-streaming mode.""" + + response = await client.chat.completions.create( + model=server_config["model_arg"], + messages=MESSAGES, + tools=TOOLS, + tool_choice="auto", + temperature=0.0, + ) + + assert response.choices + choice = response.choices[0] + message = choice.message + + assert choice.finish_reason == "tool_calls" + assert message.tool_calls is not None + + tool_call = message.tool_calls[0] + assert tool_call.type == "function" + assert tool_call.function.name == "get_current_weather" + + arguments = json.loads(tool_call.function.arguments) + assert "location" in arguments + assert "Boston" in arguments["location"] + print("\n[Non-Streaming Test Passed]") + print(f"Tool Call: {tool_call.function.name}") + print(f"Arguments: {arguments}") + + +@pytest.mark.asyncio +async def test_streaming_tool_call( + client: openai.AsyncOpenAI, server_config: ServerConfig +): + """Test tool call in streaming mode.""" + + stream = await client.chat.completions.create( + model=server_config["model_arg"], + messages=MESSAGES, + tools=TOOLS, + tool_choice="auto", + temperature=0.0, + stream=True, + ) + + tool_call_chunks = {} + async for chunk in stream: + if not chunk.choices: + continue + + delta = chunk.choices[0].delta + if not delta or not delta.tool_calls: + continue + + for tool_chunk in delta.tool_calls: + index = tool_chunk.index + if index not in tool_call_chunks: + tool_call_chunks[index] = {"name": "", "arguments": ""} + + if tool_chunk.function.name: + tool_call_chunks[index]["name"] += tool_chunk.function.name + if tool_chunk.function.arguments: + tool_call_chunks[index]["arguments"] += tool_chunk.function.arguments + + assert len(tool_call_chunks) == 1 + reconstructed_tool_call = tool_call_chunks[0] + + assert reconstructed_tool_call["name"] == "get_current_weather" + + arguments = json.loads(reconstructed_tool_call["arguments"]) + assert "location" in arguments + assert "Boston" in arguments["location"] + print("\n[Streaming Test Passed]") + print(f"Reconstructed Tool Call: {reconstructed_tool_call['name']}") + print(f"Reconstructed Arguments: {arguments}") + + +@pytest.mark.asyncio +async def test_non_streaming_product_tool_call( + client: openai.AsyncOpenAI, server_config: ServerConfig +): + """Test tool call integer and boolean parameters in non-streaming mode.""" + + response = await client.chat.completions.create( + model=server_config["model_arg"], + messages=PRODUCT_MESSAGES, + tools=PRODUCT_TOOLS, + tool_choice="auto", + temperature=0.66, + ) + + assert response.choices + choice = response.choices[0] + message = choice.message + + assert choice.finish_reason == "tool_calls" + assert message.tool_calls is not None + + tool_call = message.tool_calls[0] + assert tool_call.type == "function" + assert tool_call.function.name == "get_product_info" + + arguments = json.loads(tool_call.function.arguments) + assert "product_id" in arguments + assert "inserted" in arguments + + product_id = arguments.get("product_id") + inserted = arguments.get("inserted") + + assert isinstance(product_id, int) + assert product_id == 7355608 + assert isinstance(inserted, bool) + assert inserted is True + + print("\n[Non-Streaming Product Test Passed]") + print(f"Tool Call: {tool_call.function.name}") + print(f"Arguments: {arguments}") + + +@pytest.mark.asyncio +async def test_streaming_product_tool_call( + client: openai.AsyncOpenAI, server_config: ServerConfig +): + """Test tool call integer and boolean parameters in streaming mode.""" + + stream = await client.chat.completions.create( + model=server_config["model_arg"], + messages=PRODUCT_MESSAGES, + tools=PRODUCT_TOOLS, + tool_choice="auto", + temperature=0.66, + stream=True, + ) + + tool_call_chunks = {} + async for chunk in stream: + if not chunk.choices: + continue + + delta = chunk.choices[0].delta + if not delta or not delta.tool_calls: + continue + + for tool_chunk in delta.tool_calls: + index = tool_chunk.index + if index not in tool_call_chunks: + tool_call_chunks[index] = {"name": "", "arguments": ""} + + if tool_chunk.function.name: + tool_call_chunks[index]["name"] += tool_chunk.function.name + if tool_chunk.function.arguments: + tool_call_chunks[index]["arguments"] += tool_chunk.function.arguments + + assert len(tool_call_chunks) == 1 + reconstructed_tool_call = tool_call_chunks[0] + + assert reconstructed_tool_call["name"] == "get_product_info" + + arguments = json.loads(reconstructed_tool_call["arguments"]) + assert "product_id" in arguments + assert "inserted" in arguments + + # Handle type coercion for streaming test as well + product_id = arguments.get("product_id") + inserted = arguments.get("inserted") + + assert isinstance(product_id, int) + assert product_id == 7355608 + assert isinstance(inserted, bool) + assert inserted is True + + print("\n[Streaming Product Test Passed]") + print(f"Reconstructed Tool Call: {reconstructed_tool_call['name']}") + print(f"Reconstructed Arguments: {arguments}") diff --git a/asian_ai/performance/vLLM/tests/entrypoints/tool_parsers/test_openai_tool_parser.py b/asian_ai/performance/vLLM/tests/entrypoints/tool_parsers/test_openai_tool_parser.py new file mode 100644 index 0000000000000000000000000000000000000000..213e5e707b219ba4653f68c0f1d477950c7adff2 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/tool_parsers/test_openai_tool_parser.py @@ -0,0 +1,359 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json + +import jsonschema +import openai +import pytest +import pytest_asyncio +from rapidfuzz import fuzz + +from tests.utils import RemoteOpenAIServer + +MODEL_NAME = "openai/gpt-oss-20b" + + +@pytest.fixture(scope="module") +def server(): + args = [ + "--max-model-len", + "8192", + "--enforce-eager", + "--enable-auto-tool-choice", + "--tool-call-parser", + "openai", + ] + with RemoteOpenAIServer(MODEL_NAME, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def client(server): + """Async fixture providing an OpenAI-compatible vLLM client.""" + async with server.get_async_client() as async_client: + yield async_client + + +# ========================================================== +# Tool Definitions +# ========================================================== +TOOLS = [ + { + "type": "function", + "function": { + "name": "calculator", + "description": "Performs basic arithmetic calculations.", + "parameters": { + "type": "object", + "properties": { + "expression": { + "type": "string", + "description": ( + "Arithmetic expression to evaluate, e.g. '123 + 456'." + ), + } + }, + "required": ["expression"], + }, + }, + }, + { + "type": "function", + "function": { + "name": "get_time", + "description": "Retrieves the current local time for a given city.", + "parameters": { + "type": "object", + "properties": { + "city": { + "type": "string", + "description": "City name, e.g. 'New York'.", + } + }, + "required": ["city"], + }, + }, + }, +] + + +# ========================================================== +# Message Examples +# ========================================================== +MESSAGES_CALC = [ + {"role": "user", "content": "Calculate 123 + 456 using the calculator."} +] + +MESSAGES_GET_TIME = [ + {"role": "user", "content": "What is the current time in New York?"} +] + +MESSAGES_MULTIPLE_CALLS = [ + { + "role": "system", + "content": ( + "You can call multiple tools. " + "When using more than one, return single JSON object with tool_calls array" + "containing each tool call with its function name and arguments. " + "Do not output multiple JSON objects separately." + ), + }, + { + "role": "user", + "content": "First, calculate 7 * 8 using the calculator. " + "Then, use get_time to tell me the current time in New York.", + }, +] + +MESSAGES_INVALID_CALL = [ + { + "role": "user", + "content": "Can you help with something, " + "but don’t actually perform any calculation?", + } +] + + +# Expected outputs +FUNC_CALC = "calculator" +FUNC_ARGS_CALC = '{"expression":"123 + 456"}' + +FUNC_TIME = "get_time" +FUNC_ARGS_TIME = '{"city": "New York"}' + + +# ========================================================== +# Utility to extract reasoning and tool calls +# ========================================================== +def extract_reasoning_and_calls(chunks: list) -> tuple[str, list[str], list[str]]: + """ + Extract accumulated reasoning text and tool call arguments + from streaming chunks. + """ + reasoning: str = "" + tool_calls: dict[int, dict[str, str]] = {} + + for chunk in chunks: + choice = getattr(chunk.choices[0], "delta", None) + if not choice: + continue + + if hasattr(choice, "reasoning") and choice.reasoning: + reasoning += choice.reasoning + + for tc in getattr(choice, "tool_calls", []) or []: + idx = getattr(tc, "index", 0) + tool_entry = tool_calls.setdefault(idx, {"name": "", "arguments": ""}) + + if getattr(tc, "function", None): + func = tc.function + if getattr(func, "name", None): + tool_entry["name"] = func.name + if getattr(func, "arguments", None): + tool_entry["arguments"] += func.arguments + + function_names: list[str] = [v["name"] for _, v in sorted(tool_calls.items())] + arguments: list[str] = [v["arguments"] for _, v in sorted(tool_calls.items())] + + return reasoning, arguments, function_names + + +# ========================================================== +# Test Scenarios +# ========================================================== +@pytest.mark.asyncio +async def test_calculator_tool_call_and_argument_accuracy(client: openai.AsyncOpenAI): + """Verify calculator tool call is made and arguments are accurate.""" + + response = await client.chat.completions.create( + model=MODEL_NAME, + messages=MESSAGES_CALC, + tools=TOOLS, + temperature=0.0, + stream=False, + ) + + message = response.choices[0].message + tool_calls = getattr(message, "tool_calls", []) + assert tool_calls, "No tool calls detected" + + calc_call = next((c for c in tool_calls if c.function.name == FUNC_CALC), None) + assert calc_call, "Calculator function not called" + + raw_args = calc_call.function.arguments + assert raw_args, "Calculator arguments missing" + assert "123" in raw_args and "456" in raw_args, ( + f"Expected values not in raw arguments: {raw_args}" + ) + + try: + parsed_args = json.loads(raw_args) + except json.JSONDecodeError: + pytest.fail(f"Invalid JSON in calculator arguments: {raw_args}") + + expected_expr = "123 + 456" + actual_expr = parsed_args.get("expression", "") + similarity = fuzz.ratio(actual_expr, expected_expr) + + assert similarity > 90, ( + f"Expression mismatch: expected '{expected_expr}' " + f"got '{actual_expr}' (similarity={similarity}%)" + ) + + +@pytest.mark.asyncio +async def test_streaming_tool_call_get_time_with_reasoning(client: openai.AsyncOpenAI): + """Verify streamed reasoning and tool call behavior for get_time.""" + + stream = await client.chat.completions.create( + model=MODEL_NAME, + messages=MESSAGES_GET_TIME, + tools=TOOLS, + temperature=0.0, + stream=True, + ) + + chunks = [chunk async for chunk in stream] + reasoning, arguments, function_names = extract_reasoning_and_calls(chunks) + + assert FUNC_TIME in function_names, "get_time function not called" + + assert any("New York" in arg for arg in arguments), ( + f"Expected get_time arguments for New York not found in {arguments}" + ) + + assert len(reasoning) > 0, "Expected reasoning content missing" + + assert any(keyword in reasoning for keyword in ["New York", "time", "current"]), ( + f"Reasoning is not relevant to the request: {reasoning}" + ) + + +@pytest.mark.asyncio +async def test_streaming_multiple_tools(client: openai.AsyncOpenAI): + """Test streamed multi-tool response with reasoning.""" + stream = await client.chat.completions.create( + model=MODEL_NAME, + messages=MESSAGES_MULTIPLE_CALLS, + tools=TOOLS, + temperature=0.0, + stream=True, + ) + + chunks = [chunk async for chunk in stream] + reasoning, arguments, function_names = extract_reasoning_and_calls(chunks) + + try: + assert FUNC_CALC in function_names, ( + f"Calculator tool missing — found {function_names}" + ) + assert FUNC_TIME in function_names, ( + f"Time tool missing — found {function_names}" + ) + assert len(reasoning) > 0, "Expected reasoning content in streamed response" + except AssertionError as e: + print(f"ERROR: {e}") + + +@pytest.mark.asyncio +async def test_invalid_tool_call(client: openai.AsyncOpenAI): + """ + Verify that ambiguous instructions that should not trigger a tool + do not produce any tool calls. + """ + response = await client.chat.completions.create( + model=MODEL_NAME, + messages=MESSAGES_INVALID_CALL, + tools=TOOLS, + temperature=0.0, + stream=False, + ) + + message = response.choices[0].message + + assert message is not None, "Expected message in response" + assert hasattr(message, "content"), "Expected 'content' field in message" + + tool_calls = getattr(message, "tool_calls", []) + assert not tool_calls, ( + f"Model unexpectedly attempted a tool call on invalid input: {tool_calls}" + ) + + +@pytest.mark.asyncio +async def test_tool_call_with_temperature(client: openai.AsyncOpenAI): + """ + Verify model produces valid tool or text output + under non-deterministic sampling. + """ + response = await client.chat.completions.create( + model=MODEL_NAME, + messages=MESSAGES_CALC, + tools=TOOLS, + temperature=0.7, + stream=False, + ) + + message = response.choices[0].message + assert message is not None, "Expected non-empty message in response" + assert message.tool_calls or message.content, ( + "Response missing both text and tool calls" + ) + + print(f"\nTool calls: {message.tool_calls}") + print(f"Text: {message.content}") + + +@pytest.mark.asyncio +async def test_tool_response_schema_accuracy(client: openai.AsyncOpenAI): + """Validate that tool call arguments adhere to their declared JSON schema.""" + response = await client.chat.completions.create( + model=MODEL_NAME, + messages=MESSAGES_MULTIPLE_CALLS, + tools=TOOLS, + temperature=0.0, + ) + + calls = response.choices[0].message.tool_calls + assert calls, "No tool calls produced" + + for call in calls: + func_name = call.function.name + args = json.loads(call.function.arguments) + + schema: dict[str, object] | None = None + for tool_entry in TOOLS: + function_def = tool_entry.get("function") + if ( + function_def + and isinstance(function_def, dict) + and function_def.get("name") == func_name + ): + schema = function_def.get("parameters") + break + + assert schema is not None, f"No matching tool schema found for {func_name}" + + jsonschema.validate(instance=args, schema=schema) + + +@pytest.mark.asyncio +async def test_semantic_consistency_with_temperature(client: openai.AsyncOpenAI): + """Test that temperature variation doesn't cause contradictory reasoning.""" + responses = [] + for temp in [0.0, 0.5, 1.0]: + resp = await client.chat.completions.create( + model=MODEL_NAME, + messages=MESSAGES_CALC, + tools=TOOLS, + temperature=temp, + ) + text = (resp.choices[0].message.content or "").strip() + responses.append(text) + + # Compare fuzzy similarity between low- and mid-temperature outputs + low_mid_sim = fuzz.ratio(responses[0], responses[1]) + assert low_mid_sim > 60, ( + f"Semantic drift too large between T=0.0 and T=0.5 ({low_mid_sim}%)" + ) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_api_server_process_manager.py b/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_api_server_process_manager.py new file mode 100644 index 0000000000000000000000000000000000000000..28dbdd283727f38c71e1086d4cbd040c7b155b5d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_api_server_process_manager.py @@ -0,0 +1,398 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import multiprocessing +import socket +import threading +import time +from unittest.mock import patch + +import pytest +import zmq + +from vllm.utils.network_utils import make_zmq_socket, split_zmq_path +from vllm.v1.utils import ( + APIServerProcessManager, + get_engine_client_zmq_addr, + wait_for_completion_or_failure, +) + +# Global variables to control worker behavior +WORKER_RUNTIME_SECONDS = 0.5 + + +# Mock implementation of run_api_server_worker +def mock_run_api_server_worker(listen_address, sock, args, client_config=None): + """Mock run_api_server_worker that runs for a specific time.""" + print(f"Mock worker started with client_config: {client_config}") + time.sleep(WORKER_RUNTIME_SECONDS) + print("Mock worker completed successfully") + + +# Module-level stub for the gather_actual_addresses test. Must be +# importable by `multiprocessing.spawn` (no closures, no nesting). +def defer_addresses_stub_worker(listen_address, sock, args, client_config): + """Bind ROUTER/PULL with a kernel-assigned port, report the actual + endpoints back via the pipe, then exit.""" + ctx = zmq.Context() + try: + in_sock = make_zmq_socket( + ctx, client_config["input_address"], zmq.ROUTER, bind=True + ) + out_sock = make_zmq_socket( + ctx, client_config["output_address"], zmq.PULL, bind=True + ) + try: + pipe = client_config["actual_address_pipe"] + try: + pipe.send( + { + "input_address": in_sock.getsockopt(zmq.LAST_ENDPOINT).decode(), + "output_address": out_sock.getsockopt( + zmq.LAST_ENDPOINT + ).decode(), + } + ) + finally: + pipe.close() + finally: + in_sock.close(linger=0) + out_sock.close(linger=0) + finally: + ctx.term() + + +@pytest.fixture +def api_server_args(): + """Fixture to provide arguments for APIServerProcessManager.""" + sock = socket.socket() + return { + "target_server_fn": mock_run_api_server_worker, + "listen_address": "localhost:8000", + "sock": sock, + "args": "test_args", # Simple string to avoid pickling issues + "num_servers": 3, + "input_addresses": [ + "tcp://127.0.0.1:5001", + "tcp://127.0.0.1:5002", + "tcp://127.0.0.1:5003", + ], + "output_addresses": [ + "tcp://127.0.0.1:6001", + "tcp://127.0.0.1:6002", + "tcp://127.0.0.1:6003", + ], + "stats_update_address": "tcp://127.0.0.1:7000", + } + + +@pytest.mark.parametrize("with_stats_update", [True, False]) +def test_api_server_process_manager_init(api_server_args, with_stats_update): + """Test initializing the APIServerProcessManager.""" + # Set the worker runtime to ensure tests complete in reasonable time + global WORKER_RUNTIME_SECONDS + WORKER_RUNTIME_SECONDS = 0.5 + + # Copy the args to avoid mutating them + args = api_server_args.copy() + + if not with_stats_update: + args.pop("stats_update_address") + manager = APIServerProcessManager(**args) + + try: + # Verify the manager was initialized correctly + assert len(manager.processes) == 3 + + # Verify all processes are running + for proc in manager.processes: + assert proc.is_alive() + + print("Waiting for processes to run...") + time.sleep(WORKER_RUNTIME_SECONDS / 2) + + # They should still be alive at this point + for proc in manager.processes: + assert proc.is_alive() + + finally: + # Always clean up the processes + print("Cleaning up processes...") + manager.shutdown() + + # Give processes time to terminate + time.sleep(0.2) + + # Verify all processes were terminated + for proc in manager.processes: + assert not proc.is_alive() + + +@patch("vllm.v1.utils.run_api_server_worker_proc", mock_run_api_server_worker) +def test_wait_for_completion_or_failure(api_server_args): + """Test that wait_for_completion_or_failure works with failures.""" + global WORKER_RUNTIME_SECONDS + WORKER_RUNTIME_SECONDS = 1.0 + + # Create the manager + manager = APIServerProcessManager(**api_server_args) + + try: + assert len(manager.processes) == 3 + + # Create a result capture for the thread + result: dict[str, Exception | None] = {"exception": None} + + def run_with_exception_capture(): + try: + wait_for_completion_or_failure(api_server_manager=manager) + except Exception as e: + result["exception"] = e + finally: + manager.shutdown() + + # Start a thread to run wait_for_completion_or_failure + wait_thread = threading.Thread(target=run_with_exception_capture, daemon=True) + wait_thread.start() + + # Let all processes run for a short time + time.sleep(0.2) + + # All processes should still be running + assert all(proc.is_alive() for proc in manager.processes) + + # Now simulate a process failure + print("Simulating process failure...") + manager.processes[0].terminate() + + # Wait for the wait_for_completion_or_failure + # to detect and handle the failure + # This should trigger it to terminate all other processes + wait_thread.join(timeout=1.0) + + # The wait thread should have exited + assert not wait_thread.is_alive() + + # Verify that an exception was raised with appropriate error message + assert result["exception"] is not None + assert "died with exit code" in str(result["exception"]) + + # All processes should now be terminated + for i, proc in enumerate(manager.processes): + assert not proc.is_alive(), f"Process {i} should not be alive" + + finally: + manager.shutdown() + time.sleep(0.2) + + +@pytest.mark.timeout(30) +def test_normal_completion(api_server_args): + """Test that wait_for_completion_or_failure works in normal completion.""" + global WORKER_RUNTIME_SECONDS + WORKER_RUNTIME_SECONDS = 0.1 + + # Create the manager + manager = APIServerProcessManager(**api_server_args) + + try: + # Give processes time to terminate + # wait for processes to complete + remaining_processes = manager.processes.copy() + while remaining_processes: + for proc in remaining_processes: + if not proc.is_alive(): + remaining_processes.remove(proc) + time.sleep(0.1) + + # Verify all processes have terminated + for i, proc in enumerate(manager.processes): + assert not proc.is_alive(), f"Process {i} still alive after terminate()" + + # Now call wait_for_completion_or_failure + # since all processes have already + # terminated, it should return immediately + # with no error + try: + wait_for_completion_or_failure(api_server_manager=manager) + finally: + manager.shutdown() + + finally: + # Clean up just in case + manager.shutdown() + time.sleep(0.2) + + +@pytest.mark.timeout(30) +def test_external_process_monitoring(api_server_args): + """Test that wait_for_completion_or_failure handles additional processes.""" + global WORKER_RUNTIME_SECONDS + WORKER_RUNTIME_SECONDS = 100 + + # Create and start the external process + # (simulates local_engine_manager or coordinator) + spawn_context = multiprocessing.get_context("spawn") + external_proc = spawn_context.Process( + target=mock_run_api_server_worker, name="MockExternalProcess" + ) + external_proc.start() + + # Create the class to simulate a coordinator + class MockCoordinator: + def __init__(self, proc): + self.proc = proc + + def shutdown(self): + if self.proc.is_alive(): + self.proc.terminate() + self.proc.join(timeout=0.5) + + # Create a mock coordinator with the external process + mock_coordinator = MockCoordinator(external_proc) + + # Create the API server manager + manager = APIServerProcessManager(**api_server_args) + + try: + # Verify manager initialization + assert len(manager.processes) == 3 + + # Create a result capture for the thread + result: dict[str, Exception | None] = {"exception": None} + + def run_with_exception_capture(): + try: + wait_for_completion_or_failure( + api_server_manager=manager, coordinator=mock_coordinator + ) + except Exception as e: + result["exception"] = e + finally: + manager.shutdown() + mock_coordinator.shutdown() + + # Start a thread to run wait_for_completion_or_failure + wait_thread = threading.Thread(target=run_with_exception_capture, daemon=True) + wait_thread.start() + + # Terminate the external process to trigger a failure + time.sleep(0.2) + external_proc.terminate() + + # Wait for the thread to detect the failure + wait_thread.join(timeout=1.0) + + # The wait thread should have completed + assert not wait_thread.is_alive(), ( + "wait_for_completion_or_failure thread still running" + ) + + # Verify that an exception was raised with appropriate error message + assert result["exception"] is not None, "No exception was raised" + error_message = str(result["exception"]) + assert "died with exit code" in error_message, ( + f"Unexpected error message: {error_message}" + ) + assert "MockExternalProcess" in error_message, ( + f"Error doesn't mention external process: {error_message}" + ) + + # Verify that all API server processes were terminated as a result + for i, proc in enumerate(manager.processes): + assert not proc.is_alive(), f"API server process {i} was not terminated" + + finally: + # Clean up + manager.shutdown() + mock_coordinator.shutdown() + time.sleep(0.2) + + +@pytest.mark.timeout(60) +def test_gather_actual_addresses_end_to_end(): + """Each child binds ROUTER/PULL with a kernel-picked port and reports + the bound endpoints back via its per-child pipe; the manager surfaces + them via :py:meth:`gather_actual_addresses`.""" + host = "127.0.0.1" + num_servers = 4 + + placeholder_inputs = [ + get_engine_client_zmq_addr(local_only=False, host=host) + for _ in range(num_servers) + ] + placeholder_outputs = [ + get_engine_client_zmq_addr(local_only=False, host=host) + for _ in range(num_servers) + ] + for addr in placeholder_inputs + placeholder_outputs: + assert addr == f"tcp://{host}:0", addr + + sock = socket.socket() + manager = APIServerProcessManager( + listen_address=f"tcp://{host}:0", + sock=sock, + args="test_args", + num_servers=num_servers, + input_addresses=placeholder_inputs, + output_addresses=placeholder_outputs, + target_server_fn=defer_addresses_stub_worker, + ) + + try: + assert len(manager.processes) == num_servers + actual_inputs, actual_outputs = manager.gather_actual_addresses(timeout=15.0) + finally: + manager.shutdown() + time.sleep(0.2) + sock.close() + + assert len(actual_inputs) == num_servers + assert len(actual_outputs) == num_servers + + for addr in actual_inputs + actual_outputs: + scheme, parsed_host, port = split_zmq_path(addr) + assert scheme == "tcp", addr + assert parsed_host == host, addr + assert port and int(port) > 0, addr + + all_addrs = actual_inputs + actual_outputs + assert len(set(all_addrs)) == len(all_addrs), all_addrs + + +@pytest.mark.timeout(30) +def test_gather_actual_addresses_child_crash_before_report(): + """A child that exits before sending its endpoints must surface a + clear ``RuntimeError`` rather than hang or return ``None`` slots.""" + host = "127.0.0.1" + num_servers = 2 + placeholder_inputs = [ + get_engine_client_zmq_addr(local_only=False, host=host) + for _ in range(num_servers) + ] + placeholder_outputs = [ + get_engine_client_zmq_addr(local_only=False, host=host) + for _ in range(num_servers) + ] + + sock = socket.socket() + manager = APIServerProcessManager( + listen_address=f"tcp://{host}:0", + sock=sock, + args="test_args", + num_servers=num_servers, + input_addresses=placeholder_inputs, + output_addresses=placeholder_outputs, + # mock_run_api_server_worker exits without touching + # ``actual_address_pipe`` — simulates a child that dies before + # reporting its bound addresses. + target_server_fn=mock_run_api_server_worker, + ) + try: + # Sentinel-first vs pipe-EOF-first both produce "reporting". + with pytest.raises(RuntimeError, match="reporting"): + manager.gather_actual_addresses(timeout=10.0) + finally: + manager.shutdown() + time.sleep(0.2) + sock.close() diff --git a/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_chat_utils.py b/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_chat_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..fb99773edad2f058d2b6d0f0ca1e77d292fc4378 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_chat_utils.py @@ -0,0 +1,2744 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import warnings +from collections.abc import Mapping +from typing import Literal + +import pytest +import torch + +from vllm.assets.audio import AudioAsset +from vllm.assets.image import ImageAsset +from vllm.assets.video import VideoAsset +from vllm.config import ModelConfig +from vllm.entrypoints.chat_utils import ( + ConversationMessage, + _postprocess_messages, + parse_chat_messages, + parse_chat_messages_async, +) +from vllm.inputs import MultiModalDataDict, MultiModalUUIDDict +from vllm.multimodal.utils import ( + encode_audio_url, + encode_image_url, + encode_video_url, +) +from vllm.utils.serial_utils import tensor2base64 + +KIMI_K2_5_MODEL_ID = "moonshotai/Kimi-K2.5" +PHI3V_MODEL_ID = "microsoft/Phi-3.5-vision-instruct" +QWEN2AUDIO_MODEL_ID = "Qwen/Qwen2-Audio-7B-Instruct" +QWEN25OMNI_MODEL_ID = "Qwen/Qwen2.5-Omni-7B" +MISTRAL_MODEL_ID = "mistralai/Mistral-Small-3.1-24B-Instruct-2503" + + +@pytest.fixture(scope="function") +def kimi_k2_5_model_config(): + return ModelConfig( + KIMI_K2_5_MODEL_ID, + runner="generate", + trust_remote_code=True, + limit_mm_per_prompt={ + "image": 2, + }, + ) + + +@pytest.fixture(scope="function") +def phi3v_model_config(): + return ModelConfig( + PHI3V_MODEL_ID, + runner="generate", + trust_remote_code=True, + limit_mm_per_prompt={ + "image": 2, + }, + ) + + +@pytest.fixture(scope="function") +def phi3v_model_config_mm_interleaved(): + return ModelConfig( + PHI3V_MODEL_ID, + runner="generate", + trust_remote_code=True, + interleave_mm_strings=True, + limit_mm_per_prompt={ + "image": 2, + }, + ) + + +@pytest.fixture(scope="function") +def phi3v_model_config_image_embeds(): + return ModelConfig( + PHI3V_MODEL_ID, + runner="generate", + trust_remote_code=True, + limit_mm_per_prompt={ + "image": 2, + }, + enable_mm_embeds=True, + ) + + +@pytest.fixture(scope="function") +def qwen25omni_model_config_image_embeds(): + return ModelConfig( + QWEN25OMNI_MODEL_ID, + runner="generate", + limit_mm_per_prompt={"image": 2}, + enable_mm_embeds=True, + ) + + +@pytest.fixture(scope="function") +def qwen2_audio_model_config(): + return ModelConfig( + QWEN2AUDIO_MODEL_ID, + runner="generate", + trust_remote_code=True, + limit_mm_per_prompt={ + "audio": 1, + }, + ) + + +@pytest.fixture(scope="function") +def audio_embeds_model_config(): + return ModelConfig( + QWEN2AUDIO_MODEL_ID, + runner="generate", + trust_remote_code=True, + limit_mm_per_prompt={ + "audio": 2, + }, + enable_mm_embeds=True, + ) + + +@pytest.fixture(scope="function") +def qwen25omni_model_config_mm_interleaved(): + return ModelConfig( + QWEN25OMNI_MODEL_ID, + runner="generate", + interleave_mm_strings=True, + limit_mm_per_prompt={ + "image": 2, + "audio": 1, + "video": 1, + }, + ) + + +@pytest.fixture(scope="function") +def mistral_model_config(): + return ModelConfig( + MISTRAL_MODEL_ID, + runner="generate", + limit_mm_per_prompt={ + "image": 2, + }, + ) + + +@pytest.fixture(scope="module") +def image_url(): + image = ImageAsset("cherry_blossom") + return encode_image_url(image.pil_image) + + +@pytest.fixture(scope="module") +def video_url(): + video = VideoAsset("baby_reading", 1) + return encode_video_url(video.np_ndarrays) + + +@pytest.fixture(scope="module") +def audio_url(): + audio = AudioAsset("mary_had_lamb") + return encode_audio_url(*audio.audio_and_sample_rate) + + +def _assert_mm_data_is_image_input( + mm_data: MultiModalDataDict | None, + image_count: int, + skipped_image_indices: list | None = None, +) -> None: + assert mm_data is not None + assert set(mm_data.keys()) == {"image"} + + image_data = mm_data.get("image") + assert image_data is not None + + assert isinstance(image_data, list) and len(image_data) == image_count + if skipped_image_indices is not None: + for i in skipped_image_indices: + assert image_data[i] is None + + +def _assert_mm_data_is_vision_chunk_input( + mm_data: MultiModalDataDict | None, + vision_chunk_count: int, +) -> None: + assert mm_data is not None + assert set(mm_data.keys()) == {"vision_chunk"} + + vision_chunk_data = mm_data.get("vision_chunk") + assert vision_chunk_data is not None + + assert ( + isinstance(vision_chunk_data, list) + and len(vision_chunk_data) == vision_chunk_count + ) + + +def _assert_mm_uuids( + mm_uuids: MultiModalUUIDDict | None, + media_count: int, + expected_uuids: list[str | None], + modality: str = "image", +) -> None: + if len(expected_uuids) > 0: + assert mm_uuids is not None + assert modality in mm_uuids + + image_uuids = mm_uuids.get(modality) + assert image_uuids is not None + + assert isinstance(image_uuids, list) and len(image_uuids) == media_count + + assert image_uuids == expected_uuids + else: + assert mm_uuids is None + + +ModalityType = Literal["image", "video", "audio"] +MultiModalDataCounts = Mapping[ModalityType, int] + + +def _assert_mm_data_inputs( + mm_data: MultiModalDataDict | None, + data_count: MultiModalDataCounts, + skipped_media_indices: dict[str, list] | None = None, # modality -> list[int] +) -> None: + assert mm_data is not None + assert set(data_count.keys()) == (set(mm_data.keys())) + + for modality, n in data_count.items(): + modality_data = mm_data.get(modality) + assert modality_data is not None + assert isinstance(modality_data, list) and len(modality_data) == n + + if skipped_media_indices is not None: + skipped_media_indices_for_modality = skipped_media_indices.get(modality) + assert skipped_media_indices_for_modality is not None + for i in skipped_media_indices_for_modality: + assert modality_data[i] is None + + +def test_parse_chat_messages_single_image( + phi3v_model_config, + image_url, +): + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "What's in the image?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + {"role": "user", "content": "<|image_1|>\nWhat's in the image?"} + ] + _assert_mm_data_is_image_input(mm_data, 1) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[None]) + + +def test_parse_chat_messages_single_image_with_uuid( + phi3v_model_config, + image_url, +): + image_uuid = str(hash(image_url)) + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": { + "url": image_url, + }, + "uuid": image_uuid, + }, + {"type": "text", "text": "What's in the image?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + {"role": "user", "content": "<|image_1|>\nWhat's in the image?"} + ] + _assert_mm_data_is_image_input(mm_data, 1) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[image_uuid]) + + +def test_parse_chat_messages_single_empty_image_with_uuid( + phi3v_model_config, + image_url, +): + image_uuid = str(hash(image_url)) + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": None, + "uuid": image_uuid, + }, + {"type": "text", "text": "What's in the image?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + {"role": "user", "content": "<|image_1|>\nWhat's in the image?"} + ] + _assert_mm_data_is_image_input(mm_data, 1, skipped_image_indices=[0]) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[image_uuid]) + + +def test_parse_chat_messages_single_image_with_bad_uuid_format( + phi3v_model_config, + image_url, +): + image_uuid = str(hash(image_url)) + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": { + "url": image_url, + "uuid": image_uuid, + }, + "bad_uuid_key": image_uuid, + }, + {"type": "text", "text": "What's in the image?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + {"role": "user", "content": "<|image_1|>\nWhat's in the image?"} + ] + _assert_mm_data_is_image_input(mm_data, 1) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[None]) + + +def test_parse_chat_messages_multiple_images_with_uuids( + phi3v_model_config, + image_url, +): + image_uuid1 = "my_uuid_1" + image_uuid2 = "my_uuid_2" + + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": { + "url": image_url, + }, + "uuid": image_uuid1, + }, + { + "type": "image_url", + "image_url": { + "url": image_url, + }, + "uuid": image_uuid2, + }, + {"type": "text", "text": "What's in the image?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\n<|image_2|>\nWhat's in the image?", + } + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[image_uuid1, image_uuid2]) + + +def test_parse_chat_messages_multiple_empty_images_with_uuids( + phi3v_model_config, + image_url, +): + image_uuid1 = "my_uuid_1" + image_uuid2 = "my_uuid_2" + + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": None, + "uuid": image_uuid1, + }, + { + "type": "image_url", + "image_url": None, + "uuid": image_uuid2, + }, + {"type": "text", "text": "What's in the image?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\n<|image_2|>\nWhat's in the image?", + } + ] + _assert_mm_data_is_image_input(mm_data, 2, skipped_image_indices=[0, 1]) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[image_uuid1, image_uuid2]) + + +def test_parse_chat_messages_mixed_empty_images_with_uuids( + phi3v_model_config, + image_url, +): + image_uuid1 = "my_uuid_1" + image_uuid2 = "my_uuid_2" + + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": { + "url": image_url, + }, + "uuid": image_uuid1, + }, + { + "type": "image_url", + "image_url": None, + "uuid": image_uuid2, + }, + {"type": "text", "text": "What's in the image?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\n<|image_2|>\nWhat's in the image?", + } + ] + _assert_mm_data_is_image_input(mm_data, 2, skipped_image_indices=[1]) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[image_uuid1, image_uuid2]) + + +@pytest.mark.asyncio +async def test_parse_chat_messages_single_image_with_uuid_async( + phi3v_model_config, + image_url, +): + image_uuid = str(hash(image_url)) + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": image_uuid, + }, + {"type": "text", "text": "What's in the image?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + {"role": "user", "content": "<|image_1|>\nWhat's in the image?"} + ] + _assert_mm_data_is_image_input(mm_data, 1) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[image_uuid]) + + +@pytest.mark.asyncio +async def test_parse_chat_messages_empty_image_with_uuid_async( + phi3v_model_config, + image_url, +): + image_uuid = str(hash(image_url)) + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": None, + "uuid": image_uuid, + }, + {"type": "text", "text": "What's in the image?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + {"role": "user", "content": "<|image_1|>\nWhat's in the image?"} + ] + _assert_mm_data_is_image_input(mm_data, 1, skipped_image_indices=[0]) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[image_uuid]) + + +@pytest.mark.asyncio +async def test_parse_chat_messages_multiple_images_with_uuids_async( + phi3v_model_config, + image_url, +): + image_uuid1 = "my_uuid_1" + image_uuid2 = "my_uuid_2" + + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": image_uuid1, + }, + { + "type": "image_pil", + "image_pil": ImageAsset("cherry_blossom").pil_image, + "uuid": image_uuid2, + }, + {"type": "text", "text": "What's in these images?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\n<|image_2|>\nWhat's in these images?", + } + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[image_uuid1, image_uuid2]) + + +@pytest.mark.asyncio +async def test_parse_chat_messages_multiple_empty_images_with_uuids_async( + phi3v_model_config, + image_url, +): + image_uuid1 = "my_uuid_1" + image_uuid2 = "my_uuid_2" + + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": None, + "uuid": image_uuid1, + }, + { + "type": "image_pil", + "image_pil": None, + "uuid": image_uuid2, + }, + {"type": "text", "text": "What's in these images?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\n<|image_2|>\nWhat's in these images?", + } + ] + _assert_mm_data_is_image_input(mm_data, 2, skipped_image_indices=[0, 1]) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[image_uuid1, image_uuid2]) + + +@pytest.mark.asyncio +async def test_parse_chat_messages_multiple_images_with_partial_uuids_async( + phi3v_model_config, + image_url, +): + image_uuid2 = "my_uuid_2" + + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": {"url": image_url}, + }, + { + "type": "image_pil", + "image_pil": ImageAsset("cherry_blossom").pil_image, + "uuid": image_uuid2, + }, + {"type": "text", "text": "What's in these images?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\n<|image_2|>\nWhat's in these images?", + } + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[None, image_uuid2]) + + +def test_parse_chat_messages_empty_system( + mistral_model_config, +): + # Test string format + conversation, _, _ = parse_chat_messages( + [ + {"role": "system", "content": ""}, + { + "role": "user", + "content": [{"type": "text", "text": "Who are you?"}], + }, + ], + mistral_model_config, + content_format="string", + ) + assert conversation == [ + {"role": "system", "content": ""}, + {"role": "user", "content": "Who are you?"}, + ] + + # Test openai format + conversation, _, _ = parse_chat_messages( + [ + {"role": "system", "content": ""}, + { + "role": "user", + "content": [{"type": "text", "text": "Who are you?"}], + }, + ], + mistral_model_config, + content_format="openai", + ) + assert conversation == [ + {"role": "system", "content": [{"type": "text", "text": ""}]}, + {"role": "user", "content": [{"type": "text", "text": "Who are you?"}]}, + ] + + +@pytest.mark.asyncio +async def test_parse_chat_messages_single_image_async( + phi3v_model_config, + image_url, +): + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "What's in the image?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + {"role": "user", "content": "<|image_1|>\nWhat's in the image?"} + ] + _assert_mm_data_is_image_input(mm_data, 1) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[None]) + + +def test_parse_chat_messages_multiple_images( + phi3v_model_config, + image_url, +): + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": image_url}}, + { + "type": "image_pil", + "image_pil": ImageAsset("cherry_blossom").pil_image, + }, + {"type": "text", "text": "What's in these images?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\n<|image_2|>\nWhat's in these images?", + } + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[None, None]) + + +def test_parse_chat_messages_empty_pil_image_with_uuid( + phi3v_model_config, +): + uuid = "abcd" + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "image_pil", "image_pil": None, "uuid": uuid}, + {"type": "text", "text": "What's in this image?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\nWhat's in this image?", + } + ] + _assert_mm_data_is_image_input(mm_data, 1, skipped_image_indices=[0]) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[uuid]) + + +def test_parse_chat_messages_empty_image_embeds_with_uuid( + phi3v_model_config_image_embeds, +): + uuid = "abcd" + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "image_embeds", "image_embeds": None, "uuid": uuid}, + {"type": "text", "text": "What's in this image?"}, + ], + } + ], + phi3v_model_config_image_embeds, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\nWhat's in this image?", + } + ] + + assert mm_data is not None + assert "image" in mm_data + assert isinstance(mm_data["image"], list) + assert len(mm_data["image"]) == 1 + assert mm_data["image"][0] is None + + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[uuid]) + + +def test_parse_chat_messages_empty_audio_embeds_with_uuid( + audio_embeds_model_config, +): + """Test audio_embeds with UUID (no actual embeds data).""" + uuid = "test-audio-uuid-123" + + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Describe this audio"}, + {"type": "audio_embeds", "audio_embeds": None, "uuid": uuid}, + ], + } + ], + audio_embeds_model_config, + content_format="string", + ) + + # Should have audio in mm_data as None (UUID provided) + assert mm_data is not None + assert "audio" in mm_data + assert isinstance(mm_data["audio"], list) + assert len(mm_data["audio"]) == 1 + assert mm_data["audio"][0] is None + + # UUID should be recorded + _assert_mm_uuids(mm_uuids, 1, modality="audio", expected_uuids=[uuid]) + + +def test_parse_chat_messages_audio_embeds_with_string( + audio_embeds_model_config, +): + """Test audio_embeds with base64 string embedding data.""" + + import torch + + # Create a sample audio embedding tensor + hidden_size = audio_embeds_model_config.get_inputs_embeds_size() + audio_embedding = torch.randn(1, 128, hidden_size) + + # Encode it as base64 + base64_audio_embedding = tensor2base64(audio_embedding) + + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Describe this audio"}, + { + "type": "audio_embeds", + "audio_embeds": base64_audio_embedding, + }, + ], + } + ], + audio_embeds_model_config, + content_format="string", + ) + + # Should have audio embedding in mm_data (single tensor, not a list) + assert mm_data is not None + assert "audio" in mm_data + assert isinstance(mm_data["audio"], torch.Tensor) + assert mm_data["audio"].shape == audio_embedding.shape + # No UUID provided + assert mm_uuids is not None + assert "audio" in mm_uuids + _assert_mm_uuids(mm_uuids, 1, modality="audio", expected_uuids=[None]) + + +@pytest.mark.asyncio +async def test_parse_chat_messages_audio_embeds_async( + audio_embeds_model_config, +): + """Test audio_embeds with async futures.""" + + import torch + + # Create a sample audio embedding tensor + hidden_size = audio_embeds_model_config.get_inputs_embeds_size() + audio_embedding = torch.randn(1, 128, hidden_size) + + # Encode it as base64 + base64_audio_embedding = tensor2base64(audio_embedding) + + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Describe this audio"}, + { + "type": "audio_embeds", + "audio_embeds": base64_audio_embedding, + }, + ], + } + ], + audio_embeds_model_config, + content_format="string", + ) + + # Should have audio embedding in mm_data (single tensor, not a list) + assert mm_data is not None + assert "audio" in mm_data + assert isinstance(mm_data["audio"], torch.Tensor) + assert mm_data["audio"].shape == audio_embedding.shape + # No UUID provided + assert mm_uuids is not None + assert "audio" in mm_uuids + _assert_mm_uuids(mm_uuids, 1, modality="audio", expected_uuids=[None]) + + +def test_parse_chat_messages_multiple_image_embeds( + phi3v_model_config_image_embeds, +): + """Test that multiple image_embeds in a single message are now supported. + + This test validates the fix for the limitation that previously only allowed + one message with {'type': 'image_embeds'}. Now multiple image embeddings + can be provided in a single request, similar to regular images. + """ + # Create two sample image embedding tensors + hidden_size = phi3v_model_config_image_embeds.get_inputs_embeds_size() + image_embedding_1 = torch.randn(256, hidden_size) + image_embedding_2 = torch.randn(128, hidden_size) + + # Encode them as base64 using the convenience function + base64_image_embedding_1 = tensor2base64(image_embedding_1) + base64_image_embedding_2 = tensor2base64(image_embedding_2) + + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + { + "type": "image_embeds", + "image_embeds": base64_image_embedding_1, + }, + { + "type": "image_embeds", + "image_embeds": base64_image_embedding_2, + }, + {"type": "text", "text": "Describe these two images."}, + ], + } + ], + phi3v_model_config_image_embeds, + content_format="string", + ) + + # Verify conversation structure + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\n<|image_2|>\nDescribe these two images.", + } + ] + + # Verify mm_data contains a list of embeddings (not a single embedding) + assert mm_data is not None + assert "image" in mm_data + assert isinstance(mm_data["image"], list) + assert len(mm_data["image"]) == 2 + + # Verify each embedding has the correct shape + assert isinstance(mm_data["image"][0], torch.Tensor) + assert mm_data["image"][0].shape == image_embedding_1.shape + assert isinstance(mm_data["image"][1], torch.Tensor) + assert mm_data["image"][1].shape == image_embedding_2.shape + + # Verify UUIDs (None since we didn't provide any) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[None, None]) + + +def test_parse_chat_messages_multiple_image_embeds_with_uuids( + phi3v_model_config_image_embeds, +): + """Test multiple image_embeds with UUIDs. + + This validates that UUIDs are properly tracked for multiple embeddings. + """ + uuid1 = "image-uuid-1" + uuid2 = "image-uuid-2" + + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + { + "type": "image_embeds", + "image_embeds": None, + "uuid": uuid1, + }, + { + "type": "image_embeds", + "image_embeds": None, + "uuid": uuid2, + }, + {"type": "text", "text": "Compare these images."}, + ], + } + ], + phi3v_model_config_image_embeds, + content_format="string", + ) + + # Verify conversation structure + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\n<|image_2|>\nCompare these images.", + } + ] + + # Verify mm_data contains a list with None values (UUID references) + assert mm_data is not None + assert "image" in mm_data + assert isinstance(mm_data["image"], list) + assert len(mm_data["image"]) == 2 + assert mm_data["image"][0] is None + assert mm_data["image"][1] is None + + # Verify UUIDs are correctly tracked + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[uuid1, uuid2]) + + +@pytest.mark.asyncio +async def test_parse_chat_messages_multiple_image_embeds_async( + phi3v_model_config_image_embeds, +): + """Test multiple image_embeds with async parsing. + + This validates the AsyncMultiModalItemTracker also supports multiple embeddings. + """ + # Create two sample image embedding tensors + hidden_size = phi3v_model_config_image_embeds.get_inputs_embeds_size() + image_embedding_1 = torch.randn(200, hidden_size) + image_embedding_2 = torch.randn(150, hidden_size) + + # Encode them as base64 using the convenience function + base64_image_embedding_1 = tensor2base64(image_embedding_1) + base64_image_embedding_2 = tensor2base64(image_embedding_2) + + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + [ + { + "role": "user", + "content": [ + { + "type": "image_embeds", + "image_embeds": base64_image_embedding_1, + }, + { + "type": "image_embeds", + "image_embeds": base64_image_embedding_2, + }, + {"type": "text", "text": "What do these images show?"}, + ], + } + ], + phi3v_model_config_image_embeds, + content_format="string", + ) + + # Verify conversation structure + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\n<|image_2|>\nWhat do these images show?", + } + ] + + # Await the future and verify mm_data + assert mm_data is not None + assert "image" in mm_data + assert isinstance(mm_data["image"], list) + assert len(mm_data["image"]) == 2 + + # Verify each embedding has the correct shape + assert isinstance(mm_data["image"][0], torch.Tensor) + assert mm_data["image"][0].shape == image_embedding_1.shape + assert isinstance(mm_data["image"][1], torch.Tensor) + assert mm_data["image"][1].shape == image_embedding_2.shape + + # Verify UUIDs + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[None, None]) + + +@pytest.mark.asyncio +async def test_parse_chat_messages_empty_image_embeds_with_uuid_async( + phi3v_model_config_image_embeds, +): + uuid = "abcd" + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + [ + { + "role": "user", + "content": [ + {"type": "image_embeds", "image_embeds": None, "uuid": uuid}, + {"type": "text", "text": "What's in this image?"}, + ], + } + ], + phi3v_model_config_image_embeds, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\nWhat's in this image?", + } + ] + assert mm_data is not None + assert "image" in mm_data + assert isinstance(mm_data["image"], list) + assert len(mm_data["image"]) == 1 + assert mm_data["image"][0] is None + + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[uuid]) + + +def test_parse_chat_messages_empty_dict_image_embeds( + phi3v_model_config_image_embeds, +): + """Test that empty dictionary for image_embeds is handled without errors.""" + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "image_embeds", "image_embeds": {}}, + {"type": "text", "text": "What's in this image?"}, + ], + } + ], + phi3v_model_config_image_embeds, + content_format="string", + ) + + # Verify conversation structure + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\nWhat's in this image?", + } + ] + + # Verify mm_data contains an empty dictionary of embeddings + assert mm_data is not None + assert "image" in mm_data + assert isinstance(mm_data["image"], dict) + assert len(mm_data["image"]) == 0 + + # Verify UUIDs (None since we didn't provide any) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[None]) + + +def test_parse_chat_messages_multiple_dict_image_embeds( + qwen25omni_model_config_image_embeds, +): + """Test that multiple dictionaries for image_embeds is handled without errors.""" + # Create two sample image embedding tensors + batch_size = 2 + hidden_size = qwen25omni_model_config_image_embeds.get_inputs_embeds_size() + image_embeds = torch.randn(batch_size * 220, hidden_size) + image_grid_thw = torch.tensor([[1, 22, 40] for _ in range(batch_size)]) + + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + { + "type": "image_embeds", + "image_embeds": { + "image_embeds": tensor2base64(embeds), + "image_grid_thw": tensor2base64(grid_thw), + }, + } + for embeds, grid_thw in zip( + image_embeds.chunk(batch_size), image_grid_thw + ) + ] + + [ + {"type": "text", "text": "Describe these two images."}, + ], + } + ], + qwen25omni_model_config_image_embeds, + content_format="string", + ) + + # Verify conversation structure + assert conversation == [ + { + "role": "user", + "content": "<|vision_start|><|IMAGE|><|vision_end|>\n" + "<|vision_start|><|IMAGE|><|vision_end|>\nDescribe these two images.", + } + ] + + # Verify mm_data contains a dictionary of multi-embeddings + assert mm_data is not None + assert "image" in mm_data + assert isinstance(mm_data["image"], dict) + assert len(mm_data["image"]) == batch_size + + # Verify each embedding has the correct shape + assert isinstance(mm_data["image"]["image_embeds"], torch.Tensor) + assert mm_data["image"]["image_embeds"].shape == image_embeds.shape + assert isinstance(mm_data["image"]["image_grid_thw"], torch.Tensor) + assert mm_data["image"]["image_grid_thw"].shape == image_grid_thw.shape + + # Verify UUIDs (None since we didn't provide any) + _assert_mm_uuids(mm_uuids, batch_size, expected_uuids=[None, None]) + + +@pytest.mark.asyncio +async def test_parse_chat_messages_multiple_images_async( + phi3v_model_config, + image_url, +): + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": image_url}}, + { + "type": "image_pil", + "image_pil": ImageAsset("cherry_blossom").pil_image, + }, + {"type": "text", "text": "What's in these images?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\n<|image_2|>\nWhat's in these images?", + } + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[None, None]) + + +def test_parse_chat_messages_placeholder_already_in_prompt( + phi3v_model_config, + image_url, +): + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "image_url", "image_url": {"url": image_url}}, + { + "type": "text", + "text": "What's in <|image_1|> and how does it compare to <|image_2|>?", # noqa: E501 + }, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + assert conversation == [ + { + "role": "user", + "content": "What's in <|image_1|> and how does it compare to <|image_2|>?", + } + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[None, None]) + + +def test_parse_chat_messages_placeholder_one_already_in_prompt( + phi3v_model_config, + image_url, +): + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "image_url", "image_url": {"url": image_url}}, + { + "type": "text", + "text": "What's in <|image_1|> and how does it compare to " + "the other one?", + }, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "<|image_2|>\nWhat's in <|image_1|> and how does it compare to " + "the other one?", + } + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[None, None]) + + +def test_parse_chat_messages_multiple_images_across_messages( + phi3v_model_config, + image_url, +): + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "What's in this image?"}, + ], + }, + {"role": "assistant", "content": "Some stuff."}, + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "What about this one?"}, + ], + }, + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + {"role": "user", "content": "<|image_1|>\nWhat's in this image?"}, + {"role": "assistant", "content": "Some stuff."}, + {"role": "user", "content": "<|image_2|>\nWhat about this one?"}, + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[None, None]) + + +def test_parse_chat_messages_multiple_images_with_uuids_across_messages( + phi3v_model_config, + image_url, +): + image_uuid = str(hash(image_url)) + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": image_uuid, + }, + {"type": "text", "text": "What's in this image?"}, + ], + }, + {"role": "assistant", "content": "Some stuff."}, + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": image_uuid, + }, + {"type": "text", "text": "What about this one?"}, + ], + }, + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + {"role": "user", "content": "<|image_1|>\nWhat's in this image?"}, + {"role": "assistant", "content": "Some stuff."}, + {"role": "user", "content": "<|image_2|>\nWhat about this one?"}, + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[image_uuid, image_uuid]) + + +def test_parse_chat_messages_context_text_format( + phi3v_model_config, +): + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [{"type": "text", "text": "What's in this text?"}], + }, + {"role": "assistant", "content": "Some stuff."}, + {"role": "user", "content": "What about this one?"}, + ], + phi3v_model_config, + content_format="openai", + ) + + assert conversation == [ + { + "role": "user", + "content": [{"type": "text", "text": "What's in this text?"}], + }, + { + "role": "assistant", + "content": [{"type": "text", "text": "Some stuff."}], + }, + { + "role": "user", + "content": [{"type": "text", "text": "What about this one?"}], + }, + ] + assert mm_data is None + assert mm_uuids is None + + +def test_parse_chat_messages_openai_format_image_url( + phi3v_model_config, + image_url, +): + content = [ + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "What's in the image?"}, + ] + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": content, + } + ], + phi3v_model_config, + content_format="openai", + ) + + assert conversation == [ + { + "role": "user", + "content": [ + {"type": "image"}, + {"type": "text", "text": "What's in the image?"}, + ], + } + ] + _assert_mm_data_is_image_input(mm_data, 1) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[None]) + + +def test_parse_chat_messages_rejects_too_many_images_in_one_message( + phi3v_model_config, + image_url, +): + with warnings.catch_warnings(): + warnings.filterwarnings( + "ignore", + message="coroutine 'async_get_and_parse_image' was never awaited", + ) + with pytest.raises(ValueError, match="At most"): + parse_chat_messages( + [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": {"url": image_url}, + }, + { + "type": "image_url", + "image_url": {"url": image_url}, + }, + { + "type": "image_url", + "image_url": {"url": image_url}, + }, + {"type": "text", "text": "What's in these images?"}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + +def test_parse_chat_messages_rejects_too_many_images_across_messages( + phi3v_model_config, + image_url, +): + with warnings.catch_warnings(): + warnings.filterwarnings( + "ignore", + message="coroutine 'async_get_and_parse_image' was never awaited", + ) + with pytest.raises(ValueError, match="At most"): + parse_chat_messages( + [ + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": {"url": image_url}, + }, + {"type": "text", "text": "What's in this image?"}, + ], + }, + {"role": "assistant", "content": "Some stuff."}, + { + "role": "user", + "content": [ + { + "type": "image_url", + "image_url": {"url": image_url}, + }, + { + "type": "image_url", + "image_url": {"url": image_url}, + }, + {"type": "text", "text": "What about these two?"}, + ], + }, + ], + phi3v_model_config, + content_format="string", + ) + + +def test_parse_chat_messages_multiple_images_uncommon_input( + phi3v_model_config, + image_url, +): + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + "What's in these images?", + {"image_url": image_url}, + {"image_url": image_url}, + ], + } + ], + phi3v_model_config, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "<|image_1|>\n<|image_2|>\nWhat's in these images?", + } + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[None, None]) + + +def test_parse_chat_messages_multiple_images_interleave( + phi3v_model_config_mm_interleaved, + image_url, +): + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "I need you to compare this image", + }, + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "and this one"}, + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "Do they have differences?"}, + ], + } + ], + phi3v_model_config_mm_interleaved, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "I need you to compare this image\n<|image_1|>\nand this one\n<|image_2|>\n" # noqa: E501 + "Do they have differences?", + } + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[None, None]) + + +@pytest.mark.asyncio +async def test_parse_chat_messages_multiple_images_interleave_async( + phi3v_model_config_mm_interleaved, + image_url, +): + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "I need you to compare this image", + }, + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "and this one"}, + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "Do they have differences?"}, + ], + } + ], + phi3v_model_config_mm_interleaved, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "I need you to compare this image\n<|image_1|>\nand this one\n<|image_2|>\n" # noqa: E501 + "Do they have differences?", + } + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[None, None]) + + +@pytest.mark.asyncio +async def test_parse_chat_messages_multiple_images_with_uuids_interleave_async( + phi3v_model_config_mm_interleaved, + image_url, +): + image_uuid = str(hash(image_url)) + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "I need you to compare this image", + }, + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": image_uuid, + }, + {"type": "text", "text": "and this one"}, + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": image_uuid, + }, + {"type": "text", "text": "Do they have differences?"}, + ], + } + ], + phi3v_model_config_mm_interleaved, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "I need you to compare this image\n<|image_1|>\nand this one\n<|image_2|>\n" # noqa: E501 + "Do they have differences?", + } + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[image_uuid, image_uuid]) + + +def test_parse_chat_messages_multiple_images_multiple_messages_interleave( + phi3v_model_config_mm_interleaved, + image_url, +): + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What's on this image?"}, + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "Be accurate."}, + ], + }, + {"role": "assistant", "content": "Some stuff."}, + { + "role": "user", + "content": [ + {"type": "text", "text": "What's on this image?"}, + {"type": "image_url", "image_url": {"url": image_url}}, + ], + }, + ], + phi3v_model_config_mm_interleaved, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "What's on this image?\n<|image_1|>\nBe accurate.", + }, + {"role": "assistant", "content": "Some stuff."}, + {"role": "user", "content": "What's on this image?\n<|image_2|>"}, + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[None, None]) + + +def test_parse_chat_messages_multiple_images_with_uuids_multiple_messages_interleave( + phi3v_model_config_mm_interleaved, + image_url, +): + image_uuid = str(hash(image_url)) + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What's on this image?"}, + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": image_uuid, + }, + {"type": "text", "text": "Be accurate."}, + ], + }, + {"role": "assistant", "content": "Some stuff."}, + { + "role": "user", + "content": [ + {"type": "text", "text": "What's on this image?"}, + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": image_uuid, + }, + ], + }, + ], + phi3v_model_config_mm_interleaved, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "What's on this image?\n<|image_1|>\nBe accurate.", + }, + {"role": "assistant", "content": "Some stuff."}, + {"role": "user", "content": "What's on this image?\n<|image_2|>"}, + ] + _assert_mm_data_is_image_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[image_uuid, image_uuid]) + + +def test_parse_chat_messages_multiple_modals_multiple_messages_interleave( + qwen25omni_model_config_mm_interleaved, + image_url, + video_url, + audio_url, +): + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What's on this image?"}, + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "Now listen to this audio"}, + {"type": "audio_url", "audio_url": {"url": audio_url}}, + ], + }, + {"role": "assistant", "content": "Some stuff."}, + { + "role": "user", + "content": [ + {"type": "text", "text": "What's on this image?"}, + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "And what's in the video?"}, + {"type": "video_url", "video_url": {"url": video_url}}, + ], + }, + ], + qwen25omni_model_config_mm_interleaved, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "What's on this image?\n<|vision_start|><|IMAGE|><|vision_end|>" + "\nNow listen to this audio\nAudio 1: <|audio_bos|><|AUDIO|><|audio_eos|>", + }, + {"role": "assistant", "content": "Some stuff."}, + { + "role": "user", + "content": "What's on this image?\n<|vision_start|><|IMAGE|><|vision_end|>" + "\nAnd what's in the video?\n<|vision_start|><|VIDEO|><|vision_end|>", + }, + ] + + _assert_mm_data_inputs(mm_data, {"image": 2, "video": 1, "audio": 1}) + _assert_mm_uuids(mm_uuids, 2, modality="image", expected_uuids=[None, None]) + _assert_mm_uuids(mm_uuids, 1, modality="video", expected_uuids=[None]) + _assert_mm_uuids(mm_uuids, 1, modality="audio", expected_uuids=[None]) + + +def test_parse_chat_messages_multiple_modals_with_uuids_multiple_messages_interleave( + qwen25omni_model_config_mm_interleaved, + image_url, + video_url, + audio_url, +): + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What's on this image?"}, + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": "image_123", + }, + {"type": "text", "text": "Now listen to this audio"}, + { + "type": "audio_url", + "audio_url": {"url": audio_url}, + "uuid": "audio_123", + }, + ], + }, + {"role": "assistant", "content": "Some stuff."}, + { + "role": "user", + "content": [ + {"type": "text", "text": "What's on this image?"}, + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": "image_123", + }, + {"type": "text", "text": "And what's in the video?"}, + { + "type": "video_url", + "video_url": {"url": video_url}, + "uuid": "video_123", + }, + ], + }, + ], + qwen25omni_model_config_mm_interleaved, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "What's on this image?\n<|vision_start|><|IMAGE|><|vision_end|>" + "\nNow listen to this audio\nAudio 1: <|audio_bos|><|AUDIO|><|audio_eos|>", + }, + {"role": "assistant", "content": "Some stuff."}, + { + "role": "user", + "content": "What's on this image?\n<|vision_start|><|IMAGE|><|vision_end|>" + "\nAnd what's in the video?\n<|vision_start|><|VIDEO|><|vision_end|>", + }, + ] + + _assert_mm_data_inputs(mm_data, {"image": 2, "video": 1, "audio": 1}) + _assert_mm_uuids( + mm_uuids, 2, modality="image", expected_uuids=["image_123", "image_123"] + ) + _assert_mm_uuids(mm_uuids, 1, modality="video", expected_uuids=["video_123"]) + _assert_mm_uuids(mm_uuids, 1, modality="audio", expected_uuids=["audio_123"]) + + +def test_parse_chat_messages_multiple_modals_with_uuids_multiple_empty_media_messages_interleave( # noqa: E501 + qwen25omni_model_config_mm_interleaved, + image_url, + video_url, + audio_url, +): + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What's on this image?"}, + { + "type": "image_url", + "image_url": None, + "uuid": "image_123", + }, + {"type": "text", "text": "Now listen to this audio"}, + { + "type": "audio_url", + "audio_url": None, + "uuid": "audio_123", + }, + ], + }, + {"role": "assistant", "content": "Some stuff."}, + { + "role": "user", + "content": [ + {"type": "text", "text": "What's on this image?"}, + { + "type": "image_url", + "image_url": None, + "uuid": "image_123", + }, + {"type": "text", "text": "And what's in the video?"}, + { + "type": "video_url", + "video_url": None, + "uuid": "video_123", + }, + ], + }, + ], + qwen25omni_model_config_mm_interleaved, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "What's on this image?\n<|vision_start|><|IMAGE|><|vision_end|>" + "\nNow listen to this audio\nAudio 1: <|audio_bos|><|AUDIO|><|audio_eos|>", + }, + {"role": "assistant", "content": "Some stuff."}, + { + "role": "user", + "content": "What's on this image?\n<|vision_start|><|IMAGE|><|vision_end|>" + "\nAnd what's in the video?\n<|vision_start|><|VIDEO|><|vision_end|>", + }, + ] + + _assert_mm_data_inputs( + mm_data, + {"image": 2, "video": 1, "audio": 1}, + skipped_media_indices={"image": [0, 1], "video": [0], "audio": [0]}, + ) + _assert_mm_uuids( + mm_uuids, 2, modality="image", expected_uuids=["image_123", "image_123"] + ) + _assert_mm_uuids(mm_uuids, 1, modality="video", expected_uuids=["video_123"]) + _assert_mm_uuids(mm_uuids, 1, modality="audio", expected_uuids=["audio_123"]) + + +def test_parse_chat_messages_multiple_modals_with_partial_uuids_multiple_messages_interleave( # noqa: E501 + qwen25omni_model_config_mm_interleaved, + image_url, + video_url, + audio_url, +): + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What's on this image?"}, + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": "image_123", + }, + {"type": "text", "text": "Now listen to this audio"}, + {"type": "audio_url", "audio_url": {"url": audio_url}}, + ], + }, + {"role": "assistant", "content": "Some stuff."}, + { + "role": "user", + "content": [ + {"type": "text", "text": "What's on this image?"}, + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "text", "text": "And what's in the video?"}, + { + "type": "video_url", + "video_url": {"url": video_url}, + "uuid": "video_123", + }, + ], + }, + ], + qwen25omni_model_config_mm_interleaved, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "What's on this image?\n<|vision_start|><|IMAGE|><|vision_end|>" + "\nNow listen to this audio\nAudio 1: <|audio_bos|><|AUDIO|><|audio_eos|>", + }, + {"role": "assistant", "content": "Some stuff."}, + { + "role": "user", + "content": "What's on this image?\n<|vision_start|><|IMAGE|><|vision_end|>" + "\nAnd what's in the video?\n<|vision_start|><|VIDEO|><|vision_end|>", + }, + ] + + _assert_mm_data_inputs(mm_data, {"image": 2, "video": 1, "audio": 1}) + _assert_mm_uuids(mm_uuids, 2, modality="image", expected_uuids=["image_123", None]) + _assert_mm_uuids(mm_uuids, 1, modality="video", expected_uuids=["video_123"]) + _assert_mm_uuids(mm_uuids, 1, modality="audio", expected_uuids=[None]) + + +def test_parse_chat_messages_multiple_images_interleave_with_placeholders( + phi3v_model_config_mm_interleaved, + image_url, +): + with pytest.raises( + ValueError, + match=r"Found more '<|image_1|>' placeholders in input prompt " + "than actual multimodal data items.", + ): + parse_chat_messages( + [ + { + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": image_url}}, + {"type": "image_url", "image_url": {"url": image_url}}, + { + "type": "text", + "text": "I need you to compare this image\n<|image_1|>\nand this one\n<|image_2|>\n" # noqa: E501 + "Do they have differences?", + }, + ], + } + ], + phi3v_model_config_mm_interleaved, + content_format="string", + ) + + +def test_parse_chat_messages_include_thinking_chunk(mistral_model_config): + messages = [ + { + "role": "system", + "content": [ + {"type": "text", "text": "You are a helpful assistant."}, + { + "type": "thinking", + "closed": True, + "thinking": "Only return the answer when you are confident.", + }, + ], + }, + {"role": "user", "content": "What is 2+2?"}, + { + "role": "assistant", + "content": [ + {"type": "text", "text": "Let me think about it."}, + {"type": "thinking", "closed": True, "thinking": "2+2 = 4"}, + { + "type": "text", + "text": "The answer is 4.", + }, + ], + }, + ] + + conversation_with_thinking, _, _ = parse_chat_messages( + messages, + mistral_model_config, + content_format="openai", + ) + + expected_conversation = [ + { + "role": "system", + "content": [ + {"type": "text", "text": "You are a helpful assistant."}, + { + "type": "text", + "text": "Only return the answer when you are confident.", + }, + ], + }, + { + "role": "user", + "content": [{"type": "text", "text": "What is 2+2?"}], + }, + { + "role": "assistant", + "content": [ + {"type": "text", "text": "Let me think about it."}, + {"type": "text", "text": "2+2 = 4"}, + {"type": "text", "text": "The answer is 4."}, + ], + }, + ] + + assert conversation_with_thinking == expected_conversation + + +def test_parse_chat_messages_single_empty_audio_with_uuid( + qwen2_audio_model_config, +): + audio_uuid = "abcd" + conversation, mm_data, mm_uuids = parse_chat_messages( + [ + { + "role": "user", + "content": [ + { + "type": "input_audio", + "input_audio": {}, + "uuid": audio_uuid, + }, + {"type": "text", "text": "What does the audio say?"}, + ], + } + ], + qwen2_audio_model_config, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "Audio 1: <|audio_bos|><|AUDIO|><|audio_eos|>\nWhat does the " + "audio say?", + } + ] + _assert_mm_data_inputs(mm_data, {"audio": 1}) + _assert_mm_uuids(mm_uuids, 1, modality="audio", expected_uuids=[audio_uuid]) + + +@pytest.mark.asyncio +async def test_parse_chat_messages_single_empty_audio_with_uuid_async( + qwen2_audio_model_config, +): + audio_uuid = "abcd" + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + [ + { + "role": "user", + "content": [ + { + "type": "input_audio", + "input_audio": {}, + "uuid": audio_uuid, + }, + {"type": "text", "text": "What does the audio say?"}, + ], + } + ], + qwen2_audio_model_config, + content_format="string", + ) + + assert conversation == [ + { + "role": "user", + "content": "Audio 1: <|audio_bos|><|AUDIO|><|audio_eos|>\nWhat does the " + "audio say?", + } + ] + _assert_mm_data_inputs(mm_data, {"audio": 1}) + _assert_mm_uuids(mm_uuids, 1, modality="audio", expected_uuids=[audio_uuid]) + + +def test_parse_chat_messages_image_vision_chunk( + kimi_k2_5_model_config, + image_url, +): + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Analyze this image."}, + { + "type": "image_url", + "image_url": {"url": image_url}, + }, + ], + } + ] + + conversation, mm_data, mm_uuids = parse_chat_messages( + messages, + kimi_k2_5_model_config, + content_format="string", + ) + + placeholder = "<|media_begin|>image<|media_content|><|media_pad|><|media_end|>" + expected_conversation = [ + { + "role": "user", + "content": f"{placeholder}\nAnalyze this image.", + } + ] + + assert conversation == expected_conversation + _assert_mm_data_is_vision_chunk_input(mm_data, 1) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[None], modality="vision_chunk") + + +def test_parse_chat_messages_video_vision_chunk( + kimi_k2_5_model_config, + video_url, +): + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Analyze this video."}, + { + "type": "video_url", + "video_url": {"url": video_url}, + }, + ], + } + ] + + conversation, mm_data, mm_uuids = parse_chat_messages( + messages, + kimi_k2_5_model_config, + content_format="string", + ) + + placeholder = "<|kimi_k25_video_placeholder|>" + expected_conversation = [ + { + "role": "user", + "content": f"{placeholder}\nAnalyze this video.", + } + ] + + assert conversation == expected_conversation + _assert_mm_data_is_vision_chunk_input(mm_data, 1) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[None], modality="vision_chunk") + + +def test_parse_chat_messages_image_vision_chunk_with_uuid( + kimi_k2_5_model_config, + image_url, +): + image_uuid = "image_123" + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Analyze this image."}, + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": image_uuid, + }, + ], + } + ] + + conversation, mm_data, mm_uuids = parse_chat_messages( + messages, + kimi_k2_5_model_config, + content_format="string", + ) + + placeholder = "<|media_begin|>image<|media_content|><|media_pad|><|media_end|>" + expected_conversation = [ + { + "role": "user", + "content": f"{placeholder}\nAnalyze this image.", + } + ] + + assert conversation == expected_conversation + _assert_mm_data_is_vision_chunk_input(mm_data, 1) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[image_uuid], modality="vision_chunk") + + +def test_parse_chat_messages_video_vision_chunk_with_uuid( + kimi_k2_5_model_config, + video_url, +): + video_uuid = "video_456" + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Analyze this video."}, + { + "type": "video_url", + "video_url": {"url": video_url}, + "uuid": video_uuid, + }, + ], + } + ] + + conversation, mm_data, mm_uuids = parse_chat_messages( + messages, + kimi_k2_5_model_config, + content_format="string", + ) + + placeholder = "<|kimi_k25_video_placeholder|>" + expected_conversation = [ + { + "role": "user", + "content": f"{placeholder}\nAnalyze this video.", + } + ] + + assert conversation == expected_conversation + _assert_mm_data_is_vision_chunk_input(mm_data, 1) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[video_uuid], modality="vision_chunk") + + +def test_parse_chat_messages_mixed_vision_chunk( + kimi_k2_5_model_config, + image_url, + video_url, +): + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Analyze this image and video."}, + { + "type": "image_url", + "image_url": {"url": image_url}, + }, + { + "type": "video_url", + "video_url": {"url": video_url}, + }, + ], + } + ] + + conversation, mm_data, mm_uuids = parse_chat_messages( + messages, + kimi_k2_5_model_config, + content_format="string", + ) + + image_placeholder = ( + "<|media_begin|>image<|media_content|><|media_pad|><|media_end|>" + ) + video_placeholder = "<|kimi_k25_video_placeholder|>" + expected_conversation = [ + { + "role": "user", + "content": ( + f"{image_placeholder}\n{video_placeholder}\n" + "Analyze this image and video." + ), + } + ] + + assert conversation == expected_conversation + _assert_mm_data_is_vision_chunk_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[None, None], modality="vision_chunk") + + +def test_parse_chat_messages_mixed_vision_chunk_with_uuid( + kimi_k2_5_model_config, + image_url, + video_url, +): + image_uuid = "image_123" + video_uuid = "video_456" + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Analyze this image and video."}, + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": image_uuid, + }, + { + "type": "video_url", + "video_url": {"url": video_url}, + "uuid": video_uuid, + }, + ], + } + ] + + conversation, mm_data, mm_uuids = parse_chat_messages( + messages, + kimi_k2_5_model_config, + content_format="string", + ) + + image_placeholder = ( + "<|media_begin|>image<|media_content|><|media_pad|><|media_end|>" + ) + video_placeholder = "<|kimi_k25_video_placeholder|>" + expected_conversation = [ + { + "role": "user", + "content": ( + f"{image_placeholder}\n{video_placeholder}\n" + "Analyze this image and video." + ), + } + ] + + assert conversation == expected_conversation + _assert_mm_data_is_vision_chunk_input(mm_data, 2) + _assert_mm_uuids( + mm_uuids, 2, expected_uuids=[image_uuid, video_uuid], modality="vision_chunk" + ) + + +@pytest.mark.asyncio +async def test_parse_chat_messages_mixed_vision_chunk_async( + kimi_k2_5_model_config, + image_url, + video_url, +): + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Analyze this image and video."}, + { + "type": "image_url", + "image_url": {"url": image_url}, + }, + { + "type": "video_url", + "video_url": {"url": video_url}, + }, + ], + } + ] + + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + messages, + kimi_k2_5_model_config, + content_format="string", + ) + + image_placeholder = ( + "<|media_begin|>image<|media_content|><|media_pad|><|media_end|>" + ) + video_placeholder = "<|kimi_k25_video_placeholder|>" + expected_conversation = [ + { + "role": "user", + "content": ( + f"{image_placeholder}\n{video_placeholder}\n" + "Analyze this image and video." + ), + } + ] + + assert conversation == expected_conversation + _assert_mm_data_is_vision_chunk_input(mm_data, 2) + _assert_mm_uuids(mm_uuids, 2, expected_uuids=[None, None], modality="vision_chunk") + + +@pytest.mark.asyncio +async def test_parse_chat_messages_mixed_vision_chunk_with_uuid_async( + kimi_k2_5_model_config, + image_url, + video_url, +): + image_uuid = "image_123" + video_uuid = "video_456" + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Analyze this image and video."}, + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": image_uuid, + }, + { + "type": "video_url", + "video_url": {"url": video_url}, + "uuid": video_uuid, + }, + ], + } + ] + + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + messages, + kimi_k2_5_model_config, + content_format="string", + ) + + image_placeholder = ( + "<|media_begin|>image<|media_content|><|media_pad|><|media_end|>" + ) + video_placeholder = "<|kimi_k25_video_placeholder|>" + expected_conversation = [ + { + "role": "user", + "content": ( + f"{image_placeholder}\n{video_placeholder}\n" + "Analyze this image and video." + ), + } + ] + + assert conversation == expected_conversation + _assert_mm_data_is_vision_chunk_input(mm_data, 2) + _assert_mm_uuids( + mm_uuids, 2, expected_uuids=[image_uuid, video_uuid], modality="vision_chunk" + ) + + +@pytest.mark.asyncio +async def test_parse_chat_messages_image_vision_chunk_async( + kimi_k2_5_model_config, + image_url, +): + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Analyze this image."}, + { + "type": "image_url", + "image_url": {"url": image_url}, + }, + ], + } + ] + + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + messages, + kimi_k2_5_model_config, + content_format="string", + ) + + placeholder = "<|media_begin|>image<|media_content|><|media_pad|><|media_end|>" + expected_conversation = [ + { + "role": "user", + "content": f"{placeholder}\nAnalyze this image.", + } + ] + + assert conversation == expected_conversation + _assert_mm_data_is_vision_chunk_input(mm_data, 1) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[None], modality="vision_chunk") + + +@pytest.mark.asyncio +async def test_parse_chat_messages_video_vision_chunk_async( + kimi_k2_5_model_config, + video_url, +): + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Analyze this video."}, + { + "type": "video_url", + "video_url": {"url": video_url}, + }, + ], + } + ] + + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + messages, + kimi_k2_5_model_config, + content_format="string", + ) + + placeholder = "<|kimi_k25_video_placeholder|>" + expected_conversation = [ + { + "role": "user", + "content": f"{placeholder}\nAnalyze this video.", + } + ] + + assert conversation == expected_conversation + _assert_mm_data_is_vision_chunk_input(mm_data, 1) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[None], modality="vision_chunk") + + +@pytest.mark.asyncio +async def test_parse_chat_messages_image_vision_chunk_with_uuid_async( + kimi_k2_5_model_config, + image_url, +): + image_uuid = "image_123" + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Analyze this image."}, + { + "type": "image_url", + "image_url": {"url": image_url}, + "uuid": image_uuid, + }, + ], + } + ] + + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + messages, + kimi_k2_5_model_config, + content_format="string", + ) + + placeholder = "<|media_begin|>image<|media_content|><|media_pad|><|media_end|>" + expected_conversation = [ + { + "role": "user", + "content": f"{placeholder}\nAnalyze this image.", + } + ] + + assert conversation == expected_conversation + _assert_mm_data_is_vision_chunk_input(mm_data, 1) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[image_uuid], modality="vision_chunk") + + +@pytest.mark.asyncio +async def test_parse_chat_messages_video_vision_chunk_with_uuid_async( + kimi_k2_5_model_config, + video_url, +): + video_uuid = "video_456" + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "Analyze this video."}, + { + "type": "video_url", + "video_url": {"url": video_url}, + "uuid": video_uuid, + }, + ], + } + ] + + conversation, mm_data, mm_uuids = await parse_chat_messages_async( + messages, + kimi_k2_5_model_config, + content_format="string", + ) + + placeholder = "<|kimi_k25_video_placeholder|>" + expected_conversation = [ + { + "role": "user", + "content": f"{placeholder}\nAnalyze this video.", + } + ] + + assert conversation == expected_conversation + _assert_mm_data_is_vision_chunk_input(mm_data, 1) + _assert_mm_uuids(mm_uuids, 1, expected_uuids=[video_uuid], modality="vision_chunk") + + +def test_postprocess_messages_null_arguments_string(): + """arguments="null" must not reach the chat template as Python None. + + json.loads("null") returns None, which causes Jinja2 templates that call + tc.arguments.items() to raise 'None' has no attribute 'items'. + The function should coerce it to {} instead. + """ + messages: list[ConversationMessage] = [ + { + "role": "assistant", + "content": None, + "tool_calls": [ + { + "id": "call_1", + "type": "function", + "function": {"name": "get_current_time", "arguments": "null"}, + } + ], + } + ] + _postprocess_messages(messages) + tool_calls = messages[0]["tool_calls"] + assert tool_calls is not None + assert tool_calls[0]["function"]["arguments"] == {} diff --git a/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_context.py b/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_context.py new file mode 100644 index 0000000000000000000000000000000000000000..ad160eedd2b9789eb12dd9a284579f06b663bfaa --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_context.py @@ -0,0 +1,910 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from unittest.mock import patch + +import pytest +from openai_harmony import Author, Message, Role, TextContent + +from vllm.entrypoints.openai.responses.context import ( + HarmonyContext, + SimpleContext, + TurnMetrics, +) +from vllm.outputs import CompletionOutput, RequestOutput +from vllm.parser.harmony import ChunkResult, HarmonyParser, Segment + + +def create_mock_request_output( + prompt_token_ids=None, + output_token_ids=None, + num_cached_tokens=0, + finished=True, +): + """Helper function to create a mock RequestOutput object for testing.""" + outputs = [] + token_ids = output_token_ids if output_token_ids is not None else [] + outputs = [ + CompletionOutput( + index=0, + text="Test output", + token_ids=token_ids, + cumulative_logprob=0.0, + logprobs=None, + finish_reason=None, + stop_reason=None, + ) + ] + + return RequestOutput( + request_id="test-id", + prompt="Test prompt", + prompt_token_ids=prompt_token_ids, + prompt_logprobs=None, + outputs=outputs, + finished=finished, + num_cached_tokens=num_cached_tokens, + ) + + +async def generate_mock_outputs( + num_turns, prompt_token_counts, output_token_counts, cached_token_counts=None +): + """Generate a sequence of mock RequestOutput objects to simulate multiple + turns.""" + if cached_token_counts is None: + cached_token_counts = [0] * num_turns + + for i in range(num_turns): + # Create mock prompt token IDs and output token IDs + prompt_token_ids = list(range(1, prompt_token_counts[i] + 1)) + output_token_ids = list(range(1, output_token_counts[i] + 1)) + + # Create and yield the RequestOutput + yield create_mock_request_output( + prompt_token_ids=prompt_token_ids, + output_token_ids=output_token_ids, + num_cached_tokens=cached_token_counts[i], + ) + + +class FakeHarmonyParser(HarmonyParser): + def __init__(self): + # Skip HarmonyParser initialization and script outputs directly. + self.reasoning_parser = None + self.tool_parser = None + self._chunk_results: list[ChunkResult] = [] + self._flush_results: list[Segment | None] = [] + self.processed_chunks: list[list[int]] = [] + + def enqueue_chunk_result( + self, + segments: list[Segment] | None = None, + reasoning_token_count: int = 0, + ) -> None: + self._chunk_results.append( + ChunkResult( + segments=[] if segments is None else segments, + reasoning_token_count=reasoning_token_count, + ) + ) + + def enqueue_flush_result(self, segment: Segment | None) -> None: + self._flush_results.append(segment) + + def process_chunk(self, token_ids) -> ChunkResult: + self.processed_chunks.append(list(token_ids)) + if self._chunk_results: + return self._chunk_results.pop(0) + return ChunkResult(segments=[], reasoning_token_count=0) + + def flush(self) -> Segment | None: + if self._flush_results: + return self._flush_results.pop(0) + return None + + +def make_harmony_context( + messages=None, available_tools=None, function_tool_names=None +) -> tuple[HarmonyContext, FakeHarmonyParser]: + fake_parser = FakeHarmonyParser() + context = HarmonyContext( + messages=[] if messages is None else messages, + available_tools=[] if available_tools is None else available_tools, + function_tool_names=function_tool_names, + response_parser=fake_parser, + ) + return context, fake_parser + + +def test_single_turn_token_counting(): + """Test token counting behavior for a single turn.""" + # Create a context + context, _ = make_harmony_context() + + # Create a mock RequestOutput with specific token counts + mock_output = create_mock_request_output( + prompt_token_ids=[1, 2, 3, 4, 5], # 5 prompt tokens + output_token_ids=[6, 7, 8], # 3 output tokens + num_cached_tokens=2, # 2 cached tokens + ) + + # Append the output to the context + context.append_output(mock_output) + + # Verify the token counts + assert context.num_prompt_tokens == 5 + assert context.num_output_tokens == 3 + assert context.num_cached_tokens == 2 + assert context.num_tool_output_tokens == 0 # No tool tokens in first turn + + # Verify internal state tracking + assert not context.is_first_turn + assert len(context.all_turn_metrics) == 1 + previous_turn = context.all_turn_metrics[0] + assert previous_turn.input_tokens == 5 + assert previous_turn.output_tokens == 3 + assert previous_turn.cached_input_tokens == 2 + assert previous_turn.tool_output_tokens == 0 + + +@pytest.mark.asyncio +async def test_multi_turn_token_counting(): + """Test token counting behavior across multiple turns with tool output.""" + # Create a context + context, _ = make_harmony_context(available_tools=["browser"]) + + # Simulate a conversation with 3 turns + # Turn 1: prefill 5, decode 3, tool 7 + # Turn 2: prefill 15, cached 5, decode 4, tool 1 + # Turn 3: prefill 20, cached 15, decode 5 + prompt_token_counts = [5, 15, 20] + output_token_counts = [3, 4, 5] + cached_token_counts = [0, 5, 15] + mock_generator = generate_mock_outputs( + 3, prompt_token_counts, output_token_counts, cached_token_counts + ) + + # First turn - initial prompt and response + mock_output1 = await anext(mock_generator) + context.append_output(mock_output1) + + # At this point, we should have 5 prompt tokens and 3 output tokens + assert context.num_prompt_tokens == 5 + assert context.num_output_tokens == 3 + assert context.num_tool_output_tokens == 0 + + # Second turn - after tool output + mock_output2 = await anext(mock_generator) + context.append_output(mock_output2) + # Current prompt tokens (15) - last_turn_input_tokens (5) - + # last_turn_output_tokens (3) = 7 + expected_tool_output = 7 + + assert context.num_prompt_tokens == 5 + 15 + assert context.num_output_tokens == 3 + 4 + assert context.num_tool_output_tokens == expected_tool_output + assert context.num_cached_tokens == 5 + + # Third turn - final response + mock_output3 = await anext(mock_generator) + context.append_output(mock_output3) + # Additional tool output tokens from third turn: + # Current prompt (20) - last_turn_input_tokens (15) - + # last_turn_output_tokens (4) = 1 + expected_tool_output = 7 + 1 + + assert context.num_prompt_tokens == 5 + 15 + 20 + assert context.num_output_tokens == 3 + 4 + 5 + assert context.num_tool_output_tokens == expected_tool_output + assert context.num_cached_tokens == 5 + 15 + + # Validate all turn metrics + assert len(context.all_turn_metrics) == 3 + for i, turn in enumerate(context.all_turn_metrics): + assert turn.input_tokens == prompt_token_counts[i] + assert turn.output_tokens == output_token_counts[i] + assert turn.cached_input_tokens == cached_token_counts[i] + assert context.all_turn_metrics[1].tool_output_tokens == 7 + assert context.all_turn_metrics[2].tool_output_tokens == 1 + + +def test_empty_output_tokens(): + """Test behavior when RequestOutput has empty output tokens.""" + context, _ = make_harmony_context() + + # Create a RequestOutput with empty output tokens + mock_output = create_mock_request_output( + prompt_token_ids=[1, 2, 3], # 3 prompt tokens + output_token_ids=[], # Empty output tokens list + num_cached_tokens=1, + ) + + context.append_output(mock_output) + + # Should handle empty outputs gracefully + assert context.num_prompt_tokens == 3 + assert context.num_output_tokens == 0 # No output tokens + assert context.num_cached_tokens == 1 + assert context.num_tool_output_tokens == 0 + + +def test_missing_prompt_token_ids(): + """Test behavior when RequestOutput has None prompt_token_ids.""" + context, _ = make_harmony_context() + + mock_output = create_mock_request_output( + prompt_token_ids=None, # No prompt token IDs + output_token_ids=[1, 2], # 2 output tokens + num_cached_tokens=0, + ) + + # Logger.error will be called, but we don't need to check for warnings + # here Just ensure it doesn't raise an exception + context.append_output(mock_output) + + # Should handle missing prompt tokens gracefully + assert context.num_prompt_tokens == 0 + assert context.num_output_tokens == 2 + assert context.num_cached_tokens == 0 + assert context.num_tool_output_tokens == 0 + + +def test_reasoning_tokens_counting(): + """Test that reasoning tokens are counted correctly.""" + context, parser = make_harmony_context() + parser.enqueue_chunk_result(reasoning_token_count=4) + + mock_output = create_mock_request_output( + prompt_token_ids=[1, 2, 3], + output_token_ids=[4, 5, 6, 7], # 4 tokens, all in reasoning + num_cached_tokens=0, + ) + + context.append_output(mock_output) + + # All output tokens should be counted as reasoning + assert context.num_reasoning_tokens == 4 + assert context.num_output_tokens == 4 + + +def test_preamble_tokens_not_counted_as_reasoning(): + """Preambles (commentary with no recipient) are visible user text, + not hidden reasoning. They must NOT inflate num_reasoning_tokens.""" + context, parser = make_harmony_context() + parser.enqueue_chunk_result(reasoning_token_count=0) + + mock_output = create_mock_request_output( + prompt_token_ids=[1, 2, 3], + output_token_ids=[4, 5, 6], + num_cached_tokens=0, + ) + context.append_output(mock_output) + + assert context.num_reasoning_tokens == 0 + assert context.num_output_tokens == 3 + + +def test_commentary_with_recipient_counted_as_reasoning(): + """Commentary directed at a tool (recipient != None) is hidden from + the user, so it should still count as reasoning tokens.""" + context, parser = make_harmony_context() + parser.enqueue_chunk_result(reasoning_token_count=3) + + mock_output = create_mock_request_output( + prompt_token_ids=[1, 2, 3], + output_token_ids=[4, 5, 6], + num_cached_tokens=0, + ) + context.append_output(mock_output) + + assert context.num_reasoning_tokens == 3 + assert context.num_output_tokens == 3 + + +def test_zero_tokens_edge_case(): + """Test behavior with all zero token counts.""" + context, _ = make_harmony_context() + + # Create a request with empty lists (not None) for both prompt and + # output tokens + mock_output = create_mock_request_output( + prompt_token_ids=[], # Empty prompt tokens + output_token_ids=[], # Empty output tokens + num_cached_tokens=0, + ) + + context.append_output(mock_output) + + # All counts should be zero + assert context.num_prompt_tokens == 0 + assert context.num_output_tokens == 0 + assert context.num_cached_tokens == 0 + assert context.num_tool_output_tokens == 0 + assert context.num_reasoning_tokens == 0 + + +@pytest.mark.asyncio +async def test_single_turn_no_tool_output(): + """Test that first turn never generates tool output tokens.""" + context, _ = make_harmony_context(available_tools=["browser"]) + + # Even with large prompt in first turn, no tool tokens should be counted + mock_output = create_mock_request_output( + prompt_token_ids=list(range(100)), # 100 tokens + output_token_ids=[1, 2, 3], + num_cached_tokens=0, + ) + + context.append_output(mock_output) + + # First turn should never have tool output tokens + assert context.num_tool_output_tokens == 0 + assert context.is_first_turn is False # Should be updated after first turn + + +@pytest.mark.asyncio +async def test_negative_tool_tokens_edge_case(): + """Test edge case where calculation could result in negative tool + tokens. We should log an error and clamp the value to 0.""" + # Use patch to check if logger.error was called + with patch("vllm.entrypoints.openai.responses.context.logger.error") as mock_log: + context, _ = make_harmony_context(available_tools=["browser"]) + + # First turn + mock_output1 = create_mock_request_output( + prompt_token_ids=list(range(10)), # 10 tokens + output_token_ids=[1, 2, 3, 4, 5], # 5 tokens + ) + context.append_output(mock_output1) + + # Second turn with fewer new tokens than previous output + # This could happen in edge cases with aggressive caching + mock_output2 = create_mock_request_output( + prompt_token_ids=list(range(12)), # 12 tokens (only 2 new) + output_token_ids=[6, 7], # 2 tokens + ) + context.append_output(mock_output2) + + # Calculated negative tool tokens (12 - 10 - 5 = -3) should be clamped + # to 0 and an error should be logged + assert context.num_tool_output_tokens == 0 + assert context.num_prompt_tokens == 10 + 12 + assert context.num_output_tokens == 5 + 2 + + # Verify the error was logged properly + mock_log.assert_called_once() + + # Extract the actual log message and arguments from the call + args, _ = mock_log.call_args + log_message = args[0] + + # Check for key parts of the message + assert "Negative tool output tokens calculated" in log_message + assert "-3" in str(args) # Check that -3 is in the arguments + + +@pytest.mark.asyncio +async def test_streaming_multi_turn_token_counting(): + """Test token counting for streaming multi-turn conversations. + + This test focuses on how HarmonyContext counts tokens in a + multi-turn conversation with streaming (token-by-token) outputs and + message boundaries. + """ + # Create a streaming context + context, parser = make_harmony_context(available_tools=["browser"]) + + num_prompt_tokens = [3, 8, 13] + num_output_tokens = [3, 3, 2] + num_cached_tokens = [0, 3, 8] + + # Simulate three turns of conversation: + # Turn 1: stream tokens one by one, then finish the message + # Turn 2: new prompt, stream more tokens with a reasoning segment + # Turn 3: new prompt with tool output and cached tokens + + # First turn: 3 tokens streamed one by one + # First token of first turn + context.append_output( + create_mock_request_output( + prompt_token_ids=[1, 2, 3], # 3 prompt tokens + output_token_ids=[101], # Single token + num_cached_tokens=num_cached_tokens[0], + finished=False, # Not end of message yet + ) + ) + + # Second token of first turn + context.append_output( + create_mock_request_output( + output_token_ids=[102], + finished=False, + ) + ) + + # Last token of first turn (finished=True signals end of message) + context.append_output( + create_mock_request_output( + output_token_ids=[103], + finished=True, # End of message + ) + ) + + # Check token counts after first turn + assert context.num_prompt_tokens == 3 # Initial prompt tokens + assert context.num_output_tokens == 3 # Three output tokens + assert context.num_cached_tokens == 0 + assert context.num_tool_output_tokens == 0 # No tool output in first turn + assert context.first_tok_of_message is True # Ready for next message + + # First token of second turn + parser.enqueue_chunk_result(reasoning_token_count=1) + context.append_output( + create_mock_request_output( + prompt_token_ids=[ + 1, + 2, + 3, + 101, + 102, + 103, + 4, + 5, + ], # 8 tokens (includes previous) + output_token_ids=[201], + num_cached_tokens=num_cached_tokens[1], # Some tokens cached + finished=False, + ) + ) + + # More tokens in reasoning channel + parser.enqueue_chunk_result(reasoning_token_count=1) + context.append_output( + create_mock_request_output( + output_token_ids=[202], + finished=False, + ) + ) + + parser.enqueue_chunk_result(reasoning_token_count=1) + context.append_output( + create_mock_request_output( + output_token_ids=[203], + finished=True, # End of reasoning message + ) + ) + + # Check counts after second turn (reasoning message) + assert context.num_prompt_tokens == 3 + 8 # Initial + second prompt + assert context.num_output_tokens == 3 + 3 # First turn + second turn + assert context.num_reasoning_tokens == 3 # All tokens in analysis channel + assert context.num_cached_tokens == 3 # Cached tokens from second turn + + # Formula: this turn prompt tokens - last turn prompt - last turn output + expected_tool_tokens = 8 - 3 - 3 # = 2 + assert context.num_tool_output_tokens == expected_tool_tokens + + # Third turn (with more cached tokens) + context.append_output( + create_mock_request_output( + prompt_token_ids=[ + 1, + 2, + 3, + 101, + 102, + 103, + 4, + 5, + 201, + 202, + 203, + 6, + 7, + ], # 13 tokens + output_token_ids=[301], + num_cached_tokens=num_cached_tokens[2], # More cached tokens + finished=False, + ) + ) + + context.append_output( + create_mock_request_output( + output_token_ids=[302], + finished=True, + ) + ) + + # Final token counts check + assert context.num_prompt_tokens == sum(num_prompt_tokens) # All prompts + assert context.num_output_tokens == sum(num_output_tokens) # All outputs + assert context.num_reasoning_tokens == 3 # Unchanged from second turn + assert context.num_cached_tokens == sum( + num_cached_tokens + ) # Accumulated cached tokens + + # Additional tool tokens from third turn + # Formula: this turn prompt - last turn prompt - last turn output + additional_tool_tokens = 13 - 8 - 3 # = 2 + assert ( + context.num_tool_output_tokens == expected_tool_tokens + additional_tool_tokens + ) + + # Validate all turn metrics + assert len(context.all_turn_metrics) == 3 + for i, turn in enumerate(context.all_turn_metrics): + assert turn.input_tokens == num_prompt_tokens[i] + assert turn.output_tokens == num_output_tokens[i] + assert turn.cached_input_tokens == num_cached_tokens[i] + assert context.all_turn_metrics[1].tool_output_tokens == 2 + assert context.all_turn_metrics[2].tool_output_tokens == 2 + + +@pytest.mark.asyncio +async def test_streaming_message_synchronization(): + """Completed messages from append-local and flush segments sync into context.""" + + # Create a streaming context with some initial messages + initial_messages = [ + Message( + author=Author(role=Role.USER, name="user"), + content=[TextContent(text="Hello")], + recipient=Role.ASSISTANT, + ) + ] + context, parser = make_harmony_context(messages=initial_messages) + + # Verify initial state + assert len(context._messages) == 1 + assert context.num_init_messages == 1 + + response_text = "First response" + message = Message( + author=Author(role=Role.ASSISTANT, name="assistant"), + content=[TextContent(text=response_text)], + recipient=Role.USER, + ) + parser.enqueue_chunk_result( + segments=[ + Segment( + channel="commentary", + recipient=None, + delta="", + completed_message=message, + ) + ] + ) + + # This should sync the completed message from the latest append + context.append_output( + create_mock_request_output( + prompt_token_ids=[1, 2, 3], output_token_ids=[101], finished=False + ) + ) + + # Verify that messages were synchronized + assert len(context._messages) == 2 + + # Verify the new messages were added correctly + assert context._messages[1].content[0].text == response_text + + messages_minus_init = len(context._messages) - context.num_init_messages + assert messages_minus_init == 1 + + response_text = "Second response" + message = Message( + author=Author(role=Role.ASSISTANT, name="assistant"), + content=[TextContent(text=response_text)], + recipient=Role.USER, + ) + flush_segment = Segment( + channel="commentary", + recipient=None, + delta="", + completed_message=message, + ) + parser.enqueue_flush_result(flush_segment) + + # Create another output to trigger synchronization via flush() + context.append_output( + create_mock_request_output( + prompt_token_ids=[1, 2, 3], output_token_ids=[102], finished=True + ) + ) + + # Verify the flushed response was added, num_init_messages is still 1 + assert len(context._messages) == 3 + assert context.num_init_messages == 1 + assert context._messages[2].content[0].text == response_text + assert context.last_append_flush_status is True + assert len(context.last_append_segments) == 1 + assert context.last_append_segments[0].completed_message is message + + +def test_turn_metrics_copy_and_reset(): + """Test TurnMetrics copy and reset methods work correctly.""" + # Create a TurnMetrics with specific values + original_metrics = TurnMetrics( + input_tokens=10, + output_tokens=20, + cached_input_tokens=5, + tool_output_tokens=3, + ) + + # Test copy functionality + copied_metrics = original_metrics.copy() + + # Verify copy has same values + assert copied_metrics.input_tokens == 10 + assert copied_metrics.output_tokens == 20 + assert copied_metrics.cached_input_tokens == 5 + assert copied_metrics.tool_output_tokens == 3 + + # Verify they are separate objects + assert copied_metrics is not original_metrics + + # Modify copy to ensure independence + copied_metrics.input_tokens = 999 + assert original_metrics.input_tokens == 10 # Original unchanged + assert copied_metrics.input_tokens == 999 + + # Test reset functionality + original_metrics.reset() + + # Verify all fields are reset to zero + assert original_metrics.input_tokens == 0 + assert original_metrics.output_tokens == 0 + assert original_metrics.cached_input_tokens == 0 + assert original_metrics.tool_output_tokens == 0 + + # Verify copied metrics are unaffected by reset + assert copied_metrics.input_tokens == 999 + assert copied_metrics.output_tokens == 20 + assert copied_metrics.cached_input_tokens == 5 + assert copied_metrics.tool_output_tokens == 3 + + +# ==================== SimpleContext Tests ==================== + + +def create_simple_context_output( + text="", + token_ids=None, + prompt="Test prompt", + prompt_token_ids=None, + num_cached_tokens=0, + logprobs=None, + finished=True, +): + """Helper to create a RequestOutput with customizable text for + SimpleContext tests.""" + if token_ids is None: + token_ids = [] + return RequestOutput( + request_id="test-id", + prompt=prompt, + prompt_token_ids=prompt_token_ids, + prompt_logprobs=None, + outputs=[ + CompletionOutput( + index=0, + text=text, + token_ids=token_ids, + cumulative_logprob=0.0, + logprobs=logprobs, + finish_reason=None, + stop_reason=None, + ) + ], + finished=finished, + num_cached_tokens=num_cached_tokens, + ) + + +def test_simple_context_output_messages_empty(): + """output_messages should be empty before any output is appended.""" + context = SimpleContext() + assert context.output_messages == [] + + +def test_simple_context_output_messages_single_call(): + """Non-streaming: single append_output produces a single output message.""" + context = SimpleContext() + output = create_simple_context_output( + text="Hello world", + token_ids=[10, 20, 30], + prompt_token_ids=[1, 2, 3], + ) + context.append_output(output) + + messages = context.output_messages + assert len(messages) == 1 + assert messages[0].message == "Hello world" + assert messages[0].tokens == [10, 20, 30] + assert messages[0].type == "raw_message_tokens" + + +def test_simple_context_output_messages_streaming_consolidation(): + """Streaming: multiple append_output calls consolidate into one message.""" + context = SimpleContext() + + # Simulate 3 streaming deltas + context.append_output( + create_simple_context_output( + text="Hello", + token_ids=[10], + prompt_token_ids=[1, 2, 3], + ) + ) + context.append_output( + create_simple_context_output( + text=" world", + token_ids=[20], + prompt_token_ids=[1, 2, 3], + ) + ) + context.append_output( + create_simple_context_output( + text="!", + token_ids=[30], + prompt_token_ids=[1, 2, 3], + ) + ) + + messages = context.output_messages + assert len(messages) == 1 + assert messages[0].message == "Hello world!" + assert messages[0].tokens == [10, 20, 30] + + +def test_simple_context_output_messages_many_deltas(): + """Streaming with many small deltas still produces a single message.""" + context = SimpleContext() + + words = ["The", " quick", " brown", " fox", " jumps"] + for i, word in enumerate(words): + context.append_output( + create_simple_context_output( + text=word, + token_ids=[100 + i], + prompt_token_ids=[1, 2], + ) + ) + + messages = context.output_messages + assert len(messages) == 1 + assert messages[0].message == "The quick brown fox jumps" + assert messages[0].tokens == [100, 101, 102, 103, 104] + + +def test_simple_context_input_messages(): + """input_messages is populated on the first append_output call.""" + context = SimpleContext() + assert context.input_messages == [] + + context.append_output( + create_simple_context_output( + text="Hi", + token_ids=[10], + prompt="My prompt text", + prompt_token_ids=[1, 2, 3], + ) + ) + + assert len(context.input_messages) == 1 + assert context.input_messages[0].message == "My prompt text" + assert context.input_messages[0].tokens == [1, 2, 3] + + # Second call should not add another input message + context.append_output( + create_simple_context_output( + text=" there", + token_ids=[20], + prompt="My prompt text", + prompt_token_ids=[1, 2, 3], + ) + ) + + assert len(context.input_messages) == 1 + + +def test_simple_context_token_counting(): + """Token counting accumulates across streaming deltas.""" + context = SimpleContext() + + context.append_output( + create_simple_context_output( + text="a", + token_ids=[10, 11], + prompt_token_ids=[1, 2, 3, 4, 5], + num_cached_tokens=2, + ) + ) + context.append_output( + create_simple_context_output( + text="b", + token_ids=[12], + prompt_token_ids=[1, 2, 3, 4, 5], + num_cached_tokens=2, + ) + ) + + assert context.num_prompt_tokens == 5 + assert context.num_output_tokens == 3 # 2 + 1 + assert context.num_cached_tokens == 2 + + +def test_simple_context_final_output(): + """final_output reconstructs accumulated text and token_ids.""" + context = SimpleContext() + + context.append_output( + create_simple_context_output( + text="foo", + token_ids=[1, 2], + prompt_token_ids=[10], + ) + ) + context.append_output( + create_simple_context_output( + text="bar", + token_ids=[3], + prompt_token_ids=[10], + ) + ) + + final = context.final_output + assert final is not None + assert final.outputs[0].text == "foobar" + assert final.outputs[0].token_ids == (1, 2, 3) + + +def test_simple_context_output_messages_empty_text_with_tokens(): + """output_messages should be returned when tokens exist even if text is + empty (e.g. special tokens).""" + context = SimpleContext() + context.append_output( + create_simple_context_output( + text="", + token_ids=[99], + prompt_token_ids=[1], + ) + ) + + messages = context.output_messages + assert len(messages) == 1 + assert messages[0].message == "" + assert messages[0].tokens == [99] + + +def test_simple_context_output_messages_no_mutation(): + """Each call to output_messages returns a fresh list; callers can't + corrupt internal state.""" + context = SimpleContext() + context.append_output( + create_simple_context_output( + text="hello", + token_ids=[1], + prompt_token_ids=[10], + ) + ) + + msgs1 = context.output_messages + msgs2 = context.output_messages + assert msgs1 is not msgs2 + assert msgs1[0].message == msgs2[0].message + + # Appending more output updates the property + context.append_output( + create_simple_context_output( + text=" world", + token_ids=[2], + prompt_token_ids=[10], + ) + ) + + msgs3 = context.output_messages + assert len(msgs3) == 1 + assert msgs3[0].message == "hello world" + assert msgs3[0].tokens == [1, 2] diff --git a/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_grpc_health.py b/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_grpc_health.py new file mode 100644 index 0000000000000000000000000000000000000000..7d2b52c1e5486655ccb910367700ceafcdd9c0d0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_grpc_health.py @@ -0,0 +1,143 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +grpc = pytest.importorskip("grpc") +health_pb2 = pytest.importorskip("grpc_health.v1.health_pb2") +VllmHealthServicer = pytest.importorskip( + "smg_grpc_servicer.vllm.health_servicer" +).VllmHealthServicer + +SERVING = health_pb2.HealthCheckResponse.SERVING +NOT_SERVING = health_pb2.HealthCheckResponse.NOT_SERVING +SERVICE_UNKNOWN = health_pb2.HealthCheckResponse.SERVICE_UNKNOWN + + +@pytest.fixture +def async_llm(): + mock = MagicMock() + mock.check_health = AsyncMock() + return mock + + +@pytest.fixture +def context(): + return MagicMock(spec=grpc.aio.ServicerContext) + + +@pytest.fixture +def servicer(async_llm): + return VllmHealthServicer(async_llm) + + +@pytest.fixture +def request_msg(): + msg = MagicMock() + msg.service = "" + return msg + + +# -- Check() tests -- + + +@pytest.mark.asyncio +async def test_check_serving_overall(servicer, request_msg, context, async_llm): + request_msg.service = "" + response = await servicer.Check(request_msg, context) + assert response.status == SERVING + async_llm.check_health.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_check_serving_vllm_service(servicer, request_msg, context, async_llm): + request_msg.service = "vllm.grpc.engine.VllmEngine" + response = await servicer.Check(request_msg, context) + assert response.status == SERVING + async_llm.check_health.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_check_not_serving_engine_errored( + servicer, request_msg, context, async_llm +): + async_llm.check_health = AsyncMock(side_effect=Exception("engine dead")) + request_msg.service = "" + response = await servicer.Check(request_msg, context) + assert response.status == NOT_SERVING + + +@pytest.mark.asyncio +async def test_check_not_serving_shutting_down( + servicer, request_msg, context, async_llm +): + servicer.set_not_serving() + request_msg.service = "" + response = await servicer.Check(request_msg, context) + assert response.status == NOT_SERVING + async_llm.check_health.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_check_unknown_service_status(servicer, request_msg, context): + request_msg.service = "nonexistent.Service" + response = await servicer.Check(request_msg, context) + assert response.status == SERVICE_UNKNOWN + + +@pytest.mark.asyncio +async def test_check_unknown_service_grpc_code(servicer, request_msg, context): + request_msg.service = "fake.Svc" + await servicer.Check(request_msg, context) + context.set_code.assert_called_once_with(grpc.StatusCode.NOT_FOUND) + context.set_details.assert_called_once() + details_arg = context.set_details.call_args[0][0] + assert "fake.Svc" in details_arg + + +@pytest.mark.asyncio +@patch("smg_grpc_servicer.vllm.health_servicer.logger") +async def test_check_logs_exception_on_error( + mock_logger, servicer, request_msg, context, async_llm +): + async_llm.check_health = AsyncMock(side_effect=Exception("engine exploded")) + request_msg.service = "" + await servicer.Check(request_msg, context) + mock_logger.exception.assert_called_once() + log_args = mock_logger.exception.call_args + assert "service" in str(log_args).lower() + + +# -- Watch() tests -- + + +@pytest.mark.asyncio +async def test_watch_yields_serving(servicer, request_msg, context, async_llm): + request_msg.service = "" + watch_iter = servicer.Watch(request_msg, context) + first = await anext(watch_iter.__aiter__()) + assert first.status == SERVING + + +@pytest.mark.asyncio +async def test_watch_yields_not_serving(servicer, request_msg, context, async_llm): + async_llm.check_health = AsyncMock(side_effect=Exception("engine down")) + request_msg.service = "" + watch_iter = servicer.Watch(request_msg, context) + first = await anext(watch_iter.__aiter__()) + assert first.status == NOT_SERVING + + +@pytest.mark.asyncio +async def test_watch_unknown_service(servicer, request_msg, context): + request_msg.service = "fake.Service" + results = [] + async for response in servicer.Watch(request_msg, context): + results.append(response) + assert len(results) == 1 + assert results[0].status == SERVICE_UNKNOWN + # Watch returns SERVICE_UNKNOWN in the response body (not as a gRPC error + # code) so the stream terminates normally -- unlike Check, which sets + # NOT_FOUND on the gRPC context for unknown services. + context.set_code.assert_not_called() diff --git a/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_launch_cli.py b/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_launch_cli.py new file mode 100644 index 0000000000000000000000000000000000000000..5e73162db4a2c85e7757f3040a9c5396431acf22 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/unit_tests/test_launch_cli.py @@ -0,0 +1,111 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for the `vllm launch` CLI subcommand.""" + +import argparse +from unittest.mock import patch + +import pytest + +from vllm.entrypoints.cli.launch import ( + LaunchSubcommand, + RenderSubcommand, + cmd_init, +) +from vllm.utils.argparse_utils import FlexibleArgumentParser + + +@pytest.fixture +def launch_parser(): + parser = FlexibleArgumentParser(description="test") + subparsers = parser.add_subparsers(required=False, dest="subparser") + LaunchSubcommand().subparser_init(subparsers) + return parser + + +def test_subcommand_name(): + assert LaunchSubcommand().name == "launch" + + +def test_cmd_init_returns_subcommand(): + result = cmd_init() + assert len(result) == 1 + assert isinstance(result[0], LaunchSubcommand) + + +# -- Parsing: `vllm launch render` -- + + +def test_parse_launch_render(launch_parser): + args = launch_parser.parse_args(["launch", "render", "--model", "test-model"]) + assert args.launch_component == "render" + + +def test_parse_launch_requires_component(launch_parser): + with pytest.raises(SystemExit): + launch_parser.parse_args(["launch", "--model", "test-model"]) + + +def test_parse_launch_invalid_component(launch_parser): + with pytest.raises(SystemExit): + launch_parser.parse_args(["launch", "unknown", "--model", "test-model"]) + + +# -- Dispatch -- + + +def test_cmd_launch_render_calls_run(): + args = argparse.Namespace(model_tag=None, model="test-model") + with patch("vllm.entrypoints.cli.launch.uvloop.run") as mock_uvloop_run: + RenderSubcommand.cmd(args) + mock_uvloop_run.assert_called_once() + + +def test_cmd_launch_model_tag_overrides(): + args = argparse.Namespace( + model_tag="tag-model", + model="original-model", + launch_command=lambda a: None, + ) + LaunchSubcommand.cmd(args) + assert args.model == "tag-model" + + +def test_cmd_launch_model_tag_none(): + args = argparse.Namespace( + model_tag=None, + model="original-model", + launch_command=lambda a: None, + ) + LaunchSubcommand.cmd(args) + assert args.model == "original-model" + + +def test_cmd_dispatches(): + called = {} + + def fake_dispatch(args): + called["args"] = args + + args = argparse.Namespace(launch_command=fake_dispatch) + LaunchSubcommand.cmd(args) + assert "args" in called + + +# -- Module registration -- + + +def test_subparser_init_returns_parser(): + parser = FlexibleArgumentParser(description="test") + subparsers = parser.add_subparsers(required=False, dest="subparser") + result = LaunchSubcommand().subparser_init(subparsers) + assert isinstance(result, FlexibleArgumentParser) + + +def test_launch_registered_in_main(): + """Verify that launch module is importable as a CLI module.""" + import vllm.entrypoints.cli.launch as launch_module + + assert hasattr(launch_module, "cmd_init") + subcmds = launch_module.cmd_init() + assert any(s.name == "launch" for s in subcmds) diff --git a/asian_ai/performance/vLLM/tests/entrypoints/weight_transfer/__init__.py b/asian_ai/performance/vLLM/tests/entrypoints/weight_transfer/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..d4cea9873a5ad3bf6067f31743b09673e8d8f372 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/weight_transfer/__init__.py @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + diff --git a/asian_ai/performance/vLLM/tests/entrypoints/weight_transfer/test_weight_transfer_llm.py b/asian_ai/performance/vLLM/tests/entrypoints/weight_transfer/test_weight_transfer_llm.py new file mode 100644 index 0000000000000000000000000000000000000000..1b59e1797a7aba8832affcd0c6174b71b044bd0d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/entrypoints/weight_transfer/test_weight_transfer_llm.py @@ -0,0 +1,322 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for weight transfer APIs via LLM class. + +These tests use a mock weight transfer engine to verify that the API +calls the correct methods with the right arguments, without requiring +actual NCCL communication. +""" + +import os +from dataclasses import dataclass +from unittest.mock import patch + +import pytest +import torch + +from vllm import LLM +from vllm.config import WeightTransferConfig +from vllm.distributed.weight_transfer.base import ( + WeightTransferEngine, + WeightTransferInitInfo, + WeightTransferInitRequest, + WeightTransferUpdateInfo, + WeightTransferUpdateRequest, +) + +from ...utils import create_new_process_for_each_test + +# Use a tiny model for fast testing +MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM" + + +# --- Mock Weight Transfer Engine --- + + +@dataclass +class MockInitInfo(WeightTransferInitInfo): + """Mock initialization info.""" + + test_param: str = "test" + + +@dataclass +class MockUpdateInfo(WeightTransferUpdateInfo): + """Mock update info.""" + + names: list[str] | None = None + dtype_names: list[str] | None = None + shapes: list[list[int]] | None = None + + +class MockWeightTransferEngine(WeightTransferEngine[MockInitInfo, MockUpdateInfo]): + """Mock weight transfer engine that tracks method calls.""" + + init_info_cls = MockInitInfo + update_info_cls = MockUpdateInfo + + # Class-level tracking for verification across processes + init_transfer_engine_called: bool = False + start_called: bool = False + receive_weights_called: bool = False + finish_called: bool = False + shutdown_called: bool = False + last_init_info: MockInitInfo | None = None + last_update_info: MockUpdateInfo | None = None + + def __init__(self, config, vllm_config, device, model): + super().__init__(config, vllm_config, device, model) + # Reset tracking on init + MockWeightTransferEngine.init_transfer_engine_called = False + MockWeightTransferEngine.start_called = False + MockWeightTransferEngine.receive_weights_called = False + MockWeightTransferEngine.finish_called = False + MockWeightTransferEngine.shutdown_called = False + MockWeightTransferEngine.last_init_info = None + MockWeightTransferEngine.last_update_info = None + + def init_transfer_engine(self, init_info: MockInitInfo) -> None: + MockWeightTransferEngine.init_transfer_engine_called = True + MockWeightTransferEngine.last_init_info = init_info + + def start_weight_update(self) -> None: + MockWeightTransferEngine.start_called = True + + def finish_weight_update(self) -> None: + MockWeightTransferEngine.finish_called = True + + def receive_weights(self, update_info: MockUpdateInfo) -> None: + MockWeightTransferEngine.receive_weights_called = True + MockWeightTransferEngine.last_update_info = update_info + + def shutdown(self) -> None: + MockWeightTransferEngine.shutdown_called = True + + @staticmethod + def trainer_send_weights(*args, **kwargs): + """Mock method to simulate trainer sending weights.""" + pass + + +def mock_create_engine(config, vllm_config, device, model): + """Mock factory function that returns our mock engine.""" + return MockWeightTransferEngine(config, vllm_config, device, model) + + +# --- Tests --- + + +@create_new_process_for_each_test() +def test_get_world_size_tp1(): + """Test world_size is correctly configured for TP=1.""" + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + llm = LLM( + model=MODEL_NAME, + enforce_eager=True, + load_format="dummy", + tensor_parallel_size=1, + weight_transfer_config=WeightTransferConfig(backend="nccl"), + ) + + world_size = llm.llm_engine.vllm_config.parallel_config.world_size + assert world_size == 1 + + +@create_new_process_for_each_test() +def test_init_weight_transfer_engine_calls_engine(): + """Test that init_weight_transfer_engine calls the engine's + init_transfer_engine method.""" + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + # Run in-process so mock.patch works (spawn won't inherit the mock) + os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0" + # Enable insecure serialization to allow pickling functions for collective_rpc + os.environ["VLLM_ALLOW_INSECURE_SERIALIZATION"] = "1" + + with patch( + "vllm.v1.worker.gpu_worker.WeightTransferEngineFactory.create_engine", + mock_create_engine, + ): + llm = LLM( + model=MODEL_NAME, + enforce_eager=True, + load_format="dummy", + tensor_parallel_size=1, + weight_transfer_config=WeightTransferConfig(backend="nccl"), + ) + + # Verify engine was created + def check_engine_exists(self): + return self.weight_transfer_engine is not None + + results = llm.collective_rpc(check_engine_exists) + assert all(results), "Weight transfer engine should be initialized" + + # Call init_weight_transfer_engine + llm.init_weight_transfer_engine( + WeightTransferInitRequest(init_info={"test_param": "hello"}) + ) + + # Verify init_transfer_engine was called on the engine + def check_init_called(self): + engine = self.weight_transfer_engine + return ( + engine.init_transfer_engine_called, + engine.last_init_info.test_param if engine.last_init_info else None, + ) + + results = llm.collective_rpc(check_init_called) + for called, param in results: + assert called, "init_transfer_engine should have been called" + assert param == "hello", f"Expected 'hello', got {param}" + + +@create_new_process_for_each_test() +def test_update_weights_calls_engine(): + """Test that update_weights calls the engine's receive_weights method.""" + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + # Run in-process so mock.patch works (spawn won't inherit the mock) + os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0" + # Enable insecure serialization to allow pickling functions for collective_rpc + os.environ["VLLM_ALLOW_INSECURE_SERIALIZATION"] = "1" + + with patch( + "vllm.v1.worker.gpu_worker.WeightTransferEngineFactory.create_engine", + mock_create_engine, + ): + llm = LLM( + model=MODEL_NAME, + enforce_eager=True, + load_format="dummy", + tensor_parallel_size=1, + weight_transfer_config=WeightTransferConfig(backend="nccl"), + ) + + # First init the weight transfer + llm.init_weight_transfer_engine( + WeightTransferInitRequest(init_info={"test_param": "init"}) + ) + llm.start_weight_update() + + # Call update_weights + test_names = ["layer.weight", "layer.bias"] + test_dtypes = ["float32", "float32"] + test_shapes = [[10, 10], [10]] + + llm.update_weights( + WeightTransferUpdateRequest( + update_info={ + "names": test_names, + "dtype_names": test_dtypes, + "shapes": test_shapes, + } + ) + ) + + # Verify receive_weights was called with correct info + def check_update_called(self): + engine = self.weight_transfer_engine + if not engine.receive_weights_called: + return False, None, None, None + info = engine.last_update_info + return (True, info.names, info.dtype_names, info.shapes) + + results = llm.collective_rpc(check_update_called) + for called, names, dtypes, shapes in results: + assert called, "receive_weights should have been called" + assert names == test_names + assert dtypes == test_dtypes + assert shapes == test_shapes + + llm.finish_weight_update() + + +@create_new_process_for_each_test() +def test_full_weight_transfer_flow(): + """Test the complete weight transfer flow: init -> start -> update -> finish.""" + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + # Run in-process so mock.patch works (spawn won't inherit the mock) + os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0" + # Enable insecure serialization to allow pickling functions for collective_rpc + os.environ["VLLM_ALLOW_INSECURE_SERIALIZATION"] = "1" + + with patch( + "vllm.v1.worker.gpu_worker.WeightTransferEngineFactory.create_engine", + mock_create_engine, + ): + llm = LLM( + model=MODEL_NAME, + enforce_eager=True, + load_format="dummy", + tensor_parallel_size=1, + weight_transfer_config=WeightTransferConfig(backend="nccl"), + ) + + # Step 1: Initialize weight transfer engine + llm.init_weight_transfer_engine( + WeightTransferInitRequest(init_info={"test_param": "flow_test"}) + ) + + # Step 2: Start weight update + llm.start_weight_update() + + # Step 3: Update weights + llm.update_weights( + WeightTransferUpdateRequest( + update_info={ + "names": ["test.weight"], + "dtype_names": ["bfloat16"], + "shapes": [[100, 100]], + } + ) + ) + + # Step 4: Finish weight update + llm.finish_weight_update() + + # Verify the full flow completed + def check_flow(self): + engine = self.weight_transfer_engine + return { + "init_called": engine.init_transfer_engine_called, + "update_called": engine.receive_weights_called, + "init_param": ( + engine.last_init_info.test_param if engine.last_init_info else None + ), + "update_names": ( + engine.last_update_info.names if engine.last_update_info else None + ), + } + + results = llm.collective_rpc(check_flow) + for result in results: + assert result["init_called"], "init_transfer_engine should be called" + assert result["update_called"], "receive_weights should be called" + assert result["init_param"] == "flow_test" + assert result["update_names"] == ["test.weight"] + + +@create_new_process_for_each_test() +def test_weight_transfer_config_backend(): + """Test that WeightTransferConfig backend is properly configured.""" + if torch.accelerator.device_count() < 1: + pytest.skip("Need at least 1 GPU for this test") + + # Test with nccl backend + llm = LLM( + model=MODEL_NAME, + enforce_eager=True, + load_format="dummy", + tensor_parallel_size=1, + weight_transfer_config=WeightTransferConfig(backend="nccl"), + ) + + config = llm.llm_engine.vllm_config.weight_transfer_config + assert config.backend == "nccl" diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/README.md b/asian_ai/performance/vLLM/tests/evals/gpt_oss/README.md new file mode 100644 index 0000000000000000000000000000000000000000..57b9de35b8fb95079eb3bd27909d41ffea4e546c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/README.md @@ -0,0 +1,49 @@ +# GPQA Evaluation using GPT-OSS + +This directory contains GPQA evaluation tests using the GPT-OSS evaluation package and vLLM server. + +## Usage + +### Run tests with pytest (like buildkite) + +```bash +# H200 +pytest -s -v tests/evals/gpt_oss/test_gpqa_correctness.py \ + --config-list-file=configs/models-h200.txt + +# B200 +pytest -s -v tests/evals/gpt_oss/test_gpqa_correctness.py \ + --config-list-file=configs/models-b200.txt +``` + +## Configuration Format + +Model configs in `configs/` directory use this YAML format: + +```yaml +model_name: "openai/gpt-oss-20b" +metric_threshold: 0.568 # Minimum expected accuracy +reasoning_effort: "low" # Reasoning effort level (default: "low") +server_args: "--tensor-parallel-size 2" # Server arguments +startup_max_wait_seconds: 1800 # Max wait for server startup (default: 1800) +env: # Environment variables (optional) + SOME_VAR: "value" +``` + +The `server_args` field accepts any arguments that can be passed to `vllm serve`. + +The `env` field accepts a dictionary of environment variables to set for the server process. + +## Adding New Models + +1. Create a new YAML config file in the `configs/` directory +2. Add the filename to the appropriate `models-*.txt` file + +## Tiktoken Encoding Files + +The tiktoken encoding files required by the vLLM server are automatically downloaded from OpenAI's public blob storage on first run: + +- `cl100k_base.tiktoken` +- `o200k_base.tiktoken` + +Files are cached in the `data/` directory. The `TIKTOKEN_ENCODINGS_BASE` environment variable is automatically set to point to this directory when running evaluations. diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/__init__.py b/asian_ai/performance/vLLM/tests/evals/gpt_oss/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..0643de4834f45beb52db0a52ff79be718f6ec2dc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/__init__.py @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-baseline.yaml b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-baseline.yaml new file mode 100644 index 0000000000000000000000000000000000000000..dd4ad0ae187a4fa24ebe69a8032234bc61a79072 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-baseline.yaml @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +model_name: "openai/gpt-oss-20b" +metric_threshold: 0.568 +reasoning_effort: "low" +server_args: "--tensor-parallel-size 2" diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-flashinfer-mxfp4-bf16-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-flashinfer-mxfp4-bf16-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..e1a835d6d4ebebad432f8a852e7575311e1ed516 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-flashinfer-mxfp4-bf16-cutlass.yaml @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +model_name: "openai/gpt-oss-20b" +metric_threshold: 0.568 +reasoning_effort: "low" +server_args: "--tensor-parallel-size 2 --moe-backend flashinfer_cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-flashinfer-mxfp4-bf16-trtllm.yaml b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-flashinfer-mxfp4-bf16-trtllm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..44a83fee765bef4d04158592cad39eec68ffc475 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-flashinfer-mxfp4-bf16-trtllm.yaml @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +model_name: "openai/gpt-oss-20b" +metric_threshold: 0.568 +reasoning_effort: "low" +server_args: "--tensor-parallel-size 2 --moe-backend flashinfer_trtllm" diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-flashinfer-mxfp4-mxfp8-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-flashinfer-mxfp4-mxfp8-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..b55375f1a68960c7cd2b176c5e6657784ea4136a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-flashinfer-mxfp4-mxfp8-cutlass.yaml @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +model_name: "openai/gpt-oss-20b" +metric_threshold: 0.568 +reasoning_effort: "low" +server_args: "--tensor-parallel-size 2 --moe-backend flashinfer_cutlass --quantization-config.moe.activation mxfp8" diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-marlin.yaml b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-marlin.yaml new file mode 100644 index 0000000000000000000000000000000000000000..6b551ffd31a08eb71081c398f0cbd27d88bfc710 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-marlin.yaml @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +model_name: "openai/gpt-oss-20b" +metric_threshold: 0.568 +reasoning_effort: "low" +server_args: "--tensor-parallel-size 2 --moe-backend marlin --linear-backend marlin" diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-rocm-baseline.yaml b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-rocm-baseline.yaml new file mode 100644 index 0000000000000000000000000000000000000000..b685ca93dd0c243f51db0cb38264ba84aa03a8f1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-rocm-baseline.yaml @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +model_name: openai/gpt-oss-20b +metric_threshold: 0.568 +reasoning_effort: low +server_args: "--attention-backend ROCM_AITER_UNIFIED_ATTN --tensor-parallel-size 2" \ No newline at end of file diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-rocm-quark-mxfp4-bf16-aiter.yaml b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-rocm-quark-mxfp4-bf16-aiter.yaml new file mode 100644 index 0000000000000000000000000000000000000000..08c8abb6f115d650757bf86e5620cc1f97e9e042 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-rocm-quark-mxfp4-bf16-aiter.yaml @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +model_name: amd/gpt-oss-20b-w-mxfp4-a-bf16 +metric_threshold: 0.568 +reasoning_effort: low +server_args: "--attention-backend ROCM_AITER_UNIFIED_ATTN --moe-backend aiter --tokenizer openai/gpt-oss-20b --tensor-parallel-size 2" +env: + VLLM_ROCM_USE_AITER: "1" \ No newline at end of file diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-rocm-quark-mxfp4-bf16-triton.yaml b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-rocm-quark-mxfp4-bf16-triton.yaml new file mode 100644 index 0000000000000000000000000000000000000000..49721d5b4c08d44bfc7caf153a7441bad5d28060 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-rocm-quark-mxfp4-bf16-triton.yaml @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +model_name: amd/gpt-oss-20b-w-mxfp4-a-bf16 +metric_threshold: 0.568 +reasoning_effort: low +server_args: "--attention-backend ROCM_AITER_UNIFIED_ATTN --moe-backend triton --tokenizer openai/gpt-oss-20b --tensor-parallel-size 2" \ No newline at end of file diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-rocm-quark-mxfp4-fp8-triton.yaml b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-rocm-quark-mxfp4-fp8-triton.yaml new file mode 100644 index 0000000000000000000000000000000000000000..59f6165016fb743508cfcf68530879c3689a1fb4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-rocm-quark-mxfp4-fp8-triton.yaml @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +model_name: amd/gpt-oss-20b-MoE-Quant-W-MXFP4-A-FP8-KV-FP8 +metric_threshold: 0.568 +reasoning_effort: low +server_args: "--attention-backend ROCM_AITER_UNIFIED_ATTN --tensor-parallel-size 2" +env: + VLLM_ROCM_USE_AITER: "1" \ No newline at end of file diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-sm100-fi-mxfp4-mxfp8-trtllm.yaml b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-sm100-fi-mxfp4-mxfp8-trtllm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..9970a3483c28ad5290acc638b20bd59a6830121c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-sm100-fi-mxfp4-mxfp8-trtllm.yaml @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +model_name: "openai/gpt-oss-20b" +metric_threshold: 0.568 +reasoning_effort: "low" +server_args: "--tensor-parallel-size 2 --quantization-config.moe.activation mxfp8" diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-sm120.yaml b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-sm120.yaml new file mode 100644 index 0000000000000000000000000000000000000000..229014ad7845a3011a45fda34b81f1c8cf64918f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/gpt-oss-20b-sm120.yaml @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +model_name: "openai/gpt-oss-20b" +metric_threshold: 0.568 +reasoning_effort: "low" diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-b200.txt b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-b200.txt new file mode 100644 index 0000000000000000000000000000000000000000..adddc4bef15f0d8928b0e083dc72c2ab8145fa1b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-b200.txt @@ -0,0 +1,5 @@ +# B200 model configurations for GPQA evaluation +# Tests different environment variable combinations +gpt-oss-20b-flashinfer-mxfp4-bf16-trtllm.yaml +gpt-oss-20b-flashinfer-mxfp4-mxfp8-cutlass.yaml +gpt-oss-20b-sm100-fi-mxfp4-mxfp8-trtllm.yaml \ No newline at end of file diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-gfx942.txt b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-gfx942.txt new file mode 100644 index 0000000000000000000000000000000000000000..1bf4c01a4a0ef165a346b389660593df7794f387 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-gfx942.txt @@ -0,0 +1,3 @@ +# GFX942 model configurations for GPQA evaluation +# Tests different environment variable combinations +gpt-oss-20b-rocm-baseline.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-gfx950.txt b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-gfx950.txt new file mode 100644 index 0000000000000000000000000000000000000000..a9d20300e572f1e5e68db445227d8ea1a9dcafda --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-gfx950.txt @@ -0,0 +1,6 @@ +# GFX950 model configurations for GPQA evaluation +# Tests different environment variable combinations +gpt-oss-20b-rocm-baseline.yaml +gpt-oss-20b-rocm-quark-mxfp4-bf16-aiter.yaml +gpt-oss-20b-rocm-quark-mxfp4-bf16-triton.yaml +gpt-oss-20b-rocm-quark-mxfp4-fp8-triton.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-h100.txt b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-h100.txt new file mode 100644 index 0000000000000000000000000000000000000000..d596ef08bbf64d309f4179a705a388c076586fa5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-h100.txt @@ -0,0 +1,5 @@ +# H100 model configurations for GPQA evaluation +# Tests different environment variable combinations +gpt-oss-20b-baseline.yaml +gpt-oss-20b-flashinfer-mxfp4-bf16-cutlass.yaml +gpt-oss-20b-marlin.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-spark.txt b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-spark.txt new file mode 100644 index 0000000000000000000000000000000000000000..a56a77e2f9177b3990ce79cf1a6a2a0dd473aedb --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/configs/models-spark.txt @@ -0,0 +1,2 @@ +# DGX Spark model configurations for GPQA evaluation +gpt-oss-20b-sm120.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/conftest.py b/asian_ai/performance/vLLM/tests/evals/gpt_oss/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..b928cd0f11f36f8c4a90b5d912c83dd1d3c20d45 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/conftest.py @@ -0,0 +1,64 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Pytest configuration for GPT-OSS evaluation tests. +""" + +from pathlib import Path + + +def pytest_addoption(parser): + """Add custom command line options.""" + parser.addoption( + "--config-list-file", + required=True, + help="File containing list of config files to test", + ) + + +def pytest_generate_tests(metafunc): + """Generate test parameters from config files.""" + if "config_filename" in metafunc.fixturenames: + config_list_file = metafunc.config.getoption("--config-list-file") + + # Handle both relative and absolute paths + config_list_path = Path(config_list_file) + if not config_list_path.is_absolute(): + # If relative, try relative to test directory first + test_dir_path = Path(__file__).parent / config_list_file + if test_dir_path.exists(): + config_list_path = test_dir_path + else: + # Try relative to current working directory + config_list_path = Path.cwd() / config_list_file + + print(f"Looking for config list at: {config_list_path}") + + config_files = [] + if config_list_path.exists(): + # Determine config directory (same directory as the list file) + config_dir = config_list_path.parent + + with open(config_list_path) as f: + for line in f: + line = line.strip() + if line and not line.startswith("#"): + config_path = config_dir / line + print(f"Checking config file: {config_path}") + if config_path.exists(): + config_files.append(config_path) + print(f" Found: {config_path}") + else: + print(f" Missing: {config_path}") + else: + print(f"Config list file not found: {config_list_path}") + + # Generate test parameters + if config_files: + metafunc.parametrize( + "config_filename", + config_files, + ids=[config_file.stem for config_file in config_files], + ) + else: + print("No config files found, test will be skipped") diff --git a/asian_ai/performance/vLLM/tests/evals/gpt_oss/test_gpqa_correctness.py b/asian_ai/performance/vLLM/tests/evals/gpt_oss/test_gpqa_correctness.py new file mode 100644 index 0000000000000000000000000000000000000000..5c7eb705a0ce46358076e0c6ed2346c59e60e8fe --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gpt_oss/test_gpqa_correctness.py @@ -0,0 +1,172 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +GPQA evaluation using vLLM server and GPT-OSS evaluation package. + +Usage: +pytest -s -v tests/evals/gpt_oss/test_gpqa_correctness.py \ + --config-list-file=configs/models-h200.txt +""" + +import os +import shlex +import subprocess +import sys +import urllib.request +from pathlib import Path + +import regex as re +import yaml + +from tests.utils import RemoteOpenAIServer + +TOL = 0.05 # Absolute tolerance for accuracy comparison + +# Path to tiktoken encoding files +TIKTOKEN_DATA_DIR = Path(__file__).parent / "data" + +# Tiktoken encoding files to download +TIKTOKEN_FILES = { + "cl100k_base.tiktoken": "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken", + "o200k_base.tiktoken": "https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken", +} + + +def ensure_tiktoken_files(): + """Download tiktoken encoding files if they don't exist.""" + TIKTOKEN_DATA_DIR.mkdir(parents=True, exist_ok=True) + + for filename, url in TIKTOKEN_FILES.items(): + filepath = TIKTOKEN_DATA_DIR / filename + if not filepath.exists(): + print(f"Downloading {filename} from {url}...") + urllib.request.urlretrieve(url, filepath) + print(f" Downloaded to {filepath}") + else: + print(f" {filename} already exists.") + + +def run_gpqa_eval(model_name: str, base_url: str, reasoning_effort: str) -> float: + """Run GPQA evaluation using the gpt-oss evaluation package.""" + + # Build the command to run the evaluation + cmd = [ + sys.executable, + "-m", + "gpt_oss.evals", + "--eval", + "gpqa", + "--model", + model_name, + "--reasoning-effort", + reasoning_effort, + "--base-url", + base_url, + "--n-threads", + "200", + ] + + try: + # Set up environment for the evaluation subprocess + # Inherit current environment and add required variables + eval_env = os.environ.copy() + eval_env["OPENAI_API_KEY"] = "dummy" + + # Run the evaluation + result = subprocess.run( + cmd, + text=True, + capture_output=True, + timeout=1800, # 30 minute timeout + env=eval_env, + ) + + print("Evaluation process stdout:\n", result.stdout) + print("Evaluation process stderr:\n", result.stderr) + print(f"Evaluation process return code: {result.returncode}") + + if result.returncode != 0: + raise RuntimeError( + f"Evaluation failed with exit code {result.returncode}:\n" + f"stdout: {result.stdout}\nstderr: {result.stderr}" + ) + + # Parse the output to extract the score + match = re.search(r"'metric':\s*([\d.]+)", result.stdout) + if match: + return float(match.group(1)) + + # If we still can't find it, raise an error + raise ValueError( + f"Could not parse score from evaluation output:\n{result.stdout}" + ) + + except subprocess.TimeoutExpired as e: + raise RuntimeError("Evaluation timed out") from e + + +def test_gpqa_correctness(config_filename): + """Test GPQA correctness for a given model configuration.""" + # Ensure tiktoken files are downloaded + ensure_tiktoken_files() + + # Verify tiktoken files exist + for filename in TIKTOKEN_FILES: + filepath = TIKTOKEN_DATA_DIR / filename + assert filepath.exists(), f"Tiktoken file not found: {filepath}" + + eval_config = yaml.safe_load(config_filename.read_text(encoding="utf-8")) + + # Parse server arguments from config (use shlex to handle quoted strings) + server_args_str = eval_config.get("server_args", "") + server_args = shlex.split(server_args_str) if server_args_str else [] + + # Add standard server arguments + server_args.extend( + [ + "--trust-remote-code", + "--enforce-eager", + "--disable-uvicorn-access-log", + ] + ) + + # Build server environment with tiktoken path and any config-specified vars + server_env = {"TIKTOKEN_ENCODINGS_BASE": str(TIKTOKEN_DATA_DIR)} + if eval_config.get("env"): + server_env.update(eval_config["env"]) + + reasoning_effort = eval_config.get("reasoning_effort", "low") + + print(f"Starting GPQA evaluation for model: {eval_config['model_name']}") + print(f"Expected metric threshold: {eval_config['metric_threshold']}") + print(f"Reasoning effort: {reasoning_effort}") + print(f"Server args: {' '.join(server_args)}") + print(f"Server environment variables: {server_env}") + + # Launch server and run evaluation + with RemoteOpenAIServer( + eval_config["model_name"], + server_args, + env_dict=server_env, + max_wait_seconds=eval_config.get("startup_max_wait_seconds", 1800), + ) as remote_server: + base_url = remote_server.url_for("v1") + print(f"Server started at: {base_url}") + + measured_metric = run_gpqa_eval( + eval_config["model_name"], base_url, reasoning_effort + ) + expected_metric = eval_config["metric_threshold"] + + print(f"GPQA Results for {eval_config['model_name']}:") + print(f" Measured metric: {measured_metric:.4f}") + print(f" Expected metric: {expected_metric:.4f}") + print(f" Tolerance: {TOL:.4f}") + + # Verify metric is within tolerance + assert measured_metric >= expected_metric - TOL, ( + f"GPQA metric too low: {measured_metric:.4f} < " + f"{expected_metric:.4f} - {TOL:.4f} = {expected_metric - TOL:.4f}" + ) + + print(f"GPQA test passed for {eval_config['model_name']}") diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/README.md b/asian_ai/performance/vLLM/tests/evals/gsm8k/README.md new file mode 100644 index 0000000000000000000000000000000000000000..b07ae825f3b68ad4e6532eb5172a8367c6bf4def --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/README.md @@ -0,0 +1,40 @@ +# GSM8K Accuracy Evaluation + +This directory contains a replacement for the lm-eval-harness GSM8K evaluation, using an isolated GSM8K script and vLLM server for better performance and control. + +## Usage + +### Run tests with pytest (like buildkite) + +```bash +pytest -s -v tests/evals/gsm8k/test_gsm8k_correctness.py \ + --config-list-file=configs/models-small.txt +``` + +### Run standalone evaluation script + +```bash +# Start vLLM server first +vllm serve Qwen/Qwen2.5-1.5B-Instruct --port 8000 + +# Run evaluation +python tests/evals/gsm8k/gsm8k_eval.py --port 8000 +``` + +## Configuration Format + +Model configs in `configs/` directory use this YAML format: + +```yaml +model_name: "Qwen/Qwen2.5-1.5B-Instruct" +accuracy_threshold: 0.54 # Minimum expected accuracy +num_questions: 1319 # Number of questions (default: full test set) +num_fewshot: 5 # Few-shot examples from train set +server_args: "--max-model-len 4096 --tensor-parallel-size 2 --moe-backend flashinfer_cutlass" # Server arguments +env: # Environment variables (optional) + VLLM_LOGGING_LEVEL: "DEBUG" +``` + +The `server_args` field accepts any arguments that can be passed to `vllm serve`. + +The `env` field accepts a dictionary of environment variables to set for the server process. diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/__init__.py b/asian_ai/performance/vLLM/tests/evals/gsm8k/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..0643de4834f45beb52db0a52ff79be718f6ec2dc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/__init__.py @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-R1-DP.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-R1-DP.yaml new file mode 100644 index 0000000000000000000000000000000000000000..97017a782806bd924ee89b321f3180d396044bae --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-R1-DP.yaml @@ -0,0 +1,11 @@ +model_name: "deepseek-ai/DeepSeek-R1" +accuracy_threshold: 0.95 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1200 +server_args: >- + --enforce-eager + --max-model-len 4096 + --data-parallel-size 8 + --enable-expert-parallel + --speculative-config '{"method":"mtp","num_speculative_tokens":3}' diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-R1-DP_MI325.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-R1-DP_MI325.yaml new file mode 100644 index 0000000000000000000000000000000000000000..eba3f3324f9c16764d1e38c4eddab4f070f5b950 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-R1-DP_MI325.yaml @@ -0,0 +1,12 @@ +model_name: "deepseek-ai/DeepSeek-R1" +accuracy_threshold: 0.95 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1200 +server_args: >- + --enforce-eager + --max-model-len 4096 + --data-parallel-size 8 + --enable-expert-parallel + --attention-backend=TRITON_ATTN + --speculative-config '{"method":"mtp","num_speculative_tokens":3}' diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-R1-TP.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-R1-TP.yaml new file mode 100644 index 0000000000000000000000000000000000000000..037637ccb3cd595b536ac24a14cacf7053ef3403 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-R1-TP.yaml @@ -0,0 +1,11 @@ +model_name: "deepseek-ai/DeepSeek-R1" +accuracy_threshold: 0.95 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1200 +server_args: >- + --enforce-eager + --max-model-len 4096 + --tensor-parallel-size 8 + --enable-expert-parallel + --speculative-config '{"method":"mtp","num_speculative_tokens":3}' diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-R1-TP_MI325.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-R1-TP_MI325.yaml new file mode 100644 index 0000000000000000000000000000000000000000..6201d6fa3ce49d33924ca78293763ce2db6d0132 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-R1-TP_MI325.yaml @@ -0,0 +1,12 @@ +model_name: "deepseek-ai/DeepSeek-R1" +accuracy_threshold: 0.95 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1200 +server_args: >- + --enforce-eager + --max-model-len 4096 + --tensor-parallel-size 8 + --enable-expert-parallel + --attention-backend=TRITON_ATTN + --speculative-config '{"method":"mtp","num_speculative_tokens":3}' diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V2-Lite-Instruct-FP8.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V2-Lite-Instruct-FP8.yaml new file mode 100644 index 0000000000000000000000000000000000000000..055817e8b106c2dbdcf00a43bac448c09217d6ba --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V2-Lite-Instruct-FP8.yaml @@ -0,0 +1,6 @@ +model_name: "RedHatAI/DeepSeek-Coder-V2-Lite-Instruct-FP8" +accuracy_threshold: 0.72 +num_questions: 1319 +num_fewshot: 5 +rocm_request_timeout_seconds: 1800 +server_args: "--enforce-eager --max-model-len 4096" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V3.2-DP.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V3.2-DP.yaml new file mode 100644 index 0000000000000000000000000000000000000000..ed08f112a4193c519ed47e721864d5cfe7ac86b8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V3.2-DP.yaml @@ -0,0 +1,11 @@ +model_name: "deepseek-ai/DeepSeek-V3.2" +accuracy_threshold: 0.95 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1200 +server_args: >- + --enforce-eager + --max-model-len 4096 + --data-parallel-size 8 + --enable-expert-parallel + --speculative-config '{"method":"mtp","num_speculative_tokens":3}' diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V3.2-DP_MI325.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V3.2-DP_MI325.yaml new file mode 100644 index 0000000000000000000000000000000000000000..fb61578f599de227a3fba4e2940597be0e99f8a4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V3.2-DP_MI325.yaml @@ -0,0 +1,12 @@ +model_name: "deepseek-ai/DeepSeek-V3.2" +accuracy_threshold: 0.95 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1200 +server_args: >- + --enforce-eager + --max-model-len 4096 + --data-parallel-size 8 + --enable-expert-parallel + --attention-backend=TRITON_ATTN + --speculative-config '{"method":"mtp","num_speculative_tokens":3}' diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V3.2-TP.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V3.2-TP.yaml new file mode 100644 index 0000000000000000000000000000000000000000..e1c17e25e6eb0ba012ad98867f2edf0f6cf76675 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V3.2-TP.yaml @@ -0,0 +1,11 @@ +model_name: "deepseek-ai/DeepSeek-V3.2" +accuracy_threshold: 0.95 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1200 +server_args: >- + --enforce-eager + --max-model-len 4096 + --tensor-parallel-size 8 + --enable-expert-parallel + --speculative-config '{"method":"mtp","num_speculative_tokens":3}' diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V3.2-TP_MI325.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V3.2-TP_MI325.yaml new file mode 100644 index 0000000000000000000000000000000000000000..e943071d8d5397df2d282dfe33853d15bc79201a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DeepSeek-V3.2-TP_MI325.yaml @@ -0,0 +1,12 @@ +model_name: "deepseek-ai/DeepSeek-V3.2" +accuracy_threshold: 0.95 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1200 +server_args: >- + --enforce-eager + --max-model-len 4096 + --tensor-parallel-size 8 + --enable-expert-parallel + --attention-backend=TRITON_ATTN + --speculative-config '{"method":"mtp","num_speculative_tokens":3}' diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DiffusionGemma-26B-A4B-it-FP8-dynamic.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DiffusionGemma-26B-A4B-it-FP8-dynamic.yaml new file mode 100644 index 0000000000000000000000000000000000000000..724dcb82497c8e2b65d978cb6efba08c82577cbd --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/DiffusionGemma-26B-A4B-it-FP8-dynamic.yaml @@ -0,0 +1,11 @@ +model_name: "RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic" +accuracy_threshold: 0.84 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1200 +use_chat_completions: true +server_args: >- + --enforce-eager + --max-model-len 4096 + --tensor-parallel-size 2 + --attention-backend TRITON_ATTN diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Llama-3-8B-Instruct-nonuniform-CT.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Llama-3-8B-Instruct-nonuniform-CT.yaml new file mode 100644 index 0000000000000000000000000000000000000000..562b42e0414640321a3fe9121f4e0704ce7df250 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Llama-3-8B-Instruct-nonuniform-CT.yaml @@ -0,0 +1,5 @@ +model_name: "nm-testing/Meta-Llama-3-8B-Instruct-nonuniform-test" +accuracy_threshold: 0.74 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 4096" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Llama-3.2-1B-Instruct-INT8-CT.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Llama-3.2-1B-Instruct-INT8-CT.yaml new file mode 100644 index 0000000000000000000000000000000000000000..fbf5bb2788dd327d254d9d8b4bcb0ffcb737b31f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Llama-3.2-1B-Instruct-INT8-CT.yaml @@ -0,0 +1,5 @@ +model_name: "RedHatAI/Llama-3.2-1B-Instruct-quantized.w8a8" +accuracy_threshold: 0.31 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 4096" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Nemotron-3-Super-120B-A12B-BF16.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Nemotron-3-Super-120B-A12B-BF16.yaml new file mode 100644 index 0000000000000000000000000000000000000000..6152d8922dde00517d0384eb61e890f6154ad8ad --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Nemotron-3-Super-120B-A12B-BF16.yaml @@ -0,0 +1,12 @@ +model_name: "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16" +accuracy_threshold: 0.93 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1200 +server_args: >- + --enforce-eager + --max-model-len 4096 + --tensor-parallel-size 8 + --enable-expert-parallel + --mamba-backend flashinfer + --speculative-config '{"method":"mtp","num_speculative_tokens":5}' diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Nemotron-3-Super-120B-A12B-FP8.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Nemotron-3-Super-120B-A12B-FP8.yaml new file mode 100644 index 0000000000000000000000000000000000000000..5ee91b0ced1d6aa26883951960a5d948c957f430 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Nemotron-3-Super-120B-A12B-FP8.yaml @@ -0,0 +1,11 @@ +model_name: "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8" +accuracy_threshold: 0.93 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1200 +server_args: >- + --enforce-eager + --max-model-len 4096 + --tensor-parallel-size 8 + --enable-expert-parallel + --speculative-config '{"method":"mtp","num_speculative_tokens":5}' diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Nemotron-3-Super-120B-A12B-NVFP4.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Nemotron-3-Super-120B-A12B-NVFP4.yaml new file mode 100644 index 0000000000000000000000000000000000000000..af7647d36aab53696272d7b293e706653b6a4a8a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Nemotron-3-Super-120B-A12B-NVFP4.yaml @@ -0,0 +1,12 @@ +model_name: "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" +accuracy_threshold: 0.93 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1200 +server_args: >- + --enforce-eager + --max-model-len 4096 + --tensor-parallel-size 2 + --enable-expert-parallel + --mamba-backend flashinfer + --speculative-config '{"method":"mtp","num_speculative_tokens":5}' diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen1.5-MoE-W4A16-CT.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen1.5-MoE-W4A16-CT.yaml new file mode 100644 index 0000000000000000000000000000000000000000..4370e7d1bb02bd225c3317d9cfdb1d8f994c7ce3 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen1.5-MoE-W4A16-CT.yaml @@ -0,0 +1,6 @@ +model_name: "nm-testing/Qwen1.5-MoE-A2.7B-Chat-quantized.w4a16" +accuracy_threshold: 0.45 +num_questions: 1319 +num_fewshot: 5 +rocm_request_timeout_seconds: 1800 +server_args: "--enforce-eager --max-model-len 4096" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen2.5-VL-3B-Instruct-FP8-dynamic.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen2.5-VL-3B-Instruct-FP8-dynamic.yaml new file mode 100644 index 0000000000000000000000000000000000000000..a02abee15cd6232e9fda6ee41ad690fcba64144e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen2.5-VL-3B-Instruct-FP8-dynamic.yaml @@ -0,0 +1,5 @@ +model_name: "RedHatAI/Qwen2.5-VL-3B-Instruct-FP8-Dynamic" +accuracy_threshold: 0.60 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 4096" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-0.6B-FP8.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-0.6B-FP8.yaml new file mode 100644 index 0000000000000000000000000000000000000000..fde97e559a5168362dbd46217359159d0f461d10 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-0.6B-FP8.yaml @@ -0,0 +1,5 @@ +model_name: "Qwen/Qwen3-0.6B-FP8" +accuracy_threshold: 0.375 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 4096" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-30B-A3B-MXFP4A16.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-30B-A3B-MXFP4A16.yaml new file mode 100644 index 0000000000000000000000000000000000000000..4eb5d6988ec4f89cd1f37a80aa72bb57a9a216c9 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-30B-A3B-MXFP4A16.yaml @@ -0,0 +1,5 @@ +model_name: nm-testing/Qwen3-30B-A3B-MXFP4A16 +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 4096" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-30B-A3B-NVFP4.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-30B-A3B-NVFP4.yaml new file mode 100644 index 0000000000000000000000000000000000000000..058b4a9fef560ba8071484a7d13e1f881fac3b72 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-30B-A3B-NVFP4.yaml @@ -0,0 +1,5 @@ +model_name: "nvidia/Qwen3-30B-A3B-FP4" +accuracy_threshold: 0.89 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 4096" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-30B-A3B-Thinking-2507-FP8.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-30B-A3B-Thinking-2507-FP8.yaml new file mode 100644 index 0000000000000000000000000000000000000000..8a1e4bb98d370e938e41f32fe34f24dc762e5076 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-30B-A3B-Thinking-2507-FP8.yaml @@ -0,0 +1,6 @@ +model_name: "amd/Qwen3-30B-A3B-Thinking-2507-FP8" +accuracy_threshold: 0.81 +num_questions: 1319 +num_fewshot: 5 +max_tokens: 1024 +server_args: "--max-model-len 4096 --gpu-memory-utilization 0.85" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-30B-A3B-Thinking-2507-PTPC-FP8.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-30B-A3B-Thinking-2507-PTPC-FP8.yaml new file mode 100644 index 0000000000000000000000000000000000000000..1e01ebe4dba85edadd330de5449aee8dc621fe50 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-30B-A3B-Thinking-2507-PTPC-FP8.yaml @@ -0,0 +1,6 @@ +model_name: "amd/Qwen3-30B-A3B-Thinking-2507-PTPC-FP8" +accuracy_threshold: 0.81 +num_questions: 1319 +num_fewshot: 5 +max_tokens: 1024 +server_args: "--max-model-len 4096 --gpu-memory-utilization 0.85" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-4B-TQ-k3v4nc.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-4B-TQ-k3v4nc.yaml new file mode 100644 index 0000000000000000000000000000000000000000..7163c54ffd80bb6eb3335dce8c7e1b6d850b064d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-4B-TQ-k3v4nc.yaml @@ -0,0 +1,5 @@ +model_name: "Qwen/Qwen3-4B" +accuracy_threshold: 0.78 +num_questions: 1319 +num_fewshot: 5 +server_args: "--kv-cache-dtype turboquant_k3v4_nc --max-model-len 4096" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-4B-TQ-k8v4.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-4B-TQ-k8v4.yaml new file mode 100644 index 0000000000000000000000000000000000000000..983857799803eadfb6264dbffbee97adb7a800ba --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-4B-TQ-k8v4.yaml @@ -0,0 +1,5 @@ +model_name: "Qwen/Qwen3-4B" +accuracy_threshold: 0.80 +num_questions: 1319 +num_fewshot: 5 +server_args: "--kv-cache-dtype turboquant_k8v4 --max-model-len 4096" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-4B-TQ-t3nc.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-4B-TQ-t3nc.yaml new file mode 100644 index 0000000000000000000000000000000000000000..7c80ab13944ce7fe6a531cecea18e9e311493250 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-4B-TQ-t3nc.yaml @@ -0,0 +1,5 @@ +model_name: "Qwen/Qwen3-4B" +accuracy_threshold: 0.75 +num_questions: 1319 +num_fewshot: 5 +server_args: "--kv-cache-dtype turboquant_3bit_nc --max-model-len 4096" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-4B-TQ-t4nc.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-4B-TQ-t4nc.yaml new file mode 100644 index 0000000000000000000000000000000000000000..37549b345932240d3c0d9267191751295998a5d6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-4B-TQ-t4nc.yaml @@ -0,0 +1,5 @@ +model_name: "Qwen/Qwen3-4B" +accuracy_threshold: 0.80 +num_questions: 1319 +num_fewshot: 5 +server_args: "--kv-cache-dtype turboquant_4bit_nc --max-model-len 4096" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-Next-80B-A3B-NVFP4-EP2.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-Next-80B-A3B-NVFP4-EP2.yaml new file mode 100644 index 0000000000000000000000000000000000000000..406f922aea3d7e3f0939dcdcdd7bb357dd79acf7 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-Next-80B-A3B-NVFP4-EP2.yaml @@ -0,0 +1,11 @@ +model_name: "nm-testing/Qwen3-Next-80B-A3B-Instruct-NVFP4" +accuracy_threshold: 0.75 +num_questions: 1319 +num_fewshot: 5 +server_args: >- + --enforce-eager + --max-model-len 4096 + --tensor-parallel-size 2 + --enable-expert-parallel + --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":1}' + --moe-backend=flashinfer_trtllm diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-Next-FP8-EP2.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-Next-FP8-EP2.yaml new file mode 100644 index 0000000000000000000000000000000000000000..2de7f8702b46573fc62bf05afb297c5c5387b8b5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-Next-FP8-EP2.yaml @@ -0,0 +1,10 @@ +model_name: "Qwen/Qwen3-Next-80B-A3B-Instruct-FP8" +accuracy_threshold: 0.85 +num_questions: 1319 +num_fewshot: 5 +server_args: >- + --max-model-len 4096 + --tensor-parallel-size 2 + --enable-expert-parallel + --async-scheduling + --moe-backend=flashinfer_trtllm diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-Next-FP8-EP2_MI355.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-Next-FP8-EP2_MI355.yaml new file mode 100644 index 0000000000000000000000000000000000000000..24db74ce24206d69a44174ca51b49372452b2a2a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3-Next-FP8-EP2_MI355.yaml @@ -0,0 +1,9 @@ +model_name: "Qwen/Qwen3-Next-80B-A3B-Instruct-FP8" +accuracy_threshold: 0.85 +num_questions: 1319 +num_fewshot: 5 +server_args: >- + --max-model-len 4096 + --tensor-parallel-size 2 + --enable-expert-parallel + --async-scheduling diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-DEP2.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-DEP2.yaml new file mode 100644 index 0000000000000000000000000000000000000000..61d9b524508a0f2b338cedea4287d8ac1a2d0a23 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-DEP2.yaml @@ -0,0 +1,10 @@ +model_name: "Qwen/Qwen3.5-35B-A3B" +accuracy_threshold: 0.84 +tolerance: 0.03 +num_questions: 1319 +num_fewshot: 5 +server_args: >- + --max-model-len 4096 + --data-parallel-size 2 + --enable-expert-parallel + --no-enable-flashinfer-autotune \ No newline at end of file diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-FP8-DEP2.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-FP8-DEP2.yaml new file mode 100644 index 0000000000000000000000000000000000000000..e56dae8ca221683e2bbc08d41f72fe828b40005d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-FP8-DEP2.yaml @@ -0,0 +1,10 @@ +model_name: "Qwen/Qwen3.5-35B-A3B-FP8" +accuracy_threshold: 0.79 +tolerance: 0.03 +num_questions: 1319 +num_fewshot: 5 +server_args: >- + --max-model-len 4096 + --data-parallel-size 2 + --enable-expert-parallel + --kv-cache-dtype fp8 diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-MXFP4-AITER-TP2.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-MXFP4-AITER-TP2.yaml new file mode 100644 index 0000000000000000000000000000000000000000..02c8afb2300dfd7827cfc9dbb3241f5d134632b1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-MXFP4-AITER-TP2.yaml @@ -0,0 +1,13 @@ +model_name: "amd/Qwen3.5-35B-A3B-MXFP4" +accuracy_threshold: 0.89 +tolerance: 0.03 +num_questions: 1319 +num_fewshot: 5 +server_args: >- + --max-model-len 4096 + --tensor-parallel-size 2 + --gpu-memory-utilization 0.35 + --moe-backend aiter +env: + VLLM_ROCM_USE_AITER: "1" + ENABLE_CK: "0" # Avoid AITER CK-based MHA JIT compilation to save time diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-MXFP4-EMU-TP2.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-MXFP4-EMU-TP2.yaml new file mode 100644 index 0000000000000000000000000000000000000000..d95425960cd236cfd8cfd512533298291ee0a64b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-35B-A3B-MXFP4-EMU-TP2.yaml @@ -0,0 +1,11 @@ +model_name: "amd/Qwen3.5-35B-A3B-MXFP4" +accuracy_threshold: 0.89 +tolerance: 0.03 +num_questions: 1319 +num_fewshot: 5 +startup_max_wait_seconds: 1800 +server_args: >- + --max-model-len 4096 + --tensor-parallel-size 2 + --moe-backend emulation + --gpu-memory-utilization 0.35 diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-397B-A17B-NVFP4-DEP2-MTP.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-397B-A17B-NVFP4-DEP2-MTP.yaml new file mode 100644 index 0000000000000000000000000000000000000000..f6b0ec0c26bf8749a8f19453f657d14d263f02e8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-397B-A17B-NVFP4-DEP2-MTP.yaml @@ -0,0 +1,12 @@ +model_name: "nvidia/Qwen3.5-397B-A17B-NVFP4" +accuracy_threshold: 0.88 +tolerance: 0.03 +num_questions: 1319 +num_fewshot: 5 +server_args: >- + --max-model-len 4096 + --data-parallel-size 2 + --enable-expert-parallel + --max-num-seqs 384 + --spec-method mtp + --spec-tokens 3 diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-397B-A17B-NVFP4-DEP2.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-397B-A17B-NVFP4-DEP2.yaml new file mode 100644 index 0000000000000000000000000000000000000000..1faf138f0960651584093315fca6d91f3a04eddf --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/Qwen3.5-397B-A17B-NVFP4-DEP2.yaml @@ -0,0 +1,10 @@ +model_name: "nvidia/Qwen3.5-397B-A17B-NVFP4" +accuracy_threshold: 0.88 +tolerance: 0.03 +num_questions: 1319 +num_fewshot: 5 +server_args: >- + --max-model-len 4096 + --data-parallel-size 2 + --enable-expert-parallel + --max-num-seqs 512 diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/gemma-4-E4B-it-qat-mobile-ct.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/gemma-4-E4B-it-qat-mobile-ct.yaml new file mode 100644 index 0000000000000000000000000000000000000000..e0fc6f4ff15e83d07ef1bbd3a4a2d22f484108d7 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/gemma-4-E4B-it-qat-mobile-ct.yaml @@ -0,0 +1,5 @@ +model_name: "google/gemma-4-E4B-it-qat-mobile-ct" +accuracy_threshold: 0.50 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 4096" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/Qwen3-30B-A3B-MXFP4A16-humming-act-fp8.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/Qwen3-30B-A3B-MXFP4A16-humming-act-fp8.yaml new file mode 100644 index 0000000000000000000000000000000000000000..b62796814c7de9e141340d15baf0ac8224ab5fc4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/Qwen3-30B-A3B-MXFP4A16-humming-act-fp8.yaml @@ -0,0 +1,12 @@ +model_name: "nm-testing/Qwen3-30B-A3B-MXFP4A16" +accuracy_threshold: 0.86 +num_questions: 1319 +num_fewshot: 5 +server_args: >- + --enforce-eager + --max-model-len 8192 + --tensor-parallel-size 1 + --quantization humming + --kernel-config.enable_flashinfer_autotune=False +env: + VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"float8e4m3"}' diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/Qwen3-30B-A3B-MXFP4A16-humming.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/Qwen3-30B-A3B-MXFP4A16-humming.yaml new file mode 100644 index 0000000000000000000000000000000000000000..4c3e2c779b8141d026705c95ab5903a7bc86126d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/Qwen3-30B-A3B-MXFP4A16-humming.yaml @@ -0,0 +1,10 @@ +model_name: "nm-testing/Qwen3-30B-A3B-MXFP4A16" +accuracy_threshold: 0.86 +num_questions: 1319 +num_fewshot: 5 +server_args: >- + --enforce-eager + --max-model-len 8192 + --tensor-parallel-size 1 + --quantization humming + --kernel-config.enable_flashinfer_autotune=False diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/config-act-fp8.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/config-act-fp8.txt new file mode 100644 index 0000000000000000000000000000000000000000..b8739d6a12af2a30ebfcdc65e1d6a7ebac07f11a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/config-act-fp8.txt @@ -0,0 +1,2 @@ +gpt-oss-20b-humming-act-fp8.yaml +Qwen3-30B-A3B-MXFP4A16-humming-act-fp8.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/config.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/config.txt new file mode 100644 index 0000000000000000000000000000000000000000..d2dc3d7560916e49a4e921fc69513b6b66889e7c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/config.txt @@ -0,0 +1,2 @@ +gpt-oss-20b-humming.yaml +Qwen3-30B-A3B-MXFP4A16-humming.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/gpt-oss-20b-humming-act-fp8.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/gpt-oss-20b-humming-act-fp8.yaml new file mode 100644 index 0000000000000000000000000000000000000000..da925adbfe2a6ae62838663dba71de3a2e581fd9 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/gpt-oss-20b-humming-act-fp8.yaml @@ -0,0 +1,11 @@ +model_name: "openai/gpt-oss-20b" +accuracy_threshold: 0.30 +num_questions: 1319 +num_fewshot: 5 +server_args: >- + --enforce-eager + --max-model-len 8192 + --tensor-parallel-size 1 + --moe-backend humming +env: + VLLM_HUMMING_INPUT_QUANT_CONFIG: '{"dtype":"float8e4m3"}' diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/gpt-oss-20b-humming.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/gpt-oss-20b-humming.yaml new file mode 100644 index 0000000000000000000000000000000000000000..946aa3ee2920dacdb9084b9e985542ae572ff133 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/humming/gpt-oss-20b-humming.yaml @@ -0,0 +1,9 @@ +model_name: "openai/gpt-oss-20b" +accuracy_threshold: 0.30 +num_questions: 1319 +num_fewshot: 5 +server_args: >- + --enforce-eager + --max-model-len 8192 + --tensor-parallel-size 1 + --moe-backend humming diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-blackwell-ep.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-blackwell-ep.txt new file mode 100644 index 0000000000000000000000000000000000000000..6158ec6b5bb5944999bebbdf55637fd3575c3551 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-blackwell-ep.txt @@ -0,0 +1,3 @@ +Qwen3-Next-80B-A3B-NVFP4-EP2.yaml +Qwen3-Next-FP8-EP2.yaml +Nemotron-3-Super-120B-A12B-NVFP4.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-blackwell.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-blackwell.txt new file mode 100644 index 0000000000000000000000000000000000000000..d99bdf4c447c8a91115705cc7e53f25d174ea85b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-blackwell.txt @@ -0,0 +1,5 @@ +Qwen3-0.6B-FP8.yaml +Qwen2.5-VL-3B-Instruct-FP8-dynamic.yaml +Qwen1.5-MoE-W4A16-CT.yaml +DeepSeek-V2-Lite-Instruct-FP8.yaml +Qwen3-30B-A3B-NVFP4.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-h200.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-h200.txt new file mode 100644 index 0000000000000000000000000000000000000000..7e655a38a543e8e1363f107676d39bdbb44bb24d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-h200.txt @@ -0,0 +1,6 @@ +DeepSeek-R1-TP.yaml +DeepSeek-R1-DP.yaml +DeepSeek-V3.2-TP.yaml +DeepSeek-V3.2-DP.yaml +Nemotron-3-Super-120B-A12B-BF16.yaml +Nemotron-3-Super-120B-A12B-FP8.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-mi3xx-fp8-and-mixed.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-mi3xx-fp8-and-mixed.txt new file mode 100644 index 0000000000000000000000000000000000000000..e476f1a8c6426074697bdda0bac75b4bf52e2010 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-mi3xx-fp8-and-mixed.txt @@ -0,0 +1,7 @@ +Qwen3-0.6B-FP8.yaml +Qwen2.5-VL-3B-Instruct-FP8-dynamic.yaml +Qwen1.5-MoE-W4A16-CT.yaml +DeepSeek-V2-Lite-Instruct-FP8.yaml +Qwen3-Next-FP8-EP2_MI355.yaml +Qwen3-30B-A3B-Thinking-2507-FP8.yaml +Qwen3-30B-A3B-Thinking-2507-PTPC-FP8.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-mi3xx.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-mi3xx.txt new file mode 100644 index 0000000000000000000000000000000000000000..97ba99ec8a59723d818cee8e603e7278fc844d72 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-mi3xx.txt @@ -0,0 +1,7 @@ +DeepSeek-R1-TP_MI325.yaml +DeepSeek-R1-DP_MI325.yaml +DeepSeek-V3.2-TP_MI325.yaml +DeepSeek-V3.2-DP_MI325.yaml +Qwen3-30B-A3B-NVFP4.yaml +Qwen3.5-35B-A3B-MXFP4-AITER-TP2.yaml +Qwen3.5-35B-A3B-MXFP4-EMU-TP2.yaml \ No newline at end of file diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-qwen35-blackwell.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-qwen35-blackwell.txt new file mode 100644 index 0000000000000000000000000000000000000000..25911ed2f7faba60a4e44657cc691b3a15b002bf --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-qwen35-blackwell.txt @@ -0,0 +1,4 @@ +Qwen3.5-35B-A3B-DEP2.yaml +Qwen3.5-35B-A3B-FP8-DEP2.yaml +Qwen3.5-397B-A17B-NVFP4-DEP2.yaml +Qwen3.5-397B-A17B-NVFP4-DEP2-MTP.yaml \ No newline at end of file diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-qwen35-mi355.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-qwen35-mi355.txt new file mode 100644 index 0000000000000000000000000000000000000000..8425efb4fe3a28547a7b6502b21c4802d3332259 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-qwen35-mi355.txt @@ -0,0 +1,3 @@ +Qwen3.5-35B-A3B-DEP2.yaml +Qwen3.5-35B-A3B-MXFP4-AITER-TP2.yaml +Qwen3.5-35B-A3B-MXFP4-EMU-TP2.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-small-tp.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-small-tp.txt new file mode 100644 index 0000000000000000000000000000000000000000..2ea23f0bb872e728cb3e301ec4399c90deb5566d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-small-tp.txt @@ -0,0 +1 @@ +DiffusionGemma-26B-A4B-it-FP8-dynamic.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-small.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-small.txt new file mode 100644 index 0000000000000000000000000000000000000000..4c28772efa954738a18545613fd7b88c59064cf5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-small.txt @@ -0,0 +1,8 @@ +Qwen3-0.6B-FP8.yaml +Llama-3.2-1B-Instruct-INT8-CT.yaml +Llama-3-8B-Instruct-nonuniform-CT.yaml +Qwen2.5-VL-3B-Instruct-FP8-dynamic.yaml +Qwen1.5-MoE-W4A16-CT.yaml +DeepSeek-V2-Lite-Instruct-FP8.yaml +Qwen3-30B-A3B-MXFP4A16.yaml +gemma-4-E4B-it-qat-mobile-ct.yaml \ No newline at end of file diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-turboquant.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-turboquant.txt new file mode 100644 index 0000000000000000000000000000000000000000..7eb18c03e8da46a710ad3ddce7c12afbd0c6f9d1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/models-turboquant.txt @@ -0,0 +1,4 @@ +Qwen3-4B-TQ-k8v4.yaml +Qwen3-4B-TQ-t4nc.yaml +Qwen3-4B-TQ-k3v4nc.yaml +Qwen3-4B-TQ-t3nc.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Llama-4-Scout-Fp8-ModelOpt-triton.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Llama-4-Scout-Fp8-ModelOpt-triton.yaml new file mode 100644 index 0000000000000000000000000000000000000000..7d49996a1a920a5ab0d67624d7183a74fad2bd28 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Llama-4-Scout-Fp8-ModelOpt-triton.yaml @@ -0,0 +1,7 @@ +model_name: "nvidia/Llama-4-Scout-17B-16E-Instruct-FP8" +accuracy_threshold: 0.92 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --data-parallel-size 2 --enable-expert-parallel --moe-backend=triton" +env: + VLLM_USE_DEEP_GEMM: "0" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-BF16-triton.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-BF16-triton.yaml new file mode 100644 index 0000000000000000000000000000000000000000..f4681294cb42c300c24d2c51356d6134b62e288d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-BF16-triton.yaml @@ -0,0 +1,5 @@ +model_name: "Qwen/Qwen3-30B-A3B" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --data-parallel-size 2 --enable-expert-parallel" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm-deepep-ht.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm-deepep-ht.yaml new file mode 100644 index 0000000000000000000000000000000000000000..7e4de1c42e6c5457eb8651a76774f747384147b8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm-deepep-ht.yaml @@ -0,0 +1,8 @@ +model_name: "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --data-parallel-size 2 --enable-expert-parallel --all2all-backend deepep_high_throughput" +env: + VLLM_USE_DEEP_GEMM: "1" + VLLM_USE_DEEP_GEMM_MOE: "1" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm-deepep-ll.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm-deepep-ll.yaml new file mode 100644 index 0000000000000000000000000000000000000000..25a7171c27c2f57d6b5907a7dde2161194d6191c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm-deepep-ll.yaml @@ -0,0 +1,8 @@ +model_name: "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --data-parallel-size 2 --enable-expert-parallel --all2all-backend deepep_low_latency --disable-uvicorn-access-log" +env: + VLLM_USE_DEEP_GEMM: "1" + VLLM_USE_DEEP_GEMM_MOE: "1" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..e323ba6a593e4bd642c4fdd2ad05d57688e0c20a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm.yaml @@ -0,0 +1,8 @@ +model_name: "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --data-parallel-size 2 --enable-expert-parallel" +env: + VLLM_USE_DEEP_GEMM: "1" + VLLM_USE_DEEP_GEMM_MOE: "1" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-CT-Block-deepgemm-deepep-ht.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-CT-Block-deepgemm-deepep-ht.yaml new file mode 100644 index 0000000000000000000000000000000000000000..6a0a1532c619a3e1f50c225abe82a43bee342fb0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-CT-Block-deepgemm-deepep-ht.yaml @@ -0,0 +1,8 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-FP8-block" +accuracy_threshold: 0.85 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --data-parallel-size 2 --enable-expert-parallel --all2all-backend deepep_high_throughput" +env: + VLLM_USE_DEEP_GEMM: "1" + VLLM_USE_DEEP_GEMM_MOE: "1" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-CT-Block-deepgemm-deepep-ll.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-CT-Block-deepgemm-deepep-ll.yaml new file mode 100644 index 0000000000000000000000000000000000000000..baab0a33255005a4486555b11d3d55ed94ebe2eb --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-CT-Block-deepgemm-deepep-ll.yaml @@ -0,0 +1,8 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-FP8-block" +accuracy_threshold: 0.85 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --data-parallel-size 2 --enable-expert-parallel --all2all-backend deepep_low_latency --disable-uvicorn-access-log" +env: + VLLM_USE_DEEP_GEMM: "1" + VLLM_USE_DEEP_GEMM_MOE: "1" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-CT-Block-deepgemm.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-CT-Block-deepgemm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..f3ce97b2730a60f3b175def8b25154d4f116d0d2 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-Fp8-CT-Block-deepgemm.yaml @@ -0,0 +1,8 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-FP8-block" +accuracy_threshold: 0.85 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --data-parallel-size 2 --enable-expert-parallel" +env: + VLLM_USE_DEEP_GEMM: "1" + VLLM_USE_DEEP_GEMM_MOE: "1" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-CT-fi-cutedsl-deepep-ll.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-CT-fi-cutedsl-deepep-ll.yaml new file mode 100644 index 0000000000000000000000000000000000000000..bfb51ee0c3962de56c67c8b5008fe831502c9b68 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-CT-fi-cutedsl-deepep-ll.yaml @@ -0,0 +1,5 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-NVFP4" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --data-parallel-size 2 --enable-expert-parallel --all2all-backend deepep_low_latency --moe-backend=flashinfer_cutedsl" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-CT-fi-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-CT-fi-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..84e6aa193d02c95979d6d6f1a1b437e8aa61fdcc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-CT-fi-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-NVFP4" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --data-parallel-size 2 --enable-expert-parallel --moe-backend=flashinfer_cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-cutedsl-deepep-ll.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-cutedsl-deepep-ll.yaml new file mode 100644 index 0000000000000000000000000000000000000000..e74a66fa8ef500f45a9e44515b28672783256019 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-cutedsl-deepep-ll.yaml @@ -0,0 +1,5 @@ +model_name: "nvidia/Qwen3-30B-A3B-NVFP4" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --data-parallel-size 2 --enable-expert-parallel --all2all-backend deepep_low_latency --moe-backend=flashinfer_cutedsl" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..07ec1a880045e333bc9897db140b575dc4f528e0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "nvidia/Qwen3-30B-A3B-NVFP4" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --data-parallel-size 2 --enable-expert-parallel --moe-backend=flashinfer_cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-trtllm.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-trtllm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..4ec02793b6233e1bee8f5cbd86e112c040a62130 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-trtllm.yaml @@ -0,0 +1,5 @@ +model_name: "nvidia/Qwen3-30B-A3B-NVFP4" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --data-parallel-size 2 --enable-expert-parallel --moe-backend=flashinfer_trtllm" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/config-b200.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/config-b200.txt new file mode 100644 index 0000000000000000000000000000000000000000..52c05defe1b1981788403f68057b2bf7044d799f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor-dp-ep/config-b200.txt @@ -0,0 +1,12 @@ +Qwen3-30B-A3B-NvFp4-CT-fi-cutedsl-deepep-ll.yaml +Qwen3-30B-A3B-NvFp4-ModelOpt-fi-cutedsl-deepep-ll.yaml +Qwen3-30B-A3B-NvFp4-CT-fi-cutlass.yaml +Qwen3-30B-A3B-NvFp4-ModelOpt-fi-trtllm.yaml +Qwen3-30B-A3B-NvFp4-ModelOpt-fi-cutlass.yaml +Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm-deepep-ht.yaml +Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm-deepep-ll.yaml +Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm.yaml +Qwen3-30B-A3B-Fp8-CT-Block-deepgemm-deepep-ht.yaml +Qwen3-30B-A3B-Fp8-CT-Block-deepgemm-deepep-ll.yaml +Qwen3-30B-A3B-Fp8-CT-Block-deepgemm.yaml +Qwen3-30B-A3B-BF16-triton.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/DeepSeek-V4-Flash-deepgemm-mega-moe.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/DeepSeek-V4-Flash-deepgemm-mega-moe.yaml new file mode 100644 index 0000000000000000000000000000000000000000..c818b2c28b183a725ed9fb862b4bd3f4973bc6fa --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/DeepSeek-V4-Flash-deepgemm-mega-moe.yaml @@ -0,0 +1,5 @@ +model_name: "deepseek-ai/DeepSeek-V4-Flash" +accuracy_threshold: 0.95 +num_questions: 1319 +num_fewshot: 5 +server_args: "--trust-remote-code --kv-cache-dtype fp8 --block-size 256 --enable-expert-parallel --tensor-parallel-size 2 --attention_config.use_fp4_indexer_cache=True --moe-backend deep_gemm_mega_moe --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --speculative_config.method=mtp --speculative_config.num_speculative_tokens=2" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-BF16-fi-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-BF16-fi-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..640d7edca333c4c56fc27c65c3a109db2adcfaa1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-BF16-fi-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "meta-llama/Llama-4-Scout-17B-16E-Instruct" +accuracy_threshold: 0.92 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --enable-expert-parallel --moe-backend=flashinfer_cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-BF16-triton.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-BF16-triton.yaml new file mode 100644 index 0000000000000000000000000000000000000000..37c1b7d0de1fe45a53913a8eaa7b10abde0bd973 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-BF16-triton.yaml @@ -0,0 +1,6 @@ +model_name: "meta-llama/Llama-4-Scout-17B-16E-Instruct" +accuracy_threshold: 0.92 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2" + diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-CT-vllm-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-CT-vllm-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..394b2d3c589a200876dc00b87580f68ddcea49ce --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-CT-vllm-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic" +accuracy_threshold: 0.92 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-ModelOpt-fi-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-ModelOpt-fi-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..7638e24e7a6f8cb042cc2cda8c0ad1c9cb6e64d3 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-ModelOpt-fi-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "nvidia/Llama-4-Scout-17B-16E-Instruct-FP8" +accuracy_threshold: 0.92 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=flashinfer_cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-ModelOpt-fi-trtllm.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-ModelOpt-fi-trtllm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..443130c7a62e5e784ddc9d7a4aa8d58a84f1892b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-ModelOpt-fi-trtllm.yaml @@ -0,0 +1,5 @@ +model_name: "nvidia/Llama-4-Scout-17B-16E-Instruct-FP8" +accuracy_threshold: 0.92 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=flashinfer_trtllm" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-ModelOpt-marlin.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-ModelOpt-marlin.yaml new file mode 100644 index 0000000000000000000000000000000000000000..be08234448393328ec74a16a909360c12e366b88 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-ModelOpt-marlin.yaml @@ -0,0 +1,7 @@ +model_name: "nvidia/Llama-4-Scout-17B-16E-Instruct-FP8" +accuracy_threshold: 0.92 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2" +env: + VLLM_TEST_FORCE_FP8_MARLIN: "1" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-ModelOpt-triton.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-ModelOpt-triton.yaml new file mode 100644 index 0000000000000000000000000000000000000000..8be681be4ac69470a868c3af8ff7591dad933dc2 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Llama-4-Scout-Fp8-ModelOpt-triton.yaml @@ -0,0 +1,5 @@ +model_name: "nvidia/Llama-4-Scout-17B-16E-Instruct-FP8" +accuracy_threshold: 0.92 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=triton" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Mixtral-8x7B-BF16-fi-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Mixtral-8x7B-BF16-fi-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..4acd856fdd152448ffc63199cdebe54430325da0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Mixtral-8x7B-BF16-fi-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "mistralai/Mixtral-8x7B-v0.1" +accuracy_threshold: 0.58 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --enable-expert-parallel --moe-backend=flashinfer_cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Mixtral-8x7B-BF16-triton.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Mixtral-8x7B-BF16-triton.yaml new file mode 100644 index 0000000000000000000000000000000000000000..e9347c99a27fd7f189268ce8901c87adb0bb0153 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Mixtral-8x7B-BF16-triton.yaml @@ -0,0 +1,5 @@ +model_name: "mistralai/Mixtral-8x7B-v0.1" +accuracy_threshold: 0.58 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Mixtral-8x7B-Fp8-AutoFp8-fi-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Mixtral-8x7B-Fp8-AutoFp8-fi-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..30b3b0bd84965545eb77da5ea98f2a0063618271 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Mixtral-8x7B-Fp8-AutoFp8-fi-cutlass.yaml @@ -0,0 +1,6 @@ +# TODO(rob): enable +# model_name: "amd/Mixtral-8x7B-Instruct-v0.1-FP8-KV" +# accuracy_threshold: 0.62 +# num_questions: 1319 +# num_fewshot: 5 +# server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=flashinfer_cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Mixtral-8x7B-Fp8-AutoFp8-triton.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Mixtral-8x7B-Fp8-AutoFp8-triton.yaml new file mode 100644 index 0000000000000000000000000000000000000000..0eb9e68fac26c42aed47c0b0f2f0b61e65881228 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Mixtral-8x7B-Fp8-AutoFp8-triton.yaml @@ -0,0 +1,5 @@ +model_name: "amd/Mixtral-8x7B-Instruct-v0.1-FP8-KV" +accuracy_threshold: 0.62 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Nemotron-Nano-30B-Fp8-ModelOpt-fi-trtllm.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Nemotron-Nano-30B-Fp8-ModelOpt-fi-trtllm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..ce87c6ba085c3c56966c7de821e848425696e20e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Nemotron-Nano-30B-Fp8-ModelOpt-fi-trtllm.yaml @@ -0,0 +1,5 @@ +model_name: "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8" +accuracy_threshold: 0.29 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=flashinfer_trtllm" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Nemotron-Nano-30B-NvFp4-ModelOpt-fi-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Nemotron-Nano-30B-NvFp4-ModelOpt-fi-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..86fe4b28ec074dfbea8e89ed2ead3d351690b164 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Nemotron-Nano-30B-NvFp4-ModelOpt-fi-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4" +accuracy_threshold: 0.29 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=flashinfer_cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Nemotron-Nano-30B-NvFp4-ModelOpt-vllm-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Nemotron-Nano-30B-NvFp4-ModelOpt-vllm-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..0241949be111072b9fbd7cbbe923c7247f14b631 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Nemotron-Nano-30B-NvFp4-ModelOpt-vllm-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4" +accuracy_threshold: 0.29 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-BF16-fi-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-BF16-fi-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..3e89dc15bea6a36dc184237d9c7841ed675cc9d8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-BF16-fi-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "Qwen/Qwen3-30B-A3B" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --enable-expert-parallel --moe-backend=flashinfer_cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-BF16-triton.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-BF16-triton.yaml new file mode 100644 index 0000000000000000000000000000000000000000..7aa556aa223be56cb76083842aaa04b63797975b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-BF16-triton.yaml @@ -0,0 +1,5 @@ +model_name: "Qwen/Qwen3-30B-A3B" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..d98dc0b86910c429072d0bdc6dbc5a6eaa0b9e27 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm.yaml @@ -0,0 +1,8 @@ +model_name: "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2" +env: + VLLM_USE_DEEP_GEMM: "1" + VLLM_USE_DEEP_GEMM_MOE: "1" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-fi-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-fi-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..64d246d1d92afc41f93beb740a15079dbce0f0a6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-fi-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=flashinfer_cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-fi-trtllm.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-fi-trtllm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..4d8db9f8b7e7bf8a8340b625822fb4ce5fdb5f7b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-fi-trtllm.yaml @@ -0,0 +1,5 @@ +model_name: "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=flashinfer_trtllm" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-marlin.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-marlin.yaml new file mode 100644 index 0000000000000000000000000000000000000000..e2a0ce33f7ee01a2fa87564463361f59dbf7142b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-marlin.yaml @@ -0,0 +1,7 @@ +model_name: "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2" +env: + VLLM_TEST_FORCE_FP8_MARLIN: "1" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-triton.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-triton.yaml new file mode 100644 index 0000000000000000000000000000000000000000..3aa9942553c414e22ae7f3e65616aaa060029231 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-AutoFp8-triton.yaml @@ -0,0 +1,7 @@ +model_name: "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=triton" +env: + VLLM_USE_DEEP_GEMM: "0" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Block-deepgemm.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Block-deepgemm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..0e5109a6352f0b087acf8b7b0d85a83585b9391e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Block-deepgemm.yaml @@ -0,0 +1,8 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-FP8-block" +accuracy_threshold: 0.85 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2" +env: + VLLM_USE_DEEP_GEMM: "1" + VLLM_USE_DEEP_GEMM_MOE: "1" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Block-fi-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Block-fi-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..b3ebc14fb8859523f1d9d3d678c9cb8c594a0666 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Block-fi-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-FP8-block" +accuracy_threshold: 0.85 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=flashinfer_cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Block-marlin.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Block-marlin.yaml new file mode 100644 index 0000000000000000000000000000000000000000..a16668a49c5688747a477d96298397f66588f350 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Block-marlin.yaml @@ -0,0 +1,7 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-FP8-block" +accuracy_threshold: 0.85 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2" +env: + VLLM_TEST_FORCE_FP8_MARLIN: "1" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Block-triton.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Block-triton.yaml new file mode 100644 index 0000000000000000000000000000000000000000..f9750efe57b9c3721cd9a2bf4884038e254d18f9 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Block-triton.yaml @@ -0,0 +1,7 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-FP8-block" +accuracy_threshold: 0.85 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=triton" +env: + VLLM_USE_DEEP_GEMM: "0" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Channel-marlin.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Channel-marlin.yaml new file mode 100644 index 0000000000000000000000000000000000000000..2e402a8cb80aa1d0d17f30db49bb0dbcd8614e1f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Channel-marlin.yaml @@ -0,0 +1,7 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-FP8-dynamic" +accuracy_threshold: 0.85 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2" +env: + VLLM_TEST_FORCE_FP8_MARLIN: "1" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Channel-vllm-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Channel-vllm-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..07f64a21a8af183cd211d487bc6cfec6d463d20a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-Fp8-CT-Channel-vllm-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-FP8-dynamic" +accuracy_threshold: 0.85 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-CT-fi-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-CT-fi-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..424b8eaf7b3518012375e0b0dde0b9b158e05312 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-CT-fi-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-NVFP4" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=flashinfer_cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-CT-fi-trtllm.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-CT-fi-trtllm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..ab07f16640abe5b12af302e83160e2b79093a58d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-CT-fi-trtllm.yaml @@ -0,0 +1,5 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-NVFP4" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=flashinfer_trtllm" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-CT-marlin.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-CT-marlin.yaml new file mode 100644 index 0000000000000000000000000000000000000000..06a7c8af097801c354e55302525c52bab558b670 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-CT-marlin.yaml @@ -0,0 +1,7 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-NVFP4" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2" +env: + VLLM_TEST_FORCE_FP8_MARLIN: "1" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-CT-vllm-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-CT-vllm-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..1128abf93ead30b354f70060138047ca7bfc0327 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-CT-vllm-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "RedHatAI/Qwen3-30B-A3B-NVFP4" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..ee0f4757f16726a8ce10d8052c3c3f8c427696b6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "nvidia/Qwen3-30B-A3B-NVFP4" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=flashinfer_cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-trtllm.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-trtllm.yaml new file mode 100644 index 0000000000000000000000000000000000000000..ca23efe5055f8f01476028e0e6c417adaf75f63f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-ModelOpt-fi-trtllm.yaml @@ -0,0 +1,5 @@ +model_name: "nvidia/Qwen3-30B-A3B-NVFP4" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=flashinfer_trtllm" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-ModelOpt-marlin.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-ModelOpt-marlin.yaml new file mode 100644 index 0000000000000000000000000000000000000000..6d1142935e9540f287561ca524db567aaa6b3205 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-ModelOpt-marlin.yaml @@ -0,0 +1,7 @@ +model_name: "nvidia/Qwen3-30B-A3B-NVFP4" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2" +env: + VLLM_TEST_FORCE_FP8_MARLIN: "1" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-ModelOpt-vllm-cutlass.yaml b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-ModelOpt-vllm-cutlass.yaml new file mode 100644 index 0000000000000000000000000000000000000000..fd409f0ea863b338c9b6d8c69088bd5f75dd779f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/Qwen3-30B-A3B-NvFp4-ModelOpt-vllm-cutlass.yaml @@ -0,0 +1,5 @@ +model_name: "nvidia/Qwen3-30B-A3B-NVFP4" +accuracy_threshold: 0.88 +num_questions: 1319 +num_fewshot: 5 +server_args: "--enforce-eager --max-model-len 8192 --tensor-parallel-size 2 --moe-backend=cutlass" diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/config-b200.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/config-b200.txt new file mode 100644 index 0000000000000000000000000000000000000000..be31654b28d5e39ed23fda5335c6f2220562d614 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/config-b200.txt @@ -0,0 +1,19 @@ +Llama-4-Scout-Fp8-CT-vllm-cutlass.yaml +Llama-4-Scout-Fp8-ModelOpt-fi-trtllm.yaml +Qwen3-30B-A3B-Fp8-AutoFp8-fi-trtllm.yaml +Qwen3-30B-A3B-NvFp4-CT-vllm-cutlass.yaml +Qwen3-30B-A3B-NvFp4-CT-marlin.yaml +Qwen3-30B-A3B-NvFp4-CT-fi-trtllm.yaml +Qwen3-30B-A3B-NvFp4-CT-fi-cutlass.yaml +Qwen3-30B-A3B-NvFp4-ModelOpt-vllm-cutlass.yaml +Qwen3-30B-A3B-NvFp4-ModelOpt-marlin.yaml +Qwen3-30B-A3B-NvFp4-ModelOpt-fi-trtllm.yaml +Qwen3-30B-A3B-NvFp4-ModelOpt-fi-cutlass.yaml +Llama-4-Scout-BF16-fi-cutlass.yaml +Llama-4-Scout-BF16-triton.yaml +Mixtral-8x7B-BF16-fi-cutlass.yaml +Mixtral-8x7B-BF16-triton.yaml +Nemotron-Nano-30B-Fp8-ModelOpt-fi-trtllm.yaml +Nemotron-Nano-30B-NvFp4-ModelOpt-fi-cutlass.yaml +Nemotron-Nano-30B-NvFp4-ModelOpt-vllm-cutlass.yaml +DeepSeek-V4-Flash-deepgemm-mega-moe.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/config-h100.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/config-h100.txt new file mode 100644 index 0000000000000000000000000000000000000000..7c31ef3a7699d1ba10c8a413b96a8352bbe98b84 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/config-h100.txt @@ -0,0 +1,12 @@ +Mixtral-8x7B-Fp8-AutoFp8-triton.yaml +Qwen3-30B-A3B-Fp8-AutoFp8-deepgemm.yaml +Qwen3-30B-A3B-Fp8-AutoFp8-fi-cutlass.yaml +Qwen3-30B-A3B-Fp8-AutoFp8-marlin.yaml +Qwen3-30B-A3B-Fp8-AutoFp8-triton.yaml +Qwen3-30B-A3B-Fp8-CT-Block-deepgemm.yaml +Qwen3-30B-A3B-Fp8-CT-Block-marlin.yaml +Qwen3-30B-A3B-Fp8-CT-Block-triton.yaml +Qwen3-30B-A3B-Fp8-CT-Channel-marlin.yaml +Qwen3-30B-A3B-Fp8-CT-Channel-vllm-cutlass.yaml +Qwen3-30B-A3B-BF16-fi-cutlass.yaml +Qwen3-30B-A3B-BF16-triton.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/config-test.txt b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/config-test.txt new file mode 100644 index 0000000000000000000000000000000000000000..1816666bec0a20dc297bf7509c28ae9ec43cb301 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/configs/moe-refactor/config-test.txt @@ -0,0 +1 @@ +Qwen3-30B-A3B-NvFp4-CT-marlin.yaml \ No newline at end of file diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/conftest.py b/asian_ai/performance/vLLM/tests/evals/gsm8k/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..a6f338afe4d06d0da894f5cd4a56863be41df3ae --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/conftest.py @@ -0,0 +1,61 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from pathlib import Path + + +def pytest_addoption(parser): + """Add custom command line options.""" + parser.addoption( + "--config-list-file", + default="configs/models-small.txt", + help="File containing list of config files to test", + ) + + +def pytest_generate_tests(metafunc): + """Generate test parameters from config files.""" + if "config_filename" in metafunc.fixturenames: + config_list_file = metafunc.config.getoption("--config-list-file") + + # Handle both relative and absolute paths + config_list_path = Path(config_list_file) + if not config_list_path.is_absolute(): + # If relative, try relative to test directory first + test_dir_path = Path(__file__).parent / config_list_file + if test_dir_path.exists(): + config_list_path = test_dir_path + else: + # Try relative to current working directory + config_list_path = Path.cwd() / config_list_file + + print(f"Looking for config list at: {config_list_path}") + + config_files = [] + if config_list_path.exists(): + # Determine config directory (same directory as the list file) + config_dir = config_list_path.parent + + with open(config_list_path) as f: + for line in f: + line = line.strip() + if line and not line.startswith("#"): + config_path = config_dir / line + print(f"Checking config file: {config_path}") + if config_path.exists(): + config_files.append(config_path) + print(f" ✓ Found: {config_path}") + else: + print(f" ✗ Missing: {config_path}") + else: + print(f"Config list file not found: {config_list_path}") + + # Generate test parameters + if config_files: + metafunc.parametrize( + "config_filename", + config_files, + ids=[config_file.stem for config_file in config_files], + ) + else: + print("No config files found, test will be skipped") diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/gsm8k_eval.py b/asian_ai/performance/vLLM/tests/evals/gsm8k/gsm8k_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..f8c1f1bfd78f1659f1ad6c09741be986c15131fc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/gsm8k_eval.py @@ -0,0 +1,364 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Isolated GSM8K evaluation script for vLLM serve endpoint. +""" + +import argparse +import ast +import asyncio +import json +import os +import time +from collections.abc import Generator + +import aiohttp +import numpy as np +import regex as re +import requests +from tqdm.asyncio import tqdm + +INVALID = -9999999 + + +def download_and_cache_file(url: str, filename: str | None = None) -> str: + """Download and cache a file from a URL.""" + if filename is None: + filename = os.path.join("/tmp", url.split("/")[-1]) + + if os.path.exists(filename): + return filename + + print(f"Downloading from {url} to {filename}") + response = requests.get(url, stream=True) + response.raise_for_status() + + with open(filename, "wb") as f: + for chunk in response.iter_content(chunk_size=1024): + f.write(chunk) + + return filename + + +def load_gsm8k_data() -> tuple[list[dict], list[dict]]: + """Load GSM8K train and test data""" + train_url = "https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/train.jsonl" + test_url = "https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/test.jsonl" + + train_file = download_and_cache_file(train_url) + test_file = download_and_cache_file(test_url) + + train_data = list(read_jsonl(train_file)) + test_data = list(read_jsonl(test_file)) + + return train_data, test_data + + +def read_jsonl(filename: str) -> Generator[dict, None, None]: + """Read a JSONL file.""" + with open(filename) as fin: + for line in fin: + if not line.startswith("#"): + yield json.loads(line) + + +def get_answer_value(answer_str: str) -> int: + """Extract the numerical answer from the response.""" + answer_str = answer_str.replace(",", "") + numbers = re.findall(r"\d+", answer_str) + if len(numbers) < 1: + return INVALID + try: + return ast.literal_eval(numbers[-1]) + except SyntaxError: + return INVALID + + +async def call_vllm_api( + session: aiohttp.ClientSession, + prompt: str, + temperature: float, + max_tokens: int, + stop: list[str] | None = None, + url: str | None = None, + seed: int | None = None, +) -> tuple[str, int]: + """Call vLLM's OpenAI-compatible completions endpoint. + + Returns: + Tuple of (response_text, completion_tokens) + """ + data = { + "prompt": prompt, + "temperature": temperature, + "max_tokens": max_tokens, + "stop": stop, + } + if seed is not None: + data["seed"] = seed + + try: + async with session.post(f"{url}/v1/completions", json=data) as response: + response.raise_for_status() + result = await response.json() + text = result["choices"][0]["text"] + completion_tokens = result.get("usage", {}).get("completion_tokens", 0) + return text, completion_tokens + except Exception as e: + print(f"Error calling vLLM API ({type(e).__name__}): {e}") + return "", 0 + + +async def call_vllm_chat_api( + session: aiohttp.ClientSession, + model: str, + prompt: str, + temperature: float, + max_tokens: int, + stop: list[str] | None = None, + url: str | None = None, + seed: int | None = None, +) -> tuple[str, int]: + """Call vLLM's OpenAI-compatible chat completions endpoint.""" + data = { + "model": model, + "messages": [{"role": "user", "content": prompt}], + "temperature": temperature, + "max_tokens": max_tokens, + "stop": stop, + } + if seed is not None: + data["seed"] = seed + + try: + async with session.post(f"{url}/v1/chat/completions", json=data) as response: + response.raise_for_status() + result = await response.json() + text = result["choices"][0]["message"]["content"] or "" + completion_tokens = result.get("usage", {}).get("completion_tokens", 0) + return text, completion_tokens + except Exception as e: + print(f"Error calling vLLM chat API ({type(e).__name__}): {e}") + return "", 0 + + +def _build_gsm8k_prompts( + num_questions: int = 1319, + num_shots: int = 5, +) -> tuple[list[str], list[int]]: + """Build few-shot GSM8K completion prompts and ground-truth labels.""" + if num_questions == 0: + return [], [] + train_data, test_data = load_gsm8k_data() + num_questions = min(num_questions, len(test_data)) + + few_shot_examples = "" + for i in range(num_shots): + few_shot_examples += ( + f"Question: {train_data[i]['question']}\n" + f"Answer: {train_data[i]['answer']}\n\n" + ) + + prompts = [] + labels = [] + for i in range(num_questions): + prompts.append( + few_shot_examples + f"Question: {test_data[i]['question']}\nAnswer:" + ) + labels.append(get_answer_value(test_data[i]["answer"])) + + assert all(label != INVALID for label in labels), "Some labels are invalid" + return prompts, labels + + +def _score_gsm8k( + states: list[str], + output_tokens: list[int], + labels: list[int], + num_shots: int, + max_tokens: int, + latency: float, +) -> dict[str, float | int]: + """Score GSM8K responses and return a results dict.""" + num_questions = len(labels) + preds = [get_answer_value(state) for state in states] + accuracy = np.mean(np.array(preds) == np.array(labels)) + invalid_rate = np.mean(np.array(preds) == INVALID) + total_output_tokens = sum(output_tokens) + tokens_per_second = total_output_tokens / latency if latency > 0 else 0.0 + + return { + "accuracy": accuracy, + "invalid_rate": invalid_rate, + "latency": latency, + "questions_per_second": num_questions / latency if latency > 0 else 0.0, + "total_output_tokens": total_output_tokens, + "tokens_per_second": tokens_per_second, + "num_questions": num_questions, + "num_shots": num_shots, + "max_tokens": max_tokens, + "timestamp": time.time(), + } + + +def evaluate_gsm8k( + num_questions: int = 1319, + num_shots: int = 5, + max_tokens: int = 256, + model: str | None = None, + use_chat_completions: bool = False, + host: str = "http://127.0.0.1", + port: int = 8000, + temperature: float = 0.0, + seed: int | None = 42, + request_timeout_seconds: float = 600, +) -> dict[str, float | int]: + """ + Evaluate GSM8K accuracy using vLLM serve endpoint. + + Returns dict with accuracy, invalid_rate, latency, etc. + """ + base_url = f"{host}:{port}" + prompts, labels = _build_gsm8k_prompts(num_questions, num_shots) + num_questions = len(prompts) + + async def run_async_evaluation(): + states: list[str] = [""] * num_questions + output_tokens: list[int] = [0] * num_questions + + async def get_answer(session: aiohttp.ClientSession, i: int) -> tuple[str, int]: + stop = ["Question", "Assistant:", "<|separator|>"] + if use_chat_completions: + if model is None: + raise ValueError("model is required for chat completions") + answer, tokens = await call_vllm_chat_api( + session=session, + model=model, + prompt=prompts[i], + temperature=temperature, + max_tokens=max_tokens, + stop=stop, + url=base_url, + seed=seed, + ) + else: + answer, tokens = await call_vllm_api( + session=session, + prompt=prompts[i], + temperature=temperature, + max_tokens=max_tokens, + stop=stop, + url=base_url, + seed=seed, + ) + states[i] = answer + output_tokens[i] = tokens + return answer, tokens + + timeout = aiohttp.ClientTimeout(total=request_timeout_seconds) + async with aiohttp.ClientSession(timeout=timeout) as session: + tasks = [get_answer(session, i) for i in range(num_questions)] + await tqdm.gather(*tasks, desc="Evaluating") + + return states, output_tokens + + print(f"Running GSM8K evaluation: {num_questions} questions, {num_shots}-shot") + + tic = time.perf_counter() + states, output_tokens = asyncio.run(run_async_evaluation()) + latency = time.perf_counter() - tic + + return _score_gsm8k(states, output_tokens, labels, num_shots, max_tokens, latency) + + +def evaluate_gsm8k_offline( + llm, + num_questions: int = 1319, + num_shots: int = 5, + max_tokens: int = 256, + temperature: float = 0.0, +) -> dict[str, float | int]: + """Evaluate GSM8K accuracy using an offline vllm.LLM object. + + Same prompts and scoring as evaluate_gsm8k(), but runs generation + directly via llm.generate() instead of calling a server over HTTP. + """ + from vllm import SamplingParams + + prompts, labels = _build_gsm8k_prompts(num_questions, num_shots) + + sampling_params = SamplingParams( + temperature=temperature, + max_tokens=max_tokens, + stop=["Question", "Assistant:", "<|separator|>"], + ) + + print( + f"Running offline GSM8K evaluation: {len(prompts)} questions, {num_shots}-shot" + ) + + tic = time.perf_counter() + outputs = llm.generate(prompts, sampling_params) + latency = time.perf_counter() - tic + + states = [o.outputs[0].text for o in outputs] + output_tokens = [len(o.outputs[0].token_ids) for o in outputs] + + return _score_gsm8k(states, output_tokens, labels, num_shots, max_tokens, latency) + + +def main() -> None: + parser = argparse.ArgumentParser(description="GSM8K evaluation for vLLM serve") + parser.add_argument( + "--num-shots", type=int, default=5, help="Number of few-shot examples" + ) + parser.add_argument( + "--num-questions", + type=int, + default=1319, + help="Number of questions to evaluate", + ) + parser.add_argument( + "--max-tokens", type=int, default=256, help="Max tokens for generation" + ) + parser.add_argument("--host", type=str, default="http://127.0.0.1", help="Host URL") + parser.add_argument("--port", type=int, default=8000, help="Port number") + parser.add_argument( + "--temperature", type=float, default=0.0, help="Temperature for generation" + ) + parser.add_argument( + "--seed", type=int, default=42, help="Random seed for reproducibility" + ) + parser.add_argument("--save-results", type=str, help="Save results to JSON file") + + args = parser.parse_args() + + result = evaluate_gsm8k( + num_questions=args.num_questions, + num_shots=args.num_shots, + max_tokens=args.max_tokens, + host=args.host, + port=args.port, + temperature=args.temperature, + seed=args.seed, + ) + + # Print results to terminal + print("\nResults:") + print(f"Accuracy: {result['accuracy']:.3f}") + print(f"Invalid responses: {result['invalid_rate']:.3f}") + print(f"Total latency: {result['latency']:.3f} s") + print(f"Questions per second: {result['questions_per_second']:.3f}") + print(f"Total output tokens: {result['total_output_tokens']}") + print(f"Output tokens per second: {result['tokens_per_second']:.3f}") + + # Optional file saving + if args.save_results: + with open(args.save_results, "w") as f: + json.dump(result, f, indent=2) + print(f"Results saved to {args.save_results}") + + +if __name__ == "__main__": + main() diff --git a/asian_ai/performance/vLLM/tests/evals/gsm8k/test_gsm8k_correctness.py b/asian_ai/performance/vLLM/tests/evals/gsm8k/test_gsm8k_correctness.py new file mode 100644 index 0000000000000000000000000000000000000000..5989e75d76c55ef426542a6c214a633c7e234321 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/gsm8k/test_gsm8k_correctness.py @@ -0,0 +1,184 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +GSM8K evaluation using vLLM server and isolated GSM8K script. +Replacement for lm-eval-harness with better performance and control. + +Usage: +pytest -s -v tests/evals/gsm8k/test_gsm8k_correctness.py \ + --config-list-file=configs/models-small.txt +""" + +import importlib.metadata +import shlex +from importlib.util import find_spec + +import pytest +import torch +import yaml +from packaging import version + +from tests.utils import RemoteOpenAIServer +from vllm.platforms import current_platform + +from .gsm8k_eval import evaluate_gsm8k + +# MXFP4 via quark requires amd-quark >= 0.12 on torch >= 2.11. +# Earlier torch releases work with older quark versions. See +# https://github.com/amd/Quark/issues/34 +# TODO: Remove once amd-quark>=0.12.0 +QUARK_MXFP4_TORCH_COMPATIBLE = find_spec("quark") is not None and ( + version.parse(importlib.metadata.version("amd-quark")) >= version.parse("0.12.0") + if version.parse(torch.__version__.split("+")[0]) >= version.parse("2.11") + else True +) + +DEFAULT_STARTUP_MAX_WAIT_SECONDS = 1200 + + +def run_gsm8k_eval(eval_config: dict, server_url: str) -> dict: + """Run GSM8K evaluation using our isolated script.""" + # Extract host and port from server URL + if "://" in server_url: + server_url = server_url.split("://")[1] + + host_port = server_url.split("/")[0] # Remove path if present + if ":" in host_port: + host, p = host_port.split(":") + port = int(p) + else: + host = host_port + port = 8000 + + # Add http:// prefix if not present + if not host.startswith("http"): + host = f"http://{host}" + + # Run GSM8K evaluation + request_timeout_seconds = eval_config.get("request_timeout_seconds", 600) + if current_platform.is_rocm(): + request_timeout_seconds = eval_config.get( + "rocm_request_timeout_seconds", request_timeout_seconds + ) + + results = evaluate_gsm8k( + num_questions=eval_config["num_questions"], + num_shots=eval_config["num_fewshot"], + max_tokens=eval_config.get("max_tokens", 256), + model=eval_config["model_name"], + use_chat_completions=eval_config.get("use_chat_completions", False), + host=host, + port=port, + request_timeout_seconds=request_timeout_seconds, + ) + + return results + + +def test_gsm8k_correctness(config_filename): + """Test GSM8K correctness for a given model configuration.""" + eval_config = yaml.safe_load(config_filename.read_text(encoding="utf-8")) + + if ( + not current_platform.is_cuda() + and "Qwen3-30B-A3B-MXFP4A16" in eval_config["model_name"] + ): + pytest.skip( + "Skipping Qwen3-30B-A3B-MXFP4A16 on non-CUDA platforms. " + "Marlin kernels are not supported." + ) + + if ( + not current_platform.is_cuda() + and "gemma-4-E4B-it-qat-mobile-ct" in eval_config["model_name"] + ): + pytest.skip( + "Skipping gemma-4-E4B-it-qat-mobile-ct on non-CUDA platforms. " + "Its W2A16 (uint2b2) scheme has no kernel outside CUDA." + ) + + # TODO(akaratza): Enable DeepSeek-V3.2 and DeepSeek-R1 on ROCm platforms + if current_platform.is_rocm() and ( + "deepseek-ai/DeepSeek-V3.2" in eval_config["model_name"] + or "deepseek-ai/DeepSeek-R1" in eval_config["model_name"] + ): + pytest.skip( + "Skipping DeepSeek-V3.2 and DeepSeek-R1 on ROCm platforms " + "due to agent pool disk space issues and pod evictions." + ) + if current_platform.is_rocm() and ("Qwen3.5-35B-A3B-MXFP4" in config_filename.name): + from vllm.platforms.rocm import on_gfx950 + + if not on_gfx950() and "AITER-TP2" in config_filename.name: + pytest.skip( + "Skipping Qwen3.5-35B-A3B-MXFP4-AITER-TP2 on non-GFX950 platforms. " + "The quantization scheme is not supported on non-GFX950 platforms." + ) + if not QUARK_MXFP4_TORCH_COMPATIBLE: + pytest.skip( + "Skipping Qwen3.5-35B-A3B-MXFP4: amd-quark >= 0.12 is required " + "on torch >= 2.11." + ) + # Parse server arguments from config (use shlex to handle quoted strings) + server_args_str = eval_config.get("server_args", "") + server_args = shlex.split(server_args_str) if server_args_str else [] + + # Add standard server arguments + server_args.extend( + [ + "--trust-remote-code", + "--disable-uvicorn-access-log", + ] + ) + + startup_max_wait_seconds = eval_config.get( + "startup_max_wait_seconds", DEFAULT_STARTUP_MAX_WAIT_SECONDS + ) + env_dict = dict(eval_config.get("env") or {}) + env_dict["VLLM_ENGINE_READY_TIMEOUT_S"] = str(int(startup_max_wait_seconds)) + + print(f"Starting GSM8K evaluation for model: {eval_config['model_name']}") + print(f"Expected metric threshold: {eval_config['accuracy_threshold']}") + print(f"Number of questions: {eval_config['num_questions']}") + print(f"Number of few-shot examples: {eval_config['num_fewshot']}") + request_timeout_seconds = eval_config.get("request_timeout_seconds", 600) + if current_platform.is_rocm(): + request_timeout_seconds = eval_config.get( + "rocm_request_timeout_seconds", request_timeout_seconds + ) + print(f"Request timeout: {request_timeout_seconds}s") + print(f"Startup max wait: {startup_max_wait_seconds}s") + print(f"Server args: {' '.join(server_args)}") + print(f"Environment variables: {env_dict}") + + # Launch server and run evaluation + with RemoteOpenAIServer( + eval_config["model_name"], + server_args, + env_dict=env_dict, + max_wait_seconds=startup_max_wait_seconds, + ) as remote_server: + server_url = remote_server.url_for("v1") + print(f"Server started at: {server_url}") + + results = run_gsm8k_eval(eval_config, server_url) + + measured_metric = results["accuracy"] + expected_metric = eval_config["accuracy_threshold"] + tol = eval_config.get("tolerance", 0.08) + + print(f"GSM8K Results for {eval_config['model_name']}:") + print(f" Measured metric: {measured_metric:.4f}") + print(f" Expected metric: {expected_metric:.4f}") + print(f" Tolerance: {tol:.4f}") + print(f" Questions: {results['num_questions']}") + print(f" Invalid rate: {results['invalid_rate']:.3f}") + print(f" Latency: {results['latency']:.1f}s") + print(f" QPS: {results['questions_per_second']:.1f}") + + assert measured_metric >= expected_metric - tol, ( + f"GSM8K metric too low: {measured_metric:.4f} < " + f"{expected_metric:.4f} - {tol:.4f} = {expected_metric - tol:.4f}" + ) + + print(f"✅ GSM8K test passed for {eval_config['model_name']}") diff --git a/asian_ai/performance/vLLM/tests/evals/mrcr/README.md b/asian_ai/performance/vLLM/tests/evals/mrcr/README.md new file mode 100644 index 0000000000000000000000000000000000000000..e1215707f651f45450665c41171925480cc6de31 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/mrcr/README.md @@ -0,0 +1,44 @@ +# MRCR Long-Context Accuracy Evaluation + +Smoke test for long-context behavior using OpenAI's public [`openai/mrcr`](https://huggingface.co/datasets/openai/mrcr) dataset. The model sees a long chat with several near-duplicate "needles" and must reproduce a specific earlier assistant turn verbatim, prepended with a random anti-guessing string. + +**Scoring:** if the response doesn't start with `random_string_to_prepend`, score is 0; otherwise the prefix is stripped and the mean `SequenceMatcher.ratio()` against the reference answer is reported. + +## Usage + +```bash +# Pytest (spawns the server) +pytest -s -v tests/evals/mrcr/test_mrcr_correctness.py \ + --config-list-file=configs/models-small.txt + +# Standalone (server already running; model and context auto-discovered) +vllm serve Qwen/Qwen3-0.6B --reasoning-parser qwen3 --port 8000 +python tests/evals/mrcr/mrcr_eval.py --port 8000 +``` + +## Configuration + +```yaml +model_name: "Qwen/Qwen3-0.6B" +# Per-needle thresholds catch bucket-specific regressions (sliding window, +# chunked prefill, prefix cache) that an aggregate can hide. A scalar +# (e.g. `match_ratio_threshold: 0.20`) is also accepted and checked against +# the mean match ratio. +match_ratio_threshold: + 2: 0.30 + 4: 0.15 + 8: 0.10 +num_samples: 30 +needles: [2, 4, 8] +# max_prompt_tokens: 32768 # Optional; defaults to server max_model_len - max_tokens - 256 +max_tokens: 2048 +concurrency: 8 +server_args: "--max-model-len 32768 --reasoning-parser qwen3" +``` + +## Notes + +- Samples stream from three parquet shards (`{N}needle/{N}needle_0.parquet`); only the first few row groups are fetched, not the full 1.4 GB repo. +- `max_prompt_tokens` defaults to `max_model_len - max_tokens - 256`, i.e. fills whatever context the server advertises. Set `--max-model-len` on the server to control the smoke-test context length; override `--max-prompt-tokens` on the client to cap below that. +- Sample length is pre-filtered by `n_chars × 4 ≤ max_prompt_tokens`, then verified via the server's `/tokenize` endpoint under the actual chat template. +- Reasoning models: start the server with `--reasoning-parser ` (e.g. `qwen3`, `deepseek_r1`) so `` goes to `message.reasoning_content` and doesn't contaminate the scored answer. diff --git a/asian_ai/performance/vLLM/tests/evals/mrcr/__init__.py b/asian_ai/performance/vLLM/tests/evals/mrcr/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..0643de4834f45beb52db0a52ff79be718f6ec2dc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/mrcr/__init__.py @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project diff --git a/asian_ai/performance/vLLM/tests/evals/mrcr/configs/Qwen3.5-4B.yaml b/asian_ai/performance/vLLM/tests/evals/mrcr/configs/Qwen3.5-4B.yaml new file mode 100644 index 0000000000000000000000000000000000000000..851302b4691e07bcff26363e6027cf89244548bc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/mrcr/configs/Qwen3.5-4B.yaml @@ -0,0 +1,7 @@ +model_name: "Qwen/Qwen3.5-4B" +needles: [2, 4, 8] +match_ratio_threshold: + 2: 0.99 + 4: 0.84 + 8: 0.76 +server_args: "--max-model-len 128K --reasoning-parser qwen3" diff --git a/asian_ai/performance/vLLM/tests/evals/mrcr/configs/models-small.txt b/asian_ai/performance/vLLM/tests/evals/mrcr/configs/models-small.txt new file mode 100644 index 0000000000000000000000000000000000000000..5a5f8942c0969e73f3d17014dc8df96756c27a73 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/mrcr/configs/models-small.txt @@ -0,0 +1 @@ +Qwen3.5-4B.yaml diff --git a/asian_ai/performance/vLLM/tests/evals/mrcr/conftest.py b/asian_ai/performance/vLLM/tests/evals/mrcr/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..8a765cf1cd52e65b93335e958e94fa9577ae115d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/mrcr/conftest.py @@ -0,0 +1,54 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from pathlib import Path + + +def pytest_addoption(parser): + """Add custom command line options.""" + parser.addoption( + "--config-list-file", + default="configs/models-small.txt", + help="File containing list of config files to test", + ) + + +def pytest_generate_tests(metafunc): + """Generate test parameters from config files.""" + if "config_filename" in metafunc.fixturenames: + config_list_file = metafunc.config.getoption("--config-list-file") + + config_list_path = Path(config_list_file) + if not config_list_path.is_absolute(): + test_dir_path = Path(__file__).parent / config_list_file + if test_dir_path.exists(): + config_list_path = test_dir_path + else: + config_list_path = Path.cwd() / config_list_file + + print(f"Looking for config list at: {config_list_path}") + + config_files = [] + if config_list_path.exists(): + config_dir = config_list_path.parent + with open(config_list_path) as f: + for line in f: + line = line.strip() + if line and not line.startswith("#"): + config_path = config_dir / line + if config_path.exists(): + config_files.append(config_path) + print(f" ✓ Found: {config_path}") + else: + print(f" ✗ Missing: {config_path}") + else: + print(f"Config list file not found: {config_list_path}") + + if config_files: + metafunc.parametrize( + "config_filename", + config_files, + ids=[config_file.stem for config_file in config_files], + ) + else: + print("No config files found, test will be skipped") diff --git a/asian_ai/performance/vLLM/tests/evals/mrcr/mrcr_eval.py b/asian_ai/performance/vLLM/tests/evals/mrcr/mrcr_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..5b2777f00ea2ef8ed7890477494bcdf3b0862aba --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/mrcr/mrcr_eval.py @@ -0,0 +1,333 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""MRCR long-context evaluation for vLLM's OpenAI-compatible server. + +Streams samples from `openai/mrcr` on HuggingFace, sends chat completions to +the server, and scores each response with a prefix-gated SequenceMatcher ratio +against the reference answer. +""" + +import argparse +import asyncio +import json +import time +from difflib import SequenceMatcher + +import aiohttp +import numpy as np +import requests +from tqdm.asyncio import tqdm + +DATASET_REPO = "openai/mrcr" +NEEDLE_SHARDS = { + 2: "2needle/2needle_0.parquet", + 4: "4needle/4needle_0.parquet", + 8: "8needle/8needle_0.parquet", +} +# Reserve headroom for chat-template tokens on top of the messages. +PROMPT_SAFETY_BUFFER = 256 +# Pre-filter heuristic before the authoritative /tokenize check. +CHARS_PER_TOKEN = 4 +# Skip chain-of-thought on reasoning models; ignored by non-reasoning templates. +DEFAULT_EXTRA_BODY: dict = {"chat_template_kwargs": {"enable_thinking": False}} + + +def discover_server_model(base_url: str) -> tuple[str, int | None]: + """Return (model_id, max_model_len) from /v1/models.""" + resp = requests.get(f"{base_url}/v1/models", timeout=30) + resp.raise_for_status() + data = resp.json().get("data", []) + if not data: + raise RuntimeError(f"No models advertised at {base_url}/v1/models") + entry = data[0] + return entry["id"], entry.get("max_model_len") + + +def count_chat_tokens(base_url: str, model: str, messages: list[dict]) -> int: + """Return the chat-template-rendered token count via /tokenize.""" + resp = requests.post( + f"{base_url}/tokenize", + json={"model": model, "messages": messages, "add_generation_prompt": True}, + timeout=120, + ) + resp.raise_for_status() + return int(resp.json()["count"]) + + +def _load_mrcr_samples( + needles: list[int], + max_prompt_tokens: int, + num_samples: int, + seed: int, + base_url: str, + model_name: str, +) -> list[dict]: + """Stream MRCR samples balanced across needle buckets, token-verified.""" + try: + from datasets import load_dataset + except ImportError as e: + raise ImportError( + "MRCR eval requires `datasets`. Install with: uv pip install datasets" + ) from e + + max_chars = max_prompt_tokens * CHARS_PER_TOKEN + per_bucket = num_samples // len(needles) + leftover = num_samples - per_bucket * len(needles) + + samples: list[dict] = [] + for idx, n in enumerate(needles): + if n not in NEEDLE_SHARDS: + raise ValueError(f"Unsupported needle count {n}") + target = per_bucket + (1 if idx < leftover else 0) + if target == 0: + continue + + ds = load_dataset( + DATASET_REPO, + data_files=NEEDLE_SHARDS[n], + split="train", + streaming=True, + ).shuffle(seed=seed + n, buffer_size=16) + + taken = 0 + for row in ds: + if int(row.get("n_chars", 0)) > max_chars: + continue + prompt = row["prompt"] + messages = json.loads(prompt) if isinstance(prompt, str) else list(prompt) + n_tokens = count_chat_tokens(base_url, model_name, messages) + if n_tokens > max_prompt_tokens: + continue + samples.append( + { + "messages": messages, + "answer": row["answer"], + "random_string_to_prepend": row["random_string_to_prepend"], + "n_needles": int(row["n_needles"]), + "n_tokens": n_tokens, + } + ) + taken += 1 + if taken >= target: + break + + if taken < target: + print(f"Warning: only {taken}/{target} samples for n_needles={n}") + + if not samples: + raise RuntimeError("No MRCR samples fit; loosen max_prompt_tokens.") + return samples + + +def score_mrcr(response: str, answer: str, random_prefix: str) -> float: + """Prefix-gated SequenceMatcher ratio; 0 if the prefix is missing.""" + if not response.startswith(random_prefix): + return 0.0 + stripped = response[len(random_prefix) :] + return SequenceMatcher(a=answer, b=stripped, autojunk=False).ratio() + + +async def _call_chat( + session: aiohttp.ClientSession, + url: str, + model: str, + messages: list[dict], + max_tokens: int, + temperature: float, + seed: int | None, + extra_body: dict, +) -> tuple[str, int]: + data = { + "model": model, + "messages": messages, + "temperature": temperature, + "max_tokens": max_tokens, + **extra_body, + } + if seed is not None: + data["seed"] = seed + try: + async with session.post(f"{url}/v1/chat/completions", json=data) as resp: + resp.raise_for_status() + result = await resp.json() + text = result["choices"][0]["message"]["content"] or "" + return text, result.get("usage", {}).get("completion_tokens", 0) + except Exception as e: + print(f"chat request failed: {e}") + return "", 0 + + +def evaluate_mrcr( + model_name: str | None = None, + num_samples: int = 40, + needles: list[int] | None = None, + max_prompt_tokens: int | None = None, + max_tokens: int = 2048, + host: str = "http://127.0.0.1", + port: int = 8000, + temperature: float = 0.0, + seed: int | None = 42, + concurrency: int = 8, + extra_body: dict | None = None, +) -> dict: + """Run MRCR against a vLLM server; auto-discovers model and context.""" + needles = needles or [2, 4, 8] + extra_body = DEFAULT_EXTRA_BODY if extra_body is None else extra_body + base_url = f"{host}:{port}" + + discovered_model, server_max_len = discover_server_model(base_url) + if model_name is None: + model_name = discovered_model + if max_prompt_tokens is None: + if server_max_len is None: + raise RuntimeError( + "Server did not advertise max_model_len; pass --max-prompt-tokens." + ) + max_prompt_tokens = max(512, server_max_len - max_tokens - PROMPT_SAFETY_BUFFER) + print( + f"Model: {model_name} | max_prompt_tokens={max_prompt_tokens} " + f"(server max_model_len={server_max_len}, max_tokens={max_tokens})" + ) + + samples = _load_mrcr_samples( + needles=needles, + max_prompt_tokens=max_prompt_tokens, + num_samples=num_samples, + seed=seed or 0, + base_url=base_url, + model_name=model_name, + ) + tok_counts = [s["n_tokens"] for s in samples] + print( + f"Loaded {len(samples)} samples (needles={needles}, " + f"tokens={min(tok_counts)}-{max(tok_counts)})" + ) + + async def run(): + sem = asyncio.Semaphore(concurrency) + responses = [""] * len(samples) + out_tokens = [0] * len(samples) + + async def one(session, i): + async with sem: + text, toks = await _call_chat( + session=session, + url=base_url, + model=model_name, + messages=samples[i]["messages"], + max_tokens=max_tokens, + temperature=temperature, + seed=seed, + extra_body=extra_body, + ) + responses[i] = text + out_tokens[i] = toks + + timeout = aiohttp.ClientTimeout(total=1800) + async with aiohttp.ClientSession(timeout=timeout) as session: + await tqdm.gather( + *[one(session, i) for i in range(len(samples))], desc="MRCR" + ) + return responses, out_tokens + + tic = time.perf_counter() + responses, out_tokens = asyncio.run(run()) + latency = time.perf_counter() - tic + + scores = np.array( + [ + score_mrcr(r, s["answer"], s["random_string_to_prepend"]) + for r, s in zip(responses, samples) + ] + ) + prefix_hits = np.array( + [ + r.startswith(s["random_string_to_prepend"]) + for r, s in zip(responses, samples) + ] + ) + per_needle = { + f"match_ratio_n{n}": float( + scores[np.array([s["n_needles"] == n for s in samples])].mean() + ) + for n in needles + if any(s["n_needles"] == n for s in samples) + } + + total_out = int(sum(out_tokens)) + return { + "model": model_name, + "match_ratio": float(scores.mean()), + "prefix_hit_rate": float(prefix_hits.mean()), + "per_needle": per_needle, + "num_samples": len(samples), + "latency": latency, + "total_output_tokens": total_out, + "tokens_per_second": total_out / latency if latency > 0 else 0.0, + "max_tokens": max_tokens, + "needles": needles, + "max_prompt_tokens": max_prompt_tokens, + } + + +def main() -> None: + p = argparse.ArgumentParser(description="MRCR evaluation for vLLM serve") + p.add_argument("--model", default=None, help="Default: discovered from /v1/models") + p.add_argument("--num-samples", type=int, default=40) + p.add_argument( + "--needles", type=int, nargs="+", default=[2, 4, 8], choices=[2, 4, 8] + ) + p.add_argument( + "--max-prompt-tokens", + type=int, + default=None, + help="Default: server max_model_len - max_tokens - buffer", + ) + p.add_argument("--max-tokens", type=int, default=2048) + p.add_argument("--host", default="http://127.0.0.1") + p.add_argument("--port", type=int, default=8000) + p.add_argument("--temperature", type=float, default=0.0) + p.add_argument("--seed", type=int, default=42) + p.add_argument("--concurrency", type=int, default=8) + p.add_argument( + "--extra-body", + default=None, + help="JSON merged into each request. " + "Pass '{}' to disable the default enable_thinking=false.", + ) + p.add_argument("--save-results", default=None) + args = p.parse_args() + + extra_body = json.loads(args.extra_body) if args.extra_body else None + + result = evaluate_mrcr( + model_name=args.model, + num_samples=args.num_samples, + needles=args.needles, + max_prompt_tokens=args.max_prompt_tokens, + max_tokens=args.max_tokens, + host=args.host, + port=args.port, + temperature=args.temperature, + seed=args.seed, + concurrency=args.concurrency, + extra_body=extra_body, + ) + + print("\nResults:") + print(f" match_ratio: {result['match_ratio']:.4f}") + print(f" prefix_hit_rate: {result['prefix_hit_rate']:.4f}") + for k, v in result["per_needle"].items(): + print(f" {k}: {v:.4f}") + print(f" samples: {result['num_samples']}") + print(f" latency: {result['latency']:.1f}s") + print(f" output tok/s: {result['tokens_per_second']:.1f}") + + if args.save_results: + with open(args.save_results, "w") as f: + json.dump(result, f, indent=2) + + +if __name__ == "__main__": + main() diff --git a/asian_ai/performance/vLLM/tests/evals/mrcr/test_mrcr_correctness.py b/asian_ai/performance/vLLM/tests/evals/mrcr/test_mrcr_correctness.py new file mode 100644 index 0000000000000000000000000000000000000000..87ac5a7fcb0109cf4ebee3ad129bbf1d9223dc9a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/evals/mrcr/test_mrcr_correctness.py @@ -0,0 +1,83 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +MRCR long-context accuracy test. + +Usage: + pytest -s -v tests/evals/mrcr/test_mrcr_correctness.py \ + --config-list-file=configs/models-small.txt +""" + +import shlex + +import yaml + +from tests.utils import RemoteOpenAIServer + +from .mrcr_eval import evaluate_mrcr + + +def _split_host_port(url: str, default_port: int = 8000) -> tuple[str, int]: + if "://" in url: + url = url.split("://", 1)[1] + host_port = url.split("/", 1)[0] + if ":" in host_port: + host, p = host_port.split(":", 1) + return f"http://{host}", int(p) + return f"http://{host_port}", default_port + + +def test_mrcr_correctness(config_filename): + cfg = yaml.safe_load(config_filename.read_text(encoding="utf-8")) + + server_args = shlex.split(cfg.get("server_args", "")) + server_args += ["--trust-remote-code", "--disable-uvicorn-access-log"] + + print( + f"MRCR eval for {cfg['model_name']} (threshold {cfg['match_ratio_threshold']})" + ) + + with RemoteOpenAIServer( + cfg["model_name"], + server_args, + env_dict=cfg.get("env"), + max_wait_seconds=cfg.get("startup_max_wait_seconds", 600), + ) as server: + host, port = _split_host_port(server.url_for("v1")) + results = evaluate_mrcr( + model_name=cfg.get("model_name"), + num_samples=cfg.get("num_samples", 40), + needles=cfg.get("needles", [2, 4, 8]), + max_prompt_tokens=cfg.get("max_prompt_tokens"), + max_tokens=cfg.get("max_tokens", 2048), + host=host, + port=port, + concurrency=cfg.get("concurrency", 8), + extra_body=cfg.get("extra_body"), + ) + + threshold = cfg["match_ratio_threshold"] + tol = cfg.get("tolerance", 0.05) + + print(f" match_ratio: {results['match_ratio']:.4f}") + print(f" prefix_hit_rate: {results['prefix_hit_rate']:.4f}") + for k, v in results["per_needle"].items(): + print(f" {k}: {v:.4f}") + + failures: list[str] = [] + if isinstance(threshold, dict): + for n, expected in threshold.items(): + key = f"match_ratio_n{int(n)}" + measured = results["per_needle"].get(key) + if measured is None: + failures.append(f"{key}: no samples collected") + elif measured < expected - tol: + failures.append(f"{key}: {measured:.4f} < {expected:.4f} - {tol:.4f}") + else: + measured = results["match_ratio"] + if measured < threshold - tol: + failures.append( + f"match_ratio: {measured:.4f} < {threshold:.4f} - {tol:.4f}" + ) + + assert not failures, "MRCR thresholds failed: " + "; ".join(failures) diff --git a/asian_ai/performance/vLLM/tests/fusion/__init__.py b/asian_ai/performance/vLLM/tests/fusion/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..0643de4834f45beb52db0a52ff79be718f6ec2dc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/fusion/__init__.py @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project diff --git a/asian_ai/performance/vLLM/tests/fusion/test_quant_activation_contract.py b/asian_ai/performance/vLLM/tests/fusion/test_quant_activation_contract.py new file mode 100644 index 0000000000000000000000000000000000000000..6dc62694bf3eb87da011d17b603bfab9a0a0bb2d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/fusion/test_quant_activation_contract.py @@ -0,0 +1,131 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Contract tests for the QuantizedActivation linear-kernel integration.""" + +import pytest +import torch + +from vllm.model_executor.kernels.linear import ( + _POSSIBLE_FP8_BLOCK_KERNELS, + _POSSIBLE_FP8_KERNELS, + _POSSIBLE_INT8_KERNELS, + _POSSIBLE_NVFP4_KERNELS, +) +from vllm.model_executor.kernels.linear.nvfp4.base import ( + NvFp4LinearKernel, + NvFp4LinearLayerConfig, +) +from vllm.model_executor.kernels.linear.nvfp4.flashinfer import ( + FlashInferCutlassNvFp4LinearKernel, + FlashInferTrtllmNvFp4LinearKernel, +) +from vllm.model_executor.kernels.linear.scaled_mm.cutlass import ( + CutlassFP8ScaledMMLinearKernel, +) +from vllm.model_executor.kernels.linear.scaled_mm.flashinfer import ( + FlashInferFP8ScaledMMLinearKernel, +) +from vllm.model_executor.kernels.linear.scaled_mm.ScaledMMLinearKernel import ( + FP8ScaledMMLinearLayerConfig, + Int8ScaledMMLinearKernel, + Int8ScaledMMLinearLayerConfig, +) +from vllm.model_executor.layers.fusion.quant_activation import ( + QuantizedActivation, + as_quantized_activation, + expose_input_quant_key, +) +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + kFp8StaticTensorSym, + kNvfp4Dynamic, +) +from vllm.platforms import current_platform + +# The only backends that consume a pre-quantized activation. +SUPPORTING = { + CutlassFP8ScaledMMLinearKernel, + FlashInferFP8ScaledMMLinearKernel, + FlashInferCutlassNvFp4LinearKernel, +} + + +def _all_kernel_classes() -> list[type]: + seen: dict[type, None] = {} + for registry in ( + _POSSIBLE_FP8_KERNELS, + _POSSIBLE_FP8_BLOCK_KERNELS, + _POSSIBLE_INT8_KERNELS, + _POSSIBLE_NVFP4_KERNELS, + ): + for kernels in registry.values(): + for cls in kernels: + seen.setdefault(cls, None) + return list(seen) + + +def _probe(cls: type): + """A bare kernel instance with a plausible config, so input_quant_key() + can be queried without the hardware-gated constructor.""" + obj = cls.__new__(cls) # type: ignore[call-overload] + if issubclass(cls, NvFp4LinearKernel): + obj.config = NvFp4LinearLayerConfig() + elif issubclass(cls, Int8ScaledMMLinearKernel): + obj.config = Int8ScaledMMLinearLayerConfig( + is_static_input_scheme=True, is_channelwise=False, input_symmetric=True + ) + else: + obj.config = FP8ScaledMMLinearLayerConfig( + weight_quant_key=kFp8StaticTensorSym, + activation_quant_key=kFp8StaticTensorSym, + weight_shape=(16, 16), + input_dtype=torch.bfloat16, + out_dtype=torch.bfloat16, + ) + return obj + + +def _resolved_apply_weights(cls: type): + for base in cls.__mro__: + if "apply_weights" in base.__dict__: + return base.__dict__["apply_weights"] + raise AssertionError(f"{cls.__name__} has no apply_weights in its MRO") + + +def test_only_known_backends_support_prequantized_input(): + declarers = {c for c in _all_kernel_classes() if _probe(c).input_quant_key()} + assert declarers == SUPPORTING + + +def test_supporting_backend_declares_consume_via_helper(): + for cls in SUPPORTING: + fn = _resolved_apply_weights(cls) + assert "as_quantized_activation" in fn.__code__.co_names, cls.__name__ + + +def test_bridge_marks_supporting_and_skips_others(): + supported = _probe(FlashInferCutlassNvFp4LinearKernel) + layer = torch.nn.Module() + expose_input_quant_key(layer, supported) + assert layer.input_quant_key == kNvfp4Dynamic + + unsupported = _probe(FlashInferTrtllmNvFp4LinearKernel) + assert unsupported.input_quant_key() is None + layer = torch.nn.Module() + expose_input_quant_key(layer, unsupported) + assert not hasattr(layer, "input_quant_key") + + +def test_as_quantized_activation_validates_key(): + qa = QuantizedActivation( + data=torch.zeros(2, 4, dtype=current_platform.fp8_dtype()), + scale=torch.tensor(1.0), + orig_dtype=torch.bfloat16, + orig_shape=torch.Size([2, 4]), + quant_key=kFp8StaticTensorSym, + ) + with pytest.raises(AssertionError): + as_quantized_activation(qa, kNvfp4Dynamic) + with pytest.raises(AssertionError): + as_quantized_activation(qa, None) + assert as_quantized_activation(torch.zeros(2, 4), kFp8StaticTensorSym) is None + assert as_quantized_activation(qa, kFp8StaticTensorSym) is qa diff --git a/asian_ai/performance/vLLM/tests/ir/ir_test_utils.py b/asian_ai/performance/vLLM/tests/ir/ir_test_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..db45ec6aff6efbac17bbf608e26e5b491947cc00 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/ir/ir_test_utils.py @@ -0,0 +1,46 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Shared test utilities for vLLM IR op correctness tests. +""" + +import torch + +from vllm.ir.op import IrOp + +NUM_TOKENS = [1, 8, 17, 32, 512, 2048] +COMMON_HIDDEN_SIZES = [ + 2048, # Llama 3.2 1B, Qwen 3 MoE 30B-A3B, Gemma 3n + 4096, # Llama 3 8B, Qwen 3 8B + 5120, # Llama 4 Scout 17B-16E + 7168, # DeepSeek V3 + 8192, # Llama 3 70B +] + + +def clone_args(args: tuple) -> tuple: + return tuple(a.clone() if isinstance(a, torch.Tensor) else a for a in args) + + +def supported_providers(op: IrOp) -> list[str]: + return [ + name for name, impl in op.impls.items() if name != "native" and impl.supported + ] + + +def assert_close(op: IrOp, actual, expected): + if isinstance(actual, torch.Tensor): + tol = op.get_tolerance(actual.dtype) + try: + torch.testing.assert_close(actual, expected, **tol) + except AssertionError as e: + raise AssertionError( + f"{e}\n\nTo adjust tolerance, use:\n" + f" ir.ops.{op.name}.override_tolerance(" + f"{actual.dtype}, atol=..., rtol=...)" + ) from None + elif isinstance(actual, (tuple, list)): + for a, ex in zip(actual, expected): + assert_close(op, a, ex) + else: + assert actual == expected diff --git a/asian_ai/performance/vLLM/tests/ir/test_inplace_op.py b/asian_ai/performance/vLLM/tests/ir/test_inplace_op.py new file mode 100644 index 0000000000000000000000000000000000000000..e989d716a9be3ea51dba5f7e1c265599acecbc4c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/ir/test_inplace_op.py @@ -0,0 +1,91 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import torch +from torch import Tensor +from torch.fx.experimental.proxy_tensor import make_fx + +import vllm.ir.op +from vllm.ir.op import IrOp, IrOpInplaceOverload + + +@vllm.ir.register_op(allow_inplace=True) +def _custom_mm2(x: Tensor, w: Tensor) -> Tensor: + return x @ w + + +@_custom_mm2.register_impl("regular") +def _custom_mm2_regular(x: Tensor, w: Tensor) -> Tensor: + return x @ w + 1 + + +@_custom_mm2.register_impl("inplace", inplace=True) +def _custom_mm2_inplace(x: Tensor, w: Tensor) -> Tensor: + x.copy_(x @ w + 2) + return x + + +class TestInplaceOp: + def test_registration(self): + # Test that the inplace op is registered correctly. + assert "_custom_mm2" in IrOp.registry + assert IrOp.registry["_custom_mm2"] is _custom_mm2 + assert _custom_mm2.torch_op is torch.ops.vllm_ir._custom_mm2.default + assert isinstance(_custom_mm2.maybe_inplace, IrOpInplaceOverload) + assert ( + _custom_mm2.maybe_inplace.torch_op + is torch.ops.vllm_ir._custom_mm2.maybe_inplace + ) + + def test_inplace_dispatching(self): + # check that the correct implementation is dispatched based on priority, + # and inplace semantics hold + w = torch.randn(3, 3) + x = torch.randn(2, 3) + x1 = x.clone() + + with _custom_mm2.set_priority(["regular"]): + result_regular = _custom_mm2.maybe_inplace(x, w) + + # check that the regular op does not modify x + torch.testing.assert_close(x, x1, atol=0, rtol=0) + + with _custom_mm2.set_priority(["inplace"]): + result_inplace: Tensor = _custom_mm2.maybe_inplace(x, w) + + # check that the inplace op returns x directly + assert result_inplace.data_ptr() == x.data_ptr() + + torch.testing.assert_close(result_inplace, x1 @ w + 2) + torch.testing.assert_close(result_regular, x1 @ w + 1) + + def test_default_dispatching(self): + # check that the correct implementation is dispatched, + # and ops do not modify inputs when using the default overload + w = torch.randn(3, 3) + x = torch.randn(2, 3) + x1 = x.clone() + + with _custom_mm2.set_priority(["regular"]): + result_regular = _custom_mm2(x, w) + + with _custom_mm2.set_priority(["inplace"]): + result_inplace = _custom_mm2(x, w) + + # check that x was not modified by either impl + torch.testing.assert_close(x, x1, atol=0, rtol=0) + + torch.testing.assert_close(result_inplace, x1 @ w + 2) + torch.testing.assert_close(result_regular, x1 @ w + 1) + + def test_trace(self): + # Test that the inplace op can be used in a graph. + def func(x: Tensor, y: Tensor) -> Tensor: + return _custom_mm2.maybe_inplace(x, y) + + x = torch.randn(2, 3) + y = torch.randn(3, 4) + graph = make_fx(func)(x, y) + assert any( + node.target == torch.ops.vllm_ir._custom_mm2.maybe_inplace + for node in graph.graph.nodes + ) diff --git a/asian_ai/performance/vLLM/tests/ir/test_op.py b/asian_ai/performance/vLLM/tests/ir/test_op.py new file mode 100644 index 0000000000000000000000000000000000000000..61189e5af070f26f54e51fe3d81098922688d2a1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/ir/test_op.py @@ -0,0 +1,766 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import importlib.util +import logging +from pathlib import Path +from typing import Any + +import pytest +import torch +from torch import fx +from torch.fx.experimental.proxy_tensor import make_fx + +import vllm.ir.op +from vllm.ir.op import RESERVED_PROVIDERS, IrOp, IrOpImpl + + +class CustomError(Exception): + pass + + +@pytest.fixture +def custom_add_op(fake_vllm_ir): + """Register ``_custom_add`` plus impl_a, impl_b, impl_even for this test.""" + + @vllm.ir.register_op(allow_inplace=True) + def _custom_add(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return x + y + + @_custom_add.register_impl("impl_a") + def impl_a(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return x + y + 10 + + @_custom_add.register_impl("impl_b", inplace=True) + def impl_b(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + """Computes x+y+20""" + x.add_(y) + x.add_(20) + return x + + @_custom_add.register_impl( + "impl_even", supports_args=lambda x, y: x.size(1) % 2 == 0 + ) + def impl_even(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return x + y + 50 + + return _custom_add + + +def test_registration_overloads(fake_vllm_ir): + assert all( + n not in IrOp.registry for n in ["_custom_sub", "_custom_mul", "_custom_div"] + ) + + # Calling with decorator + @vllm.ir.register_op() + def _custom_sub(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return x - y + + assert _custom_sub.name == "_custom_sub" + assert _custom_sub is IrOp.registry["_custom_sub"] + + # Custom name + @vllm.ir.register_op(name="_custom_mul") + def custom_mul(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return x * y + + assert custom_mul.name == "_custom_mul" + assert custom_mul is IrOp.registry["_custom_mul"] + + # Direct construction does not register directly + def _custom_div(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return x / y + + custom_div = IrOp("_custom_div", _custom_div) + assert custom_div.name == "_custom_div" + assert "_custom_div" not in IrOp.registry + + # Duplicate op registration not allowed + with pytest.raises(AssertionError): + + @vllm.ir.register_op + def _custom_mul(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return x * y - 100 + + +def test_no_kw_only_args(fake_vllm_ir): + # kw-only args not supported + with pytest.raises(ValueError, match="keyword-only arguments"): + + @vllm.ir.register_op + def _custom_kwarg_op( + x: torch.Tensor, y: torch.Tensor, *, kwarg: int = 0 + ) -> torch.Tensor: + return x + y + kwarg + + assert "_custom_kwarg_op" not in IrOp.registry + + +class TestIrOpCustomAdd: + # Registration invariants + def test_decorated_object(self, custom_add_op): + """Make sure that referring directly to an op is correct""" + _custom_add = custom_add_op + assert isinstance(_custom_add, IrOp) + assert "_custom_add" in IrOp.registry + assert _custom_add is IrOp.registry["_custom_add"] + + def test_torch_op_is_registered(self, custom_add_op): + _custom_add = custom_add_op + torch_ops = getattr(torch.ops, vllm.ir.op.vllm_ir_torch_lib.ns) + assert hasattr(torch_ops, "_custom_add") + assert callable(torch_ops._custom_add.default) + assert _custom_add.torch_op is torch_ops._custom_add.default + + # Semantic correctness + def test_semantics_match_native(self, custom_add_op): + _custom_add = custom_add_op + x = torch.randn(4, 5) + y = torch.randn(4, 5) + + # Calls native by default + out = _custom_add(x, y) + ref = x + y + + torch.testing.assert_close(out, ref) + + # ------------------------- + # Implementation registration + # ------------------------- + + def test_register_impl_is_non_intrusive(self, custom_add_op): + _custom_add = custom_add_op + + @_custom_add.register_impl("dummy_provider") + def dummy_impl(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return x + y + 123 + + assert "dummy_provider" in _custom_add.impls + assert isinstance(_custom_add.impls["dummy_provider"], IrOpImpl) + + x = torch.ones(2, 2) + y = torch.ones(2, 2) + + # Native semantics must still hold + torch.testing.assert_close(_custom_add(x, y), x + y) + + def test_schema_contains_tensor_signature(self, custom_add_op): + _custom_add = custom_add_op + schema = _custom_add._schema_str + + assert "Tensor" in schema + assert "-> Tensor" in schema + + # ------------------------- + # FX visibility + # ------------------------- + + @pytest.mark.parametrize("enable_torch_wrap", [True, False]) + @pytest.mark.parametrize("symbolic_trace", [True, False]) + @pytest.mark.parametrize("overload", ["default", "maybe_inplace"]) + def test_trace_sees_single_custom_op( + self, + custom_add_op, + symbolic_trace: bool, + enable_torch_wrap: bool, + overload: str, + ): + _custom_add = custom_add_op + op_fn = _custom_add if overload == "default" else _custom_add.maybe_inplace + torch_op = ( + _custom_add.torch_op + if overload == "default" + else _custom_add.maybe_inplace.torch_op + ) + + def fn(x, y): + return op_fn(x, y) + + def find_fn(target: Any, gm: fx.GraphModule): + return gm.graph.find_nodes(op="call_function", target=target) + + with pytest.raises(CustomError), vllm.ir.enable_torch_wrap(enable_torch_wrap): + if symbolic_trace: + gm = torch.fx.symbolic_trace(fn) + else: + gm = make_fx(fn)(torch.randn(2, 2), torch.randn(2, 2)) + + x1, y1 = torch.rand(5, 4), torch.rand(5, 4) + out_fx = gm(x1, y1) + out_eager = fn(x1, y1) + + # raise error to check enable_torch_wrap context restored correctly + raise CustomError + + # check behavior matches eager in all cases + torch.testing.assert_close(out_fx, out_eager) + + # check that IR nodes only appear if enable_torch_wrap=True + ir_nodes = find_fn(torch_op, gm) + if enable_torch_wrap: + assert len(ir_nodes) == 1, gm.code + else: + assert len(ir_nodes) == 0, gm.code + + # with torch wrapping enabled (default), IR nodes appear + if symbolic_trace: + gm = torch.fx.symbolic_trace(fn) + else: + gm = make_fx(fn)(torch.randn(2, 2), torch.randn(2, 2)) + + ir_nodes = find_fn(torch_op, gm) + assert len(ir_nodes) == 1, gm.code + + +class TestIrOpImplDispatch: + def test_register_impl(self, custom_add_op): + _custom_add = custom_add_op + assert "impl_a" in _custom_add.impls + impl = _custom_add.impls["impl_a"] + + assert impl is _custom_add.impls["impl_a"] + assert impl.op is _custom_add + assert impl.provider == "impl_a" + assert callable(impl.impl_fn) + + # Test duplicate registration rejected + with pytest.raises(AssertionError): + + @_custom_add.register_impl("impl_a") + def impl_a_dup(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return x + y + 30 + + # Check the original impl is still intact + assert _custom_add.impls["impl_a"] is impl + + # Check support all args + assert _custom_add.impls["impl_a"].supports_all_args + assert _custom_add.impls["impl_b"].supports_all_args + assert not _custom_add.impls["impl_even"].supports_all_args + + def test_reserved_provider_rejected(self, custom_add_op): + _custom_add = custom_add_op + for provider in RESERVED_PROVIDERS: + with pytest.raises(AssertionError): + + @_custom_add.register_impl(provider) + def bad_impl(x, y): + return x + y + + def test_set_priority_scoped(self, custom_add_op): + _custom_add = custom_add_op + assert _custom_add.get_priority() == [] + + with _custom_add.set_priority(["impl_even", "impl_b"]): + assert _custom_add.get_priority() == ["impl_even", "impl_b"] + + # Check nesting + with _custom_add.set_priority(["impl_b"]): + assert _custom_add.get_priority() == ["impl_b"] + + # Restored + assert _custom_add.get_priority() == ["impl_even", "impl_b"] + + # Check that exception restores priority + with pytest.raises(CustomError), _custom_add.set_priority(["impl_a"]): + assert _custom_add.get_priority() == ["impl_a"] + raise CustomError + + # Restored again + assert _custom_add.get_priority() == ["impl_even", "impl_b"] + + # Restored to empty + assert _custom_add.get_priority() == [] + + @pytest.mark.parametrize( + "default,override", + [ + (["impl_even", "impl_b"], ["impl_a"]), + (["impl_a"], ["impl_even", "impl_b"]), + ], + ) + def test_set_default_priority( + self, custom_add_op, default: list[str], override: list[str] + ): + _custom_add = custom_add_op + assert _custom_add.get_priority() == [] + + _custom_add.set_default(default) + assert _custom_add.get_priority() == default + + # Priority doesn't change after exiting the set_priority context. + with _custom_add.set_priority(override): + assert _custom_add.get_priority() == override + assert _custom_add.get_priority() == default + + # Should override the previous default. + _custom_add.set_default(override) + assert _custom_add.get_priority() == override + + @pytest.mark.parametrize("overload", ["default", "maybe_inplace"]) + def test_dispatch_priority_order(self, custom_add_op, overload: str): + _custom_add = custom_add_op + op_fn = _custom_add if overload == "default" else _custom_add.maybe_inplace + torch_op = ( + _custom_add.torch_op + if overload == "default" + else _custom_add.maybe_inplace.torch_op + ) + + x = torch.tensor(1, dtype=torch.int32) + y = torch.tensor(2, dtype=torch.int32) + + with _custom_add.set_priority(["impl_b", "impl_a"]): + assert _custom_add.dispatch(x, y) is _custom_add.impls["impl_b"] + out1 = op_fn(x.clone(), y) + out2 = torch_op(x.clone(), y) + + with _custom_add.set_priority(["impl_a"]): + assert _custom_add.dispatch(x, y) is _custom_add.impls["impl_a"] + out3 = op_fn(x.clone(), y) + out4 = torch_op(x.clone(), y) + + # impl_b + assert out1.item() == 1 + 2 + 20 + assert out2.item() == 1 + 2 + 20 + # impl_a + assert out3.item() == 1 + 2 + 10 + assert out4.item() == 1 + 2 + 10 + + def test_unsupported_impl_filtered(self, custom_add_op): + _custom_add = custom_add_op + + @_custom_add.register_impl("impl_unsupported", supported=False) + def impl_unsupported(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return x + y + 999 + + x = torch.tensor(1, dtype=torch.int32) + y = torch.tensor(2, dtype=torch.int32) + + with _custom_add.set_priority(["impl_unsupported", "impl_a"]): + assert _custom_add.get_priority() == ["impl_a"] + out = _custom_add(x, y) + + # impl_unsupported skipped → impl_a + assert out.item() == 1 + 2 + 10 + + def test_supports_args_runtime_dispatch_and_warning( + self, custom_add_op, caplog_vllm: pytest.LogCaptureFixture + ): + _custom_add = custom_add_op + x1 = torch.ones((2, 2), dtype=torch.int32) + y1 = torch.full((2, 2), 2, dtype=torch.int32) + + x2 = torch.ones((2, 3), dtype=torch.int32) + y2 = torch.full((2, 3), 2, dtype=torch.int32) + + with ( + caplog_vllm.at_level(logging.WARNING), + _custom_add.set_priority(["impl_even"]), + ): + # Test the warning about native fallback is logged (before even dispatching) + assert len(caplog_vllm.records) == 1 + message = caplog_vllm.records[0].message + assert "_custom_add" in message + assert "fallback to native" in message + assert "priority" in message + + # Check dispatching + assert _custom_add.get_priority() == ["impl_even", "native"] + assert _custom_add.dispatch(x1, y1) is _custom_add.impls["impl_even"] + assert _custom_add.dispatch(x2, y2) is _custom_add.impls["native"] + + out1 = _custom_add(x1, y1) # size(1) == 2 → impl_even + out2 = _custom_add(x2, y2) # size(1) == 3 → native fallback + + # no other warnings + assert len(caplog_vllm.records) == 1 + assert torch.all(out1 == 1 + 2 + 50) + assert torch.all(out2 == 1 + 2) + + def test_default_priority( + self, + custom_add_op, + caplog_vllm: pytest.LogCaptureFixture, + disable_log_dedup, + ): + _custom_add = custom_add_op + # Make sure logs are not deduplicated to properly test the warning + x = torch.tensor([3], dtype=torch.int32) + y = torch.tensor([4], dtype=torch.int32) + + # No priority set → falls back to native + assert _custom_add.get_priority() == [] + with caplog_vllm.at_level(logging.WARNING): + # Native by default + assert _custom_add.dispatch(x, y) is _custom_add.impls["native"] + out = _custom_add(x, y) + + # Check dispatching to native by default + assert out.item() == 3 + 4 + + # Check warning + assert len(caplog_vllm.records) == 2 + message = caplog_vllm.records[0].message.lower() + assert "_custom_add" in message + assert "priority not set" in message + + +@pytest.mark.parametrize("default", [True, False]) +def test_set_default_torch_wrap(default: bool): + """set_default_torch_wrap permanently flips the global flag.""" + original = vllm.ir.op._ENABLE_TORCH_WRAP + try: + vllm.ir.set_default_torch_wrap(default) + assert vllm.ir.op._ENABLE_TORCH_WRAP is default + + # Flag doesn't change after exiting the enable_torch_wrap context. + with vllm.ir.enable_torch_wrap(not default): + assert vllm.ir.op._ENABLE_TORCH_WRAP is (not default) + assert vllm.ir.op._ENABLE_TORCH_WRAP is default + + # Should override the previous default. + vllm.ir.set_default_torch_wrap(not default) + assert vllm.ir.op._ENABLE_TORCH_WRAP is (not default) + finally: + vllm.ir.op._ENABLE_TORCH_WRAP = original + + +@pytest.fixture +def custom_mm_op(fake_vllm_ir): + """Fixture that registers ``_custom_mm`` (isolated by ``fake_vllm_ir``).""" + + @vllm.ir.register_op + def _custom_mm( + x: torch.Tensor, y: torch.Tensor, bias: torch.Tensor | None = None + ) -> torch.Tensor: + tmp = x @ y + return tmp if bias is None else tmp + bias + + return _custom_mm + + +def test_default_args(custom_mm_op): + _custom_mm = custom_mm_op + + # Test that default args are properly applied when dispatching and calling + @_custom_mm.register_impl("impl_mm", supports_args=lambda x, y, bias=None: True) + def impl_mm( + x: torch.Tensor, y: torch.Tensor, bias: torch.Tensor | None = None + ) -> torch.Tensor: + tmp = x @ y + return tmp + 50 if bias is None else tmp + bias + 100 + + x1 = torch.tensor([1, 2], dtype=torch.int32) + x2 = torch.tensor([3, 4], dtype=torch.int32) + + # Test that supports_args receives the defaulted args + assert impl_mm.supports_args(x1, x2) + with _custom_mm.set_priority(["impl_mm", "native"]): + assert _custom_mm.dispatch(x1, x2) is impl_mm + + +def test_bad_impl_registrations(custom_mm_op): + _custom_mm = custom_mm_op + # Check bad schema + with pytest.raises(ValueError, match="does not match native schema"): + + @_custom_mm.register_impl("impl_mm_bad_schema") + def impl_mm_bad_schema(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return x @ y - 1 + + with pytest.raises(ValueError, match="does not match native schema"): + + @_custom_mm.register_impl("impl_mm_bad_schema_2") + def impl_mm_bad_schema_2( + x: torch.Tensor, y: torch.Tensor, b: torch.Tensor | None = None + ) -> torch.Tensor: + return x @ y + b - 2 + + with pytest.raises(ValueError, match="does not match native schema"): + + @_custom_mm.register_impl("impl_mm_bad_schema_3") + def impl_mm_bad_schema_3( + x: torch.Tensor, y: torch.Tensor, bias: torch.Tensor + ) -> torch.Tensor: + return x @ y + bias - 5 + + # check supports_args with incorrect params + with pytest.raises(ValueError, match="supports_args must be a callable"): + + @_custom_mm.register_impl("impl_mm_bad_supports_args", supports_args=True) + def impl_mm_bad_supports_args( + x: torch.Tensor, y: torch.Tensor, bias: torch.Tensor | None = None + ) -> torch.Tensor: + return x @ y + 10 + + with pytest.raises(ValueError, match="number of parameters"): + + @_custom_mm.register_impl( + "impl_mm_bad_supports_args_2", supports_args=lambda x, y: True + ) + def impl_mm_bad_supports_args( + x: torch.Tensor, y: torch.Tensor, bias: torch.Tensor | None = None + ) -> torch.Tensor: + return x @ y + 10 + + with pytest.raises(ValueError, match="keyword-only parameters"): + + @_custom_mm.register_impl( + "impl_mm_bad_supports_args_3", supports_args=lambda x, y, *, b: True + ) + def impl_mm_bad_supports_args_2( + x: torch.Tensor, y: torch.Tensor, bias: torch.Tensor | None = None + ) -> torch.Tensor: + return x @ y + 20 + + with pytest.raises(ValueError, match="does not match native parameter"): + + @_custom_mm.register_impl( + "impl_mm_bad_supports_args_4", supports_args=lambda x, y, b: True + ) + def impl_mm_bad_supports_args_4( + x: torch.Tensor, y: torch.Tensor, bias: torch.Tensor | None = None + ) -> torch.Tensor: + return x @ y + 30 + + with pytest.raises(ValueError, match="does not match native default"): + + @_custom_mm.register_impl( + "impl_mm_bad_supports_args_5", supports_args=lambda x, y, bias=1: True + ) + def impl_mm_bad_supports_args_5( + x: torch.Tensor, y: torch.Tensor, bias: torch.Tensor | None = None + ) -> torch.Tensor: + return x @ y + 40 + + # With fixture, each test gets a fresh op with only "native" impl + assert set(_custom_mm.impls.keys()) == {"native"} + + +IMPL_OOT_SRC = """ +import torch + +@_custom_mm.register_impl("impl_mm_oot") +def impl_mm_oot( + x: torch.Tensor, y: torch.Tensor, bias: torch.Tensor | None = None +) -> torch.Tensor: + return x @ y - 99 +""" + + +def load_custom_mm_module(file_path: Path, custom_mm_op): + spec = importlib.util.spec_from_file_location("_custom_mm_oot", file_path) + assert spec is not None + module = importlib.util.module_from_spec(spec) + + # Inject the variable into the module's global namespace + # This allows the @_custom_mm.register_impl decorator to work + module._custom_mm = custom_mm_op # type: ignore[attr-defined] + + # Execute the file; this triggers the decorator + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +def test_uuid_and_oot(custom_mm_op, tmp_path: Path): + _custom_mm = custom_mm_op + file_path = tmp_path / "_custom_mm_oot.py" + file_path.write_text(IMPL_OOT_SRC) + + assert "impl_mm_oot" not in _custom_mm.impls + _ = load_custom_mm_module(file_path, _custom_mm) + assert "impl_mm_oot" in _custom_mm.impls + + uuid = _custom_mm.impls["impl_mm_oot"].uuid() + del _custom_mm.impls["impl_mm_oot"] + + # Replace file source + file_path.write_text(IMPL_OOT_SRC + " # added file source") + assert "impl_mm_oot" not in _custom_mm.impls + _ = load_custom_mm_module(file_path, _custom_mm) + assert "impl_mm_oot" in _custom_mm.impls + + uuid1 = _custom_mm.impls["impl_mm_oot"].uuid() + assert uuid1 != uuid + del _custom_mm.impls["impl_mm_oot"] + + # Back to original + file_path.write_text(IMPL_OOT_SRC) + assert "impl_mm_oot" not in _custom_mm.impls + _ = load_custom_mm_module(file_path, _custom_mm) + assert "impl_mm_oot" in _custom_mm.impls + + uuid2 = _custom_mm.impls["impl_mm_oot"].uuid() + assert uuid2 == uuid + assert uuid2 != uuid1 + del _custom_mm.impls["impl_mm_oot"] + + +def _test_native(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return x + y + + +def _make_op_with_generator(name: str = "_ig_test"): + op = IrOp(name, _test_native) + + @op.register_input_generator + def _gen(n: int = 4): + x = torch.randn(n, 3) + y = torch.randn(n, 3) + return x, y + + return op + + +def _test_native_single(x: torch.Tensor) -> torch.Tensor: + return x + + +class TestInputGenerator: + def test_no_input_generator_by_default(self): + op = IrOp("_ig_test_no_gen", _test_native_single) + assert not op.has_input_generator + + def test_register_input_generator(self): + op = _make_op_with_generator("_ig_test_reg") + assert op.has_input_generator + + def test_generate_inputs_returns_tuple(self): + op = _make_op_with_generator("_ig_test_tuple") + result = op.generate_inputs(n=2) + assert isinstance(result, tuple) + assert len(result) == 2 + assert result[0].shape == (2, 3) + assert result[1].shape == (2, 3) + + def test_generate_inputs_default_kwargs(self): + op = _make_op_with_generator("_ig_test_default") + result = op.generate_inputs() + assert result[0].shape == (4, 3) + + def test_generate_inputs_without_registration_raises(self): + op = IrOp("_ig_test_no_gen_raises", _test_native_single) + with pytest.raises(RuntimeError, match="No input generator"): + op.generate_inputs() + + +class TestTolerance: + def test_override_and_get_tolerance(self): + op = IrOp("_tol_test", _test_native) + + tol = op.get_tolerance(torch.float32) + assert tol == {"atol": 1e-5, "rtol": 1.3e-6} + + op.override_tolerance(torch.float32, atol=0.1, rtol=0.2) + assert op.get_tolerance(torch.float32) == {"atol": 0.1, "rtol": 0.2} + assert op.get_tolerance(torch.float16) == {"atol": 1e-3, "rtol": 1e-3} + + def test_get_tolerance_raises_for_unknown_dtype(self): + op = IrOp("_tol_test_unknown", _test_native) + with pytest.raises(ValueError, match="No tolerance defined"): + op.get_tolerance(torch.complex64) + + +def test_naming_validation(fake_vllm_ir): + """Test that op and provider names are validated ([a-z_][a-z_0-9]*).""" + + # Valid op and provider names + @vllm.ir.register_op + def _valid_name_123(x: torch.Tensor) -> torch.Tensor: + return x + + @_valid_name_123.register_impl("valid_provider_123") + def valid_impl(x: torch.Tensor) -> torch.Tensor: + return x + 1 + + # Invalid op names should fail + with pytest.raises(ValueError, match="name.*invalid"): + + @vllm.ir.register_op + def InvalidName(x: torch.Tensor) -> torch.Tensor: + return x + + with pytest.raises(ValueError, match="name.*invalid"): + + @vllm.ir.register_op(name="123invalid") + def some_func(x: torch.Tensor) -> torch.Tensor: + return x + + # Invalid provider names should fail + with pytest.raises(ValueError, match="name.*invalid"): + + @_valid_name_123.register_impl("Invalid-Provider") + def invalid_impl(x: torch.Tensor) -> torch.Tensor: + return x + 1 + + +def test_registration_stack_traces(fake_vllm_ir): + """Test that stack traces are captured for ops and impls.""" + + @vllm.ir.register_op + def _test_stack(x: torch.Tensor) -> torch.Tensor: + return x + + @_test_stack.register_impl("test_provider") + def test_impl(x: torch.Tensor) -> torch.Tensor: + return x + 1 + + # Verify op stack trace + assert hasattr(_test_stack, "_registration_stack") + assert len(_test_stack._registration_stack) > 0 + op_stack_str = "".join(_test_stack._registration_stack) + assert "test_op.py" in op_stack_str + # Last frame should be the decorator in user code, not internal decorator logic + assert "@vllm.ir.register_op" in _test_stack._registration_stack[-1] + assert "return decorator(f)" not in op_stack_str + + # Verify impl stack trace + impl = _test_stack.impls["test_provider"] + assert hasattr(impl, "_registration_stack") + assert len(impl._registration_stack) > 0 + impl_stack_str = "".join(impl._registration_stack) + assert "test_op.py" in impl_stack_str + # Last frame should be the decorator in user code + assert '@_test_stack.register_impl("test_provider")' in impl._registration_stack[-1] + + +def test_op_repr_uses_docstring(fake_vllm_ir): + """Test that __str__ uses the function's docstring and __repr__ is simple.""" + + @vllm.ir.register_op + def _test_repr_with_doc(x: torch.Tensor) -> torch.Tensor: + """First line of docstring. + + Additional details here. + """ + return x + + @vllm.ir.register_op + def _test_repr_no_doc(x: torch.Tensor) -> torch.Tensor: + return x + + # __str__ with docstring: uses first line only + str_with = str(_test_repr_with_doc) + assert "IrOp('_test_repr_with_doc')" in str_with + assert "First line of docstring." in str_with + assert "Additional details" not in str_with + + # __str__ without docstring: simple format + assert str(_test_repr_no_doc) == "IrOp('_test_repr_no_doc')" + + # __repr__ should be simple for both + assert repr(_test_repr_with_doc) == "IrOp('_test_repr_with_doc')" + assert repr(_test_repr_no_doc) == "IrOp('_test_repr_no_doc')" + + +def test_vllm_ir_fixture(fake_vllm_ir): + """Test that the fake_vllm_ir fixture provides test isolation.""" + + @vllm.ir.register_op + def _test_fixture(x: torch.Tensor) -> torch.Tensor: + return x + + assert "_test_fixture" in IrOp.registry + # Fixture will automatically clean up after test diff --git a/asian_ai/performance/vLLM/tests/kernels/__init__.py b/asian_ai/performance/vLLM/tests/kernels/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/kernels/allclose_default.py b/asian_ai/performance/vLLM/tests/kernels/allclose_default.py new file mode 100644 index 0000000000000000000000000000000000000000..b8209e87294198eb21a3ebf07cb67e9234c4c6f1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/allclose_default.py @@ -0,0 +1,17 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import torch + +# Reference default values of atol and rtol are from +# https://github.com/pytorch/pytorch/blob/6d96beb6bec24d73ee3f080bac54d2104068f675/test/test_transformers.py#L67 +default_atol = {torch.float16: 1e-3, torch.bfloat16: 1e-3, torch.float: 1e-5} +default_rtol = {torch.float16: 1e-3, torch.bfloat16: 1.6e-2, torch.float: 1.3e-6} + + +def get_default_atol(output) -> float: + return default_atol[output.dtype] + + +def get_default_rtol(output) -> float: + return default_rtol[output.dtype] diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/conftest.py b/asian_ai/performance/vLLM/tests/kernels/attention/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..9e912fb722f06a31e4432447c07e23a43c68f337 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/conftest.py @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from vllm.utils.torch_utils import ( + create_kv_caches_with_random, + create_kv_caches_with_random_flash, +) + + +@pytest.fixture() +def kv_cache_factory(): + return create_kv_caches_with_random + + +@pytest.fixture() +def kv_cache_factory_flashinfer(): + return create_kv_caches_with_random_flash diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_aiter_flash_attn.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_aiter_flash_attn.py new file mode 100644 index 0000000000000000000000000000000000000000..8e6af45a5e0e6bb3362e0b913f0d8a2ac28abb56 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_aiter_flash_attn.py @@ -0,0 +1,258 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import pytest +import torch + +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +# Import AITER backend if on ROCm and aiter is available +if current_platform.is_rocm(): + from vllm._aiter_ops import is_aiter_found_and_supported + + if is_aiter_found_and_supported(): + import aiter + + from vllm.v1.attention.backends.rocm_aiter_fa import cp_mha_gather_cache + +NUM_HEADS = [(4, 4), (8, 2)] +HEAD_SIZES = [128, 256] +BLOCK_SIZES = [16] +DTYPES = [torch.bfloat16] +QDTYPES = [None] +# one value large enough to test overflow in index calculation. +# one value small enough to test the schema op check +NUM_BLOCKS = [32768, 2048] + + +def ref_paged_attn( + query: torch.Tensor, + key_cache: torch.Tensor, + value_cache: torch.Tensor, + query_lens: list[int], + kv_lens: list[int], + block_tables: torch.Tensor, + scale: float, + sliding_window: int | None = None, + soft_cap: float | None = None, +) -> torch.Tensor: + num_seqs = len(query_lens) + block_tables = block_tables.cpu().numpy() + _, block_size, num_kv_heads, head_size = key_cache.shape + + outputs: list[torch.Tensor] = [] + start_idx = 0 + for i in range(num_seqs): + query_len = query_lens[i] + kv_len = kv_lens[i] + q = query[start_idx : start_idx + query_len] + q *= scale + + num_kv_blocks = (kv_len + block_size - 1) // block_size + block_indices = block_tables[i, :num_kv_blocks] + + k = key_cache[block_indices].view(-1, num_kv_heads, head_size) + k = k[:kv_len] + v = value_cache[block_indices].view(-1, num_kv_heads, head_size) + v = v[:kv_len] + + if q.shape[1] != k.shape[1]: + k = torch.repeat_interleave(k, q.shape[1] // k.shape[1], dim=1) + v = torch.repeat_interleave(v, q.shape[1] // v.shape[1], dim=1) + attn = torch.einsum("qhd,khd->hqk", q, k).float() + empty_mask = torch.ones(query_len, kv_len) + mask = torch.triu(empty_mask, diagonal=kv_len - query_len + 1).bool() + if sliding_window is not None: + sliding_window_mask = ( + torch.triu( + empty_mask, diagonal=kv_len - (query_len + sliding_window) + 1 + ) + .bool() + .logical_not() + ) + mask |= sliding_window_mask + if soft_cap is not None: + attn = soft_cap * torch.tanh(attn / soft_cap) + attn.masked_fill_(mask, float("-inf")) + attn = torch.softmax(attn, dim=-1).to(v.dtype) + out = torch.einsum("hqk,khd->qhd", attn, v) + + outputs.append(out) + start_idx += query_len + + return torch.cat(outputs, dim=0) + + +@pytest.mark.skipif(not current_platform.is_rocm(), reason="Only ROCm is supported") +@pytest.mark.parametrize( + "seq_lens", [[(10, 1328), (5, 18), (129, 463)], [(8, 523), (24, 37), (3, 2011)]] +) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("sliding_window", [None, 256]) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("soft_cap", [None]) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("q_dtype", QDTYPES) +@torch.inference_mode() +def test_varlen_with_paged_kv( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + num_blocks: int, + q_dtype: torch.dtype | None, +) -> None: + from vllm._aiter_ops import is_aiter_found_and_supported + + if not is_aiter_found_and_supported(): + pytest.skip("aiter package required for this test.") + + torch.set_default_device("cuda") + set_random_seed(0) + num_seqs = len(seq_lens) + query_lens = [x[0] for x in seq_lens] + kv_lens = [x[1] for x in seq_lens] + num_query_heads = num_heads[0] + num_kv_heads = num_heads[1] + assert num_query_heads % num_kv_heads == 0 + max_query_len = max(query_lens) + max_kv_len = max(kv_lens) + window_size = (sliding_window - 1, 0) if sliding_window is not None else (-1, -1) + scale = head_size**-0.5 + + query = torch.randn(sum(query_lens), num_query_heads, head_size, dtype=dtype) + key_cache = torch.randn( + num_blocks, block_size, num_kv_heads, head_size, dtype=dtype + ) + value_cache = torch.randn_like(key_cache) + cu_query_lens = torch.tensor([0] + query_lens, dtype=torch.int32).cumsum( + dim=0, dtype=torch.int32 + ) + + cu_seq_lens = torch.tensor([0] + kv_lens, dtype=torch.int32).cumsum( + dim=0, dtype=torch.int32 + ) + # Save kv_lens as list before converting to tensor + kv_lens_list = kv_lens + kv_lens = torch.tensor(kv_lens, dtype=torch.int32) + + max_num_blocks_per_seq = (max_kv_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, num_blocks, (num_seqs, max_num_blocks_per_seq), dtype=torch.int32 + ) + + output = torch.empty_like(query) + + maybe_quantized_query = query + maybe_quantized_key_cache = key_cache + maybe_quantized_value_cache = value_cache + k_scale_tensor = None + v_scale_tensor = None + dequant = False + + if q_dtype is not None: + # QKV are drawn from N(0, 1): no need for a fp8 scaling factor + maybe_quantized_query = query.to(q_dtype) + maybe_quantized_key_cache = key_cache.to(q_dtype) + maybe_quantized_value_cache = value_cache.to(q_dtype) + dequant = True + scale_shape = (num_seqs, num_kv_heads) + + # For per-seq-per-head scales (matching AITER backend expectation) + k_scale_tensor = torch.ones(scale_shape, dtype=torch.float32) + v_scale_tensor = torch.ones(scale_shape, dtype=torch.float32) + + # Prepare metadata for cp_mha_gather_cache + # token_to_batch: maps each token to its batch index + token_to_batch = torch.zeros(sum(kv_lens_list), dtype=torch.int32) + seq_starts = torch.zeros(num_seqs, dtype=torch.int32) + + token_idx = 0 + for batch_idx, kv_len in enumerate(kv_lens_list): + token_to_batch[token_idx : token_idx + kv_len] = batch_idx + seq_starts[batch_idx] = 0 # Assuming all sequences start at 0 in their blocks + token_idx += kv_len + + # Allocate buffers for gathered KV + total_kv_tokens = sum(kv_lens_list) + gathered_key = torch.empty( + total_kv_tokens, num_kv_heads, head_size, dtype=maybe_quantized_key_cache.dtype + ) + gathered_value = torch.empty( + total_kv_tokens, + num_kv_heads, + head_size, + dtype=maybe_quantized_value_cache.dtype, + ) + + # Gather paged KV cache into contiguous tensors using triton kernel + cp_mha_gather_cache( + key_cache=maybe_quantized_key_cache, + value_cache=maybe_quantized_value_cache, + key=gathered_key, + value=gathered_value, + block_tables=block_tables, + k_scales=k_scale_tensor + if k_scale_tensor is not None + else torch.ones(1, dtype=torch.float32), + v_scales=v_scale_tensor + if v_scale_tensor is not None + else torch.ones(1, dtype=torch.float32), + cu_seqlens_kv=cu_seq_lens, + token_to_batch=token_to_batch, + seq_starts=seq_starts, + dequant=dequant, + kv_cache_layout="NHD", + total_tokens=total_kv_tokens, + ) + + # Call aiter flash attention with gathered KV + aiter.flash_attn_varlen_func( + q=maybe_quantized_query, + k=gathered_key, + v=gathered_value, + cu_seqlens_q=cu_query_lens, + cu_seqlens_k=cu_seq_lens, + max_seqlen_q=max_query_len, + max_seqlen_k=max_kv_len, + min_seqlen_q=1, + dropout_p=0.0, + softmax_scale=scale, + causal=True, + window_size=window_size, + alibi_slopes=None, + return_lse=False, + out=output, + ) + + ref_output = ref_paged_attn( + query=query, + key_cache=key_cache, + value_cache=value_cache, + query_lens=query_lens, + kv_lens=kv_lens_list, + block_tables=block_tables, + scale=scale, + sliding_window=sliding_window, + soft_cap=soft_cap, + ) + + atol, rtol = 2e-2, 2e-2 + if q_dtype is not None: + atol, rtol = 1.5e-1, 1.5e-1 + ( + torch.testing.assert_close(output, ref_output, atol=atol, rtol=rtol), + f"{torch.max(torch.abs(output - ref_output))}", + ) + + # Log diff stats for tracking changes + print(f"Max abs diff: {torch.max(torch.abs(output - ref_output))}") + print(f"Mean diff: {torch.mean(torch.abs(output - ref_output))}") + print(f"Min diff: {torch.std(torch.abs(output - ref_output))}") diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_attention.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_attention.py new file mode 100644 index 0000000000000000000000000000000000000000..0e87ae0918866dfd7f8dbf60da0072d70b5dca87 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_attention.py @@ -0,0 +1,346 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import random + +import pytest +import torch + +from tests.kernels.allclose_default import get_default_atol, get_default_rtol +from tests.kernels.utils import opcheck +from vllm import _custom_ops as ops +from vllm.model_executor.layers.attention import Attention, MMEncoderAttention +from vllm.platforms import current_platform +from vllm.utils.mem_utils import get_max_shared_memory_bytes +from vllm.utils.torch_utils import set_random_seed + +FLOAT32_BYTES = torch.finfo(torch.float).bits // 8 +# This will change depending on the compute capability. +# - 512 as a buffer +MAX_SEQ_LEN = get_max_shared_memory_bytes() // FLOAT32_BYTES - 512 +# There may not be enough gpu memory due to large NUM_BLOCKS. +# Reduce NUM_BLOCKS when it happens. +NUM_BLOCKS = 4321 # Arbitrary values for testing +PARTITION_SIZE_ROCM = 256 +DTYPES = [torch.bfloat16] +NUM_GEN_SEQS = [7] # Arbitrary values for testing +NUM_PREFILL_SEQS = [3] # Arbitrary values for testing +NUM_HEADS = [(32, 8), (40, 40), (64, 8)] # Arbitrary values for testing + +# Head sizes supported by the ROCm paged attention kernel. +HEAD_SIZES = [64, 128] + +BLOCK_SIZES = [16, 32] +USE_ALIBI = [False, True] +KV_CACHE_DTYPE = ["auto", "fp8"] +SEEDS = [0] +CUDA_DEVICES = [ + f"cuda:{i}" for i in range(1 if torch.accelerator.device_count() == 1 else 2) +] + + +def ref_masked_attention( + query: torch.Tensor, + key: torch.Tensor, + value: torch.Tensor, + scale: float, + attn_mask: torch.Tensor | None = None, +) -> torch.Tensor: + attn_weights = scale * torch.einsum("qhd,khd->hqk", query, key).float() + if attn_mask is not None: + attn_weights = attn_weights + attn_mask.float() + attn_weights = torch.softmax(attn_weights, dim=-1).to(value.dtype) + out = torch.einsum("hqk,khd->qhd", attn_weights, value) + return out + + +def ref_single_query_cached_kv_attention( + output: torch.Tensor, + query: torch.Tensor, + num_queries_per_kv: int, + key_cache: torch.Tensor, + value_cache: torch.Tensor, + block_tables: torch.Tensor, + seq_lens: torch.Tensor, + scale: float, + alibi_slopes: torch.Tensor | None, +) -> None: + num_query_heads = query.shape[1] + num_kv_heads = value_cache.shape[1] + head_size = value_cache.shape[2] + block_size = value_cache.shape[3] + num_seqs = query.shape[0] + + block_tables_lst = block_tables.cpu().tolist() + seq_lens_lst = seq_lens.cpu().tolist() + for i in range(num_seqs): + q = query[i].unsqueeze(0) + block_table = block_tables_lst[i] + seq_len = int(seq_lens_lst[i]) + + keys_lst: list[torch.Tensor] = [] + values_lst: list[torch.Tensor] = [] + for j in range(seq_len): + block_number = int(block_table[j // block_size]) + block_offset = j % block_size + + k = key_cache[block_number, :, :, block_offset, :] + k = k.reshape(num_kv_heads, head_size) + keys_lst.append(k) + + v = value_cache[block_number, :, :, block_offset] + values_lst.append(v) + keys = torch.stack(keys_lst, dim=0) + values = torch.stack(values_lst, dim=0) + if num_queries_per_kv > 1: + # Handle MQA and GQA + keys = torch.repeat_interleave(keys, num_queries_per_kv, dim=1) + values = torch.repeat_interleave(values, num_queries_per_kv, dim=1) + + alibi_bias = None + if alibi_slopes is not None: + # Create the ALiBi bias used in the paged attention kernel. + position_ids = torch.arange(seq_len).int() + alibi_bias = (position_ids - seq_len + 1).float() + alibi_bias = alibi_slopes.view(-1, 1, 1) * alibi_bias.view(1, 1, -1) + + out = ref_masked_attention(q, keys, values, scale, alibi_bias) + out = out.view(num_query_heads, head_size) + output[i].copy_(out, non_blocking=True) + + +@pytest.mark.skipif( + not current_platform.is_rocm(), reason="ROCm-only paged attention kernel" +) +@pytest.mark.parametrize("num_seqs", NUM_GEN_SEQS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("use_alibi", USE_ALIBI) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("kv_cache_dtype", KV_CACHE_DTYPE) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +def test_paged_attention( + kv_cache_factory, + num_seqs: int, + num_heads: tuple[int, int], + head_size: int, + use_alibi: bool, + block_size: int, + dtype: torch.dtype, + kv_cache_dtype: str, + seed: int, + device: str, +) -> None: + if current_platform.is_navi() and ( + kv_cache_dtype == "fp8" or head_size != 128 or block_size != 16 or use_alibi + ): + pytest.skip() + + set_random_seed(seed) + torch.set_default_device(device) + scale = float(1.0 / (head_size**0.5)) + num_query_heads, num_kv_heads = num_heads + query = torch.empty(num_seqs, num_query_heads, head_size, dtype=dtype) + query.uniform_(-scale, scale) + + assert num_query_heads % num_kv_heads == 0 + num_queries_per_kv = num_query_heads // num_kv_heads + alibi_slopes = None + if use_alibi: + alibi_slopes = torch.randn(num_query_heads, dtype=torch.float) + + seq_lens = [random.randint(1, MAX_SEQ_LEN) for _ in range(num_seqs)] + seq_lens[-1] = MAX_SEQ_LEN + max_seq_len = max(seq_lens) + seq_lens = torch.tensor(seq_lens, dtype=torch.int) + + # Create the block tables. + max_num_blocks_per_seq = (max_seq_len + block_size - 1) // block_size + block_tables_lst: list[list[int]] = [] + for _ in range(num_seqs): + block_table = [ + random.randint(0, NUM_BLOCKS - 1) for _ in range(max_num_blocks_per_seq) + ] + block_tables_lst.append(block_table) + + block_tables = torch.tensor(block_tables_lst, dtype=torch.int) + + # Create the KV caches. + key_caches, value_caches = kv_cache_factory( + NUM_BLOCKS, + block_size, + 1, + num_kv_heads, + head_size, + kv_cache_dtype, + dtype, + seed, + device, + ) + key_cache, value_cache = key_caches[0], value_caches[0] + + # Using default kv_scale + k_scale = v_scale = torch.tensor(1.0, dtype=torch.float32, device=device) + + # Call the paged attention kernel. + output = torch.empty_like(query) + num_partitions = (max_seq_len + PARTITION_SIZE_ROCM - 1) // PARTITION_SIZE_ROCM + assert PARTITION_SIZE_ROCM % block_size == 0 + num_seqs, num_heads, head_size = output.shape + tmp_output = torch.empty( + size=(num_seqs, num_heads, num_partitions, head_size), + dtype=output.dtype, + ) + exp_sums = torch.empty( + size=(num_seqs, num_heads, num_partitions), + dtype=torch.float32, + ) + max_logits = torch.empty_like(exp_sums) + ops.paged_attention_rocm( + output, + exp_sums, + max_logits, + tmp_output, + query, + key_cache, + value_cache, + num_kv_heads, + scale, + block_tables, + seq_lens, + None, + block_size, + max_seq_len, + alibi_slopes, + kv_cache_dtype, + k_scale, + v_scale, + ) + + opcheck( + torch.ops._rocm_C.paged_attention, + ( + output, + exp_sums, + max_logits, + tmp_output, + query, + key_cache, + value_cache, + num_kv_heads, + scale, + block_tables, + seq_lens, + None, + block_size, + max_seq_len, + alibi_slopes, + kv_cache_dtype, + k_scale, + v_scale, + None, + "f16", + ), + cond=(head_size == 64 and block_size == BLOCK_SIZES[0]), + ) + + # Run the reference implementation. + if kv_cache_dtype == "fp8": + # Convert cache data back to dtype. + x = 16 // torch.tensor([], dtype=dtype).element_size() + key_cache_shape = (NUM_BLOCKS, num_kv_heads, head_size // x, block_size, x) + dequantized_key_cache = torch.empty( + size=key_cache_shape, dtype=dtype, device=device + ) + ops.convert_fp8(dequantized_key_cache, key_cache) + key_cache = dequantized_key_cache + + value_cache_shape = value_cache.shape + dequantized_value_cache = torch.empty( + size=value_cache_shape, dtype=dtype, device=device + ) + ops.convert_fp8(dequantized_value_cache, value_cache) + value_cache = dequantized_value_cache + + ref_output = torch.empty_like(query) + ref_single_query_cached_kv_attention( + ref_output, + query, + num_queries_per_kv, + key_cache, + value_cache, + block_tables, + seq_lens, + scale, + alibi_slopes, + ) + + # NOTE(woosuk): Due to the kernel-level differences in the two + # implementations, there is a small numerical difference in the two + # outputs. Thus, we use a relaxed tolerance for the test. + atol = get_default_atol(output) if current_platform.is_rocm() else 1e-3 + rtol = get_default_rtol(output) if current_platform.is_rocm() else 1e-5 + + # NOTE(zhaoyang): FP8 KV Cache will introduce quantization error, + # so we use a relaxed tolerance for the test. + atol, rtol = 1e-3, 1e-5 + if kv_cache_dtype == "fp8": + atol, rtol = 1e-2, 1e-5 + torch.testing.assert_close(output, ref_output, atol=atol, rtol=rtol) + + +def ref_multi_query_kv_attention( + cu_seq_lens: list[int], + query: torch.Tensor, + key: torch.Tensor, + value: torch.Tensor, + scale: float, + alibi_bias: list[torch.Tensor] | None, + dtype: torch.dtype, +) -> torch.Tensor: + num_seqs = len(cu_seq_lens) - 1 + ref_outputs: list[torch.Tensor] = [] + if alibi_bias: + assert len(alibi_bias) == num_seqs + for i in range(num_seqs): + start_idx = cu_seq_lens[i] + end_idx = cu_seq_lens[i + 1] + seq_len = end_idx - start_idx + + # Create attention mask. ALiBi already includes a tril causal mask. + if alibi_bias: + attn_mask = alibi_bias[i] + else: + attn_mask = torch.triu( + torch.ones(seq_len, seq_len, dtype=dtype), diagonal=1 + ) + attn_mask = attn_mask * torch.finfo(dtype).min + attn_mask = attn_mask.to(dtype=dtype) + + ref_output = ref_masked_attention( + query[start_idx:end_idx], + key[start_idx:end_idx], + value[start_idx:end_idx], + scale, + attn_mask=attn_mask, + ) + ref_outputs.append(ref_output) + + return torch.cat(ref_outputs, dim=0) + + +@pytest.mark.parametrize("attention_cls", [Attention, MMEncoderAttention]) +def test_num_heads_not_divisible_by_num_kv_heads(attention_cls: type) -> None: + head_size = 64 + scale = float(1.0 / (head_size**0.5)) + num_heads = 16 + num_kv_heads = 5 + with pytest.raises(AssertionError): + _ = attention_cls( + num_heads=num_heads, + head_size=head_size, + scale=scale, + num_kv_heads=num_kv_heads, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_attention_selector.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_attention_selector.py new file mode 100644 index 0000000000000000000000000000000000000000..f371f1a5566f8ae0f43b7935b3a04285b022f5db --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_attention_selector.py @@ -0,0 +1,553 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from unittest.mock import patch + +import pytest +import torch + +from vllm.config import ( + AttentionConfig, + CacheConfig, + VllmConfig, + set_current_vllm_config, +) +from vllm.platforms import current_platform +from vllm.platforms.cpu import CpuPlatform + +if current_platform.is_cuda(): + from vllm.platforms.cuda import CudaPlatform +else: + CudaPlatform = None + +if current_platform.is_rocm(): + from vllm.platforms.rocm import RocmPlatform +else: + RocmPlatform = None + +from vllm.v1.attention.backends.registry import AttentionBackendEnum +from vllm.v1.attention.selector import _cached_get_attn_backend, get_attn_backend + + +@pytest.fixture(autouse=True) +def clear_cache(): + """Clear lru cache to ensure each test case runs without caching.""" + _cached_get_attn_backend.cache_clear() + + +# Define MLA and non-MLA backends separately +DEVICE_MLA_BACKENDS = { + "cuda": [ + "TRITON_MLA", + "FLASHMLA", + "FLASHINFER_MLA", + "FLASH_ATTN_MLA", + "CUTLASS_MLA", + ], + "hip": ["TRITON_MLA", "ROCM_AITER_MLA"], + "cpu": [], +} + +DEVICE_REGULAR_ATTN_BACKENDS = { + "cuda": ["FLASHINFER", "FLASH_ATTN"], + "hip": ["ROCM_ATTN"], + "cpu": ["CPU_ATTN"], +} + +DEVICE_MLA_BLOCK_SIZES = { + "cuda": [16, 64], # CUDA supports both standard and extended block sizes + "hip": [16, 1], # HIP requires special handling for block_size=1 + # "cpu": [16] # CPU uses fixed block size from test cases + "cpu": [], # FIXME(woosuk): Temporarily disable CPU tests +} + + +def generate_params(): + is_rocm = current_platform.is_rocm() + params = [] + device_list = ["cuda", "cpu"] if not is_rocm else ["hip", "cpu"] + for use_mla in [True, False]: + for device in device_list: + backends = ( + DEVICE_MLA_BACKENDS[device] + if use_mla + else DEVICE_REGULAR_ATTN_BACKENDS[device] + ) + for name in backends: + block_sizes = DEVICE_MLA_BLOCK_SIZES[device] if use_mla else [16] + for block_size in block_sizes: + params.append( + pytest.param( + device, + name, + use_mla, + block_size, + id=f"{device}_{name}_mla_{str(use_mla)[0]}_blks{block_size}", + ) + ) + return params + + +@pytest.mark.parametrize("device, name, use_mla, block_size", generate_params()) +def test_backend_selection( + device: str, + name: str, + use_mla: bool, + block_size: int, +): + """Test attention backend selection with valid device-backend pairs.""" + # Create AttentionConfig with the specified backend + attention_config = AttentionConfig(backend=AttentionBackendEnum[name]) + cache_config = CacheConfig(block_size=block_size) + vllm_config = VllmConfig( + attention_config=attention_config, cache_config=cache_config + ) + + with set_current_vllm_config(vllm_config): + if device == "cpu": + with patch("vllm.platforms.current_platform", CpuPlatform()): + backend = get_attn_backend(16, torch.float16, None) + assert backend.get_name() == "CPU_ATTN" + + elif device == "hip": + if RocmPlatform is None: + pytest.skip("RocmPlatform not available") + with patch("vllm.platforms.current_platform", RocmPlatform()): + if use_mla: + # ROCm MLA backend logic: + # - TRITON_MLA: supported when block_size != 1 + # - ROCM_AITER_MLA: supported when block_size == 1 + # If backend is forced but doesn't match block_size, + # should raise ValueError + + if name == "TRITON_MLA" and block_size == 1: + # TRITON_MLA doesn't support block_size == 1 + with pytest.raises(ValueError): + get_attn_backend(576, torch.float16, None, use_mla=use_mla) + else: + # Valid backend-block_size combination + backend = get_attn_backend( + 576, torch.float16, None, use_mla=use_mla + ) + expected = name + assert backend.get_name() == expected + else: + backend = get_attn_backend(32, torch.float16, None, use_mla=use_mla) + expected = "ROCM_ATTN" + assert backend.get_name() == expected + + elif device == "cuda": + if CudaPlatform is None: + pytest.skip("CudaPlatform not available") + with patch("vllm.platforms.current_platform", CudaPlatform()): + capability = torch.cuda.get_device_capability() + if use_mla: + # CUDA MLA backend logic: + # - CUTLASS_MLA: only supported with block_size == 128 + # and Blackwell GPUs (SM 10.x), V1 only + # - FLASHINFER_MLA: only supported on Blackwell GPUs + # (SM 10.x), V1 only + # - FLASHMLA: only supported with block_size == 64 + # - FLASH_ATTN_MLA: V1 only + # - TRITON_MLA: fallback for other cases + + if name == "CUTLASS_MLA": + if block_size != 128: + # CUTLASS_MLA only supports block_size == 128 + pytest.skip("CUTLASS_MLA only supports block_size 128") + if capability[0] != 10: + pytest.skip("CUTLASS MLA is not supported on this platform") + backend = get_attn_backend( + 576, torch.float16, None, use_mla=use_mla + ) + expected = "CUTLASS_MLA" + assert backend.get_name() == expected + elif name == "FLASHINFER_MLA": + if capability[0] != 10: + pytest.skip( + "FlashInfer MLA is not supported on this platform" + ) + if block_size not in [32, 64]: + # FlashInfer MLA only supports block_size 32 or 64 + pytest.skip( + "FlashInfer MLA only supports block_size 32 or 64" + ) + backend = get_attn_backend( + 576, torch.float16, None, use_mla=use_mla + ) + expected = "FLASHINFER_MLA" + assert backend.get_name() == expected + elif name == "FLASHMLA": + if block_size != 64: + # FlashMLA only supports block_size == 64 + pytest.skip("FlashMLA only supports block_size 64") + from vllm.v1.attention.backends.mla.flashmla import ( + is_flashmla_dense_supported, + ) + + is_supported, _ = is_flashmla_dense_supported() + if not is_supported: + pytest.skip("FlashMLA not supported on this platform") + backend = get_attn_backend( + 576, + torch.float16, + None, + use_mla=use_mla, + ) + expected = name + assert backend.get_name() == expected + elif name == "FLASH_ATTN_MLA": + from vllm.v1.attention.backends.fa_utils import ( + flash_attn_supports_mla, + ) + + if not flash_attn_supports_mla(): + pytest.skip( + "FlashAttention MLA not supported on this platform" + ) + backend = get_attn_backend( + 576, torch.float16, None, use_mla=use_mla + ) + expected = "FLASH_ATTN_MLA" + assert backend.get_name() == expected + else: + # TRITON_MLA or other fallback + backend = get_attn_backend( + 576, torch.float16, None, use_mla=use_mla + ) + expected = "TRITON_MLA" + assert backend.get_name() == expected + elif name == "FLASHINFER": + backend = get_attn_backend(64, torch.float16, None, use_mla=use_mla) + expected = "FLASHINFER" + assert backend.get_name() == expected + elif name == "FLASH_ATTN": + backend = get_attn_backend(32, torch.float16, None, use_mla=use_mla) + expected = "FLASH_ATTN" + assert backend.get_name() == expected + + +@pytest.mark.parametrize("device", ["cpu", "cuda", "hip"]) +def test_fp32_fallback(device: str): + """Test attention backend selection with fp32.""" + # Use default config (no backend specified) + vllm_config = VllmConfig() + + with set_current_vllm_config(vllm_config): + if device == "cpu": + with patch("vllm.platforms.current_platform", CpuPlatform()): + backend = get_attn_backend(16, torch.float32, None) + assert backend.get_name() == "CPU_ATTN" + + elif device == "cuda": + if CudaPlatform is None: + pytest.skip("CudaPlatform not available") + with patch("vllm.platforms.current_platform", CudaPlatform()): + backend = get_attn_backend(16, torch.float32, None) + assert backend.get_name() == "FLEX_ATTENTION" + + elif device == "hip": + if RocmPlatform is None: + pytest.skip("RocmPlatform not available") + # ROCm backends do not support head_size=16 (minimum is 32). + # No known HuggingFace transformer model uses head_size=16. + # Revisit if a real model with this head size is identified + # and accuracy-tested. + with ( + patch("vllm.platforms.current_platform", RocmPlatform()), + pytest.raises(ValueError, match="No valid attention backend"), + ): + get_attn_backend(16, torch.float32, None) + + +def test_flash_attn(monkeypatch: pytest.MonkeyPatch): + """Test FlashAttn validation.""" + pytest.skip( + "Skipping as current backend selector does not " + "handle fallbacks when a backend is explicitly set." + ) + + attention_config = AttentionConfig(backend=AttentionBackendEnum.FLASH_ATTN) + cache_config = CacheConfig(block_size=16) + vllm_config = VllmConfig( + attention_config=attention_config, cache_config=cache_config + ) + + with set_current_vllm_config(vllm_config): + # Unsupported CUDA arch + monkeypatch.setattr(torch.cuda, "get_device_capability", lambda _=None: (7, 5)) + backend = get_attn_backend(16, torch.float16, None) + assert backend.get_name() != "FLASH_ATTN" + + # Reset the monkeypatch for subsequent tests + monkeypatch.undo() + + # Unsupported data type + backend = get_attn_backend(16, torch.float8_e4m3fn, None) + assert backend.get_name() != "FLASH_ATTN" + + # Unsupported kv cache data type + backend = get_attn_backend(16, torch.float16, "fp8") + assert backend.get_name() != "FLASH_ATTN" + + # Unsupported block size + vllm_config.cache_config.block_size = 8 + backend = get_attn_backend(16, torch.float16, None) + assert backend.get_name() != "FLASH_ATTN" + + # flash-attn is not installed + import sys + + vllm_config.cache_config.block_size = 16 + original_module = sys.modules.get("vllm_flash_attn") + monkeypatch.setitem(sys.modules, "vllm_flash_attn", None) + backend = get_attn_backend(16, torch.float16, None) + assert backend.get_name() != "FLASH_ATTN" + + # Restore the original module if it existed + if original_module is not None: + monkeypatch.setitem(sys.modules, "vllm_flash_attn", original_module) + else: + monkeypatch.delitem(sys.modules, "vllm_flash_attn", raising=False) + + # Unsupported head size + backend = get_attn_backend(17, torch.float16, None) + assert backend.get_name() != "FLASH_ATTN" + + +def test_invalid_backend(): + """Test that invalid attention backend names raise ValueError.""" + with ( + pytest.raises(ValueError), + ): + # Invalid backend name should raise ValueError when creating enum + AttentionConfig(backend=AttentionBackendEnum["INVALID"]) + + +@pytest.mark.parametrize("auto_value", ["auto", "AUTO", "Auto"]) +def test_auto_backend_string(auto_value: str): + """Test that 'auto' string value triggers automatic backend selection.""" + # Using "auto" should result in backend=None (automatic selection) + attention_config = AttentionConfig(backend=auto_value) + assert attention_config.backend is None + + +def test_auto_backend_selection_behavior(): + """Test that 'auto' backend behaves same as None (automatic selection).""" + # Create config with explicit "auto" + auto_config = AttentionConfig(backend="auto") + + # Create config with None (default) + none_config = AttentionConfig(backend=None) + + # Both should have backend=None + assert auto_config.backend is None + assert none_config.backend is None + + # Both configs should result in the same automatic backend selection + vllm_config_auto = VllmConfig(attention_config=auto_config) + vllm_config_none = VllmConfig(attention_config=none_config) + + with ( + set_current_vllm_config(vllm_config_auto), + patch("vllm.platforms.current_platform", CpuPlatform()), + ): + backend_auto = get_attn_backend(16, torch.float16, None) + + _cached_get_attn_backend.cache_clear() + + with ( + set_current_vllm_config(vllm_config_none), + patch("vllm.platforms.current_platform", CpuPlatform()), + ): + backend_none = get_attn_backend(16, torch.float16, None) + + # Both should select the same backend + assert backend_auto.get_name() == backend_none.get_name() + + +@pytest.mark.parametrize( + "backend_name,flash_attn_version,should_succeed", + [ + ("FLASH_ATTN", 3, True), # FA3 supports per-head quant scales + ("FLASH_ATTN", 2, False), # FA2 does not support per-head quant scales + ("FLASHINFER", None, False), # FlashInfer does not support + ("FLEX_ATTENTION", None, False), # Flex does not support + ], +) +@pytest.mark.skipif( + current_platform.is_rocm(), + reason="Attention backend FA3 is not supported on ROCm. This test can't succeed.", +) +def test_per_head_quant_scales_backend_selection( + backend_name: str, flash_attn_version: int | None, should_succeed: bool +): + """Test backend selection when use_per_head_quant_scales=True.""" + # Clear cache to ensure fresh backend selection + _cached_get_attn_backend.cache_clear() + + attention_config = AttentionConfig( + backend=AttentionBackendEnum[backend_name], + flash_attn_version=flash_attn_version, + ) + cache_config = CacheConfig(block_size=64) + vllm_config = VllmConfig( + attention_config=attention_config, cache_config=cache_config + ) + + if CudaPlatform is None: + pytest.skip("CudaPlatform not available") + with ( + set_current_vllm_config(vllm_config), + patch("vllm.platforms.current_platform", CudaPlatform()), + ): + if backend_name == "FLASH_ATTN" and flash_attn_version == 3: + if not torch.cuda.is_available(): + pytest.skip("FA3 requires CUDA") + capability = torch.cuda.get_device_capability() + if capability[0] != 9: + pytest.skip("FA3 is only supported on Hopper (SM 9.x) GPUs") + + if should_succeed: + backend = get_attn_backend( + head_size=128, + dtype=torch.float16, + kv_cache_dtype="fp8", + use_per_head_quant_scales=True, + ) + assert backend.get_name() == backend_name + else: + with pytest.raises(ValueError) as exc_info: + get_attn_backend( + head_size=128, + dtype=torch.float16, + kv_cache_dtype="fp8", + use_per_head_quant_scales=True, + ) + assert backend_name in str(exc_info.value) + + +@pytest.mark.parametrize( + "backend_name,use_non_causal,should_succeed", + [ + ("FLASH_ATTN", True, True), # FlashAttn supports non-causal + ("FLASH_ATTN", False, True), # FlashAttn also works with causal + ] + + ( + [ + ("FLASHINFER", True, True), # FlashInfer supports non-causal + ("FLASHINFER", False, True), # FlashInfer works with causal + ] + if CudaPlatform is not None + else [] + ), +) +def test_non_causal_backend_selection( + backend_name: str, use_non_causal: bool, should_succeed: bool +): + """Test that use_non_causal on AttentionConfig controls backend filtering. + + DFlashProposer sets use_non_causal=True on the draft model's + AttentionConfig so only non-causal-capable backends are selected. + The target model keeps use_non_causal=False (default) and can use + any backend. + """ + _cached_get_attn_backend.cache_clear() + + attention_config = AttentionConfig( + backend=AttentionBackendEnum[backend_name], + use_non_causal=use_non_causal, + ) + cache_config = CacheConfig(block_size=16) + vllm_config = VllmConfig( + attention_config=attention_config, cache_config=cache_config + ) + + platform = CudaPlatform or RocmPlatform + if platform is None: + pytest.skip("CudaPlatform and RocmPlatform are not available") + with ( + set_current_vllm_config(vllm_config), + patch("vllm.platforms.current_platform", platform()), + ): + if should_succeed: + backend = get_attn_backend( + head_size=128, + dtype=torch.float16, + kv_cache_dtype=None, + ) + assert backend.get_name() == backend_name + else: + with pytest.raises(ValueError) as exc_info: + get_attn_backend( + head_size=128, + dtype=torch.float16, + kv_cache_dtype=None, + ) + assert "non-causal" in str(exc_info.value).lower() + + +def test_non_causal_autoselect_backend(): + """Test that when backend=None with use_non_causal=True, auto-selection + picks a compatible backend. + + This simulates the DFlash scenario where the user doesn't specify + --attention-backend or --speculative-config.attention_backend. + The drafter inherits backend=None and auto-selects a backend that + supports non-causal attention. + """ + _cached_get_attn_backend.cache_clear() + + attention_config = AttentionConfig( + backend=None, + use_non_causal=True, + ) + cache_config = CacheConfig(block_size=16) + vllm_config = VllmConfig( + attention_config=attention_config, cache_config=cache_config + ) + + if CudaPlatform is None: + pytest.skip("CudaPlatform not available") + with ( + set_current_vllm_config(vllm_config), + patch("vllm.platforms.current_platform", CudaPlatform()), + ): + backend = get_attn_backend( + head_size=128, + dtype=torch.float16, + kv_cache_dtype=None, + ) + assert backend.supports_non_causal() + + +@pytest.mark.parametrize( + "kv_cache_dtype", + [ + "fp8_e5m2", + "fp8_ds_mla", + "fp8_inc", + "nvfp4", + "fp8_per_token_head", + "int8_per_token_head", + ], +) +def test_flash_attn_rejects_unhandled_kv_cache_dtypes(kv_cache_dtype: str): + """FlashAttentionBackend must not claim support for kv_cache dtypes + that it cannot handle.""" + from vllm.v1.attention.backends.flash_attn import FlashAttentionBackend + + assert not FlashAttentionBackend.supports_kv_cache_dtype(kv_cache_dtype) + + +@pytest.mark.parametrize("kv_cache_dtype", ["fp8", "fp8_e4m3"]) +def test_flash_attn_accepts_handled_fp8_variants( + kv_cache_dtype: str, monkeypatch: pytest.MonkeyPatch +): + """FlashAttentionBackend must accept the two fp8 dtypes it can actually + handle: 'fp8' (alias for fp8_e4m3fn) and 'fp8_e4m3'.""" + import vllm.v1.attention.backends.flash_attn as fa_mod + from vllm.v1.attention.backends.flash_attn import FlashAttentionBackend + + monkeypatch.setattr(fa_mod.current_platform, "is_xpu", lambda: True) + assert FlashAttentionBackend.supports_kv_cache_dtype(kv_cache_dtype) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_cache.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_cache.py new file mode 100644 index 0000000000000000000000000000000000000000..c9159aca862caf828ae968b83caa5c807f3defe0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_cache.py @@ -0,0 +1,1224 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import random + +import pytest +import torch + +from tests.kernels.utils import DEFAULT_OPCHECK_TEST_UTILS, opcheck +from vllm import _custom_ops as ops +from vllm.model_executor.layers.quantization.utils.quant_utils import scaled_dequantize +from vllm.platforms import current_platform +from vllm.utils.torch_utils import nvfp4_kv_cache_split_views, set_random_seed + +COPYING_DIRECTION = [("cuda", "cpu"), ("cuda", "cuda"), ("cpu", "cuda")] +DTYPES = [torch.bfloat16, torch.float] +NUM_TOKENS = [42] # Arbitrary values for testing +NUM_LAYERS = [1] # Arbitrary values for testing +NUM_HEADS = [8] # Arbitrary values for testing +HEAD_SIZES = [64, 80, 256] +BLOCK_SIZES = [8, 16, 32] +CACHE_LAYOUTS = ["NHD", "HND"] +KV_SCALE_TYPES = ["tensor", "attn_head"] + +# Parameters for MLA tests. +KV_LORA_RANKS = [256, 512] +QK_ROPE_HEAD_DIMS = [64] +NUM_TOKENS_MLA = [42] +BLOCK_SIZES_MLA = [16] +NUM_BLOCKS_MLA = [8] + +# Arbitrary values for testing +# don't make it too large. e.g. [1024, 36000] will OOM +NUM_BLOCKS = [1024, 10000] + +NUM_MAPPINGS = [256] # Arbitrary values for testing +SEEDS = [0] +CUDA_DEVICES = [ + f"cuda:{i}" for i in range(1 if torch.accelerator.device_count() == 1 else 2) +] + +# We assume fp8 is always enabled for testing. +KV_CACHE_DTYPE = ["auto", "fp8"] + +RESHAPE_FLASH_IMPLEMENTATIONS = ["cuda", "triton"] + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("kv_cache_dtype", KV_CACHE_DTYPE) +@torch.inference_mode() +def test_reshape_and_cache( + kv_cache_factory, + num_tokens: int, + num_heads: int, + head_size: int, + block_size: int, + num_blocks: int, + dtype: torch.dtype, + seed: int, + device: str, + kv_cache_dtype: str, +) -> None: + if kv_cache_dtype == "fp8" and head_size % 16: + pytest.skip() + set_random_seed(seed) + torch.set_default_device(device) + torch.accelerator.set_device_index(device) + # Create a random slot mapping. + num_slots = block_size * num_blocks + slot_mapping_lst = random.sample(range(num_slots), num_tokens) + slot_mapping = torch.tensor(slot_mapping_lst, dtype=torch.long) + + qkv = torch.randn(num_tokens, 3, num_heads, head_size, dtype=dtype) + _, key, value = qkv.unbind(dim=1) + + # Create the KV caches. + key_caches, value_caches = kv_cache_factory( + num_blocks, + block_size, + 1, + num_heads, + head_size, + kv_cache_dtype, + dtype, + seed, + device, + ) + key_cache, value_cache = key_caches[0], value_caches[0] + + # Using default kv_scale + k_scale = (key.amax() / 64.0).to(torch.float32) + v_scale = (value.amax() / 64.0).to(torch.float32) + + # Clone the KV caches. + if kv_cache_dtype == "fp8": + cloned_key_cache = torch.empty_like(key_cache, dtype=torch.float16) + ops.convert_fp8(cloned_key_cache, key_cache, k_scale.item()) + cloned_value_cache = torch.empty_like(value_cache, dtype=torch.float16) + ops.convert_fp8(cloned_value_cache, value_cache, v_scale.item()) + else: + cloned_key_cache = key_cache.clone() + cloned_value_cache = value_cache.clone() + + # Call the reshape_and_cache kernel. + opcheck( + torch.ops._C_cache_ops.reshape_and_cache, + ( + key, + value, + key_cache, + value_cache, + slot_mapping, + kv_cache_dtype, + k_scale, + v_scale, + ), + cond=(head_size == HEAD_SIZES[0]), + ) + ops.reshape_and_cache( + key, + value, + key_cache, + value_cache, + slot_mapping, + kv_cache_dtype, + k_scale, + v_scale, + ) + + if kv_cache_dtype == "fp8": + result_key_cache = torch.empty_like(key_cache, dtype=torch.float16) + ops.convert_fp8(result_key_cache, key_cache, k_scale.item()) + result_value_cache = torch.empty_like(value_cache, dtype=torch.float16) + ops.convert_fp8(result_value_cache, value_cache, v_scale.item()) + + # Run the reference implementation. + reshaped_key = key.reshape(num_tokens, *key_cache[0, :, :, 0, :].shape) + block_indices = torch.div(slot_mapping, block_size, rounding_mode="floor") + block_indices_lst = block_indices.cpu().tolist() + block_offsets = slot_mapping % block_size + block_offsets_lst = block_offsets.cpu().tolist() + for i in range(num_tokens): + block_idx = block_indices_lst[i] + block_offset = block_offsets_lst[i] + cloned_key_cache[block_idx, :, :, block_offset, :] = reshaped_key[i] + cloned_value_cache[block_idx, :, :, block_offset] = value[i] + + if kv_cache_dtype == "fp8": + torch.testing.assert_close( + result_key_cache, cloned_key_cache, atol=0.001, rtol=0.1 + ) + torch.testing.assert_close( + result_value_cache, cloned_value_cache, atol=0.001, rtol=0.1 + ) + else: + torch.testing.assert_close(key_cache, cloned_key_cache) + torch.testing.assert_close(value_cache, cloned_value_cache) + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("kv_cache_dtype", KV_CACHE_DTYPE + ["nvfp4"]) +@pytest.mark.parametrize("kv_cache_layout", CACHE_LAYOUTS) +@pytest.mark.parametrize("kv_scale_type", KV_SCALE_TYPES) +@pytest.mark.parametrize("implementation", RESHAPE_FLASH_IMPLEMENTATIONS) +@torch.inference_mode() +def test_reshape_and_cache_flash( + kv_cache_factory_flashinfer, + num_tokens: int, + num_heads: int, + head_size: int, + block_size: int, + num_blocks: int, + dtype: torch.dtype, + seed: int, + device: str, + kv_cache_dtype: str, + kv_cache_layout: str, + kv_scale_type: str, + implementation: str, +) -> None: + set_random_seed(seed) + torch.set_default_device(device) + torch.accelerator.set_device_index(device) + assert implementation in ["cuda", "triton"] + if implementation == "triton" and kv_cache_layout == "HND": + pytest.skip("Triton implementation only supports NHD layout.") + + if kv_scale_type == "attn_head" and implementation != "cuda": + pytest.skip("Only CUDA implementation supports attn_head scaling.") + + if kv_cache_dtype == "nvfp4": + if not current_platform.has_device_capability(100): + pytest.skip("NVFP4 requires compute capability >= 10.0 (Blackwell).") + if implementation != "cuda": + pytest.skip("NVFP4 only supports CUDA implementation.") + if kv_scale_type != "tensor": + pytest.skip("NVFP4 only supports per-tensor scaling.") + if head_size % 16 != 0: + pytest.skip("NVFP4 requires head_size divisible by 16.") + if (head_size // 16) % 4 != 0: + pytest.skip( + "NVFP4 requires (head_size // 16) divisible by 4 " + "for 4x4 block scale swizzle." + ) + if block_size % 4 != 0: + pytest.skip("NVFP4 requires block_size divisible by 4.") + if dtype not in (torch.float16, torch.bfloat16): + pytest.skip("NVFP4 quantization only supports fp16/bf16 input.") + + # fp8 conversion requires continugous memory buffer. Reduce the number of + # blocks and tokens to consume less memory. + num_tokens = num_tokens // 2 + num_blocks = num_blocks // 2 + # Create a random slot mapping. + num_slots = block_size * num_blocks + slot_mapping_lst = random.sample(range(num_slots), num_tokens) + slot_mapping = torch.tensor(slot_mapping_lst, dtype=torch.long, device=device) + qkv = torch.randn(num_tokens, 3, num_heads, head_size, dtype=dtype, device=device) + _, key, value = qkv.unbind(dim=1) + + # Create the KV caches. + key_caches, value_caches = kv_cache_factory_flashinfer( + num_blocks, + block_size, + 1, + num_heads, + head_size, + kv_cache_dtype, + dtype, + device=device, + cache_layout=kv_cache_layout, + ) + key_cache, value_cache = key_caches[0], value_caches[0] + del key_caches + del value_caches + + # For nvfp4, the factory returns kv[:, 0] and kv[:, 1] like all dtypes. + # Split views are still needed for dequant verification. + key_scale_cache = None + value_scale_cache = None + nvfp4_key_data = None + nvfp4_value_data = None + if kv_cache_dtype == "nvfp4": + (nvfp4_key_data,), (key_scale_cache,) = nvfp4_kv_cache_split_views(key_cache) + (nvfp4_value_data,), (value_scale_cache,) = nvfp4_kv_cache_split_views( + value_cache + ) + + if kv_cache_dtype == "nvfp4": + # Global scale = amax / 448 (per-tensor) + k_scale = (key.abs().amax() / 448.0).to(torch.float32) + v_scale = (value.abs().amax() / 448.0).to(torch.float32) + elif kv_scale_type == "tensor": + k_scale = (key.amax() / 64.0).to(torch.float32) + v_scale = (value.amax() / 64.0).to(torch.float32) + else: # "attn_head" + k_scale = (key.amax(dim=(0, 2)) / 64.0).to(torch.float32) + v_scale = (value.amax(dim=(0, 2)) / 64.0).to(torch.float32) + + def permute_and_compact(x): + y = x if kv_cache_layout == "NHD" else x.permute(0, 2, 1, 3) + return y.contiguous() + + if kv_cache_dtype != "nvfp4": + key_cache_compact = permute_and_compact(key_cache) + value_cache_compact = permute_and_compact(value_cache) + + def convert_fp8_local(output, input, scale, kv_dtype): + fp8_input = input.view(current_platform.fp8_dtype()) + if scale.numel() == 1: # per-tensor + result = scaled_dequantize( + fp8_input.flatten(0, 2), scale, group_shape=None, out_dtype=output.dtype + ).reshape(*input.shape) + else: # per-head: broadcast scale along the head dimension + # Original code uses dim 2 for NHD, dim 1 for HND + if kv_cache_layout == "NHD": + result = fp8_input.to(output.dtype) * scale.view(1, 1, -1, 1) + else: + result = fp8_input.to(output.dtype) * scale.view(1, -1, 1, 1) + output.copy_(result) + + # Clone the KV caches (for non-nvfp4, used as reference baseline). + if kv_cache_dtype == "fp8": + cloned_key_cache = torch.empty_like(key_cache_compact, dtype=torch.float16) + convert_fp8_local(cloned_key_cache, key_cache_compact, k_scale, kv_cache_dtype) + cloned_value_cache = torch.empty_like(value_cache_compact, dtype=torch.float16) + convert_fp8_local( + cloned_value_cache, value_cache_compact, v_scale, kv_cache_dtype + ) + elif kv_cache_dtype != "nvfp4": + cloned_key_cache = key_cache_compact.clone() + cloned_value_cache = value_cache_compact.clone() + + # Call the reshape_and_cache kernel. + if implementation == "cuda": + if kv_cache_dtype != "nvfp4": + opcheck( + torch.ops._C_cache_ops.reshape_and_cache_flash, + ( + key, + value, + key_cache, + value_cache, + slot_mapping, + kv_cache_dtype, + k_scale, + v_scale, + ), + cond=(head_size == HEAD_SIZES[0]), + ) + ops.reshape_and_cache_flash( + key, + value, + key_cache, + value_cache, + slot_mapping, + kv_cache_dtype, + k_scale, + v_scale, + ) + elif implementation == "triton": + from vllm.v1.attention.ops.triton_reshape_and_cache_flash import ( + triton_reshape_and_cache_flash, + ) + + triton_reshape_and_cache_flash( + key, + value, + key_cache, + value_cache, + slot_mapping, + kv_cache_dtype, + k_scale, + v_scale, + ) + + if kv_cache_dtype == "nvfp4": + # Verify NVFP4 by dequantizing the entire cache and comparing + # the written positions against original bf16 values. + # Same pattern as FP8: dequant whole cache, then extract and compare. + from tests.kernels.quantization.nvfp4_utils import ( + dequant_nvfp4_kv_cache, + ) + + def dequant_nvfp4_cache_nhd(data_cache, scale_cache, global_scale): + # data_cache: [N, T, H, data_dim] NHD (contiguous inner dims) + # scale_cache: [N, T, H, scale_dim] NHD (contiguous inner dims) + # Permute to HND layout for the dequant utility. + data_hnd = data_cache.permute(0, 2, 1, 3) + scale_hnd = scale_cache.permute(0, 2, 1, 3) + result_hnd = dequant_nvfp4_kv_cache( + data_hnd, scale_hnd, global_scale, head_size, block_size + ) + return result_hnd.permute(0, 2, 1, 3) # back to [N, T, H, D] + + result_key_cache = dequant_nvfp4_cache_nhd( + nvfp4_key_data, key_scale_cache, k_scale.item() + ) + result_value_cache = dequant_nvfp4_cache_nhd( + nvfp4_value_data, value_scale_cache, v_scale.item() + ) + + # Flatten [num_blocks, block_size] → [num_slots] and index by slot_mapping. + num_slots = num_blocks * block_size + result_key_flat = result_key_cache.reshape(num_slots, num_heads, head_size) + result_value_flat = result_value_cache.reshape(num_slots, num_heads, head_size) + + torch.testing.assert_close( + result_key_flat[slot_mapping], key.float(), atol=1.5, rtol=0.5 + ) + torch.testing.assert_close( + result_value_flat[slot_mapping], value.float(), atol=1.5, rtol=0.5 + ) + return + + key_cache_compact = permute_and_compact(key_cache) + value_cache_compact = permute_and_compact(value_cache) + + if kv_cache_dtype == "fp8": + result_key_cache = torch.empty_like(key_cache_compact, dtype=torch.float16) + convert_fp8_local(result_key_cache, key_cache_compact, k_scale, kv_cache_dtype) + result_value_cache = torch.empty_like(value_cache_compact, dtype=torch.float16) + convert_fp8_local( + result_value_cache, + value_cache_compact, + v_scale, + kv_cache_dtype, + ) + + # Run the reference implementation. + block_indices = torch.div(slot_mapping, block_size, rounding_mode="floor") + block_indices_lst = block_indices.cpu().tolist() + block_offsets = slot_mapping % block_size + block_offsets_lst = block_offsets.cpu().tolist() + for i in range(num_tokens): + block_idx = block_indices_lst[i] + block_offset = block_offsets_lst[i] + if kv_cache_layout == "NHD": + cloned_key_cache[block_idx, block_offset, :, :] = key[i] + cloned_value_cache[block_idx, block_offset, :, :] = value[i] + else: + cloned_key_cache[block_idx, :, block_offset, :] = key[i] + cloned_value_cache[block_idx, :, block_offset, :] = value[i] + + if kv_cache_dtype == "fp8": + torch.testing.assert_close( + result_key_cache, cloned_key_cache, atol=0.001, rtol=0.1 + ) + torch.testing.assert_close( + result_value_cache, cloned_value_cache, atol=0.001, rtol=0.1 + ) + else: + torch.testing.assert_close(key_cache_compact, cloned_key_cache) + torch.testing.assert_close(value_cache_compact, cloned_value_cache) + + +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("kv_cache_dtype", KV_CACHE_DTYPE) +@pytest.mark.parametrize("kv_cache_layout", CACHE_LAYOUTS) +@pytest.mark.parametrize("implementation", RESHAPE_FLASH_IMPLEMENTATIONS) +@torch.inference_mode() +def test_reshape_and_cache_flash_unaligned_rows( + kv_cache_factory_flashinfer, + dtype: torch.dtype, + kv_cache_dtype: str, + kv_cache_layout: str, + implementation: str, +) -> None: + """Regression test for https://github.com/vllm-project/vllm/issues/41257. + + head_size=46 with num_heads=13 places KV-cache rows at byte offsets + that are not a multiple of the vector width (NHD row pitch + 13*46*itemsize, HND head pitch 46*itemsize), unlike HEAD_SIZES above + which are all 16-byte multiples. The CUDA kernel used to issue + vectorized stores to those rows -> CUDA misaligned address. + """ + test_reshape_and_cache_flash( + kv_cache_factory_flashinfer, + num_tokens=42, + num_heads=13, + head_size=46, + block_size=16, + num_blocks=128, + dtype=dtype, + seed=0, + device=CUDA_DEVICES[0], + kv_cache_dtype=kv_cache_dtype, + kv_cache_layout=kv_cache_layout, + kv_scale_type="tensor", + implementation=implementation, + ) + + +@pytest.mark.parametrize("direction", COPYING_DIRECTION) +@pytest.mark.parametrize("num_mappings", NUM_MAPPINGS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("kv_cache_dtype", KV_CACHE_DTYPE) +@torch.inference_mode() +def test_swap_blocks( + kv_cache_factory, + direction: tuple[str, str], + num_mappings: int, + num_heads: int, + head_size: int, + block_size: int, + num_blocks: int, + dtype: torch.dtype, + seed: int, + device: str, + kv_cache_dtype: str, +) -> None: + if kv_cache_dtype == "fp8" and "cpu" in direction: + pytest.skip() + if kv_cache_dtype == "fp8" and head_size % 16: + pytest.skip() + + set_random_seed(seed) + + src_device = device if direction[0] == "cuda" else "cpu" + dst_device = device if direction[1] == "cuda" else "cpu" + + src_blocks = random.sample(range(num_blocks), num_mappings) + # For the same device, mapping must not overlap + if src_device == dst_device: + remaining_blocks = list(set(range(num_blocks)) - set(src_blocks)) + dst_blocks = random.sample(remaining_blocks, num_mappings) + else: + dst_blocks = random.sample(range(num_blocks), num_mappings) + + block_mapping = list(zip(src_blocks, dst_blocks)) + block_mapping_tensor = torch.tensor( + block_mapping, dtype=torch.int64, device="cpu" + ).view(-1, 2) + + # Create the KV caches on the first device. + src_key_caches, src_value_caches = kv_cache_factory( + num_blocks, + block_size, + 1, + num_heads, + head_size, + kv_cache_dtype, + dtype, + seed, + src_device, + ) + + # Create the KV caches on the second device. + dist_key_caches, dist_value_caches = kv_cache_factory( + num_blocks, + block_size, + 1, + num_heads, + head_size, + kv_cache_dtype, + dtype, + seed, + dst_device, + ) + + src_key_caches_clone = src_key_caches[0].clone() + src_value_caches_clone = src_value_caches[0].clone() + + # Call the swap_blocks kernel. + do_opcheck = head_size == HEAD_SIZES[0] + src_cache = src_key_caches[0] + block_size_in_bytes = src_cache.element_size() * src_cache.stride(0) + opcheck( + torch.ops._C_cache_ops.swap_blocks, + ( + src_key_caches[0], + dist_key_caches[0], + block_size_in_bytes, + block_mapping_tensor, + ), + cond=do_opcheck, + ) + opcheck( + torch.ops._C_cache_ops.swap_blocks, + ( + src_value_caches[0], + dist_value_caches[0], + block_size_in_bytes, + block_mapping_tensor, + ), + cond=do_opcheck, + ) + + ops.swap_blocks( + src_key_caches[0], + dist_key_caches[0], + block_size_in_bytes, + block_mapping_tensor, + ) + ops.swap_blocks( + src_value_caches[0], + dist_value_caches[0], + block_size_in_bytes, + block_mapping_tensor, + ) + + for src, dst in block_mapping: + torch.testing.assert_close( + src_key_caches_clone[src].cpu(), dist_key_caches[0][dst].cpu() + ) + torch.testing.assert_close( + src_value_caches_clone[src].cpu(), dist_value_caches[0][dst].cpu() + ) + + +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@torch.inference_mode() +def test_fp8_e4m3_conversion( + num_heads: int, + head_size: int, + block_size: int, + num_blocks: int, + dtype: torch.dtype, + seed: int, + device: str, +) -> None: + set_random_seed(seed) + + low = -224.0 + high = 224.0 + shape = (num_blocks, num_heads, head_size, block_size) + cache = torch.empty(shape, dtype=dtype, device=device) + cache.uniform_(low, high) + + cache_fp8 = torch.empty_like(cache, dtype=torch.uint8) + ops.convert_fp8(cache_fp8, cache) + + converted_cache = torch.empty_like(cache) + ops.convert_fp8(converted_cache, cache_fp8) + + torch.testing.assert_close(cache, converted_cache, atol=0.001, rtol=0.1) + + +def _create_mla_cache( + num_blocks: int, + block_size: int, + entry_size: int, + dtype: torch.dtype, + kv_cache_dtype: str, + device: str, +) -> torch.Tensor: + cache_dtype = torch.uint8 if kv_cache_dtype == "fp8" else dtype + return torch.zeros( + num_blocks, block_size, entry_size, dtype=cache_dtype, device=device + ) + + +def _fill_mla_cache(cache: torch.Tensor, kv_cache_dtype: str): + rand_dtype = torch.float16 if kv_cache_dtype == "fp8" else cache.dtype + + vals = torch.randn(*cache.shape, device=cache.device, dtype=rand_dtype) + if kv_cache_dtype == "fp8": + temp = torch.zeros_like(cache) + ops.convert_fp8(temp, vals, 1.0, kv_dtype=kv_cache_dtype) + vals = temp + cache.copy_(vals) + + +@pytest.mark.parametrize("kv_lora_rank", KV_LORA_RANKS) +@pytest.mark.parametrize("qk_rope_head_dim", QK_ROPE_HEAD_DIMS) +@pytest.mark.parametrize("num_tokens", NUM_TOKENS_MLA) +@pytest.mark.parametrize("block_size", BLOCK_SIZES_MLA) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS_MLA) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("kv_cache_dtype", KV_CACHE_DTYPE) +@torch.inference_mode() +def test_concat_and_cache_mla( + kv_lora_rank: int, + qk_rope_head_dim: int, + num_tokens: int, + block_size: int, + num_blocks: int, + dtype: torch.dtype, + seed: int, + device: str, + kv_cache_dtype: str, +) -> None: + set_random_seed(seed) + torch.set_default_device(device) + torch.accelerator.set_device_index(device) + + total_slots = num_blocks * block_size + slot_mapping_lst = random.sample(range(total_slots), num_tokens) + slot_mapping = torch.tensor(slot_mapping_lst, dtype=torch.long, device=device) + + kv_c = torch.randn(num_tokens, kv_lora_rank, dtype=dtype, device=device) + k_pe = torch.randn(num_tokens, qk_rope_head_dim, dtype=dtype, device=device) + entry_size = kv_lora_rank + qk_rope_head_dim + + scale = torch.tensor(0.1, dtype=torch.float32, device=device) + kv_cache = _create_mla_cache( + num_blocks, block_size, entry_size, dtype, kv_cache_dtype, device + ) + ref_temp = torch.zeros(*kv_cache.shape, dtype=dtype, device=device) + + for i in range(num_tokens): + slot = slot_mapping[i].item() + block_idx = slot // block_size + block_offset = slot % block_size + ref_temp[block_idx, block_offset, :kv_lora_rank] = kv_c[i] + ref_temp[block_idx, block_offset, kv_lora_rank:] = k_pe[i] + + if kv_cache_dtype == "fp8": + ref_kv_cache = torch.empty_like(ref_temp, dtype=kv_cache.dtype) + ops.convert_fp8(ref_kv_cache, ref_temp, scale.item(), kv_dtype=kv_cache_dtype) + else: + ref_kv_cache = ref_temp + + opcheck( + torch.ops._C_cache_ops.concat_and_cache_mla, + (kv_c, k_pe, kv_cache, slot_mapping, kv_cache_dtype, scale), + test_utils=DEFAULT_OPCHECK_TEST_UTILS, + ) + + ops.concat_and_cache_mla(kv_c, k_pe, kv_cache, slot_mapping, kv_cache_dtype, scale) + + if kv_cache_dtype == "fp8": + result_temp = torch.empty_like(kv_cache, dtype=torch.float16) + ops.convert_fp8( + result_temp, kv_cache.contiguous(), scale.item(), kv_dtype=kv_cache_dtype + ) + expected_temp = torch.empty_like(ref_kv_cache, dtype=torch.float16) + ops.convert_fp8( + expected_temp, ref_kv_cache, scale.item(), kv_dtype=kv_cache_dtype + ) + torch.testing.assert_close(result_temp, expected_temp, atol=0.001, rtol=0.1) + else: + torch.testing.assert_close(kv_cache, ref_kv_cache) + + +@pytest.mark.parametrize("kv_lora_rank", KV_LORA_RANKS) +@pytest.mark.parametrize("qk_rope_head_dim", QK_ROPE_HEAD_DIMS) +@pytest.mark.parametrize("num_tokens", NUM_TOKENS_MLA) +@pytest.mark.parametrize("block_size", BLOCK_SIZES_MLA) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS_MLA) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@torch.inference_mode() +def test_concat_and_cache_ds_mla( + kv_lora_rank: int, + qk_rope_head_dim: int, + num_tokens: int, + block_size: int, + num_blocks: int, + dtype: torch.dtype, + seed: int, + device: str, +) -> None: + if current_platform.is_rocm(): + pytest.skip("concat_and_cache_mla doesn't support fp8_ds_mla on ROCm") + if dtype.itemsize != 2: + pytest.skip("ds_mla only supports 16-bit input") + if kv_lora_rank != 512: + pytest.skip("fp8_ds_mla requires kv_lora_rank == 512") + kv_cache_dtype = "fp8_ds_mla" + set_random_seed(seed) + torch.set_default_device(device) + torch.accelerator.set_device_index(device) + + total_slots = num_blocks * block_size + slot_mapping_lst = random.sample(range(total_slots), num_tokens) + slot_mapping = torch.tensor(slot_mapping_lst, dtype=torch.long, device=device) + + kv_c = torch.randn(num_tokens, kv_lora_rank, dtype=dtype, device=device) + k_pe = torch.randn(num_tokens, qk_rope_head_dim, dtype=dtype, device=device) + entry_size = kv_lora_rank + (4 * 4) + (2 * qk_rope_head_dim) + + scale = torch.tensor(1.0, dtype=torch.float32, device=device) + kv_cache = _create_mla_cache( + num_blocks, + block_size, + entry_size, + dtype=torch.uint8, + kv_cache_dtype=kv_cache_dtype, + device=device, + ) + + ref_cache = torch.zeros_like(kv_cache, dtype=kv_cache.dtype) + tile_data = torch.zeros(128, dtype=dtype, device=device) + + for i in range(num_tokens): + slot = slot_mapping[i].item() + block_idx = slot // block_size + block_offset = slot % block_size + + ref_cache_slice = ref_cache[block_idx, block_offset] + ref_cache_16bit = ref_cache_slice.view(dtype) + ref_cache_32bit = ref_cache_slice.view(torch.float32) + + kv_c_data = kv_c[i] + num_tiles = kv_lora_rank // 128 + for tile_idx in range(num_tiles): + tile_start = tile_idx * 128 + tile_end = (tile_idx + 1) * 128 + tile_data[:] = kv_c_data[tile_start:tile_end] + + # tile_scale = tile_data.amax().to(torch.float32) / 448. + # NOTE: Using torch's amax() gives different results, + # so this must be manually computed. + tile_data_float = tile_data.to(torch.float32) + manual_max = abs(tile_data_float[0]) + for j in range(1, 128): + manual_max = max(manual_max, abs(tile_data_float[j])) + tile_scale = manual_max / 448.0 + + ref_cache_32bit[kv_lora_rank // 4 + tile_idx] = tile_scale + + ops.convert_fp8( + ref_cache_slice[tile_start:tile_end], + tile_data, + tile_scale.item(), + kv_dtype="fp8", + ) + + for j in range(qk_rope_head_dim): + ref_cache_16bit[kv_lora_rank // 2 + 8 + j] = k_pe[i, j] + + opcheck( + torch.ops._C_cache_ops.concat_and_cache_mla, + (kv_c, k_pe, kv_cache, slot_mapping, kv_cache_dtype, scale), + test_utils=DEFAULT_OPCHECK_TEST_UTILS, + ) + + ops.concat_and_cache_mla(kv_c, k_pe, kv_cache, slot_mapping, kv_cache_dtype, scale) + + for i in range(num_tokens): + slot = slot_mapping[i].item() + block_idx = slot // block_size + block_offset = slot % block_size + kv_cache_slice = kv_cache[block_idx, block_offset] + ref_cache_slice = ref_cache[block_idx, block_offset] + + kv_nope = kv_cache_slice[:kv_lora_rank] + ref_nope = ref_cache_slice[:kv_lora_rank] + kv_scales = kv_cache_slice.view(torch.float32)[ + kv_lora_rank // 4 : kv_lora_rank // 4 + 4 + ] + ref_scales = ref_cache_slice.view(torch.float32)[ + kv_lora_rank // 4 : kv_lora_rank // 4 + 4 + ] + kv_rope = kv_cache_slice.view(dtype)[kv_lora_rank // 2 + 8 :] + ref_rope = ref_cache_slice.view(dtype)[kv_lora_rank // 2 + 8 :] + + torch.testing.assert_close(kv_nope, ref_nope, atol=0.001, rtol=0.1) + torch.testing.assert_close(kv_scales, ref_scales, atol=0.001, rtol=0.1) + torch.testing.assert_close(kv_rope, ref_rope, atol=0.001, rtol=0.1) + + +@pytest.mark.parametrize("kv_lora_rank", KV_LORA_RANKS) +@pytest.mark.parametrize("qk_rope_head_dim", QK_ROPE_HEAD_DIMS) +@pytest.mark.parametrize("block_size", BLOCK_SIZES_MLA) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS_MLA) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("kv_cache_dtype", KV_CACHE_DTYPE) +@torch.inference_mode() +def test_swap_blocks_mla( + kv_lora_rank: int, + qk_rope_head_dim: int, + block_size: int, + num_blocks: int, + dtype: torch.dtype, + seed: int, + device: str, + kv_cache_dtype: str, +) -> None: + set_random_seed(seed) + torch.set_default_device(device) + torch.accelerator.set_device_index(device) + + entry_size = kv_lora_rank + qk_rope_head_dim + + src_cache = _create_mla_cache( + num_blocks, block_size, entry_size, dtype, kv_cache_dtype, device + ) + dst_cache = _create_mla_cache( + num_blocks, block_size, entry_size, dtype, kv_cache_dtype, device + ) + + _fill_mla_cache(src_cache, kv_cache_dtype) + _fill_mla_cache(dst_cache, kv_cache_dtype) + + src_cache_clone = src_cache.clone() + + num_mappings = min(2, num_blocks // 2) + src_blocks = random.sample(range(num_blocks), num_mappings) + remaining_blocks = list(set(range(num_blocks)) - set(src_blocks)) + dst_blocks = random.sample(remaining_blocks, num_mappings) + block_mapping = list(zip(src_blocks, dst_blocks)) + block_mapping_tensor = torch.tensor( + block_mapping, dtype=torch.int64, device="cpu" + ).view(-1, 2) + + block_size_in_bytes = src_cache.element_size() * src_cache.stride(0) + opcheck( + torch.ops._C_cache_ops.swap_blocks, + (src_cache, dst_cache, block_size_in_bytes, block_mapping_tensor), + test_utils=DEFAULT_OPCHECK_TEST_UTILS, + ) + + ops.swap_blocks(src_cache, dst_cache, block_size_in_bytes, block_mapping_tensor) + + for src, dst in block_mapping: + torch.testing.assert_close( + src_cache_clone[src].cpu(), + dst_cache[dst].cpu(), + msg=f"Block {src} from src should have been swapped to block " + f"{dst} in dst_cache.", + ) + + +@pytest.mark.parametrize("kv_lora_rank", [512]) +@pytest.mark.parametrize("qk_rope_head_dim", [64]) +@pytest.mark.parametrize("block_size", [16]) +@pytest.mark.parametrize("num_blocks", [1024]) +@pytest.mark.parametrize("max_seq_len", [512]) +@pytest.mark.parametrize("batch_size", [8]) +@pytest.mark.parametrize("dtype", [torch.float32]) +@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8"]) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@torch.inference_mode() +def test_gather_and_maybe_dequant_cache_mla( + kv_lora_rank, + qk_rope_head_dim, + block_size, + num_blocks, + max_seq_len, + batch_size, + dtype, + kv_cache_dtype, + device, +): + entry_size = kv_lora_rank + qk_rope_head_dim + scale = torch.tensor(0.1, dtype=torch.float32, device=device) + src_cache = _create_mla_cache( + num_blocks, block_size, entry_size, dtype, kv_cache_dtype, device + ) + _fill_mla_cache(src_cache, kv_cache_dtype=kv_cache_dtype) + + seq_len_tensor = torch.randint( + max_seq_len, max_seq_len + 1, (batch_size,), device=device + ) + + total_tokens = seq_len_tensor.sum() + cu_seq_lens = torch.empty((batch_size + 1), dtype=torch.int32, device=device) + cu_seq_lens[0] = 0 + cu_seq_lens[1:] = seq_len_tensor.cumsum(dim=0).to(dtype=torch.int32) + token_to_seq = torch.arange(0, batch_size, dtype=torch.int32, device=device) + token_to_seq = torch.repeat_interleave(token_to_seq, seq_len_tensor) + print("seq_len_tensor", seq_len_tensor) + + tot_blocks_tensor = (seq_len_tensor + block_size - 1) // block_size + block_table = torch.empty( + (batch_size, num_blocks), dtype=torch.int32, device=device + ) + + for b in range(batch_size): + perm = torch.randperm(num_blocks, device=device) + block_table[b, :] = perm + + dst = torch.zeros((total_tokens, entry_size), dtype=dtype, device=device) + + expected_batches = [] + for b in range(batch_size): + s = seq_len_tensor[b] + if s == 0: + continue + tot = tot_blocks_tensor[b] + blocks = block_table[b, :tot].tolist() + + gathered_rows = [] + for i in range(tot - 1): + block_data = src_cache[blocks[i]] + if kv_cache_dtype == "fp8": + dequantized_block = torch.empty_like(block_data, dtype=dtype) + ops.convert_fp8(dequantized_block, block_data, scale.item()) + gathered_rows.append(dequantized_block) + else: + gathered_rows.append(block_data) + remaining = s - (tot - 1) * block_size + last_block_data = src_cache[blocks[-1], :remaining, :] + if kv_cache_dtype == "fp8": + dequantized_last_block = torch.empty_like(last_block_data, dtype=dtype) + ops.convert_fp8(dequantized_last_block, last_block_data, scale.item()) + gathered_rows.append(dequantized_last_block) + else: + gathered_rows.append(last_block_data) + + batch_expected = torch.cat(gathered_rows, dim=0) + expected_batches.append(batch_expected) + expected = torch.cat(expected_batches, dim=0) + + opcheck( + torch.ops._C_cache_ops.gather_and_maybe_dequant_cache, + ( + src_cache, + dst, + block_table, + cu_seq_lens, + token_to_seq, + total_tokens, + kv_cache_dtype, + scale, + None, + ), + test_utils=DEFAULT_OPCHECK_TEST_UTILS, + ) + + ops.gather_and_maybe_dequant_cache( + src_cache, + dst, + block_table, + cu_seq_lens, + token_to_seq, + total_tokens, + kv_cache_dtype, + scale, + None, + ) + torch.testing.assert_close(dst, expected) + + +@pytest.mark.parametrize("kv_lora_rank", [512]) +@pytest.mark.parametrize("qk_rope_head_dim", [64]) +@pytest.mark.parametrize("block_size", [16]) +@pytest.mark.parametrize("num_blocks", [128]) +@pytest.mark.parametrize("dtype", [torch.float32]) +@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8"]) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@torch.inference_mode() +def test_gather_and_maybe_dequant_cache_mla_with_seq_starts( + kv_lora_rank, + qk_rope_head_dim, + block_size, + num_blocks, + dtype, + kv_cache_dtype, + device, +): + entry_size = kv_lora_rank + qk_rope_head_dim + scale = torch.tensor(0.1, dtype=torch.float32, device=device) + src_cache = _create_mla_cache( + num_blocks, block_size, entry_size, dtype, kv_cache_dtype, device + ) + _fill_mla_cache(src_cache, kv_cache_dtype=kv_cache_dtype) + + seq_starts = torch.tensor([3, 17, 5], dtype=torch.int32, device=device) + seq_lens = torch.tensor([20, 10, 16], dtype=torch.int32, device=device) + batch_size = seq_lens.shape[0] + total_tokens = seq_lens.sum().item() + cu_seq_lens = torch.empty((batch_size + 1), dtype=torch.int32, device=device) + cu_seq_lens[0] = 0 + cu_seq_lens[1:] = seq_lens.cumsum(dim=0) + token_to_seq = torch.repeat_interleave( + torch.arange(batch_size, dtype=torch.int32, device=device), seq_lens + ) + + block_table = torch.empty( + (batch_size, num_blocks), dtype=torch.int32, device=device + ) + for b in range(batch_size): + block_table[b, :] = torch.randperm(num_blocks, device=device) + + if kv_cache_dtype == "fp8": + dequant_src_cache = torch.empty_like(src_cache, dtype=dtype) + ops.convert_fp8(dequant_src_cache, src_cache, scale.item()) + else: + dequant_src_cache = src_cache + + expected_rows = [] + for b in range(batch_size): + start = seq_starts[b].item() + length = seq_lens[b].item() + for offset in range(start, start + length): + block_id = block_table[b, offset // block_size] + slot = offset % block_size + expected_rows.append(dequant_src_cache[block_id, slot]) + expected = torch.stack(expected_rows) + + dst = torch.zeros((total_tokens, entry_size), dtype=dtype, device=device) + ops.gather_and_maybe_dequant_cache( + src_cache, + dst, + block_table, + cu_seq_lens, + token_to_seq, + total_tokens, + kv_cache_dtype, + scale, + seq_starts, + ) + torch.testing.assert_close(dst, expected) + + +@pytest.mark.parametrize("kv_lora_rank", [512]) +@pytest.mark.parametrize("qk_rope_head_dim", [64]) +@pytest.mark.parametrize("block_size", [16]) +@pytest.mark.parametrize("num_blocks", [1024]) +@pytest.mark.parametrize("max_seq_len", [512]) +@pytest.mark.parametrize("batch_size", [8]) +@pytest.mark.parametrize("dtype", [torch.float32]) +@pytest.mark.parametrize( + "kv_cache_dtype", ["auto"] +) # You can also test "fp8" if needed. +@pytest.mark.parametrize("device", CUDA_DEVICES) +@torch.inference_mode() +def test_cp_gather_cache_mla( + kv_lora_rank, + qk_rope_head_dim, + block_size, + num_blocks, + max_seq_len, + batch_size, + dtype, + kv_cache_dtype, + device, +): + entry_size = kv_lora_rank + qk_rope_head_dim + src_cache = _create_mla_cache( + num_blocks, block_size, entry_size, dtype, kv_cache_dtype, device + ) + _fill_mla_cache(src_cache, kv_cache_dtype=kv_cache_dtype) + + seq_len_tensor = torch.randint(0, max_seq_len + 1, (batch_size,), device=device) + + total_tokens = seq_len_tensor.sum() + cu_seq_lens = torch.empty((batch_size + 1), dtype=torch.int32, device=device) + cu_seq_lens[0] = 0 + cu_seq_lens[1:] = seq_len_tensor.cumsum(dim=0).to(dtype=torch.int32) + print("seq_len_tensor", seq_len_tensor) + + tot_blocks_tensor = (seq_len_tensor + block_size - 1) // block_size + block_table = torch.empty( + (batch_size, num_blocks), dtype=torch.int32, device=device + ) + + for b in range(batch_size): + perm = torch.randperm(num_blocks, device=device) + block_table[b, :] = perm + + dst = torch.zeros((total_tokens, entry_size), dtype=src_cache.dtype, device=device) + + expected_batches = [] + for b in range(batch_size): + s = seq_len_tensor[b] + if s == 0: + continue + tot = tot_blocks_tensor[b] + blocks = block_table[b, :tot].tolist() + + gathered_rows = [] + for i in range(tot - 1): + gathered_rows.append(src_cache[blocks[i]]) + remaining = s - (tot - 1) * block_size + gathered_rows.append(src_cache[blocks[-1], :remaining, :]) + + batch_expected = torch.cat(gathered_rows, dim=0) + expected_batches.append(batch_expected) + expected = torch.cat(expected_batches, dim=0) + + opcheck( + torch.ops._C_cache_ops.cp_gather_cache, + (src_cache, dst, block_table, cu_seq_lens, batch_size, None), + test_utils=DEFAULT_OPCHECK_TEST_UTILS, + ) + + ops.cp_gather_cache(src_cache, dst, block_table, cu_seq_lens, batch_size) + torch.testing.assert_close(dst, expected) + + +@pytest.mark.parametrize("kv_lora_rank", KV_LORA_RANKS) +@pytest.mark.parametrize("qk_rope_head_dim", QK_ROPE_HEAD_DIMS) +@pytest.mark.parametrize("num_tokens", NUM_TOKENS_MLA) +@pytest.mark.parametrize("block_size", BLOCK_SIZES_MLA) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS_MLA) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.cpu_model +@pytest.mark.skipif(not current_platform.is_cpu(), reason="CPU only") +@torch.inference_mode() +def test_concat_and_cache_mla_cpu( + kv_lora_rank: int, + qk_rope_head_dim: int, + num_tokens: int, + block_size: int, + num_blocks: int, + dtype: torch.dtype, + seed: int, +) -> None: + device = "cpu" + kv_cache_dtype = "auto" + set_random_seed(seed) + torch.set_default_device(device) + + total_slots = num_blocks * block_size + slot_mapping_lst = random.sample(range(total_slots), num_tokens) + slot_mapping = torch.tensor(slot_mapping_lst, dtype=torch.long, device=device) + + kv_c = torch.randn(num_tokens, kv_lora_rank, dtype=dtype, device=device) + k_pe = torch.randn(num_tokens, qk_rope_head_dim, dtype=dtype, device=device) + entry_size = kv_lora_rank + qk_rope_head_dim + + scale = torch.tensor(0.1, dtype=torch.float32, device=device) + kv_cache = _create_mla_cache( + num_blocks, block_size, entry_size, dtype, kv_cache_dtype, device + ) + ref_temp = torch.zeros(*kv_cache.shape, dtype=dtype, device=device) + + for i in range(num_tokens): + slot = slot_mapping[i].item() + block_idx = slot // block_size + block_offset = slot % block_size + ref_temp[block_idx, block_offset, :kv_lora_rank] = kv_c[i] + ref_temp[block_idx, block_offset, kv_lora_rank:] = k_pe[i] + + if kv_cache_dtype == "fp8": + ref_kv_cache = torch.empty_like(ref_temp, dtype=kv_cache.dtype) + ops.convert_fp8(ref_kv_cache, ref_temp, scale.item(), kv_dtype=kv_cache_dtype) + else: + ref_kv_cache = ref_temp + + opcheck( + torch.ops._C_cache_ops.concat_and_cache_mla, + (kv_c, k_pe, kv_cache, slot_mapping, kv_cache_dtype, scale), + test_utils=DEFAULT_OPCHECK_TEST_UTILS, + ) + + ops.concat_and_cache_mla(kv_c, k_pe, kv_cache, slot_mapping, kv_cache_dtype, scale) + torch.testing.assert_close(kv_cache, ref_kv_cache) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_cascade_flash_attn.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_cascade_flash_attn.py new file mode 100644 index 0000000000000000000000000000000000000000..540a8bc65c4b4476a58e1aa025a675013e4cf641 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_cascade_flash_attn.py @@ -0,0 +1,187 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import pytest +import torch + +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.attention.backends.flash_attn import cascade_attention, merge_attn_states + +try: + from vllm.vllm_flash_attn import ( + fa_version_unsupported_reason, + flash_attn_varlen_func, + is_fa_version_supported, + ) +except ImportError: + if current_platform.is_rocm(): + pytest.skip( + "vllm_flash_attn is not supported for vLLM on ROCm.", + allow_module_level=True, + ) + +NUM_HEADS = [(4, 4), (8, 2), (16, 2)] +HEAD_SIZES = [128, 192, 256] +BLOCK_SIZES = [16] +DTYPES = [torch.float16, torch.bfloat16] + + +@pytest.mark.parametrize("num_tokens", [1, 39, 16912]) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@torch.inference_mode() +def test_merge_kernel( + num_tokens: int, + num_heads: tuple[int, int], + head_size: int, + dtype: torch.dtype, +): + torch.set_default_device("cuda") + set_random_seed(0) + num_query_heads = num_heads[0] + num_kv_heads = num_heads[1] + assert num_query_heads % num_kv_heads == 0 + + # Prepare inputs. + prefix_output = torch.randn(num_tokens, num_query_heads, head_size, dtype=dtype) + suffix_output = torch.randn(num_tokens, num_query_heads, head_size, dtype=dtype) + prefix_lse = torch.randn(num_query_heads, num_tokens, dtype=torch.float32) + suffix_lse = torch.randn(num_query_heads, num_tokens, dtype=torch.float32) + + # Run the kernel. + output = torch.empty(num_tokens, num_query_heads, head_size, dtype=dtype) + merge_attn_states(output, prefix_output, prefix_lse, suffix_output, suffix_lse) + + # Reference implementation. + max_lse = torch.maximum(prefix_lse, suffix_lse) + p_lse = torch.exp(prefix_lse - max_lse) + s_lse = torch.exp(suffix_lse - max_lse) + p_scale = p_lse / (p_lse + s_lse) + s_scale = s_lse / (p_lse + s_lse) + p_scale = p_scale.transpose(0, 1).unsqueeze(2) + s_scale = s_scale.transpose(0, 1).unsqueeze(2) + ref_output = p_scale * prefix_output + s_scale * suffix_output + ref_output = ref_output.to(dtype) + + # Compare the results. + torch.testing.assert_close(output, ref_output, atol=1e-2, rtol=1e-2) + + +CASES = [ + # Case 1. A general case. + ([(129, 871), (18, 280), (37, 988), (1023, 2304), (1, 257)], 256), + # Case 2. Flash-decoding case. + ([(1, 1023), (1, 879), (1, 778), (1, 1777)] * 100, 512), +] + + +@pytest.mark.parametrize("seq_lens_and_common_prefix", CASES) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("soft_cap", [None, 50]) +@pytest.mark.parametrize("num_blocks", [2048]) +@pytest.mark.parametrize("fa_version", [2, 3]) +@torch.inference_mode() +def test_cascade( + seq_lens_and_common_prefix: tuple[list[tuple[int, int]], int], + num_heads: tuple[int, int], + head_size: int, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + num_blocks: int, + fa_version: int, +) -> None: + torch.set_default_device("cuda") + if not is_fa_version_supported(fa_version): + pytest.skip( + f"Flash attention version {fa_version} not supported due " + f'to: "{fa_version_unsupported_reason(fa_version)}"' + ) + + set_random_seed(0) + + window_size = (-1, -1) + scale = head_size**-0.5 + num_query_heads = num_heads[0] + num_kv_heads = num_heads[1] + assert num_query_heads % num_kv_heads == 0 + key_cache = torch.randn( + num_blocks, block_size, num_kv_heads, head_size, dtype=dtype + ) + value_cache = torch.randn_like(key_cache) + + seq_lens, common_prefix_len = seq_lens_and_common_prefix + num_seqs = len(seq_lens) + query_lens = [x[0] for x in seq_lens] + kv_lens = [x[1] for x in seq_lens] + max_query_len = max(query_lens) + max_kv_len = max(kv_lens) + + total_num_query_tokens = sum(query_lens) + query = torch.randn(total_num_query_tokens, num_query_heads, head_size, dtype=dtype) + cu_query_lens = torch.tensor([0] + query_lens, dtype=torch.int32).cumsum( + dim=0, dtype=torch.int32 + ) + kv_lens_tensor = torch.tensor(kv_lens, dtype=torch.int32) + max_num_blocks_per_seq = (max_kv_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, num_blocks, (num_seqs, max_num_blocks_per_seq), dtype=torch.int32 + ) + + assert common_prefix_len > 0 + assert common_prefix_len % block_size == 0 + num_common_kv_blocks = common_prefix_len // block_size + # Make sure the first `num_common_kv_blocks` blocks are the same. + block_tables[:, :num_common_kv_blocks] = block_tables[0, :num_common_kv_blocks] + + # Run the regular attention. + ref_output = flash_attn_varlen_func( + q=query, + k=key_cache, + v=value_cache, + cu_seqlens_q=cu_query_lens, + seqused_k=kv_lens_tensor, + max_seqlen_q=max_query_len, + max_seqlen_k=max_kv_len, + softmax_scale=scale, + causal=True, + window_size=window_size, + block_table=block_tables, + softcap=soft_cap if soft_cap is not None else 0, + ) + + # Run cascade attention. + assert all(common_prefix_len < kv_len for kv_len in kv_lens) + cu_prefix_query_lens = torch.tensor([0, total_num_query_tokens], dtype=torch.int32) + prefix_kv_lens = torch.tensor([common_prefix_len], dtype=torch.int32) + suffix_kv_lens = kv_lens_tensor - common_prefix_len + output = torch.empty_like(query) + cascade_attention( + output=output, + query=query, + key_cache=key_cache, + value_cache=value_cache, + cu_query_lens=cu_query_lens, + max_query_len=max_query_len, + cu_prefix_query_lens=cu_prefix_query_lens, + prefix_kv_lens=prefix_kv_lens, + suffix_kv_lens=suffix_kv_lens, + max_kv_len=max_kv_len, + softmax_scale=scale, + alibi_slopes=None, + sliding_window=window_size, + logits_soft_cap=soft_cap if soft_cap is not None else 0, + block_table=block_tables, + common_prefix_len=common_prefix_len, + max_num_splits=0, # no max + fa_version=fa_version, + ) + + # Compare the results. + torch.testing.assert_close(output, ref_output, atol=1e-2, rtol=1e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_cpu_attn.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_cpu_attn.py new file mode 100644 index 0000000000000000000000000000000000000000..2cf924d009545066aac2e0e681c82e60bd757c57 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_cpu_attn.py @@ -0,0 +1,1112 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import functools +import math + +import pytest +import torch + +from vllm.platforms import CpuArchEnum, current_platform +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.attention.backends.cpu_attn import _get_attn_isa + +if not current_platform.is_cpu(): + pytest.skip("skipping CPU-only tests", allow_module_level=True) + +from vllm._custom_ops import ( + cpu_attention_with_kv_cache, + cpu_attn_get_scheduler_metadata, + cpu_attn_reshape_and_cache, +) + +# Enable AMX tile data registers so isolated runs (e.g. -k fp8_amx) don't rely +# on ref_paged_attn's einsum to trigger oneDNN's _init_amx() first. +if torch.cpu._is_amx_tile_supported(): + torch.cpu._init_amx() + +NUM_HEADS = [ + (4, 4), + (8, 2), + (9, 3), +] +HEAD_SIZES = [96, 128, 512] +HEAD_SIZES_VEC16 = [96, 80, 112, 128] +QTYPES = [torch.bfloat16, torch.half, torch.float32] +SLIDING_WINDOWS = [None, 256] +NUM_BLOCKS = [ + 1024, +] +SEQ_LENS = [ # (q_len, kv_len) + [(1, 213), (1, 1), (1, 312), (1, 7), (1, 7812)], # decode batch + [(2345, 2345), (5, 5), (3, 16), (134, 5131)], # prefill batch + [(992, 2456), (1, 1234), (98, 1145), (1, 4162), (2345, 2345)], # mixed batch +] +_FP8_ATOL = {"fp8_e4m3": 0.2, "fp8_e5m2": 0.3} +_FP8_RTOL = 0.1 +ENCODER_SEQ_LENS = [ + [1, 678, 2367, 145, 4162, 36, 7812], +] + + +def get_attn_isa( + block_size: int | None = None, + dtype: torch.dtype | None = None, +): + # Delegate to _get_attn_isa so the fallback path applies the same arch + # gating (e.g. RISC-V RVV is only chosen when the build's hardcoded + # VLEN=128 kernel is actually present; on VLEN=256 / scalar hosts it + # correctly falls through to vec/vec16). + return _get_attn_isa( + dtype if dtype is not None else torch.bfloat16, + block_size if block_size else 32, + ) + + +# rand number generation takes too much time, cache rand tensors +@functools.lru_cache(maxsize=128, typed=False) +def tensor_cache(elem_num: int, dtype: torch.dtype, tag: str = "none") -> torch.Tensor: + tensor = torch.randn(elem_num, dtype=dtype) + + return tensor + + +def _get_alibi_slopes(total_num_heads: int) -> torch.Tensor: + closest_power_of_2 = 2 ** math.floor(math.log2(total_num_heads)) + base = torch.tensor( + 2 ** (-(2 ** -(math.log2(closest_power_of_2) - 3))), + dtype=torch.float32, + ) + powers = torch.arange(1, 1 + closest_power_of_2, dtype=torch.int32) + slopes = torch.pow(base, powers) + + if closest_power_of_2 != total_num_heads: + extra_base = torch.tensor( + 2 ** (-(2 ** -(math.log2(2 * closest_power_of_2) - 3))), + dtype=torch.float32, + ) + num_remaining_heads = min( + closest_power_of_2, total_num_heads - closest_power_of_2 + ) + extra_powers = torch.arange( + start=1, end=1 + 2 * num_remaining_heads, step=2, dtype=torch.int32 + ) + slopes = torch.cat([slopes, torch.pow(extra_base, extra_powers)], dim=0) + return slopes.float() + + +def ref_paged_attn( + query: torch.Tensor, + key_cache: torch.Tensor, + value_cache: torch.Tensor, + query_lens: list[int], + kv_lens: list[int], + block_tables: torch.Tensor, + scale: float, + sliding_window: int | None = None, + soft_cap: float | None = None, + alibi_slopes: torch.Tensor | None = None, + s_aux: torch.Tensor | None = None, + dynamic_causal: list[bool] | None = None, +) -> torch.Tensor: + num_seqs = len(query_lens) + block_tables = block_tables.cpu().numpy() + _, block_size, num_kv_heads, head_size = key_cache.shape + dtype = query.dtype + + outputs: list[torch.Tensor] = [] + start_idx = 0 + + if alibi_slopes is not None: + alibi_slopes = alibi_slopes[:, None, None] + + if s_aux is not None: + s_aux = s_aux.float() + s_aux = s_aux[:, None, None] + + for i in range(num_seqs): + query_len = query_lens[i] + kv_len = kv_lens[i] + q = query[start_idx : start_idx + query_len].float() + q *= scale + + num_kv_blocks = (kv_len + block_size - 1) // block_size + block_indices = block_tables[i, :num_kv_blocks] + + k = key_cache[block_indices].view(-1, num_kv_heads, head_size) + k = k[:kv_len].float() + v = value_cache[block_indices].view(-1, num_kv_heads, head_size) + v = v[:kv_len].float() + + if q.shape[1] != k.shape[1]: + k = torch.repeat_interleave(k, q.shape[1] // k.shape[1], dim=1) + v = torch.repeat_interleave(v, q.shape[1] // v.shape[1], dim=1) + attn = torch.einsum("qhd,khd->hqk", q, k).float() + empty_mask = torch.ones(query_len, kv_len) + + if dynamic_causal is None or dynamic_causal[i]: + mask = torch.triu(empty_mask, diagonal=kv_len - query_len + 1).bool() + if sliding_window is not None: + sliding_window_mask = ( + torch.triu( + empty_mask, diagonal=kv_len - (query_len + sliding_window) + 1 + ) + .bool() + .logical_not() + ) + mask |= sliding_window_mask + else: + if sliding_window is not None: + mask = ( + torch.triu( + empty_mask, diagonal=1 - sliding_window + kv_len - query_len + ).bool() + ^ torch.triu( + empty_mask, diagonal=sliding_window + kv_len - query_len + ).bool() + ).logical_not() + else: + mask = empty_mask.logical_not() + + if soft_cap is not None: + attn = soft_cap * torch.tanh(attn / soft_cap) + + if alibi_slopes is not None: + q_start_pos = kv_len - query_len + q_pos = q_start_pos + torch.arange(0, query_len)[None, :, None] + kv_pos = torch.arange(0, kv_len)[None, None, :] + dist = q_pos - kv_pos + alibi_bias = -alibi_slopes * dist + attn += alibi_bias + + attn.masked_fill_(mask, float("-inf")) + + if s_aux is not None: + s_aux_ext = s_aux.repeat(1, query_len, 1) + attn = torch.cat((s_aux_ext, attn), dim=-1) + + attn = torch.softmax(attn, dim=-1) + + if s_aux is not None: + attn = attn[:, :, 1:] + + out = torch.einsum("hqk,khd->qhd", attn, v).to(dtype=dtype) + + outputs.append(out) + start_idx += query_len + + return torch.cat(outputs, dim=0) + + +def ref_varlen_encoder_attn( + query: torch.Tensor, # [token, q_head_num, head_dim] + key: torch.Tensor, # [token, kv_head_num, head_dim] + value: torch.Tensor, + seq_lens: list[int], + scale: float, + sliding_window: int | None = None, +) -> torch.Tensor: + num_seqs = len(seq_lens) + dtype = query.dtype + + output = torch.empty_like(query) + + start_idx = 0 + for i in range(num_seqs): + seq_len = seq_lens[i] + q = query[start_idx : start_idx + seq_len].float() + k = key[start_idx : start_idx + seq_len].float() + v = value[start_idx : start_idx + seq_len].float() + q *= scale + + if q.shape[1] != k.shape[1]: + k = torch.repeat_interleave(k, q.shape[1] // k.shape[1], dim=1) + v = torch.repeat_interleave(v, q.shape[1] // v.shape[1], dim=1) + attn = torch.einsum("qhd,khd->hqk", q, k).float() + empty_mask = torch.ones(seq_len, seq_len) + if sliding_window is not None: + mask = ( + torch.triu(empty_mask, diagonal=1 - sliding_window).bool() + ^ torch.triu(empty_mask, diagonal=sliding_window).bool() + ).logical_not() + else: + mask = empty_mask.logical_not() + + attn.masked_fill_(mask, float("-inf")) + attn = torch.softmax(attn, dim=-1) + out = torch.einsum("hqk,khd->qhd", attn, v).to(dtype=dtype) + output[start_idx : start_idx + seq_len].copy_(out) + + start_idx += seq_len + + return output + + +@torch.inference_mode() +def varlen_encoder_attention( + seq_lens: list[int], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + isa: str, +) -> None: + set_random_seed(0) + num_seqs = len(seq_lens) + num_query_heads = num_heads[0] + num_kv_heads = num_heads[1] + assert num_query_heads % num_kv_heads == 0 + scale = head_size**-0.5 + token_num = sum(seq_lens) + + seq_lens_tensor = torch.tensor(seq_lens, dtype=torch.int32) + query_start_loc = torch.zeros(num_seqs, dtype=torch.int32) + torch.cumsum(seq_lens_tensor[:-1], 0, out=query_start_loc[1:]) + block_nums = (seq_lens_tensor + block_size - 1) // block_size + start_block_ids = torch.zeros_like(seq_lens_tensor) + torch.cumsum(block_nums[:-1], 0, out=start_block_ids[1:]) + total_block_num: int = block_nums.sum().item() + max_block_num = block_nums.max().item() + block_offsets = torch.arange(0, max_block_num, dtype=torch.int32) + encoder_block_table = start_block_ids[:, None] + block_offsets[None, :] + slot_mapping_list = [] + slot_start_idx = 0 + for i in range(num_seqs): + block_num = block_nums[i].item() + seq_len = seq_lens[i] + slot_mapping_list.append(torch.arange(slot_start_idx, slot_start_idx + seq_len)) + slot_start_idx += block_num * block_size + slot_mapping = torch.cat(slot_mapping_list) + + query = tensor_cache( + elem_num=token_num * num_query_heads * head_size, + dtype=dtype, + tag="query", + ) + query = query.view( + token_num, + num_query_heads, + head_size, + ) + + key_value = tensor_cache( + elem_num=2 * token_num * num_kv_heads * head_size, + dtype=dtype, + tag="kv", + ) + key_value = key_value.view( + 2, + token_num, + num_kv_heads, + head_size, + ) + key, value = key_value.unbind(0) + + # KV cache for CPU attention + packed_key_value_cache = torch.zeros( + total_block_num, num_kv_heads, block_size, head_size * 2, dtype=dtype + ) + packed_key_value_cache = packed_key_value_cache.view( + (total_block_num, num_kv_heads, block_size * 2, -1) + ) + packed_key_cache, packed_value_cache = packed_key_value_cache.chunk(2, dim=2) + + cu_query_lens = torch.tensor([0] + seq_lens, dtype=torch.int32).cumsum( + dim=0, dtype=torch.int32 + ) + kv_lens_tensor = torch.tensor(seq_lens, dtype=torch.int32) + + # use reshape_and_cache to pack key_cache and value_cache + cpu_attn_reshape_and_cache( + key=key.view(-1, num_kv_heads, head_size), + value=value.view(-1, num_kv_heads, head_size), + key_cache=packed_key_cache, + value_cache=packed_value_cache, + slot_mapping=slot_mapping, + isa=isa, + ) + + metadata = cpu_attn_get_scheduler_metadata( + num_reqs=num_seqs, + num_heads=num_query_heads, + num_kv_heads=num_kv_heads, + head_dim=head_size, + seq_lens=kv_lens_tensor, + dtype=dtype, + query_start_loc=cu_query_lens, + causal=False, + sliding_window_size=sliding_window if sliding_window is not None else -1, + isa=isa, + enable_kv_split=False, + ) + + out_without_split = torch.empty_like(query) + cpu_attention_with_kv_cache( + query=query, + key_cache=packed_key_cache, + value_cache=packed_value_cache, + output=out_without_split, + query_start_loc=cu_query_lens, + seq_lens=kv_lens_tensor, + scale=scale, + causal=False, + alibi_slopes=None, + sliding_window=sliding_window if sliding_window is not None else -1, + block_table=encoder_block_table, + softcap=0, + scheduler_metadata=metadata, + s_aux=None, + ) + + metadata = cpu_attn_get_scheduler_metadata( + num_reqs=num_seqs, + num_heads=num_query_heads, + num_kv_heads=num_kv_heads, + head_dim=head_size, + seq_lens=kv_lens_tensor, + dtype=dtype, + query_start_loc=cu_query_lens, + causal=False, + sliding_window_size=sliding_window if sliding_window is not None else -1, + isa=isa, + enable_kv_split=True, + ) + + out_with_split = torch.empty_like(query) + cpu_attention_with_kv_cache( + query=query, + key_cache=packed_key_cache, + value_cache=packed_value_cache, + output=out_with_split, + query_start_loc=cu_query_lens, + seq_lens=kv_lens_tensor, + scale=scale, + causal=False, + alibi_slopes=None, + sliding_window=sliding_window if sliding_window is not None else -1, + block_table=encoder_block_table, + softcap=0, + scheduler_metadata=metadata, + s_aux=None, + ) + + ref_output = ref_varlen_encoder_attn( + query=query, + key=key, + value=value, + seq_lens=seq_lens, + scale=scale, + sliding_window=sliding_window, + ) + atol, rtol = 1.5e-2, 1e-2 + + ( + torch.testing.assert_close(out_with_split, ref_output, atol=atol, rtol=rtol), + f"{torch.max(torch.abs(out_with_split - ref_output))}", + ) + ( + torch.testing.assert_close(out_without_split, ref_output, atol=atol, rtol=rtol), + f"{torch.max(torch.abs(out_without_split - ref_output))}", + ) + + +@torch.inference_mode() +def varlen_with_paged_kv( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + num_blocks: int, + use_alibi: bool, + use_sink: bool, + isa: str, + kv_cache_dtype: str = "auto", + k_scale: float = 1.0, + v_scale: float = 1.0, + dynamic_causal: list[bool] | None = None, +) -> None: + set_random_seed(0) + num_seqs = len(seq_lens) + query_lens = [x[0] for x in seq_lens] + kv_lens = [x[1] for x in seq_lens] + num_query_heads = num_heads[0] + num_kv_heads = num_heads[1] + assert num_query_heads % num_kv_heads == 0 + max_kv_len = max(kv_lens) + scale = head_size**-0.5 + token_num = sum(query_lens) + dynamic_causal_tensor = ( + torch.tensor(dynamic_causal, dtype=torch.bool) + if dynamic_causal is not None + else None + ) + + # for n heads the set of slopes is the geometric sequence that starts + # 2^(-8/n) + alibi_slopes = _get_alibi_slopes(num_query_heads) if use_alibi else None + + s_aux = ( + 15 * torch.rand((num_query_heads,), dtype=torch.bfloat16) if use_sink else None + ) + + is_fp8 = kv_cache_dtype != "auto" + if is_fp8 and current_platform.get_cpu_architecture() != CpuArchEnum.X86: + pytest.skip("FP8 KV cache only supported on x86") + + query = tensor_cache( + elem_num=token_num * num_query_heads * head_size, + dtype=dtype, + ) + query = query.view( + token_num, + num_query_heads, + head_size, + ) + + key_value = tensor_cache( + elem_num=2 * num_blocks * num_kv_heads * block_size * head_size, + dtype=dtype, + ) + key_value = key_value.view( + 2, + num_blocks, + block_size, + num_kv_heads, + head_size, + ) + if is_fp8: + # Clamp KV to [-1, 1] so FP8 quantization error (<=12.5% for E4M3, + # <=25% for E5M2) stays within the test tolerances regardless of + # which tensor_cache values happen to be in use. + key_value = key_value.clamp(-1, 1) + key_cache, value_cache = key_value.unbind(0) + + # KV cache for CPU attention + cache_dtype = torch.uint8 if is_fp8 else dtype + packed_key_value_cache = torch.empty( + num_blocks, num_kv_heads, block_size, head_size * 2, dtype=cache_dtype + ) + packed_key_value_cache = packed_key_value_cache.view( + (num_blocks, num_kv_heads, block_size * 2, -1) + ) + packed_key_cache, packed_value_cache = packed_key_value_cache.chunk(2, dim=2) + + cu_query_lens = torch.tensor([0] + query_lens, dtype=torch.int32).cumsum( + dim=0, dtype=torch.int32 + ) + kv_lens_tensor = torch.tensor(kv_lens, dtype=torch.int32) + max_num_blocks_per_seq = (max_kv_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, num_blocks, (num_seqs, max_num_blocks_per_seq), dtype=torch.int32 + ) + + # use reshape_and_cache to pack key_cache and value_cache + slot_mapping = torch.arange(0, num_blocks * block_size, dtype=torch.int64) + fp8_kwargs: dict = ( + dict(k_scale=k_scale, v_scale=v_scale, kv_cache_dtype=kv_cache_dtype) + if is_fp8 + else {} + ) + cpu_attn_reshape_and_cache( + key=key_cache.view(-1, num_kv_heads, head_size), + value=value_cache.view(-1, num_kv_heads, head_size), + key_cache=packed_key_cache, + value_cache=packed_value_cache, + slot_mapping=slot_mapping, + isa=isa, + **fp8_kwargs, + ) + + metadata = cpu_attn_get_scheduler_metadata( + num_reqs=num_seqs, + num_heads=num_query_heads, + num_kv_heads=num_kv_heads, + head_dim=head_size, + seq_lens=kv_lens_tensor, + dtype=dtype, + query_start_loc=cu_query_lens, + causal=dynamic_causal is None, + sliding_window_size=sliding_window if sliding_window is not None else -1, + isa=isa, + enable_kv_split=False, + dynamic_causal=dynamic_causal_tensor, + ) + + out_without_split = torch.empty_like(query) + cpu_attention_with_kv_cache( + query=query, + key_cache=packed_key_cache, + value_cache=packed_value_cache, + output=out_without_split, + query_start_loc=cu_query_lens, + seq_lens=kv_lens_tensor, + scale=scale, + causal=dynamic_causal is None, + alibi_slopes=alibi_slopes, + sliding_window=sliding_window if sliding_window is not None else -1, + block_table=block_tables, + softcap=soft_cap if soft_cap is not None else 0, + scheduler_metadata=metadata, + s_aux=s_aux, + dynamic_causal=dynamic_causal_tensor, + **fp8_kwargs, + ) + + metadata = cpu_attn_get_scheduler_metadata( + num_reqs=num_seqs, + num_heads=num_query_heads, + num_kv_heads=num_kv_heads, + head_dim=head_size, + seq_lens=kv_lens_tensor, + dtype=dtype, + query_start_loc=cu_query_lens, + causal=dynamic_causal is None, + sliding_window_size=sliding_window if sliding_window is not None else -1, + isa=isa, + enable_kv_split=True, + dynamic_causal=dynamic_causal_tensor, + ) + + out_with_split = torch.empty_like(query) + cpu_attention_with_kv_cache( + query=query, + key_cache=packed_key_cache, + value_cache=packed_value_cache, + output=out_with_split, + query_start_loc=cu_query_lens, + seq_lens=kv_lens_tensor, + scale=scale, + causal=dynamic_causal is None, + alibi_slopes=alibi_slopes, + sliding_window=sliding_window if sliding_window is not None else -1, + block_table=block_tables, + softcap=soft_cap if soft_cap is not None else 0, + scheduler_metadata=metadata, + s_aux=s_aux, + dynamic_causal=dynamic_causal_tensor, + **fp8_kwargs, + ) + + if is_fp8: + # Build a float KV cache via the non-FP8 path and run float attention + # to use as the reference. + ref_key_cache = torch.empty( + num_blocks, num_kv_heads, block_size, head_size, dtype=dtype + ) + ref_value_cache = torch.empty_like(ref_key_cache) + cpu_attn_reshape_and_cache( + key=key_cache.view(-1, num_kv_heads, head_size), + value=value_cache.view(-1, num_kv_heads, head_size), + key_cache=ref_key_cache, + value_cache=ref_value_cache, + slot_mapping=slot_mapping, + isa=isa, + ) + ref_output = torch.empty_like(query) + cpu_attention_with_kv_cache( + query=query, + key_cache=ref_key_cache, + value_cache=ref_value_cache, + output=ref_output, + query_start_loc=cu_query_lens, + seq_lens=kv_lens_tensor, + scale=scale, + causal=dynamic_causal is None, + alibi_slopes=alibi_slopes, + sliding_window=sliding_window if sliding_window is not None else -1, + block_table=block_tables, + softcap=soft_cap if soft_cap is not None else 0, + scheduler_metadata=metadata, + s_aux=s_aux, + dynamic_causal=dynamic_causal_tensor, + ) + atol = _FP8_ATOL[kv_cache_dtype] + rtol = _FP8_RTOL + else: + ref_output = ref_paged_attn( + query=query, + key_cache=key_cache, + value_cache=value_cache, + query_lens=query_lens, + kv_lens=kv_lens, + block_tables=block_tables, + scale=scale, + sliding_window=sliding_window, + soft_cap=soft_cap, + alibi_slopes=alibi_slopes, + s_aux=s_aux, + dynamic_causal=dynamic_causal, + ) + atol, rtol = 1.5e-2, 1e-2 + + ( + torch.testing.assert_close(out_with_split, ref_output, atol=atol, rtol=rtol), + f"{torch.max(torch.abs(out_with_split - ref_output))}", + ) + ( + torch.testing.assert_close(out_without_split, ref_output, atol=atol, rtol=rtol), + f"{torch.max(torch.abs(out_without_split - ref_output))}", + ) + + +@pytest.mark.parametrize("seq_lens", ENCODER_SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize( + "block_size", + [ + 128, + ], +) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOWS) +@pytest.mark.parametrize("dtype", QTYPES) +@pytest.mark.parametrize("isa", ["vec"]) +def test_varlen_encoder_attention_vec( + seq_lens: list[int], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + isa: str, +) -> None: + varlen_encoder_attention( + seq_lens=seq_lens, + num_heads=num_heads, + head_size=head_size, + sliding_window=sliding_window, + dtype=dtype, + block_size=block_size, + isa=isa, + ) + + +@pytest.mark.parametrize("seq_lens", ENCODER_SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize( + "block_size", + [ + 128, + ], +) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOWS) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("isa", ["amx"]) +@pytest.mark.skipif(not torch.cpu._is_amx_tile_supported(), reason="no AMX support.") +def test_varlen_encoder_attention_amx( + seq_lens: list[int], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + isa: str, +) -> None: + varlen_encoder_attention( + seq_lens=seq_lens, + num_heads=num_heads, + head_size=head_size, + sliding_window=sliding_window, + dtype=dtype, + block_size=block_size, + isa=isa, + ) + + +@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8_e4m3", "fp8_e5m2"]) +@pytest.mark.parametrize("seq_lens", SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", [96, 128]) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOWS) +@pytest.mark.parametrize("dtype", QTYPES) +@pytest.mark.parametrize("soft_cap", [None]) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("use_alibi", [False]) +@pytest.mark.parametrize("use_sink", [False]) +@pytest.mark.parametrize("isa", ["vec"]) +def test_varlen_with_paged_kv_normal_vec( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + num_blocks: int, + use_alibi: bool, + use_sink: bool, + isa: str, + kv_cache_dtype: str, +) -> None: + varlen_with_paged_kv( + seq_lens=seq_lens, + num_heads=num_heads, + head_size=head_size, + sliding_window=sliding_window, + dtype=dtype, + block_size=block_size, + soft_cap=soft_cap, + num_blocks=num_blocks, + use_alibi=use_alibi, + use_sink=use_sink, + isa=isa, + kv_cache_dtype=kv_cache_dtype, + ) + + +@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8_e4m3", "fp8_e5m2"]) +@pytest.mark.parametrize("seq_lens", SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", [96, 128]) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOWS) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("soft_cap", [None]) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("use_alibi", [False]) +@pytest.mark.parametrize("use_sink", [False]) +@pytest.mark.parametrize("isa", ["amx"]) +@pytest.mark.skipif(not torch.cpu._is_amx_tile_supported(), reason="no AMX support.") +def test_varlen_with_paged_kv_normal_amx( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + num_blocks: int, + use_alibi: bool, + use_sink: bool, + isa: str, + kv_cache_dtype: str, +) -> None: + varlen_with_paged_kv( + seq_lens=seq_lens, + num_heads=num_heads, + head_size=head_size, + sliding_window=sliding_window, + dtype=dtype, + block_size=block_size, + soft_cap=soft_cap, + num_blocks=num_blocks, + use_alibi=use_alibi, + use_sink=use_sink, + isa=isa, + kv_cache_dtype=kv_cache_dtype, + ) + + +@pytest.mark.parametrize("seq_lens", SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES_VEC16) +@pytest.mark.parametrize("block_size", [48]) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOWS) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("soft_cap", [None]) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("use_alibi", [False]) +@pytest.mark.parametrize("use_sink", [False]) +@pytest.mark.parametrize("isa", ["vec16"]) +def test_varlen_with_paged_kv_normal_vec16( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + num_blocks: int, + use_alibi: bool, + use_sink: bool, + isa: str, +) -> None: + varlen_with_paged_kv( + seq_lens=seq_lens, + num_heads=num_heads, + head_size=head_size, + sliding_window=sliding_window, + dtype=dtype, + block_size=block_size, + soft_cap=soft_cap, + num_blocks=num_blocks, + use_alibi=use_alibi, + use_sink=use_sink, + isa=isa, + ) + + +@pytest.mark.parametrize("seq_lens", SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", [96, 128]) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOWS) +@pytest.mark.parametrize("dtype", QTYPES) +@pytest.mark.parametrize("soft_cap", [None]) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("use_alibi", [False]) +@pytest.mark.parametrize("use_sink", [False]) +@pytest.mark.parametrize("isa", ["neon"]) +@pytest.mark.skipif( + current_platform.get_cpu_architecture() != CpuArchEnum.ARM, + reason="Not an Arm CPU.", +) +def test_varlen_with_paged_kv_normal_neon( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + num_blocks: int, + use_alibi: bool, + use_sink: bool, + isa: str, +) -> None: + varlen_with_paged_kv( + seq_lens=seq_lens, + num_heads=num_heads, + head_size=head_size, + sliding_window=sliding_window, + dtype=dtype, + block_size=block_size, + soft_cap=soft_cap, + num_blocks=num_blocks, + use_alibi=use_alibi, + use_sink=use_sink, + isa=isa, + ) + + +@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8_e4m3"]) +@pytest.mark.parametrize("seq_lens", SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", [96, 128]) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOWS) +@pytest.mark.parametrize("dtype", QTYPES) +@pytest.mark.parametrize("soft_cap", [None]) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("use_alibi", [False]) +@pytest.mark.parametrize("use_sink", [False]) +@pytest.mark.parametrize("isa", ["rvv"]) +@pytest.mark.skipif( + current_platform.get_cpu_architecture() != CpuArchEnum.RISCV, + reason="Not a RISC-V CPU.", +) +def test_varlen_with_paged_kv_normal_rvv( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + num_blocks: int, + use_alibi: bool, + use_sink: bool, + isa: str, + kv_cache_dtype: str, +) -> None: + varlen_with_paged_kv( + seq_lens=seq_lens, + num_heads=num_heads, + head_size=head_size, + sliding_window=sliding_window, + dtype=dtype, + block_size=block_size, + soft_cap=soft_cap, + num_blocks=num_blocks, + use_alibi=use_alibi, + use_sink=use_sink, + isa=isa, + kv_cache_dtype=kv_cache_dtype, + ) + + +@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8_e4m3"]) +@pytest.mark.parametrize("seq_lens", SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", [96]) +@pytest.mark.parametrize("block_size", [128]) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOWS) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("soft_cap", [50]) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("use_alibi", [False]) +@pytest.mark.parametrize("use_sink", [False]) +@pytest.mark.parametrize("isa", [get_attn_isa()]) +def test_varlen_with_paged_kv_softcap( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + num_blocks: int, + use_alibi: bool, + use_sink: bool, + isa: str, + kv_cache_dtype: str, +) -> None: + varlen_with_paged_kv( + seq_lens=seq_lens, + num_heads=num_heads, + head_size=head_size, + sliding_window=sliding_window, + dtype=dtype, + block_size=block_size, + soft_cap=soft_cap, + num_blocks=num_blocks, + use_alibi=use_alibi, + use_sink=use_sink, + isa=isa, + kv_cache_dtype=kv_cache_dtype, + ) + + +@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8_e4m3"]) +@pytest.mark.parametrize("seq_lens", SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", [96]) +@pytest.mark.parametrize("block_size", [128]) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOWS) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("soft_cap", [None]) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("use_alibi", [True]) +@pytest.mark.parametrize("use_sink", [False]) +@pytest.mark.parametrize("isa", [get_attn_isa()]) +def test_varlen_with_paged_kv_alibi( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + num_blocks: int, + use_alibi: bool, + use_sink: bool, + isa: str, + kv_cache_dtype: str, +) -> None: + varlen_with_paged_kv( + seq_lens=seq_lens, + num_heads=num_heads, + head_size=head_size, + sliding_window=sliding_window, + dtype=dtype, + block_size=block_size, + soft_cap=soft_cap, + num_blocks=num_blocks, + use_alibi=use_alibi, + use_sink=use_sink, + isa=isa, + kv_cache_dtype=kv_cache_dtype, + ) + + +@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8_e4m3"]) +@pytest.mark.parametrize("seq_lens", SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", [96]) +@pytest.mark.parametrize("block_size", [128]) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOWS) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("soft_cap", [None]) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("use_alibi", [False]) +@pytest.mark.parametrize("use_sink", [True]) +@pytest.mark.parametrize("isa", [get_attn_isa()]) +def test_varlen_with_paged_kv_sink( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + num_blocks: int, + use_alibi: bool, + use_sink: bool, + isa: str, + kv_cache_dtype: str, +) -> None: + varlen_with_paged_kv( + seq_lens=seq_lens, + num_heads=num_heads, + head_size=head_size, + sliding_window=sliding_window, + dtype=dtype, + block_size=block_size, + soft_cap=soft_cap, + num_blocks=num_blocks, + use_alibi=use_alibi, + use_sink=use_sink, + isa=isa, + kv_cache_dtype=kv_cache_dtype, + ) + + +@pytest.mark.parametrize( + "kv_cache_dtype", + [ + "auto", + ], +) +@pytest.mark.parametrize("seq_lens", SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize( + "head_size", + [ + 128, + ], +) +@pytest.mark.parametrize("block_size", [96, 128]) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOWS) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("soft_cap", [None]) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("use_alibi", [False]) +@pytest.mark.parametrize("use_sink", [False]) +@pytest.mark.parametrize("isa", ["amx"]) +@pytest.mark.skipif(not torch.cpu._is_amx_tile_supported(), reason="no AMX support.") +def test_varlen_with_paged_kv_dynamic_causal( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + num_blocks: int, + use_alibi: bool, + use_sink: bool, + isa: str, + kv_cache_dtype: str, +) -> None: + dynamic_causal = [bool(i % 2) for i in range(len(seq_lens))] + varlen_with_paged_kv( + seq_lens=seq_lens, + num_heads=num_heads, + head_size=head_size, + sliding_window=sliding_window, + dtype=dtype, + block_size=block_size, + soft_cap=soft_cap, + num_blocks=num_blocks, + use_alibi=use_alibi, + use_sink=use_sink, + isa=isa, + kv_cache_dtype=kv_cache_dtype, + dynamic_causal=dynamic_causal, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_cutlass_mla_decode.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_cutlass_mla_decode.py new file mode 100644 index 0000000000000000000000000000000000000000..e2adf8da42ebc469c2f933928fc84cef2cd67939 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_cutlass_mla_decode.py @@ -0,0 +1,280 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import math +import random + +import pytest +import torch + +import vllm._custom_ops as ops +from vllm.platforms import current_platform +from vllm.triton_utils import triton +from vllm.utils.platform_utils import num_compute_units + + +def cal_diff( + x: torch.Tensor, + y: torch.Tensor, + name: str, + use_fp8: bool = False, + diff_threshold: float | None = None, +) -> None: + x, y = x.double(), y.double() + cos_diff = 1 - 2 * (x * y).sum().item() / max((x * x + y * y).sum().item(), 1e-12) + if diff_threshold is not None: + # directly compare the cos_diff with the threshold + assert cos_diff < diff_threshold + else: + # use the default threshold + if use_fp8: + assert cos_diff < 1e-4 + else: + assert cos_diff < 1e-5 + + +CUTLASS_MLA_UNSUPPORTED_REASON = ( + "Cutlass MLA Requires compute capability of 100 or above." + if not current_platform.is_device_capability_family(100) + else "Cutlass MLA is supported" +) + + +@pytest.mark.skipif( + not current_platform.has_device_capability(100), + reason=CUTLASS_MLA_UNSUPPORTED_REASON, +) +@pytest.mark.parametrize("b", [128]) +@pytest.mark.parametrize("s_q", [1]) +@pytest.mark.parametrize("mean_sk", [4096, 8192, 16384]) +@pytest.mark.parametrize("h_q", [16, 32, 64, 128]) +@pytest.mark.parametrize("h_kv", [1]) +@pytest.mark.parametrize("d", [576]) +@pytest.mark.parametrize("dv", [512]) +@pytest.mark.parametrize("block_size", [64]) +@pytest.mark.parametrize("causal", [True]) +@pytest.mark.parametrize("varlen", [False, True]) +@pytest.mark.parametrize( + "torch_dtype", + [ + torch.bfloat16, + # fp8 can have occasional precision-related failures. + pytest.param(torch.float8_e4m3fn, marks=pytest.mark.flaky(reruns=2)), + ], +) +@torch.inference_mode() +def test_cutlass_mla_decode( + b, s_q, mean_sk, h_q, h_kv, d, dv, block_size, causal, varlen, torch_dtype +): + device = torch.device("cuda:0") + init_dtype = torch.bfloat16 if torch_dtype == torch.float8_e4m3fn else torch_dtype + torch.set_default_dtype(init_dtype) + torch.set_default_device(device) + torch.accelerator.set_device_index(device) + torch.manual_seed(42) + random.seed(42) + + print( + f"{b=}, {s_q=}, {mean_sk=}, {h_q=}, {h_kv=}, " + f"{d=}, {dv=}, {causal=}, {varlen=}, {torch_dtype=}" + ) + + use_fp8 = torch_dtype == torch.float8_e4m3fn + scale = math.sqrt(d) ** (-1) + cache_seqlens = torch.full((b,), mean_sk, dtype=torch.int32) + if varlen: + for i in range(b): + cache_seqlens[i] = max(random.normalvariate(mean_sk, mean_sk / 2), s_q) + total_seqlens = cache_seqlens.sum().item() + max_seqlen = cache_seqlens.max().item() + max_seqlen_pad = triton.cdiv(max_seqlen, 256) * 256 + + q = torch.randn(b, s_q, h_q, d) + block_table = torch.arange( + b * max_seqlen_pad // block_size, dtype=torch.int32 + ).view(b, max_seqlen_pad // block_size) + blocked_k = torch.randn(block_table.numel(), block_size, h_kv, d) + blocked_v = blocked_k[..., :dv] + + init_dtype = q.dtype + if use_fp8: + fp8_dtype = torch.float8_e4m3fn + descale_q = torch.ones((1), dtype=torch.float32) + descale_k = torch.ones((1), dtype=torch.float32) + + q = q.to(fp8_dtype) + blocked_k = blocked_k.to(fp8_dtype) + blocked_v = blocked_v.to(fp8_dtype) + else: + descale_q = None + descale_k = None + + def cutlass_mla(): + MAX_HEADS = 128 + + q_reshaped = q.squeeze(1) + q_nope = q_reshaped[:, :, :dv].clone() + q_pe = q_reshaped[:, :, dv:].clone() + + if h_q < MAX_HEADS: + q_nope_padded = q_nope.new_empty((b, MAX_HEADS, dv)) + q_nope_padded[:, :h_q] = q_nope + q_nope = q_nope_padded + + q_pe_padded = q_pe.new_empty((b, MAX_HEADS, d - dv)) + q_pe_padded[:, :h_q] = q_pe + q_pe = q_pe_padded + + kv_cache_flat = blocked_k.squeeze(2) + sm_count = num_compute_units(device.index) + workspace_size = ops.sm100_cutlass_mla_get_workspace_size( + max_seqlen * block_size, b, sm_count, num_kv_splits=1 + ) + workspace = torch.empty(workspace_size, device="cuda", dtype=torch.uint8) + + out_ans = torch.empty(b, MAX_HEADS, dv, dtype=init_dtype) + output_lse = torch.empty( + (b, MAX_HEADS), dtype=torch.float32, device=q_nope.device + ) + ops.sm100_cutlass_mla_decode( + out_ans, + output_lse, + q_nope, + q_pe, + kv_cache_flat, + cache_seqlens, + block_table, + workspace, + scale, + 1, + ) + return out_ans[:, :h_q].contiguous(), output_lse[:, :h_q].contiguous() + + def scaled_dot_product_attention(query, key, value, is_causal=False): + query = query.float() + key = key.float() + value = value.float() + key = key.repeat_interleave(h_q // h_kv, dim=0) + value = value.repeat_interleave(h_q // h_kv, dim=0) + attn_weight = query @ key.transpose(-2, -1) / math.sqrt(query.size(-1)) + if is_causal: + s_q = query.shape[-2] + s_k = key.shape[-2] + attn_bias = torch.zeros(s_q, s_k, dtype=query.dtype) + temp_mask = torch.ones(s_q, s_k, dtype=torch.bool).tril(diagonal=s_k - s_q) + attn_bias.masked_fill_(temp_mask.logical_not(), float("-inf")) + attn_bias.to(query.dtype) + attn_weight += attn_bias + lse = attn_weight.logsumexp(dim=-1) + attn_weight = torch.softmax(attn_weight, dim=-1, dtype=torch.float32) + return attn_weight @ value, lse + + def ref_mla(): + q_ = (q.to(torch.float) * descale_q).to(init_dtype) if use_fp8 else q + blocked_k_ = ( + (blocked_k.to(torch.float) * descale_k).to(init_dtype) + if use_fp8 + else blocked_k + ) + blocked_v_ = ( + (blocked_v.to(torch.float) * descale_k).to(init_dtype) + if use_fp8 + else blocked_v + ) + out = torch.empty(b, s_q, h_q, dv, dtype=torch.float32) + lse = torch.empty(b, h_q, s_q, dtype=torch.float32) + for i in range(b): + begin = i * max_seqlen_pad + end = begin + cache_seqlens[i] + out_i, lse_i = scaled_dot_product_attention( + q_[i].transpose(0, 1), + blocked_k_.view(-1, h_kv, d)[begin:end].transpose(0, 1), + blocked_v_.view(-1, h_kv, dv)[begin:end].transpose(0, 1), + is_causal=causal, + ) + out[i] = out_i.transpose(0, 1) + lse[i] = lse_i + return out, lse + + out_cutlass, lse_cutlass = cutlass_mla() + out_torch, lse_torch = ref_mla() + # Extract the single token (s_q=1) slice to match cutlass output shape + out_torch_slice = out_torch[:, 0, :, :] # [b, h_q, dv] + lse_torch_slice = lse_torch[:, 0, :] # [b, h_q] + cal_diff(out_cutlass, out_torch_slice, "out", use_fp8) + # lse has larger numerical error, so use a larger threshold + cal_diff(lse_cutlass, lse_torch_slice, "lse", use_fp8, diff_threshold=1e-3) + + t = triton.testing.do_bench(cutlass_mla) + FLOPS = s_q * total_seqlens * h_q * (d + dv) * 2 + bytes = (total_seqlens * h_kv * d + b * s_q * h_q * d) * ( + torch.finfo(torch_dtype).bits // 8 + ) + (b * s_q * h_q * dv) * (torch.finfo(init_dtype).bits // 8) + print( + f"{t:.3f} ms, {FLOPS / 10**9 / t:.0f} TFLOPS,", f"{bytes / 10**6 / t:.0f} GB/s" + ) + + +@pytest.mark.skipif( + not current_platform.has_device_capability(100), + reason=CUTLASS_MLA_UNSUPPORTED_REASON, +) +@torch.inference_mode() +def test_cutlass_mla_decode_cross_layer_view(): + """The kernel must read the cache's page-dim stride instead of assuming + pages are packed back-to-back. A per-layer view into a cross-layer + (block-major) cache has stride(0) inflated by num_layers; outputs must + match a contiguous cache holding the same data exactly.""" + device = torch.device("cuda:0") + torch.set_default_dtype(torch.bfloat16) + torch.set_default_device(device) + torch.manual_seed(42) + + b, mean_sk, d, dv, block_size = 4, 512, 576, 512, 64 + num_layers, layer_idx = 3, 1 + scale = math.sqrt(d) ** (-1) + + num_pages = b * (mean_sk // block_size) + cache_seqlens = torch.full((b,), mean_sk, dtype=torch.int32) + block_table = torch.arange(num_pages, dtype=torch.int32).view( + b, mean_sk // block_size + ) + + kv_contig = torch.randn(num_pages, block_size, d) + # Neighbor layers hold random data so packed-pages addressing reads + # garbage rather than zeros. + kv_cross_layer = torch.randn(num_pages, num_layers, block_size, d) + kv_view = kv_cross_layer[:, layer_idx] + kv_view.copy_(kv_contig) + assert kv_view.stride(0) == num_layers * block_size * d + + q_nope = torch.randn(b, 128, dv) + q_pe = torch.randn(b, 128, d - dv) + sm_count = num_compute_units(device.index) + workspace_size = ops.sm100_cutlass_mla_get_workspace_size( + mean_sk, b, sm_count, num_kv_splits=1 + ) + workspace = torch.empty(workspace_size, dtype=torch.uint8) + + def run(cache): + out = torch.empty(b, 128, dv) + lse = torch.empty(b, 128, dtype=torch.float32) + ops.sm100_cutlass_mla_decode( + out, + lse, + q_nope, + q_pe, + cache, + cache_seqlens, + block_table, + workspace, + scale, + 1, + ) + return out, lse + + out_contig, lse_contig = run(kv_contig) + out_view, lse_view = run(kv_view) + + # Same data and same compute order; only addressing differs. + assert torch.equal(out_contig, out_view) + assert torch.equal(lse_contig, lse_view) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_deepgemm_attention.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_deepgemm_attention.py new file mode 100644 index 0000000000000000000000000000000000000000..ab6760a7d83570748a2204b3af03804994335bb3 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_deepgemm_attention.py @@ -0,0 +1,310 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import random + +import pytest +import torch + +from vllm.platforms import current_platform +from vllm.utils.deep_gemm import ( + _ceil_to_ue8m0, + calc_diff, + fp8_fp4_mqa_logits, + fp8_fp4_paged_mqa_logits, + get_num_sms, + get_paged_mqa_logits_metadata, +) +from vllm.utils.import_utils import has_deep_gemm +from vllm.utils.math_utils import cdiv + + +def kv_cache_cast_to_fp8(x: torch.Tensor) -> torch.Tensor: + # x: (num_blocks, block_size, 1, head_dim) + num_blocks, block_size, num_heads, head_dim = x.shape + assert num_heads == 1 + x_amax = x.abs().float().amax(dim=3, keepdim=True).clamp(1e-4) + sf = x_amax / 448.0 + x_scaled = (x * (1.0 / sf)).to(torch.float8_e4m3fn) + x_fp8 = torch.empty( + (num_blocks, block_size * (head_dim + 4)), + device=x.device, + dtype=torch.uint8, + ) + x_fp8[:, : block_size * head_dim] = x_scaled.view( + num_blocks, block_size * head_dim + ).view(dtype=torch.uint8) + x_fp8[:, block_size * head_dim :] = sf.view(num_blocks, block_size).view( + dtype=torch.uint8 + ) + return x_fp8.view(num_blocks, block_size, num_heads, head_dim + 4) + + +def per_custom_dims_cast_to_fp8( + x: torch.Tensor, dims: tuple, use_ue8m0: bool +) -> tuple[torch.Tensor, torch.Tensor]: + excluded_dims = tuple([i for i in range(x.dim()) if i not in set(dims)]) + x_amax = x.abs().float().amax(dim=excluded_dims, keepdim=True).clamp(1e-4) + sf = x_amax / 448.0 + sf = _ceil_to_ue8m0(sf) if use_ue8m0 else sf + x_scaled = (x * (1.0 / sf)).to(torch.float8_e4m3fn) + return x_scaled, sf.squeeze() + + +def _generate_cp_test_data(seq_len: int, seq_len_kv: int): + assert seq_len_kv % seq_len == 0 and seq_len % 2 == 0 + chunk_size = seq_len // 2 + cp_size = seq_len_kv // seq_len + cp_id = cp_size // 3 + ks = torch.zeros(seq_len, dtype=torch.int, device="cuda") + ke = torch.zeros(seq_len, dtype=torch.int, device="cuda") + for i in range(chunk_size): + ke[i] = cp_id * chunk_size + i + ke[i + chunk_size] = (cp_size * 2 - 1 - cp_id) * chunk_size + i + return ks, ke + + +def _ref_fp8_mqa_logits( + q: torch.Tensor, + kv: torch.Tensor, + weights: torch.Tensor, + cu_seqlen_ks: torch.Tensor, + cu_seqlen_ke: torch.Tensor, +): + seq_len_kv = kv.shape[0] + + k = kv + q = q.float() + k = k.float() + + mask_lo = ( + torch.arange(0, seq_len_kv, device="cuda")[None, :] >= cu_seqlen_ks[:, None] + ) + mask_hi = ( + torch.arange(0, seq_len_kv, device="cuda")[None, :] < cu_seqlen_ke[:, None] + ) + mask = mask_lo & mask_hi + score = torch.einsum("mhd,nd->hmn", q, k) + logits = (score.relu() * weights.unsqueeze(-1).transpose(0, 1)).sum(dim=0) + logits = logits.masked_fill(~mask, float("-inf")) + + return logits + + +@pytest.mark.skipif(not current_platform.is_cuda(), reason="CUDA only") +@pytest.mark.skipif(not has_deep_gemm(), reason="DeepGEMM not available") +@pytest.mark.skipif( + not current_platform.has_device_capability(90), reason="SM90 and SM100 only" +) +@pytest.mark.parametrize("clean_logits", [True, False]) +def test_deepgemm_fp8_mqa_logits(clean_logits: bool): + torch.manual_seed(0) + random.seed(0) + num_heads, head_dim = 32, 128 + for seq_len in (512,): + for seq_len_kv in (1024,): + for disable_cp in (False, True): + q = torch.randn( + seq_len, + num_heads, + head_dim, + device="cuda", + dtype=torch.bfloat16, + ) + kv = torch.randn( + seq_len_kv, head_dim, device="cuda", dtype=torch.bfloat16 + ) + weights = torch.randn( + seq_len, num_heads, device="cuda", dtype=torch.float32 + ) + + if disable_cp: + ks = torch.zeros(seq_len, dtype=torch.int, device="cuda") + ke = torch.arange(seq_len, dtype=torch.int, device="cuda") + ( + seq_len_kv - seq_len + ) + else: + ks, ke = _generate_cp_test_data(seq_len, seq_len_kv) + + q_fp8 = q.to(torch.float8_e4m3fn) + kv_fp8 = per_custom_dims_cast_to_fp8(kv, (0,), False) + logits = fp8_fp4_mqa_logits( + (q_fp8, None), kv_fp8, weights, ks, ke, clean_logits=clean_logits + ) + + ref_logits = _ref_fp8_mqa_logits( + q=q, + kv=kv, + weights=weights, + cu_seqlen_ks=ks, + cu_seqlen_ke=ke, + ) + ref_neginf_mask = ref_logits == float("-inf") + + if clean_logits: + neginf_mask = logits == float("-inf") + assert torch.equal(neginf_mask, ref_neginf_mask) + + ref_logits = ref_logits.masked_fill(ref_neginf_mask, 0) + logits = logits.masked_fill(ref_neginf_mask, 0) + diff = calc_diff(logits, ref_logits) + assert diff < 1e-3, f"{diff=}" + + +def _ref_fp8_fp4_paged_mqa_logits( + q: torch.Tensor, + kv_cache: torch.Tensor, + weights: torch.Tensor, + context_lens: torch.Tensor, + block_tables: torch.Tensor, + max_model_len: int, +): + batch_size, next_n, _, _ = q.size() + _, block_size, _, _ = kv_cache.size() + logits = torch.full( + [batch_size * next_n, max_model_len], + float("-inf"), + device=q.device, + dtype=torch.float32, + ) + context_lens_list = context_lens.tolist() + for i in range(batch_size): + context_len = context_lens_list[i] + q_offsets = torch.arange(context_len - next_n, context_len, device="cuda") + weight_slice = ( + weights[i * next_n : (i + 1) * next_n, :].transpose(0, 1).contiguous() + ) + for block_rk in range(cdiv(context_len, block_size)): + block_idx = block_tables[i][block_rk] + qx, kx = q[i], kv_cache[block_idx] + k_offsets = torch.arange( + block_rk * block_size, + (block_rk + 1) * block_size, + device="cuda", + ) + mask = (k_offsets[None, :] < context_len) & ( + k_offsets[None, :] <= q_offsets[:, None] + ) + s = torch.where( + mask[None, :, :], + (qx.transpose(0, 1) @ kx.transpose(0, 1).transpose(1, 2)).to( + logits.dtype + ), + float("-inf"), + ) + s = torch.relu(s) * weight_slice[..., None] + s = s.sum(dim=0) + logits[ + i * next_n : (i + 1) * next_n, + block_rk * block_size : (block_rk + 1) * block_size, + ] = torch.where(k_offsets[None, :] <= q_offsets[:, None], s, float("-inf")) + return logits + + +@pytest.mark.skipif(not current_platform.is_cuda(), reason="CUDA only") +@pytest.mark.skipif(not has_deep_gemm(), reason="DeepGEMM not available") +@pytest.mark.skipif( + not current_platform.has_device_capability(90), reason="SM90 and SM100 only" +) +def test_deepgemm_fp8_fp4_paged_mqa_logits(): + # NOTE: clean_logits=True is incompatible with the 2D context_lens + # required by csrc/apis/attention.hpp; only the False path is exercised. + clean_logits = False + torch.manual_seed(0) + random.seed(0) + + max_model_len = 4096 + for batch_size, next_n in [(4, 1), (2, 2)]: + for heads, index_dim in [(32, 128)]: + for avg_kv in (2048,): + num_blocks, blocksize = max_model_len * 2, 64 + + q = torch.randn( + (batch_size, next_n, heads, index_dim), + device="cuda", + dtype=torch.bfloat16, + ) + kv_cache = torch.randn( + (num_blocks, blocksize, 1, index_dim), + device="cuda", + dtype=torch.bfloat16, + ) + weights = torch.randn( + (batch_size * next_n, heads), + device="cuda", + dtype=torch.float32, + ) + + context_lens = ( + torch.randint(int(0.8 * avg_kv), int(1.2 * avg_kv), (batch_size,)) + .cuda() + .to(torch.int32) + ) + max_block_len = ( + (context_lens.max().item() + blocksize - 1) // blocksize * blocksize + ) + block_tables = torch.zeros( + (batch_size, max_block_len), + device="cuda", + dtype=torch.int32, + ) + + counter = 0 + block_idx_pool = list(range(num_blocks)) + random.shuffle(block_idx_pool) + for i in range(batch_size): + ctx_len = int(context_lens[i].item()) + for j in range((ctx_len + blocksize - 1) // blocksize): + block_tables[i][j] = block_idx_pool[counter] + counter += 1 + + q_fp8 = q.to(torch.float8_e4m3fn) + kv_cache_fp8 = kv_cache_cast_to_fp8(kv_cache) + + # deep_gemm paged MQA logits requires 2D context_lens of + # shape (B, next_n) (csrc/apis/attention.hpp:332-335); + # see indexer.py:607-608. For each batch/next_n token, the + # effective context length is context_lens[b] - next_n + j + 1. + next_n_arange = torch.arange(next_n, device="cuda", dtype=torch.int32) + context_lens_2d = ( + context_lens.unsqueeze(-1) - next_n + 1 + next_n_arange + ).contiguous() + schedule_metadata = get_paged_mqa_logits_metadata( + context_lens_2d, blocksize, get_num_sms() + ) + logits = fp8_fp4_paged_mqa_logits( + (q_fp8, None), + kv_cache_fp8, + weights, + context_lens_2d, + block_tables, + schedule_metadata, + max_model_len, + clean_logits=clean_logits, + ) + + ref_logits = _ref_fp8_fp4_paged_mqa_logits( + q, + kv_cache, + weights, + context_lens, + block_tables, + max_model_len, + ) + + positions = ( + torch.arange(max_model_len, device="cuda") + .unsqueeze(0) + .expand(batch_size * next_n, -1) + ) + row_indices = torch.arange(batch_size * next_n, device="cuda") // next_n + next_n_offset = ( + torch.arange(batch_size * next_n, device="cuda") % next_n + ) + mask = positions <= ( + context_lens[row_indices] - next_n + next_n_offset + ).unsqueeze(1) + + logits = logits.masked_fill(~mask, 0) + ref_logits = ref_logits.masked_fill(~mask, 0) + diff = calc_diff(logits, ref_logits) + assert diff < 1e-3, f"{diff=}" diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_flash_attn.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_flash_attn.py new file mode 100644 index 0000000000000000000000000000000000000000..3ab7e5096791a95f994cc9759f9560077133be04 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_flash_attn.py @@ -0,0 +1,217 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import pytest +import torch + +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +try: + from vllm.vllm_flash_attn import ( + fa_version_unsupported_reason, + flash_attn_varlen_func, + is_fa_version_supported, + ) +except ImportError: + if current_platform.is_rocm(): + pytest.skip( + "vllm_flash_attn is not supported for vLLM on ROCm.", + allow_module_level=True, + ) + + +NUM_HEADS = [(4, 4), (8, 2)] +HEAD_SIZES = [40, 72, 80, 128, 256] +BLOCK_SIZES = [16] +DTYPES = [torch.bfloat16] +QDTYPES = [None, torch.float8_e4m3fn] +# one value large enough to test overflow in index calculation. +# one value small enough to test the schema op check +NUM_BLOCKS = [32768, 2048] +SOFT_CAPS = [None] +SLIDING_WINDOWS = [None, 256] + + +def ref_paged_attn( + query: torch.Tensor, + key_cache: torch.Tensor, + value_cache: torch.Tensor, + query_lens: list[int], + kv_lens: list[int], + block_tables: torch.Tensor, + scale: float, + sliding_window: int | None = None, + soft_cap: float | None = None, +) -> torch.Tensor: + num_seqs = len(query_lens) + block_tables = block_tables.cpu().numpy() + _, block_size, num_kv_heads, head_size = key_cache.shape + + outputs: list[torch.Tensor] = [] + start_idx = 0 + for i in range(num_seqs): + query_len = query_lens[i] + kv_len = kv_lens[i] + q = query[start_idx : start_idx + query_len] + q *= scale + + num_kv_blocks = (kv_len + block_size - 1) // block_size + block_indices = block_tables[i, :num_kv_blocks] + + k = key_cache[block_indices].view(-1, num_kv_heads, head_size) + k = k[:kv_len] + v = value_cache[block_indices].view(-1, num_kv_heads, head_size) + v = v[:kv_len] + + if q.shape[1] != k.shape[1]: + k = torch.repeat_interleave(k, q.shape[1] // k.shape[1], dim=1) + v = torch.repeat_interleave(v, q.shape[1] // v.shape[1], dim=1) + attn = torch.einsum("qhd,khd->hqk", q, k).float() + empty_mask = torch.ones(query_len, kv_len) + mask = torch.triu(empty_mask, diagonal=kv_len - query_len + 1).bool() + if sliding_window is not None: + sliding_window_mask = ( + torch.triu( + empty_mask, diagonal=kv_len - (query_len + sliding_window) + 1 + ) + .bool() + .logical_not() + ) + mask |= sliding_window_mask + if soft_cap is not None: + attn = soft_cap * torch.tanh(attn / soft_cap) + attn.masked_fill_(mask, float("-inf")) + attn = torch.softmax(attn, dim=-1).to(v.dtype) + out = torch.einsum("hqk,khd->qhd", attn, v) + + outputs.append(out) + start_idx += query_len + + return torch.cat(outputs, dim=0) + + +@pytest.mark.parametrize("use_out", [True, False]) +@pytest.mark.parametrize( + "seq_lens", [[(1, 1328), (5, 18), (129, 463)], [(1, 523), (1, 37), (1, 2011)]] +) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOWS) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("soft_cap", SOFT_CAPS) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("fa_version", [2, 3]) +@pytest.mark.parametrize("q_dtype", QDTYPES) +@torch.inference_mode() +def test_varlen_with_paged_kv( + use_out: bool, + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + num_blocks: int, + fa_version: int, + q_dtype: torch.dtype | None, +) -> None: + torch.set_default_device("cuda") + if not is_fa_version_supported(fa_version): + pytest.skip( + f"Flash attention version {fa_version} not supported due " + f'to: "{fa_version_unsupported_reason(fa_version)}"' + ) + if q_dtype is not None and (dtype != torch.bfloat16 or fa_version == 2): + pytest.skip( + "Flash attention with quantized inputs is only " + "supported on version 3 with bfloat16 base type" + ) + set_random_seed(0) + num_seqs = len(seq_lens) + query_lens = [x[0] for x in seq_lens] + kv_lens = [x[1] for x in seq_lens] + num_query_heads = num_heads[0] + num_kv_heads = num_heads[1] + assert num_query_heads % num_kv_heads == 0 + max_query_len = max(query_lens) + max_kv_len = max(kv_lens) + window_size = (sliding_window - 1, 0) if sliding_window is not None else (-1, -1) + scale = head_size**-0.5 + + query = torch.randn(sum(query_lens), num_query_heads, head_size, dtype=dtype) + key_cache = torch.randn( + num_blocks, block_size, num_kv_heads, head_size, dtype=dtype + ) + value_cache = torch.randn_like(key_cache) + cu_query_lens = torch.tensor([0] + query_lens, dtype=torch.int32).cumsum( + dim=0, dtype=torch.int32 + ) + kv_lens = torch.tensor(kv_lens, dtype=torch.int32) + + max_num_blocks_per_seq = (max_kv_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, num_blocks, (num_seqs, max_num_blocks_per_seq), dtype=torch.int32 + ) + + out = torch.empty_like(query) if use_out else None + + maybe_quantized_query = query + maybe_quantized_key_cache = key_cache + maybe_quantized_value_cache = value_cache + q_descale = None + k_descale = None + v_descale = None + if q_dtype is not None: + # QKV are drawn from N(0, 1): no need for a fp8 scaling factor + maybe_quantized_query = query.to(q_dtype) + maybe_quantized_key_cache = key_cache.to(q_dtype) + maybe_quantized_value_cache = value_cache.to(q_dtype) + + scale_shape = (num_seqs, num_kv_heads) + q_descale = torch.ones(scale_shape, dtype=torch.float32) + k_descale = torch.ones(scale_shape, dtype=torch.float32) + v_descale = torch.ones(scale_shape, dtype=torch.float32) + + output = flash_attn_varlen_func( + q=maybe_quantized_query, + k=maybe_quantized_key_cache, + v=maybe_quantized_value_cache, + out=out, + cu_seqlens_q=cu_query_lens, + seqused_k=kv_lens, + max_seqlen_q=max_query_len, + max_seqlen_k=max_kv_len, + softmax_scale=scale, + causal=True, + window_size=window_size, + block_table=block_tables, + softcap=soft_cap if soft_cap is not None else 0, + fa_version=fa_version, + q_descale=q_descale, + k_descale=k_descale, + v_descale=v_descale, + ) + output = output if not use_out else out + + ref_output = ref_paged_attn( + query=query, + key_cache=key_cache, + value_cache=value_cache, + query_lens=query_lens, + kv_lens=kv_lens, + block_tables=block_tables, + scale=scale, + sliding_window=sliding_window, + soft_cap=soft_cap, + ) + atol, rtol = 1.5e-2, 1e-2 + if q_dtype is not None: + atol, rtol = 1.5e-1, 1.5e-1 + ( + torch.testing.assert_close(output, ref_output, atol=atol, rtol=rtol), + f"{torch.max(torch.abs(output - ref_output))}", + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_flashinfer.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_flashinfer.py new file mode 100644 index 0000000000000000000000000000000000000000..9a9a36479f491c7bf3883ddae1a8ebca212eb774 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_flashinfer.py @@ -0,0 +1,701 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import pytest + +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +try: + import flashinfer +except ImportError: + if current_platform.is_rocm(): + pytest.skip( + "flashinfer is not supported for vLLM on ROCm.", allow_module_level=True + ) + +import torch + +NUM_HEADS = [(32, 8), (6, 1)] +HEAD_SIZES = [128, 256] +BLOCK_SIZES = [16, 32] +DTYPES = [torch.bfloat16] +NUM_BLOCKS = 32768 # Large enough to test overflow in index calculation. +SOFT_CAPS = [None, 30.0] +SLIDING_WINDOWS = [None, 64] + + +def ref_paged_attn( + query: torch.Tensor, + key_cache: torch.Tensor, + value_cache: torch.Tensor, + query_lens: list[int], + kv_lens: list[int], + block_tables: torch.Tensor, + scale: float, + sliding_window: int | None = None, + soft_cap: float | None = None, +) -> torch.Tensor: + num_seqs = len(query_lens) + block_tables = block_tables.cpu().numpy() + _, block_size, num_kv_heads, head_size = key_cache.shape + + outputs: list[torch.Tensor] = [] + start_idx = 0 + for i in range(num_seqs): + query_len = query_lens[i] + kv_len = kv_lens[i] + q = query[start_idx : start_idx + query_len] + q *= scale + + num_kv_blocks = (kv_len + block_size - 1) // block_size + block_indices = block_tables[i, :num_kv_blocks] + + k = key_cache[block_indices].view(-1, num_kv_heads, head_size) + k = k[:kv_len] + v = value_cache[block_indices].view(-1, num_kv_heads, head_size) + v = v[:kv_len] + + if q.shape[1] != k.shape[1]: + k = torch.repeat_interleave(k, q.shape[1] // k.shape[1], dim=1) + v = torch.repeat_interleave(v, q.shape[1] // v.shape[1], dim=1) + attn = torch.einsum("qhd,khd->hqk", q, k).float() + empty_mask = torch.ones(query_len, kv_len) + mask = torch.triu(empty_mask, diagonal=kv_len - query_len + 1).bool() + if sliding_window is not None: + sliding_window_mask = ( + torch.triu( + empty_mask, diagonal=kv_len - (query_len + sliding_window) + 1 + ) + .bool() + .logical_not() + ) + mask |= sliding_window_mask + if soft_cap is not None: + attn = soft_cap * torch.tanh(attn / soft_cap) + attn.masked_fill_(mask, float("-inf")) + attn = torch.softmax(attn, dim=-1).to(v.dtype) + out = torch.einsum("hqk,khd->qhd", attn, v) + + outputs.append(out) + start_idx += query_len + + return torch.cat(outputs, dim=0) + + +def _make_paged_kv_metadata( + kv_lens: list[int], + block_size: int, + num_blocks: int, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + """Build paged-KV metadata tensors for fast_plan_decode tests. + + Returns: + kv_indptr – CPU int32, shape [num_seqs + 1] + kv_indices – CUDA int32, shape [total_blocks] + kv_last_page_lens – CPU int32, shape [num_seqs] + block_tables – CUDA int32, shape [num_seqs, max_blocks_per_seq] + """ + num_seqs = len(kv_lens) + max_blocks = (max(kv_lens) + block_size - 1) // block_size + block_tables = torch.randint( + 0, num_blocks, (num_seqs, max_blocks), dtype=torch.int32, device="cuda" + ) + + indptr_list = [0] + indices_list: list[int] = [] + last_lens_list: list[int] = [] + for i, seq_len in enumerate(kv_lens): + n = (seq_len + block_size - 1) // block_size + indices_list.extend(block_tables[i, :n].cpu().tolist()) + indptr_list.append(indptr_list[-1] + n) + last_lens_list.append(seq_len % block_size or block_size) + + return ( + torch.tensor(indptr_list, dtype=torch.int32, device="cpu"), + torch.tensor(indices_list, dtype=torch.int32, device="cuda"), + torch.tensor(last_lens_list, dtype=torch.int32, device="cpu"), + block_tables, + ) + + +def _make_cg_decode_wrapper( + num_seqs: int, + kv_indices_buffer: torch.Tensor, + workspace_buffer: torch.Tensor, + use_tensor_cores: bool = True, +) -> "flashinfer.BatchDecodeWithPagedKVCacheWrapper": + """Create a cudagraph-enabled BatchDecodeWithPagedKVCacheWrapper. + + *kv_indices_buffer* is shared with the caller so that fast_plan_decode + can avoid the device-to-device index copy on subsequent (cudagraph) calls. + """ + return flashinfer.BatchDecodeWithPagedKVCacheWrapper( + workspace_buffer, + "NHD", + use_cuda_graph=True, + paged_kv_indptr_buffer=torch.zeros( + num_seqs + 1, dtype=torch.int32, device="cuda" + ), + paged_kv_indices_buffer=kv_indices_buffer, + paged_kv_last_page_len_buffer=torch.zeros( + num_seqs, dtype=torch.int32, device="cuda" + ), + use_tensor_cores=use_tensor_cores, + ) + + +def test_fast_decode_plan_importable() -> None: + """fast_decode_plan must be importable from flashinfer.decode. + + This is a forward-compatibility smoke test: if FlashInfer reorganises its + public API the import will fail before any other test does. + """ + from flashinfer.decode import fast_decode_plan # noqa: F401 + + assert callable(fast_decode_plan) + + +@pytest.mark.parametrize("dtype", DTYPES) +@torch.inference_mode +def test_fast_plan_decode_warmup_uses_full_plan(dtype: torch.dtype) -> None: + """On the first call fast_plan_decode must route through self.plan() and + flip vllm_first_call to False on the wrapper object.""" + from unittest.mock import patch + + from vllm.v1.attention.backends.flashinfer import fast_plan_decode + + torch.set_default_device("cuda") + set_random_seed(0) + + kv_lens = [128, 64] + block_size = 16 + num_seqs = len(kv_lens) + num_query_heads, num_kv_heads = 8, 2 + head_size = 128 + + kv_indptr, kv_indices, kv_last_page_lens, _ = _make_paged_kv_metadata( + kv_lens, block_size, NUM_BLOCKS + ) + + workspace = torch.empty(128 * 1024 * 1024, dtype=torch.int8) + wrapper = _make_cg_decode_wrapper(num_seqs, kv_indices.clone(), workspace) + + assert getattr(wrapper, "vllm_first_call", True) is True + + with patch.object(wrapper, "plan", wraps=wrapper.plan) as mock_plan: + fast_plan_decode( + wrapper, + indptr_cpu=kv_indptr, + indices=kv_indices, + last_page_len_cpu=kv_last_page_lens, + num_qo_heads=num_query_heads, + num_kv_heads=num_kv_heads, + head_dim=head_size, + page_size=block_size, + q_data_type=dtype, + kv_data_type=dtype, + ) + mock_plan.assert_called_once() + + assert wrapper.vllm_first_call is False, ( + "vllm_first_call should be False after the first fast_plan_decode call" + ) + + +@pytest.mark.parametrize("kv_lens", [[1328, 18, 463], [1, 54, 293, 70]]) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@torch.inference_mode +def test_fast_plan_decode_matches_full_plan( + kv_lens: list[int], + num_heads: tuple[int, int], + head_size: int, + block_size: int, + dtype: torch.dtype, +) -> None: + """fast_plan_decode's cudagraph path (delegating to FlashInfer's + fast_decode_plan) must produce attention output numerically identical to + a standard plan() call. + + Both the warmup call (self.plan) and the subsequent fast call + (fast_decode_plan) are verified against the same reference. + """ + from vllm.v1.attention.backends.flashinfer import fast_plan_decode + + torch.set_default_device("cuda") + set_random_seed(0) + num_seqs = len(kv_lens) + num_query_heads, num_kv_heads = num_heads + + query = torch.randn(num_seqs, num_query_heads, head_size, dtype=dtype) + key_value_cache = torch.randn( + NUM_BLOCKS, 2, block_size, num_kv_heads, head_size, dtype=dtype + ) + + kv_indptr, kv_indices, kv_last_page_lens, _ = _make_paged_kv_metadata( + kv_lens, block_size, NUM_BLOCKS + ) + + # Reference output via the standard plan() + workspace_ref = torch.empty(128 * 1024 * 1024, dtype=torch.int8) + ref_wrapper = flashinfer.BatchDecodeWithPagedKVCacheWrapper( + workspace_ref, "NHD", use_tensor_cores=True + ) + ref_wrapper.plan( + kv_indptr, + kv_indices, + kv_last_page_lens, + num_query_heads, + num_kv_heads, + head_size, + block_size, + "NONE", + q_data_type=dtype, + kv_data_type=dtype, + ) + ref_output = ref_wrapper.run(query, key_value_cache) + + # CUDAGraph wrapper exercised through fast_plan_decode + kv_indices_buf = kv_indices.clone() + workspace_cg = torch.empty(128 * 1024 * 1024, dtype=torch.int8) + cg_wrapper = _make_cg_decode_wrapper(num_seqs, kv_indices_buf, workspace_cg) + + plan_kwargs: dict = dict( + indptr_cpu=kv_indptr, + indices=kv_indices_buf, + last_page_len_cpu=kv_last_page_lens, + num_qo_heads=num_query_heads, + num_kv_heads=num_kv_heads, + head_dim=head_size, + page_size=block_size, + q_data_type=dtype, + kv_data_type=dtype, + ) + + # First call – warmup path (routes through self.plan) + fast_plan_decode(cg_wrapper, **plan_kwargs) + warmup_output = cg_wrapper.run(query, key_value_cache) + torch.testing.assert_close(warmup_output, ref_output, atol=1e-2, rtol=1e-2) + + # Second call – fast path (routes through fast_decode_plan from FlashInfer) + fast_plan_decode(cg_wrapper, **plan_kwargs) + fast_output = cg_wrapper.run(query, key_value_cache) + torch.testing.assert_close(fast_output, ref_output, atol=1e-2, rtol=1e-2) + + +@pytest.mark.parametrize("kv_lens", [[1328, 18, 463], [1, 54, 293, 70]]) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("soft_cap", SOFT_CAPS) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOWS) +@torch.inference_mode +def test_flashinfer_decode_with_paged_kv( + kv_lens: list[int], + num_heads: tuple[int, int], + head_size: int, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + sliding_window: int | None, +) -> None: + torch.set_default_device("cuda") + set_random_seed(0) + num_seqs = len(kv_lens) + num_query_heads = num_heads[0] + num_kv_heads = num_heads[1] + assert num_query_heads % num_kv_heads == 0 + max_kv_len = max(kv_lens) + scale = head_size**-0.5 + + query = torch.randn(num_seqs, num_query_heads, head_size, dtype=dtype) + + key_value_cache = torch.randn( + NUM_BLOCKS, 2, block_size, num_kv_heads, head_size, dtype=dtype + ) + key_cache = key_value_cache[:, 0, :, :, :].squeeze(1) + value_cache = key_value_cache[:, 1, :, :, :].squeeze(1) + + max_num_blocks_per_seq = (max_kv_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, NUM_BLOCKS, (num_seqs, max_num_blocks_per_seq), dtype=torch.int32 + ) + + kv_indptr = [0] + kv_indices = [] + kv_last_page_lens = [] + for i in range(num_seqs): + seq_len = kv_lens[i] + assert seq_len > 0 + num_blocks = (seq_len + block_size - 1) // block_size + kv_indices.extend(block_tables[i, :num_blocks]) + kv_indptr.append(kv_indptr[-1] + num_blocks) + kv_last_page_len = seq_len % block_size + if kv_last_page_len == 0: + kv_last_page_len = block_size + kv_last_page_lens.append(kv_last_page_len) + + kv_indptr = torch.tensor(kv_indptr, dtype=torch.int32) + kv_indices = torch.tensor(kv_indices, dtype=torch.int32) + kv_last_page_lens = torch.tensor(kv_last_page_lens, dtype=torch.int32) + + workspace_buffer = torch.empty(128 * 1024 * 1024, dtype=torch.int8) + wrapper = flashinfer.BatchDecodeWithPagedKVCacheWrapper( + workspace_buffer, "NHD", use_tensor_cores=True + ) + wrapper.plan( + kv_indptr, + kv_indices, + kv_last_page_lens, + num_query_heads, + num_kv_heads, + head_size, + block_size, + "NONE", + window_left=sliding_window - 1 if sliding_window is not None else -1, + q_data_type=dtype, + kv_data_type=dtype, + logits_soft_cap=soft_cap, + ) + + output = wrapper.run(query, key_value_cache) + + ref_output = ref_paged_attn( + query=query, + key_cache=key_cache, + value_cache=value_cache, + query_lens=[1] * num_seqs, + kv_lens=kv_lens, + block_tables=block_tables, + scale=scale, + soft_cap=soft_cap, + sliding_window=sliding_window, + ) + ( + torch.testing.assert_close(output, ref_output, atol=1e-2, rtol=1e-2), + f"{torch.max(torch.abs(output - ref_output))}", + ) + + +@pytest.mark.parametrize("seq_lens", [[(1, 1328), (5, 18), (129, 463)]]) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("soft_cap", SOFT_CAPS) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOWS) +@torch.inference_mode +def test_flashinfer_prefill_with_paged_kv( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + sliding_window: int | None, +) -> None: + torch.set_default_device("cuda") + set_random_seed(0) + num_seqs = len(seq_lens) + query_lens = [x[0] for x in seq_lens] + kv_lens = [x[1] for x in seq_lens] + num_query_heads = num_heads[0] + num_kv_heads = num_heads[1] + assert num_query_heads % num_kv_heads == 0 + max_kv_len = max(kv_lens) + scale = head_size**-0.5 + + query = torch.randn(sum(query_lens), num_query_heads, head_size, dtype=dtype) + key_value_cache = torch.randn( + NUM_BLOCKS, 2, block_size, num_kv_heads, head_size, dtype=dtype + ) + key_cache = key_value_cache[:, 0, :, :, :].squeeze(1) + value_cache = key_value_cache[:, 1, :, :, :].squeeze(1) + + # Normalize the scale of the key and value caches to mitigate + # numerical instability. + key_cache /= head_size**0.5 + value_cache /= head_size**0.5 + + max_num_blocks_per_seq = (max_kv_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, NUM_BLOCKS, (num_seqs, max_num_blocks_per_seq), dtype=torch.int32 + ) + + qo_indptr = [0] + kv_indptr = [0] + kv_indices = [] + kv_last_page_lens = [] + for i in range(num_seqs): + seq_len = kv_lens[i] + assert seq_len > 0 + num_blocks = (seq_len + block_size - 1) // block_size + kv_indices.extend(block_tables[i, :num_blocks]) + kv_indptr.append(kv_indptr[-1] + num_blocks) + kv_last_page_len = seq_len % block_size + if kv_last_page_len == 0: + kv_last_page_len = block_size + kv_last_page_lens.append(kv_last_page_len) + qo_indptr.append(qo_indptr[-1] + query_lens[i]) + + qo_indptr = torch.tensor(qo_indptr, dtype=torch.int32) + kv_indptr = torch.tensor(kv_indptr, dtype=torch.int32) + kv_indices = torch.tensor(kv_indices, dtype=torch.int32) + kv_last_page_lens = torch.tensor(kv_last_page_lens, dtype=torch.int32) + + workspace_buffer = torch.empty(128 * 1024 * 1024, dtype=torch.int8) + wrapper = flashinfer.BatchPrefillWithPagedKVCacheWrapper(workspace_buffer, "NHD") + wrapper.plan( + qo_indptr, + kv_indptr, + kv_indices, + kv_last_page_lens, + num_query_heads, + num_kv_heads, + head_size, + block_size, + window_left=sliding_window - 1 if sliding_window is not None else -1, + q_data_type=dtype, + kv_data_type=dtype, + logits_soft_cap=soft_cap, + ) + + output = wrapper.run( + query, + key_value_cache, + ) + + ref_output = ref_paged_attn( + query=query, + key_cache=key_cache, + value_cache=value_cache, + query_lens=query_lens, + kv_lens=kv_lens, + block_tables=block_tables, + scale=scale, + soft_cap=soft_cap, + sliding_window=sliding_window, + ) + ( + torch.testing.assert_close(output, ref_output, atol=5e-2, rtol=1e-2), + f"{torch.max(torch.abs(output - ref_output))}", + ) + + +@pytest.mark.parametrize("seq_lens", [[(1, 132), (5, 18)]]) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("soft_cap", SOFT_CAPS) +def test_flashinfer_prefill_with_paged_fp8_kv( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, +) -> None: + pytest.skip("TODO: fix the accuracy issue") + torch.set_default_device("cuda") + set_random_seed(0) + num_seqs = len(seq_lens) + query_lens = [x[0] for x in seq_lens] + kv_lens = [x[1] for x in seq_lens] + num_query_heads = num_heads[0] + num_kv_heads = num_heads[1] + assert num_query_heads % num_kv_heads == 0 + max_kv_len = max(kv_lens) + scale = head_size**-0.5 + + kv_cache_dtype = torch.float8_e4m3fn + + query = torch.randn(sum(query_lens), num_query_heads, head_size, dtype=dtype) + NUM_BLOCKS_FP8 = 2048 + key_value_cache = torch.randn( + NUM_BLOCKS_FP8, 2, block_size, num_kv_heads, head_size, dtype=dtype + ) + key_cache, value_cache = torch.chunk(key_value_cache, 2, dim=1) + key_cache /= head_size**0.5 + value_cache /= head_size**0.5 + + k_scale = key_cache.amax().item() / 448.0 + v_scale = value_cache.amax().item() / 448.0 + + kv_cache_fp8 = torch.cat([key_cache / k_scale, value_cache / v_scale], dim=1).to( + kv_cache_dtype + ) + + assert kv_cache_fp8.shape == key_value_cache.shape + max_num_blocks_per_seq = (max_kv_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, NUM_BLOCKS_FP8, (num_seqs, max_num_blocks_per_seq), dtype=torch.int32 + ) + + qo_indptr = [0] + kv_indptr = [0] + kv_indices = [] + kv_last_page_lens = [] + for i in range(num_seqs): + seq_len = kv_lens[i] + assert seq_len > 0 + num_blocks = (seq_len + block_size - 1) // block_size + kv_indices.extend(block_tables[i, :num_blocks]) + kv_indptr.append(kv_indptr[-1] + num_blocks) + kv_last_page_len = seq_len % block_size + if kv_last_page_len == 0: + kv_last_page_len = block_size + kv_last_page_lens.append(kv_last_page_len) + qo_indptr.append(qo_indptr[-1] + query_lens[i]) + + qo_indptr = torch.tensor(qo_indptr, dtype=torch.int32) + kv_indptr = torch.tensor(kv_indptr, dtype=torch.int32) + kv_indices = torch.tensor(kv_indices, dtype=torch.int32) + kv_last_page_lens = torch.tensor(kv_last_page_lens, dtype=torch.int32) + + workspace_buffer = torch.empty(128 * 1024 * 1024, dtype=torch.int8) + wrapper = flashinfer.BatchPrefillWithPagedKVCacheWrapper(workspace_buffer, "NHD") + wrapper.plan( + qo_indptr, + kv_indptr, + kv_indices, + kv_last_page_lens, + num_query_heads, + num_kv_heads, + head_size, + block_size, + q_data_type=dtype, + kv_data_type=kv_cache_dtype, + logits_soft_cap=soft_cap, + ) + + output = wrapper.run(query, kv_cache_fp8, k_scale=k_scale, v_scale=v_scale) + + ref_output = ref_paged_attn( + query=query, + key_cache=key_cache.squeeze(1), + value_cache=value_cache.squeeze(1), + query_lens=query_lens, + kv_lens=kv_lens, + block_tables=block_tables, + scale=scale, + soft_cap=soft_cap, + ) + del query + del block_tables + # verify prefill fp8 + ( + torch.testing.assert_close(output, ref_output, atol=5e-2, rtol=1e-2), + f"{torch.max(torch.abs(output - ref_output))}", + ) + + +@pytest.mark.parametrize("kv_lens", [[1328, 18, 463], [1, 54, 293, 70]]) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("soft_cap", SOFT_CAPS) +@pytest.mark.skip(reason="TODO: fix the accuracy issue") +@torch.inference_mode +def test_flashinfer_decode_with_paged_fp8_kv( + kv_lens: list[int], + num_heads: tuple[int, int], + head_size: int, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, +) -> None: + # test doesn't work for num_heads = (16,16) + torch.set_default_device("cuda") + set_random_seed(0) + num_seqs = len(kv_lens) + num_query_heads = num_heads[0] + num_kv_heads = num_heads[1] + assert num_query_heads % num_kv_heads == 0 + max_kv_len = max(kv_lens) + scale = head_size**-0.5 + use_tensor_cores = True + kv_cache_dtype = torch.float8_e4m3fn + + query = torch.randn(num_seqs, num_query_heads, head_size, dtype=dtype) + NUM_BLOCKS_FP8 = 2048 + key_value_cache = torch.randn( + NUM_BLOCKS_FP8, 2, block_size, num_kv_heads, head_size, dtype=dtype + ) + key_cache, value_cache = torch.chunk(key_value_cache, 2, dim=1) + key_cache /= head_size**0.5 + value_cache /= head_size**0.5 + + k_scale = key_cache.amax().item() / 448.0 + v_scale = value_cache.amax().item() / 448.0 + + key_cache_fp8 = (key_cache / k_scale).to(kv_cache_dtype) + value_cache_fp8 = (value_cache / v_scale).to(kv_cache_dtype) + assert key_cache_fp8.shape[1] == 1 and value_cache_fp8.shape[1] == 1 + kv_cache_fp8 = torch.cat([key_cache_fp8, value_cache_fp8], dim=1) + + max_num_blocks_per_seq = (max_kv_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, NUM_BLOCKS_FP8, (num_seqs, max_num_blocks_per_seq), dtype=torch.int32 + ) + + kv_indptr = [0] + kv_indices = [] + kv_last_page_lens = [] + for i in range(num_seqs): + seq_len = kv_lens[i] + assert seq_len > 0 + num_blocks = (seq_len + block_size - 1) // block_size + kv_indices.extend(block_tables[i, :num_blocks]) + kv_indptr.append(kv_indptr[-1] + num_blocks) + kv_last_page_len = seq_len % block_size + if kv_last_page_len == 0: + kv_last_page_len = block_size + kv_last_page_lens.append(kv_last_page_len) + + kv_indptr = torch.tensor(kv_indptr, dtype=torch.int32) + kv_indices = torch.tensor(kv_indices, dtype=torch.int32) + kv_last_page_lens = torch.tensor(kv_last_page_lens, dtype=torch.int32) + + workspace_buffer = torch.empty(128 * 1024 * 1024, dtype=torch.int8) + wrapper = flashinfer.BatchDecodeWithPagedKVCacheWrapper( + workspace_buffer, "NHD", use_tensor_cores=use_tensor_cores + ) + wrapper.plan( + kv_indptr, + kv_indices, + kv_last_page_lens, + num_query_heads, + num_kv_heads, + head_size, + block_size, + "NONE", + q_data_type=dtype, + kv_data_type=kv_cache_dtype, + logits_soft_cap=soft_cap, + ) + output = wrapper.run(query, kv_cache_fp8, k_scale=k_scale, v_scale=v_scale) + key_cache = key_value_cache[:, 0, :, :, :].squeeze(1) + value_cache = key_value_cache[:, 1, :, :, :].squeeze(1) + + ref_output = ref_paged_attn( + query=query, + key_cache=key_cache, + value_cache=value_cache, + query_lens=[1] * num_seqs, + kv_lens=kv_lens, + block_tables=block_tables, + scale=scale, + soft_cap=soft_cap, + ) + # Temporary fix: Increasing the tolerance. Seems like a flashinfer issue + ( + torch.testing.assert_close(output, ref_output, atol=2e-2, rtol=1e-2), + f"{torch.max(torch.abs(output - ref_output))}", + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_flashinfer_mla_decode.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_flashinfer_mla_decode.py new file mode 100644 index 0000000000000000000000000000000000000000..fb176a8fba8cf85580c6b4e6498b36eb1af5545c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_flashinfer_mla_decode.py @@ -0,0 +1,119 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch +import torch.nn.functional as F +from torch import Tensor + +from vllm.platforms import current_platform + +FLASHINFER_WORKSPACE_BUFFER_SIZE = 128 * 1024 * 1024 + +if not current_platform.has_device_capability(100): + pytest.skip( + reason="FlashInfer MLA Requires compute capability of 10 or above.", + allow_module_level=True, + ) +else: + from flashinfer.decode import trtllm_batch_decode_with_kv_cache_mla + + +def ref_mla( + out: Tensor, # (bs, num_heads, v_head_dim) + query: Tensor, # (bs, num_heads, head_dim) + kv_cache: Tensor, # (num_blocks, block_size, head_dim) + scale: float, + block_tables: Tensor, # (bs, max_num_blocks) + seq_lens: Tensor, # (bs,) +): + bs, num_heads, v_head_dim = out.shape + head_dim = query.shape[2] + + for i in range(bs): + # gather and flatten KV-cache + kv = kv_cache[block_tables[i]] # (max_num_blocks, block_size, head_dim) + kv = kv.view(1, -1, head_dim)[:, : seq_lens[i]] # (1, seq_len, head_dim) + v = kv[:, :, :v_head_dim] + + q = query[i].view(num_heads, 1, head_dim) + o = F.scaled_dot_product_attention(q, kv, v, scale=scale, enable_gqa=True) + out[i] = o.view(num_heads, v_head_dim) + + return out + + +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("bs", [1, 2, 4, 16]) +@pytest.mark.parametrize("block_size", [32, 64]) +def test_flashinfer_mla_decode(dtype: torch.dtype, bs: int, block_size: int): + torch.set_default_device("cuda") + torch.manual_seed(42) + + # Deepseek R1 config + num_heads = 128 + kv_lora_rank = 512 + qk_nope_head_dim = 128 + qk_rope_head_dim = 64 + qk_head_dim = kv_lora_rank + qk_rope_head_dim + scale = (qk_nope_head_dim + qk_rope_head_dim) ** -0.5 + + MAX_SEQ_LEN = 1024 + + seq_lens = [torch.randint(2, MAX_SEQ_LEN, (1,)).item() for _ in range(bs)] + seq_lens[-1] = MAX_SEQ_LEN + max_seq_len = max(seq_lens) + seq_lens_tensor = torch.tensor(seq_lens, dtype=torch.int32) + + # Generate block tables with random but unique block IDs + # From https://github.com/flashinfer-ai/flashinfer/pull/1222 + blocks_per_seq = (seq_lens_tensor + block_size - 1) // block_size + max_num_blocks_per_seq = max(blocks_per_seq.max().item(), 4) + total_blocks_needed = sum(blocks_per_seq) + # Get random unique IDs for all blocks + all_block_ids = torch.randperm(total_blocks_needed) + + block_id = 0 + block_tables = torch.zeros( + (bs, max_num_blocks_per_seq), + dtype=torch.int32, + ) + + # Populate block tables and track block assignments + block_id = 0 + for i in range(bs): + num_blocks_needed = blocks_per_seq[i] + block_tables[i, :num_blocks_needed] = all_block_ids[ + block_id : block_id + num_blocks_needed + ] + block_id += num_blocks_needed + + kv_cache = torch.randn(block_tables.numel(), block_size, qk_head_dim).to(dtype) + q = torch.randn(bs, num_heads, qk_head_dim).to(dtype) + + out_ref = q.new_zeros(bs, num_heads, kv_lora_rank) + ref_mla(out_ref, q, kv_cache, scale, block_tables, seq_lens_tensor) + + workspace_buffer = torch.zeros( + FLASHINFER_WORKSPACE_BUFFER_SIZE, + dtype=torch.uint8, + device=q.device, + ) + # Flashinfer MLA expects the query to be of shape + # (bs, q_len_per_request, num_heads, qk_head_dim), + # where q_len_per_request is the MTP query length (=1 without MTP) + q = q.unsqueeze(1) + + out_ans = trtllm_batch_decode_with_kv_cache_mla( + query=q, + kv_cache=kv_cache.unsqueeze(1), + workspace_buffer=workspace_buffer, + qk_nope_head_dim=qk_nope_head_dim, + kv_lora_rank=kv_lora_rank, + qk_rope_head_dim=qk_rope_head_dim, + block_tables=block_tables, + seq_lens=seq_lens_tensor, + max_seq_len=max_seq_len, + bmm1_scale=scale, + ) + out_ans = out_ans.squeeze(1) + torch.testing.assert_close(out_ans, out_ref, atol=1e-2, rtol=1e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_flashinfer_trtllm_attention.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_flashinfer_trtllm_attention.py new file mode 100644 index 0000000000000000000000000000000000000000..a5dabcfddf1dd6af221443dbaa86977e76ded837 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_flashinfer_trtllm_attention.py @@ -0,0 +1,554 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from tests.kernels.quantization.nvfp4_utils import ( + dequant_nvfp4_kv_cache, + dequantize_nvfp4_to_dtype, + get_nvfp4_global_scale, +) +from vllm.platforms import current_platform +from vllm.utils.math_utils import round_up +from vllm.utils.torch_utils import ( + nvfp4_kv_cache_full_dim, + nvfp4_kv_cache_split_views, + set_random_seed, +) + +if not current_platform.is_device_capability_family(100): + pytest.skip( + "This TRTLLM kernel requires NVIDIA Blackwell.", allow_module_level=True + ) +else: + import flashinfer + +FLOAT32_BYTES = torch.finfo(torch.float).bits // 8 +FP8_DTYPE = current_platform.fp8_dtype() +FP4_DTYPE = torch.uint8 + + +def to_float8(x, dtype=torch.float8_e4m3fn): + finfo = torch.finfo(dtype) + min_val, max_val = x.aminmax() + amax = torch.maximum(min_val.abs(), max_val.abs()).clamp(min=1e-12) + scale = finfo.max / amax * 0.1 + x_scl_sat = (x * scale).clamp(min=finfo.min, max=finfo.max) + return x_scl_sat.to(dtype), scale.float().reciprocal() + + +def build_paged_kv_metadata( + seq_lens: torch.Tensor, + block_tables: torch.Tensor, + block_size: int, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """Build paged-KV indptr/indices/last_page_lens from seq_lens + block_tables.""" + kv_indptr = [0] + kv_indices = [] + kv_last_page_lens = [] + for i in range(len(seq_lens)): + sl = int(seq_lens[i]) + assert sl > 0 + nb = (sl + block_size - 1) // block_size + kv_indices.extend(block_tables[i, :nb].tolist()) + kv_indptr.append(kv_indptr[-1] + nb) + kv_last_page_lens.append(sl % block_size or block_size) + return ( + torch.tensor(kv_indptr, dtype=torch.int32), + torch.tensor(kv_indices, dtype=torch.int32), + torch.tensor(kv_last_page_lens, dtype=torch.int32), + ) + + +def make_nvfp4_kv_cache( + kv_bf16_hnd: torch.Tensor, block_size: int, head_size: int +) -> tuple: + """Quantize bf16 KV cache to nvfp4 via reshape_and_cache_flash. + + Returns (k_data, v_data), (k_scales, v_scales), kv_scale, ref_kv_bf16. + """ + num_blocks, _, num_kv_heads, _, _ = kv_bf16_hnd.shape + kv_scale_val = (kv_bf16_hnd.abs().amax() / 448.0).item() + kv_scale_tensor = torch.tensor( + kv_scale_val, dtype=torch.float32, device=kv_bf16_hnd.device + ) + + # Allocate in HND physical order, permute to NHD logical order. + # hnd_order swaps dims 2↔3; it is its own inverse. + full_dim = nvfp4_kv_cache_full_dim(head_size) + hnd_order = (0, 1, 3, 2, 4) + kv_cache = torch.zeros( + (num_blocks, 2, num_kv_heads, block_size, full_dim), + dtype=torch.uint8, + device=kv_bf16_hnd.device, + ).permute(*hnd_order) + + # Flatten NHD [N, T, H, D] → token tensors [N*T, H, D] for the kernel. + num_tokens = num_blocks * block_size + k_tokens = ( + kv_bf16_hnd[:, 0] + .permute(0, 2, 1, 3) + .reshape(num_tokens, num_kv_heads, head_size) + ) + v_tokens = ( + kv_bf16_hnd[:, 1] + .permute(0, 2, 1, 3) + .reshape(num_tokens, num_kv_heads, head_size) + ) + slot_mapping = torch.arange(num_tokens, dtype=torch.long, device=kv_bf16_hnd.device) + + # reshape_and_cache_flash: kernel receives kv_cache[:, 0] and [:, 1] + # (full K/V buffers containing both data and scale). + torch.ops._C_cache_ops.reshape_and_cache_flash( + k_tokens, + v_tokens, + kv_cache[:, 0], + kv_cache[:, 1], + slot_mapping, + "nvfp4", + kv_scale_tensor, + kv_scale_tensor, + ) + + # Split in HND order for trtllm kernel (expects HND numTokensPerPage). + kv_cache_hnd = kv_cache.permute(*hnd_order) + (k_data, v_data), (k_scales, v_scales) = nvfp4_kv_cache_split_views(kv_cache_hnd) + + # Dequantize for the FA2 reference baseline. + ref_k = dequant_nvfp4_kv_cache( + k_data, k_scales, kv_scale_val, head_size, block_size + ).to(torch.bfloat16) + ref_v = dequant_nvfp4_kv_cache( + v_data, v_scales, kv_scale_val, head_size, block_size + ).to(torch.bfloat16) + ref_kv_bf16 = torch.stack([ref_k, ref_v], dim=1) # [N, 2, H, T, D] + + return (k_data, v_data), (k_scales, v_scales), kv_scale_val, ref_kv_bf16 + + +def make_quantized_kv_cache( + kv_cache: torch.Tensor, + kv_quant_dtype: torch.dtype, + block_size: int, + head_size: int, +) -> tuple: + """Quantize kv_cache based on dtype. Returns (kv_cache, kv_cache_sf, + kv_scale, ref_kv_cache, is_nvfp4_kv).""" + is_nvfp4_kv = kv_quant_dtype == FP4_DTYPE + if is_nvfp4_kv: + data, scales, kv_scale, ref = make_nvfp4_kv_cache( + kv_cache, block_size, head_size + ) + return data, scales, kv_scale, ref, True + elif kv_quant_dtype == FP8_DTYPE: + kv_fp8, kv_scale = to_float8(kv_cache) + ref = kv_fp8.to(kv_cache.dtype) * kv_scale + return kv_fp8, None, kv_scale, ref, False + else: + return kv_cache, None, 1.0, kv_cache, False + + +DTYPE = [torch.bfloat16] +QUANT_DTYPES = [ + # (q_quant_dtype, kv_quant_dtype, o_quant_dtype) + (None, None, None), + (None, FP8_DTYPE, None), + (FP8_DTYPE, FP8_DTYPE, None), + (FP8_DTYPE, FP8_DTYPE, FP8_DTYPE), + (FP8_DTYPE, FP8_DTYPE, FP4_DTYPE), + (FP8_DTYPE, FP4_DTYPE, FP8_DTYPE), # nvfp4 KV cache +] +BATCH_SIZE = [4, 12] +MAX_SEQ_LENS = [(1024, 4096)] +NUM_HEADS = [(64, 8), (40, 8)] +HEAD_SIZE = [128] +KV_LAYOUT = ["HND"] # currently only HND is supported +BLOCK_SIZE = [16] +WINDOW_LEFT = [-1, 127] +SOFT_CAP = [None, 50.0] +HAS_SINKS = [True, False] + +NUM_BLOCKS = 32768 # Large enough to test overflow in index calculation. + + +@pytest.mark.parametrize("dtype", DTYPE) +@pytest.mark.parametrize("quant_dtypes", QUANT_DTYPES) +@pytest.mark.parametrize("batch_size", BATCH_SIZE) +@pytest.mark.parametrize("max_seq_lens", MAX_SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZE) +@pytest.mark.parametrize("kv_layout", KV_LAYOUT) +@pytest.mark.parametrize("block_size", BLOCK_SIZE) +@pytest.mark.parametrize("window_left", WINDOW_LEFT) +@pytest.mark.parametrize("soft_cap", SOFT_CAP) +@pytest.mark.parametrize("has_sinks", HAS_SINKS) +@torch.inference_mode +def test_flashinfer_trtllm_decode_with_baseline( + dtype: torch.dtype, + quant_dtypes: tuple[torch.dtype | None, torch.dtype | None, torch.dtype | None], + batch_size: int, + max_seq_lens: tuple[int, int], + num_heads: tuple[int, int], + head_size: int, + kv_layout: str, + block_size: int, + window_left: int, + soft_cap: float | None, + has_sinks: bool, +) -> None: + torch.set_default_device("cuda") + set_random_seed(42) + + q_quant_dtype, kv_quant_dtype, o_quant_dtype = quant_dtypes + q_quant_dtype = q_quant_dtype or dtype + kv_quant_dtype = kv_quant_dtype or dtype + o_quant_dtype = o_quant_dtype or dtype + + _, max_kv_len = max_seq_lens + + num_qo_heads, num_kv_heads = num_heads + assert num_qo_heads % num_kv_heads == 0 + + sm_scale = float(1.0 / (head_size**0.5)) + + kv_cache_shape = None + if kv_layout == "NHD": + kv_cache_shape = (NUM_BLOCKS, 2, block_size, num_kv_heads, head_size) + elif kv_layout == "HND": + kv_cache_shape = (NUM_BLOCKS, 2, num_kv_heads, block_size, head_size) + else: + raise ValueError(f"Invalid kv_layout: {kv_layout}") + + # max_q_len = 1 + q_lens = torch.ones((batch_size,), dtype=torch.int32) + q_indptr = torch.cat( + [ + torch.tensor([0], dtype=torch.int32), + torch.cumsum(q_lens, dim=0, dtype=torch.int32), + ] + ) + + query = torch.randn(torch.sum(q_lens).item(), num_qo_heads, head_size, dtype=dtype) + if q_quant_dtype == FP8_DTYPE: + query, q_scale = to_float8(query) + ref_query = query.to(dtype) * q_scale + else: + q_scale = 1.0 + ref_query = query + + kv_lens = torch.randint(1, max_kv_len, (batch_size,), dtype=torch.int32) + kv_lens[-1] = max_kv_len + + seq_lens = kv_lens + q_lens + max_seq_len = torch.max(seq_lens).item() + + kv_cache = torch.randn(kv_cache_shape, dtype=dtype) + kv_cache, kv_cache_sf, kv_scale, ref_kv_cache, is_nvfp4_kv = ( + make_quantized_kv_cache(kv_cache, kv_quant_dtype, block_size, head_size) + ) + + k_scale = v_scale = kv_scale + + max_num_blocks_per_seq = (max_seq_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, NUM_BLOCKS, (batch_size, max_num_blocks_per_seq), dtype=torch.int32 + ) + kv_indptr, kv_indices, kv_last_page_lens = build_paged_kv_metadata( + seq_lens, block_tables, block_size + ) + workspace_buffer = torch.zeros(128 * 1024 * 1024, dtype=torch.int8) + + # Baseline Decode + if has_sinks: + sinks = torch.rand(num_qo_heads, dtype=torch.float32) * 5 + wrapper = flashinfer.BatchAttentionWithAttentionSinkWrapper( + float_workspace_buffer=workspace_buffer, kv_layout=kv_layout, backend="fa2" + ) + else: + sinks = None + wrapper = flashinfer.BatchPrefillWithPagedKVCacheWrapper( + float_workspace_buffer=workspace_buffer, kv_layout=kv_layout, backend="fa2" + ) + + wrapper.plan( + qo_indptr=q_indptr, + paged_kv_indptr=kv_indptr, + paged_kv_indices=kv_indices, + paged_kv_last_page_len=kv_last_page_lens, + num_qo_heads=num_qo_heads, + num_kv_heads=num_kv_heads, + head_dim_qk=head_size, + page_size=block_size, + causal=True, + sm_scale=sm_scale, + window_left=window_left, + logits_soft_cap=soft_cap, + q_data_type=dtype, + kv_data_type=dtype, + ) + output = torch.empty(ref_query.shape, dtype=dtype) + wrapper.run(ref_query, ref_kv_cache, sinks, sm_scale, out=output) + + o_scale = 1.0 + o_sf_scale_float = None + if o_quant_dtype == FP8_DTYPE: + _, o_scale = to_float8(output) + elif o_quant_dtype == FP4_DTYPE: + o_sf_scale = get_nvfp4_global_scale(output) + o_sf_scale_float = o_sf_scale.item() + + # TRTLLM Decode + if o_quant_dtype == FP4_DTYPE: + output_trtllm = flashinfer.utils.FP4Tensor( + torch.empty(query.shape[:-1] + (query.shape[-1] // 2,), dtype=torch.uint8), + torch.empty( + ( + round_up(query.shape[0], 128), + round_up(query.shape[1] * query.shape[2] // 16, 4), + ), + dtype=torch.float8_e4m3fn, + ), + ) + else: + output_trtllm = torch.empty(query.shape, dtype=o_quant_dtype) + + flashinfer.decode.trtllm_batch_decode_with_kv_cache( + query=query, + kv_cache=kv_cache, + workspace_buffer=workspace_buffer, + block_tables=block_tables, + seq_lens=seq_lens, + max_seq_len=max_seq_len, + bmm1_scale=q_scale * k_scale * sm_scale, + bmm2_scale=v_scale / o_scale, + window_left=window_left, + sinks=sinks, + o_sf_scale=o_sf_scale_float, + out=output_trtllm, + kv_cache_sf=kv_cache_sf, + ) + if o_quant_dtype == FP8_DTYPE: + output_trtllm = output_trtllm.to(dtype) * o_scale + elif o_quant_dtype == FP4_DTYPE: + output_trtllm.data = output_trtllm.data.reshape( + -1, query.shape[1] * query.shape[2] // 2 + ) + output_trtllm = dequantize_nvfp4_to_dtype( + output_trtllm.data, output_trtllm.scale, o_sf_scale, dtype, query.device + ) + output_trtllm = output_trtllm.reshape(-1, query.shape[1], query.shape[2]) + + if is_nvfp4_kv: + rtol, atol = 1.0, 1.0 # nvfp4 has higher quantization error + elif q_quant_dtype == FP8_DTYPE and o_quant_dtype == FP4_DTYPE: + rtol, atol = 7e-2, 9e-2 + elif q_quant_dtype == FP8_DTYPE and o_quant_dtype == FP8_DTYPE: + rtol, atol = 3e-2, 4e-2 + elif q_quant_dtype == FP8_DTYPE and o_quant_dtype == dtype: + rtol, atol = 2e-2, 2e-2 + elif kv_quant_dtype == FP8_DTYPE: + rtol, atol = 4e-2, 6e-2 + else: + rtol, atol = 1e-2, 1e-2 + + ( + torch.testing.assert_close(output, output_trtllm, atol=atol, rtol=rtol), + f"{torch.max(torch.abs(output - output_trtllm))}", + ) + + +@pytest.mark.parametrize("dtype", DTYPE) +@pytest.mark.parametrize("quant_dtypes", QUANT_DTYPES) +@pytest.mark.parametrize("batch_size", BATCH_SIZE) +@pytest.mark.parametrize("max_seq_lens", MAX_SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZE) +@pytest.mark.parametrize("kv_layout", KV_LAYOUT) +@pytest.mark.parametrize("block_size", BLOCK_SIZE) +@pytest.mark.parametrize("window_left", WINDOW_LEFT) +@pytest.mark.parametrize("soft_cap", [None]) +@pytest.mark.parametrize("has_sinks", HAS_SINKS) +@torch.inference_mode +def test_flashinfer_trtllm_prefill_with_baseline( + dtype: torch.dtype, + quant_dtypes: tuple[torch.dtype | None, torch.dtype | None, torch.dtype | None], + batch_size: int, + max_seq_lens: tuple[int, int], + num_heads: tuple[int, int], + head_size: int, + kv_layout: str, + block_size: int, + window_left: int, + soft_cap: float | None, + has_sinks: bool, +) -> None: + torch.set_default_device("cuda") + set_random_seed(42) + + q_quant_dtype, kv_quant_dtype, o_quant_dtype = quant_dtypes + q_quant_dtype = q_quant_dtype or dtype + kv_quant_dtype = kv_quant_dtype or dtype + o_quant_dtype = o_quant_dtype or dtype + + # FP8 Q + nvfp4 KV is the required combination for the nvfp4 KV path. + # All other mixed Q/KV dtype combinations are unsupported. + is_nvfp4_kv = kv_quant_dtype == FP4_DTYPE + if q_quant_dtype != kv_quant_dtype and not ( + q_quant_dtype == FP8_DTYPE and is_nvfp4_kv + ): + pytest.skip("Skipped mixed QKV dtypes for prefill") + + max_q_len, max_kv_len = max_seq_lens + + num_qo_heads, num_kv_heads = num_heads + assert num_qo_heads % num_kv_heads == 0 + + sm_scale = float(1.0 / (head_size**0.5)) + + kv_cache_shape = None + if kv_layout == "NHD": + kv_cache_shape = (NUM_BLOCKS, 2, block_size, num_kv_heads, head_size) + elif kv_layout == "HND": + kv_cache_shape = (NUM_BLOCKS, 2, num_kv_heads, block_size, head_size) + else: + raise ValueError(f"Invalid kv_layout: {kv_layout}") + + q_lens = torch.randint(1, max_q_len, (batch_size,), dtype=torch.int32) + q_lens[-1] = max_q_len + q_indptr = torch.cat( + [ + torch.tensor([0], dtype=torch.int32), + torch.cumsum(q_lens, dim=0, dtype=torch.int32), + ] + ) + + query = torch.randn(torch.sum(q_lens).item(), num_qo_heads, head_size, dtype=dtype) + if q_quant_dtype == FP8_DTYPE: + query, q_scale = to_float8(query) + ref_query = query.to(dtype) * q_scale + else: + q_scale = 1.0 + ref_query = query + + kv_lens = torch.randint(1, max_kv_len, (batch_size,), dtype=torch.int32) + kv_lens[-1] = max_kv_len + + seq_lens = kv_lens + q_lens + max_seq_len = torch.max(seq_lens).item() + + kv_cache = torch.randn(kv_cache_shape, dtype=dtype) + kv_cache, kv_cache_sf, kv_scale, ref_kv_cache, is_nvfp4_kv = ( + make_quantized_kv_cache(kv_cache, kv_quant_dtype, block_size, head_size) + ) + + k_scale = v_scale = kv_scale + + max_num_blocks_per_seq = (max_seq_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, NUM_BLOCKS, (batch_size, max_num_blocks_per_seq), dtype=torch.int32 + ) + kv_indptr, kv_indices, kv_last_page_lens = build_paged_kv_metadata( + seq_lens, block_tables, block_size + ) + workspace_buffer = torch.zeros(128 * 1024 * 1024, dtype=torch.int8) + + # Baseline Prefill + if has_sinks: + sinks = torch.rand(num_qo_heads, dtype=torch.float32) * 5 + wrapper = flashinfer.BatchAttentionWithAttentionSinkWrapper( + float_workspace_buffer=workspace_buffer, kv_layout=kv_layout, backend="fa2" + ) + else: + sinks = None + wrapper = flashinfer.BatchPrefillWithPagedKVCacheWrapper( + float_workspace_buffer=workspace_buffer, kv_layout=kv_layout, backend="fa2" + ) + + wrapper.plan( + qo_indptr=q_indptr, + paged_kv_indptr=kv_indptr, + paged_kv_indices=kv_indices, + paged_kv_last_page_len=kv_last_page_lens, + num_qo_heads=num_qo_heads, + num_kv_heads=num_kv_heads, + head_dim_qk=head_size, + page_size=block_size, + causal=True, + sm_scale=sm_scale, + window_left=window_left, + logits_soft_cap=soft_cap, + q_data_type=dtype, + kv_data_type=dtype, + ) + output = torch.empty(ref_query.shape, dtype=dtype) + wrapper.run(ref_query, ref_kv_cache, sinks, sm_scale, out=output) + + o_scale = 1.0 + o_sf_scale_float = None + if o_quant_dtype == FP8_DTYPE: + _, o_scale = to_float8(output) + elif o_quant_dtype == FP4_DTYPE: + o_sf_scale = get_nvfp4_global_scale(output) + o_sf_scale_float = o_sf_scale.item() + + # TRTLLM Prefill + if o_quant_dtype == FP4_DTYPE: + output_trtllm = flashinfer.utils.FP4Tensor( + torch.empty(query.shape[:-1] + (query.shape[-1] // 2,), dtype=torch.uint8), + torch.empty( + ( + round_up(query.shape[0], 128), + round_up(query.shape[1] * query.shape[2] // 16, 4), + ), + dtype=torch.float8_e4m3fn, + ), + ) + else: + output_trtllm = torch.empty(query.shape, dtype=o_quant_dtype) + + flashinfer.prefill.trtllm_batch_context_with_kv_cache( + query=query, + kv_cache=kv_cache, + workspace_buffer=workspace_buffer, + block_tables=block_tables, + seq_lens=seq_lens, + max_q_len=max_q_len, + max_kv_len=max_seq_len, + bmm1_scale=q_scale * k_scale * sm_scale, + bmm2_scale=v_scale / o_scale, + batch_size=batch_size, + cum_seq_lens_q=q_indptr, + cum_seq_lens_kv=kv_indptr, + window_left=window_left, + sinks=sinks, + o_sf_scale=o_sf_scale_float, + out=output_trtllm, + kv_cache_sf=kv_cache_sf, + ) + if o_quant_dtype == FP8_DTYPE: + output_trtllm = output_trtllm.to(dtype) * o_scale + elif o_quant_dtype == FP4_DTYPE: + output_trtllm.data = output_trtllm.data.reshape( + -1, query.shape[1] * query.shape[2] // 2 + ) + output_trtllm = dequantize_nvfp4_to_dtype( + output_trtllm.data, output_trtllm.scale, o_sf_scale, dtype, query.device + ) + output_trtllm = output_trtllm.reshape(-1, query.shape[1], query.shape[2]) + + if is_nvfp4_kv: + rtol, atol = 1.0, 1.5 # nvfp4 has higher quantization error + elif q_quant_dtype == FP8_DTYPE and o_quant_dtype == FP4_DTYPE: + rtol, atol = 3e-1, 4e-1 + elif q_quant_dtype == FP8_DTYPE and o_quant_dtype == FP8_DTYPE: + rtol, atol = 4e-2, 6e-2 + elif q_quant_dtype == FP8_DTYPE and o_quant_dtype == dtype: + rtol, atol = 2e-2, 3e-2 + else: + rtol, atol = 1e-2, 1e-2 + + ( + torch.testing.assert_close(output, output_trtllm, atol=atol, rtol=rtol), + f"{torch.max(torch.abs(output - output_trtllm))}", + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_flashmla.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_flashmla.py new file mode 100644 index 0000000000000000000000000000000000000000..4b18343bc1b793c40eeb9e1c47051db06433ce13 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_flashmla.py @@ -0,0 +1,178 @@ +# Adapted from: https://github.com/deepseek-ai/FlashMLA/blob/main/tests/test_flash_mla.py +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import math +import random + +import pytest +import torch + +from vllm.triton_utils import triton +from vllm.v1.attention.ops.flashmla import ( + flash_mla_with_kvcache, + get_mla_metadata, + is_flashmla_dense_supported, +) + + +def cal_diff( + x: torch.Tensor, y: torch.Tensor, name: str, use_fp8: bool = False +) -> None: + x, y = x.double(), y.double() + cos_diff = 1 - 2 * (x * y).sum().item() / max((x * x + y * y).sum().item(), 1e-12) + if use_fp8: + assert cos_diff < 1e-4 + else: + assert cos_diff < 1e-5 + + +FLASH_MLA_UNSUPPORTED_REASON = ( + is_flashmla_dense_supported()[1] + if not is_flashmla_dense_supported()[0] + else "FlashMLA is supported" +) + + +@pytest.mark.skipif( + not is_flashmla_dense_supported()[0], reason=FLASH_MLA_UNSUPPORTED_REASON +) +@pytest.mark.parametrize("b", [128]) +@pytest.mark.parametrize("s_q", [1, 2]) +@pytest.mark.parametrize("mean_sk", [4096, 8192, 16384]) +@pytest.mark.parametrize("h_q", [16, 32, 64, 128]) +@pytest.mark.parametrize("h_kv", [1]) +@pytest.mark.parametrize("d", [576]) +@pytest.mark.parametrize("dv", [512]) +@pytest.mark.parametrize("block_size", [64]) +@pytest.mark.parametrize("causal", [True]) +@pytest.mark.parametrize("varlen", [False, True]) +@pytest.mark.parametrize( + "torch_dtype", [torch.bfloat16, torch.float16, torch.float8_e4m3fn] +) +@torch.inference_mode() +def test_flash_mla( + b, s_q, mean_sk, h_q, h_kv, d, dv, block_size, causal, varlen, torch_dtype +): + device = torch.device("cuda:0") + init_dtype = torch.bfloat16 if torch_dtype == torch.float8_e4m3fn else torch_dtype + torch.set_default_dtype(init_dtype) + torch.set_default_device(device) + torch.accelerator.set_device_index(device) + torch.manual_seed(0) + random.seed(0) + + print( + f"{b=}, {s_q=}, {mean_sk=}, {h_q=}, {h_kv=}, " + f"{d=}, {dv=}, {causal=}, {varlen=}, {torch_dtype=}" + ) + + use_fp8 = torch_dtype == torch.float8_e4m3fn + cache_seqlens = torch.full((b,), mean_sk, dtype=torch.int32) + if varlen: + for i in range(b): + cache_seqlens[i] = max(random.normalvariate(mean_sk, mean_sk / 2), s_q) + total_seqlens = cache_seqlens.sum().item() + max_seqlen = cache_seqlens.max().item() + max_seqlen_pad = triton.cdiv(max_seqlen, 256) * 256 + + q = torch.randn(b, s_q, h_q, d) + block_table = torch.arange( + b * max_seqlen_pad // block_size, dtype=torch.int32 + ).view(b, max_seqlen_pad // block_size) + blocked_k = torch.randn(block_table.numel(), block_size, h_kv, d) + for i in range(b): + blocked_k.view(b, max_seqlen_pad, h_kv, d)[i, cache_seqlens[i].item() :] = ( + float("nan") + ) + blocked_v = blocked_k[..., :dv] + + tile_scheduler_metadata, num_splits = get_mla_metadata( + cache_seqlens, s_q * h_q // h_kv, h_kv + ) + + init_dtype = q.dtype + if use_fp8: + fp8_dtype = torch.float8_e4m3fn + descale_q = torch.ones((1), dtype=torch.float32) + descale_k = torch.ones((1), dtype=torch.float32) + + q = q.to(fp8_dtype) + blocked_k = blocked_k.to(fp8_dtype) + blocked_v = blocked_v.to(fp8_dtype) + else: + descale_q = None + descale_k = None + + def flash_mla(): + return flash_mla_with_kvcache( + q, + blocked_k, + block_table, + cache_seqlens, + dv, + tile_scheduler_metadata, + num_splits, + causal=causal, + descale_q=descale_q, + descale_k=descale_k, + ) + + def scaled_dot_product_attention(query, key, value, is_causal=False): + query = query.float() + key = key.float() + value = value.float() + key = key.repeat_interleave(h_q // h_kv, dim=0) + value = value.repeat_interleave(h_q // h_kv, dim=0) + attn_weight = query @ key.transpose(-2, -1) / math.sqrt(query.size(-1)) + if is_causal: + s_q = query.shape[-2] + s_k = key.shape[-2] + attn_bias = torch.zeros(s_q, s_k, dtype=query.dtype) + temp_mask = torch.ones(s_q, s_k, dtype=torch.bool).tril(diagonal=s_k - s_q) + attn_bias.masked_fill_(temp_mask.logical_not(), float("-inf")) + attn_bias.to(query.dtype) + attn_weight += attn_bias + lse = attn_weight.logsumexp(dim=-1) + attn_weight = torch.softmax(attn_weight, dim=-1, dtype=torch.float32) + return attn_weight @ value, lse + + def ref_mla(): + q_ = (q.to(torch.float) * descale_q).to(init_dtype) if use_fp8 else q + blocked_k_ = ( + (blocked_k.to(torch.float) * descale_k).to(init_dtype) + if use_fp8 + else blocked_k + ) + blocked_v_ = ( + (blocked_v.to(torch.float) * descale_k).to(init_dtype) + if use_fp8 + else blocked_v + ) + out = torch.empty(b, s_q, h_q, dv, dtype=torch.float32) + lse = torch.empty(b, h_q, s_q, dtype=torch.float32) + for i in range(b): + begin = i * max_seqlen_pad + end = begin + cache_seqlens[i] + out_i, lse_i = scaled_dot_product_attention( + q_[i].transpose(0, 1), + blocked_k_.view(-1, h_kv, d)[begin:end].transpose(0, 1), + blocked_v_.view(-1, h_kv, dv)[begin:end].transpose(0, 1), + is_causal=causal, + ) + out[i] = out_i.transpose(0, 1) + lse[i] = lse_i + return out, lse + + out_flash, lse_flash = flash_mla() + out_torch, lse_torch = ref_mla() + cal_diff(out_flash, out_torch, "out", use_fp8) + cal_diff(lse_flash, lse_torch, "lse") + + t = triton.testing.do_bench(flash_mla) + FLOPS = s_q * total_seqlens * h_q * (d + dv) * 2 + bytes = (total_seqlens * h_kv * d + b * s_q * h_q * d) * ( + torch.finfo(torch_dtype).bits // 8 + ) + (b * s_q * h_q * dv) * (torch.finfo(init_dtype).bits // 8) + print( + f"{t:.3f} ms, {FLOPS / 10**9 / t:.0f} TFLOPS,", f"{bytes / 10**6 / t:.0f} GB/s" + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_flashmla_sparse.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_flashmla_sparse.py new file mode 100644 index 0000000000000000000000000000000000000000..e3886d1645de2c31ed1a50207706621208d40967 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_flashmla_sparse.py @@ -0,0 +1,292 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch + + +def test_sparse_flashmla_metadata_smoke(): + import vllm.v1.attention.ops.flashmla as fm + + ok, reason = fm.is_flashmla_sparse_supported() + if not ok: + pytest.skip(reason) + + device = torch.device("cuda") + batch_size = 1 + seqlen_q = 1 + num_heads_q = 128 + num_heads_k = 1 + q_seq_per_hk = seqlen_q * num_heads_q // num_heads_k + topk = 128 + + cache_seqlens = torch.zeros(batch_size, dtype=torch.int32, device=device) + + tile_md, num_splits = fm.get_mla_metadata( + cache_seqlens, + q_seq_per_hk, + num_heads_k, + num_heads_q=num_heads_q, + topk=topk, + is_fp8_kvcache=True, + ) + assert isinstance(tile_md, fm.FlashMLASchedMeta) + assert tile_md.tile_scheduler_metadata is None + assert tile_md.num_splits is None + assert num_splits is None + + +def test_sparse_flashmla_decode_smoke(): + import vllm.v1.attention.ops.flashmla as fm + + ok, reason = fm.is_flashmla_sparse_supported() + if not ok: + pytest.skip(reason) + + device = torch.device("cuda") + batch_size = 1 + seqlen_q = 1 + num_heads_q = 64 + head_dim_k = 576 + head_dim_v = 512 + num_heads_k = 1 + page_block_size = 64 + bytes_per_token = 656 + topk = 128 + + # Metadata + q_seq_per_hk = seqlen_q * num_heads_q // num_heads_k + # q_heads_per_hk = num_heads_q // num_heads_k + cache_seqlens = torch.zeros(batch_size, dtype=torch.int32, device=device) + tile_md, num_splits = fm.get_mla_metadata( + cache_seqlens, + q_seq_per_hk, + num_heads_k, + num_heads_q=num_heads_q, + topk=topk, + is_fp8_kvcache=True, + ) + + # Inputs + q = torch.zeros( + (batch_size, seqlen_q, num_heads_q, head_dim_k), + dtype=torch.bfloat16, + device=device, + ) + k_cache = torch.zeros( + (1, page_block_size, num_heads_k, bytes_per_token), + dtype=torch.uint8, + device=device, + ) + indices = torch.zeros( + (batch_size, seqlen_q, topk), dtype=torch.int32, device=device + ) + + block_table = torch.zeros((batch_size, 128), dtype=torch.int32, device=device) + out, lse = fm.flash_mla_with_kvcache( + q, + k_cache, + block_table, + cache_seqlens, + head_dim_v, + tile_md, + num_splits, + indices=indices, + is_fp8_kvcache=True, + ) + assert out.shape[0] == batch_size + assert out.shape[-1] == head_dim_v + assert lse.shape[0] == batch_size + + +def test_sparse_flashmla_prefill_smoke(): + import vllm.v1.attention.ops.flashmla as fm + + ok, reason = fm.is_flashmla_sparse_supported() + if not ok: + pytest.skip(reason) + + device = torch.device("cuda") + s_q = 1 + s_kv = 1 + h_q = 64 # kernel expects multiple of 64 + h_kv = 1 + d_qk = 576 + d_v = 512 + topk = 128 + + q = torch.zeros((s_q, h_q, d_qk), dtype=torch.bfloat16, device=device) + kv = torch.zeros((s_kv, h_kv, d_qk), dtype=torch.bfloat16, device=device) + indices = torch.zeros((s_q, h_kv, topk), dtype=torch.int32, device=device) + + out, max_logits, lse = fm.flash_mla_sparse_fwd(q, kv, indices, 1.0, d_v) + assert out.shape == (s_q, h_q, d_v) + assert max_logits.shape == (s_q, h_q) + assert lse.shape == (s_q, h_q) + + +def test_deepseek_v4_prefill_chunk_planning_expands_for_short_sequences(): + from vllm.v1.attention.backends.mla.sparse_swa import DeepseekSparseSWAMetadata + + metadata = DeepseekSparseSWAMetadata( + block_table=torch.empty(0, dtype=torch.int32), + slot_mapping=torch.empty(0, dtype=torch.int32), + block_size=64, + num_prefills=5, + prefill_seq_lens_cpu=torch.tensor([80, 96, 112, 128, 144], dtype=torch.int32), + prefill_query_lens_cpu=torch.tensor([4, 4, 4, 4, 4], dtype=torch.int32), + prefill_window_size=64, + prefill_max_model_len=1024, + prefill_max_num_batched_tokens=128, + ) + + chunk_plan = metadata.get_prefill_chunk_plan(compress_ratio=4, prefill_chunk_size=4) + + # the adaptive plan keeps all 5 in one chunk + assert chunk_plan == [(0, 5, 36, 103)] + + +def test_flashinfer_sparse_indices_cache(monkeypatch): + from vllm.models.deepseek_v4.nvidia import flashinfer_sparse as flashinfer_mod + from vllm.models.deepseek_v4.sparse_mla import DeepseekV4FlashMLAMetadata + from vllm.v1.attention.backends.mla.sparse_swa import DeepseekSparseSWAMetadata + + builder_calls = 0 + + def fake_build(*args, **kwargs): + nonlocal builder_calls + builder_calls += 1 + return ( + torch.tensor([[builder_calls]], dtype=torch.int32), + torch.tensor([builder_calls], dtype=torch.int32), + ) + + monkeypatch.setattr( + flashinfer_mod, "build_flashinfer_mixed_sparse_indices", fake_build + ) + + def make_attn(compress_ratio: int, topk_width: int): + attn = object.__new__(flashinfer_mod.DeepseekV4FlashInferMLAAttention) + attn.compress_ratio = compress_ratio + attn.window_size = 4 + attn.topk_indices_buffer = torch.tensor( + [[0, 1], [2, 3], [4, 5]], dtype=torch.int32 + )[:, :topk_width] + return attn + + def make_swa_metadata(): + return DeepseekSparseSWAMetadata( + block_table=torch.tensor([[0, 1], [2, 3]], dtype=torch.int32), + slot_mapping=torch.tensor([0, 1], dtype=torch.int64), + block_size=64, + seq_lens=torch.tensor([8, 10], dtype=torch.int32), + query_start_loc=torch.tensor([0, 1, 3], dtype=torch.int32), + query_start_loc_cpu=torch.tensor([0, 1, 3], dtype=torch.int32), + token_to_req_indices=torch.tensor([0, 1, 1], dtype=torch.int32), + decode_swa_indices=torch.tensor([[5, 6, -1, -1]], dtype=torch.int32), + decode_swa_lens=torch.tensor([2], dtype=torch.int32), + is_valid_token=torch.tensor([True], dtype=torch.bool), + num_decodes=1, + num_prefills=1, + num_decode_tokens=1, + num_prefill_tokens=2, + ) + + def make_flashmla_metadata(): + return DeepseekV4FlashMLAMetadata( + num_reqs=2, + max_query_len=2, + max_seq_len=10, + num_actual_tokens=3, + query_start_loc=torch.tensor([0, 1, 3], dtype=torch.int32), + slot_mapping=torch.tensor([0, 1, 2], dtype=torch.int64), + block_table=torch.tensor([[0, 1], [2, 3]], dtype=torch.int32), + req_id_per_token=torch.tensor([0, 1, 1], dtype=torch.int32), + block_size=256, + topk_tokens=2, + c128a_global_decode_topk_indices=torch.tensor( + [[[9, 10]]], dtype=torch.int32 + ), + c128a_decode_topk_lens=torch.tensor([2], dtype=torch.int32), + c128a_prefill_topk_indices=torch.tensor( + [[0, 1], [1, 2]], dtype=torch.int32 + ), + ) + + swa_attn = make_attn(1, 0) + swa_metadata = make_swa_metadata() + _, _, sparse_indices_first, sparse_lens_first = ( + swa_attn._build_sparse_index_metadata( + kv_cache=None, + swa_k_cache=torch.empty((1, 64, 512), dtype=torch.bfloat16), + swa_metadata=swa_metadata, + attn_metadata=None, + swa_only=True, + ) + ) + _, _, sparse_indices_second, sparse_lens_second = ( + swa_attn._build_sparse_index_metadata( + kv_cache=None, + swa_k_cache=torch.empty((1, 64, 512), dtype=torch.bfloat16), + swa_metadata=swa_metadata, + attn_metadata=None, + swa_only=True, + ) + ) + assert builder_calls == 1 + assert sparse_indices_first is sparse_indices_second + assert sparse_lens_first is sparse_lens_second + + c128a_attn = make_attn(128, 2) + c128a_metadata = make_swa_metadata() + c128a_flashmla_md = make_flashmla_metadata() + _, _, sparse_indices_first, sparse_lens_first = ( + c128a_attn._build_sparse_index_metadata( + kv_cache=torch.empty((1, 2, 512), dtype=torch.bfloat16), + swa_k_cache=torch.empty((1, 64, 512), dtype=torch.bfloat16), + swa_metadata=c128a_metadata, + attn_metadata=c128a_flashmla_md, + swa_only=False, + ) + ) + _, _, sparse_indices_second, sparse_lens_second = ( + c128a_attn._build_sparse_index_metadata( + kv_cache=torch.empty((1, 2, 512), dtype=torch.bfloat16), + swa_k_cache=torch.empty((1, 64, 512), dtype=torch.bfloat16), + swa_metadata=c128a_metadata, + attn_metadata=c128a_flashmla_md, + swa_only=False, + ) + ) + + assert builder_calls == 2 + assert sparse_indices_first is sparse_indices_second + assert sparse_lens_first is sparse_lens_second + + c4a_attn = make_attn(4, 2) + c4a_metadata = make_swa_metadata() + c4a_flashmla_md = make_flashmla_metadata() + c4a_flashmla_md.c128a_global_decode_topk_indices = None + c4a_flashmla_md.c128a_decode_topk_lens = None + c4a_flashmla_md.c128a_prefill_topk_indices = None + _, _, sparse_indices_third, sparse_lens_third = ( + c4a_attn._build_sparse_index_metadata( + kv_cache=torch.empty((1, 2, 512), dtype=torch.bfloat16), + swa_k_cache=torch.empty((1, 64, 512), dtype=torch.bfloat16), + swa_metadata=c4a_metadata, + attn_metadata=c4a_flashmla_md, + swa_only=False, + ) + ) + _, _, sparse_indices_fourth, sparse_lens_fourth = ( + c4a_attn._build_sparse_index_metadata( + kv_cache=torch.empty((1, 2, 512), dtype=torch.bfloat16), + swa_k_cache=torch.empty((1, 64, 512), dtype=torch.bfloat16), + swa_metadata=c4a_metadata, + attn_metadata=c4a_flashmla_md, + swa_only=False, + ) + ) + + assert builder_calls == 4 + assert sparse_indices_third is not sparse_indices_fourth + assert sparse_lens_third is not sparse_lens_fourth diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_lightning_attn.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_lightning_attn.py new file mode 100644 index 0000000000000000000000000000000000000000..ef7aca48e8d521f9fd2df2c8d4a5fc5af72a6f4f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_lightning_attn.py @@ -0,0 +1,268 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm.model_executor.layers.lightning_attn import linear_decode_forward_triton +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +DEVICE = current_platform.device_type + +pytestmark = pytest.mark.skipif( + not (current_platform.is_cuda_alike() or current_platform.is_xpu()), + reason="Lightning attention Triton kernels require CUDA/ROCm or XPU.", +) + +NUM_HEADS = [4, 8] +HEAD_SIZES = [64] +BATCH_SIZES = [1, 2] +SEQ_LENGTHS = [16] +DTYPES = [torch.float32] + + +def reference_lightning_attention(q, k, v, ed, block_size, kv_history): + """Reference implementation of lightning attention core algorithm + + The difference from the main implementation is that this processes + each step sequentially, instead of using parallelized triton kernels + """ + B, H, S, D = q.shape + E = v.shape[-1] + dtype = q.dtype + output = torch.zeros((B, H, S, E), dtype=dtype, device=q.device) + + # Use clone() to ensure an independent copy + if kv_history is None: + kv_cache = torch.zeros((B, H, D, E), dtype=dtype, device=q.device) + else: + kv_cache = kv_history.clone() + + # More efficient implementation + # Convert decay factors to matrix form + decay = torch.exp(-ed).view(1, -1, 1, 1) if ed.dim() == 1 else torch.exp(-ed) + + for b in range(B): + for step in range(S): + # Process all heads at once for this position + q_bs = q[b, :, step] # [H, D] + k_bs = k[b, :, step] # [H, D] + v_bs = v[b, :, step] # [H, E] + + # Calculate KV outer products for all heads + for h in range(H): + # Calculate KV outer product + kv_outer = torch.outer(k_bs[h], v_bs[h]) + + # Update KV cache with decay + # Note: Using the same order as in the Triton kernel + kv_cache[b, h] = decay[0, h, 0, 0] * kv_cache[b, h] + kv_outer + + # Calculate attention output + output[b, h, step] = torch.matmul(q_bs[h], kv_cache[b, h]) + + # Match the shape returned by the actual implementation + # The actual implementation returns a tensor of shape [B, H, 2, D, E] + # where dimension 2 contains both KV and KV history + kv_reshaped = kv_cache.unsqueeze(2) # [B, H, 1, D, E] + final_kv_cache = torch.cat([kv_reshaped, kv_reshaped], dim=2) # [B, H, 2, D, E] + + return output, final_kv_cache + + +def reference_linear_decode(q, k, v, kv_caches, slope_rate, slot_idx): + """Reference implementation: linear attention decode function""" + B, H, _, D = q.shape + output = torch.zeros(B, H * D, dtype=q.dtype, device=q.device) + + # Calculate decay factors once (more efficient) + decay = torch.exp(-slope_rate).view(-1, 1, 1) # [H, 1, 1] + + # Process each batch + for b in range(B): + slot_id = slot_idx[b].item() + + # Skip padding positions + if slot_id == -1: + continue + + # Process all heads at once for this batch + q_b = q[b, :, 0] # [H, D] + k_b = k[b, :, 0] # [H, D] + v_b = v[b, :, 0] # [H, D] + + # Process each attention head + for h in range(H): + # Get current query, key and value + q_bh = q_b[h] + k_bh = k_b[h] + v_bh = v_b[h] + + # Get cache + kv_cache_old = kv_caches[b, h] + + # Calculate new key-value outer product + kv_outer = torch.outer(k_bh, v_bh) + + # Apply decay and update cache + kv_new = kv_outer + decay[h, 0, 0] * kv_cache_old + + # Calculate output + out_h = torch.matmul(q_bh, kv_new) + + # Update output and cache + output[b, h * D : (h + 1) * D] = out_h + kv_caches[b, h] = kv_new + + return output + + +@pytest.mark.parametrize("batch_size", BATCH_SIZES) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@torch.inference_mode() +def test_linear_decode_forward_triton( + batch_size: int, + num_heads: int, + head_size: int, + dtype: torch.dtype, +): + torch.set_default_device(DEVICE) + set_random_seed(42) + base = 0.01 + q = base * torch.randn(batch_size, num_heads, 1, head_size, dtype=dtype) + k = base * torch.randn(batch_size, num_heads, 1, head_size, dtype=dtype) + v = base * torch.randn(batch_size, num_heads, 1, head_size, dtype=dtype) + + kv_caches = base * torch.randn( + batch_size, num_heads, head_size, head_size, dtype=dtype, device=DEVICE + ) + + kv_caches_copy = kv_caches.clone() + + slope_rate = torch.zeros(num_heads, device=DEVICE) + for h in range(num_heads): + slope_rate[h] = 0.1 * (h + 1) + + slot_idx = torch.arange(batch_size, device=DEVICE) + + triton_output = linear_decode_forward_triton( + q, k, v, kv_caches, slope_rate, slot_idx + ) + + reference_output = reference_linear_decode( + q, k, v, kv_caches_copy, slope_rate, slot_idx + ) + torch.testing.assert_close(triton_output, reference_output, rtol=1e-1, atol=1e-1) + torch.testing.assert_close(kv_caches, kv_caches_copy, rtol=1e-1, atol=1e-1) + + assert triton_output.shape == (batch_size, num_heads * head_size) + + +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@torch.inference_mode() +def test_linear_decode_forward_triton_with_padding( + num_heads: int, + head_size: int, + dtype: torch.dtype, +): + torch.set_default_device(DEVICE) + set_random_seed(42) + + batch_size = 4 + base = 0.01 + q = base * torch.randn(batch_size, num_heads, 1, head_size, dtype=dtype) + k = base * torch.randn(batch_size, num_heads, 1, head_size, dtype=dtype) + v = base * torch.randn(batch_size, num_heads, 1, head_size, dtype=dtype) + + kv_caches = base * torch.randn( + batch_size, num_heads, head_size, head_size, dtype=dtype, device=DEVICE + ) + + kv_caches_copy = kv_caches.clone() + + slope_rate = torch.zeros(num_heads, device=DEVICE) + for h in range(num_heads): + slope_rate[h] = 0.1 * (h + 1) + + slot_idx = torch.tensor([0, 1, -1, 2], device=DEVICE) + + triton_output = linear_decode_forward_triton( + q, k, v, kv_caches, slope_rate, slot_idx + ) + + reference_output = reference_linear_decode( + q, k, v, kv_caches_copy, slope_rate, slot_idx + ) + + padding_mask = (slot_idx != -1).unsqueeze(1).expand(-1, num_heads * head_size) + + triton_masked = triton_output[padding_mask] + reference_masked = reference_output[padding_mask] + + atol, rtol = 1.5e-1, 1.5e-1 + + valid_indices = slot_idx != -1 + + for i in range(batch_size): + if valid_indices[i] > 0: + torch.testing.assert_close( + kv_caches[i], kv_caches_copy[i], rtol=rtol, atol=atol + ) + + torch.testing.assert_close(triton_masked, reference_masked, rtol=rtol, atol=atol) + + assert triton_output.shape == (batch_size, num_heads * head_size) + + +@pytest.mark.parametrize("batch_size", BATCH_SIZES) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("seq_len", SEQ_LENGTHS) +@pytest.mark.parametrize("dtype", DTYPES) +@torch.inference_mode() +def test_lightning_attention_reference( + batch_size: int, + num_heads: int, + head_size: int, + seq_len: int, + dtype: torch.dtype, +): + torch.set_default_device(DEVICE) + set_random_seed(42) + + base = 0.01 + q = base * torch.randn(batch_size, num_heads, seq_len, head_size, dtype=dtype) + k = base * torch.randn(batch_size, num_heads, seq_len, head_size, dtype=dtype) + v = base * torch.randn(batch_size, num_heads, seq_len, head_size, dtype=dtype) + + ed = torch.zeros(num_heads, device=DEVICE) + for h in range(num_heads): + ed[h] = 0.1 * (h + 1) + + kv_history = base * torch.randn( + batch_size, num_heads, head_size, head_size, dtype=dtype, device=DEVICE + ) + + kv_history_clone = kv_history.clone() + + ref_output, ref_kv_cache = reference_lightning_attention( + q, k, v, ed, 256, kv_history + ) + + from vllm.model_executor.layers.lightning_attn import lightning_attention + + actual_output, actual_kv_cache = lightning_attention( + q, k, v, ed, 256, kv_history_clone + ) + + atol, rtol = 1.5e-1, 1.5e-1 + torch.testing.assert_close(ref_output, actual_output, rtol=rtol, atol=atol) + torch.testing.assert_close(ref_kv_cache, actual_kv_cache, rtol=rtol, atol=atol) + + assert ref_output.shape == (batch_size, num_heads, seq_len, head_size) + assert ref_kv_cache.shape == actual_kv_cache.shape diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_merge_attn_states.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_merge_attn_states.py new file mode 100644 index 0000000000000000000000000000000000000000..a734064053ea2492665750ca3e65158440f6244c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_merge_attn_states.py @@ -0,0 +1,392 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm._custom_ops import ( + merge_attn_states as merge_attn_states_cuda, +) +from vllm._custom_ops import ( + scaled_fp8_quant, +) +from vllm.platforms import current_platform +from vllm.v1.attention.ops.triton_merge_attn_states import ( + merge_attn_states as merge_attn_states_triton, +) + + +# Naive PyTorch Implements section 2.2 of https://www.arxiv.org/pdf/2501.01005 +# can be used to combine partial attention results (in the split-KV case) +def merge_attn_states_torch( + output: torch.Tensor, # [NUM_TOKENS, NUM_HEADS, HEAD_SIZE] + prefix_output: torch.Tensor, # [NUM_TOKENS, NUM_HEADS, HEAD_SIZE] + prefix_lse: torch.Tensor, # [NUM_HEADS, NUM_TOKENS] + suffix_output: torch.Tensor, # [NUM_TOKENS, NUM_HEADS, HEAD_SIZE] + suffix_lse: torch.Tensor, # [NUM_HEADS, NUM_TOKENS] + output_lse: torch.Tensor | None = None, # [NUM_HEADS, NUM_TOKENS] + prefill_tokens_with_context: int | None = None, + output_scale: torch.Tensor | None = None, # scalar, per-tensor FP8 scale +): + # Apply prefill_tokens_with_context mask if needed + if prefill_tokens_with_context is None: + prefill_tokens_with_context = output.shape[0] + p_lse = prefix_lse + s_lse = suffix_lse + # inf -> -inf + p_lse[p_lse == torch.inf] = -torch.inf + s_lse[s_lse == torch.inf] = -torch.inf + # max_lse [NUM_HEADS, NUM_TOKENS] + max_lse = torch.maximum(p_lse, s_lse) + + mask = torch.ones((prefix_lse.shape[1], 1, 1), device=p_lse.device) + mask[prefill_tokens_with_context:].fill_(0) + p_lse = p_lse - max_lse + s_lse = s_lse - max_lse + p_lse_exp = torch.exp(p_lse) + s_lse_exp = torch.exp(s_lse) + out_se = p_lse_exp + s_lse_exp + if output_lse is not None: + output_lse = torch.log(out_se) + max_lse + output_lse[prefill_tokens_with_context:] = suffix_lse[ + prefill_tokens_with_context: + ] + p_scale = p_lse_exp / out_se # [NUM_HEADS, NUM_TOKENS] + s_scale = s_lse_exp / out_se # [NUM_HEADS, NUM_TOKENS] + p_scale = torch.transpose(p_scale, 0, 1).unsqueeze(2) # [NUM_TOKENS, NUM_HEADS, 1] + s_scale = torch.transpose(s_scale, 0, 1).unsqueeze(2) # [NUM_TOKENS, NUM_HEADS, 1] + output = prefix_output * p_scale * mask + suffix_output * ( + s_scale * mask + (1 - mask) + ) + if output_scale is not None: + shape = output.shape + output, _ = scaled_fp8_quant(output.float().view(-1, shape[-1]), output_scale) + output = output.view(shape) + return output, output_lse + + +NUM_BATCH_TOKENS = [256, 512, 613, 1024, 1536, 4096] +NUM_QUERY_HEADS = [4, 8, 16, 32, 48, 64] +HEAD_SIZES = [32, 48, 64, 96, 128, 256] +DTYPES = [torch.float32, torch.half, torch.bfloat16] + +all_case_info: list[tuple] = [] + + +def generate_markdown_table(): + global all_case_info + table_header = ( + "| tokens | heads | headsize | dtype " + "| device | torch | triton | cuda | speedup |" + ) + table_separator = "| --- | --- | --- | --- | --- | --- | --- | --- | --- |" + + def shortly_dtype(dtype: torch.dtype) -> str: + return str(dtype).removeprefix("torch.") + + def shortly_device(device: str) -> str: + return device.removeprefix("NVIDIA").strip() + + print(table_header) + print(table_separator) + for info in all_case_info: + ( + num_tokens, + num_heads, + head_size, + dtype, + device, + avg_time_torch_kernel, + avg_time_triton_kernel, + avg_time_cuda_kernel, + performance_improved, + ) = info + dtype = shortly_dtype(dtype) + device = shortly_device(device) + print( + f"| {num_tokens} | {num_heads} | {head_size} " + f"| {dtype} | {device} | {avg_time_torch_kernel:.5f}ms " + f"| {avg_time_triton_kernel:.5f}ms " + f"| {avg_time_cuda_kernel:.5f}ms " + f"| {performance_improved:.4f}x |" + ) + + +@pytest.mark.parametrize("use_fp8", [False, True]) +@pytest.mark.parametrize("prefill_tokens_with_context", [None, 128]) +@pytest.mark.parametrize("num_tokens", NUM_BATCH_TOKENS) +@pytest.mark.parametrize("num_query_heads", NUM_QUERY_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("input_dtype", DTYPES) +@torch.inference_mode() +def test_merge_attn_states( + prefill_tokens_with_context: int | None, + num_tokens: int, + num_query_heads: int, + head_size: int, + input_dtype: torch.dtype, + use_fp8: bool, +): + if not current_platform.is_cuda(): + pytest.skip( + "Currently only support compare triton merge_attn_states " + "with custom cuda merge_attn_states kernel" + ) + + NUM_TOKENS = num_tokens + NUM_HEADS = num_query_heads + HEAD_SIZE = head_size + + # When use_fp8 is set, inputs stay as input_dtype (bf16/fp16/fp32) + # and output becomes FP8. + output_dtype = input_dtype + output_scale = None + if use_fp8: + output_dtype = current_platform.fp8_dtype() + output_scale = torch.tensor([0.05], dtype=torch.float32, device="cuda") + + print( + f"\nNUM_TOKENS:{NUM_TOKENS}, NUM_HEADS:{NUM_HEADS}, " + f"HEAD_SIZE:{HEAD_SIZE}, input_dtype: {input_dtype}, " + f"output_dtype: {output_dtype}, use_fp8: {use_fp8}, " + f"prefill_tokens_with_context: {prefill_tokens_with_context}, " + f"Device: {current_platform.get_device_name()}" + ) + + # prefix_lse and suffix_lse contain inf and normal values + prefix_lse = torch.randn(NUM_HEADS, NUM_TOKENS, dtype=torch.float32, device="cuda") + suffix_lse = torch.randn(NUM_HEADS, NUM_TOKENS, dtype=torch.float32, device="cuda") + + # Generate boolean masks + mask_prefix = torch.rand(NUM_HEADS, NUM_TOKENS) < 0.1 + mask_suffix = torch.rand(NUM_HEADS, NUM_TOKENS) < 0.1 + # Ensure that the same position is not True at the same time + combined_mask = torch.logical_and(mask_prefix, mask_suffix) + mask_prefix = torch.logical_and(mask_prefix, ~combined_mask) + mask_suffix = torch.logical_and(mask_suffix, ~combined_mask) + + prefix_lse[mask_prefix] = float("inf") + suffix_lse[mask_suffix] = float("inf") + + # Other input tensors (need to be initialized but + # no actual calculation needed) + output = torch.zeros( + (NUM_TOKENS, NUM_HEADS, HEAD_SIZE), dtype=output_dtype, device="cuda" + ) + output_lse = torch.zeros( + (NUM_HEADS, NUM_TOKENS), dtype=torch.float32, device="cuda" + ) + prefix_output = torch.randn( + (NUM_TOKENS, NUM_HEADS, HEAD_SIZE), dtype=input_dtype, device="cuda" + ) + suffix_output = torch.randn( + (NUM_TOKENS, NUM_HEADS, HEAD_SIZE), dtype=input_dtype, device="cuda" + ) + + warmup_times = 2 + repeat_times = 20 + + output_torch = output.clone() + output_lse_torch = output_lse.clone() + total_time_torch_kernel = 0 + start = torch.Event(enable_timing=True) + end = torch.Event(enable_timing=True) + + # 0. Run the Torch kernel + prefix_lse_torch = prefix_lse.clone() + suffix_lse_torch = suffix_lse.clone() + for _ in range(warmup_times): + output_torch, output_lse_torch = merge_attn_states_torch( + output_torch, + prefix_output, + prefix_lse_torch, + suffix_output, + suffix_lse_torch, + output_lse_torch, + prefill_tokens_with_context, + output_scale, + ) + torch.accelerator.synchronize() + + for _ in range(repeat_times): + start.record() + output_torch, output_lse_torch = merge_attn_states_torch( + output_torch, + prefix_output, + prefix_lse_torch, + suffix_output, + suffix_lse_torch, + output_lse_torch, + prefill_tokens_with_context, + output_scale, + ) + end.record() + torch.accelerator.synchronize() + total_time_torch_kernel += start.elapsed_time(end) + + avg_time_torch_kernel = total_time_torch_kernel / repeat_times + + # 1. Run the Triton kernel + output_ref_triton = output.clone() + output_lse_ref_triton = output_lse.clone() + + total_time_triton_kernel = 0 + start = torch.Event(enable_timing=True) + end = torch.Event(enable_timing=True) + + for _ in range(warmup_times): + merge_attn_states_triton( + output_ref_triton, + prefix_output, + prefix_lse, + suffix_output, + suffix_lse, + output_lse_ref_triton, + prefill_tokens_with_context, + output_scale, + ) + torch.accelerator.synchronize() + + for _ in range(repeat_times): + start.record() + merge_attn_states_triton( + output_ref_triton, + prefix_output, + prefix_lse, + suffix_output, + suffix_lse, + output_lse_ref_triton, + prefill_tokens_with_context, + output_scale, + ) + end.record() + torch.accelerator.synchronize() + total_time_triton_kernel += start.elapsed_time(end) + + avg_time_triton_kernel = total_time_triton_kernel / repeat_times + + # 2. Run the CUDA kernel + total_time_cuda_kernel = 0 + output_cuda = output.clone() + output_lse_cuda = output_lse.clone() + + for _ in range(warmup_times): + merge_attn_states_cuda( + output_cuda, + prefix_output, + prefix_lse, + suffix_output, + suffix_lse, + output_lse_cuda, + prefill_tokens_with_context, + output_scale, + ) + torch.accelerator.synchronize() + + for _ in range(repeat_times): + start.record() + merge_attn_states_cuda( + output_cuda, + prefix_output, + prefix_lse, + suffix_output, + suffix_lse, + output_lse_cuda, + prefill_tokens_with_context, + output_scale, + ) + end.record() + torch.accelerator.synchronize() + total_time_cuda_kernel += start.elapsed_time(end) + + avg_time_cuda_kernel = total_time_cuda_kernel / repeat_times + + # 3. Performance compare + performance_improved = avg_time_triton_kernel / avg_time_cuda_kernel + print(f" Torch time: {avg_time_torch_kernel:.6f}ms") + print(f"Triton time: {avg_time_triton_kernel:.6f}ms") + print( + f" CUDA time: {avg_time_cuda_kernel:.6f}ms, " + f"Performance: {performance_improved:.5f}x" + ) + print("-" * 100) + + # 4. Correctness compare + # Liger Kernel: Efficient Triton Kernels for LLM Training + # https://arxiv.org/pdf/2410.10989, 3.3 Correctness + # use rtol = 1e-2 for bfloat16. + if use_fp8: + # Compare in dequantized space (multiply back by scale) so that + # absolute differences reflect real precision, not amplified FP8 + # quantization steps. + atol, rtol = 1e-1, 1e-1 + assert output_scale is not None + scale = output_scale.item() + elif output_dtype == torch.bfloat16: + atol, rtol = 1e-3, 1e-2 + scale = 1.0 + else: + atol, rtol = 1e-3, 1e-3 + scale = 1.0 + + def diff(a: torch.Tensor, b: torch.Tensor): + max_diff = torch.max(torch.abs(a.float() - b.float())) + return max_diff + + # Use Triton output as reference because we want to replace + # the Triton kernel with custom CUDA kernel for merge attn + # states operation. + output_ref = output_ref_triton + output_lse_ref = output_lse_ref_triton + torch.testing.assert_close( + output_cuda.float() * scale, + output_ref.float() * scale, + atol=atol, + rtol=rtol, + ) + print( + "Output all match, max abs diff (dequantized):" + if use_fp8 + else "Output all match, max abs diff:" + ) + _diff = diff(output_ref.float() * scale, output_torch.float() * scale) + print(f"(Triton vs Torch) : {_diff}") + _diff = diff(output_torch.float() * scale, output_cuda.float() * scale) + print(f" (CUDA vs Torch) : {_diff}") + _diff = diff(output_ref.float() * scale, output_cuda.float() * scale) + print(f" (CUDA vs Triton): {_diff}") + print("-" * 100) + + torch.testing.assert_close( + output_lse_cuda.float(), output_lse_ref.float(), atol=atol, rtol=rtol + ) + print("Output LSE all match, max abs diff:") + print(f"(Triton vs Torch) : {diff(output_lse_torch, output_lse_ref)}") + print(f" (CUDA vs Torch) : {diff(output_lse_torch, output_lse_cuda)}") + print(f" (CUDA vs Triton): {diff(output_lse_ref, output_lse_cuda)}") + print("-" * 100) + + print( + "All output values test passed! All inf values " + "are correctly replaced with -inf." + ) + print("-" * 100) + + device = current_platform.get_device_name() + all_case_info.append( + ( + NUM_TOKENS, + NUM_HEADS, + HEAD_SIZE, + output_dtype, + device, + avg_time_torch_kernel, + avg_time_triton_kernel, + avg_time_cuda_kernel, + performance_improved, + ) + ) + if len(all_case_info) == ( + len(NUM_BATCH_TOKENS) * len(HEAD_SIZES) * len(NUM_QUERY_HEADS) * len(DTYPES) + ): + generate_markdown_table() diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_mha_attn.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_mha_attn.py new file mode 100644 index 0000000000000000000000000000000000000000..b4789ec0b983b2ab4a067c10eb603648606ad51b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_mha_attn.py @@ -0,0 +1,348 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Test: + +* Tests for MMEncoderAttention layer +""" + +import itertools +from unittest.mock import patch + +import numpy as np +import pytest +import torch + +from vllm.config import get_current_vllm_config +from vllm.config.multimodal import MultiModalConfig +from vllm.model_executor.layers.attention import MMEncoderAttention +from vllm.platforms import current_platform +from vllm.platforms.cpu import CpuPlatform +from vllm.platforms.cuda import CudaPlatform +from vllm.platforms.interface import DeviceCapability +from vllm.platforms.rocm import RocmPlatform +from vllm.utils.torch_utils import set_default_torch_dtype, set_random_seed +from vllm.v1.attention.backends.registry import AttentionBackendEnum +from vllm.v1.attention.selector import _cached_get_attn_backend + + +@pytest.fixture(autouse=True) +def clear_cache(): + """Clear lru cache to ensure each test case runs without caching.""" + _cached_get_attn_backend.cache_clear() + + +devices = ["cpu"] +if current_platform.is_cuda(): + devices.append("cuda") +if current_platform.is_rocm(): + devices.append("hip") + + +@pytest.mark.parametrize("device", devices) +def test_mha_attn_platform(default_vllm_config, device: str): + """ + Test the attention selector between different platform and device. + """ + torch.set_default_dtype(torch.float16) + + if device == "cpu": + with ( + patch("vllm.model_executor.models.vision.current_platform", CpuPlatform()), + ): + attn = MMEncoderAttention(16, 64, scale=1) + assert attn.attn_backend == AttentionBackendEnum.TORCH_SDPA + elif device == "hip": + with ( + patch("vllm.model_executor.models.vision.current_platform", RocmPlatform()), + ): + attn = MMEncoderAttention(16, 64, scale=1) + assert attn.attn_backend == AttentionBackendEnum.FLASH_ATTN + else: + # Test CUDA with head_size=64 (divisible by 32) + # - should use vLLM's FlashAttention + with ( + patch("vllm.model_executor.models.vision.current_platform", CudaPlatform()), + ): + attn = MMEncoderAttention(16, 64, scale=1) + assert attn.attn_backend == AttentionBackendEnum.FLASH_ATTN + + # Test CUDA with head_size=72 (not divisible by 32) + # - should use vLLM's FlashAttention + with ( + patch("vllm.model_executor.models.vision.current_platform", CudaPlatform()), + ): + attn = MMEncoderAttention(16, 72, scale=1) + assert attn.attn_backend == AttentionBackendEnum.FLASH_ATTN + + # Test CUDA with head_size=72 (not divisible by 32) + # - should use vLLM's FlashAttention + with ( + patch("vllm.model_executor.models.vision.current_platform", CudaPlatform()), + set_default_torch_dtype(torch.float32), + ): + attn = MMEncoderAttention(16, 72, scale=1) + assert attn.attn_backend == AttentionBackendEnum.TRITON_ATTN + + # Test Turing (pre-Ampere, sm_75): FlashAttention requires sm>=80, + # and Triton no longer supports MMA on Turing, so we expect that + # TORCH_SDPA is used for MMEncoderAttention. + with ( + patch("vllm.model_executor.models.vision.current_platform", CudaPlatform()), + patch.object( + CudaPlatform, + "get_device_capability", + return_value=DeviceCapability(major=7, minor=5), + ), + ): + attn = MMEncoderAttention(16, 64, scale=1) + assert attn.attn_backend == AttentionBackendEnum.TORCH_SDPA + + +def ref_attention( + query: torch.Tensor, + key: torch.Tensor, + value: torch.Tensor, + scale: float, +) -> torch.Tensor: + """ + Native implementation of scaled dot product attention without mask: + - query, key, value: [batch_size, seq_len, num_heads, head_size] + - attn_mask: [batch_size, seq_len, seq_len] + """ + query, key, value = (x.transpose(1, 2) for x in (query, key, value)) + attn_weights = scale * torch.matmul(query, key.transpose(2, 3)) + attn_weights = torch.softmax(attn_weights, dim=-1).to(value.dtype) + out = torch.matmul(attn_weights, value).transpose(1, 2) + return out + + +BATCH_SIZES = [1, 16] +SEQ_LENS = [1] +VAR_SEQ_LENS = [ + [2, 2], + [2, 3, 4], +] +NUM_HEADS = [1, 16] +NUM_KV_HEADS = [1] +HEAD_SIZES = [64, 80] +# flshattF and tritonflashattF supported: {torch.float16, torch.bfloat16} +DTYPES = ( + [torch.half, torch.bfloat16, torch.float] + if not current_platform.is_rocm() + else [torch.half, torch.bfloat16] +) +CUDA_DEVICES = ["cuda"] + + +@pytest.mark.parametrize("batch_size", BATCH_SIZES) +@pytest.mark.parametrize("seq_len", SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("num_kv_heads", NUM_KV_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("device", CUDA_DEVICES) +def test_mha_attn_forward( + default_vllm_config, + batch_size: int, + seq_len: int, + num_heads: int, + num_kv_heads: int, + head_size: int, + dtype: torch.dtype, + device: str, +): + set_random_seed(0) + torch.set_default_device(device) + torch.set_default_dtype(dtype) + + q = torch.randn(batch_size, seq_len, num_heads * head_size) + k = torch.randn(batch_size, seq_len, num_kv_heads * head_size) + v = torch.randn(batch_size, seq_len, num_kv_heads * head_size) + scale = 1.0 / head_size**0.5 + attn = MMEncoderAttention( + num_heads, head_size, scale=scale, num_kv_heads=num_kv_heads + ) + output = attn(q, k, v) + + assert num_heads % num_kv_heads == 0 + num_queries_per_kv = num_heads // num_kv_heads + q = q.reshape(batch_size, seq_len, num_heads, head_size) + k = k.reshape(batch_size, seq_len, num_kv_heads, head_size) + v = v.reshape(batch_size, seq_len, num_kv_heads, head_size) + if num_queries_per_kv > 1: + k = torch.repeat_interleave(k, num_queries_per_kv, dim=2) + v = torch.repeat_interleave(v, num_queries_per_kv, dim=2) + + ref_output = ref_attention( + q, + k, + v, + scale=scale, + ).reshape(batch_size, seq_len, num_heads * head_size) + tol_kwargs = ( + dict(rtol=1e-3, atol=1e-3) + if attn.attn_backend == AttentionBackendEnum.TRITON_ATTN + else {} + ) + torch.testing.assert_close(output, ref_output, **tol_kwargs) + + +@pytest.mark.parametrize("var_seq_len", VAR_SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("num_kv_heads", NUM_KV_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("device", CUDA_DEVICES) +def test_mha_attn_varlen_forward( + default_vllm_config, + var_seq_len: list[int], + num_heads: int, + num_kv_heads: int, + head_size: int, + dtype: torch.dtype, + device: str, +): + set_random_seed(0) + torch.set_default_device(device) + torch.set_default_dtype(dtype) + + q = torch.randn(1, sum(var_seq_len), num_heads, head_size) + k = torch.randn(1, sum(var_seq_len), num_kv_heads, head_size) + v = torch.randn(1, sum(var_seq_len), num_kv_heads, head_size) + cu_seqlens = torch.tensor( + [0] + list(itertools.accumulate(var_seq_len)), dtype=torch.int32 + ) + scale = 1.0 / head_size**0.5 + attn = MMEncoderAttention( + num_heads, head_size, scale=scale, num_kv_heads=num_kv_heads + ) + output = attn( + q, k, v, cu_seqlens=cu_seqlens, max_seqlen=torch.tensor(max(var_seq_len)) + ) + + assert num_heads % num_kv_heads == 0 + num_queries_per_kv = num_heads // num_kv_heads + if num_queries_per_kv > 1: + k = torch.repeat_interleave(k, num_queries_per_kv, dim=2) + v = torch.repeat_interleave(v, num_queries_per_kv, dim=2) + + ref_output = [] + for q_i, k_i, v_i in zip( + torch.split(q, var_seq_len, dim=1), + torch.split(k, var_seq_len, dim=1), + torch.split(v, var_seq_len, dim=1), + ): + output_i = ref_attention( + q_i, + k_i, + v_i, + scale=scale, + ) + ref_output.append(output_i) + ref_output = torch.cat(ref_output, dim=1) + torch.testing.assert_close(output, ref_output, atol=1e-2, rtol=1e-2) + + +@pytest.mark.parametrize("var_seq_len", VAR_SEQ_LENS) +@pytest.mark.parametrize( + "dtype", + [torch.bfloat16, torch.half], +) +@pytest.mark.parametrize("device", CUDA_DEVICES) +def test_mha_attn_varlen_forward_flashinfer( + default_vllm_config, + var_seq_len: list[int], + dtype: torch.dtype, + device: str, +): + """Test MMEncoderAttention varlen forward with FLASHINFER backend (head_size=72). + + Exercises the path that uses --mm-encoder-attn-backend=FLASHINFER with + recomputed cu_seqlens, max_seqlen, and sequence_lengths as in qwen3_vl + vision encoder. + """ + pytest.importorskip("flashinfer") + + num_heads = 16 + head_size = 72 + set_random_seed(0) + torch.set_default_device(device) + torch.set_default_dtype(dtype) + + # Override vllm config so get_vit_attn_backend returns FLASHINFER (simulates + # --mm-encoder-attn-backend=FLASHINFER). + vllm_config = get_current_vllm_config() + old_model_config = getattr(vllm_config, "model_config", None) + minimal_model_config = type( + "MinimalModelConfig", + (), + { + "multimodal_config": MultiModalConfig( + mm_encoder_attn_backend=AttentionBackendEnum.FLASHINFER + ), + }, + )() + vllm_config.model_config = minimal_model_config + try: + total_len = sum(var_seq_len) + # Stride of second dim = 3 * num_heads * head_size (same as qwen2_5_vl + # after qkv rearrange and unbind: qkv shape (b, s, 3, head, head_dim)). + qkv = torch.randn(1, total_len, 3, num_heads, head_size) + q, k, v = qkv.unbind(dim=2) + + cu_seqlens_np = np.array( + [0] + list(itertools.accumulate(var_seq_len)), dtype=np.int32 + ) + hidden_size = num_heads * head_size + tp_size = 1 + + sequence_lengths = MMEncoderAttention.maybe_compute_seq_lens( + AttentionBackendEnum.FLASHINFER, + cu_seqlens_np, + device, + ) + + max_seqlen_val = MMEncoderAttention.compute_max_seqlen( + AttentionBackendEnum.FLASHINFER, cu_seqlens_np + ) + max_seqlen = torch.tensor(max_seqlen_val, device=device, dtype=torch.int32) + + cu_seqlens = MMEncoderAttention.maybe_recompute_cu_seqlens( + AttentionBackendEnum.FLASHINFER, + cu_seqlens_np, + hidden_size, + tp_size, + device, + ) + + scale = 1.0 / head_size**0.5 + attn = MMEncoderAttention( + num_heads, + head_size, + scale=scale, + num_kv_heads=num_heads, + ) + assert attn.attn_backend == AttentionBackendEnum.FLASHINFER + + output = attn( + q, + k, + v, + cu_seqlens=cu_seqlens, + max_seqlen=max_seqlen, + sequence_lengths=sequence_lengths, + ) + + ref_output = [] + for q_i, k_i, v_i in zip( + torch.split(q, var_seq_len, dim=1), + torch.split(k, var_seq_len, dim=1), + torch.split(v, var_seq_len, dim=1), + ): + output_i = ref_attention(q_i, k_i, v_i, scale=scale) + ref_output.append(output_i) + ref_output = torch.cat(ref_output, dim=1) + torch.testing.assert_close(output, ref_output, atol=1e-2, rtol=1e-2) + finally: + vllm_config.model_config = old_model_config diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_minimax_m3.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_minimax_m3.py new file mode 100644 index 0000000000000000000000000000000000000000..835b5da0a2f6be87373bbbdfab0914ccce97ae0c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_minimax_m3.py @@ -0,0 +1,1226 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Correctness tests for MiniMax M3 sparse prefill attention kernels.""" + +import pytest +import torch + +from vllm import _custom_ops as ops +from vllm.models.minimax_m3.common.indexer import ( + MiniMaxM3IndexerBackend, +) +from vllm.models.minimax_m3.common.ops.index_topk import ( + minimax_m3_index_decode, + minimax_m3_index_score, + minimax_m3_index_topk, +) +from vllm.models.minimax_m3.common.ops.sparse_attn import ( + _FP8_DTYPES, + minimax_m3_sparse_attn, + minimax_m3_sparse_attn_decode, +) +from vllm.models.minimax_m3.common.sparse_attention import ( + MiniMaxM3SparseBackend, + MiniMaxM3SparseTritonImpl, +) +from vllm.platforms import current_platform +from vllm.v1.attention.backends.utils import set_kv_cache_layout +from vllm.v1.kv_cache_interface import FullAttentionSpec, MLAAttentionSpec +from vllm.v1.worker.gpu.attn_utils import _reshape_kv_cache +from vllm.v1.worker.utils import AttentionGroup + +if not (current_platform.is_cuda() or current_platform.is_rocm()): + pytest.skip( + "MiniMax M3 attention kernels require CUDA or ROCm.", + allow_module_level=True, + ) + + +@pytest.fixture +def kv_layout(request): + """Set the global KV cache layout for one test and restore it after.""" + set_kv_cache_layout(request.param) + try: + yield request.param + finally: + set_kv_cache_layout(None) + + +def _stride_order_for(backend: type[MiniMaxM3SparseBackend], ndim: int) -> tuple: + """Mirror the allocator's stride-order resolution (identity fallback).""" + try: + stride_order = backend.get_kv_cache_stride_order() + assert len(stride_order) == ndim + except (AttributeError, NotImplementedError): + stride_order = tuple(range(ndim)) + return stride_order + + +def _allocate_main_kv_via_contract( + num_pages: int, device: torch.device | str = "cuda" +) -> torch.Tensor: + """Build the main KV cache exactly as the production allocator does for the + currently active layout: allocate the physical (permuted) tensor, then + expose the inverse-permuted logical-NHD view the backend sees.""" + logical_shape = MiniMaxM3SparseBackend.get_kv_cache_shape( + num_pages, BLOCK_SIZE, NUM_KV_HEADS, HEAD_DIM + ) + stride_order = _stride_order_for(MiniMaxM3SparseBackend, len(logical_shape)) + physical_shape = tuple(logical_shape[i] for i in stride_order) + inv_order = [stride_order.index(i) for i in range(len(stride_order))] + raw = torch.randn(physical_shape, device=device, dtype=DTYPE) + return raw.permute(*inv_order) + + +NUM_Q_HEADS = 32 +NUM_KV_HEADS = 2 +HEAD_DIM = 128 +BLOCK_SIZE = 128 +DTYPE = torch.bfloat16 +SM_SCALE = HEAD_DIM**-0.5 +TOPK = 16 + + +@pytest.mark.parametrize( + ("kv_cache_dtype", "expected_dtype"), + [ + ("fp8", current_platform.fp8_dtype()), + ("fp8_e4m3", current_platform.fp8_dtype()), + ( + "fp8_e5m2", + torch.float8_e5m2fnuz + if current_platform.is_fp8_fnuz() + else torch.float8_e5m2, + ), + ], +) +def test_sparse_impl_uses_platform_fp8_dtype( + kv_cache_dtype: str, + expected_dtype: torch.dtype, +): + impl = MiniMaxM3SparseTritonImpl( + num_heads=NUM_Q_HEADS, + head_size=HEAD_DIM, + scale=SM_SCALE, + num_kv_heads=NUM_KV_HEADS, + kv_cache_dtype=kv_cache_dtype, + topk_blocks=TOPK, + sparse_block_size=BLOCK_SIZE, + ) + assert impl.kv_cache_fp8_dtype == expected_dtype + + +@pytest.mark.parametrize( + "dtype", + [ + torch.float8_e4m3fn, + torch.float8_e4m3fnuz, + torch.float8_e5m2, + torch.float8_e5m2fnuz, + ], +) +def test_sparse_kernels_recognize_fp8_dtypes(dtype: torch.dtype): + assert dtype in _FP8_DTYPES + + +# Index top-k kernels. +def _reference_index_topk( + idx_q: torch.Tensor, + index_kv_cache: torch.Tensor, + block_table: torch.Tensor, + q_lens: torch.Tensor, + seq_lens: torch.Tensor, + prefix_lens: torch.Tensor, + topk: int, + init_blocks: int, + local_blocks: int, + sm_scale: float = 1.0, +) -> torch.Tensor: + total_q, num_idx_heads, _ = idx_q.shape + out = torch.full( + (num_idx_heads, total_q, topk), -1, device=idx_q.device, dtype=torch.int32 + ) + + q_start = 0 + for req_id, (q_len, seq_len, prefix_len) in enumerate( + zip(q_lens.tolist(), seq_lens.tolist(), prefix_lens.tolist()) + ): + q_end = q_start + q_len + q = idx_q[q_start:q_end] + num_blocks = (seq_len + BLOCK_SIZE - 1) // BLOCK_SIZE + pages = block_table[req_id, :num_blocks] + k = index_kv_cache[pages].reshape(num_blocks * BLOCK_SIZE, -1) + score = torch.einsum("qhd,kd->hqk", q.float(), k.float()) * sm_scale + + q_pos = prefix_len + torch.arange(q_len, device=idx_q.device) + k_pos = torch.arange(k.shape[0], device=idx_q.device) + score.masked_fill_(k_pos[None, :] > q_pos[:, None], -float("inf")) + score = score.reshape(num_idx_heads, q_len, num_blocks, BLOCK_SIZE) + score_tensor = score.max(dim=3).values + + valid_blocks = (q_pos + BLOCK_SIZE) // BLOCK_SIZE + for local_q, num_valid_blocks in enumerate(valid_blocks.tolist()): + end = min(init_blocks, num_valid_blocks) + score_tensor[:, local_q, :end] = 1e30 + start = max(0, num_valid_blocks - local_blocks) + score_tensor[:, local_q, start:num_valid_blocks] = 1e29 + + k = min(topk, num_valid_blocks) + topk_idx = score_tensor[:, local_q].topk(k, dim=1).indices + out[:, q_start + local_q, :k] = topk_idx + q_start = q_end + + return out + + +def _assert_topk_indices_equal_unordered( + actual: torch.Tensor, + expected: torch.Tensor, +) -> None: + """Compare selected sparse blocks without requiring a deterministic order.""" + assert actual.shape == expected.shape + actual_flat = actual.cpu().reshape(-1, actual.shape[-1]).tolist() + expected_flat = expected.cpu().reshape(-1, expected.shape[-1]).tolist() + for actual_row, expected_row in zip(actual_flat, expected_flat): + assert set(actual_row) == set(expected_row) + + +def test_prefill_index_topk_correctness(): + topk = 6 + init_blocks = 0 + local_blocks = 1 + num_idx_heads = 2 + head_dim = 16 + q_lens = torch.tensor((4, 3), device="cuda", dtype=torch.int32) + prefix_lens = torch.tensor((0, 1024), device="cuda", dtype=torch.int32) + seq_lens = prefix_lens + q_lens + batch = q_lens.numel() + max_seq_len = seq_lens.max().item() + max_blocks = (max_seq_len + BLOCK_SIZE - 1) // BLOCK_SIZE + num_pages = batch * max_blocks + + cu_seqlens = torch.zeros(batch + 1, device="cuda", dtype=torch.int32) + cu_seqlens[1:] = q_lens.cumsum(0) + block_table = torch.randperm(num_pages, device="cuda", dtype=torch.int32).reshape( + batch, max_blocks + ) + idx_q = torch.ones(q_lens.sum().item(), num_idx_heads, head_dim, device="cuda") + index_kv_cache = torch.empty(num_pages, BLOCK_SIZE, head_dim, device="cuda") + for req_id in range(batch): + for block_id in range(max_blocks): + page = block_table[req_id, block_id] + index_kv_cache[page].fill_(block_id + 1) + + score = minimax_m3_index_score( + idx_q, + index_kv_cache, + block_table, + cu_seqlens, + seq_lens, + prefix_lens, + max_query_len=q_lens.max().item(), + max_seq_len=max_seq_len, + num_kv_heads=num_idx_heads, + ) + actual = minimax_m3_index_topk( + score, + cu_seqlens, + prefix_lens, + max_query_len=q_lens.max().item(), + topk=topk, + init_blocks=init_blocks, + local_blocks=local_blocks, + ) + expected = _reference_index_topk( + idx_q, + index_kv_cache, + block_table, + q_lens, + seq_lens, + prefix_lens, + topk, + init_blocks, + local_blocks, + ) + _assert_topk_indices_equal_unordered(actual, expected) + + +# MSA indexer (SM100): fmha_sm100 OnlyScore for the per-block scores, then the +# Triton minimax_m3_index_topk for selection (no sparse_topk_select). Uses a +# deterministic construction (idx_q == 1, distinct e4m3-exact per-block values) +# so scores are strictly monotonic in the block id -> exact top-k agreement. +def _fmha_indexer_topk( + idx_q: torch.Tensor, # [total_q, H, 128] bf16/e4m3 + index_cache: torch.Tensor, # [num_pages, 128, 128] bf16/e4m3 + block_table: torch.Tensor, + q_lens: torch.Tensor, + seq_lens: torch.Tensor, + prefix_lens: torch.Tensor, + sm_scale: float, + topk: int, +) -> torch.Tensor: + """Replicate MiniMaxM3IndexerMSAImpl's score path (single decode/prefill side).""" + from vllm.third_party.fmha_sm100.api import _fmha_sm100, _fmha_sm100_plan + + num_idx_heads, head_dim = idx_q.shape[1], idx_q.shape[2] + nvp = [(s + 127) // 128 for s in seq_lens.tolist()] + kv_indices = torch.cat([block_table[r, : nvp[r]] for r in range(len(nvp))]).to( + torch.int32 + ) + + qo = q_lens.cpu().to(torch.int32) + kv = seq_lens.cpu().to(torch.int32) + plan = _fmha_sm100_plan( + qo, + kv, + num_idx_heads, + num_kv_heads=1, + qo_offset=kv - qo, + page_size=128, + output_maxscore=True, + causal=True, + num_kv_splits=1, + ) + k_pages = index_cache.view(index_cache.shape[0], 1, 128, head_dim) + _, max_score = _fmha_sm100( + idx_q, + k_pages, + k_pages, + plan, + kv_indices=kv_indices, + output_o=False, + output_maxscore=True, + sm_scale=sm_scale, + ) + + batch = q_lens.numel() + cu = torch.zeros(batch + 1, dtype=torch.int32, device=idx_q.device) + cu[1:] = q_lens.to(torch.int32).cumsum(0) + # max_score [H, k_tiles, total_q] -> transpose to [H, total_q, k_tiles]. + return minimax_m3_index_topk( + max_score.transpose(1, 2), + cu, + prefix_lens.to(torch.int32), + int(q_lens.max()), + topk, + 0, # init_blocks + 0, # local_blocks + ) + + +# e4m3-exact, strictly-increasing per-block values: with idx_q == 1 (also exact) +# the per-block scores are exact and distinct in BOTH bf16 and e4m3, so the fp8 +# score path selects the same top-k as the reference (no quantization ties). +_E4M3_EXACT_VALUES = [ + *range(1, 17), # 1..16 (step 1) + *range(18, 33, 2), # 18..32 (step 2) + *range(36, 65, 4), # 36..64 (step 4) + *range(72, 129, 8), # 72..128 (step 8) +] + + +@pytest.mark.skipif( + not current_platform.is_device_capability_family(100), + reason="fmha_sm100 indexer requires SM100 (Blackwell).", +) +@pytest.mark.parametrize("index_dtype", [torch.bfloat16, torch.float8_e4m3fn]) +@pytest.mark.parametrize( + ("q_lens", "prefix_lens"), + [ + ((4, 3), (2048, 2560)), # prefill: every token sees >= 16 causal blocks + ((1, 1, 1), (2048, 3000, 4096)), # decode: one query token per request + ], +) +def test_fmha_sm100_indexer_matches_reference(q_lens, prefix_lens, index_dtype): + torch.manual_seed(0) + num_idx_heads, head_dim = 4, HEAD_DIM + device = "cuda" + + q_lens_t = torch.tensor(q_lens, device=device, dtype=torch.int32) + prefix_lens_t = torch.tensor(prefix_lens, device=device, dtype=torch.int32) + seq_lens = prefix_lens_t + q_lens_t + batch = len(q_lens) + max_blocks = (int(seq_lens.max()) + BLOCK_SIZE - 1) // BLOCK_SIZE + assert max_blocks <= len(_E4M3_EXACT_VALUES) + num_pages = batch * max_blocks + block_table = torch.randperm(num_pages, device=device, dtype=torch.int32).reshape( + batch, max_blocks + ) + + idx_q = torch.ones( + int(q_lens_t.sum()), num_idx_heads, head_dim, device=device, dtype=index_dtype + ) + index_cache = torch.empty( + num_pages, BLOCK_SIZE, head_dim, device=device, dtype=index_dtype + ) + for r in range(batch): + for b in range(max_blocks): + index_cache[block_table[r, b]] = float(_E4M3_EXACT_VALUES[b]) + + sm_scale = head_dim**-0.5 + actual = _fmha_indexer_topk( + idx_q, + index_cache, + block_table, + q_lens_t, + seq_lens, + prefix_lens_t, + sm_scale, + TOPK, + ) + expected = _reference_index_topk( + idx_q, + index_cache, + block_table, + q_lens_t, + seq_lens, + prefix_lens_t, + TOPK, + init_blocks=0, + local_blocks=0, + sm_scale=sm_scale, + ) + _assert_topk_indices_equal_unordered(actual, expected) + + +# Full impl-level parity: drive both MiniMaxM3IndexerMSAImpl (fmha_sm100 score + +# Triton top-k) and MiniMaxM3IndexerTritonImpl through their real metadata +# builders on the SAME CommonAttentionMetadata + index cache, and assert the +# selected blocks agree. This exercises all the metadata the impl/kernels consume +# (decode/prefill split, cu_seqlens_q rebasing, prefix_lens, kv_indices gather, +# decode_pages split) -- a metadata bug on either side shifts the causal window +# or the block->page mapping and breaks the comparison. +@pytest.mark.skipif( + not current_platform.is_device_capability_family(100), + reason="fmha_sm100 indexer requires SM100 (Blackwell).", +) +@pytest.mark.parametrize("topk", [8, 16]) +def test_msa_indexer_impl_matches_triton(topk, monkeypatch): + import vllm.models.minimax_m3.common.indexer as indexer_mod + from tests.v1.attention.utils import ( + BatchSpec, + create_common_attn_metadata, + create_vllm_config, + ) + from vllm.config import set_current_vllm_config + from vllm.forward_context import set_forward_context + from vllm.models.minimax_m3.common.indexer import ( + MiniMaxM3IndexerTritonImpl, + MiniMaxM3IndexerTritonMetadataBuilder, + ) + from vllm.models.minimax_m3.nvidia.indexer_msa import ( + MiniMaxM3IndexerMSAImpl, + MiniMaxM3IndexerMSAMetadataBuilder, + ) + + torch.manual_seed(0) + device = torch.device("cuda") + num_idx_heads, head_dim = 4, HEAD_DIM + # TP=1: avoid requiring an initialized distributed group in a unit test. + monkeypatch.setattr(indexer_mod, "get_tensor_model_parallel_world_size", lambda: 1) + + vllm_config = create_vllm_config( + block_size=BLOCK_SIZE, max_model_len=8192, max_num_batched_tokens=8192 + ) + vllm_config.model_config.hf_config.sparse_attention_config = { + "sparse_num_index_heads": num_idx_heads + } + + # Decode-first mixed batch: 2 decode reqs (q_len 1) then 2 prefill reqs. Long + # prefixes so every token sees > TOPK causal blocks (non-trivial selection). + batch = BatchSpec(seq_lens=[2305, 2561, 2624, 2720], query_lens=[1, 1, 64, 96]) + common = create_common_attn_metadata( + batch, BLOCK_SIZE, device, arange_block_indices=True + ) + num_tokens = batch.compute_num_tokens() + + # Deterministic index cache: distinct, monotonic per-logical-block values so + # the top-k is unambiguous (both kernels pick the same blocks, no fp ties). + block_table = common.block_table_tensor + num_pages = int(block_table.max().item()) + 1 + index_cache = torch.zeros( + num_pages, BLOCK_SIZE, head_dim, device=device, dtype=DTYPE + ) + for r, seq_len in enumerate(batch.seq_lens): + for b in range((seq_len + BLOCK_SIZE - 1) // BLOCK_SIZE): + index_cache[block_table[r, b]] = float(b + 1) + index_q = torch.ones( + num_tokens, num_idx_heads * head_dim, device=device, dtype=DTYPE + ) + + spec = MLAAttentionSpec( + block_size=BLOCK_SIZE, num_kv_heads=1, head_size=head_dim, dtype=DTYPE + ) + impl_kwargs = dict( + num_kv_heads=num_idx_heads, + scale=head_dim**-0.5, + topk_blocks=topk, + sparse_block_size=BLOCK_SIZE, + num_index_heads=num_idx_heads, + index_head_dim=head_dim, + init_blocks=0, + local_blocks=0, + ) + + with set_current_vllm_config(vllm_config): + msa_impl = MiniMaxM3IndexerMSAImpl(prefix="idx_msa", **impl_kwargs) + triton_impl = MiniMaxM3IndexerTritonImpl(prefix="idx_triton", **impl_kwargs) + msa_builder = MiniMaxM3IndexerMSAMetadataBuilder( + spec, [msa_impl.index_cache.prefix], vllm_config, device + ) + triton_builder = MiniMaxM3IndexerTritonMetadataBuilder( + spec, [triton_impl.index_cache.prefix], vllm_config, device + ) + + # Both impls score against the same index keys. + msa_impl.index_cache.kv_cache = index_cache + triton_impl.index_cache.kv_cache = index_cache + + # Exercise the shared persistent top-k buffer for BOTH impls: each must write + # decode ([:, :nd]) and prefill ([:, nd:]) into its buffer and return views. + # Separate buffers so the two forwards don't clobber each other. + nd = sum(q for q in batch.query_lens if q <= 1) + msa_impl.topk_indices_buffer = torch.full( + (num_idx_heads, num_tokens, topk), -2, dtype=torch.int32, device=device + ) + triton_impl.topk_indices_buffer = torch.full( + (num_idx_heads, num_tokens, topk), -2, dtype=torch.int32, device=device + ) + + attn_metadata = { + msa_impl.index_cache.prefix: msa_builder.build(0, common), + triton_impl.index_cache.prefix: triton_builder.build(0, common), + } + with set_forward_context(attn_metadata, vllm_config): + msa_decode, msa_prefill = msa_impl(index_q) + tri_decode, tri_prefill = triton_impl(index_q) + + assert msa_decode is not None and tri_decode is not None + assert msa_prefill is not None and tri_prefill is not None + _assert_topk_indices_equal_unordered(msa_decode, tri_decode) + _assert_topk_indices_equal_unordered(msa_prefill, tri_prefill) + # decode/prefill outputs are views into each impl's persistent buffer. + for impl, dec, pre in ( + (msa_impl, msa_decode, msa_prefill), + (triton_impl, tri_decode, tri_prefill), + ): + buf = impl.topk_indices_buffer + assert dec.data_ptr() == buf[:, :nd, :].data_ptr() + assert pre.data_ptr() == buf[:, nd:, :].data_ptr() + + +@pytest.mark.parametrize( + ("decode_query_len", "max_decode_query_len"), + [ + (1, 1), + (1, 4), + (4, 4), + ], +) +@pytest.mark.parametrize("num_padded_reqs", [0, 2]) +def test_decode_index_topk_correctness( + decode_query_len: int, + max_decode_query_len: int, + num_padded_reqs: int, +): + topk = 6 + init_blocks = 0 + local_blocks = 1 + num_idx_heads = 2 + head_dim = 16 + active_seq_lens = torch.tensor((7, 129, 1025), device="cuda", dtype=torch.int32) + q_lens = torch.full_like(active_seq_lens, decode_query_len) + prefix_lens = active_seq_lens - decode_query_len + active_batch = active_seq_lens.numel() + batch = active_batch + num_padded_reqs + seq_lens = torch.cat( + [ + active_seq_lens, + torch.zeros(num_padded_reqs, device="cuda", dtype=torch.int32), + ] + ) + max_seq_len = active_seq_lens.max().item() + max_blocks = (max_seq_len + BLOCK_SIZE - 1) // BLOCK_SIZE + num_pages = active_batch * max_blocks + + active_block_table = torch.randperm( + num_pages, device="cuda", dtype=torch.int32 + ).reshape(active_batch, max_blocks) + block_table = torch.zeros(batch, max_blocks, device="cuda", dtype=torch.int32) + block_table[:active_batch] = active_block_table + idx_q = torch.randn( + batch * decode_query_len, num_idx_heads, head_dim, device="cuda" + ) + index_kv_cache = torch.randn(num_pages, BLOCK_SIZE, head_dim, device="cuda") + + actual = minimax_m3_index_decode( + idx_q, + index_kv_cache, + block_table, + seq_lens, + max_seq_len=max_seq_len, + topk=topk, + init_blocks=init_blocks, + local_blocks=local_blocks, + num_kv_heads=num_idx_heads, + decode_query_len=decode_query_len, + max_decode_query_len=max_decode_query_len, + ) + expected = torch.full_like(actual, -1) + active_tokens = active_batch * decode_query_len + expected[:, :active_tokens] = _reference_index_topk( + idx_q[:active_tokens], + index_kv_cache, + block_table[:active_batch], + q_lens, + active_seq_lens, + prefix_lens, + topk, + init_blocks, + local_blocks, + ) + _assert_topk_indices_equal_unordered(actual, expected) + + +@pytest.mark.skipif( + not current_platform.is_device_capability_family(100), + reason="fp8 e4m3 indexer cache is the SM100 (MSA) path.", +) +@pytest.mark.parametrize("num_idx_heads", [1, 4]) +def test_decode_index_topk_fp8(num_idx_heads: int): + """The fp8 (e4m3) indexer cache feeds the Triton decode kernel on the MSA + path. The kernel must score in fp32 (no scaling) so its top-k matches a + reference computed from the dequantized fp8 values.""" + torch.manual_seed(0) + topk, init_blocks, local_blocks, head_dim = 8, 0, 1, 128 + decode_query_len = 1 + active_seq_lens = torch.tensor((129, 1025, 4097), device="cuda", dtype=torch.int32) + q_lens = torch.full_like(active_seq_lens, decode_query_len) + prefix_lens = active_seq_lens - decode_query_len + batch = active_seq_lens.numel() + max_seq_len = int(active_seq_lens.max()) + max_blocks = (max_seq_len + BLOCK_SIZE - 1) // BLOCK_SIZE + num_pages = batch * max_blocks + block_table = torch.randperm(num_pages, device="cuda", dtype=torch.int32).reshape( + batch, max_blocks + ) + idx_q = torch.randn( + batch * decode_query_len, num_idx_heads, head_dim, device="cuda" + ).to(torch.float8_e4m3fn) + index_kv_cache = torch.randn(num_pages, BLOCK_SIZE, head_dim, device="cuda").to( + torch.float8_e4m3fn + ) + + actual = minimax_m3_index_decode( + idx_q, + index_kv_cache, + block_table, + active_seq_lens, + max_seq_len=max_seq_len, + topk=topk, + init_blocks=init_blocks, + local_blocks=local_blocks, + num_kv_heads=num_idx_heads, + sm_scale=head_dim**-0.5, + decode_query_len=decode_query_len, + ) + # Reference from the DEQUANTIZED fp8 values (the kernel computes the fp8 QK + # in fp32, so it must match an fp32 matmul of the same e4m3 values). + expected = _reference_index_topk( + idx_q.float(), + index_kv_cache.float(), + block_table, + q_lens, + active_seq_lens, + prefix_lens, + topk, + init_blocks, + local_blocks, + head_dim**-0.5, + ) + _assert_topk_indices_equal_unordered(actual, expected) + + +# Sparse attention kernels. +def _reference_sparse_attn( + q: torch.Tensor, + kv_cache: torch.Tensor, + topk_idx: torch.Tensor, + block_table: torch.Tensor, + q_lens: torch.Tensor, + seq_lens: torch.Tensor, + prefix_lens: torch.Tensor, +) -> torch.Tensor: + out = torch.empty_like(q, dtype=torch.float32) + gqa_group_size = NUM_Q_HEADS // NUM_KV_HEADS + q_start = 0 + for req_id, (q_len, seq_len, prefix_len) in enumerate( + zip(q_lens.tolist(), seq_lens.tolist(), prefix_lens.tolist()) + ): + q_end = q_start + q_len + q_req = q[q_start:q_end] + positions = torch.arange(seq_len, device="cuda") + pages = block_table[req_id, positions // BLOCK_SIZE] + rows = positions % BLOCK_SIZE + k_req = kv_cache[pages, 0, rows] + v_req = kv_cache[pages, 1, rows].float() + + q_pos = prefix_len + torch.arange(q_len, device="cuda") + key_blocks = positions // BLOCK_SIZE + causal_mask = positions.unsqueeze(0) <= q_pos.unsqueeze(1) + + for kv_head in range(NUM_KV_HEADS): + selected = topk_idx[kv_head, q_start:q_end] + selected_mask = (key_blocks[None, :, None] == selected[:, None, :]).any(-1) + mask = causal_mask & selected_mask + head_start = kv_head * gqa_group_size + head_end = head_start + gqa_group_size + + q_heads = q_req[:, head_start:head_end].transpose(0, 1) + k_head = k_req[:, kv_head].T.expand(gqa_group_size, -1, -1) + scores = torch.bmm(q_heads, k_head, out_dtype=torch.float32) + scores = scores.transpose(0, 1) * SM_SCALE + probs = torch.softmax( + scores.masked_fill(~mask[:, None, :], -float("inf")), -1 + ) + out[q_start:q_end, head_start:head_end] = torch.einsum( + "qhk,kd->qhd", probs, v_req[:, kv_head] + ) + q_start += q_len + return out.to(q.dtype) + + +@pytest.mark.parametrize("kv_layout", ["NHD", "HND"], indirect=True) +@pytest.mark.parametrize( + ("q_lens", "kv_lens"), + [ + ((129, 257), (129, 257)), + ((65, 129, 257), (129, 257, 385)), + ], +) +def test_prefill_sparse_attention_correctness( + kv_layout: str, + q_lens: tuple[int, ...], + kv_lens: tuple[int, ...], +): + assert len(q_lens) == len(kv_lens) + assert all(kv_len >= q_len for q_len, kv_len in zip(q_lens, kv_lens)) + + # Build paged-KV metadata, including a non-identity page order. + batch = len(q_lens) + pages_per_req = [(kv_len + BLOCK_SIZE - 1) // BLOCK_SIZE for kv_len in kv_lens] + max_blocks = max(pages_per_req) + num_pages = sum(pages_per_req) + physical_pages = torch.randperm(num_pages, device="cuda", dtype=torch.int32) + block_table = torch.zeros(batch, max_blocks, device="cuda", dtype=torch.int32) + base_page = 0 + for req_id, num_req_pages in enumerate(pages_per_req): + block_table[req_id, :num_req_pages] = physical_pages[ + base_page : base_page + num_req_pages + ] + base_page += num_req_pages + + q_lens_t = torch.tensor(q_lens, device="cuda", dtype=torch.int32) + seq_lens = torch.tensor(kv_lens, device="cuda", dtype=torch.int32) + prefix_lens = seq_lens - q_lens_t + cu_seqlens = torch.zeros(batch + 1, device="cuda", dtype=torch.int32) + cu_seqlens[1:] = q_lens_t.cumsum(0) + total_q = sum(q_lens) + max_seqlen_q = max(q_lens) + + q_shape = (total_q, NUM_Q_HEADS, HEAD_DIM) + q = torch.randn(q_shape, device="cuda", dtype=DTYPE) + # Allocate the main KV cache through the backend layout contract so the + # physical storage matches the active layout (contiguous NHD or strided + # HND), while the kernels and reference see the logical-NHD view. + kv_cache = _allocate_main_kv_via_contract(num_pages) + + # Build sparse block indices with the same contract as the real M3 indexer: + # one forced local block, then score-selected older causal blocks. + topk_shape = (NUM_KV_HEADS, total_q, TOPK) + topk_idx = torch.full(topk_shape, -1, device="cuda", dtype=torch.int32) + q_start = 0 + for q_len, prefix_len in zip(q_lens_t.tolist(), prefix_lens.tolist()): + for local_q in range(q_len): + current_block = (prefix_len + local_q) // BLOCK_SIZE + older_blocks = torch.randperm( + current_block, device="cuda", dtype=torch.int32 + ) + selected = torch.cat( + [ + torch.tensor([current_block], device="cuda", dtype=torch.int32), + older_blocks[: TOPK - 1], + ] + ) + topk_idx[:, q_start + local_q, : selected.numel()] = selected + q_start += q_len + + actual = torch.empty_like(q) + minimax_m3_sparse_attn( + q, + kv_cache, + topk_idx, + block_table, + cu_seqlens, + seq_lens, + prefix_lens, + max_seqlen_q, + NUM_KV_HEADS, + SM_SCALE, + actual, + ) + + expected = _reference_sparse_attn( + q, + kv_cache, + topk_idx, + block_table, + q_lens_t, + seq_lens, + prefix_lens, + ) + torch.accelerator.synchronize() + + error = (actual.float() - expected.float()).abs() + assert error.mean().item() < 2.5e-4 + assert error.max().item() < 1.7e-2 + + +def test_main_backend_layout_contract(): + """The main sparse backend exposes the logical-NHD shape and the + flash_attn-style stride order for each layout.""" + nb, bs, h, d = 7, BLOCK_SIZE, NUM_KV_HEADS, HEAD_DIM + logical = MiniMaxM3SparseBackend.get_kv_cache_shape(nb, bs, h, d) + assert logical == (nb, 2, bs, h, d) + # The old HND-ordered shape is no longer the logical shape. + assert logical != (nb, 2, h, bs, d) + + try: + set_kv_cache_layout("HND") + assert MiniMaxM3SparseBackend.get_kv_cache_stride_order() == (0, 1, 3, 2, 4) + set_kv_cache_layout("NHD") + assert MiniMaxM3SparseBackend.get_kv_cache_stride_order() == (0, 1, 2, 3, 4) + finally: + set_kv_cache_layout(None) + + for layout in ("NHD", "HND"): + try: + set_kv_cache_layout(layout) + order = MiniMaxM3SparseBackend.get_kv_cache_stride_order() + finally: + set_kv_cache_layout(None) + # Valid permutation: no duplicates, covers every axis. + assert set(order) == set(range(len(order))) + + # M3 has no cross-layer KV blocks. + with pytest.raises(NotImplementedError): + MiniMaxM3SparseBackend.get_kv_cache_stride_order( + include_num_layers_dimension=True + ) + + +def test_main_backend_unknown_layout_raises(monkeypatch): + """An unrecognized layout (injected past env-var validation) is rejected.""" + import vllm.models.minimax_m3.common.sparse_attention as sparse_attn_mod + + monkeypatch.setattr(sparse_attn_mod, "get_kv_cache_layout", lambda: "BOGUS") + with pytest.raises(ValueError, match="Unknown cache layout format"): + MiniMaxM3SparseBackend.get_kv_cache_stride_order() + + +def test_indexer_backend_stride_order_is_identity(): + """The 3-dim indexer cache must not inherit the parent's 5-element stride + order; it overrides to the 3-element identity so the allocator keeps the + contiguous layout.""" + assert MiniMaxM3IndexerBackend.get_kv_cache_stride_order() == (0, 1, 2) + + # Cross-layer (per-layer-stacked) KV blocks are not supported. + with pytest.raises(NotImplementedError): + MiniMaxM3IndexerBackend.get_kv_cache_stride_order( + include_num_layers_dimension=True + ) + + # The stride order matches the 3-dim indexer shape rank. + indexer_shape = MiniMaxM3IndexerBackend.get_kv_cache_shape( + 5, BLOCK_SIZE, 1, HEAD_DIM + ) + assert len(indexer_shape) == 3 + assert _stride_order_for(MiniMaxM3IndexerBackend, len(indexer_shape)) == (0, 1, 2) + + +def test_hnd_allocation_is_byte_identical_to_transpose(): + """Under HND the backend-visible logical view is byte-identical to the + pre-change allocate-HND-then-transpose(2, 3) workaround.""" + nb, bs, h, d = 4, BLOCK_SIZE, NUM_KV_HEADS, HEAD_DIM + logical = MiniMaxM3SparseBackend.get_kv_cache_shape(nb, bs, h, d) + try: + set_kv_cache_layout("HND") + stride_order = MiniMaxM3SparseBackend.get_kv_cache_stride_order() + finally: + set_kv_cache_layout(None) + + physical_shape = tuple(logical[i] for i in stride_order) + # The physical (permuted) shape equals the old hardcoded HND shape. + assert physical_shape == (nb, 2, h, bs, d) + + inv_order = [stride_order.index(i) for i in range(len(stride_order))] + raw = torch.empty(physical_shape, device="cuda", dtype=DTYPE) + view = raw.permute(*inv_order) + expected = raw.view((nb, 2, h, bs, d)).transpose(2, 3) + + assert view.shape == expected.shape + assert view.stride() == expected.stride() + assert view.storage_offset() == expected.storage_offset() + + # Negative: the identity (wrong) stride order under HND does not reproduce + # the transpose view. + wrong_view = raw.view(logical) + assert wrong_view.stride() != expected.stride() + + +def test_main_cache_is_block_first_and_unpadded(): + """The allocator's contiguous-view branch (not the padded-strided branch) + is used for the main GQA cache: its spec is unpadded and the physical + layout keeps num_blocks as the first dimension under both layouts.""" + from vllm.v1.kv_cache_interface import FullAttentionSpec + + spec = FullAttentionSpec( + block_size=BLOCK_SIZE, + num_kv_heads=NUM_KV_HEADS, + head_size=HEAD_DIM, + head_size_v=HEAD_DIM, + dtype=DTYPE, + ) + # Unpadded -> allocator uses kv_tensor.view(...) rather than as_strided(). + assert spec.page_size_padded is None + + logical = MiniMaxM3SparseBackend.get_kv_cache_shape( + 4, BLOCK_SIZE, NUM_KV_HEADS, HEAD_DIM + ) + for layout in ("NHD", "HND"): + try: + set_kv_cache_layout(layout) + order = MiniMaxM3SparseBackend.get_kv_cache_stride_order() + finally: + set_kv_cache_layout(None) + inv_order = [order.index(i) for i in range(len(order))] + # Physical first dim is num_blocks (block-first); required by the + # padded-strided branch's block-first assumption if it were ever taken. + assert inv_order[0] == 0 + assert logical[order[0]] == logical[0] + + +def _build_decode_inputs( + seq_lens_list: tuple[int, ...], + decode_query_len: int = 1, + num_padded_reqs: int = 0, +): + """Shared decode setup: uniform query tokens per request, a non-identity + block table, and topk indices selecting the current block plus older causal + blocks for each query token.""" + active_batch = len(seq_lens_list) + batch = active_batch + num_padded_reqs + pages_per_req = [(s + BLOCK_SIZE - 1) // BLOCK_SIZE for s in seq_lens_list] + max_blocks = max(pages_per_req) + num_pages = sum(pages_per_req) + physical_pages = torch.randperm(num_pages, device="cuda", dtype=torch.int32) + block_table = torch.zeros(batch, max_blocks, device="cuda", dtype=torch.int32) + base_page = 0 + for req_id, num_req_pages in enumerate(pages_per_req): + block_table[req_id, :num_req_pages] = physical_pages[ + base_page : base_page + num_req_pages + ] + base_page += num_req_pages + + seq_lens = torch.tensor( + (*seq_lens_list, *([0] * num_padded_reqs)), + device="cuda", + dtype=torch.int32, + ) + q = torch.randn( + batch * decode_query_len, NUM_Q_HEADS, HEAD_DIM, device="cuda", dtype=DTYPE + ) + + topk_idx = torch.full( + (NUM_KV_HEADS, batch * decode_query_len, TOPK), + -1, + device="cuda", + dtype=torch.int32, + ) + token_id = 0 + for req_id, seq_len in enumerate(seq_lens_list): + for local_q in range(decode_query_len): + query_pos = seq_len - decode_query_len + local_q + current_block = query_pos // BLOCK_SIZE + older_blocks = torch.randperm( + current_block, device="cuda", dtype=torch.int32 + ) + selected = torch.cat( + [ + torch.tensor([current_block], device="cuda", dtype=torch.int32), + older_blocks[: TOPK - 1], + ] + ) + topk_idx[:, token_id, : selected.numel()] = selected + token_id += 1 + + return q, block_table, seq_lens, topk_idx, num_pages + + +@pytest.mark.parametrize("kv_layout", ["NHD", "HND"], indirect=True) +@pytest.mark.parametrize( + "seq_lens_list", + [(130, 257), (129, 200, 384)], +) +@pytest.mark.parametrize("decode_query_len", [1, 4]) +@pytest.mark.parametrize("num_padded_reqs", [0, 2]) +def test_decode_sparse_attention_correctness( + kv_layout: str, + seq_lens_list: tuple[int, ...], + decode_query_len: int, + num_padded_reqs: int, +): + """Decode (split-K) parity under both layouts: this is the only coverage of + the decode-site cache feed, and the strided HND case fails if the kernel + ignores the cache strides.""" + torch.manual_seed(0) + q, block_table, seq_lens, topk_idx, num_pages = _build_decode_inputs( + seq_lens_list, decode_query_len, num_padded_reqs + ) + kv_cache = _allocate_main_kv_via_contract(num_pages) + + actual = torch.empty_like(q) + minimax_m3_sparse_attn_decode( + q, + kv_cache, + topk_idx, + block_table, + seq_lens, + NUM_KV_HEADS, + SM_SCALE, + actual, + decode_query_len, + ) + + # Reuse the prefill reference: decode is a uniform query chunk ending at + # seq_len - 1 for each request. + active_batch = len(seq_lens_list) + active_tokens = active_batch * decode_query_len + q_lens_t = torch.full( + (len(seq_lens_list),), decode_query_len, device="cuda", dtype=torch.int32 + ) + active_seq_lens = seq_lens[:active_batch] + prefix_lens = active_seq_lens - q_lens_t + expected = _reference_sparse_attn( + q[:active_tokens], + kv_cache, + topk_idx[:, :active_tokens], + block_table[:active_batch], + q_lens_t, + active_seq_lens, + prefix_lens, + ) + torch.accelerator.synchronize() + + error = (actual[:active_tokens].float() - expected.float()).abs() + assert error.mean().item() < 2.5e-4 + assert error.max().item() < 1.7e-2 + + +def test_decode_wrong_layout_breaks_parity(): + """Negative (AC-3/AC-5): consuming the physical HND buffer as if it were + already contiguous-NHD (i.e. skipping the allocator's inverse permute) + reorders the K/V content, so the decode output no longer matches the + reference computed on the correct logical view. The mislabeled tensor keeps + the same shape as the correct view, so the kernel stays in bounds.""" + torch.manual_seed(0) + seq_lens_list = (130, 257) + q, block_table, seq_lens, topk_idx, num_pages = _build_decode_inputs(seq_lens_list) + + # Physical HND storage [blocks, 2, heads, block, dim]. + phys = torch.randn( + (num_pages, 2, NUM_KV_HEADS, BLOCK_SIZE, HEAD_DIM), device="cuda", dtype=DTYPE + ) + # Correct logical-NHD view (strided) vs. the same bytes mislabeled as a + # contiguous-NHD cache — same shape, different content mapping. + correct = phys.permute(0, 1, 3, 2, 4) + wrong = phys.reshape(num_pages, 2, BLOCK_SIZE, NUM_KV_HEADS, HEAD_DIM) + + q_lens_t = torch.ones(len(seq_lens_list), device="cuda", dtype=torch.int32) + prefix_lens = seq_lens - q_lens_t + expected = _reference_sparse_attn( + q, correct, topk_idx, block_table, q_lens_t, seq_lens, prefix_lens + ) + + actual = torch.empty_like(q) + minimax_m3_sparse_attn_decode( + q, wrong, topk_idx, block_table, seq_lens, NUM_KV_HEADS, SM_SCALE, actual, 1 + ) + torch.accelerator.synchronize() + assert (actual.float() - expected.float()).abs().max().item() > 1.7e-2 + + +def _make_attn_group(backend, spec): + return AttentionGroup( + backend=backend, + layer_names=["main"], + kv_cache_spec=spec, + kv_cache_group_id=0, + ) + + +def test_main_cache_byte_identical_through_production_allocator(): + """AC-2: drive the real allocator (`_reshape_kv_cache`) for the M3 main + `FullAttentionSpec` under HND and assert the backend-visible view has the + same shape, stride, and storage offset as the pre-change + allocate-HND-then-transpose path; the indexer `MLAAttentionSpec` allocates + through the same path to its 3-dim shape.""" + nb = 4 + spec = FullAttentionSpec( + block_size=BLOCK_SIZE, + num_kv_heads=NUM_KV_HEADS, + head_size=HEAD_DIM, + head_size_v=HEAD_DIM, + dtype=DTYPE, + ) + raw = torch.zeros(nb * spec.page_size_bytes, dtype=torch.int8) + group = _make_attn_group(MiniMaxM3SparseBackend, spec) + try: + set_kv_cache_layout("HND") + kv_caches = _reshape_kv_cache([group], {"main": raw}, "auto", [BLOCK_SIZE], {}) + finally: + set_kv_cache_layout(None) + view = kv_caches["main"] + + oracle = raw.view(DTYPE).view((nb, 2, NUM_KV_HEADS, BLOCK_SIZE, HEAD_DIM)) + oracle = oracle.transpose(2, 3) + assert tuple(view.shape) == tuple(oracle.shape) + assert view.stride() == oracle.stride() + assert view.storage_offset() == oracle.storage_offset() + + # Indexer cache allocates through the same path under both layouts. + ispec = MLAAttentionSpec( + block_size=BLOCK_SIZE, num_kv_heads=1, head_size=HEAD_DIM, dtype=DTYPE + ) + for layout in ("NHD", "HND"): + iraw = torch.zeros(nb * ispec.page_size_bytes, dtype=torch.int8) + igroup = AttentionGroup( + backend=MiniMaxM3IndexerBackend, + layer_names=["idx"], + kv_cache_spec=ispec, + kv_cache_group_id=0, + ) + try: + set_kv_cache_layout(layout) + iout = _reshape_kv_cache([igroup], {"idx": iraw}, "auto", [BLOCK_SIZE], {}) + finally: + set_kv_cache_layout(None) + assert tuple(iout["idx"].shape) == (nb, BLOCK_SIZE, HEAD_DIM) + + +def test_indexer_inherited_stride_order_trips_allocator_assert(): + """AC-4 negative: without the indexer override, the inherited 5-element + stride order trips the allocator's `len(stride_order) == len(shape)` assert + for the 3-dim indexer shape; the `AssertionError` is NOT swallowed by the + allocator's `(AttributeError, NotImplementedError)` fallback.""" + + class _BrokenIndexerBackend(MiniMaxM3IndexerBackend): + # Simulate inheriting the parent's 5-element stride order. + get_kv_cache_stride_order = staticmethod( + MiniMaxM3SparseBackend.get_kv_cache_stride_order + ) + + nb = 4 + ispec = MLAAttentionSpec( + block_size=BLOCK_SIZE, num_kv_heads=1, head_size=HEAD_DIM, dtype=DTYPE + ) + iraw = torch.zeros(nb * ispec.page_size_bytes, dtype=torch.int8) + igroup = AttentionGroup( + backend=_BrokenIndexerBackend, + layer_names=["idx"], + kv_cache_spec=ispec, + kv_cache_group_id=0, + ) + try: + set_kv_cache_layout("HND") + with pytest.raises(AssertionError): + _reshape_kv_cache([igroup], {"idx": iraw}, "auto", [BLOCK_SIZE], {}) + finally: + set_kv_cache_layout(None) + + +def test_padded_main_cache_is_flagged(): + """AC-2.1 negative: the M3 main cache relies on the allocator's + contiguous-view branch (`page_size_padded is None`). A spec that sets + `page_size_padded` is explicitly flagged rather than silently wrong-strided.""" + + def _require_unpadded_block_first(spec, stride_order): + inv_order = [stride_order.index(i) for i in range(len(stride_order))] + assert spec.page_size_padded is None, ( + "main GQA cache must be unpadded to use the contiguous-view " + "allocator branch" + ) + assert inv_order[0] == 0, "main GQA cache must remain block-first" + + try: + set_kv_cache_layout("HND") + stride_order = MiniMaxM3SparseBackend.get_kv_cache_stride_order() + finally: + set_kv_cache_layout(None) + + good = FullAttentionSpec( + block_size=BLOCK_SIZE, + num_kv_heads=NUM_KV_HEADS, + head_size=HEAD_DIM, + head_size_v=HEAD_DIM, + dtype=DTYPE, + ) + _require_unpadded_block_first(good, stride_order) # passes + + padded = FullAttentionSpec( + block_size=BLOCK_SIZE, + num_kv_heads=NUM_KV_HEADS, + head_size=HEAD_DIM, + head_size_v=HEAD_DIM, + dtype=DTYPE, + page_size_padded=good.page_size_bytes + 128, + ) + with pytest.raises(AssertionError): + _require_unpadded_block_first(padded, stride_order) + + +@pytest.mark.parametrize("kv_layout", ["NHD", "HND"], indirect=True) +def test_reshape_and_cache_flash_write_persists(kv_layout: str): + """AC-5 write path: the `reshape_and_cache_flash` write site now consumes + `self.kv_cache.unbind(1)` directly. Writing through those views must persist + into the bound storage (read back through an independent logical view) under + both layouts — a `.contiguous()` copy of the unbind slice would leave the + bound storage unchanged.""" + torch.manual_seed(0) + num_pages = 4 + kv_cache = _allocate_main_kv_via_contract(num_pages) + with torch.no_grad(): + kv_cache.zero_() + + # Exactly the production write-site code under test. + key_cache, value_cache = kv_cache.unbind(1) + + num_tokens = 12 + slot_mapping = torch.randperm(num_pages * BLOCK_SIZE, device="cuda")[ + :num_tokens + ].to(torch.int64) + key = torch.randn(num_tokens, NUM_KV_HEADS, HEAD_DIM, device="cuda", dtype=DTYPE) + value = torch.randn(num_tokens, NUM_KV_HEADS, HEAD_DIM, device="cuda", dtype=DTYPE) + scale = torch.ones((), device="cuda") + ops.reshape_and_cache_flash( + key, value, key_cache, value_cache, slot_mapping, "auto", scale, scale + ) + torch.accelerator.synchronize() + + # Read back through the independent logical view; proves the writes landed + # in the engine-bound storage, not a detached copy. + for t in range(num_tokens): + slot = int(slot_mapping[t].item()) + blk, intra = divmod(slot, BLOCK_SIZE) + torch.testing.assert_close(kv_cache[blk, 0, intra], key[t]) + torch.testing.assert_close(kv_cache[blk, 1, intra], value[t]) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_mixed_causal_attn.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_mixed_causal_attn.py new file mode 100644 index 0000000000000000000000000000000000000000..07e0a7e2ca8a99915ddb4f4540f58ec2cd933858 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_mixed_causal_attn.py @@ -0,0 +1,318 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for per-request causal/non-causal attention (mixed batches). + +Validates that both triton and flash-attention backends correctly handle +batches where some sequences use causal masking and others use non-causal +(bidirectional) masking — needed by DiffusionGemma. +""" + +import pytest +import torch + +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +# Mixed causal/non-causal attention is only validated on a subset of GPUs: +# the Triton path on Hopper (SM90) and B200 (SM100); the FA4 path on Hopper +# (SM90) only. +_device_capability = current_platform.get_device_capability() +_major = _device_capability.major if _device_capability is not None else None + +NUM_HEADS = [(4, 4), (8, 2)] +HEAD_SIZES = [128] +BLOCK_SIZES = [16] +DTYPES = [torch.bfloat16] + + +def ref_paged_attn( + query: torch.Tensor, + key_cache: torch.Tensor, + value_cache: torch.Tensor, + query_lens: list[int], + kv_lens: list[int], + block_tables: torch.Tensor, + scale: float, + per_seq_causal: list[bool], + sliding_window: int | None = None, +) -> torch.Tensor: + num_seqs = len(query_lens) + block_tables_np = block_tables.cpu().numpy() + _, block_size, num_kv_heads, head_size = key_cache.shape + + outputs: list[torch.Tensor] = [] + start_idx = 0 + for i in range(num_seqs): + query_len = query_lens[i] + kv_len = kv_lens[i] + q = query[start_idx : start_idx + query_len] + q = q * scale + + num_kv_blocks = (kv_len + block_size - 1) // block_size + block_indices = block_tables_np[i, :num_kv_blocks] + k = key_cache[block_indices].view(-1, num_kv_heads, head_size)[:kv_len] + v = value_cache[block_indices].view(-1, num_kv_heads, head_size)[:kv_len] + + if q.shape[1] != k.shape[1]: + k = torch.repeat_interleave(k, q.shape[1] // k.shape[1], dim=1) + v = torch.repeat_interleave(v, q.shape[1] // v.shape[1], dim=1) + + attn = torch.einsum("qhd,khd->hqk", q, k).float() + + if per_seq_causal[i]: + mask = torch.triu( + torch.ones(query_len, kv_len, device=attn.device), + diagonal=kv_len - query_len + 1, + ).bool() + else: + mask = torch.zeros(query_len, kv_len, device=attn.device).bool() + + if sliding_window is not None: + sw_mask = ( + torch.triu( + torch.ones(query_len, kv_len, device=attn.device), + diagonal=kv_len - (query_len + sliding_window) + 1, + ) + .bool() + .logical_not() + ) + mask |= sw_mask + + attn.masked_fill_(mask, float("-inf")) + attn = torch.softmax(attn, dim=-1).to(v.dtype) + out = torch.einsum("hqk,khd->qhd", attn, v) + outputs.append(out) + start_idx += query_len + + return torch.cat(outputs, dim=0) + + +# ---- Triton backend test ---- + + +@pytest.mark.skipif( + _major not in (9, 10), + reason="Triton mixed causal attention requires Hopper (SM90) or B200 (SM100).", +) +@pytest.mark.parametrize( + "seq_lens", + [[(1, 128), (5, 64), (1, 256)]], +) +@pytest.mark.parametrize( + "per_seq_causal", + [[True, False, True], [False, True, False], [True, True, False]], +) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@torch.inference_mode() +def test_triton_mixed_causal( + seq_lens: list[tuple[int, int]], + per_seq_causal: list[bool], + num_heads: tuple[int, int], + head_size: int, + dtype: torch.dtype, + block_size: int, +): + if not current_platform.is_cuda(): + pytest.skip("Triton attention requires CUDA") + + from vllm.v1.attention.ops.triton_unified_attention import unified_attention + + set_random_seed(42) + device = "cuda" + + num_query_heads, num_kv_heads = num_heads + assert len(seq_lens) == len(per_seq_causal) + + query_lens = [s[0] for s in seq_lens] + kv_lens = [s[1] for s in seq_lens] + num_seqs = len(seq_lens) + + num_query_tokens = sum(query_lens) + max_kv_len = max(kv_lens) + max_num_blocks = (max_kv_len + block_size - 1) // block_size + num_blocks = max_num_blocks * num_seqs + 10 + + scale = head_size**-0.5 + query = torch.randn( + num_query_tokens, num_query_heads, head_size, dtype=dtype, device=device + ) + key_cache = torch.randn( + num_blocks, block_size, num_kv_heads, head_size, dtype=dtype, device=device + ) + value_cache = torch.randn( + num_blocks, block_size, num_kv_heads, head_size, dtype=dtype, device=device + ) + + block_tables_list = [] + for i in range(num_seqs): + n_blocks = (kv_lens[i] + block_size - 1) // block_size + blocks = list(range(i * max_num_blocks, i * max_num_blocks + n_blocks)) + blocks += [0] * (max_num_blocks - n_blocks) + block_tables_list.append(blocks) + block_tables = torch.tensor(block_tables_list, dtype=torch.int32, device=device) + + cu_seqlens_q = torch.zeros(num_seqs + 1, dtype=torch.int32, device=device) + for i, ql in enumerate(query_lens): + cu_seqlens_q[i + 1] = cu_seqlens_q[i] + ql + + seqused_k = torch.tensor(kv_lens, dtype=torch.int32, device=device) + max_seqlen_q = max(query_lens) + max_seqlen_k = max(kv_lens) + + causal_tensor = torch.tensor(per_seq_causal, dtype=torch.bool, device=device) + + output = torch.empty_like(query) + unified_attention( + q=query, + k=key_cache, + v=value_cache, + out=output, + cu_seqlens_q=cu_seqlens_q, + max_seqlen_q=max_seqlen_q, + seqused_k=seqused_k, + max_seqlen_k=max_seqlen_k, + softmax_scale=scale, + causal=causal_tensor, + window_size=(-1, -1), + block_table=block_tables, + softcap=0.0, + q_descale=None, + k_descale=1.0, + v_descale=1.0, + ) + + ref_output = ref_paged_attn( + query, + key_cache, + value_cache, + query_lens, + kv_lens, + block_tables, + scale, + per_seq_causal, + ) + + torch.testing.assert_close(output, ref_output, atol=1e-2, rtol=1e-2) + + +# ---- Flash Attention 4 backend test (native per_seq_causal) ---- + + +@pytest.mark.skipif( + _major != 9, + reason="FA4 mixed causal attention requires Hopper (SM90).", +) +@pytest.mark.parametrize( + "seq_lens", + [[(1, 128), (5, 64), (1, 256)]], +) +@pytest.mark.parametrize( + "per_seq_causal", + [[True, False, True], [False, True, False]], +) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@torch.inference_mode() +def test_flash_attn4_mixed_causal( + seq_lens: list[tuple[int, int]], + per_seq_causal: list[bool], + num_heads: tuple[int, int], + head_size: int, + dtype: torch.dtype, + block_size: int, +): + if not current_platform.is_cuda(): + pytest.skip("Flash attention requires CUDA") + + try: + from vllm.vllm_flash_attn import ( + fa_version_unsupported_reason, + flash_attn_varlen_func, + is_fa_version_supported, + ) + except ImportError: + pytest.skip("vllm_flash_attn not available") + + if not is_fa_version_supported(4): + reason = fa_version_unsupported_reason(4) + pytest.skip(f"FA4 not supported: {reason}") + + set_random_seed(42) + device = "cuda" + + num_query_heads, num_kv_heads = num_heads + assert len(seq_lens) == len(per_seq_causal) + + query_lens = [s[0] for s in seq_lens] + kv_lens = [s[1] for s in seq_lens] + num_seqs = len(seq_lens) + + num_query_tokens = sum(query_lens) + max_kv_len = max(kv_lens) + max_num_blocks = (max_kv_len + block_size - 1) // block_size + num_blocks = max_num_blocks * num_seqs + 10 + + scale = head_size**-0.5 + query = torch.randn( + num_query_tokens, num_query_heads, head_size, dtype=dtype, device=device + ) + key_cache = torch.randn( + num_blocks, block_size, num_kv_heads, head_size, dtype=dtype, device=device + ) + value_cache = torch.randn( + num_blocks, block_size, num_kv_heads, head_size, dtype=dtype, device=device + ) + + block_tables_list = [] + for i in range(num_seqs): + n_blocks = (kv_lens[i] + block_size - 1) // block_size + blocks = list(range(i * max_num_blocks, i * max_num_blocks + n_blocks)) + blocks += [0] * (max_num_blocks - n_blocks) + block_tables_list.append(blocks) + block_tables = torch.tensor(block_tables_list, dtype=torch.int32, device=device) + + cu_seqlens_q = torch.zeros(num_seqs + 1, dtype=torch.int32, device=device) + for i, ql in enumerate(query_lens): + cu_seqlens_q[i + 1] = cu_seqlens_q[i] + ql + + seqused_k = torch.tensor(kv_lens, dtype=torch.int32, device=device) + per_seq_causal_tensor = torch.tensor( + per_seq_causal, dtype=torch.int32, device=device + ) + + ref_output = ref_paged_attn( + query, + key_cache, + value_cache, + query_lens, + kv_lens, + block_tables, + scale, + per_seq_causal, + ) + + output = torch.empty_like(query) + flash_attn_varlen_func( + q=query, + k=key_cache, + v=value_cache, + out=output, + cu_seqlens_q=cu_seqlens_q, + max_seqlen_q=max(query_lens), + seqused_k=seqused_k, + max_seqlen_k=max(kv_lens), + softmax_scale=scale, + # The kernel must be compiled causal for `dynamic_causal` to take effect. + causal=True, + block_table=block_tables, + softcap=0.0, + dynamic_causal=per_seq_causal_tensor, + fa_version=4, + ) + + torch.testing.assert_close(output, ref_output, atol=1e-2, rtol=1e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_mla_cross_layer_kernel_equivalence.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_mla_cross_layer_kernel_equivalence.py new file mode 100644 index 0000000000000000000000000000000000000000..13166bde34a220aaeb344ddb0560b6abb33ff41c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_mla_cross_layer_kernel_equivalence.py @@ -0,0 +1,566 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Bit-exact kernel equivalence for MLA decode/write kernels on the +cross-layer (block-major) KV cache layout. + +The cross-layer layout carves each layer's per-block page out of a single +unified slot, so the per-layer view has an inflated ``stride(0)`` (the full +unified slot) and a non-zero storage offset. These tests confirm the MLA +kernels behind the backends that opt in to the layout (FlashMLA dense, +FlashInfer MLA dense, FlashMLA fp8 sparse, plus the ``concat_and_cache_mla`` +write) honor that strided view bit-identically to a contiguous per-layer +cache, and that writes do not bleed into neighbouring layers' segments. +""" + +import pytest +import torch + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available(), reason="MLA cache kernels require CUDA" +) + + +def test_concat_and_cache_mla_into_unified_slot_view(): + """concat_and_cache_mla must write correctly into a per-layer view whose + block stride is the full unified slot (block-major), with zero bleed into + the other layers' segments of the same slot.""" + from vllm import _custom_ops as ops + + torch.manual_seed(0) + dev = "cuda" + kv_lora_rank = 512 + pe = 64 + entry = kv_lora_rank + pe + page = 64 + num_blocks = 32 + ntok = 200 + + kv_c = torch.randn(ntok, kv_lora_rank, device=dev, dtype=torch.bfloat16) + k_pe = torch.randn(ntok, pe, device=dev, dtype=torch.bfloat16) + slot = torch.randperm(num_blocks * page, device=dev, dtype=torch.int64)[:ntok] + scale = torch.tensor(1.0, device=dev) + + def write(cache): + ops.concat_and_cache_mla(kv_c, k_pe, cache, slot, "auto", scale) + + # Contiguous per-layer reference: (num_blocks, page, entry). + ref = torch.zeros(num_blocks, page, entry, device=dev, dtype=torch.bfloat16) + write(ref) + + # Unified slot holding three layer pages per block. Carve the middle + # layer's view (non-zero offset, block stride == full unified slot). + layer_page_elems = page * entry + n_layers = 3 + unified_slot_elems = n_layers * layer_page_elems + big = torch.zeros(num_blocks, unified_slot_elems, device=dev, dtype=torch.bfloat16) + flat = big.view(-1) + offset = layer_page_elems # middle layer + view = torch.as_strided( + flat, + size=(num_blocks, page, entry), + stride=(unified_slot_elems, entry, 1), + storage_offset=offset, + ) + assert not view.is_contiguous() + assert view.stride(0) == unified_slot_elems + write(view) + + # Bit-exact equivalence and zero bleed into the neighbour segments. + max_diff = (ref.float() - view.float()).abs().max().item() + assert max_diff == 0.0, f"max|Δ| = {max_diff}" + + neighbour_lo = torch.as_strided( + flat, (num_blocks, layer_page_elems), (unified_slot_elems, 1), 0 + ) + neighbour_hi = torch.as_strided( + flat, + (num_blocks, layer_page_elems), + (unified_slot_elems, 1), + 2 * layer_page_elems, + ) + assert neighbour_lo.abs().max().item() == 0.0 + assert neighbour_hi.abs().max().item() == 0.0 + + +def test_flashmla_dense_decode_unified_slot_view(): + """FlashMLA dense decode (FLASHMLA backend, e.g. Kimi-K2-style dense MLA + on Hopper) must read a unified-slot block-major view bit-identically to a + contiguous per-layer cache.""" + import vllm.v1.attention.ops.flashmla as fm + + ok, reason = fm.is_flashmla_dense_supported() + if not ok: + pytest.skip(reason) + + torch.manual_seed(0) + dev = "cuda" + dt = torch.bfloat16 + head_dim = 576 + hdv = 512 + h_q = 128 + page = 64 + num_blocks = 64 + bs = 4 + n_layers = 3 + layer = 1 + + q = torch.randn(bs, 1, h_q, head_dim, device=dev, dtype=dt) * 0.1 + kv_data = torch.randn(num_blocks, page, 1, head_dim, device=dev, dtype=dt) * 0.1 + + # (A) contiguous per-layer reference. + cache_contiguous = kv_data.clone().contiguous() + + # (B) unified slot: view one layer -> inflated stride(0), non-zero offset. + unified = ( + torch.randn(num_blocks, n_layers, page, 1, head_dim, device=dev, dtype=dt) * 0.1 + ) + unified[:, layer].copy_(kv_data) + cache_view = unified[:, layer] + assert not cache_view.is_contiguous() + assert cache_view.stride(0) == n_layers * page * 1 * head_dim + + max_blk = num_blocks // bs + block_table = torch.arange(num_blocks, device=dev, dtype=torch.int32).view( + bs, max_blk + ) + cache_seqlens = torch.full((bs,), max_blk * page, device=dev, dtype=torch.int32) + + def run(kc): + meta, num_splits = fm.get_mla_metadata() + out, _ = fm.flash_mla_with_kvcache( + q=q, + k_cache=kc, + block_table=block_table, + cache_seqlens=cache_seqlens, + head_dim_v=hdv, + tile_scheduler_metadata=meta, + num_splits=num_splits, + softmax_scale=head_dim**-0.5, + causal=True, + ) + return out.clone().float() + + out_ref = run(cache_contiguous) + out_view = run(cache_view) + assert torch.isfinite(out_ref).all() + assert out_ref.abs().max().item() > 0.0 + assert (out_ref - out_view).abs().max().item() == 0.0 + + +def test_flashinfer_mla_dense_decode_unified_slot_view(): + """FlashInfer MLA dense decode must read a unified-slot block-major view + (inflated stride(0), non-zero storage offset) bit-identically to a + contiguous per-layer cache.""" + try: + from flashinfer.decode import trtllm_batch_decode_with_kv_cache_mla + except ImportError: + pytest.skip("flashinfer is not available") + from vllm.platforms import current_platform + + if not current_platform.is_device_capability_family(100): + pytest.skip("FlashInfer trtllm-gen MLA requires sm100") + + torch.manual_seed(0) + dev = "cuda" + dt = torch.bfloat16 + kv_lora_rank = 512 + qk_rope_head_dim = 64 + qk_nope_head_dim = 128 + head_dim = kv_lora_rank + qk_rope_head_dim # 576 + num_qo_heads = 128 + page = 64 + num_blocks = 64 + bs = 4 + n_layers = 3 # >1 so the per-layer view's block stride is inflated. + layer = 1 + + q = torch.randn(bs, 1, num_qo_heads, head_dim, device=dev, dtype=dt) + kv_data = torch.randn(num_blocks, 1, page, head_dim, device=dev, dtype=dt) + + # (A) contiguous per-layer reference. + kv_contiguous = kv_data.clone().contiguous() + + # (B) unified slot: block b of every layer packed together; view one layer + # -> stride(0) is n_layers x larger and storage offset is non-zero. + unified = torch.randn(num_blocks, n_layers, 1, page, head_dim, device=dev, dtype=dt) + unified[:, layer].copy_(kv_data) + kv_view = unified[:, layer] + assert not kv_view.is_contiguous() + assert kv_view.stride(0) == n_layers * 1 * page * head_dim + + max_blk = num_blocks // bs + block_tables = torch.arange(num_blocks, device=dev, dtype=torch.int32).view( + bs, max_blk + ) + seq_lens = torch.full((bs,), max_blk * page, device=dev, dtype=torch.int32) + ws = torch.empty(128 * 1024 * 1024, dtype=torch.int8, device=dev) + scale = head_dim**-0.5 + + def run(kv): + return trtllm_batch_decode_with_kv_cache_mla( + query=q, + kv_cache=kv, + workspace_buffer=ws, + qk_nope_head_dim=qk_nope_head_dim, + kv_lora_rank=kv_lora_rank, + qk_rope_head_dim=qk_rope_head_dim, + block_tables=block_tables, + seq_lens=seq_lens, + max_seq_len=int(seq_lens.max().item()), + bmm1_scale=scale, + bmm2_scale=1.0, + ).clone() + + out_ref = run(kv_contiguous).float() + out_view = run(kv_view).float() + assert torch.isfinite(out_ref).all() + assert (out_ref - out_view).abs().max().item() == 0.0 + + +def test_flashmla_fp8_sparse_decode_unified_slot_view(): + """FlashMLA fp8 sparse decode (DeepSeek V3.2/V4 DSA path) must read a + unified-slot block-major view bit-identically to a contiguous fp8_ds_mla + cache, with finite nonzero output.""" + import vllm.v1.attention.ops.flashmla as fm + + ok, reason = fm.is_flashmla_sparse_supported() + if not ok: + pytest.skip(reason) + + torch.manual_seed(0) + dev = "cuda" + entry = 656 # fp8_ds_mla bytes per token + page = 64 + num_blocks = 32 + h_q = 128 + head_dim = 576 + hdv = 512 + batch = 2 + topk = 128 + n_layers = 3 + layer = 1 + + q = torch.randn(batch, 1, h_q, head_dim, device=dev, dtype=torch.bfloat16) * 0.1 + + # Structurally valid fp8 ds_mla payload: 512B fp8 + 16B f32 scales + 128B + # bf16 rope (random bytes corrupt the scale region and yield NaNs). + nope = (torch.randn(num_blocks, page, 1, 512, device=dev) * 0.1).to( + torch.float8_e4m3fn + ) + scales = torch.ones(num_blocks, page, 1, 4, device=dev, dtype=torch.float32) + rope = (torch.randn(num_blocks, page, 1, 64, device=dev) * 0.1).to(torch.bfloat16) + payload = torch.cat( + [ + nope.view(torch.uint8).view(num_blocks, page, 1, 512), + scales.view(torch.uint8).view(num_blocks, page, 1, 16), + rope.view(torch.uint8).view(num_blocks, page, 1, 128), + ], + dim=-1, + ).contiguous() + assert payload.shape[-1] == entry and payload.dtype == torch.uint8 + + # (A) contiguous reference. + cache_contiguous = payload.clone().contiguous() + + # (B) unified slot: view one layer -> inflated stride(0), non-zero offset. + unified = torch.randint( + 0, 256, (num_blocks, n_layers, page, 1, entry), device=dev, dtype=torch.uint8 + ) + unified[:, layer].copy_(payload) + cache_view = unified[:, layer] + assert not cache_view.is_contiguous() + assert cache_view.stride(0) == n_layers * page * 1 * entry + + # Sparse indices: each batch uses its own disjoint blocks. + blocks_per_batch = num_blocks // batch + idx = torch.full((batch, 1, topk), -1, device=dev, dtype=torch.int32) + for b in range(batch): + slots: list[int] = [] + for blk in range(b * blocks_per_batch, (b + 1) * blocks_per_batch): + slots.extend(blk * page + off for off in range(page)) + slots_t = torch.tensor(slots[:topk], device=dev, dtype=torch.int32) + idx[b, 0, : slots_t.numel()] = slots_t + + def run(kc): + meta, num_splits = fm.get_mla_metadata() + out, _ = fm.flash_mla_with_kvcache( + q=q, + k_cache=kc, + block_table=None, + cache_seqlens=None, + head_dim_v=hdv, + tile_scheduler_metadata=meta, + is_fp8_kvcache=True, + indices=idx, + softmax_scale=head_dim**-0.5, + ) + return out.clone().float() + + out_ref = run(cache_contiguous) + out_view = run(cache_view) + assert torch.isfinite(out_ref).all() + assert out_ref.abs().max().item() > 0.0 + assert (out_ref - out_view).abs().max().item() == 0.0 + + +def test_indexer_k_quant_and_cache_into_unified_slot_view(): + """indexer_k_quant_and_cache (DeepSeek V3.2/V4 DSA indexer K write) must + write correctly into a per-layer view whose block stride is the full + unified slot, with zero bleed into the other layers' segments.""" + from vllm import _custom_ops as ops + + torch.manual_seed(0) + dev = "cuda" + head_dim = 128 + quant_block_size = 128 + block_size = 64 + num_blocks = 16 + ntok = 100 + # Indexer cache layout per token: head_dim fp8 bytes followed by + # head_dim * 4 / quant_block_size scale bytes. + cache_stride = head_dim + head_dim * 4 // quant_block_size + + k = torch.randn(ntok, head_dim, device=dev, dtype=torch.bfloat16) + slot = torch.randperm(num_blocks * block_size, device=dev, dtype=torch.int64)[:ntok] + + def write(cache): + ops.indexer_k_quant_and_cache(k, cache, slot, quant_block_size, "ue8m0") + + # Contiguous per-layer reference. + ref = torch.zeros( + num_blocks, block_size, cache_stride, device=dev, dtype=torch.uint8 + ) + write(ref) + + # Unified slot holding three layer pages per block; carve the middle one. + n_layers = 3 + layer = 1 + unified = torch.zeros( + num_blocks, n_layers, block_size, cache_stride, device=dev, dtype=torch.uint8 + ) + view = unified[:, layer] + assert not view.is_contiguous() + assert view.stride(0) == n_layers * block_size * cache_stride + write(view) + + assert torch.equal(ref, view.contiguous()) + # Zero bleed into the neighbour layers' segments. + assert unified[:, 0].abs().max().item() == 0 + assert unified[:, 2].abs().max().item() == 0 + + +def test_flashattn_mla_dense_decode_unified_slot_view(): + """FA3 decode (FLASH_ATTN_MLA backend) must read a unified-slot + block-major view bit-identically to a contiguous per-layer cache.""" + try: + from vllm.vllm_flash_attn import flash_attn_varlen_func + except ImportError: + pytest.skip("vllm_flash_attn is not available") + from vllm.v1.attention.backends.fa_utils import flash_attn_supports_mla + + if not flash_attn_supports_mla(): + pytest.skip("FA3 MLA requires a Hopper device") + + torch.manual_seed(0) + dev = "cuda" + dt = torch.bfloat16 + kv_lora_rank = 512 + rope_dim = 64 + entry = kv_lora_rank + rope_dim # 576 + h_q = 16 + page = 64 + num_blocks = 64 + bs = 4 + n_layers = 3 + layer = 1 + + q_pe = torch.randn(bs, h_q, rope_dim, device=dev, dtype=dt) * 0.1 + q_nope = torch.randn(bs, h_q, kv_lora_rank, device=dev, dtype=dt) * 0.1 + kv_data = torch.randn(num_blocks, page, entry, device=dev, dtype=dt) * 0.1 + + # (A) contiguous per-layer reference. + cache_contiguous = kv_data.clone().contiguous() + + # (B) unified slot: view one layer -> inflated stride(0), non-zero offset. + unified = torch.randn(num_blocks, n_layers, page, entry, device=dev, dtype=dt) * 0.1 + unified[:, layer].copy_(kv_data) + cache_view = unified[:, layer] + assert not cache_view.is_contiguous() + assert cache_view.stride(0) == n_layers * page * entry + + max_blk = num_blocks // bs + block_table = torch.arange(num_blocks, device=dev, dtype=torch.int32).view( + bs, max_blk + ) + seq_lens = torch.full((bs,), max_blk * page, device=dev, dtype=torch.int32) + cu_seqlens_q = torch.arange(bs + 1, device=dev, dtype=torch.int32) + + def run(cache): + kv_c_cache = cache[..., :kv_lora_rank] + k_pe_cache = cache[..., kv_lora_rank:] + out = flash_attn_varlen_func( + q=q_pe, + k=k_pe_cache.unsqueeze(-2), # Add head dim of 1 + v=kv_c_cache.unsqueeze(-2), # Add head dim of 1 + q_v=q_nope, + max_seqlen_q=1, + cu_seqlens_q=cu_seqlens_q, + max_seqlen_k=int(seq_lens.max().item()), + seqused_k=seq_lens, + block_table=block_table, + softmax_scale=entry**-0.5, + causal=True, + fa_version=3, + ) + return out.clone().float() + + out_ref = run(cache_contiguous) + out_view = run(cache_view) + assert torch.isfinite(out_ref).all() + assert out_ref.abs().max().item() > 0.0 + assert (out_ref - out_view).abs().max().item() == 0.0 + + +def test_flashmla_dense_fp8_decode_unified_slot_view(): + """FlashMLA dense fp8 decode (FLASHMLA backend with quantized KV cache) + must read a unified-slot block-major view bit-identically to a contiguous + per-layer fp8 cache.""" + import vllm.v1.attention.ops.flashmla as fm + + ok, reason = fm.is_flashmla_dense_supported() + if not ok: + pytest.skip(reason) + + torch.manual_seed(0) + dev = "cuda" + head_dim = 576 + hdv = 512 + h_q = 128 + page = 64 + num_blocks = 64 + bs = 4 + n_layers = 3 + layer = 1 + + q = torch.randn(bs, 1, h_q, head_dim, device=dev, dtype=torch.bfloat16) * 0.1 + kv_data = (torch.randn(num_blocks, page, head_dim, device=dev) * 0.1).to( + torch.float8_e4m3fn + ) + + # (A) contiguous per-layer reference. + cache_contiguous = kv_data.clone().contiguous() + + # (B) unified slot: view one layer -> inflated stride(0), non-zero offset. + unified = (torch.randn(num_blocks, n_layers, page, head_dim, device=dev) * 0.1).to( + torch.float8_e4m3fn + ) + unified[:, layer].copy_(kv_data) + cache_view = unified[:, layer] + assert not cache_view.is_contiguous() + assert cache_view.stride(0) == n_layers * page * head_dim + + max_blk = num_blocks // bs + block_table = torch.arange(num_blocks, device=dev, dtype=torch.int32).view( + bs, max_blk + ) + cache_seqlens = torch.full((bs,), max_blk * page, device=dev, dtype=torch.int32) + descale = torch.ones(1, device=dev, dtype=torch.float32) + + def run(kc): + tile_md, num_splits = fm.get_mla_metadata_dense_fp8(cache_seqlens, h_q, 1) + out, _ = fm.flash_mla_with_kvcache_fp8( + q=q, + k_cache=kc.unsqueeze(-2), # Add head dim of 1 + block_table=block_table, + cache_seqlens=cache_seqlens, + head_dim_v=hdv, + tile_scheduler_metadata=tile_md, + num_splits=num_splits, + softmax_scale=head_dim**-0.5, + causal=True, + descale_q=descale, + descale_k=descale, + ) + return out.clone().float() + + out_ref = run(cache_contiguous) + out_view = run(cache_view) + assert torch.isfinite(out_ref).all() + assert out_ref.abs().max().item() > 0.0 + assert (out_ref - out_view).abs().max().item() == 0.0 + + +def test_flashinfer_mla_dense_fp8_decode_unified_slot_view(): + """FlashInfer MLA dense decode with an fp8 KV cache must read a + unified-slot block-major view bit-identically to a contiguous per-layer + cache.""" + try: + from flashinfer.decode import trtllm_batch_decode_with_kv_cache_mla + except ImportError: + pytest.skip("flashinfer is not available") + from vllm.platforms import current_platform + + if not current_platform.is_device_capability_family(100): + pytest.skip("FlashInfer trtllm-gen MLA requires sm100") + + torch.manual_seed(0) + dev = "cuda" + kv_lora_rank = 512 + qk_rope_head_dim = 64 + qk_nope_head_dim = 128 + head_dim = kv_lora_rank + qk_rope_head_dim # 576 + num_qo_heads = 128 + page = 64 + num_blocks = 64 + bs = 4 + n_layers = 3 + layer = 1 + + # With a quantized KV cache the decode query is quantized to fp8 as well + # (trtllm-gen has no bf16-query x fp8-cache decode kernel). + q = (torch.randn(bs, 1, num_qo_heads, head_dim, device=dev) * 0.1).to( + torch.float8_e4m3fn + ) + kv_data = (torch.randn(num_blocks, 1, page, head_dim, device=dev) * 0.1).to( + torch.float8_e4m3fn + ) + + # (A) contiguous per-layer reference. + kv_contiguous = kv_data.clone().contiguous() + + # (B) unified slot: view one layer -> inflated stride(0), non-zero offset. + unified = ( + torch.randn(num_blocks, n_layers, 1, page, head_dim, device=dev) * 0.1 + ).to(torch.float8_e4m3fn) + unified[:, layer].copy_(kv_data) + kv_view = unified[:, layer] + assert not kv_view.is_contiguous() + assert kv_view.stride(0) == n_layers * 1 * page * head_dim + + max_blk = num_blocks // bs + block_tables = torch.arange(num_blocks, device=dev, dtype=torch.int32).view( + bs, max_blk + ) + seq_lens = torch.full((bs,), max_blk * page, device=dev, dtype=torch.int32) + ws = torch.empty(128 * 1024 * 1024, dtype=torch.int8, device=dev) + scale = head_dim**-0.5 + + def run(kv): + return trtllm_batch_decode_with_kv_cache_mla( + query=q, + kv_cache=kv, + workspace_buffer=ws, + qk_nope_head_dim=qk_nope_head_dim, + kv_lora_rank=kv_lora_rank, + qk_rope_head_dim=qk_rope_head_dim, + block_tables=block_tables, + seq_lens=seq_lens, + max_seq_len=int(seq_lens.max().item()), + bmm1_scale=scale, + bmm2_scale=1.0, + ).clone() + + out_ref = run(kv_contiguous).float() + out_view = run(kv_view).float() + assert torch.isfinite(out_ref).all() + assert (out_ref - out_view).abs().max().item() == 0.0 diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_mla_decode_cpu.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_mla_decode_cpu.py new file mode 100644 index 0000000000000000000000000000000000000000..4d5fca89a808d20df5ec21fdaeab96c428e87590 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_mla_decode_cpu.py @@ -0,0 +1,84 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch +import torch.nn.functional as F +from torch import Tensor + +import vllm._custom_ops as ops +from vllm.platforms import current_platform +from vllm.utils.math_utils import cdiv + + +def ref_mla( + out: Tensor, # (bs, num_heads, v_head_dim) + query: Tensor, # (bs, num_heads, head_dim) + kv_cache: Tensor, # (num_blocks, block_size, head_dim) + scale: float, + block_tables: Tensor, # (bs, max_num_blocks) + seq_lens: Tensor, # (bs,) +): + bs, num_heads, v_head_dim = out.shape + head_dim = query.shape[2] + + for i in range(bs): + # gather and flatten KV-cache + kv = kv_cache[block_tables[i]] # (max_num_blocks, block_size, head_dim) + kv = kv.view(1, -1, head_dim)[:, : seq_lens[i]] # (1, seq_len, head_dim) + v = kv[:, :, :v_head_dim] + + q = query[i].view(num_heads, 1, head_dim) + o = F.scaled_dot_product_attention(q, kv, v, scale=scale, enable_gqa=True) + out[i] = o.view(num_heads, v_head_dim) + + return out + + +@pytest.mark.parametrize("bs", [4]) +@pytest.mark.parametrize("mean_seq_len", [256]) +@pytest.mark.parametrize("h_q", [16]) +@pytest.mark.parametrize("d", [576]) +@pytest.mark.parametrize("dv", [512]) +@pytest.mark.parametrize("block_size", [16]) +@pytest.mark.parametrize("dtype", [torch.float, torch.half, torch.bfloat16]) +@pytest.mark.parametrize("varlen", [False, True]) +@pytest.mark.cpu_model +@pytest.mark.skipif(not current_platform.is_cpu(), reason="CPU only") +def test_mla_decode_cpu( + bs: int, + mean_seq_len: int, + h_q: int, + d: int, + dv: int, + block_size: int, + dtype: torch.dtype, + varlen: bool, +): + torch.set_default_dtype(dtype) + torch.manual_seed(0) + + scale = d ** (-0.5) + if varlen: + seq_lens = torch.empty(bs).normal_(mean_seq_len, mean_seq_len / 2) + seq_lens = seq_lens.clip(2).to(torch.int32) + else: + seq_lens = torch.full((bs,), mean_seq_len, dtype=torch.int32) + max_seq_len = seq_lens.max().item() + seqlen_pad = cdiv(max_seq_len, 256) * 256 # is this necessary? + + q = torch.randn(bs, h_q, d) + block_table = torch.arange(bs * seqlen_pad // block_size, dtype=torch.int32) + block_table = block_table.view(bs, seqlen_pad // block_size) + + kv_cache = torch.randn(block_table.numel(), block_size, d) + for i, seq_len in enumerate(seq_lens.tolist()): + kv_cache.view(bs, seqlen_pad, d)[i, seq_len:] = float("nan") + + out_mla = q.new_zeros(bs, h_q, dv) + ops.mla_decode_kvcache_cpu(out_mla, q, kv_cache, scale, block_table, seq_lens) + + out_ref = q.new_zeros(bs, h_q, dv) + ref_mla(out_ref, q, kv_cache, scale, block_table, seq_lens) + + assert not out_mla.isnan().any(), "Likely read out of bounds" + torch.testing.assert_close(out_mla, out_ref) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_pack_unpack_triton.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_pack_unpack_triton.py new file mode 100644 index 0000000000000000000000000000000000000000..037dd93fc9ff6469701ba002178711c2d89d1310 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_pack_unpack_triton.py @@ -0,0 +1,234 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import torch +from torch.testing import assert_close + +from vllm.v1.attention.ops.common import pack_seq_triton, unpack_seq_triton + + +def test_pack_seq_basic_fp8(): + """Test basic functionality of pack_seq_triton with fp8 and 3D tensors.""" + device = "cuda" + dtype = torch.float8_e4m3fn + + # Test cases with 3D tensors (N, H, D) + test_cases = [ + (6, 8, 4, 2, [3, 3]), # (6, 8, 4) -> (2, 3, 8, 4) + (10, 4, 8, 3, [2, 4, 4]), # (10, 4, 8) -> (3, 4, 4, 8) + (20, 16, 32, 4, [5, 5, 5, 5]), # (20, 16, 32) -> (4, 5, 16, 32) + ] + + for N, H, D, B, lengths_list in test_cases: + # Create input tensor with small values for fp8 + x = torch.randn(N, H, D, dtype=torch.float32, device=device) * 0.1 + x = x.to(dtype=dtype) + lengths = torch.tensor(lengths_list, device=device) + + # Pack the data + packed = pack_seq_triton(x, lengths) + + # Check output shape and properties + expected_shape = (B, max(lengths_list), H, D) + assert packed.shape == expected_shape + assert packed.dtype == dtype + assert packed.device == x.device + + # Check that valid data is preserved (within fp8 precision) + for b in range(B): + start_idx = sum(lengths_list[:b]) + seq_len = lengths_list[b] + + expected_data = x[start_idx : start_idx + seq_len].to(torch.float32) + actual_data = packed[b, :seq_len].to(torch.float32) + + assert_close(actual_data, expected_data, rtol=1e-1, atol=1e-2) + + +def test_pack_seq_custom_padding_fp8(): + """Test pack_seq_triton with custom padding values for fp8.""" + device = "cuda" + dtype = torch.float8_e4m3fn + N, H, D, B = 20, 8, 16, 2 + lengths = torch.tensor([10, 10], device=device) + + x = torch.randn(N, H, D, dtype=torch.float32, device=device) * 0.1 + x = x.to(dtype=dtype) + + # Test with different padding values + for pad_value in [-100.0, -10.0, 0.0, 10.0, 100.0]: + result = pack_seq_triton(x, lengths, pad_value=pad_value) + + # Check valid data + for b in range(B): + start_idx = b * 10 + expected_data = x[start_idx : start_idx + 10].to(torch.float32) + actual_data = result[b, :10].to(torch.float32) + assert_close(actual_data, expected_data, rtol=1e-1, atol=1e-2) + + # Check padding (fp8 has limited range, so check for large values) + padded_data = result[:, 10:].to(torch.float32) + if pad_value < 0: + assert torch.all(padded_data < -50) # Large negative values + elif pad_value > 0: + assert torch.all(padded_data > 50) # Large positive values + else: + assert torch.allclose(padded_data, torch.zeros_like(padded_data), atol=1e-2) + + +def test_pack_seq_default_negative_inf_padding_fp8(): + """Test that pack_seq_triton uses -inf padding by default for fp8.""" + device = "cuda" + dtype = torch.float8_e4m3fn + # B = 2 + N, H, D = 20, 8, 16 + lengths = torch.tensor([10, 10], device=device) + + x = torch.randn(N, H, D, dtype=torch.float32, device=device) * 0.1 + x = x.to(dtype=dtype) + result = pack_seq_triton(x, lengths) + + # Check that padding is large negative values (fp8 representation of -inf) + padded_data = result[:, 10:].to(torch.float32) + assert torch.all( + padded_data < -100 + ) # fp8 -inf is represented as large negative number + + +def test_pack_seq_edge_cases_fp8(): + """Test pack_seq_triton with edge cases for fp8.""" + device = "cuda" + dtype = torch.float8_e4m3fn + + # Test with single batch element + x = torch.randn(10, 8, 16, dtype=torch.float32, device=device) * 0.1 + x = x.to(dtype=dtype) + lengths = torch.tensor([10], device=device) + result = pack_seq_triton(x, lengths) + assert result.shape == (1, 10, 8, 16) + + # Test with very short sequences + x = torch.randn(20, 4, 8, dtype=torch.float32, device=device) * 0.1 + x = x.to(dtype=dtype) + lengths = torch.tensor([1, 1, 1], device=device) + result = pack_seq_triton(x, lengths) + assert result.shape == (3, 1, 4, 8) + + # Test with different sequence lengths + x = torch.randn(15, 8, 16, dtype=torch.float32, device=device) * 0.1 + x = x.to(dtype=dtype) + lengths = torch.tensor([5, 7, 3], device=device) + result = pack_seq_triton(x, lengths) + assert result.shape == (3, 7, 8, 16) + + +def test_pack_seq_different_block_sizes_fp8(): + """Test pack_seq_triton with different block sizes for fp8.""" + device = "cuda" + dtype = torch.float8_e4m3fn + N, H, D, B = 100, 16, 32, 4 + lengths = torch.tensor([25, 25, 25, 25], device=device) + + x = torch.randn(N, H, D, dtype=torch.float32, device=device) * 0.1 + x = x.to(dtype=dtype) + + # Test different block sizes + for block_t, block_d in [(32, 32), (64, 64), (128, 128)]: + result = pack_seq_triton(x, lengths, block_t=block_t, block_d=block_d) + + assert result.shape == (B, 25, H, D) + + # Check that valid data is preserved (within fp8 precision) + for b in range(B): + start_idx = b * 25 + expected_data = x[start_idx : start_idx + 25].to(torch.float32) + actual_data = result[b, :25].to(torch.float32) + assert_close(actual_data, expected_data, rtol=1e-1, atol=1e-2) + + +def test_pack_seq_shape_consistency(): + """Test that pack_seq_triton maintains shape consistency.""" + device = "cuda" + dtype = torch.float8_e4m3fn + N, H, D, B = 20, 8, 16, 2 + lengths = torch.tensor([10, 10], device=device) + + x = torch.randn(N, H, D, dtype=torch.float32, device=device) * 0.1 + x = x.to(dtype=dtype) + + result = pack_seq_triton(x, lengths) + + # Check shape consistency + assert result.shape[0] == B # Batch dimension + assert result.shape[1] == lengths.max().item() # Max sequence length + assert result.shape[2:] == x.shape[1:] # Feature dimensions preserved + + +def test_pack_unpack_roundtrip_fp8(): + """Test that pack -> unpack gives us back the original data for fp8.""" + device = "cuda" + dtype = torch.float8_e4m3fn + + # Test cases with 3D tensors + test_cases = [ + (6, 8, 4, 2, [3, 3]), + (10, 4, 8, 3, [2, 4, 4]), + (20, 16, 32, 4, [5, 5, 5, 5]), + (15, 8, 16, 3, [7, 5, 3]), + ] + + for N, H, D, B, lengths_list in test_cases: + # Create input tensor with small values for fp8 + x = torch.randn(N, H, D, dtype=torch.float32, device=device) * 0.1 + x = x.to(dtype=dtype) + lengths = torch.tensor(lengths_list, device=device) + + # Pack the data + packed = pack_seq_triton(x, lengths) + + # Unpack the data + unpacked = unpack_seq_triton(packed, lengths) + + # Check that we get back the original data (within fp8 precision) + assert unpacked.shape == x.shape + x_f32 = x.to(torch.float32) + unpacked_f32 = unpacked.to(torch.float32) + assert_close(x_f32, unpacked_f32, rtol=1e-3, atol=1e-3) + + # Unpack without explicit start locations (computed in kernel) + unpacked_with_loc = unpack_seq_triton(packed, lengths) + assert_close(x_f32, unpacked_with_loc.to(torch.float32), rtol=1e-3, atol=1e-2) + + +def test_unpack_seq_triton_edge_cases_fp8(): + """Test unpack function with edge cases for fp8.""" + device = "cuda" + dtype = torch.float8_e4m3fn + + # Test with single batch element + x = torch.randn(10, 8, 16, dtype=torch.float32, device=device) * 0.1 + x = x.to(dtype=dtype) + lengths = torch.tensor([10], device=device) + packed = pack_seq_triton(x, lengths) + unpacked = unpack_seq_triton(packed, lengths) + assert unpacked.shape == x.shape + assert_close(x.to(torch.float32), unpacked.to(torch.float32), rtol=1e-1, atol=1e-2) + + # Test with very short sequences + x = torch.randn(20, 4, 8, dtype=torch.float32, device=device) * 0.1 + x = x.to(dtype=dtype) + lengths = torch.tensor([1, 1, 1], device=device) + packed = pack_seq_triton(x, lengths) + unpacked = unpack_seq_triton(packed, lengths) + # Only compare the first 3 elements that were actually packed + assert_close( + x[:3].to(torch.float32), unpacked.to(torch.float32), rtol=1e-1, atol=1e-2 + ) + + x = torch.randn(15, 8, 16, dtype=torch.float32, device=device) * 0.1 + x = x.to(dtype=dtype) + lengths = torch.tensor([5, 7, 3], device=device) + packed = pack_seq_triton(x, lengths) + unpacked = unpack_seq_triton(packed, lengths) + assert unpacked.shape == x.shape + assert_close(x.to(torch.float32), unpacked.to(torch.float32), rtol=1e-1, atol=1e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_prefix_prefill.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_prefix_prefill.py new file mode 100644 index 0000000000000000000000000000000000000000..f8fbcc2695fbe61ec5b1b1653f79320b2de28dd3 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_prefix_prefill.py @@ -0,0 +1,682 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import math +import random +import time +from collections.abc import Callable +from contextlib import nullcontext + +import pytest +import torch +import torch.nn.functional as F +from torch.nn.attention import SDPBackend, sdpa_kernel + +from vllm.platforms import current_platform +from vllm.utils.torch_utils import STR_DTYPE_TO_TORCH_DTYPE, set_random_seed +from vllm.v1.attention.ops.chunked_prefill_paged_decode import ( + chunked_prefill_paged_decode, +) +from vllm.v1.attention.ops.prefix_prefill import context_attention_fwd + +NUM_HEADS = [64] +NUM_QUERIES_PER_KV = [1, 64] +HEAD_SIZES = [24, 128] +DTYPES = [torch.float16] +CUDA_DEVICES = [ + f"cuda:{i}" for i in range(1 if torch.accelerator.device_count() == 1 else 2) +] +SLIDING_WINDOW = [0, 16, 2048] +KV_CACHE_DTYPES = ["auto", "fp8", "fp8_e5m2"] + +OPS = [chunked_prefill_paged_decode, context_attention_fwd] + + +def create_causal_attention_mask_for_sdpa( + query_lens: list[int], + seq_lens: list[int], + sliding_window: int = 0, + device: torch.device = None, + dtype: torch.dtype = None, +) -> torch.Tensor: + total_queries = sum(query_lens) + total_keys = sum(seq_lens) + + # Create a mask filled with -inf + mask = torch.full( + (total_queries, total_keys), float("-inf"), device=device, dtype=dtype + ) + + query_start = 0 + key_start = 0 + + for query_len, seq_len in zip(query_lens, seq_lens): + query_end = query_start + query_len + key_end = key_start + seq_len + q_indices = torch.arange(query_len, device=device) + k_indices = torch.arange(seq_len, device=device) + q_pos_in_seq = seq_len - query_len + q_indices + + valid_mask = k_indices[None, :] <= q_pos_in_seq[:, None] + + if sliding_window > 0: + valid_mask &= k_indices[None, :] >= ( + q_pos_in_seq[:, None] - sliding_window + 1 + ) + + mask[query_start:query_end, key_start:key_end][valid_mask] = 0.0 + + query_start = query_end + key_start = key_end + + return mask + + +def create_alibi_causal_mask( + query_len: int, + seq_len: int, + alibi_slopes: torch.Tensor, + device: torch.device, + dtype: torch.dtype, +) -> torch.Tensor: + query_pos = torch.arange( + seq_len - query_len, seq_len, device=device, dtype=torch.float32 + ) + key_pos = torch.arange(seq_len, device=device, dtype=torch.float32) + + rel_pos = key_pos[None, :] - query_pos[:, None] + + # Apply ALiBi slopes: [num_heads, query_len, seq_len] + alibi_bias = alibi_slopes[:, None, None] * rel_pos[None, :, :] + alibi_bias = alibi_bias.to(dtype) + + # Apply causal mask: prevent attending to future positions + # causal_mask[i, j] = True if key_pos[j] <= query_pos[i] + causal_mask = key_pos[None, :] <= query_pos[:, None] + alibi_bias = alibi_bias.masked_fill(~causal_mask[None, :, :], float("-inf")) + + # Add batch dimension: [1, num_heads, query_len, seq_len] + # SDPA expects batch dimension even for single sequences + return alibi_bias.unsqueeze(0) + + +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("num_queries_per_kv", NUM_QUERIES_PER_KV) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("kv_cache_dtype", KV_CACHE_DTYPES) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOW) +@pytest.mark.parametrize("op", OPS) +@torch.inference_mode() +def test_contexted_kv_attention( + num_heads: int, + num_queries_per_kv: int, + head_size: int, + sliding_window: int, + dtype: torch.dtype, + kv_cache_dtype: str, + device: str, + op: Callable, + block_size: int = 32, +) -> None: + if "fp8" in kv_cache_dtype and not current_platform.has_device_capability(89): + pytest.skip( + "Triton limitation: fp8e4nv data type is not supported on CUDA arch < 89" + ) + + if ( + current_platform.is_rocm() + and op is chunked_prefill_paged_decode + and kv_cache_dtype == "fp8_e5m2" + ): + pytest.skip("ROCm custom paged attention does not support fp8_e5m2 KV cache") + + set_random_seed(0) + torch.set_default_device(device) + + # Need this, otherwise when we capture the graph the process + # for GPU 1 would run on both GPU0 and GPU1 and things would hang + # + # see also similar issue: https://github.com/Dao-AILab/flash-attention/issues/523 + torch.accelerator.set_device_index(device) + + MAX_SEQ_LEN = 1024 + MAX_CTX_LEN = 1024 + BS = 10 + cache_size = 640 + max_block_per_request = 64 + query_lens = [random.randint(16, MAX_SEQ_LEN) for _ in range(BS)] + # ensure one sequence in batch is a decode + query_lens[-1] = 1 + + ctx_lens = [random.randint(16, MAX_CTX_LEN) for _ in range(BS)] + seq_lens = [a + b for a, b in zip(query_lens, ctx_lens)] + num_kv_heads = num_heads // num_queries_per_kv + + num_tokens = sum(query_lens) + query = torch.empty(num_tokens, num_heads, head_size, dtype=dtype) + query.uniform_(-1e-3, 1e-3) + output = torch.empty(num_tokens, num_heads, head_size, dtype=dtype) + + kv = torch.empty(sum(seq_lens), 2, num_kv_heads, head_size, dtype=dtype) + kv.uniform_(-1e-3, 1e-3) + key, value = kv.unbind(dim=1) + + if kv_cache_dtype == "auto": + cache_dtype = dtype + else: + cache_dtype = STR_DTYPE_TO_TORCH_DTYPE[kv_cache_dtype] + k_cache = torch.zeros( + cache_size, block_size, num_kv_heads, head_size, dtype=cache_dtype + ) + v_cache = torch.zeros( + cache_size, block_size, num_kv_heads, head_size, dtype=cache_dtype + ) + k = torch.zeros(sum(query_lens), num_kv_heads, head_size, dtype=dtype) + v = torch.zeros(sum(query_lens), num_kv_heads, head_size, dtype=dtype) + values = torch.arange(0, cache_size, dtype=torch.int32) + values = values[torch.randperm(cache_size)] + block_table = values[: BS * max_block_per_request].view(BS, max_block_per_request) + b_seq_len = torch.tensor(seq_lens, dtype=torch.int32) + b_ctx_len = torch.tensor(ctx_lens, dtype=torch.int32) + b_start_loc = torch.cumsum(torch.tensor([0] + query_lens), dim=0).to(torch.int32) + max_input_len = MAX_SEQ_LEN + # copy kv to cache + b_seq_start_loc = torch.cumsum(torch.tensor([0] + seq_lens[:-1]), dim=0).to( + torch.int32 + ) + for i in range(BS): + for j in range(query_lens[i]): + k[b_start_loc[i] + j].copy_(key[b_seq_start_loc[i] + b_ctx_len[i] + j]) + v[b_start_loc[i] + j].copy_(value[b_seq_start_loc[i] + b_ctx_len[i] + j]) + cur_ctx = 0 + block_id = 0 + while cur_ctx < b_ctx_len[i]: + start_loc = b_seq_start_loc[i] + cur_ctx + if cur_ctx + block_size > b_ctx_len[i]: + end_loc = b_seq_start_loc[i] + b_ctx_len[i] + else: + end_loc = start_loc + block_size + start_slot = block_table[i, block_id] * block_size + end_slot = start_slot + end_loc - start_loc + k_cache.view(-1, num_kv_heads, head_size)[start_slot:end_slot].copy_( + key[start_loc:end_loc] + ) + v_cache.view(-1, num_kv_heads, head_size)[start_slot:end_slot].copy_( + value[start_loc:end_loc] + ) + cur_ctx += block_size + block_id += 1 + # transpose K_cache[num_blocks, block_size, num_kv_heads, head_size] + # to K_cache[num_blocks, num_kv_heads, head_size/8, block_size, 8] + k_cache = ( + k_cache.view(-1, block_size, num_kv_heads, head_size // 8, 8) + .permute(0, 2, 3, 1, 4) + .contiguous() + ) + # transpose V_cache[num_blocks, block_size, num_kv_heads, head_size] + # to V_cache[num_blocks, num_kv_heads, head_size, block_size] + v_cache = ( + v_cache.view(-1, block_size, num_kv_heads, head_size) + .permute(0, 2, 3, 1) + .contiguous() + ) + k_scale = v_scale = torch.tensor(1.0, dtype=torch.float32, device=device) + + # Warm up the Triton kernel by calling it once before actually measuring + # generation time + op( + query, + k, + v, + output, + kv_cache_dtype, + k_cache, + v_cache, + block_table, + b_start_loc, + b_seq_len, + MAX_CTX_LEN, + max_input_len, + k_scale, + v_scale, + sliding_window=sliding_window, + ) + torch.accelerator.synchronize() + start_time = time.time() + op( + query, + k, + v, + output, + kv_cache_dtype, + k_cache, + v_cache, + block_table, + b_start_loc, + b_seq_len, + MAX_CTX_LEN, + max_input_len, + k_scale, + v_scale, + sliding_window=sliding_window, + ) + torch.accelerator.synchronize() + end_time = time.time() + print(f"triton Time: {(end_time - start_time) * 1000:.2f} ms") + + scale = float(1.0 / (head_size**0.5)) + + # Reshape for SDPA: (seq_len, num_heads, head_size) -> + # (1, num_heads, seq_len, head_size) + query_sdpa = query.view(num_tokens, num_kv_heads, num_queries_per_kv, head_size) + query_sdpa = query_sdpa.permute(1, 2, 0, 3).reshape( + 1, num_heads, num_tokens, head_size + ) + + # Expand key and value for GQA/MQA to match query heads + key_sdpa = key[:, :, None, :].expand( + key.shape[0], num_kv_heads, num_queries_per_kv, key.shape[-1] + ) + key_sdpa = key_sdpa.permute(1, 2, 0, 3).reshape( + 1, num_heads, sum(seq_lens), head_size + ) + + value_sdpa = value[:, :, None, :].expand( + value.shape[0], num_kv_heads, num_queries_per_kv, value.shape[-1] + ) + value_sdpa = value_sdpa.permute(1, 2, 0, 3).reshape( + 1, num_heads, sum(seq_lens), head_size + ) + + attn_mask = create_causal_attention_mask_for_sdpa( + query_lens, seq_lens, sliding_window, device=device, dtype=dtype + ) + + output_ref = F.scaled_dot_product_attention( + query_sdpa, + key_sdpa, + value_sdpa, + attn_mask=attn_mask, + dropout_p=0.0, + scale=scale, + ) + torch.accelerator.synchronize() + start_time = time.time() + output_ref = F.scaled_dot_product_attention( + query_sdpa, + key_sdpa, + value_sdpa, + attn_mask=attn_mask, + dropout_p=0.0, + scale=scale, + ) + torch.accelerator.synchronize() + end_time = time.time() + print(f"PyTorch SDPA Time: {(end_time - start_time) * 1000:.2f} ms") + + # Reshape output back to (num_tokens, num_heads, head_size) + output_ref = output_ref.view(num_heads, num_tokens, head_size) + output_ref = output_ref.permute(1, 0, 2).contiguous() + atol = 1e-3 if "fp8" in kv_cache_dtype else 1e-4 + torch.testing.assert_close(output, output_ref, atol=atol, rtol=0) + + +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("num_queries_per_kv", NUM_QUERIES_PER_KV) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("kv_cache_dtype", KV_CACHE_DTYPES) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("op", OPS) +@torch.inference_mode() +def test_contexted_kv_attention_alibi( + num_heads: int, + num_queries_per_kv: int, + head_size: int, + dtype: torch.dtype, + kv_cache_dtype: str, + device: str, + op: Callable, + block_size: int = 32, +) -> None: + if "fp8" in kv_cache_dtype and not current_platform.has_device_capability(89): + pytest.skip( + "Triton limitation: fp8e4nv data type is not supported on CUDA arch < 89" + ) + + if ( + current_platform.is_rocm() + and op is chunked_prefill_paged_decode + and kv_cache_dtype == "fp8_e5m2" + ): + pytest.skip("ROCm custom paged attention does not support fp8_e5m2 KV cache") + + set_random_seed(0) + torch.set_default_device(device) + + # Need this, otherwise when we capture the graph the process + # for GPU 1 would run on both GPU0 and GPU1 and things would hang + # + # see also similar issue: https://github.com/Dao-AILab/flash-attention/issues/523 + torch.accelerator.set_device_index(device) + + def _get_alibi_slopes(total_num_heads: int) -> torch.Tensor: + # Fork from: vllm/vllm/model_executor/models/bloom.py#L44 + closest_power_of_2 = 2 ** math.floor(math.log2(total_num_heads)) + base = torch.tensor( + 2 ** (-(2 ** -(math.log2(closest_power_of_2) - 3))), + dtype=torch.float32, + ) + powers = torch.arange(1, 1 + closest_power_of_2, dtype=torch.int32) + slopes = torch.pow(base, powers) + + if closest_power_of_2 != total_num_heads: + extra_base = torch.tensor( + 2 ** (-(2 ** -(math.log2(2 * closest_power_of_2) - 3))), + dtype=torch.float32, + ) + num_remaining_heads = min( + closest_power_of_2, total_num_heads - closest_power_of_2 + ) + extra_powers = torch.arange( + start=1, end=1 + 2 * num_remaining_heads, step=2, dtype=torch.int32 + ) + slopes = torch.cat([slopes, torch.pow(extra_base, extra_powers)], dim=0) + return slopes + + alibi_slopes = _get_alibi_slopes(num_heads).to(device) + + MAX_SEQ_LEN = 1024 + MAX_CTX_LEN = 1024 + BS = 10 + cache_size = 640 + max_block_per_request = 64 + query_lens = [random.randint(16, MAX_SEQ_LEN) for _ in range(BS)] + ctx_lens = [random.randint(16, MAX_CTX_LEN) for _ in range(BS)] + seq_lens = [a + b for a, b in zip(query_lens, ctx_lens)] + num_kv_heads = num_heads // num_queries_per_kv + + num_tokens = sum(query_lens) + query = torch.empty(num_tokens, num_heads, head_size, dtype=dtype) + query.uniform_(-1e-3, 1e-3) + output = torch.empty(num_tokens, num_heads, head_size, dtype=dtype) + + kv = torch.empty(sum(seq_lens), 2, num_kv_heads, head_size, dtype=dtype) + kv.uniform_(-1e-3, 1e-3) + key, value = kv.unbind(dim=1) + if kv_cache_dtype == "auto": + cache_dtype = dtype + else: + cache_dtype = STR_DTYPE_TO_TORCH_DTYPE[kv_cache_dtype] + k_cache = torch.zeros( + cache_size, block_size, num_kv_heads, head_size, dtype=cache_dtype + ) + v_cache = torch.zeros( + cache_size, block_size, num_kv_heads, head_size, dtype=cache_dtype + ) + k = torch.zeros(sum(query_lens), num_kv_heads, head_size, dtype=dtype) + v = torch.zeros(sum(query_lens), num_kv_heads, head_size, dtype=dtype) + values = torch.arange(0, cache_size, dtype=torch.int32) + values = values[torch.randperm(cache_size)] + block_table = values[: BS * max_block_per_request].view(BS, max_block_per_request) + b_seq_len = torch.tensor(seq_lens, dtype=torch.int32) + b_ctx_len = torch.tensor(ctx_lens, dtype=torch.int32) + b_start_loc = torch.cumsum(torch.tensor([0] + query_lens), dim=0).to(torch.int32) + max_input_len = MAX_SEQ_LEN + # copy kv to cache + b_seq_start_loc = torch.cumsum(torch.tensor([0] + seq_lens[:-1]), dim=0).to( + torch.int32 + ) + for i in range(BS): + for j in range(query_lens[i]): + k[b_start_loc[i] + j].copy_(key[b_seq_start_loc[i] + b_ctx_len[i] + j]) + v[b_start_loc[i] + j].copy_(value[b_seq_start_loc[i] + b_ctx_len[i] + j]) + cur_ctx = 0 + block_id = 0 + while cur_ctx < b_ctx_len[i]: + start_loc = b_seq_start_loc[i] + cur_ctx + if cur_ctx + block_size > b_ctx_len[i]: + end_loc = b_seq_start_loc[i] + b_ctx_len[i] + else: + end_loc = start_loc + block_size + start_slot = block_table[i, block_id] * block_size + end_slot = start_slot + end_loc - start_loc + k_cache.view(-1, num_kv_heads, head_size)[start_slot:end_slot].copy_( + key[start_loc:end_loc] + ) + v_cache.view(-1, num_kv_heads, head_size)[start_slot:end_slot].copy_( + value[start_loc:end_loc] + ) + cur_ctx += block_size + block_id += 1 + # transpose K_cache[num_blocks, block_size, num_kv_heads, head_size] + # to K_cache[num_blocks, num_kv_heads, head_size/8, block_size, 8] + k_cache = ( + k_cache.view(-1, block_size, num_kv_heads, head_size // 8, 8) + .permute(0, 2, 3, 1, 4) + .contiguous() + ) + # transpose V_cache[num_blocks, block_size, num_kv_heads, head_size] + # to V_cache[num_blocks, num_kv_heads, head_size, block_size] + v_cache = ( + v_cache.view(-1, block_size, num_kv_heads, head_size) + .permute(0, 2, 3, 1) + .contiguous() + ) + k_scale = v_scale = torch.tensor(1.0, dtype=torch.float32, device=device) + + # Warm up the Triton kernel by calling it once before actually measuring + # generation time + op( + query, + k, + v, + output, + kv_cache_dtype, + k_cache, + v_cache, + block_table, + b_start_loc, + b_seq_len, + MAX_CTX_LEN, + max_input_len, + k_scale, + v_scale, + alibi_slopes=alibi_slopes, + ) + torch.accelerator.synchronize() + start_time = time.time() + op( + query, + k, + v, + output, + kv_cache_dtype, + k_cache, + v_cache, + block_table, + b_start_loc, + b_seq_len, + MAX_CTX_LEN, + max_input_len, + k_scale, + v_scale, + alibi_slopes=alibi_slopes, + ) + torch.accelerator.synchronize() + end_time = time.time() + print(f"triton Time: {(end_time - start_time) * 1000:.2f} ms") + scale = float(1.0 / (head_size**0.5)) + + # Prepare query, key, value for SDPA + # Expand key and value for GQA/MQA to match query heads + key_expanded = key[:, :, None, :].expand( + key.shape[0], num_kv_heads, num_queries_per_kv, key.shape[-1] + ) + value_expanded = value[:, :, None, :].expand( + value.shape[0], num_kv_heads, num_queries_per_kv, value.shape[-1] + ) + + output_ref = torch.empty_like(output) + + torch.accelerator.synchronize() + start_time = time.time() + + query_start = 0 + key_start = 0 + for i, (query_len, seq_len) in enumerate(zip(query_lens, seq_lens)): + query_end = query_start + query_len + key_end = key_start + seq_len + + # Get query, key, value for this sequence + q = query[query_start:query_end] # [query_len, num_heads, head_size] + k = key_expanded[ + key_start:key_end + ] # [seq_len, num_kv_heads, num_queries_per_kv, head_size] + v = value_expanded[ + key_start:key_end + ] # [seq_len, num_kv_heads, num_queries_per_kv, head_size] + + # Reshape for SDPA: (batch=1, num_heads, seq_len, head_size) + q_sdpa = q.view(query_len, num_kv_heads, num_queries_per_kv, head_size) + q_sdpa = ( + q_sdpa.permute(1, 2, 0, 3) + .reshape(1, num_heads, query_len, head_size) + .contiguous() + ) + + k_sdpa = ( + k.permute(1, 2, 0, 3).reshape(1, num_heads, seq_len, head_size).contiguous() + ) + v_sdpa = ( + v.permute(1, 2, 0, 3).reshape(1, num_heads, seq_len, head_size).contiguous() + ) + + # Create ALiBi causal mask for this sequence using utility function + alibi_mask = create_alibi_causal_mask( + query_len, seq_len, alibi_slopes, device, dtype + ) + + # Compute attention. On ROCm we force use of the Math SDPA backend rather than + # the Flash or Mem-Efficient backends for increased numerical accuracy + if current_platform.is_rocm(): + sdpa_context = sdpa_kernel(SDPBackend.MATH) + else: + sdpa_context = nullcontext() + with sdpa_context: + out = F.scaled_dot_product_attention( + q_sdpa, + k_sdpa, + v_sdpa, + attn_mask=alibi_mask, + dropout_p=0.0, + scale=scale, + ) + + # Reshape output back to [query_len, num_heads, head_size] + out = out.view(num_heads, query_len, head_size).permute(1, 0, 2) + output_ref[query_start:query_end].copy_(out) + + query_start = query_end + key_start = key_end + + torch.accelerator.synchronize() + end_time = time.time() + print(f"PyTorch SDPA Time: {(end_time - start_time) * 1000:.2f} ms") + atol = 1e-3 if "fp8" in kv_cache_dtype else 1e-6 + torch.testing.assert_close(output, output_ref, atol=atol, rtol=0) + + +# These tests are optional to only run when explicitly invoked +# +# pytest -v -s --optional \ +# tests/kernels/test_prefix_prefill.py::test_contexted_kv_attention_f32 +# +# These tests are useful to test model dtype float32 on Turing devices. +# We skip them to not increase the time when running tests on CI +@pytest.mark.optional +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("num_queries_per_kv", NUM_QUERIES_PER_KV) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("dtype", [torch.float32]) +@pytest.mark.parametrize("kv_cache_dtype", KV_CACHE_DTYPES) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("sliding_window", SLIDING_WINDOW) +@pytest.mark.parametrize("op", OPS) +@torch.inference_mode() +def test_contexted_kv_attention_f32( + num_heads: int, + num_queries_per_kv: int, + head_size: int, + sliding_window: int, + dtype: torch.dtype, + kv_cache_dtype: str, + device: str, + op: Callable, +) -> None: + test_contexted_kv_attention( + num_heads, + num_queries_per_kv, + head_size, + sliding_window, + dtype, + kv_cache_dtype, + device, + op, + ) + + +@pytest.mark.optional +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("num_queries_per_kv", NUM_QUERIES_PER_KV) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("dtype", [torch.float32]) +@pytest.mark.parametrize("kv_cache_dtype", KV_CACHE_DTYPES) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("op", OPS) +@torch.inference_mode() +def test_contexted_kv_attention_alibi_f32( + num_heads: int, + num_queries_per_kv: int, + head_size: int, + dtype: torch.dtype, + kv_cache_dtype: str, + device: str, + op: Callable, +) -> None: + test_contexted_kv_attention_alibi( + num_heads, num_queries_per_kv, head_size, dtype, kv_cache_dtype, device, op + ) + + +@pytest.mark.parametrize("head_size", [128]) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("op", OPS) +@torch.inference_mode() +def test_qwen3_nonstandard_block_size( + head_size: int, + dtype: torch.dtype, + device: str, + op: Callable, +) -> None: + """ + A separate test function specifically added + for Qwen3-Next-80B (Block Size 544). + """ + if not current_platform.is_rocm(): + pytest.skip("544 block size optimization is only for ROCm.") + + test_contexted_kv_attention( + num_heads=64, + num_queries_per_kv=1, + head_size=head_size, + block_size=544, + sliding_window=0, + dtype=dtype, + kv_cache_dtype="auto", + device=device, + op=op, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_rocm_aiter_mla_decode_metadata.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_rocm_aiter_mla_decode_metadata.py new file mode 100644 index 0000000000000000000000000000000000000000..4a695f11a4a9df64b0772e046c21cdb61ab17428 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_rocm_aiter_mla_decode_metadata.py @@ -0,0 +1,202 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Regression test for AITER MLA persistent decode metadata dtypes. + +For the gfx950 fp8/fp8 nhead=32 qlen=1 fold path, the split/reduce metadata +layout depends on the q/kv element size. The builder must forward dtype_q/dtype_kv +to ``get_mla_metadata_v1``; omitting them lays out the work for the wrong dtype +and corrupts decode output. The test pins the builder's metadata to a golden +recomputed at runtime with the explicit correct dtypes. +""" + +import types +from unittest.mock import patch + +import pytest +import torch + +from vllm._aiter_ops import is_aiter_found +from vllm.platforms import current_platform + + +def _on_gfx950() -> bool: + if not (current_platform.is_rocm() and is_aiter_found()): + return False + from vllm.platforms.rocm import on_gfx950 + + return on_gfx950() + + +pytestmark = pytest.mark.skipif( + not _on_gfx950(), + reason="AITER MLA fp8 persistent decode metadata is gfx950-only", +) + +# The fold path that the bug corrupted: fp8 query + fp8 KV-cache, 32 query +# heads, single-token decode, batch 128, context 8192, page_size 1. +NUM_QUERY_HEADS = 32 +DECODE_QLEN = 1 +BATCH_SIZE = 128 +CONTEXT_LEN = 8192 +PAGE_SIZE = 1 + +# Expected dtypes for this fold path: bf16 model dtype -> bf16 query; fp8 +# KV-cache -> fp8_e4m3 kv. +EXPECTED_Q_DTYPE = torch.bfloat16 +EXPECTED_KV_DTYPE = torch.float8_e4m3fn + +# The split/reduce content tensors filled by get_mla_metadata_v1. work_meta_data +# is excluded: it holds raw device pointers, never equal across allocations. +_CONTENT_METADATA_FIELDS = ( + "work_indptr", + "work_info_set", + "reduce_indptr", + "reduce_final_map", + "reduce_partial_map", +) + +# The builder's get_mla_metadata_v1 call passes 6 input args then 6 output +# buffers (see AiterMLAMetadataBuilder._build_decode). Output order -> field. +_NUM_INPUT_ARGS = 6 +_OUTPUT_ARG_FIELDS = ( + "work_meta_data", + "work_info_set", + "work_indptr", + "reduce_indptr", + "reduce_final_map", + "reduce_partial_map", +) + + +def _build_decode_metadata(): + """Build AITER MLA decode metadata for the fp8/fp8 nhead=32 fold path. + + Returns ``(metadata, captured)`` where ``captured`` records the positional + args/kwargs the builder passed to ``get_mla_metadata_v1``, so the golden can + be recomputed from the identical inputs. + """ + from tests.v1.attention.utils import ( + BatchSpec, + create_common_attn_metadata, + create_vllm_config, + ) + from vllm.config.vllm import set_current_vllm_config + from vllm.v1.attention.backends.registry import AttentionBackendEnum + from vllm.v1.kv_cache_interface import MLAAttentionSpec + from vllm.v1.worker.workspace import init_workspace_manager + + device = torch.device("cuda:0") + + vllm_config = create_vllm_config( + model_name="deepseek-ai/DeepSeek-R1", + max_model_len=CONTEXT_LEN, + # One flat page per token (page_size=1); +buffer for the null block. + num_gpu_blocks=BATCH_SIZE * CONTEXT_LEN + 200, + block_size=PAGE_SIZE, + max_num_seqs=BATCH_SIZE, + max_num_batched_tokens=8192, + hf_config_override={"num_attention_heads": NUM_QUERY_HEADS}, + ) + vllm_config.cache_config.cache_dtype = "fp8" + + spec = MLAAttentionSpec( + block_size=PAGE_SIZE, + num_kv_heads=1, + head_size=vllm_config.model_config.get_head_size(), + dtype=vllm_config.model_config.dtype, + cache_dtype_str="fp8", + ) + + builder_cls = AttentionBackendEnum.ROCM_AITER_MLA.get_class().get_builder_cls() + + # The builder reads layer.prefill_backend from static_forward_context; a + # stub with the attribute is enough for metadata construction. + layer_name = "placeholder" + vllm_config.compilation_config.static_forward_context[layer_name] = ( + types.SimpleNamespace(prefill_backend=None) + ) + + init_workspace_manager(device) + + batch_spec = BatchSpec( + seq_lens=[CONTEXT_LEN] * BATCH_SIZE, + query_lens=[DECODE_QLEN] * BATCH_SIZE, + ) + + captured: dict = {} + + with set_current_vllm_config(vllm_config): + builder = builder_cls(spec, [layer_name], vllm_config, device) + common_attn_metadata = create_common_attn_metadata( + batch_spec, PAGE_SIZE, device, arange_block_indices=True + ) + + import aiter + + real_get_mla_metadata_v1 = aiter.get_mla_metadata_v1 + + def spy(*args, **kwargs): + captured["args"] = args + captured["kwargs"] = dict(kwargs) + return real_get_mla_metadata_v1(*args, **kwargs) + + with patch("aiter.get_mla_metadata_v1", spy): + metadata = builder.build( + common_prefix_len=0, + common_attn_metadata=common_attn_metadata, + ) + + return metadata, captured + + +def _compute_golden_metadata(captured: dict) -> dict[str, torch.Tensor]: + """Recompute the persistent metadata with explicit fp8/bf16 dtypes. + + Replays ``get_mla_metadata_v1`` on the builder's exact input tensors with + fresh output buffers and the explicitly-correct dtypes. This reference must + match the builder's output when the fix is in place. + """ + import aiter + + args = captured["args"] + inputs = args[:_NUM_INPUT_ARGS] + # Fresh copies so the golden does not alias the builder's persistent buffers. + fresh_outputs = [arg.clone() for arg in args[_NUM_INPUT_ARGS:]] + + golden_kwargs = dict(captured["kwargs"]) + golden_kwargs["dtype_q"] = EXPECTED_Q_DTYPE + golden_kwargs["dtype_kv"] = EXPECTED_KV_DTYPE + + aiter.get_mla_metadata_v1(*inputs, *fresh_outputs, **golden_kwargs) + + return dict(zip(_OUTPUT_ARG_FIELDS, fresh_outputs)) + + +def test_persistent_decode_metadata_matches_fp8_golden(): + """The builder's metadata must match the dtype-correct golden. + + Regression guard: the fixed builder forwards fp8/bf16 dtypes so its + split/reduce metadata matches the golden recomputed with those explicit + dtypes. Dropping the dtypes (the original bug) produces a different layout + and fails this test. + """ + metadata, captured = _build_decode_metadata() + + # qlen=1 must take the persistent-metadata path for this to be meaningful. + assert metadata.decode is not None + assert metadata.decode.has_persistent_metadata + assert metadata.work_meta_data is not None + + golden = _compute_golden_metadata(captured) + + mismatched = [ + name + for name in _CONTENT_METADATA_FIELDS + if getattr(metadata, name).shape != golden[name].shape + or not torch.equal(getattr(metadata, name), golden[name]) + ] + assert not mismatched, ( + "AITER MLA persistent decode metadata does not match the fp8/bf16 " + f"golden for fields {mismatched}; the builder must forward " + "dtype_q/dtype_kv to get_mla_metadata_v1." + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_rocm_aiter_unified_attn.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_rocm_aiter_unified_attn.py new file mode 100644 index 0000000000000000000000000000000000000000..4c635076374a34b47471fcc52e5fb22b0d9d1f67 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_rocm_aiter_unified_attn.py @@ -0,0 +1,340 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""ROCm kernel correctness tests for AITER unified attention. + +Compares ``aiter.ops.triton.unified_attention`` against ``ref_paged_attn`` under +decode, prefill, and mixed batches with varied shapes. +""" + +from typing import Any, Literal + +import pytest +import torch + +from tests.kernels.attention.test_triton_unified_attention import ref_paged_attn +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +_SKIP_NON_MI3XX = True +if current_platform.is_rocm(): + from vllm.platforms.rocm import on_mi3xx + + _SKIP_NON_MI3XX = not on_mi3xx() + +pytestmark = [ + pytest.mark.skipif(not current_platform.is_rocm(), reason="ROCm-specific tests"), + pytest.mark.skipif(_SKIP_NON_MI3XX, reason="MI300/MI350 ROCm only"), +] + +NUM_Q_HEADS = 8 +NUM_KV_HEADS = 8 +HEAD_SIZES = [128, 256] +BLOCK_SIZES = [16, 64] +# TODO: re-add torch.float16 once AITER reenables fp16 unified attention. +DTYPES = [torch.bfloat16] +FP8_DTYPE = current_platform.fp8_dtype() + +# (query_len, kv_len) per sequence +MIXED_SEQ_LENS = [ + [(1, 128), (5, 18), (129, 463)], + [(10, 256), (5, 64), (32, 128)], + [(1, 1024), (5, 18), (129, 1328)], +] +DECODE_SEQ_LENS = [ + [(1, 128), (1, 256), (1, 384), (1, 512)], + [(1, 1024), (1, 1536), (1, 2048)], +] +PREFILL_SEQ_LENS = [ + [(256, 256), (128, 512)], + [(64, 128), (32, 256), (16, 512)], + [(256, 1024), (128, 2048)], +] + +DEFAULT_ATOL, DEFAULT_RTOL = 1.5e-2, 1e-2 +FP8_ATOL, FP8_RTOL = 1.5e-1, 1.5e-1 +# Non-unity scale so q_descale handling is exercised explicitly. +Q_SCALE = 0.75 +K_SCALE, V_SCALE = 0.5, 0.25 + +Fp8Variant = Literal["fp8_kv", "fp8_query", "fp8_query_kv"] + +FP8_VARIANTS = [ + pytest.param("fp8_kv", id="fp8_kv"), + pytest.param("fp8_query", id="fp8_query"), + pytest.param("fp8_query_kv", id="fp8_query_kv"), +] + +FP8_SEQ_LENS = [ + MIXED_SEQ_LENS[0], + DECODE_SEQ_LENS[0], + DECODE_SEQ_LENS[1], + PREFILL_SEQ_LENS[0], + PREFILL_SEQ_LENS[2], +] + + +def _require_aiter() -> None: + from vllm._aiter_ops import is_aiter_found_and_supported + + if not is_aiter_found_and_supported(): + pytest.skip("aiter is required on supported ROCm hardware for this test") + + +def _make_case( + *, + seq_lens: list[tuple[int, int]], + head_size: int, + block_size: int, + dtype: torch.dtype, + num_blocks: int = 2048, + kv_cache_dtype: torch.dtype | None = None, + k_scale: float = 1.0, + v_scale: float = 1.0, + q_dtype: torch.dtype | None = None, + q_scale: float = Q_SCALE, +) -> dict[str, Any]: + torch.set_default_device("cuda") + + query_lens = [q for q, _ in seq_lens] + kv_lens = [k for _, k in seq_lens] + num_seqs = len(seq_lens) + max_query_len = max(query_lens) + max_kv_len = max(kv_lens) + scale = head_size**-0.5 + + query = torch.randn(sum(query_lens), NUM_Q_HEADS, head_size, dtype=dtype) + if kv_cache_dtype is None: + key_cache = torch.randn( + num_blocks, block_size, NUM_KV_HEADS, head_size, dtype=dtype + ) + value_cache = torch.randn_like(key_cache) + else: + key_cache = torch.clamp( + torch.randn(num_blocks, block_size, NUM_KV_HEADS, head_size), + -1.0, + 1.0, + ).to(kv_cache_dtype) + value_cache = torch.clamp( + torch.randn(num_blocks, block_size, NUM_KV_HEADS, head_size), + -1.0, + 1.0, + ).to(kv_cache_dtype) + + cu_seqlens_q = torch.tensor([0] + query_lens, dtype=torch.int32).cumsum( + dim=0, dtype=torch.int32 + ) + seq_lens_tensor = torch.tensor(kv_lens, dtype=torch.int32) + + max_num_blocks = (max_kv_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, num_blocks, (num_seqs, max_num_blocks), dtype=torch.int32 + ) + + descale_shape = (num_seqs, NUM_KV_HEADS) + k_descale = torch.full(descale_shape, k_scale, dtype=torch.float32, device="cuda") + v_descale = torch.full(descale_shape, v_scale, dtype=torch.float32, device="cuda") + + kernel_query = query + q_descale = None + if q_dtype is not None: + q_descale = torch.tensor(q_scale, dtype=torch.float32, device="cuda") + kernel_query = (query / q_scale).to(q_dtype) + + return { + "query": query, + "kernel_query": kernel_query, + "key_cache": key_cache, + "value_cache": value_cache, + "block_tables": block_tables, + "query_lens": query_lens, + "kv_lens": kv_lens, + "seq_lens_tensor": seq_lens_tensor, + "cu_seqlens_q": cu_seqlens_q, + "q_descale": q_descale, + "k_descale": k_descale, + "v_descale": v_descale, + "scale": scale, + "max_query_len": max_query_len, + "max_kv_len": max_kv_len, + "query_dtype": dtype, + "k_scale": k_scale, + "v_scale": v_scale, + } + + +def _make_fp8_case( + *, + seq_lens: list[tuple[int, int]], + head_size: int, + block_size: int, + variant: Fp8Variant, +) -> dict[str, Any]: + use_fp8_kv = variant in ("fp8_kv", "fp8_query_kv") + use_fp8_query = variant in ("fp8_query", "fp8_query_kv") + return _make_case( + seq_lens=seq_lens, + head_size=head_size, + block_size=block_size, + dtype=torch.bfloat16, + kv_cache_dtype=FP8_DTYPE if use_fp8_kv else None, + k_scale=K_SCALE if use_fp8_kv else 1.0, + v_scale=V_SCALE if use_fp8_kv else 1.0, + q_dtype=FP8_DTYPE if use_fp8_query else None, + ) + + +def _run_aiter_unified_attention(case: dict[str, Any]) -> torch.Tensor: + from aiter.ops.triton.unified_attention import unified_attention + + kernel_query = case["kernel_query"] + # Kernel writes high-precision output even when Q is FP8 (matches vLLM usage). + output = torch.empty_like(case["query"]) + unified_attention( + q=kernel_query, + k=case["key_cache"], + v=case["value_cache"], + out=output, + cu_seqlens_q=case["cu_seqlens_q"], + max_seqlen_q=case["max_query_len"], + seqused_k=case["seq_lens_tensor"], + max_seqlen_k=case["max_kv_len"], + softmax_scale=case["scale"], + causal=True, + alibi_slopes=None, + window_size=(-1, -1), + block_table=case["block_tables"], + softcap=0, + q_descale=case["q_descale"], + k_descale=case["k_descale"], + v_descale=case["v_descale"], + sinks=None, + output_scale=None, + ) + return output + + +def _ref_output(case: dict[str, Any]) -> torch.Tensor: + key_cache = case["key_cache"] + value_cache = case["value_cache"] + if key_cache.dtype != case["query_dtype"]: + key_cache = key_cache.to(case["query_dtype"]) * case["k_scale"] + value_cache = value_cache.to(case["query_dtype"]) * case["v_scale"] + + return ref_paged_attn( + query=case["query"], + key_cache=key_cache, + value_cache=value_cache, + query_lens=case["query_lens"], + kv_lens=case["kv_lens"], + block_tables=case["block_tables"], + scale=case["scale"], + ) + + +def _assert_matches_reference( + case: dict[str, Any], + *, + atol: float = DEFAULT_ATOL, + rtol: float = DEFAULT_RTOL, +) -> None: + output = _run_aiter_unified_attention(case) + output_ref = _ref_output(case) + torch.testing.assert_close(output, output_ref, atol=atol, rtol=rtol) + + +@pytest.mark.parametrize("seq_lens", MIXED_SEQ_LENS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@torch.inference_mode() +def test_aiter_unified_attn_mixed_batch( + seq_lens: list[tuple[int, int]], + head_size: int, + block_size: int, + dtype: torch.dtype, +) -> None: + """Decode + prefill sequences in one batch (native dtypes).""" + _require_aiter() + set_random_seed(0) + + case = _make_case( + seq_lens=seq_lens, + head_size=head_size, + block_size=block_size, + dtype=dtype, + ) + _assert_matches_reference(case) + + +@pytest.mark.parametrize("seq_lens", DECODE_SEQ_LENS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@torch.inference_mode() +def test_aiter_unified_attn_decode( + seq_lens: list[tuple[int, int]], + head_size: int, + block_size: int, + dtype: torch.dtype, +) -> None: + """Single-token decode (native dtypes).""" + _require_aiter() + set_random_seed(0) + + case = _make_case( + seq_lens=seq_lens, + head_size=head_size, + block_size=block_size, + dtype=dtype, + ) + _assert_matches_reference(case) + + +@pytest.mark.parametrize("seq_lens", PREFILL_SEQ_LENS) +@pytest.mark.parametrize("head_size", [128]) +@pytest.mark.parametrize("block_size", [16]) +@torch.inference_mode() +def test_aiter_unified_attn_prefill( + seq_lens: list[tuple[int, int]], + head_size: int, + block_size: int, +) -> None: + """Prefill-only batches with query_len > 1 (native dtypes).""" + _require_aiter() + set_random_seed(0) + + case = _make_case( + seq_lens=seq_lens, + head_size=head_size, + block_size=block_size, + dtype=torch.bfloat16, + ) + _assert_matches_reference(case) + + +@pytest.mark.skipif( + not current_platform.supports_fp8(), + reason="FP8 not supported on this hardware", +) +@pytest.mark.parametrize("variant", FP8_VARIANTS) +@pytest.mark.parametrize("seq_lens", FP8_SEQ_LENS) +@pytest.mark.parametrize("head_size", [128]) +@pytest.mark.parametrize("block_size", [16, 64]) +@torch.inference_mode() +def test_aiter_unified_attn_fp8( + variant: Fp8Variant, + seq_lens: list[tuple[int, int]], + head_size: int, + block_size: int, +) -> None: + """FP8 KV cache, FP8 query, or both; compared at bf16 reference precision.""" + _require_aiter() + set_random_seed(0) + + case = _make_fp8_case( + seq_lens=seq_lens, + head_size=head_size, + block_size=block_size, + variant=variant, + ) + _assert_matches_reference(case, atol=FP8_ATOL, rtol=FP8_RTOL) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_rocm_attention_selector.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_rocm_attention_selector.py new file mode 100644 index 0000000000000000000000000000000000000000..b6470dc305eada59fc23de6c039d7e8c1ea1b352 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_rocm_attention_selector.py @@ -0,0 +1,73 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm.config import AttentionConfig, VllmConfig, set_current_vllm_config +from vllm.platforms.rocm import RocmPlatform +from vllm.v1.attention.backends.registry import AttentionBackendEnum +from vllm.v1.attention.selector import _cached_get_attn_backend, get_attn_backend + + +@pytest.fixture(autouse=True) +def clear_cache(): + """Clear lru cache to ensure each test case runs without caching.""" + _cached_get_attn_backend.cache_clear() + + +@pytest.mark.skip(reason="Skipped for now. Should be revisited.") +def test_selector(monkeypatch: pytest.MonkeyPatch): + # Set the current platform to ROCm using monkeypatch + monkeypatch.setattr("vllm.v1.attention.selector.current_platform", RocmPlatform()) + + # Test standard ROCm attention + attention_config = AttentionConfig(backend=AttentionBackendEnum.ROCM_ATTN) + vllm_config = VllmConfig(attention_config=attention_config) + + with set_current_vllm_config(vllm_config): + backend = get_attn_backend(16, torch.float16, torch.float16, 16, False) + assert backend.get_name() == "ROCM_FLASH" or backend.get_name() == "TRITON_ATTN" + + # MLA test for deepseek related + # Change the attention backend to triton MLA + attention_config = AttentionConfig(backend=AttentionBackendEnum.TRITON_MLA) + vllm_config = VllmConfig(attention_config=attention_config) + + with set_current_vllm_config(vllm_config): + backend = get_attn_backend(576, torch.bfloat16, "auto", 16, False, use_mla=True) + assert backend.get_name() == "TRITON_MLA" + + # If attention backend is None + # If use_mla is true + # The selected backend is triton MLA + attention_config = AttentionConfig(backend=None) + vllm_config = VllmConfig(attention_config=attention_config) + + with set_current_vllm_config(vllm_config): + backend = get_attn_backend(576, torch.bfloat16, "auto", 16, False, use_mla=True) + assert backend.get_name() == "TRITON_MLA" + + # Change the attention backend to AITER MLA + attention_config = AttentionConfig(backend=AttentionBackendEnum.ROCM_AITER_MLA) + vllm_config = VllmConfig(attention_config=attention_config) + + with set_current_vllm_config(vllm_config): + backend = get_attn_backend(576, torch.bfloat16, "auto", 1, False, use_mla=True) + assert backend.get_name() == "ROCM_AITER_MLA" + + # If attention backend is None + # If use_mla is true + # If VLLM_ROCM_USE_AITER is enabled + # The selected backend is ROCM_AITER_MLA + with monkeypatch.context() as m: + m.setenv("VLLM_ROCM_USE_AITER", "1") + + attention_config = AttentionConfig(backend=None) + vllm_config = VllmConfig(attention_config=attention_config) + + with set_current_vllm_config(vllm_config): + backend = get_attn_backend( + 576, torch.bfloat16, "auto", 1, False, use_mla=True + ) + assert backend.get_name() == "ROCM_AITER_MLA" diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_rocm_triton_attn_dsv4.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_rocm_triton_attn_dsv4.py new file mode 100644 index 0000000000000000000000000000000000000000..f3340647c4cd6aa0f071afb454d4d7bfa82948cf --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_rocm_triton_attn_dsv4.py @@ -0,0 +1,740 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm.platforms import current_platform + +pytestmark = pytest.mark.skipif( + not current_platform.is_rocm(), reason="Only used by ROCm" +) + + +def _on_gfx950() -> bool: + if not current_platform.is_rocm(): + return False + try: + from vllm.platforms.rocm import _ON_GFX950 + + return bool(_ON_GFX950) + except Exception: + return False + + +# The flash-decode split-K decode path is only tuned for AMD gfx950; other +# architectures take the fallback decode kernel, so its tests are skipped there. +requires_gfx950 = pytest.mark.skipif( + not _on_gfx950(), + reason="split-K decode kernel is only tuned for AMD gfx950", +) + +NOPE_HEAD_DIM = 448 +ROPE_HEAD_DIM = 64 +HEAD_DIM = NOPE_HEAD_DIM + ROPE_HEAD_DIM + + +def _ref_global_topk_ragged( + topk_indices: torch.Tensor, + token_to_req_indices: torch.Tensor, + block_table: torch.Tensor, + block_size: int, + is_valid_token: torch.Tensor, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + topk = topk_indices.reshape(topk_indices.shape[0], -1) + valid = (topk >= 0) & is_valid_token[:, None] + lens = valid.sum(dim=1, dtype=torch.int32) + indptr = torch.zeros(lens.shape[0] + 1, dtype=torch.int32, device=topk.device) + torch.cumsum(lens, dim=0, out=indptr[1:]) + + safe_topk = torch.clamp(topk, min=0) + block_indices = safe_topk // block_size + block_offsets = safe_topk % block_size + req_indices = token_to_req_indices[:, None].expand_as(topk) + slot_ids = block_table[req_indices, block_indices] * block_size + block_offsets + + offsets = torch.arange(topk.shape[1], dtype=torch.int32, device=topk.device) + positions = indptr[:-1, None] + offsets[None, :] + return slot_ids[valid], positions[valid].to(torch.long), indptr, lens + + +def _ref_sparse_prefill_ragged( + q: torch.Tensor, + kv: torch.Tensor, + rows: list[list[int]], + scale: float, + attn_sink: torch.Tensor | None, +) -> torch.Tensor: + q_f32 = q.float() + kv_f32 = kv.float() + out = torch.empty_like(q_f32) + + for query_idx in range(q.shape[0]): + row_indices = rows[query_idx] + for head_idx in range(q.shape[1]): + if row_indices: + selected_kv = kv_f32[row_indices] + scores = torch.mv(selected_kv, q_f32[query_idx, head_idx]) * scale + if attn_sink is not None: + scores_with_sink = torch.cat( + [scores, attn_sink[head_idx].float().reshape(1)] + ) + probs = torch.softmax(scores_with_sink, dim=0)[:-1] + else: + probs = torch.softmax(scores, dim=0) + out[query_idx, head_idx] = torch.sum( + probs[:, None] * selected_kv, dim=0 + ) + else: + out[query_idx, head_idx] = 0 + return out.to(torch.bfloat16) + + +def _pack_fp8_ds_mla_cache( + kv: torch.Tensor, block_size: int, is_extra: bool = False +) -> torch.Tensor: + assert kv.shape[-1] == HEAD_DIM + num_tokens = kv.shape[0] + num_blocks = (num_tokens + block_size - 1) // block_size + cache = torch.zeros( + (num_blocks, block_size, 584), + dtype=torch.uint8, + device=kv.device, + ) + cache_flat = cache.view(torch.uint8).flatten() + kv_nope_fp8 = ( + kv[:, :NOPE_HEAD_DIM] + .to(torch.float8_e4m3fn if is_extra else current_platform.fp8_dtype()) + .view(torch.uint8) + ) + kv_rope_u8 = kv[:, NOPE_HEAD_DIM:].contiguous().view(torch.uint8) + + for slot in range(num_tokens): + block_idx = slot // block_size + pos = slot % block_size + block_base = block_idx * cache.stride(0) + token_base = block_base + pos * 576 + scale_base = block_base + block_size * 576 + pos * 8 + cache_flat[token_base : token_base + NOPE_HEAD_DIM].copy_(kv_nope_fp8[slot]) + cache_flat[ + token_base + NOPE_HEAD_DIM : token_base + NOPE_HEAD_DIM + ROPE_HEAD_DIM * 2 + ].copy_(kv_rope_u8[slot]) + cache_flat[scale_base : scale_base + 7].fill_(127) + return cache + + +def _read_fp8_ds_mla_cache( + cache: torch.Tensor, slot: int, block_size: int, is_extra: bool = False +) -> torch.Tensor: + cache_flat = cache.view(torch.uint8).flatten() + block_idx = slot // block_size + pos = slot % block_size + block_base = block_idx * cache.stride(0) + token_base = block_base + pos * 576 + + nope_u8 = cache_flat[token_base : token_base + NOPE_HEAD_DIM] + nope = nope_u8.view( + torch.float8_e4m3fn if is_extra else current_platform.fp8_dtype() + ).to(torch.float32) + rope_u8 = cache_flat[ + token_base + NOPE_HEAD_DIM : token_base + NOPE_HEAD_DIM + ROPE_HEAD_DIM * 2 + ] + rope = rope_u8.view(torch.bfloat16).to(torch.float32) + return torch.cat([nope, rope]) + + +def _ref_sparse_decode_ragged( + q: torch.Tensor, + main_cache: torch.Tensor, + main_rows: list[list[int]], + scale: float, + attn_sink: torch.Tensor | None, + block_size: int, + extra_cache: torch.Tensor | None = None, + extra_rows: list[list[int]] | None = None, +) -> torch.Tensor: + q_f32 = q.float() + out = torch.empty_like(q_f32) + + for query_idx in range(q.shape[0]): + row_kv = [ + _read_fp8_ds_mla_cache(main_cache, int(slot), block_size) + for slot in main_rows[query_idx] + ] + if extra_cache is not None and extra_rows is not None: + row_kv.extend( + _read_fp8_ds_mla_cache( + extra_cache, int(slot), block_size, is_extra=True + ) + for slot in extra_rows[query_idx] + ) + + kv = torch.stack(row_kv).to(q.device) + for head_idx in range(q.shape[1]): + scores = torch.mv(kv, q_f32[query_idx, head_idx]) * scale + if attn_sink is not None: + scores_with_sink = torch.cat( + [scores, attn_sink[head_idx].float().reshape(1)] + ) + probs = torch.softmax(scores_with_sink, dim=0)[:-1] + else: + probs = torch.softmax(scores, dim=0) + out[query_idx, head_idx] = torch.sum(probs[:, None] * kv, dim=0) + return out.to(torch.bfloat16) + + +def _ragged_from_rows( + rows: list[list[int]], device: torch.device +) -> tuple[torch.Tensor, torch.Tensor]: + """Flatten per-query slot lists into ragged (indices, indptr) tensors.""" + flat = [slot for row in rows for slot in row] + indptr = [0] + for row in rows: + indptr.append(indptr[-1] + len(row)) + return ( + torch.tensor(flat, dtype=torch.int32, device=device), + torch.tensor(indptr, dtype=torch.int32, device=device), + ) + + +def _ref_combine_topk_swa_ragged( + device: torch.device, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + expected_ragged = torch.tensor( + [ + 100, + 101, + 7, + 8, + 9, + 110, + 111, + 8, + 9, + 10, + 120, + 121, + 122, + 9, + 10, + 11, + 150, + 27, + 28, + 29, + 160, + 161, + 28, + 29, + 30, + ], + dtype=torch.int32, + device=device, + ) + expected_lens = torch.tensor([5, 5, 6, 4, 5], dtype=torch.int32, device=device) + expected_indptr = torch.zeros(6, dtype=torch.int32, device=device) + torch.cumsum(expected_lens, dim=0, out=expected_indptr[1:]) + return expected_ragged, expected_indptr, expected_lens + + +@torch.inference_mode() +def test_compute_global_topk_ragged_indices_and_indptr() -> None: + from vllm.models.deepseek_v4.amd.rocm import ( + compute_global_topk_ragged_indices_and_indptr, + ) + + device = torch.device("cuda") + block_size = 4 + topk_indices = torch.tensor( + [ + [0, 3, 4, -1], + [5, 8, -1, -1], + [2, 7, 9, -1], + ], + dtype=torch.int32, + device=device, + ) + token_to_req_indices = torch.tensor([0, 1, 1], dtype=torch.int32, device=device) + block_table = torch.tensor( + [ + [10, 11, 12], + [20, 21, 22], + ], + dtype=torch.int32, + device=device, + ) + is_valid_token = torch.tensor([True, False, True], dtype=torch.bool, device=device) + + actual_ragged, actual_indptr, actual_lens = ( + compute_global_topk_ragged_indices_and_indptr( + topk_indices, + token_to_req_indices, + block_table, + block_size, + is_valid_token, + ) + ) + expected_values, expected_positions, expected_indptr, expected_lens = ( + _ref_global_topk_ragged( + topk_indices, + token_to_req_indices, + block_table, + block_size, + is_valid_token, + ) + ) + + torch.testing.assert_close(actual_ragged[expected_positions], expected_values) + torch.testing.assert_close(actual_indptr, expected_indptr) + torch.testing.assert_close(actual_lens, expected_lens) + + +@torch.inference_mode() +def test_sparse_attn_prefill_ragged_kernel() -> None: + from vllm.v1.attention.ops.rocm_aiter_mla_sparse import ( + _rocm_sparse_attn_prefill_ragged_triton, + ) + + device = torch.device("cuda") + torch.manual_seed(0) + q = torch.randn(3, 3, HEAD_DIM, dtype=torch.bfloat16, device=device) * 0.125 + kv = torch.randn(5, HEAD_DIM, dtype=torch.bfloat16, device=device) * 0.125 + indices = torch.tensor([0, 2, 1, 3, 4], dtype=torch.int32, device=device) + indptr = torch.tensor([0, 2, 5, 5], dtype=torch.int32, device=device) + attn_sink = torch.tensor([-0.25, 0.0, 0.25], dtype=torch.float32, device=device) + scale = HEAD_DIM**-0.5 + + actual = _rocm_sparse_attn_prefill_ragged_triton( + q=q, + kv=kv, + indices=indices, + indptr=indptr, + scale=scale, + attn_sink=attn_sink, + nope_head_dim=NOPE_HEAD_DIM, + rope_head_dim=ROPE_HEAD_DIM, + ) + expected = _ref_sparse_prefill_ragged( + q, kv, [[0, 2], [1, 3, 4], []], scale, attn_sink + ) + + torch.testing.assert_close(actual, expected, atol=2e-2, rtol=2e-2) + + +@torch.inference_mode() +def test_sparse_attn_decode_ragged_kernel() -> None: + from vllm.v1.attention.ops.rocm_aiter_mla_sparse import ( + _rocm_sparse_attn_decode_ragged_triton, + ) + + device = torch.device("cuda") + torch.manual_seed(1) + block_size = 4 + q = torch.randn(2, 3, HEAD_DIM, dtype=torch.bfloat16, device=device) * 0.125 + main_kv = torch.randn(6, HEAD_DIM, dtype=torch.bfloat16, device=device) * 0.125 + extra_kv = torch.randn(5, HEAD_DIM, dtype=torch.bfloat16, device=device) * 0.125 + main_cache = _pack_fp8_ds_mla_cache(main_kv, block_size) + extra_cache = _pack_fp8_ds_mla_cache(extra_kv, block_size, is_extra=True) + main_indices = torch.tensor([0, 2, 4, 1], dtype=torch.int32, device=device) + main_indptr = torch.tensor([0, 2, 4], dtype=torch.int32, device=device) + extra_indices = torch.tensor([1, 3, 0], dtype=torch.int32, device=device) + extra_indptr = torch.tensor([0, 1, 3], dtype=torch.int32, device=device) + attn_sink = torch.tensor([-0.1, 0.0, 0.1], dtype=torch.float32, device=device) + scale = HEAD_DIM**-0.5 + + actual = _rocm_sparse_attn_decode_ragged_triton( + q=q, + main_cache=main_cache, + main_indices=main_indices, + main_indptr=main_indptr, + scale=scale, + attn_sink=attn_sink, + nope_head_dim=NOPE_HEAD_DIM, + rope_head_dim=ROPE_HEAD_DIM, + extra_cache=extra_cache, + extra_indices=extra_indices, + extra_indptr=extra_indptr, + ) + expected = _ref_sparse_decode_ragged( + q=q, + main_cache=main_cache, + main_rows=[[0, 2], [4, 1]], + scale=scale, + attn_sink=attn_sink, + block_size=block_size, + extra_cache=extra_cache, + extra_rows=[[1], [3, 0]], + ) + + torch.testing.assert_close(actual, expected, atol=2e-2, rtol=2e-2) + + +@torch.inference_mode() +def test_combine_topk_swa_indices_ragged() -> None: + from vllm.models.deepseek_v4.amd.rocm import ( + combine_topk_swa_indices_ragged, + ) + + device = torch.device("cuda") + topk_indices = torch.tensor( + [ + [100, 101, 102, 103], + [110, 111, 112, 113], + [120, 121, 122, 123], + [130, 131, 132, 133], + [140, 141, 142, 143], + ], + dtype=torch.int32, + device=device, + ) + query_start_loc = torch.tensor([0, 3, 5], dtype=torch.int32, device=device) + seq_lens = torch.tensor([6, 4], dtype=torch.int32, device=device) + gather_lens = torch.tensor([4, 3], dtype=torch.int32, device=device) + window_size = 3 + compress_ratio = 2 + topk = 4 + M = 20 + N = 8 + + actual_ragged, actual_indptr, actual_lens = combine_topk_swa_indices_ragged( + topk_indices, + query_start_loc, + seq_lens, + gather_lens, + window_size, + compress_ratio, + topk, + M, + N, + ) + expected_ragged, expected_indptr, expected_lens = _ref_combine_topk_swa_ragged( + device + ) + + torch.testing.assert_close( + actual_ragged[: expected_ragged.numel()], expected_ragged + ) + torch.testing.assert_close(actual_indptr, expected_indptr) + torch.testing.assert_close(actual_lens, expected_lens) + + +@requires_gfx950 +@torch.inference_mode() +def test_decode_num_splits_heuristic(monkeypatch) -> None: + """Split-count heuristic added with the flash-decode split-K decode path.""" + from vllm.v1.attention.ops import rocm_aiter_mla_sparse as mod + + # Pin the CU count so the heuristic is deterministic off-device. + monkeypatch.setattr(mod, "_decode_cu_count", lambda: 256) + + # A batch that already fills the device should not be split. + assert mod._decode_num_splits(256, 1, avg_main_len=128.0, avg_extra_len=0.0) == 1 + # A tiny batch on a large device should split to add parallelism. + assert mod._decode_num_splits(2, 1, avg_main_len=256.0, avg_extra_len=0.0) > 1 + + # The chosen count always stays within the searched [1, 16] range, and a + # zero-length workload never splits (no work to parallelize). + for num_queries in (1, 4, 24, 224, 1024): + splits = mod._decode_num_splits( + num_queries, 1, avg_main_len=512.0, avg_extra_len=128.0 + ) + assert 1 <= splits <= 16 + assert mod._decode_num_splits(2, 1, avg_main_len=0.0, avg_extra_len=0.0) >= 1 + + +@requires_gfx950 +@pytest.mark.parametrize("num_splits", [1, 2, 3, 4, 8]) +@pytest.mark.parametrize("with_extra", [True, False]) +@pytest.mark.parametrize("with_sink", [True, False]) +@torch.inference_mode() +def test_sparse_attn_decode_split_k_kernel( + monkeypatch, num_splits: int, with_extra: bool, with_sink: bool +) -> None: + """Flash-decode split-K decode path (partial + reduce kernels). + + This path is the gfx950 production path (``_ON_GFX950``), so the test only + runs on gfx950. The split count is pinned so the partial/reduce kernels are + exercised across split counts. ``num_splits=8`` drives splits past the + shortest segment length, covering the empty-split edge case handled by the + reduce kernel. + """ + from vllm.v1.attention.ops import rocm_aiter_mla_sparse as mod + + device = torch.device("cuda") + torch.manual_seed(7) + block_size = 4 + num_heads = 3 + + main_rows = [[0, 2, 4, 6, 1, 3, 7, 5], [4, 1, 6, 0, 2]] + num_queries = len(main_rows) + q = ( + torch.randn( + num_queries, num_heads, HEAD_DIM, dtype=torch.bfloat16, device=device + ) + * 0.125 + ) + main_kv = torch.randn(8, HEAD_DIM, dtype=torch.bfloat16, device=device) * 0.125 + main_cache = _pack_fp8_ds_mla_cache(main_kv, block_size) + main_indices, main_indptr = _ragged_from_rows(main_rows, device) + + extra_rows: list[list[int]] | None = None + extra_cache: torch.Tensor | None = None + extra_indices: torch.Tensor | None = None + extra_indptr: torch.Tensor | None = None + if with_extra: + rows = [[1, 3, 0, 5, 2, 4], [3, 0, 6]] + extra_kv = torch.randn(7, HEAD_DIM, dtype=torch.bfloat16, device=device) * 0.125 + extra_rows = rows + extra_cache = _pack_fp8_ds_mla_cache(extra_kv, block_size, is_extra=True) + extra_indices, extra_indptr = _ragged_from_rows(rows, device) + + attn_sink = ( + torch.tensor([-0.1, 0.0, 0.1], dtype=torch.float32, device=device) + if with_sink + else None + ) + scale = HEAD_DIM**-0.5 + + # Pin the split count so each parametrized value is exercised deterministically. + monkeypatch.setattr(mod, "_decode_num_splits", lambda *args, **kwargs: num_splits) + + actual = mod._rocm_sparse_attn_decode_ragged_triton( + q=q, + main_cache=main_cache, + main_indices=main_indices, + main_indptr=main_indptr, + scale=scale, + attn_sink=attn_sink, + nope_head_dim=NOPE_HEAD_DIM, + rope_head_dim=ROPE_HEAD_DIM, + extra_cache=extra_cache, + extra_indices=extra_indices, + extra_indptr=extra_indptr, + ) + expected = _ref_sparse_decode_ragged( + q=q, + main_cache=main_cache, + main_rows=main_rows, + scale=scale, + attn_sink=attn_sink, + block_size=block_size, + extra_cache=extra_cache, + extra_rows=extra_rows, + ) + + torch.testing.assert_close(actual, expected, atol=2e-2, rtol=2e-2) + + +# --------------------------------------------------------------------------- +# o-projection: fused inverse-RoPE + cached bf16 wo_a (rocm_inv_rope_einsum) +# --------------------------------------------------------------------------- + + +# Cache rows = max_position_embeddings * scaling_factor. +_ROTARY_MAX_POS = 1024 +_ROTARY_SCALING_FACTOR = 4.0 +_ROTARY_CACHE_LEN = int(_ROTARY_MAX_POS * _ROTARY_SCALING_FACTOR) + + +def _make_dsv4_rotary(device: torch.device): + """The official DSv4 rotary embedding, sized down for unit tests.""" + from vllm.model_executor.layers.rotary_embedding.deepseek_scaling_rope import ( + DeepseekV4ScalingRotaryEmbedding, + ) + + # The model loader constructs layers under a default-device context; + # mirror that so the fp32 cos_sin_cache lands on the GPU. + with torch.device(device): + rotary_emb = DeepseekV4ScalingRotaryEmbedding( + head_size=ROPE_HEAD_DIM, + rotary_dim=ROPE_HEAD_DIM, + max_position_embeddings=_ROTARY_MAX_POS, + base=10000, + is_neox_style=False, + scaling_factor=_ROTARY_SCALING_FACTOR, + dtype=torch.bfloat16, + mscale=1.0, + mscale_all_dim=1.0, + ) + rotary_emb = rotary_emb.to(device) + assert rotary_emb.cos_sin_cache.shape == (_ROTARY_CACHE_LEN, ROPE_HEAD_DIM) + return rotary_emb + + +def _inv_rope_via_rotary_native( + rotary_emb: torch.nn.Module, + o: torch.Tensor, + positions: torch.Tensor, +) -> torch.Tensor: + """Reference: the official ``forward_native(inverse=True)`` path.""" + expected, _ = rotary_emb.forward_native(positions, o.clone(), None, inverse=True) + return expected.to(torch.bfloat16) + + +class _FakeWoA(torch.nn.Module): + """Stand-in for the wo_a linear layer holding the (optionally fp8) weight.""" + + def __init__( + self, weight: torch.Tensor, weight_scale_inv: torch.Tensor | None = None + ) -> None: + super().__init__() + self.weight = weight + if weight_scale_inv is not None: + self.weight_scale_inv = weight_scale_inv + + +@pytest.mark.parametrize("num_tokens", [1, 7, 64]) +@pytest.mark.parametrize("num_heads", [1, 8]) +@pytest.mark.parametrize("pos_dtype", [torch.int32, torch.int64]) +@torch.inference_mode() +def test_fused_inverse_rope_gptj_matches_rotary_native( + num_tokens: int, num_heads: int, pos_dtype: torch.dtype, default_vllm_config +) -> None: + from vllm.v1.attention.ops.rocm_aiter_mla_sparse import _fused_inverse_rope_gptj + + device = torch.device("cuda") + torch.manual_seed(0) + rotary_emb = _make_dsv4_rotary(device) + o = torch.randn( + num_tokens, num_heads, HEAD_DIM, dtype=torch.bfloat16, device=device + ) + positions = torch.randint( + 0, _ROTARY_CACHE_LEN, (num_tokens,), dtype=pos_dtype, device=device + ) + + actual = _fused_inverse_rope_gptj( + o, positions, rotary_emb.cos_sin_cache, ROPE_HEAD_DIM + ) + expected = _inv_rope_via_rotary_native(rotary_emb, o, positions) + + assert actual.dtype == torch.bfloat16 + assert actual.shape == o.shape + # NoPE lanes are a pure bf16 passthrough -> must be bit-exact. + assert torch.equal(actual[..., :NOPE_HEAD_DIM], expected[..., :NOPE_HEAD_DIM]) + # RoPE lanes: tolerate at most ~1 bf16 ulp from fp32 fma ordering. + torch.testing.assert_close(actual, expected, atol=2e-2, rtol=2e-2) + + +@torch.inference_mode() +def test_fused_inverse_rope_gptj_empty(default_vllm_config) -> None: + from vllm.v1.attention.ops.rocm_aiter_mla_sparse import _fused_inverse_rope_gptj + + device = torch.device("cuda") + rotary_emb = _make_dsv4_rotary(device) + o = torch.empty(0, 8, HEAD_DIM, dtype=torch.bfloat16, device=device) + positions = torch.empty(0, dtype=torch.int32, device=device) + + out = _fused_inverse_rope_gptj( + o, positions, rotary_emb.cos_sin_cache, ROPE_HEAD_DIM + ) + assert out.shape == (0, 8, HEAD_DIM) + assert out.dtype == torch.bfloat16 + + +@torch.inference_mode() +def test_rocm_inv_rope_einsum_matches_rotary_native(default_vllm_config) -> None: + from vllm.v1.attention.ops.rocm_aiter_mla_sparse import rocm_inv_rope_einsum + + device = torch.device("cuda") + torch.manual_seed(2) + num_tokens, num_heads = 5, 8 + n_local_groups = num_heads + o_lora_rank = 16 + hidden_dim = num_heads * HEAD_DIM // n_local_groups # 512 + + rotary_emb = _make_dsv4_rotary(device) + o = ( + torch.randn( + num_tokens, num_heads, HEAD_DIM, dtype=torch.bfloat16, device=device + ) + * 0.125 + ) + positions = torch.randint( + 0, _ROTARY_CACHE_LEN, (num_tokens,), dtype=torch.int32, device=device + ) + weight = ( + torch.randn(n_local_groups * o_lora_rank, hidden_dim, device=device) * 0.125 + ).to(torch.bfloat16) + wo_a = _FakeWoA(weight) + + actual = rocm_inv_rope_einsum( + rotary_emb, o, positions, ROPE_HEAD_DIM, n_local_groups, o_lora_rank, wo_a + ) + + o_ref = _inv_rope_via_rotary_native(rotary_emb, o, positions) + o_ref = o_ref.view(num_tokens, n_local_groups, -1) + wo_a_ref = weight.view(n_local_groups, o_lora_rank, hidden_dim).to(torch.bfloat16) + expected = torch.einsum("tgd,grd->tgr", o_ref, wo_a_ref) + + assert actual.shape == (num_tokens, n_local_groups, o_lora_rank) + torch.testing.assert_close(actual, expected, atol=2e-2, rtol=2e-2) + + +@torch.inference_mode() +def test_get_cached_wo_a_bf16_plain_caches() -> None: + from vllm.v1.attention.ops.rocm_aiter_mla_sparse import _get_cached_wo_a_bf16 + + device = torch.device("cuda") + torch.manual_seed(4) + n_local_groups, o_lora_rank, hidden_dim = 2, 4, 8 + weight = torch.randn( + n_local_groups * o_lora_rank, hidden_dim, dtype=torch.bfloat16, device=device + ) + wo_a = _FakeWoA(weight) + + out1 = _get_cached_wo_a_bf16(wo_a, n_local_groups, o_lora_rank, hidden_dim) + expected = weight.view(n_local_groups, o_lora_rank, hidden_dim).to(torch.bfloat16) + assert out1.shape == (n_local_groups, o_lora_rank, hidden_dim) + torch.testing.assert_close(out1, expected, atol=0, rtol=0) + assert hasattr(wo_a, "_dsv4_wo_a_bf16") + + # Mutate the source weight: the cached tensor must be returned unchanged + # (proving the dequant is not recomputed per call). + wo_a.weight.zero_() + out2 = _get_cached_wo_a_bf16(wo_a, n_local_groups, o_lora_rank, hidden_dim) + assert out2 is out1 + torch.testing.assert_close(out2, expected, atol=0, rtol=0) + + +@torch.inference_mode() +def test_get_cached_wo_a_bf16_fp8_blockscale_caches() -> None: + from vllm.v1.attention.ops.rocm_aiter_mla_sparse import _get_cached_wo_a_bf16 + + device = torch.device("cuda") + torch.manual_seed(5) + n_local_groups, o_lora_rank, hidden_dim = 2, 4, 8 + row_block, col_block = 2, 2 + row_blocks = o_lora_rank // row_block + col_blocks = hidden_dim // col_block + + fp8_dtype = current_platform.fp8_dtype() + weight_f32 = ( + torch.randn( + n_local_groups, o_lora_rank, hidden_dim, dtype=torch.float32, device=device + ) + * 0.1 + ) + weight_fp8 = weight_f32.to(fp8_dtype) + scale = ( + torch.rand( + n_local_groups, row_blocks, col_blocks, dtype=torch.float32, device=device + ) + * 0.5 + + 0.5 + ) + wo_a = _FakeWoA( + weight_fp8.reshape(n_local_groups * o_lora_rank, hidden_dim), + weight_scale_inv=scale.reshape(n_local_groups * row_blocks, col_blocks), + ) + + out = _get_cached_wo_a_bf16(wo_a, n_local_groups, o_lora_rank, hidden_dim) + + scale_full = scale.repeat_interleave(row_block, dim=-2).repeat_interleave( + col_block, dim=-1 + ) + expected = (weight_fp8.to(torch.float32) * scale_full).to(torch.bfloat16) + assert out.shape == (n_local_groups, o_lora_rank, hidden_dim) + torch.testing.assert_close(out, expected, atol=0, rtol=0) + + # Second call returns the same cached object. + assert _get_cached_wo_a_bf16(wo_a, n_local_groups, o_lora_rank, hidden_dim) is out diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_triton_decode_attention.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_triton_decode_attention.py new file mode 100644 index 0000000000000000000000000000000000000000..f746f613b0d7cbcac42bfca153e3e693cdb1a75f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_triton_decode_attention.py @@ -0,0 +1,325 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm.platforms import current_platform +from vllm.utils.math_utils import cdiv +from vllm.v1.attention.ops.triton_decode_attention import decode_attention_fwd + +DEVICE_TYPE = current_platform.device_type + + +@pytest.mark.parametrize("B", [3, 5]) +@pytest.mark.parametrize("L", [1027, 1025]) +@pytest.mark.parametrize("H_Q", [32]) +@pytest.mark.parametrize("H_KV", [32, 8]) +@pytest.mark.parametrize("D_QK", [128, 192, 576]) +@pytest.mark.parametrize("D_V", [128, 512]) +@pytest.mark.parametrize("CACHE_SIZE", [16384]) +@pytest.mark.parametrize("PAGE_SIZE", [1, 16]) +def test_decode_attention(B, L, H_Q, H_KV, D_QK, D_V, CACHE_SIZE, PAGE_SIZE): + assert CACHE_SIZE % PAGE_SIZE == 0 + dtype = torch.bfloat16 + seq_len = L # This represents the number of tokens already in the sequence + sm_scale = 1.0 / (D_QK**0.5) + num_kv_splits = 8 + + num_pages_per_batch = cdiv(seq_len, PAGE_SIZE) + req_to_page = torch.randint( + 0, CACHE_SIZE // PAGE_SIZE, (B, num_pages_per_batch, 1), device=DEVICE_TYPE + ) + req_to_token = req_to_page * PAGE_SIZE + req_to_token = req_to_token.expand(B, num_pages_per_batch, PAGE_SIZE) + req_to_token = req_to_token + torch.arange(PAGE_SIZE, device=DEVICE_TYPE).view( + 1, 1, -1 + ) + req_to_token = req_to_token.view(B, -1) + req_to_token = req_to_token[:, :seq_len].contiguous() + + # q represents the new token being generated, one per batch + q = torch.randn(B, H_Q, D_QK, dtype=dtype, device=DEVICE_TYPE) + + # k_buffer and v_buffer represent all previous tokens + # Page size is 1. + k_buffer = torch.randn(CACHE_SIZE, H_KV, D_QK, dtype=dtype, device=DEVICE_TYPE) + v_buffer = torch.randn(CACHE_SIZE, H_KV, D_V, dtype=dtype, device=DEVICE_TYPE) + + # o will have the same shape as q + o = torch.zeros(B, H_Q, D_V, dtype=dtype, device=DEVICE_TYPE) + + lse = torch.zeros(B, H_Q, dtype=dtype, device=DEVICE_TYPE) + + b_seq_len = torch.full((B,), seq_len, device=DEVICE_TYPE) + + attn_logits = torch.empty( + (B, H_Q, num_kv_splits, D_V + 1), + dtype=torch.float32, + device=DEVICE_TYPE, + ) + + # Call the original implementation. + decode_attention_fwd( + q, + k_buffer, + v_buffer, + o, + lse, + req_to_token, + b_seq_len, + attn_logits, + num_kv_splits, + sm_scale, + ) + + # Page size can be larger than 1. + k_buffer = k_buffer.view(CACHE_SIZE // PAGE_SIZE, PAGE_SIZE, H_KV, D_QK) + v_buffer = v_buffer.view(CACHE_SIZE // PAGE_SIZE, PAGE_SIZE, H_KV, D_V) + + o1 = torch.zeros_like(o) + lse1 = torch.zeros_like(lse) + + decode_attention_fwd( + q, + k_buffer, + v_buffer, + o1, + lse1, + req_to_page, + b_seq_len, + attn_logits, + num_kv_splits, + sm_scale, + PAGE_SIZE, + ) + + assert torch.allclose(o, o1) + + +def _quantize_to_fp8(tensor: torch.Tensor): + """Quantize a BF16 tensor to FP8 e4m3fn with per-tensor scale. + + Returns (fp8_tensor, scale) where: + fp8_tensor ≈ tensor / scale (stored as float8_e4m3fn) + tensor ≈ fp8_tensor.to(float32) * scale (dequantized) + """ + amax = tensor.abs().amax() + # float8_e4m3fn max representable value is 448.0 + scale = (amax / 448.0).clamp(min=1e-12).to(torch.float32) + fp8_tensor = ( + (tensor.to(torch.float32) / scale).clamp(-448.0, 448.0).to(torch.float8_e4m3fn) + ) + return fp8_tensor, scale + + +@pytest.mark.parametrize("B", [3]) +@pytest.mark.parametrize("L", [1025]) +@pytest.mark.parametrize("H_Q", [32]) +@pytest.mark.parametrize("H_KV", [32, 8]) +@pytest.mark.parametrize("D_QK", [128, 576]) +@pytest.mark.parametrize("D_V", [128, 512]) +@pytest.mark.parametrize("CACHE_SIZE", [16384]) +@pytest.mark.parametrize("PAGE_SIZE", [1, 16]) +def test_decode_attention_fp8(B, L, H_Q, H_KV, D_QK, D_V, CACHE_SIZE, PAGE_SIZE): + """Test FP8 KV cache path: quantize K/V to FP8, run kernel with scales, + and compare against BF16 reference output.""" + assert CACHE_SIZE % PAGE_SIZE == 0 + dtype = torch.bfloat16 + seq_len = L + sm_scale = 1.0 / (D_QK**0.5) + num_kv_splits = 8 + + num_pages_per_batch = cdiv(seq_len, PAGE_SIZE) + req_to_page = torch.randint( + 0, CACHE_SIZE // PAGE_SIZE, (B, num_pages_per_batch, 1), device=DEVICE_TYPE + ) + req_to_token = req_to_page * PAGE_SIZE + req_to_token = req_to_token.expand(B, num_pages_per_batch, PAGE_SIZE) + req_to_token = req_to_token + torch.arange(PAGE_SIZE, device=DEVICE_TYPE).view( + 1, 1, -1 + ) + req_to_token = req_to_token.view(B, -1) + req_to_token = req_to_token[:, :seq_len].contiguous() + + q = torch.randn(B, H_Q, D_QK, dtype=dtype, device=DEVICE_TYPE) + + # Create BF16 K/V as reference + k_bf16 = torch.randn(CACHE_SIZE, H_KV, D_QK, dtype=dtype, device=DEVICE_TYPE) + v_bf16 = torch.randn(CACHE_SIZE, H_KV, D_V, dtype=dtype, device=DEVICE_TYPE) + + # --- BF16 reference --- + o_ref = torch.zeros(B, H_Q, D_V, dtype=dtype, device=DEVICE_TYPE) + lse_ref = torch.zeros(B, H_Q, dtype=dtype, device=DEVICE_TYPE) + attn_logits = torch.empty( + (B, H_Q, num_kv_splits, D_V + 1), dtype=torch.float32, device=DEVICE_TYPE + ) + + if PAGE_SIZE == 1: + decode_attention_fwd( + q, + k_bf16, + v_bf16, + o_ref, + lse_ref, + req_to_token, + b_seq_len=torch.full((B,), seq_len, device=DEVICE_TYPE), + attn_logits=attn_logits, + num_kv_splits=num_kv_splits, + sm_scale=sm_scale, + ) + else: + k_paged = k_bf16.view(CACHE_SIZE // PAGE_SIZE, PAGE_SIZE, H_KV, D_QK) + v_paged = v_bf16.view(CACHE_SIZE // PAGE_SIZE, PAGE_SIZE, H_KV, D_V) + decode_attention_fwd( + q, + k_paged, + v_paged, + o_ref, + lse_ref, + req_to_page, + b_seq_len=torch.full((B,), seq_len, device=DEVICE_TYPE), + attn_logits=attn_logits, + num_kv_splits=num_kv_splits, + sm_scale=sm_scale, + page_size=PAGE_SIZE, + ) + + # --- FP8 path --- + k_fp8, k_scale = _quantize_to_fp8(k_bf16) + v_fp8, v_scale = _quantize_to_fp8(v_bf16) + + o_fp8 = torch.zeros(B, H_Q, D_V, dtype=dtype, device=DEVICE_TYPE) + lse_fp8 = torch.zeros(B, H_Q, dtype=dtype, device=DEVICE_TYPE) + attn_logits_fp8 = torch.empty( + (B, H_Q, num_kv_splits, D_V + 1), dtype=torch.float32, device=DEVICE_TYPE + ) + + if PAGE_SIZE == 1: + decode_attention_fwd( + q, + k_fp8, + v_fp8, + o_fp8, + lse_fp8, + req_to_token, + b_seq_len=torch.full((B,), seq_len, device=DEVICE_TYPE), + attn_logits=attn_logits_fp8, + num_kv_splits=num_kv_splits, + sm_scale=sm_scale, + k_scale=k_scale, + v_scale=v_scale, + ) + else: + k_fp8_paged = k_fp8.view(CACHE_SIZE // PAGE_SIZE, PAGE_SIZE, H_KV, D_QK) + v_fp8_paged = v_fp8.view(CACHE_SIZE // PAGE_SIZE, PAGE_SIZE, H_KV, D_V) + decode_attention_fwd( + q, + k_fp8_paged, + v_fp8_paged, + o_fp8, + lse_fp8, + req_to_page, + b_seq_len=torch.full((B,), seq_len, device=DEVICE_TYPE), + attn_logits=attn_logits_fp8, + num_kv_splits=num_kv_splits, + sm_scale=sm_scale, + page_size=PAGE_SIZE, + k_scale=k_scale, + v_scale=v_scale, + ) + + # FP8 tolerances match test_mla_backends.py test_backend_correctness. + torch.testing.assert_close(o_ref, o_fp8, atol=5e-1, rtol=1e-2) + + +@pytest.mark.parametrize( + "H_Q,H_KV,D_QK,D_V,is_mla", + [ + (16, 1, 576, 512, True), # MLA path (grouped kernel, v = trans(k)) + (32, 8, 128, 128, False), # GQA path (grouped kernel) + (32, 32, 128, 128, False), # MHA path (normal kernel) + ], +) +@pytest.mark.parametrize("PAGE_SIZE", [16]) +def test_decode_attention_cross_layer_view(H_Q, H_KV, D_QK, D_V, is_mla, PAGE_SIZE): + """The kernel must honor the cache's page-dim stride, not assume pages are + packed back-to-back. A per-layer view into a cross-layer (block-major) + cache has stride(0) inflated by num_layers; outputs must match a + contiguous cache holding the same data exactly.""" + B = 3 + seq_len = 1027 + CACHE_SIZE = 16384 + NUM_LAYERS = 3 + LAYER_IDX = 1 + dtype = torch.bfloat16 + sm_scale = 1.0 / (D_QK**0.5) + num_kv_splits = 8 + num_pages = CACHE_SIZE // PAGE_SIZE + + num_pages_per_batch = cdiv(seq_len, PAGE_SIZE) + req_to_page = torch.randint( + 0, num_pages, (B, num_pages_per_batch), device=DEVICE_TYPE + ) + + q = torch.randn(B, H_Q, D_QK, dtype=dtype, device=DEVICE_TYPE) + b_seq_len = torch.full((B,), seq_len, device=DEVICE_TYPE) + + # Reference: contiguous paged cache. + k_ref = torch.randn( + num_pages, PAGE_SIZE, H_KV, D_QK, dtype=dtype, device=DEVICE_TYPE + ) + if is_mla: + v_ref = k_ref[..., :D_V] + else: + v_ref = torch.randn( + num_pages, PAGE_SIZE, H_KV, D_V, dtype=dtype, device=DEVICE_TYPE + ) + + # Cross-layer cache: all layers' pages for a block are adjacent. The + # per-layer view has the same shape as the contiguous cache but + # stride(0) is NUM_LAYERS x larger. Neighbor layers hold random data so + # any packed-pages addressing reads garbage rather than zeros. + k_xl = torch.randn( + num_pages, NUM_LAYERS, PAGE_SIZE, H_KV, D_QK, dtype=dtype, device=DEVICE_TYPE + ) + k_view = k_xl[:, LAYER_IDX] + k_view.copy_(k_ref) + assert k_view.stride(0) == NUM_LAYERS * PAGE_SIZE * H_KV * D_QK + if is_mla: + v_view = k_view[..., :D_V] + else: + v_xl = torch.randn( + num_pages, NUM_LAYERS, PAGE_SIZE, H_KV, D_V, dtype=dtype, device=DEVICE_TYPE + ) + v_view = v_xl[:, LAYER_IDX] + v_view.copy_(v_ref) + + def run(k_buffer, v_buffer): + o = torch.zeros(B, H_Q, D_V, dtype=dtype, device=DEVICE_TYPE) + lse = torch.zeros(B, H_Q, dtype=dtype, device=DEVICE_TYPE) + attn_logits = torch.empty( + (B, H_Q, num_kv_splits, D_V + 1), dtype=torch.float32, device=DEVICE_TYPE + ) + decode_attention_fwd( + q, + k_buffer, + v_buffer, + o, + lse, + req_to_page, + b_seq_len, + attn_logits, + num_kv_splits, + sm_scale, + PAGE_SIZE, + is_mla=is_mla, + ) + return o, lse + + o_ref, lse_ref = run(k_ref, v_ref) + o_xl, lse_xl = run(k_view, v_view) + + # Same data and same compute order; only addressing differs. + assert torch.equal(o_ref, o_xl) + assert torch.equal(lse_ref, lse_xl) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_triton_prefill_attention.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_triton_prefill_attention.py new file mode 100644 index 0000000000000000000000000000000000000000..28a8c9fde895c27fcc394dce90c2d2bac67fb987 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_triton_prefill_attention.py @@ -0,0 +1,232 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch +import torch.nn.functional as F + +from vllm.platforms import current_platform +from vllm.v1.attention.ops.triton_prefill_attention import context_attention_fwd + +DEVICE_TYPE = current_platform.device_type + + +def ref_masked_attention( + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + is_causal: bool = True, + sliding_window_q: int | None = None, + sliding_window_k: int | None = None, +) -> torch.Tensor: + """Reference implementation using PyTorch SDPA.""" + # q, k, v: [total_tokens, num_heads, head_dim] + # SDPA expects [batch, num_heads, seq_len, head_dim] + + total_tokens = q.shape[0] + + # Add batch dimension and transpose + q = q.unsqueeze(0).transpose(1, 2) # [1, num_heads, total_tokens, head_dim] + k = k.unsqueeze(0).transpose(1, 2) # [1, num_heads, total_tokens, head_dim] + v = v.unsqueeze(0).transpose(1, 2) # [1, num_heads, total_tokens, head_dim] + + # Create attention mask if needed + attn_mask = None + use_causal = is_causal + + # If we have sliding window or need custom masking, create explicit mask + sliding_window_q = sliding_window_q if sliding_window_q is not None else 0 + sliding_window_k = sliding_window_k if sliding_window_k is not None else 0 + if (sliding_window_q > 0) or (sliding_window_k > 0): + # Position indices + pos_q = torch.arange(total_tokens, device=q.device).unsqueeze(1) + pos_k = torch.arange(total_tokens, device=q.device).unsqueeze(0) + + # Start with valid mask (False = no masking) + mask = torch.ones( + (total_tokens, total_tokens), dtype=torch.bool, device=q.device + ) + + # Apply causal mask + if is_causal: + mask = mask & (pos_q >= pos_k) + + # Apply sliding window masks + sliding_window_mask = torch.ones_like(mask) + if sliding_window_q > 0: + sliding_window_mask &= pos_q - pos_k <= sliding_window_q + + if sliding_window_k > 0: + sliding_window_mask &= pos_k - pos_q <= sliding_window_k + + mask = mask & sliding_window_mask + + attn_mask = torch.where(mask, 0.0, float("-inf")).to(q.dtype) + use_causal = False # Don't use is_causal when providing explicit mask + + # Use SDPA + output = F.scaled_dot_product_attention( + q, k, v, attn_mask=attn_mask, is_causal=use_causal, dropout_p=0.0 + ) + + # Convert back to original shape: [total_tokens, num_heads, head_dim] + output = output.transpose(1, 2).squeeze(0) + + return output + + +@pytest.mark.parametrize("B", [5]) +@pytest.mark.parametrize("max_seq_len", [1024]) +@pytest.mark.parametrize("H_Q", [32]) +@pytest.mark.parametrize("H_KV", [32, 8]) +@pytest.mark.parametrize("D", [128]) +@pytest.mark.parametrize("is_causal", [True, False]) +@pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16]) +def test_context_attention( + B: int, + max_seq_len: int, + H_Q: int, + H_KV: int, + D: int, + is_causal: bool, + dtype: torch.dtype, +): + """Test basic context attention without sliding window.""" + torch.manual_seed(42) + + # Generate random sequence lengths for each batch + seq_lens = torch.randint( + max_seq_len // 2, max_seq_len + 1, (B,), device=DEVICE_TYPE + ) + total_tokens = seq_lens.sum().item() + + # Create batch start locations + b_start_loc = torch.zeros(B, dtype=torch.int32, device=DEVICE_TYPE) + b_start_loc[1:] = torch.cumsum(seq_lens[:-1], dim=0) + + # Create input tensors + q = torch.randn(total_tokens, H_Q, D, dtype=dtype, device=DEVICE_TYPE) + k = torch.randn(total_tokens, H_KV, D, dtype=dtype, device=DEVICE_TYPE) + v = torch.randn(total_tokens, H_KV, D, dtype=dtype, device=DEVICE_TYPE) + o = torch.zeros_like(q) + + # Call Triton kernel + context_attention_fwd( + q, + k, + v, + o, + b_start_loc, + seq_lens, + max_seq_len, + is_causal=is_causal, + sliding_window_q=None, + sliding_window_k=None, + ) + + # Compute reference output for each sequence in batch + o_ref = torch.zeros_like(q) + for i in range(B): + start = b_start_loc[i].item() + end = start + seq_lens[i].item() + + q_seq = q[start:end] + k_seq = k[start:end] + v_seq = v[start:end] + + # Expand KV heads if using GQA + if H_Q != H_KV: + kv_group_num = H_Q // H_KV + k_seq = k_seq.repeat_interleave(kv_group_num, dim=1) + v_seq = v_seq.repeat_interleave(kv_group_num, dim=1) + + o_ref[start:end] = ref_masked_attention( + q_seq, + k_seq, + v_seq, + is_causal=is_causal, + sliding_window_q=None, + sliding_window_k=None, + ) + + # Compare outputs + torch.testing.assert_close(o, o_ref, rtol=1e-2, atol=1e-2) + + +@pytest.mark.parametrize("B", [4]) +@pytest.mark.parametrize("max_seq_len", [1024]) +@pytest.mark.parametrize("H_Q", [32]) +@pytest.mark.parametrize("H_KV", [32, 8]) +@pytest.mark.parametrize("D", [128]) +@pytest.mark.parametrize("sliding_window", [(32, 32), (32, 0), (0, 32)]) +@pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16]) +def test_context_attention_sliding_window( + B: int, + max_seq_len: int, + H_Q: int, + H_KV: int, + D: int, + sliding_window: tuple[int, int], + dtype: torch.dtype, +): + sliding_window_q, sliding_window_k = sliding_window + """Test context attention with sliding window.""" + torch.manual_seed(42) + + # Generate random sequence lengths for each batch + seq_lens = torch.randint( + max_seq_len // 2, max_seq_len + 1, (B,), device=DEVICE_TYPE + ) + total_tokens = seq_lens.sum().item() + + # Create batch start locations + b_start_loc = torch.zeros(B, dtype=torch.int32, device=DEVICE_TYPE) + b_start_loc[1:] = torch.cumsum(seq_lens[:-1], dim=0) + + # Create input tensors + q = torch.randn(total_tokens, H_Q, D, dtype=dtype, device=DEVICE_TYPE) + k = torch.randn(total_tokens, H_KV, D, dtype=dtype, device=DEVICE_TYPE) + v = torch.randn(total_tokens, H_KV, D, dtype=dtype, device=DEVICE_TYPE) + o = torch.zeros_like(q) + + # Call Triton kernel + context_attention_fwd( + q, + k, + v, + o, + b_start_loc, + seq_lens, + max_seq_len, + is_causal=False, + sliding_window_q=sliding_window_q, + sliding_window_k=sliding_window_k, + ) + + # Compute reference output for each sequence in batch + o_ref = torch.zeros_like(q) + for i in range(B): + start = b_start_loc[i].item() + end = start + seq_lens[i].item() + + q_seq = q[start:end] + k_seq = k[start:end] + v_seq = v[start:end] + + # Expand KV heads if using GQA + if H_Q != H_KV: + kv_group_num = H_Q // H_KV + k_seq = k_seq.repeat_interleave(kv_group_num, dim=1) + v_seq = v_seq.repeat_interleave(kv_group_num, dim=1) + + o_ref[start:end] = ref_masked_attention( + q_seq, + k_seq, + v_seq, + is_causal=False, + sliding_window_q=sliding_window_q if sliding_window_q > 0 else None, + sliding_window_k=sliding_window_k if sliding_window_k > 0 else None, + ) + + # Compare outputs + torch.testing.assert_close(o, o_ref, rtol=2e-2, atol=2e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_triton_unified_attention.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_triton_unified_attention.py new file mode 100644 index 0000000000000000000000000000000000000000..e889fc261466232a36be3b9aa1ee1dfb3629225a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_triton_unified_attention.py @@ -0,0 +1,645 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import pytest +import torch + +from vllm.platforms import current_platform +from vllm.utils.math_utils import next_power_of_2 +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.attention.ops.triton_unified_attention import unified_attention +from vllm.v1.kv_cache_interface import KVQuantMode + +DEVICE_TYPE = current_platform.device_type + +NUM_HEADS = [(4, 4), (8, 2), (5, 1)] +HEAD_SIZES = [128, 256] +BLOCK_SIZES = [16] + +DTYPES = [torch.bfloat16] +QDTYPES = [None, current_platform.fp8_dtype()] +FP8_DTYPE = current_platform.fp8_dtype() + +# one value large enough to test overflow in index calculation. +# one value small enough to test the schema op check +NUM_BLOCKS = [32768, 2048] + +# 0: use 2D kernel for decode +# 8: use 3D kernel for decode +SEQ_THRESHOLD_3D_VALUES = [0, 8] + + +def ref_paged_attn( + query: torch.Tensor, + key_cache: torch.Tensor, + value_cache: torch.Tensor, + query_lens: list[int], + kv_lens: list[int], + block_tables: torch.Tensor, + scale: float, + sliding_window: int | None = None, + soft_cap: float | None = None, +) -> torch.Tensor: + num_seqs = len(query_lens) + block_tables = block_tables.cpu().numpy() + _, block_size, num_kv_heads, head_size = key_cache.shape + + outputs: list[torch.Tensor] = [] + start_idx = 0 + for i in range(num_seqs): + query_len = query_lens[i] + kv_len = kv_lens[i] + q = query[start_idx : start_idx + query_len] + q *= scale + + num_kv_blocks = (kv_len + block_size - 1) // block_size + block_indices = block_tables[i, :num_kv_blocks] + + k = key_cache[block_indices].view(-1, num_kv_heads, head_size) + k = k[:kv_len] + v = value_cache[block_indices].view(-1, num_kv_heads, head_size) + v = v[:kv_len] + + if q.shape[1] != k.shape[1]: + k = torch.repeat_interleave(k, q.shape[1] // k.shape[1], dim=1) + v = torch.repeat_interleave(v, q.shape[1] // v.shape[1], dim=1) + attn = torch.einsum("qhd,khd->hqk", q, k).float() + empty_mask = torch.ones(query_len, kv_len) + mask = torch.triu(empty_mask, diagonal=kv_len - query_len + 1).bool() + if sliding_window is not None: + sliding_window_mask = ( + torch.triu( + empty_mask, diagonal=kv_len - (query_len + sliding_window) + 1 + ) + .bool() + .logical_not() + ) + mask |= sliding_window_mask + if soft_cap is not None and soft_cap > 0: + attn = soft_cap * torch.tanh(attn / soft_cap) + attn.masked_fill_(mask, float("-inf")) + attn = torch.softmax(attn, dim=-1).to(v.dtype) + out = torch.einsum("hqk,khd->qhd", attn, v) + + outputs.append(out) + start_idx += query_len + + return torch.cat(outputs, dim=0) + + +@pytest.mark.parametrize( + "seq_lens", [[(1, 1328), (5, 18), (129, 463)], [(1, 523), (1, 37), (1, 2011)]] +) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("sliding_window", [None, 64, 128, 256]) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("soft_cap", [None, 50.0]) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("q_dtype", QDTYPES) +@pytest.mark.parametrize("seq_threshold_3D", SEQ_THRESHOLD_3D_VALUES) +@torch.inference_mode() +def test_triton_unified_attn( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + dtype: torch.dtype, + block_size: int, + soft_cap: float | None, + num_blocks: int, + q_dtype: torch.dtype | None, + seq_threshold_3D: int, +) -> None: + torch.set_default_device(DEVICE_TYPE) + + set_random_seed(0) + num_seqs = len(seq_lens) + query_lens = [x[0] for x in seq_lens] + kv_lens = [x[1] for x in seq_lens] + num_query_heads = num_heads[0] + num_kv_heads = num_heads[1] + assert num_query_heads % num_kv_heads == 0 + max_query_len = max(query_lens) + max_kv_len = max(kv_lens) + window_size = (sliding_window - 1, 0) if sliding_window is not None else (-1, -1) + scale = head_size**-0.5 + + query = torch.randn(sum(query_lens), num_query_heads, head_size, dtype=dtype) + key_cache = torch.randn( + num_blocks, block_size, num_kv_heads, head_size, dtype=dtype + ) + value_cache = torch.randn_like(key_cache) + cu_query_lens = torch.tensor([0] + query_lens, dtype=torch.int32).cumsum( + dim=0, dtype=torch.int32 + ) + kv_lens = torch.tensor(kv_lens, dtype=torch.int32) + + max_num_blocks_per_seq = (max_kv_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, num_blocks, (num_seqs, max_num_blocks_per_seq), dtype=torch.int32 + ) + + output = torch.empty_like(query) + + maybe_quantized_query = query + maybe_quantized_key_cache = key_cache + maybe_quantized_value_cache = value_cache + q_descale = None + k_descale = None + v_descale = None + kv_quant_mode = KVQuantMode.NONE + if q_dtype is not None: + # Use non-1 scales so FP8 Q/K/V descale handling is tested explicitly. + q_scale = torch.tensor(0.75, dtype=torch.float32) + k_scale = torch.tensor(0.5, dtype=torch.float32) + v_scale = torch.tensor(0.25, dtype=torch.float32) + q_descale = q_scale + scale_shape = (num_seqs, num_kv_heads) + k_descale = torch.full(scale_shape, k_scale.item(), dtype=torch.float32) + v_descale = torch.full(scale_shape, v_scale.item(), dtype=torch.float32) + maybe_quantized_query = (query / q_scale).to(q_dtype) + maybe_quantized_key_cache = (key_cache / k_scale).to(q_dtype) + maybe_quantized_value_cache = (value_cache / v_scale).to(q_dtype) + kv_quant_mode = KVQuantMode.FP8_PER_TENSOR + + num_par_softmax_segments = 16 + head_size_padded = next_power_of_2(head_size) + softmax_segm_output = torch.empty( + (seq_threshold_3D, num_query_heads, num_par_softmax_segments, head_size_padded), + dtype=torch.float32, + ) + softmax_segm_max = torch.empty( + (seq_threshold_3D, num_query_heads, num_par_softmax_segments), + dtype=torch.float32, + ) + softmax_segm_expsum = torch.empty( + (seq_threshold_3D, num_query_heads, num_par_softmax_segments), + dtype=torch.float32, + ) + + unified_attention( + q=maybe_quantized_query, + k=maybe_quantized_key_cache, + v=maybe_quantized_value_cache, + out=output, + cu_seqlens_q=cu_query_lens, + seqused_k=kv_lens, + max_seqlen_q=max_query_len, + max_seqlen_k=max_kv_len, + softmax_scale=scale, + causal=True, + window_size=window_size, + block_table=block_tables, + softcap=soft_cap if soft_cap is not None else 0, + q_descale=q_descale, + k_descale=k_descale, + v_descale=v_descale, + seq_threshold_3D=seq_threshold_3D, + num_par_softmax_segments=num_par_softmax_segments, + softmax_segm_output=softmax_segm_output, + softmax_segm_max=softmax_segm_max, + softmax_segm_expsum=softmax_segm_expsum, + kv_quant_mode=kv_quant_mode, + ) + + ref_output = ref_paged_attn( + query=query, + key_cache=key_cache, + value_cache=value_cache, + query_lens=query_lens, + kv_lens=kv_lens, + block_tables=block_tables, + scale=scale, + sliding_window=sliding_window, + soft_cap=soft_cap, + ) + atol, rtol = 1.5e-2, 1e-2 + if q_dtype is not None: + atol, rtol = 1.5e-1, 1.5e-1 + ( + torch.testing.assert_close(output, ref_output, atol=atol, rtol=rtol), + f"{torch.max(torch.abs(output - ref_output))}", + ) + + +@pytest.mark.parametrize( + "seq_lens", [[(1, 1328), (5, 18), (129, 463)], [(1, 523), (1, 37), (1, 2011)]] +) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("seq_threshold_3D", SEQ_THRESHOLD_3D_VALUES) +@torch.inference_mode() +def test_triton_unified_attn_bf16_query_fp8_kv( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + block_size: int, + num_blocks: int, + seq_threshold_3D: int, +) -> None: + """Test bf16 Q with FP8 per-tensor KV cache (dequant via _cast_kv_tile).""" + torch.set_default_device(DEVICE_TYPE) + set_random_seed(0) + + num_seqs = len(seq_lens) + query_lens = [x[0] for x in seq_lens] + kv_lens = [x[1] for x in seq_lens] + num_query_heads = num_heads[0] + num_kv_heads = num_heads[1] + assert num_query_heads % num_kv_heads == 0 + max_query_len = max(query_lens) + max_kv_len = max(kv_lens) + window_size = (-1, -1) + scale = head_size**-0.5 + + dtype = torch.bfloat16 + query = torch.randn(sum(query_lens), num_query_heads, head_size, dtype=dtype) + key_cache = torch.randn( + num_blocks, block_size, num_kv_heads, head_size, dtype=dtype + ) + value_cache = torch.randn_like(key_cache) + + k_scale = torch.tensor(0.5, dtype=torch.float32) + v_scale = torch.tensor(0.25, dtype=torch.float32) + fp8_key_cache = (key_cache / k_scale).to(FP8_DTYPE) + fp8_value_cache = (value_cache / v_scale).to(FP8_DTYPE) + + scale_shape = (num_seqs, num_kv_heads) + k_descale = torch.full(scale_shape, k_scale.item(), dtype=torch.float32) + v_descale = torch.full(scale_shape, v_scale.item(), dtype=torch.float32) + + cu_query_lens = torch.tensor([0] + query_lens, dtype=torch.int32).cumsum( + dim=0, dtype=torch.int32 + ) + kv_lens_t = torch.tensor(kv_lens, dtype=torch.int32) + + max_num_blocks_per_seq = (max_kv_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, num_blocks, (num_seqs, max_num_blocks_per_seq), dtype=torch.int32 + ) + + output = torch.empty_like(query) + + num_par_softmax_segments = 16 + head_size_padded = next_power_of_2(head_size) + softmax_segm_output = torch.empty( + (seq_threshold_3D, num_query_heads, num_par_softmax_segments, head_size_padded), + dtype=torch.float32, + ) + softmax_segm_max = torch.empty( + (seq_threshold_3D, num_query_heads, num_par_softmax_segments), + dtype=torch.float32, + ) + softmax_segm_expsum = torch.empty( + (seq_threshold_3D, num_query_heads, num_par_softmax_segments), + dtype=torch.float32, + ) + + unified_attention( + q=query, + k=fp8_key_cache, + v=fp8_value_cache, + out=output, + cu_seqlens_q=cu_query_lens, + seqused_k=kv_lens_t, + max_seqlen_q=max_query_len, + max_seqlen_k=max_kv_len, + softmax_scale=scale, + causal=True, + window_size=window_size, + block_table=block_tables, + softcap=0, + q_descale=None, + k_descale=k_descale, + v_descale=v_descale, + seq_threshold_3D=seq_threshold_3D, + num_par_softmax_segments=num_par_softmax_segments, + softmax_segm_output=softmax_segm_output, + softmax_segm_max=softmax_segm_max, + softmax_segm_expsum=softmax_segm_expsum, + kv_quant_mode=KVQuantMode.FP8_PER_TENSOR, + ) + + ref_output = ref_paged_attn( + query=query, + key_cache=key_cache, + value_cache=value_cache, + query_lens=query_lens, + kv_lens=kv_lens, + block_tables=block_tables, + scale=scale, + ) + + atol, rtol = 1.5e-1, 1.5e-1 + ( + torch.testing.assert_close(output, ref_output, atol=atol, rtol=rtol), + f"{torch.max(torch.abs(output - ref_output))}", + ) + + +@pytest.mark.parametrize( + "seq_lens", + [ + [(1, 1328), (5, 18), (129, 463)], + [(1, 523), (1, 37), (1, 2011)], + [(1, 1)] * 533, + [(533, 533)] * 533, + ], +) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("sliding_window", [None, 64, 128, 256]) +@pytest.mark.parametrize("soft_cap", [None, 50.0]) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("seq_threshold_3D", SEQ_THRESHOLD_3D_VALUES) +@torch.inference_mode() +def test_triton_unified_attn_fp16_input_fp8_output( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + block_size: int, + soft_cap: float | None, + num_blocks: int, + seq_threshold_3D: int, +) -> None: + """Test with fp16 input and fp8 output using output_scale.""" + torch.set_default_device(DEVICE_TYPE) + + set_random_seed(0) + num_seqs = len(seq_lens) + query_lens = [x[0] for x in seq_lens] + kv_lens = [x[1] for x in seq_lens] + num_query_heads = num_heads[0] + num_kv_heads = num_heads[1] + assert num_query_heads % num_kv_heads == 0 + max_query_len = max(query_lens) + max_kv_len = max(kv_lens) + window_size = (sliding_window - 1, 0) if sliding_window is not None else (-1, -1) + scale = head_size**-0.5 + + dtype = torch.float16 + query = torch.randn(sum(query_lens), num_query_heads, head_size, dtype=dtype) + key_cache = torch.randn( + num_blocks, block_size, num_kv_heads, head_size, dtype=dtype + ) + value_cache = torch.randn_like(key_cache) + cu_query_lens = torch.tensor([0] + query_lens, dtype=torch.int32).cumsum( + dim=0, dtype=torch.int32 + ) + kv_lens_tensor = torch.tensor(kv_lens, dtype=torch.int32) + + max_num_blocks_per_seq = (max_kv_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, num_blocks, (num_seqs, max_num_blocks_per_seq), dtype=torch.int32 + ) + + output = torch.empty(sum(query_lens), num_query_heads, head_size, dtype=FP8_DTYPE) + + output_scale = torch.tensor(0.5, dtype=torch.float32) + + num_par_softmax_segments = 16 + head_size_padded = next_power_of_2(head_size) + softmax_segm_output = torch.empty( + (seq_threshold_3D, num_query_heads, num_par_softmax_segments, head_size_padded), + dtype=torch.float32, + ) + softmax_segm_max = torch.empty( + (seq_threshold_3D, num_query_heads, num_par_softmax_segments), + dtype=torch.float32, + ) + softmax_segm_expsum = torch.empty( + (seq_threshold_3D, num_query_heads, num_par_softmax_segments), + dtype=torch.float32, + ) + + unified_attention( + q=query, + k=key_cache, + v=value_cache, + out=output, + cu_seqlens_q=cu_query_lens, + seqused_k=kv_lens_tensor, + max_seqlen_q=max_query_len, + max_seqlen_k=max_kv_len, + softmax_scale=scale, + causal=True, + window_size=window_size, + block_table=block_tables, + softcap=soft_cap if soft_cap is not None else 0, + q_descale=None, + k_descale=None, + v_descale=None, + output_scale=output_scale, + seq_threshold_3D=seq_threshold_3D, + num_par_softmax_segments=num_par_softmax_segments, + softmax_segm_output=softmax_segm_output, + softmax_segm_max=softmax_segm_max, + softmax_segm_expsum=softmax_segm_expsum, + ) + + ref_output = ref_paged_attn( + query=query, + key_cache=key_cache, + value_cache=value_cache, + query_lens=query_lens, + kv_lens=kv_lens, + block_tables=block_tables, + scale=scale, + sliding_window=sliding_window, + soft_cap=soft_cap, + ) + + output_fp16 = output.to(torch.float32) * output_scale.item() + output_fp16 = output_fp16.to(torch.float16) + + atol, rtol = 2e-1, 2e-1 + ( + torch.testing.assert_close(output_fp16, ref_output, atol=atol, rtol=rtol), + f"{torch.max(torch.abs(output_fp16 - ref_output))}", + ) + + +# USE_TD path covers two head-size regimes: +# - pow2 (HEAD_SIZE == HEAD_SIZE_PADDED): full TD path including Q/O. +# - non-pow2 (96, HEAD_SIZE_PADDED=128): gates USE_TD_QO off — Q load +# and output store fall back to pointer path, KV tile TD load remains. +# The non-pow2 case mirrors real models like Phi-3-mini (head_size=96). +HEAD_SIZES_USE_TD = [128, 256, 96] + + +def _run_use_td_case( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + block_size: int, + sliding_window: int | None, + soft_cap: float | None, + seq_threshold_3D: int, + dtype: torch.dtype = torch.bfloat16, + num_blocks: int = 2048, +) -> None: + """Shared driver for the USE_TD test cases. + + Runs ``unified_attention(..., use_td=True)`` and compares against the + reference paged-attention implementation that the sibling non-TD + tests use. + """ + torch.set_default_device(DEVICE_TYPE) + set_random_seed(0) + + num_seqs = len(seq_lens) + query_lens = [x[0] for x in seq_lens] + kv_lens = [x[1] for x in seq_lens] + num_query_heads, num_kv_heads = num_heads + assert num_query_heads % num_kv_heads == 0 + max_query_len = max(query_lens) + max_kv_len = max(kv_lens) + window_size = (sliding_window - 1, 0) if sliding_window is not None else (-1, -1) + scale = head_size**-0.5 + + query = torch.randn(sum(query_lens), num_query_heads, head_size, dtype=dtype) + key_cache = torch.randn( + num_blocks, block_size, num_kv_heads, head_size, dtype=dtype + ) + value_cache = torch.randn_like(key_cache) + cu_query_lens = torch.tensor([0] + query_lens, dtype=torch.int32).cumsum( + dim=0, dtype=torch.int32 + ) + kv_lens_tensor = torch.tensor(kv_lens, dtype=torch.int32) + + max_num_blocks_per_seq = (max_kv_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, num_blocks, (num_seqs, max_num_blocks_per_seq), dtype=torch.int32 + ) + + output = torch.empty_like(query) + + num_par_softmax_segments = 16 + head_size_padded = next_power_of_2(head_size) + softmax_segm_output = torch.empty( + (seq_threshold_3D, num_query_heads, num_par_softmax_segments, head_size_padded), + dtype=torch.float32, + ) + softmax_segm_max = torch.empty( + (seq_threshold_3D, num_query_heads, num_par_softmax_segments), + dtype=torch.float32, + ) + softmax_segm_expsum = torch.empty( + (seq_threshold_3D, num_query_heads, num_par_softmax_segments), + dtype=torch.float32, + ) + + unified_attention( + q=query, + k=key_cache, + v=value_cache, + out=output, + cu_seqlens_q=cu_query_lens, + seqused_k=kv_lens_tensor, + max_seqlen_q=max_query_len, + max_seqlen_k=max_kv_len, + softmax_scale=scale, + causal=True, + window_size=window_size, + block_table=block_tables, + softcap=soft_cap if soft_cap is not None else 0, + q_descale=None, + k_descale=None, + v_descale=None, + seq_threshold_3D=seq_threshold_3D, + num_par_softmax_segments=num_par_softmax_segments, + softmax_segm_output=softmax_segm_output, + softmax_segm_max=softmax_segm_max, + softmax_segm_expsum=softmax_segm_expsum, + use_td=True, + ) + + ref_output = ref_paged_attn( + query=query, + key_cache=key_cache, + value_cache=value_cache, + query_lens=query_lens, + kv_lens=kv_lens, + block_tables=block_tables, + scale=scale, + sliding_window=sliding_window, + soft_cap=soft_cap, + ) + torch.testing.assert_close(output, ref_output, atol=1.5e-2, rtol=1e-2) + + +@pytest.mark.parametrize( + "seq_lens", [[(1, 1328), (5, 18), (129, 463)], [(1, 523), (1, 37), (1, 2011)]] +) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES_USE_TD) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("sliding_window", [None, 128]) +@pytest.mark.parametrize("soft_cap", [None, 50.0]) +@pytest.mark.parametrize("num_blocks", NUM_BLOCKS) +@pytest.mark.parametrize("seq_threshold_3D", SEQ_THRESHOLD_3D_VALUES) +@torch.inference_mode() +def test_triton_unified_attn_use_td( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_size: int, + sliding_window: int | None, + block_size: int, + soft_cap: float | None, + num_blocks: int, + seq_threshold_3D: int, +) -> None: + """Exercise the USE_TD (tensor-descriptor) Q/K/V load/store path. + + Covers both 2D and 3D kernels via ``seq_threshold_3D``. Two routes + to the USE_TD_QO=False fallback (pointer path for Q/O with TD still + active for KV tile loads): + + - non-pow2 ``num_queries_per_kv`` via ``NUM_HEADS`` entry ``(5, 1)``, + - non-pow2 ``head_size`` via ``HEAD_SIZES_USE_TD`` entry ``96``. + """ + _run_use_td_case( + seq_lens=seq_lens, + num_heads=num_heads, + head_size=head_size, + block_size=block_size, + sliding_window=sliding_window, + soft_cap=soft_cap, + seq_threshold_3D=seq_threshold_3D, + num_blocks=num_blocks, + ) + + +# Prefill-heavy shape: long query drives the prefill kernel path where +# ``_get_tile_size`` returns 32, which exceeds block_size=16 and must be +# clamped by the fix in 'clamp TILE_SIZE to block_size when USE_TD'. +# Only the prefill launch exercises the clamp, so parameterize only over +# the (num_heads, seq_threshold_3D=0) combinations needed to cover it. +@pytest.mark.parametrize("num_heads", [(4, 4), (5, 1)]) +@torch.inference_mode() +def test_triton_unified_attn_use_td_tile_clamp( + num_heads: tuple[int, int], +) -> None: + """Regression guard: ``USE_TD`` needs ``BLOCK_SIZE % TILE_SIZE == 0``. + + With ``block_size=16`` and ``head_size=128`` (non-Gemma3), + ``_get_tile_size`` returns 32 for prefill, which violates the + ``USE_TD`` constraint unless clamped to ``block_size``. Without + the clamp the triton kernel ``static_assert`` fires at compile time. + """ + _run_use_td_case( + seq_lens=[(256, 256), (128, 128)], + num_heads=num_heads, + head_size=128, + block_size=16, + sliding_window=None, + soft_cap=None, + seq_threshold_3D=0, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_triton_unified_attention_diffkv.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_triton_unified_attention_diffkv.py new file mode 100644 index 0000000000000000000000000000000000000000..1fd1b35e87bb9d7aa0993a7404553e8890043884 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_triton_unified_attention_diffkv.py @@ -0,0 +1,189 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Unit tests for the Triton DiffKV unified-attention kernel. +""" + +import pytest +import torch + +from vllm.platforms import current_platform +from vllm.utils.math_utils import next_power_of_2 +from vllm.utils.torch_utils import ( + canonicalize_singleton_dim_strides, + set_random_seed, +) +from vllm.v1.attention.backends.fa_utils import ( + get_flash_attn_version, + is_flash_attn_varlen_func_available, +) +from vllm.v1.attention.ops.triton_unified_attention_diffkv import ( + unified_attention_diffkv, +) + +DEVICE_TYPE = current_platform.device_type + +# (num_query_heads, num_kv_heads): MHA, GQA, and the num_kv_heads==1 +# (degenerate-stride) case. +NUM_HEADS = [(4, 4), (8, 2), (5, 1)] +# (head_size_qk, head_size_v). (192, 128) is the canonical asymmetric +# DiffKV shape; FA4 on Blackwell only supports head_size>128 when it is +# 192, and FA3 on Hopper supports it too -- so this pair is runnable on +# both. (128, 128) keeps the equal-dim path covered through the DiffKV +# kernel. +HEAD_SIZES = [(128, 128), (192, 128)] +BLOCK_SIZES = [16] +DTYPES = [torch.bfloat16] + +NUM_BLOCKS = 2048 + +# 0: 2D decode kernel; 8: 3D (split-KV) decode kernel. +SEQ_THRESHOLD_3D_VALUES = [0, 8] + +NUM_PAR_SOFTMAX_SEGMENTS = 16 + + +def _alloc_segm_buffers(seq_threshold_3D: int, num_query_heads: int, head_size_v: int): + """Allocate the split-KV softmax scratch (last dim == head_size_v).""" + head_size_v_padded = next_power_of_2(head_size_v) + segm_output = torch.empty( + ( + seq_threshold_3D, + num_query_heads, + NUM_PAR_SOFTMAX_SEGMENTS, + head_size_v_padded, + ), + dtype=torch.float32, + ) + segm_max = torch.empty( + (seq_threshold_3D, num_query_heads, NUM_PAR_SOFTMAX_SEGMENTS), + dtype=torch.float32, + ) + segm_expsum = torch.empty( + (seq_threshold_3D, num_query_heads, NUM_PAR_SOFTMAX_SEGMENTS), + dtype=torch.float32, + ) + return segm_output, segm_max, segm_expsum + + +@pytest.mark.parametrize( + "seq_lens", + [ + [(1, 1328), (5, 18), (129, 463)], # mixed prefill + decode + [(1, 523), (1, 37), (1, 2011)], # decode-only (exercises 3D path) + ], +) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_sizes", HEAD_SIZES) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("sliding_window", [None, 128]) +@pytest.mark.parametrize("soft_cap", [None, 50.0]) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seq_threshold_3D", SEQ_THRESHOLD_3D_VALUES) +@torch.inference_mode() +def test_triton_unified_attn_diffkv_vs_fa( + seq_lens: list[tuple[int, int]], + num_heads: tuple[int, int], + head_sizes: tuple[int, int], + sliding_window: int | None, + soft_cap: float | None, + dtype: torch.dtype, + block_size: int, + seq_threshold_3D: int, +) -> None: + head_size_qk, head_size_v = head_sizes + + # DiffKV requires FA3 (Hopper) / FA4 (Blackwell) as the reference. + fa_version = get_flash_attn_version(head_size=head_size_qk, head_size_v=head_size_v) + if not is_flash_attn_varlen_func_available() or fa_version not in (3, 4): + pytest.skip(f"FA DiffKV needs FA3/FA4 (got version {fa_version}).") + + from vllm.v1.attention.backends.fa_utils import flash_attn_varlen_func + + torch.set_default_device(DEVICE_TYPE) + set_random_seed(0) + + num_seqs = len(seq_lens) + query_lens = [x[0] for x in seq_lens] + kv_lens = [x[1] for x in seq_lens] + num_query_heads, num_kv_heads = num_heads + assert num_query_heads % num_kv_heads == 0 + max_query_len = max(query_lens) + max_kv_len = max(kv_lens) + window_size = (sliding_window - 1, 0) if sliding_window is not None else (-1, -1) + scale = head_size_qk**-0.5 + + query = torch.randn(sum(query_lens), num_query_heads, head_size_qk, dtype=dtype) + # Packed KV cache: [num_blocks, block_size, num_kv_heads, hqk + hv]. + kv_cache = torch.randn( + NUM_BLOCKS, + block_size, + num_kv_heads, + head_size_qk + head_size_v, + dtype=dtype, + ) + key_cache = kv_cache[..., :head_size_qk] + value_cache = kv_cache[..., head_size_qk:] + + cu_query_lens = torch.tensor([0] + query_lens, dtype=torch.int32).cumsum( + dim=0, dtype=torch.int32 + ) + kv_lens_t = torch.tensor(kv_lens, dtype=torch.int32) + + max_num_blocks_per_seq = (max_kv_len + block_size - 1) // block_size + block_tables = torch.randint( + 0, NUM_BLOCKS, (num_seqs, max_num_blocks_per_seq), dtype=torch.int32 + ) + + # ---- FlashAttention DiffKV (ground truth) --------------------------- + # Mirror the backend: fix degenerate strides on size-1 dims so FA's + # TMA path sees ≥16-byte-aligned strides (matters for num_kv_heads==1). + fa_k = canonicalize_singleton_dim_strides(key_cache) + fa_v = canonicalize_singleton_dim_strides(value_cache) + fa_out = torch.empty(sum(query_lens), num_query_heads, head_size_v, dtype=dtype) + flash_attn_varlen_func( + q=query, + k=fa_k, + v=fa_v, + out=fa_out, + cu_seqlens_q=cu_query_lens, + max_seqlen_q=max_query_len, + seqused_k=kv_lens_t, + max_seqlen_k=max_kv_len, + softmax_scale=scale, + causal=True, + window_size=list(window_size), + block_table=block_tables, + softcap=soft_cap if soft_cap is not None else 0, + fa_version=fa_version, + ) + + # ---- Triton DiffKV -------------------------------------------------- + segm_output, segm_max, segm_expsum = _alloc_segm_buffers( + seq_threshold_3D, num_query_heads, head_size_v + ) + triton_out = torch.empty(sum(query_lens), num_query_heads, head_size_v, dtype=dtype) + unified_attention_diffkv( + q=query, + k=key_cache, + v=value_cache, + out=triton_out, + cu_seqlens_q=cu_query_lens, + seqused_k=kv_lens_t, + softmax_scale=scale, + causal=True, + window_size=window_size, + block_table=block_tables, + softcap=soft_cap if soft_cap is not None else 0, + max_seqlen_q=max_query_len, + seq_threshold_3D=seq_threshold_3D, + num_par_softmax_segments=NUM_PAR_SOFTMAX_SEGMENTS, + softmax_segm_output=segm_output, + softmax_segm_max=segm_max, + softmax_segm_expsum=segm_expsum, + ) + + ( + torch.testing.assert_close(triton_out, fa_out, atol=2e-2, rtol=2e-2), + f"triton vs FA max abs diff: {torch.max(torch.abs(triton_out - fa_out))}", + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_trtllm_kvfp8_dequant.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_trtllm_kvfp8_dequant.py new file mode 100644 index 0000000000000000000000000000000000000000..d37fee77d73a2cc370405085496a549921900915 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_trtllm_kvfp8_dequant.py @@ -0,0 +1,440 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Standalone unit tests for trtllm_prefill_attn_kvfp8_dequant. + +Tests both contiguous and non-contiguous (cross-layer unified) KV cache +layouts against a pure-PyTorch reference implementation. +""" + +import pytest +import torch + +from vllm.platforms import current_platform + +if current_platform.is_rocm(): + pytest.skip( + "trtllm kvfp8 dequant is not supported on ROCm.", + allow_module_level=True, + ) + +FP8_DTYPE = current_platform.fp8_dtype() + +NUM_BLOCKS = 128 + + +def to_float8(x, dtype=None): + if dtype is None: + dtype = FP8_DTYPE + finfo = torch.finfo(dtype) + min_val, max_val = x.aminmax() + amax = torch.maximum(min_val.abs(), max_val.abs()).clamp(min=1e-12) + scale = finfo.max / amax * 0.1 + x_scl_sat = (x * scale).clamp(min=finfo.min, max=finfo.max) + return x_scl_sat.to(dtype), scale.float().reciprocal() + + +def make_contiguous_kv_cache(num_blocks, num_kv_heads, block_size, head_size): + """Create a standard contiguous fp8 KV cache (HND layout).""" + raw = torch.randn( + num_blocks, + 2, + num_kv_heads, + block_size, + head_size, + dtype=torch.bfloat16, + device="cuda", + ) + kv_cache, scale = to_float8(raw) + return kv_cache, scale + + +def make_cross_layer_kv_cache( + num_blocks, + num_kv_heads, + block_size, + head_size, + num_layers=4, +): + """ + Create a non-contiguous per-layer view mimicking cross-layer allocation. + + Physical layout: (num_blocks, 2, num_kv_heads, num_layers, block_size, head_size) + Returned view: (num_blocks, 2, num_kv_heads, block_size, head_size) + with non-contiguous strides on dims 0, 1, 2 (they skip over num_layers). + """ + raw = torch.randn( + num_blocks, + 2, + num_kv_heads, + num_layers, + block_size, + head_size, + dtype=torch.bfloat16, + device="cuda", + ) + fp8_full, scale = to_float8(raw) + layer_view = fp8_full[:, :, :, 0, :, :] + assert not layer_view.is_contiguous(), ( + f"Expected non-contiguous view, got strides {layer_view.stride()}" + ) + return layer_view, scale + + +def ref_dequant(kv_cache, block_tables, k_scale, v_scale, dequant_dtype): + """Pure PyTorch reference: gather pages and dequantize fp8 -> dequant_dtype.""" + batch_size, num_pages_per_seq = block_tables.shape + s = kv_cache.shape + out = torch.zeros( + batch_size * num_pages_per_seq + 1, + s[1], + s[2], + s[3], + s[4], + dtype=dequant_dtype, + device=kv_cache.device, + ) + for b in range(batch_size): + for p in range(num_pages_per_seq): + page_idx = block_tables[b, p].item() + if page_idx <= 0: + continue + mock_idx = b * num_pages_per_seq + p + 1 + out[mock_idx, 0] = (kv_cache[page_idx, 0].float() * k_scale.item()).to( + dequant_dtype + ) + out[mock_idx, 1] = (kv_cache[page_idx, 1].float() * v_scale.item()).to( + dequant_dtype + ) + return out + + +@pytest.mark.parametrize("num_kv_heads", [1, 8]) +@pytest.mark.parametrize("head_size", [64, 128]) +@pytest.mark.parametrize("block_size", [16, 32]) +@pytest.mark.parametrize("batch_size", [1, 4]) +@pytest.mark.parametrize("num_pages_per_seq", [3, 8]) +@pytest.mark.parametrize("contiguous", [True, False]) +@torch.inference_mode() +def test_trtllm_kvfp8_dequant( + num_kv_heads: int, + head_size: int, + block_size: int, + batch_size: int, + num_pages_per_seq: int, + contiguous: bool, +): + from vllm.v1.attention.backends.flashinfer import ( + trtllm_prefill_attn_kvfp8_dequant, + ) + + torch.set_default_device("cuda") + + if contiguous: + kv_cache, scale = make_contiguous_kv_cache( + NUM_BLOCKS, + num_kv_heads, + block_size, + head_size, + ) + else: + kv_cache, scale = make_cross_layer_kv_cache( + NUM_BLOCKS, + num_kv_heads, + block_size, + head_size, + ) + + k_scale = scale.clone() + v_scale = scale.clone() + + block_tables = torch.randint( + 1, + NUM_BLOCKS, + (batch_size, num_pages_per_seq), + dtype=torch.int32, + ) + + mock_kv_cache, mock_block_table = trtllm_prefill_attn_kvfp8_dequant( + kv_cache, + block_tables, + k_scale, + v_scale, + torch.bfloat16, + ) + + ref = ref_dequant(kv_cache, block_tables, k_scale, v_scale, torch.bfloat16) + + expected_bt = torch.arange( + 1, + batch_size * num_pages_per_seq + 1, + dtype=torch.int32, + device="cuda", + ).reshape(batch_size, num_pages_per_seq) + torch.testing.assert_close(mock_block_table, expected_bt) + + # Page 0 is padding (never written), compare only pages 1+ + torch.testing.assert_close(mock_kv_cache[1:], ref[1:], atol=1e-3, rtol=1e-3) + + +@torch.inference_mode() +def test_block_tables_with_zero_pages(): + """Pages with index <= 0 must be skipped (early return in kernel).""" + from vllm.v1.attention.backends.flashinfer import ( + trtllm_prefill_attn_kvfp8_dequant, + ) + + torch.set_default_device("cuda") + num_kv_heads, block_size, head_size = 8, 16, 64 + + kv_cache, scale = make_contiguous_kv_cache( + NUM_BLOCKS, + num_kv_heads, + block_size, + head_size, + ) + k_scale = v_scale = scale.clone() + + # Mix of valid pages and zeros (padding) + block_tables = torch.tensor( + [[5, 0, 10], [0, 0, 0], [3, 7, 0]], + dtype=torch.int32, + device="cuda", + ) + + mock_kv_cache, _ = trtllm_prefill_attn_kvfp8_dequant( + kv_cache, + block_tables, + k_scale, + v_scale, + torch.bfloat16, + ) + ref = ref_dequant(kv_cache, block_tables, k_scale, v_scale, torch.bfloat16) + + # Only compare pages that were actually written (non-zero page indices) + for b in range(block_tables.shape[0]): + for p in range(block_tables.shape[1]): + if block_tables[b, p].item() > 0: + idx = b * block_tables.shape[1] + p + 1 + torch.testing.assert_close( + mock_kv_cache[idx], + ref[idx], + atol=1e-3, + rtol=1e-3, + ) + + +@torch.inference_mode() +def test_all_zero_block_tables(): + """All-zero block_tables: kernel should write nothing.""" + from vllm.v1.attention.backends.flashinfer import ( + trtllm_prefill_attn_kvfp8_dequant, + ) + + torch.set_default_device("cuda") + num_kv_heads, block_size, head_size = 4, 16, 64 + + kv_cache, scale = make_contiguous_kv_cache( + NUM_BLOCKS, + num_kv_heads, + block_size, + head_size, + ) + k_scale = v_scale = scale.clone() + + block_tables = torch.zeros(2, 4, dtype=torch.int32, device="cuda") + + # Should not crash even though no pages are valid + mock_kv_cache, mock_block_table = trtllm_prefill_attn_kvfp8_dequant( + kv_cache, + block_tables, + k_scale, + v_scale, + torch.bfloat16, + ) + assert mock_kv_cache.shape[0] == 2 * 4 + 1 + assert mock_block_table.shape == (2, 4) + + +@torch.inference_mode() +def test_different_k_v_scales(): + """Verify K and V are dequantized with independent scales.""" + from vllm.v1.attention.backends.flashinfer import ( + trtllm_prefill_attn_kvfp8_dequant, + ) + + torch.set_default_device("cuda") + num_kv_heads, block_size, head_size = 8, 16, 64 + + kv_cache, _ = make_contiguous_kv_cache( + NUM_BLOCKS, + num_kv_heads, + block_size, + head_size, + ) + k_scale = torch.tensor([0.5], dtype=torch.float32, device="cuda") + v_scale = torch.tensor([2.0], dtype=torch.float32, device="cuda") + + block_tables = torch.tensor([[1, 2]], dtype=torch.int32, device="cuda") + + mock_kv_cache, _ = trtllm_prefill_attn_kvfp8_dequant( + kv_cache, + block_tables, + k_scale, + v_scale, + torch.bfloat16, + ) + ref = ref_dequant(kv_cache, block_tables, k_scale, v_scale, torch.bfloat16) + + torch.testing.assert_close(mock_kv_cache[1:], ref[1:], atol=1e-3, rtol=1e-3) + + +@torch.inference_mode() +def test_single_page_per_seq(): + """Minimum grid dim 1 = 1 page per sequence.""" + from vllm.v1.attention.backends.flashinfer import ( + trtllm_prefill_attn_kvfp8_dequant, + ) + + torch.set_default_device("cuda") + num_kv_heads, block_size, head_size = 8, 16, 128 + + kv_cache, scale = make_contiguous_kv_cache( + NUM_BLOCKS, + num_kv_heads, + block_size, + head_size, + ) + k_scale = v_scale = scale.clone() + + block_tables = torch.tensor([[5], [10], [20]], dtype=torch.int32, device="cuda") + + mock_kv_cache, _ = trtllm_prefill_attn_kvfp8_dequant( + kv_cache, + block_tables, + k_scale, + v_scale, + torch.bfloat16, + ) + ref = ref_dequant(kv_cache, block_tables, k_scale, v_scale, torch.bfloat16) + + torch.testing.assert_close(mock_kv_cache[1:], ref[1:], atol=1e-3, rtol=1e-3) + + +@torch.inference_mode() +def test_large_page_indices(): + """Page indices near the top of the buffer stress offset arithmetic.""" + from vllm.v1.attention.backends.flashinfer import ( + trtllm_prefill_attn_kvfp8_dequant, + ) + + torch.set_default_device("cuda") + num_kv_heads, block_size, head_size = 8, 16, 128 + large_num_blocks = 32768 + + kv_cache, scale = make_contiguous_kv_cache( + large_num_blocks, + num_kv_heads, + block_size, + head_size, + ) + k_scale = v_scale = scale.clone() + + # Use page indices near the top of the buffer + block_tables = torch.tensor( + [[large_num_blocks - 1, large_num_blocks - 2, 1]], + dtype=torch.int32, + device="cuda", + ) + + mock_kv_cache, _ = trtllm_prefill_attn_kvfp8_dequant( + kv_cache, + block_tables, + k_scale, + v_scale, + torch.bfloat16, + ) + ref = ref_dequant(kv_cache, block_tables, k_scale, v_scale, torch.bfloat16) + + torch.testing.assert_close(mock_kv_cache[1:], ref[1:], atol=1e-3, rtol=1e-3) + + +@torch.inference_mode() +def test_large_block_size(): + """block_size=64 -> HEAD_STRIDE=8192, large tl.arange per thread block.""" + from vllm.v1.attention.backends.flashinfer import ( + trtllm_prefill_attn_kvfp8_dequant, + ) + + torch.set_default_device("cuda") + num_kv_heads, block_size, head_size = 4, 64, 128 + + kv_cache, scale = make_contiguous_kv_cache( + NUM_BLOCKS, + num_kv_heads, + block_size, + head_size, + ) + k_scale = v_scale = scale.clone() + + block_tables = torch.randint( + 1, + NUM_BLOCKS, + (2, 4), + dtype=torch.int32, + device="cuda", + ) + + mock_kv_cache, _ = trtllm_prefill_attn_kvfp8_dequant( + kv_cache, + block_tables, + k_scale, + v_scale, + torch.bfloat16, + ) + ref = ref_dequant(kv_cache, block_tables, k_scale, v_scale, torch.bfloat16) + + torch.testing.assert_close(mock_kv_cache[1:], ref[1:], atol=1e-3, rtol=1e-3) + + +@torch.inference_mode() +def test_cross_layer_many_layers(): + """ + Non-contiguous with 36 layers -- matches real gpt-oss-120b. + Strides are far from contiguous (factor of 36 in the gaps). + """ + from vllm.v1.attention.backends.flashinfer import ( + trtllm_prefill_attn_kvfp8_dequant, + ) + + torch.set_default_device("cuda") + num_kv_heads, block_size, head_size = 8, 16, 64 + num_layers = 36 + + kv_cache, scale = make_cross_layer_kv_cache( + NUM_BLOCKS, + num_kv_heads, + block_size, + head_size, + num_layers=num_layers, + ) + k_scale = v_scale = scale.clone() + + block_tables = torch.randint( + 1, + NUM_BLOCKS, + (4, 6), + dtype=torch.int32, + device="cuda", + ) + + mock_kv_cache, _ = trtllm_prefill_attn_kvfp8_dequant( + kv_cache, + block_tables, + k_scale, + v_scale, + torch.bfloat16, + ) + ref = ref_dequant(kv_cache, block_tables, k_scale, v_scale, torch.bfloat16) + + torch.testing.assert_close(mock_kv_cache[1:], ref[1:], atol=1e-3, rtol=1e-3) diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_use_trtllm_attention.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_use_trtllm_attention.py new file mode 100644 index 0000000000000000000000000000000000000000..110e312faff9be45bb046e8debd806ac955149c5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_use_trtllm_attention.py @@ -0,0 +1,218 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from unittest.mock import patch + +import pytest +import torch + +from vllm.platforms import current_platform +from vllm.utils.flashinfer import ( + can_use_trtllm_attention, + supports_trtllm_attention, + use_trtllm_attention, +) + +if not current_platform.is_cuda(): + pytest.skip( + "TRTLLM attention is only supported on CUDA platforms.", + allow_module_level=True, + ) + +MODEL_CONFIGS = { + "Llama-3-70B": dict(num_qo_heads=64, num_kv_heads=8), + "Llama-3-8B": dict(num_qo_heads=32, num_kv_heads=8), + "Qwen2.5-0.5B": dict(num_qo_heads=14, num_kv_heads=2), + "Mistral-7B": dict(num_qo_heads=32, num_kv_heads=8), + "Gemma-2-9B": dict(num_qo_heads=8, num_kv_heads=4), + "Falcon-40B": dict(num_qo_heads=128, num_kv_heads=8), +} + + +def get_config(model: str) -> dict: + """Return the attention config for a model.""" + return MODEL_CONFIGS[model] + + +DEFAULT_KWARGS = dict( + **get_config("Llama-3-70B"), + num_tokens=128, + max_seq_len=4096, + dcp_world_size=1, + kv_cache_dtype="auto", + q_dtype=torch.bfloat16, + is_prefill=False, + force_use_trtllm=None, + has_sinks=False, + has_spec=False, +) + + +def _call(**overrides) -> bool: + kwargs = {**DEFAULT_KWARGS, **overrides} + return use_trtllm_attention(**kwargs) + + +@pytest.fixture(autouse=True) +def _clear_supports_cache(): + """Clear functools.cache to ensure each test runs independently.""" + supports_trtllm_attention.cache_clear() + + +# supports_trtllm_attention + + +@patch("vllm.envs.VLLM_BATCH_INVARIANT", True) +def test_supports_batch_invariant_disables(): + assert supports_trtllm_attention() is False + + +@patch("vllm.envs.VLLM_BATCH_INVARIANT", False) +@patch( + "vllm.utils.flashinfer.current_platform.is_device_capability_family", + return_value=True, +) +@patch("vllm.utils.flashinfer.has_nvidia_artifactory", return_value=True) +def test_supports_sm100_with_artifactory(_art, _cap): + assert supports_trtllm_attention() is True + + +@patch("vllm.envs.VLLM_BATCH_INVARIANT", False) +@patch( + "vllm.utils.flashinfer.current_platform.is_device_capability", return_value=False +) +@patch( + "vllm.utils.flashinfer.current_platform.is_device_capability_family", + return_value=False, +) +def test_supports_unsupported_platform(_family, _cap): + assert supports_trtllm_attention() is False + + +@patch("vllm.envs.VLLM_BATCH_INVARIANT", False) +@patch("vllm.utils.flashinfer.current_platform.is_device_capability", return_value=True) +@patch( + "vllm.utils.flashinfer.current_platform.is_device_capability_family", + return_value=False, +) +@patch("vllm.utils.flashinfer.has_nvidia_artifactory", return_value=True) +def test_supports_sm90_decode_only(_art, _family, _cap): + assert supports_trtllm_attention(is_prefill=False) is True + assert supports_trtllm_attention(is_prefill=True) is False + + +@patch("vllm.envs.VLLM_BATCH_INVARIANT", False) +@patch( + "vllm.utils.flashinfer.current_platform.is_device_capability_family", + return_value=True, +) +@patch("vllm.utils.flashinfer.has_nvidia_artifactory", return_value=False) +def test_supports_sm100_without_artifactory(_art, _cap): + assert supports_trtllm_attention() is False + + +# can_use_trtllm_attention + + +@patch("vllm.utils.flashinfer.force_use_trtllm_attention", return_value=False) +def test_can_use_force_disabled(_mock): + cfg = get_config("Llama-3-70B") + assert can_use_trtllm_attention(cfg["num_qo_heads"], cfg["num_kv_heads"]) is False + + +@patch("vllm.utils.flashinfer.force_use_trtllm_attention", return_value=None) +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=True) +def test_can_use_compatible_heads(_sup, _force): + cfg = get_config("Llama-3-70B") + assert can_use_trtllm_attention(cfg["num_qo_heads"], cfg["num_kv_heads"]) is True + + +@patch("vllm.utils.flashinfer.force_use_trtllm_attention", return_value=None) +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=True) +def test_can_use_incompatible_heads(_sup, _force): + assert can_use_trtllm_attention(40, 6) is False + + +@pytest.mark.parametrize("model", list(MODEL_CONFIGS.keys())) +@patch("vllm.utils.flashinfer.force_use_trtllm_attention", return_value=None) +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=False) +def test_can_use_platform_unsupported(_sup, _force, model): + cfg = get_config(model) + assert can_use_trtllm_attention(cfg["num_qo_heads"], cfg["num_kv_heads"]) is False + + +# use_trtllm_attention + + +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=True) +def test_use_force_off(_mock): + assert _call(force_use_trtllm=False) is False + + +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=True) +def test_use_dcp_fallback(_mock): + assert _call(dcp_world_size=2) is False + + +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=False) +def test_use_platform_unsupported(_mock): + assert _call() is False + + +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=False) +def test_use_platform_unsupported_force_on_still_false(_mock): + assert _call(force_use_trtllm=True) is False + + +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=True) +def test_use_incompatible_heads(_mock): + assert _call(num_qo_heads=40, num_kv_heads=6) is False + + +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=True) +def test_use_incompatible_heads_force_on_still_false(_mock): + assert _call(num_qo_heads=40, num_kv_heads=6, force_use_trtllm=True) is False + + +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=True) +def test_use_spec_decode_enables(_mock): + assert _call(has_spec=True, is_prefill=False) is True + + +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=True) +@patch( + "vllm.utils.flashinfer.current_platform.fp8_dtype", + return_value=torch.float8_e4m3fn, +) +def test_use_fp8_query_forces_trtllm(_fp8, _sup): + assert _call(q_dtype=torch.float8_e4m3fn) is True + + +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=True) +def test_use_sinks_force_trtllm(_mock): + assert _call(has_sinks=True) is True + + +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=True) +def test_use_auto_prefill_kv_auto(_mock): + assert _call(is_prefill=True, kv_cache_dtype="auto") is True + + +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=True) +def test_use_auto_prefill_kv_fp8(_mock): + assert _call(is_prefill=True, kv_cache_dtype="fp8") is False + + +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=True) +def test_use_auto_decode_small_batch(_mock): + assert _call(is_prefill=False, num_tokens=128, kv_cache_dtype="auto") is True + + +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=True) +def test_use_auto_decode_large_batch(_mock): + assert _call(is_prefill=False, num_tokens=512, kv_cache_dtype="auto") is False + + +@patch("vllm.utils.flashinfer.supports_trtllm_attention", return_value=True) +def test_use_force_on(_mock): + assert _call(force_use_trtllm=True) is True diff --git a/asian_ai/performance/vLLM/tests/kernels/attention/test_xpu_mla_sparse.py b/asian_ai/performance/vLLM/tests/kernels/attention/test_xpu_mla_sparse.py new file mode 100644 index 0000000000000000000000000000000000000000..262e635e6a44e5d12ece8ac30de11bdaf4f6ba86 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/attention/test_xpu_mla_sparse.py @@ -0,0 +1,118 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm.v1.attention.ops.xpu_mla_sparse import triton_bf16_mla_sparse_interface + + +# https://github.com/deepseek-ai/FlashMLA/blob/main/tests/ref.py#L7 +def _merge_two_lse( + lse0: torch.Tensor, lse1: torch.Tensor | None, s_q: int, h_q: int +) -> torch.Tensor: + if lse1 is None: + return lse0 + else: + return torch.logsumexp( + torch.stack([lse0.view(s_q, h_q), lse1.broadcast_to(s_q, h_q)], dim=0), + dim=0, + ) + + +# Adapted from https://github.com/deepseek-ai/FlashMLA/blob/main/tests/ref.py#L19 +def reference_mla_sparse_prefill( + q: torch.Tensor, + kv: torch.Tensor, + indices: torch.Tensor, + sm_scale: float, + d_v: int, + topk_length: torch.Tensor | None = None, + attn_sink: torch.Tensor | None = None, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + """ + Returns: + - o: [s_q, h_q, dv] + - o_fp32: [s_q, h_q, dv] + - max_logits: [s_q, h_q] + - lse: [s_q, h_q] + """ + s_q, h_q, d_qk = q.shape + s_kv, _, _ = kv.shape + _, _, topk = indices.shape + + indices = indices.clone().squeeze(1) + if topk_length is not None: + mask = torch.arange(topk, device=topk_length.device).unsqueeze(0).broadcast_to( + s_q, topk + ) >= topk_length.unsqueeze(1) # [s_q, topk] + indices[mask] = -1 + invalid_mask = (indices < 0) | (indices >= s_kv) # [s_q, topk] + indices[invalid_mask] = 0 + + q = q.float() + gathered_kv = ( + kv.index_select(dim=0, index=indices.flatten()).reshape(s_q, topk, d_qk).float() + ) # [s_q, topk, d_qk] + P = q @ gathered_kv.transpose(1, 2) # [s_q, h_q, topk] + P *= sm_scale + P[invalid_mask.unsqueeze(1).broadcast_to(P.shape)] = float("-inf") + + orig_lse = torch.logsumexp(P, dim=-1) # [s_q, h_q] + max_logits = P.max(dim=-1).values # [s_q, h_q] + + lse_for_o = _merge_two_lse(orig_lse, attn_sink, s_q, h_q) + if not torch.is_inference_mode_enabled(): + lse_for_o = lse_for_o.clone() + lse_for_o[lse_for_o == float("-inf")] = float( + "+inf" + ) # So that corresponding O will be 0 + s_for_o = torch.exp(P - lse_for_o.unsqueeze(-1)) + out = s_for_o @ gathered_kv[..., :d_v] # [s_q, h_q, dv] + + lonely_q_mask = orig_lse == float("-inf") # [s_q, h_q] + orig_lse[lonely_q_mask] = float("+inf") + return (out.to(kv.dtype), out, max_logits, orig_lse) + + +@pytest.mark.parametrize("device_str", ["xpu"]) +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@pytest.mark.skipif( + not torch.xpu.is_available(), + reason="XPU is required", +) +def test_bf16_triton_sparse_mla(device_str, dtype): + device = torch.device(device_str) + s_q = 1 + s_kv = 256 + h_q = 64 # kernel expects multiple of 64 + h_kv = 1 + d_qk = 576 + d_v = 512 + topk = 128 + + torch.random.manual_seed(1234) + + q = torch.randn((s_q, h_q, d_qk), dtype=dtype, device=device) + kv = torch.randn((s_kv, h_kv, d_qk), dtype=dtype, device=device) + indices = torch.full((s_q, h_kv, topk), -1, dtype=torch.int32, device=device) + for t in range(s_q): + for h in range(h_kv): + i_i = torch.randperm(max(1, t))[:topk] + indices[t, h, : len(i_i)] = i_i + + sm_scale = d_qk**-0.5 + + out, max_logits, lse = triton_bf16_mla_sparse_interface( + q, kv, indices, sm_scale, d_v + ) + assert out.shape == (s_q, h_q, d_v) + assert max_logits.shape == (s_q, h_q) + assert lse.shape == (s_q, h_q) + + ref_out, ref_out_fp32, ref_max_logits, ref_lse = reference_mla_sparse_prefill( + q, kv, indices, sm_scale, d_v + ) + assert torch.allclose(out, ref_out, atol=1e-2, rtol=1e-2) + assert torch.allclose(max_logits, ref_max_logits, atol=1e-3, rtol=1e-3) + assert torch.allclose(lse, ref_lse, atol=1e-3, rtol=1e-3) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_activation.py b/asian_ai/performance/vLLM/tests/kernels/core/test_activation.py new file mode 100644 index 0000000000000000000000000000000000000000..823e6ae731273502de029d9cd67751667ce0c226 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_activation.py @@ -0,0 +1,234 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import random + +import pytest +import torch + +from tests.kernels.allclose_default import get_default_atol, get_default_rtol +from tests.kernels.utils import opcheck +from vllm.model_executor.layers.activation import ( + FastGELU, + FatreluAndMul, + GeluAndMul, + MulAndSilu, + NewGELU, + QuickGELU, + SiluAndMul, + SiluAndMulWithClamp, + SwigluOAIAndMul, + SwigluStepAndMul, + swiglustep_and_mul_triton, +) +from vllm.utils.torch_utils import set_random_seed + +DTYPES = [torch.half, torch.bfloat16, torch.float] +NUM_TOKENS = [7, 83, 2048] # Arbitrary values for testing +D = [512, 13824] # Arbitrary values for testing +SEEDS = [0] +CUDA_DEVICES = [ + f"cuda:{i}" for i in range(1 if torch.accelerator.device_count() == 1 else 2) +] + + +@pytest.mark.parametrize( + "activation", + [ + "silu_and_mul", + "mul_and_silu", + "gelu", + "gelu_tanh", + "fatrelu", + "swigluoai_and_mul", + "swiglustep_and_mul", + ], +) +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("d", D) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@torch.inference_mode() +def test_act_and_mul( + default_vllm_config, + activation: str, + num_tokens: int, + d: int, + dtype: torch.dtype, + seed: int, + device: str, +) -> None: + set_random_seed(seed) + torch.set_default_device(device) + x = torch.randn(num_tokens, 2 * d, dtype=dtype) + if activation == "silu_and_mul": + layer = SiluAndMul(compile_native=False) + fn = torch.ops._C.silu_and_mul + if activation == "mul_and_silu": + layer = MulAndSilu() + fn = torch.ops._C.mul_and_silu + elif activation == "gelu": + layer = GeluAndMul(approximate="none") + fn = torch.ops._C.gelu_and_mul + elif activation == "gelu_tanh": + layer = GeluAndMul(approximate="tanh") + fn = torch.ops._C.gelu_tanh_and_mul + elif activation == "fatrelu": + threshold = random.uniform(0, 1) + layer = FatreluAndMul(threshold) + fn = torch.ops._C.fatrelu_and_mul + elif activation == "swigluoai_and_mul": + layer = SwigluOAIAndMul() + fn = torch.ops._C.swigluoai_and_mul + elif activation == "swiglustep_and_mul": + layer = SwigluStepAndMul() + fn = swiglustep_and_mul_triton + out = layer(x) + ref_out = layer.forward_native(x) + if activation in ["swigluoai_and_mul", "swiglustep_and_mul"]: + rtol = { + # For fp16, change the relative tolerance from 1e-3 to 2e-3 + torch.float16: 2e-3, + torch.bfloat16: 2e-2, + torch.float: 1.3e-6, + } + + def _get_rtol(output) -> float: + return rtol[output.dtype] + + torch.testing.assert_close( + out, ref_out, atol=get_default_atol(out), rtol=_get_rtol(out) + ) + else: + # The SiluAndMul, MulAndSilu, GELU and FatReLU implementations are + # equivalent to the native PyTorch implementations, so we can do exact + # comparison. + torch.testing.assert_close(out, ref_out, atol=0.0, rtol=0.0) + + d = x.shape[-1] // 2 + output_shape = x.shape[:-1] + (d,) + out = torch.empty(output_shape, dtype=x.dtype, device=x.device) + if activation == "fatrelu": + opcheck(fn, (out, x, threshold)) + elif activation == "swigluoai_and_mul": + opcheck(fn, (out, x, layer.alpha, layer.limit)) + elif activation != "swiglustep_and_mul": + opcheck(fn, (out, x)) + + +SWIGLU_LIMITS = [3.0, 7.0, 15.0] + + +@pytest.mark.parametrize("swiglu_limit", SWIGLU_LIMITS) +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("d", D) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@torch.inference_mode() +def test_silu_and_mul_with_clamp( + default_vllm_config, + swiglu_limit: float, + num_tokens: int, + d: int, + dtype: torch.dtype, + seed: int, + device: str, +) -> None: + """SiluAndMulWithClamp: cuda kernel must match native reference.""" + set_random_seed(seed) + torch.set_default_device(device) + # Use large values to ensure clamping is exercised. + x = torch.randn(num_tokens, 2 * d, dtype=dtype) * swiglu_limit * 2 + + layer = SiluAndMulWithClamp(swiglu_limit, compile_native=False) + out = layer(x) + ref_out = layer.forward_native(x) + + rtol = { + torch.float16: 2e-3, + torch.bfloat16: 2e-2, + torch.float: 1.3e-6, + } + torch.testing.assert_close( + out, ref_out, atol=get_default_atol(out), rtol=rtol[out.dtype] + ) + + # Verify clamping is actually being applied: the clamped output should + # differ from the unclamped SiluAndMul output when inputs are large. + unclamped_out = SiluAndMul.forward_native(x) + assert not torch.equal(ref_out.float(), unclamped_out.float()), ( + "Input was not large enough to exercise the clamp; increase scale" + ) + + # Verify gate clamping semantics with a controlled scalar case. + # gate=large_val is clamped to limit first, then silu(limit) * 1.0. + x_gate = torch.tensor( + [[swiglu_limit * 20.0, 1.0]], dtype=torch.float32, device=device + ) + out_gate = SiluAndMulWithClamp(swiglu_limit, compile_native=False)(x_gate) + expected_gate = torch.nn.functional.silu( + torch.tensor(swiglu_limit, dtype=torch.float32) + ).item() + torch.testing.assert_close( + out_gate, + torch.tensor([[expected_gate]], dtype=torch.float32, device=device), + atol=1e-3, + rtol=1e-3, + ) + + # Verify up clamping semantics: up >> limit gets clamped to limit. + x_up = torch.tensor( + [[1.0, swiglu_limit * 20.0]], dtype=torch.float32, device=device + ) + out_up = SiluAndMulWithClamp(swiglu_limit, compile_native=False)(x_up) + silu_1 = torch.nn.functional.silu(torch.tensor(1.0)).item() + torch.testing.assert_close( + out_up, + torch.tensor([[silu_1 * swiglu_limit]], dtype=torch.float32, device=device), + atol=1e-3, + rtol=1e-3, + ) + + # opcheck + out_buf = torch.empty(x.shape[:-1] + (d,), dtype=dtype, device=device) + opcheck(torch.ops._C.silu_and_mul_with_clamp, (out_buf, x, swiglu_limit)) + + +@pytest.mark.parametrize( + "activation", + [ + (FastGELU, torch.ops._C.gelu_fast), + (NewGELU, torch.ops._C.gelu_new), + (QuickGELU, torch.ops._C.gelu_quick), + ], +) +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("d", D) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@torch.inference_mode() +def test_activation( + default_vllm_config, + activation: type[torch.nn.Module], + num_tokens: int, + d: int, + dtype: torch.dtype, + seed: int, + device: str, +) -> None: + set_random_seed(seed) + torch.set_default_device(device) + x = torch.randn(num_tokens, d, dtype=dtype) + layer = activation[0]() + fn = activation[1] + out = layer(x) + ref_out = layer.forward_native(x) + torch.testing.assert_close( + out, ref_out, atol=get_default_atol(out), rtol=get_default_rtol(out) + ) + + out = torch.empty_like(x) + opcheck(fn, (out, x)) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_apply_rotary_emb.py b/asian_ai/performance/vLLM/tests/kernels/core/test_apply_rotary_emb.py new file mode 100644 index 0000000000000000000000000000000000000000..71215f0743e442ba9690af28202f0f44bb80ab1d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_apply_rotary_emb.py @@ -0,0 +1,203 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Tests for ApplyRotaryEmb CustomOp dispatch behavior. + +This test ensures that RotaryEmbedding classes correctly call the appropriate +ApplyRotaryEmb methods based on the calling context: + +1. RotaryEmbedding.forward_native() -> ApplyRotaryEmb.forward_native() +2. RotaryEmbedding.forward_cuda() -> ApplyRotaryEmb.forward() (auto-dispatch) +3. RotaryEmbedding.forward_hip() -> ApplyRotaryEmb.forward() (auto-dispatch) +""" + +from dataclasses import dataclass + +import pytest +import torch + +from vllm.config import ( + CompilationConfig, + VllmConfig, + get_cached_compilation_config, + set_current_vllm_config, +) +from vllm.platforms import current_platform + +CUDA_DEVICES = ["cuda:0"] + + +@dataclass +class RotaryEmbeddingTestCase: + """Test case configuration for RotaryEmbedding dispatch tests.""" + + name: str + rope_class: type + rope_kwargs: dict + method_name: str # forward_native, forward_cuda, forward + positions_shape: tuple # (num_tokens,) or (3, num_tokens) or (4, num_tokens) + expect_forward_native: bool # Should call ApplyRotaryEmb.forward_native() + expect_forward: bool # Should call ApplyRotaryEmb.forward() + + +def get_test_cases() -> list[RotaryEmbeddingTestCase]: + """Generate test cases for all RotaryEmbedding classes.""" + from vllm.model_executor.layers.rotary_embedding.ernie45_vl_rope import ( + Ernie4_5_VLRotaryEmbedding, + ) + from vllm.model_executor.layers.rotary_embedding.mrope import MRotaryEmbedding + from vllm.model_executor.layers.rotary_embedding.xdrope import XDRotaryEmbedding + + common_kwargs = { + "head_size": 128, + "rotary_dim": 128, + "max_position_embeddings": 4096, + "base": 10000, + "is_neox_style": True, + "dtype": torch.bfloat16, + } + + return [ + # MRotaryEmbedding tests + RotaryEmbeddingTestCase( + name="MRotaryEmbedding.forward_native", + rope_class=MRotaryEmbedding, + rope_kwargs={**common_kwargs, "mrope_section": [16, 24, 24]}, + method_name="forward_native", + positions_shape=(3, 32), # 2D for multimodal + expect_forward_native=True, + expect_forward=False, + ), + RotaryEmbeddingTestCase( + name="MRotaryEmbedding.forward_cuda_1d", + rope_class=MRotaryEmbedding, + rope_kwargs={**common_kwargs, "mrope_section": [16, 24, 24]}, + method_name="forward_cuda", + positions_shape=(32,), # 1D triggers apply_rotary_emb path + expect_forward_native=False, + expect_forward=True, + ), + # XDRotaryEmbedding tests + RotaryEmbeddingTestCase( + name="XDRotaryEmbedding.forward", + rope_class=XDRotaryEmbedding, + rope_kwargs={ + **common_kwargs, + "scaling_alpha": 1.0, + "xdrope_section": [16, 16, 16, 16], + }, + method_name="forward", + positions_shape=(4, 32), # 4D for P/W/H/T + expect_forward_native=False, + expect_forward=True, + ), + # Ernie4_5_VLRotaryEmbedding tests + RotaryEmbeddingTestCase( + name="Ernie4_5_VLRotaryEmbedding.forward_native", + rope_class=Ernie4_5_VLRotaryEmbedding, + rope_kwargs={**common_kwargs, "mrope_section": [22, 22, 20]}, + method_name="forward_native", + positions_shape=(3, 32), # 2D for multimodal + expect_forward_native=True, + expect_forward=False, + ), + ] + + +def run_dispatch_test( + test_case: RotaryEmbeddingTestCase, + device: str, +): + """Run a dispatch test for a RotaryEmbedding class.""" + vllm_config = VllmConfig( + compilation_config=CompilationConfig(custom_ops=["all", "+apply_rotary_emb"]) + ) + get_cached_compilation_config.cache_clear() + + with set_current_vllm_config(vllm_config): + rope = test_case.rope_class(**test_case.rope_kwargs).to(device=device) + + apply_rotary_emb = rope.apply_rotary_emb + + # Verify custom op is enabled + if test_case.expect_forward_native: + assert ( + apply_rotary_emb._forward_method != apply_rotary_emb.forward_native + ), "Test setup error: ApplyRotaryEmb custom op should be enabled" + + # Setup call tracking + call_tracker = {"forward_native_called": False, "forward_called": False} + original_forward_native = apply_rotary_emb.forward_native + original_forward = apply_rotary_emb.forward + + def tracked_forward_native(*args, **kwargs): + call_tracker["forward_native_called"] = True + return original_forward_native(*args, **kwargs) + + def tracked_forward(*args, **kwargs): + call_tracker["forward_called"] = True + return original_forward(*args, **kwargs) + + apply_rotary_emb.forward_native = tracked_forward_native + apply_rotary_emb.forward = tracked_forward + + try: + num_tokens = test_case.positions_shape[-1] + num_q_heads = 8 + num_kv_heads = 2 + head_size = test_case.rope_kwargs["head_size"] + max_position = test_case.rope_kwargs["max_position_embeddings"] + + positions = torch.randint( + 0, max_position // 4, test_case.positions_shape, device=device + ) + query = torch.randn( + num_tokens, num_q_heads * head_size, dtype=torch.bfloat16, device=device + ) + key = torch.randn( + num_tokens, + num_kv_heads * head_size, + dtype=torch.bfloat16, + device=device, + ) + + # Call the method under test + method = getattr(rope, test_case.method_name) + method(positions, query.clone(), key.clone()) + + # Verify expectations + if test_case.expect_forward_native: + assert call_tracker["forward_native_called"], ( + f"{test_case.name} should call ApplyRotaryEmb.forward_native()" + ) + if not test_case.expect_forward: + assert not call_tracker["forward_called"], ( + f"{test_case.name} should NOT call ApplyRotaryEmb.forward(). " + "Bug: when +apply_rotary_emb is enabled, forward_native() " + "incorrectly dispatches to CUDA/HIP kernels." + ) + if test_case.expect_forward: + assert call_tracker["forward_called"], ( + f"{test_case.name} should call ApplyRotaryEmb.forward()" + ) + finally: + apply_rotary_emb.forward_native = original_forward_native + apply_rotary_emb.forward = original_forward + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), reason="Skipping CUDA/ROCm only tests." +) +@pytest.mark.parametrize("test_case", get_test_cases(), ids=lambda tc: tc.name) +@pytest.mark.parametrize("device", CUDA_DEVICES) +def test_rotary_embedding_dispatch( + test_case: RotaryEmbeddingTestCase, + device: str, +): + """ + Test that RotaryEmbedding classes dispatch to the correct ApplyRotaryEmb method. + + - forward_native methods should call ApplyRotaryEmb.forward_native() + - forward_cuda/forward methods should call ApplyRotaryEmb.forward() + """ + run_dispatch_test(test_case, device) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_batched_weight_rms_norm.py b/asian_ai/performance/vLLM/tests/kernels/core/test_batched_weight_rms_norm.py new file mode 100644 index 0000000000000000000000000000000000000000..06b3a7673900471d608369bd9ea6564c840c67c5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_batched_weight_rms_norm.py @@ -0,0 +1,70 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the batched-weight RMS norm kernel (vllm._custom_ops.rms_norm). + +``rms_norm`` can use the outermost input batch index to select the corresponding +weight row. The result must match that of looping ``rms_norm`` over that dimension. +""" + +import pytest +import torch + +from vllm import _custom_ops as ops +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +pytestmark = pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="rms_norm requires a CUDA/ROCm device", +) + + +@pytest.mark.parametrize( + "shape", + [ + (28, 17, 128), # 3D: [num_rows, tokens, hidden] + (1, 5, 2, 128), # 4D: single row (edge case) + (28, 13, 8, 128), # 4D: [L, num_ctx, nkv, hd] (DFlash K-norm) + (6, 3, 4, 769), # 4D: non-power-of-two hidden size + ], +) +@pytest.mark.parametrize("dtype", [torch.half, torch.bfloat16, torch.float]) +@pytest.mark.parametrize("seed", [42]) +@torch.inference_mode() +def test_rms_norm_matches_loop( + shape: tuple[int, ...], dtype: torch.dtype, seed: int +) -> None: + set_random_seed(seed) + torch.set_default_device("cuda") + + num_rows, hidden = shape[0], shape[-1] + eps = 1e-6 + + x = torch.randn(*shape, dtype=dtype) * 0.1 + # Distinct weight per row so that a wrong row index would be caught. + weight = torch.randn(num_rows, hidden, dtype=dtype) * 0.1 + 1.0 + + # Reference batched-weight rms norm. + out_ref = torch.empty_like(x) + for i in range(x.shape[0]): + ops.rms_norm(out_ref[i], x[i], weight[i], eps) + + out = torch.empty_like(x) + ops.rms_norm(out, x, weight, eps) + + # Expect bitwise-identical results. + torch.testing.assert_close(out, out_ref, atol=0, rtol=0) + + +@torch.inference_mode() +def test_rms_norm_validates_shapes() -> None: + torch.set_default_device("cuda") + + x = torch.randn(4, 8, 128, dtype=torch.float) + out = torch.empty_like(x) + # Expect num rows mismatch. + with pytest.raises(RuntimeError): + ops.rms_norm(out, x, torch.randn(3, 128), 1e-6) + # Expect hidden size mismatch. + with pytest.raises(RuntimeError): + ops.rms_norm(out, x, torch.randn(4, 64), 1e-6) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_cpu_activation.py b/asian_ai/performance/vLLM/tests/kernels/core/test_cpu_activation.py new file mode 100644 index 0000000000000000000000000000000000000000..214a86ab437ddf4a5abf6daad22d891c07a96912 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_cpu_activation.py @@ -0,0 +1,212 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import pytest +import torch + +from tests.kernels.allclose_default import get_default_atol, get_default_rtol +from tests.kernels.utils import opcheck +from vllm.platforms import CpuArchEnum, current_platform +from vllm.utils.torch_utils import set_random_seed + +if not current_platform.is_cpu(): + pytest.skip("skipping CPU-only tests", allow_module_level=True) + +from vllm.model_executor.layers.activation import ( + GELU, + FastGELU, + GeluAndMul, + NewGELU, + QuickGELU, + SiluAndMul, +) + +DTYPES = [torch.bfloat16, torch.float32] +NUM_TOKENS = [7, 83] +D = [512, 2048] +SEEDS = [0] + + +@pytest.mark.parametrize( + ("activation_cls", "fn"), + [ + (SiluAndMul, torch.ops._C.silu_and_mul), + (GeluAndMul, torch.ops._C.gelu_and_mul), + (GeluAndMul, torch.ops._C.gelu_tanh_and_mul), + ], +) +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("d", D) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@torch.inference_mode() +def test_cpu_act_and_mul( + default_vllm_config, + activation_cls: type[torch.nn.Module], + fn: object, + num_tokens: int, + d: int, + dtype: torch.dtype, + seed: int, +) -> None: + set_random_seed(seed) + x = torch.randn(num_tokens, 2 * d, dtype=dtype) + + layer = activation_cls() + out = layer(x) + ref_out = layer.forward_native(x) + + torch.testing.assert_close( + out, ref_out, atol=get_default_atol(out), rtol=get_default_rtol(out) + ) + + output_shape = x.shape[:-1] + (x.shape[-1] // 2,) + raw_out = torch.empty(output_shape, dtype=x.dtype, device=x.device) + opcheck(fn, (raw_out, x)) + + +@pytest.mark.parametrize( + ("activation_cls", "fn", "op_args"), + [ + (NewGELU, torch.ops._C.gelu_new, ()), + (FastGELU, torch.ops._C.gelu_fast, ()), + (QuickGELU, torch.ops._C.gelu_quick, ()), + pytest.param( + GELU, + getattr(torch.ops._C, "activation_lut_bf16", None), + ("gelu",), + marks=pytest.mark.skipif( + current_platform.get_cpu_architecture() != CpuArchEnum.ARM, + reason="activation_lut_bf16 is only built on Arm CPU", + ), + ), + ], +) +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("d", D) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@torch.inference_mode() +def test_cpu_unary_activation( + default_vllm_config, + activation_cls: type[torch.nn.Module], + fn: object, + op_args: tuple[str, ...], + num_tokens: int, + d: int, + dtype: torch.dtype, + seed: int, +) -> None: + set_random_seed(seed) + x = torch.randn(num_tokens, d, dtype=dtype) + layer = activation_cls() + out = layer(x) + ref_out = layer.forward_native(x) + torch.testing.assert_close( + out, ref_out, atol=get_default_atol(out), rtol=get_default_rtol(out) + ) + # gelu with activation_lut_bf16 only makes sense for BF16 + if not (activation_cls is GELU and dtype != torch.bfloat16): + raw_out = torch.empty_like(x) + opcheck(fn, (raw_out, x, *op_args)) + + +@pytest.mark.parametrize("dtype", DTYPES) +@torch.inference_mode() +def test_cpu_gelu_tanh_and_mul( + default_vllm_config, + dtype: torch.dtype, +) -> None: + gate = torch.tensor( + [ + [ + -12.0, + -10.0, + -9.01, + -5.0, + -2.0, + -1.0, + -0.0, + 0.0, + 0.5, + 1.0, + 2.0, + 5.0, + 9.01, + 10.0, + 12.0, + 11.0, + ], + [ + -7.5, + -4.5, + -3.0, + -1.5, + -0.75, + -0.25, + 0.25, + 0.75, + 1.5, + 3.0, + 4.5, + 7.5, + -11.0, + 11.0, + 8.75, + -8.75, + ], + ], + dtype=dtype, + ) + val = torch.tensor( + [ + [ + 0.25, + -0.5, + 0.75, + -1.0, + 1.25, + -1.5, + 1.75, + -2.0, + 2.25, + -2.5, + 2.75, + -3.0, + 3.25, + -3.5, + 3.75, + -4.0, + ], + [ + -0.4, + 0.6, + -0.8, + 1.0, + -1.2, + 1.4, + -1.6, + 1.8, + -2.0, + 2.2, + -2.4, + 2.6, + -2.8, + 3.0, + -3.2, + 3.4, + ], + ], + dtype=dtype, + ) + + x = torch.cat((val, gate), dim=-1).contiguous() + kernel_out = torch.empty_like(val) + torch.ops._C.gelu_tanh_and_mul(kernel_out, x) + + torch_ref = torch.nn.functional.gelu(val, approximate="tanh") * gate + + atol = get_default_atol(kernel_out) + rtol = get_default_rtol(kernel_out) + torch.testing.assert_close(kernel_out, torch_ref, atol=atol, rtol=rtol) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_fused_allreduce_gemma_rms_norm.py b/asian_ai/performance/vLLM/tests/kernels/core/test_fused_allreduce_gemma_rms_norm.py new file mode 100644 index 0000000000000000000000000000000000000000..be0ce7f17bc089605e66c693a1f0a6d29d6ebf5c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_fused_allreduce_gemma_rms_norm.py @@ -0,0 +1,109 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the manual AllReduce + GemmaRMSNorm fusion used by MiniMax M3. + +``fused_allreduce_gemma_rms_norm`` must match the unfused model path, i.e. +``GemmaRMSNorm(all_reduce(partial), residual)``, both on the flashinfer fast +path (TP>1 with flashinfer + NVSwitch) and on the eager fallback (TP==1, or when +flashinfer is unavailable / the GPU has no NVSwitch). +""" + +import pytest +import torch +from torch.multiprocessing import spawn + +from tests.utils import ensure_current_vllm_config, init_test_distributed_environment +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.distributed.communication_op import tensor_model_parallel_all_reduce +from vllm.model_executor.layers.fused_allreduce_gemma_rms_norm import ( + fused_allreduce_gemma_rms_norm, +) +from vllm.model_executor.layers.layernorm import GemmaRMSNorm +from vllm.platforms import current_platform +from vllm.utils.network_utils import get_open_port +from vllm.utils.torch_utils import set_random_seed + + +@ensure_current_vllm_config() +def _worker_fused_ar_norm( + local_rank, + world_size, + port, + num_tokens, + hidden_size, + dtype, + seed, + eps, +): + """Per-rank worker: compare the fused helper vs all_reduce + GemmaRMSNorm.""" + device = torch.device(f"cuda:{local_rank}") + torch.accelerator.set_device_index(device) + init_test_distributed_environment( + world_size, 1, local_rank, port, local_rank=local_rank + ) + + # Norm weights are identical across ranks (replicated GemmaRMSNorm). + set_random_seed(seed) + norm = GemmaRMSNorm(hidden_size, eps=eps).cuda().to(dtype) + with torch.no_grad(): + norm.weight.normal_(mean=0.0, std=0.1) + + # Residual is shared across ranks; the partial o_proj output differs per rank + # (each rank holds a partial sum that all_reduce combines). + torch.manual_seed(seed + 7) + residual = torch.randn(num_tokens, hidden_size, dtype=dtype, device=device) + torch.manual_seed(seed + 1000 + local_rank) + partial = torch.randn(num_tokens, hidden_size, dtype=dtype, device=device) + + # Reference: the unfused model path. + reduced = tensor_model_parallel_all_reduce(partial.clone()) + ref_out, ref_res = norm(reduced, residual.clone()) + + # Fused helper (flashinfer fast path when available, else fallback). + out, res = fused_allreduce_gemma_rms_norm(partial.clone(), residual.clone(), norm) + torch.accelerator.synchronize() + + torch.testing.assert_close(out, ref_out, atol=2e-2, rtol=2e-2) + torch.testing.assert_close(res, ref_res, atol=2e-2, rtol=2e-2) + + cleanup_dist_env_and_memory() + + +@pytest.mark.skipif( + not current_platform.is_cuda(), + reason="CUDA required", +) +# world_size=1 exercises the TP==1 identity branch on a single GPU; >1 exercises +# the all_reduce + GemmaRMSNorm equivalence (flashinfer kernel or fallback). +@pytest.mark.parametrize("world_size", [1, 2, 4]) +@pytest.mark.parametrize("num_tokens", [1, 128, 333]) +@pytest.mark.parametrize("hidden_size", [2048, 4096]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("eps", [1e-6]) +@pytest.mark.parametrize("seed", [42]) +def test_fused_allreduce_gemma_rms_norm( + world_size, + num_tokens, + hidden_size, + dtype, + eps, + seed, +): + num_gpus = current_platform.device_count() + if num_gpus < world_size: + pytest.skip(f"Need >= {world_size} GPUs, have {num_gpus}") + port = str(get_open_port()) + spawn( + _worker_fused_ar_norm, + args=( + world_size, + port, + num_tokens, + hidden_size, + dtype, + seed, + eps, + ), + nprocs=world_size, + join=True, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_fused_q_kv_rmsnorm.py b/asian_ai/performance/vLLM/tests/kernels/core/test_fused_q_kv_rmsnorm.py new file mode 100644 index 0000000000000000000000000000000000000000..b846e973afb7d5dfb2918ec9444953d399aa4f0b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_fused_q_kv_rmsnorm.py @@ -0,0 +1,81 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Correctness + large-token-count launch tests for fused_q_kv_rmsnorm. + +Before the grid-dim fix the kernel used grid ``(2, num_tokens)``, which hit +CUDA's 65535 grid-y cap for ``num_tokens >= 65536`` and failed with +``Triton Error [CUDA]: invalid argument`` at every large chunked-prefill +profile run. These tests pin the new grid layout. +""" + +from __future__ import annotations + +import pytest +import torch + +from vllm.models.deepseek_v4.common.ops import fused_q_kv_rmsnorm +from vllm.platforms import current_platform + +pytestmark = pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="fused_q_kv_rmsnorm requires a CUDA/ROCm device", +) + + +def _ref_rmsnorm(x: torch.Tensor, w: torch.Tensor, eps: float) -> torch.Tensor: + x_f32 = x.to(torch.float32) + variance = x_f32.pow(2).mean(dim=-1, keepdim=True) + y = x_f32 * torch.rsqrt(variance + eps) * w.to(torch.float32) + return y.to(x.dtype) + + +@pytest.mark.parametrize("num_tokens", [1, 17, 1024, 8192]) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) +def test_fused_q_kv_rmsnorm_correctness(num_tokens: int, dtype: torch.dtype): + torch.manual_seed(0) + device = "cuda" + q_size, kv_size = 192, 576 + qr = torch.randn(num_tokens, q_size, dtype=dtype, device=device) + kv = torch.randn(num_tokens, kv_size, dtype=dtype, device=device) + qw = torch.randn(q_size, dtype=dtype, device=device) + kvw = torch.randn(kv_size, dtype=dtype, device=device) + eps = 1e-6 + + qr_out, kv_out = fused_q_kv_rmsnorm(qr, kv, qw, kvw, eps) + + qr_ref = _ref_rmsnorm(qr, qw, eps) + kv_ref = _ref_rmsnorm(kv, kvw, eps) + + tol = dict(rtol=1e-2, atol=1e-2) + torch.testing.assert_close(qr_out, qr_ref, **tol) + torch.testing.assert_close(kv_out, kv_ref, **tol) + + +@pytest.mark.parametrize("num_tokens", [65535, 65536, 131072]) +def test_fused_q_kv_rmsnorm_launches_past_grid_y_cap(num_tokens: int): + """Regression guard: grid used to be (2, num_tokens), hitting CUDA's + 65535 grid-y cap at num_tokens >= 65536. The new grid (num_tokens, 2) + lifts that bound to 2**31-1.""" + device = "cuda" + dtype = torch.bfloat16 + q_size, kv_size = 192, 576 + qr = torch.randn(num_tokens, q_size, dtype=dtype, device=device) + kv = torch.randn(num_tokens, kv_size, dtype=dtype, device=device) + qw = torch.randn(q_size, dtype=dtype, device=device) + kvw = torch.randn(kv_size, dtype=dtype, device=device) + + qr_out, kv_out = fused_q_kv_rmsnorm(qr, kv, qw, kvw, 1e-6) + # spot-check a couple of rows against the torch reference + for row in (0, num_tokens // 2, num_tokens - 1): + torch.testing.assert_close( + qr_out[row], + _ref_rmsnorm(qr[row : row + 1], qw, 1e-6)[0], + rtol=1e-2, + atol=1e-2, + ) + torch.testing.assert_close( + kv_out[row], + _ref_rmsnorm(kv[row : row + 1], kvw, 1e-6)[0], + rtol=1e-2, + atol=1e-2, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_fused_qk_norm_rope.py b/asian_ai/performance/vLLM/tests/kernels/core/test_fused_qk_norm_rope.py new file mode 100644 index 0000000000000000000000000000000000000000..9a609cce99e99a0296e2578f804d6361040f62b9 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_fused_qk_norm_rope.py @@ -0,0 +1,146 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from tests.kernels.utils import opcheck +from vllm.model_executor.layers.layernorm import RMSNorm +from vllm.model_executor.layers.rotary_embedding import RotaryEmbedding +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +DTYPES = [torch.bfloat16, torch.float16] +IS_NEOX = [True, False] +EPS_VALUES = [1e-5, 1e-6] +SEEDS = [13] +PARTIAL_ROPE = [True, False] +CUDA_DEVICES = ["cuda:0"] + + +def _apply_qk_norm_rope( + qkv: torch.Tensor, + positions: torch.Tensor, + q_norm: RMSNorm, + k_norm: RMSNorm, + rope: RotaryEmbedding, + num_heads_q: int, + num_heads_kv: int, + head_dim: int, +) -> torch.Tensor: + q_size = num_heads_q * head_dim + kv_size = num_heads_kv * head_dim + + q, k, v = qkv.split([q_size, kv_size, kv_size], dim=-1) + + q_by_head = q.view(*q.shape[:-1], q.shape[-1] // head_dim, head_dim) + q_by_head = q_norm.forward_native(q_by_head) + q = q_by_head.view(q.shape) + + k_by_head = k.view(*k.shape[:-1], k.shape[-1] // head_dim, head_dim) + k_by_head = k_norm.forward_native(k_by_head) + k = k_by_head.view(k.shape) + + q, k = rope.forward_native(positions, q, k) + return torch.cat([q, k, v], dim=-1) + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="fused_qk_norm_rope custom op requires cuda and rocm platform", +) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("is_neox", IS_NEOX) +@pytest.mark.parametrize("eps", EPS_VALUES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("rotary_ratio", [1.0, 0.5, 0.25]) +@torch.inference_mode() +def test_fused_qk_norm_rope_matches_reference( + default_vllm_config, + device: str, + dtype: torch.dtype, + is_neox: bool, + eps: float, + seed: int, + rotary_ratio: float, +): + torch.set_default_device(device) + set_random_seed(seed) + num_heads, num_kv_heads, head_dim = 16, 4, 128 + num_tokens = 4 + + total_dim = (num_heads + 2 * num_kv_heads) * head_dim + qkv_base = torch.randn(num_tokens, total_dim, dtype=dtype, device=device) + qkv_fused = qkv_base.clone() + positions = torch.arange(num_tokens, dtype=torch.long, device=device) + + q_norm = RMSNorm(head_dim, eps=eps).to(device=device, dtype=dtype) + k_norm = RMSNorm(head_dim, eps=eps).to(device=device, dtype=dtype) + q_norm.weight.data.normal_(mean=1.0, std=0.1) + k_norm.weight.data.normal_(mean=1.0, std=0.1) + q_weight = q_norm.weight.data + k_weight = k_norm.weight.data + rotary_dim = int(head_dim * rotary_ratio) + rope = RotaryEmbedding( + head_size=head_dim, + rotary_dim=rotary_dim, + max_position_embeddings=4096, + base=10000.0, + is_neox_style=is_neox, + dtype=dtype, + ).to(device) + + ref_result = _apply_qk_norm_rope( + qkv=qkv_base, + positions=positions, + q_norm=q_norm, + k_norm=k_norm, + rope=rope, + num_heads_q=num_heads, + num_heads_kv=num_kv_heads, + head_dim=head_dim, + ) + + opcheck( + torch.ops._C.fused_qk_norm_rope, + ( + qkv_fused.clone(), + num_heads, + num_kv_heads, + num_kv_heads, + head_dim, + eps, + q_weight, + k_weight, + rope.cos_sin_cache, + is_neox, + positions.view(-1), + ), + ) + + torch.ops._C.fused_qk_norm_rope( + qkv_fused, + num_heads, + num_kv_heads, + num_kv_heads, + head_dim, + eps, + q_weight, + k_weight, + rope.cos_sin_cache, + is_neox, + positions.view(-1), + ) + + if dtype == torch.float16: + ATOL, RTOL = (2e-3, 2e-3) + else: + ATOL, RTOL = (1e-2, 1e-2) + + torch.testing.assert_close( + qkv_fused, + ref_result, + atol=ATOL, + rtol=RTOL, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_fused_quant_layernorm.py b/asian_ai/performance/vLLM/tests/kernels/core/test_fused_quant_layernorm.py new file mode 100644 index 0000000000000000000000000000000000000000..a4ac835d567468a0ef0623abf47cfb4b9c6f3627 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_fused_quant_layernorm.py @@ -0,0 +1,335 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import itertools + +import pytest +import torch + +import vllm._custom_ops as ops +from tests.kernels.utils import fp8_ulp_distance, opcheck +from vllm.model_executor.layers.layernorm import RMSNorm +from vllm.model_executor.layers.quantization.utils.fp8_utils import ( + per_token_group_quant_fp8, +) +from vllm.model_executor.layers.quantization.utils.int8_utils import ( + per_token_group_quant_int8, +) +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +DTYPES = [torch.bfloat16, torch.float] +QUANT_DTYPES = [torch.int8, current_platform.fp8_dtype()] +VEC_HIDDEN_SIZES = [1024, 1025, 1027, 1029] +# Avoid combinatorial explosion with full Cartesian product +NUM_TOKENS_HIDDEN_SIZES = [ + *[(1, i) for i in [1, 64, 128, *VEC_HIDDEN_SIZES, 5120, 5137]], + *[(2048, i) for i in [1, 64, *VEC_HIDDEN_SIZES, 5137]], + *[(4096, i) for i in [1, 64, 5137]], +] + +ADD_RESIDUAL = [False, True] +SCALE_UBS = [True, False] +GROUP_SIZES = [None, [1, 64], [1, 128]] +TMA_ALIGNMENTS = [0, 4] +SEEDS = [0] +CUDA_DEVICES = [ + f"cuda:{i}" for i in range(1 if torch.accelerator.device_count() == 1 else 2) +] + +EPS = 1e-6 + +## Helpers + + +def as_float32_tensor(x: float | torch.Tensor) -> torch.Tensor: + return torch.as_tensor(x, dtype=torch.float32, device="cuda") + + +def ref_rms_norm( + rms_norm_layer: RMSNorm, x: torch.Tensor, residual: torch.Tensor | None +) -> tuple[torch.Tensor, torch.Tensor | None]: + if residual is not None: + residual = residual.clone() + out, residual = rms_norm_layer.forward_native(x, residual) + else: + out = rms_norm_layer.forward_native(x) + + return out, residual + + +def ref_dynamic_per_token_or_block_quant( + rms_norm_layer: RMSNorm, + x: torch.Tensor, + quant_dtype: torch.dtype, + residual: torch.Tensor | None, + scale_ub: torch.Tensor | None, + group_size: list[int] | None, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor | None]: + if scale_ub is not None: + assert quant_dtype == current_platform.fp8_dtype() + + # Norm + torch_out, residual = ref_rms_norm(rms_norm_layer, x, residual) + + # Quant + if group_size is not None: + if quant_dtype == current_platform.fp8_dtype(): + torch_out, scales = per_token_group_quant_fp8( + torch_out, group_size=group_size[1], use_ue8m0=False + ) + else: + assert quant_dtype == torch.int8 + torch_out, scales = per_token_group_quant_int8( + torch_out, group_size=group_size[1] + ) + else: + if quant_dtype == current_platform.fp8_dtype(): + torch_out, scales = ops.scaled_fp8_quant( + torch_out, scale_ub=scale_ub, use_per_token_if_dynamic=True + ) + else: + assert quant_dtype == torch.int8 + torch_out, scales, _ = ops.scaled_int8_quant(torch_out) + + return torch_out, scales, residual + + +def ref_impl( + rms_norm_layer: RMSNorm, + x: torch.Tensor, + quant_dtype: torch.dtype, + residual: torch.Tensor | None, + scale_ub: torch.Tensor | None, + group_size: list[int] | None, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor | None]: + return ref_dynamic_per_token_or_block_quant( + rms_norm_layer, x, quant_dtype, residual, scale_ub, group_size + ) + + +def ops_dynamic_per_token_or_block_quant( + weight: torch.Tensor, + x: torch.Tensor, + quant_dtype: torch.dtype, + residual: torch.Tensor | None, + scale_ub: torch.Tensor | None, + group_size: list[int] | None, + tma_alignment: int, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor | None]: + if residual is not None: + residual = residual.clone() + if group_size is not None: + out, scales = ops.rms_norm_per_block_quant( + x, + weight, + EPS, + quant_dtype, + group_size, + scale_ub, + residual, + True, + tma_alignment, + ) + scales = scales.contiguous() + else: + out, scales = ops.rms_norm_dynamic_per_token_quant( + x, weight, EPS, quant_dtype, scale_ub, residual + ) + return out, scales, residual + + +def ops_impl( + weight: torch.Tensor, + x: torch.Tensor, + quant_dtype: torch.dtype, + residual: torch.Tensor | None, + scale_ub: torch.Tensor | None, + group_size: list[int] | None, + tma_alignment: int, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor | None]: + return ops_dynamic_per_token_or_block_quant( + weight, x, quant_dtype, residual, scale_ub, group_size, tma_alignment + ) + + +@pytest.mark.parametrize("num_tokens, hidden_size", NUM_TOKENS_HIDDEN_SIZES) +@pytest.mark.parametrize("add_residual", ADD_RESIDUAL) +@pytest.mark.parametrize("has_scale_ub", SCALE_UBS) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("quant_dtype", QUANT_DTYPES) +@pytest.mark.parametrize( + "group_size, tma_alignment", + [(None, 0), *itertools.product(GROUP_SIZES, TMA_ALIGNMENTS)], +) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("strided_input", [False, True]) +@torch.inference_mode() +def test_rms_norm( + default_vllm_config, + num_tokens: int, + hidden_size: int, + add_residual: bool, + has_scale_ub: bool, + dtype: torch.dtype, + quant_dtype: torch.dtype, + group_size: list[int] | None, + tma_alignment: int, + seed: int, + device: str, + strided_input: bool, +) -> None: + set_random_seed(seed) + torch.set_default_device(device) + torch.accelerator.set_device_index(device) + + if group_size is not None and hidden_size % group_size[1] != 0: + # skip + pytest.skip("Skip non-divisible group sizes") + + if group_size is not None and has_scale_ub: + # blockwise baseline doesn't support scale_ub + pytest.skip("scale_ub not supported for blockwise/group quantization") + + if ( + group_size is None or quant_dtype != current_platform.fp8_dtype() + ) and tma_alignment != 0: + # TMA alignment is only supported for groupwise fp8 kernels + pytest.skip("tma alignment not supported for per-token or int8 quantization") + + if ( + group_size is not None + and tma_alignment != 0 + and hidden_size // group_size[1] % tma_alignment == 0 + ): + # Skip tests where TMA alignment doesn't create extra padding to save time + pytest.skip("Skip TMA alignment cases where no extra padding is added") + + if has_scale_ub and quant_dtype != current_platform.fp8_dtype(): + # skip + pytest.skip("scale_ub only supported for fp8 quantization") + + layer = RMSNorm(hidden_size, EPS).to(dtype=dtype) + + # Make weights + layer.weight.data.normal_(mean=1.0, std=0.1) + + # Make inputs: use a wider tensor and slice to create a non-contiguous + # (strided) input when strided_input=True. The last dimension stride + # remains 1, which the kernel requires. + scale = 1 / (hidden_size) + last_dim = 2 * hidden_size if strided_input else hidden_size + x = torch.randn(num_tokens, last_dim, dtype=dtype) * scale + x = x[:, :hidden_size] + + # dim 1 gets special-cased + x_is_strided = strided_input and num_tokens != 1 + # check that the input is strided iff we expect it to be + assert x.is_contiguous() != x_is_strided + + # Residual must still be contiguous + residual = ( + torch.randn(num_tokens, hidden_size, dtype=dtype) * scale + if add_residual + else None + ) + if has_scale_ub: + rms_x, _ = ref_rms_norm(layer, x, residual) + scale_ub = torch.mean(rms_x).to(dtype=torch.float32, device="cuda") + else: + scale_ub = None + + ref_out, ref_scales, ref_residual = ref_impl( + layer, x, quant_dtype, residual, scale_ub, group_size + ) + ops_out, ops_scales, ops_residual = ops_impl( + layer.weight, x, quant_dtype, residual, scale_ub, group_size, tma_alignment + ) + + assert ref_out.dtype == quant_dtype + assert ops_out.dtype == quant_dtype + + # Per-block bf16 scales: allow a small relative tolerance for a few groups + # whose abs-max flips by one ULP between the fused and reference paths. The + # per-token and fp32 paths stay strict. + relax_block_rocm = ( + group_size is not None + and dtype == torch.bfloat16 + and current_platform.is_rocm() + ) + + def scales_close(rtol: float, atol: float) -> bool: + if torch.allclose(ref_scales, ops_scales, rtol=rtol, atol=atol): + return True + return relax_block_rocm and torch.allclose( + ref_scales, ops_scales, rtol=1e-2, atol=atol + ) + + if quant_dtype == torch.int8: + assert scales_close(rtol=1e-5, atol=1e-6) + # big atol to account for round-off errors. + assert torch.allclose(ref_out, ops_out, atol=1) + else: + assert scales_close(rtol=1e-5, atol=1e-8) + a = ref_out.to(dtype=torch.float32) + b = ops_out.to(dtype=torch.float32) + ok = torch.allclose(a, b, atol=1e-6) + if not ok: + if relax_block_rocm: + # ULP-flipped group scale can cross an E4M3 tie; tolerate a + # bounded count of isolated fp8 outliers. + ulp = fp8_ulp_distance(ref_out, ops_out) + max_outliers = ulp.numel() // 100_000 + 8 + ok = int((ulp > 0).sum().item()) <= max_outliers + else: + # CUDA (& non-bf16): compare dequantized values with relaxed tolerance. + if group_size is None: + a_deq = a * ref_scales.view(-1, 1) + b_deq = b * ops_scales.view(-1, 1) + else: + a_deq = a * ref_scales.repeat_interleave(group_size[1], dim=1) + b_deq = b * ops_scales.repeat_interleave(group_size[1], dim=1) + # NOTE: It is possible that some future test cases trigger this + # max diff due to precision issues. If such an error is + # encountered, it's recommended to inspect the differences between + # all corresponding elements from each tensor (e.g. by looping over + # them) and checking how many the max diff error shows up on (just + # a few bad elements should still be considered acceptable). + ok = torch.allclose(a_deq, b_deq, rtol=5e-2, atol=5e-2) + assert ok + if add_residual: + assert torch.allclose(ref_residual, ops_residual) + + output = torch.empty(x.shape, dtype=quant_dtype, device=x.device) + if group_size is None: + scales = torch.empty( + (x.numel() // x.shape[-1], 1), device=x.device, dtype=torch.float32 + ) + opcheck( + torch.ops._C.rms_norm_dynamic_per_token_quant, + (output, x, layer.weight, scales, 1e-5, scale_ub, residual), + ) + else: + assert hidden_size % group_size[1] == 0 + num_groups = hidden_size // group_size[1] + scales = torch.empty( + (num_groups, num_tokens), + device=x.device, + dtype=torch.float32, + ).transpose(0, 1) + opcheck( + torch.ops._C.rms_norm_per_block_quant, + ( + output, + x, + layer.weight, + scales, + 1e-5, + scale_ub, + residual, + group_size[1], + True, # is_scale_transposed + ), + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_fused_rms_norm_gated.py b/asian_ai/performance/vLLM/tests/kernels/core/test_fused_rms_norm_gated.py new file mode 100644 index 0000000000000000000000000000000000000000..57d4264c00946bbf8ef52a3a3c6139b1efdaef1d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_fused_rms_norm_gated.py @@ -0,0 +1,103 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Tests that FusedRMSNormGated decomposes correctly under torch.compile, +matching the eager triton kernel output.""" + +import pytest +import torch + +from vllm.model_executor.layers.fla.ops.kda import FusedRMSNormGated +from vllm.utils.torch_utils import set_random_seed + +DTYPES = [torch.bfloat16] +HIDDEN_SIZES = [128, 512] +NUM_TOKENS = [64, 128] +ACTIVATIONS = ["swish", "sigmoid"] +ELEMENTWISE_AFFINE = [True, False] +SEEDS = [0] + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", HIDDEN_SIZES) +@pytest.mark.parametrize("activation", ACTIVATIONS) +@pytest.mark.parametrize("elementwise_affine", ELEMENTWISE_AFFINE) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@torch.inference_mode() +def test_compiled_vs_eager( + default_vllm_config, + num_tokens: int, + hidden_size: int, + activation: str, + elementwise_affine: bool, + dtype: torch.dtype, + seed: int, +) -> None: + """forward_native decomposition matches forward_cuda triton kernel.""" + torch._dynamo.reset() + set_random_seed(seed) + device = torch.device("cuda:0") + + module = FusedRMSNormGated( + hidden_size, + elementwise_affine=elementwise_affine, + eps=1e-5, + activation=activation, + device=device, + dtype=dtype, + ) + x = torch.randn(num_tokens, hidden_size, dtype=dtype, device=device) + g = torch.randn(num_tokens, hidden_size, dtype=dtype, device=device) + + # forward_cuda may modify x in-place, so clone inputs + cuda_out = module.forward_cuda(x.clone(), g.clone()) + compiled_native = torch.compile(module.forward_native, fullgraph=True) + native_out = compiled_native(x.clone(), g.clone()) + + torch.testing.assert_close(native_out, cuda_out, atol=1e-3, rtol=1e-2) + + +@pytest.mark.parametrize( + "shape", + [ + (1, 16, 32, 128), + (2, 8, 16, 64), + ], +) +@pytest.mark.parametrize("activation", ACTIVATIONS) +@pytest.mark.parametrize("elementwise_affine", ELEMENTWISE_AFFINE) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@torch.inference_mode() +def test_compiled_vs_eager_multidim( + default_vllm_config, + shape: tuple, + activation: str, + elementwise_affine: bool, + dtype: torch.dtype, + seed: int, +) -> None: + """forward_native decomposition handles multi-dimensional inputs.""" + torch._dynamo.reset() + set_random_seed(seed) + device = torch.device("cuda:0") + head_dim = shape[-1] + + module = FusedRMSNormGated( + head_dim, + elementwise_affine=elementwise_affine, + eps=1e-5, + activation=activation, + device=device, + dtype=dtype, + ) + x = torch.randn(*shape, dtype=dtype, device=device) + g = torch.randn(*shape, dtype=dtype, device=device) + + # forward_cuda may modify x in-place, so clone inputs + cuda_out = module.forward_cuda(x.clone(), g.clone()) + compiled_native = torch.compile(module.forward_native, fullgraph=True) + native_out = compiled_native(x.clone(), g.clone()) + + torch.testing.assert_close(native_out, cuda_out, atol=1e-3, rtol=1e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_fused_silu_mul_block_quant.py b/asian_ai/performance/vLLM/tests/kernels/core/test_fused_silu_mul_block_quant.py new file mode 100644 index 0000000000000000000000000000000000000000..58143aa2ac87a9e8f210564c070c5b3a9d436de4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_fused_silu_mul_block_quant.py @@ -0,0 +1,189 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch +import torch.nn.functional as F + +import vllm._custom_ops as ops +from tests.kernels.utils import opcheck +from vllm.model_executor.layers.quantization.utils.fp8_utils import ( + per_token_group_quant_fp8, +) +from vllm.model_executor.layers.quantization.utils.int8_utils import ( + per_token_group_quant_int8, +) +from vllm.platforms import current_platform + +DTYPES = [torch.float16, torch.bfloat16] +QUANT_DTYPES = [current_platform.fp8_dtype(), torch.int8] +VEC_HIDDEN_SIZES = [1024, 1025, 1027, 1029] +NUM_TOKENS_HIDDEN_SIZES = [ + *[(1, i) for i in [64, *VEC_HIDDEN_SIZES, 2048, 5120]], + *[(16, i) for i in [64, *VEC_HIDDEN_SIZES, 5120]], + *[(128, i) for i in [64, *VEC_HIDDEN_SIZES]], + *[(512, i) for i in [64, 5120]], +] +SCALE_UBS = [False] +GROUP_SIZES = [64, 128] +IS_SCALE_TRANSPOSED = [False, True] +SEEDS = [0] +CUDA_DEVICES = [i for i in range(1 if torch.accelerator.device_count() == 1 else 2)] + + +def ref_silu_and_mul_per_block_quant( + x: torch.Tensor, + quant_dtype: torch.dtype, + group_size: int, +) -> tuple[torch.Tensor, torch.Tensor]: + """Reference implementation: unfused SiLU+Mul then group quantization.""" + hidden = x.shape[-1] // 2 + gate, up = x.split(hidden, dim=-1) + silu_out = F.silu(gate) * up + + if quant_dtype == current_platform.fp8_dtype(): + return per_token_group_quant_fp8( + silu_out, group_size=group_size, use_ue8m0=False + ) + elif quant_dtype == torch.int8: + return per_token_group_quant_int8(silu_out, group_size=group_size) + else: + raise ValueError(f"Unsupported quant_dtype: {quant_dtype}") + + +@pytest.mark.parametrize("num_tokens, hidden_size", NUM_TOKENS_HIDDEN_SIZES) +@pytest.mark.parametrize("has_scale_ub", SCALE_UBS) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("quant_dtype", QUANT_DTYPES) +@pytest.mark.parametrize("group_size", GROUP_SIZES) +@pytest.mark.parametrize("is_scale_transposed", IS_SCALE_TRANSPOSED) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device_idx", CUDA_DEVICES) +@torch.inference_mode() +def test_silu_and_mul_per_block_quant( + default_vllm_config, + num_tokens: int, + hidden_size: int, + has_scale_ub: bool, + dtype: torch.dtype, + quant_dtype: torch.dtype, + group_size: int, + is_scale_transposed: bool, + seed: int, + device_idx: str, +) -> None: + """Test SiLU+Mul+Block Quantization kernel correctness.""" + torch.accelerator.set_device_index(device_idx) + device = f"cuda:{device_idx}" + torch.random.manual_seed(seed) + torch.set_default_device(device) + + if hidden_size % group_size != 0: + return + + if has_scale_ub: + pytest.skip("Scale upper bound not yet supported") + + scale = 1 / hidden_size + x = torch.randn(num_tokens, hidden_size * 2, dtype=dtype, device=device) * scale + + # Reference implementation + ref_out, ref_scales = ref_silu_and_mul_per_block_quant(x, quant_dtype, group_size) + + # Fused kernel implementation + ops_out, ops_scales = ops.silu_and_mul_per_block_quant( + x, group_size, quant_dtype, None, is_scale_transposed + ) + + # Check for NaN/Inf + assert not torch.isnan(ops_out.float()).any(), "Kernel output contains NaN" + assert not torch.isinf(ops_out.float()).any(), "Kernel output contains Inf" + assert not torch.isnan(ops_scales).any(), "Kernel scales contain NaN" + assert not torch.isinf(ops_scales).any(), "Kernel scales contain Inf" + + # Check dtypes + assert ref_out.dtype == quant_dtype + assert ops_out.dtype == quant_dtype + + # Check scales match + torch.testing.assert_close(ref_scales, ops_scales, rtol=1e-5, atol=1e-5) + + # Check output correctness via dequantized values + ref_scales_expanded = ref_scales.repeat_interleave(group_size, dim=1) + ops_scales_expanded = ops_scales.repeat_interleave(group_size, dim=1) + ref_deq = ref_out.to(dtype=torch.float32) * ref_scales_expanded + ops_deq = ops_out.to(dtype=torch.float32) * ops_scales_expanded + torch.testing.assert_close(ref_deq, ops_deq, atol=5e-2, rtol=5e-2) + + # opcheck + output = torch.empty(num_tokens, hidden_size, device=device, dtype=quant_dtype) + num_groups = hidden_size // group_size + if is_scale_transposed: + scales = torch.empty(num_groups, num_tokens, device=device, dtype=torch.float32) + else: + scales = torch.empty(num_tokens, num_groups, device=device, dtype=torch.float32) + opcheck( + torch.ops._C.silu_and_mul_per_block_quant, + (output, x, scales, group_size, None, is_scale_transposed), + ) + + +@pytest.mark.parametrize("dtype", [torch.float16]) +@pytest.mark.parametrize("hidden_size", [4096]) +@pytest.mark.parametrize("num_tokens", [128]) +@pytest.mark.parametrize("group_size", [128]) +def test_silu_block_quant_shapes( + default_vllm_config, + dtype: torch.dtype, + hidden_size: int, + num_tokens: int, + group_size: int, +): + """Test that output shapes are correct.""" + torch.set_default_device("cuda") + x = torch.randn(num_tokens, hidden_size * 2, dtype=dtype, device="cuda") + + # Row-major scales + out, scales = ops.silu_and_mul_per_block_quant( + x, + group_size=group_size, + quant_dtype=current_platform.fp8_dtype(), + is_scale_transposed=False, + ) + assert out.shape == (num_tokens, hidden_size) + assert scales.shape == (num_tokens, hidden_size // group_size) + + # Column-major scales (logical shape same after .t() in _custom_ops) + out, scales = ops.silu_and_mul_per_block_quant( + x, + group_size=group_size, + quant_dtype=current_platform.fp8_dtype(), + is_scale_transposed=True, + ) + assert out.shape == (num_tokens, hidden_size) + assert scales.shape == (num_tokens, hidden_size // group_size) + + +@pytest.mark.parametrize("dtype", [torch.float16]) +@pytest.mark.parametrize("batch_size", [1, 16, 256]) +@pytest.mark.parametrize("hidden_size", [1024, 5120, 14336]) +def test_silu_block_quant_edge_cases( + default_vllm_config, dtype: torch.dtype, batch_size: int, hidden_size: int +): + """Test edge cases: single token, large batch, large hidden size.""" + torch.set_default_device("cuda") + x = torch.randn(batch_size, hidden_size * 2, dtype=dtype, device="cuda") + + out, scales = ops.silu_and_mul_per_block_quant( + x, + group_size=128, + quant_dtype=current_platform.fp8_dtype(), + is_scale_transposed=False, + ) + + assert out.shape == (batch_size, hidden_size) + assert out.dtype == current_platform.fp8_dtype() + assert scales.dtype == torch.float32 + assert not torch.isnan(out.float()).any() + assert not torch.isnan(scales).any() + assert not torch.isinf(scales).any() diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_layernorm.py b/asian_ai/performance/vLLM/tests/kernels/core/test_layernorm.py new file mode 100644 index 0000000000000000000000000000000000000000..84d00d866b8b5efe0198be5ae3aecc3670aa9b8a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_layernorm.py @@ -0,0 +1,250 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from tests.kernels.quant_utils import FP8_DTYPE +from tests.kernels.utils import fp8_ulp_distance, opcheck +from vllm import ir +from vllm.model_executor.layers.layernorm import GemmaRMSNorm, RMSNorm +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +if current_platform.is_rocm(): + from vllm.platforms.rocm import on_gfx90a + + on_mi250 = on_gfx90a() +else: + on_mi250 = False + +DTYPES = [torch.half, torch.bfloat16, torch.float] +NUM_TOKENS = [7, 83, 4096] # Arbitrary values for testing +HIDDEN_SIZES = [8, 768, 769, 5120, 5125, 8192] # Arbitrary values for testing +ADD_RESIDUAL = [False, True] if not on_mi250 else [True] +SEEDS = [0] +CUDA_DEVICES = [ + f"cuda:{i}" for i in range(1 if torch.accelerator.device_count() == 1 else 2) +] + + +def _rms_norm_tolerance(dtype: torch.dtype) -> dict[str, float]: + return ir.ops.rms_norm.get_tolerance(dtype) + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", HIDDEN_SIZES) +@pytest.mark.parametrize("add_residual", ADD_RESIDUAL) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("strided_input", [False, True]) +@torch.inference_mode() +def test_rms_norm( + default_vllm_config, + num_tokens: int, + hidden_size: int, + add_residual: bool, + dtype: torch.dtype, + seed: int, + device: str, + strided_input: bool, +) -> None: + set_random_seed(seed) + torch.set_default_device(device) + layer = RMSNorm(hidden_size).to(dtype=dtype) + layer.weight.data.normal_(mean=1.0, std=0.1) + scale = 1 / (2 * hidden_size) + last_dim = 2 * hidden_size if strided_input else hidden_size + x = torch.randn(num_tokens, last_dim, dtype=dtype) + x = x[..., :hidden_size] + assert x.is_contiguous() != strided_input + x *= scale + residual = torch.randn_like(x) * scale if add_residual else None + + # NOTE(woosuk): The reference implementation should be executed first + # because the custom kernel is in-place. + ref_out = layer.forward_native(x, residual) + out = layer(x, residual) + # NOTE(woosuk): LayerNorm operators (including RMS) typically have larger + # numerical errors than other operators because they involve reductions. + # Therefore, we use a larger tolerance. + if add_residual: + torch.testing.assert_close(out[0], ref_out[0], atol=1e-2, rtol=1e-2) + torch.testing.assert_close(out[1], ref_out[1], atol=1e-2, rtol=1e-2) + else: + torch.testing.assert_close(out, ref_out, atol=1e-2, rtol=1e-2) + + if residual is not None: + opcheck( + torch.ops._C.fused_add_rms_norm, + (x, residual, layer.weight.data, layer.variance_epsilon), + ) + else: + opcheck( + torch.ops._C.rms_norm, (out, x, layer.weight.data, layer.variance_epsilon) + ) + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", HIDDEN_SIZES) +@pytest.mark.parametrize("add_residual", ADD_RESIDUAL) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@torch.inference_mode() +def test_rms_norm_weightless( + default_vllm_config, + num_tokens: int, + hidden_size: int, + add_residual: bool, + dtype: torch.dtype, + seed: int, + device: str, +) -> None: + set_random_seed(seed) + torch.set_default_device(device) + layer = RMSNorm(hidden_size, has_weight=False).to(dtype=dtype) + x = torch.randn(num_tokens, hidden_size, dtype=dtype) + residual = torch.randn_like(x) if add_residual else None + + ref_out = layer.forward_native(x, residual) + out = layer(x, residual) + tol = _rms_norm_tolerance(dtype) + if add_residual: + torch.testing.assert_close(out[0], ref_out[0], **tol) + torch.testing.assert_close(out[1], ref_out[1], **tol) + else: + torch.testing.assert_close(out, ref_out, **tol) + + if residual is not None: + opcheck( + torch.ops._C.fused_add_rms_norm, + (x, residual, None, layer.variance_epsilon), + ) + else: + opcheck( + torch.ops._C.rms_norm, + (out, x, None, layer.variance_epsilon), + ) + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", HIDDEN_SIZES) +@pytest.mark.parametrize("add_residual", ADD_RESIDUAL) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("quant_scale", [0.01, 1.0, 10.0]) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("strided_input", [False, True]) +def test_fused_rms_norm_quant( + num_tokens: int, + hidden_size: int, + add_residual: bool, + dtype: torch.dtype, + quant_scale: float, + seed: int, + device: str, + strided_input: bool, +) -> None: + set_random_seed(seed) + torch.set_default_device(device) + + weight = torch.empty(hidden_size, dtype=dtype).normal_(mean=1.0, std=0.1) + scale = 1 / (2 * hidden_size) + last_dim = 2 * hidden_size if strided_input else hidden_size + x_base = torch.randn(num_tokens, last_dim, dtype=dtype) + x = x_base[..., :hidden_size] + assert x.is_contiguous() != strided_input + + x *= scale + if add_residual: + residual = torch.randn_like(x) * scale + residual_fused = residual.clone() + else: + residual = residual_fused = None + + out_norm = torch.empty_like(x) + out_quant = torch.empty_like(x, dtype=FP8_DTYPE) + out_quant_fused = torch.empty_like(out_quant) + + quant_scale_t = torch.tensor(quant_scale, dtype=torch.float32) + + if add_residual: + torch.ops._C.fused_add_rms_norm_static_fp8_quant( + out_quant_fused, x, residual_fused, weight, quant_scale_t, 1e-6 + ) + + # Unfused kernel is in-place so it goes second + # Also use a separate clone of x to avoid modifying the input + x_unfused_base = x_base.clone() + x_unfused = x_unfused_base[..., :hidden_size] + assert x_unfused.is_contiguous() != strided_input + torch.ops._C.fused_add_rms_norm(x_unfused, residual, weight, 1e-6) + torch.ops._C.static_scaled_fp8_quant( + out_quant, x_unfused.contiguous(), quant_scale_t + ) + + torch.accelerator.synchronize() + torch.testing.assert_close(residual_fused, residual, atol=1e-2, rtol=1e-2) + opcheck( + torch.ops._C.fused_add_rms_norm_static_fp8_quant, + (out_quant_fused, x, residual_fused, weight, quant_scale_t, 1e-6), + ) + else: + torch.ops._C.rms_norm_static_fp8_quant( + out_quant_fused, x, weight, quant_scale_t, 1e-6 + ) + + torch.ops._C.rms_norm(out_norm, x, weight, 1e-6) + torch.ops._C.static_scaled_fp8_quant(out_quant, out_norm, quant_scale_t) + + opcheck( + torch.ops._C.rms_norm_static_fp8_quant, + (out_quant_fused, x, weight, quant_scale_t, 1e-6), + ) + + if current_platform.is_rocm(): + # Fused and unfused FP8 paths can land on opposite sides of an E4M3 tie; + # tolerate a tiny number of isolated fp8 outliers on ROCm. + ulp = fp8_ulp_distance(out_quant, out_quant_fused) + max_outliers = ulp.numel() // 100_000 + 8 + num_outliers = int((ulp > 0).sum().item()) + assert num_outliers <= max_outliers, ( + f"FP8 quant mismatch: {num_outliers} fp8 outliers (allowed {max_outliers})" + ) + else: + torch.testing.assert_close( + out_quant.to(dtype=torch.float32), + out_quant_fused.to(dtype=torch.float32), + atol=1e-3, + rtol=1e-3, + ) + + +@torch.inference_mode() +def test_gemma_rms_norm_mixed_input_weight_dtype(default_vllm_config) -> None: + if not torch.cuda.is_available(): + pytest.skip("CUDA required") + + device = CUDA_DEVICES[0] + torch.set_default_device(device) + + num_tokens, hidden_size = 32, 1024 + x = torch.randn(num_tokens, hidden_size, dtype=torch.bfloat16, device=device) + layer = GemmaRMSNorm(hidden_size, eps=1e-6).to(device=device) + layer.weight.data.normal_(mean=0.0, std=0.1) + + # Gemma uses fp32 weight parameter while activations can be bf16. + assert layer.weight.dtype == torch.float32 + out = layer(x) + + x_fp32 = x.float() + weight_fp32 = layer.weight.data.float() + 1.0 + variance = x_fp32.pow(2).mean(dim=-1, keepdim=True) + ref = (x_fp32 * torch.rsqrt(variance + layer.variance_epsilon) * weight_fp32).to( + x.dtype + ) + + assert out.dtype == x.dtype + torch.testing.assert_close(out, ref, atol=1e-2, rtol=1e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_minimax_reduce_rms.py b/asian_ai/performance/vLLM/tests/kernels/core/test_minimax_reduce_rms.py new file mode 100644 index 0000000000000000000000000000000000000000..55739a7ab4f3acc790943ed414854b23fedaf017 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_minimax_reduce_rms.py @@ -0,0 +1,208 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for MiniMax QK RMS-norm: NCCL reference vs Lamport fused kernel.""" + +import pytest +import torch +import torch.nn as nn +from torch.multiprocessing import spawn + +from tests.kernels.utils import opcheck +from tests.utils import ensure_current_vllm_config, init_test_distributed_environment +from vllm.distributed import cleanup_dist_env_and_memory +from vllm.model_executor.layers.minimax_rms_norm import ( + MiniMaxText01RMSNormTP, + rms_norm_tp, +) +from vllm.platforms import current_platform +from vllm.triton_utils import HAS_TRITON +from vllm.utils.network_utils import get_open_port +from vllm.utils.torch_utils import set_random_seed + + +@ensure_current_vllm_config() +def _worker_forward_qk( + local_rank, + world_size, + port, + num_tokens, + hidden_q_full, + hidden_k_full, + dtype, + seed, + eps, +): + """Per-rank worker: compare NCCL allreduce path vs Lamport fused kernel.""" + + if not hasattr(torch.ops._C, "minimax_allreduce_rms_qk"): + cleanup_dist_env_and_memory() + return + device = torch.device(f"cuda:{local_rank}") + torch.accelerator.set_device_index(device) + init_test_distributed_environment( + world_size, 1, local_rank, port, local_rank=local_rank + ) + + hq = hidden_q_full // world_size + hk = hidden_k_full // world_size + + q_norm = MiniMaxText01RMSNormTP(hidden_q_full, eps=eps).cuda() + k_norm = MiniMaxText01RMSNormTP(hidden_k_full, eps=eps).cuda() + + set_random_seed(seed) + qw = torch.randn(hidden_q_full, dtype=dtype, device="cuda") + kw = torch.randn(hidden_k_full, dtype=dtype, device="cuda") + q_norm.weight = nn.Parameter(qw[local_rank * hq : (local_rank + 1) * hq]) + k_norm.weight = nn.Parameter(kw[local_rank * hk : (local_rank + 1) * hk]) + + torch.manual_seed(seed + 1000 + local_rank) + qkv = torch.randn(num_tokens, hq + hk + hk, dtype=dtype, device="cuda") + + # Reference: eager all-reduce path. ``forward_qk`` no longer all-reduces + # the variance (it is the tp==1 / already-reduced building block), so the + # multi-rank reference must use the eager path that performs the global + # variance all-reduce, matching the fused kernel below. + ref_q, ref_k = rms_norm_tp._minimax_qk_norm_tp_eager( + qkv.clone(), + q_norm.weight, + k_norm.weight, + hq, + hk, + world_size, + eps, + ) + + # Set up Lamport workspace. + from vllm.distributed.parallel_state import get_tp_group + from vllm.model_executor.layers.minimax_rms_norm.lamport_workspace import ( + get_allreduce_workspace, + ) + + workspace = get_allreduce_workspace( + rank=local_rank, + world_size=world_size, + max_tokens=num_tokens, + process_group=get_tp_group().cpu_group, + ) + + opcheck( + torch.ops._C.minimax_allreduce_rms_qk, + ( + qkv.clone(), + q_norm.weight, + k_norm.weight, + workspace, + hq, + hk, + local_rank, + world_size, + eps, + ), + ) + fused_q, fused_k = torch.ops._C.minimax_allreduce_rms_qk( + qkv.clone(), + q_norm.weight, + k_norm.weight, + workspace, + hq, + hk, + local_rank, + world_size, + eps, + ) + _, _, fused_v = qkv.split([hq, hk, hk], dim=-1) + torch.accelerator.synchronize() + + torch.testing.assert_close( + fused_q, + ref_q, + atol=3e-2, + rtol=3e-2, + ) + torch.testing.assert_close(fused_k, ref_k, atol=3e-2, rtol=3e-2) + + cleanup_dist_env_and_memory() + + +@pytest.mark.skipif( + not current_platform.is_cuda(), + reason="CUDA required", +) +@pytest.mark.parametrize("world_size", [2, 4, 8]) +@pytest.mark.parametrize("num_tokens", [1, 128, 333]) +@pytest.mark.parametrize( + "hidden_dims", + [(6144, 1024)], +) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) +@pytest.mark.parametrize("eps", [1e-6]) +@pytest.mark.parametrize("seed", [42]) +def test_minimax_reduce_rms_qk( + world_size, + num_tokens, + hidden_dims, + dtype, + eps, + seed, +): + num_gpus = current_platform.device_count() + if num_gpus < world_size: + pytest.skip(f"Need >= {world_size} GPUs, have {num_gpus}") + hidden_q_full, hidden_k_full = hidden_dims + port = str(get_open_port()) + spawn( + _worker_forward_qk, + args=( + world_size, + port, + num_tokens, + hidden_q_full, + hidden_k_full, + dtype, + seed, + eps, + ), + nprocs=world_size, + join=True, + ) + + +@pytest.mark.skipif( + not current_platform.is_cuda() or not HAS_TRITON, + reason="CUDA and Triton required", +) +@pytest.mark.parametrize("num_tokens", [1, 7, 128, 333, 2049]) +@pytest.mark.parametrize("hidden_dims", [(3072, 512), (768, 256), (3000, 500)]) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) +@pytest.mark.parametrize("tp_world", [1, 4, 8]) +@pytest.mark.parametrize("eps", [1e-6]) +@pytest.mark.parametrize("seed", [42]) +def test_minimax_qk_norm_triton_fallback( + monkeypatch, num_tokens, hidden_dims, dtype, tp_world, eps, seed +): + """Single-GPU check: Triton fallback kernels vs the pure-torch reference. + + The all-reduce is a TP communication barrier, so it is monkeypatched to + identity here; both the Triton path and the reference see the same + (patched) reduction. This validates the kernel math and the folded + ``/ tp_world`` scaling without needing multiple ranks -- ``hidden_dims`` + are the per-rank q/k segment widths. + """ + monkeypatch.setattr(rms_norm_tp, "_all_reduce_variance", lambda v: v) + + q_size, kv_size = hidden_dims + device = "cuda" + torch.manual_seed(seed) + qkv = torch.randn(num_tokens, q_size + 2 * kv_size, dtype=dtype, device=device) + q_weight = torch.randn(q_size, dtype=dtype, device=device) + k_weight = torch.randn(kv_size, dtype=dtype, device=device) + + q_triton, k_triton = rms_norm_tp._minimax_qk_norm_tp_fallback( + qkv, q_weight, k_weight, q_size, kv_size, 0, tp_world, eps + ) + q_ref, k_ref = rms_norm_tp._minimax_qk_norm_tp_eager( + qkv, q_weight, k_weight, q_size, kv_size, tp_world, eps + ) + + torch.testing.assert_close(q_triton, q_ref, atol=3e-2, rtol=3e-2) + torch.testing.assert_close(k_triton, k_ref, atol=3e-2, rtol=3e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_mrope.py b/asian_ai/performance/vLLM/tests/kernels/core/test_mrope.py new file mode 100644 index 0000000000000000000000000000000000000000..b9f55705fad3565bdc0443b65b3e8b8a035dfec1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_mrope.py @@ -0,0 +1,236 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from typing import NamedTuple + +import pytest +import torch + +from vllm.model_executor.layers.rotary_embedding import get_rope +from vllm.platforms import current_platform +from vllm.transformers_utils.config import get_config +from vllm.utils.torch_utils import set_random_seed + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + + +def generate_test_data( + num_tokens: int, + num_q_heads: int, + num_kv_heads: int, + head_size: int, + max_position_embeddings: int, + dtype: torch.dtype, + device: torch.device, +): + """Generate test data for given configuration.""" + set_random_seed(42) + # Create 2D positions (3, num_tokens) for multimodal case + positions = torch.randint( + 0, max_position_embeddings // 4, (3, num_tokens), device=device + ) + + # Create query and key tensors + query = torch.randn(num_tokens, num_q_heads * head_size, dtype=dtype, device=device) + key = torch.randn(num_tokens, num_kv_heads * head_size, dtype=dtype, device=device) + + return positions, query, key + + +class MRoPETestInfo(NamedTuple): + model_name: str + # https://github.com/pytorch/pytorch/blob/main/torch/testing/_comparison.py#L1317 + atol: float = 1e-2 + rtol: float = 1.6e-2 + marks: list[pytest.MarkDecorator] = [] + + +MODELS_TO_TEST = [ + MRoPETestInfo(model_name="zai-org/GLM-4.1V-9B-Thinking"), + MRoPETestInfo(model_name="Qwen/Qwen2-VL-7B-Instruct"), + MRoPETestInfo(model_name="Qwen/Qwen2-VL-72B-Instruct"), + MRoPETestInfo(model_name="Qwen/Qwen2.5-VL-72B-Instruct"), + MRoPETestInfo(model_name="Qwen/Qwen3-VL-4B-Instruct"), + MRoPETestInfo(model_name="Qwen/Qwen3-VL-30B-A3B-Instruct"), +] + +num_tokens_list = [11, 8192] + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), reason="Skipping CUDA/ROCm only tests." +) +@pytest.mark.parametrize( + "model_info, model_name", + [ + pytest.param(test_config, test_config.model_name, marks=test_config.marks) + for test_config in MODELS_TO_TEST + ], +) +@pytest.mark.parametrize("tp_size", [1, 2]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("num_tokens", num_tokens_list) +def test_mrope( + default_vllm_config, + model_name: str, + model_info: MRoPETestInfo, + tp_size: int, + dtype: torch.dtype, + num_tokens: int, +): + atol = model_info.atol + rtol = model_info.rtol + + config = get_config(model_name, False).get_text_config() + + # get the model config + total_num_kv_heads = config.num_key_value_heads + total_num_heads = config.num_attention_heads + num_heads = total_num_heads // tp_size + num_kv_heads = max(1, total_num_kv_heads // tp_size) + head_dim = ( + config.head_dim + if hasattr(config, "head_dim") + else config.hidden_size // total_num_heads + ) + is_neox_style = True + + max_position = config.max_position_embeddings + + mrope_helper_class = get_rope( + head_size=head_dim, + max_position=max_position, + is_neox_style=is_neox_style, + rope_parameters=config.rope_parameters, + dtype=dtype, + ).to(device=device) + + # create q k v input tensors + # create rotary pos emb input tensors + positions, query, key = generate_test_data( + num_tokens, num_heads, num_kv_heads, head_dim, max_position, dtype, device + ) + + query_native, key_native = mrope_helper_class.forward_native( + positions, + query.clone(), + key.clone(), + ) + + query_cuda, key_cuda = mrope_helper_class.forward_cuda( + positions, + query.clone(), + key.clone(), + ) + + torch.testing.assert_close(query_native, query_cuda, atol=atol, rtol=rtol) + torch.testing.assert_close(key_native, key_cuda, atol=atol, rtol=rtol) + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), reason="Skipping CUDA/ROCm only tests." +) +@pytest.mark.parametrize( + "model_info, model_name", + [ + pytest.param(test_config, test_config.model_name, marks=test_config.marks) + for test_config in MODELS_TO_TEST + ], +) +@pytest.mark.parametrize("tp_size", [1, 2]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("num_tokens", num_tokens_list) +def test_mrope_torch_compile_tracing( + default_vllm_config, + model_name: str, + model_info: MRoPETestInfo, + tp_size: int, + dtype: torch.dtype, + num_tokens: int, +): + atol = model_info.atol + rtol = model_info.rtol + + config = get_config(model_name, False).get_text_config() + + # get the model config + total_num_kv_heads = config.num_key_value_heads + total_num_heads = config.num_attention_heads + num_heads = total_num_heads // tp_size + num_kv_heads = max(1, total_num_kv_heads // tp_size) + head_dim = ( + config.head_dim + if hasattr(config, "head_dim") + else config.hidden_size // total_num_heads + ) + is_neox_style = True + max_position = config.max_position_embeddings + + mrope_helper_class = get_rope( + head_size=head_dim, + max_position=max_position, + is_neox_style=is_neox_style, + rope_parameters=config.rope_parameters, + dtype=dtype, + ).to(device=device) + + # Generate test data + positions, query, key = generate_test_data( + num_tokens, num_heads, num_kv_heads, head_dim, max_position, dtype, device + ) + + # Create a wrapper that makes the in-place function appear functional + def functional_forward_cuda(pos, q, k): + """Wrapper that converts in-place operation to functional style + + CUDA Graph does not support in-place operations. + This wrapper creates working copies of the + input tensors and modifies them. + """ + q_work = q.clone() # Create working copies + k_work = k.clone() + # Your in-place function modifies q_work and k_work + mrope_helper_class.forward_cuda(pos, q_work, k_work) + return q_work, k_work # Return the modified tensors + + # Get reference results + query_native, key_native = mrope_helper_class.forward_native( + positions, + query.clone(), + key.clone(), + ) + + try: + compiled_forward_cuda = torch.compile( + functional_forward_cuda, + fullgraph=True, + backend="inductor", + mode="reduce-overhead", + dynamic=False, + ) + + # Run compiled version + query_compiled_cuda, key_compiled_cuda = compiled_forward_cuda( + positions, + query, + key, + ) + + # Run original version for comparison + query_cuda = query.clone() + key_cuda = key.clone() + mrope_helper_class.forward_cuda(positions, query_cuda, key_cuda) + + # Verify results + torch.testing.assert_close( + query_compiled_cuda, query_cuda, atol=atol, rtol=rtol + ) + torch.testing.assert_close(key_compiled_cuda, key_cuda, atol=atol, rtol=rtol) + torch.testing.assert_close( + query_compiled_cuda, query_native, atol=atol, rtol=rtol + ) + torch.testing.assert_close(key_compiled_cuda, key_native, atol=atol, rtol=rtol) + + print("✓ forward_cuda successfully traced with torch.compile inductor") + + except Exception as e: + pytest.fail(f"forward_cuda failed to trace with torch.compile inductor: {e}") diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_opcheck.py b/asian_ai/performance/vLLM/tests/kernels/core/test_opcheck.py new file mode 100644 index 0000000000000000000000000000000000000000..c5cad8da48904dde2a00b118f5a93de667f1897c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_opcheck.py @@ -0,0 +1,26 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Tests for miscellaneous utilities +""" + +import torch + +from tests.kernels.utils import opcheck + + +def test_convert_fp8_opcheck(): + data = torch.randn((256, 256), dtype=torch.float32, device="cuda") + result = torch.empty_like(data, dtype=torch.float8_e4m3fn) + opcheck(torch.ops._C_cache_ops.convert_fp8, (result, data, 1.0, "fp8")) + + +# TODO: Add this back, currently fails with +# csrc/cuda_utils_kernels.cu:15 'invalid argument' +# @pytest.mark.skipif(not current_platform.is_cuda(), +# reason="Only supported for CUDA") +# def test_cuda_utils_opcheck(): +# opcheck(torch.ops._C_cuda_utils.get_device_attribute, (0, 0)) +# opcheck( +# torch.ops._C_cuda_utils. +# get_max_shared_memory_per_block_device_attribute, (0, )) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_permute_cols.py b/asian_ai/performance/vLLM/tests/kernels/core/test_permute_cols.py new file mode 100644 index 0000000000000000000000000000000000000000..1880d88883dcc9b6267e28640d1a84fdb429698e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_permute_cols.py @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from tests.kernels.utils import opcheck +from vllm._custom_ops import permute_cols + +if not hasattr(torch.ops._C, "permute_cols"): + pytest.skip(reason="permute_cols is not supported on ROCm", allow_module_level=True) + + +@pytest.mark.parametrize("shape", [(1, 512), (544, 4096), (67, 8192)]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +def test_permute_cols(shape, dtype): + x = torch.randn(shape, dtype=dtype).cuda() + perm = torch.randperm(x.shape[1]).to(torch.int).cuda() + opcheck(torch.ops._C.permute_cols, (x, perm)) + y = permute_cols(x, perm) + torch.testing.assert_close(y, x[:, perm]) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_pos_encoding.py b/asian_ai/performance/vLLM/tests/kernels/core/test_pos_encoding.py new file mode 100644 index 0000000000000000000000000000000000000000..0cae6e8164b290cce6b893bc0f0abeb6921fa79d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_pos_encoding.py @@ -0,0 +1,193 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from collections.abc import Callable +from itertools import product + +import pytest +import torch + +from tests.kernels.allclose_default import get_default_atol, get_default_rtol +from vllm.model_executor.layers.rotary_embedding import get_rope +from vllm.utils.torch_utils import set_random_seed + +IS_NEOX_STYLE = [True, False] +DTYPES = [torch.bfloat16, torch.float] +HEAD_SIZES = [64, 80, 120, 256] +ROTARY_DIMS = [None, 32] # None means rotary dim == head size +NUM_HEADS = [17] # Arbitrary values for testing +BATCH_SIZES = [5] # Arbitrary values for testing +SEQ_LENS = [11, 8192] # Arbitrary values for testing +SEEDS = [0] +CUDA_DEVICES = [ + f"cuda:{i}" for i in range(1 if torch.accelerator.device_count() == 1 else 2) +] +USE_KEY = [True, False] + + +def _get_flat_tensor_shape( + batch_size: int, seq_len: int, num_heads: int, head_size: int +) -> tuple[int, ...]: + return (batch_size, seq_len, num_heads * head_size) + + +# For testing sliced tensors +def _get_padded_tensor_shape( + batch_size: int, seq_len: int, num_heads: int, head_size: int +) -> tuple[int, ...]: + return (batch_size, seq_len, num_heads, head_size + 64) + + +def _get_batch_tensor_shape( + batch_size: int, seq_len: int, num_heads: int, head_size: int +) -> tuple[int, ...]: + return (batch_size, seq_len, num_heads, head_size) + + +TENSORS_SHAPES_FN = [ + _get_batch_tensor_shape, + _get_flat_tensor_shape, + _get_padded_tensor_shape, +] + + +@pytest.mark.parametrize("is_neox_style", IS_NEOX_STYLE) +@pytest.mark.parametrize("tensor_shape_fn", TENSORS_SHAPES_FN) +@pytest.mark.parametrize("batch_size", BATCH_SIZES) +@pytest.mark.parametrize("seq_len", SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_size", HEAD_SIZES) +@pytest.mark.parametrize("rotary_dim", ROTARY_DIMS) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("use_key", USE_KEY) +@torch.inference_mode() +def test_rotary_embedding( + default_vllm_config, + is_neox_style: bool, + tensor_shape_fn: Callable[[int, int, int, int], tuple[int, ...]], + batch_size: int, + seq_len: int, + num_heads: int, + head_size: int, + rotary_dim: int | None, + dtype: torch.dtype, + seed: int, + device: str, + use_key: bool, + max_position: int = 8192, + rope_theta: float = 10000, +) -> None: + if rotary_dim is None: + rotary_dim = head_size + + set_random_seed(seed) + torch.set_default_device(device) + if rotary_dim is None: + rotary_dim = head_size + rope_parameters = { + "rope_type": "default", + "rope_theta": rope_theta, + "partial_rotary_factor": rotary_dim / head_size, + } + rope = get_rope(head_size, max_position, is_neox_style, rope_parameters) + rope = rope.to(dtype=dtype, device=torch.get_default_device()) + + positions = torch.randint(0, max_position, (batch_size, seq_len)) + query_shape = tensor_shape_fn(batch_size, seq_len, num_heads, head_size) + # slice tensor if required, noop otherwise + query = torch.randn(query_shape, dtype=dtype)[..., :head_size] + key = torch.randn_like(query)[..., :head_size] if use_key else None + + # NOTE(woosuk): The reference implementation should be executed first + # because the custom kernel is in-place. + ref_query, ref_key = rope.forward_native(positions, query, key) + out_query, out_key = rope.forward(positions, query, key) + # Compare the results. + torch.testing.assert_close( + out_query, + ref_query, + atol=get_default_atol(out_query), + rtol=get_default_rtol(out_query), + ) + if use_key: + torch.testing.assert_close( + out_key, + ref_key, + atol=get_default_atol(out_key), + rtol=get_default_rtol(out_key), + ) + else: + assert ref_key is None and out_key is None, "expected returned key to be None" + + +@torch.inference_mode() +def test_rope_module_cache(default_vllm_config): + MAX_POSITIONS = [123, 1234] + ROPE_THETAS = [10000, 1000000] + ROPE_PARAMETERS = ( + {"rope_type": "default"}, + {"rope_type": "linear", "factor": (1,)}, + {"rope_type": "dynamic", "factor": 1}, + ) + settings = ( + HEAD_SIZES, + ROTARY_DIMS, + MAX_POSITIONS, + ROPE_THETAS, + IS_NEOX_STYLE, + ROPE_PARAMETERS, + DTYPES, + ) + rope_setting_id_map: dict[str, int] = {} + for setting in product(*settings): + ( + head_size, + rotary_dim, + max_position, + rope_theta, + is_neox_style, + rope_parameters, + dtype, + ) = setting + if rotary_dim is None: + rotary_dim = head_size + rope_parameters["rope_theta"] = rope_theta + rope_parameters["partial_rotary_factor"] = rotary_dim / head_size + rope = get_rope( + head_size, + max_position, + is_neox_style, + rope_parameters, + dtype, + ) + # different settings cannot share the same rope module + assert id(rope) not in rope_setting_id_map.values() + assert all(x.dtype == dtype for x in rope.buffers()) + assert all(x.dtype == dtype for x in rope.parameters()) + rope_setting_id_map[str(setting)] = id(rope) + + for setting in product(*settings): + ( + head_size, + rotary_dim, + max_position, + rope_theta, + is_neox_style, + rope_parameters, + dtype, + ) = setting + if rotary_dim is None: + rotary_dim = head_size + rope_parameters["rope_theta"] = rope_theta + rope_parameters["partial_rotary_factor"] = rotary_dim / head_size + rope = get_rope( + head_size, + max_position, + is_neox_style, + rope_parameters, + dtype, + ) + # check if cache take effect + assert id(rope) == rope_setting_id_map[str(setting)] diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_rotary_embedding.py b/asian_ai/performance/vLLM/tests/kernels/core/test_rotary_embedding.py new file mode 100644 index 0000000000000000000000000000000000000000..1dcc4b0e036f0c636be6da825b85a29fb43e98c2 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_rotary_embedding.py @@ -0,0 +1,79 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Tests for miscellaneous utilities +""" + +import pytest +import torch + +from tests.kernels.utils import opcheck +from vllm.model_executor.layers.rotary_embedding import RotaryEmbedding + + +def rotary_embedding_opcheck( + rot, + positions: torch.Tensor, + query: torch.Tensor, + key: torch.Tensor | None = None, +): + cos_sin_cache = rot.cos_sin_cache.to(query.device, dtype=query.dtype) + + # ops.rotary_embedding() is a in-place operation + # that updates the query and key tensors. + opcheck( + torch.ops._C.rotary_embedding, + (positions, query, key, rot.head_size, cos_sin_cache, rot.is_neox_style), + ) + + +@pytest.mark.parametrize("device", ["cuda"]) +@pytest.mark.parametrize("max_position", [11, 4096, 32768]) +@pytest.mark.parametrize("is_neox_style", [True, False]) +@pytest.mark.parametrize("rotary_dim", [32]) +@pytest.mark.parametrize("head_size", [32, 108]) +@pytest.mark.parametrize("seq_len", [11, 1024]) +@pytest.mark.parametrize("use_key", [True, False]) +@pytest.mark.parametrize("head_stride_is_contiguous", [True, False]) +@pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16]) +def test_rotary_embedding_opcheck( + default_vllm_config, + dist_init, + device, + max_position, + is_neox_style, + rotary_dim, + head_size, + seq_len, + use_key, + head_stride_is_contiguous, + dtype, +): + batch_size = 1 + base = 10000 + num_heads = 7 + rot = RotaryEmbedding( + head_size, rotary_dim, max_position, base, is_neox_style, dtype + ) + + positions = torch.randint(0, max_position, (batch_size, seq_len), device=device) + head_stride = head_size + (64 if head_stride_is_contiguous else 0) + + query = torch.randn( + batch_size, seq_len, num_heads, head_stride, dtype=dtype, device=device + ) + key = torch.randn_like(query) if use_key else None + query = query[..., :head_size] + key = key[..., :head_size] if key is not None else None + + rotary_embedding_opcheck(rot, positions, query, key) + + # if we have a contiguous head stride, test the alternate + # [..., num_heads * head_dim] shape/layout + if head_stride_is_contiguous: + rotary_embedding_opcheck( + rot, + positions, + query.flatten(start_dim=-2), + key.flatten(start_dim=-2) if key is not None else None, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_rotary_embedding_mla_cache_fused.py b/asian_ai/performance/vLLM/tests/kernels/core/test_rotary_embedding_mla_cache_fused.py new file mode 100644 index 0000000000000000000000000000000000000000..ec97cea5bed9a6fb9b096463fee8609a293c5956 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_rotary_embedding_mla_cache_fused.py @@ -0,0 +1,210 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Tests for fused MLA KV-cache write and RoPE fused kernel +""" + +import random + +import pytest +import torch + +from tests.kernels.allclose_default import get_default_atol, get_default_rtol +from tests.kernels.utils import DEFAULT_OPCHECK_TEST_UTILS, opcheck +from vllm import _custom_ops as ops +from vllm.model_executor.layers.rotary_embedding import RotaryEmbedding +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + + +@pytest.fixture +def default_vllm_config(monkeypatch): + """Enable the AITER triton rope on ROCm for fp16-consistent numerics. + + The fused CUDA kernel runs native fp16 while forward_native upcasts to + fp32, so on ROCm we route through the AITER triton rope (+rotary_embedding) + to match. Its env gates are cached at import, hence refresh_env_variables(). + """ + from vllm._aiter_ops import rocm_aiter_ops + from vllm.config import CompilationConfig, VllmConfig, set_current_vllm_config + + is_rocm = current_platform.is_rocm() + if is_rocm: + config = VllmConfig( + compilation_config=CompilationConfig(custom_ops=["+rotary_embedding"]) + ) + else: + config = VllmConfig() + try: + with monkeypatch.context() as m, set_current_vllm_config(config): + if is_rocm: + m.setenv("VLLM_ROCM_USE_AITER", "1") + m.setenv("VLLM_ROCM_USE_AITER_TRITON_ROPE", "1") + rocm_aiter_ops.refresh_env_variables() + yield config + finally: + if is_rocm: + rocm_aiter_ops.refresh_env_variables() + + +@pytest.mark.parametrize("dtype", [torch.half, torch.bfloat16, torch.float]) +@pytest.mark.parametrize("is_neox_style", [False, True]) +@pytest.mark.parametrize("seq_len", [11, 42]) +@pytest.mark.parametrize("qk_rope_head_dim", [64, 128]) +@pytest.mark.parametrize("num_q_heads", [128]) +@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8"]) +@pytest.mark.parametrize("kv_lora_rank", [512]) +@pytest.mark.parametrize("num_blocks", [64]) +@pytest.mark.parametrize("block_size", [16, 64, 256]) +@pytest.mark.parametrize("seed", [0]) +@pytest.mark.parametrize( + "device", + [f"cuda:{i}" for i in range(1 if torch.accelerator.device_count() == 1 else 2)], +) +@torch.inference_mode() +def test_concat_and_cache_mla_rope_fused( + default_vllm_config, + dtype: torch.dtype, + is_neox_style: bool, + seq_len: int, + qk_rope_head_dim: int, + num_q_heads: int, + kv_cache_dtype: str, + kv_lora_rank: int, + num_blocks: int, + block_size: int, + seed: int, + device: str, + max_position: int = 8192, + base: float = 10000, +) -> None: + set_random_seed(seed) + torch.set_default_device(device) + + rope = RotaryEmbedding( + qk_rope_head_dim, + qk_rope_head_dim, + max_position, + base, + is_neox_style, + torch.float32, + ) + + rope = rope.to(dtype=dtype, device=torch.get_default_device()) + + positions = torch.randint(0, max_position, (seq_len,)) + + query = torch.randn(seq_len, num_q_heads, qk_rope_head_dim, dtype=dtype) + key = torch.randn(seq_len, 1, qk_rope_head_dim + kv_lora_rank, dtype=dtype) + + k_pe = torch.flatten(key[..., :qk_rope_head_dim], start_dim=1).to(device=device) + kv_c = torch.flatten(key[..., qk_rope_head_dim:], start_dim=1).to(device=device) + + if current_platform.is_rocm(): + # We use forward_hip for the same numerics as the fused custom kernel on ROCm + # when dtype is FP16. The torch-native implementation implicitly upcasts + # FP16 x FP16 multiplications to FP32 before downcasting them, which leads + # to notable output divergences. + # Clone the tensors because the implementation modifies them in-place + ref_q_pe, ref_k_pe = rope.forward_hip(positions, query.clone(), k_pe.clone()) + else: + # NOTE(woosuk): The reference implementation should be executed first + # because the custom kernel is in-place. + ref_q_pe, ref_k_pe = rope.forward_native(positions, query, k_pe) + assert ref_k_pe is not None + + ref_k_pe = torch.flatten(ref_k_pe, start_dim=1).to(device=device) + ref_k_rope = ref_k_pe[..., :qk_rope_head_dim] + + total_available_slots = num_blocks * block_size + total_needed_slots = seq_len + assert total_available_slots >= total_needed_slots, "Not enough kv slots!" + + slot_mapping_lst = random.sample(range(total_available_slots), total_needed_slots) + slot_mapping = torch.tensor(slot_mapping_lst, dtype=torch.long, device=device) + + entry_size = kv_lora_rank + qk_rope_head_dim + + kv_cache_scale = torch.tensor([0.1], dtype=torch.float32, device=device) + + kv_cache = torch.zeros( + num_blocks, + block_size, + entry_size, + dtype=torch.uint8 if kv_cache_dtype == "fp8" else dtype, + device=device, + ) + + ref_temp = torch.zeros(*kv_cache.shape, dtype=dtype, device=device) + + for i in range(seq_len): + slot = slot_mapping[i].item() + block_idx = slot // block_size + block_offset = slot % block_size + ref_temp[block_idx, block_offset] = torch.cat((kv_c[i], ref_k_rope[i]), -1) + + if kv_cache_dtype == "fp8": + ref_kv_cache = torch.empty_like(ref_temp, dtype=kv_cache.dtype) + ops.convert_fp8( + ref_kv_cache, ref_temp, kv_cache_scale.item(), kv_dtype=kv_cache_dtype + ) + else: + ref_kv_cache = ref_temp + + opcheck( + torch.ops._C_cache_ops.concat_and_cache_mla_rope_fused, + ( + positions, + query, + k_pe, + kv_c, + rope.cos_sin_cache, + is_neox_style, + slot_mapping, + kv_cache, + kv_cache_dtype, + kv_cache_scale, + ), + test_utils=DEFAULT_OPCHECK_TEST_UTILS, + ) + + ops.concat_and_cache_mla_rope_fused( + positions, + query, + k_pe, + kv_c, + rope.cos_sin_cache, + is_neox_style, + slot_mapping, + kv_cache, + kv_cache_dtype, + kv_cache_scale, + ) + + # ROCm neox-style Triton FMA diverges slightly from the fused kernel, so + # relax the affected tolerance: rtol for fp8 (one e4m3 ULP ~12.5%) and atol + # otherwise (bounded ~6e-4). Other paths use the CUDA defaults. + rocm_neox = current_platform.is_rocm() and is_neox_style + if kv_cache_dtype == "fp8": + result_temp = torch.empty_like(kv_cache, dtype=torch.float16) + ops.convert_fp8( + result_temp, + kv_cache.contiguous(), + kv_cache_scale.item(), + kv_dtype=kv_cache_dtype, + ) + expected_temp = torch.empty_like(ref_kv_cache, dtype=torch.float16) + ops.convert_fp8( + expected_temp, ref_kv_cache, kv_cache_scale.item(), kv_dtype=kv_cache_dtype + ) + torch.testing.assert_close( + result_temp, expected_temp, atol=0.001, rtol=0.15 if rocm_neox else 0.1 + ) + elif rocm_neox: + torch.testing.assert_close(kv_cache, ref_kv_cache, atol=1e-3, rtol=1e-3) + else: + torch.testing.assert_close(kv_cache, ref_kv_cache) + + torch.testing.assert_close( + query, ref_q_pe, atol=get_default_atol(query), rtol=get_default_rtol(query) + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_uva.py b/asian_ai/performance/vLLM/tests/kernels/core/test_uva.py new file mode 100644 index 0000000000000000000000000000000000000000..9c577457e39019f266a1ddbb410b5bc83d730a88 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_uva.py @@ -0,0 +1,55 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch + +from vllm.utils.platform_utils import is_uva_available +from vllm.utils.torch_utils import get_accelerator_view_from_cpu_tensor + +CUDA_DEVICES = [ + f"cuda:{i}" for i in range(1 if torch.accelerator.device_count() == 1 else 2) +] + + +@pytest.mark.skipif(not is_uva_available(), reason="UVA is not available.") +@pytest.mark.parametrize("device", CUDA_DEVICES) +def test_cpu_write(device): + torch.set_default_device(device) + cpu_tensor = torch.zeros(10, 10, device="cpu", pin_memory=True, dtype=torch.int32) + cuda_view = get_accelerator_view_from_cpu_tensor(cpu_tensor) + assert cuda_view.device.type == "cuda" + + assert cuda_view[0, 0] == 0 + assert cuda_view[2, 3] == 0 + assert cuda_view[4, 5] == 0 + + cpu_tensor[0, 0] = 1 + cpu_tensor[2, 3] = 2 + cpu_tensor[4, 5] = -1 + + cuda_view.mul_(2) + assert cuda_view[0, 0] == 2 + assert cuda_view[2, 3] == 4 + assert cuda_view[4, 5] == -2 + + +@pytest.mark.skipif(not is_uva_available(), reason="UVA is not available.") +@pytest.mark.parametrize("device", CUDA_DEVICES) +def test_gpu_write(device): + torch.set_default_device(device) + cpu_tensor = torch.zeros(10, 10, device="cpu", pin_memory=True, dtype=torch.int32) + cuda_view = get_accelerator_view_from_cpu_tensor(cpu_tensor) + assert cuda_view.device.type == "cuda" + + assert cuda_view[0, 0] == 0 + assert cuda_view[2, 3] == 0 + assert cuda_view[4, 5] == 0 + + cuda_view[0, 0] = 1 + cuda_view[2, 3] = 2 + cuda_view[4, 5] = -1 + cuda_view.mul_(2) + + assert cpu_tensor[0, 0] == 2 + assert cpu_tensor[2, 3] == 4 + assert cpu_tensor[4, 5] == -2 diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_vit_bilinear_pos_embed.py b/asian_ai/performance/vLLM/tests/kernels/core/test_vit_bilinear_pos_embed.py new file mode 100644 index 0000000000000000000000000000000000000000..9974ea09eff4d3ade2725eec445ea5e1d54bdff6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_vit_bilinear_pos_embed.py @@ -0,0 +1,120 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Accuracy tests for the fused Triton bilinear position-embedding kernel. + +Compares ``triton_pos_embed_interpolate`` against the pure-PyTorch +``pos_embed_interpolate_native`` across a variety of grid shapes and dtypes. +""" + +import pytest +import torch + +from vllm.triton_utils import HAS_TRITON + +if HAS_TRITON: + from vllm.model_executor.models.qwen3_vl import ( + pos_embed_interpolate_native, + triton_pos_embed_interpolate, + ) + + +DTYPES = [torch.float32, torch.bfloat16] +# Qwen3-VL default +NUM_GRID_PER_SIDE = 48 +SPATIAL_MERGE_SIZE = 2 +HIDDEN_DIM = 1152 + +# 4 square + 4 non-square grids (h, w divisible by spatial_merge_size=2) +SQUARE_GRIDS = [(1, 4, 4), (1, 16, 16), (1, 32, 32), (1, 48, 48)] +NON_SQUARE_GRIDS = [(1, 8, 16), (1, 14, 20), (1, 32, 48), (1, 60, 80)] +ALL_GRIDS = SQUARE_GRIDS + NON_SQUARE_GRIDS + + +@pytest.mark.skipif(not HAS_TRITON, reason="Triton not available") +@pytest.mark.parametrize("dtype", DTYPES, ids=lambda d: str(d).split(".")[-1]) +@pytest.mark.parametrize( + "grid_thw", + ALL_GRIDS, + ids=[f"{t}x{h}x{w}" for t, h, w in ALL_GRIDS], +) +def test_triton_matches_native( + grid_thw: tuple[int, int, int], + dtype: torch.dtype, +) -> None: + """Triton kernel output must match the native PyTorch implementation.""" + t, h, w = grid_thw + device = "cuda" + + # Scale to match real Qwen3-VL pos_embed weight distribution (std~0.23). + torch.manual_seed(42) + embed_weight = ( + torch.randn( + NUM_GRID_PER_SIDE * NUM_GRID_PER_SIDE, + HIDDEN_DIM, + device=device, + dtype=dtype, + ) + * 0.25 + ) + + native_out = pos_embed_interpolate_native( + embed_weight, t, h, w, NUM_GRID_PER_SIDE, SPATIAL_MERGE_SIZE, dtype + ) + triton_out = triton_pos_embed_interpolate( + embed_weight, t, h, w, NUM_GRID_PER_SIDE, SPATIAL_MERGE_SIZE, dtype + ) + + assert native_out.shape == triton_out.shape, ( + f"Shape mismatch: native {native_out.shape} vs triton {triton_out.shape}" + ) + + # Small numerical differences arise from the precomputed h/w_scale + # in the triton kernel vs torch.linspace in the native path, which can + # cause single-ULP output differences + # in a handful of elements. + atol = {torch.float32: 5e-5, torch.bfloat16: 1e-2}[dtype] + rtol = {torch.float32: 1e-5, torch.bfloat16: 1e-2}[dtype] + torch.testing.assert_close(triton_out, native_out, atol=atol, rtol=rtol) + + +@pytest.mark.skipif(not HAS_TRITON, reason="Triton not available") +@pytest.mark.parametrize("dtype", DTYPES, ids=lambda d: str(d).split(".")[-1]) +def test_temporal_repeat(dtype: torch.dtype) -> None: + """Verify temporal dimension t > 1 correctly repeats the spatial pattern.""" + device = "cuda" + h, w = 16, 16 + t_single, t_multi = 1, 3 + + # Scale to match real Qwen3-VL pos_embed weight distribution (std~0.23). + torch.manual_seed(42) + embed_weight = ( + torch.randn( + NUM_GRID_PER_SIDE * NUM_GRID_PER_SIDE, + HIDDEN_DIM, + device=device, + dtype=dtype, + ) + * 0.25 + ) + + out_single = triton_pos_embed_interpolate( + embed_weight, + t_single, + h, + w, + NUM_GRID_PER_SIDE, + SPATIAL_MERGE_SIZE, + dtype, + ) + out_multi = triton_pos_embed_interpolate( + embed_weight, + t_multi, + h, + w, + NUM_GRID_PER_SIDE, + SPATIAL_MERGE_SIZE, + dtype, + ) + + expected = out_single.repeat(t_multi, 1) + torch.testing.assert_close(out_multi, expected, atol=0, rtol=0) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_vit_fp8_attn.py b/asian_ai/performance/vLLM/tests/kernels/core/test_vit_fp8_attn.py new file mode 100644 index 0000000000000000000000000000000000000000..ae1a8fa9867d21b9e72cf1865ea5c909ac062c29 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_vit_fp8_attn.py @@ -0,0 +1,279 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the full FP8 ViT attention path (quantize -> cuDNN -> un-pad).""" + +import contextlib + +import pytest +import torch + +from vllm.triton_utils import HAS_TRITON +from vllm.utils.flashinfer import ( + is_flashinfer_cudnn_fp8_prefill_attn_supported, +) +from vllm.v1.attention.backends.registry import AttentionBackendEnum + + +def _has_flashinfer_cudnn() -> bool: + """Check if FlashInfer cuDNN backend is available.""" + try: + from flashinfer.prefill import ( + cudnn_batch_prefill_with_kv_cache, # noqa: F401 + ) + + return True + except ImportError: + return False + + +HEAD_DIMS = [72, 80] +SEQ_LENS = [256] +NUM_HEADS = [16] + + +@pytest.fixture +def _fp8_attention(): + """Create FP8-enabled MMEncoderAttention via config.""" + from types import SimpleNamespace + from unittest.mock import patch + + from vllm.config import VllmConfig, set_current_vllm_config + from vllm.config.multimodal import MultiModalConfig + + if not is_flashinfer_cudnn_fp8_prefill_attn_supported(): + pytest.skip("FlashInfer cuDNN FP8 prefill attention not supported") + + mm_config = MultiModalConfig(mm_encoder_attn_dtype="fp8") + vllm_config = VllmConfig() + vllm_config.model_config = SimpleNamespace(multimodal_config=mm_config) + + # MMEncoderAttention reads torch.get_default_dtype() during init + # to determine the output dtype. In real model loading this is bf16. + old_dtype = torch.get_default_dtype() + torch.set_default_dtype(torch.bfloat16) + + with ( + set_current_vllm_config(vllm_config), + patch( + "vllm.model_executor.layers.attention.mm_encoder_attention" + ".get_vit_attn_backend", + return_value=AttentionBackendEnum.FLASHINFER, + ), + ): + yield + + torch.set_default_dtype(old_dtype) + + +def _build_cu_seqlens_and_meta( + seq_len: int, + num_heads: int, + head_dim: int, + fp8_padded_hidden_size: int | None = None, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """Build cu_seqlens, max_seqlen, sequence_lengths for a single sequence.""" + import numpy as np + + from vllm.model_executor.layers.attention.mm_encoder_attention import ( + MMEncoderAttention, + ) + + cu_seqlens_np = np.array([0, seq_len], dtype=np.int32) + + sequence_lengths = MMEncoderAttention.maybe_compute_seq_lens( + AttentionBackendEnum.FLASHINFER, + cu_seqlens_np, + torch.device("cuda"), + ) + + max_seqlen = torch.tensor( + MMEncoderAttention.compute_max_seqlen( + AttentionBackendEnum.FLASHINFER, cu_seqlens_np + ), + dtype=torch.int32, + ) + + cu_seqlens = MMEncoderAttention.maybe_recompute_cu_seqlens( + AttentionBackendEnum.FLASHINFER, + cu_seqlens_np, + num_heads * head_dim, + 1, # tp_size + torch.device("cuda"), + fp8_padded_hidden_size=fp8_padded_hidden_size, + ) + + return cu_seqlens, max_seqlen, sequence_lengths + + +@pytest.mark.skipif( + not (HAS_TRITON and _has_flashinfer_cudnn()), + reason="Triton and FlashInfer cuDNN required", +) +@pytest.mark.parametrize("head_dim", HEAD_DIMS) +@pytest.mark.parametrize("seq_len", SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +def test_fp8_attn_output_shape( + head_dim: int, + seq_len: int, + num_heads: int, + _fp8_attention, +) -> None: + """Verify FP8 attention produces correct output shape after un-padding.""" + from vllm.model_executor.layers.attention.mm_encoder_attention import ( + MMEncoderAttention, + ) + from vllm.utils.math_utils import round_up + + attn = None + with contextlib.suppress(ValueError, ImportError): + attn = MMEncoderAttention( + num_heads=num_heads, + head_size=head_dim, + prefix="visual.blocks.0.attn", + ).to("cuda") + + if attn is None or not attn.fp8_enabled: + pytest.skip("FP8 MMEncoderAttention not available") + assert attn is not None # mypy narrowing + + # FP8 always needs fp8_padded_hidden_size for correct cu_seqlens + fp8_padded_hidden_size = num_heads * round_up(head_dim, 16) + + cu_seqlens, max_seqlen, sequence_lengths = _build_cu_seqlens_and_meta( + seq_len, num_heads, head_dim, fp8_padded_hidden_size=fp8_padded_hidden_size + ) + + q = torch.randn( + seq_len, + num_heads, + head_dim, + device="cuda", + dtype=torch.bfloat16, + ) + k = torch.randn_like(q) + v = torch.randn_like(q) + + output = attn._forward_flashinfer(q, k, v, cu_seqlens, max_seqlen, sequence_lengths) + + # Output should have original head_dim (un-padded) + assert output.shape[-1] == head_dim + assert output.dtype == torch.bfloat16 + + +@pytest.mark.skipif( + not (HAS_TRITON and _has_flashinfer_cudnn()), + reason="Triton and FlashInfer cuDNN required", +) +@pytest.mark.parametrize("head_dim", HEAD_DIMS) +@pytest.mark.parametrize("seq_len", SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +def test_fp8_vs_bf16_close( + head_dim: int, seq_len: int, num_heads: int, _fp8_attention +) -> None: + """FP8 attention output should be reasonably close to BF16 baseline.""" + from vllm.model_executor.layers.attention.mm_encoder_attention import ( + MMEncoderAttention, + ) + from vllm.utils.math_utils import round_up + + torch.manual_seed(42) + q = torch.randn( + 1, + seq_len, + num_heads, + head_dim, + device="cuda", + dtype=torch.bfloat16, + ) + k = torch.randn_like(q) + v = torch.randn_like(q) + + # FP8 path + attn_fp8 = None + with contextlib.suppress(ValueError, ImportError): + attn_fp8 = MMEncoderAttention( + num_heads=num_heads, + head_size=head_dim, + prefix="visual.blocks.0.attn", + ).to("cuda") + + if attn_fp8 is None or not attn_fp8.fp8_enabled: + pytest.skip("FP8 MMEncoderAttention not available") + assert attn_fp8 is not None # mypy narrowing + + fp8_padded_hidden_size = num_heads * round_up(head_dim, 16) + cu_seqlens, max_seqlen, seq_lengths = _build_cu_seqlens_and_meta( + seq_len, + num_heads, + head_dim, + fp8_padded_hidden_size=fp8_padded_hidden_size, + ) + + out_fp8 = attn_fp8._forward_flashinfer( + q.clone(), + k.clone(), + v.clone(), + cu_seqlens, + max_seqlen, + seq_lengths, + ) + + # BF16 baseline (create non-FP8 attention by using scale=attn_fp8.scale + # and calling the wrapper directly without FP8 quantization) + from vllm.model_executor.layers.attention.mm_encoder_attention import ( + _get_flashinfer_workspace_buffer, + ) + from vllm.v1.attention.ops.vit_attn_wrappers import ( + vit_flashinfer_wrapper, + ) + + out_bf16 = vit_flashinfer_wrapper( + q=q.clone(), + k=k.clone(), + v=v.clone(), + scale=attn_fp8.scale, + workspace_buffer=_get_flashinfer_workspace_buffer(), + cu_seqlens=cu_seqlens, + max_seqlen=max_seqlen, + sequence_lengths=seq_lengths, + ) + + out_fp8_f = out_fp8.float() + out_bf16_f = out_bf16.float() + + abs_diff = (out_fp8_f - out_bf16_f).abs() + abs_diff_flat = abs_diff.flatten() + + # Relative diff (avoid division by zero) + denom = out_bf16_f.abs().clamp(min=1e-6) + rel_diff_flat = (abs_diff / denom).flatten() + + cosine_sim = torch.nn.functional.cosine_similarity( + out_fp8_f.flatten().unsqueeze(0), + out_bf16_f.flatten().unsqueeze(0), + ).item() + + pcts = [50, 90, 95, 99, 99.9] + abs_pct = {p: torch.quantile(abs_diff_flat, p / 100).item() for p in pcts} + rel_pct = {p: torch.quantile(rel_diff_flat, p / 100).item() for p in pcts} + + print(f"\nFP8 vs BF16 (head_dim={head_dim}, seq_len={seq_len}):") + print(f" cosine_sim={cosine_sim:.6f}") + print( + f" abs_diff: max={abs_diff_flat.max().item():.6f}, " + f"mean={abs_diff_flat.mean().item():.6f}, " + + ", ".join(f"p{p}={abs_pct[p]:.6f}" for p in pcts) + ) + print( + f" rel_diff: max={rel_diff_flat.max().item():.6f}, " + f"mean={rel_diff_flat.mean().item():.6f}, " + + ", ".join(f"p{p}={rel_pct[p]:.6f}" for p in pcts) + ) + + assert abs_diff_flat.max().item() < 0.3, ( + f"FP8 vs BF16 max abs diff too large: {abs_diff_flat.max().item()}" + ) + assert abs_diff_flat.mean().item() < 0.03, ( + f"FP8 vs BF16 mean abs diff too large: {abs_diff_flat.mean().item()}" + ) + assert cosine_sim > 0.99, f"Cosine similarity too low: {cosine_sim:.6f}" diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_vit_fp8_quant.py b/asian_ai/performance/vLLM/tests/kernels/core/test_vit_fp8_quant.py new file mode 100644 index 0000000000000000000000000000000000000000..f5a884de36e707a1a7b461a9d9de41c13cdba09f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_vit_fp8_quant.py @@ -0,0 +1,126 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the stride-aware FP8 quantization kernel with head_dim padding.""" + +import pytest +import torch + +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + get_fp8_min_max, +) +from vllm.platforms import current_platform +from vllm.triton_utils import HAS_TRITON + +if HAS_TRITON: + from vllm.kernels.triton.qkv_padded_fp8_quant import ( + quantize_fp8_pad_head_dim_triton, + ) + +HEAD_DIMS = [72, 80, 128] +SEQ_LENS = [64, 256] +NUM_HEADS = [16] +SCALES = [0.01, 0.1, 1.0] + + +def _naive_fp8_quantize( + tensor: torch.Tensor, scale: torch.Tensor, skip_scale: bool +) -> torch.Tensor: + """Reference FP8 quantization in PyTorch.""" + fp8_dtype = current_platform.fp8_dtype() + fp8_min, fp8_max = get_fp8_min_max() + + x = tensor.float() + if not skip_scale: + x = x / scale.item() + x = x.clamp(fp8_min, fp8_max) + return x.to(fp8_dtype) + + +@pytest.mark.skipif(not HAS_TRITON, reason="Triton not available") +@pytest.mark.parametrize("head_dim", HEAD_DIMS) +@pytest.mark.parametrize("seq_len", SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("scale_val", SCALES) +def test_quantize_contiguous( + head_dim: int, seq_len: int, num_heads: int, scale_val: float +) -> None: + """Test quantization of contiguous 3D tensors.""" + torch.manual_seed(42) + tensor = torch.randn( + seq_len, num_heads, head_dim, device="cuda", dtype=torch.bfloat16 + ) + scale = torch.tensor([scale_val], dtype=torch.float32, device="cuda").view( + 1, 1, 1, 1 + ) + + result = quantize_fp8_pad_head_dim_triton(tensor, scale) + + padded_dim = (head_dim + 15) // 16 * 16 + assert result.shape == (seq_len, num_heads, padded_dim) + assert result.is_contiguous() + assert result.dtype == current_platform.fp8_dtype() + + # Compare unpadded portion against reference + ref = _naive_fp8_quantize(tensor, scale, skip_scale=False) + torch.testing.assert_close(result[:, :, :head_dim].float(), ref.float()) + + # Padded region should be zero + if padded_dim > head_dim: + assert (result[:, :, head_dim:].float() == 0).all() + + +@pytest.mark.skipif(not HAS_TRITON, reason="Triton not available") +@pytest.mark.parametrize("head_dim", [72, 80]) +def test_quantize_non_contiguous(head_dim: int) -> None: + """Test quantization from non-contiguous QKV views (interleaved buffer).""" + seq_len, num_heads = 64, 16 + # Simulate interleaved QKV buffer: shape (seq_len, 3 * num_heads, head_dim) + qkv = torch.randn( + seq_len, 3 * num_heads, head_dim, device="cuda", dtype=torch.bfloat16 + ) + # Q is every 3rd head slice - non-contiguous view + q = qkv[:, 0::3, :] + assert not q.is_contiguous() + + scale = torch.tensor([0.1], dtype=torch.float32, device="cuda").view(1, 1, 1, 1) + result = quantize_fp8_pad_head_dim_triton(q, scale) + + padded_dim = (head_dim + 15) // 16 * 16 + assert result.shape == (seq_len, num_heads, padded_dim) + assert result.is_contiguous() + + # Compare against contiguous reference + ref = _naive_fp8_quantize(q.contiguous(), scale, skip_scale=False) + torch.testing.assert_close(result[:, :, :head_dim].float(), ref.float()) + + +@pytest.mark.skipif(not HAS_TRITON, reason="Triton not available") +def test_skip_scale() -> None: + """Test skip_scale=True produces cast-only output (no division).""" + seq_len, num_heads, head_dim = 32, 8, 80 + tensor = torch.randn( + seq_len, num_heads, head_dim, device="cuda", dtype=torch.bfloat16 + ) + scale = torch.tensor([0.5], dtype=torch.float32, device="cuda").view(1, 1, 1, 1) + + result_skip = quantize_fp8_pad_head_dim_triton(tensor, scale, skip_scale=True) + result_noskip = quantize_fp8_pad_head_dim_triton(tensor, scale, skip_scale=False) + + # skip_scale should just cast, not divide + ref_cast = _naive_fp8_quantize(tensor, scale, skip_scale=True) + torch.testing.assert_close(result_skip[:, :, :head_dim].float(), ref_cast.float()) + + # With scale != 1.0, skip and no-skip should differ + assert not torch.equal(result_skip.float(), result_noskip.float()) + + +@pytest.mark.skipif(not HAS_TRITON, reason="Triton not available") +def test_4d_input() -> None: + """Test that 4D input (B, S, H, D) is handled correctly.""" + B, S, H, D = 2, 32, 8, 72 + tensor = torch.randn(B, S, H, D, device="cuda", dtype=torch.bfloat16) + scale = torch.tensor([0.1], dtype=torch.float32, device="cuda").view(1, 1, 1, 1) + + result = quantize_fp8_pad_head_dim_triton(tensor, scale) + padded_dim = (D + 15) // 16 * 16 + assert result.shape == (B, S, H, padded_dim) diff --git a/asian_ai/performance/vLLM/tests/kernels/core/test_vit_fp8_scaling.py b/asian_ai/performance/vLLM/tests/kernels/core/test_vit_fp8_scaling.py new file mode 100644 index 0000000000000000000000000000000000000000..00ab5a18ff5407a9ea2446e5880d60d75581a55f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/core/test_vit_fp8_scaling.py @@ -0,0 +1,251 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for FP8 scaling (dynamic and static) in MMEncoderAttention.""" + +import contextlib +import json +from types import SimpleNamespace +from unittest.mock import patch + +import pytest +import torch + +from vllm.model_executor.layers.attention.mm_encoder_attention import ( + _FP8_AMAX_HISTORY_LEN, + _FP8_MAX, +) +from vllm.utils.flashinfer import ( + is_flashinfer_cudnn_fp8_prefill_attn_supported, +) + +LAYER_0 = "visual.blocks.0.attn.attn" +LAYER_1 = "visual.blocks.1.attn.attn" +NUM_HEADS = 16 +HEAD_DIM = 72 + + +@contextlib.contextmanager +def _build_attention(mm_config): + """Yield an MMEncoderAttention with the given multimodal config. + + The VllmConfig context stays active while the test runs so that + ``get_multimodal_config()`` calls during the forward path resolve. Also + invokes ``process_weights_after_loading`` to simulate the model loader's + auto-scan. Yields ``None`` if FlashInfer cuDNN is not available. + """ + from vllm.config import VllmConfig, set_current_vllm_config + from vllm.model_executor.layers.attention.mm_encoder_attention import ( + MMEncoderAttention, + ) + from vllm.v1.attention.backends.registry import AttentionBackendEnum + + if not is_flashinfer_cudnn_fp8_prefill_attn_supported(): + yield None + return + + vllm_config = VllmConfig() + vllm_config.model_config = SimpleNamespace(multimodal_config=mm_config) + + with ( + set_current_vllm_config(vllm_config), + patch( + "vllm.model_executor.layers.attention.mm_encoder_attention" + ".get_vit_attn_backend", + return_value=AttentionBackendEnum.FLASHINFER, + ), + ): + attn = MMEncoderAttention( + num_heads=NUM_HEADS, + head_size=HEAD_DIM, + prefix=LAYER_0, + ) + attn.process_weights_after_loading(torch.bfloat16) + yield attn + + +@pytest.fixture +def _make_attention(): + """Create an MMEncoderAttention with dynamic FP8 scaling.""" + from vllm.config.multimodal import MultiModalConfig + + with _build_attention(MultiModalConfig(mm_encoder_attn_dtype="fp8")) as attn: + yield attn + + +@pytest.fixture +def _make_static_attention(tmp_path): + """Create an MMEncoderAttention with static FP8 scales from a file.""" + from vllm.config.multimodal import MultiModalConfig + + scale_file = tmp_path / "scales.json" + scale_file.write_text( + json.dumps( + { + LAYER_0: {"q": 224.0, "k": 198.0, "v": 210.0}, + LAYER_1: {"q": 100.0, "k": 110.0, "v": 120.0}, + } + ) + ) + with _build_attention( + MultiModalConfig( + mm_encoder_attn_dtype="fp8", + mm_encoder_fp8_scale_path=str(scale_file), + ) + ) as attn: + yield attn + + +def test_dynamic_scaling_updates_scales(_make_attention) -> None: + """Verify that _record_amax_and_update_scales updates scale buffers.""" + attn = _make_attention + if attn is None or not attn.fp8_enabled: + pytest.skip("FP8 attention not available (FlashInfer backend required)") + + attn = attn.to("cuda") + + S, H, D = 32, NUM_HEADS, HEAD_DIM + q = torch.full((S, H, D), 2.0, device="cuda", dtype=torch.bfloat16) + k = torch.full((S, H, D), 3.0, device="cuda", dtype=torch.bfloat16) + v = torch.full((S, H, D), 4.0, device="cuda", dtype=torch.bfloat16) + + attn._record_amax_and_update_scales(q, k, v) + + expected_q_scale = 2.0 / _FP8_MAX + expected_k_scale = 3.0 / _FP8_MAX + expected_v_scale = 4.0 / _FP8_MAX + + torch.testing.assert_close(attn._fp8_q_scale.item(), expected_q_scale) + torch.testing.assert_close(attn._fp8_k_scale.item(), expected_k_scale) + torch.testing.assert_close(attn._fp8_v_scale.item(), expected_v_scale) + + +def test_circular_buffer_wraps(_make_attention) -> None: + """Verify the amax circular buffer wraps at HISTORY_LEN.""" + attn = _make_attention + if attn is None or not attn.fp8_enabled: + pytest.skip("FP8 attention not available (FlashInfer backend required)") + + attn = attn.to("cuda") + S, H, D = 16, NUM_HEADS, HEAD_DIM + + for i in range(_FP8_AMAX_HISTORY_LEN + 2): + mag = float(i + 1) + q = torch.full((S, H, D), mag, device="cuda", dtype=torch.bfloat16) + k = torch.full((S, H, D), mag, device="cuda", dtype=torch.bfloat16) + v = torch.full((S, H, D), mag, device="cuda", dtype=torch.bfloat16) + attn._record_amax_and_update_scales(q, k, v) + + assert attn._fp8_amax_pos == 2 + + expected_max = float(_FP8_AMAX_HISTORY_LEN + 2) + expected_scale = expected_max / _FP8_MAX + torch.testing.assert_close(attn._fp8_q_scale.item(), expected_scale) + + +def test_static_scales_loaded(_make_static_attention) -> None: + """Verify static scales are loaded from the JSON file.""" + attn = _make_static_attention + if attn is None or not attn.fp8_enabled: + pytest.skip("FP8 attention not available (FlashInfer backend required)") + + assert attn.fp8_enabled + assert not attn._fp8_dynamic_scale + + # Layer 0 scales (the layer this attention was created with). + assert attn._fp8_q_scale.item() == 224.0 + assert attn._fp8_k_scale.item() == 198.0 + assert attn._fp8_v_scale.item() == 210.0 + + assert not attn.skip_scale_q + assert not attn.skip_scale_k + assert not attn.skip_scale_v + + # No amax history buffers for static scaling. + assert not hasattr(attn, "_fp8_q_amax") + + +def test_static_scales_missing_layer(tmp_path) -> None: + """Verify error when requested layer is not in the scale file.""" + from vllm.config import VllmConfig, set_current_vllm_config + from vllm.config.multimodal import MultiModalConfig + from vllm.v1.attention.backends.registry import AttentionBackendEnum + + if not is_flashinfer_cudnn_fp8_prefill_attn_supported(): + pytest.skip("FlashInfer cuDNN not available") + + scale_file = tmp_path / "wrong_layer.json" + scale_file.write_text( + json.dumps({"visual.blocks.99.attn": {"q": 1.0, "k": 1.0, "v": 1.0}}) + ) + mm_config = MultiModalConfig( + mm_encoder_attn_dtype="fp8", + mm_encoder_fp8_scale_path=str(scale_file), + ) + vllm_config = VllmConfig() + vllm_config.model_config = SimpleNamespace(multimodal_config=mm_config) + + from vllm.model_executor.layers.attention.mm_encoder_attention import ( + MMEncoderAttention, + ) + + with ( + set_current_vllm_config(vllm_config), + patch( + "vllm.model_executor.layers.attention.mm_encoder_attention" + ".get_vit_attn_backend", + return_value=AttentionBackendEnum.FLASHINFER, + ), + ): + attn = MMEncoderAttention( + num_heads=NUM_HEADS, + head_size=HEAD_DIM, + prefix=LAYER_0, + ) + with pytest.raises(ValueError, match="scales not found for layer"): + attn.process_weights_after_loading(torch.bfloat16) + + +def test_dynamic_scales_auto_save(tmp_path) -> None: + """Verify scales are saved to disk after the amax buffer fills.""" + import vllm.model_executor.layers.attention.mm_encoder_attention as _mod + from vllm.config.multimodal import MultiModalConfig + + if not is_flashinfer_cudnn_fp8_prefill_attn_supported(): + pytest.skip("FlashInfer cuDNN not available") + + # Reset module-level state between runs (other tests may have left + # state behind after triggering a save). + _mod._fp8_scale_save_path = None + _mod._fp8_saved_scale_refs.clear() + + save_file = tmp_path / "auto_scales.json" + with _build_attention( + MultiModalConfig( + mm_encoder_attn_dtype="fp8", + mm_encoder_fp8_scale_save_path=str(save_file), + ) + ) as attn: + if attn is None or not attn.fp8_enabled: + pytest.skip("FP8 attention not available") + + attn = attn.to("cuda") + S, H, D = 16, NUM_HEADS, HEAD_DIM + + # Run exactly _FP8_AMAX_HISTORY_LEN forward passes. + for i in range(_FP8_AMAX_HISTORY_LEN): + mag = float(i + 1) + q = torch.full((S, H, D), mag, device="cuda", dtype=torch.bfloat16) + k = torch.full((S, H, D), mag * 0.5, device="cuda", dtype=torch.bfloat16) + v = torch.full((S, H, D), mag * 0.3, device="cuda", dtype=torch.bfloat16) + attn._record_amax_and_update_scales(q, k, v) + + # File should have been written on the 16th call (buffer wrap). + assert save_file.is_file(), "Scale file was not saved" + scales = json.loads(save_file.read_text()) + assert LAYER_0 in scales + assert set(scales[LAYER_0].keys()) == {"q", "k", "v"} + for val in scales[LAYER_0].values(): + assert isinstance(val, float) and val > 0 + + # Path is cleared after the one-shot save fires. + assert _mod._fp8_scale_save_path is None diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/helpers.py b/asian_ai/performance/vLLM/tests/kernels/helion/helpers.py new file mode 100644 index 0000000000000000000000000000000000000000..635c250c6c7232203fd74bd07d233a49d970900b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/helpers.py @@ -0,0 +1,75 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import json +import tempfile +from collections.abc import Callable +from contextlib import contextmanager +from pathlib import Path +from typing import Any +from unittest.mock import patch + +import helion + +from vllm.kernels.helion.case_key import CaseKey +from vllm.kernels.helion.config_manager import ConfigManager +from vllm.kernels.helion.register import register_kernel +from vllm.kernels.helion.utils import get_canonical_gpu_name + +GPU_PLATFORM = get_canonical_gpu_name() + +DEFAULT_CONFIGS: dict[CaseKey, helion.Config] = { + CaseKey.default(): helion.Config(block_sizes=[32]), +} + + +@contextmanager +def dummy_kernel_registry( + configs: dict[CaseKey, helion.Config] | None = None, +): + """Context manager providing a register function with automatic config setup. + + Yields a ``register`` callable with the same signature as + ``register_kernel``. Before applying the real decorator it writes a + config JSON for the kernel name (from ``op_name`` or ``fn.__name__``) + into a temporary directory backed by a fresh ``ConfigManager``. + """ + if configs is None: + configs = DEFAULT_CONFIGS + + def _to_config_entries(cfgs: dict) -> list[dict[str, Any]]: + pairs: list[dict[str, Any]] = [] + for k, v in cfgs.items(): + config_data = v.__dict__["config"] + pairs.append({"key": dict(k), "config": config_data}) + return pairs + + with tempfile.TemporaryDirectory() as tmpdir: + config_dir = Path(tmpdir) + ConfigManager.reset_instance() + cm = ConfigManager(base_dir=config_dir) + + with patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=cm, + ): + + def register( + op_name: str | None = None, + **kwargs, + ) -> Callable: + def decorator(fn: Callable) -> Callable: + name = op_name or fn.__name__ + kernel_dir = config_dir / name + kernel_dir.mkdir(parents=True, exist_ok=True) + (kernel_dir / f"{GPU_PLATFORM}.json").write_text( + json.dumps(_to_config_entries(configs)) + ) + return register_kernel(op_name, **kwargs)(fn) + + return decorator + + try: + yield register + finally: + ConfigManager.reset_instance() diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/test_autotune.py b/asian_ai/performance/vLLM/tests/kernels/helion/test_autotune.py new file mode 100644 index 0000000000000000000000000000000000000000..fc01c91d82a9af696275e9f3ed3addac2b98fe48 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/test_autotune.py @@ -0,0 +1,91 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for autotuning Helion kernels, including disabled kernels with no configs.""" + +import pytest +import torch + +from vllm.utils.import_utils import has_helion + +if not has_helion(): + pytest.skip( + "Helion is not installed. Install with: pip install vllm[helion]", + allow_module_level=True, + ) + +import helion +import helion.language as hl +from helion.autotuner.base_search import BaseSearch + +from tests.kernels.helion.helpers import dummy_kernel_registry +from vllm.kernels.helion.register import create_helion_decorated_kernel + + +def _add_kernel(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + out = torch.empty_like(x) + for tile in hl.tile(x.size()): + out[tile] = x[tile] + y[tile] + return out + + +class NoCompileSearch(BaseSearch): + """Autotuner that returns the default config without GPU compilation. + + Modeled after helion's test BasicSearch (pytorch/helion#1649). + """ + + def autotune(self, *, skip_cache: bool = False): + return self.config_spec.default_config() + + +def _no_compile_autotuner_fn(bound_kernel, args, **kwargs): + return NoCompileSearch(bound_kernel, args, **kwargs) + + +class TestAutotuneDisabledKernel: + """Test autotuning flow on disabled kernels (no platform configs).""" + + def setup_method(self): + from vllm.kernels.helion.register import _REGISTERED_KERNELS + + self._saved_registry = dict(_REGISTERED_KERNELS) + _REGISTERED_KERNELS.clear() + + def teardown_method(self): + from vllm.kernels.helion.register import _REGISTERED_KERNELS + + _REGISTERED_KERNELS.clear() + _REGISTERED_KERNELS.update(self._saved_registry) + + def test_autotune_disabled_kernel_produces_valid_config(self): + """Register a kernel with no configs (disabled), run autotune, + verify it produces a valid helion.Config.""" + with dummy_kernel_registry(configs={}) as register: + wrapper = register( + "autotune_test_kernel", + config_picker=lambda args, keys: None, + fake_impl=lambda *a, **kw: None, + input_generator=lambda: { + "small": ( + torch.randn(4, 4, device="cuda"), + torch.randn(4, 4, device="cuda"), + ), + }, + )(_add_kernel) + + assert wrapper._disabled is True + + inputs = wrapper.get_inputs() + assert "small" in inputs + + settings = helion.Settings() + settings.autotuner_fn = _no_compile_autotuner_fn + wrapper.helion_settings = settings + + config = wrapper.run_autotune(inputs["small"]) + expected_default = ( + create_helion_decorated_kernel(_add_kernel, helion_settings=settings) + .bind(inputs["small"]) + .config_spec.default_config() + ) + assert config == expected_default diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/test_case_key.py b/asian_ai/performance/vLLM/tests/kernels/helion/test_case_key.py new file mode 100644 index 0000000000000000000000000000000000000000..6fd9b1b2d91c4d4df6f7efa420308b1dbd49d5b4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/test_case_key.py @@ -0,0 +1,68 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from vllm.utils.import_utils import has_helion + +if not has_helion(): + pytest.skip( + "Helion is not installed. Install with: pip install vllm[helion]", + allow_module_level=True, + ) + +from vllm.kernels.helion.case_key import CaseKey + + +class TestCaseKey: + """Test suite for CaseKey class.""" + + def test_construction_with_dict(self): + key = CaseKey({"intermediate": 2048, "numtokens": 256}) + assert key["intermediate"] == 2048 + assert key["numtokens"] == 256 + + def test_empty_construction_raises(self): + with pytest.raises(TypeError, match="at least one key-value pair"): + CaseKey() + with pytest.raises(TypeError, match="at least one key-value pair"): + CaseKey({}) + + def test_default_construction(self): + key = CaseKey.default() + assert len(key) == 0 + assert key.is_default() + + def test_non_default_is_not_default(self): + key = CaseKey({"intermediate": 2048}) + assert not key.is_default() + + def test_hashable_and_equality(self): + a = CaseKey({"intermediate": 2048, "numtokens": 256}) + b = CaseKey({"numtokens": 256, "intermediate": 2048}) + assert a == b + assert hash(a) == hash(b) + assert a != CaseKey({"intermediate": 4096}) + assert CaseKey.default() == CaseKey.default() + + configs = { + CaseKey.default(): "default_config", + a: "a_config", + } + assert configs[b] == "a_config" + assert configs[CaseKey.default()] == "default_config" + + def test_str_is_sorted_json(self): + assert str(CaseKey({"z": 1, "a": 2})) == '{"a":2,"z":1}' + assert str(CaseKey.default()) == "{}" + + def test_immutable(self): + key = CaseKey({"intermediate": 2048}) + with pytest.raises(TypeError, match="immutable"): + key["intermediate"] = 4096 + with pytest.raises(TypeError, match="immutable"): + del key["intermediate"] + with pytest.raises(TypeError, match="immutable"): + key.update({"numtokens": 256}) + with pytest.raises(TypeError, match="immutable"): + key.clear() diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/test_config_manager.py b/asian_ai/performance/vLLM/tests/kernels/helion/test_config_manager.py new file mode 100644 index 0000000000000000000000000000000000000000..7e0b3889646cd06f03801b8fe5f36dcf760690b4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/test_config_manager.py @@ -0,0 +1,409 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Unit tests for Helion ConfigManager and ConfigSet. + +Tests the simplified configuration management system for Helion custom kernels. +""" + +import json +import tempfile +from pathlib import Path + +import pytest + +from vllm.utils.import_utils import has_helion + +# Skip entire module if helion is not available +if not has_helion(): + pytest.skip( + "Helion is not installed. Install with: pip install vllm[helion]", + allow_module_level=True, + ) + +import helion + +from vllm.kernels.helion.case_key import CaseKey +from vllm.kernels.helion.config_manager import ( + ConfigManager, + ConfigSet, +) + + +@pytest.fixture(autouse=True) +def reset_config_manager_singleton(): + """Reset ConfigManager singleton before each test.""" + ConfigManager.reset_instance() + yield + ConfigManager.reset_instance() + + +class TestConfigSet: + """Test suite for ConfigSet class.""" + + def test_config_set_creation(self): + """Test creating an empty ConfigSet.""" + config_set = ConfigSet("test_kernel") + + assert config_set.kernel_name == "test_kernel" + assert config_set.get_platforms() == [] + + def test_config_set_from_dict(self): + """Test creating ConfigSet from dictionary data.""" + config_data = { + "block_sizes": [32, 16], + "num_warps": 4, + "num_stages": 3, + "pid_type": "persistent_interleaved", + } + data = { + "h100": [ + {"key": {"batch": 32, "hidden": 4096}, "config": config_data}, + ] + } + + config_set = ConfigSet.from_dict("test_kernel", data) + + assert config_set.kernel_name == "test_kernel" + assert config_set.get_platforms() == ["h100"] + + internal_key = CaseKey({"batch": 32, "hidden": 4096}) + config = config_set.get_config("h100", internal_key) + assert isinstance(config, helion.Config) + assert config.block_sizes == [32, 16] + assert config.num_warps == 4 + assert config.num_stages == 3 + assert config.pid_type == "persistent_interleaved" + + def test_config_set_get_config_keyerror(self): + """Test that accessing non-existent configs raises informative KeyErrors.""" + config_set = ConfigSet("test_kernel") + + with pytest.raises(KeyError, match="platform 'h100' not found"): + config_set.get_config("h100", "nonexistent") + + config_data = {"num_warps": 8, "num_stages": 4} + data = { + "h100": [ + {"key": {"batch": 64, "hidden": 2048}, "config": config_data}, + ] + } + config_set = ConfigSet.from_dict("test_kernel", data) + + nonexistent_key = CaseKey({"batch": 32, "hidden": 4096}) + with pytest.raises(KeyError, match="config_key .* not found"): + config_set.get_config("h100", nonexistent_key) + + def test_config_set_get_platforms(self): + """Test get_platforms method.""" + # Use realistic config data + config1 = {"num_warps": 4, "num_stages": 3} + config2 = {"num_warps": 8, "num_stages": 5} + + data = { + "h100": [ + {"key": {"batch": 32, "hidden": 4096}, "config": config1}, + ], + "a100": [ + {"key": {"batch": 16, "hidden": 2048}, "config": config2}, + ], + } + config_set = ConfigSet.from_dict("test_kernel", data) + + platforms = config_set.get_platforms() + assert platforms == ["a100", "h100"] # Should be sorted + + def test_config_set_get_config_keys(self): + """Test get_config_keys method.""" + config1 = {"num_warps": 4, "num_stages": 3} + config2 = {"num_warps": 8, "num_stages": 5} + + data = { + "h100": [ + {"key": {"batch": 32, "hidden": 4096}, "config": config1}, + {"key": {"batch": 64, "hidden": 2048}, "config": config2}, + ] + } + config_set = ConfigSet.from_dict("test_kernel", data) + + config_keys = config_set.get_config_keys("h100") + expected_keys = sorted( + [ + CaseKey({"batch": 32, "hidden": 4096}), + CaseKey({"batch": 64, "hidden": 2048}), + ], + key=lambda k: str(k) if k is not None else "", + ) + assert config_keys == expected_keys + + assert config_set.get_config_keys("v100") == [] + + def test_config_set_to_dict(self): + """Test converting ConfigSet to dictionary.""" + original_config = { + "block_sizes": [64, 32], + "num_warps": 16, + "num_stages": 4, + "pid_type": "persistent_blocked", + } + original_data = { + "h100": [ + {"key": {"batch": 32, "hidden": 4096}, "config": original_config}, + ] + } + + config_set = ConfigSet.from_dict("test_kernel", original_data) + result_data = config_set.to_dict() + + internal_key = CaseKey({"batch": 32, "hidden": 4096}) + assert internal_key in result_data["h100"] + assert result_data["h100"][internal_key] == original_config + + +class TestConfigManager: + """Test suite for ConfigManager class.""" + + def test_config_manager_creation_default_base_dir(self): + """Test creating ConfigManager with default base directory.""" + manager = ConfigManager() + assert manager._base_dir.name == "configs" + + def test_config_manager_creation_custom_base_dir(self): + """Test creating ConfigManager with custom base directory.""" + custom_dir = "/tmp/custom_configs" + manager = ConfigManager(base_dir=custom_dir) + + # Paths are resolved, so compare with resolved path + assert manager._base_dir == Path(custom_dir).resolve() + + def test_get_config_file_path(self): + """Test getting config file path for a kernel.""" + manager = ConfigManager(base_dir="/tmp") + + dir_path = manager.get_config_file_path("silu_mul_fp8") + assert dir_path == Path("/tmp/silu_mul_fp8") + + file_path = manager.get_config_file_path("silu_mul_fp8", "nvidia_h100") + assert file_path == Path("/tmp/silu_mul_fp8/nvidia_h100.json") + + def test_ensure_base_dir_exists(self): + """Test ensuring base directory exists.""" + with tempfile.TemporaryDirectory() as temp_dir: + base_dir = Path(temp_dir) / "non_existent" / "configs" + manager = ConfigManager(base_dir=base_dir) + assert not base_dir.exists() + + returned_path = manager.ensure_base_dir_exists() + + assert base_dir.exists() + assert base_dir.is_dir() + assert returned_path == base_dir + + def test_load_config_set_file_not_exists(self): + """Test loading config set when file doesn't exist.""" + with tempfile.TemporaryDirectory() as temp_dir: + manager = ConfigManager(base_dir=temp_dir) + config_set = manager.load_config_set("non_existent_kernel") + + assert isinstance(config_set, ConfigSet) + assert config_set.kernel_name == "non_existent_kernel" + assert config_set.get_platforms() == [] + + def test_load_config_set_valid_file(self): + """Test loading config set from per-platform files.""" + with tempfile.TemporaryDirectory() as temp_dir: + kernel_config = { + "block_sizes": [128, 64], + "num_warps": 8, + "num_stages": 6, + "pid_type": "persistent_interleaved", + } + kernel_dir = Path(temp_dir) / "test_kernel" + kernel_dir.mkdir() + platform_file = kernel_dir / "h100.json" + with open(platform_file, "w") as f: + json.dump( + [{"key": {"batch": 32, "hidden": 4096}, "config": kernel_config}], + f, + ) + + manager = ConfigManager(base_dir=temp_dir) + config_set = manager.load_config_set("test_kernel") + + assert isinstance(config_set, ConfigSet) + assert config_set.kernel_name == "test_kernel" + assert config_set.get_platforms() == ["h100"] + + internal_key = CaseKey({"batch": 32, "hidden": 4096}) + config = config_set.get_config("h100", internal_key) + assert isinstance(config, helion.Config) + assert config.block_sizes == [128, 64] + assert config.num_warps == 8 + + def test_load_config_set_invalid_json(self): + """Test loading config set from file with invalid JSON.""" + with tempfile.TemporaryDirectory() as temp_dir: + kernel_dir = Path(temp_dir) / "test_kernel" + kernel_dir.mkdir() + config_file = kernel_dir / "h100.json" + with open(config_file, "w") as f: + f.write("invalid json content {") + + manager = ConfigManager(base_dir=temp_dir) + config_set = manager.load_config_set("test_kernel") + + assert isinstance(config_set, ConfigSet) + assert config_set.kernel_name == "test_kernel" + assert config_set.get_platforms() == [] + + def test_save_config_set(self): + """Test saving ConfigSet to per-platform files.""" + with tempfile.TemporaryDirectory() as temp_dir: + kernel_config = { + "block_sizes": [256, 128], + "num_warps": 16, + "num_stages": 8, + "pid_type": "persistent_blocked", + } + data = { + "h100": [ + {"key": {"batch": 32, "hidden": 4096}, "config": kernel_config}, + ] + } + config_set = ConfigSet.from_dict("test_kernel", data) + + manager = ConfigManager(base_dir=temp_dir) + saved_path = manager.save_config_set(config_set) + + expected_dir = Path(temp_dir) / "test_kernel" + assert saved_path == expected_dir + assert saved_path.is_dir() + + platform_file = expected_dir / "h100.json" + assert platform_file.exists() + with open(platform_file) as f: + loaded_data = json.load(f) + assert isinstance(loaded_data, list) + assert len(loaded_data) == 1 + entry = loaded_data[0] + assert entry["key"] == {"batch": 32, "hidden": 4096} + assert entry["config"] == kernel_config + + def test_save_config_set_creates_directory(self): + """Test that save_config_set creates parent directories if needed.""" + with tempfile.TemporaryDirectory() as temp_dir: + nested_dir = Path(temp_dir) / "nested" / "configs" + data = { + "h100": [ + {"key": {}, "config": {"num_warps": 4}}, + ] + } + config_set = ConfigSet.from_dict("test_kernel", data) + + manager = ConfigManager(base_dir=nested_dir) + saved_path = manager.save_config_set(config_set) + + assert nested_dir.exists() + assert nested_dir.is_dir() + assert saved_path.is_dir() + assert (saved_path / "h100.json").exists() + + def test_get_platform_configs(self): + """Test getting all configs for a specific platform.""" + with tempfile.TemporaryDirectory() as temp_dir: + config_1 = {"num_warps": 4, "num_stages": 3, "block_sizes": [64, 32]} + config_2 = {"num_warps": 8, "num_stages": 5, "block_sizes": [128, 64]} + default_config = { + "num_warps": 16, + "num_stages": 7, + "block_sizes": [256, 128], + } + config_3 = {"num_warps": 2, "num_stages": 2, "block_sizes": [32, 16]} + + kernel_dir = Path(temp_dir) / "test_kernel" + kernel_dir.mkdir() + with open(kernel_dir / "h100.json", "w") as f: + json.dump( + [ + {"key": {"batch": 32, "hidden": 4096}, "config": config_1}, + {"key": {"batch": 64, "hidden": 2048}, "config": config_2}, + {"key": {}, "config": default_config}, + ], + f, + ) + with open(kernel_dir / "a100.json", "w") as f: + json.dump( + [{"key": {"batch": 16, "hidden": 1024}, "config": config_3}], + f, + ) + + manager = ConfigManager(base_dir=temp_dir) + + key_b32_h4096 = CaseKey({"batch": 32, "hidden": 4096}) + key_b64_h2048 = CaseKey({"batch": 64, "hidden": 2048}) + key_b16_h1024 = CaseKey({"batch": 16, "hidden": 1024}) + + h100_configs = manager.get_platform_configs("test_kernel", "h100") + assert len(h100_configs) == 3 + assert key_b32_h4096 in h100_configs + assert key_b64_h2048 in h100_configs + assert CaseKey.default() in h100_configs + for config in h100_configs.values(): + assert isinstance(config, helion.Config) + + assert h100_configs[key_b32_h4096].num_warps == 4 + assert h100_configs[CaseKey.default()].num_stages == 7 + + a100_configs = manager.get_platform_configs("test_kernel", "a100") + assert len(a100_configs) == 1 + assert key_b16_h1024 in a100_configs + assert isinstance(a100_configs[key_b16_h1024], helion.Config) + assert a100_configs[key_b16_h1024].num_warps == 2 + + nonexistent_configs = manager.get_platform_configs("test_kernel", "v100") + assert len(nonexistent_configs) == 0 + + def test_singleton_returns_same_instance(self): + """Test that ConfigManager returns the same instance on repeated calls.""" + manager1 = ConfigManager(base_dir="/tmp/test_singleton") + manager2 = ConfigManager(base_dir="/tmp/test_singleton") + + assert manager1 is manager2 + + def test_singleton_with_default_base_dir(self): + """Test singleton behavior with default base directory.""" + manager1 = ConfigManager() + manager2 = ConfigManager() + + assert manager1 is manager2 + assert manager1._base_dir == manager2._base_dir + + def test_singleton_error_on_different_base_dir(self): + """Test that ConfigManager raises error when created with different base_dir.""" + ConfigManager(base_dir="/tmp/first_dir") + + with pytest.raises(ValueError, match="singleton already exists"): + ConfigManager(base_dir="/tmp/different_dir") + + def test_reset_instance_allows_new_base_dir(self): + """Test that reset_instance allows creating with a new base_dir.""" + manager1 = ConfigManager(base_dir="/tmp/first_dir") + assert manager1._base_dir == Path("/tmp/first_dir").resolve() + + ConfigManager.reset_instance() + + manager2 = ConfigManager(base_dir="/tmp/second_dir") + assert manager2._base_dir == Path("/tmp/second_dir").resolve() + assert manager1 is not manager2 + + def test_get_instance_returns_existing(self): + """Test that get_instance returns the existing singleton.""" + manager1 = ConfigManager(base_dir="/tmp/test_get_instance") + manager2 = ConfigManager.get_instance() + + assert manager1 is manager2 + + def test_get_instance_raises_if_not_initialized(self): + """Test that get_instance raises RuntimeError if no instance exists.""" + with pytest.raises(RuntimeError, match="has not been created"): + ConfigManager.get_instance() diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/test_dynamic_per_token_scaled_fp8_quant.py b/asian_ai/performance/vLLM/tests/kernels/helion/test_dynamic_per_token_scaled_fp8_quant.py new file mode 100644 index 0000000000000000000000000000000000000000..713025e3bb59b329d230cc360061c6bbcc39f611 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/test_dynamic_per_token_scaled_fp8_quant.py @@ -0,0 +1,165 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the dynamic_per_token_scaled_fp8_quant helion kernel + +Run `pytest tests/kernels/helion/test_dynamic_per_token_scaled_fp8_quant.py`. +""" + +from typing import Any + +import pytest +import torch +from torch._subclasses.fake_tensor import FakeTensorMode + +from tests.kernels.helion.utils import skip_if_platform_unsupported +from tests.kernels.quant_utils import FP8_DTYPE +from vllm.kernels.helion.case_key import CaseKey +from vllm.kernels.helion.config_manager import ConfigManager +from vllm.kernels.helion.ops.dynamic_per_token_scaled_fp8_quant import ( + _pick_cache, + baseline, + dynamic_per_token_scaled_fp8_quant, + pick_config, +) +from vllm.platforms import current_platform +from vllm.utils.import_utils import has_helion +from vllm.utils.torch_utils import set_random_seed + +if not has_helion(): + pytest.skip( + "Helion is not installed. Install with: pip install vllm[helion]", + allow_module_level=True, + ) + + +def _generate_fake_input(num_tokens: int, hidden_size: int) -> tuple[Any, ...]: + with FakeTensorMode(): + input = torch.randn( + num_tokens, hidden_size, device="cuda", dtype=torch.bfloat16 + ) + result = torch.empty( + input.shape, device=input.device, dtype=current_platform.fp8_dtype() + ) + scale = torch.empty((num_tokens, 1), device=input.device, dtype=torch.float32) + scale_ub = torch.mean(input).to(torch.float32) + args = (result, input, scale, scale_ub) + return args + + +@pytest.fixture(autouse=True) +def reset_config_manager_singleton(): + ConfigManager.reset_instance() + ConfigManager() + yield + ConfigManager.reset_instance() + + +class TestDynamicPerTokenScaledFp8QuantConfigPicker: + def setup_method(self): + _pick_cache.clear() + + def test_config_picker_exact_match(self): + config_keys = [ + CaseKey({"hidden_size": 2048, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "num_tokens": 16}), + ] + + args = _generate_fake_input(16, 4096) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey({"hidden_size": 4096, "num_tokens": 16}) + + def test_config_picker_closest_match(self): + config_keys = [ + CaseKey({"hidden_size": 2048, "num_tokens": 16}), + CaseKey({"hidden_size": 2048, "num_tokens": 32}), + CaseKey({"hidden_size": 4096, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "num_tokens": 32}), + ] + + args = _generate_fake_input(20, 3000) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey({"hidden_size": 2048, "num_tokens": 32}) + + def test_config_picker_no_configs(self): + config_keys: list[dict] = [] + + args = _generate_fake_input(16, 4096) + selected_key = pick_config(args, config_keys) + assert selected_key is None + + def test_config_picker_fallback_to_largest(self): + config_keys = [ + CaseKey({"hidden_size": 2048, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "num_tokens": 16}), + ] + + args = _generate_fake_input(32, 8192) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey({"hidden_size": 4096, "num_tokens": 16}) + + +class TestDynamicPerTokenScaledFp8QuantCorrectness: + @pytest.mark.parametrize("num_tokens", [1, 7, 4096]) + @pytest.mark.parametrize("hidden_size", [17, 1024, 1025, 1026, 5137, 8193]) + @pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float]) + @pytest.mark.parametrize("has_scale_ub", [True, False]) + @pytest.mark.parametrize("seed", [0]) + def test_dynamic_per_token_fp8_quant( + self, + num_tokens: int, + hidden_size: int, + dtype: torch.dtype, + has_scale_ub: bool, + seed: int, + ) -> None: + skip_if_platform_unsupported("dynamic_per_token_scaled_fp8_quant") + set_random_seed(seed) + + x = ( + torch.rand(num_tokens, hidden_size, dtype=dtype, device="cuda") + 1e-6 + ) # avoid nans + + scale_ub = ( + torch.mean(x).to(dtype=torch.float32, device="cuda") + if has_scale_ub + else None + ) + + ref_out = torch.empty(x.shape, device="cuda", dtype=FP8_DTYPE) + ref_scales = torch.empty((x.shape[0], 1), device="cuda", dtype=torch.float32) + baseline(ref_out, x, ref_scales, scale_ub) + + ops_out = torch.empty(x.shape, device="cuda", dtype=FP8_DTYPE) + ops_scales = torch.empty((x.shape[0], 1), device="cuda", dtype=torch.float32) + dynamic_per_token_scaled_fp8_quant(ops_out, x, ops_scales, scale_ub) + + torch.testing.assert_close(ref_scales, ops_scales) + # allow 1 ULP difference + assert ( + ref_out.view(torch.uint8).to(torch.int16) + - ops_out.view(torch.uint8).to(torch.int16) + ).abs().max() <= 1 + + +class TestDynamicPerTokenScaledFp8QuantIntegration: + def test_kernel_registration_integration(self): + from vllm.kernels.helion.register import get_registered_kernels + + registered_kernels = get_registered_kernels() + assert "dynamic_per_token_scaled_fp8_quant" in registered_kernels + + kernel_wrapper = registered_kernels["dynamic_per_token_scaled_fp8_quant"] + assert kernel_wrapper.op_name == "dynamic_per_token_scaled_fp8_quant" + assert kernel_wrapper._config_picker is not None + assert kernel_wrapper._mutates_args == ["result", "scale"] + + def test_fake_impl_functionality(self): + skip_if_platform_unsupported("dynamic_per_token_scaled_fp8_quant") + from vllm.kernels.helion.register import get_registered_kernels + + registered_kernels = get_registered_kernels() + kernel_wrapper = registered_kernels["dynamic_per_token_scaled_fp8_quant"] + fake_impl = kernel_wrapper._fake_impl + + args = _generate_fake_input(16, 4096) + assert fake_impl(*args) is None diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/test_fused_qk_norm_rope.py b/asian_ai/performance/vLLM/tests/kernels/helion/test_fused_qk_norm_rope.py new file mode 100644 index 0000000000000000000000000000000000000000..16b4e1db863f437d782fc17ef9bf7d344336c57f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/test_fused_qk_norm_rope.py @@ -0,0 +1,261 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the fused_qk_norm_rope helion kernel + +Run `pytest tests/kernels/helion/test_fused_qk_norm_rope.py`. +""" + +from typing import Any + +import pytest +import torch +from torch._subclasses.fake_tensor import FakeTensorMode + +from tests.kernels.helion.utils import skip_if_platform_unsupported +from vllm.benchmarks.lib.utils import default_vllm_config +from vllm.kernels.helion.case_key import CaseKey +from vllm.kernels.helion.config_manager import ConfigManager +from vllm.kernels.helion.ops.fused_qk_norm_rope import ( + _pick_cache, + baseline, + fused_qk_norm_rope, + pick_config, +) +from vllm.model_executor.layers.rotary_embedding import RotaryEmbedding +from vllm.utils.import_utils import has_helion + +if not has_helion(): + pytest.skip( + "Helion is not installed. Install with: pip install vllm[helion]", + allow_module_level=True, + ) + + +@default_vllm_config() +def _generate_fake_input( + num_tokens: int, num_q_heads: int, num_kv_heads: int +) -> tuple[Any, ...]: + with FakeTensorMode(): + head_dim = 128 + eps = 1e-6 + is_neox = True + rotary_ratio = 1.0 + device = "cuda" + dtype = torch.bfloat16 + total_dim = (num_q_heads + 2 * num_kv_heads) * head_dim + qkv = torch.randn(num_tokens, total_dim, dtype=dtype, device=device) + positions = torch.arange(num_tokens, dtype=torch.long, device=device) + q_weight = torch.normal( + mean=1.0, + std=1.0, + size=(head_dim,), + dtype=qkv.dtype, + device=device, + ) + k_weight = torch.normal( + mean=1.0, + std=1.0, + size=(head_dim,), + dtype=qkv.dtype, + device=device, + ) + rotary_dim = int(head_dim * rotary_ratio) + rope = RotaryEmbedding( + head_size=head_dim, + rotary_dim=rotary_dim, + max_position_embeddings=4096, + base=10000.0, + is_neox_style=is_neox, + dtype=dtype, + ).to(device) + args = ( + qkv, + num_q_heads, + num_kv_heads, + num_kv_heads, + head_dim, + eps, + q_weight, + k_weight, + rope.cos_sin_cache, + is_neox, + positions.view(-1), + ) + return args + + +@pytest.fixture(autouse=True) +def reset_config_manager_singleton(): + ConfigManager.reset_instance() + ConfigManager() + yield + ConfigManager.reset_instance() + + +class TestFusedQkNormRopeConfigPicker: + def setup_method(self): + _pick_cache.clear() + + def test_config_picker_exact_match(self): + config_keys = [ + CaseKey({"q_heads": 2048, "kv_heads": 64, "num_tokens": 16}), + CaseKey({"q_heads": 4096, "kv_heads": 128, "num_tokens": 16}), + ] + + args = _generate_fake_input(16, 4096, 128) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey( + {"q_heads": 4096, "kv_heads": 128, "num_tokens": 16} + ) + + def test_config_picker_closest_match(self): + config_keys = [ + CaseKey({"q_heads": 2048, "kv_heads": 64, "num_tokens": 16}), + CaseKey({"q_heads": 2048, "kv_heads": 64, "num_tokens": 32}), + CaseKey({"q_heads": 2048, "kv_heads": 128, "num_tokens": 16}), + CaseKey({"q_heads": 2048, "kv_heads": 128, "num_tokens": 32}), + CaseKey({"q_heads": 4096, "kv_heads": 64, "num_tokens": 16}), + CaseKey({"q_heads": 4096, "kv_heads": 64, "num_tokens": 32}), + CaseKey({"q_heads": 4096, "kv_heads": 128, "num_tokens": 16}), + CaseKey({"q_heads": 4096, "kv_heads": 128, "num_tokens": 32}), + ] + + args = _generate_fake_input(20, 3000, 70) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey( + {"q_heads": 2048, "kv_heads": 64, "num_tokens": 32} + ) + + def test_config_picker_no_configs(self): + config_keys: list[dict] = [] + + args = _generate_fake_input(16, 4096, 128) + selected_key = pick_config(args, config_keys) + assert selected_key is None + + def test_config_picker_fallback_to_largest(self): + config_keys = [ + CaseKey({"q_heads": 2048, "kv_heads": 64, "num_tokens": 16}), + CaseKey({"q_heads": 2048, "kv_heads": 64, "num_tokens": 32}), + CaseKey({"q_heads": 2048, "kv_heads": 128, "num_tokens": 16}), + CaseKey({"q_heads": 2048, "kv_heads": 128, "num_tokens": 32}), + CaseKey({"q_heads": 4096, "kv_heads": 64, "num_tokens": 16}), + CaseKey({"q_heads": 4096, "kv_heads": 64, "num_tokens": 32}), + CaseKey({"q_heads": 4096, "kv_heads": 128, "num_tokens": 16}), + CaseKey({"q_heads": 4096, "kv_heads": 128, "num_tokens": 32}), + ] + + args = _generate_fake_input(64, 8192, 256) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey( + {"q_heads": 4096, "kv_heads": 128, "num_tokens": 32} + ) + + +class TestFusedQkNormRopeCorrectness: + @pytest.mark.parametrize( + "num_heads, num_kv_heads, head_dim", [(16, 4, 128), (64, 8, 128)] + ) + @pytest.mark.parametrize("num_tokens", [1, 7, 1024, 1025]) + @pytest.mark.parametrize("is_neox", [False, True]) + @pytest.mark.parametrize("rotary_ratio", [1.0, 0.5, 0.25]) + @pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) + @default_vllm_config() + def test_fused_qk_norm_rope( + self, + num_heads: int, + num_kv_heads: int, + head_dim: int, + num_tokens: int, + is_neox: bool, + rotary_ratio: float, + dtype: torch.dtype, + ): + skip_if_platform_unsupported("fused_qk_norm_rope") + + torch.manual_seed(42) + eps = 1e-6 + device = "cuda" + total_dim = (num_heads + 2 * num_kv_heads) * head_dim + ref_qkv = torch.empty( + num_tokens, total_dim, dtype=dtype, device=device + ).uniform_(-0.1, 0.1) + ops_qkv = ref_qkv.clone() + positions = torch.arange(num_tokens, dtype=torch.long, device=device) + q_weight = torch.empty(head_dim, dtype=dtype, device=device).uniform_(0.8, 1.2) + k_weight = torch.empty(head_dim, dtype=dtype, device=device).uniform_(0.8, 1.2) + rotary_dim = int(head_dim * rotary_ratio) + rope = RotaryEmbedding( + head_size=head_dim, + rotary_dim=rotary_dim, + max_position_embeddings=40960, + base=10000.0, + is_neox_style=is_neox, + dtype=dtype, + ).to(device) + + baseline( + ref_qkv, + num_heads, + num_kv_heads, + num_kv_heads, + head_dim, + eps, + q_weight, + k_weight, + rope.cos_sin_cache, + is_neox, + positions.view(-1), + ) + + fused_qk_norm_rope( + ops_qkv, + num_heads, + num_kv_heads, + num_kv_heads, + head_dim, + eps, + q_weight, + k_weight, + rope.cos_sin_cache, + is_neox, + positions.view(-1), + ) + + if dtype == torch.bfloat16: + atol = 5e-2 + rtol = 5e-2 + else: + atol = 1e-2 + rtol = 1e-2 + + torch.testing.assert_close( + ref_qkv, + ops_qkv, + atol=atol, + rtol=rtol, + ) + + +class TestFusedQkNormRopeIntegration: + def test_kernel_registration_integration(self): + from vllm.kernels.helion.register import get_registered_kernels + + registered_kernels = get_registered_kernels() + assert "fused_qk_norm_rope" in registered_kernels + + kernel_wrapper = registered_kernels["fused_qk_norm_rope"] + assert kernel_wrapper.op_name == "fused_qk_norm_rope" + assert kernel_wrapper._config_picker is not None + assert kernel_wrapper._mutates_args == ["qkv"] + + def test_fake_impl_functionality(self): + skip_if_platform_unsupported("fused_qk_norm_rope") + from vllm.kernels.helion.register import get_registered_kernels + + registered_kernels = get_registered_kernels() + kernel_wrapper = registered_kernels["fused_qk_norm_rope"] + fake_impl = kernel_wrapper._fake_impl + + args = _generate_fake_input(16, 4096, 128) + assert fake_impl(*args) is None diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/test_helion_available.py b/asian_ai/performance/vLLM/tests/kernels/helion/test_helion_available.py new file mode 100644 index 0000000000000000000000000000000000000000..4cc37755c1195283a94207931dbe4d7ddb2a372e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/test_helion_available.py @@ -0,0 +1,45 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Tests for Helion kernel availability and basic functionality. + +This module demonstrates the pattern for testing optional Helion kernels. +Tests in this directory will be skipped if Helion is not installed. +""" + +import pytest + +from vllm.utils.import_utils import has_helion + +# Skip entire module if helion is not available +if not has_helion(): + pytest.skip( + "Helion is not installed. Install with: pip install vllm[helion]", + allow_module_level=True, + ) + +import helion +import helion.language as hl +import torch + + +def test_helion_kernel_compilation_smoke(): + """Smoke test: compile and run a simple Helion kernel.""" + + @helion.kernel(autotune_effort="none") + def add_kernel(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + out = torch.empty_like(x) + for tile in hl.tile(x.size()): + out[tile] = x[tile] + y[tile] + return out + + # Create test tensors + x = torch.randn(1024, device="cuda", dtype=torch.float32) + y = torch.randn(1024, device="cuda", dtype=torch.float32) + + # Run the helion kernel + result = add_kernel(x, y) + + # Verify correctness + expected = x + y + assert torch.allclose(result, expected), "Helion kernel output mismatch" diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/test_pattern_matching.py b/asian_ai/performance/vLLM/tests/kernels/helion/test_pattern_matching.py new file mode 100644 index 0000000000000000000000000000000000000000..025cccd947f09196f5208d91e35e7675e26d9b98 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/test_pattern_matching.py @@ -0,0 +1,205 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Test make_fx tracing and inductor pattern matching with HelionKernelWrapper.""" + +import contextlib +from unittest.mock import Mock, patch + +import pytest +import torch + +from vllm.utils.import_utils import has_helion + +if not has_helion(): + pytest.skip( + "Helion is not installed. Install with: pip install vllm[helion]", + allow_module_level=True, + ) + +import helion +import helion.language as hl +from helion._compat import requires_torch_version + +if not requires_torch_version("2.11"): + pytest.skip( + "HigherOrderOp requires PyTorch >= 2.11", + allow_module_level=True, + ) + +from helion._compiler._dynamo.higher_order_ops import ( + helion_kernel_side_table, + helion_kernel_wrapper_mutation, +) +from torch._inductor.pattern_matcher import ( + PatternMatcherPass, + fwd_only, + register_replacement, + select_decomp_table, +) +from torch.fx.experimental.proxy_tensor import make_fx + +from vllm.kernels.helion.config_manager import ConfigManager +from vllm.kernels.helion.register import HelionKernelWrapper + + +@contextlib.contextmanager +def _helion_mock_context(): + configs = { + "default": helion.Config(block_sizes=[64], num_warps=2, num_stages=2), + } + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock(return_value=configs) + + with ( + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=mock_config_manager, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value="nvidia_h200", + ), + ): + yield + + +class TestMakeFxHop: + def setup_method(self): + helion_kernel_side_table.reset_table() + + @pytest.mark.skip(reason="SymInt proxy tracking issue with PyTorch 2.11+") + def test_make_fx_symbolic(self): + def raw_add_scale( + x: torch.Tensor, y: torch.Tensor, scale: float + ) -> tuple[torch.Tensor, int, torch.Tensor]: + out_x = torch.empty_like(x) + out_y = torch.empty_like(x) + for tile in hl.tile(x.size()): + out_x[tile] = x[tile] + y[tile] * scale + out_y[tile] = out_x[tile] * 2.0 + return out_x, 42, out_y + + input_x = torch.randn(7, 13) + input_y = torch.randn(7, 13) + scale = 0.5 + + with _helion_mock_context(): + wrapper = HelionKernelWrapper( + raw_kernel_func=raw_add_scale, + op_name="test_make_fx", + fake_impl=lambda *a, **kw: None, + config_picker=lambda args, keys: "default", + ) + + def fn(x, y): + return wrapper(x, y, scale) + + gm = make_fx(fn, tracing_mode="symbolic")(input_x, input_y) + + hop_nodes = [ + n + for n in gm.graph.nodes + if n.op == "call_function" and n.target is helion_kernel_wrapper_mutation + ] + assert len(hop_nodes) == 1 + node = hop_nodes[0] + + assert node.kwargs["constant_args"]["scale"] == scale + assert set(node.kwargs["tensor_args"]) == {"x", "y"} + + specs = node.kwargs["output_spec"]["leaf_specs"] + tensor_specs = [s for s in specs if s["type"] == "tensor"] + scalar_specs = [s for s in specs if s["type"] == "scalar"] + assert len(tensor_specs) == 2 + assert len(scalar_specs) == 1 + + for spec in tensor_specs: + assert spec["dtype"] == input_x.dtype + + assert scalar_specs[0]["scalar_value"] == 42 + + for val in node.meta["val"]: + assert all(isinstance(s, torch.SymInt) for s in val.shape) + + # Both out_x and out_y are empty_like(x), so output shapes == input shape + input_node = next(n for n in gm.graph.nodes if n.op == "placeholder") + input_shape = input_node.meta["val"].shape + for val in node.meta["val"]: + assert len(val.shape) == len(input_shape) + for out_s, in_s in zip(val.shape, input_shape): + assert out_s == in_s + + @pytest.mark.skip(reason="SymInt proxy tracking issue with PyTorch 2.11+") + def test_pattern_matcher_replaces_with_helion_hop(self): + def raw_silu_mul(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + M, N = x.size() + out = torch.empty_like(x) + for tile_m, tile_n in hl.tile([M, N]): + out[tile_m, tile_n] = ( + torch.nn.functional.silu(x[tile_m, tile_n]) * y[tile_m, tile_n] + ) + return out + + with _helion_mock_context(): + wrapper = HelionKernelWrapper( + raw_kernel_func=raw_silu_mul, + op_name="test_pm_silu_mul", + fake_impl=lambda *a, **kw: None, + config_picker=lambda args, keys: "default", + ) + + def pattern(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return torch.nn.functional.silu(x) * y + + def replacement(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + return wrapper(x, y) + + inputs = [torch.randn(8, 16), torch.randn(8, 16)] + + pm_pass = PatternMatcherPass(pass_name="test_helion_replacement") + register_replacement(pattern, replacement, inputs, fwd_only, pm_pass) + + def model(x, y): + return torch.nn.functional.silu(x) * y + + decompositions = select_decomp_table() + input_x = torch.randn(8, 16) + input_y = torch.randn(8, 16) + gm = make_fx(model, decompositions, tracing_mode="symbolic")( + input_x, input_y + ) + + def count_hop_nodes(graph): + return sum( + 1 + for n in graph.nodes + if n.op == "call_function" + and n.target is helion_kernel_wrapper_mutation + ) + + assert count_hop_nodes(gm.graph) == 0 + + match_count = pm_pass.apply(gm.graph) + gm.graph.lint() + gm.recompile() + + assert match_count == 1 + assert count_hop_nodes(gm.graph) == 1 + + hop_node = next( + n + for n in gm.graph.nodes + if n.op == "call_function" + and n.target is helion_kernel_wrapper_mutation + ) + + # raw_silu_mul returns empty_like(x), so output shape == input shape + for val in hop_node.meta["val"]: + assert all(isinstance(s, torch.SymInt) for s in val.shape) + + input_node = next(n for n in gm.graph.nodes if n.op == "placeholder") + input_shape = input_node.meta["val"].shape + output_shape = hop_node.meta["val"][0].shape + assert len(output_shape) == len(input_shape) + for out_s, in_s in zip(output_shape, input_shape): + assert out_s == in_s diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/test_per_token_group_fp8_quant.py b/asian_ai/performance/vLLM/tests/kernels/helion/test_per_token_group_fp8_quant.py new file mode 100644 index 0000000000000000000000000000000000000000..62b0dfb0356dee15167f5e6d1b05fa8df1f1ee9c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/test_per_token_group_fp8_quant.py @@ -0,0 +1,243 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the per_token_group_fp8_quant helion kernel + +Run `pytest tests/kernels/helion/test_per_token_group_fp8_quant.py`. +""" + +from typing import Any + +import pytest +import torch +from torch._subclasses.fake_tensor import FakeTensorMode + +from tests.kernels.helion.utils import skip_if_platform_unsupported +from tests.kernels.quant_utils import FP8_DTYPE +from vllm.kernels.helion.case_key import CaseKey +from vllm.kernels.helion.config_manager import ConfigManager +from vllm.kernels.helion.ops.per_token_group_fp8_quant import ( + _pick_cache, + baseline, + per_token_group_fp8_quant, + pick_config, +) +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + get_fp8_min_max, +) +from vllm.utils.import_utils import has_helion + +if not has_helion(): + pytest.skip( + "Helion is not installed. Install with: pip install vllm[helion]", + allow_module_level=True, + ) + + +def _generate_fake_input( + num_tokens: int, hidden_size: int, group_size: int +) -> tuple[Any, ...]: + with FakeTensorMode(): + input = torch.randn( + (num_tokens, hidden_size), device="cuda", dtype=torch.bfloat16 + ) + output_q = torch.empty(input.shape, device=input.device, dtype=FP8_DTYPE) + output_s = torch.empty( + (num_tokens, hidden_size // group_size), + device=input.device, + dtype=torch.float32, + ) + use_ue8m0 = False + column_major = False + fp8_min, fp8_max = get_fp8_min_max() + eps = 1e-10 + args = ( + input, + output_q, + output_s, + group_size, + eps, + fp8_min, + fp8_max, + use_ue8m0, + column_major, + ) + return args + + +@pytest.fixture(autouse=True) +def reset_config_manager_singleton(): + ConfigManager.reset_instance() + ConfigManager() + yield + ConfigManager.reset_instance() + + +class TestPerTokenGroupFp8QuantConfigPicker: + def setup_method(self): + _pick_cache.clear() + + def test_config_picker_exact_match(self): + config_keys = [ + CaseKey({"hidden_size": 2048, "group_size": 64, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "group_size": 128, "num_tokens": 16}), + ] + + args = _generate_fake_input(16, 4096, 128) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey( + {"hidden_size": 4096, "group_size": 128, "num_tokens": 16} + ) + + def test_config_picker_closest_match(self): + config_keys = [ + CaseKey({"hidden_size": 2048, "group_size": 64, "num_tokens": 16}), + CaseKey({"hidden_size": 2048, "group_size": 64, "num_tokens": 32}), + CaseKey({"hidden_size": 2048, "group_size": 128, "num_tokens": 16}), + CaseKey({"hidden_size": 2048, "group_size": 128, "num_tokens": 32}), + CaseKey({"hidden_size": 4096, "group_size": 64, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "group_size": 64, "num_tokens": 32}), + CaseKey({"hidden_size": 4096, "group_size": 128, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "group_size": 128, "num_tokens": 32}), + ] + + args = _generate_fake_input(20, 3000, 70) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey( + {"hidden_size": 2048, "group_size": 64, "num_tokens": 32} + ) + + def test_config_picker_no_configs(self): + config_keys: list[dict] = [] + + args = _generate_fake_input(16, 4096, 128) + selected_key = pick_config(args, config_keys) + assert selected_key is None + + def test_config_picker_fallback_to_largest(self): + config_keys = [ + CaseKey({"hidden_size": 2048, "group_size": 64, "num_tokens": 16}), + CaseKey({"hidden_size": 2048, "group_size": 64, "num_tokens": 32}), + CaseKey({"hidden_size": 2048, "group_size": 128, "num_tokens": 16}), + CaseKey({"hidden_size": 2048, "group_size": 128, "num_tokens": 32}), + CaseKey({"hidden_size": 4096, "group_size": 64, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "group_size": 64, "num_tokens": 32}), + CaseKey({"hidden_size": 4096, "group_size": 128, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "group_size": 128, "num_tokens": 32}), + ] + + args = _generate_fake_input(64, 8192, 256) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey( + {"hidden_size": 4096, "group_size": 128, "num_tokens": 32} + ) + + +class TestPerTokenGroupFp8QuantCorrectness: + @pytest.mark.parametrize( + "shape", [(31, 128), (32, 128), (63, 256), (64, 256), (16, 512), (2048, 5120)] + ) + @pytest.mark.parametrize("column_major", [False, True]) + @pytest.mark.parametrize("tma_aligned", [False, True]) + @pytest.mark.parametrize("scale_ue8m0", [False, True]) + @pytest.mark.parametrize("group_size", [64, 128]) + def test_per_token_group_fp8_quant( + self, + shape, + column_major: bool, + tma_aligned: bool, + scale_ue8m0: bool, + group_size: int, + ): + skip_if_platform_unsupported("per_token_group_fp8_quant") + + torch.manual_seed(42) + num_tokens, hidden_size = shape + fp8_min, fp8_max = get_fp8_min_max() + eps = 1e-10 + input = ( + torch.randn((num_tokens, hidden_size), device="cuda", dtype=torch.bfloat16) + * 8 + ) + ref_q = torch.empty(input.shape, device=input.device, dtype=FP8_DTYPE) + ops_q = ref_q.clone() + + groups_per_row = hidden_size // group_size + if column_major: + if tma_aligned: + tma_alignment = 4 + tma_aligned_m = ( + (num_tokens + tma_alignment - 1) // tma_alignment * tma_alignment + ) + shape = (num_tokens, groups_per_row) + stride = (1, tma_aligned_m) + ref_s = torch.empty_strided( + shape, stride, device=input.device, dtype=torch.float32 + ) + else: + ref_s = torch.empty( + (groups_per_row, num_tokens), + device=input.device, + dtype=torch.float32, + ).transpose(0, 1) + else: + ref_s = torch.empty( + (num_tokens, groups_per_row), device=input.device, dtype=torch.float32 + ) + + ops_s = ref_s.clone() + + baseline( + input, + ref_q, + ref_s, + group_size, + eps, + fp8_min, + fp8_max, + scale_ue8m0, + column_major, + tma_aligned, + ) + per_token_group_fp8_quant( + input, + ops_q, + ops_s, + group_size, + eps, + fp8_min, + fp8_max, + scale_ue8m0, + column_major, + tma_aligned, + ) + + assert torch.allclose(ref_s, ops_s) + # allow 1 ULP difference + assert ( + ref_q.view(torch.uint8).to(torch.int16) + - ops_q.view(torch.uint8).to(torch.int16) + ).abs().max() <= 1 + + +class TestPerTokenGroupFp8QuantIntegration: + def test_kernel_registration_integration(self): + from vllm.kernels.helion.register import get_registered_kernels + + registered_kernels = get_registered_kernels() + assert "per_token_group_fp8_quant" in registered_kernels + + kernel_wrapper = registered_kernels["per_token_group_fp8_quant"] + assert kernel_wrapper.op_name == "per_token_group_fp8_quant" + assert kernel_wrapper._config_picker is not None + assert kernel_wrapper._mutates_args == ["output_q", "output_s"] + + def test_fake_impl_functionality(self): + skip_if_platform_unsupported("per_token_group_fp8_quant") + from vllm.kernels.helion.register import get_registered_kernels + + registered_kernels = get_registered_kernels() + kernel_wrapper = registered_kernels["per_token_group_fp8_quant"] + fake_impl = kernel_wrapper._fake_impl + + args = _generate_fake_input(16, 4096, 128) + assert fake_impl(*args) is None diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/test_register.py b/asian_ai/performance/vLLM/tests/kernels/helion/test_register.py new file mode 100644 index 0000000000000000000000000000000000000000..0997e592bcd3e463948c9466f263f8a3f3beba69 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/test_register.py @@ -0,0 +1,1063 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Unit tests for Helion kernel registration. + +Tests ConfiguredHelionKernel, HelionKernelWrapper, and PresetConfigSearch +including config picker registration and custom autotuner integration. +""" + +from unittest.mock import Mock, patch + +import pytest +import torch + +from vllm.utils.import_utils import has_helion + +if not has_helion(): + pytest.skip( + "Helion is not installed. Install with: pip install vllm[helion]", + allow_module_level=True, + ) + +import helion +import helion.language as hl + +from tests.kernels.helion.helpers import dummy_kernel_registry +from vllm.kernels.helion.case_key import CaseKey +from vllm.kernels.helion.config_manager import ConfigManager +from vllm.kernels.helion.register import ( + _HOP_AVAILABLE, + ConfiguredHelionKernel, + HelionKernelWrapper, + get_kernel_by_name, + get_registered_kernels, + register_kernel, + validate_helion_settings, +) + +if _HOP_AVAILABLE: + from helion._compat import supports_torch_compile_fusion + from helion._compiler._dynamo.higher_order_ops import ( + helion_kernel_wrapper_mutation, + ) + from torch._inductor.utils import run_and_get_code + + +def _add_kernel(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + out = torch.empty_like(x) + for tile in hl.tile(x.size()): + out[tile] = x[tile] + y[tile] + return out + + +@pytest.fixture +def sample_configs(): + """Create real Helion config objects for testing.""" + return { + CaseKey({"batchsize": 32, "hiddensize": 4096}): helion.Config( + block_sizes=[128], + num_warps=4, + num_stages=3, + ), + CaseKey({"batchsize": 64, "hiddensize": 4096}): helion.Config( + block_sizes=[256], + num_warps=8, + num_stages=4, + ), + CaseKey({"batchsize": 128, "hiddensize": 4096}): helion.Config( + block_sizes=[512], + num_warps=16, + num_stages=2, + ), + CaseKey.default(): helion.Config( + block_sizes=[64], + num_warps=2, + num_stages=2, + ), + } + + +@pytest.fixture +def sample_kernel(): + """Create a simple test kernel function.""" + + def test_kernel(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: + """Simple test kernel that adds two tensors.""" + return x + y + + return test_kernel + + +@pytest.fixture +def config_manager_with_test_configs(sample_configs): + """Set up ConfigManager with test configs for nvidia_h200 platform.""" + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock(return_value=sample_configs) + return mock_config_manager + + +@pytest.fixture +def configured_kernel(sample_kernel, sample_configs, config_manager_with_test_configs): + """Create a ConfiguredHelionKernel for testing.""" + + def test_config_picker(args, config_keys): + return None + + with ( + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=config_manager_with_test_configs, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value="nvidia_h200", + ), + patch("vllm.kernels.helion.register.helion.kernel") as mock_kernel, + ): + mock_decorated = Mock() + mock_kernel.return_value = Mock(return_value=mock_decorated) + + return ConfiguredHelionKernel( + op_name="test_kernel", + config_picker=test_config_picker, + raw_kernel_func=sample_kernel, + helion_settings=None, + ) + + +class TestValidateHelionSettings: + """Test suite for validate_helion_settings utility function.""" + + def test_accepts_none_settings(self): + """Test that None settings are accepted without error.""" + validate_helion_settings(None, "test_kernel") # Should not raise + + def test_accepts_valid_settings(self): + """Test that valid settings without conflicts are accepted.""" + settings = helion.Settings() + settings.static_shapes = False + settings.print_output_code = True + validate_helion_settings(settings, "test_kernel") # Should not raise + + def test_rejects_autotuner_fn(self): + """Test that settings with custom autotuner_fn raise ValueError.""" + settings = helion.Settings() + settings.autotuner_fn = lambda *args: None # Set custom autotuner function + + with pytest.raises(ValueError, match="uses a custom autotuner"): + validate_helion_settings(settings, "test_kernel") + + def test_warns_on_static_shapes_true(self): + """Test that static_shapes=True emits a warning about being overridden.""" + settings = helion.Settings() + settings.static_shapes = True + + with patch("vllm.kernels.helion.register.logger") as mock_logger: + validate_helion_settings(settings, "test_kernel") + mock_logger.warning.assert_called_once() + assert "overridden to False" in mock_logger.warning.call_args[0][0] + + +def create_configured_kernel_with_configs( + op_name, + config_picker, + kernel_func, + configs, + platform="nvidia_h200", + helion_settings=None, +): + """Helper to create ConfiguredHelionKernel with real config objects.""" + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock(return_value=configs) + + with ( + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=mock_config_manager, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value=platform, + ), + patch("vllm.kernels.helion.register.helion.kernel") as mock_kernel, + ): + mock_decorated = Mock() + mock_kernel.return_value = Mock(return_value=mock_decorated) + + return ConfiguredHelionKernel( + op_name=op_name, + config_picker=config_picker, + raw_kernel_func=kernel_func, + helion_settings=helion_settings, + ) + + +class TestConfiguredHelionKernel: + """Test suite for ConfiguredHelionKernel.""" + + def test_init_raises_without_picker(self, sample_kernel, sample_configs): + """Test that __init__ raises when no picker registered.""" + configs: dict[CaseKey, helion.Config] = { + CaseKey.default(): sample_configs[CaseKey.default()] + } + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock(return_value=configs) + + with ( + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=mock_config_manager, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value="nvidia_h200", + ), + pytest.raises(RuntimeError, match="No config picker registered"), + ): + ConfiguredHelionKernel( + op_name="test_kernel", + config_picker=None, # No picker registered + raw_kernel_func=sample_kernel, + helion_settings=None, + ) + + def test_config_selector_validates_picker_result( + self, sample_kernel, sample_configs + ): + """Test that config selector validates picker returns valid key.""" + + def invalid_picker(args, config_keys): + return {"invalid": 999} + + kernel = create_configured_kernel_with_configs( + op_name="test_kernel", + config_picker=invalid_picker, + kernel_func=sample_kernel, + configs=sample_configs, + ) + + key_computer = kernel._create_key_computer() + selector = kernel._create_config_selector(key_computer) + + with pytest.raises( + ValueError, match="Config picker returned invalid config key" + ): + selector((torch.randn(32, 4096),)) + + def test_config_selector_handles_none_from_picker( + self, sample_kernel, sample_configs + ): + """Test that config selector falls back to 'default' on None.""" + + def none_picker(args, config_keys): + return None + + kernel = create_configured_kernel_with_configs( + op_name="test_kernel", + config_picker=none_picker, + kernel_func=sample_kernel, + configs=sample_configs, + ) + + key_computer = kernel._create_key_computer() + selector = kernel._create_config_selector(key_computer) + + result = selector((torch.randn(32, 4096),)) + assert result is kernel.configs[CaseKey.default()] + + def test_create_decorated_kernel_passes_helion_settings( + self, sample_kernel, sample_configs + ): + """Test that _create_decorated_kernel passes helion_settings.""" + + def default_picker(args, config_keys): + return None + + settings = helion.Settings() + settings.print_output_code = True + + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock(return_value=sample_configs) + + with ( + patch("vllm.kernels.helion.register.helion.kernel") as mock_kernel, + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=mock_config_manager, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value="nvidia_h200", + ), + ): + mock_decorated = Mock() + mock_kernel.return_value = Mock(return_value=mock_decorated) + + ConfiguredHelionKernel( + op_name="test_kernel", + config_picker=default_picker, + raw_kernel_func=sample_kernel, + helion_settings=settings, + ) + + call_kwargs = mock_kernel.call_args[1] + assert "print_output_code" in call_kwargs + assert call_kwargs["print_output_code"] is True + # static_shapes is always forced to False by vLLM + assert call_kwargs["static_shapes"] is False + + def test_key_and_config_selector_use_same_logic( + self, sample_kernel, sample_configs + ): + """Test that key and config_selector produce identical results.""" + + def tracking_picker(args, config_keys): + x = args[0] + batch_size = x.shape[0] + if batch_size <= 32: + return CaseKey({"batchsize": 32, "hiddensize": 4096}) + elif batch_size <= 64: + return CaseKey({"batchsize": 64, "hiddensize": 4096}) + return CaseKey({"batchsize": 128, "hiddensize": 4096}) + + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock(return_value=sample_configs) + + with ( + patch("vllm.kernels.helion.register.helion.kernel") as mock_helion_kernel, + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=mock_config_manager, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value="nvidia_h200", + ), + ): + mock_decorated = Mock() + mock_helion_kernel.return_value = Mock(return_value=mock_decorated) + + kernel = ConfiguredHelionKernel( + op_name="test_kernel", + config_picker=tracking_picker, + raw_kernel_func=sample_kernel, + helion_settings=None, + ) + + call_kwargs = mock_helion_kernel.call_args[1] + key_fn = call_kwargs["key"] + autotuner_fn = call_kwargs["autotuner_fn"] + + tensor = torch.randn(50, 4096) # batch=50, should select batchsize_64 + + key_result = key_fn(tensor) + autotuner = autotuner_fn(None, (tensor,)) + config = autotuner.autotune() + + expected_key = CaseKey({"batchsize": 64, "hiddensize": 4096}) + assert key_result == str(expected_key) + assert config is kernel.configs[expected_key] + + +class TestHelionKernelWrapper: + """Test suite for HelionKernelWrapper.""" + + def test_init_disables_on_missing_configs(self, sample_kernel): + """Test __init__ marks wrapper as disabled when configs are missing.""" + + def fake_impl(*args, **kwargs): + return torch.zeros_like(args[0]) + + def default_picker(args, config_keys): + return None + + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock( + return_value={} + ) # Empty configs + + with ( + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=mock_config_manager, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value="nvidia_h200", + ), + patch("vllm.kernels.helion.register.helion.kernel") as mock_kernel, + ): + mock_kernel.return_value = Mock(return_value=sample_kernel) + + wrapper = HelionKernelWrapper( + raw_kernel_func=sample_kernel, + op_name="test_kernel", + fake_impl=fake_impl, + config_picker=default_picker, + ) + + assert wrapper._disabled is True + assert "No configs available" in wrapper._disabled_reason + + def test_disabled_wrapper_raises_on_call(self, sample_kernel): + """Test __call__ raises RuntimeError on a disabled wrapper.""" + + def fake_impl(*args, **kwargs): + return torch.zeros_like(args[0]) + + def default_picker(args, config_keys): + return None + + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock(return_value={}) + + with ( + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=mock_config_manager, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value="nvidia_h200", + ), + patch("vllm.kernels.helion.register.helion.kernel") as mock_kernel, + ): + mock_kernel.return_value = Mock(return_value=sample_kernel) + + wrapper = HelionKernelWrapper( + raw_kernel_func=sample_kernel, + op_name="test_kernel", + fake_impl=fake_impl, + config_picker=default_picker, + ) + + with pytest.raises(RuntimeError, match="is disabled"): + wrapper(torch.randn(4, 4), torch.randn(4, 4)) + + def test_disabled_wrapper_get_configured_op_raises(self, sample_kernel): + """Test get_configured_op raises RuntimeError on a disabled wrapper.""" + + def fake_impl(*args, **kwargs): + return torch.zeros_like(args[0]) + + def default_picker(args, config_keys): + return None + + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock(return_value={}) + + with ( + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=mock_config_manager, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value="nvidia_h200", + ), + patch("vllm.kernels.helion.register.helion.kernel") as mock_kernel, + ): + mock_kernel.return_value = Mock(return_value=sample_kernel) + + wrapper = HelionKernelWrapper( + raw_kernel_func=sample_kernel, + op_name="test_kernel", + fake_impl=fake_impl, + config_picker=default_picker, + ) + + with pytest.raises(RuntimeError, match="is disabled"): + wrapper.get_configured_op() + + def test_disabled_wrapper_supports_get_inputs(self, sample_kernel): + """Test get_inputs works on a disabled wrapper.""" + + def fake_impl(*args, **kwargs): + return torch.zeros_like(args[0]) + + def default_picker(args, config_keys): + return None + + expected_inputs = {"key1": (torch.randn(4),)} + input_gen = Mock(return_value=expected_inputs) + + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock(return_value={}) + + with ( + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=mock_config_manager, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value="nvidia_h200", + ), + patch("vllm.kernels.helion.register.helion.kernel") as mock_kernel, + ): + mock_kernel.return_value = Mock(return_value=sample_kernel) + + wrapper = HelionKernelWrapper( + raw_kernel_func=sample_kernel, + op_name="test_kernel", + fake_impl=fake_impl, + config_picker=default_picker, + input_generator=input_gen, + ) + + assert wrapper._disabled is True + result = wrapper.get_inputs() + assert result is expected_inputs + + def test_disabled_wrapper_supports_run_autotune(self, sample_kernel): + """Test run_autotune works on a disabled wrapper.""" + + def fake_impl(*args, **kwargs): + return torch.zeros_like(args[0]) + + def default_picker(args, config_keys): + return None + + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock(return_value={}) + + mock_config = Mock() + + with ( + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=mock_config_manager, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value="nvidia_h200", + ), + patch("vllm.kernels.helion.register.helion.kernel") as mock_kernel, + ): + mock_kernel.return_value = Mock(return_value=sample_kernel) + + wrapper = HelionKernelWrapper( + raw_kernel_func=sample_kernel, + op_name="test_kernel", + fake_impl=fake_impl, + config_picker=default_picker, + ) + + assert wrapper._disabled is True + + with patch( + "vllm.kernels.helion.register.create_helion_decorated_kernel" + ) as mock_create: + mock_autotune_kernel = Mock() + mock_autotune_kernel.autotune.return_value = mock_config + mock_create.return_value = mock_autotune_kernel + + inputs = (torch.randn(4, 4),) + result = wrapper.run_autotune(inputs) + assert result is mock_config + + def test_init_caches_configured_kernel(self, sample_kernel, sample_configs): + """Test __init__ eagerly builds and caches ConfiguredHelionKernel.""" + + def fake_impl(*args, **kwargs): + return torch.zeros_like(args[0]) + + def default_picker(args, config_keys): + return None + + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock(return_value=sample_configs) + + with ( + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=mock_config_manager, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value="nvidia_h200", + ), + patch("vllm.kernels.helion.register.helion.kernel") as mock_kernel, + ): + mock_kernel.return_value = Mock(return_value=sample_kernel) + + wrapper = HelionKernelWrapper( + raw_kernel_func=sample_kernel, + op_name="test_kernel", + fake_impl=fake_impl, + config_picker=default_picker, + ) + + assert wrapper._configured_kernel is not None + result1 = wrapper.get_configured_op() + result2 = wrapper.get_configured_op() + assert result1 is result2 + + @pytest.mark.skipif( + not _HOP_AVAILABLE, reason="HOP path only used when HOP available" + ) + def test_init_eagerly_initializes_hop_path(self): + """Test that register_kernel eagerly builds the configured kernel + on the HOP path (no custom op registration needed).""" + from vllm.kernels.helion.utils import get_canonical_gpu_name + + configs: dict[CaseKey, helion.Config] = { + CaseKey.default(): helion.Config(block_sizes=[4, 4]) + } + with ( + dummy_kernel_registry(configs=configs) as register, + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + wraps=get_canonical_gpu_name, + ) as mock_gpu, + ): + wrapper = register( + config_picker=lambda args, keys: None, + )(_add_kernel) + + mock_gpu.assert_called_once() + assert wrapper._configured_kernel is not None + + with patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + side_effect=AssertionError("get_canonical_gpu_name called during __call__"), + ): + x = torch.randn(4, 4, device="cuda") + y = torch.randn(4, 4, device="cuda") + result = wrapper(x, y) + expected = x + y + assert torch.allclose(result, expected) + + @pytest.mark.skipif( + _HOP_AVAILABLE, reason="CustomOp path not used when HOP available" + ) + def test_init_eagerly_initializes(self): + """Test that register_kernel eagerly loads configs and detects GPU + during construction so __call__ needs no further initialization.""" + from vllm.kernels.helion.utils import get_canonical_gpu_name + + with ( + dummy_kernel_registry() as register, + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + wraps=get_canonical_gpu_name, + ) as mock_gpu, + ): + wrapper = register( + config_picker=lambda args, keys: None, + )(_add_kernel) + + # Init must have detected GPU and built the kernel + mock_gpu.assert_called_once() + assert wrapper._configured_kernel is not None + assert hasattr(torch.ops.vllm_helion, wrapper.op_name) + + @pytest.mark.skipif( + _HOP_AVAILABLE, reason="CustomOp path not used when HOP available" + ) + def test_get_or_register_custom_op_returns_cached_op( + self, sample_kernel, sample_configs + ): + def fake_impl(*args, **kwargs): + return torch.zeros_like(args[0]) + + def default_picker(args, config_keys): + return None + + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock(return_value=sample_configs) + + existing_op = Mock() + mock_namespace = Mock() + mock_namespace.test_kernel = existing_op + + with ( + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=mock_config_manager, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value="nvidia_h200", + ), + patch.object(torch.ops, "vllm_helion", mock_namespace), + patch("vllm.kernels.helion.register.helion.kernel") as mock_kernel, + ): + mock_decorated = Mock() + mock_kernel.return_value = Mock(return_value=mock_decorated) + + wrapper = HelionKernelWrapper( + raw_kernel_func=sample_kernel, + op_name="test_kernel", + fake_impl=fake_impl, + config_picker=default_picker, + ) + result = wrapper._get_or_register_custom_op() + assert result is existing_op + + @pytest.mark.skipif( + _HOP_AVAILABLE, reason="CustomOp path not used when HOP available" + ) + def test_get_or_register_custom_op_registers_new_op( + self, sample_kernel, sample_configs + ): + def fake_impl(*args, **kwargs): + return torch.zeros_like(args[0]) + + def default_picker(args, config_keys): + return None + + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock(return_value=sample_configs) + + new_op = Mock() + registered_ops: dict[str, Mock] = {} + mutates_args = ["y"] + + class MockNamespace: + def __getattr__(self, name): + if name in registered_ops: + return registered_ops[name] + raise AttributeError(name) + + mock_namespace = MockNamespace() + + def register_side_effect(op_name, op_func, **kwargs): + registered_ops[op_name] = new_op + + with ( + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=mock_config_manager, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value="nvidia_h200", + ), + patch.object(torch.ops, "vllm_helion", mock_namespace), + patch( + "vllm.kernels.helion.register.direct_register_custom_op", + side_effect=register_side_effect, + ) as mock_register, + patch("vllm.kernels.helion.register.helion.kernel") as mock_kernel, + ): + mock_decorated = Mock() + mock_kernel.return_value = Mock(return_value=mock_decorated) + + wrapper = HelionKernelWrapper( + raw_kernel_func=sample_kernel, + op_name="test_kernel", + fake_impl=fake_impl, + mutates_args=mutates_args, + config_picker=default_picker, + ) + result = wrapper._get_or_register_custom_op() + + mock_register.assert_called_once() + assert result is new_op + assert mock_register.call_args[1]["op_func"] is mock_decorated + assert mock_register.call_args[1]["mutates_args"] is mutates_args + + +class TestKernelRegistry: + """Test suite for kernel registry functionality.""" + + def setup_method(self): + """Save and clear the registry before each test.""" + from vllm.kernels.helion.register import _REGISTERED_KERNELS + + self._saved_registry = dict(_REGISTERED_KERNELS) + _REGISTERED_KERNELS.clear() + + def teardown_method(self): + """Restore the registry after each test.""" + from vllm.kernels.helion.register import _REGISTERED_KERNELS + + _REGISTERED_KERNELS.clear() + _REGISTERED_KERNELS.update(self._saved_registry) + + def test_get_registered_kernels_returns_copy(self): + """Test get_registered_kernels returns copy of registry.""" + result1 = get_registered_kernels() + result2 = get_registered_kernels() + + # Should be separate objects + assert result1 is not result2 + # Should have same content + assert result1 == result2 + + def test_get_kernel_by_name_returns_kernel(self): + """Test get_kernel_by_name returns registered kernel.""" + with dummy_kernel_registry() as register: + wrapper = register("test_kernel", config_picker=lambda args, keys: None)( + _add_kernel + ) + + from vllm.kernels.helion.register import _REGISTERED_KERNELS + + _REGISTERED_KERNELS["test_kernel"] = wrapper + + result = get_kernel_by_name("test_kernel") + assert result is wrapper + + def test_get_kernel_by_name_returns_none_for_missing(self): + """Test get_kernel_by_name returns None for missing kernel.""" + result = get_kernel_by_name("nonexistent") + assert result is None + + def test_register_kernel_auto_generates_fake_impl(self): + """Test register_kernel auto-generates fake_impl when not provided.""" + with ( + dummy_kernel_registry() as register, + patch("vllm.kernels.helion.register.infer_fake_impl") as mock_infer, + ): + mock_fake = Mock() + mock_infer.return_value = mock_fake + wrapper = register( + config_picker=lambda args, keys: None, + )(_add_kernel) + + mock_infer.assert_called_once_with(_add_kernel, None) + assert wrapper._fake_impl is mock_fake + + def test_register_kernel_creates_wrapper(self): + """Test register_kernel creates HelionKernelWrapper.""" + with dummy_kernel_registry() as register: + result = register("test_name", config_picker=lambda args, keys: None)( + _add_kernel + ) + + assert isinstance(result, HelionKernelWrapper) + assert result.op_name == "test_name" + assert result.raw_kernel_func is _add_kernel + + def test_register_kernel_auto_detects_name(self): + """Test register_kernel uses function name when no name provided.""" + with dummy_kernel_registry() as register: + wrapper = register(config_picker=lambda args, keys: None)(_add_kernel) + + assert wrapper.op_name == "_add_kernel" + + def test_register_kernel_registers_in_global_registry(self): + """Test register_kernel adds wrapper to global registry.""" + with dummy_kernel_registry() as register: + wrapper = register("test_kernel", config_picker=lambda args, keys: None)( + _add_kernel + ) + + registered_kernels = get_registered_kernels() + assert "test_kernel" in registered_kernels + assert registered_kernels["test_kernel"] is wrapper + + def test_register_kernel_passes_helion_settings(self): + """Test register_kernel passes helion_settings to wrapper.""" + settings = helion.Settings() + settings.print_output_code = True + + with dummy_kernel_registry() as register: + result = register( + "test_name", + config_picker=lambda args, keys: None, + helion_settings=settings, + )(_add_kernel) + + assert result.helion_settings is settings + + def test_register_kernel_supports_decorator_syntax(self): + """Test register_kernel works with decorator arguments.""" + mock_fake = Mock() + + with dummy_kernel_registry() as register: + result = register( + "custom_name", + config_picker=lambda args, keys: None, + fake_impl=mock_fake, + )(_add_kernel) + + assert result.op_name == "custom_name" + assert result._fake_impl is mock_fake + + def test_register_kernel_raises_on_duplicate_registration(self): + """Test register_kernel raises error on duplicate names.""" + with dummy_kernel_registry() as register: + register("duplicate_name", config_picker=lambda args, keys: None)( + _add_kernel + ) + + with pytest.raises(ValueError, match="already registered"): + register("duplicate_name", config_picker=lambda args, keys: None)( + _add_kernel + ) + + def test_register_kernel_rejects_autotuner_fn_in_settings(self): + """Test register_kernel rejects conflicting autotuner_fn.""" + mock_settings = Mock() + mock_settings.to_dict.return_value = {"autotuner_fn": Mock()} + + with pytest.raises(ValueError, match="uses a custom autotuner"): + + @register_kernel( + "test", + config_picker=lambda args, keys: None, + helion_settings=mock_settings, + ) + def test_kernel(x): + return x + + def test_register_kernel_no_warning_with_static_shapes_false(self): + """Test register_kernel doesn't warn with static_shapes=False.""" + mock_settings = Mock() + mock_settings.to_dict.return_value = {"static_shapes": False} + + with ( + dummy_kernel_registry() as register, + patch("vllm.kernels.helion.register.logger") as mock_logger, + ): + register( + "test", + config_picker=lambda args, keys: None, + helion_settings=mock_settings, + )(_add_kernel) + + mock_logger.warning.assert_not_called() + + def test_disabled_kernel_appears_in_registry(self): + """Test that a disabled wrapper is still in the global registry.""" + + def fake_impl(*args, **kwargs): + return torch.zeros_like(args[0]) + + mock_config_manager = Mock(spec=ConfigManager) + mock_config_manager.get_platform_configs = Mock(return_value={}) + + with ( + patch( + "vllm.kernels.helion.config_manager.ConfigManager", + return_value=mock_config_manager, + ), + patch( + "vllm.kernels.helion.utils.get_canonical_gpu_name", + return_value="nvidia_h200", + ), + patch("vllm.kernels.helion.register.helion.kernel") as mock_kernel, + ): + mock_kernel.return_value = Mock(return_value=_add_kernel) + + wrapper = register_kernel( + "disabled_kernel", + config_picker=lambda args, keys: None, + fake_impl=fake_impl, + )(_add_kernel) + + assert wrapper._disabled is True + registered = get_registered_kernels() + assert "disabled_kernel" in registered + assert registered["disabled_kernel"] is wrapper + + +@pytest.mark.skipif(not _HOP_AVAILABLE, reason="Requires PyTorch >= 2.11 for HOP") +class TestTorchCompileHOP: + """Test that HelionKernelWrapper emits the correct HOP under torch.compile.""" + + def test_compiled_graph_contains_helion_hop(self): + """Verify torch.compile on a HelionKernelWrapper emits a + helion_kernel_wrapper_mutation HOP node in the FX graph.""" + configs: dict[CaseKey, helion.Config] = { + CaseKey.default(): helion.Config(block_sizes=[4, 4]) + } + + with dummy_kernel_registry(configs=configs) as register: + add_helion_kernel = register( + op_name="test_torch_compile_add_kernel", + config_picker=lambda args, keys: None, + )(_add_kernel) + + captured_graph: torch.fx.GraphModule | None = None + + def capturing_backend(gm, example_inputs): + nonlocal captured_graph + assert captured_graph is None, "Backend called multiple times" + captured_graph = gm + return gm.forward + + def f(x, y): + return add_helion_kernel(x, y) + + torch._dynamo.reset() + compiled_f = torch.compile(f, backend=capturing_backend, fullgraph=True) + + x = torch.randn(4, 4, device="cuda") + y = torch.randn(4, 4, device="cuda") + + # Run compiled version and capture graph + compiled_result = compiled_f(x, y) + + assert captured_graph is not None + hop_nodes = [ + node + for node in captured_graph.graph.nodes + if node.op == "call_function" + and node.target is helion_kernel_wrapper_mutation + ] + assert len(hop_nodes) > 0, ( + "Expected helion_kernel_wrapper_mutation HOP node in compiled graph, " + f"but found none. Graph nodes: " + f"{[(n.op, n.target) for n in captured_graph.graph.nodes]}" + ) + + # Verify compiled result matches eager execution + eager_result = f(x, y) # Run in eager mode + + assert torch.allclose(compiled_result, eager_result, atol=1e-5, rtol=1e-5), ( + "Compiled execution result doesn't match eager execution. " + f"Max difference: {torch.max(torch.abs(compiled_result - eager_result))}" + ) + + @pytest.mark.skipif( + not (_HOP_AVAILABLE and supports_torch_compile_fusion()), + reason="Requires PyTorch with Helion inductor fusion support", + ) + def test_inductor_backend_compiles_helion_hop(self): + """Test torch.compile with inductor backend and Helion fusion enabled.""" + + configs: dict[CaseKey, helion.Config] = { + CaseKey.default(): helion.Config(block_sizes=[4, 4]) + } + + with dummy_kernel_registry(configs=configs) as register: + add_helion_kernel = register( + op_name="test_inductor_add_kernel", + config_picker=lambda args, keys: None, + helion_settings=helion.Settings( + torch_compile_fusion=True, static_shapes=False + ), + )(_add_kernel) + + def f(x, y): + x = x * 2.0 + y = y + 1.0 + out = add_helion_kernel(x, y) + return out.relu() + + torch._dynamo.reset() + compiled_f = torch.compile(f, backend="inductor", fullgraph=True) + + x = torch.randn(4, 4, device="cuda") + y = torch.randn(4, 4, device="cuda") + + compiled_result, source_codes = run_and_get_code(compiled_f, x, y) + eager_result = f(x, y) + + assert torch.allclose(compiled_result, eager_result, atol=1e-5, rtol=1e-5), ( + "Inductor-compiled result doesn't match eager execution. " + f"Max difference: {torch.max(torch.abs(compiled_result - eager_result))}" + ) + + # With fusion enabled, prologue/epilogue ops should be fused into + # a single triton kernel rather than generating separate kernels. + kernel_count = sum(code.count("@triton.jit") for code in source_codes) + assert kernel_count == 1, ( + f"Expected 1 fused triton kernel, got {kernel_count}. " + "Prologue/epilogue ops were not fused into the Helion kernel." + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/test_rms_norm_dynamic_per_token_quant.py b/asian_ai/performance/vLLM/tests/kernels/helion/test_rms_norm_dynamic_per_token_quant.py new file mode 100644 index 0000000000000000000000000000000000000000..46d26bd5b57044a05a5cfefaf2ec3f812a837ffd --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/test_rms_norm_dynamic_per_token_quant.py @@ -0,0 +1,207 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the rms_norm_dynamic_per_token_quant helion kernel + +Run `pytest tests/kernels/helion/test_rms_norm_dynamic_per_token_quant.py`. +""" + +from typing import Any + +import pytest +import torch +from torch._subclasses.fake_tensor import FakeTensorMode + +from tests.kernels.helion.utils import skip_if_platform_unsupported +from vllm.kernels.helion.case_key import CaseKey +from vllm.kernels.helion.config_manager import ConfigManager +from vllm.kernels.helion.ops.rms_norm_dynamic_per_token_quant import ( + _pick_cache, + baseline, + pick_config, + rms_norm_dynamic_per_token_quant, +) +from vllm.platforms import current_platform +from vllm.utils.import_utils import has_helion +from vllm.utils.torch_utils import set_random_seed + +if not has_helion(): + pytest.skip( + "Helion is not installed. Install with: pip install vllm[helion]", + allow_module_level=True, + ) + + +def _generate_fake_input(num_tokens: int, hidden_size: int) -> tuple[Any, ...]: + with FakeTensorMode(): + input = torch.randn( + num_tokens, hidden_size, device="cuda", dtype=torch.bfloat16 + ) + result = torch.empty( + input.shape, device=input.device, dtype=current_platform.fp8_dtype() + ) + scale = torch.empty((num_tokens, 1), device=input.device, dtype=torch.float32) + scale_ub = torch.mean(input).to(torch.float32) + residual = torch.randn_like(input) + weight = torch.normal( + mean=1.0, + std=1.0, + size=(hidden_size,), + dtype=input.dtype, + device=input.device, + ) + epsilon = 1e-6 + args = (result, input, weight, scale, epsilon, scale_ub, residual) + return args + + +@pytest.fixture(autouse=True) +def reset_config_manager_singleton(): + ConfigManager.reset_instance() + ConfigManager() + yield + ConfigManager.reset_instance() + + +class TestRmsNormDynamicPerTokenQuantConfigPicker: + def setup_method(self): + _pick_cache.clear() + + def test_config_picker_exact_match(self): + config_keys = [ + CaseKey({"hidden_size": 2048, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "num_tokens": 16}), + ] + + args = _generate_fake_input(16, 4096) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey({"hidden_size": 4096, "num_tokens": 16}) + + def test_config_picker_closest_match(self): + config_keys = [ + CaseKey({"hidden_size": 2048, "num_tokens": 16}), + CaseKey({"hidden_size": 2048, "num_tokens": 32}), + CaseKey({"hidden_size": 4096, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "num_tokens": 32}), + ] + + args = _generate_fake_input(20, 3000) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey({"hidden_size": 2048, "num_tokens": 32}) + + def test_config_picker_no_configs(self): + config_keys: list[dict] = [] + + args = _generate_fake_input(16, 4096) + selected_key = pick_config(args, config_keys) + assert selected_key is None + + def test_config_picker_fallback_to_largest(self): + config_keys = [ + CaseKey({"hidden_size": 2048, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "num_tokens": 16}), + ] + + args = _generate_fake_input(32, 8192) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey({"hidden_size": 4096, "num_tokens": 16}) + + +DTYPES = [torch.bfloat16, torch.float] +QUANT_DTYPES = [torch.int8, current_platform.fp8_dtype()] +VEC_HIDDEN_SIZES = [1024, 1025, 1027, 1029] +# Avoid combinatorial explosion with full Cartesian product +NUM_TOKENS_HIDDEN_SIZES = [ + *[(1, i) for i in [1, 64, *VEC_HIDDEN_SIZES, 5120, 5137]], + *[(2048, i) for i in [1, 64, *VEC_HIDDEN_SIZES, 5137]], + *[(4096, i) for i in [1, 64, 5137]], +] + +ADD_RESIDUAL = [False, True] +SCALE_UBS = [True, False] +SEEDS = [0] + +EPS = 1e-6 + + +class TestRmsNormDynamicPerTokenQuantCorrectness: + @pytest.mark.parametrize("num_tokens, hidden_size", NUM_TOKENS_HIDDEN_SIZES) + @pytest.mark.parametrize("add_residual", ADD_RESIDUAL) + @pytest.mark.parametrize("has_scale_ub", SCALE_UBS) + @pytest.mark.parametrize("dtype", DTYPES) + @pytest.mark.parametrize("quant_dtype", QUANT_DTYPES) + @pytest.mark.parametrize("seed", SEEDS) + def test_rms_norm_dynamic_per_token_quant( + self, + num_tokens: int, + hidden_size: int, + add_residual: bool, + has_scale_ub: bool, + dtype: torch.dtype, + quant_dtype: torch.dtype, + seed: int, + ) -> None: + skip_if_platform_unsupported("rms_norm_dynamic_per_token_quant") + + set_random_seed(seed) + + if has_scale_ub and quant_dtype != current_platform.fp8_dtype(): + # skip + return + + scale = 1 / (hidden_size) + x = torch.randn(num_tokens, hidden_size, dtype=dtype, device="cuda") * scale + weight = torch.normal( + mean=1.0, std=1.0, size=(hidden_size,), dtype=dtype, device=x.device + ) + residual = torch.randn_like(x) * scale if add_residual else None + scale_ub = ( + torch.mean(x).to(dtype=torch.float32, device="cuda") + if has_scale_ub + else None + ) + + ref_out = torch.empty(x.shape, device=x.device, dtype=quant_dtype) + ref_scales = torch.empty((x.shape[0], 1), device=x.device, dtype=torch.float32) + ref_residual = residual.clone() if residual is not None else None + baseline(ref_out, x, weight, ref_scales, EPS, scale_ub, ref_residual) + + ops_out = torch.empty(x.shape, device=x.device, dtype=quant_dtype) + ops_scales = torch.empty((x.shape[0], 1), device=x.device, dtype=torch.float32) + ops_residual = residual.clone() if residual is not None else None + rms_norm_dynamic_per_token_quant( + ops_out, x, weight, ops_scales, EPS, scale_ub, ops_residual + ) + + torch.testing.assert_close(ref_scales, ops_scales) + # allow 1 ULP difference + assert ( + ref_out.view(torch.uint8).to(torch.int16) + - ops_out.view(torch.uint8).to(torch.int16) + ).abs().max() <= 1 + + if add_residual: + torch.testing.assert_close(ref_residual, ops_residual) + + +class TestRmsNormDynamicPerTokenQuantIntegration: + def test_kernel_registration_integration(self): + from vllm.kernels.helion.register import get_registered_kernels + + registered_kernels = get_registered_kernels() + assert "rms_norm_dynamic_per_token_quant" in registered_kernels + + kernel_wrapper = registered_kernels["rms_norm_dynamic_per_token_quant"] + assert kernel_wrapper.op_name == "rms_norm_dynamic_per_token_quant" + assert kernel_wrapper._config_picker is not None + assert kernel_wrapper._mutates_args == ["result", "scale", "residual"] + + def test_fake_impl_functionality(self): + skip_if_platform_unsupported("rms_norm_dynamic_per_token_quant") + from vllm.kernels.helion.register import get_registered_kernels + + registered_kernels = get_registered_kernels() + kernel_wrapper = registered_kernels["rms_norm_dynamic_per_token_quant"] + fake_impl = kernel_wrapper._fake_impl + + args = _generate_fake_input(16, 4096) + assert fake_impl(*args) is None diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/test_rms_norm_per_block_quant.py b/asian_ai/performance/vLLM/tests/kernels/helion/test_rms_norm_per_block_quant.py new file mode 100644 index 0000000000000000000000000000000000000000..ebe7286dc874d159f4013f48c88d005bb656a4a9 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/test_rms_norm_per_block_quant.py @@ -0,0 +1,298 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the rms_norm_per_block_quant helion kernel + +Run `pytest tests/kernels/helion/test_rms_norm_per_block_quant.py`. +""" + +import itertools +from typing import Any + +import pytest +import torch +from torch._subclasses.fake_tensor import FakeTensorMode + +from tests.kernels.helion.utils import skip_if_platform_unsupported +from tests.kernels.quant_utils import FP8_DTYPE +from vllm.kernels.helion.case_key import CaseKey +from vllm.kernels.helion.config_manager import ConfigManager +from vllm.kernels.helion.ops.rms_norm_per_block_quant import ( + _pick_cache, + baseline, + pick_config, + rms_norm_per_block_quant, +) +from vllm.utils.import_utils import has_helion +from vllm.utils.torch_utils import set_random_seed + +if not has_helion(): + pytest.skip( + "Helion is not installed. Install with: pip install vllm[helion]", + allow_module_level=True, + ) + + +def _generate_fake_input( + num_tokens: int, hidden_size: int, group_size: int +) -> tuple[Any, ...]: + with FakeTensorMode(): + input = torch.randn( + (num_tokens, hidden_size), device="cuda", dtype=torch.bfloat16 + ) + result = torch.empty(input.shape, device=input.device, dtype=FP8_DTYPE) + scale = torch.empty( + (num_tokens, hidden_size // group_size), + device=input.device, + dtype=torch.float32, + ) + scale_ub = torch.mean(input).to(scale.dtype) + residual = torch.randn_like(input) + weight = torch.normal( + mean=1.0, + std=1.0, + size=(hidden_size,), + dtype=input.dtype, + device=input.device, + ) + epsilon = 1e-6 + args = ( + result, + input, + weight, + scale, + epsilon, + scale_ub, + residual, + group_size, + False, + ) + return args + + +@pytest.fixture(autouse=True) +def reset_config_manager_singleton(): + ConfigManager.reset_instance() + ConfigManager() + yield + ConfigManager.reset_instance() + + +class TestRmsNormPerBlockQuantConfigPicker: + def setup_method(self): + _pick_cache.clear() + + def test_config_picker_exact_match(self): + config_keys = [ + CaseKey({"hidden_size": 2048, "group_size": 64, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "group_size": 128, "num_tokens": 16}), + ] + + args = _generate_fake_input(16, 4096, 128) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey( + {"hidden_size": 4096, "group_size": 128, "num_tokens": 16} + ) + + def test_config_picker_closest_match(self): + config_keys = [ + CaseKey({"hidden_size": 2048, "group_size": 64, "num_tokens": 16}), + CaseKey({"hidden_size": 2048, "group_size": 64, "num_tokens": 32}), + CaseKey({"hidden_size": 2048, "group_size": 128, "num_tokens": 16}), + CaseKey({"hidden_size": 2048, "group_size": 128, "num_tokens": 32}), + CaseKey({"hidden_size": 4096, "group_size": 64, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "group_size": 64, "num_tokens": 32}), + CaseKey({"hidden_size": 4096, "group_size": 128, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "group_size": 128, "num_tokens": 32}), + ] + + args = _generate_fake_input(20, 3000, 70) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey( + {"hidden_size": 2048, "group_size": 64, "num_tokens": 32} + ) + + def test_config_picker_no_configs(self): + config_keys: list[dict] = [] + + args = _generate_fake_input(16, 4096, 128) + selected_key = pick_config(args, config_keys) + assert selected_key is None + + def test_config_picker_fallback_to_largest(self): + config_keys = [ + CaseKey({"hidden_size": 2048, "group_size": 64, "num_tokens": 16}), + CaseKey({"hidden_size": 2048, "group_size": 64, "num_tokens": 32}), + CaseKey({"hidden_size": 2048, "group_size": 128, "num_tokens": 16}), + CaseKey({"hidden_size": 2048, "group_size": 128, "num_tokens": 32}), + CaseKey({"hidden_size": 4096, "group_size": 64, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "group_size": 64, "num_tokens": 32}), + CaseKey({"hidden_size": 4096, "group_size": 128, "num_tokens": 16}), + CaseKey({"hidden_size": 4096, "group_size": 128, "num_tokens": 32}), + ] + + args = _generate_fake_input(64, 8192, 256) + selected_key = pick_config(args, config_keys) + assert selected_key == CaseKey( + {"hidden_size": 4096, "group_size": 128, "num_tokens": 32} + ) + + +DTYPES = [torch.bfloat16, torch.float] +QUANT_DTYPES = [torch.int8, FP8_DTYPE] +VEC_HIDDEN_SIZES = [64, 1024] +# Avoid combinatorial explosion with full Cartesian product +NUM_TOKENS_HIDDEN_SIZES = [ + *[(1, i) for i in [64, 128, 1024, 5120]], + *[(2048, i) for i in [64, 1024]], + *[(4096, i) for i in [64]], +] + +ADD_RESIDUAL = [False, True] +SCALE_UBS = [True, False] +GROUP_SIZES = [64, 128] +TMA_ALIGNMENTS = [0, 4] +SEEDS = [0] +EPS = 1e-6 + + +class TestRmsNormPerBlockQuantCorrectness: + @pytest.mark.parametrize("num_tokens, hidden_size", NUM_TOKENS_HIDDEN_SIZES) + @pytest.mark.parametrize("add_residual", ADD_RESIDUAL) + @pytest.mark.parametrize("has_scale_ub", SCALE_UBS) + @pytest.mark.parametrize("dtype", DTYPES) + @pytest.mark.parametrize("quant_dtype", QUANT_DTYPES) + @pytest.mark.parametrize("is_scale_transposed", [False, True]) + @pytest.mark.parametrize( + "group_size, tma_alignment", + [*itertools.product(GROUP_SIZES, TMA_ALIGNMENTS)], + ) + @pytest.mark.parametrize("seed", SEEDS) + def test_rms_norm_per_block_quant( + self, + num_tokens: int, + hidden_size: int, + add_residual: bool, + has_scale_ub: bool, + dtype: torch.dtype, + quant_dtype: torch.dtype, + is_scale_transposed: bool, + group_size: int, + tma_alignment: int, + seed: int, + ) -> None: + skip_if_platform_unsupported("rms_norm_per_block_quant") + + set_random_seed(seed) + + if hidden_size % group_size != 0: + # skip + return + + if tma_alignment != 0 and hidden_size // group_size % tma_alignment == 0: + # Skip tests where TMA alignment doesn't create extra padding to save time + return + + if has_scale_ub and quant_dtype != FP8_DTYPE: + # skip + return + + scale = 1 / (hidden_size) + input = torch.randn(num_tokens, hidden_size, dtype=dtype, device="cuda") * scale + weight = torch.normal( + mean=1.0, std=1.0, size=(hidden_size,), dtype=dtype, device=input.device + ) + residual = torch.randn_like(input) * scale if add_residual else None + scale_ub = ( + torch.mean(input).to(dtype=torch.float32, device="cuda") + if has_scale_ub + else None + ) + groups_per_row = hidden_size // group_size + + ref_residual = residual.clone() if residual is not None else None + ops_residual = residual.clone() if residual is not None else None + ref_out = torch.empty(input.shape, device=input.device, dtype=quant_dtype) + ops_out = ref_out.clone() + + if is_scale_transposed: + if tma_alignment == 0: + ref_scales = torch.empty( + (groups_per_row, num_tokens), + device=input.device, + dtype=torch.float32, + ).transpose(0, 1) + else: + tma_aligned_m = ( + (num_tokens + tma_alignment - 1) // tma_alignment * tma_alignment + ) + shape = (num_tokens, groups_per_row) + stride = (1, tma_aligned_m) + ref_scales = torch.empty_strided( + shape, stride, device=input.device, dtype=torch.float32 + ) + else: + ref_scales = torch.empty( + (num_tokens, groups_per_row), device=input.device, dtype=torch.float32 + ) + + ops_scales = ref_scales.clone() + + baseline( + ref_out, + input, + weight, + ref_scales, + EPS, + scale_ub, + ref_residual, + group_size, + is_scale_transposed, + ) + ref_scales = ref_scales.contiguous() + + rms_norm_per_block_quant( + ops_out, + input, + weight, + ops_scales, + EPS, + scale_ub, + ops_residual, + group_size, + is_scale_transposed, + ) + ops_scales = ops_scales.contiguous() + + torch.testing.assert_close(ref_scales, ops_scales) + # allow 1 ULP difference + assert ( + ref_out.view(torch.uint8).to(torch.int16) + - ops_out.view(torch.uint8).to(torch.int16) + ).abs().max() <= 1 + + if add_residual: + torch.testing.assert_close(ref_residual, ops_residual) + + +class TestRmsNormPerBlockQuantIntegration: + def test_kernel_registration_integration(self): + from vllm.kernels.helion.register import get_registered_kernels + + registered_kernels = get_registered_kernels() + assert "rms_norm_per_block_quant" in registered_kernels + + kernel_wrapper = registered_kernels["rms_norm_per_block_quant"] + assert kernel_wrapper.op_name == "rms_norm_per_block_quant" + assert kernel_wrapper._config_picker is not None + assert kernel_wrapper._mutates_args == ["result", "scale", "residual"] + + def test_fake_impl_functionality(self): + skip_if_platform_unsupported("rms_norm_per_block_quant") + from vllm.kernels.helion.register import get_registered_kernels + + registered_kernels = get_registered_kernels() + kernel_wrapper = registered_kernels["rms_norm_per_block_quant"] + fake_impl = kernel_wrapper._fake_impl + + args = _generate_fake_input(16, 4096, 128) + assert fake_impl(*args) is None diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/test_silu_mul_fp8.py b/asian_ai/performance/vLLM/tests/kernels/helion/test_silu_mul_fp8.py new file mode 100644 index 0000000000000000000000000000000000000000..fccdf15729075f3ea856ffb73806f657c49fce9b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/test_silu_mul_fp8.py @@ -0,0 +1,364 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch +import torch.nn.functional as F + +from vllm.utils.import_utils import has_helion + +if not has_helion(): + pytest.skip( + "Helion is not installed. Install with: pip install vllm[helion]", + allow_module_level=True, + ) + +from vllm.kernels.helion.case_key import CaseKey +from vllm.kernels.helion.config_manager import ConfigManager +from vllm.kernels.helion.ops.silu_mul_fp8 import ( + _pick_cache, + pick_silu_mul_fp8_config, + silu_mul_fp8, + silu_mul_fp8_baseline, +) + + +def skip_if_platform_unsupported(): + try: + from vllm.kernels.helion.utils import get_canonical_gpu_name + + if not torch.cuda.is_available(): + pytest.skip("CUDA not available") + + platform = get_canonical_gpu_name() + + try: + config_manager = ConfigManager.get_instance() + except RuntimeError: + config_manager = ConfigManager() + + configs = config_manager.get_platform_configs("silu_mul_fp8", platform) + if len(configs) == 0: + pytest.skip("Current GPU platform not supported for silu_mul_fp8 kernel") + + except (ImportError, RuntimeError, KeyError): + pytest.skip("Error detecting platform support for silu_mul_fp8 kernel") + + +@pytest.fixture(autouse=True) +def reset_config_manager_singleton(): + ConfigManager.reset_instance() + ConfigManager() + yield + ConfigManager.reset_instance() + + +class TestSiluMulFp8ConfigPicker: + def setup_method(self): + _pick_cache.clear() + + def test_config_picker_exact_match(self): + config_keys = [ + CaseKey({"intermediate": 2048, "numtokens": 256}), + CaseKey({"intermediate": 4096, "numtokens": 256}), + ] + + input_tensor = torch.randn(32, 4096, dtype=torch.bfloat16, device="cuda") + scale = torch.tensor([0.5], dtype=torch.float32, device="cuda") + args = (input_tensor, scale) + + selected_key = pick_silu_mul_fp8_config(args, config_keys) + assert selected_key == CaseKey({"intermediate": 2048, "numtokens": 256}) + + def test_config_picker_closest_match(self): + config_keys = [ + CaseKey({"intermediate": 2048, "numtokens": 256}), + CaseKey({"intermediate": 4096, "numtokens": 256}), + ] + input_tensor = torch.randn(32, 7000, dtype=torch.bfloat16, device="cuda") + scale = torch.tensor([0.5], dtype=torch.float32, device="cuda") + args = (input_tensor, scale) + + selected_key = pick_silu_mul_fp8_config(args, config_keys) + assert selected_key == CaseKey({"intermediate": 4096, "numtokens": 256}) + + def test_config_picker_no_configs(self): + config_keys: list[dict] = [] + + input_tensor = torch.randn(32, 4096, dtype=torch.bfloat16, device="cuda") + scale = torch.tensor([0.5], dtype=torch.float32, device="cuda") + args = (input_tensor, scale) + + selected_key = pick_silu_mul_fp8_config(args, config_keys) + assert selected_key is None + + @pytest.mark.parametrize("intermediate_size", [2048, 4096, 5120]) + def test_config_picker_different_sizes(self, intermediate_size): + config_keys = [ + CaseKey({"intermediate": 2048, "numtokens": 256}), + CaseKey({"intermediate": 4096, "numtokens": 256}), + CaseKey({"intermediate": 5120, "numtokens": 256}), + ] + + input_tensor = torch.randn( + 32, 2 * intermediate_size, dtype=torch.bfloat16, device="cuda" + ) + scale = torch.tensor([0.5], dtype=torch.float32, device="cuda") + args = (input_tensor, scale) + + selected_key = pick_silu_mul_fp8_config(args, config_keys) + assert selected_key == { + "intermediate": intermediate_size, + "numtokens": 256, + } + + def test_config_picker_numtokens_ceiling(self): + config_keys = [ + CaseKey({"intermediate": 4096, "numtokens": 8}), + CaseKey({"intermediate": 4096, "numtokens": 32}), + CaseKey({"intermediate": 4096, "numtokens": 128}), + CaseKey({"intermediate": 4096, "numtokens": 256}), + ] + input_tensor = torch.randn(20, 8192, dtype=torch.bfloat16, device="cuda") + scale = torch.tensor([0.5], dtype=torch.float32, device="cuda") + + selected_key = pick_silu_mul_fp8_config((input_tensor, scale), config_keys) + assert selected_key == CaseKey({"intermediate": 4096, "numtokens": 32}) + + def test_config_picker_numtokens_exact(self): + config_keys = [ + CaseKey({"intermediate": 4096, "numtokens": 8}), + CaseKey({"intermediate": 4096, "numtokens": 32}), + CaseKey({"intermediate": 4096, "numtokens": 128}), + ] + input_tensor = torch.randn(32, 8192, dtype=torch.bfloat16, device="cuda") + scale = torch.tensor([0.5], dtype=torch.float32, device="cuda") + + selected_key = pick_silu_mul_fp8_config((input_tensor, scale), config_keys) + assert selected_key == CaseKey({"intermediate": 4096, "numtokens": 32}) + + def test_config_picker_numtokens_fallback_to_largest(self): + config_keys = [ + CaseKey({"intermediate": 4096, "numtokens": 8}), + CaseKey({"intermediate": 4096, "numtokens": 32}), + CaseKey({"intermediate": 4096, "numtokens": 128}), + ] + input_tensor = torch.randn(512, 8192, dtype=torch.bfloat16, device="cuda") + scale = torch.tensor([0.5], dtype=torch.float32, device="cuda") + + selected_key = pick_silu_mul_fp8_config((input_tensor, scale), config_keys) + assert selected_key == CaseKey({"intermediate": 4096, "numtokens": 128}) + + +class TestSiluMulFp8Correctness: + @pytest.mark.parametrize("batch_size", [1, 8, 32, 128]) + @pytest.mark.parametrize("intermediate_size", [2048, 3000, 3500, 4096, 5000]) + @pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) + def test_silu_mul_fp8_correctness(self, batch_size, intermediate_size, dtype): + skip_if_platform_unsupported() + + input_size = 2 * intermediate_size + input_tensor = torch.randn(batch_size, input_size, dtype=dtype, device="cuda") + scale = torch.tensor([0.5], dtype=torch.float32, device="cuda") + + reference_output = silu_mul_fp8_baseline(input_tensor, scale) + helion_output = silu_mul_fp8(input_tensor, scale) + + assert helion_output.shape == reference_output.shape + assert helion_output.dtype == torch.float8_e4m3fn + assert reference_output.dtype == torch.float8_e4m3fn + + ref_f32 = reference_output.to(torch.float32) + helion_f32 = helion_output.to(torch.float32) + # FP8 E4M3 has limited precision. Values near quantization boundaries + # can round differently due to intermediate precision differences. + torch.testing.assert_close( + helion_f32, + ref_f32, + atol=0.05, + rtol=0.05, + msg=f"Mismatch at batch={batch_size}, size={intermediate_size}", + ) + + def test_silu_mul_fp8_shape_inference(self): + skip_if_platform_unsupported() + batch_size, input_size = 32, 8192 + intermediate_size = input_size // 2 + + input_tensor = torch.randn( + batch_size, input_size, dtype=torch.bfloat16, device="cuda" + ) + scale = torch.tensor([0.5], dtype=torch.float32, device="cuda") + + output = silu_mul_fp8(input_tensor, scale) + + expected_shape = (batch_size, intermediate_size) + assert output.shape == expected_shape + assert output.dtype == torch.float8_e4m3fn + + def test_silu_mul_fp8_scale_variations(self): + skip_if_platform_unsupported() + batch_size, input_size = 16, 4096 + + input_tensor = torch.randn( + batch_size, input_size, dtype=torch.bfloat16, device="cuda" + ) + + scales = [0.1, 0.5, 1.0, 2.0, 10.0] + + for scale_val in scales: + scale = torch.tensor([scale_val], dtype=torch.float32, device="cuda") + + reference_output = silu_mul_fp8_baseline(input_tensor, scale) + helion_output = silu_mul_fp8(input_tensor, scale) + ref_f32 = reference_output.to(torch.float32) + helion_f32 = helion_output.to(torch.float32) + + torch.testing.assert_close( + helion_f32, + ref_f32, + atol=0.05, + rtol=0.05, + msg=f"Mismatch for scale={scale_val}", + ) + + @pytest.mark.parametrize( + "shape", + [ + (1, 4096), + (16, 4096), + (128, 4096), + (1024, 4096), + (1, 8192), + (16, 8192), + (128, 8192), + ], + ) + def test_silu_mul_fp8_various_shapes(self, shape): + skip_if_platform_unsupported() + + input_tensor = torch.randn(*shape, dtype=torch.bfloat16, device="cuda") + scale = torch.tensor([0.5], dtype=torch.float32, device="cuda") + + reference_output = silu_mul_fp8_baseline(input_tensor, scale) + helion_output = silu_mul_fp8(input_tensor, scale) + + assert helion_output.shape == reference_output.shape + + ref_f32 = reference_output.to(torch.float32) + helion_f32 = helion_output.to(torch.float32) + + torch.testing.assert_close( + helion_f32, ref_f32, atol=0.05, rtol=0.05, msg=f"Mismatch for shape={shape}" + ) + + +def silu_mul_fp8_pytorch(input: torch.Tensor, scale: torch.Tensor) -> torch.Tensor: + """Pure PyTorch reference using F.silu. + + This matches vLLM's SiluAndMul.forward_native exactly: + F.silu(x[..., :d]) * x[..., d:] + """ + d = input.shape[-1] // 2 + result = F.silu(input[..., :d]) * input[..., d:] + return (result.to(torch.float32) / scale).to(torch.float8_e4m3fn) + + +class TestSiluMulFp8PytorchReference: + """Tests comparing Helion kernel against pure PyTorch implementation. + + Uses tighter tolerance since both use PyTorch's FP8 conversion + (same rounding mode), unlike the vLLM C++ baseline which uses + NVIDIA's hardware FP8 conversion with different rounding. + """ + + @pytest.mark.parametrize("batch_size", [1, 8, 32, 128, 256]) + @pytest.mark.parametrize("intermediate_size", [1024, 2048, 4096]) + @pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) + def test_silu_mul_fp8_vs_pytorch(self, batch_size, intermediate_size, dtype): + skip_if_platform_unsupported() + + input_tensor = torch.randn( + batch_size, 2 * intermediate_size, dtype=dtype, device="cuda" + ) + scale = torch.tensor([0.5], dtype=torch.float32, device="cuda") + + pytorch_output = silu_mul_fp8_pytorch(input_tensor, scale) + helion_output = silu_mul_fp8(input_tensor, scale) + + assert helion_output.shape == pytorch_output.shape + assert helion_output.dtype == torch.float8_e4m3fn + + pytorch_f32 = pytorch_output.to(torch.float32) + helion_f32 = helion_output.to(torch.float32) + + # Tolerance accounts for FP8 quantization boundary effects + torch.testing.assert_close( + helion_f32, + pytorch_f32, + atol=0.05, + rtol=0.05, + msg=( + f"Mismatch at batch={batch_size}, size={intermediate_size}, " + f"dtype={dtype}" + ), + ) + + @pytest.mark.parametrize( + "shape", + [ + (1, 2, 4096), # 3D input + (2, 4, 2048), # 3D input + (1, 1, 1, 8192), # 4D input + ], + ) + def test_silu_mul_fp8_multidim_vs_pytorch(self, shape): + skip_if_platform_unsupported() + + input_tensor = torch.randn(*shape, dtype=torch.bfloat16, device="cuda") + scale = torch.tensor([0.5], dtype=torch.float32, device="cuda") + + pytorch_output = silu_mul_fp8_pytorch(input_tensor, scale) + helion_output = silu_mul_fp8(input_tensor, scale) + + assert helion_output.shape == pytorch_output.shape + + pytorch_f32 = pytorch_output.to(torch.float32) + helion_f32 = helion_output.to(torch.float32) + + torch.testing.assert_close( + helion_f32, + pytorch_f32, + atol=0.05, + rtol=0.05, + msg=f"Mismatch for shape={shape}", + ) + + +class TestSiluMulFp8Integration: + def test_kernel_registration_integration(self): + from vllm.kernels.helion.register import get_registered_kernels + + registered_kernels = get_registered_kernels() + assert "silu_mul_fp8" in registered_kernels + + kernel_wrapper = registered_kernels["silu_mul_fp8"] + assert kernel_wrapper.op_name == "silu_mul_fp8" + assert kernel_wrapper._config_picker is not None + + def test_fake_impl_functionality(self): + skip_if_platform_unsupported() + from vllm.kernels.helion.register import get_registered_kernels + + input_tensor = torch.randn(32, 4096, dtype=torch.bfloat16, device="cuda") + scale = torch.tensor([0.5], dtype=torch.float32, device="cuda") + registered_kernels = get_registered_kernels() + kernel_wrapper = registered_kernels["silu_mul_fp8"] + fake_impl = kernel_wrapper._fake_impl + + fake_output = fake_impl(input_tensor, scale) + + expected_shape = (32, 2048) + assert fake_output.shape == expected_shape + assert fake_output.dtype == torch.float8_e4m3fn + assert fake_output.device == input_tensor.device diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/test_utils.py b/asian_ai/performance/vLLM/tests/kernels/helion/test_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..54d282281b3200a12118e0ed334b8ef702c6da10 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/test_utils.py @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for Helion utility functions.""" + +import pytest + +from vllm.kernels.helion.utils import canonicalize_gpu_name + + +@pytest.mark.parametrize( + "driver_reported_name,expected", + [ + ("NVIDIA H200", "nvidia_h200"), + ("NVIDIA A100-SXM4-80GB", "nvidia_a100"), + ("NVIDIA H100 80GB HBM3", "nvidia_h100"), + ("NVIDIA H100 PCIe", "nvidia_h100"), + ("NVIDIA H100 SXM5", "nvidia_h100"), + ("NVIDIA GeForce RTX 4090", "nvidia_geforce_rtx_4090"), + ("AMD Instinct MI300X", "amd_instinct_mi300x"), + ("AMD Instinct MI250X / MI250", "amd_instinct_mi250x_mi250"), + ("Tesla V100-SXM2-32GB", "tesla_v100"), + ], +) +def test_canonicalize_gpu_name(driver_reported_name, expected): + """Test GPU name canonicalization.""" + assert canonicalize_gpu_name(driver_reported_name) == expected + + +@pytest.mark.parametrize("invalid_name", ["", " ", "\t", "\n"]) +def test_canonicalize_gpu_name_rejects_empty(invalid_name): + """Test that empty or whitespace-only names are rejected.""" + with pytest.raises(ValueError, match="cannot be empty"): + canonicalize_gpu_name(invalid_name) diff --git a/asian_ai/performance/vLLM/tests/kernels/helion/utils.py b/asian_ai/performance/vLLM/tests/kernels/helion/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..316977b61813b2c032cfa58afec39cbc4d5f214f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/helion/utils.py @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Helion Kernel test utils""" + +import pytest +import torch + +from vllm.kernels.helion.config_manager import ConfigManager + + +def skip_if_platform_unsupported(op_name: str): + try: + from vllm.kernels.helion.utils import get_canonical_gpu_name + + if not torch.cuda.is_available(): + pytest.skip("CUDA not available") + + platform = get_canonical_gpu_name() + + try: + config_manager = ConfigManager.get_instance() + except RuntimeError: + config_manager = ConfigManager() + + configs = config_manager.get_platform_configs(op_name, platform) + if len(configs) == 0: + pytest.skip(f"Current GPU platform not supported for {op_name} kernel") + + except (ImportError, RuntimeError, KeyError): + pytest.skip(f"Error detecting platform support for {op_name} kernel") diff --git a/asian_ai/performance/vLLM/tests/kernels/ir/test_ir_ops.py b/asian_ai/performance/vLLM/tests/kernels/ir/test_ir_ops.py new file mode 100644 index 0000000000000000000000000000000000000000..cee67b404eb82421225dcea935ccaef5a9dab1e5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/ir/test_ir_ops.py @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Meta-tests for vLLM IR op infrastructure. + +Ensures all registered ops have input generators defined. +Per-op correctness tests live alongside their op definitions +(e.g. tests/kernels/ir/test_layernorm.py). +""" + +import vllm.kernels # noqa: F401 — registers provider implementations +from vllm.ir.op import IrOp + + +def test_all_ops_have_input_generator(): + missing = [name for name, op in IrOp.registry.items() if not op.has_input_generator] + assert not missing, ( + f"IR ops without input generators: {missing}. " + f"Register one with @ir.ops..register_input_generator" + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/ir/test_layernorm.py b/asian_ai/performance/vLLM/tests/kernels/ir/test_layernorm.py new file mode 100644 index 0000000000000000000000000000000000000000..757327df3db8ee3cc584243398ef59ca244a2e47 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/ir/test_layernorm.py @@ -0,0 +1,386 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch + +# This registers op implementations +import vllm.kernels # noqa: F401 +from tests.ir.ir_test_utils import ( + COMMON_HIDDEN_SIZES, + NUM_TOKENS, + assert_close, + clone_args, + supported_providers, +) +from tests.kernels.allclose_default import get_default_rtol +from vllm import ir +from vllm.platforms import current_platform + +rms_norm_native = ir.ops.rms_norm.impls["native"].impl_fn + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike() and not current_platform.is_xpu(), + reason="Currently only kernels on CUDA, ROCm and XPU", +) +def test_rms_norm_registration(): + expected = { + "native": True, + "vllm_c": current_platform.is_cuda_alike(), + "aiter": current_platform.is_rocm(), + "oink": current_platform.has_device_capability(100) + and hasattr(torch.ops, "oink") + and hasattr(torch.ops.oink, "rmsnorm"), + "xpu_kernels": current_platform.is_xpu(), + } + + actual = { + provider: impl.supported for provider, impl in ir.ops.rms_norm.impls.items() + } + + assert actual == expected + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32]) +@pytest.mark.parametrize("n_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", COMMON_HIDDEN_SIZES) +@pytest.mark.parametrize("epsilon", [1e-6, 1e-5]) +@pytest.mark.skipif( + not current_platform.is_cuda_alike() and not current_platform.is_xpu(), + reason="Currently only kernels on CUDA, ROCm and XPU", +) +class TestRMSNorm: + @classmethod + def setup_class(cls, **kwargs): + torch.set_default_device(current_platform.device_type) + + def test_native_semantics(self, dtype, n_tokens, hidden_size, epsilon): + x, weight, epsilon = ir.ops.rms_norm.generate_inputs( + num_tokens=4, hidden_size=8, dtype=dtype, epsilon=epsilon + ) + out = rms_norm_native(x, weight, epsilon=epsilon) + + # Check shape, dtype, device + assert out.shape == x.shape + assert out.dtype == x.dtype + assert out.device == x.device + + # Check the scaling property of rms norm + out2 = rms_norm_native(x * 2.0, weight, epsilon=epsilon) + torch.testing.assert_close(out2, out, rtol=get_default_rtol(out), atol=1e-3) + + # Mean square should be approximately 1 (ignoring epsilon and weight scaling) + combined_norm = out.float() / weight.float() + variance = combined_norm.pow(2).mean(dim=-1) + # After RMS normalization, variance should be close to 1 + torch.testing.assert_close( + variance, torch.ones_like(variance), rtol=1e-2, atol=1e-2 + ) + + # Check behavior with and without weight + weight1 = torch.ones_like(weight) + out3 = rms_norm_native(x, weight1, epsilon=epsilon) + out4 = rms_norm_native(x, None, epsilon=epsilon) + torch.testing.assert_close(out3, out4) + + @pytest.mark.parametrize("provider", supported_providers(ir.ops.rms_norm)) + def test_impls(self, dtype, n_tokens, hidden_size, epsilon, provider): + impl = ir.ops.rms_norm.impls[provider] + x, weight, eps = ir.ops.rms_norm.generate_inputs( + num_tokens=n_tokens, hidden_size=hidden_size, dtype=dtype, epsilon=epsilon + ) + args = (x, weight, eps) + + if not impl.supports_args(*args): + pytest.skip(f"{provider} does not support args") + + ref_output = rms_norm_native(*clone_args(args)) + output = impl.impl_fn(*clone_args(args)) + assert_close(ir.ops.rms_norm, output, ref_output) + + # check that dispatched call matches direct call + with ir.ops.rms_norm.set_priority([provider, "native"]): + out_dispatched = ir.ops.rms_norm(*args) + out_direct = impl.impl_fn(*args) + torch.testing.assert_close(out_dispatched, out_direct, rtol=0.0, atol=0.0) + + # none of these support variance_size override + assert not impl.supports_args(x, weight, eps, 4) + assert not impl.supports_args(x, weight, eps, variance_size=4) + + # test weight=None behavior + out_no_weight = impl.impl_fn(x, None, eps) + out_unit_weight = impl.impl_fn(x, torch.ones_like(weight), eps) + assert_close(ir.ops.rms_norm, out_no_weight, out_unit_weight) + + @pytest.mark.parametrize("provider", ["vllm_c", "aiter", "xpu_kernels", "native"]) + def test_torch_opcheck(self, dtype, n_tokens, hidden_size, epsilon, provider): + if not ir.ops.rms_norm.impls[provider].supported: + pytest.skip(f"{provider} impl not supported on this platform") + + args = ir.ops.rms_norm.generate_inputs( + num_tokens=n_tokens, hidden_size=hidden_size, dtype=dtype, epsilon=epsilon + ) + + # When checking the torch op, we have to set priority and use dispatch + with ir.ops.rms_norm.set_priority([provider, "native"]): + torch.library.opcheck(torch.ops.vllm_ir.rms_norm, args) + + +@pytest.mark.skipif( + not current_platform.is_rocm(), + reason="aiter is only supported on ROCm", +) +def test_aiter_rejects_unsupported_dtypes(): + torch.set_default_device(current_platform.device_type) + impl = ir.ops.rms_norm.impls["aiter"] + for dtype in [torch.float32, torch.float64]: + args = ir.ops.rms_norm.generate_inputs( + num_tokens=8, hidden_size=4096, dtype=dtype, epsilon=1e-5 + ) + assert not impl.supports_args(*args), f"aiter should reject dtype={dtype}" + + +@pytest.mark.skipif( + not current_platform.is_rocm(), + reason="ROCm vllm_c RMSNorm needs explicit ND input handling", +) +def test_vllm_c_rms_norm_accepts_nd_input(): + torch.set_default_device(current_platform.device_type) + impl = ir.ops.rms_norm.impls["vllm_c"] + if not impl.supported: + pytest.skip("vllm_c impl not supported on this platform") + + base = torch.randn(3, 8, 192, dtype=torch.float16) + x = base.split(64, dim=-1)[0].view(3, 8, 4, 16) + assert not x.is_contiguous() + weight = torch.randn(16, dtype=torch.float16) + epsilon = 1e-5 + + output = impl.impl_fn(x, weight, epsilon) + ref_output = rms_norm_native(x, weight, epsilon) + + assert output.shape == x.shape + assert_close(ir.ops.rms_norm, output, ref_output) + + +fused_add_rms_norm_native = ir.ops.fused_add_rms_norm.impls["native"].impl_fn + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike() and not current_platform.is_xpu(), + reason="Currently only kernels on CUDA, ROCm and XPU", +) +def test_fused_add_rms_norm_registration(): + expected = { + "native": True, + "vllm_c": current_platform.is_cuda_alike(), + "aiter": current_platform.is_rocm(), + "oink": current_platform.has_device_capability(100) + and hasattr(torch.ops, "oink") + and hasattr(torch.ops.oink, "fused_add_rms_norm"), + "xpu_kernels": current_platform.is_xpu(), + } + + actual = { + provider: impl.supported + for provider, impl in ir.ops.fused_add_rms_norm.impls.items() + } + + assert actual == expected + + +@pytest.mark.skipif( + not current_platform.is_rocm(), + reason="ROCm vllm_c fused_add_rms_norm needs explicit ND input handling", +) +def test_vllm_c_fused_add_rms_norm_accepts_nd_input(): + torch.set_default_device(current_platform.device_type) + impl = ir.ops.fused_add_rms_norm.impls["vllm_c"] + if not impl.supported: + pytest.skip("vllm_c impl not supported on this platform") + + base = torch.randn(3, 8, 192, dtype=torch.float16) + residual_base = torch.randn(3, 8, 192, dtype=torch.float16) + x = base.split(64, dim=-1)[0].view(3, 8, 4, 16) + x_residual = residual_base.split(64, dim=-1)[0].view(3, 8, 4, 16) + assert not x.is_contiguous() + assert not x_residual.is_contiguous() + weight = torch.randn(16, dtype=torch.float16) + epsilon = 1e-5 + + output, residual = impl.impl_fn(x.clone(), x_residual.clone(), weight, epsilon) + ref_output, ref_residual = fused_add_rms_norm_native(x, x_residual, weight, epsilon) + + assert output.shape == x.shape + assert residual.shape == x_residual.shape + assert_close(ir.ops.fused_add_rms_norm, output, ref_output) + assert_close(ir.ops.fused_add_rms_norm, residual, ref_residual) + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32]) +@pytest.mark.parametrize("n_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", COMMON_HIDDEN_SIZES) +@pytest.mark.parametrize("epsilon", [1e-6, 1e-5]) +@pytest.mark.skipif( + not current_platform.is_cuda_alike() and not current_platform.is_xpu(), + reason="Currently only kernels on CUDA, ROCm and XPU", +) +class TestFusedAddRMSNorm: + @classmethod + def setup_class(cls, **kwargs): + torch.set_default_device(current_platform.device_type) + + def test_native_semantics(self, dtype, n_tokens, hidden_size, epsilon): + x, x_residual, weight, eps = ir.ops.fused_add_rms_norm.generate_inputs( + num_tokens=4, hidden_size=8, dtype=dtype, epsilon=epsilon + ) + out, residual_out = fused_add_rms_norm_native(x, x_residual, weight, eps) + + # Check shape, dtype, device + assert out.shape == x.shape + assert out.dtype == x.dtype + assert out.device == x.device + assert residual_out.shape == x_residual.shape + assert residual_out.dtype == x_residual.dtype + assert residual_out.device == x_residual.device + + # Check that residual_out = x + x_residual + expected_residual = (x.float() + x_residual.float()).to(dtype) + torch.testing.assert_close( + residual_out, expected_residual, rtol=1e-3, atol=1e-3 + ) + + # Verify that the output is RMS normalized version of (x + x_residual) + expected_out = rms_norm_native(expected_residual, weight, epsilon) + assert_close( + ir.ops.fused_add_rms_norm, + (out, residual_out), + (expected_out, expected_residual), + ) + + # Check the scaling property of rms norm + out1, _ = fused_add_rms_norm_native( + x, torch.zeros_like(x), weight, epsilon=epsilon + ) + out2, _ = fused_add_rms_norm_native( + x * 2.0, torch.zeros_like(x), weight, epsilon=epsilon + ) + torch.testing.assert_close(out2, out1, rtol=get_default_rtol(out), atol=1e-3) + + # Check behavior with and without weight + weight1 = torch.ones_like(weight) + out3, _ = fused_add_rms_norm_native(x, x_residual, weight1, eps) + out4, _ = fused_add_rms_norm_native(x, x_residual, None, eps) + torch.testing.assert_close(out3, out4) + + @pytest.mark.parametrize("provider", supported_providers(ir.ops.fused_add_rms_norm)) + def test_impls(self, dtype, n_tokens, hidden_size, epsilon, provider): + impl = ir.ops.fused_add_rms_norm.impls[provider] + x, x_residual, weight, eps = ir.ops.fused_add_rms_norm.generate_inputs( + num_tokens=n_tokens, hidden_size=hidden_size, dtype=dtype, epsilon=epsilon + ) + args = (x, x_residual, weight, eps, None) + + if not impl.supports_args(*args): + pytest.skip(f"{provider} does not support args") + + ref_output, ref_residual = fused_add_rms_norm_native(*clone_args(args)) + output, residual = impl.impl_fn(*clone_args(args)) + assert_close(ir.ops.fused_add_rms_norm, output, ref_output) + assert_close(ir.ops.fused_add_rms_norm, residual, ref_residual) + + # check that dispatched call matches direct call + with ir.ops.fused_add_rms_norm.set_priority([provider, "native"]): + out_dispatched, residual_dispatched = ir.ops.fused_add_rms_norm(*args[:4]) + out_direct, residual_direct = impl.impl_fn(*clone_args(args)) + torch.testing.assert_close(out_dispatched, out_direct, rtol=0.0, atol=0.0) + torch.testing.assert_close( + residual_dispatched, residual_direct, rtol=0.0, atol=0.0 + ) + + # none of these support variance_size override + assert not impl.supports_args(x, x_residual, weight, epsilon, 4) + assert not impl.supports_args(x, x_residual, weight, epsilon, variance_size=4) + + # test weight=None behavior + out_no_weight, residual_no_weight = impl.impl_fn( + x.clone(), x_residual.clone(), None, epsilon + ) + out_unit_weight, residual_unit_weight = impl.impl_fn( + x.clone(), x_residual.clone(), torch.ones_like(weight), epsilon + ) + assert_close(ir.ops.fused_add_rms_norm, out_no_weight, out_unit_weight) + assert_close( + ir.ops.fused_add_rms_norm, residual_no_weight, residual_unit_weight + ) + + @pytest.mark.parametrize("provider", ["vllm_c"]) + def test_inplace_semantics(self, dtype, n_tokens, hidden_size, epsilon, provider): + """Test that inplace implementations reuse inputs, + for maybe_inplace overload but not for default overload.""" + impl = ir.ops.fused_add_rms_norm.impls[provider] + if not impl.supported: + pytest.skip(f"{provider} impl not supported on this platform") + + x, x_residual, weight, eps = ir.ops.fused_add_rms_norm.generate_inputs( + num_tokens=n_tokens, hidden_size=hidden_size, dtype=dtype, epsilon=epsilon + ) + + # Test default overload - should NOT modify inputs even with inplace impl + x_default = x.clone() + x_residual_default = x_residual.clone() + x_default_ptr = x_default.data_ptr() + x_residual_default_ptr = x_residual_default.data_ptr() + + with ir.ops.fused_add_rms_norm.set_priority([provider, "native"]): + out_default, residual_default = ir.ops.fused_add_rms_norm( + x_default, x_residual_default, weight, eps + ) + + # Default should NOT be inplace (even with inplace implementation) + assert out_default.data_ptr() != x_default_ptr + assert residual_default.data_ptr() != x_residual_default_ptr + torch.testing.assert_close(x, x_default, rtol=0.0, atol=0.0) + torch.testing.assert_close(x_residual, x_residual_default, rtol=0.0, atol=0.0) + + # Test maybe_inplace overload - should modify inputs with inplace impl + x_inplace = x.clone() + x_residual_inplace = x_residual.clone() + x_inplace_ptr = x_inplace.data_ptr() + x_residual_inplace_ptr = x_residual_inplace.data_ptr() + + with ir.ops.fused_add_rms_norm.set_priority([provider, "native"]): + out_inplace, residual_inplace = ir.ops.fused_add_rms_norm.maybe_inplace( + x_inplace, x_residual_inplace, weight, eps + ) + + # maybe_inplace should be inplace + assert out_inplace.data_ptr() == x_inplace_ptr + assert residual_inplace.data_ptr() == x_residual_inplace_ptr + + # Both should produce same results + torch.testing.assert_close(out_default, out_inplace, atol=0.0, rtol=0.0) + torch.testing.assert_close( + residual_default, residual_inplace, atol=0.0, rtol=0.0 + ) + + @pytest.mark.parametrize("provider", supported_providers(ir.ops.fused_add_rms_norm)) + def test_torch_opcheck(self, dtype, n_tokens, hidden_size, epsilon, provider): + args = ir.ops.fused_add_rms_norm.generate_inputs( + num_tokens=n_tokens, hidden_size=hidden_size, dtype=dtype, epsilon=epsilon + ) + args = args + (None,) # Add variance_size parameter + + # When checking the torch op, we have to set priority and use dispatch + with ir.ops.fused_add_rms_norm.set_priority([provider, "native"]): + torch.library.opcheck(torch.ops.vllm_ir.fused_add_rms_norm.default, args) + + # Only test maybe_inplace with non-inplace implementations + # Inplace implementations return aliases of inputs which is not allowed. + # We break this invariant, but we also convert maybe_inplace to the default + # overload during compilation, so maybe_inplace never reaches Inductor. + if not ir.ops.fused_add_rms_norm.impls[provider].inplace: + torch.library.opcheck( + torch.ops.vllm_ir.fused_add_rms_norm.maybe_inplace, args + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/mamba/__init__.py b/asian_ai/performance/vLLM/tests/kernels/mamba/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/kernels/mamba/cpu/test_cpu_gdn_ops.py b/asian_ai/performance/vLLM/tests/kernels/mamba/cpu/test_cpu_gdn_ops.py new file mode 100644 index 0000000000000000000000000000000000000000..2874c58f88c856a5e956111e757053101732f876 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/mamba/cpu/test_cpu_gdn_ops.py @@ -0,0 +1,538 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import functools + +import pytest +import torch +import torch.nn.functional as F + +import vllm._custom_ops as ops +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +if not current_platform.is_cpu(): + pytest.skip("skipping CPU-only tests", allow_module_level=True) + +set_random_seed(12345) + +NUM_HEADS = [ + (2, 4), + (4, 4), +] +HEAD_DIMS = [ + (32, 32), + (64, 32), +] +CHUNK_SIZE = 64 +CONV_DIM = 128 +CONV_KERNEL = 4 +PREFILL_SEQ_LENS = [ + [1], + [1, 2, 3], + [CHUNK_SIZE - 1], + [CHUNK_SIZE], + [CHUNK_SIZE + 1], + [CHUNK_SIZE - 1, CHUNK_SIZE, CHUNK_SIZE + 1], + [2 * CHUNK_SIZE - 1, 2 * CHUNK_SIZE, 2 * CHUNK_SIZE + 1], + [4 * CHUNK_SIZE + 17], +] +DECODE_BATCH_SIZES = [1, 3, 5] + + +@functools.lru_cache(maxsize=128, typed=False) +def tensor_cache( + elem_num: int, + dtype: torch.dtype, +) -> torch.Tensor: + tensor = torch.rand(elem_num, dtype=dtype) + return tensor + + +def ref_l2norm( + x: torch.Tensor, + dim: int = -1, + eps: float = 1e-5, +) -> torch.Tensor: + inv_norm = torch.rsqrt((x * x).sum(dim=dim, keepdim=True) + eps) + return x * inv_norm + + +def ref_gdn_gating( + A_log: torch.Tensor, + a: torch.Tensor, + b: torch.Tensor, + dt_bias: torch.Tensor, +) -> tuple[torch.Tensor, torch.Tensor]: + softplus_x = F.softplus(a.float() + dt_bias.float(), beta=1.0, threshold=20.0) + g = -torch.exp(A_log.float()) * softplus_x + beta = torch.sigmoid(b.float()).to(dtype=b.dtype) + return g, beta + + +def ref_gated_delta_rule( + query: torch.Tensor, + key: torch.Tensor, + value: torch.Tensor, + a: torch.Tensor, + b: torch.Tensor, + A_log: torch.Tensor, + dt_bias: torch.Tensor, + initial_state: torch.Tensor, + cu_seqlens: torch.Tensor, + use_qk_l2norm_in_kernel: bool = False, +) -> tuple[torch.Tensor, torch.Tensor]: + g, beta = ref_gdn_gating(A_log, a, b, dt_bias) + out = torch.empty_like(value) + final_state = torch.empty_like(initial_state) + + for seq_idx in range(cu_seqlens.numel() - 1): + begin = int(cu_seqlens[seq_idx].item()) + end = int(cu_seqlens[seq_idx + 1].item()) + q_seq = query[:, begin:end] + k_seq = key[:, begin:end] + v_seq = value[:, begin:end] + g_seq = g[begin:end].unsqueeze(0) + beta_seq = beta[begin:end].unsqueeze(0) + initial_dtype = q_seq.dtype + + if use_qk_l2norm_in_kernel: + q_seq = ref_l2norm(q_seq, dim=-1) + k_seq = ref_l2norm(k_seq, dim=-1) + + if q_seq.shape[2] != v_seq.shape[2]: + repeat_factor = v_seq.shape[2] // q_seq.shape[2] + q_seq = q_seq.repeat_interleave(repeat_factor, dim=2) + k_seq = k_seq.repeat_interleave(repeat_factor, dim=2) + + q_seq, k_seq, v_seq, beta_seq, g_seq = [ + x.transpose(1, 2).contiguous().to(torch.float32) + for x in (q_seq, k_seq, v_seq, beta_seq, g_seq) + ] + + batch_size, num_heads, seq_len, head_dim = q_seq.shape + v_head_dim = v_seq.shape[-1] + q_seq = q_seq * (1 / (head_dim**0.5)) + out_seq = torch.empty( + batch_size, + num_heads, + seq_len, + v_head_dim, + dtype=v_seq.dtype, + ) + state = initial_state[seq_idx : seq_idx + 1].to(v_seq) + + for token_idx in range(seq_len): + q_t = q_seq[:, :, token_idx] + k_t = k_seq[:, :, token_idx] + v_t = v_seq[:, :, token_idx] + g_t = g_seq[:, :, token_idx].exp().unsqueeze(-1).unsqueeze(-1) + beta_t = beta_seq[:, :, token_idx].unsqueeze(-1) + + state = state * g_t + kv_mem = (state * k_t.unsqueeze(-2)).sum(dim=-1) + delta = (v_t - kv_mem) * beta_t + state = state + delta.unsqueeze(-1) * k_t.unsqueeze(-2) + out_seq[:, :, token_idx] = (state * q_t.unsqueeze(-2)).sum(dim=-1) + + out[:, begin:end] = out_seq.transpose(1, 2).contiguous().to(initial_dtype) + final_state[seq_idx] = state.squeeze(0) + + return out, final_state + + +def gdn_inputs( + num_tokens: int, + num_heads: tuple[int, int], + head_dims: tuple[int, int], +) -> tuple[torch.Tensor, ...]: + num_qk_heads, num_v_heads = num_heads + head_dim, v_head_dim = head_dims + q_shape = (1, num_tokens, num_qk_heads, head_dim) + q_numel = num_tokens * num_qk_heads * head_dim + q = tensor_cache(q_numel, torch.bfloat16).view(q_shape) + k = tensor_cache(q_numel, torch.bfloat16).view(q_shape) + + v_shape = (1, num_tokens, num_v_heads, v_head_dim) + v = tensor_cache(num_tokens * num_v_heads * v_head_dim, torch.bfloat16).view( + v_shape + ) + + gate_shape = (num_tokens, num_v_heads) + gate_numel = num_tokens * num_v_heads + a = tensor_cache(gate_numel, torch.bfloat16).view(gate_shape) + b = tensor_cache(gate_numel, torch.bfloat16).view(gate_shape) + A_log = tensor_cache(num_v_heads, torch.float32) + dt_bias = tensor_cache(num_v_heads, torch.bfloat16) + return q, k, v, a, b, A_log, dt_bias + + +@pytest.mark.parametrize("num_tokens", [1, 9]) +@pytest.mark.parametrize("num_v_heads", [4, 8]) +@torch.inference_mode() +def test_fused_gdn_gating_cpu( + num_tokens: int, + num_v_heads: int, +) -> None: + gate_shape = (num_tokens, num_v_heads) + gate_numel = num_tokens * num_v_heads + a = tensor_cache(gate_numel, torch.bfloat16).view(gate_shape) + b = tensor_cache(gate_numel, torch.bfloat16).view(gate_shape) + A_log = tensor_cache(num_v_heads, torch.float32) + dt_bias = tensor_cache(num_v_heads, torch.bfloat16) + + g_ref, beta_ref = ref_gdn_gating(A_log, a, b, dt_bias) + g, beta = ops.fused_gdn_gating_cpu(A_log, a, b, dt_bias) + + torch.testing.assert_close(g, g_ref.unsqueeze(0), atol=1e-4, rtol=1e-4) + torch.testing.assert_close( + beta.float(), beta_ref.unsqueeze(0).float(), atol=5e-3, rtol=5e-3 + ) + + +# decode path +@pytest.mark.parametrize("batch_size", DECODE_BATCH_SIZES) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_dims", HEAD_DIMS) +@torch.inference_mode() +def test_fused_sigmoid_gating_delta_rule_update_cpu( + batch_size: int, + num_heads: tuple[int, int], + head_dims: tuple[int, int], +) -> None: + q, k, v, a, b, A_log, dt_bias = gdn_inputs( + num_tokens=batch_size, + num_heads=num_heads, + head_dims=head_dims, + ) + _, num_v_heads = num_heads + head_dim, v_head_dim = head_dims + state_indices = torch.arange(batch_size, dtype=torch.int32) + cu_seqlens = torch.arange(batch_size + 1, dtype=torch.int32) + state_shape = (batch_size, num_v_heads, head_dim, v_head_dim) + state = tensor_cache( + batch_size * num_v_heads * head_dim * v_head_dim, torch.float32 + ).view(state_shape) + state_ref = state[state_indices].transpose(-1, -2).contiguous() + + out_ref, final_state_ref = ref_gated_delta_rule( + query=q, + key=k, + value=v, + a=a, + b=b, + A_log=A_log, + dt_bias=dt_bias, + initial_state=state_ref, + cu_seqlens=cu_seqlens, + use_qk_l2norm_in_kernel=True, + ) + out_ref = out_ref.transpose(0, 1).contiguous() + + state_out = state.clone() + out = ops.fused_sigmoid_gating_delta_rule_update_cpu( + A_log=A_log, + dt_bias=dt_bias, + q=q, + k=k, + v=v, + a=a, + b=b, + initial_state_source=state_out, + initial_state_indices=state_indices, + cu_seqlens=cu_seqlens, + use_qk_l2norm_in_kernel=True, + ) + + torch.testing.assert_close(out, out_ref, atol=1e-2, rtol=1e-2) + torch.testing.assert_close( + state_out[state_indices].transpose(-1, -2), + final_state_ref, + atol=1e-2, + rtol=1e-2, + ) + + +# prefill path +@pytest.mark.parametrize("seq_lens", PREFILL_SEQ_LENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_dims", HEAD_DIMS) +@torch.inference_mode() +def test_chunk_gated_delta_rule_cpu( + seq_lens: list[int], + num_heads: tuple[int, int], + head_dims: tuple[int, int], +) -> None: + total_tokens = sum(seq_lens) + q, k, v, a, b, A_log, dt_bias = gdn_inputs( + num_tokens=total_tokens, + num_heads=num_heads, + head_dims=head_dims, + ) + _, num_v_heads = num_heads + head_dim, v_head_dim = head_dims + cu_seqlens = torch.tensor( + [0, *torch.tensor(seq_lens).cumsum(0).tolist()], dtype=torch.int32 + ) + initial_state_shape = (len(seq_lens), num_v_heads, head_dim, v_head_dim) + initial_state = tensor_cache( + len(seq_lens) * num_v_heads * head_dim * v_head_dim, torch.float32 + ).view(initial_state_shape) + initial_state_ref = initial_state.transpose(-1, -2).contiguous() + + out_ref, final_state_ref = ref_gated_delta_rule( + query=q, + key=k, + value=v, + a=a, + b=b, + A_log=A_log, + dt_bias=dt_bias, + initial_state=initial_state_ref, + cu_seqlens=cu_seqlens, + use_qk_l2norm_in_kernel=True, + ) + + g, beta = ref_gdn_gating(A_log, a, b, dt_bias) + out, final_state = ops.chunk_gated_delta_rule_cpu( + query=q, + key=k, + value=v, + g=g.unsqueeze(0), + beta=beta.unsqueeze(0), + initial_state=initial_state, + output_final_state=True, + cu_seqlens=cu_seqlens, + head_first=False, + use_qk_l2norm_in_kernel=True, + ) + + torch.testing.assert_close(out, out_ref, atol=1e-2, rtol=1e-2) + torch.testing.assert_close( + final_state.transpose(-1, -2), + final_state_ref, + atol=1e-2, + rtol=1e-2, + ) + + +# (total_tokens, split) pairs mimicking where chunked prefill breaks a sequence +# across two scheduler steps: chunk-aligned and non-aligned splits. +TWO_CALL_SPLITS = [ + (2 * CHUNK_SIZE, CHUNK_SIZE), + (2 * CHUNK_SIZE + 17, CHUNK_SIZE), + (2 * CHUNK_SIZE + 17, CHUNK_SIZE + 9), + (4 * CHUNK_SIZE + 17, 2 * CHUNK_SIZE), + (3 * CHUNK_SIZE, CHUNK_SIZE + 1), +] + + +@pytest.mark.parametrize("total_tokens, split", TWO_CALL_SPLITS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("head_dims", HEAD_DIMS) +@torch.inference_mode() +def test_chunk_gated_delta_rule_cpu_two_call_split( + total_tokens: int, + split: int, + num_heads: tuple[int, int], + head_dims: tuple[int, int], +) -> None: + """A prefill split into two calls (the second seeded with the first's + ``final_state`` and a rebased ``cu_seqlens``) must match the single-call + result, mimicking the cross-scheduler-step handoff in + ``cpu_gdn_attention_core``. + """ + q, k, v, a, b, A_log, dt_bias = gdn_inputs( + num_tokens=total_tokens, + num_heads=num_heads, + head_dims=head_dims, + ) + _, num_v_heads = num_heads + head_dim, v_head_dim = head_dims + + g, beta = ref_gdn_gating(A_log, a, b, dt_bias) + g = g.unsqueeze(0) # [1, T, HV] + beta = beta.unsqueeze(0) + + zero_state = torch.zeros(1, num_v_heads, head_dim, v_head_dim, dtype=torch.float32) + + # Reference: whole sequence in one call, no initial state. + out_full, final_full = ops.chunk_gated_delta_rule_cpu( + query=q, + key=k, + value=v, + g=g, + beta=beta, + initial_state=zero_state, + output_final_state=True, + cu_seqlens=torch.tensor([0, total_tokens], dtype=torch.int32), + head_first=False, + use_qk_l2norm_in_kernel=True, + ) + + # Call 1: tokens [0:split], no initial state, capture final state. + out1, state1 = ops.chunk_gated_delta_rule_cpu( + query=q[:, :split], + key=k[:, :split], + value=v[:, :split], + g=g[:, :split], + beta=beta[:, :split], + initial_state=zero_state, + output_final_state=True, + cu_seqlens=torch.tensor([0, split], dtype=torch.int32), + head_first=False, + use_qk_l2norm_in_kernel=True, + ) + # Call 2: tokens [split:T] seeded with call 1's final state and a cu_seqlens + # rebased to start at 0, as cpu_gdn_attention_core continues a prefill chunk. + tail = total_tokens - split + out2, state2 = ops.chunk_gated_delta_rule_cpu( + query=q[:, split:], + key=k[:, split:], + value=v[:, split:], + g=g[:, split:], + beta=beta[:, split:], + initial_state=state1.to(torch.float32), + output_final_state=True, + cu_seqlens=torch.tensor([0, tail], dtype=torch.int32), + head_first=False, + use_qk_l2norm_in_kernel=True, + ) + + out_split = torch.cat([out1, out2], dim=1) + + # State must be near-exact; output allows a looser bound for the bf16 round-trip. + torch.testing.assert_close(state2, final_full, atol=1e-3, rtol=1e-3) + torch.testing.assert_close(out_split, out_full, atol=2e-2, rtol=2e-2) + + +def _conv_inputs(total_tokens: int): + x = tensor_cache(total_tokens * CONV_DIM, torch.bfloat16).view( + total_tokens, CONV_DIM + ) + weight = tensor_cache(CONV_DIM * CONV_KERNEL, torch.bfloat16).view( + CONV_DIM, CONV_KERNEL + ) + bias = tensor_cache(CONV_DIM, torch.bfloat16) + return x, weight, bias + + +@pytest.mark.parametrize("total_tokens, split", TWO_CALL_SPLITS) +@torch.inference_mode() +def test_causal_conv1d_torch_two_call_split(total_tokens: int, split: int) -> None: + """Non-AMX conv-state handoff: a two-call split (the second seeded via + ``has_initial_state=True`` from the conv_states the first wrote back) must + match the single-call result. + """ + from vllm.model_executor.layers.mamba.ops.cpu.causal_conv1d import ( + causal_conv1d_torch, + ) + + x, weight, bias = _conv_inputs(total_tokens) + state_len = CONV_KERNEL - 1 + # [num_slots, conv_dim, state_len]; slot 0 used here. + conv_states_full = torch.zeros(1, CONV_DIM, state_len, dtype=x.dtype) + conv_states_split = torch.zeros(1, CONV_DIM, state_len, dtype=x.dtype) + + # x is [conv_dim, T] for causal_conv1d_torch. + xt = x.transpose(0, 1).contiguous() + + out_full = causal_conv1d_torch( + x=xt, + weight=weight, + bias=bias, + conv_states=conv_states_full, + query_start_loc=torch.tensor([0, total_tokens], dtype=torch.int32), + cache_indices=torch.tensor([0], dtype=torch.int32), + has_initial_state=torch.tensor([False]), + activation="silu", + ) + + out1 = causal_conv1d_torch( + x=xt[:, :split], + weight=weight, + bias=bias, + conv_states=conv_states_split, + query_start_loc=torch.tensor([0, split], dtype=torch.int32), + cache_indices=torch.tensor([0], dtype=torch.int32), + has_initial_state=torch.tensor([False]), + activation="silu", + ) + out2 = causal_conv1d_torch( + x=xt[:, split:], + weight=weight, + bias=bias, + conv_states=conv_states_split, + query_start_loc=torch.tensor([0, total_tokens - split], dtype=torch.int32), + cache_indices=torch.tensor([0], dtype=torch.int32), + has_initial_state=torch.tensor([True]), + activation="silu", + ) + out_split = torch.cat([out1, out2], dim=1) + + torch.testing.assert_close(out_split, out_full, atol=1e-2, rtol=1e-2) + + +@pytest.mark.skipif( + not torch.cpu._is_amx_tile_supported(), + reason="causal_conv1d_fwd_cpu requires AMX/AVX512", +) +@pytest.mark.parametrize("total_tokens, split", TWO_CALL_SPLITS) +@torch.inference_mode() +def test_causal_conv1d_fwd_cpu_two_call_split(total_tokens: int, split: int) -> None: + """AMX prefill conv op must honor ``has_initial_state`` so a two-call split + matches the single-call result. + + Regression test for ``causal_conv1d_fwd_varlen_kernel_impl`` (``conv.cpp``) + ignoring the carried conv state on continued chunks. + """ + state_len = CONV_KERNEL - 1 + x, weight, bias = _conv_inputs(total_tokens) + + def amx(x_seg, conv_states, has_init): + seq = x_seg.shape[0] + return ops.causal_conv1d_fwd_cpu( + x=x_seg.transpose(0, 1), # [dim, seq]; stride(-2)==1 (view of [seq,dim]) + weight=weight, + bias=bias, + conv_states=conv_states, + query_start_loc=torch.tensor([0, seq], dtype=torch.int32), + cache_indices=torch.tensor([0], dtype=torch.int32), + has_initial_state=torch.tensor([has_init]), + silu_activation=True, + is_vnni=False, + ).contiguous() + + # conv_state layout passed by the AMX branch: [num_slots, dim, state_len]. + cs_full = torch.zeros(1, CONV_DIM, state_len, dtype=x.dtype) + out_full = amx(x, cs_full, False) + + cs_split = torch.zeros(1, CONV_DIM, state_len, dtype=x.dtype) + out1 = amx(x[:split], cs_split, False) + out2 = amx(x[split:], cs_split, True) + out_split = torch.cat([out1, out2], dim=1) + + torch.testing.assert_close(out_split, out_full, atol=1e-2, rtol=1e-2) + + +@torch.inference_mode() +def test_batch_memcpy_cpu_fallback() -> None: + """The ctypes batch_memcpy fallback (used when triton-cpu is absent) must + copy each src into its dst, validating the (src_ptrs, dst_ptrs, sizes) + argument order against ctypes.memmove(dst, src, size). + """ + from vllm.utils.cpu_triton_utils import batch_memcpy_kernel + + # Varied byte sizes, including a non-power-of-two run. + sizes_bytes = [256, 1024, 17 * 4, 4096] + srcs = [torch.rand(n // 4, dtype=torch.float32) for n in sizes_bytes] + dsts = [torch.zeros_like(s) for s in srcs] + + src_ptrs = torch.tensor([s.data_ptr() for s in srcs], dtype=torch.uint64) + dst_ptrs = torch.tensor([d.data_ptr() for d in dsts], dtype=torch.uint64) + sizes = torch.tensor(sizes_bytes, dtype=torch.int32) + + batch_memcpy_kernel[(len(srcs),)](src_ptrs, dst_ptrs, sizes, BLOCK_SIZE=1024) + + for src, dst in zip(srcs, dsts): + torch.testing.assert_close(dst, src) diff --git a/asian_ai/performance/vLLM/tests/kernels/mamba/test_causal_conv1d.py b/asian_ai/performance/vLLM/tests/kernels/mamba/test_causal_conv1d.py new file mode 100644 index 0000000000000000000000000000000000000000..1c0bc3393ebddd114c03f9312f76b5a3d50c23d3 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/mamba/test_causal_conv1d.py @@ -0,0 +1,389 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import pytest +import torch +import torch.nn.functional as F +from einops import rearrange + +from vllm.model_executor.layers.mamba.ops.causal_conv1d import ( + causal_conv1d_fn, + causal_conv1d_update, +) +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.attention.backends.utils import NULL_BLOCK_ID + +DEVICE = current_platform.device_type + +pytestmark = pytest.mark.skipif( + not (current_platform.is_cuda_alike() or current_platform.is_xpu()), + reason="causal_conv1d Triton kernels require CUDA-alike or XPU", +) + + +def causal_conv1d_ref( + x: torch.Tensor, + weight: torch.Tensor, + bias: torch.Tensor | None = None, + initial_states: torch.Tensor | None = None, + return_final_states: bool = False, + final_states_out: torch.Tensor | None = None, + activation: str | None = "silu", +): + """ + x: (batch, dim, seqlen) + weight: (dim, width) + bias: (dim,) + initial_states: (batch, dim, width - 1) + final_states_out: (batch, dim, width - 1) + + out: (batch, dim, seqlen) + """ + if activation not in [None, "silu", "swish"]: + raise NotImplementedError("activation must be None, silu, or swish") + dtype_in = x.dtype + x = x.to(weight.dtype) + seqlen = x.shape[-1] + dim, width = weight.shape + if initial_states is None: + out = F.conv1d(x, weight.unsqueeze(1), bias, padding=width - 1, groups=dim) + else: + x = torch.cat([initial_states, x], dim=-1) + out = F.conv1d(x, weight.unsqueeze(1), bias, padding=0, groups=dim) + out = out[..., :seqlen] + if return_final_states: + final_states = F.pad(x, (width - 1 - x.shape[-1], 0)).to( + dtype_in + ) # (batch, dim, width - 1) + if final_states_out is not None: + final_states_out.copy_(final_states) + else: + final_states_out = final_states + out = (out if activation is None else F.silu(out)).to(dtype=dtype_in) + return (out, None) if not return_final_states else (out, final_states_out) + + +def causal_conv1d_update_ref( + x, conv_state, weight, bias=None, activation=None, cache_seqlens=None +): + """ + x: (batch, dim) or (batch, dim, seqlen) + conv_state: (batch, dim, state_len), where state_len >= width - 1 + weight: (dim, width) + bias: (dim,) + cache_seqlens: (batch,), dtype int32. + If not None, the conv_state is treated as a circular buffer. + The conv_state will be updated by copying x to the + conv_state starting at the index + @cache_seqlens % state_len before performing the convolution. + + out: (batch, dim) or (batch, dim, seqlen) + """ + if activation not in [None, "silu", "swish"]: + raise NotImplementedError("activation must be None, silu, or swish") + dtype_in = x.dtype + unsqueeze = x.dim() == 2 + if unsqueeze: + x = x.unsqueeze(-1) + batch, dim, seqlen = x.shape + width = weight.shape[1] + state_len = conv_state.shape[-1] + assert conv_state.shape == (batch, dim, state_len) + assert weight.shape == (dim, width) + if cache_seqlens is None: + x_new = torch.cat([conv_state, x], dim=-1).to( + weight.dtype + ) # (batch, dim, state_len + seqlen) + conv_state.copy_(x_new[:, :, -state_len:]) + else: + width_idx = torch.arange( + -(width - 1), 0, dtype=torch.long, device=x.device + ).unsqueeze(0) + cache_seqlens.unsqueeze(1) + width_idx = ( + torch.remainder(width_idx, state_len).unsqueeze(1).expand(-1, dim, -1) + ) + x_new = torch.cat([conv_state.gather(2, width_idx), x], dim=-1).to(weight.dtype) + copy_idx = torch.arange(seqlen, dtype=torch.long, device=x.device).unsqueeze( + 0 + ) + cache_seqlens.unsqueeze(1) + copy_idx = torch.remainder(copy_idx, state_len).unsqueeze(1).expand(-1, dim, -1) + conv_state.scatter_(2, copy_idx, x) + out = F.conv1d(x_new, weight.unsqueeze(1), bias, padding=0, groups=dim)[ + :, :, -seqlen: + ] + if unsqueeze: + out = out.squeeze(-1) + return (out if activation is None else F.silu(out)).to(dtype=dtype_in) + + +@pytest.mark.parametrize("itype", [torch.bfloat16, torch.float]) +@pytest.mark.parametrize("silu_activation", [True]) +@pytest.mark.parametrize("has_bias", [True]) +def causal_conv1d_opcheck_fn( + x: torch.Tensor, + weight: torch.Tensor, + bias: torch.Tensor | None = None, + cu_seq_len: torch.Tensor | None = None, + cache_indices: torch.Tensor | None = None, + has_initial_state: torch.Tensor | None = None, + conv_states: torch.Tensor | None = None, + activation: str | None = "silu", + null_block_id: int = NULL_BLOCK_ID, +): + """ + x: (batch, dim, seqlen) + weight: (dim, width) + bias: (dim,) + seq_idx: (batch, seqlen) + initial_states: (batch, dim, width - 1) + final_states_out: (batch, dim, width - 1), to be written to + activation: either None or "silu" or "swish" + + out: (batch, dim, seqlen) + """ + if activation not in [None, "silu", "swish"]: + raise NotImplementedError("activation must be None, silu, or swish") + if x.stride(-1) != 1: + x = x.contiguous() + bias = bias.contiguous() if bias is not None else None + + +@pytest.mark.parametrize("itype", [torch.bfloat16]) +@pytest.mark.parametrize("silu_activation", [False, True]) +@pytest.mark.parametrize("has_bias", [False, True]) +@pytest.mark.parametrize("seqlen", [1]) +@pytest.mark.parametrize("width", [4]) +@pytest.mark.parametrize("dim", [2048, 2048 + 16, 4096]) +def test_causal_conv1d_update(dim, width, seqlen, has_bias, silu_activation, itype): + device = DEVICE + rtol, atol = (3e-4, 1e-3) if itype == torch.float32 else (3e-3, 5e-3) + if itype == torch.bfloat16: + rtol, atol = 1e-2, 5e-2 + # set seed + set_random_seed(0) + batch = 2 + x = torch.randn(batch, dim, seqlen, device=device, dtype=itype) + x_ref = x.clone() + # +1 entry to reserve index 0 as null block + conv_state = torch.randn(batch + 1, dim, width - 1, device=device, dtype=itype) + + weight = torch.randn(dim, width, device=device, dtype=itype) + bias = torch.randn(dim, device=device, dtype=itype) if has_bias else None + # Start indices from 1, skipping null block at index 0 + conv_state_indices = torch.arange(1, batch + 1, dtype=torch.int32, device=device) + conv_state_ref = conv_state[conv_state_indices].detach().clone() + activation = None if not silu_activation else "silu" + + out = causal_conv1d_update( + x, + conv_state, + weight, + bias, + activation=activation, + conv_state_indices=conv_state_indices, + ) + out_ref = causal_conv1d_update_ref( + x_ref, conv_state_ref, weight, bias, activation=activation + ) + + assert torch.equal(conv_state[conv_state_indices], conv_state_ref) + assert torch.allclose(out, out_ref, rtol=rtol, atol=atol) + + +@pytest.mark.parametrize("itype", [torch.float32, torch.bfloat16]) +@pytest.mark.parametrize("silu_activation", [False, True]) +@pytest.mark.parametrize("has_bias", [False, True]) +@pytest.mark.parametrize("seqlen", [1, 3]) +@pytest.mark.parametrize("width", [3, 4]) +@pytest.mark.parametrize("dim", [2048 + 16, 4096]) +# tests correctness in case subset of the sequences are padded +@pytest.mark.parametrize("with_padding", [True, False]) +@pytest.mark.parametrize("batch_size", [3]) +def test_causal_conv1d_update_with_batch_gather( + batch_size, with_padding, dim, width, seqlen, has_bias, silu_activation, itype +): + device = DEVICE + rtol, atol = (3e-4, 1e-3) if itype == torch.float32 else (3e-3, 5e-3) + if itype == torch.bfloat16: + rtol, atol = 1e-2, 5e-2 + + # set seed + set_random_seed(0) + + padding = 5 if with_padding else 0 + padded_batch_size = batch_size + padding + # total_entries = number of cache line + total_entries = 10 * batch_size + + # x will be (batch, dim, seqlen) with contiguous along dim-axis + x = torch.randn( + padded_batch_size, seqlen, dim, device=device, dtype=itype + ).transpose(1, 2) + + x_ref = x.clone() + + # +1 to exclude index 0 (null block) + conv_state_indices = (torch.randperm(total_entries - 1)[:batch_size] + 1).to( + dtype=torch.int32, device=device + ) + unused_states_bool = torch.ones(total_entries, dtype=torch.bool, device=device) + unused_states_bool[conv_state_indices] = False + padded_state_indices = torch.concat( + [ + conv_state_indices, + torch.as_tensor( + [NULL_BLOCK_ID] * padding, dtype=torch.int32, device=device + ), + ], + dim=0, + ) + + # conv_state will be (cache_lines, dim, state_len) + # with contiguous along dim-axis + conv_state = torch.randn( + total_entries, width - 1, dim, device=device, dtype=itype + ).transpose(1, 2) + + conv_state_for_padding_test = conv_state.clone() + + weight = torch.randn(dim, width, device=device, dtype=itype) + bias = torch.randn(dim, device=device, dtype=itype) if has_bias else None + conv_state_ref = conv_state[conv_state_indices, :].detach().clone() + activation = None if not silu_activation else "silu" + + out = causal_conv1d_update( + x, + conv_state, + weight, + bias, + activation=activation, + conv_state_indices=padded_state_indices, + ) + out_ref = causal_conv1d_update_ref( + x_ref[:batch_size], conv_state_ref, weight, bias, activation=activation + ) + + assert torch.equal(conv_state[conv_state_indices, :], conv_state_ref) + assert torch.equal( + conv_state[unused_states_bool], conv_state_for_padding_test[unused_states_bool] + ) + assert torch.allclose(out[:batch_size], out_ref, rtol=rtol, atol=atol) + + +@pytest.mark.parametrize("itype", [torch.bfloat16]) +@pytest.mark.parametrize("silu_activation", [True]) +@pytest.mark.parametrize("has_bias", [True]) +@pytest.mark.parametrize("width", [4]) +@pytest.mark.parametrize("seqlen", [8, 249, 4096]) +@pytest.mark.parametrize("dim", [64, 4096]) +@pytest.mark.parametrize("with_padding", [True, False]) +@pytest.mark.parametrize("batch", [4, 10]) +def test_causal_conv1d_varlen( + batch, with_padding, dim, seqlen, width, has_bias, silu_activation, itype +): + device = DEVICE + torch.accelerator.empty_cache() + rtol, atol = (3e-4, 1e-3) if itype == torch.float32 else (3e-3, 5e-3) + if itype == torch.bfloat16: + rtol, atol = 1e-2, 5e-2 + # set seed + set_random_seed(0) + seqlens = [] + batch_size = batch + padding = 3 if with_padding else 0 + padded_batch_size = batch_size + padding + nsplits = padded_batch_size - 1 + + eos_pos = torch.randperm(seqlen - 1)[:nsplits].sort().values + + seqlens.append( + torch.diff( + torch.cat([torch.tensor([-1]), eos_pos, torch.tensor([seqlen - 1])]) + ).tolist() + ) + assert sum(seqlens[-1]) == seqlen + assert all(s > 0 for s in seqlens[-1]) + + total_entries = batch_size * 10 + cumsum = torch.cumsum(torch.tensor(seqlens[0]), dim=0).to(torch.int32) + cumsum = torch.concat([torch.tensor([0], dtype=torch.int32), cumsum], dim=0) + x = rearrange( + torch.randn(1, seqlen, 4096 + dim + 64, device=device, dtype=itype), + "b s d -> b d s", + )[:, 4096 : 4096 + dim, :] + + weight = torch.randn(dim, width, device=device, dtype=itype) + + bias = torch.randn(dim, device=device, dtype=itype) if has_bias else None + x_ref = x.clone() + weight_ref = weight.clone() + bias_ref = bias.clone() if bias is not None else None + activation = None if not silu_activation else "silu" + final_states = torch.randn( + total_entries, width - 1, dim, device=x.device, dtype=x.dtype + ).transpose(1, 2) + final_states_ref = final_states.clone() + has_initial_states = torch.randint( + 0, 2, (cumsum.shape[0] - 1,), dtype=torch.bool, device=x.device + ) + # +1 to exclude index 0 (null block) + state_indices = ( + torch.randperm(total_entries - 1, dtype=torch.int32, device=x.device)[ + :batch_size + ] + + 1 + ) + padded_state_indices = torch.concat( + [ + state_indices, + torch.as_tensor( + [NULL_BLOCK_ID] * padding, dtype=torch.int32, device=device + ), + ], + dim=-1, + ) + out = causal_conv1d_fn( + x.squeeze(0), + weight, + bias=bias, + conv_states=final_states, + query_start_loc=cumsum.to(device), + cache_indices=padded_state_indices, + has_initial_state=has_initial_states, + activation=activation, + ) + + out_ref = [] + out_ref_b = [] + + splits = [torch.split(var, seqlens[0], dim=-1) for var in (x_ref)] + for i in range(len(seqlens[0])): + x_s = [v[i].unsqueeze(0) for v in splits][0] + if padded_state_indices[i] == NULL_BLOCK_ID: + continue + out_ref_b.append( + causal_conv1d_ref( + x_s, + weight_ref, + bias_ref, + activation=activation, + return_final_states=True, + final_states_out=final_states_ref[padded_state_indices[i]].unsqueeze(0), + initial_states=final_states_ref[padded_state_indices[i]].unsqueeze(0) + if has_initial_states[i] + else None, + ) + ) + out_ref.append(torch.cat([t[0] for t in out_ref_b], dim=2)) + out_ref_tensor = torch.cat(out_ref, dim=0) + + assert torch.allclose( + final_states[state_indices], + final_states_ref[state_indices], + rtol=rtol, + atol=atol, + ) + unpadded_out = out[:, : out_ref_tensor.shape[-1]] + assert torch.allclose(unpadded_out, out_ref_tensor, rtol=rtol, atol=atol) diff --git a/asian_ai/performance/vLLM/tests/kernels/mamba/test_gdn_forward_core_split.py b/asian_ai/performance/vLLM/tests/kernels/mamba/test_gdn_forward_core_split.py new file mode 100644 index 0000000000000000000000000000000000000000..61f9aac28e2ea9bc8513c86798ebdf35d9e85fab --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/mamba/test_gdn_forward_core_split.py @@ -0,0 +1,296 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Integration test for the non-spec decode split in +``GatedDeltaNet._forward_core``. + +On a pure non-spec batch that mixes prefills with 1-token decodes, the layer +peels the decodes (the contiguous decode-first front slice) off to +``fused_sigmoid_gating_delta_rule_update`` -- the same recurrent update kernel +the spec-decode path uses -- and runs only the prefill tail through +``chunk_gated_delta_rule``. This must produce the same core-attention output and +the same ssm-state pool update as running *everything* through +``chunk_gated_delta_rule`` (the previous behavior). + +Both paths are exercised through the REAL ``_forward_core``: + +* ``meta_split`` is built by the real ``GDNAttentionMetadataBuilder`` for a + mixed batch, so ``num_decodes > 0`` triggers the peel (and the builder rebases + ``chunk_indices``/``chunk_offsets`` to the prefill-only tail). +* ``meta_unified`` is the same metadata with the decodes reclassified as + prefills and full-batch chunk metadata, which forces ``_forward_core`` through + the existing chunk-only path on identical inputs (the conv is unified over all + non-spec tokens in both paths, so it cancels out and only the recurrent split + is compared). + +The Triton/FLA chunk backend is forced so the prefill-only ``chunk_indices`` +must stay consistent with the rebased ``cu_seqlens`` (a stringent, backend +portable check of the split wiring). +""" + +from __future__ import annotations + +import dataclasses +import types +from unittest.mock import patch + +import pytest +import torch + +from vllm.platforms import current_platform + +if not ( + current_platform.is_cuda() and current_platform.is_device_capability_family(100) +): + pytest.skip( + reason="GDN _forward_core split test uses the CuteDSL prefill backend " + "(requires CUDA SM10x).", + allow_module_level=True, + ) + +from tests.v1.attention.utils import ( # noqa: E402 + BatchSpec, + create_common_attn_metadata, + create_vllm_config, +) +from vllm.config import set_current_vllm_config # noqa: E402 +from vllm.model_executor.layers.fla.ops.index import ( # noqa: E402 + prepare_chunk_indices, + prepare_chunk_offsets, +) +from vllm.model_executor.layers.fla.ops.utils import FLA_CHUNK_SIZE # noqa: E402 +from vllm.model_executor.layers.mamba.gdn import qwen_gdn_linear_attn # noqa: E402 +from vllm.model_executor.layers.mamba.gdn.qwen_gdn_linear_attn import ( # noqa: E402 + ChunkGatedDeltaRule, + QwenGatedDeltaNetAttention, +) +from vllm.model_executor.layers.mamba.mamba_utils import ( # noqa: E402 + MambaStateShapeCalculator, +) +from vllm.v1.attention.backends.gdn_attn import ( # noqa: E402 + GDNAttentionMetadataBuilder, +) +from vllm.v1.kv_cache_interface import MambaSpec # noqa: E402 + +# Small GDN dims; head_k_dim/head_v_dim=128 keeps the chunk/update kernels happy. +H = 4 # num key heads +HV = 8 # num value heads +K = 128 # head_k_dim +V = 128 # head_v_dim +CONV_KERNEL = 4 +KEY_DIM = H * K +VALUE_DIM = HV * V +CONV_DIM = 2 * KEY_DIM + VALUE_DIM +BLOCK_SIZE = 16 +PREFIX = "model.layers.0.linear_attn" + + +def _make_vllm_config(): + # A small, ungated GDN model whose config is cached locally; only the config + # (scheduler/cache/compilation/hf) is used here, never the weights. Inject + # linear_key_head_dim=128 and request the CuteDSL prefill backend -- the + # supported GDN chunk kernel on Blackwell (the Triton/FLA chunk kernel is + # unsupported on SM10x). CuteDSL consumes chunk_indices/chunk_offsets, so + # this also exercises the prefill-only chunk-metadata wiring. + cfg = create_vllm_config( + model_name="Qwen/Qwen3.5-0.8B", + block_size=BLOCK_SIZE, + hf_config_override={"linear_key_head_dim": K}, + ) + cfg.additional_config = {"gdn_prefill_backend": "cutedsl"} + return cfg + + +def _build_layer( + vllm_config, conv_state, ssm_state, A_log, dt_bias, conv_weight, conv_bias +): + """A minimal object that runs the real ``_forward_core`` bound to it.""" + layer = types.SimpleNamespace() + layer.prefix = PREFIX + layer.enable_packed_recurrent_decode = False + layer.tp_size = 1 + layer.num_k_heads = H + layer.num_v_heads = HV + layer.head_k_dim = K + layer.head_v_dim = V + layer.key_dim = KEY_DIM + layer.value_dim = VALUE_DIM + layer.activation = "silu" + layer.A_log = A_log + layer.dt_bias = dt_bias + layer.conv1d = types.SimpleNamespace(weight=conv_weight, bias=conv_bias) + layer.kv_cache = (conv_state, ssm_state) + with set_current_vllm_config(vllm_config): + layer.chunk_gated_delta_rule = ChunkGatedDeltaRule() + for name in ( + "rearrange_mixed_qkv", + "_forward_core", + ): + setattr( + layer, + name, + types.MethodType(getattr(QwenGatedDeltaNetAttention, name), layer), + ) + return layer + + +def _run_forward_core(layer, meta, mixed_qkv, b, a, num_tokens): + core_attn_out = torch.zeros( + num_tokens, HV, V, dtype=mixed_qkv.dtype, device=mixed_qkv.device + ) + ctx = types.SimpleNamespace(attn_metadata={PREFIX: meta}) + with patch.object(qwen_gdn_linear_attn, "get_forward_context", return_value=ctx): + layer._forward_core( + mixed_qkv=mixed_qkv.clone(), + b=b.clone(), + a=a.clone(), + core_attn_out=core_attn_out, + ) + return core_attn_out + + +@pytest.mark.parametrize("state_dtype", [torch.bfloat16, torch.float32]) +@pytest.mark.parametrize("num_decodes,prefill_lens", [(3, [512, 300]), (4, [64, 5])]) +@pytest.mark.parametrize("fresh_prefill", [False, True]) +def test_forward_core_split_matches_unified( + state_dtype: torch.dtype, + num_decodes: int, + prefill_lens: list[int], + fresh_prefill: bool, +) -> None: + torch.manual_seed(0) + device = torch.device("cuda") + vllm_config = _make_vllm_config() + + # Decode-first batch: D 1-token decodes (with context), then the prefills. + decode_seq_lens = [64] * num_decodes + prefill_seq_lens = [ + pl if (fresh_prefill and i == 0) else pl + 37 + for i, pl in enumerate(prefill_lens) + ] + seq_lens = decode_seq_lens + prefill_seq_lens + query_lens = [1] * num_decodes + list(prefill_lens) + batch = BatchSpec(seq_lens=seq_lens, query_lens=query_lens) + + builder = GDNAttentionMetadataBuilder( + kv_cache_spec=MambaSpec( + block_size=BLOCK_SIZE, shapes=((16, 64),), dtypes=(torch.float16,) + ), + layer_names=[PREFIX], + vllm_config=vllm_config, + device=device, + ) + common = create_common_attn_metadata( + batch, BLOCK_SIZE, device, arange_block_indices=True + ) + with set_current_vllm_config(vllm_config): + meta_split = builder.build(common_prefix_len=0, common_attn_metadata=common) + + assert meta_split.spec_sequence_masks is None + assert meta_split.num_decodes == num_decodes + assert meta_split.num_prefills == len(prefill_lens) + assert meta_split.num_decode_tokens == num_decodes + assert builder.gdn_prefill_backend == "cutedsl" + + num_tokens = sum(query_lens) + + # Full-batch chunk metadata for the unified reference path, built the same + # way the builder would for a non-split batch (backend-matched). + cu_full = meta_split.non_spec_query_start_loc + if builder.gdn_prefill_backend == "cutedsl": + from vllm.model_executor.layers.mamba.ops.gdn_chunk_cutedsl import ( + prepare_metadata_cutedsl, + ) + + full_ci, full_co = prepare_metadata_cutedsl( + cu_full, int(cu_full[-1].item()), FLA_CHUNK_SIZE + ) + else: + cu_full_cpu = cu_full.cpu() + full_ci = prepare_chunk_indices(cu_full_cpu, FLA_CHUNK_SIZE).to(device) + full_co = prepare_chunk_offsets(cu_full_cpu, FLA_CHUNK_SIZE).to(device) + meta_unified = dataclasses.replace( + meta_split, + num_decodes=0, + num_decode_tokens=0, + num_prefills=meta_split.num_decodes + meta_split.num_prefills, + num_prefill_tokens=( + meta_split.num_decode_tokens + meta_split.num_prefill_tokens + ), + chunk_indices=full_ci, + chunk_offsets=full_co, + # Unified path: the chunk kernel processes the full non-spec batch. + prefill_query_start_loc=meta_split.non_spec_query_start_loc, + prefill_state_indices=meta_split.non_spec_state_indices_tensor, + prefill_has_initial_state=meta_split.has_initial_state, + ) + + # Size the state pools from the indices the builder actually produced. + pool_size = int(meta_split.non_spec_state_indices_tensor.max().item()) + 1 + conv_state_shape, temporal_state_shape = ( + MambaStateShapeCalculator.gated_delta_net_state_shape( + 1, H, HV, K, V, CONV_KERNEL, num_spec=0 + ) + ) + conv_state0 = ( + torch.randn(pool_size, *conv_state_shape, dtype=torch.bfloat16, device=device) + * 0.05 + ) + ssm_state0 = ( + torch.randn(pool_size, *temporal_state_shape, dtype=state_dtype, device=device) + * 0.05 + ) + + A_log = torch.randn(HV, dtype=torch.float32, device=device) * 0.1 + dt_bias = torch.randn(HV, dtype=torch.float32, device=device) * 0.1 + conv_weight = ( + torch.randn(CONV_DIM, 1, CONV_KERNEL, dtype=torch.bfloat16, device=device) * 0.1 + ) + conv_bias = torch.randn(CONV_DIM, dtype=torch.bfloat16, device=device) * 0.1 + + mixed_qkv = ( + torch.randn(num_tokens, CONV_DIM, dtype=torch.bfloat16, device=device) * 0.1 + ) + a = torch.randn(num_tokens, HV, dtype=torch.bfloat16, device=device) * 0.1 + b = torch.randn(num_tokens, HV, dtype=torch.bfloat16, device=device) * 0.1 + + # ---- Split path (real _forward_core, meta_split) ---- + conv_state_split = conv_state0.clone() + ssm_state_split = ssm_state0.clone() + layer_split = _build_layer( + vllm_config, + conv_state_split, + ssm_state_split, + A_log, + dt_bias, + conv_weight, + conv_bias, + ) + out_split = _run_forward_core(layer_split, meta_split, mixed_qkv, b, a, num_tokens) + + # ---- Unified path (real _forward_core, meta_unified) ---- + conv_state_unified = conv_state0.clone() + ssm_state_unified = ssm_state0.clone() + layer_unified = _build_layer( + vllm_config, + conv_state_unified, + ssm_state_unified, + A_log, + dt_bias, + conv_weight, + conv_bias, + ) + out_unified = _run_forward_core( + layer_unified, meta_unified, mixed_qkv, b, a, num_tokens + ) + + # Conv is unified in both paths, so the conv-state update must be identical. + torch.testing.assert_close(conv_state_split, conv_state_unified, atol=0, rtol=0) + + # Chunk vs. recurrent update accumulate in different orders; mirror the + # tolerances used by the kernel-level parity test. + if state_dtype == torch.float32: + atol = rtol = 2e-2 + else: + atol = rtol = 6e-2 + torch.testing.assert_close(out_split, out_unified, atol=atol, rtol=rtol) + torch.testing.assert_close(ssm_state_split, ssm_state_unified, atol=atol, rtol=rtol) diff --git a/asian_ai/performance/vLLM/tests/kernels/mamba/test_gdn_prefill_cutedsl.py b/asian_ai/performance/vLLM/tests/kernels/mamba/test_gdn_prefill_cutedsl.py new file mode 100644 index 0000000000000000000000000000000000000000..12945c0f347cc0bc70130396b1bb2623330416c4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/mamba/test_gdn_prefill_cutedsl.py @@ -0,0 +1,199 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import math + +import pytest +import torch +import torch.nn.functional as F + +from vllm.platforms import current_platform + +if not ( + current_platform.is_cuda() and current_platform.is_device_capability_family(100) +): + pytest.skip( + reason="GDN CuteDSL prefill requires CUDA SM10x.", + allow_module_level=True, + ) + +from vllm.model_executor.layers.fla.ops import ( # noqa: E402 + chunk_gated_delta_rule, +) +from vllm.model_executor.layers.fla.ops.index import ( # noqa: E402 + prepare_chunk_indices, + prepare_chunk_offsets, +) +from vllm.model_executor.layers.mamba.ops.gdn_chunk_cutedsl import ( # noqa: E402 + chunk_gated_delta_rule_cutedsl, + prepare_metadata_cutedsl, +) + + +@pytest.mark.parametrize("num_seqs", [1, 5, 257]) +@pytest.mark.parametrize("state_dtype", [torch.bfloat16, torch.float32]) +def test_gdn_chunk_cutedsl_correctness(num_seqs: int, state_dtype: torch.dtype): + seq_lens = torch.randint( + 1, + 130, + (num_seqs,), + dtype=torch.int32, + ) + cu_seqlens = torch.zeros(num_seqs + 1, device="cuda", dtype=torch.int32) + cu_seqlens[1:] = seq_lens.to(device="cuda").cumsum(0) + total_tokens = int(cu_seqlens[-1].item()) + + num_k_heads = 4 + num_v_heads = 8 + head_k_dim = 128 + head_v_dim = 128 + dtype = torch.bfloat16 + + q = torch.randn( + 1, + total_tokens, + num_k_heads, + head_k_dim, + device="cuda", + dtype=dtype, + ) + k = torch.randn_like(q) + v = torch.randn( + 1, + total_tokens, + num_v_heads, + head_v_dim, + device="cuda", + dtype=dtype, + ) + q = F.normalize(q.float(), p=2, dim=-1).to(dtype) + k = F.normalize(k.float(), p=2, dim=-1).to(dtype) + a = torch.randn( + 1, + total_tokens, + num_v_heads, + device="cuda", + dtype=dtype, + ) + b = torch.randn( + 1, + total_tokens, + num_v_heads, + device="cuda", + dtype=dtype, + ) + # Match upstream FLA GatedDeltaNet synthetic initialization: + # https://github.com/fla-org/flash-linear-attention/blob/main/fla/layers/gated_deltanet.py + A = torch.empty(num_v_heads, device="cuda", dtype=torch.float32).uniform_(0, 16) + A_log = torch.log(A) + dt = torch.exp( + torch.rand(num_v_heads, device="cuda", dtype=torch.float32) + * (math.log(0.1) - math.log(0.001)) + + math.log(0.001) + ) + dt = torch.clamp(dt, min=1e-4) + dt_bias = dt + torch.log(-torch.expm1(-dt)) + g = -A_log.exp().view(1, 1, num_v_heads) * F.softplus( + a.float() + dt_bias.view(1, 1, num_v_heads) + ) + beta = torch.sigmoid(b.float()) + initial_state = ( + torch.randn( + num_seqs, + num_v_heads, + head_v_dim, + head_k_dim, + device="cuda", + dtype=state_dtype, + ) + * 0.05 + ) + + # check metadata kernel + chunk_indices, chunk_offsets = prepare_metadata_cutedsl(cu_seqlens, total_tokens) + torch.accelerator.synchronize() + + expected_indices = prepare_chunk_indices(cu_seqlens, 64) + expected_offsets = prepare_chunk_offsets(cu_seqlens, 64) + total_chunks = int(expected_offsets[-1].item()) + + torch.testing.assert_close(chunk_offsets, expected_offsets.to(torch.int32)) + torch.testing.assert_close( + chunk_indices[:total_chunks], + expected_indices, + ) + + ref_o, ref_state = chunk_gated_delta_rule( + q=q, + k=k, + v=v, + g=g, + beta=beta, + initial_state=initial_state, + output_final_state=True, + cu_seqlens=cu_seqlens, + use_qk_l2norm_in_kernel=False, + ) + actual_core_attn_out = torch.empty( + total_tokens, + num_v_heads, + head_v_dim, + device="cuda", + dtype=dtype, + ) + actual_o, actual_state = chunk_gated_delta_rule_cutedsl( + q=q, + k=k, + v=v, + g=g, + beta=beta, + initial_state=initial_state, + cu_seqlens=cu_seqlens, + chunk_indices=chunk_indices, + chunk_offsets=chunk_offsets, + core_attn_out=actual_core_attn_out, + ) + torch.accelerator.synchronize() + + # check main kernel + o_error = (actual_o.float() - ref_o.float()).abs() + state_error = ( + actual_state.float() - ref_state.to(actual_state.dtype).float() + ).abs() + assert o_error.max().item() < 2e-3 + assert o_error.mean().item() < 6e-5 + assert state_error.max().item() < 2e-2 + assert state_error.mean().item() < 6e-4 + core_attn_out_error = ( + actual_core_attn_out.float() - actual_o.squeeze(0).float() + ).abs() + assert core_attn_out_error.max().item() == 0 + + # check main kernel when core_attn_out is not passed + no_buffer_o, no_buffer_state = chunk_gated_delta_rule_cutedsl( + q=q, + k=k, + v=v, + g=g, + beta=beta, + initial_state=initial_state, + cu_seqlens=cu_seqlens, + chunk_indices=chunk_indices, + chunk_offsets=chunk_offsets, + ) + torch.accelerator.synchronize() + + no_buffer_o_error = (no_buffer_o.float() - ref_o.float()).abs() + no_buffer_state_error = ( + no_buffer_state.float() - ref_state.to(no_buffer_state.dtype).float() + ).abs() + buffer_o_error = (no_buffer_o.float() - actual_o.float()).abs() + buffer_state_error = ( + no_buffer_state.float() - actual_state.to(no_buffer_state.dtype).float() + ).abs() + assert no_buffer_o_error.max().item() < 2e-3 + assert no_buffer_o_error.mean().item() < 6e-5 + assert no_buffer_state_error.max().item() < 2e-2 + assert no_buffer_state_error.mean().item() < 6e-4 + assert buffer_o_error.max().item() == 0 + assert buffer_state_error.max().item() == 0 diff --git a/asian_ai/performance/vLLM/tests/kernels/mamba/test_mamba_mixer2.py b/asian_ai/performance/vLLM/tests/kernels/mamba/test_mamba_mixer2.py new file mode 100644 index 0000000000000000000000000000000000000000..da4c4877f1ec35a22f0dc6498015da9952354be5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/mamba/test_mamba_mixer2.py @@ -0,0 +1,138 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import unittest + +import pytest +import torch + +from tests.utils import ensure_current_vllm_config, multi_gpu_test +from vllm.distributed.parallel_state import ( + init_distributed_environment, + initialize_model_parallel, +) +from vllm.model_executor.layers.mamba.mamba_mixer2 import Mixer2RMSNormGated +from vllm.utils.system_utils import update_environment_variables +from vllm.utils.torch_utils import set_random_seed + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize("batch_size", [8]) +@pytest.mark.parametrize("seq_len", [128]) +@pytest.mark.parametrize( + "hidden_size_n_groups", + [ + (64, 1), + (64, 2), + (64, 4), # hidden_size be divisible by num_gpus + ], +) +@pytest.mark.parametrize("dtype", [torch.float16]) +def test_mixer2_gated_norm_multi_gpu( + batch_size: int, + seq_len: int, + hidden_size_n_groups: tuple[int, int], + dtype: torch.dtype, + device: str = "cuda", +): + hidden_size, n_groups = hidden_size_n_groups + num_processes = 2 + + def run_torch_spawn(fn, nprocs): + # need to use torch.mp.spawn otherwise will have problems with + # torch.distributed and cuda + torch.multiprocessing.spawn( + fn, + args=( + num_processes, + batch_size, + seq_len, + hidden_size, + n_groups, + dtype, + device, + ), + nprocs=nprocs, + ) + + run_torch_spawn(mixer2_gated_norm_tensor_parallel, 2) + + +def mixer2_gated_norm_tensor_parallel( + local_rank: int, + world_size: int, + batch_size: int, + seq_len: int, + hidden_size: int, + n_groups: int, + dtype: torch.dtype, + device: str, +): + set_random_seed(0) + + device = torch.device(f"cuda:{local_rank}") + torch.accelerator.set_device_index(device) + torch.set_default_device(device) + torch.set_default_dtype(dtype) + + update_environment_variables( + { + "RANK": str(local_rank), + "LOCAL_RANK": str(local_rank), + "WORLD_SIZE": str(world_size), + "MASTER_ADDR": "localhost", + "MASTER_PORT": "12345", + } + ) + + # initialize distributed + init_distributed_environment() + with ensure_current_vllm_config(): + initialize_model_parallel(tensor_model_parallel_size=world_size) + + # create random weights an inputs + weight = torch.rand((hidden_size,), dtype=dtype, device=device) + hidden_states = torch.randn(batch_size, seq_len, hidden_size) + gate_states = torch.randn(batch_size, seq_len, hidden_size) + + # create gated-norm with TP + mixer = Mixer2RMSNormGated( + full_hidden_size=hidden_size, + full_n_groups=n_groups, + ) + mixer.weight.weight_loader(mixer.weight, weight) # load + + # create gated-norm without TP to compute reference + # - utilize mock patching to disable TP when + with ( + unittest.mock.patch( + "vllm.model_executor.layers.mamba.mamba_mixer2." + "get_tensor_model_parallel_world_size", + return_value=1, + ), + unittest.mock.patch( + "vllm.model_executor.layers.mamba.mamba_mixer2." + "get_tensor_model_parallel_rank", + return_value=0, + ), + ): + mixer_single_gpu = Mixer2RMSNormGated( + full_hidden_size=hidden_size, + full_n_groups=n_groups, + ) + # assign weight to single-gpu mixer + mixer_single_gpu.weight.data = weight + + # generate and compare + N = hidden_size // world_size + output = mixer( + hidden_states[..., local_rank * N : (local_rank + 1) * N], + gate_states[..., local_rank * N : (local_rank + 1) * N], + ) + ref_output = mixer_single_gpu(hidden_states, gate_states) + torch.testing.assert_close( + output, + ref_output[..., local_rank * N : (local_rank + 1) * N], + atol=5e-3, + rtol=1e-3, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/mamba/test_mamba_ssm.py b/asian_ai/performance/vLLM/tests/kernels/mamba/test_mamba_ssm.py new file mode 100644 index 0000000000000000000000000000000000000000..9b8ad77966c9bea37bf7782cb3cbcd820af3c074 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/mamba/test_mamba_ssm.py @@ -0,0 +1,1104 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch +import torch.nn.functional as F +from einops import rearrange, repeat + +from tests.kernels.mamba.utils import selective_state_update_ref +from tests.kernels.utils import opcheck +from vllm import _custom_ops as ops # noqa: F401 +from vllm.model_executor.layers.mamba.ops.mamba_ssm import ( + selective_scan_fn, + selective_state_update, +) +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.attention.backends.utils import NULL_BLOCK_ID + +DEVICE = current_platform.device_type + +pytestmark = pytest.mark.skipif( + not (current_platform.is_cuda_alike() or current_platform.is_xpu()), + reason="mamba_ssm kernels require CUDA-alike or XPU", +) + +# selective_scan_fn is backed by the CUDA-only `ops.selective_scan_fwd` C++ op, +# so tests exercising it must be skipped on XPU. selective_state_update is +# pure Triton and runs on both CUDA-alike and XPU. +skip_unless_cuda_alike = pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="selective_scan_fn uses CUDA-only custom op", +) + + +def selective_scan_ref( + u, + delta, + A, + B, + C, + D=None, + z=None, + delta_bias=None, + delta_softplus=False, + return_last_state=False, + prev_state=None, + final_state_out=None, +): + """ + u: r(B D L) + delta: r(B D L) + A: c(D N) or r(D N) + B: c(D N) or r(B N L) or r(B N 2L) or r(B G N L) or (B G N L) + C: c(D N) or r(B N L) or r(B N 2L) or r(B G N L) or (B G N L) + D: r(D) + z: r(B D L) + delta_bias: r(D), fp32 + prev_state: r(B D N), fp32 + + out: r(B D L) + last_state (optional): r(B D dstate) or c(B D dstate) + """ + dtype_in = u.dtype + u = u.float() + delta = delta.float() + if delta_bias is not None: + delta = delta + delta_bias[..., None].float() + if delta_softplus: + delta = F.softplus(delta) + batch, dim, dstate = u.shape[0], A.shape[0], A.shape[1] + is_variable_B = B.dim() >= 3 + is_variable_C = C.dim() >= 3 + B = B.float() + C = C.float() + x = A.new_zeros((batch, dim, dstate)) if prev_state is None else prev_state + ys = [] + deltaA = torch.exp(torch.einsum("bdl,dn->bdln", delta, A)) + if not is_variable_B: + deltaB_u = torch.einsum("bdl,dn,bdl->bdln", delta, B, u) + else: + if B.dim() == 3: + deltaB_u = torch.einsum("bdl,bnl,bdl->bdln", delta, B, u) + else: + B = repeat(B, "B G N L -> B (G H) N L", H=dim // B.shape[1]) + deltaB_u = torch.einsum("bdl,bdnl,bdl->bdln", delta, B, u) + if is_variable_C and C.dim() == 4: + C = repeat(C, "B G N L -> B (G H) N L", H=dim // C.shape[1]) + for i in range(u.shape[2]): + x = deltaA[:, :, i] * x + deltaB_u[:, :, i] + if not is_variable_C: + y = torch.einsum("bdn,dn->bd", x, C) + else: + if C.dim() == 3: + y = torch.einsum("bdn,bn->bd", x, C[:, :, i]) + else: + y = torch.einsum("bdn,bdn->bd", x, C[:, :, :, i]) + if i == u.shape[2] - 1: + if final_state_out is None: + final_state_out = x + else: + final_state_out.copy_(x) + ys.append(y) + y = torch.stack(ys, dim=2) # (batch dim L) + out = y if D is None else y + u * rearrange(D, "d -> d 1") + if z is not None: + out = out * F.silu(z) + out = out.to(dtype=dtype_in) + return out if not return_last_state else (out, final_state_out) + + +def selective_scan_opcheck_fn( + u, + delta, + A, + B, + C, + D=None, + z=None, + delta_bias=None, + delta_softplus=False, + cu_seq_len=None, + cache_indices=None, + has_initial_state=None, + ssm_states=None, + null_block_id=NULL_BLOCK_ID, + block_size=2048, + block_idx_first_scheduled_token=None, + block_idx_last_scheduled_token=None, + initial_state_idx=None, + cu_chunk_seqlen=None, + last_chunk_indices=None, +): + """if return_last_state is True, returns (out, last_state) + last_state has shape (batch, dim, dstate). + """ + if u.stride(-1) != 1: + u = u.contiguous() + if delta.stride(-1) != 1: + delta = delta.contiguous() + if D is not None: + D = D.contiguous() + if B.stride(-1) != 1: + B = B.contiguous() + if C.stride(-1) != 1: + C = C.contiguous() + if z is not None and z.stride(-1) != 1: + z = z.contiguous() + if B.dim() == 3 and cu_seq_len is None: + B = B.unsqueeze(1) + if B.dim() == 2 and cu_seq_len is not None: + B = B.unsqueeze(0) + if C.dim() == 3 and cu_seq_len is None: + C = C.unsqueeze(1) + if C.dim() == 2 and cu_seq_len is not None: + C = C.unsqueeze(0) + + # Disable test_autograd_registration for now as it seems to trigger + # a bogus error. + opcheck( + torch.ops._C.selective_scan_fwd, + ( + u, + delta, + A, + B, + C, + D, + z, + delta_bias, + delta_softplus, + cu_seq_len, + cache_indices, + has_initial_state, + ssm_states, + null_block_id, + block_size, + block_idx_first_scheduled_token, + block_idx_last_scheduled_token, + initial_state_idx, + cu_chunk_seqlen, + last_chunk_indices, + ), + test_utils=["test_schema", "test_faketensor"], + ) + + +@pytest.mark.parametrize("wtype", [torch.float32]) +@pytest.mark.parametrize("itype", [torch.float32, torch.bfloat16]) +@pytest.mark.parametrize("seqlen", [128, 1024, 4096]) +@pytest.mark.parametrize("has_delta_bias", [True]) +@pytest.mark.parametrize("delta_softplus", [True]) +@pytest.mark.parametrize("has_z", [True]) +@pytest.mark.parametrize("has_D", [True]) +@pytest.mark.parametrize("varBC_groups", [1, 2]) +@pytest.mark.parametrize("is_variable_C", [True]) +@pytest.mark.parametrize("is_variable_B", [True]) +@pytest.mark.parametrize("scan_chunks", [1, 3]) +@skip_unless_cuda_alike +def test_selective_scan( + is_variable_B, + is_variable_C, + varBC_groups, + has_D, + has_z, + has_delta_bias, + delta_softplus, + seqlen, + itype, + wtype, + scan_chunks, +): + if varBC_groups > 1 and (not is_variable_B or not is_variable_C): + pytest.skip() # This config is not applicable + device = "cuda" + rtol, atol = (6e-4, 2e-3) if itype == torch.float32 else (3e-3, 5e-3) + if itype == torch.bfloat16: + rtol, atol = 3e-2, 5e-2 + rtolw, atolw = (1e-3, 1e-3) + if has_z: # If we have z, the errors on the weights seem higher + rtolw = max(rtolw, rtol) + atolw = max(atolw, atol) + # set seed + set_random_seed(0) + batch_size = 1 + dim = 4 + dstate = 8 + A = -0.5 * torch.rand(dim, dstate, device=device, dtype=wtype) + A_ref = A.clone() + if not is_variable_B: + B_shape = [dim, dstate] + elif varBC_groups == 1: + B_shape = [batch_size, dstate, seqlen] + else: + B_shape = [batch_size, varBC_groups, dstate, seqlen] + B = torch.randn(B_shape, device=device, dtype=wtype if not is_variable_B else itype) + B_ref = B.clone() + if not is_variable_C: + C_shape = [dim, dstate] + elif varBC_groups == 1: + C_shape = [batch_size, dstate, seqlen] + else: + C_shape = [batch_size, varBC_groups, dstate, seqlen] + C = torch.randn(C_shape, device=device, dtype=wtype if not is_variable_C else itype) + C_ref = C.clone() + D = torch.randn(dim, device=device, dtype=torch.float32) if has_D else None + D_ref = D.clone() if D is not None else None + z = ( + torch.randn(batch_size, dim, seqlen, device=device, dtype=itype) + if has_z + else None + ) + z_ref = z.clone() if z is not None else None + delta_bias = ( + (0.5 * torch.rand(dim, device=device, dtype=torch.float32)) + if has_delta_bias + else None + ) + u = torch.randn(batch_size, dim, seqlen, device=device, dtype=itype) + u_ref = u.clone() + delta = 0.5 * torch.rand(batch_size, dim, seqlen, device=device, dtype=itype) + delta_ref = delta.clone() + state_shape = (batch_size, u.shape[1], int(A.shape[1])) + state = torch.randn(state_shape, device=u.device, dtype=itype, requires_grad=False) + state_ref = state.clone() + out = None + out_ref = None + outs = [] + for c in range(scan_chunks): + chunked_prompt_len = seqlen // scan_chunks + chunk_start = chunked_prompt_len * c + chunk_end = chunked_prompt_len * (c + 1) + if c == scan_chunks - 1: + chunk_end = seqlen + _B = B + if is_variable_B: + _B = B[..., chunk_start:chunk_end] + _C = C + if is_variable_B: + _C = C[..., chunk_start:chunk_end] + _z = z + if has_z: + assert z is not None + _z = z[..., chunk_start:chunk_end] + out = selective_scan_fn( + u[..., chunk_start:chunk_end], + state, + delta[..., chunk_start:chunk_end], + A, + _B, + _C, + D, + z=_z, + delta_bias=delta_bias, + delta_softplus=delta_softplus, + has_initial_state=torch.ones(batch_size, device=u.device, dtype=torch.bool) + if c > 0 + else None, + block_size=2048, + block_idx_first_scheduled_token=None, + block_idx_last_scheduled_token=None, + initial_state_idx=None, + ) + outs.append(out) + if len(outs) > 1: + out = torch.cat(outs, dim=-1) + + out_ref, state_ref, *rest = selective_scan_ref( + u_ref, + delta_ref, + A_ref, + B_ref, + C_ref, + D_ref, + z=z_ref, + delta_bias=delta_bias, + delta_softplus=delta_softplus, + return_last_state=True, + ) + + assert out is not None and out_ref is not None + assert torch.allclose(out, out_ref, rtol=rtol, atol=atol) + assert state is not None and state_ref is not None + assert torch.allclose(state, state_ref.to(itype), rtol=rtol, atol=atol) + + selective_scan_opcheck_fn( + u, + delta, + A, + B, + C, + D, + z, + delta_bias=delta_bias, + delta_softplus=delta_softplus, + ssm_states=state, + block_size=2048, + ) + + +@pytest.mark.parametrize("itype", [torch.float32, torch.bfloat16]) +@pytest.mark.parametrize("has_z", [False, True]) +@pytest.mark.parametrize("dstate", [16, 64]) +@pytest.mark.parametrize("dim", [2048, 2048 + 16, 4096]) +def test_selective_state_update(dim, dstate, has_z, itype): + device = DEVICE + rtol, atol = (3e-4, 1e-3) if itype == torch.float32 else (5e-3, 1e-2) + if itype == torch.bfloat16: + rtol, atol = 1e-2, 5e-2 + if current_platform.is_rocm() or current_platform.is_xpu(): + atol *= 2 + # set seed + set_random_seed(0) + batch_size = 1 + state = torch.randn(batch_size, dim, dstate, dtype=itype, device=device) + x = torch.randn(batch_size, dim, device=device, dtype=itype) + out = torch.empty_like(x) + dt = torch.randn(batch_size, dim, device=device, dtype=itype) + dt_bias = torch.rand(dim, device=device) - 4.0 + A = -torch.rand(dim, dstate, device=device) - 1.0 + B = torch.randn(batch_size, dstate, device=device) + C = torch.randn(batch_size, dstate, device=device) + D = torch.randn(dim, device=device) + z = torch.randn_like(x) if has_z else None + state_ref = state.detach().clone() + selective_state_update( + state, x, dt, A, B, C, D=D, z=z, dt_bias=dt_bias, dt_softplus=True, out=out + ) + out_ref = selective_state_update_ref( + state_ref, x, dt, A, B, C, D=D, z=z, dt_bias=dt_bias, dt_softplus=True + ) + + assert torch.allclose(state, state_ref, rtol=rtol, atol=atol) + assert torch.allclose(out, out_ref, rtol=rtol, atol=atol) + + +@pytest.mark.parametrize("philox_rounds", [0, 4]) +@pytest.mark.parametrize("has_z", [False, True]) +@pytest.mark.parametrize("dstate", [16, 64]) +@pytest.mark.parametrize("dim", [2048, 4096]) +@pytest.mark.skipif( + not ( + current_platform.is_cuda() and current_platform.is_device_capability_family(100) + ), + reason="Stochastic rounding in triton is only supported" + " on compute capability 10.0 CUDA devices.", +) +def test_selective_state_update_stochastic_rounding(dim, dstate, has_z, philox_rounds): + device = DEVICE + rtol, atol = 5e-3, 1e-1 + # set seed + set_random_seed(0) + batch_size = 1 + state = torch.randn(batch_size, dim, dstate, dtype=torch.float16, device=device) + x = torch.randn(batch_size, dim, device=device, dtype=torch.bfloat16) + out = torch.empty_like(x) + dt = torch.randn(batch_size, dim, device=device, dtype=torch.bfloat16) + dt_bias = torch.rand(dim, device=device) - 4.0 + A = -torch.rand(dim, dstate, device=device) - 1.0 + B = torch.randn(batch_size, dstate, device=device) + C = torch.randn(batch_size, dstate, device=device) + D = torch.randn(dim, device=device) + z = torch.randn_like(x) if has_z else None + # Reference uses fp32 state to get ground truth + state_ref = state.float() + selective_state_update( + state, + x, + dt, + A, + B, + C, + D=D, + z=z, + dt_bias=dt_bias, + dt_softplus=True, + out=out, + enable_stochastic_rounding=True, + cache_philox_rounds=philox_rounds, + ) + out_ref = selective_state_update_ref( + state_ref, x, dt, A, B, C, D=D, z=z, dt_bias=dt_bias, dt_softplus=True + ) + + assert state.dtype == torch.float16 + assert torch.allclose(state, state_ref.to(torch.float16), rtol=rtol, atol=atol) + assert torch.allclose(out, out_ref, rtol=rtol, atol=atol) + + +@pytest.mark.parametrize("itype", [torch.float32, torch.bfloat16]) +@pytest.mark.parametrize("has_z", [False, True]) +@pytest.mark.parametrize("dstate", [16, 64]) +@pytest.mark.parametrize("dim", [2048, 2048 + 16, 4096]) +@pytest.mark.parametrize("max_seq_len", [1, 2, 4]) +def test_selective_state_update_varlen(dim, dstate, has_z, itype, max_seq_len): + device = DEVICE + rtol, atol = (3e-4, 1e-3) if itype == torch.float32 else (5e-3, 1e-2) + if itype == torch.bfloat16: + rtol, atol = 5e-2, 1.5e-1 + if current_platform.is_rocm() or current_platform.is_xpu(): + atol *= 2 + # set seed + set_random_seed(0) + batch_size = 4 + token_counts = torch.randint(1, max_seq_len + 1, (batch_size,), device=device) + total_tokens = int(token_counts.sum().item()) + cu_seqlens = torch.tensor( + [0] + torch.cumsum(token_counts, dim=0).tolist(), + dtype=torch.int32, + device=device, + ) + state = torch.randn(batch_size, dim, dstate, dtype=itype, device=device) + x = torch.randn(total_tokens, dim, device=device, dtype=itype) + out = torch.empty_like(x) + dt = torch.randn(total_tokens, dim, device=device, dtype=itype) + dt_bias = torch.rand(dim, device=device) - 4.0 + A = -torch.rand(dim, dstate, device=device) - 1.0 + B = torch.randn(total_tokens, dstate, device=device) + C = torch.randn(total_tokens, dstate, device=device) + D = torch.randn(dim, device=device) + z = torch.randn_like(x) if has_z else None + state_ref = state.detach().clone() + selective_state_update( + state, + x, + dt, + A, + B, + C, + D=D, + z=z, + dt_bias=dt_bias, + dt_softplus=True, + out=out, + cu_seqlens=cu_seqlens, + ) + + out_ref_list = [] + for seq_idx in range(batch_size): + start_idx = cu_seqlens[seq_idx].item() + end_idx = cu_seqlens[seq_idx + 1].item() + num_tokens = end_idx - start_idx + for token_idx in range(num_tokens): + idx = start_idx + token_idx + out_ref_list.append( + selective_state_update_ref( + state_ref[seq_idx : seq_idx + 1], + x[idx : idx + 1], + dt[idx : idx + 1], + A, + B[idx : idx + 1], + C[idx : idx + 1], + D=D, + z=z[idx : idx + 1] if z is not None else None, + dt_bias=dt_bias, + dt_softplus=True, + ) + ) + out_ref = torch.cat(out_ref_list, dim=0) + assert torch.allclose(state, state_ref, rtol=rtol, atol=atol) + assert torch.allclose(out, out_ref, rtol=rtol, atol=atol) + + +@pytest.mark.parametrize("wtype", [torch.float32]) +@pytest.mark.parametrize("itype", [torch.float32]) +@pytest.mark.parametrize("seqlen", [1, 256, 1024, 4096]) +@pytest.mark.parametrize("return_last_state", [True]) +@pytest.mark.parametrize("has_delta_bias", [True]) +@pytest.mark.parametrize("delta_softplus", [True]) +@pytest.mark.parametrize("has_z", [True]) +@pytest.mark.parametrize("has_D", [True]) +@pytest.mark.parametrize("varBC_groups", [1, 2]) +@pytest.mark.parametrize("is_variable_C", [True]) +@pytest.mark.parametrize("is_variable_B", [True]) +# tests correctness in case subset of the sequences are padded +@pytest.mark.parametrize("with_padding", [False, True]) +@skip_unless_cuda_alike +def test_selective_scan_varlen( + with_padding, + is_variable_B, + is_variable_C, + varBC_groups, + has_D, + has_z, + has_delta_bias, + delta_softplus, + return_last_state, + seqlen, + itype, + wtype, +): + if varBC_groups > 1 and (not is_variable_B or not is_variable_C): + pytest.skip() # This config is not applicable + device = "cuda" + rtol, atol = (6e-4, 2e-3) if itype == torch.float32 else (3e-3, 5e-3) + if itype == torch.bfloat16: + rtol, atol = 3e-2, 5e-2 + rtolw, atolw = (1e-3, 1e-3) + if has_z: # If we have z, the errors on the weights seem higher + rtolw = max(rtolw, rtol) + atolw = max(atolw, atol) + # set seed + torch.random.manual_seed(0) + seqlens = [] + batch_size = 4 + if seqlen < 10: + batch_size = 1 + padding = 3 if with_padding else 0 + padded_batch_size = batch_size + padding + + if with_padding and seqlen < padded_batch_size: + pytest.skip() + + nsplits = padded_batch_size - 1 + eos_pos = torch.randperm(seqlen - 1)[:nsplits].sort().values + seqlens.append( + torch.diff( + torch.cat([torch.tensor([-1]), eos_pos, torch.tensor([seqlen - 1])]) + ).tolist() + ) + + assert sum(seqlens[-1]) == seqlen + assert all(s > 0 for s in seqlens[-1]) + + total_entries = batch_size * 10 + cumsum = torch.cumsum(torch.tensor(seqlens[0]), dim=0).to(torch.int32) + cumsum = torch.concat([torch.tensor([0], dtype=torch.int32), cumsum], dim=0).cuda() + + dim = 4 + dstate = 8 + A = -0.5 * torch.rand(dim, dstate, device=device, dtype=wtype) + A_ref = A.clone() + B_shape = [varBC_groups, dstate, seqlen] + B = torch.randn(B_shape, device=device, dtype=wtype if not is_variable_B else itype) + B_ref = B.clone() + C_shape = [varBC_groups, dstate, seqlen] + C = torch.randn(C_shape, device=device, dtype=wtype if not is_variable_C else itype) + C_ref = C.clone() + D = torch.randn(dim, device=device, dtype=torch.float32) if has_D else None + D_ref = D.clone() if D is not None else None + z = torch.randn(dim, seqlen, device=device, dtype=itype) + z_ref = z.clone() + delta_bias = ( + (0.5 * torch.rand(dim, device=device, dtype=torch.float32)) + if has_delta_bias + else None + ) + u = torch.randn(dim, seqlen, device=device, dtype=itype) + u_ref = u.clone() + delta = 0.5 * torch.rand(dim, seqlen, device=device, dtype=itype) + delta_ref = delta.clone() + out = None + out_ref = None + + prev_state_shape = (total_entries, u.shape[0], int(A.shape[1])) + prev_state = torch.randn( + prev_state_shape, device=u.device, dtype=itype, requires_grad=False + ) + prev_state_ref = prev_state.clone() + # +1 to exclude index 0 (null block) + state_indices = ( + torch.randperm(total_entries - 1, dtype=torch.int32, device=u.device)[ + :batch_size + ] + + 1 + ) + unused_states_bool = torch.ones(total_entries, dtype=torch.bool, device=device) + unused_states_bool[state_indices] = False + padded_state_indices = torch.concat( + [ + state_indices, + torch.as_tensor( + [NULL_BLOCK_ID] * padding, dtype=torch.int32, device=device + ), + ], + dim=-1, + ) + + has_initial_state = torch.randint( + 0, 2, (cumsum.shape[0] - 1,), dtype=torch.bool, device=u.device + ) + out = selective_scan_fn( + u, + prev_state, + delta, + A, + B, + C, + D, + z, + delta_bias, + delta_softplus, + cumsum, + padded_state_indices, + has_initial_state, + ) + outs_ref = [] + splits = [ + torch.split(var, seqlens[0], dim=-1) + for var in (u_ref, delta_ref, B_ref, C_ref, z_ref) + ] + for i in range(len(seqlens[0])): + u_s, delta_s, B_s, C_s, z_s = (v[i].unsqueeze(0) for v in splits) + if padded_state_indices[i] == NULL_BLOCK_ID: + continue + out_ref_s, _ = selective_scan_ref( + u_s, + delta_s, + A_ref, + B_s, + C_s, + D_ref, + z=z_s, + delta_bias=delta_bias, + delta_softplus=delta_softplus, + return_last_state=return_last_state, + prev_state=prev_state_ref[padded_state_indices[i]].unsqueeze(0) + if has_initial_state[i] + else None, + final_state_out=prev_state_ref[padded_state_indices[i]].unsqueeze(0), + ) + outs_ref.append(out_ref_s) + out_ref = torch.cat(outs_ref, dim=-1)[0] + + unpadded_out = out[:, : out_ref[0].shape[-1]] + print("Output diff max", (unpadded_out - out_ref).max()) + print("Output diff mean", (unpadded_out - out_ref).mean()) + print("Output state diff max", (prev_state - prev_state_ref).max()) + print("Output state diff mean", (prev_state - prev_state_ref).mean()) + assert torch.allclose(prev_state, prev_state_ref, rtol=rtol, atol=atol) + assert torch.allclose(unpadded_out, out_ref, rtol=rtol, atol=atol) + selective_scan_opcheck_fn( + u, + delta, + A, + B, + C, + D, + z, + delta_bias, + delta_softplus, + cumsum, + padded_state_indices, + has_initial_state, + prev_state, + block_size=2048, + ) + + +@pytest.mark.parametrize("itype", [torch.float32, torch.bfloat16]) +@pytest.mark.parametrize("has_z", [True]) +@pytest.mark.parametrize("dstate", [16, 64]) +@pytest.mark.parametrize("dim", [2048, 2048 + 16, 4096]) +# tests correctness in case subset of the sequences are padded +@pytest.mark.parametrize("with_padding", [True, False]) +def test_selective_state_update_with_batch_indices( + with_padding, dim, dstate, has_z, itype +): + device = DEVICE + rtol, atol = (3e-4, 1e-3) if itype == torch.float32 else (5e-3, 1e-2) + if itype == torch.bfloat16: + rtol, atol = 1e-1, 1e-1 + if current_platform.is_rocm() or current_platform.is_xpu(): + atol *= 2 + # set seed + torch.random.manual_seed(0) + batch_size = 3 + padding = 5 if with_padding else 0 + padded_batch_size = batch_size + padding + total_entries = 10 * batch_size + state = torch.randn(total_entries, dim, dstate, dtype=itype, device=device) + # +1 to exclude index 0 (null block) + state_indices = (torch.randperm(total_entries - 1)[:batch_size] + 1).to( + dtype=torch.int32, device=device + ) + unused_states_bool = torch.ones(total_entries, dtype=torch.bool, device=device) + unused_states_bool[state_indices] = False + padded_state_indices = torch.concat( + [ + state_indices, + torch.as_tensor( + [NULL_BLOCK_ID] * padding, dtype=torch.int32, device=device + ), + ], + dim=0, + ) + x = torch.randn(padded_batch_size, dim, device=device, dtype=itype) + out = torch.empty_like(x) + dt = torch.randn(padded_batch_size, dim, device=device, dtype=itype) + dt_bias = torch.rand(dim, device=device) - 4.0 + A = -torch.rand(dim, dstate, device=device) - 1.0 + B = torch.randn(padded_batch_size, dstate, device=device) + C = torch.randn(padded_batch_size, dstate, device=device) + D = torch.randn(dim, device=device) + z = torch.randn_like(x) if has_z else None + state_ref = state[state_indices, :].clone() + state_before = state.clone() + selective_state_update( + state, + x, + dt, + A, + B, + C, + D=D, + z=z, + dt_bias=dt_bias, + dt_softplus=True, + state_batch_indices=padded_state_indices, + out=out, + ) + out_ref = selective_state_update_ref( + state_ref, + x[:batch_size], + dt[:batch_size], + A, + B[:batch_size], + C[:batch_size], + D=D, + z=z[:batch_size] if z is not None else None, + dt_bias=dt_bias, + dt_softplus=True, + ) + + print("Output diff max", (out[:batch_size] - out_ref).max()) + print("Output diff mean", (out[:batch_size] - out_ref).mean()) + print("Output state diff max", (state[state_indices, :] - state_ref).max()) + print("Output state diff mean", (state[state_indices, :] - state_ref).mean()) + # test padded entries stay the same + if with_padding: + assert torch.equal(state_before[unused_states_bool], state[unused_states_bool]) + assert torch.equal(x[batch_size + 1 :], x[batch_size + 1 :]) + assert torch.equal(dt[batch_size + 1 :], dt[batch_size + 1 :]) + assert torch.equal(B[batch_size + 1 :], B[batch_size + 1 :]) + assert torch.equal(C[batch_size + 1 :], C[batch_size + 1 :]) + + # test "real" entries + assert torch.allclose(state[state_indices, :], state_ref, rtol=rtol, atol=atol) + assert torch.allclose(out[:batch_size], out_ref, rtol=rtol, atol=atol) + + +@pytest.mark.parametrize("itype", [torch.float32, torch.bfloat16]) +@pytest.mark.parametrize("has_z", [False, True]) +@pytest.mark.parametrize("tie_hdim", [False, True]) +@pytest.mark.parametrize("ngroups", [1, 4]) +@pytest.mark.parametrize("dstate", [16, 64]) +@pytest.mark.parametrize("dim", [2048, 4096]) +def test_selective_state_update_with_heads_with_batch_indices( + dim, dstate, ngroups, has_z, tie_hdim, itype +): + device = DEVICE + rtol, atol = (3e-4, 1e-3) if itype == torch.float32 else (5e-3, 3e-2) + if itype == torch.bfloat16: + rtol, atol = 1e-1, 1e-1 + # set seed + torch.random.manual_seed(0) + batch_size = 3 + headdim = 64 + nheads = dim // headdim + + total_entries = 10 * batch_size + state = torch.randn( + total_entries, nheads, headdim, dstate, dtype=itype, device=device + ) + # +1 to exclude index 0 (null block) + state_indices = (torch.randperm(total_entries - 1)[:batch_size] + 1).to( + dtype=torch.int32, device=device + ) + + x = torch.randn(batch_size, nheads, headdim, device=device, dtype=itype) + out = torch.empty_like(x) + if not tie_hdim: + dt = torch.randn(batch_size, nheads, headdim, device=device, dtype=itype) + dt_bias = torch.rand(nheads, headdim, device=device) - 4.0 + A = -torch.rand(nheads, headdim, dstate, device=device) - 1.0 + D = torch.randn(nheads, headdim, device=device) + else: + dt = repeat( + torch.randn(batch_size, nheads, device=device, dtype=itype), + "b h -> b h p", + p=headdim, + ) + dt_bias = repeat(torch.rand(nheads, device=device) - 4.0, "h -> h p", p=headdim) + A = repeat( + -torch.rand(nheads, device=device) - 1.0, "h -> h p n", p=headdim, n=dstate + ) + D = repeat(torch.randn(nheads, device=device), "h -> h p", p=headdim) + B = torch.randn(batch_size, ngroups, dstate, device=device) + C = torch.randn(batch_size, ngroups, dstate, device=device) + z = torch.randn_like(x) if has_z else None + state_ref = state[state_indices, :].detach().clone() + selective_state_update( + state, + x, + dt, + A, + B, + C, + D=D, + z=z, + dt_bias=dt_bias, + dt_softplus=True, + state_batch_indices=state_indices, + out=out, + ) + out_ref = selective_state_update_ref( + state_ref, x, dt, A, B, C, D=D, z=z, dt_bias=dt_bias, dt_softplus=True + ) + + print(f"Output max diff: {(out - out_ref).abs().max().item()}") + print(f"Output mean diff: {(out - out_ref).abs().mean().item()}") + assert torch.allclose(state[state_indices, :], state_ref, rtol=rtol, atol=atol) + assert torch.allclose(out, out_ref, rtol=rtol, atol=atol) + + +@pytest.mark.parametrize("itype", [torch.float32, torch.bfloat16]) +@pytest.mark.parametrize("has_z", [False, True]) +@pytest.mark.parametrize("dstate", [16, 64]) +@pytest.mark.parametrize("dim", [2048, 4096]) +@pytest.mark.parametrize("max_seq_len", [2, 4]) +def test_selective_state_update_with_num_accepted_tokens( + dim, dstate, has_z, itype, max_seq_len +): + device = DEVICE + rtol, atol = (3e-4, 1e-3) if itype == torch.float32 else (5e-3, 1e-2) + if itype == torch.bfloat16: + rtol, atol = 5e-2, 1.5e-1 + if current_platform.is_rocm() or current_platform.is_xpu(): + atol *= 2 + + set_random_seed(0) + batch_size = 4 + + tokens_per_seq = torch.randint(1, max_seq_len + 1, (batch_size,), device=device) + total_tokens = int(tokens_per_seq.sum().item()) + + num_accepted_tokens = torch.randint(0, max_seq_len, (batch_size,), device=device) + num_accepted_tokens[0] = 0 # Add edge-case of no accepted tokens + num_accepted_tokens[1] = max_seq_len # Add edge-case of all tokens accepted + + cu_seqlens = torch.tensor( + [0] + torch.cumsum(tokens_per_seq, dim=0).tolist(), + dtype=torch.int32, + device=device, + ) + + total_state_slots = 50 + state = torch.randn(total_state_slots, dim, dstate, dtype=itype, device=device) + + state_batch_indices = torch.full( + (batch_size, max_seq_len), NULL_BLOCK_ID, dtype=torch.int32, device=device + ) + # Start from 1 to exclude null block at index 0 + initial_state_slots = torch.randint( + 1, 15, (batch_size,), device=device, dtype=torch.int32 + ) + for seq_idx in range(batch_size): + token_pos = max(num_accepted_tokens[seq_idx].item() - 1, 0) + state_batch_indices[seq_idx, token_pos] = initial_state_slots[seq_idx] + + dst_state_batch_indices = torch.full( + (batch_size, max_seq_len), NULL_BLOCK_ID, dtype=torch.int32, device=device + ) + slot_offset = 15 + dst_slots_map = {} + for seq_idx in range(batch_size): + for token_idx in range(tokens_per_seq[seq_idx].item()): + dst_state_batch_indices[seq_idx, token_idx] = slot_offset + dst_slots_map[(seq_idx, token_idx)] = slot_offset + slot_offset += 1 + + x = torch.randn(total_tokens, dim, device=device, dtype=itype) + out = torch.empty_like(x) + dt = torch.randn(total_tokens, dim, device=device, dtype=itype) + dt_bias = torch.rand(dim, device=device) - 4.0 + A = -torch.rand(dim, dstate, device=device) - 1.0 + B = torch.randn(total_tokens, dstate, device=device) + C = torch.randn(total_tokens, dstate, device=device) + D = torch.randn(dim, device=device) + z = torch.randn_like(x) if has_z else None + + state_ref_intermediate = {} + out_ref_list = [] + + for seq_idx in range(batch_size): + seq_start = cu_seqlens[seq_idx].item() + seq_end = cu_seqlens[seq_idx + 1].item() + num_tokens = seq_end - seq_start + + token_pos = max(num_accepted_tokens[seq_idx].item() - 1, 0) + initial_slot = state_batch_indices[seq_idx, token_pos].item() + state_seq = state[initial_slot : initial_slot + 1].clone() + + for token_idx in range(num_tokens): + global_idx = seq_start + token_idx + + out_token = selective_state_update_ref( + state_seq, + x[global_idx : global_idx + 1], + dt[global_idx : global_idx + 1], + A, + B[global_idx : global_idx + 1], + C[global_idx : global_idx + 1], + D=D, + z=z[global_idx : global_idx + 1] if z is not None else None, + dt_bias=dt_bias, + dt_softplus=True, + ) + out_ref_list.append(out_token) + state_ref_intermediate[(seq_idx, token_idx)] = state_seq.clone() + + out_ref = torch.cat(out_ref_list, dim=0) + + selective_state_update( + state, + x, + dt, + A, + B, + C, + D=D, + z=z, + dt_bias=dt_bias, + dt_softplus=True, + out=out, + cu_seqlens=cu_seqlens, + state_batch_indices=state_batch_indices, + dst_state_batch_indices=dst_state_batch_indices, + num_accepted_tokens=num_accepted_tokens, + ) + + assert torch.allclose(out, out_ref, rtol=rtol, atol=atol) + + for seq_idx in range(batch_size): + num_tokens = tokens_per_seq[seq_idx].item() + for token_idx in range(num_tokens): + dst_slot = dst_slots_map[(seq_idx, token_idx)] + state_ref = state_ref_intermediate[(seq_idx, token_idx)].squeeze(0) + assert torch.allclose(state[dst_slot], state_ref, rtol=rtol, atol=atol) + + +@pytest.mark.parametrize("itype", [torch.float32, torch.bfloat16]) +@pytest.mark.parametrize("has_z", [False, True]) +@pytest.mark.parametrize("dstate", [16, 64]) +@pytest.mark.parametrize("dim", [2048, 4096]) +@pytest.mark.parametrize("max_seq_len", [2, 4]) +def test_selective_state_update_varlen_with_num_accepted( + dim, dstate, has_z, itype, max_seq_len +): + device = DEVICE + rtol, atol = (3e-4, 1e-3) if itype == torch.float32 else (5e-3, 1e-2) + if itype == torch.bfloat16: + rtol, atol = 5e-2, 1.5e-1 + if current_platform.is_rocm() or current_platform.is_xpu(): + atol *= 2 + + set_random_seed(0) + batch_size = 4 + + tokens_per_seq = torch.randint(1, max_seq_len + 1, (batch_size,), device=device) + total_tokens = int(tokens_per_seq.sum().item()) + + num_accepted_tokens = torch.randint(0, max_seq_len, (batch_size,), device=device) + num_accepted_tokens[0] = 0 # Add edge-case of no accepted tokens + num_accepted_tokens[1] = max_seq_len # Add edge-case of all tokens accepted + + cu_seqlens = torch.tensor( + [0] + torch.cumsum(tokens_per_seq, dim=0).tolist(), + dtype=torch.int32, + device=device, + ) + + total_state_slots = 50 + state = torch.randn(total_state_slots, dim, dstate, dtype=itype, device=device) + + state_batch_indices = torch.full( + (batch_size, max_seq_len), NULL_BLOCK_ID, dtype=torch.int32, device=device + ) + + # Start from 1 to exclude null block at index 0 + initial_state_slots = torch.randint( + 1, 15, (batch_size,), device=device, dtype=torch.int32 + ) + for seq_idx in range(batch_size): + token_pos = max(num_accepted_tokens[seq_idx].item() - 1, 0) + state_batch_indices[seq_idx, token_pos] = initial_state_slots[seq_idx] + + dst_state_batch_indices = torch.full( + (batch_size, max_seq_len), NULL_BLOCK_ID, dtype=torch.int32, device=device + ) + + slot_offset = 15 + dst_slots_map = {} + for seq_idx in range(batch_size): + for token_idx in range(tokens_per_seq[seq_idx].item()): + dst_state_batch_indices[seq_idx, token_idx] = slot_offset + dst_slots_map[(seq_idx, token_idx)] = slot_offset + slot_offset += 1 + + x = torch.randn(total_tokens, dim, device=device, dtype=itype) + out = torch.empty_like(x) + dt = torch.randn(total_tokens, dim, device=device, dtype=itype) + dt_bias = torch.rand(dim, device=device) - 4.0 + A = -torch.rand(dim, dstate, device=device) - 1.0 + B = torch.randn(total_tokens, dstate, device=device) + C = torch.randn(total_tokens, dstate, device=device) + D = torch.randn(dim, device=device) + z = torch.randn_like(x) if has_z else None + + state_ref_intermediate = {} + + for seq_idx in range(batch_size): + seq_start = cu_seqlens[seq_idx].item() + seq_end = cu_seqlens[seq_idx + 1].item() + num_tokens = seq_end - seq_start + + token_pos = max(num_accepted_tokens[seq_idx].item() - 1, 0) + initial_slot = state_batch_indices[seq_idx, token_pos].item() + state_seq = state[initial_slot : initial_slot + 1].clone() + + for token_idx in range(num_tokens): + global_idx = seq_start + token_idx + + selective_state_update_ref( + state_seq, + x[global_idx : global_idx + 1], + dt[global_idx : global_idx + 1], + A, + B[global_idx : global_idx + 1], + C[global_idx : global_idx + 1], + D=D, + z=z[global_idx : global_idx + 1] if z is not None else None, + dt_bias=dt_bias, + dt_softplus=True, + ) + + state_ref_intermediate[(seq_idx, token_idx)] = state_seq.clone() + + selective_state_update( + state, + x, + dt, + A, + B, + C, + D=D, + z=z, + dt_bias=dt_bias, + dt_softplus=True, + out=out, + cu_seqlens=cu_seqlens, + state_batch_indices=state_batch_indices, + dst_state_batch_indices=dst_state_batch_indices, + num_accepted_tokens=num_accepted_tokens, + ) + + for seq_idx in range(batch_size): + num_tokens = tokens_per_seq[seq_idx].item() + + for token_idx in range(num_tokens): + dst_slot = dst_slots_map[(seq_idx, token_idx)] + state_ref = state_ref_intermediate[(seq_idx, token_idx)].squeeze(0) + + assert torch.allclose(state[dst_slot], state_ref, rtol=rtol, atol=atol) diff --git a/asian_ai/performance/vLLM/tests/kernels/mamba/test_mamba_ssm_configs.py b/asian_ai/performance/vLLM/tests/kernels/mamba/test_mamba_ssm_configs.py new file mode 100644 index 0000000000000000000000000000000000000000..8bd32bb0b5d87e38574c1151c39fcd1adb13f2c5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/mamba/test_mamba_ssm_configs.py @@ -0,0 +1,212 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Unit tests for the JSON-based config loader added to selective_state_update. + +Tests cover: + - Flat MoE-style filename generation + - VLLM_TUNED_CONFIG_FOLDER env-var override + - Fallback to heuristic when no config file exists + - Nearest effective_batch interpolation + - Edge cases: non-dict JSON, empty config +""" + +import json + +from vllm.model_executor.layers.mamba.ops.mamba_ssm import ( + _get_default_ssm_launch_config, + _try_get_optimal_ssm_config_cached, + get_ssm_config_file_name, + get_ssm_configs, + get_ssm_device_name, + try_get_optimal_ssm_config, +) + +# Common kwargs for try_get_optimal_ssm_config. Tests pick (batch, nheads) so +# their product (effective_batch) matches the value being probed. +_HEADDIM = 64 +_CACHE_DTYPE = "float32" + + +def _clear_caches() -> None: + get_ssm_configs.cache_clear() + _try_get_optimal_ssm_config_cached.cache_clear() + + +def _write_config(tmp_path, dstate: int, payload: dict) -> None: + """Write payload as the bundled config for (headdim, dstate, cache_dtype).""" + device_name = get_ssm_device_name() + config_path = tmp_path / get_ssm_config_file_name( + _HEADDIM, dstate, _CACHE_DTYPE, device_name + ) + with open(config_path, "w") as f: + json.dump(payload, f) + + +# --------------------------------------------------------------------------- +# Config filename generation +# --------------------------------------------------------------------------- + + +def test_config_file_name_format(): + name = get_ssm_config_file_name( + headdim=64, dstate=128, cache_dtype="float32", device_name="NVIDIA_B200" + ) + assert name == ( + "headdim=64,dstate=128,device_name=NVIDIA_B200,cache_dtype=float32.json" + ) + + +# --------------------------------------------------------------------------- +# VLLM_TUNED_CONFIG_FOLDER override +# --------------------------------------------------------------------------- + + +def test_env_override_loads_custom_config(monkeypatch, tmp_path): + """VLLM_TUNED_CONFIG_FOLDER should take precedence over the bundled dir.""" + _write_config( + tmp_path, + dstate=16, + payload={ + "1": {"BLOCK_SIZE_M": 4, "num_warps": 1}, + }, + ) + + monkeypatch.setenv("VLLM_TUNED_CONFIG_FOLDER", str(tmp_path)) + _clear_caches() + + cfg = get_ssm_configs(_HEADDIM, 16, _CACHE_DTYPE) + assert cfg is not None + assert cfg[1] == {"BLOCK_SIZE_M": 4, "num_warps": 1} + + _clear_caches() + + +# --------------------------------------------------------------------------- +# Fallback to heuristic when no config file exists +# --------------------------------------------------------------------------- + + +def test_fallback_when_no_config(monkeypatch, tmp_path): + """try_get_optimal_ssm_config must fall back to _get_default_ssm_launch_config + when no JSON file is found for the current + (device, headdim, dstate, cache_dtype) combination. + """ + monkeypatch.setenv("VLLM_TUNED_CONFIG_FOLDER", str(tmp_path)) + monkeypatch.setattr( + "vllm.model_executor.layers.mamba.ops.mamba_ssm._CONFIGS_DIR", + str(tmp_path), + ) + + for dstate in (8, 16, 32, 64, 128, 256): + for is_blackwell in (False, True): + _clear_caches() + block_m, warps = try_get_optimal_ssm_config( + headdim=_HEADDIM, + dstate=dstate, + batch=1, + nheads=1, + cache_dtype=_CACHE_DTYPE, + is_blackwell=is_blackwell, + ) + assert (block_m, warps) == _get_default_ssm_launch_config( + dstate, is_blackwell=is_blackwell + ) + + _clear_caches() + + +# --------------------------------------------------------------------------- +# Nearest effective_batch interpolation +# --------------------------------------------------------------------------- + + +def test_nearest_effective_batch_interpolation(monkeypatch, tmp_path): + """When effective_batch = batch*nheads is not an exact key, the closest + key should be selected.""" + _write_config( + tmp_path, + dstate=32, + payload={ + "64": {"BLOCK_SIZE_M": 8, "num_warps": 1}, + "4096": {"BLOCK_SIZE_M": 32, "num_warps": 4}, + }, + ) + + monkeypatch.setenv("VLLM_TUNED_CONFIG_FOLDER", str(tmp_path)) + _clear_caches() + + # effective_batch = 1*128 = 128 -> closer to 64 than to 4096 + block_m, warps = try_get_optimal_ssm_config( + headdim=_HEADDIM, + dstate=32, + batch=1, + nheads=128, + cache_dtype=_CACHE_DTYPE, + is_blackwell=False, + ) + assert block_m == 8 and warps == 1 + + # effective_batch = 4*1024 = 4096 -> exact match on 4096 + block_m, warps = try_get_optimal_ssm_config( + headdim=_HEADDIM, + dstate=32, + batch=4, + nheads=1024, + cache_dtype=_CACHE_DTYPE, + is_blackwell=False, + ) + assert block_m == 32 and warps == 4 + + _clear_caches() + + +# --------------------------------------------------------------------------- +# Edge cases: malformed / empty config files +# --------------------------------------------------------------------------- + + +def test_non_dict_json_returns_none(monkeypatch, tmp_path): + """A valid JSON file that is not a dict (e.g. a list) must be ignored + and return None rather than raising AttributeError.""" + device_name = get_ssm_device_name() + config_path = tmp_path / get_ssm_config_file_name( + _HEADDIM, 16, _CACHE_DTYPE, device_name + ) + with open(config_path, "w") as f: + json.dump([1, 2, 3], f) + + monkeypatch.setenv("VLLM_TUNED_CONFIG_FOLDER", str(tmp_path)) + monkeypatch.setattr( + "vllm.model_executor.layers.mamba.ops.mamba_ssm._CONFIGS_DIR", + str(tmp_path), + ) + _clear_caches() + + assert get_ssm_configs(_HEADDIM, 16, _CACHE_DTYPE) is None + + _clear_caches() + + +def test_empty_config_falls_back_to_heuristic(monkeypatch, tmp_path): + """An empty JSON object {} must not crash min() — should fall back + to the hard-coded heuristic.""" + _write_config(tmp_path, dstate=64, payload={}) + + monkeypatch.setenv("VLLM_TUNED_CONFIG_FOLDER", str(tmp_path)) + _clear_caches() + + dstate = 64 + block_m, warps = try_get_optimal_ssm_config( + headdim=_HEADDIM, + dstate=dstate, + batch=1, + nheads=64, + cache_dtype=_CACHE_DTYPE, + is_blackwell=False, + ) + assert (block_m, warps) == _get_default_ssm_launch_config( + dstate=dstate, is_blackwell=False + ) + + _clear_caches() diff --git a/asian_ai/performance/vLLM/tests/kernels/mamba/test_mamba_ssm_ssd.py b/asian_ai/performance/vLLM/tests/kernels/mamba/test_mamba_ssm_ssd.py new file mode 100644 index 0000000000000000000000000000000000000000..ce859b89f966bff0e43fb180f2e72c5d71761da6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/mamba/test_mamba_ssm_ssd.py @@ -0,0 +1,579 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch +import torch.nn.functional as F +from einops import rearrange, repeat + +from vllm.model_executor.layers.mamba.ops.ssd_combined import ( + mamba_chunk_scan_combined_varlen, +) +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.attention.backends.mamba2_attn import compute_varlen_chunk_metadata + +# All kernels exercised here are pure Triton, so they run on any backend +# that the vLLM platform layer treats as a CUDA-alike device or as XPU. +DEVICE = current_platform.device_type + +pytestmark = pytest.mark.skipif( + not (current_platform.is_cuda_alike() or current_platform.is_xpu()), + reason="Mamba2 SSD Triton kernels require a CUDA-alike or XPU device.", +) + +# Added by the IBM Team, 2024 + +# Adapted from https://github.com/state-spaces/mamba/blob/v2.2.4/mamba_ssm/modules/ssd_minimal.py + + +# this is the segsum implementation taken from above +def segsum(x): + """Calculates segment sum.""" + T = x.size(-1) + x = repeat(x, "... d -> ... d e", e=T) + mask = torch.tril(torch.ones(T, T, device=x.device, dtype=bool), diagonal=-1) + x = x.masked_fill(~mask, 0) + x_segsum = torch.cumsum(x, dim=-2) + mask = torch.tril(torch.ones(T, T, device=x.device, dtype=bool), diagonal=0) + x_segsum = x_segsum.masked_fill(~mask, -torch.inf) + return x_segsum + + +def ssd_minimal_discrete(X, A, B, C, block_len, initial_states=None): + """ + Arguments: + X: (batch, length, n_heads, d_head) + A: (batch, length, n_heads) + B: (batch, length, n_heads, d_state) + C: (batch, length, n_heads, d_state) + Return: + Y: (batch, length, n_heads, d_head) + """ + assert X.dtype == A.dtype == B.dtype == C.dtype + assert X.shape[1] % block_len == 0 + + # Rearrange into blocks/chunks + X, A, B, C = ( + rearrange(x, "b (c l) ... -> b c l ...", l=block_len) for x in (X, A, B, C) + ) + + A = rearrange(A, "b c l h -> b h c l") + A_cumsum = torch.cumsum(A, dim=-1) + + # 1. Compute the output for each intra-chunk (diagonal blocks) + L = torch.exp(segsum(A)) + Y_diag = torch.einsum("bclhn,bcshn,bhcls,bcshp->bclhp", C, B, L, X) + + # 2. Compute the state for each intra-chunk + # (right term of low-rank factorization of off-diagonal blocks; B terms) + decay_states = torch.exp(A_cumsum[:, :, :, -1:] - A_cumsum) + states = torch.einsum("bclhn,bhcl,bclhp->bchpn", B, decay_states, X) + + # 3. Compute the inter-chunk SSM recurrence; produces correct SSM states at + # chunk boundaries + # (middle term of factorization of off-diag blocks; A terms) + if initial_states is None: + initial_states = torch.zeros_like(states[:, :1]) + states = torch.cat([initial_states, states], dim=1) + decay_chunk = torch.exp(segsum(F.pad(A_cumsum[:, :, :, -1], (1, 0)))) + new_states = torch.einsum("bhzc,bchpn->bzhpn", decay_chunk, states) + states, final_state = new_states[:, :-1], new_states[:, -1] + + # 4. Compute state -> output conversion per chunk + # (left term of low-rank factorization of off-diagonal blocks; C terms) + state_decay_out = torch.exp(A_cumsum) + Y_off = torch.einsum("bclhn,bchpn,bhcl->bclhp", C, states, state_decay_out) + + # Add output of intra-chunk and inter-chunk terms + # (diagonal and off-diagonal blocks) + Y = rearrange(Y_diag + Y_off, "b c l h p -> b (c l) h p") + return Y, final_state + + +def generate_random_inputs(batch_size, seqlen, n_heads, d_head, itype, device=DEVICE): + set_random_seed(0) + A = -torch.exp(torch.rand(n_heads, dtype=itype, device=device)) + dt = F.softplus( + torch.randn(batch_size, seqlen, n_heads, dtype=itype, device=device) - 4 + ) + X = torch.randn((batch_size, seqlen, n_heads, d_head), dtype=itype, device=device) + B = torch.randn((batch_size, seqlen, n_heads, d_head), dtype=itype, device=device) + C = torch.randn((batch_size, seqlen, n_heads, d_head), dtype=itype, device=device) + + return A, dt, X, B, C + + +def generate_continuous_batched_examples( + example_lens_by_batch, + num_examples, + full_length, + last_taken, + exhausted, + n_heads, + d_head, + itype, + device=DEVICE, + return_naive_ref=True, +): + # this function generates a random examples of certain length + # and then cut according to "example_lens_by_batch" and feed + # them in continuous batches to the kernels. + # If if return_naive_ref=True, the naive torch implementation + # ssd_minimal_discrete will be used to compute and return + # reference output. + + # generate the full-length example + A, dt, X, B, C = generate_random_inputs( + num_examples, full_length, n_heads, d_head, itype + ) + + if return_naive_ref: + Y_min, final_state_min = ssd_minimal_discrete( + X * dt.unsqueeze(-1), A * dt, B, C, block_len=full_length // 4 + ) + + # internal function that outputs a cont batch of examples + # given a tuple of lengths for each example in the batch + # e.g., example_lens=(8, 4) means take 8 samples from first eg, + # 4 examples from second eg, etc + def get_continuous_batch(example_lens: tuple[int, ...]): + indices = [] + for i, x in enumerate(example_lens): + c = last_taken.get(i, 0) + indices.append((c, c + x)) + last_taken[i] = (c + x) % full_length + exhausted[i] = last_taken[i] == 0 + + return ( + torch.concat([x[i, s:e] for i, (s, e) in enumerate(indices)]).unsqueeze(0) + for x in (dt, X, B, C) + ) + + # internal function that maps "n" to the appropriate right boundary + # value when forming continuous batches from examples of length given + # by "full_length". + # - e.g., when n > full_length, returns n % full_length + # when n == full_length, returns full_length + def end_boundary(n: int): + return n - ((n - 1) // full_length) * full_length + + IND_E = None + for spec in example_lens_by_batch: + # get the (maybe partial) example seen in this cont batch + dt2, X2, B2, C2 = get_continuous_batch(spec) + + # get the metadata + cu_seqlens = torch.tensor((0,) + spec, device=device).cumsum(dim=0) + seq_idx = torch.zeros( + cu_seqlens[-1], dtype=torch.int32, device=cu_seqlens.device + ) + for i, (srt, end) in enumerate( + zip( + cu_seqlens, + cu_seqlens[1:], + ) + ): + seq_idx[srt:end] = i + + # for cont batch + if IND_E is None: + IND_S = [0 for _ in range(len(spec))] + else: + IND_S = [x % full_length for x in IND_E] + IND_E = [end_boundary(x + y) for x, y in zip(IND_S, spec)] + + # varlen has implicit batch=1 + dt2 = dt2.squeeze(0) + X2 = X2.squeeze(0) + B2 = B2.squeeze(0) + C2 = C2.squeeze(0) + yield ( + [Y_min[s, IND_S[s] : IND_E[s]] for s in range(num_examples)] + if return_naive_ref + else None, + cu_seqlens, + seq_idx, + (A, dt2, X2, B2, C2), + ) + + +@pytest.mark.parametrize("itype", [torch.float32, torch.bfloat16]) +@pytest.mark.parametrize("n_heads", [4, 16, 32]) +@pytest.mark.parametrize("d_head", [5, 8, 32, 128]) +@pytest.mark.parametrize("seq_len_chunk_size", [(112, 16), (128, 32)]) +def test_mamba_chunk_scan_single_example(d_head, n_heads, seq_len_chunk_size, itype): + # this tests the kernels on a single example (bs=1) + + # TODO: the bfloat16 case requires higher thresholds. To be investigated + + if itype == torch.bfloat16: + atol, rtol = 5e-2, 5e-2 + else: + atol, rtol = 8e-3, 5e-3 + + # set seed + batch_size = 1 # batch_size + # ssd_minimal_discrete requires chunk_size divide seqlen + # - this is only required for generating the reference seqs, + # it is not an operational limitation. + seqlen, chunk_size = seq_len_chunk_size + + A, dt, X, B, C = generate_random_inputs(batch_size, seqlen, n_heads, d_head, itype) + + Y_min, final_state_min = ssd_minimal_discrete( + X * dt.unsqueeze(-1), A * dt, B, C, chunk_size + ) + + cu_seqlens = torch.tensor((0, seqlen), device=DEVICE).cumsum(dim=0) + cu_chunk_seqlens, last_chunk_indices, seq_idx_chunks = ( + compute_varlen_chunk_metadata(cu_seqlens, chunk_size) + ) + # varlen has implicit batch=1 + X = X.squeeze(0) + dt = dt.squeeze(0) + A = A.squeeze(0) + B = B.squeeze(0) + C = C.squeeze(0) + Y = torch.empty_like(X) + final_state = mamba_chunk_scan_combined_varlen( + X, + dt, + A, + B, + C, + chunk_size, + cu_seqlens=cu_seqlens.to(torch.int32), + cu_chunk_seqlens=cu_chunk_seqlens, + last_chunk_indices=last_chunk_indices, + seq_idx=seq_idx_chunks, + out=Y, + D=None, + ) + + # just test the last in sequence + torch.testing.assert_close(Y[-1], Y_min[0, -1], atol=atol, rtol=rtol) + + # just test the last head + # NOTE, in the kernel we always cast states to fp32 + torch.testing.assert_close( + final_state[:, -1].to(torch.float32), + final_state_min[:, -1].to(torch.float32), + atol=atol, + rtol=rtol, + ) + + +@pytest.mark.parametrize("itype", [torch.float32]) +@pytest.mark.parametrize("n_heads", [4, 8]) +@pytest.mark.parametrize("d_head", [5, 16, 32]) +@pytest.mark.parametrize( + "seq_len_chunk_size_cases", + [ + # small-ish chunk_size (8) + (64, 8, 2, [(64, 32), (64, 32)]), + (64, 8, 2, [(8, 8), (8, 8), (8, 8)]), # chunk size boundary + ( + 64, + 8, + 2, + [(4, 4), (4, 4), (4, 4), (4, 4)], + ), # chunk_size larger than cont batches + (64, 8, 5, [(64, 32, 16, 8, 8)]), + # large-ish chunk_size (256) + (64, 256, 1, [(5,), (1,), (1,), (1,)]), # irregular sizes with small sequences + ( + 64, + 256, + 2, + [(5, 30), (1, 2), (1, 2), (1, 2)], + ), # irregular sizes with small sequences + # we also need to test some large seqlen + # to catch errors with init states decay + (768, 128, 2, [(138, 225), (138, 225)]), + ], +) +def test_mamba_chunk_scan_cont_batch(d_head, n_heads, seq_len_chunk_size_cases, itype): + # this test with multiple examples in a continuous batch + # (i.e. chunked prefill) + + seqlen, chunk_size, num_examples, cases = seq_len_chunk_size_cases + + # This test can have larger error for longer sequences + if seqlen > 256: + atol, rtol = 1e-2, 5e-3 + else: + atol, rtol = 5e-3, 5e-3 + + # hold state during the cutting process so we know if an + # example has been exhausted and needs to cycle + last_taken: dict = {} # map: eg -> pointer to last taken sample + exhausted: dict = {} # map: eg -> boolean indicating example is exhausted + + states = None + for Y_min, cu_seqlens, _token_seq_idx, ( + A, + dt, + X, + B, + C, + ) in generate_continuous_batched_examples( + cases, num_examples, seqlen, last_taken, exhausted, n_heads, d_head, itype + ): + cu_chunk_seqlens, last_chunk_indices, seq_idx_chunks = ( + compute_varlen_chunk_metadata(cu_seqlens, chunk_size) + ) + + Y = torch.empty_like(X) + new_states = mamba_chunk_scan_combined_varlen( + X, + dt, + A, + B, + C, + chunk_size, + cu_seqlens=cu_seqlens.to(torch.int32), + cu_chunk_seqlens=cu_chunk_seqlens, + last_chunk_indices=last_chunk_indices, + seq_idx=seq_idx_chunks, + out=Y, + D=None, + initial_states=states, + ) + + # just test the last in sequence + for i in range(num_examples): + # just test one dim and dstate + Y_eg = Y[cu_seqlens[i] : cu_seqlens[i + 1], 0, 0] + Y_min_eg = Y_min[i][:, 0, 0] + torch.testing.assert_close(Y_eg, Y_min_eg, atol=atol, rtol=rtol) + + # update states + states = new_states + for i, clear in exhausted.items(): + if clear: + states[i].fill_(0.0) + exhausted[i] = False + + +@pytest.mark.parametrize("chunk_size", [8, 256]) +@pytest.mark.parametrize( + "seqlens", + [(16, 20), (270, 88, 212, 203)], +) +def test_mamba_chunk_scan_cont_batch_prefill_chunking(chunk_size, seqlens): + # This test verifies the correctness of the chunked prefill implementation + # in the mamba2 ssd kernels, by comparing concatenation (in the sequence + # dimension) of chunked results with the full sequence result. + # It is different from test_mamba_chunk_scan_cont_batch by: + # 1. Not using the naive torch implementation (ssd_minimal_discrete) to get + # reference outputs. Instead, it compares chunked kernel outputs to full + # sequence kernel outputs. This is the most straightforward way to + # assert chunked prefill correctness. + # 2. It focuses on cases where sequences change in the middle of mamba + # chunks, and not necessarily on chunk boundaries. + + max_seqlen = max(seqlens) + # This test can have larger error for longer sequences + if max_seqlen > 256: + atol, rtol = 1e-2, 5e-3 + else: + atol, rtol = 5e-3, 5e-3 + + num_sequences = len(seqlens) + n_heads = 16 + d_head = 64 + itype = torch.float32 + + # hold state during the cutting process so we know if an + # example has been exhausted and needs to cycle + last_taken: dict = {} # map: eg -> pointer to last taken sample + exhausted: dict = {} # map: eg -> boolean indicating example is exhausted + _, cu_seqlens, seq_idx, (A, dt, X, B, C) = next( + generate_continuous_batched_examples( + [seqlens], + num_sequences, + max_seqlen, + last_taken, + exhausted, + n_heads, + d_head, + itype, + return_naive_ref=False, + ) + ) + seqlens = torch.tensor(seqlens, dtype=torch.int32, device=X.device) + device = X.device + + ## full seqlen computation + cu_chunk_seqlens, last_chunk_indices, seq_idx_chunks = ( + compute_varlen_chunk_metadata(cu_seqlens, chunk_size) + ) + Y_ref = torch.empty_like(X) + state_ref = mamba_chunk_scan_combined_varlen( + X, + dt, + A, + B, + C, + chunk_size, + cu_seqlens=cu_seqlens.to(torch.int32), + cu_chunk_seqlens=cu_chunk_seqlens, + last_chunk_indices=last_chunk_indices, + seq_idx=seq_idx_chunks, + out=Y_ref, + D=None, + initial_states=None, + ) + + ## chunked seqlen computation + # first chunk + chunked_seqlens = seqlens // 2 + chunked_cu_seqlens = torch.cat( + [torch.tensor([0], device=device), torch.cumsum(chunked_seqlens, dim=0)], dim=0 + ) + chunked_input_seq_len = chunked_cu_seqlens[-1] + X_chunked = torch.zeros_like(X)[:chunked_input_seq_len, ...] + dt_chunked = torch.zeros_like(dt)[:chunked_input_seq_len, ...] + B_chunked = torch.zeros_like(B)[:chunked_input_seq_len, ...] + C_chunked = torch.zeros_like(C)[:chunked_input_seq_len, ...] + for i in range(num_sequences): + chunk_f = lambda x, i: x[ + cu_seqlens[i] : cu_seqlens[i] + chunked_seqlens[i], ... + ] + + X_chunked[chunked_cu_seqlens[i] : chunked_cu_seqlens[i + 1], ...] = chunk_f( + X, i + ) + dt_chunked[chunked_cu_seqlens[i] : chunked_cu_seqlens[i + 1], ...] = chunk_f( + dt, i + ) + B_chunked[chunked_cu_seqlens[i] : chunked_cu_seqlens[i + 1], ...] = chunk_f( + B, i + ) + C_chunked[chunked_cu_seqlens[i] : chunked_cu_seqlens[i + 1], ...] = chunk_f( + C, i + ) + + cu_chunk_seqlens, last_chunk_indices, seq_idx_chunks = ( + compute_varlen_chunk_metadata(chunked_cu_seqlens, chunk_size) + ) + Y_partial = torch.empty_like(X_chunked) + partial_state = mamba_chunk_scan_combined_varlen( + X_chunked, + dt_chunked, + A, + B_chunked, + C_chunked, + chunk_size, + cu_seqlens=chunked_cu_seqlens.to(torch.int32), + cu_chunk_seqlens=cu_chunk_seqlens, + last_chunk_indices=last_chunk_indices, + seq_idx=seq_idx_chunks, + out=Y_partial, + D=None, + initial_states=None, + ) + + # remaining chunk + remaining_chunked_seqlens = seqlens - chunked_seqlens + remaining_chunked_cu_seqlens = torch.cat( + [ + torch.tensor([0], device=device), + torch.cumsum(remaining_chunked_seqlens, dim=0), + ], + dim=0, + ) + remaining_chunked_input_seq_len = remaining_chunked_cu_seqlens[-1] + remaining_X_chunked = torch.zeros_like(X)[:remaining_chunked_input_seq_len, ...] + remaining_dt_chunked = torch.zeros_like(dt)[:remaining_chunked_input_seq_len, ...] + remaining_B_chunked = torch.zeros_like(B)[:remaining_chunked_input_seq_len, ...] + remaining_C_chunked = torch.zeros_like(C)[:remaining_chunked_input_seq_len, ...] + for i in range(num_sequences): + remaining_chunk_f = lambda x, i: x[ + cu_seqlens[i] + chunked_seqlens[i] : cu_seqlens[i + 1], ... + ] + + remaining_X_chunked[ + remaining_chunked_cu_seqlens[i] : remaining_chunked_cu_seqlens[i + 1], ... + ] = remaining_chunk_f(X, i) + remaining_dt_chunked[ + remaining_chunked_cu_seqlens[i] : remaining_chunked_cu_seqlens[i + 1], ... + ] = remaining_chunk_f(dt, i) + remaining_B_chunked[ + remaining_chunked_cu_seqlens[i] : remaining_chunked_cu_seqlens[i + 1], ... + ] = remaining_chunk_f(B, i) + remaining_C_chunked[ + remaining_chunked_cu_seqlens[i] : remaining_chunked_cu_seqlens[i + 1], ... + ] = remaining_chunk_f(C, i) + + # assert input chunking is correct + concat_chunk_f = lambda pt1, pt2, i: torch.cat( + [ + pt1[chunked_cu_seqlens[i] : chunked_cu_seqlens[i + 1], ...], + pt2[ + remaining_chunked_cu_seqlens[i] : remaining_chunked_cu_seqlens[i + 1], + ..., + ], + ], + dim=0, + ) + concat_batch_f = lambda pt1, pt2: torch.cat( + [concat_chunk_f(pt1, pt2, i) for i in range(num_sequences)], dim=0 + ) + + assert concat_batch_f(X_chunked, remaining_X_chunked).equal(X) + assert concat_batch_f(dt_chunked, remaining_dt_chunked).equal(dt) + assert concat_batch_f(B_chunked, remaining_B_chunked).equal(B) + assert concat_batch_f(C_chunked, remaining_C_chunked).equal(C) + + cu_chunk_seqlens, last_chunk_indices, seq_idx_chunks = ( + compute_varlen_chunk_metadata(remaining_chunked_cu_seqlens, chunk_size) + ) + + Y_chunked = torch.empty_like(remaining_X_chunked) + state_chunked = mamba_chunk_scan_combined_varlen( + remaining_X_chunked, + remaining_dt_chunked, + A, + remaining_B_chunked, + remaining_C_chunked, + chunk_size, + cu_seqlens=remaining_chunked_cu_seqlens.to(torch.int32), + cu_chunk_seqlens=cu_chunk_seqlens, + last_chunk_indices=last_chunk_indices, + seq_idx=seq_idx_chunks, + out=Y_chunked, + D=None, + initial_states=partial_state, + ) + Y = concat_batch_f(Y_partial, Y_chunked) + + # kernel chunked is same as kernel overall + for i in range(num_sequences): + Y_seq = Y[cu_seqlens[i] : cu_seqlens[i + 1], ...] + Y_ref_seq = Y_ref[cu_seqlens[i] : cu_seqlens[i + 1], ...] + torch.testing.assert_close( + Y_seq[: chunked_seqlens[i], ...], + Y_ref_seq[: chunked_seqlens[i], ...], + atol=atol, + rtol=rtol, + msg=lambda x, i=i: f"seq{i} output part1 " + x, + ) + torch.testing.assert_close( + Y_seq[chunked_seqlens[i] :, ...], + Y_ref_seq[chunked_seqlens[i] :, ...], + atol=atol, + rtol=rtol, + msg=lambda x, i=i: f"seq{i} output part2 " + x, + ) + + state_seq = state_chunked[i] + state_seq_ref = state_ref[i] + torch.testing.assert_close( + state_seq, + state_seq_ref, + atol=atol, + rtol=rtol, + msg=lambda x, i=i: f"seq{i} state " + x, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/mamba/test_precopy_mamba_align.py b/asian_ai/performance/vLLM/tests/kernels/mamba/test_precopy_mamba_align.py new file mode 100644 index 0000000000000000000000000000000000000000..823d7f8ed9aad6a9454d1d5c1deda004ba6affa0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/mamba/test_precopy_mamba_align.py @@ -0,0 +1,180 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Equivalence test for ``precopy_mamba_align_fused_kernel``. + +The V2 "align" pre-copy must migrate mamba state across block boundaries with +byte-identical semantics to the V1 copy specs (``get_conv_copy_spec`` / +``get_temporal_copy_spec``): + +* conv state (SD layout, conv_width > 0): shift the sliding window by + ``token_bias`` tokens -- ``state[bt[src_col], token_bias:]`` -> + ``state[bt[dst_col], :conv_width - token_bias]``. +* temporal state (conv_width == 0): ``token_bias`` selects the accepted + speculative column -- ``state[bt[src_col + token_bias]]`` -> + ``state[bt[dst_col]]``. + +The kernel must also no-op when ``src_col < 0`` (fresh request) or +``src_col == dst_col`` (no boundary crossed). +""" + +from __future__ import annotations + +import torch + +from vllm.platforms import current_platform +from vllm.v1.worker.mamba_utils import precopy_mamba_align_fused_kernel + +try: + import pytest + + pytestmark = pytest.mark.skipif( + not current_platform.is_cuda(), + reason="precopy_mamba_align_fused_kernel needs CUDA/Triton", + ) + _parametrize = pytest.mark.parametrize +except ModuleNotFoundError: # allow running directly as ``python `` + pytest = None + + def _parametrize(_name, _values): + def _deco(fn): + return fn + + return _deco + + +NUM_LAYERS = 3 +CONV_WIDTH = 4 # conv_kernel - 1 + num_spec +CONV_DIM = 96 +SSM_SHAPE = (4, 16, 16) +MAX_COLS = 8 + + +def _build_state(num_blocks, device): + """Per-layer (conv SD [nb, width, dim] bf16, ssm [nb, *shape] fp32) pools.""" + convs, ssms = [], [] + for _ in range(NUM_LAYERS): + convs.append( + torch.randn( + num_blocks, CONV_WIDTH, CONV_DIM, dtype=torch.bfloat16, device=device + ) + ) + ssms.append( + torch.randn(num_blocks, *SSM_SHAPE, dtype=torch.float32, device=device) + ) + return convs, ssms + + +def _build_meta(convs, ssms, device): + """Flattened per-(layer, state-type) metadata, ordered conv, ssm per layer.""" + n = NUM_LAYERS * 2 + base = torch.zeros(n, dtype=torch.int64, device=device) + blk_stride = torch.zeros(n, dtype=torch.int64, device=device) + elem = torch.zeros(n, dtype=torch.int32, device=device) + inner = torch.zeros(n, dtype=torch.int64, device=device) + width = torch.zeros(n, dtype=torch.int32, device=device) + group = torch.zeros(n, dtype=torch.int32, device=device) + drc = torch.zeros(n, dtype=torch.int32, device=device) # DS rows (unused, SD) + drs = torch.zeros(n, dtype=torch.int64, device=device) + i = 0 + for layer in range(NUM_LAYERS): + conv, ssm = convs[layer], ssms[layer] + # conv (SD): width = size(1), inner = stride(1) + base[i] = conv.data_ptr() + blk_stride[i] = conv.stride(0) * conv.element_size() + elem[i] = conv.element_size() + width[i] = conv.size(1) + inner[i] = conv.stride(1) + i += 1 + # ssm (temporal): width = 0, inner = elems per block + base[i] = ssm.data_ptr() + blk_stride[i] = ssm.stride(0) * ssm.element_size() + elem[i] = ssm.element_size() + width[i] = 0 + inner[i] = ssm[0].numel() + i += 1 + return base, blk_stride, elem, inner, width, group, drc, drs + + +def _reference(convs, ssms, bt, src_col, dst_col, bias, num_reqs): + """Apply the V1 copy semantics on clones, reading from the pre-copy state.""" + conv_pre = [c.clone() for c in convs] + ssm_pre = [s.clone() for s in ssms] + conv_ref = [c.clone() for c in convs] + ssm_ref = [s.clone() for s in ssms] + for r in range(num_reqs): + sc, dc, tb = int(src_col[r]), int(dst_col[r]), int(bias[r]) + if sc < 0 or sc == dc: + continue + sblk, dblk = int(bt[r, sc]), int(bt[r, dc]) + tblk = int(bt[r, sc + tb]) # temporal src column shifted by bias + for layer in range(NUM_LAYERS): + conv_ref[layer][dblk, : CONV_WIDTH - tb] = conv_pre[layer][sblk, tb:] + ssm_ref[layer][dblk] = ssm_pre[layer][tblk] + return conv_ref, ssm_ref + + +@_parametrize("num_reqs", [1, 4, 16]) +@_parametrize("token_bias", [0, 1, 2]) +def test_precopy_matches_v1_copy_specs(num_reqs, token_bias): + device = torch.device("cuda") + torch.manual_seed(0) + # Distinct physical block per (req, col) so copies never alias. + num_blocks = num_reqs * MAX_COLS + 1 + bt = torch.empty(num_reqs, MAX_COLS, dtype=torch.int32, device=device) + for r in range(num_reqs): + bt[r] = torch.arange( + 1 + r * MAX_COLS, 1 + (r + 1) * MAX_COLS, dtype=torch.int32, device=device + ) + + # Per-req columns: req 0 fresh (src=-1, skip), req 1 same block (skip), + # the rest cross from col 1 -> col 0 with the given spec token bias. + src_col = torch.full((num_reqs,), 1, dtype=torch.int32, device=device) + dst_col = torch.zeros(num_reqs, dtype=torch.int32, device=device) + bias = torch.full((num_reqs,), token_bias, dtype=torch.int32, device=device) + if num_reqs >= 1: + src_col[0] = -1 # fresh -> no copy + if num_reqs >= 2: + dst_col[1] = 1 # src_col == dst_col -> no copy + + convs, ssms = _build_state(num_blocks, device) + conv_ref, ssm_ref = _reference( + convs, ssms, bt.cpu(), src_col.cpu(), dst_col.cpu(), bias.cpu(), num_reqs + ) + + base, blk_stride, elem, inner, width, group, drc, drs = _build_meta( + convs, ssms, device + ) + bt_ptrs = torch.tensor([bt.data_ptr()], dtype=torch.int64, device=device) + idx_mapping = torch.arange(num_reqs, dtype=torch.int32, device=device) + grid = (num_reqs, NUM_LAYERS * 2) + precopy_mamba_align_fused_kernel[grid]( + dst_col, + src_col, + bias, + bt_ptrs, + bt.stride(0), + base, + blk_stride, + elem, + inner, + width, + group, + drc, + drs, + idx_mapping, + num_reqs, + COPY_BLOCK_SIZE=1024, + CONV_STATE_DIM_FIRST=False, + ) + torch.accelerator.synchronize() + + for layer in range(NUM_LAYERS): + torch.testing.assert_close(convs[layer], conv_ref[layer], rtol=0, atol=0) + torch.testing.assert_close(ssms[layer], ssm_ref[layer], rtol=0, atol=0) + + +if __name__ == "__main__": + for nr in (1, 4, 16): + for tb in (0, 1, 2): + test_precopy_matches_v1_copy_specs(nr, tb) + print(f"OK num_reqs={nr} token_bias={tb}") diff --git a/asian_ai/performance/vLLM/tests/kernels/mamba/test_ssu_dispatch.py b/asian_ai/performance/vLLM/tests/kernels/mamba/test_ssu_dispatch.py new file mode 100644 index 0000000000000000000000000000000000000000..f65a1d247fece23663ae0433447182c010211580 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/mamba/test_ssu_dispatch.py @@ -0,0 +1,144 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm.config.mamba import MambaBackendEnum, MambaConfig +from vllm.model_executor.layers.mamba.ops.ssu_dispatch import ( + FlashInferSSUBackend, + TritonSSUBackend, + get_mamba_ssu_backend, + initialize_mamba_ssu_backend, + selective_state_update, +) +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.attention.backends.registry import MambaAttentionBackendEnum +from vllm.v1.kv_cache_interface import ( + KVCacheConfig, + KVCacheGroupSpec, + MambaSpec, +) + +try: + import flashinfer.mamba # noqa: F401 + + HAS_FLASHINFER = True +except ImportError: + HAS_FLASHINFER = False + + +def _kv_cache_config_with_ssu( + mamba_type: MambaAttentionBackendEnum = MambaAttentionBackendEnum.MAMBA2, +) -> KVCacheConfig: + spec = MambaSpec( + block_size=16, + shapes=((16, 64),), + dtypes=(torch.float16,), + mamba_type=mamba_type, + ) + return KVCacheConfig( + num_blocks=1, + kv_cache_tensors=[], + kv_cache_groups=[KVCacheGroupSpec(layer_names=["l0"], kv_cache_spec=spec)], + ) + + +def test_default_backend_is_triton(): + initialize_mamba_ssu_backend(MambaConfig(), _kv_cache_config_with_ssu()) + backend = get_mamba_ssu_backend() + assert isinstance(backend, TritonSSUBackend) + assert backend.name == "triton" + + +def test_explicit_triton_backend(): + initialize_mamba_ssu_backend( + MambaConfig(backend=MambaBackendEnum.TRITON), _kv_cache_config_with_ssu() + ) + backend = get_mamba_ssu_backend() + assert isinstance(backend, TritonSSUBackend) + + +@pytest.mark.skipif(not HAS_FLASHINFER, reason="flashinfer not installed") +def test_flashinfer_backend_init(): + initialize_mamba_ssu_backend( + MambaConfig(backend=MambaBackendEnum.FLASHINFER), _kv_cache_config_with_ssu() + ) + backend = get_mamba_ssu_backend() + assert isinstance(backend, FlashInferSSUBackend) + assert backend.name == "flashinfer" + + +def test_uninitialized_backend_raises(): + import vllm.model_executor.layers.mamba.ops.ssu_dispatch as mod + + old = mod._mamba_ssu_backend + mod._mamba_ssu_backend = None + with pytest.raises(RuntimeError, match="not been initialized"): + get_mamba_ssu_backend() + mod._mamba_ssu_backend = old + + +@pytest.mark.parametrize( + "mamba_type", + [ + MambaAttentionBackendEnum.LINEAR, + MambaAttentionBackendEnum.GDN_ATTN, + MambaAttentionBackendEnum.SHORT_CONV, + ], +) +def test_init_is_noop_for_non_ssu_mamba_type(mamba_type): + import vllm.model_executor.layers.mamba.ops.ssu_dispatch as mod + + old = mod._mamba_ssu_backend + mod._mamba_ssu_backend = None + try: + initialize_mamba_ssu_backend( + MambaConfig(), _kv_cache_config_with_ssu(mamba_type) + ) + assert mod._mamba_ssu_backend is None + with pytest.raises(RuntimeError, match="not been initialized"): + get_mamba_ssu_backend() + finally: + mod._mamba_ssu_backend = old + + +@pytest.mark.skipif(HAS_FLASHINFER, reason="flashinfer is installed") +def test_flashinfer_import_error(): + with pytest.raises(ImportError, match="FlashInfer is required"): + FlashInferSSUBackend(MambaConfig()) + + +def test_triton_basic_call(): + set_random_seed(0) + initialize_mamba_ssu_backend( + MambaConfig(backend=MambaBackendEnum.TRITON), _kv_cache_config_with_ssu() + ) + device = "cuda" + batch_size = 2 + dim = 64 + dstate = 16 + + state = torch.randn(batch_size, dim, dstate, device=device) + x = torch.randn(batch_size, dim, device=device) + out = torch.empty_like(x) + dt = torch.randn(batch_size, dim, device=device) + dt_bias = torch.rand(dim, device=device) - 4.0 + A = -torch.rand(dim, dstate, device=device) + B = torch.randn(batch_size, dstate, device=device) + C = torch.randn(batch_size, dstate, device=device) + D = torch.randn(dim, device=device) + + selective_state_update( + state, + x, + dt, + A, + B, + C, + D=D, + dt_bias=dt_bias, + dt_softplus=True, + out=out, + ) + assert not torch.isnan(out).any() diff --git a/asian_ai/performance/vLLM/tests/kernels/mamba/utils.py b/asian_ai/performance/vLLM/tests/kernels/mamba/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..280d272d442567ef8e3aa8b4d24047ee82994473 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/mamba/utils.py @@ -0,0 +1,78 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import torch +import torch.nn.functional as F +from einops import rearrange, repeat + + +def selective_state_update_ref( + state, x, dt, A, B, C, D=None, z=None, dt_bias=None, dt_softplus=False +): + """ + Argument: + state: (batch, dim, dstate) or (batch, nheads, dim, dstate) + x: (batch, dim) or (batch, nheads, dim) + dt: (batch, dim) or (batch, nheads, dim) + A: (dim, dstate) or (nheads, dim, dstate) + B: (batch, dstate) or (batch, ngroups, dstate) + C: (batch, dstate) or (batch, ngroups, dstate) + D: (dim,) or (nheads, dim) + z: (batch, dim) or (batch, nheads, dim) + dt_bias: (dim,) or (nheads, dim) + Return: + out: (batch, dim) or (batch, nheads, dim) + """ + has_heads = state.dim() > 3 + if state.dim() == 3: + state = state.unsqueeze(1) + if x.dim() == 2: + x = x.unsqueeze(1) + if dt.dim() == 2: + dt = dt.unsqueeze(1) + if A.dim() == 2: + A = A.unsqueeze(0) + if B.dim() == 2: + B = B.unsqueeze(1) + if C.dim() == 2: + C = C.unsqueeze(1) + if D is not None and D.dim() == 1: + D = D.unsqueeze(0) + if z is not None and z.dim() == 2: + z = z.unsqueeze(1) + if dt_bias is not None and dt_bias.dim() == 1: + dt_bias = dt_bias.unsqueeze(0) + batch, nheads, dim, dstate = state.shape + assert x.shape == (batch, nheads, dim) + assert dt.shape == x.shape + assert A.shape == (nheads, dim, dstate) + ngroups = B.shape[1] + assert nheads % ngroups == 0, "nheads must be divisible by ngroups" + assert B.shape == (batch, ngroups, dstate) + assert C.shape == B.shape + if D is not None: + assert D.shape == (nheads, dim) + if z is not None: + assert z.shape == x.shape + if dt_bias is not None: + assert dt_bias.shape == (nheads, dim) + dt = dt + dt_bias + dt = F.softplus(dt) if dt_softplus else dt + dA = torch.exp( + rearrange(dt, "b h d -> b h d 1") * A + ) # (batch, nheads, dim, dstate) + B = repeat(B, "b g n -> b (g h) n", h=nheads // ngroups) # (batch, nheads, dstate) + C = repeat(C, "b g n -> b (g h) n", h=nheads // ngroups) # (batch, nheads, dstate) + dB = rearrange(dt, "b h d -> b h d 1") * rearrange( + B, "b h n -> b h 1 n" + ) # (batch, nheads, dim, dstate) + state.copy_( + state * dA + dB * rearrange(x, "b h d -> b h d 1") + ) # (batch, dim, dstate + out = torch.einsum("bhdn,bhn->bhd", state.to(C.dtype), C) + if D is not None: + out += (x * D).to(out.dtype) + out = (out if z is None else out * F.silu(z)).to(x.dtype) + if not has_heads: + out = out.squeeze(1) + return out diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/__init__.py b/asian_ai/performance/vLLM/tests/kernels/moe/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/conftest.py b/asian_ai/performance/vLLM/tests/kernels/moe/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..930deee087e5df3b0f27e184f7ef5e125d40837c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/conftest.py @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest + + +def pytest_addoption(parser): + parser.addoption( + "--subtests", action="store", type=str, default=None, help="subtest ids" + ) + + +@pytest.fixture +def subtests(request): + return request.config.getoption("--subtests") diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/__init__.py b/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/cli_args.py b/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/cli_args.py new file mode 100644 index 0000000000000000000000000000000000000000..93bcf389f4b4166c67e2fa22307f84eab573adf2 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/cli_args.py @@ -0,0 +1,158 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import argparse + +import torch + +import vllm.model_executor.layers.fused_moe.modular_kernel as mk +from vllm.model_executor.layers.fused_moe.config import FusedMoEQuantConfig + +from .common import Config +from .mk_objects import ( + MK_ALL_PREPARE_FINALIZE_TYPES, + MK_FUSED_EXPERT_TYPES, + MK_SINGLE_GPU_PREPARE_FINALIZE_TYPES, +) + + +def make_config_arg_parser(description: str): + def to_pf_class_type(s: str) -> mk.FusedMoEPrepareAndFinalizeModular: + for pf in MK_ALL_PREPARE_FINALIZE_TYPES: + if pf.__name__ == s: + return pf + raise ValueError(f"Cannot find a PrepareFinalize type that matches {s}") + + def to_experts_class_type(s: str) -> mk.FusedMoEExpertsModular: + for fe in MK_FUSED_EXPERT_TYPES: + if fe.__name__ == s: + return fe + raise ValueError(f"Cannot find a FusedExperts type that matches {s}") + + def to_quant_torch_dtype(s: str) -> torch.dtype: + if s == "torch.float8_e4m3fn": + return torch.float8_e4m3fn + raise ValueError(f"Unsupported quant type {s}") + + parser = argparse.ArgumentParser(description=description) + + parser.add_argument( + "--world-size", + type=int, + default=2, + help="Number of ranks that participate in all2all", + ) + parser.add_argument( + "--pf-type", + type=to_pf_class_type, + required=True, + help=( + "Choose a PrepareFinalize Type : " + f"{[x.__name__ for x in MK_ALL_PREPARE_FINALIZE_TYPES]}" + ), + ) + parser.add_argument( + "--experts-type", + type=to_experts_class_type, + required=True, + help=( + f"Choose a FusedExpert type : {[x.__name__ for x in MK_FUSED_EXPERT_TYPES]}" + ), + ) + parser.add_argument( + "-m", + nargs="+", + type=int, + default=[64], + help="num tokens per rank", + ) + parser.add_argument( + "-k", + type=int, + default=7168, + help="hidden-size", + ) + parser.add_argument( + "-n", + type=int, + default=1024, + help="N dimension of the first fused-moe matmul", + ) + parser.add_argument( + "--num-experts", type=int, default=32, help="Global num experts" + ) + parser.add_argument("--topk", nargs="+", type=int, default=[4, 1], help="num topk") + + # Quant args + parser.add_argument( + "--quant-dtype", type=to_quant_torch_dtype, help="Quant datatype" + ) + parser.add_argument( + "--per-token-quantized-activations", + action="store_true", + help=("The input activations must be per-token quantized"), + ) + parser.add_argument( + "--per-channel-quantized-weights", + action="store_true", + help="The weights must be per-channel quantized.", + ) + parser.add_argument( + "--block-shape", nargs="+", type=int, help="Quantization block shape" + ) + + # Torch trace profile generation args + parser.add_argument( + "--torch-trace-dir-path", + type=str, + default=None, + help="Get torch trace for single execution", + ) + + return parser + + +def _validate_args(args: argparse.Namespace): + if args.quant_dtype is not None: + assert args.quant_dtype == torch.float8_e4m3fn + if args.block_shape is not None: + assert len(args.block_shape) == 2, ( + f"block shape must have 2 elements. got {args.block_shape}" + ) + + if args.experts_type in MK_SINGLE_GPU_PREPARE_FINALIZE_TYPES: + assert args.world_size == 1, "Single GPU objects need world size set to 1" + + if args.torch_trace_dir_path is not None: + from pathlib import Path + + assert Path(args.torch_trace_dir_path).is_dir(), ( + f"Please create {args.torch_trace_dir_path}" + ) + + +def make_config(args: argparse.Namespace) -> Config: + _validate_args(args) + + quant_config = None + if args.quant_dtype is not None: + quant_config = FusedMoEQuantConfig.make( + quant_dtype=args.quant_dtype, + per_act_token_quant=args.per_token_quantized_activations, + per_out_ch_quant=args.per_channel_quantized_weights, + block_shape=args.block_shape, + ) + + return Config( + Ms=args.m, + K=args.k, + N=args.n, + E=args.num_experts, + topks=args.topk, + dtype=torch.bfloat16, # hard-code + quant_config=quant_config, + prepare_finalize_type=args.pf_type, + fused_experts_type=args.experts_type, + world_size=args.world_size, + torch_trace_dir_path=args.torch_trace_dir_path, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/common.py b/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/common.py new file mode 100644 index 0000000000000000000000000000000000000000..df3f026ff182d60f5d457b230cb632374b649bec --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/common.py @@ -0,0 +1,795 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from dataclasses import dataclass +from types import SimpleNamespace +from typing import Any + +import torch + +import vllm._custom_ops as ops +import vllm.model_executor.layers.fused_moe.modular_kernel as mk +from tests.kernels.moe.utils import make_test_weights, per_token_cast_to_fp8 +from tests.kernels.quantization.nvfp4_utils import ( + FLOAT4_E2M1_MAX, + FLOAT8_E4M3_MAX, + dequantize_nvfp4_to_dtype, +) +from tests.kernels.utils import torch_experts +from vllm.config import VllmConfig +from vllm.distributed import ( + get_dp_group, + get_pcp_group, + get_tensor_model_parallel_world_size, +) +from vllm.forward_context import set_forward_context +from vllm.model_executor.layers.fused_moe import fused_topk +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.all2all_utils import ( + maybe_make_prepare_finalize, +) +from vllm.model_executor.layers.fused_moe.config import ( + FusedMoEConfig, + FusedMoEParallelConfig, + FusedMoEQuantConfig, + RoutingMethodType, +) +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + kFp8Dynamic128Sym, + kFp8DynamicTensorSym, + kFp8DynamicTokenSym, + kFp8Static128BlockSym, + kFp8StaticChannelSym, + kFp8StaticTensorSym, +) +from vllm.utils.import_utils import ( + has_aiter, + has_deep_ep, + has_deep_ep_v2, + has_deep_gemm, + has_mori, +) +from vllm.utils.math_utils import next_power_of_2 + +from .mk_objects import ( + TestMoEQuantConfig, + expert_info, + make_fused_experts, + prepare_finalize_info, +) +from .parallel_utils import ProcessGroupInfo + + +def _describe_tensor(t: torch.Tensor | None, name: str) -> str: + if t is None: + return f"{name} : None" + else: + return f"{name} : {t.shape} {t.dtype} {t.device}" + + +@dataclass +class Config: + Ms: list[int] | int + K: int + N: int + E: int + topks: list[int] | int + dtype: torch.dtype + quant_config: TestMoEQuantConfig | None + + prepare_finalize_type: mk.FusedMoEPrepareAndFinalize + fused_experts_type: mk.FusedMoEExperts + + world_size: int + + torch_trace_dir_path: str | None = None + + def __post_init__(self): + if self.quant_config is None: + self.quant_config = TestMoEQuantConfig(None, False, False, None) + + def describe(self) -> str: + s = "" + s += "== Config:\n" + s += f" world_size={self.world_size}\n" + s += f" PF={self.prepare_finalize_type.__name__}\n" + s += f" FE={self.fused_experts_type.__name__}\n" + s += f" E={self.E}\n" + s += f" Ms={self.Ms}\n" + s += f" N={self.N}\n" + s += f" K={self.K}\n" + s += f" topk={self.topks}\n" + s += f" dtype={self.dtype}\n" + s += " Quant:\n" + if self.quant_config is not None: + s += f" q_dtype={self.quant_dtype}\n" + s += f" q_block_shape={self.quant_block_shape}\n" + s += f" q_per_out_ch_quant={self.is_per_out_ch_quant}\n" + s += f" q_per_act_token={self.is_per_act_token_quant}\n" + else: + s += " quant=None\n" + return s + + @property + def M(self) -> int: + assert isinstance(self.Ms, int) + return self.Ms + + @property + def quant_dtype(self) -> torch.dtype | str | None: + assert self.quant_config is not None + return self.quant_config.quant_dtype + + @property + def is_per_act_token_quant(self) -> bool: + assert self.quant_config is not None + return self.quant_config.per_act_token_quant + + @property + def is_per_tensor_act_quant(self) -> bool: + return not self.is_per_act_token_quant and self.quant_block_shape is None + + @property + def is_per_out_ch_quant(self) -> bool: + assert self.quant_config is not None + return self.quant_config.per_out_ch_quant + + @property + def quant_block_shape(self) -> list[int] | None: + assert self.quant_config is not None + return self.quant_config.block_shape + + @property + def topk(self) -> int: + assert isinstance(self.topks, int) + return self.topks + + @property + def num_local_experts(self) -> int: + return self.E // self.world_size + + def make_env_data(self) -> tuple[VllmConfig, dict[Any, Any]]: + """ + make env data for vllm launch. + """ + vllm_config = VllmConfig() + vllm_config.model_config = SimpleNamespace( + enforce_eager=True, + is_moe=True, + ) + vllm_config.parallel_config.data_parallel_size = self.world_size + vllm_config.parallel_config.enable_expert_parallel = True + + env_dict = { + "VLLM_USE_DEEP_GEMM": str(int(self.needs_deep_gemm())), + } + + vllm_config.parallel_config.all2all_backend = self.all2all_backend() + + return vllm_config, env_dict + + def fe_supports_quant_scheme(self) -> bool: + """Check if the fused experts class supports this quant config. + See https://github.com/ROCm/aiter/issues/2419 for AITER gaps.""" + if self.quant_config is None or self.quant_dtype is None: + return True + if self.quant_dtype != torch.float8_e4m3fn: + return True + # Derive QuantKeys from test config + if self.quant_block_shape is not None: + w_key = kFp8Static128BlockSym + a_key = kFp8Dynamic128Sym + elif self.is_per_out_ch_quant: + w_key = kFp8StaticChannelSym + a_key = ( + kFp8DynamicTokenSym + if self.is_per_act_token_quant + else kFp8StaticTensorSym + ) + else: + w_key = kFp8StaticTensorSym + a_key = ( + kFp8DynamicTensorSym + if self.is_per_act_token_quant + else kFp8StaticTensorSym + ) + fe_cls = self.fused_experts_type + if hasattr(fe_cls, "_supports_quant_scheme"): + try: + return fe_cls._supports_quant_scheme(w_key, a_key) + except NotImplementedError: + pass + return True + + def is_fp8_block_quantized(self): + return ( + self.quant_dtype == torch.float8_e4m3fn + and self.quant_block_shape is not None + ) + + def is_batched_prepare_finalize(self): + info = prepare_finalize_info(self.prepare_finalize_type) + return mk.FusedMoEActivationFormat.BatchedExperts == info.activation_format + + def is_batched_fused_experts(self): + info = expert_info(self.fused_experts_type) + return mk.FusedMoEActivationFormat.BatchedExperts == info.activation_format + + def is_standard_fused_experts(self): + info = expert_info(self.fused_experts_type) + return mk.FusedMoEActivationFormat.Standard == info.activation_format + + def fe_supported_types(self): + info = expert_info(self.fused_experts_type) + return info.supported_dtypes + + def pf_supported_types(self): + info = prepare_finalize_info(self.prepare_finalize_type) + return info.supported_dtypes + + def is_block_quant_supported(self): + info = expert_info(self.fused_experts_type) + return info.blocked_quantization_support + + def supports_apply_weight_on_input(self): + info = prepare_finalize_info(self.prepare_finalize_type) + return info.supports_apply_weight_on_input + + def needs_deep_gemm(self): + info = expert_info(self.fused_experts_type) + return info.needs_deep_gemm + + def needs_deep_ep(self): + info = prepare_finalize_info(self.prepare_finalize_type) + return ( + info.backend == "deepep_high_throughput" + or info.backend == "deepep_low_latency" + ) + + def needs_deep_ep_v2(self): + info = prepare_finalize_info(self.prepare_finalize_type) + return info.backend == "deepep_v2" + + def needs_aiter(self): + info = expert_info(self.fused_experts_type) + return info.needs_aiter + + def needs_mori(self): + info = prepare_finalize_info(self.prepare_finalize_type) + return info.backend in ("mori_high_throughput", "mori_low_latency") + + def all2all_backend(self): + info = prepare_finalize_info(self.prepare_finalize_type) + return info.backend + + def is_valid(self) -> tuple[bool, str | None]: + # Check prepare-finalize and fused-experts compatibility + if self.is_batched_prepare_finalize(): + if not self.is_batched_fused_experts(): + return False, "Mismatched format." + else: + if not self.is_standard_fused_experts(): + return False, "Mismatched format." + + # Check quantization sanity + if ( + int(self.is_per_act_token_quant) + + int(self.is_per_tensor_act_quant) + + int(self.quant_block_shape is not None) + ) > 1: + # invalid quant config + return False, f"Bad quant_config {self.quant_config}." + + # check type support + if self.quant_dtype is None: + if ( + self.dtype not in self.pf_supported_types() + or self.dtype not in self.fe_supported_types() + ): + return False, ( + f"Unsupported type {self.dtype} not in " + f"{self.pf_supported_types()} and " + f"{self.fe_supported_types()}." + ) + else: + if ( + self.quant_dtype not in self.pf_supported_types() + or self.quant_dtype not in self.fe_supported_types() + ): + return False, ( + f"Unsupported quant type {self.quant_dtype} " + f"not in {self.pf_supported_types()} and " + f"{self.fe_supported_types()}." + ) + + # Check quant scheme compatibility with fused experts class + if not self.fe_supports_quant_scheme(): + return False, ( + f"FE {self.fused_experts_type.__name__} does not support " + f"quant scheme (per_out_ch={self.is_per_out_ch_quant}, " + f"per_act_token={self.is_per_act_token_quant}, " + f"block={self.quant_block_shape})" + ) + + # Check block quantization support + is_block_quantized = self.quant_block_shape is not None + if is_block_quantized and self.quant_dtype is None: + return False, "No block quantization support." + + if is_block_quantized and not self.is_block_quant_supported(): + return False, "Mismatched block quantization support." + + # deep_gemm only works with block-quantized + if self.needs_deep_gemm() and not is_block_quantized: + return False, "Needs DeepGEMM but not block quantized." + + # Check dependencies (turn into asserts?) + if self.needs_deep_ep() and not has_deep_ep(): + return False, "Needs DeepEP, but DeepEP not available." + if self.needs_deep_ep_v2() and not has_deep_ep_v2(): + return False, "Needs DeepEP v2, but DeepEP v2 not available." + if self.needs_deep_gemm() and not has_deep_gemm(): + return ( + False, + "Needs DeepGEMM, but the current vLLM environment does not provide it.", + ) + if self.needs_aiter() and not has_aiter(): # noqa: SIM103 + return False, "Needs Aiter, but Aiter not available." + if self.needs_mori() and not has_mori(): # noqa: SIM103 + return False, "Needs MoRI, but MoRI not available." + + try: + if not self.fused_experts_type._supports_current_device(): + return ( + False, + f"{self.fused_experts_type} not supported on the current device.", + ) + except NotImplementedError: + pass + + return True, None + + +@dataclass +class WeightTensors: + w1: torch.Tensor + w2: torch.Tensor + w1_scale: torch.Tensor | None + w2_scale: torch.Tensor | None + w1_gs: torch.Tensor | None = None + w2_gs: torch.Tensor | None = None + + def describe(self): + s = "" + s += "== Weight Tensors: \n" + s += f" - {_describe_tensor(self.w1, 'w1')} \n" + s += f" - {_describe_tensor(self.w2, 'w2')} \n" + s += f" - {_describe_tensor(self.w1_scale, 'w1_scale')} \n" + s += f" - {_describe_tensor(self.w2_scale, 'w2_scale')} \n" + s += f" - {_describe_tensor(self.w1_gs, 'w1_gs')} \n" + s += f" - {_describe_tensor(self.w2_gs, 'w2_gs')} \n" + return s + + def is_quantized(self) -> bool: + # or w1_scale is not None? + return ( + self.w1.dtype == torch.float8_e4m3fn + or self.w1.dtype == torch.uint8 + or self.w1.dtype == torch.int8 + ) + + def to_current_device(self): + device = torch.accelerator.current_device_index() + self.w1 = self.w1.to(device=device) + self.w2 = self.w2.to(device=device) + + if self.w1_scale is not None: + self.w1_scale = self.w1_scale.to(device=device) + if self.w2_scale is not None: + self.w2_scale = self.w2_scale.to(device=device) + + if self.w1_gs is not None: + self.w1_gs = self.w1_gs.to(device=device) + if self.w2_gs is not None: + self.w2_gs = self.w2_gs.to(device=device) + + def slice_weights(self, rank: int, num_local_experts: int) -> "WeightTensors": + s = rank * num_local_experts + e = s + num_local_experts + w1 = self.w1[s:e, :, :] + w2 = self.w2[s:e, :, :] + w1_scale = self.w1_scale[s:e, :, :] if self.w1_scale is not None else None + w2_scale = self.w2_scale[s:e, :, :] if self.w2_scale is not None else None + w1_gs = self.w1_gs[s:e] if self.w1_gs is not None else None + w2_gs = self.w2_gs[s:e] if self.w2_gs is not None else None + + return WeightTensors(w1, w2, w1_scale, w2_scale, w1_gs, w2_gs) + + @staticmethod + def make(config: Config) -> "WeightTensors": + (_, w1, w1_scale, w1_gs), (_, w2, w2_scale, w2_gs) = make_test_weights( + e=config.E, + n=config.N, + k=config.K, + in_dtype=config.dtype, + quant_dtype=config.quant_dtype, + block_shape=config.quant_block_shape, + # or config.is_per_out_ch_quant + per_out_ch_quant=config.is_per_act_token_quant, + ) + return WeightTensors( + w1=w1, w2=w2, w1_scale=w1_scale, w2_scale=w2_scale, w1_gs=w1_gs, w2_gs=w2_gs + ) + + +@dataclass +class RankTensors: + hidden_states: torch.Tensor + hidden_states_scale: torch.Tensor | None + + topk_weights: torch.Tensor + topk_ids: torch.Tensor + expert_map: torch.Tensor | None + + def describe(self): + s = "" + s += "== Rank Tensors: \n" + s += f" - {_describe_tensor(self.hidden_states, 'HS')} \n" + s += f" - {_describe_tensor(self.hidden_states_scale, 'HS_scale')} \n" + s += f" - {_describe_tensor(self.topk_weights, 'topk_weights')} \n" + s += f" - {_describe_tensor(self.topk_ids, 'topk_ids')} \n" + s += f" - {_describe_tensor(self.expert_map, 'expert_map')} \n" + return s + + @staticmethod + def make_hidden_states( + config: Config, + ) -> tuple[torch.Tensor, torch.Tensor | None]: + """ + Return hidden_states + """ + m, k, dtype = (config.M, config.K, config.dtype) + device = torch.accelerator.current_device_index() + a = torch.randn((m, k), device=device, dtype=dtype) / 15.0 + + if config.quant_dtype is None: + return a, None + + # We dequant and use that as hidden_states so the tests are stable. + # quantizing and dequantizing yield slightly different results + # depending on the hardware. Here we, quantize and dequantize + # first - so further quantize and dequantize will yield the same + # values. + if config.is_per_tensor_act_quant: + a_q, a_scales = ops.scaled_fp8_quant(a, use_per_token_if_dynamic=False) + return a_q.float().mul(a_scales).to(dtype), a_scales + + if config.is_per_act_token_quant: + a_q, a_scales = ops.scaled_fp8_quant(a, use_per_token_if_dynamic=True) + return a_q.float().mul(a_scales).to(dtype), None + + assert config.quant_block_shape is not None + block_k = config.quant_block_shape[1] + a_q, a_scales = per_token_cast_to_fp8(a, block_size=block_k) + return a_q.float().view((-1, block_k)).mul(a_scales.view(-1, 1)).view(m, k).to( + dtype + ), None + + @staticmethod + def make(config: Config, pgi: ProcessGroupInfo): + dtype = config.dtype + topk, m, _ = (config.topk, config.M, config.K) + hidden_states, hidden_states_scale = RankTensors.make_hidden_states(config) + + num_local_experts, global_num_experts = (config.num_local_experts, config.E) + score = torch.randn((m, global_num_experts), device="cuda", dtype=dtype) + topk_weights, topk_ids, _ = fused_topk(hidden_states, score, topk, False) + + # distribute topk_ids evenly + device = torch.accelerator.current_device_index() + for mi in range(m): + topk_ids[mi] = torch.randperm(config.E)[:topk] + topk_ids = topk_ids.to(device=device) + + expert_map = None + if config.world_size > 1: + expert_map = torch.full( + (global_num_experts,), fill_value=-1, dtype=torch.int32 + ) + s = pgi.rank * num_local_experts + e = s + num_local_experts + expert_map[s:e] = torch.tensor(list(range(num_local_experts))) + expert_map = expert_map.to(device=device, dtype=torch.int32) + + return RankTensors( + hidden_states=hidden_states, + hidden_states_scale=hidden_states_scale, + topk_weights=topk_weights, + topk_ids=topk_ids, + expert_map=expert_map, + ) + + +def reference_moe_impl( + config: Config, weights: WeightTensors, rank_tensors: RankTensors +) -> torch.Tensor: + if config.quant_dtype == "nvfp4": + quant_blocksize = 16 + dtype = config.dtype + + w1_q = weights.w1 + w1_blockscale = weights.w1_scale + w1_gs = weights.w1_gs + + w2_q = weights.w2 + w2_blockscale = weights.w2_scale + w2_gs = weights.w2_gs + + a_global_scale = ( + (FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX) + / torch.amax(rank_tensors.hidden_states.flatten(), dim=-1) + ).to(torch.float32) + + assert w1_gs is not None + assert w2_gs is not None + assert w1_blockscale is not None + assert w2_blockscale is not None + + assert w1_blockscale.shape[1] % 128 == 0 + assert w1_blockscale.shape[2] % 4 == 0 + assert w2_blockscale.shape[1] % 128 == 0 + assert w2_blockscale.shape[2] % 4 == 0 + + a_fp4, a_scale_interleaved = ops.scaled_fp4_quant( + rank_tensors.hidden_states, a_global_scale + ) + + a = dequantize_nvfp4_to_dtype( + a_fp4, + a_scale_interleaved, + a_global_scale, + dtype=dtype, + device=a_fp4.device, + block_size=quant_blocksize, + ) + + e = w1_q.shape[0] + n = w1_q.shape[1] // 2 + k = w2_q.shape[1] + + w1 = torch.zeros((e, 2 * n, k), device="cuda", dtype=dtype) + w2 = torch.zeros((e, k, n), device="cuda", dtype=dtype) + + for idx in range(0, e): + w1[idx] = dequantize_nvfp4_to_dtype( + w1_q[idx], + w1_blockscale[idx], + w1_gs[idx], + dtype=dtype, + device=w1_q.device, + block_size=quant_blocksize, + ) + w2[idx] = dequantize_nvfp4_to_dtype( + w2_q[idx], + w2_blockscale[idx], + w2_gs[idx], + dtype=dtype, + device=w2_q.device, + block_size=quant_blocksize, + ) + a_scale = None + w1_scale = None + w2_scale = None + quant_dtype = None + per_act_token_quant = False + block_shape = None + else: + a = rank_tensors.hidden_states + a_scale = rank_tensors.hidden_states_scale + w1 = weights.w1 + w1_scale = weights.w1_scale + w2 = weights.w2 + w2_scale = weights.w2_scale + quant_dtype = config.quant_dtype + per_act_token_quant = config.is_per_act_token_quant + block_shape = config.quant_block_shape + + return torch_experts( + a=a, + w1=w1, + w2=w2, + topk_weight=rank_tensors.topk_weights, + topk_ids=rank_tensors.topk_ids, + global_num_experts=config.E, + expert_map=None, + w1_scale=w1_scale, + w2_scale=w2_scale, + a1_scale=a_scale, + quant_dtype=quant_dtype, + per_act_token_quant=per_act_token_quant, + block_shape=block_shape, + apply_router_weights_on_input=config.topk == 1 + and config.supports_apply_weight_on_input(), + ) + + +def _make_gscale(num_experts: int) -> torch.Tensor: + return torch.ones( + (num_experts,), + device=torch.accelerator.current_device_index(), + dtype=torch.float32, + ) + + +def make_modular_kernel( + config: Config, + vllm_config: VllmConfig, + quant_config: FusedMoEQuantConfig, +) -> mk.FusedMoEKernel: + # make moe config + moe_parallel_config: FusedMoEParallelConfig = FusedMoEParallelConfig.make( + tp_size_=get_tensor_model_parallel_world_size(), + pcp_size_=get_pcp_group().world_size, + dp_size_=get_dp_group().world_size, + sp_size_=1, + vllm_parallel_config=vllm_config.parallel_config, + ) + + moe = FusedMoEConfig( + num_experts=config.E, + experts_per_token=config.topk, + hidden_dim=config.K, + intermediate_size=config.N, + num_local_experts=config.num_local_experts, + num_logical_experts=config.E, + moe_parallel_config=moe_parallel_config, + in_dtype=config.dtype, + max_num_tokens=next_power_of_2(config.M), + activation=MoEActivation.SILU, + device=vllm_config.device_config.device, + routing_method=RoutingMethodType.DeepSeekV3, + ) + + prepare_finalize = maybe_make_prepare_finalize( + moe=moe, + quant_config=quant_config, + allow_new_interface=True, + ) + assert prepare_finalize is not None + + fused_experts = make_fused_experts( + config.fused_experts_type, + moe, + quant_config, + prepare_finalize.num_dispatchers(), + config.N, + ) + + modular_kernel = mk.FusedMoEKernel( + prepare_finalize=prepare_finalize, + fused_experts=fused_experts, + ) + + return modular_kernel + + +def _maybe_convert_weights_for_experts( + config: Config, + rank_weights: WeightTensors, +) -> WeightTensors: + """Convert weights to expert-specific format (e.g., TrtLLM BlockMajorK).""" + from vllm.model_executor.layers.fused_moe.oracle.fp8 import ( + Fp8MoeBackend, + convert_to_fp8_moe_kernel_format, + ) + + fe_type = config.fused_experts_type + fe_name = getattr(fe_type, "__name__", "") + + backend: Fp8MoeBackend | None = None + if fe_name == "TrtLlmFp8ExpertsModular": + backend = Fp8MoeBackend.FLASHINFER_TRTLLM + elif fe_name == "FlashInferExperts": + backend = Fp8MoeBackend.FLASHINFER_CUTLASS + + if backend is None or not rank_weights.is_quantized(): + return rank_weights + + mock_layer = SimpleNamespace( + weight_block_size=config.quant_block_shape, + moe_config=SimpleNamespace( + is_act_and_mul=True, + intermediate_size_per_partition=config.N, + ), + activation=SimpleNamespace(is_gated=True), + ) + + w1, w2, w1_scale, w2_scale = convert_to_fp8_moe_kernel_format( + fp8_backend=backend, + layer=mock_layer, + w13=rank_weights.w1, + w2=rank_weights.w2, + w13_scale=rank_weights.w1_scale, + w2_scale=rank_weights.w2_scale, + w13_input_scale=None, + w2_input_scale=None, + ) + + return WeightTensors( + w1=w1, + w2=w2, + w1_scale=w1_scale, + w2_scale=w2_scale, + w1_gs=rank_weights.w1_gs, + w2_gs=rank_weights.w2_gs, + ) + + +def run_modular_kernel( + pgi: ProcessGroupInfo, + vllm_config: VllmConfig, + config: Config, + weights: WeightTensors, + rank_tensors: RankTensors, +) -> torch.Tensor: + assert isinstance(config.Ms, int) + assert isinstance(config.topks, int) + + # weights for rank + rank_weights = weights.slice_weights(pgi.rank, config.num_local_experts) + rank_weights = _maybe_convert_weights_for_experts(config, rank_weights) + + if config.quant_dtype == "nvfp4": + gscale = _make_gscale(config.num_local_experts) + else: + gscale = None + + quant_config = FusedMoEQuantConfig.make( + config.quant_dtype, + w1_scale=rank_weights.w1_scale, + w2_scale=rank_weights.w2_scale, + a1_scale=rank_tensors.hidden_states_scale, + g1_alphas=(1 / rank_weights.w1_gs) if rank_weights.w1_gs is not None else None, + g2_alphas=(1 / rank_weights.w2_gs) if rank_weights.w2_gs is not None else None, + a1_gscale=gscale, + a2_gscale=gscale, + block_shape=config.quant_block_shape, + per_act_token_quant=config.is_per_act_token_quant, + per_out_ch_quant=config.is_per_out_ch_quant, + ) + + mk = make_modular_kernel(config, vllm_config, quant_config) + + # impls might update the tensor in place + hidden_states = rank_tensors.hidden_states.clone() + + topk_ids = rank_tensors.topk_ids.to(mk.prepare_finalize.topk_indices_dtype()) + + mk_kwargs = { + "hidden_states": hidden_states, + "w1": rank_weights.w1, + "w2": rank_weights.w2, + "topk_weights": rank_tensors.topk_weights, + "topk_ids": topk_ids, + "activation": MoEActivation.SILU, + "expert_map": rank_tensors.expert_map, + "global_num_experts": config.E, + "apply_router_weight_on_input": config.topk == 1 + and config.supports_apply_weight_on_input(), + } + + num_tokens = rank_tensors.hidden_states.shape[0] + num_tokens_across_dp = torch.tensor( + [num_tokens] * config.world_size, device="cuda", dtype=torch.int + ) + + torch.distributed.barrier() + + with set_forward_context( + None, + vllm_config, + num_tokens=num_tokens, + num_tokens_across_dp=num_tokens_across_dp, + ): + out = mk.apply(**mk_kwargs) + + return out diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/make_feature_matrix.py b/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/make_feature_matrix.py new file mode 100644 index 0000000000000000000000000000000000000000..aaa29811cd7ec8b152e74aac57c2c08bc80cbad6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/make_feature_matrix.py @@ -0,0 +1,189 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import copy +from enum import Enum +from itertools import product + +import torch +from tqdm import tqdm + +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.model_executor.layers.fused_moe.config import FUSED_MOE_UNQUANTIZED_CONFIG +from vllm.utils.torch_utils import set_random_seed + +from .common import ( + Config, + RankTensors, + WeightTensors, + reference_moe_impl, + run_modular_kernel, +) +from .mk_objects import ( + MK_FUSED_EXPERT_TYPES, + MK_MULTI_GPU_PREPARE_FINALIZE_TYPES, + MK_QUANT_CONFIGS, +) +from .parallel_utils import ProcessGroupInfo, parallel_launch_with_config + + +class Result(Enum): + PASS = 1 + FAIL = 2 + SKIP = 3 + + +def rank_worker( + pgi: ProcessGroupInfo, + vllm_config: VllmConfig, + cpu_group, + config: Config, + weights: WeightTensors, +): + set_random_seed(pgi.rank) + + # get weights to this device + weights.to_current_device() + + Ms = config.Ms + assert isinstance(Ms, list) + TOPKs = config.topks + assert isinstance(TOPKs, list) + + for m, topk in product(Ms, TOPKs): + print(f"Running m={m}, topk={topk} ...") + # override m and topk + cfgx = copy.deepcopy(config) + cfgx.Ms = m + cfgx.topks = topk + + # inputs for rank + rank_tensors = RankTensors.make(cfgx, pgi) + + # modular kernel out + mk_out = run_modular_kernel(pgi, vllm_config, cfgx, weights, rank_tensors) + + with set_current_vllm_config(vllm_config): + ref_out = reference_moe_impl(cfgx, weights, rank_tensors) + + torch.testing.assert_close(ref_out, mk_out, atol=3e-2, rtol=3e-2) + + +def make_feature_matrix(csv_file_path: str): + from dataclasses import asdict + + import pandas as pd + + def add_to_results( + config: Config, success: Result, results_df: pd.DataFrame | None = None + ): + config_dict = asdict(config) + config_dict["prepare_finalize_type"] = config_dict[ + "prepare_finalize_type" + ].__name__ + config_dict["fused_experts_type"] = config_dict["fused_experts_type"].__name__ + config_dict["per_tensor_act_quant"] = config.is_per_tensor_act_quant + quant_config_dict = config_dict["quant_config"] + del config_dict["quant_config"] + if quant_config_dict is None: + quant_config = FUSED_MOE_UNQUANTIZED_CONFIG + quant_config_dict = asdict(quant_config) + + config_dict |= quant_config_dict + result_dict = config_dict | {"success": success.name} + + result_df = pd.DataFrame([result_dict]) + if results_df is None: + results_df = result_df + else: + results_df = pd.concat([results_df, result_df], ignore_index=True) + + return results_df + + Ms = [64] + Ks = [7168] # hidden sizes + Ns = [2048] + TOPKs = [[4, 1]] + Es = [32] + DTYPEs = [torch.bfloat16] + PF_TYPES = MK_MULTI_GPU_PREPARE_FINALIZE_TYPES + FE_TYPES = MK_FUSED_EXPERT_TYPES + Q_TYPES = MK_QUANT_CONFIGS + + combinations = list( + product(Ms, Ks, Ns, Es, TOPKs, DTYPEs, PF_TYPES, FE_TYPES, Q_TYPES) + ) + + results_df: pd.DataFrame | None = None + for m, k, n, e, topks, dtype, pf_type, experts_type, quant_config in tqdm( + combinations + ): + config = Config( + Ms=[m], + K=k, + N=n, + E=e, + topks=topks, + dtype=dtype, + prepare_finalize_type=pf_type, + fused_experts_type=experts_type, + quant_config=quant_config, + world_size=2, + ) + + success = None + if config.is_valid()[0]: + print(f"Running config : {config.describe()} ...") + try: + weights: WeightTensors = WeightTensors.make(config) + vllm_config, env_dict = config.make_env_data() + parallel_launch_with_config( + config.world_size, + rank_worker, + vllm_config, + env_dict, + config, + weights, + ) + success = Result.PASS + except Exception as _: + success = Result.FAIL + else: + success = Result.SKIP + + results_df = add_to_results(config, success, results_df) + + if results_df is not None: + results_df.to_csv(f"{csv_file_path}") + + +if __name__ == "__main__": + import argparse + from pathlib import Path + + parser = argparse.ArgumentParser( + description=( + "Make ModularKernel feature matrix \n" + "Example : python3 -m tests.kernels.moe.modular_kernel_tools.make_feature_matrix " # noqa: E501 + "-f ./feature_matrices/feature_matrix.csv" + ) + ) + + parser.add_argument( + "-f", + "--feature-matrix-csv-file-path", + type=str, + required=True, + help="File name to Generate a .csv file", + ) + args = parser.parse_args() + + csv_path = args.feature_matrix_csv_file_path + assert csv_path.endswith("csv"), ( + f"Need a file path ending with .csv, got {csv_path}" + ) + assert Path(csv_path).parent.is_dir(), ( + f"Cannot find parent directory for {Path(csv_path).parent}" + ) + + make_feature_matrix(args.feature_matrix_csv_file_path) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/mk_objects.py b/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/mk_objects.py new file mode 100644 index 0000000000000000000000000000000000000000..2ec89195052e7357d52ca872381d2dc01103dca2 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/mk_objects.py @@ -0,0 +1,497 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from dataclasses import dataclass + +import torch + +# Fused experts and PrepareFinalize imports +import vllm.model_executor.layers.fused_moe.modular_kernel as mk +from vllm.model_executor.layers.fused_moe import TritonExperts +from vllm.model_executor.layers.fused_moe.config import ( + FusedMoEConfig, + FusedMoEQuantConfig, +) +from vllm.model_executor.layers.fused_moe.experts.batched_deep_gemm_moe import ( + BatchedDeepGemmExperts, +) +from vllm.model_executor.layers.fused_moe.experts.deep_gemm_moe import DeepGemmExperts +from vllm.model_executor.layers.fused_moe.experts.fused_batched_moe import ( + BatchedTritonExperts, + NaiveBatchedExperts, +) +from vllm.model_executor.layers.fused_moe.experts.triton_deep_gemm_moe import ( + TritonOrDeepGemmExperts, +) +from vllm.model_executor.layers.fused_moe.prepare_finalize import ( + MoEPrepareAndFinalizeNoDPEPModular, +) +from vllm.model_executor.layers.quantization.utils.nvfp4_utils import ( + cutlass_fp4_supported, +) +from vllm.model_executor.layers.quantization.utils.w8a8_utils import ( + cutlass_fp8_supported, +) +from vllm.platforms import current_platform +from vllm.utils.deep_gemm import is_deep_gemm_supported +from vllm.utils.flashinfer import ( + has_flashinfer_cutlass_fused_moe, + has_flashinfer_nvlink_one_sided, + has_flashinfer_trtllm_fused_moe, +) +from vllm.utils.import_utils import ( + has_aiter, + has_deep_ep, + has_deep_ep_v2, + has_deep_gemm, + has_mori, +) + + +@dataclass +class TestMoEQuantConfig: + quant_dtype: torch.dtype | str | None + per_out_ch_quant: bool + per_act_token_quant: bool + block_shape: list[int] | None + + +@dataclass +class PrepareFinalizeInfo: + activation_format: mk.FusedMoEActivationFormat + supported_dtypes: list[torch.dtype | str] + blocked_quantization_support: bool + backend: str | None + supports_apply_weight_on_input: bool = True + + +@dataclass +class ExpertInfo: + activation_format: mk.FusedMoEActivationFormat + supported_dtypes: list[torch.dtype | str] + blocked_quantization_support: bool + needs_matching_quant: bool = False + needs_deep_gemm: bool = False + needs_aiter: bool = False + + +PREPARE_FINALIZE_INFO: dict[ + mk.FusedMoEPrepareAndFinalizeModular, PrepareFinalizeInfo +] = {} +EXPERT_INFO: dict[mk.FusedMoEExpertsModular, ExpertInfo] = {} +MK_ALL_PREPARE_FINALIZE_TYPES: list[mk.FusedMoEPrepareAndFinalizeModular] = [] +MK_MULTI_GPU_PREPARE_FINALIZE_TYPES: list[mk.FusedMoEPrepareAndFinalizeModular] = [] +MK_SINGLE_GPU_PREPARE_FINALIZE_TYPES: list[mk.FusedMoEPrepareAndFinalizeModular] = [] +MK_FUSED_EXPERT_TYPES: list[mk.FusedMoEExpertsModular] = [] + +standard_format = mk.FusedMoEActivationFormat.Standard +batched_format = mk.FusedMoEActivationFormat.BatchedExperts +common_float_types: list[torch.dtype | str] = [ + torch.float8_e4m3fn, + torch.bfloat16, + torch.float16, + torch.float32, +] +common_float_and_int_types = common_float_types + [torch.int8] +nvfp4_types = ["nvfp4"] +fp8_types = [torch.float8_e4m3fn] + + +def register_prepare_and_finalize( + kind, + activation_format: mk.FusedMoEActivationFormat, + supported_dtypes: list[torch.dtype | str], + blocked_quantization_support: bool, + backend: str | None, + force_multigpu: bool = False, + supports_apply_weight_on_input: bool = True, +): + global PREPARE_FINALIZE_INFO + global MK_ALL_PREPARE_FINALIZE_TYPES + global MK_MULTI_GPU_PREPARE_FINALIZE_TYPES + global MK_SINGLE_GPU_PREPARE_FINALIZE_TYPES + assert kind not in PREPARE_FINALIZE_INFO + + PREPARE_FINALIZE_INFO[kind] = PrepareFinalizeInfo( + activation_format, + supported_dtypes, + blocked_quantization_support, + backend, + supports_apply_weight_on_input, + ) + MK_ALL_PREPARE_FINALIZE_TYPES.append(kind) + if backend is not None or force_multigpu: + MK_MULTI_GPU_PREPARE_FINALIZE_TYPES.append(kind) + else: + MK_SINGLE_GPU_PREPARE_FINALIZE_TYPES.append(kind) + + +def register_experts( + kind, + activation_format: mk.FusedMoEActivationFormat, + supported_dtypes: list[torch.dtype | str], + blocked_quantization_support: bool, + needs_matching_quant: bool = False, + needs_deep_gemm: bool = False, + needs_aiter: bool = False, +): + global EXPERT_INFO + global MK_FUSED_EXPERT_TYPES + assert kind not in EXPERT_INFO + + EXPERT_INFO[kind] = ExpertInfo( + activation_format, + supported_dtypes, + blocked_quantization_support, + needs_matching_quant, + needs_deep_gemm, + needs_aiter, + ) + + MK_FUSED_EXPERT_TYPES.append(kind) + + +def prepare_finalize_info(kind) -> PrepareFinalizeInfo: + info = PREPARE_FINALIZE_INFO.get(kind) + assert info is not None + return info + + +def expert_info(kind) -> ExpertInfo: + info = EXPERT_INFO.get(kind) + assert info is not None + return info + + +register_prepare_and_finalize( + MoEPrepareAndFinalizeNoDPEPModular, + standard_format, + common_float_types, + blocked_quantization_support=True, + backend=None, +) + +register_experts( + BatchedTritonExperts, + batched_format, + common_float_types, + blocked_quantization_support=True, + needs_matching_quant=True, +) + +register_experts( + TritonExperts, + standard_format, + common_float_and_int_types, + blocked_quantization_support=True, + needs_matching_quant=True, +) + +register_experts( + NaiveBatchedExperts, + batched_format, + common_float_and_int_types, + blocked_quantization_support=True, +) + +# Disable on blackwell for now +if has_deep_ep() and not current_platform.has_device_capability(100): + from vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ht import ( + DeepEPHTPrepareAndFinalize, + ) + from vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ll import ( + DeepEPLLPrepareAndFinalize, + ) + + register_prepare_and_finalize( + DeepEPHTPrepareAndFinalize, + standard_format, + common_float_types, + blocked_quantization_support=True, + backend="deepep_high_throughput", + ) + + register_prepare_and_finalize( + DeepEPLLPrepareAndFinalize, + batched_format, + common_float_types, + blocked_quantization_support=True, + backend="deepep_low_latency", + ) + +if has_deep_ep_v2() and current_platform.has_device_capability(100): + from vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_v2 import ( + DeepEPV2PrepareAndFinalize, + ) + + register_prepare_and_finalize( + DeepEPV2PrepareAndFinalize, + standard_format, + common_float_types, + blocked_quantization_support=True, + backend="deepep_v2", + ) + +if has_mori(): + from vllm.model_executor.layers.fused_moe.prepare_finalize.mori import ( + MoriPrepareAndFinalize, + ) + + register_prepare_and_finalize( + MoriPrepareAndFinalize, + standard_format, + fp8_types, + blocked_quantization_support=True, + backend="mori_high_throughput", + supports_apply_weight_on_input=False, + ) + +if has_flashinfer_cutlass_fused_moe() and current_platform.has_device_capability(100): + from vllm.model_executor.layers.fused_moe.experts.flashinfer_cutlass_moe import ( + FlashInferExperts, + ) + from vllm.model_executor.layers.fused_moe.prepare_finalize.flashinfer_nvlink_two_sided import ( # noqa: E501 + FlashInferNVLinkTwoSidedPrepareAndFinalize, + ) + + register_prepare_and_finalize( + FlashInferNVLinkTwoSidedPrepareAndFinalize, + standard_format, + nvfp4_types + fp8_types, + blocked_quantization_support=True, + backend=None, + force_multigpu=True, + supports_apply_weight_on_input=False, + ) + + register_experts( + FlashInferExperts, + standard_format, + nvfp4_types + fp8_types, + blocked_quantization_support=True, + # Note: this is a hack to get it to run for now + ) +else: + FlashInferCutlassMoEPrepareAndFinalize = None + FlashInferExperts = None + +if ( + has_flashinfer_nvlink_one_sided() + and has_flashinfer_cutlass_fused_moe() + and current_platform.has_device_capability(100) +): + from vllm.model_executor.layers.fused_moe.prepare_finalize.flashinfer_nvlink_one_sided import ( # noqa: E501 + FlashInferNVLinkOneSidedPrepareAndFinalize, + ) + + register_prepare_and_finalize( + FlashInferNVLinkOneSidedPrepareAndFinalize, + standard_format, + nvfp4_types, + blocked_quantization_support=False, + backend="flashinfer_nvlink_one_sided", + supports_apply_weight_on_input=False, + ) + +if has_flashinfer_cutlass_fused_moe() and current_platform.has_device_capability(100): + from vllm.model_executor.layers.fused_moe.experts.trtllm_nvfp4_moe import ( + TrtLlmNvFp4ExpertsModular, + ) + + register_experts( + TrtLlmNvFp4ExpertsModular, + standard_format, + nvfp4_types, + blocked_quantization_support=False, + ) + +if has_flashinfer_trtllm_fused_moe() and current_platform.has_device_capability(100): + from vllm.model_executor.layers.fused_moe.experts.trtllm_fp8_moe import ( + TrtLlmFp8ExpertsModular, + ) + + register_experts( + TrtLlmFp8ExpertsModular, + standard_format, + fp8_types, + blocked_quantization_support=True, + ) + +if has_aiter(): + from vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe import ( + AiterExperts, + ) + + register_experts( + AiterExperts, + standard_format, + fp8_types, + blocked_quantization_support=True, + needs_aiter=True, + ) +else: + AiterExperts = None + +if has_deep_gemm() and is_deep_gemm_supported(): + register_experts( + BatchedDeepGemmExperts, + batched_format, + fp8_types, + blocked_quantization_support=True, + needs_matching_quant=False, + needs_deep_gemm=True, + ) + register_experts( + DeepGemmExperts, + standard_format, + fp8_types, + blocked_quantization_support=True, + needs_matching_quant=False, + needs_deep_gemm=True, + ) + register_experts( + TritonOrDeepGemmExperts, + standard_format, + common_float_and_int_types, + blocked_quantization_support=True, + needs_matching_quant=True, + needs_deep_gemm=True, + ) + +if cutlass_fp8_supported(): + from vllm.model_executor.layers.fused_moe import ( + CutlassBatchedExpertsFp8, + CutlassExpertsFp8, + ) + + register_experts( + CutlassExpertsFp8, + standard_format, + fp8_types, + blocked_quantization_support=False, + ) + register_experts( + CutlassBatchedExpertsFp8, + batched_format, + fp8_types, + blocked_quantization_support=False, + ) +else: + CutlassBatchedExpertsFp8 = None + CutlassExpertsFp8 = None + +if cutlass_fp4_supported(): + from vllm.model_executor.layers.fused_moe.experts.cutlass_moe import ( + CutlassExpertsFp4, + ) + + register_experts( + CutlassExpertsFp4, + standard_format, + nvfp4_types, + blocked_quantization_support=True, + ) +else: + CutlassExpertsFp4 = None + +MK_QUANT_CONFIGS: list[TestMoEQuantConfig | None] = [ + None, + # per-channel / per-column weights and per-tensor activations + TestMoEQuantConfig( + quant_dtype=torch.float8_e4m3fn, + per_out_ch_quant=True, + per_act_token_quant=False, + block_shape=None, + ), + # per-channel / per-column weights and per-token activations + TestMoEQuantConfig( + quant_dtype=torch.float8_e4m3fn, + per_out_ch_quant=True, + per_act_token_quant=True, + block_shape=None, + ), + # per-tensor weights and per-tensor activations + TestMoEQuantConfig( + quant_dtype=torch.float8_e4m3fn, + per_out_ch_quant=False, + per_act_token_quant=False, + block_shape=None, + ), + # per-tensor weights and per-token activations + TestMoEQuantConfig( + quant_dtype=torch.float8_e4m3fn, + per_out_ch_quant=False, + per_act_token_quant=True, + block_shape=None, + ), + # block-quantized weights and 128 block per-token activations + TestMoEQuantConfig( + quant_dtype=torch.float8_e4m3fn, + per_out_ch_quant=False, + per_act_token_quant=False, + block_shape=[128, 128], + ), + # TODO (varun) : Should we test the following combinations ? + # block-quantized weights and per-token activations + # block-quantized weights and per-tensor activations +] + +if cutlass_fp4_supported() or has_flashinfer_cutlass_fused_moe(): + MK_QUANT_CONFIGS += [ + TestMoEQuantConfig( + quant_dtype="nvfp4", + per_out_ch_quant=False, + per_act_token_quant=False, + block_shape=None, + ), + ] + + +def _slice(rank: int, num_local_experts: int, t: torch.Tensor) -> torch.Tensor: + s = rank * num_local_experts + e = s + num_local_experts + return t[s:e] + + +def make_cutlass_strides( + e: int, + n: int, + k: int, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + ab_strides1 = torch.full((e,), k, device="cuda", dtype=torch.int64) + ab_strides2 = torch.full((e,), n, device="cuda", dtype=torch.int64) + c_strides1 = torch.full((e,), 2 * n, device="cuda", dtype=torch.int64) + c_strides2 = torch.full((e,), k, device="cuda", dtype=torch.int64) + return ab_strides1, ab_strides2, c_strides1, c_strides2 + + +def make_fused_experts( + fused_experts_type: mk.FusedMoEExpertsModular, + moe: FusedMoEConfig, + quant_config: FusedMoEQuantConfig, + num_dispatchers: int, + N: int, +) -> mk.FusedMoEExpertsModular: + if ( + fused_experts_type.activation_format() + == mk.FusedMoEActivationFormat.BatchedExperts + ): + kwargs = { + "moe_config": moe, + "quant_config": quant_config, + "max_num_tokens": moe.max_num_tokens, + "num_dispatchers": num_dispatchers, + } + else: + kwargs = { + "moe_config": moe, + "quant_config": quant_config, + } + + torch.set_printoptions(threshold=0, edgeitems=0, linewidth=10000) + + print(f"Making {fused_experts_type.__class__.__name__} {kwargs} ...") + experts = fused_experts_type(**kwargs) + + torch.set_printoptions(threshold=1000, edgeitems=5, linewidth=80) + + return experts diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/parallel_utils.py b/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/parallel_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..05cec45a2d48dfb2496d0db20a7d1ea6113b52b6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/parallel_utils.py @@ -0,0 +1,166 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import dataclasses +import os +import traceback +from collections.abc import Callable +from typing import Any, Concatenate + +import torch +from torch.multiprocessing import spawn # pyright: ignore[reportPrivateImportUsage] +from typing_extensions import ParamSpec + +import vllm.envs as envs +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.distributed import ( + cleanup_dist_env_and_memory, + init_distributed_environment, + initialize_model_parallel, +) +from vllm.utils.network_utils import get_open_port + +## Parallel Processes Utils + +P = ParamSpec("P") + + +@dataclasses.dataclass +class ProcessGroupInfo: + world_size: int + world_local_size: int + rank: int + node_rank: int + local_rank: int + device: torch.device + + +def _set_vllm_config( + vllm_config: VllmConfig, world_size: int, rank: int, local_rank: int +): + import tempfile + + temp_file = tempfile.mkstemp()[1] + + # When DP is enabled, processes are organized as: + # rank = dp_rank * tp_pp_world_size + tp_pp_rank + tp_pp_world_size = vllm_config.parallel_config.world_size + vllm_config.parallel_config.data_parallel_rank = rank // tp_pp_world_size + tp_pp_rank = rank % tp_pp_world_size + vllm_config.parallel_config.rank = tp_pp_rank + + with set_current_vllm_config(vllm_config): + init_distributed_environment( + world_size=tp_pp_world_size, + rank=tp_pp_rank, + distributed_init_method=f"file://{temp_file}", + local_rank=local_rank, + backend="nccl", + ) + + initialize_model_parallel( + tensor_model_parallel_size=vllm_config.parallel_config.tensor_parallel_size, + pipeline_model_parallel_size=vllm_config.parallel_config.pipeline_parallel_size, + ) + if envs.VLLM_DISTRIBUTED_USE_SPLIT_GROUP: + cpu_group = torch.distributed.split_group( + split_ranks=[list(range(world_size))], + group_desc="moe_test_cpu", + ) + else: + cpu_group = torch.distributed.new_group( + list(range(world_size)), backend="gloo" + ) + return cpu_group + + +def _worker_parallel_launch( + local_rank: int, + world_size: int, + world_local_size: int, + node_rank: int, + init_method: str, + worker: Callable[..., None], + vllm_config: VllmConfig | None, + env_dict: dict | None, + worker_kwargs: dict[str, Any], + *args: Any, +) -> None: + rank = node_rank * world_local_size + local_rank + device = torch.device("cuda", local_rank) + torch.accelerator.set_device_index(device) + torch.set_default_device(device) + torch.distributed.init_process_group( + backend="cpu:gloo,cuda:nccl", + init_method=init_method, + rank=rank, + world_size=world_size, + device_id=device, + ) + barrier = torch.tensor([rank], device=device) + torch.distributed.all_reduce(barrier) + + if env_dict is not None: + os.environ.update(env_dict) + + cpu_group = None + if vllm_config is not None: + cpu_group = _set_vllm_config(vllm_config, world_size, rank, local_rank) + + def _run_worker(): + worker( + ProcessGroupInfo( + world_size=world_size, + world_local_size=world_local_size, + rank=rank, + node_rank=node_rank, + local_rank=local_rank, + device=device, + ), + vllm_config, + cpu_group, + *args, + **worker_kwargs, + ) + + try: + if vllm_config is not None: + with set_current_vllm_config(vllm_config): + _run_worker() + else: + _run_worker() + except Exception as ex: + print(ex) + traceback.print_exc() + raise + finally: + torch.accelerator.synchronize() + if vllm_config is not None: + cleanup_dist_env_and_memory() + else: + torch.distributed.destroy_process_group() + + +def parallel_launch_with_config( + world_size: int, + worker: Callable[Concatenate[ProcessGroupInfo, VllmConfig, Any, P], None], + vllm_config: VllmConfig, + env_dict: dict[Any, Any] | None, + *args: P.args, + **kwargs: P.kwargs, +) -> None: + spawn( + _worker_parallel_launch, + args=( + world_size, + world_size, + 0, + f"tcp://{os.getenv('LOCALHOST', 'localhost')}:{get_open_port()}", + worker, + vllm_config, + env_dict, + kwargs, + ) + + args, + nprocs=world_size, + join=True, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/profile_modular_kernel.py b/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/profile_modular_kernel.py new file mode 100644 index 0000000000000000000000000000000000000000..a1bfad430d169a846e037f1607763071ede24313 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/modular_kernel_tools/profile_modular_kernel.py @@ -0,0 +1,182 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import copy +from collections.abc import Callable +from itertools import product +from typing import Any + +import torch + +from vllm.config import VllmConfig +from vllm.forward_context import set_forward_context +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.config import FusedMoEQuantConfig +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.worker.workspace import init_workspace_manager + +from .common import ( + Config, + RankTensors, + WeightTensors, + _make_gscale, + make_modular_kernel, +) +from .parallel_utils import ProcessGroupInfo, parallel_launch_with_config + + +def do_profile( + fn: Callable, + fn_kwargs: dict[Any, Any], + pgi: ProcessGroupInfo, + config: Config, + num_warmups: int = 5, +): + for _ in range(num_warmups): + fn(**fn_kwargs) + + with torch.profiler.profile( + activities=[ + torch.profiler.ProfilerActivity.CPU, + torch.profiler.ProfilerActivity.CUDA, + ], + with_stack=True, + record_shapes=True, + ) as tprof: + fn(**fn_kwargs) + device = torch.accelerator.current_device_index() + torch.accelerator.synchronize(device) + + # TODO (varun): Add a descriptive trace file name + tprof.export_chrome_trace( + f"{config.torch_trace_dir_path}/m{config.M}_{pgi.rank}_trace.json" + ) + + +def profile_modular_kernel( + pgi: ProcessGroupInfo, + vllm_config: VllmConfig, + config: Config, + weights: WeightTensors, + rank_tensors: RankTensors, +) -> None: + assert isinstance(config.Ms, int) + assert isinstance(config.topks, int) + + # weights for rank + rank_weights = weights.slice_weights(pgi.rank, config.num_local_experts) + + if config.quant_dtype == "nvfp4": + gscale = _make_gscale(config.num_local_experts) + else: + gscale = None + + quant_config = FusedMoEQuantConfig.make( + config.quant_dtype, + w1_scale=rank_weights.w1_scale, + w2_scale=rank_weights.w2_scale, + a1_scale=rank_tensors.hidden_states_scale, + g1_alphas=(1 / rank_weights.w1_gs) if rank_weights.w1_gs is not None else None, + g2_alphas=(1 / rank_weights.w2_gs) if rank_weights.w2_gs is not None else None, + a1_gscale=gscale, + a2_gscale=gscale, + block_shape=config.quant_block_shape, + per_act_token_quant=config.is_per_act_token_quant, + per_out_ch_quant=config.is_per_out_ch_quant, + ) + + # make modular kernel + mk = make_modular_kernel(config, vllm_config, quant_config) + + topk_ids = rank_tensors.topk_ids.to( + mk.prepare_finalize.topk_indices_dtype() or rank_tensors.topk_ids.dtype + ) + + # impls might update the tensor in place + hidden_states = rank_tensors.hidden_states.clone() + + mk_kwargs = { + "hidden_states": hidden_states, + "w1": rank_weights.w1, + "w2": rank_weights.w2, + "topk_weights": rank_tensors.topk_weights, + "topk_ids": topk_ids, + "activation": MoEActivation.SILU, + "expert_map": rank_tensors.expert_map, + "global_num_experts": config.E, + "apply_router_weight_on_input": config.topk == 1 + and config.supports_apply_weight_on_input(), + } + + num_tokens = hidden_states.shape[0] + num_tokens_across_dp = torch.tensor( + [num_tokens] * config.world_size, device="cpu", dtype=torch.int + ) + + with set_forward_context( + None, + vllm_config, + num_tokens=num_tokens, + num_tokens_across_dp=num_tokens_across_dp, + ): + do_profile(mk.apply, mk_kwargs, pgi, config) + + +def rank_worker( + pgi: ProcessGroupInfo, + vllm_config: VllmConfig, + cpu_group, + config: Config, + weights: WeightTensors, +): + set_random_seed(pgi.rank) + + # workspace manager is normally initialized by GPUModelRunner; we initialize + # it here for the standalone benchmark process. + init_workspace_manager(torch.device(f"cuda:{pgi.local_rank}")) + + # get weights to this device + weights.to_current_device() + + Ms = config.Ms + assert isinstance(Ms, list) + TOPKs = config.topks + assert isinstance(TOPKs, list) + + for m, topk in product(Ms, TOPKs): + print(f"Running m={m}, topk={topk} ...") + # override m and topk + cfgx = copy.deepcopy(config) + cfgx.Ms = m + cfgx.topks = topk + + # inputs for rank + rank_tensors = RankTensors.make(cfgx, pgi) + profile_modular_kernel(pgi, vllm_config, cfgx, weights, rank_tensors) + + +def run(config: Config): + weights: WeightTensors = WeightTensors.make(config) + vllm_config, env_dict = config.make_env_data() + parallel_launch_with_config( + config.world_size, rank_worker, vllm_config, env_dict, config, weights + ) + + +if __name__ == "__main__": + from .cli_args import make_config, make_config_arg_parser + + parser = make_config_arg_parser( + description=( + "Run single prepare-finalize & fused-experts combination test" + "Example : python3 -m tests.kernels.moe.modular_kernel_tools.profile_modular_kernel " # noqa: E501 + "--pf-type DeepEPLLPrepareAndFinalize --experts-type BatchedTritonExperts" + ) + ) + args = parser.parse_args() + assert args.torch_trace_dir_path is not None, ( + "Please pass in a directory to store torch traces" + ) + config = make_config(args) + + run(config) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/parallel_utils.py b/asian_ai/performance/vLLM/tests/kernels/moe/parallel_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..7b4b831ea5429aac6c647640e74e7f80c334278e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/parallel_utils.py @@ -0,0 +1,245 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +DeepEP test utilities +""" + +import dataclasses +import os +import traceback +from collections.abc import Callable +from typing import Concatenate + +import torch +from torch.distributed import ProcessGroup +from torch.multiprocessing import spawn # pyright: ignore[reportPrivateImportUsage] +from typing_extensions import ParamSpec + +from vllm.utils.import_utils import has_deep_ep, has_deep_ep_v2 +from vllm.utils.network_utils import get_open_port + +if has_deep_ep(): + from vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ht import ( + DeepEPHTPrepareAndFinalize, + ) + from vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ll import ( + DeepEPLLPrepareAndFinalize, + ) + +if has_deep_ep_v2(): + from vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_v2 import ( + DeepEPV2PrepareAndFinalize, + ) + +## Parallel Processes Utils + +P = ParamSpec("P") + + +@dataclasses.dataclass +class ProcessGroupInfo: + world_size: int + world_local_size: int + rank: int + node_rank: int + local_rank: int + device: torch.device + + +def _worker_parallel_launch( + local_rank: int, + world_size: int, + world_local_size: int, + node_rank: int, + init_method: str, + worker: Callable[Concatenate[ProcessGroupInfo, P], None], + *args: P.args, + **kwargs: P.kwargs, +) -> None: + rank = node_rank * world_local_size + local_rank + torch.accelerator.set_device_index(local_rank) + device = torch.device("cuda", local_rank) + torch.distributed.init_process_group( + backend="nccl", + init_method=init_method, + rank=rank, + world_size=world_size, + ) + barrier = torch.tensor([rank], device=device) + torch.distributed.all_reduce(barrier) + + try: + worker( + ProcessGroupInfo( + world_size=world_size, + world_local_size=world_local_size, + rank=rank, + node_rank=node_rank, + local_rank=local_rank, + device=device, + ), + *args, + **kwargs, + ) + except Exception as ex: + print(ex) + traceback.print_exc() + raise + finally: + torch.distributed.destroy_process_group() + + +def parallel_launch( + world_size: int, + worker: Callable[Concatenate[ProcessGroupInfo, P], None], + *args: P.args, + **kwargs: P.kwargs, +) -> None: + assert not kwargs + spawn( + _worker_parallel_launch, + args=( + world_size, + world_size, + 0, + f"tcp://{os.getenv('LOCALHOST', 'localhost')}:{get_open_port()}", + worker, + ) + + args, + nprocs=world_size, + join=True, + ) + + +## DeepEP specific utils + + +@dataclasses.dataclass +class DeepEPHTArgs: + num_local_experts: int + + +@dataclasses.dataclass +class DeepEPLLArgs: + max_tokens_per_rank: int + hidden_size: int + num_experts: int + use_fp8_dispatch: bool + + +def make_deepep_ht_a2a( + pg: ProcessGroup, + pgi: ProcessGroupInfo, + dp_size: int, + ht_args: DeepEPHTArgs, + q_dtype: torch.dtype | None = None, + block_shape: list[int] | None = None, +): + import deep_ep + + # high throughput a2a + num_nvl_bytes = 1024 * 1024 * 1024 # 1GB + num_rdma_bytes, low_latency_mode, num_qps_per_rank = 0, False, 1 + buffer = deep_ep.Buffer( + group=pg, + num_nvl_bytes=num_nvl_bytes, + num_rdma_bytes=num_rdma_bytes, + low_latency_mode=low_latency_mode, + num_qps_per_rank=num_qps_per_rank, + ) + return DeepEPHTPrepareAndFinalize( + buffer=buffer, + num_dispatchers=pgi.world_size, + dp_size=dp_size, + rank_expert_offset=pgi.rank * ht_args.num_local_experts, + ) + + +def make_deepep_ll_a2a( + pg: ProcessGroup, + pgi: ProcessGroupInfo, + deepep_ll_args: DeepEPLLArgs, + q_dtype: torch.dtype | None = None, + block_shape: list[int] | None = None, +): + import deep_ep + + # low-latency a2a + num_rdma_bytes = deep_ep.Buffer.get_low_latency_rdma_size_hint( + deepep_ll_args.max_tokens_per_rank, + deepep_ll_args.hidden_size, + pgi.world_size, + deepep_ll_args.num_experts, + ) + + buffer = deep_ep.Buffer( + group=pg, + num_rdma_bytes=num_rdma_bytes, + low_latency_mode=True, + num_qps_per_rank=deepep_ll_args.num_experts // pgi.world_size, + ) + + return DeepEPLLPrepareAndFinalize( + buffer=buffer, + num_dispatchers=pgi.world_size, + max_tokens_per_rank=deepep_ll_args.max_tokens_per_rank, + use_fp8_dispatch=deepep_ll_args.use_fp8_dispatch, + ) + + +def make_deepep_a2a( + pg: ProcessGroup, + pgi: ProcessGroupInfo, + dp_size: int, + deepep_ht_args: DeepEPHTArgs | None, + deepep_ll_args: DeepEPLLArgs | None, + q_dtype: torch.dtype | None = None, + block_shape: list[int] | None = None, +): + if deepep_ht_args is not None: + assert deepep_ll_args is None + return make_deepep_ht_a2a( + pg, pgi, dp_size, deepep_ht_args, q_dtype, block_shape + ) + + assert deepep_ll_args is not None + return make_deepep_ll_a2a(pg, pgi, deepep_ll_args, q_dtype, block_shape) + + +@dataclasses.dataclass +class DeepEPV2Args: + num_local_experts: int + num_experts: int + num_topk: int + hidden_size: int + max_tokens_per_rank: int + use_fp8_dispatch: bool + + +def make_deepep_v2_a2a( + pg: ProcessGroup, + pgi: ProcessGroupInfo, + dp_size: int, + v2_args: DeepEPV2Args, + use_cudagraph: bool = False, +): + import deep_ep + + buffer = deep_ep.ElasticBuffer( + group=pg, + num_max_tokens_per_rank=v2_args.max_tokens_per_rank, + hidden=v2_args.hidden_size, + num_topk=v2_args.num_topk, + use_fp8_dispatch=v2_args.use_fp8_dispatch, + explicitly_destroy=True, + ) + return DeepEPV2PrepareAndFinalize( + buffer=buffer, + num_dispatchers=pgi.world_size, + dp_size=dp_size, + rank_expert_offset=pgi.rank * v2_args.num_local_experts, + num_experts=v2_args.num_experts, + num_topk=v2_args.num_topk, + use_fp8_dispatch=v2_args.use_fp8_dispatch, + use_cudagraph=use_cudagraph, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_batched_deepgemm.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_batched_deepgemm.py new file mode 100644 index 0000000000000000000000000000000000000000..032a65fe8be869f3b84ea57a5b17bcbda96dc93f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_batched_deepgemm.py @@ -0,0 +1,122 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.config import fp8_w8a8_moe_quant_config +from vllm.model_executor.layers.fused_moe.experts.batched_deep_gemm_moe import ( + BatchedDeepGemmExperts, +) +from vllm.model_executor.layers.fused_moe.experts.fused_batched_moe import ( + BatchedTritonExperts, +) +from vllm.model_executor.layers.fused_moe.modular_kernel import FusedMoEKernel +from vllm.model_executor.layers.fused_moe.prepare_finalize.batched import ( + BatchedPrepareAndFinalize, +) +from vllm.utils.deep_gemm import calc_diff, is_deep_gemm_supported + +from .test_deepgemm import make_block_quant_fp8_weights +from .utils import make_dummy_moe_config + +BLOCK_SIZE = [128, 128] + + +@pytest.mark.skipif(not is_deep_gemm_supported(), reason="Requires deep_gemm kernels") +@pytest.mark.parametrize("E", [16, 32]) # number of experts +@pytest.mark.parametrize("T", [256, 512]) # tokens per expert +@pytest.mark.parametrize("K", [128, 256]) # hidden dim +@pytest.mark.parametrize("N", [512, 1024]) # intermediate dim per expert +@pytest.mark.parametrize("topk", [2, 4]) +def test_batched_deepgemm_vs_triton( + E: int, T: int, K: int, N: int, topk: int, monkeypatch, workspace_init +): + """Compare BatchedDeepGemmExperts to BatchedTritonExperts.""" + + monkeypatch.setenv("VLLM_USE_DEEP_GEMM", "1") + + device = "cuda" + w1, w2, w1_s, w2_s = make_block_quant_fp8_weights(E, N, K, BLOCK_SIZE) + + M = E * T # total tokens + a = torch.randn(M, K, device=device, dtype=torch.bfloat16) / 10.0 + fp8_info = torch.finfo(torch.float8_e4m3fn) + a.clamp_(fp8_info.min, fp8_info.max) + + # random router outputs → top-k indices / weights + router_logits = torch.randn(M, E, device=device, dtype=torch.float32) + topk_weights, topk_ids = torch.topk(router_logits, k=topk, dim=-1) + topk_weights = torch.nn.functional.softmax(topk_weights, dim=-1) + + # token number for each expert + cnt = torch.bincount(topk_ids.flatten(), minlength=E) + max_cnt = int(cnt.max().item()) + # next power of 2 for max token number + max_num_tokens = 1 << (max_cnt - 1).bit_length() + + prep_finalize = BatchedPrepareAndFinalize( + max_num_tokens=max_num_tokens, + num_local_experts=E, + num_dispatchers=1, + rank=0, + ) + + quant_config = fp8_w8a8_moe_quant_config( + w1_scale=w1_s, + w2_scale=w2_s, + per_act_token_quant=False, + block_shape=BLOCK_SIZE, + ) + + # triton (reference) + triton_experts = BatchedTritonExperts( + max_num_tokens=max_num_tokens, + num_dispatchers=1, + quant_config=quant_config, + moe_config=make_dummy_moe_config(), + ) + mk_triton = FusedMoEKernel( + prep_finalize, + triton_experts, + ) + + out_triton = mk_triton.apply( + hidden_states=a, + w1=w1, + w2=w2, + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=MoEActivation.SILU, + global_num_experts=E, + expert_map=None, + apply_router_weight_on_input=False, + ) + + # deepgemm + deepgemm_experts = BatchedDeepGemmExperts( + max_num_tokens=max_num_tokens, + num_dispatchers=1, + quant_config=quant_config, + moe_config=make_dummy_moe_config(), + ) + mk_deepgemm = FusedMoEKernel( + prep_finalize, + deepgemm_experts, + ) + + out_deepgemm = mk_deepgemm.apply( + hidden_states=a, + w1=w1, + w2=w2, + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=MoEActivation.SILU, + global_num_experts=E, + expert_map=None, + apply_router_weight_on_input=False, + ) + + diff = calc_diff(out_deepgemm, out_triton) + assert diff < 1e-3, f"Output diff too large: {diff}" diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_batched_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_batched_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..7155f641e053e13a3d253256be8e53f41ac130dd --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_batched_moe.py @@ -0,0 +1,353 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +from dataclasses import dataclass + +import pytest +import torch + +from tests.kernels.moe.utils import ( + batched_moe, + make_quantized_test_activations, + make_test_weights, + naive_batched_moe, +) +from tests.kernels.quant_utils import native_batched_masked_quant_matmul +from tests.kernels.utils import torch_experts +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.model_executor.layers.fused_moe import fused_topk +from vllm.model_executor.layers.fused_moe.experts.fused_batched_moe import ( + invoke_moe_batched_triton_kernel, +) +from vllm.platforms import current_platform +from vllm.triton_utils import tl +from vllm.utils.torch_utils import set_random_seed + +MNK_FACTORS = [ + (1, 128, 128), + (1, 512, 512), + (1, 1024, 2048), + (32, 128, 128), + (32, 512, 512), + (32, 1024, 2048), + (45, 128, 2048), + (45, 1024, 128), + (64, 512, 512), + (64, 1024, 2048), + (222, 128, 2048), + (222, 1024, 2048), +] +NUM_EXPERTS = [8, 64] +TOP_KS = [1, 2, 6] + +DTYPES = [torch.bfloat16] + +if not current_platform.is_fp8_fnuz(): + DTYPES.append(torch.float8_e4m3fn) + +vllm_config = VllmConfig() + + +@dataclass +class BatchedMMConfig: + in_dtype: torch.dtype + quant_dtype: torch.dtype | None + out_dtype: torch.dtype + num_experts: int + max_tokens_per_expert: int + K: int + N: int + + +@dataclass +class BatchedMMTensors: + A: torch.Tensor # [E, max_tokens, K] + B: torch.Tensor # [E, K, N] - column major + C: torch.Tensor # [E, max_tokens, N] + num_expert_tokens: torch.Tensor # [E] + + @staticmethod + def make_tensors(config: BatchedMMConfig): + A = ( + torch.randn( + (config.num_experts, config.max_tokens_per_expert, config.K), + device="cuda", + dtype=config.in_dtype, + ) + / 10 + ) + B = torch.randn( + (config.num_experts, config.N, config.K), + device="cuda", + dtype=config.in_dtype, + ) + C = torch.zeros( + (config.num_experts, config.max_tokens_per_expert, config.N), + device="cuda", + dtype=config.out_dtype, + ) + + num_expert_tokens = torch.randint( + low=0, + high=config.max_tokens_per_expert, + size=(config.num_experts,), + device="cuda", + dtype=torch.int32, + ) + + return BatchedMMTensors(A, B, C, num_expert_tokens) + + +@pytest.mark.parametrize("num_experts", [8, 32]) +@pytest.mark.parametrize("max_tokens_per_expert", [32, 224, 512]) +@pytest.mark.parametrize("K", [128, 1024]) +@pytest.mark.parametrize("N", [128, 1024]) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("block_shape", [None, [128, 128]]) +@pytest.mark.parametrize("per_act_token_quant", [False, True]) +def test_batched_mm( + num_experts: int, + max_tokens_per_expert: int, + K: int, + N: int, + dtype: torch.dtype, + block_shape: list[int] | None, + per_act_token_quant: bool, +): + """Note: float8_e4m3fn is not supported on CUDA architecture < 89, + and those tests will be skipped on unsupported hardware.""" + set_random_seed(7) + + use_fp8_w8a8 = dtype == torch.float8_e4m3fn + + if (dtype == torch.float8_e4m3fn) and not current_platform.has_device_capability( + 89 + ): + pytest.skip( + "Triton limitation: fp8e4nv data type is not supported on CUDA arch < 89" + ) + + if (per_act_token_quant or block_shape is not None) and not use_fp8_w8a8: + pytest.skip("Don't test blocking for non-quantized types.") + + if per_act_token_quant and block_shape is not None: + pytest.skip("Skip illegal quantization test.") + + if dtype.itemsize == 1: + act_dtype = torch.bfloat16 + quant_dtype = dtype + else: + act_dtype = dtype + quant_dtype = None + + num_expert_tokens = torch.randint( + low=0, + high=max_tokens_per_expert, + size=(num_experts,), + device="cuda", + dtype=torch.int32, + ) + + A, A_q, A_scale = make_quantized_test_activations( + num_experts, + max_tokens_per_expert, + K, + in_dtype=act_dtype, + quant_dtype=quant_dtype, + block_shape=block_shape, + per_act_token_quant=per_act_token_quant, + ) + + (B, B_q, B_scale, _), _ = make_test_weights( + num_experts, + N // 2, + K, + in_dtype=act_dtype, + quant_dtype=quant_dtype, + block_shape=block_shape, + per_out_ch_quant=per_act_token_quant, + ) + + out_shape = (num_experts, max_tokens_per_expert, N) + test_output = torch.zeros(out_shape, dtype=act_dtype, device="cuda") + ref_output = torch.zeros(out_shape, dtype=act_dtype, device="cuda") + q_ref_output = torch.zeros(out_shape, dtype=act_dtype, device="cuda") + + compute_tl_dtype = { + torch.float16: tl.float16, + torch.bfloat16: tl.bfloat16, + torch.float32: tl.float32, + }[test_output.dtype] + + assert A_q.dtype == B_q.dtype + + invoke_moe_batched_triton_kernel( + A_q, + B_q, + test_output, + num_expert_tokens, + compute_tl_dtype, + # Quantization data + A_scale, + B_scale, + None, + # Quantization schemes + use_fp8_w8a8, + False, + False, + config={ + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 16, + "BLOCK_SIZE_K": 16 if dtype.itemsize > 1 else 32, + }, + per_act_token_quant=per_act_token_quant, + block_shape=block_shape, + ) + + ref_output = native_batched_masked_quant_matmul( + A, + B, + ref_output, + num_expert_tokens, + ) + + q_ref_output = native_batched_masked_quant_matmul( + A_q, + B_q, + q_ref_output, + num_expert_tokens, + A_scale, + B_scale, + block_shape, + per_act_token_quant, + ) + + rtol, atol = { + torch.float16: (6e-2, 6e-2), + torch.bfloat16: (6e-2, 6e-2), + torch.float32: (1e-2, 1e-2), + }[test_output.dtype] + + torch.testing.assert_close(ref_output, q_ref_output, atol=atol, rtol=rtol) + torch.testing.assert_close(test_output, q_ref_output, atol=atol, rtol=rtol) + + +@pytest.mark.parametrize(("m", "n", "k"), MNK_FACTORS) +@pytest.mark.parametrize("e", NUM_EXPERTS) +@pytest.mark.parametrize("topk", TOP_KS) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("per_act_token_quant", [False, True]) +@pytest.mark.parametrize("block_shape", [None, [128, 128]]) +@pytest.mark.parametrize("input_scales", [False]) +def test_fused_moe_batched_experts( + m: int, + n: int, + k: int, + e: int, + topk: int, + dtype: torch.dtype, + per_act_token_quant: bool, + block_shape: list[int] | None, + input_scales: bool, + workspace_init, +): + """Note: float8_e4m3fn is not supported on CUDA architecture < 89, + and those tests will be skipped on unsupported hardware.""" + set_random_seed(7) + + use_fp8_w8a8 = dtype == torch.float8_e4m3fn + + if (dtype == torch.float8_e4m3fn) and not current_platform.has_device_capability( + 89 + ): + pytest.skip( + "Triton limitation: fp8e4nv data type is not supported on CUDA arch < 89" + ) + + if topk > e: + pytest.skip("topk > e") + + if not use_fp8_w8a8 and (per_act_token_quant or block_shape is not None): + pytest.skip("Skip quantization test for non-quantized type") + + if per_act_token_quant and block_shape is not None: + pytest.skip("Skip illegal quantization test.") + + a = torch.randn((m, k), device="cuda", dtype=torch.bfloat16) / 10 + score = torch.randn((m, e), device="cuda", dtype=torch.bfloat16) + + if dtype.itemsize == 1: + act_dtype = torch.bfloat16 + quant_dtype = dtype + else: + act_dtype = dtype + quant_dtype = None + + (w1_16, w1, w1_s, _), (w2_16, w2, w2_s, _) = make_test_weights( + e, + n, + k, + block_shape=block_shape, + in_dtype=act_dtype, + quant_dtype=quant_dtype, + per_out_ch_quant=per_act_token_quant, + ) + + if input_scales and quant_dtype is not None: + a1_scale = torch.tensor(1, device="cuda", dtype=torch.float32) + a2_scale = torch.tensor(1, device="cuda", dtype=torch.float32) + else: + a1_scale = None + a2_scale = None + + with set_current_vllm_config(vllm_config): + topk_weight, topk_ids, _ = fused_topk(a, score, topk, False) + + baseline_output = torch_experts( + a, + w1, + w2, + topk_weight, + topk_ids, + w1_scale=w1_s, + w2_scale=w2_s, + a1_scale=a1_scale, + a2_scale=a2_scale, + quant_dtype=quant_dtype, + per_act_token_quant=per_act_token_quant, + block_shape=block_shape, + ) + + batched_output = naive_batched_moe( + a, + w1, + w2, + topk_weight, + topk_ids, + w1_scale=w1_s, + w2_scale=w2_s, + a1_scale=a1_scale, + a2_scale=a2_scale, + quant_dtype=quant_dtype, + per_act_token_quant=per_act_token_quant, + block_shape=block_shape, + ) + + triton_output = batched_moe( + a, + w1, + w2, + topk_weight, + topk_ids, + w1_scale=w1_s, + w2_scale=w2_s, + a1_scale=a1_scale, + a2_scale=a2_scale, + quant_dtype=quant_dtype, + per_act_token_quant=per_act_token_quant, + block_shape=block_shape, + ) + + torch.testing.assert_close(batched_output, baseline_output, atol=3e-2, rtol=2e-2) + + torch.testing.assert_close(triton_output, batched_output, atol=2e-2, rtol=2e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_block_fp8.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_block_fp8.py new file mode 100644 index 0000000000000000000000000000000000000000..b4ba09a3063181ed7bf7a09432af2527a2556cd8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_block_fp8.py @@ -0,0 +1,317 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +import vllm.model_executor.layers.fused_moe.modular_kernel as mk +from tests.kernels.moe.utils import ( + make_dummy_moe_config, + make_test_quant_config, + make_test_weights, + modular_triton_fused_moe, +) +from tests.kernels.quant_utils import ( + native_per_token_group_quant_fp8, + native_w8a8_block_matmul, +) +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.model_executor.layers.activation import SiluAndMul +from vllm.model_executor.layers.fused_moe import ( + fused_experts, + fused_topk, +) +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.all2all_utils import ( + maybe_make_prepare_finalize, +) +from vllm.model_executor.layers.fused_moe.config import ( + fp8_w8a8_moe_quant_config, +) +from vllm.model_executor.layers.fused_moe.experts.deep_gemm_moe import ( + _valid_deep_gemm_shape, +) +from vllm.model_executor.layers.fused_moe.experts.triton_deep_gemm_moe import ( + TritonOrDeepGemmExperts, +) +from vllm.platforms import current_platform +from vllm.utils.deep_gemm import ( + get_mk_alignment_for_contiguous_layout, + is_deep_gemm_e8m0_used, +) +from vllm.utils.import_utils import has_deep_gemm + +dg_available = has_deep_gemm() + +if current_platform.get_device_capability() < (9, 0): + pytest.skip("FP8 Triton requires CUDA 9.0 or higher", allow_module_level=True) +if current_platform.is_fp8_fnuz(): + pytest.skip( + "Tests in this file require float8_e4m3fn and platform does not support", + allow_module_level=True, + ) + +vllm_config = VllmConfig() + +# Test configurations +DTYPES = [torch.bfloat16] # [torch.half, torch.bfloat16, torch.float32] +# Deepseek-V3's intermediate size 18432, so N is 18432*2/8=4608 at TP8 +# and its hidden size is 7168. +MNK_FACTORS = [ + (1, 128, 128), + (1, 128, 7168), + (1, 1024, 7168), + (1, 4608, 128), + (1, 4608, 7168), + (83, 128, 128), + (83, 512, 512), + (83, 4608, 512), + (83, 4608, 7168), + (128, 512, 512), + (128, 1024, 7168), + (128, 4608, 7168), + (2048, 128, 128), + (2048, 1024, 7168), + (2048, 4608, 512), + (2048, 4608, 7168), + (8192, 128, 128), + (8192, 128, 7168), + (8192, 1024, 7168), + (8192, 4608, 7168), +] + +MNK_FACTORS_DG = [ + (128, 128, 128), + (128, 128, 7168), + (128, 1024, 7168), + (128, 4608, 128), + (128, 4608, 7168), + (192, 512, 512), + (192, 1024, 7168), + (192, 4608, 7168), + (1335, 128, 128), + (1335, 1024, 7168), + (1335, 4608, 512), + (1335, 4608, 7168), + (2048, 128, 128), + (2048, 128, 7168), + (2048, 1024, 7168), + (2048, 4608, 7168), +] + +BLOCK_SIZE = [[128, 128]] +E = [2, 8, 16] # [128, 256] +TOP_KS = [1, 2, 6] +SEEDS = [0] + + +def torch_w8a8_block_fp8_moe(a, w1, w2, w1_s, w2_s, topk_weight, topk_ids, block_shape): + """Fused moe with block-wise quantization using native torch.""" + B, D = a.shape + topk = topk_ids.size(1) + a = a.view(B, -1, D).repeat(1, topk, 1).reshape(-1, D) + out = torch.zeros(B * topk, w2.shape[1], dtype=a.dtype, device=a.device) + + topk_weight = topk_weight.view(-1) + topk_ids = topk_ids.view(-1) + + _, block_k = block_shape[0], block_shape[1] + a_q, a_s = native_per_token_group_quant_fp8(a, block_k) + a_q = a_q.to(torch.float32) + for i in range(w1.shape[0]): + mask = topk_ids == i + if mask.sum(): + inter_out = native_w8a8_block_matmul( + a_q[mask], w1[i], a_s[mask], w1_s[i], block_shape, output_dtype=a.dtype + ) + act_out = SiluAndMul().forward_native(inter_out) + act_out_q, act_out_s = native_per_token_group_quant_fp8(act_out, block_k) + out[mask] = native_w8a8_block_matmul( + act_out_q, w2[i], act_out_s, w2_s[i], block_shape, output_dtype=a.dtype + ) + return ( + out.view(B, -1, w2.shape[1]) * topk_weight.view(B, -1, 1).to(out.dtype) + ).sum(dim=1) + + +# Skip all tests if CUDA is not available +pytest.importorskip("torch.cuda") + + +@pytest.fixture(autouse=True) +def setup_cuda(): + torch.set_default_device("cuda") + + +@pytest.mark.parametrize(("M", "N", "K"), MNK_FACTORS) +@pytest.mark.parametrize("E", E) +@pytest.mark.parametrize("topk", TOP_KS) +@pytest.mark.parametrize("block_size", BLOCK_SIZE) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@torch.inference_mode() +def test_w8a8_block_fp8_fused_moe( + M, N, K, E, topk, block_size, dtype, seed, monkeypatch, workspace_init +): + if topk > E: + pytest.skip(f"Skipping test; topk={topk} > E={E}") + + torch.manual_seed(seed) + + a = torch.randn((M, K), dtype=dtype) / 10 + score = torch.randn((M, E), dtype=dtype) + + w1, w2, quant_config = make_test_quant_config( + E, + N, + K, + dtype, + quant_dtype=torch.float8_e4m3fn, + per_act_token_quant=False, + block_shape=block_size, + ) + + m_fused_moe = modular_triton_fused_moe(make_dummy_moe_config(), quant_config) + + topk_weights, topk_ids, _ = fused_topk(a, score.float(), topk, False) + + # Set the context to avoid lots of warning spam. + with set_current_vllm_config(vllm_config): + ref_out = torch_w8a8_block_fp8_moe( + a, + w1, + w2, + quant_config.w1_scale, + quant_config.w2_scale, + topk_weights, + topk_ids, + block_size, + ) + + out = fused_experts( + a, w1, w2, topk_weights, topk_ids, quant_config=quant_config + ) + + m_out = m_fused_moe.apply( + a, + w1, + w2, + topk_weights, + topk_ids, + activation=MoEActivation.SILU, + apply_router_weight_on_input=False, + expert_map=None, + global_num_experts=w1.shape[0], + ) + + # 0.039 only needed for M >= 8192 + tol = 0.035 if M < 8192 else 0.039 + torch.testing.assert_close(out, ref_out, atol=tol, rtol=tol) + torch.testing.assert_close(m_out, ref_out, atol=tol, rtol=tol) + + +@pytest.mark.parametrize(("M", "N", "K"), MNK_FACTORS_DG) +@pytest.mark.parametrize("E", E) +@pytest.mark.parametrize("topk", TOP_KS) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.skipif(not dg_available, reason="DeepGemm kernels not available.") +@pytest.mark.skipif(is_deep_gemm_e8m0_used(), reason="Not E8M0 scale MOE") +@torch.inference_mode() +def test_w8a8_block_fp8_deep_gemm_fused_moe(M, N, K, E, topk, seed, monkeypatch): + if topk > E: + pytest.skip(f"Skipping test: topk={topk} > E={E}") + + if not _valid_deep_gemm_shape(M, N, K): + pytest.skip(f"Skipping test: invalid size m={M}, n={N}, k={K}") + + torch.manual_seed(seed) + + block_size = get_mk_alignment_for_contiguous_layout() + dtype = torch.bfloat16 + + a = torch.randn((M, K), dtype=dtype) / 10 + score = torch.randn((M, E), dtype=dtype) + + (_, w1, w1_s, _), (_, w2, w2_s, _) = make_test_weights( + E, + N, + K, + dtype, + torch.float8_e4m3fn, + per_out_ch_quant=False, + block_shape=block_size, + ) + + # Note: for now use_compile will error out if the problem size is + # large enough to trigger chunking. I'm leaving the flag and + # setup code in case we are able to revisit this later. + use_compile = False + + use_cudagraph = N >= 1024 and K >= 1024 and current_platform.is_cuda_alike() + + topk_weights, topk_ids, _ = fused_topk(a, score.float(), topk, False) + + quant_config = fp8_w8a8_moe_quant_config( + w1_scale=w1_s, + w2_scale=w2_s, + block_shape=block_size, + ) + moe_config = make_dummy_moe_config() + + deep_gemm_experts = mk.FusedMoEKernel( + prepare_finalize=maybe_make_prepare_finalize( + moe=moe_config, + quant_config=quant_config, + allow_new_interface=True, + use_monolithic=False, + ), + fused_experts=TritonOrDeepGemmExperts( + moe_config=moe_config, + quant_config=quant_config, + ), + ) + + def deep_gemm_moe_fp8(a, w1, w2, w1_s, w2_s, topk_weights, topk_ids): + return deep_gemm_experts.apply( + hidden_states=a, + w1=w1, + w2=w2, + topk_weights=topk_weights, + topk_ids=topk_ids, + global_num_experts=E, + activation=MoEActivation.SILU, + apply_router_weight_on_input=False, + expert_map=False, + ) + + # Set the context to avoid lots of warning spam. + with set_current_vllm_config(vllm_config): + ref_out = torch_w8a8_block_fp8_moe( + a, w1, w2, w1_s, w2_s, topk_weights, topk_ids, block_size + ) + + if use_compile: + deep_gemm_moe_fp8_fn = torch.compile( + deep_gemm_moe_fp8, backend="inductor", fullgraph=True + ) + torch._dynamo.mark_dynamic(a, 0) + torch._dynamo.mark_dynamic(topk_weights, 0) + torch._dynamo.mark_dynamic(topk_ids, 0) + else: + deep_gemm_moe_fp8_fn = deep_gemm_moe_fp8 + + out = deep_gemm_moe_fp8_fn(a, w1, w2, w1_s, w2_s, topk_weights, topk_ids) + + if use_cudagraph: + out.fill_(0) + stream = torch.cuda.Stream() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + out = deep_gemm_moe_fp8_fn( + a, w1, w2, w1_s, w2_s, topk_weights, topk_ids + ) + torch.accelerator.synchronize() + graph.replay() + torch.accelerator.synchronize() + + torch.testing.assert_close(out, ref_out, atol=0.035, rtol=0.035) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_block_int8.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_block_int8.py new file mode 100644 index 0000000000000000000000000000000000000000..f15a1530ba6b394fed6cea9e0b5c12d7599e325e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_block_int8.py @@ -0,0 +1,134 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from tests.kernels.moe.utils import make_test_quant_config +from tests.kernels.quant_utils import ( + native_per_token_group_quant_int8, + native_w8a8_block_matmul, +) +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.model_executor.layers.activation import SiluAndMul +from vllm.model_executor.layers.fused_moe import fused_experts, fused_topk +from vllm.platforms import current_platform + +if current_platform.get_device_capability() < (7, 0): + pytest.skip("INT8 Triton requires CUDA 7.0 or higher", allow_module_level=True) + +vllm_config = VllmConfig() + +DTYPES = [torch.bfloat16] + +MNK_FACTORS = [ + (1, 128, 128), + (1, 128, 7168), + (1, 1024, 7168), + (1, 4096, 512), + (1, 4096, 7168), + (33, 512, 512), + (33, 128, 7168), + (33, 1024, 7168), + (33, 4096, 128), + (33, 4096, 7168), + (128, 128, 128), + (128, 1024, 7168), + (128, 4096, 512), + (128, 4096, 7168), + (222, 512, 512), + (222, 1024, 7168), + (222, 4096, 7168), + (2048, 128, 128), + (2048, 1024, 7168), + (2048, 4096, 4096), +] + +E = [8, 24] +TOP_KS = [2, 6] +# BLOCK_SIZE = [[64, 64], [64, 128], [128, 64], [128, 128]] +BLOCK_SIZE = [[128, 128]] +SEEDS = [0] + + +# For test +def torch_w8a8_block_int8_moe(a, w1, w2, w1_s, w2_s, score, topk, block_shape): + """This function performs fused moe with block-wise quantization using + native torch.""" + B, D = a.shape + a = a.view(B, -1, D).repeat(1, topk, 1).reshape(-1, D) + out = torch.zeros(B * topk, w2.shape[1], dtype=a.dtype, device=a.device) + score = torch.softmax(score, dim=-1, dtype=torch.float32) + topk_weight, topk_ids = torch.topk(score, topk) + topk_weight = topk_weight.view(-1) + topk_ids = topk_ids.view(-1) + + _, block_k = block_shape[0], block_shape[1] + a_q, a_s = native_per_token_group_quant_int8(a, block_k) + for i in range(w1.shape[0]): + mask = topk_ids == i + if mask.sum(): + inter_out = native_w8a8_block_matmul( + a_q[mask], w1[i], a_s[mask], w1_s[i], block_shape, output_dtype=a.dtype + ) + act_out = SiluAndMul().forward_native(inter_out) + act_out_q, act_out_s = native_per_token_group_quant_int8(act_out, block_k) + act_out = act_out.to(torch.float32) + out[mask] = native_w8a8_block_matmul( + act_out_q, w2[i], act_out_s, w2_s[i], block_shape, output_dtype=a.dtype + ) + return ( + out.view(B, -1, w2.shape[1]) * topk_weight.view(B, -1, 1).to(out.dtype) + ).sum(dim=1) + + +@pytest.fixture(autouse=True, scope="module") +def setup_cuda(): + """Sets the default CUDA device for all tests in this module.""" + torch.set_default_device("cuda") + + +@pytest.mark.parametrize(("M", "N", "K"), MNK_FACTORS) +@pytest.mark.parametrize("E", E) +@pytest.mark.parametrize("topk", TOP_KS) +@pytest.mark.parametrize("block_size", BLOCK_SIZE) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@torch.inference_mode() +def test_w8a8_block_int8_fused_moe(M, N, K, E, topk, block_size, dtype, seed): + """Tests the fused_moe kernel with W8A8 INT8 block quantization against a + native torch reference.""" + torch.manual_seed(seed) + + a = torch.randn((M, K), dtype=dtype) / 10 + score = torch.randn((M, E), dtype=dtype) + topk_weights, topk_ids, _ = fused_topk(a, score.float(), topk, False) + + w1, w2, quant_config = make_test_quant_config( + E, + N, + K, + dtype, + quant_dtype=torch.int8, + per_act_token_quant=False, + block_shape=block_size, + ) + + # Set the context to avoid lots of warning spam. + with set_current_vllm_config(vllm_config): + out = fused_experts( + a, w1, w2, topk_weights, topk_ids, quant_config=quant_config + ) + ref_out = torch_w8a8_block_int8_moe( + a, + w1, + w2, + quant_config.w1_scale, + quant_config.w2_scale, + score, + topk, + block_size, + ) + + # Check results + torch.testing.assert_close(out, ref_out, atol=0.065, rtol=0.065) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_count_expert_num_tokens.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_count_expert_num_tokens.py new file mode 100644 index 0000000000000000000000000000000000000000..9768b84c1f7bf82cddb0dac1e2264ee0910fe888 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_count_expert_num_tokens.py @@ -0,0 +1,143 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Tests compute_expert_num_tokens kernels +""" + +import dataclasses + +import pytest +import torch + +from vllm.model_executor.layers.fused_moe.utils import count_expert_num_tokens + + +@dataclasses.dataclass +class TestTensors: + topk_ids: torch.Tensor + expert_map: torch.Tensor | None = None + + def to_device(self, device: str): + self.topk_ids = self.topk_ids.to(device=device) + if self.expert_map is not None: + self.expert_map = self.expert_map.to(device=device) + + @staticmethod + def make( + num_tokens: int, + num_topk: int, + num_experts: int, + device: str, + topk_ids_dtype: torch.dtype, + ) -> "TestTensors": + # make topk ids + topk_ids = torch.empty((num_tokens, num_topk), device=device, dtype=torch.int64) + for x in range(num_tokens): + topk_ids[x] = torch.randperm(num_experts)[:num_topk] + topk_ids = topk_ids.to(dtype=torch.int64) + return TestTensors(topk_ids=topk_ids) + + def with_ep_rank( + self, ep_rank: int, num_global_experts: int, num_local_experts: int, device: str + ): + # make an expert map + expert_map = torch.empty((num_global_experts), device=device, dtype=torch.int32) + expert_map.fill_(-1) + s = ep_rank * num_local_experts + e = s + num_local_experts + expert_map[s:e] = torch.tensor(list(range(num_local_experts)), device=device) + + return TestTensors(topk_ids=self.topk_ids.clone(), expert_map=expert_map) + + +def ref_impl(tt: TestTensors, expert_num_tokens: torch.Tensor): + # do the reference in cpu + tt.to_device("cpu") + expert_ids, counts = tt.topk_ids.unique(return_counts=True) + + for eid, count in zip(expert_ids, counts): + if eid != -1 and tt.expert_map is not None: + eid = tt.expert_map[eid] + + if eid == -1: + continue + + expert_num_tokens[eid] += count + + +def do_test_compute_expert_num_tokens( + num_tokens: int, + num_topk: int, + num_experts: int, + ep_size: int, + topk_ids_dtype: torch.dtype, +): + assert num_topk <= num_experts + + tt = TestTensors.make( + num_tokens, num_topk, num_experts, topk_ids_dtype=topk_ids_dtype, device="cpu" + ) + + num_global_experts = num_experts + assert num_global_experts % ep_size == 0 + num_local_experts = num_global_experts // ep_size + for ep_rank in range(ep_size): + tt_rank = tt.with_ep_rank(ep_rank, num_global_experts, num_local_experts, "cpu") + + ref_expert_num_tokens = torch.zeros( + (num_local_experts), device="cpu", dtype=torch.int32 + ) + ref_impl(tt_rank, ref_expert_num_tokens) + ref_expert_num_tokens = ref_expert_num_tokens.to("cuda") + + tt_rank.to_device("cuda") + # Test with expert_map + triton_expert_num_tokens_w_emap = count_expert_num_tokens( + tt_rank.topk_ids, num_local_experts, tt_rank.expert_map + ) + + # Test without expert map + topk_ids = tt_rank.expert_map[tt_rank.topk_ids].to(topk_ids_dtype) + triton_expert_num_tokens_wo_emap = count_expert_num_tokens( + topk_ids, num_local_experts, expert_map=None + ) + + torch.testing.assert_close( + ref_expert_num_tokens, triton_expert_num_tokens_w_emap, atol=0, rtol=0 + ) + torch.testing.assert_close( + ref_expert_num_tokens, triton_expert_num_tokens_wo_emap, atol=0, rtol=0 + ) + + +@pytest.mark.parametrize("num_tokens", [1, 4, 8, 11, 127, 128, 3333, 7317]) +@pytest.mark.parametrize("num_topk", [2, 6, 8]) +@pytest.mark.parametrize("num_experts", [64]) +@pytest.mark.parametrize("ep_size", [1, 2, 4]) +@pytest.mark.parametrize("topk_ids_dtype", [torch.int64]) +def test_compute_expert_num_tokens( + num_tokens: int, + num_topk: int, + num_experts: int, + ep_size: int, + topk_ids_dtype: torch.dtype, +): + do_test_compute_expert_num_tokens( + num_tokens, num_topk, num_experts, ep_size, topk_ids_dtype + ) + + +@pytest.mark.parametrize("numel", list(range(1, 8192, 111))) +@pytest.mark.parametrize("num_experts", [32]) +@pytest.mark.parametrize("ep_size", [2]) +@pytest.mark.parametrize("topk_ids_dtype", [torch.int64]) +def test_compute_expert_num_tokens_from_numel( + numel: int, num_experts: int, ep_size: int, topk_ids_dtype: torch.dtype +): + do_test_compute_expert_num_tokens( + num_tokens=numel, + num_topk=1, + num_experts=num_experts, + ep_size=ep_size, + topk_ids_dtype=topk_ids_dtype, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_cpu_fused_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_cpu_fused_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..79efa09e58b93529990be351c692d8fd2134302d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_cpu_fused_moe.py @@ -0,0 +1,178 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from tests.kernels.allclose_default import get_default_atol, get_default_rtol +from vllm._custom_ops import ( + cpu_fused_moe, + cpu_prepack_moe_weight, +) +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.cpu_fused_moe import _CPU_MOE_ACT_FN +from vllm.platforms import CpuArchEnum, current_platform +from vllm.utils.torch_utils import set_random_seed + +if not current_platform.is_cpu(): + pytest.skip("skipping CPU-only tests", allow_module_level=True) + +EXPERT_NUM = [ + 8, +] +HIDDEN_DIM = [128, 2880] +INTERMEDIATE_DIM = [128, 2880] +BATCH_SIZE = [1, 64, 256] +ACT = [ + MoEActivation.SILU, + MoEActivation.SWIGLUOAI, + MoEActivation.GELU, + MoEActivation.GELU_TANH, +] +USE_BIAS = [False, True] +ISA = ["vec"] +if current_platform.get_cpu_architecture() == CpuArchEnum.ARM: + ISA.append("neon") +if torch.cpu._is_amx_tile_supported(): + ISA.append("amx") + +DTYPE = [torch.bfloat16] + + +def ref_fused_moe( + input: torch.Tensor, + w13: torch.Tensor, + w2: torch.Tensor, + w13_bias: torch.Tensor | None, + w2_bias: torch.Tensor | None, + topk_weights: torch.Tensor, + topk_ids: torch.Tensor, + activation: MoEActivation, +) -> torch.Tensor: + len_experts = w13.size(0) + + cnts = topk_ids.new_zeros((topk_ids.shape[0], len_experts)) + cnts.scatter_(1, topk_ids.to(torch.int64), 1) + tokens_per_expert = cnts.sum(dim=0) + idxs = topk_ids.view(-1).argsort() + + sorted_tokens = input[idxs // topk_ids.shape[1]] + tokens_per_expert = tokens_per_expert.cpu().numpy() + + outputs = [] + start_idx = 0 + + for i, num_tokens in enumerate(tokens_per_expert): + end_idx = start_idx + num_tokens + if num_tokens == 0: + continue + tokens_for_this_expert = sorted_tokens[start_idx:end_idx].float() + curr_w13 = w13[i].float() + curr_w2 = w2[i].float() + + curr_w13_bias = None + if w13_bias is not None: + curr_w13_bias = w13_bias[i].float() + + curr_w2_bias = None + if w2_bias is not None: + curr_w2_bias = w2_bias[i].float() + + gate_up = torch.nn.functional.linear( + tokens_for_this_expert, curr_w13, curr_w13_bias + ) + # Note: to simulate the kernel implementation + gate_up = _CPU_MOE_ACT_FN[activation](gate_up).to(dtype=input.dtype).float() + expert_out = torch.nn.functional.linear(gate_up, curr_w2, curr_w2_bias) + + outputs.append(expert_out) + start_idx = end_idx + + outs = torch.cat(outputs, dim=0) if len(outputs) else sorted_tokens.new_empty(0) + new_x = torch.empty_like(outs) + + new_x[idxs] = outs + final_out = ( + new_x.view(*topk_ids.shape, -1) + .mul_(topk_weights.unsqueeze(dim=-1)) + .sum(dim=1) + .type(input.dtype) + ) + return final_out + + +@pytest.mark.parametrize("batch_size", BATCH_SIZE) +@pytest.mark.parametrize("expert_num", EXPERT_NUM) +@pytest.mark.parametrize("hidden_size", HIDDEN_DIM) +@pytest.mark.parametrize("intermediate_size", INTERMEDIATE_DIM) +@pytest.mark.parametrize("use_bias", USE_BIAS) +@pytest.mark.parametrize("dtype", DTYPE) +@pytest.mark.parametrize("act", ACT) +@pytest.mark.parametrize("isa", ISA) +def test_cpu_fused_moe( + default_vllm_config, + batch_size: int, + expert_num: int, + hidden_size: int, + intermediate_size: int, + use_bias: bool, + dtype: torch.dtype, + act: MoEActivation, + isa: str, +): + set_random_seed(0) + + topk_num = max(expert_num // 2, 1) + up_dim = 2 * intermediate_size + + input = torch.randn((batch_size, hidden_size), dtype=dtype) / ( + 0.5 * hidden_size**0.5 + ) + w13 = torch.randn((expert_num, up_dim, hidden_size), dtype=dtype) / ( + 0.5 * hidden_size**0.5 + ) + w2 = torch.randn((expert_num, hidden_size, intermediate_size), dtype=dtype) / ( + 0.5 * intermediate_size**0.5 + ) + router_logits = torch.randn((batch_size, expert_num), dtype=dtype) + w13_bias = None + w2_bias = None + if use_bias: + w13_bias = torch.randn((expert_num, up_dim), dtype=dtype) / (0.5 * up_dim**0.5) + w2_bias = torch.randn((expert_num, hidden_size), dtype=dtype) / ( + 0.5 * hidden_size**0.5 + ) + score = torch.softmax(router_logits, dim=-1, dtype=torch.float32) + topk_weight, topk_ids = torch.topk(score, topk_num) + topk_ids = topk_ids.to(torch.int32) + + ref_output = ref_fused_moe( + input, + w13, + w2, + w13_bias, + w2_bias, + topk_weight, + topk_ids, + act, + ) + + packed_w13 = cpu_prepack_moe_weight(w13, isa) + packed_w2 = cpu_prepack_moe_weight(w2, isa) + output = cpu_fused_moe( + input, + packed_w13, + packed_w2, + w13_bias, + w2_bias, + topk_weight, + topk_ids, + act.value, + isa, + ) + + atol, rtol = get_default_atol(output), get_default_rtol(output) + ( + torch.testing.assert_close(output, ref_output, atol=atol, rtol=rtol), + f"{torch.max(torch.abs(output - ref_output))}", + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_cpu_int4_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_cpu_int4_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..1aa0e0f6ab099ee9965eb22f4102a2f76e63b2de --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_cpu_int4_moe.py @@ -0,0 +1,296 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for CPU INT4 W4A8 dynamic quantized fused MoE kernel (CPUExpertsInt4).""" + +import sys + +import pytest +import torch +import torch.nn.functional as F + +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +if not current_platform.is_cpu(): + pytest.skip("skipping CPU-only tests", allow_module_level=True) + +# Check if the dynamic_4bit_int_moe op is available +if not hasattr(torch.ops._C, "dynamic_4bit_int_moe"): + pytest.skip("dynamic_4bit_int_moe op not available", allow_module_level=True) + +# Check if KleidiAI ops are available +if not hasattr(torch.ops.aten, "_dyn_quant_pack_4bit_weight"): + pytest.skip("KleidiAI 4-bit ops not available", allow_module_level=True) + + +# Tolerance for INT4 W4A8 +INT4_W4A8_ATOL = 2e-2 +INT4_W4A8_RTOL = 2e-2 + + +def _silu_and_mul(x: torch.Tensor) -> torch.Tensor: + """SwiGLU activation: SiLU(gate) * up.""" + d = x.shape[-1] // 2 + return F.silu(x[..., :d]) * x[..., d:] + + +def _pack_int4_weight_to_kleidi( + int4_as_int8: torch.Tensor, + scales: torch.Tensor, + bias: torch.Tensor | None, + group_size: int, + in_features: int, + out_features: int, +) -> torch.Tensor: + """Pack INT4 weights (stored as int8 in [-8,7]) to KleidiAI format. + + Args: + int4_as_int8: [out, in] int8 tensor with values in [-8, 7] + scales: [out, in//group_size] or [out, 1] for channel-wise + bias: [out] optional bias + group_size: Quantization group size (-1 for channel-wise) + in_features: Input dimension + out_features: Output dimension + + Returns: + Packed weight tensor in KleidiAI format + """ + # Shift to unsigned nibble [0, 15] + tmp = int4_as_int8.add(8) + # Pack pairs along input dimension + uint8_nibbles = ((tmp[:, 1::2] << 4) | tmp[:, ::2]).to(torch.uint8) + + # Determine scale dtype based on group_size + scale_dtype = torch.float32 if group_size == -1 else torch.bfloat16 + scales_typed = scales.to(scale_dtype) + bias_typed = None if bias is None else bias.to(torch.float32) + + # Pack using KleidiAI op + actual_group_size = in_features if group_size == -1 else group_size + return torch.ops.aten._dyn_quant_pack_4bit_weight( + uint8_nibbles, + scales_typed, + bias_typed, + actual_group_size, + in_features, + out_features, + ) + + +def _make_int4_moe_weights( + E: int, + N: int, + K: int, + group_size: int, + has_bias: bool = False, +) -> tuple[ + torch.Tensor, + torch.Tensor, + torch.Tensor, + torch.Tensor, + torch.Tensor | None, + torch.Tensor | None, +]: + """Generate random INT4 MoE weights with random scales. + + Args: + E: Number of experts + N: Intermediate size + K: Hidden size + group_size: Quantization group size (-1 for channel-wise) + has_bias: Whether to include bias + + Returns: + (w13_packed, w2_packed, w13_ref, w2_ref, w13_bias, w2_bias) + where *_ref are the dequantized float reference weights + """ + # Generate INT4 weights as int8 values in [-8, 7] + w13_int4 = torch.randint(-8, 8, (E, 2 * N, K), dtype=torch.int8) + w2_int4 = torch.randint(-8, 8, (E, K, N), dtype=torch.int8) + + # Determine number of scale columns + def _n_scale_cols(in_features: int) -> int: + return 1 if group_size == -1 else (in_features // group_size) + + # Generate random scales + scale_dtype = torch.float32 if group_size == -1 else torch.bfloat16 + w13_scales = torch.rand(E, 2 * N, _n_scale_cols(K), dtype=scale_dtype) * 0.01 + w2_scales = torch.rand(E, K, _n_scale_cols(N), dtype=scale_dtype) * 0.01 + + # Generate biases if needed + w13_bias = None + w2_bias = None + if has_bias: + w13_bias = torch.randn(E, 2 * N, dtype=torch.float32) * 0.01 + w2_bias = torch.randn(E, K, dtype=torch.float32) * 0.01 + + # Pack weights for each expert + w13_packed_list = [] + w2_packed_list = [] + + for e in range(E): + w13_packed_list.append( + _pack_int4_weight_to_kleidi( + w13_int4[e], + w13_scales[e], + w13_bias[e] if (has_bias and w13_bias is not None) else None, + group_size, + K, + 2 * N, + ) + ) + w2_packed_list.append( + _pack_int4_weight_to_kleidi( + w2_int4[e], + w2_scales[e], + w2_bias[e] if (has_bias and w2_bias is not None) else None, + group_size, + N, + K, + ) + ) + + w13_packed = torch.stack(w13_packed_list, dim=0) + w2_packed = torch.stack(w2_packed_list, dim=0) + + # Create reference dequantized weights + w13_ref = torch.zeros(E, 2 * N, K, dtype=torch.float32) + w2_ref = torch.zeros(E, K, N, dtype=torch.float32) + + for e in range(E): + # Dequantize w13 + for i in range(2 * N): + for j in range(K): + group_idx = 0 if group_size == -1 else (j // group_size) + w13_ref[e, i, j] = ( + w13_int4[e, i, j].float() * w13_scales[e, i, group_idx].float() + ) + if has_bias and w13_bias is not None: + w13_ref[e, i, j] += w13_bias[e, i].float() + + # Dequantize w2 + for i in range(K): + for j in range(N): + group_idx = 0 if group_size == -1 else (j // group_size) + w2_ref[e, i, j] = ( + w2_int4[e, i, j].float() * w2_scales[e, i, group_idx].float() + ) + if has_bias and w2_bias is not None: + w2_ref[e, i, j] += w2_bias[e, i].float() + + return w13_packed, w2_packed, w13_ref, w2_ref, w13_bias, w2_bias + + +def ref_int4_moe( + a: torch.Tensor, + w13_ref: torch.Tensor, + w2_ref: torch.Tensor, + topk_weight: torch.Tensor, + topk_ids: torch.Tensor, +) -> torch.Tensor: + """Reference INT4 W4A8 fused MoE using dequantized weights. + + Steps: + 1. Use dequantized float weights + 2. For each expert: matmul → SwiGLU → matmul + 3. Weighted sum across top-k experts + """ + B, D = a.shape + topk = topk_ids.size(1) + + a_exp = a.view(B, -1, D).repeat(1, topk, 1).reshape(-1, D).float() + out = torch.zeros(B * topk, w2_ref.shape[1], dtype=torch.float32) + + topk_weight_flat = topk_weight.view(-1) + topk_ids_flat = topk_ids.view(-1) + + for i in range(w13_ref.shape[0]): + mask = topk_ids_flat == i + if mask.sum(): + # w13: [2N, K], input: [B, K] -> output: [B, 2N] + gate_up = torch.matmul(a_exp[mask], w13_ref[i].transpose(0, 1)) + # SwiGLU activation + hidden = _silu_and_mul(gate_up) + # w2: [K, N], hidden: [B, N] -> output: [B, K] + out[mask] = torch.matmul(hidden, w2_ref[i].transpose(0, 1)) + + return ( + (out.view(B, -1, w2_ref.shape[1]) * topk_weight_flat.view(B, -1, 1)) + .sum(dim=1) + .to(a.dtype) + ) + + +NUM_TOKENS = [1, 2, 64, 128] +# (intermediate_size N, hidden_size K, num_experts E, topk, group_size) +MoE_CONFIGS = [ + (256, 512, 8, 2, 128), + (256, 512, 8, 2, 64), + (256, 512, 8, 2, -1), # channel-wise + (512, 256, 8, 4, 128), + (512, 512, 8, 2, 128), + (768, 2048, 8, 2, 128), + (768, 2048, 16, 4, 64), +] +SEEDS = [0, 42] + + +@pytest.mark.parametrize("M", NUM_TOKENS) +@pytest.mark.parametrize("N,K,E,topk,group_size", MoE_CONFIGS) +@pytest.mark.parametrize("seed", SEEDS) +def test_cpu_int4_moe_kernel(M, N, K, E, topk, group_size, seed): + """Test dynamic_4bit_int_moe kernel against dequantized torch reference.""" + set_random_seed(seed) + + # Generate input activations + a = torch.randn(M, K, dtype=torch.bfloat16) / (K**0.5) + + # Generate INT4 weights + w13_packed, w2_packed, w13_ref, w2_ref, w13_bias, w2_bias = _make_int4_moe_weights( + E, N, K, group_size, has_bias=False + ) + + # Generate router logits and topk + score = torch.randn(M, E, dtype=torch.bfloat16) + score = torch.softmax(score, dim=-1, dtype=torch.float32) + topk_weight, topk_ids = torch.topk(score, topk) + topk_ids = topk_ids.to(torch.long) + + # Reference output using dequantized weights + ref_out = ref_int4_moe( + a, + w13_ref, + w2_ref, + topk_weight, + topk_ids, + ) + + # Test dynamic_4bit_int_moe kernel + # Activation kind: 1 = SwiGLU_Ug (SiLU(u)*g) for OAI-style + activation_kind = 1 + apply_router_weight_on_input = False + + out = torch.ops._C.dynamic_4bit_int_moe( + a, + topk_ids, + topk_weight, + w13_packed, + w2_packed, + K, # H (hidden_size / w2_out_features) + N, # I (intermediate_size / w2_in_features) + 2 * N, # I2 (2*intermediate_size / w13_out_features) + group_size, + apply_router_weight_on_input, + activation_kind, + ) + + torch.testing.assert_close( + ref_out.bfloat16(), + out, + atol=INT4_W4A8_ATOL, + rtol=INT4_W4A8_RTOL, + ) + + +if __name__ == "__main__": + sys.exit(pytest.main([__file__, "-v"])) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_cpu_quant_fused_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_cpu_quant_fused_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..40848781f89af995e31c77534152e96efb5af998 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_cpu_quant_fused_moe.py @@ -0,0 +1,888 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for CPU quantized fused MoE kernels.""" + +import math +import sys + +import pytest +import torch +import torch.nn.functional as F + +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +if not current_platform.is_cpu(): + pytest.skip("skipping CPU-only tests", allow_module_level=True) + +import vllm._custom_ops as ops # noqa: E402 + +if not hasattr(torch.ops._C, "fused_experts_cpu"): + pytest.skip("fused_experts_cpu op not available", allow_module_level=True) + + +def _silu_and_mul(x: torch.Tensor) -> torch.Tensor: + d = x.shape[-1] // 2 + return F.silu(x[..., :d]) * x[..., d:] + + +def _prepack_experts(w: torch.Tensor) -> torch.Tensor: + """VNNI-prepack expert weights via ``convert_weight_packed``.""" + return torch.ops._C.convert_weight_packed(w) + + +# =========================================================================== +# FP8 W8A16 MoE +# =========================================================================== + + +BLOCK_SIZE = [128, 128] # [block_n, block_k] + +_FP8_INFO = torch.finfo(torch.float8_e4m3fn) +FP8_SCALE = _FP8_INFO.max # 448.0 +FACTOR_FOR_SCALE = 1e-3 + + +def _block_dequant_weight( + weight: torch.Tensor, + scales: torch.Tensor, + block_size: list[int], +) -> torch.Tensor: + """Block-dequantize FP8 weight [E, N, K] -> float [E, N, K].""" + E, N, K = weight.shape + block_n, block_k = block_size + pad_N = (block_n - N % block_n) % block_n + pad_K = (block_k - K % block_k) % block_k + + if pad_N > 0 or pad_K > 0: + weight = F.pad(weight, (0, pad_K, 0, pad_N)) + + n_tiles = math.ceil(N / block_n) + k_tiles = math.ceil(K / block_k) + + weight_block = ( + weight.view(E, n_tiles, block_n, k_tiles, block_k) + .permute(0, 1, 3, 2, 4) + .float() + .contiguous() + ) + weight_scaled = ( + (weight_block * scales.view(E, n_tiles, k_tiles, 1, 1)) + .permute(0, 1, 3, 2, 4) + .contiguous() + ) + if pad_N > 0 or pad_K > 0: + weight_scaled = weight_scaled.view(E, N + pad_N, K + pad_K) + weight_scaled = weight_scaled[..., :N, :K].contiguous() + else: + weight_scaled = weight_scaled.view(E, N, K) + return weight_scaled + + +def ref_w8a16_block_fp8_moe( + a: torch.Tensor, + w1: torch.Tensor, + w2: torch.Tensor, + w1_s: torch.Tensor, + w2_s: torch.Tensor, + topk_weight: torch.Tensor, + topk_ids: torch.Tensor, + block_size: list[int], +) -> torch.Tensor: + """Reference FP8 W8A16 block-scaled fused MoE in pure torch.""" + B, D = a.shape + topk = topk_ids.size(1) + + w1_dq = _block_dequant_weight(w1, w1_s, block_size) + w2_dq = _block_dequant_weight(w2, w2_s, block_size) + + a_exp = a.view(B, -1, D).repeat(1, topk, 1).reshape(-1, D).float() + out = torch.zeros(B * topk, w2_dq.shape[1], dtype=torch.float32) + + topk_weight_flat = topk_weight.view(-1) + topk_ids_flat = topk_ids.view(-1) + + for i in range(w1_dq.shape[0]): + mask = topk_ids_flat == i + if mask.sum(): + ic0 = torch.matmul(a_exp[mask], w1_dq[i].transpose(0, 1)) + ic1 = _silu_and_mul(ic0) + out[mask] = torch.matmul(ic1, w2_dq[i].transpose(0, 1)) + + return ( + (out.view(B, -1, w2_dq.shape[1]) * topk_weight_flat.view(B, -1, 1)) + .sum(dim=1) + .to(a.dtype) + ) + + +def _make_fp8_moe_weights( + E: int, + N: int, + K: int, + block_size: list[int], +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + """Generate random FP8 MoE weights with random block scales.""" + block_n, block_k = block_size + + w1 = ( + (torch.randn(E, 2 * N, K) * FP8_SCALE) + .clamp(min=-FP8_SCALE, max=FP8_SCALE) + .to(torch.float8_e4m3fn) + ) + w2 = ( + (torch.randn(E, K, N) * FP8_SCALE) + .clamp(min=-FP8_SCALE, max=FP8_SCALE) + .to(torch.float8_e4m3fn) + ) + + w1_s = ( + torch.randn(E, math.ceil(2 * N / block_n), math.ceil(K / block_k)) + * FACTOR_FOR_SCALE + ) + w2_s = ( + torch.randn(E, math.ceil(K / block_n), math.ceil(N / block_k)) + * FACTOR_FOR_SCALE + ) + return w1, w2, w1_s, w2_s + + +FP8_NUM_TOKENS = [1, 2, 64, 121] +FP8_MOE_CONFIGS = [ + (256, 512, 8, 2), + (256, 512, 8, 4), + (512, 256, 8, 2), + (512, 256, 8, 4), + (512, 512, 8, 2), + (512, 512, 8, 4), + (768, 2048, 8, 2), + (768, 2048, 8, 4), + (768, 2048, 128, 8), +] + + +@pytest.mark.parametrize("M", FP8_NUM_TOKENS) +@pytest.mark.parametrize("N,K,E,topk", FP8_MOE_CONFIGS) +@pytest.mark.parametrize("seed", [0]) +def test_w8a16_block_fp8_cpu_fused_moe(M, N, K, E, topk, seed): + """Test fused_experts_cpu FP8 W8A16 against dequantised torch reference.""" + set_random_seed(seed) + + a = torch.randn(M, K, dtype=torch.bfloat16) / math.sqrt(K) + w1, w2, w1_s, w2_s = _make_fp8_moe_weights(E, N, K, BLOCK_SIZE) + + score = torch.randn(M, E, dtype=torch.bfloat16) + score = torch.softmax(score, dim=-1, dtype=torch.float32) + topk_weight, topk_ids = torch.topk(score, topk) + topk_ids = topk_ids.to(torch.int32) + + ref_out = ref_w8a16_block_fp8_moe( + a, w1, w2, w1_s, w2_s, topk_weight, topk_ids, BLOCK_SIZE + ) + + pw1, pw2 = _prepack_experts(w1), _prepack_experts(w2) + + # Test inplace=False against reference + out = ops.fused_experts_cpu( + a.clone(), + pw1, + pw2, + topk_weight, + topk_ids, + False, + ops.CPUQuantMethod.FP8_W8A16, + w1_s, + w2_s, + None, + None, + BLOCK_SIZE, + is_vnni=True, + ) + torch.testing.assert_close(ref_out.bfloat16(), out, atol=1e-2, rtol=1e-2) + + # Test inplace=True produces identical output + out_inplace = ops.fused_experts_cpu( + a.clone(), + pw1, + pw2, + topk_weight, + topk_ids, + True, + ops.CPUQuantMethod.FP8_W8A16, + w1_s, + w2_s, + None, + None, + BLOCK_SIZE, + is_vnni=True, + ) + torch.testing.assert_close(out_inplace, out, atol=0, rtol=0) + + +# =========================================================================== +# MXFP4 W4A16 MoE +# =========================================================================== + + +class MXFP4QuantizeUtil: + """MXFP4 quantization utility.""" + + E2M1_max = 6.0 + E2M1_values = [0, 0.5, 1, 1.5, 2, 3, 4, 6] + E2M1_bounds = torch.tensor([0.25, 0.75, 1.25, 1.75, 2.5, 3.5, 5]) + block_size = 32 + + @classmethod + def quantize(cls, input: torch.Tensor) -> tuple: + """Quantize BF16 tensor to MXFP4 packed uint8 format.""" + + def cast_fp4(x): + sign = torch.sign(x) + sign_bit = (2 - sign) // 2 + ord_ = torch.sum( + (x.abs().unsqueeze(-1) - cls.E2M1_bounds.to(x.device)) > 0, dim=-1 + ) + fp4_val = (sign_bit * 0b1000 + ord_).to(torch.uint8) + return fp4_val + + def fuse_uint4_to_uint8(x): + left_side = x[..., 0::2] + right_side = x[..., 1::2] + new_data = right_side.clone() << 4 + new_data[..., : left_side.shape[-1]] += left_side + return new_data + + original_shape = input.shape + input = input.view(-1, cls.block_size) + input_amax = input.abs().max(dim=-1, keepdim=True).values + descale = input_amax / cls.E2M1_max + min_value = torch.tensor(-127.0, device=descale.device) + e8m0_scale = torch.ceil(torch.maximum(torch.log2(descale), min_value)) + + input = (input / torch.exp2(e8m0_scale)).view(original_shape) + input_q = cast_fp4(input) + input_q = fuse_uint4_to_uint8(input_q) + e8m0_scale = (e8m0_scale + 127).to(torch.uint8) + return input_q, e8m0_scale + + @classmethod + def dequantize(cls, quantized_data, dtype: torch.dtype, scale): + """Dequantize MXFP4 packed tensor back to float.""" + + def unfuse_uint8_to_uint4(x): + left_side = x & 0x0F + right_side = (x >> 4) & 0x0F + shape = list(x.shape) + shape[-1] = shape[-1] * 2 + result = torch.zeros(shape, dtype=torch.uint8, device=x.device) + result[..., 0::2] = left_side + result[..., 1::2] = right_side + return result + + e8m0_scale = scale + x_unfused = unfuse_uint8_to_uint4(quantized_data) + sign = 1 - 2 * ((x_unfused & 0b1000) >> 3).to(torch.float32) + magnitude = (x_unfused & 0b0111).to(torch.long) + values = torch.tensor(cls.E2M1_values, device=quantized_data.device) + original_shape = magnitude.shape + x_float = values[magnitude.reshape(-1)].reshape(original_shape) + x_float = sign.float() * x_float + x_float = x_float.reshape(-1, cls.block_size) + scale_factor = torch.exp2(e8m0_scale.float() - 127) + scale_factor = scale_factor.reshape(-1, 1) + x_float = x_float * scale_factor + return x_float.reshape(original_shape).to(dtype) + + +def _swiglu(x: torch.Tensor, alpha: float, limit: float) -> torch.Tensor: + """SwigLU activation used in GPT-OSS. + Input is interleaved: [gate_0, up_0, gate_1, up_1, ...] in last dim. + """ + gate = x[..., 0::2] + up = x[..., 1::2] + gate_clamped = torch.clamp(gate, max=limit) + up_clamped = torch.clamp(up, min=-limit, max=limit) + return gate_clamped * torch.sigmoid(alpha * gate_clamped) * (up_clamped + 1) + + +def ref_mxfp4_fused_moe( + a: torch.Tensor, + w1_dq: torch.Tensor, + w2_dq: torch.Tensor, + topk_weight: torch.Tensor, + topk_ids: torch.Tensor, + topk: int, +) -> torch.Tensor: + """Reference MXFP4 fused MoE with SiLU activation.""" + B, D = a.shape + a_f = a.float() + out = torch.zeros(B * topk, w2_dq.shape[1], dtype=torch.float32) + topk_ids_flat = topk_ids.view(-1) + + for i in range(w1_dq.shape[0]): + mask = topk_ids_flat == i + if mask.sum() == 0: + continue + token_indices = torch.where(mask)[0] + source_indices = token_indices // topk + ic0 = torch.matmul(a_f[source_indices], w1_dq[i].float().T) + ic1 = _silu_and_mul(ic0) + out[mask] = torch.matmul(ic1, w2_dq[i].float().T) + + return (out.view(B, topk, -1) * topk_weight.unsqueeze(-1)).sum(dim=1).to(a.dtype) + + +def ref_mxfp4_fused_moe_gptoss( + a: torch.Tensor, + w1_dq: torch.Tensor, + w2_dq: torch.Tensor, + w1_bias: torch.Tensor, + w2_bias: torch.Tensor, + topk_weight: torch.Tensor, + topk_ids: torch.Tensor, + alpha: float, + limit: float, +) -> torch.Tensor: + """Reference MXFP4 fused MoE with SwigLU+bias (GPT-OSS style).""" + B, D = a.shape + topk = topk_ids.shape[1] + a_f = a.float() + E = w1_dq.shape[0] + out = torch.zeros(B * topk, w2_dq.shape[1], dtype=torch.float32) + topk_ids_flat = topk_ids.view(-1) + + for i in range(E): + mask = topk_ids_flat == i + if mask.sum() == 0: + continue + token_indices = torch.where(mask)[0] + source_indices = token_indices // topk + ic0 = torch.matmul(a_f[source_indices], w1_dq[i].float().T) + ic0 = ic0 + w1_bias[i].float() + ic1 = _swiglu(ic0, alpha, limit) + ic2 = torch.matmul(ic1, w2_dq[i].float().T) + ic2 = ic2 + w2_bias[i].float() + out[mask] = ic2 + + return (out.view(B, topk, -1) * topk_weight.unsqueeze(-1)).sum(dim=1).to(a.dtype) + + +def _prepack_mxfp4_experts( + w: torch.Tensor, w_scale: torch.Tensor +) -> tuple[torch.Tensor, torch.Tensor]: + """VNNI-prepack MXFP4 weights and repack scales.""" + packed_w = torch.ops._C.convert_weight_packed(w) + packed_s = torch.ops._C.convert_scale_packed(w_scale) + return packed_w, packed_s + + +MXFP4_NUM_TOKENS = [1, 2, 32, 121] +MXFP4_MOE_CONFIGS = [ + (128, 128, 4, 2), + (256, 256, 8, 4), + (352, 256, 8, 4), + (512, 320, 8, 4), +] + + +@pytest.mark.parametrize("M", MXFP4_NUM_TOKENS) +@pytest.mark.parametrize("N,K,E,topk", MXFP4_MOE_CONFIGS) +@pytest.mark.parametrize("seed", [0]) +def test_mxfp4_cpu_fused_moe(M, N, K, E, topk, seed): + """Test fused_experts_mxfp4_cpu against dequantized torch reference.""" + set_random_seed(seed) + dtype = torch.bfloat16 + + a = torch.randn(M, K, dtype=dtype) / 10 + + # Generate and quantize weights + w1_bf16 = torch.randn(E, 2 * N, K, dtype=dtype) / 10 + w1q, w1s = MXFP4QuantizeUtil.quantize(w1_bf16) + w1s = w1s.reshape(E, 2 * N, K // 32) + w1dq = MXFP4QuantizeUtil.dequantize(w1q, dtype, w1s) + + w2_bf16 = torch.randn(E, K, N, dtype=dtype) / 10 + w2q, w2s = MXFP4QuantizeUtil.quantize(w2_bf16) + w2s = w2s.reshape(E, K, N // 32) + w2dq = MXFP4QuantizeUtil.dequantize(w2q, dtype, w2s) + + # Routing + score = torch.randn(M, E, dtype=dtype) + score = torch.softmax(score, dim=-1, dtype=torch.float32) + topk_weight, topk_ids = torch.topk(score, topk) + topk_ids = topk_ids.to(torch.int32) + + # Reference + ref_out = ref_mxfp4_fused_moe(a, w1dq, w2dq, topk_weight, topk_ids, topk) + + # Pack weights for kernel + pw1, pw1s = _prepack_mxfp4_experts(w1q, w1s) + pw2, pw2s = _prepack_mxfp4_experts(w2q, w2s) + + # Kernel + out = ops.fused_experts_cpu( + a.clone(), + pw1, + pw2, + topk_weight, + topk_ids, + False, # inplace + ops.CPUQuantMethod.MXFP4, + pw1s, # w1_scale + pw2s, # w2_scale + None, # w1_zero + None, # w2_zero + None, # block_size + ) + + torch.testing.assert_close(ref_out.bfloat16(), out, atol=1e-2, rtol=1e-2) + + +@pytest.mark.parametrize("M", [1, 32]) +@pytest.mark.parametrize("N,K,E,topk", [(128, 128, 4, 2), (64, 64, 4, 2)]) +@pytest.mark.parametrize("seed", [0]) +def test_mxfp4_cpu_fused_moe_bias_swiglu(M, N, K, E, topk, seed): + """Test fused_experts_mxfp4_cpu with bias and SwigLU activation (GPT-OSS).""" + set_random_seed(seed) + dtype = torch.bfloat16 + alpha = 1.702 + limit = 7.0 + + a = torch.randn(M, K, dtype=dtype) / 10 + + # Generate and quantize weights + w1_bf16 = torch.randn(E, 2 * N, K, dtype=dtype) / 10 + w1q, w1s = MXFP4QuantizeUtil.quantize(w1_bf16) + w1s = w1s.reshape(E, 2 * N, K // 32) + w1dq = MXFP4QuantizeUtil.dequantize(w1q, dtype, w1s) + w1_b = torch.randn(E, 2 * N, dtype=torch.float32) / 10 + + w2_bf16 = torch.randn(E, K, N, dtype=dtype) / 10 + w2q, w2s = MXFP4QuantizeUtil.quantize(w2_bf16) + w2s = w2s.reshape(E, K, N // 32) + w2dq = MXFP4QuantizeUtil.dequantize(w2q, dtype, w2s) + w2_b = torch.randn(E, K, dtype=torch.float32) / 10 + + # Routing + score = torch.randn(M, E, dtype=dtype) + score = torch.softmax(score, dim=-1, dtype=torch.float32) + topk_weight, topk_ids = torch.topk(score, topk) + topk_ids = topk_ids.to(torch.int32) + + # Reference + ref_out = ref_mxfp4_fused_moe_gptoss( + a, w1dq, w2dq, w1_b, w2_b, topk_weight, topk_ids, alpha, limit + ) + + # Pack weights for kernel + pw1, pw1s = _prepack_mxfp4_experts(w1q, w1s) + pw2, pw2s = _prepack_mxfp4_experts(w2q, w2s) + + # Kernel + out = ops.fused_experts_cpu( + a.clone(), + pw1, + pw2, + topk_weight, + topk_ids, + False, # inplace + ops.CPUQuantMethod.MXFP4, + pw1s, # w1_scale + pw2s, # w2_scale + None, # w1_zero + None, # w2_zero + None, # block_size + w1_bias=w1_b, + w2_bias=w2_b, + alpha=alpha, + limit=limit, + ) + + torch.testing.assert_close(ref_out.bfloat16(), out, atol=1e-2, rtol=1e-2) + + +# =========================================================================== +# INT4 W4A16 MoE +# =========================================================================== + + +def _pack_int4_gptq(w_int4: torch.Tensor) -> torch.Tensor: + """Pack INT4 values [N, K] → [N, K//8] int32 along K dim (GPTQ format).""" + N, K = w_int4.shape + assert K % 8 == 0 + w = w_int4.to(torch.int32) + w_packed = torch.zeros(N, K // 8, dtype=torch.int32) + for j in range(8): + w_packed |= (w[:, j::8] & 0xF) << (j * 4) + return w_packed + + +def _pack_int4_awq(w_int4: torch.Tensor) -> torch.Tensor: + """Pack INT4 values [..., N] → [..., N//8] int32 along last dim (AWQ format).""" + # AWQ packing bitshifts: indices {0,4,1,5,2,6,3,7} * 4 bits each + _AWQ_BITSHIFTS = [0, 16, 4, 20, 8, 24, 12, 28] + + N = w_int4.shape[-1] + assert N % 8 == 0 + w = w_int4.to(torch.int32) + w_packed = torch.zeros(*w.shape[:-1], N // 8, dtype=torch.int32) + for j, shift in enumerate(_AWQ_BITSHIFTS): + w_packed |= (w[..., j::8] & 0xF) << shift + return w_packed + + +def _ref_int4_moe( + a: torch.Tensor, + w1_int4: torch.Tensor, + w2_int4: torch.Tensor, + w1_zeros: torch.Tensor | None, + w2_zeros: torch.Tensor | None, + w1_s: torch.Tensor, + w2_s: torch.Tensor, + topk_weight: torch.Tensor, + topk_ids: torch.Tensor, + group_size: int, +) -> torch.Tensor: + """Reference INT4 W4A16 group-quantized fused MoE in pure torch.""" + B = a.shape[0] + topk = topk_ids.size(1) + K_out = a.shape[1] + + out = torch.zeros(B, topk, K_out, dtype=torch.float32) + for b in range(B): + for t in range(topk): + eid = topk_ids[b, t].item() + x = a[b : b + 1].float() + + # Dequantize w1: [K, 2*N], groups along K (input dim) + K_dim = w1_int4.shape[1] + w1_dq = torch.zeros(K_dim, w1_int4.shape[2], dtype=torch.float32) + for g in range(w1_s.shape[1]): + k_start = g * group_size + k_end = min((g + 1) * group_size, K_dim) + zp = w1_zeros[eid, g, :].float() if w1_zeros is not None else 8.0 + w1_dq[k_start:k_end, :] = ( + w1_int4[eid, k_start:k_end, :].float() - zp + ) * w1_s[eid, g, :].float() + + ic = torch.matmul(x, w1_dq) # [1, K] @ [K, 2*N] → [1, 2*N] + ic = _silu_and_mul(ic) # [1, N] + + # Dequantize w2: [N, K], groups along N (input dim) + N_dim = w2_int4.shape[1] + w2_dq = torch.zeros(N_dim, w2_int4.shape[2], dtype=torch.float32) + for g in range(w2_s.shape[1]): + n_start = g * group_size + n_end = min((g + 1) * group_size, N_dim) + zp = w2_zeros[eid, g, :].float() if w2_zeros is not None else 8.0 + w2_dq[n_start:n_end, :] = ( + w2_int4[eid, n_start:n_end, :].float() - zp + ) * w2_s[eid, g, :].float() + + oc = torch.matmul(ic, w2_dq) # [1, N] @ [N, K] → [1, K] + out[b, t] = oc.squeeze(0) + + return (out * topk_weight.unsqueeze(-1)).sum(dim=1).to(a.dtype) + + +def _make_int4_moe_weights(E, N, K, group_size, quant_algo): + """Create INT4 MoE weights in GPTQ or AWQ packed format. + + Canonical layout (input × output): + w1_int4: [E, K, 2*N] w2_int4: [E, N, K] + + GPTQ packed (pack transposed weight along input/K dim): + w1_packed: [E, K//8, 2*N] w2_packed: [E, N//8, K] + zeros: actual int4 zero points, same packing as weights + + AWQ packed (pack along output/N dim): + w1_packed: [E, K, 2*N//8] w2_packed: [E, N, K//8] + zeros: actual int4 zero points, same packing as weights + + Returns: + w1_int4, w2_int4, + w1_packed, w2_packed, + w1_zeros, w2_zeros, + w1_zeros_packed, w2_zeros_packed, + w1_s, w2_s + """ + w1_int4 = torch.randint(0, 16, (E, K, 2 * N), dtype=torch.int32) + w2_int4 = torch.randint(0, 16, (E, N, K), dtype=torch.int32) + + num_groups_w1 = K // group_size + num_groups_w2 = N // group_size + w1_s = ( + torch.randn(E, num_groups_w1, 2 * N, dtype=torch.bfloat16) * 0.01 + ).abs() + 0.001 + w2_s = (torch.randn(E, num_groups_w2, K, dtype=torch.bfloat16) * 0.01).abs() + 0.001 + + if quant_algo == ops.CPUQuantAlgo.GPTQ: + # Pack: canonical [E, K, 2*N] → transpose [E, 2*N, K] → GPTQ pack + # [E, 2*N, K//8] → transpose [E, K//8, 2*N] + w1_t = w1_int4.transpose(1, 2).contiguous() # [E, 2*N, K] + w1_packed = ( + torch.stack([_pack_int4_gptq(w1_t[e]) for e in range(E)]) + .transpose(1, 2) + .contiguous() + ) # [E, K//8, 2*N] + w2_t = w2_int4.transpose(1, 2).contiguous() # [E, K, N] + w2_packed = ( + torch.stack([_pack_int4_gptq(w2_t[e]) for e in range(E)]) + .transpose(1, 2) + .contiguous() + ) # [E, N//8, K] + w1_zeros = w2_zeros = None + w1_zeros_packed = torch.full( + (E, num_groups_w1, 2 * N // 8), 0x77777777, dtype=torch.int32 + ) + w2_zeros_packed = torch.full( + (E, num_groups_w2, K // 8), 0x77777777, dtype=torch.int32 + ) + else: # AWQ + # Asymmetric: actual zero points, packed along output dim. + w1_zeros = torch.randint(1, 15, (E, num_groups_w1, 2 * N), dtype=torch.int32) + w2_zeros = torch.randint(1, 15, (E, num_groups_w2, K), dtype=torch.int32) + w1_packed = torch.stack( + [_pack_int4_awq(w1_int4[e]) for e in range(E)] + ) # [E, K, 2*N//8] + w2_packed = torch.stack( + [_pack_int4_awq(w2_int4[e]) for e in range(E)] + ) # [E, N, K//8] + w1_zeros_packed = torch.stack( + [_pack_int4_awq(w1_zeros[e]) for e in range(E)] + ) # [E, K//gs, 2*N//8] + w2_zeros_packed = torch.stack( + [_pack_int4_awq(w2_zeros[e]) for e in range(E)] + ) # [E, N//gs, K//8] + + return ( + w1_int4, + w2_int4, + w1_packed, + w2_packed, + w1_zeros, + w2_zeros, + w1_zeros_packed, + w2_zeros_packed, + w1_s, + w2_s, + ) + + +INT4_MOE_CONFIGS = [ + # (N, K, E, topk, group_size) + (256, 512, 8, 2, 128), + (512, 256, 8, 2, 128), + (512, 512, 8, 4, 128), + (768, 2048, 8, 2, 128), +] + + +@pytest.mark.parametrize("M", [1, 2, 64, 121]) +@pytest.mark.parametrize("N,K,E,topk,group_size", INT4_MOE_CONFIGS) +@pytest.mark.parametrize("quant_algo", [ops.CPUQuantAlgo.GPTQ, ops.CPUQuantAlgo.AWQ]) +@pytest.mark.parametrize("seed", [0]) +def test_int4_w4a16_cpu_fused_moe(M, N, K, E, topk, group_size, quant_algo, seed): + """Test fused_experts_cpu INT4 W4A16 for both GPTQ and AWQ quant formats.""" + set_random_seed(seed) + + a = torch.randn(M, K, dtype=torch.bfloat16) / (0.5 * K**0.5) + ( + w1_int4, + w2_int4, + w1_packed, + w2_packed, + w1_zeros, + w2_zeros, + w1_zeros_packed, + w2_zeros_packed, + w1_s, + w2_s, + ) = _make_int4_moe_weights(E, N, K, group_size, quant_algo) + + score = torch.randn(M, E, dtype=torch.bfloat16) + score = torch.softmax(score, dim=-1, dtype=torch.float32) + topk_weight, topk_ids = torch.topk(score, topk) + topk_ids = topk_ids.to(torch.int32) + + ref_out = _ref_int4_moe( + a, + w1_int4, + w2_int4, + w1_zeros, + w2_zeros, + w1_s, + w2_s, + topk_weight, + topk_ids, + group_size, + ) + + from vllm.model_executor.layers.fused_moe.experts.cpu_moe import ( + prepare_int4_moe_layer_for_cpu, + ) + + (blocked_w1, blocked_w2, blocked_s1, blocked_s2, blocked_z1, blocked_z2) = ( + prepare_int4_moe_layer_for_cpu( + w1_packed, + w2_packed, + w1_s, + w2_s, + quant_algo=quant_algo, + w13_zeros=w1_zeros_packed, + w2_zeros=w2_zeros_packed, + ) + ) + + out = ops.fused_experts_cpu( + a.clone(), + blocked_w1, + blocked_w2, + topk_weight, + topk_ids, + False, # inplace + ops.CPUQuantMethod.INT4_W4A8, + blocked_s1, + blocked_s2, + blocked_z1, + blocked_z2, + None, # block_size + None, # w1_bias + None, # w2_bias + None, # alpha + None, # limit + True, # is_vnni + ) + torch.testing.assert_close(ref_out.bfloat16(), out, atol=1e-2, rtol=1e-2) + + +# =========================================================================== +# INT8 W8A8 MoE +# =========================================================================== + + +def _quantize_per_channel(w): + """Symmetric per-channel INT8 quantisation. w: [N, K] -> (int8, scale).""" + amax = w.abs().amax(dim=-1, keepdim=True).clamp(min=1e-12) + scale = amax / 127.0 + w_q = (w / scale).round().clamp(-128, 127).to(torch.int8) + return w_q, scale.float() + + +def _quantize_per_token(x): + """Symmetric per-token INT8 quantisation. x: [M, K] -> (int8, scale).""" + amax = x.abs().amax(dim=-1, keepdim=True).clamp(min=1e-12) + scale = amax / 127.0 + x_q = (x / scale).round().clamp(-128, 127).to(torch.int8) + return x_q, scale.float() + + +def _ref_int8_moe(a, w1, w2, w1_s, w2_s, topk_weight, topk_ids): + """Reference INT8 W8A8 per-channel fused MoE in pure torch.""" + B, D = a.shape + topk = topk_ids.size(1) + + out = torch.zeros(B, topk, w2.shape[1], dtype=torch.float32) + for b in range(B): + for t in range(topk): + eid = topk_ids[b, t].item() + + x = a[b : b + 1].float() + x_q, x_s = _quantize_per_token(x) + ic = torch.matmul(x_q.float(), w1[eid].float().t()) + ic = ic * x_s * w1_s[eid].view(1, -1) + ic = _silu_and_mul(ic) + + ic_q, ic_s = _quantize_per_token(ic) + oc = torch.matmul(ic_q.float(), w2[eid].float().t()) + oc = oc * ic_s * w2_s[eid].view(1, -1) + out[b, t] = oc.squeeze(0) + + result = (out * topk_weight.unsqueeze(-1)).sum(dim=1) + return result.to(a.dtype) + + +def _make_int8_moe_weights(E, N, K): + factor = 1e-2 + w1_f = (torch.randn(E, 2 * N, K) - 0.5) * 2 + w2_f = (torch.randn(E, K, N) - 0.5) * 2 + + w1_q_list, w1_s_list = [], [] + w2_q_list, w2_s_list = [], [] + for e in range(E): + q, s = _quantize_per_channel(w1_f[e]) + w1_q_list.append(q) + w1_s_list.append(s) + q, s = _quantize_per_channel(w2_f[e]) + w2_q_list.append(q) + w2_s_list.append(s) + + return ( + torch.stack(w1_q_list), + torch.stack(w2_q_list), + torch.stack(w1_s_list) * factor, + torch.stack(w2_s_list) * factor, + ) + + +INT8_NUM_TOKENS = [1, 2, 64, 121] +INT8_MOE_CONFIGS = [ + # (N, K, E, topk) + (256, 512, 8, 2), + (512, 256, 8, 2), + (512, 512, 8, 4), + (768, 2048, 8, 2), +] + + +@pytest.mark.parametrize("M", INT8_NUM_TOKENS) +@pytest.mark.parametrize("N,K,E,topk", INT8_MOE_CONFIGS) +@pytest.mark.parametrize("seed", [0]) +@pytest.mark.parametrize("is_vnni", [False, True]) +@pytest.mark.parametrize("inplace", [False, True]) +def test_int8_w8a8_cpu_fused_moe(M, N, K, E, topk, seed, is_vnni, inplace): + """Test fused_experts_cpu INT8 W8A8 against torch reference.""" + set_random_seed(seed) + + a = torch.randn(M, K, dtype=torch.bfloat16) / (0.5 * K**0.5) + w1_q, w2_q, w1_s, w2_s = _make_int8_moe_weights(E, N, K) + + score = torch.randn(M, E, dtype=torch.bfloat16) + score = torch.softmax(score, dim=-1, dtype=torch.float32) + topk_weight, topk_ids = torch.topk(score, topk) + topk_ids = topk_ids.to(torch.int32) + + ref_out = _ref_int8_moe(a, w1_q, w2_q, w1_s, w2_s, topk_weight, topk_ids) + + w1 = _prepack_experts(w1_q) if is_vnni else w1_q + w2 = _prepack_experts(w2_q) if is_vnni else w2_q + + out = ops.fused_experts_cpu( + a.clone(), + w1, + w2, + topk_weight, + topk_ids, + inplace, + ops.CPUQuantMethod.INT8_W8A8, + w1_s, + w2_s, + None, # w1_zero + None, # w2_zero + None, # block_size + None, # w1_bias + None, # w2_bias + None, # alpha + None, # limit + is_vnni, + ) + torch.testing.assert_close( + ref_out.bfloat16(), + out, + atol=2e-1, + rtol=2e-1, + ) + + +if __name__ == "__main__": + sys.exit(pytest.main([__file__, "-v"])) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_cutedsl_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_cutedsl_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..650f6eb5008c3d03397ccd44381c59193eed6549 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_cutedsl_moe.py @@ -0,0 +1,588 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import pytest + +from vllm.platforms import current_platform + +if not current_platform.has_device_capability(100): + pytest.skip( + reason="Nvfp4 Requires compute capability of 10 or above.", + allow_module_level=True, + ) + +import torch +from flashinfer import fp4_quantize +from torch.nn import functional as F + +from vllm.model_executor.layers.activation import SiluAndMul +from vllm.model_executor.layers.fused_moe.experts.flashinfer_cutedsl_batched_moe import ( # noqa: E501 + flashinfer_cutedsl_moe_masked, +) +from vllm.utils.flashinfer import ( + flashinfer_cutedsl_grouped_gemm_nt_masked as cutedsl_gmm_masked, +) +from vllm.utils.flashinfer import ( + scaled_fp4_grouped_quantize, +) + +kE2M1ToFloat = torch.tensor( + [0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0], dtype=torch.float32 +) + +FLOAT8_E4M3_MAX = 448.0 +FLOAT4_E2M1_MAX = 6.0 + + +def convert_swizzled_to_linear(a_sf_swizzled: torch.Tensor, m, k, block_size): + m_tiles = (m + 128 - 1) // 128 + f = block_size * 4 + k_tiles = (k + f - 1) // f + tmp = torch.reshape(a_sf_swizzled, (1, m_tiles, k_tiles, 32, 4, 4)) + tmp = torch.permute(tmp, (0, 1, 4, 3, 2, 5)) + out = tmp.reshape(m_tiles * 128, k_tiles * f // block_size) + return out[0:m, 0:k] + + +def dequantize_nvfp4_to_dtype( + tensor_fp4, tensor_sf, global_scale, dtype, device, block_size=16 +): + """Dequantize the fp4 tensor back to high precision.""" + # Two fp4 values are packed into one uint8. + assert tensor_fp4.dtype == torch.uint8 + m, packed_k = tensor_fp4.shape + k = packed_k * 2 + tensor_f32 = break_fp4_bytes(tensor_fp4, dtype) + tensor_f32 = tensor_f32.reshape(m, k // block_size, block_size) + tensor_sf = tensor_sf.view(torch.float8_e4m3fn) + tensor_sf = convert_swizzled_to_linear(tensor_sf, m, k, block_size) + tensor_sf_dtype = tensor_sf.to(torch.float32) / global_scale + + # scale the tensor + out = (tensor_f32 * tensor_sf_dtype.unsqueeze(-1)).reshape(m, k) + return out.to(dtype=dtype) + + +def break_fp4_bytes(a, dtype): + assert a.dtype == torch.uint8 + m, n = a.shape + + # Vectorized nibble processing + a_flat = a.flatten() + high = (a_flat & 0xF0) >> 4 # Upper nibbles + low = a_flat & 0x0F # Lower nibbles + + # Combine nibbles for batch processing + combined = torch.stack((low, high), dim=1).flatten() + + # Vectorized sign and magnitude extraction + signs = (combined & 0x08).to(torch.bool) # Sign bits + abs_vals = (combined & 0x07).to(torch.long) # Magnitude indices + + # Device-aware lookup and sign application + kE2M1 = kE2M1ToFloat.to(device=a.device) + values = kE2M1[abs_vals] * torch.where(signs, -1.0, 1.0) + + # Reshape to final form + return values.reshape(m, n * 2).to(dtype=dtype) + + +def generate_balanced_routing( + hidden_states: torch.Tensor, num_experts: int, top_k: int +): + """ + Generate routing weights and topk indices such that every expert is active. + Returns routing_weights, topk_idx + """ + + num_tokens, hidden_dim = hidden_states.shape + # num_tokens = batch_size * seq_len + + # First, assign at least one token per expert + tokens_per_expert = torch.arange(num_tokens) % num_experts + tokens_per_expert = tokens_per_expert[torch.randperm(num_tokens)] # shuffle + + # Each token has top_k experts — start with one guaranteed expert + topk_idx = torch.full((num_tokens, top_k), -1, dtype=torch.long) + topk_idx[:, 0] = tokens_per_expert + + # For remaining top_k - 1 experts, pick randomly (allowing repeats) + if top_k > 1: + random_choices = torch.randint(0, num_experts, (num_tokens, top_k - 1)) + topk_idx[:, 1:] = random_choices + + # Normalize routing weights so each token's weights sum to 1 + routing_weights = torch.rand(num_tokens, top_k) + routing_weights /= routing_weights.sum(dim=-1, keepdim=True) + + # Reshape back if needed + routing_weights = routing_weights.view(num_tokens, top_k) + topk_idx = topk_idx.view(num_tokens, top_k) + + return routing_weights, topk_idx + + +def prepare_inputs( + hidden_states: torch.Tensor, + router_logits: torch.Tensor, + num_experts: int, + topk: int, +): + routing_weights, topk_idx = generate_balanced_routing( + router_logits, num_experts, topk + ) + + masked_m = [] + for i in range(num_experts): + mask = topk_idx.view(-1) == i + masked_m.append(mask.sum()) + + masked_m = torch.tensor(masked_m, dtype=torch.int32) + # Initialize the hidden_states_3d with ones instead of empty to avoid nan + # issue. + hidden_states_3d = torch.ones( + (num_experts, max(masked_m), hidden_states.shape[1]), + dtype=hidden_states.dtype, + device=hidden_states.device, + ) + for i in range(num_experts): + hidden_states_3d[i, : masked_m[i], :] = hidden_states[topk_idx.view(-1) == i] + + return hidden_states_3d, masked_m, topk_idx, routing_weights + + +MNK_FACTORS = [ + (2, 1024, 1024), + (2, 1024, 1536), + (2, 3072, 1024), + (2, 3072, 1536), + (64, 1024, 1024), + (64, 1024, 1536), + (64, 3072, 1024), + (64, 2048, 1024), + (224, 1024, 1024), + (224, 1024, 1536), +] + + +# Reference implementation of torch_moe +def torch_moe(a, w1, w2, score, topk, expert_map): + B, D = a.shape + a = a.view(B, -1, D).repeat(1, topk, 1).reshape(-1, D) + out = torch.zeros(B * topk, w2.shape[1], dtype=a.dtype, device=a.device) + score = torch.softmax(score, dim=-1, dtype=torch.float32) + topk_weight, topk_ids = torch.topk(score, topk) + topk_weight = topk_weight.view(-1) + topk_ids = topk_ids.view(-1) + if expert_map is not None: + topk_ids = expert_map[topk_ids] + for i in range(w1.shape[0]): + mask = topk_ids == i + if mask.sum(): + out[mask] = SiluAndMul()(a[mask] @ w1[i].transpose(0, 1)) @ w2[i].transpose( + 0, 1 + ) + return ( + out.view(B, -1, w2.shape[1]) * topk_weight.view(B, -1, 1).to(out.dtype) + ).sum(dim=1) + + +def torch_moe_nvfp4(a, w1, w2, topk, topk_weight, topk_ids): + B, D = a.shape + a = a.view(B, -1, D).repeat(1, topk, 1).reshape(-1, D) + out = torch.zeros(B * topk, w2.shape[1], dtype=a.dtype, device=a.device) + + topk_weight = topk_weight.view(-1) + topk_ids = topk_ids.view(-1) + + for i in range(w1.shape[0]): + mask = topk_ids == i + if mask.sum(): + m = w1[i].shape[0] + assert m % 2 == 0 + # Note: w1 and w3 are swapped! + w3_expert, w1_expert = w1[i][m // 2 :, :], w1[i][: m // 2, :] + inter = F.silu(a[mask] @ w1_expert.t()) * (a[mask] @ w3_expert.t()) + inter_gs = torch.tensor(1.0).cuda() + inter_q, inter_blockscale = fp4_quantize(inter, inter_gs) + inter = dequantize_nvfp4_to_dtype( + inter_q, + inter_blockscale, + inter_gs, + dtype=inter.dtype, + device=inter.device, + block_size=16, + ).cuda() + out[mask] = inter @ w2[i].transpose(0, 1) + return ( + out.view(B, -1, w2.shape[1]) * topk_weight.view(B, -1, 1).to(out.dtype) + ).sum(dim=1) + + +def grouped_gemm_ref( + hidden_states_expanded: torch.Tensor, + hidden_states_3d: torch.Tensor, + weights: torch.Tensor, + topk_idx: torch.Tensor, + masked_m: torch.Tensor, + B: int, + topk: int, + num_experts: int, + *, + block_size: int = 16, +) -> torch.Tensor: + """ + Computes the reference grouped GEMM (fp4 quantized per-expert loop), + computes flashinfer grouped GEMM (for scale consistency), + and returns ONLY the repacked reference output: out_ref. + + Returns: + out_ref: Tensor [num_experts, max_m, n_out] + """ + device_hs = hidden_states_expanded.device + device_w = weights.device + out_dtype = weights.dtype + n_out = weights.shape[1] + + # Flattened reference output (B*topk, n_out) + out = torch.zeros((B * topk, n_out), dtype=out_dtype, device=device_w) + + # Per-expert reference compute loop + for i in range(num_experts): + mask = topk_idx.view(-1) == i + if mask.any(): + lhs = hidden_states_expanded[mask] + rhs = weights[i] + + a_amax = lhs.abs().max().to(torch.float32).to(device_hs) + b_amax = rhs.abs().max().to(torch.float32).to(device_w) + + a_gs = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX / a_amax + b_gs = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX / b_amax + + lhsq, lhsq_sf = fp4_quantize(lhs, a_gs) + rhsq, rhsq_sf = fp4_quantize(rhs, b_gs) + + lhs_in_dtype = dequantize_nvfp4_to_dtype( + lhsq, + lhsq_sf, + a_gs, + dtype=lhs.dtype, + device=device_hs, + block_size=block_size, + ) + rhs_in_dtype = dequantize_nvfp4_to_dtype( + rhsq, + rhsq_sf, + b_gs, + dtype=rhs.dtype, + device=device_w, + block_size=block_size, + ) + + out[mask] = lhs_in_dtype @ rhs_in_dtype.t() + + # Determine per-expert max_m + max_m_val = int(masked_m.max().item()) + + # Repack into [num_experts, max_m, n_out] + out_ref = torch.zeros( + (num_experts, max_m_val, n_out), + dtype=out.dtype, + device=out.device, + ) + expert_slot = [0] * num_experts + + for i, expert_id in enumerate(topk_idx.view(-1).tolist()): + slot = expert_slot[expert_id] + if slot < max_m_val: + out_ref[expert_id, slot, :] = out[i] + expert_slot[expert_id] += 1 + else: + raise IndexError( + f"Expert {expert_id} exceeded max slots ({max_m_val}). " + "Increase max_m or check masked_m." + ) + + return out_ref + + +def flashinfer_cutedsl_grouped_gemm_nt_masked( + hidden_states: torch.Tensor, # 3d + input_global_scale: torch.Tensor, # (l,) + weights: torch.Tensor, + w_global_scale: torch.Tensor, # (l,) + masked_m: torch.Tensor, +): + # hidden_states: [l, m, k] + # weights: [l, n, k] + aq, aq_sf = scaled_fp4_grouped_quantize( + hidden_states, + masked_m.to(hidden_states.device), + input_global_scale, + ) + num_experts, n, k = weights.shape + bq, bq_sf = scaled_fp4_grouped_quantize( + weights, + torch.full((num_experts,), n, device=weights.device, dtype=torch.int32), + w_global_scale, + ) + + out = torch.zeros( + (num_experts, max(masked_m), n), dtype=weights.dtype, device=aq.device + ) + out = out.permute(1, 2, 0) # requirement of kernel + sf_vec_size = 16 + ab_dtype = "float4_e2m1fn" + sf_dtype = "float8_e4m3fn" + c_dtype = "bfloat16" + alpha = 1.0 / (input_global_scale * w_global_scale).to(out.dtype).view( + 1, 1, num_experts + ) + + def get_cute_dtype(input: torch.Tensor) -> str: + if input.dtype == torch.bfloat16: + return "bfloat16" + elif input.dtype == torch.float16: + return "float16" + elif input.dtype == torch.float32: + return "float32" + else: + raise ValueError(f"Unsupported cute dtype {input.dtype}") + + cutedsl_gmm_masked( + (aq, aq_sf), + (bq, bq_sf), + out, + masked_m.to(aq.device), + ab_dtype=ab_dtype, + sf_dtype=sf_dtype, + c_dtype=c_dtype, + sf_vec_size=sf_vec_size, + alpha=alpha, + alpha_dtype=get_cute_dtype(alpha), + ) + + return out + + +@pytest.mark.parametrize("bs, hidden_dim, inter_dim", [(2, 128, 256), (16, 128, 512)]) +@pytest.mark.parametrize("topk", [1, 2, 4]) +@torch.inference_mode() +def test_flashinfer_cutedsl_moe_masked( + bs: int, hidden_dim: int, inter_dim: int, topk: int +): + torch.manual_seed(42) + device = "cuda" + num_experts = 8 + hidden_states = ( + torch.randn(bs, hidden_dim, dtype=torch.bfloat16, device=device) / 5.0 + ) + w1 = ( + torch.randn( + num_experts, 2 * inter_dim, hidden_dim, dtype=torch.bfloat16, device=device + ) + / 10.0 + ) + w2 = ( + torch.randn( + num_experts, hidden_dim, inter_dim, dtype=torch.bfloat16, device=device + ) + / 10.0 + ) + router_logits = torch.randn(bs, num_experts, dtype=torch.float32) + + hidden_states_expanded = ( + hidden_states.view(bs, -1, hidden_dim) + .repeat(1, topk, 1) + .reshape(-1, hidden_dim) + ) + hidden_states_3d, masked_m, topk_idx, routing_weights = prepare_inputs( + hidden_states_expanded, router_logits, num_experts, topk + ) + + w1_amax = w1.abs().amax(dim=(1, 2)).to(torch.float32).to(w1.device) + w2_amax = w2.abs().amax(dim=(1, 2)).to(torch.float32).to(w2.device) + input_global_scale = torch.ones( + (num_experts,), dtype=torch.float32, device=hidden_states.device + ) + + w1_global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX / w1_amax + w2_global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX / w2_amax + a2_global_scale = torch.ones( + (num_experts,), dtype=torch.float32, device=hidden_states.device + ) # assume intermediate scale is 1.0 + + w1_fp4, w1_blockscale = scaled_fp4_grouped_quantize( + w1, + torch.ones(num_experts, dtype=torch.int32, device=w1.device) * 2 * inter_dim, + w1_global_scale, + ) + w2_fp4, w2_blockscale = scaled_fp4_grouped_quantize( + w2, + torch.ones(num_experts, dtype=torch.int32, device=w2.device) * hidden_dim, + w2_global_scale, + ) + + w1_alpha = 1.0 / (input_global_scale * w1_global_scale) + w2_alpha = 1.0 / (a2_global_scale * w2_global_scale) + + out = torch.empty_like(hidden_states_3d, device=hidden_states.device) + # Note: the 1st dim shouldn't be bs + wk = torch.empty( + num_experts, + hidden_states_3d.shape[1], + inter_dim * 2, + dtype=hidden_states_3d.dtype, + device=hidden_states.device, + ) + flashinfer_cutedsl_moe_masked( + hidden_states_3d.to(hidden_states.device), + input_global_scale, + w1_fp4.permute(2, 0, 1), + w1_blockscale, + w1_alpha, + w2_fp4.permute(2, 0, 1), + a2_global_scale, + w2_blockscale, + w2_alpha, + masked_m.to(hidden_states.device), + wk, + out, + ) + + # reference + # input_global_scale is per-expert ([num_experts]); fp4_quantize and + # dequantize_nvfp4_to_dtype are non-grouped APIs that expect [1] or + # [num_tokens]. Use a single element since all values are uniform here. + a_global = input_global_scale[:1].contiguous() + a_fp4, a_scale_interleaved = fp4_quantize(hidden_states, a_global) + a_in_dtype = dequantize_nvfp4_to_dtype( + a_fp4, + a_scale_interleaved, + a_global, + dtype=hidden_states.dtype, + device=hidden_states.device, + block_size=16, + ) + w1_d = torch.empty( + (num_experts, 2 * inter_dim, hidden_dim), device=w1.device, dtype=w1.dtype + ) + w2_d = torch.empty( + (num_experts, hidden_dim, inter_dim), device=w2.device, dtype=w2.dtype + ) + + for idx in range(0, num_experts): + w1_fp4_sliced, w1_blockscale_sliced = fp4_quantize( + w1[idx], w1_global_scale[idx] + ) + w2_fp4_sliced, w2_blockscale_sliced = fp4_quantize( + w2[idx], w2_global_scale[idx] + ) + w1_d[idx] = dequantize_nvfp4_to_dtype( + w1_fp4_sliced, + w1_blockscale_sliced, + w1_global_scale[idx], + dtype=w1.dtype, + device=w1.device, + block_size=16, + ) + w2_d[idx] = dequantize_nvfp4_to_dtype( + w2_fp4_sliced, + w2_blockscale_sliced, + w2_global_scale[idx], + dtype=w2.dtype, + device=w2.device, + block_size=16, + ) + + ref_output = torch_moe_nvfp4( + a_in_dtype, + w1_d, + w2_d, + topk, + routing_weights.to(a_in_dtype.device), + topk_idx.to(a_in_dtype.device), + ) + out_weighted = torch.zeros_like(ref_output, device=out.device, dtype=out.dtype) + + positions = torch.nonzero(masked_m[topk_idx], as_tuple=False) + rows, cols = positions[:, 0], positions[:, 1] + experts = topk_idx[rows, cols] + for i in range(num_experts): + mask = experts == i + if mask.any(): + idx = torch.nonzero(mask, as_tuple=False).squeeze(-1) + r, c = rows[idx], cols[idx] + out_weighted[r] += out[i, : len(r), :] * routing_weights[r, c].to( + out.device + ).unsqueeze(-1) + torch.testing.assert_close( + out_weighted.cpu(), ref_output.cpu(), atol=2e-1, rtol=2e-1 + ) + + +@pytest.mark.parametrize( + "bs, hidden_dim, inter_dim, topk", [(2, 128, 256, 2), (16, 128, 512, 5)] +) +@torch.inference_mode() +def test_grouped_gemm_nt_masked( + bs: int, hidden_dim: int, inter_dim: int, topk: int +) -> None: + torch.manual_seed(42) + B = bs + D = hidden_dim + N = inter_dim + # CuteDSL group gemm has issue when not all experts are active. + # i.e. masked = [2, 3, 0, 0, 1] where the 2nd and 3rd experts are inactive + # see https://github.com/flashinfer-ai/flashinfer/issues/1856 + num_experts = bs + hidden_states = torch.randn(B, D, dtype=torch.bfloat16, device="cuda") + weights = torch.randn(num_experts, N, D, dtype=torch.bfloat16, device="cuda") + router_logits = torch.randn(B, num_experts, dtype=torch.float32) + + hidden_states_expanded = ( + hidden_states.view(B, -1, D).repeat(1, topk, 1).reshape(-1, D) + ) + hidden_states_3d, masked_m, topk_idx, _ = prepare_inputs( + hidden_states_expanded, router_logits, num_experts, topk + ) + + a_amax = ( + hidden_states_3d.abs() + .amax(dim=(1, 2)) + .to(torch.float32) + .to(hidden_states.device) + ) + b_amax = weights.abs().amax(dim=(1, 2)).to(torch.float32).to(weights.device) + a_gs = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX / a_amax + b_gs = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX / b_amax + out_flashinfer = flashinfer_cutedsl_grouped_gemm_nt_masked( + hidden_states_3d.to(hidden_states.device), a_gs, weights, b_gs, masked_m + ) + # reference + out_ref = grouped_gemm_ref( + hidden_states_expanded=hidden_states_expanded, + hidden_states_3d=hidden_states_3d, + weights=weights, + topk_idx=topk_idx, + masked_m=masked_m, + B=B, + topk=topk, + num_experts=num_experts, + ) + # Note: just to compare the masked position due to cutedsl may write nan + # into unmasked position. + for i in range(num_experts): + torch.testing.assert_close( + out_flashinfer.permute(2, 0, 1)[i, : masked_m[i]], + out_ref.to(out_flashinfer.device)[i, : masked_m[i]], + atol=1e-1, + rtol=1e-1, + ) + + +if __name__ == "__main__": + test_flashinfer_cutedsl_moe_masked(16, 128, 512, 4) + test_grouped_gemm_nt_masked(16, 128, 512, 4) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_cutlass_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_cutlass_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..64a5a79aee92e44fdd3d31b38e476a86d0f82bdc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_cutlass_moe.py @@ -0,0 +1,608 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import copy +import dataclasses +from math import prod + +import pytest +import torch + +import vllm.model_executor.layers.fused_moe.modular_kernel as mk +from tests.kernels.moe.utils import make_dummy_moe_config +from vllm import _custom_ops as ops +from vllm.config import ParallelConfig, VllmConfig, set_current_vllm_config +from vllm.model_executor.layers.fused_moe import fused_experts, fused_topk +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.all2all_utils import ( + maybe_make_prepare_finalize, +) +from vllm.model_executor.layers.fused_moe.config import ( + FUSED_MOE_UNQUANTIZED_CONFIG, + FusedMoEQuantConfig, + fp8_w8a8_moe_quant_config, +) +from vllm.model_executor.layers.fused_moe.experts.cutlass_moe import ( + CutlassExpertsFp4, + CutlassExpertsFp8, + run_cutlass_moe_fp8, +) +from vllm.model_executor.layers.fused_moe.utils import moe_kernel_quantize_input +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +NUM_EXPERTS = [40, 64] +TOP_KS = [6, 8] + +MNK_FACTORS = [ + (2, 1024, 1024), + (2, 3072, 1024), + (2, 3072, 1536), + (7, 3072, 1536), + (64, 1024, 1024), + (64, 1024, 1536), + (64, 3072, 1024), + (224, 1024, 1024), + (224, 3072, 1024), + (224, 3072, 1536), + (32768, 1024, 1024), + # These sizes trigger wrong answers. + # (7232, 2048, 5120), + # (40000, 2048, 5120), +] + +vllm_config = VllmConfig(parallel_config=ParallelConfig(pipeline_parallel_size=1)) + + +def test_cutlass_moe_supports_gelu_tanh_activation_metadata(): + assert CutlassExpertsFp8._supports_activation(MoEActivation.GELU_TANH) + assert CutlassExpertsFp4._supports_activation(MoEActivation.GELU_TANH) + assert CutlassExpertsFp4._supports_activation(MoEActivation.GELU_TANH_NO_MUL) + + +@dataclasses.dataclass +class MOETensors: + a: torch.Tensor + w1: torch.Tensor + w2: torch.Tensor + ab_strides1: torch.Tensor + c_strides1: torch.Tensor + ab_strides2: torch.Tensor + c_strides2: torch.Tensor + + @staticmethod + def make_moe_tensors( + m: int, k: int, n: int, e: int, dtype: torch.dtype + ) -> "MOETensors": + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + w1 = torch.randn((e, 2 * n, k), device="cuda", dtype=dtype) / 10 + w2 = torch.randn((e, k, n), device="cuda", dtype=dtype) / 10 + ab_strides1 = torch.full((e,), k, device="cuda", dtype=torch.int64) + c_strides1 = torch.full((e,), 2 * n, device="cuda", dtype=torch.int64) + ab_strides2 = torch.full((e,), n, device="cuda", dtype=torch.int64) + c_strides2 = torch.full((e,), k, device="cuda", dtype=torch.int64) + return MOETensors( + a=a, + w1=w1, + w2=w2, + ab_strides1=ab_strides1, + c_strides1=c_strides1, + ab_strides2=ab_strides2, + c_strides2=c_strides2, + ) + + +@dataclasses.dataclass +class MOETensors8Bit(MOETensors): + # quantized + a_q: torch.Tensor | None = None # a -> a_q + w1_q: torch.Tensor | None = None # w1 -> w1_q + w2_q: torch.Tensor | None = None # w2 -> w2_q + a_scale: torch.Tensor | None = None + w1_scale: torch.Tensor | None = None + w2_scale: torch.Tensor | None = None + # dequantized + a_d: torch.Tensor | None = None # a -> a_q -> a_d + w1_d: torch.Tensor | None = None # w1 -> w1_q -> w1_d + w2_d: torch.Tensor | None = None # w2 -> w2_q -> w2_d + + @staticmethod + def make_moe_tensors_8bit( + m: int, k: int, n: int, e: int, per_act_token: bool, per_out_channel: bool + ) -> "MOETensors8Bit": + dtype = torch.half + q_dtype = torch.float8_e4m3fn + + moe_tensors_fp16 = MOETensors.make_moe_tensors(m, k, n, e, dtype) + + # a -> a_q, w1 -> w1_q, w2 -> w2_q + n_b_scales = 2 * n if per_out_channel else 1 + k_b_scales = k if per_out_channel else 1 + # Get the right scale for tests. + a_q, a_scale = ops.scaled_fp8_quant( + moe_tensors_fp16.a, None, use_per_token_if_dynamic=per_act_token + ) + + w1_q = torch.empty((e, 2 * n, k), device="cuda", dtype=q_dtype) + w2_q = torch.empty((e, k, n), device="cuda", dtype=q_dtype) + + w1_scale = torch.empty((e, n_b_scales, 1), device="cuda", dtype=torch.float32) + w2_scale = torch.empty((e, k_b_scales, 1), device="cuda", dtype=torch.float32) + for expert in range(e): + w1_q[expert], w1_scale[expert] = ops.scaled_fp8_quant( + moe_tensors_fp16.w1[expert], use_per_token_if_dynamic=per_out_channel + ) + w2_q[expert], w2_scale[expert] = ops.scaled_fp8_quant( + moe_tensors_fp16.w2[expert], use_per_token_if_dynamic=per_out_channel + ) + + # a_q -> a_d, w1_q -> w1_d, w2_q -> w2_d + a_d = a_q.float().mul(a_scale).to(dtype) + w1_d = torch.empty_like(moe_tensors_fp16.w1) + w2_d = torch.empty_like(moe_tensors_fp16.w2) + for expert in range(e): + w1_d[expert] = (w1_q[expert].float() * w1_scale[expert]).half() + w2_d[expert] = (w2_q[expert].float() * w2_scale[expert]).half() + + return MOETensors8Bit( + a=moe_tensors_fp16.a, + w1=moe_tensors_fp16.w1, + w2=moe_tensors_fp16.w2, + ab_strides1=moe_tensors_fp16.ab_strides1, + c_strides1=moe_tensors_fp16.c_strides1, + ab_strides2=moe_tensors_fp16.ab_strides2, + c_strides2=moe_tensors_fp16.c_strides2, + a_q=a_q, + w1_q=w1_q, + w2_q=w2_q, + a_scale=a_scale, + w1_scale=w1_scale, + w2_scale=w2_scale, + a_d=a_d, + w1_d=w1_d, + w2_d=w2_d, + ) + + +def run_with_expert_maps( + num_experts: int, + num_local_experts: int, + quant_config: FusedMoEQuantConfig, + **cutlass_moe_kwargs, +): + def slice_experts(): + slice_params = [ + "w1", + "w2", + ] + full_tensors = { + k: v + for k, v in cutlass_moe_kwargs.items() + if k in slice_params and k in cutlass_moe_kwargs + } + + for i in range(0, num_experts, num_local_experts): + s, e = i, i + num_local_experts + + # make expert map + expert_map = [-1] * num_experts + expert_map[s:e] = list(range(num_local_experts)) + expert_map = torch.tensor(expert_map, dtype=torch.int32, device="cuda") + + # update cutlass moe arg with expert_map + cutlass_moe_kwargs["expert_map"] = expert_map + # update cutlass moe arg tensors + for k, t in full_tensors.items(): + cutlass_moe_kwargs[k] = t[s:e] + + new_quant_config = copy.deepcopy(quant_config) + new_quant_config._w1.scale = quant_config.w1_scale[s:e] + new_quant_config._w2.scale = quant_config.w2_scale[s:e] + + yield cutlass_moe_kwargs, new_quant_config + + out_tensor = torch.zeros_like(cutlass_moe_kwargs["hidden_states"]) + for kwargs, new_quant_config in slice_experts(): + w2 = kwargs["w2"] + a = kwargs["hidden_states"] + moe_config = make_dummy_moe_config( + max_num_tokens=kwargs.get("hidden_states").shape[0], + experts_per_token=kwargs.get("topk_ids").shape[1], + num_experts=num_experts, + num_local_experts=num_local_experts, + hidden_dim=w2.shape[1], + intermediate_size=w2.shape[2], + in_dtype=a.dtype, + ) + kernel = mk.FusedMoEKernel( + maybe_make_prepare_finalize( + moe=moe_config, + quant_config=new_quant_config, + allow_new_interface=True, + use_monolithic=False, + ), + CutlassExpertsFp8( + moe_config=moe_config, + quant_config=new_quant_config, + ), + ) + out_tensor = out_tensor + kernel.apply(**kwargs) + + return out_tensor + + +def run_8_bit( + moe_tensors: MOETensors8Bit, + topk_weights: torch.Tensor, + topk_ids: torch.Tensor, + per_act_token: bool, + per_out_ch: bool, + num_local_experts: int | None = None, +) -> torch.Tensor: + assert not any( + [ + t is None + for t in [ + moe_tensors.w1_q, + moe_tensors.w2_q, + moe_tensors.w1_scale, + moe_tensors.w2_scale, + moe_tensors.a_scale, + ] + ] + ) + + quant_config = fp8_w8a8_moe_quant_config( + w1_scale=moe_tensors.w1_scale, + w2_scale=moe_tensors.w2_scale, + per_act_token_quant=per_act_token, + per_out_ch_quant=per_out_ch, + # Set to moe_tensors.a_scale iff static scales + per tensor. + # This is not currently being tested. + a1_scale=None, + ) + + num_experts = moe_tensors.w1.size(0) # type: ignore[attr-defined] + with_ep = num_local_experts is not None or num_local_experts == num_experts + + kwargs = { + "hidden_states": moe_tensors.a, + "w1": moe_tensors.w1_q, # type: ignore[union-attr] + "w2": moe_tensors.w2_q, # type: ignore[union-attr] + "topk_weights": topk_weights, + "topk_ids": topk_ids, + "global_num_experts": num_experts, + "activation": MoEActivation.SILU, + "expert_map": None, + "apply_router_weight_on_input": False, + } + + if not with_ep: + moe_config = make_dummy_moe_config( + max_num_tokens=moe_tensors.a.shape[0], + experts_per_token=topk_ids.shape[1], + num_experts=num_experts, + num_local_experts=num_local_experts, + hidden_dim=moe_tensors.w2_q.shape[1], # type: ignore[union-attr] + intermediate_size=moe_tensors.w2_q.shape[2], # type: ignore[union-attr] + in_dtype=moe_tensors.a.dtype, + ) + kernel = mk.FusedMoEKernel( + maybe_make_prepare_finalize( + moe=moe_config, + quant_config=quant_config, + allow_new_interface=True, + use_monolithic=False, + ), + CutlassExpertsFp8( + moe_config=moe_config, + quant_config=quant_config, + ), + ) + return kernel.apply(**kwargs) + + assert num_local_experts is not None + return run_with_expert_maps( + num_experts, + num_local_experts, # type: ignore[arg-type] + quant_config, + **kwargs, + ) + + +@pytest.mark.parametrize("m,n,k", MNK_FACTORS) +@pytest.mark.parametrize("e", NUM_EXPERTS) +@pytest.mark.parametrize("topk", TOP_KS) +@pytest.mark.parametrize("per_act_token", [True, False]) +@pytest.mark.parametrize("per_out_ch", [True, False]) +@pytest.mark.skipif( + (lambda x: x is None or not ops.cutlass_group_gemm_supported(x.to_int()))( + current_platform.get_device_capability() + ), + reason="Grouped gemm is not supported on this GPU type.", +) +def test_cutlass_moe_8_bit_no_graph( + m: int, + n: int, + k: int, + e: int, + topk: int, + per_act_token: bool, + per_out_ch: bool, + monkeypatch, + workspace_init, + ep_size: int | None = None, +): + set_random_seed(7) + with set_current_vllm_config(vllm_config): + mt = MOETensors8Bit.make_moe_tensors_8bit(m, k, n, e, per_act_token, per_out_ch) + + score = torch.randn((m, e), device="cuda", dtype=torch.half) + topk_weights, topk_ids, _ = fused_topk(mt.a, score, topk, renormalize=False) + + # Note that we are using the dequantized versions of the tensors. + # Using a, w1 and w2 directly results in minor output differences. + + quant_config = FUSED_MOE_UNQUANTIZED_CONFIG + triton_output = fused_experts( + mt.a_d, mt.w1_d, mt.w2_d, topk_weights, topk_ids, quant_config=quant_config + ) + + if ep_size is not None: + assert e % ep_size == 0, "Cannot distribute experts evenly" + number_local_experts = e // ep_size + else: + number_local_experts = None + + cutlass_output = run_8_bit( + mt, topk_weights, topk_ids, per_act_token, per_out_ch, number_local_experts + ) + + # Note 5.5 only needed for larger problem sizes, 5 works ok for + # the rest. + torch.testing.assert_close( + triton_output, cutlass_output, atol=5.5e-2, rtol=1e-2 + ) + + +@pytest.mark.parametrize("m,n,k", MNK_FACTORS) +@pytest.mark.parametrize("e", NUM_EXPERTS) +@pytest.mark.parametrize("topk", TOP_KS) +@pytest.mark.parametrize("per_act_token", [True, False]) +@pytest.mark.parametrize("per_out_ch", [True, False]) +@pytest.mark.skipif( + (lambda x: x is None or not ops.cutlass_group_gemm_supported(x.to_int()))( + current_platform.get_device_capability() + ), + reason="Grouped gemm is not supported on this GPU type.", +) +def test_cutlass_moe_8_bit_cuda_graph( + m: int, + n: int, + k: int, + e: int, + topk: int, + per_act_token: bool, + per_out_ch: bool, + monkeypatch, + workspace_init, +): + set_random_seed(7) + with set_current_vllm_config(vllm_config): + dtype = torch.half + + mt = MOETensors8Bit.make_moe_tensors_8bit(m, k, n, e, per_act_token, per_out_ch) + + score = torch.randn((m, e), device="cuda", dtype=dtype) + topk_weights, topk_ids, _ = fused_topk(mt.a, score, topk, renormalize=False) + + # Note that we are using the dequantized versions of the tensors. + # Using a, w1 and w2 directly results in minor output differences. + quant_config = FUSED_MOE_UNQUANTIZED_CONFIG + triton_output = fused_experts( + mt.a_d, mt.w1_d, mt.w2_d, topk_weights, topk_ids, quant_config=quant_config + ) + + stream = torch.cuda.Stream() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + cutlass_output = run_8_bit( + mt, topk_weights, topk_ids, per_act_token, per_out_ch + ) + + torch.accelerator.synchronize() + graph.replay() + torch.accelerator.synchronize() + + torch.testing.assert_close(triton_output, cutlass_output, atol=9e-2, rtol=1e-2) + + +@pytest.mark.parametrize("m", [64]) +@pytest.mark.parametrize("n", [1024]) +@pytest.mark.parametrize("k", [4096]) +@pytest.mark.parametrize("e", [16]) +@pytest.mark.parametrize("topk", [1, 8]) +@pytest.mark.parametrize("per_act_token", [True]) +@pytest.mark.parametrize("per_out_channel", [True]) +@pytest.mark.parametrize("ep_size", [1, 2, 4, 8, 16]) +@pytest.mark.skipif( + (lambda x: x is None or not ops.cutlass_group_gemm_supported(x.to_int()))( + current_platform.get_device_capability() + ), + reason="Grouped gemm is not supported on this GPU type.", +) +def test_cutlass_moe_8_bit_EP( + m: int, + n: int, + k: int, + e: int, + topk: int, + per_act_token: bool, + per_out_channel: bool, + ep_size: int, + monkeypatch, + workspace_init, +): + test_cutlass_moe_8_bit_no_graph( + m, + n, + k, + e, + topk, + per_act_token, + per_out_channel, + monkeypatch, + workspace_init, + ep_size, + ) + + +LARGE_MNK_FACTORS = [ + (1, 8192, 5120, 31), + (32768, 1024, 1024, 16), + (65536, 512, 1024, 16), +] + + +@pytest.mark.parametrize("m,n,k,topk", LARGE_MNK_FACTORS) +@pytest.mark.parametrize("e", [128]) +@pytest.mark.parametrize("per_act_token", [False]) +@pytest.mark.parametrize("per_out_channel", [True]) +@pytest.mark.parametrize("ep_size", [8]) +@pytest.mark.skipif( + (lambda x: x is None or not ops.cutlass_group_gemm_supported(x.to_int()))( + current_platform.get_device_capability() + ), + reason="Grouped gemm is not supported on this GPU type.", +) +def test_cutlass_moe_8_bit_EP_large( + m: int, + n: int, + k: int, + e: int, + topk: int, + per_act_token: bool, + per_out_channel: bool, + ep_size: int, + monkeypatch, + workspace_init, +): + test_cutlass_moe_8_bit_no_graph( + m, + n, + k, + e, + topk, + per_act_token, + per_out_channel, + monkeypatch, + workspace_init, + ep_size, + ) + + +@pytest.mark.parametrize("m,n,k,topk", [(1, 8192, 5120, 31)]) +@pytest.mark.parametrize("e", [128]) +@pytest.mark.parametrize("per_act_token", [False]) +@pytest.mark.parametrize("per_out_channel", [True]) +@pytest.mark.parametrize("ep_size", [8]) +@pytest.mark.skipif( + (lambda x: x is None or not ops.cutlass_group_gemm_supported(x.to_int()))( + current_platform.get_device_capability() + ), + reason="Grouped gemm is not supported on this GPU type.", +) +def test_run_cutlass_moe_fp8( + m: int, + n: int, + k: int, + e: int, + topk: int, + per_act_token: bool, + per_out_channel: bool, + ep_size: int, + workspace_init, +): + set_random_seed(7) + with set_current_vllm_config(vllm_config): + mt = MOETensors8Bit.make_moe_tensors_8bit( + m, k, n, e, per_act_token, per_out_channel + ) + + score = torch.randn((m, e), device="cuda", dtype=torch.half) + topk_weights, topk_ids, _ = fused_topk(mt.a, score, topk, renormalize=False) + # we want to make sure there is at least one token that's generated in + # this expert shard and at least one token that's NOT generated in this + # expert shard + topk_ids[0][0] = -1 + topk_ids[0][1] = 1 + + workspace13_shape = (m * topk, max(2 * n, k)) + workspace2_shape = (m * topk, max(n, k)) + output_shape = (m, k) + + workspace13 = torch.empty( + prod(workspace13_shape), device="cuda", dtype=mt.a.dtype + ) + workspace2 = torch.empty( + prod(workspace2_shape), device="cuda", dtype=mt.a.dtype + ) + + num_local_experts = e // ep_size + start, end = 0, num_local_experts + expert_map = [-1] * e + expert_map[start:end] = list(range(num_local_experts)) + expert_map = torch.tensor(expert_map, dtype=torch.int32, device="cuda") + + ab_strides1 = torch.full((e,), k, device="cuda", dtype=torch.int64) + ab_strides2 = torch.full((e,), n, device="cuda", dtype=torch.int64) + c_strides1 = torch.full((e,), 2 * n, device="cuda", dtype=torch.int64) + c_strides2 = torch.full((e,), k, device="cuda", dtype=torch.int64) + + activation = MoEActivation.SILU + a1q, a1q_scale = moe_kernel_quantize_input( + mt.a, mt.a_scale, torch.float8_e4m3fn, per_act_token + ) + global_num_experts = -1 if mt.w1_q is None else mt.w1_q.size(0) + func = lambda output: run_cutlass_moe_fp8( + output, + a1q, + mt.w1_q, + mt.w2_q, + topk_ids, + activation, + global_num_experts, + expert_map, + mt.w1_scale, + mt.w2_scale, + a1q_scale, + None, + ab_strides1, + ab_strides2, + c_strides1, + c_strides2, + workspace13, + workspace2, + None, + mt.a.dtype, + per_act_token, + per_out_channel, + False, + topk_weights, + None, + ) + + workspace13.random_() + output_random_workspace = torch.empty( + output_shape, device="cuda", dtype=mt.a.dtype + ) + func(output_random_workspace) + + workspace13.fill_(0) + output_zero_workspace = torch.zeros( + output_shape, device="cuda", dtype=mt.a.dtype + ) + func(output_zero_workspace) + + torch.testing.assert_close( + output_random_workspace, output_zero_workspace, atol=5e-3, rtol=1e-3 + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_deepep_deepgemm_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_deepep_deepgemm_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..4dc2ebcc73e596ef2da23e4a91199c9405f08912 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_deepep_deepgemm_moe.py @@ -0,0 +1,573 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Test DeepEP + DeepGEMM integration +DeepGEMM are gemm kernels specialized for the +fp8 block-quantized case. +""" + +import dataclasses +from contextlib import contextmanager + +import pytest +import torch.distributed +from torch.distributed import ProcessGroup +from typing_extensions import ParamSpec + +import vllm.envs as envs +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.forward_context import set_forward_context +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.config import ( + FusedMoEQuantConfig, + fp8_w8a8_moe_quant_config, +) +from vllm.model_executor.layers.fused_moe.fused_moe import fused_experts +from vllm.model_executor.layers.fused_moe.modular_kernel import FusedMoEKernel +from vllm.utils.deep_gemm import ( + get_mk_alignment_for_contiguous_layout, + is_deep_gemm_e8m0_used, + is_deep_gemm_supported, +) +from vllm.utils.import_utils import has_deep_ep, has_deep_gemm +from vllm.utils.math_utils import next_power_of_2 +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.worker.workspace import init_workspace_manager + +from ...utils import multi_gpu_test +from .parallel_utils import ProcessGroupInfo, parallel_launch +from .utils import make_dummy_moe_config, make_test_weights + +if has_deep_ep(): + from vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ht import ( + DeepEPHTPrepareAndFinalize, + ) + from vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ll import ( + DeepEPLLPrepareAndFinalize, + ) + + from .parallel_utils import DeepEPHTArgs, DeepEPLLArgs, make_deepep_a2a + +if has_deep_gemm(): + from vllm.model_executor.layers.fused_moe.experts.batched_deep_gemm_moe import ( + BatchedDeepGemmExperts, + ) + from vllm.model_executor.layers.fused_moe.experts.deep_gemm_moe import ( + DeepGemmExperts, + ) + +requires_deep_ep = pytest.mark.skipif( + not has_deep_ep(), + reason="Requires deep_ep kernels", +) + +requires_deep_gemm = pytest.mark.skipif( + not is_deep_gemm_supported(), + reason="Requires deep_gemm kernels", +) + +P = ParamSpec("P") + + +@contextmanager +def with_dp_metadata(M: int, world_size: int): + num_tokens_across_dp = torch.tensor([M] * world_size, device="cpu", dtype=torch.int) + + vllm_config = VllmConfig() + vllm_config.parallel_config.data_parallel_size = world_size + vllm_config.parallel_config.enable_expert_parallel = True + + with set_forward_context( + None, + vllm_config, + num_tokens=M, + num_tokens_across_dp=num_tokens_across_dp, + ): + yield + + +def make_block_quant_fp8_weights( + e: int, + n: int, + k: int, + block_size: list[int], +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + """ + Return weights w1q, w2q, w1_scale, w2_scale + """ + (_, w1q, w1_scale, _), (_, w2q, w2_scale, _) = make_test_weights( + e, n, k, torch.bfloat16, torch.float8_e4m3fn, block_shape=block_size + ) + return w1q, w2q, w1_scale, w2_scale + + +@dataclasses.dataclass +class TestConfig: + topk: int + m: int + k: int + n: int + num_experts: int + per_act_token_quant: bool + block_size: list[int] + # configs for testing low-latency kernels + low_latency: bool + use_fp8_dispatch: bool | None = False + + +@dataclasses.dataclass +class TestTensors: + rank_tokens: torch.Tensor # all ranks make this many tokens + rank_token_scales: torch.Tensor | None + topk: torch.Tensor + topk_weights: torch.Tensor + config: TestConfig + + @staticmethod + def make(config: TestConfig, rank) -> "TestTensors": + dtype = torch.bfloat16 + topk, m, k = (config.topk, config.m, config.k) + + fp8_info = torch.finfo(torch.float8_e4m3fn) + fp8_max, fp8_min = fp8_info.max, fp8_info.min + device = torch.accelerator.current_device_index() + rank_tokens = torch.randn((m, k), device=device, dtype=dtype) / 10.0 + rank_tokens = rank_tokens.clamp(min=fp8_min, max=fp8_max) + rank_token_scales = None + + topk_ids = torch.randint( + low=0, + high=config.num_experts, + size=(m, topk), + device=device, + ).to(dtype=torch.int64) + + topk_weights = torch.randn( + topk_ids.shape, + dtype=torch.float32, + device=device, + ) + + return TestTensors( + rank_tokens=rank_tokens, + rank_token_scales=rank_token_scales, + topk=topk_ids, + topk_weights=topk_weights, + config=config, + ) + + +def make_ll_modular_kernel( + pg: ProcessGroup, + pgi: ProcessGroupInfo, + max_tokens_per_rank: int, + dp_size: int, + hidden_size: int, + q_dtype: torch.dtype | None, + test_config: TestConfig, + quant_config: FusedMoEQuantConfig, +) -> FusedMoEKernel: + assert test_config.low_latency + assert test_config.use_fp8_dispatch is not None + + a2a: DeepEPLLPrepareAndFinalize = make_deepep_a2a( + pg=pg, + pgi=pgi, + dp_size=dp_size, + deepep_ht_args=None, + deepep_ll_args=DeepEPLLArgs( + max_tokens_per_rank=max_tokens_per_rank, + hidden_size=hidden_size, + num_experts=test_config.num_experts, + use_fp8_dispatch=test_config.use_fp8_dispatch, + ), + q_dtype=q_dtype, + block_shape=test_config.block_size, + ) + + fused_experts = BatchedDeepGemmExperts( + max_num_tokens=max_tokens_per_rank, + num_dispatchers=pgi.world_size // dp_size, + quant_config=quant_config, + moe_config=make_dummy_moe_config(), + ) + return FusedMoEKernel( + prepare_finalize=a2a, + fused_experts=fused_experts, + ) + + +def make_ht_modular_kernel( + pg: ProcessGroup, + pgi: ProcessGroupInfo, + dp_size: int, + num_local_experts: int, + q_dtype: torch.dtype | None, + test_config: TestConfig, + quant_config: FusedMoEQuantConfig, +) -> FusedMoEKernel: + assert not test_config.low_latency + assert test_config.use_fp8_dispatch is None + + a2a: DeepEPHTPrepareAndFinalize = make_deepep_a2a( + pg=pg, + pgi=pgi, + dp_size=dp_size, + deepep_ht_args=DeepEPHTArgs(num_local_experts=num_local_experts), + deepep_ll_args=None, + q_dtype=q_dtype, + block_shape=test_config.block_size, + ) + + fused_experts = DeepGemmExperts( + moe_config=make_dummy_moe_config(), + quant_config=quant_config, + ) + return FusedMoEKernel( + prepare_finalize=a2a, + fused_experts=fused_experts, + ) + + +def make_modular_kernel( + pg: ProcessGroup, + pgi: ProcessGroupInfo, + dp_size: int, + num_local_experts: int, + test_tensors: TestTensors, + quant_config: FusedMoEQuantConfig, +) -> FusedMoEKernel: + q_dtype = torch.float8_e4m3fn + test_config = test_tensors.config + + mk: FusedMoEKernel + # Make modular kernel + if test_config.low_latency: + max_tokens_per_rank = max(64, next_power_of_2(test_tensors.rank_tokens.size(0))) + hidden_size = test_tensors.rank_tokens.size(-1) + + mk = make_ll_modular_kernel( + pg=pg, + pgi=pgi, + max_tokens_per_rank=max_tokens_per_rank, + dp_size=dp_size, + hidden_size=hidden_size, + q_dtype=q_dtype, + test_config=test_config, + quant_config=quant_config, + ) + else: + mk = make_ht_modular_kernel( + pg, + pgi, + dp_size, + num_local_experts, + q_dtype, + test_config, + quant_config=quant_config, + ) + + return mk + + +def deepep_deepgemm_moe_impl( + pg: ProcessGroup, + pgi: ProcessGroupInfo, + dp_size: int, + test_tensors: TestTensors, + w1: torch.Tensor, + w2: torch.Tensor, + w1_scale: torch.Tensor | None, + w2_scale: torch.Tensor | None, +) -> torch.Tensor: + test_config = test_tensors.config + num_experts = test_config.num_experts + num_local_experts = w1.size(0) + + def build_expert_map(): + num_local_experts = w1.size(0) + expert_map = torch.full((num_experts,), fill_value=-1, dtype=torch.int32) + s = pgi.rank * num_local_experts + e = s + num_local_experts + expert_map[s:e] = torch.tensor(list(range(num_local_experts))) + device = torch.accelerator.current_device_index() + return expert_map.to(device=device, dtype=torch.int32) + + quant_config = fp8_w8a8_moe_quant_config( + w1_scale=w1_scale, + w2_scale=w2_scale, + # Low-Latency kernels can't dispatch scales. + a1_scale=(None if test_config.low_latency else test_tensors.rank_token_scales), + block_shape=test_config.block_size, + ) + + # Make modular kernel + mk: FusedMoEKernel = make_modular_kernel( + pg=pg, + pgi=pgi, + dp_size=dp_size, + num_local_experts=num_local_experts, + test_tensors=test_tensors, + quant_config=quant_config, + ) + + with with_dp_metadata( + M=test_tensors.rank_tokens.size(0), world_size=pgi.world_size + ): + out = mk.apply( + hidden_states=test_tensors.rank_tokens, + w1=w1, + w2=w2, + topk_weights=test_tensors.topk_weights, + topk_ids=test_tensors.topk, + activation=MoEActivation.SILU, + global_num_experts=num_experts, + expert_map=build_expert_map(), + apply_router_weight_on_input=False, + ) + return out + + +def triton_impl( + a: torch.Tensor, + topk_ids: torch.Tensor, + topk_weights: torch.Tensor, + w1: torch.Tensor, + w2: torch.Tensor, + w1_scale: torch.Tensor, + w2_scale: torch.Tensor, + a1_scale: torch.Tensor, + block_shape: list[int], +): + quant_config = fp8_w8a8_moe_quant_config( + w1_scale=w1_scale, + w2_scale=w2_scale, + a1_scale=a1_scale, + block_shape=block_shape, + ) + + return fused_experts( + hidden_states=a, + w1=w1, + w2=w2, + topk_weights=topk_weights, + topk_ids=topk_ids, + quant_config=quant_config, + ) + + +def _test_deepep_deepgemm_moe( + pgi: ProcessGroupInfo, + dp_size: int, + config: TestConfig, + w1: torch.Tensor, + w2: torch.Tensor, + w1_scale: torch.Tensor, + w2_scale: torch.Tensor, +): + device = torch.device(f"cuda:{pgi.local_rank}") + init_workspace_manager(device) + + set_random_seed(pgi.rank) + + device = torch.accelerator.current_device_index() + w1 = w1.to(device=device) + w2 = w2.to(device=device) + w1_scale = w1_scale.to(device=device) + w2_scale = w2_scale.to(device=device) + + if envs.VLLM_DISTRIBUTED_USE_SPLIT_GROUP: + pg = torch.distributed.split_group( + split_ranks=[list(range(pgi.world_size))], + group_desc="deepep_deepgemm_test", + ) + else: + pg = torch.distributed.new_group(list(range(pgi.world_size))) + test_tensors = TestTensors.make(config, pgi.rank) + block_shape = [w1.size(1) // w1_scale.size(1), w1.size(2) // w1_scale.size(2)] + + with set_current_vllm_config(VllmConfig()): + # Reference + triton_moe = triton_impl( + a=test_tensors.rank_tokens, + topk_ids=test_tensors.topk, + topk_weights=test_tensors.topk_weights, + w1=w1, + w2=w2, + w1_scale=w1_scale, + w2_scale=w2_scale, + a1_scale=test_tensors.rank_token_scales, + block_shape=block_shape, + ) + + # Slice experts for this rank. + num_local_experts = config.num_experts // pgi.world_size + e_start = num_local_experts * pgi.rank + e_end = e_start + num_local_experts + w1_ep = w1[e_start:e_end] + w2_ep = w2[e_start:e_end] + w1_scale_ep = w1_scale[e_start:e_end] + w2_scale_ep = w2_scale[e_start:e_end] + + deepep_moe = deepep_deepgemm_moe_impl( + pg, + pgi, + dp_size, + test_tensors, + w1_ep, + w2_ep, + w1_scale_ep, + w2_scale_ep, + ) + + torch.testing.assert_close( + triton_moe, + deepep_moe, + atol=6e-2, + rtol=6e-2, + ) + + +MNKs = [ + (8, 128, 128), + (8, 128, 512), + (3, 1024, 2048), + (32, 128, 1024), + (45, 512, 2048), + (64, 1024, 1024), + (129, 128, 256), + (129, 1024, 2048), + (222, 1024, 2048), +] + +TOPKS = [2, 6] +NUM_EXPERTS = [32] + + +@pytest.mark.parametrize("mnk", MNKs) +@pytest.mark.parametrize("num_experts", NUM_EXPERTS) +@pytest.mark.parametrize("topk", TOPKS) +@pytest.mark.parametrize("world_dp_size", [(2, 1)]) +@multi_gpu_test(num_gpus=2) +@requires_deep_ep +@requires_deep_gemm +def test_ht_deepep_deepgemm_moe( + mnk: tuple[int, int, int], + num_experts: int, + topk: int, + world_dp_size: tuple[int, int], + disable_deepgemm_ue8m0, + workspace_init, +): + """ + Tests for High-Throughput DeepEP + DeepGemm integration. + """ + + m, n, k = mnk + set_random_seed(7) + + if topk > num_experts: + pytest.skip(f"Skipping test: topk={topk} > E={num_experts}") + + block_m = get_mk_alignment_for_contiguous_layout()[0] + block_size = [block_m, block_m] + + world_size, dp_size = world_dp_size + config = TestConfig( + topk=topk, + m=m, + k=k, + n=n, + num_experts=num_experts, + per_act_token_quant=False, + block_size=block_size, + low_latency=False, + use_fp8_dispatch=None, + ) + + w1, w2, w1_scale, w2_scale = make_block_quant_fp8_weights( + num_experts, n, k, block_size + ) + + parallel_launch( + world_size, + _test_deepep_deepgemm_moe, + dp_size, + config, + w1, + w2, + w1_scale, + w2_scale, + ) + + +MNKs = [ + (1, 128, 2560), + (2, 128, 2560), + (3, 1024, 2560), + (32, 128, 2560), + (45, 512, 2560), + (64, 1024, 2560), + (222, 1024, 2560), +] +# Fix tests for USE_FP8_DISPATCH=True +USE_FP8_DISPATCH = [False] + + +@pytest.mark.parametrize("mnk", MNKs) +@pytest.mark.parametrize("num_experts", NUM_EXPERTS) +@pytest.mark.parametrize("topk", TOPKS) +@pytest.mark.parametrize("use_fp8_dispatch", USE_FP8_DISPATCH) +@pytest.mark.parametrize("block_size", [[128, 128]]) +@pytest.mark.parametrize("world_dp_size", [(2, 1)]) +@multi_gpu_test(num_gpus=2) +@requires_deep_ep +@requires_deep_gemm +def test_ll_deepep_deepgemm_moe( + mnk: tuple[int, int, int], + num_experts: int, + topk: int, + use_fp8_dispatch: bool, + block_size: list[int], + world_dp_size: tuple[int, int], + disable_deepgemm_ue8m0, + workspace_init, +): + """ + Tests for Low-Latency DeepEP + DeepGemm integration. + """ + assert not is_deep_gemm_e8m0_used() + + m, n, k = mnk + set_random_seed(7) + + if topk > num_experts: + pytest.skip(f"Skipping test: topk={topk} > E={num_experts}") + + world_size, dp_size = world_dp_size + config = TestConfig( + topk=topk, + m=m, + k=k, + n=n, + num_experts=num_experts, + per_act_token_quant=False, + block_size=block_size, + low_latency=True, + use_fp8_dispatch=use_fp8_dispatch, + ) + + w1, w2, w1_scale, w2_scale = make_block_quant_fp8_weights( + num_experts, n, k, block_size + ) + + parallel_launch( + world_size, + _test_deepep_deepgemm_moe, + dp_size, + config, + w1, + w2, + w1_scale, + w2_scale, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_deepep_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_deepep_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..7b0dd4ca3383f0dd3509fbab7d092d8457186769 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_deepep_moe.py @@ -0,0 +1,576 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Test deepep dispatch-combine logic +""" + +import dataclasses + +import pytest +import torch.distributed +from torch.distributed import ProcessGroup + +import vllm.envs as envs +from tests.kernels.moe.utils import check_accuracy, make_dummy_moe_config +from vllm import _custom_ops as ops +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.model_executor.layers.activation import SiluAndMul +from vllm.model_executor.layers.fused_moe import TritonExperts +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.config import ( + FusedMoEQuantConfig, +) +from vllm.model_executor.layers.fused_moe.experts.fused_batched_moe import ( + BatchedTritonExperts, +) +from vllm.model_executor.layers.fused_moe.modular_kernel import FusedMoEKernel +from vllm.model_executor.layers.quantization.utils.fp8_utils import ( + per_token_group_quant_fp8, +) +from vllm.platforms import current_platform +from vllm.utils.import_utils import has_deep_ep +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.worker.workspace import init_workspace_manager + +from ...utils import multi_gpu_test +from .parallel_utils import ProcessGroupInfo, parallel_launch + +if has_deep_ep(): + from vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ht import ( + DeepEPHTPrepareAndFinalize, + ) + from vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ll import ( + DeepEPLLPrepareAndFinalize, + ) + + from .parallel_utils import DeepEPHTArgs, DeepEPLLArgs, make_deepep_a2a + +requires_deep_ep = pytest.mark.skipif( + not has_deep_ep(), + reason="Requires deep_ep kernels", +) + +MAX_TOKENS_PER_RANK = 64 + + +def make_weights( + e, n, k, dtype +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + """ + Return weights w1, w2, w1_scale, w2_scale + """ + if dtype in [torch.float16, torch.bfloat16]: + w1 = torch.randn((e, 2 * n, k), device="cuda", dtype=dtype) / 10 + w2 = torch.randn((e, k, n), device="cuda", dtype=dtype) / 10 + return w1, w2, None, None + + # per-out-channel weight quantization + assert dtype == current_platform.fp8_dtype() + w1 = torch.empty((e, 2 * n, k), device="cuda", dtype=torch.float16) + w2 = torch.empty((e, k, n), device="cuda", dtype=torch.float16) + + n_b_scales = 2 * n + k_b_scales = k + w1_q = torch.empty_like(w1, dtype=dtype) + w2_q = torch.empty_like(w2, dtype=dtype) + w1_scale = torch.empty((e, n_b_scales, 1), device="cuda", dtype=torch.float32) + w2_scale = torch.empty((e, k_b_scales, 1), device="cuda", dtype=torch.float32) + for expert in range(e): + w1_q[expert], w1_scale[expert] = ops.scaled_fp8_quant( + w1[expert], use_per_token_if_dynamic=True + ) + w2_q[expert], w2_scale[expert] = ops.scaled_fp8_quant( + w2[expert], use_per_token_if_dynamic=True + ) + return w1_q, w2_q, w1_scale, w2_scale + + +@dataclasses.dataclass +class TestConfig: + dtype: torch.dtype + topk: int + m: int + k: int + n: int + num_experts: int + + +@dataclasses.dataclass +class TestTensors: + rank_tokens: torch.Tensor # all ranks make this many tokens + rank_token_scales: torch.Tensor | None + topk: torch.Tensor + topk_weights: torch.Tensor + config: TestConfig + + @staticmethod + def make(config: TestConfig, low_latency_mode: bool) -> "TestTensors": + # TODO (varun) - check that float16 works ? + assert config.dtype in [torch.bfloat16, current_platform.fp8_dtype()] + token_dtype = ( + torch.bfloat16 + if config.dtype == current_platform.fp8_dtype() + else config.dtype + ) + rank_tokens = ( + torch.randn((config.m, config.k), device="cuda", dtype=token_dtype) / 10 + ) + rank_token_scales = None + + topk = torch.randint( + low=0, high=config.num_experts, size=(config.m, config.topk), device="cuda" + ).to(dtype=torch.int64) + topk_weights = torch.randn(topk.shape, dtype=torch.float32, device="cuda") + return TestTensors( + rank_tokens=rank_tokens, + rank_token_scales=rank_token_scales, + topk=topk, + topk_weights=topk_weights, + config=config, + ) + + +def make_modular_kernel( + pg: ProcessGroup, + pgi: ProcessGroupInfo, + low_latency_mode: bool, + hidden_size: int, + dp_size: int, + num_experts: int, + num_local_experts: int, + q_dtype: torch.dtype | None, + use_fp8_dispatch: bool, + quant_config: FusedMoEQuantConfig, +) -> FusedMoEKernel: + ht_args: DeepEPHTArgs | None = None + ll_args: DeepEPLLArgs | None = None + + if low_latency_mode: + ll_args = DeepEPLLArgs( + max_tokens_per_rank=MAX_TOKENS_PER_RANK, + hidden_size=hidden_size, + num_experts=num_experts, + use_fp8_dispatch=use_fp8_dispatch, + ) + else: + assert not use_fp8_dispatch, ( + "FP8 Dispatch is valid only for low-latency kernels" + ) + ht_args = DeepEPHTArgs(num_local_experts=num_local_experts) + + a2a: DeepEPHTPrepareAndFinalize | DeepEPLLPrepareAndFinalize = make_deepep_a2a( + pg=pg, + pgi=pgi, + dp_size=dp_size, + q_dtype=q_dtype, + block_shape=None, + deepep_ht_args=ht_args, + deepep_ll_args=ll_args, + ) + + num_dispatchers = pgi.world_size // dp_size + + moe_config = make_dummy_moe_config() + + if low_latency_mode: + assert not quant_config.per_act_token_quant, "not supported in ll mode" + fused_experts = BatchedTritonExperts( + max_num_tokens=MAX_TOKENS_PER_RANK, + num_dispatchers=num_dispatchers, + moe_config=moe_config, + quant_config=quant_config, + ) + else: + fused_experts = TritonExperts( + moe_config=moe_config, + quant_config=quant_config, + ) + + mk = FusedMoEKernel( + prepare_finalize=a2a, + fused_experts=fused_experts, + ) + return mk + + +def deep_ep_moe_impl( + pg: ProcessGroup, + pgi: ProcessGroupInfo, + low_latency_mode: bool, + dp_size: int, + test_tensors: TestTensors, + w1: torch.Tensor, + w2: torch.Tensor, + w1_scale: torch.Tensor | None, + w2_scale: torch.Tensor | None, + num_experts: int, + use_fp8_dispatch: bool, + per_act_token_quant: bool, +) -> torch.Tensor: + num_local_experts = w1.size(0) + + def build_expert_map(): + num_local_experts = w1.size(0) + expert_map = torch.full((num_experts,), fill_value=-1, dtype=torch.int32) + s = pgi.rank * num_local_experts + e = s + num_local_experts + expert_map[s:e] = torch.tensor(list(range(num_local_experts))) + device = torch.accelerator.current_device_index() + return expert_map.to(device=device, dtype=torch.int32) + + hidden_size = test_tensors.rank_tokens.size(1) + is_quantized = w1.dtype == current_platform.fp8_dtype() + q_dtype = None + if is_quantized: + q_dtype = current_platform.fp8_dtype() + + out_hidden_states = torch.empty_like(test_tensors.rank_tokens) + total_num_tokens = test_tensors.rank_tokens.size(0) + + quant_config = FusedMoEQuantConfig.make( + q_dtype, + w1_scale=w1_scale, + w2_scale=w2_scale, + per_act_token_quant=per_act_token_quant, + a1_scale=test_tensors.rank_token_scales, + ) + + # Build the kernel (and its DeepEP buffer) once and reuse it across chunks. + # Re-creating it per chunk re-inits rocSHMEM, which only allows one + # allocation per process on ROCm. The buffer is sized by max_tokens_per_rank + # so it is valid for every chunk (mirrors production's cached all2all handle). + mk: FusedMoEKernel = make_modular_kernel( + pg, + pgi, + low_latency_mode, + hidden_size, + dp_size, + num_experts, + num_local_experts, + q_dtype, + use_fp8_dispatch, + quant_config, + ) + + def process_chunk(chunk_start, chunk_end, skip_result_store=False): + rank_tokens_chunk = test_tensors.rank_tokens[chunk_start:chunk_end] + topk_weights_chunk = test_tensors.topk_weights[chunk_start:chunk_end] + topk_chunk = test_tensors.topk[chunk_start:chunk_end] + + if low_latency_mode: + # Reusing one buffer leaves it dirty; the low-latency kernels need + # the zero-initialized regions reset before each dispatch. + mk.prepare_finalize.buffer.clean_low_latency_buffer( + MAX_TOKENS_PER_RANK, + hidden_size, + num_experts, + ) + + out = mk.apply( + hidden_states=rank_tokens_chunk, + w1=w1, + w2=w2, + topk_weights=topk_weights_chunk, + topk_ids=topk_chunk, + activation=MoEActivation.SILU, + global_num_experts=num_experts, + expert_map=build_expert_map(), + apply_router_weight_on_input=False, + ) + + if not skip_result_store: + out_hidden_states[chunk_start:chunk_end, :].copy_(out, non_blocking=True) + + max_num_tokens_per_dp = ( + MAX_TOKENS_PER_RANK if low_latency_mode else total_num_tokens + ) + + for chunk_start_ in range(0, total_num_tokens, max_num_tokens_per_dp): + chunk_start = chunk_start_ + chunk_end = min(chunk_start + max_num_tokens_per_dp, total_num_tokens) + # clamp start and end + chunk_start = min(chunk_start, total_num_tokens - 1) + chunk_end = min(chunk_end, total_num_tokens) + + process_chunk( + chunk_start, chunk_end, skip_result_store=chunk_start_ >= total_num_tokens + ) + + return out_hidden_states + + +def torch_moe_impl( + test_tensors: TestTensors, + w1: torch.Tensor, + w2: torch.Tensor, + w1_scale: torch.Tensor | None, + w2_scale: torch.Tensor | None, + using_fp8_dispatch: bool, + per_act_token_quant: bool, +): + a, topk_ids, topk_weights = ( + test_tensors.rank_tokens, + test_tensors.topk, + test_tensors.topk_weights, + ) + if using_fp8_dispatch: + # The DeepEP implementation is requested to dispatch using FP8. + # For numerical stability for testing, emulate the fp8 dispatch by + # blockwise quant and de-quant. + assert not per_act_token_quant + a = test_tensors.rank_tokens + aq, aq_scale = per_token_group_quant_fp8(a, 128, use_ue8m0=False) + a = ( + (aq.view(-1, 128).to(torch.float32) * aq_scale.view(-1, 1)) + .view(a.shape) + .to(a.dtype) + ) + + is_quantized = w1.dtype == current_platform.fp8_dtype() + a_dtype = a.dtype + if is_quantized: + w1 = w1.to(dtype=torch.float32) * w1_scale + w2 = w2.to(dtype=torch.float32) * w2_scale + a = a.to(dtype=torch.float32) + + m, _ = a.shape + topk = topk_ids.size(1) + out = torch.zeros_like(a) + + for i in range(m): + a_i = a[i] + o_i = out[i] + for j in range(topk): + e = topk_ids[i][j] + e_w = topk_weights[i][j] + w1_e = w1[e] + w2_e = w2[e] + o_i += ( + SiluAndMul()(a_i @ w1_e.transpose(0, 1)) @ w2_e.transpose(0, 1) + ) * e_w + + if is_quantized: + out = out.to(dtype=a_dtype) + + return out + + +def assert_deepep_close( + expected: torch.Tensor, + actual: torch.Tensor, + k: int, + use_fp8_dispatch: bool, +) -> None: + if use_fp8_dispatch and current_platform.is_fp8_fnuz(): + # ROCm e4m3fnuz rounds differently than the reference quant, + # so DeepEP's fp8 dispatch can yield a few outliers even with + # a correct kernel; allow a small fraction of mismatches here. + atol = rtol = 1.5e-1 + check_accuracy(expected, actual, atol=atol, rtol=rtol, percent=0.95) + return + + torch.testing.assert_close( + expected, + actual, + atol=6e-2, + rtol=6e-2, + ) + + +def _deep_ep_moe( + pgi: ProcessGroupInfo, + low_latency_mode: bool, + dp_size: int, + config: TestConfig, + w1: torch.Tensor, + w2: torch.Tensor, + w1_scale: torch.Tensor | None, + w2_scale: torch.Tensor | None, + use_fp8_dispatch: bool, + per_act_token_quant: bool, +): + # Set seed in worker process for deterministic tensor generation. + set_random_seed(7) + + device = torch.device(f"cuda:{pgi.local_rank}") + init_workspace_manager(device) + + if not low_latency_mode: + assert not use_fp8_dispatch, ( + "FP8 dispatch interface is available only in low-latency mode" + ) + + is_quantized = w1.dtype == current_platform.fp8_dtype() + device_idx = torch.accelerator.current_device_index() + w1 = w1.to(device=device_idx) + w2 = w2.to(device=device_idx) + if is_quantized: + assert w1_scale is not None and w2_scale is not None + w1_scale = w1_scale.to(device=device_idx) + w2_scale = w2_scale.to(device=device_idx) + + if envs.VLLM_DISTRIBUTED_USE_SPLIT_GROUP: + pg = torch.distributed.split_group( + split_ranks=[list(range(pgi.world_size))], + group_desc="deepep_test", + ) + else: + pg = torch.distributed.new_group(list(range(pgi.world_size))) + test_tensors = TestTensors.make(config, low_latency_mode) + + with set_current_vllm_config(VllmConfig()): + # Reference + torch_combined = torch_moe_impl( + test_tensors, + w1, + w2, + w1_scale, + w2_scale, + use_fp8_dispatch, + per_act_token_quant, + ) + + # Splice experts for this rank. + num_local_experts = config.num_experts // pgi.world_size + e_start = num_local_experts * pgi.rank + e_end = e_start + num_local_experts + w1_ep = w1[e_start:e_end] + w2_ep = w2[e_start:e_end] + + w1_scale_ep, w2_scale_ep = None, None + if is_quantized: + w1_scale_ep = w1_scale[e_start:e_end] # type: ignore + w2_scale_ep = w2_scale[e_start:e_end] # type: ignore + deepep_combined = deep_ep_moe_impl( + pg, + pgi, + low_latency_mode, + dp_size, + test_tensors, + w1_ep, + w2_ep, + w1_scale_ep, + w2_scale_ep, + config.num_experts, + use_fp8_dispatch, + per_act_token_quant, + ) + + assert_deepep_close(torch_combined, deepep_combined, config.k, use_fp8_dispatch) + + +MNKs = [ + (1, 128, 128), + (2, 128, 512), + (3, 1024, 2048), + (32, 128, 1024), + (45, 512, 2048), + (64, 1024, 1024), + (222, 1024, 2048), +] + +DTYPES = [torch.bfloat16, current_platform.fp8_dtype()] + + +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("m,n,k", MNKs) +@pytest.mark.parametrize("num_experts", [32]) +@pytest.mark.parametrize("topk", [6]) +@pytest.mark.parametrize("world_dp_size", [(2, 1)]) +@pytest.mark.parametrize("per_act_token_quant", [False, True]) +@multi_gpu_test(num_gpus=2) +@requires_deep_ep +def test_deep_ep_moe( + dtype: torch.dtype, + m: int, + n: int, + k: int, + num_experts: int, + topk: int, + world_dp_size: tuple[int, int], + per_act_token_quant: bool, + workspace_init, +): + low_latency_mode = False + use_fp8_dispatch = False + + set_random_seed(7) + world_size, dp_size = world_dp_size + config = TestConfig(dtype=dtype, topk=topk, m=m, k=k, n=n, num_experts=num_experts) + + w1, w2, w1_scale, w2_scale = make_weights(num_experts, n, k, dtype) + + parallel_launch( + world_size, + _deep_ep_moe, + low_latency_mode, + dp_size, + config, + w1, + w2, + w1_scale, + w2_scale, + use_fp8_dispatch, + per_act_token_quant, + ) + + +MNKs = [ + (1, 128, 2560), + (2, 128, 2560), + (3, 1024, 2560), + (32, 128, 2560), + (45, 512, 2560), + (64, 1024, 2560), + (222, 1024, 2560), +] +DTYPES = [current_platform.fp8_dtype(), torch.bfloat16] +USE_FP8_DISPATCH = [True, False] + + +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("m,n,k", MNKs) +@pytest.mark.parametrize("num_experts", [32]) +@pytest.mark.parametrize("topk", [6]) +@pytest.mark.parametrize("world_dp_size", [(2, 1)]) +@pytest.mark.parametrize("use_fp8_dispatch", USE_FP8_DISPATCH) +@multi_gpu_test(num_gpus=2) +@requires_deep_ep +def test_low_latency_deep_ep_moe( + dtype: torch.dtype, + m: int, + n: int, + k: int, + num_experts: int, + topk: int, + world_dp_size: tuple[int, int], + use_fp8_dispatch: bool, + workspace_init, +): + low_latency_mode = True + + if low_latency_mode and k not in DeepEPLLPrepareAndFinalize.SUPPORTED_HIDDEN_SIZES: + pytest.skip( + f"Skipping test as hidden size {k} is not in list of supported " + f"hidden sizes {DeepEPLLPrepareAndFinalize.SUPPORTED_HIDDEN_SIZES}" + ) + + set_random_seed(7) + world_size, dp_size = world_dp_size + config = TestConfig(dtype=dtype, topk=topk, m=m, k=k, n=n, num_experts=num_experts) + + w1, w2, w1_scale, w2_scale = make_weights(num_experts, n, k, dtype) + + parallel_launch( + world_size, + _deep_ep_moe, + low_latency_mode, + dp_size, + config, + w1, + w2, + w1_scale, + w2_scale, + use_fp8_dispatch, + False, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_deepep_v2_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_deepep_v2_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..03c8e3c5e4eb15ed97cb028131a7694070922839 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_deepep_v2_moe.py @@ -0,0 +1,542 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Test DeepEP v2 (ElasticBuffer) dispatch-combine logic. +Compares against a pure-PyTorch reference MoE implementation. +""" + +import dataclasses + +import pytest +import torch.distributed +from torch.distributed import ProcessGroup + +from tests.kernels.moe.utils import make_dummy_moe_config, make_test_weights +from tests.kernels.utils import torch_experts +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.model_executor.layers.fused_moe import TritonExperts +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.config import ( + FusedMoEQuantConfig, +) +from vllm.model_executor.layers.fused_moe.modular_kernel import FusedMoEKernel +from vllm.utils.import_utils import has_deep_ep_v2 +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.worker.workspace import init_workspace_manager + +from ...utils import multi_gpu_test +from .parallel_utils import ProcessGroupInfo, parallel_launch + +if has_deep_ep_v2(): + from .parallel_utils import DeepEPV2Args, make_deepep_v2_a2a + +requires_deep_ep_v2 = pytest.mark.skipif( + not has_deep_ep_v2(), + reason="Requires DeepEP v2 (ElasticBuffer)", +) + + +@dataclasses.dataclass +class TestConfig: + dtype: torch.dtype + topk: int + m: int + k: int + n: int + num_experts: int + + +@dataclasses.dataclass +class TestTensors: + rank_tokens: torch.Tensor + rank_token_scales: torch.Tensor | None + topk: torch.Tensor + topk_weights: torch.Tensor + config: TestConfig + + @staticmethod + def make(config: TestConfig) -> "TestTensors": + assert config.dtype in [torch.bfloat16, torch.float8_e4m3fn] + token_dtype = ( + torch.bfloat16 if config.dtype == torch.float8_e4m3fn else config.dtype + ) + rank_tokens = ( + torch.randn((config.m, config.k), device="cuda", dtype=token_dtype) / 10 + ) + + topk = torch.stack( + [ + torch.randperm(config.num_experts, device="cuda")[: config.topk] + for _ in range(config.m) + ] + ).to(dtype=torch.int64) + topk_weights = torch.randn(topk.shape, dtype=torch.float32, device="cuda") + return TestTensors( + rank_tokens=rank_tokens, + rank_token_scales=None, + topk=topk, + topk_weights=topk_weights, + config=config, + ) + + +def make_modular_kernel( + pg: ProcessGroup, + pgi: ProcessGroupInfo, + dp_size: int, + hidden_size: int, + num_experts: int, + num_local_experts: int, + topk: int, + q_dtype: torch.dtype | None, + use_fp8_dispatch: bool, + quant_config: FusedMoEQuantConfig, + use_cudagraph: bool = False, +) -> FusedMoEKernel: + v2_args = DeepEPV2Args( + num_local_experts=num_local_experts, + num_experts=num_experts, + num_topk=topk, + hidden_size=hidden_size, + max_tokens_per_rank=8192, + use_fp8_dispatch=use_fp8_dispatch, + ) + + a2a = make_deepep_v2_a2a( + pg=pg, + pgi=pgi, + dp_size=dp_size, + v2_args=v2_args, + use_cudagraph=use_cudagraph, + ) + + moe_config = make_dummy_moe_config( + num_experts=num_local_experts, + experts_per_token=topk, + hidden_dim=hidden_size, + ) + + fused_experts = TritonExperts( + moe_config=moe_config, + quant_config=quant_config, + ) + + mk = FusedMoEKernel( + prepare_finalize=a2a, + fused_experts=fused_experts, + inplace=False, + ) + return mk + + +def deepep_v2_moe_impl( + pg: ProcessGroup, + pgi: ProcessGroupInfo, + dp_size: int, + test_tensors: TestTensors, + w1: torch.Tensor, + w2: torch.Tensor, + w1_scale: torch.Tensor | None, + w2_scale: torch.Tensor | None, + num_experts: int, + topk: int, + use_fp8_dispatch: bool, + per_act_token_quant: bool, +) -> torch.Tensor: + num_local_experts = w1.size(0) + + def build_expert_map(): + expert_map = torch.full((num_experts,), fill_value=-1, dtype=torch.int32) + s = pgi.rank * num_local_experts + e = s + num_local_experts + expert_map[s:e] = torch.tensor(list(range(num_local_experts))) + device = torch.accelerator.current_device_index() + return expert_map.to(device=device, dtype=torch.int32) + + is_quantized = w1.dtype == torch.float8_e4m3fn + q_dtype = torch.float8_e4m3fn if is_quantized else None + + quant_config = FusedMoEQuantConfig.make( + q_dtype, + w1_scale=w1_scale, + w2_scale=w2_scale, + per_act_token_quant=per_act_token_quant, + a1_scale=test_tensors.rank_token_scales, + ) + + hidden_size = test_tensors.rank_tokens.size(1) + + mk: FusedMoEKernel = make_modular_kernel( + pg, + pgi, + dp_size, + hidden_size, + num_experts, + num_local_experts, + topk, + q_dtype, + use_fp8_dispatch, + quant_config, + ) + + out = mk.apply( + hidden_states=test_tensors.rank_tokens, + w1=w1, + w2=w2, + topk_weights=test_tensors.topk_weights, + topk_ids=test_tensors.topk, + activation=MoEActivation.SILU, + global_num_experts=num_experts, + expert_map=build_expert_map(), + apply_router_weight_on_input=False, + ) + + return out + + +def _deep_ep_v2_moe( + pgi: ProcessGroupInfo, + dp_size: int, + config: TestConfig, + w1: torch.Tensor, + w2: torch.Tensor, + w1_scale: torch.Tensor | None, + w2_scale: torch.Tensor | None, + use_fp8_dispatch: bool, + per_act_token_quant: bool, +): + device = torch.device(f"cuda:{pgi.local_rank}") + init_workspace_manager(device) + + is_quantized = w1.dtype == torch.float8_e4m3fn + device_idx = torch.accelerator.current_device_index() + w1 = w1.to(device=device_idx) + w2 = w2.to(device=device_idx) + if is_quantized: + assert w1_scale is not None and w2_scale is not None + w1_scale = w1_scale.to(device=device_idx) + w2_scale = w2_scale.to(device=device_idx) + + pg = torch.distributed.new_group(list(range(pgi.world_size))) + test_tensors = TestTensors.make(config) + + with set_current_vllm_config(VllmConfig()): + # Reference + q_dtype = torch.float8_e4m3fn if is_quantized else None + torch_combined = torch_experts( + test_tensors.rank_tokens, + w1, + w2, + test_tensors.topk_weights, + test_tensors.topk, + w1_scale=w1_scale, + w2_scale=w2_scale, + quant_dtype=q_dtype, + per_act_token_quant=per_act_token_quant, + ) + + # Splice experts for this rank + num_local_experts = config.num_experts // pgi.world_size + e_start = num_local_experts * pgi.rank + e_end = e_start + num_local_experts + w1_ep = w1[e_start:e_end] + w2_ep = w2[e_start:e_end] + + w1_scale_ep, w2_scale_ep = None, None + if is_quantized: + w1_scale_ep = w1_scale[e_start:e_end] # type: ignore + w2_scale_ep = w2_scale[e_start:e_end] # type: ignore + + deepep_combined = deepep_v2_moe_impl( + pg, + pgi, + dp_size, + test_tensors, + w1_ep, + w2_ep, + w1_scale_ep, + w2_scale_ep, + config.num_experts, + config.topk, + use_fp8_dispatch, + per_act_token_quant, + ) + + torch.testing.assert_close( + torch_combined, + deepep_combined, + atol=6e-2, + rtol=6e-2, + ) + + +MNKs = [ + (1, 256, 256), + (2, 256, 512), + (3, 1024, 2048), + (32, 256, 1024), + (45, 512, 2048), + (64, 1024, 1024), + (222, 1024, 2048), +] + +DTYPES = [torch.bfloat16, torch.float8_e4m3fn] + + +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("m,n,k", MNKs) +@pytest.mark.parametrize("num_experts", [32]) +@pytest.mark.parametrize("topk", [6]) +@pytest.mark.parametrize("world_dp_size", [(2, 1)]) +@multi_gpu_test(num_gpus=2) +@requires_deep_ep_v2 +def test_deep_ep_v2_moe( + dtype: torch.dtype, + m: int, + n: int, + k: int, + num_experts: int, + topk: int, + world_dp_size: tuple[int, int], + workspace_init, +): + per_act_token_quant = False + use_fp8_dispatch = False + + set_random_seed(7) + world_size, dp_size = world_dp_size + config = TestConfig(dtype=dtype, topk=topk, m=m, k=k, n=n, num_experts=num_experts) + + quant_dtype = dtype if dtype == torch.float8_e4m3fn else None + (_, w1, w1_scale, _), (_, w2, w2_scale, _) = make_test_weights( + num_experts, + n, + k, + quant_dtype=quant_dtype, + per_out_ch_quant=True, + ) + + parallel_launch( + world_size, + _deep_ep_v2_moe, + dp_size, + config, + w1, + w2, + w1_scale, + w2_scale, + use_fp8_dispatch, + per_act_token_quant, + ) + + +def _deep_ep_v2_moe_cudagraph( + pgi: ProcessGroupInfo, + dp_size: int, + config: TestConfig, + w1: torch.Tensor, + w2: torch.Tensor, + w1_scale: torch.Tensor | None, + w2_scale: torch.Tensor | None, +): + """Worker function: verify DeepEP v2 + TrtLLM FP8 with do_expand=False.""" + import tempfile + + from vllm.distributed import ( + init_distributed_environment, + initialize_model_parallel, + ) + + device = torch.device(f"cuda:{pgi.local_rank}") + init_workspace_manager(device) + + pg = torch.distributed.new_group(list(range(pgi.world_size))) + test_tensors = TestTensors.make(config) + num_local_experts = config.num_experts // pgi.world_size + hidden_size = config.k + + # Create FP8 weights directly, then dequantize for bf16 reference. + w1_fp8 = torch.randn( + (config.num_experts, 2 * config.n, config.k), + device="cuda", + dtype=torch.bfloat16, + ).to(torch.float8_e4m3fn) + w2_fp8 = torch.randn( + (config.num_experts, config.k, config.n), + device="cuda", + dtype=torch.bfloat16, + ).to(torch.float8_e4m3fn) + w1_ref = w1_fp8.to(torch.bfloat16) + w2_ref = w2_fp8.to(torch.bfloat16) + + from vllm.config import KernelConfig + + vllm_cfg = VllmConfig() + vllm_cfg.kernel_config = KernelConfig(moe_backend="flashinfer_trtllm") + + with set_current_vllm_config(vllm_cfg): + # Initialize vLLM parallel state (needed by FusedMoE layer) + temp_file = tempfile.mktemp() + init_distributed_environment( + world_size=pgi.world_size, + rank=pgi.rank, + distributed_init_method=f"file://{temp_file}", + local_rank=pgi.local_rank, + backend="nccl", + ) + initialize_model_parallel(tensor_model_parallel_size=1) + # Reference MoE using dequantized bf16 weights + torch_combined = torch_experts( + test_tensors.rank_tokens, + w1_ref, + w2_ref, + test_tensors.topk_weights, + test_tensors.topk, + ) + + # Use the production pipeline: make_fused_moe_layer creates + # a FusedMoE layer, quantizes weights, runs + # process_weights_after_loading (TrtLLM W31 swap + BlockMajorK + # shuffle), and selects the kernel. + # Quantize weights using production helper, EP-slice, then + # convert to TrtLLM format. + from tests.kernels.moe.test_moe_layer import _quantize_fp8_halves + from vllm.model_executor.layers.fused_moe.experts.trtllm_fp8_moe import ( + TrtLlmFp8ExpertsModular, + ) + from vllm.model_executor.layers.fused_moe.oracle.fp8 import ( + Fp8MoeBackend, + convert_to_fp8_moe_kernel_format, + ) + + block_shape = [128, 128] + qw = _quantize_fp8_halves(w1_ref, w2_ref, block_shape) + + # EP-slice before format conversion + e_start = num_local_experts * pgi.rank + e_end = e_start + num_local_experts + w1_ep = qw.w13_weight[e_start:e_end] + w2_ep = qw.w2_weight[e_start:e_end] + assert qw.w13_weight_scale is not None + assert qw.w2_weight_scale is not None + w1_scale_ep = qw.w13_weight_scale[e_start:e_end] + w2_scale_ep = qw.w2_weight_scale[e_start:e_end] + + # Convert to TrtLLM format (W31 swap + BlockMajorK shuffle) + class _MockLayer: + weight_block_size = block_shape + + class moe_config: + is_act_and_mul = True + intermediate_size_per_partition = config.n + + class activation: + is_gated = True + + w1_ep, w2_ep, w1_scale_ep, w2_scale_ep = convert_to_fp8_moe_kernel_format( + fp8_backend=Fp8MoeBackend.FLASHINFER_TRTLLM, + layer=_MockLayer(), + w13=w1_ep, + w2=w2_ep, + w13_scale=w1_scale_ep, + w2_scale=w2_scale_ep, + w13_input_scale=None, + w2_input_scale=None, + ) + + # Build TrtLLM expert with correct EP params + quant_config = FusedMoEQuantConfig.make( + torch.float8_e4m3fn, + block_shape=block_shape, + w1_scale=w1_scale_ep, + w2_scale=w2_scale_ep, + ) + moe_config = make_dummy_moe_config( + num_experts=num_local_experts, + experts_per_token=config.topk, + hidden_dim=hidden_size, + intermediate_size=config.n, + ) + fused_experts = TrtLlmFp8ExpertsModular( + moe_config=moe_config, + quant_config=quant_config, + ) + + v2_args = DeepEPV2Args( + num_local_experts=num_local_experts, + num_experts=config.num_experts, + num_topk=config.topk, + hidden_size=hidden_size, + max_tokens_per_rank=8192, + use_fp8_dispatch=False, + ) + a2a = make_deepep_v2_a2a( + pg=pg, + pgi=pgi, + dp_size=dp_size, + v2_args=v2_args, + use_cudagraph=True, + ) + mk_kernel = FusedMoEKernel( + prepare_finalize=a2a, + fused_experts=fused_experts, + inplace=False, + ) + + for _ in range(3): + out = mk_kernel.apply( + hidden_states=test_tensors.rank_tokens, + w1=w1_ep, + w2=w2_ep, + topk_weights=test_tensors.topk_weights, + topk_ids=test_tensors.topk, + activation=MoEActivation.SILU, + global_num_experts=config.num_experts, + expert_map=None, + apply_router_weight_on_input=False, + ) + + torch.testing.assert_close( + torch_combined, + out, + atol=6e-2, + rtol=6e-2, + ) + + +@pytest.mark.parametrize("m,n,k", [(32, 256, 1024)]) +@pytest.mark.parametrize("num_experts", [32]) +@pytest.mark.parametrize("topk", [6]) +@pytest.mark.parametrize("world_dp_size", [(2, 1)]) +@multi_gpu_test(num_gpus=2) +@requires_deep_ep_v2 +def test_deep_ep_v2_moe_cudagraph( + m: int, + n: int, + k: int, + num_experts: int, + topk: int, + world_dp_size: tuple[int, int], + workspace_init, +): + set_random_seed(7) + world_size, dp_size = world_dp_size + config = TestConfig( + dtype=torch.float8_e4m3fn, + topk=topk, + m=m, + k=k, + n=n, + num_experts=num_experts, + ) + + parallel_launch( + world_size, + _deep_ep_v2_moe_cudagraph, + dp_size, + config, + None, # weights created inside worker + None, + None, + None, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_deepgemm.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_deepgemm.py new file mode 100644 index 0000000000000000000000000000000000000000..3eba3f5ebefcb3984ddfb2718b78f50d5a298b86 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_deepgemm.py @@ -0,0 +1,397 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Unit-test DeepGEMM FP8 and FP4 kernels (no DeepEP). +Compare DeepGEMM path against the Triton fallback inside vLLM's fused_experts. +""" + +import importlib +import math + +import pytest +import torch + +# vLLM fused-expert reference (Triton fallback + DeepGEMM option) +import vllm.model_executor.layers.fused_moe.modular_kernel as mk +from tests.kernels.moe.utils import make_dummy_moe_config +from vllm.model_executor.layers.fused_moe.activation import ( + MoEActivation, +) +from vllm.model_executor.layers.fused_moe.all2all_utils import ( + maybe_make_prepare_finalize, +) +from vllm.model_executor.layers.fused_moe.config import ( + FusedMoEQuantConfig, + FusedMoEQuantDesc, + fp8_w8a8_moe_quant_config, +) +from vllm.model_executor.layers.fused_moe.experts.triton_deep_gemm_moe import ( + TritonOrDeepGemmExperts, +) +from vllm.model_executor.layers.fused_moe.fused_moe import fused_experts +from vllm.model_executor.layers.quantization.utils.fp8_utils import ( + per_token_group_quant_fp8, +) +from vllm.utils.deep_gemm import ( + calc_diff, + is_deep_gemm_supported, + per_block_cast_to_fp8, +) + +BLOCK_SIZE = [128, 128] + + +def make_block_quant_fp8_weights( + e: int, + n: int, + k: int, + block_size: list[int], +): + """ + Generate (w1, w2) expert weights and their per-block scale tensors + in FP8 block-quantized format. + + w1 shape: (E, 2N, K) + w2 shape: (E, K, N) + """ + dtype = torch.bfloat16 + fp8_max, fp8_min = ( + torch.finfo(torch.float8_e4m3fn).max, + torch.finfo(torch.float8_e4m3fn).min, + ) + + # bf16 reference weights + w1_bf16 = torch.randn(e, 2 * n, k, device="cuda", dtype=dtype) / 10 + w2_bf16 = torch.randn(e, k, n, device="cuda", dtype=dtype) / 10 + w1_bf16.clamp_(fp8_min, fp8_max) + w2_bf16.clamp_(fp8_min, fp8_max) + + block_n, block_k = block_size + n_tiles_w1 = math.ceil((2 * n) / block_n) + k_tiles_w1 = math.ceil(k / block_k) + n_tiles_w2 = math.ceil(k / block_n) + k_tiles_w2 = math.ceil(n / block_k) + + w1 = torch.empty_like(w1_bf16, dtype=torch.float8_e4m3fn) + w2 = torch.empty_like(w2_bf16, dtype=torch.float8_e4m3fn) + w1_s = torch.empty(e, n_tiles_w1, k_tiles_w1, device="cuda", dtype=torch.float32) + w2_s = torch.empty(e, n_tiles_w2, k_tiles_w2, device="cuda", dtype=torch.float32) + + for i in range(e): + w1[i], w1_s[i] = per_block_cast_to_fp8( + w1_bf16[i], block_size=block_size, use_ue8m0=True + ) + w2[i], w2_s[i] = per_block_cast_to_fp8( + w2_bf16[i], block_size=block_size, use_ue8m0=True + ) + + return w1, w2, w1_s, w2_s + + +def run_single_case(m, n, k, topk, num_experts, block_size): + """ + Run one (M,N,K) configuration on a single GPU and assert DeepGEMM == + Triton baseline within tolerance. + """ + tokens_bf16 = ( + torch.randn(m, k, device="cuda", dtype=torch.bfloat16) + .clamp_min_(-1) + .clamp_max_(1) + ) + _, a1_scale = per_token_group_quant_fp8(tokens_bf16, block_size[1]) + + # expert weight tensors + w1, w2, w1_s, w2_s = make_block_quant_fp8_weights(num_experts, n, k, block_size) + + router_logits = torch.randn(m, num_experts, device="cuda", dtype=torch.float32) + topk_weights, topk_ids = torch.topk(router_logits, k=topk, dim=-1) + topk_weights = torch.nn.functional.softmax(topk_weights, dim=-1) + + quant_config = fp8_w8a8_moe_quant_config( + w1_scale=w1_s, + w2_scale=w2_s, + a1_scale=a1_scale, + block_shape=block_size, + ) + moe_config = make_dummy_moe_config() + + deep_gemm_experts = mk.FusedMoEKernel( + prepare_finalize=maybe_make_prepare_finalize( + moe=moe_config, + quant_config=quant_config, + allow_new_interface=True, + use_monolithic=False, + ), + fused_experts=TritonOrDeepGemmExperts( + moe_config=moe_config, + quant_config=quant_config, + ), + ) + + # triton reference + out_triton = fused_experts( + hidden_states=tokens_bf16, + w1=w1, + w2=w2, + topk_weights=topk_weights, + topk_ids=topk_ids, + quant_config=quant_config, + ) + + # DeepGemm + out_deepgemm = deep_gemm_experts.apply( + hidden_states=tokens_bf16, + w1=w1, + w2=w2, + topk_weights=topk_weights, + topk_ids=topk_ids, + global_num_experts=num_experts, + activation=MoEActivation.SILU, + apply_router_weight_on_input=False, + expert_map=None, + ) + diff = calc_diff(out_deepgemm, out_triton) + assert diff < 0.001, f"Diff exceeded 1%: {diff}" + + +# Note: N <= 512 will disable the deepgemm path due to performance issues. +MNKs = [ + (1024, 768, 128), + (2048, 768, 512), + (512, 1024, 1024), + (4096, 4096, 1024), +] + +TOPKS = [2, 6] +NUM_EXPERTS = [32] + + +@pytest.mark.parametrize(("m", "n", "k"), MNKs) +@pytest.mark.parametrize("topk", TOPKS) +@pytest.mark.parametrize("num_experts", NUM_EXPERTS) +@pytest.mark.skipif(not is_deep_gemm_supported(), reason="Requires deep_gemm kernels") +def test_deepgemm_vs_triton(m, n, k, topk, num_experts, monkeypatch, workspace_init): + with monkeypatch.context() as mp: + mp.setenv("VLLM_USE_DEEP_GEMM", "1") + + _DeepGemmExperts = importlib.import_module( + "vllm.model_executor.layers.fused_moe.experts.deep_gemm_moe" + ).DeepGemmExperts + + call_counter = {"cnt": 0} + + orig_fn = _DeepGemmExperts.apply + + def _spy_apply(*args, **kwargs): + call_counter["cnt"] += 1 + return orig_fn(*args, **kwargs) + + monkeypatch.setattr(_DeepGemmExperts, "apply", _spy_apply) + if topk > num_experts: + pytest.skip(f"topk={topk} > num_experts={num_experts}") + + run_single_case( + m=m, + n=n, + k=k, + topk=topk, + num_experts=num_experts, + block_size=BLOCK_SIZE, + ) + + # ensure that the DeepGEMM path was indeed taken. + assert call_counter["cnt"] == 1, ( + f"DeepGEMM path was not executed during the test. " + f"Call counter: {call_counter['cnt']}" + ) + + +# --------------------------------------------------------------------------- +# FP4 weight tests (DeepGEMM m_grouped_fp8_fp4_gemm_nt_contiguous) +# --------------------------------------------------------------------------- + + +def make_mxfp4_weights( + e: int, + n: int, + k: int, +): + """ + Generate (w1, w2) expert weights in MXFP4 packed format with float32 scales, + plus BF16 reference weights for validation. + + w1 shape: (E, 2N, K//2) uint8 — packed FP4 + w2 shape: (E, K, N//2) uint8 — packed FP4 + w1_s shape: (E, 2N, K//32) float32 — per-row block-32 scales + w2_s shape: (E, K, N//32) float32 — per-row block-32 scales + w1_bf16: (E, 2N, K) — original BF16 for reference + w2_bf16: (E, K, N) — original BF16 for reference + """ + from deep_gemm.utils.math import per_token_cast_to_fp4 + + dtype = torch.bfloat16 + gran_k = 32 # MXFP4 block size + + # bf16 reference weights — scale by 1/sqrt(dim) for numerical stability + w1_bf16 = torch.randn(e, 2 * n, k, device="cuda", dtype=dtype) * (k**-0.5) + w2_bf16 = torch.randn(e, k, n, device="cuda", dtype=dtype) * (n**-0.5) + + # Quantize per-expert to FP4 + w1 = torch.empty(e, 2 * n, k // 2, device="cuda", dtype=torch.uint8) + w2 = torch.empty(e, k, n // 2, device="cuda", dtype=torch.uint8) + w1_s = torch.empty( + e, 2 * n, math.ceil(k / gran_k), device="cuda", dtype=torch.float32 + ) + w2_s = torch.empty(e, k, math.ceil(n / gran_k), device="cuda", dtype=torch.float32) + + for i in range(e): + w1[i], w1_s[i] = per_token_cast_to_fp4( + w1_bf16[i].float(), use_ue8m0=True, gran_k=gran_k + ) + w2[i], w2_s[i] = per_token_cast_to_fp4( + w2_bf16[i].float(), use_ue8m0=True, gran_k=gran_k + ) + + return w1, w2, w1_s, w2_s, w1_bf16, w2_bf16 + + +def _bf16_moe_reference(x, w1, w2, topk_weights, topk_ids): + """BF16 token-loop MoE reference for correctness testing.""" + import torch.nn.functional as F + + num_tokens, hidden_size = x.shape + intermediate = w1.shape[1] // 2 + top_k = topk_ids.shape[1] + + output = torch.zeros(num_tokens, hidden_size, dtype=torch.float32, device=x.device) + for t in range(num_tokens): + for kk in range(top_k): + e = topk_ids[t, kk].item() + w = topk_weights[t, kk].item() + fc1 = x[t : t + 1].float() @ w1[e].float().T + linear = fc1[:, :intermediate] + gate = fc1[:, intermediate:] + act = F.silu(gate) * linear + fc2 = act @ w2[e].float().T + output[t] += w * fc2[0] + return output.to(torch.bfloat16) + + +def run_single_fp4_case(m, n, k, topk, num_experts): + """ + Run one (M,N,K) configuration with FP4 weights on DeepGEMM and assert + DeepGEMM FP4 == BF16 reference within tolerance. + """ + tokens_bf16 = torch.randn(m, k, device="cuda", dtype=torch.bfloat16) * (k**-0.5) + + # FP4 expert weight tensors + BF16 originals for reference + w1, w2, w1_s, w2_s, w1_bf16, w2_bf16 = make_mxfp4_weights(num_experts, n, k) + + router_logits = torch.randn(m, num_experts, device="cuda", dtype=torch.float32) + topk_weights, topk_ids = torch.topk(router_logits, k=topk, dim=-1) + topk_weights = torch.nn.functional.softmax(topk_weights, dim=-1) + + from vllm.model_executor.layers.quantization.utils.quant_utils import ( + GroupShape, + ) + from vllm.platforms import current_platform + + _fp8_dtype = current_platform.fp8_dtype() + _block_shape = GroupShape(128, 128) + quant_config = FusedMoEQuantConfig( + _a1=FusedMoEQuantDesc(_fp8_dtype, _block_shape, None, None, None, None), + _a2=FusedMoEQuantDesc(_fp8_dtype, _block_shape, None, None, None, None), + _w1=FusedMoEQuantDesc("mxfp4", None, w1_s, None, None, None), + _w2=FusedMoEQuantDesc("mxfp4", None, w2_s, None, None, None), + ) + moe_config = make_dummy_moe_config() + + from vllm.model_executor.layers.fused_moe.experts.deep_gemm_moe import ( + DeepGemmFP4Experts, + ) + + deep_gemm_fp4_experts = mk.FusedMoEKernel( + prepare_finalize=maybe_make_prepare_finalize( + moe=moe_config, + quant_config=quant_config, + allow_new_interface=True, + use_monolithic=False, + ), + fused_experts=DeepGemmFP4Experts( + moe_config=moe_config, + quant_config=quant_config, + ), + ) + + # DeepGEMM FP4 path + out_deepgemm_fp4 = deep_gemm_fp4_experts.apply( + hidden_states=tokens_bf16, + w1=w1, + w2=w2, + topk_weights=topk_weights, + topk_ids=topk_ids, + global_num_experts=num_experts, + activation=MoEActivation.SILU, + apply_router_weight_on_input=False, + expert_map=None, + ) + + # BF16 reference using the same original weights + out_ref = _bf16_moe_reference(tokens_bf16, w1_bf16, w2_bf16, topk_weights, topk_ids) + + # FP4 vs BF16 reference: quantization error from FP4 weights + FP8 activations + diff = calc_diff(out_deepgemm_fp4, out_ref) + assert diff < 0.05, f"FP4 diff exceeded 5%: {diff}" + + +# DeepSeek V4 dims: H=4096, I=2048, so N=2*I=4096, K=H=4096. +# FP4 quantization with block_k=32 needs large K for good accuracy. +FP4_MNKs = [ + (128, 4096, 4096), # DeepSeek V4 shape + (256, 2048, 2048), # Half-size variant +] + +FP4_TOPKS = [2] +FP4_NUM_EXPERTS = [8] + + +@pytest.mark.parametrize(("m", "n", "k"), FP4_MNKs) +@pytest.mark.parametrize("topk", FP4_TOPKS) +@pytest.mark.parametrize("num_experts", FP4_NUM_EXPERTS) +@pytest.mark.skipif(not is_deep_gemm_supported(), reason="Requires deep_gemm kernels") +def test_deepgemm_fp4_vs_triton( + m, n, k, topk, num_experts, monkeypatch, workspace_init +): + pytest.importorskip("deep_gemm.utils.math") + with monkeypatch.context() as mp: + mp.setenv("VLLM_USE_DEEP_GEMM", "1") + + _DeepGemmFP4Experts = importlib.import_module( + "vllm.model_executor.layers.fused_moe.experts.deep_gemm_moe" + ).DeepGemmFP4Experts + + call_counter = {"cnt": 0} + + orig_fn = _DeepGemmFP4Experts.apply + + def _spy_apply(*args, **kwargs): + call_counter["cnt"] += 1 + return orig_fn(*args, **kwargs) + + monkeypatch.setattr(_DeepGemmFP4Experts, "apply", _spy_apply) + if topk > num_experts: + pytest.skip(f"topk={topk} > num_experts={num_experts}") + + run_single_fp4_case( + m=m, + n=n, + k=k, + topk=topk, + num_experts=num_experts, + ) + + # ensure that the DeepGEMM FP4 path was indeed taken. + assert call_counter["cnt"] == 1, ( + f"DeepGEMM FP4 path was not executed during the test. " + f"Call counter: {call_counter['cnt']}" + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_flashinfer.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_flashinfer.py new file mode 100644 index 0000000000000000000000000000000000000000..f09360d00e87d7be5aae4c5415328e89c604cee3 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_flashinfer.py @@ -0,0 +1,424 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from dataclasses import dataclass + +import pytest +import torch + +import vllm.model_executor.layers.fused_moe.modular_kernel as mk +from vllm.config import ParallelConfig, VllmConfig, set_current_vllm_config +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.all2all_utils import ( + maybe_make_prepare_finalize, +) +from vllm.model_executor.layers.fused_moe.config import ( + FusedMoEConfig, + FusedMoEParallelConfig, + FusedMoEQuantConfig, + RoutingMethodType, + fp8_w8a8_moe_quant_config, +) +from vllm.model_executor.layers.fused_moe.experts.flashinfer_cutlass_moe import ( + FlashInferExperts, +) +from vllm.model_executor.layers.fused_moe.experts.trtllm_fp8_moe import ( + TrtLlmFp8ExpertsMonolithic, +) +from vllm.model_executor.layers.fused_moe.fused_moe import fused_experts +from vllm.model_executor.layers.quantization.utils.flashinfer_utils import ( + rotate_weights_for_fi_trtllm_fp8_per_tensor_moe, + swap_w13_to_w31, +) +from vllm.model_executor.layers.quantization.utils.fp8_utils import input_to_float8 +from vllm.model_executor.models.llama4 import Llama4MoE +from vllm.platforms import current_platform +from vllm.utils.math_utils import next_power_of_2 +from vllm.utils.torch_utils import set_random_seed + +try: + from vllm.utils.flashinfer import has_flashinfer_cutlass_fused_moe +except ImportError: + if current_platform.is_rocm(): + pytest.skip( + "flashinfer not supported for vLLM on ROCm", allow_module_level=True + ) + +if not has_flashinfer_cutlass_fused_moe() or not current_platform.has_device_capability( + 90 +): + pytest.skip( + "Supported for sm >= 90", + allow_module_level=True, + ) + +NUM_EXPERTS = [16] +TOP_KS = [1] + +MNK_FACTORS = [ + (256, 8192, 5120), + (127, 4096, 5120), + (10, 8192, 5120), + (10, 4096, 5120), + (1, 8192, 5120), + (1, 4096, 5120), +] + +vllm_config = VllmConfig(parallel_config=ParallelConfig(pipeline_parallel_size=1)) + + +def quant_fp8_per_tensor_batches(a): + num_batches = a.size(0) + a_quant = [] + a_scales = [] + + for i in range(num_batches): + a_fp8, a_global_sf = input_to_float8(a[i]) + if a_global_sf.numel() == 1: + a_global_sf = a_global_sf.view(1, 1) + a_quant.append(a_fp8) + a_scales.append(a_global_sf) + + result_a_quant = torch.stack(a_quant) + result_a_scales = torch.stack(a_scales) + + return result_a_quant, result_a_scales + + +def check_accuracy(ref_output, actual_output, atol=0.1, rtol=0.85, percent=0.925): + close = torch.isclose(ref_output, actual_output, atol=atol, rtol=rtol) + match_ratio = close.float().mean() + assert match_ratio >= percent, ( + f"Match ratio {match_ratio:.4f} is below the threshold {percent:.4f}" + ) + + mismatch_percent = 1.0 - match_ratio.item() + assert mismatch_percent <= 1 - percent, ( + f"Mismatch percentage {mismatch_percent:.4f} is above the threshold " + f"{1 - percent:.4f}" + ) + + +@dataclass +class TestData: + hidden_states: torch.Tensor + w13_quantized: torch.Tensor + w2_quantized: torch.Tensor + a1_scale: torch.Tensor + a2_scale: torch.Tensor + w13_weight_scale: torch.Tensor + w2_weight_scale: torch.Tensor + layer: torch.nn.Module + + @staticmethod + def make_moe_tensors_8bit( + m: int, + k: int, + n: int, + e: int, + is_trtllm: bool, + activation: MoEActivation = MoEActivation.SILU, + topk: int = 1, + ) -> "TestData": + is_gated = activation.is_gated + + hidden_states = torch.randn((m, k), device="cuda", dtype=torch.bfloat16) / 10 + w13 = ( + torch.randn( + (e, (2 * n) if is_gated else n, k), device="cuda", dtype=torch.bfloat16 + ) + / 10 + ) + w2 = torch.randn((e, k, n), device="cuda", dtype=torch.bfloat16) / 10 + + # Scale to fp8 + _, a1_scale = input_to_float8(hidden_states) + a2_scale = torch.scalar_tensor(1.0).to(device="cuda").to(dtype=torch.float32) + w13_quantized, w13_weight_scale = quant_fp8_per_tensor_batches(w13) + w2_quantized, w2_weight_scale = quant_fp8_per_tensor_batches(w2) + + layer = torch.nn.Module() + layer.orig_dtype = torch.bfloat16 + layer.w13_weight = w13_quantized.clone() + layer.w2_weight = w2_quantized.clone() + layer.w13_input_scale = a1_scale + layer.w2_input_scale = a2_scale + layer.w13_weight_scale = w13_weight_scale + layer.w2_weight_scale = w2_weight_scale + layer.activation = activation + # Setup dummy config. + layer.moe_parallel_config = mk.FusedMoEParallelConfig.make_no_parallel() + + # flashinfer expects swapped rows for w13 + if is_gated: + layer.w13_weight.data = swap_w13_to_w31(layer.w13_weight.data) + if is_trtllm: + rotate_weights_for_fi_trtllm_fp8_per_tensor_moe( + layer.w13_weight, layer.w2_weight, is_gated + ) + layer.custom_routing_function = Llama4MoE.custom_routing_function + layer.routing_method_type = RoutingMethodType.Llama4 + layer.renormalize = False + layer.intermediate_size_per_partition = n + layer.ep_rank = 0 + layer.local_num_experts = e + + layer.moe = FusedMoEConfig( + num_experts=e, + experts_per_token=topk, + hidden_dim=k, + intermediate_size=n, + num_local_experts=e, + num_logical_experts=e, + moe_parallel_config=layer.moe_parallel_config, + in_dtype=hidden_states.dtype, + routing_method=layer.routing_method_type, + activation=activation, + device=w13_quantized.device, + max_num_tokens=next_power_of_2(m), + ) + + return TestData( + hidden_states=hidden_states, + w13_quantized=w13_quantized, + w2_quantized=w2_quantized, + a1_scale=a1_scale, + a2_scale=a2_scale, + w13_weight_scale=w13_weight_scale, + w2_weight_scale=w2_weight_scale, + layer=layer, + ) + + +@pytest.mark.parametrize("m,n,k", MNK_FACTORS) +@pytest.mark.parametrize("e", NUM_EXPERTS) +@pytest.mark.parametrize("topk", TOP_KS) +@pytest.mark.parametrize("activation", [MoEActivation.SILU, MoEActivation.RELU2_NO_MUL]) +def test_flashinfer_per_tensor_moe_fp8_no_graph( + m: int, + n: int, + k: int, + e: int, + topk: int, + activation: MoEActivation, + monkeypatch, +): + if not current_platform.has_device_capability(100): + pytest.skip("Test is only supported for sm >= 100") + set_random_seed(7) + with set_current_vllm_config(vllm_config): + td = TestData.make_moe_tensors_8bit( + m, k, n, e, is_trtllm=True, activation=activation + ) + + score = torch.randn((m, e), device="cuda", dtype=torch.bfloat16) + topk_weights, topk_ids = Llama4MoE.custom_routing_function( + hidden_states=td.hidden_states, + gating_output=score, + topk=topk, + renormalize=False, + ) + + quant_config = fp8_w8a8_moe_quant_config( + w1_scale=td.w13_weight_scale, + w2_scale=td.w2_weight_scale, + a1_scale=td.a1_scale, + a2_scale=td.a2_scale, + per_act_token_quant=False, + ) + + output = fused_experts( + td.hidden_states, + td.w13_quantized, + td.w2_quantized, + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=activation, + global_num_experts=e, + expert_map=None, + apply_router_weight_on_input=True, + quant_config=quant_config, + ) + + kernel = mk.FusedMoEKernel( + maybe_make_prepare_finalize( + moe=td.layer.moe, + quant_config=quant_config, + allow_new_interface=True, + use_monolithic=True, + ), + TrtLlmFp8ExpertsMonolithic( + moe_config=td.layer.moe, + quant_config=quant_config, + ), + ) + + flashinfer_output = kernel.apply_monolithic( + hidden_states=td.hidden_states, + w1=td.layer.w13_weight, + w2=td.layer.w2_weight, + router_logits=score, + activation=activation, + global_num_experts=e, + expert_map=None, + apply_router_weight_on_input=True, + routed_scaling_factor=1.0, + ) + + check_accuracy( + ref_output=output, + actual_output=flashinfer_output, + atol=0.1, + rtol=0.85, + percent=0.925, + ) + + +@pytest.mark.parametrize("m,n,k", MNK_FACTORS) +@pytest.mark.parametrize("e", NUM_EXPERTS) +@pytest.mark.parametrize("topk", TOP_KS) +@pytest.mark.parametrize("activation", [MoEActivation.SILU, MoEActivation.RELU2_NO_MUL]) +def test_flashinfer_cutlass_moe_fp8_no_graph( + m: int, + n: int, + k: int, + e: int, + topk: int, + activation: MoEActivation, + monkeypatch, + workspace_init, +): + set_random_seed(7) + with set_current_vllm_config(vllm_config): + td = TestData.make_moe_tensors_8bit( + m, k, n, e, is_trtllm=False, activation=activation + ) + + score = torch.randn((m, e), device="cuda", dtype=torch.bfloat16) + topk_weights, topk_ids = Llama4MoE.custom_routing_function( + hidden_states=td.hidden_states, + gating_output=score, + topk=topk, + renormalize=False, + ) + + quant_config = fp8_w8a8_moe_quant_config( + w1_scale=td.w13_weight_scale, + g1_alphas=(td.w13_weight_scale * td.a1_scale).squeeze(), + w2_scale=td.w2_weight_scale, + g2_alphas=(td.w2_weight_scale * td.a2_scale).squeeze(), + a1_scale=td.a1_scale, + a1_gscale=td.a1_scale, + a2_scale=td.a2_scale, + a2_gscale=1.0 / td.a2_scale, + per_act_token_quant=False, + ) + + output = fused_experts( + td.hidden_states, + td.w13_quantized, + td.w2_quantized, + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=activation, + global_num_experts=e, + expert_map=None, + apply_router_weight_on_input=True, + quant_config=quant_config, + ) + + td.layer.dp_size = 1 + + def get_fused_moe_quant_config(n: torch.nn.Module) -> FusedMoEQuantConfig: + return quant_config + + td.layer.get_fused_moe_quant_config = get_fused_moe_quant_config + td.layer.quant_method = td.layer + + moe_config = FusedMoEConfig( + num_experts=e, + experts_per_token=topk, + hidden_dim=k, + intermediate_size=n, + num_local_experts=e, + num_logical_experts=e, + activation=activation, + device="cuda", + moe_parallel_config=FusedMoEParallelConfig.make_no_parallel(), + in_dtype=torch.bfloat16, + routing_method=RoutingMethodType.TopK, + max_num_tokens=next_power_of_2(m), + ) + + kernel = mk.FusedMoEKernel( + maybe_make_prepare_finalize( + moe=moe_config, + quant_config=quant_config, + allow_new_interface=True, + use_monolithic=False, + ), + FlashInferExperts( + moe_config=moe_config, + quant_config=quant_config, + ), + ) + + flashinfer_cutlass_output = kernel.apply( + td.hidden_states, + td.layer.w13_weight, + td.layer.w2_weight, + topk_weights, + topk_ids, + activation=activation, + global_num_experts=e, + expert_map=None, + apply_router_weight_on_input=True, + ) + + check_accuracy( + ref_output=output, + actual_output=flashinfer_cutlass_output, + atol=0.1, + rtol=0.85, + percent=0.925, + ) + + +@pytest.mark.parametrize( + "num_experts,intermediate,hidden", + [ + (8, 2048, 1536), + (64, 4096, 4096), + ], +) +def test_convert_moe_weights_to_flashinfer_trtllm_block_layout( + num_experts, intermediate, hidden +): + from vllm.model_executor.layers.quantization.utils.flashinfer_utils import ( + convert_moe_weights_to_flashinfer_trtllm_block_layout, + ) + + w13 = torch.randn( + (num_experts, 2 * intermediate, hidden), dtype=torch.bfloat16, device="cuda" + ) + w2 = torch.randn( + (num_experts, hidden, intermediate), dtype=torch.bfloat16, device="cuda" + ) + + cache: dict[torch.Size, torch.Tensor] = {} + w13_converted, w2_converted = convert_moe_weights_to_flashinfer_trtllm_block_layout( + cache, w13, w2 + ) + + assert w13_converted.ndim == 4, ( + f"Expected 4D tensor, got shape {w13_converted.shape}" + ) + assert w2_converted.ndim == 4, f"Expected 4D tensor, got shape {w2_converted.shape}" + + assert w13_converted.numel() == w13.numel(), "W13 element count should be preserved" + assert w2_converted.numel() == w2.numel(), "W2 element count should be preserved" + + assert w13_converted.dtype == torch.bfloat16 + assert w2_converted.dtype == torch.bfloat16 + + assert w13_converted.shape[0] == num_experts + assert w2_converted.shape[0] == num_experts diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_flashinfer_b12x_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_flashinfer_b12x_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..63c55007d714728447813d5ba5792acd43cab911 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_flashinfer_b12x_moe.py @@ -0,0 +1,228 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm.platforms import current_platform + +if not current_platform.is_device_capability_family(120): + pytest.skip( + reason="FlashInfer CuteDSL SM12x MoE requires SM120 " + "(RTX Pro 6000 / DGX Spark).", + allow_module_level=True, + ) + +from vllm.utils.flashinfer import has_flashinfer_b12x_moe + +if not has_flashinfer_b12x_moe(): + pytest.skip( + reason=( + "FlashInfer cute_dsl_fused_moe_nvfp4 / convert_sf_to_mma_layout " + "not available in installed FlashInfer (needs PRs #3051 and #3066)." + ), + allow_module_level=True, + ) + +# Import fp4_quantize after the skip guard — FlashInfer must be installed. +from flashinfer.fp4_quantization import fp4_quantize + +import vllm.model_executor.layers.fused_moe.modular_kernel as mk +from tests.kernels.moe.utils import make_dummy_moe_config +from tests.kernels.utils import torch_moe +from vllm.config import ParallelConfig, VllmConfig, set_current_vllm_config +from vllm.model_executor.layers.fused_moe import fused_topk +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.all2all_utils import ( + maybe_make_prepare_finalize, +) +from vllm.model_executor.layers.fused_moe.config import nvfp4_moe_quant_config +from vllm.model_executor.layers.fused_moe.experts.flashinfer_b12x_moe import ( + FlashInferB12xExperts, +) +from vllm.utils.flashinfer import flashinfer_convert_sf_to_mma_layout +from vllm.utils.torch_utils import set_random_seed + +# Dimensions chosen to satisfy FP4 alignment requirements (k multiple of 256, +# n multiple of 128) while keeping tests fast. +MNK_FACTORS = [ + (2, 128, 256), + (2, 256, 512), + (16, 128, 256), + (64, 256, 512), +] + + +def _reorder_gate_up_to_up_gate( + w: torch.Tensor, + w_s: torch.Tensor, +) -> tuple[torch.Tensor, torch.Tensor]: + """Swap gate and up-projection halves along dim=1 to [up, gate] order. + + The SM12x kernel expects weights in [up (w3), gate (w1)] order while the + BF16 reference uses [gate (w1), up (w3)]. This replicates the reordering + done at model-load time by ``prepare_nvfp4_moe_layer_for_fi_or_cutlass``. + """ + n = w.shape[1] // 2 + return ( + torch.cat([w[:, n:, :], w[:, :n, :]], dim=1), + torch.cat([w_s[:, n:, :], w_s[:, :n, :]], dim=1), + ) + + +@pytest.mark.parametrize("m,n,k", MNK_FACTORS) +@pytest.mark.parametrize("e", [8, 16]) +@pytest.mark.parametrize("topk", [1, 2, 4]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@torch.inference_mode() +def test_flashinfer_b12x_moe( + m: int, + n: int, + k: int, + e: int, + topk: int, + dtype: torch.dtype, + workspace_init, +): + """Test FlashInferB12xExperts against a BF16 torch reference. + + The SM12x kernel takes BF16 hidden states directly and fuses token + dispatch, W1 GEMM, SwiGLU, and W2 GEMM into one call. We verify + correctness against ``torch_moe`` using generous tolerances to account + for the internal FP4 quantization of activations and weights. + + Scale convention + ---------------- + The SM12x kernel uses ``w1_alpha`` as *both* the activation-quantisation + global scale and the weight dequantisation factor. These two roles are + conflated into a single parameter in ``launch_sm120_moe``, so they must + equal the same value. We use ``global_scale = 1.0`` for + ``fp4_quantize`` so that ``w1_alpha = ones`` satisfies both roles + simultaneously. The alternative — vLLM's convention of baking a large + ``w_gs`` into block-scale values and compensating with + ``g1_alphas = 1/w_gs`` — is incompatible with this kernel. + """ + set_random_seed(7) + with set_current_vllm_config( + VllmConfig(parallel_config=ParallelConfig(pipeline_parallel_size=1)) + ): + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + + # Generate BF16 reference weights in [gate, up] order. + # Shape: w1=(e, 2n, k), w2=(e, k, n). + w1_bf16 = torch.randn((e, 2 * n, k), device="cuda", dtype=dtype) / 15 + w2_bf16 = torch.randn((e, k, n), device="cuda", dtype=dtype) / 15 + + # ------------------------------------------------------------------ # + # Quantise weights for the SM12x kernel using FlashInfer's convention: + # global_scale = 1.0 → block_scale = max_abs_block / fp4_max + # w1_alpha = 1.0 (no extra global factor to compensate) + # + # The scale factors returned by fp4_quantize(..., is_sf_swizzled_layout=True) + # are already in the swizzled 2D layout expected by convert_sf_to_mma_layout. + # No additional swizzle_blockscale() call is needed. + # ------------------------------------------------------------------ # + gs = torch.ones(1, device="cuda", dtype=torch.float32) + sf_vec_size = 16 + + # W1: reorder BF16 from [gate, up] → [up, gate], then quantise. + w1_reordered = torch.cat( + [w1_bf16[:, n:, :], w1_bf16[:, :n, :]], dim=1 + ) # shape (e, 2n, k), [up, gate] + w1_flat = w1_reordered.reshape(e * 2 * n, k) + w1_q_flat, w1_sf_flat = fp4_quantize( + w1_flat, + global_scale=gs, + sf_vec_size=sf_vec_size, + is_sf_swizzled_layout=True, + ) + w1_q = w1_q_flat.view(e, 2 * n, k // 2) # uint8, packed FP4 + w1_blockscale = w1_sf_flat.view(e, 2 * n, w1_sf_flat.shape[1]) # float8 + + # W2: no row reordering needed for the down-projection. + w2_flat = w2_bf16.reshape(e * k, n) + w2_q_flat, w2_sf_flat = fp4_quantize( + w2_flat, + global_scale=gs, + sf_vec_size=sf_vec_size, + is_sf_swizzled_layout=True, + ) + w2_q = w2_q_flat.view(e, k, n // 2) # uint8, packed FP4 + w2_blockscale = w2_sf_flat.view(e, k, w2_sf_flat.shape[1]) # float8 + + # All per-expert alphas are 1.0 (global_scale = 1.0, no compensation). + ones_e = torch.ones(e, device="cuda", dtype=torch.float32) + + quant_config = nvfp4_moe_quant_config( + g1_alphas=ones_e, + g2_alphas=ones_e, + a1_gscale=ones_e, + a2_gscale=ones_e, + w1_scale=w1_blockscale, + w2_scale=w2_blockscale, + ) + + moe_config = make_dummy_moe_config( + num_experts=e, + experts_per_token=topk, + hidden_dim=k, + intermediate_size=n, + in_dtype=dtype, + ) + + experts = FlashInferB12xExperts( + moe_config=moe_config, + quant_config=quant_config, + ) + # In production, process_weights_after_loading computes these after + # normalizing block scales. In the test the scales are already in final + # form (global_scale=1.0), so we compute the MMA layouts directly. + num_experts_w1, m1, k1_sf = w1_blockscale.shape + experts.w1_sf_mma = flashinfer_convert_sf_to_mma_layout( + w1_blockscale.reshape(num_experts_w1 * m1, k1_sf), + m=m1, + k=k1_sf * 16, + num_groups=num_experts_w1, + ) + num_experts_w2, m2, k2_sf = w2_blockscale.shape + experts.w2_sf_mma = flashinfer_convert_sf_to_mma_layout( + w2_blockscale.reshape(num_experts_w2 * m2, k2_sf), + m=m2, + k=k2_sf * 16, + num_groups=num_experts_w2, + ) + + kernel = mk.FusedMoEKernel( + maybe_make_prepare_finalize( + moe=moe_config, + quant_config=quant_config, + allow_new_interface=True, + use_monolithic=False, + ), + experts, + ) + + score = torch.randn((m, e), device="cuda", dtype=dtype) + topk_weights, topk_ids, _ = fused_topk(a, score, topk, renormalize=False) + + sm12x_output = kernel.apply( + hidden_states=a, + w1=w1_q, + w2=w2_q, + topk_weights=topk_weights, + topk_ids=topk_ids, + global_num_experts=e, + activation=MoEActivation.SILU, + apply_router_weight_on_input=False, + expert_map=None, + ) + + # Reference: BF16 torch MoE using original [gate, up] BF16 weights. + # torch_moe's SiluAndMul expects [gate, up] order, matching w1_bf16. + torch_output = torch_moe(a, w1_bf16, w2_bf16, score, topk) + + torch.testing.assert_close(sm12x_output, torch_output, atol=2e-1, rtol=2e-1) + + +if __name__ == "__main__": + test_flashinfer_b12x_moe(16, 128, 256, 8, 2, torch.bfloat16) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_flashinfer_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_flashinfer_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..41e8b8b7ad59f5da017a7c8eab27b8667d11e539 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_flashinfer_moe.py @@ -0,0 +1,181 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch + +from tests.kernels.moe.utils import make_test_quant_config +from tests.kernels.quantization.nvfp4_utils import ( + FLOAT4_E2M1_MAX, + FLOAT8_E4M3_MAX, + dequantize_nvfp4_to_dtype, +) +from tests.kernels.utils import torch_moe +from vllm import _custom_ops as ops +from vllm.config import ParallelConfig, VllmConfig, set_current_vllm_config +from vllm.model_executor.layers.fused_moe import fused_topk +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.all2all_utils import ( + maybe_make_prepare_finalize, +) +from vllm.model_executor.layers.fused_moe.config import ( + FusedMoEConfig, + FusedMoEParallelConfig, + RoutingMethodType, +) +from vllm.model_executor.layers.fused_moe.experts.flashinfer_cutlass_moe import ( + FlashInferExperts, + is_valid_flashinfer_cutlass_fused_moe, +) +from vllm.model_executor.layers.fused_moe.modular_kernel import FusedMoEKernel +from vllm.platforms import current_platform +from vllm.utils.flashinfer import has_flashinfer_cutlass_fused_moe +from vllm.utils.math_utils import next_power_of_2 +from vllm.utils.torch_utils import set_random_seed + +if not has_flashinfer_cutlass_fused_moe() or not current_platform.has_device_capability( + 100 +): + pytest.skip( + "Requires flashinfer_cutlass_fused_moe and nvfp4 support", + allow_module_level=True, + ) + +MNK_FACTORS = [ + (2, 1024, 1024), + (2, 3072, 1024), + (2, 3072, 1536), + (64, 1024, 1536), + (64, 3072, 1024), + (64, 2048, 1536), + (224, 1024, 1024), + (224, 1024, 1536), +] + + +@pytest.mark.parametrize("m,n,k", MNK_FACTORS) +@pytest.mark.parametrize("e", [40, 64, 256]) +@pytest.mark.parametrize("topk", [1, 6, 8]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("activation", [MoEActivation.SILU, MoEActivation.RELU2_NO_MUL]) +@torch.inference_mode() +def test_flashinfer_fp4_moe_no_graph( + m: int, + n: int, + k: int, + e: int, + topk: int, + dtype: torch.dtype, + activation: MoEActivation, + workspace_init, +): + set_random_seed(7) + with set_current_vllm_config( + VllmConfig(parallel_config=ParallelConfig(pipeline_parallel_size=1)) + ): + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + + quant_blocksize = 16 + is_gated_act = activation.is_gated + + w1_q, w2_q, quant_config = make_test_quant_config( + e, + n, + k, + in_dtype=dtype, + quant_dtype="nvfp4", + block_shape=None, + per_act_token_quant=False, + make_gate=is_gated_act, + ) + + score = torch.randn((m, e), device="cuda", dtype=dtype) + topk_weights, topk_ids, _ = fused_topk(a, score, topk, renormalize=False) + + assert is_valid_flashinfer_cutlass_fused_moe(a, w1_q, w2_q) + + moe_config = FusedMoEConfig( + num_experts=e, + experts_per_token=topk, + hidden_dim=k, + intermediate_size=n, + num_local_experts=e, + num_logical_experts=e, + activation=activation, + device="cuda", + moe_parallel_config=FusedMoEParallelConfig.make_no_parallel(), + in_dtype=dtype, + routing_method=RoutingMethodType.TopK, + max_num_tokens=next_power_of_2(m), + ) + + flashinfer_experts = FusedMoEKernel( + maybe_make_prepare_finalize( + moe=moe_config, + quant_config=quant_config, + allow_new_interface=True, + use_monolithic=False, + ), + FlashInferExperts(moe_config=moe_config, quant_config=quant_config), + ) + + flashinfer_output = flashinfer_experts.apply( + hidden_states=a, + w1=w1_q, + w2=w2_q, + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=activation, + global_num_experts=e, + expert_map=None, + apply_router_weight_on_input=False, + ) + + # Reference check: + a_global_scale = ( + (FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX) / torch.amax(a.flatten(), dim=-1) + ).to(torch.float32) + a_fp4, a_scale_interleaved = ops.scaled_fp4_quant(a, a_global_scale) + _, m_k = a_fp4.shape + a_in_dtype = dequantize_nvfp4_to_dtype( + a_fp4, + a_scale_interleaved, + a_global_scale, + dtype=a.dtype, + device=a.device, + block_size=quant_blocksize, + ) + + w1_d = torch.empty( + (e, (2 if is_gated_act else 1) * n, k), device="cuda", dtype=dtype + ) + w2_d = torch.empty((e, k, n), device="cuda", dtype=dtype) + + for idx in range(0, e): + w1_d[idx] = dequantize_nvfp4_to_dtype( + w1_q[idx], + quant_config.w1_scale[idx], + (1 / quant_config.g1_alphas[idx]), + dtype=dtype, + device=w1_q.device, + block_size=quant_blocksize, + ) + w2_d[idx] = dequantize_nvfp4_to_dtype( + w2_q[idx], + quant_config.w2_scale[idx], + (1 / quant_config.g2_alphas[idx]), + dtype=dtype, + device=w2_q.device, + block_size=quant_blocksize, + ) + + torch_output = torch_moe( + a_in_dtype, w1_d, w2_d, score, topk, activation=activation + ) + + torch.testing.assert_close( + torch_output, flashinfer_output, atol=1e-1, rtol=1e-1 + ) + + +if __name__ == "__main__": + test_flashinfer_fp4_moe_no_graph((2, 1024, 1024), 40, 1, torch.half) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_flydsl_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_flydsl_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..4886aa4ea72331dcc3f8061013a7bc84bf83fc69 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_flydsl_moe.py @@ -0,0 +1,179 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# Copyright (c) 2025 FlyDSL Project Contributors + + +import importlib.util + +import pytest +import torch + +from vllm.model_executor.layers.fused_moe import fused_experts +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.config import ( + int4_w4a16_moe_quant_config, +) +from vllm.platforms import current_platform +from vllm.platforms.rocm import on_gfx950 + +if not (current_platform.is_rocm() and on_gfx950()): + pytest.skip("This test can only run on ROCm and gfx950.", allow_module_level=True) + +aiter_available = importlib.util.find_spec("aiter") is not None + +if not aiter_available: + pytest.skip("These tests require AITER to run.", allow_module_level=True) + +from vllm.model_executor.layers.fused_moe.fused_flydsl_moe import ( # noqa: E402 + fused_flydsl_moe, +) +from vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors_moe import ( # noqa: E402, E501 + compressed_tensors_moe_w4a16_flydsl, +) + +RoutingBuffers = tuple[ + torch.Tensor, # sorted_token_ids + torch.Tensor, # sorted_weights + torch.Tensor, # sorted_expert_ids + torch.Tensor, # num_valid_ids (shape [1], i32) + int, # sorted_size + int, # blocks +] + + +@pytest.mark.parametrize( + "num_tokens", [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384] +) +@pytest.mark.parametrize("inter_dim", [256, 512]) +def test_flydsl_moe(num_tokens: int, inter_dim: int): + device = "cuda" + topk = 8 + num_experts = 384 + hidden_size = 7168 + packed_factor = 8 + w13_num_shards = 2 + params_dtype = torch.bfloat16 + group_size = 32 + w2_scales_size = inter_dim + scale_factor = 0.01 + + num_groups_w2 = w2_scales_size // group_size + num_groups_w13 = hidden_size // group_size + + w13_weight = torch.randint( + 0, + 255, + (num_experts, hidden_size // packed_factor, w13_num_shards * inter_dim), + dtype=torch.int32, + device=device, + ) + + w2_weight = torch.randint( + 0, + 255, + (num_experts, inter_dim // packed_factor, hidden_size), + dtype=torch.int32, + device=device, + ) + w13_scale = scale_factor * torch.randn( + num_experts, + num_groups_w13, + w13_num_shards * inter_dim, + dtype=params_dtype, + device=device, + ) + w2_scale = scale_factor * torch.randn( + num_experts, num_groups_w2, hidden_size, dtype=params_dtype, device=device + ) + + w13_weight_packed = w13_weight.transpose(1, 2).contiguous().view(torch.uint8) + w2_weight_packed = w2_weight.transpose(1, 2).contiguous().view(torch.uint8) + w13_weight_scale = w13_scale.transpose(1, 2).contiguous() + w2_weight_scale = w2_scale.transpose(1, 2).contiguous() + + moe_quant_config = int4_w4a16_moe_quant_config( + w1_scale=w13_weight_scale, + w2_scale=w2_weight_scale, + w1_zp=None, + w2_zp=None, + block_shape=[0, group_size], + ) + score = torch.rand((num_tokens, num_experts), device=device, dtype=torch.float32) + topk_vals, topk_ids = torch.topk(score, k=topk, dim=1) + topk_weights = torch.softmax(topk_vals, dim=1).to(torch.float32) + x = torch.randn((num_tokens, hidden_size), dtype=torch.bfloat16, device=device) + out_ref = fused_experts( + x, + w13_weight_packed, + w2_weight_packed, + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=MoEActivation.SILU, + apply_router_weight_on_input=False, + global_num_experts=num_experts, + expert_map=None, + quant_config=moe_quant_config, + ) + + w13 = w13_weight + w13 = compressed_tensors_moe_w4a16_flydsl._gptq_int32_to_flydsl_packed(w13) + w13 = w13.view(-1).contiguous() + + w2 = w2_weight + w2 = compressed_tensors_moe_w4a16_flydsl._gptq_int32_to_flydsl_packed(w2) + w2 = w2.view(-1).contiguous() + + w13_scale_flydsl = w13_scale + w2_scale_flydsl = w2_scale + + if group_size > 0 and w13_scale.dim() == 3 and w13_scale.shape[1] > 1: + E, G, N = w13_scale.shape + w13_scale_flydsl = ( + w13_scale_flydsl.view(E, G // 2, 2, N) + .permute(0, 1, 3, 2) + .contiguous() + .view(-1) + .contiguous() + ) + elif w13_scale.dim() == 3 and w13_scale.shape[1] == 1: + w13_scale_flydsl = w13_scale_flydsl.squeeze(1) + + if group_size > 0 and w2_scale.dim() == 3 and w2_scale.shape[1] > 1: + E, G, N = w2_scale.shape + w2_scale_flydsl = ( + w2_scale_flydsl.view(E, G // 2, 2, N) + .permute(0, 1, 3, 2) + .contiguous() + .view(-1) + .contiguous() + ) + elif w2_scale.dim() == 3 and w2_scale.shape[1] == 1: + w2_scale_flydsl = w2_scale_flydsl.squeeze(1) + + w13_scale_flydsl = w13_scale_flydsl.contiguous() + w2_scale_flydsl = w2_scale_flydsl.contiguous() + + w13.is_shuffled = True + w2.is_shuffled = True + + out = fused_flydsl_moe( + x, + w13, + w2, + num_experts, + inter_dim, + topk_weights, + topk_ids, + w1_scale=w13_scale_flydsl, + w2_scale=w2_scale_flydsl, + topk=topk_weights.shape[-1], + group_size=group_size, + doweight_stage1=False, + scale_is_bf16=True, + ) + + assert torch.allclose(out, out_ref, atol=0.5, rtol=0.1) + + +if __name__ == "__main__": + test_flydsl_moe(512, 256) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_fused_topk.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_fused_topk.py new file mode 100644 index 0000000000000000000000000000000000000000..2589cc3de94caf5a30d6ed3ead08e3d2ddb4cb0d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_fused_topk.py @@ -0,0 +1,273 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the MoE fused topk kernel + +Run `pytest tests/kernels/moe/test_fused_topk.py`. +""" + +import pytest +import torch + +from vllm.model_executor.layers.fused_moe.router.fused_topk_bias_router import ( + fused_topk_bias, +) +from vllm.model_executor.layers.fused_moe.router.fused_topk_router import fused_topk +from vllm.platforms import current_platform + + +def torch_topk( + gating_output: torch.Tensor, + topk: int, + renormalize: bool, + e_score_correction_bias: torch.Tensor = None, + scoring_func: str = "softmax", +): + if scoring_func == "softmax": + scores = torch.softmax(gating_output.float(), dim=-1) + else: + assert scoring_func == "sigmoid" + scores = torch.sigmoid(gating_output.float()) + + if e_score_correction_bias is not None: + num_experts = gating_output.shape[-1] + scores_for_choice = scores.view( + -1, num_experts + ) + e_score_correction_bias.unsqueeze(0) + _, topk_ids = torch.topk(scores_for_choice, k=topk, dim=-1) + topk_weights = scores.gather(1, topk_ids) + else: + topk_weights, topk_ids = torch.topk(scores, k=topk, dim=-1) + + if renormalize: + topk_weights = topk_weights / topk_weights.sum(dim=-1, keepdim=True) + + return topk_weights, topk_ids + + +@pytest.mark.skipif( + not current_platform.is_cuda(), reason="This test is skipped on non-CUDA platform." +) +@pytest.mark.parametrize("num_tokens", [1, 33, 56]) +@pytest.mark.parametrize("hidden_size", [1024, 2048]) +@pytest.mark.parametrize("num_experts", [6, 16]) +@pytest.mark.parametrize("topk", [3, 4]) +@pytest.mark.parametrize("renormalize", [True, False]) +@pytest.mark.parametrize("scoring_func", ["softmax", "sigmoid"]) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.half, torch.float32]) +def test_fused_topk( + num_tokens: int, + hidden_size: int, + num_experts: int, + topk: int, + renormalize: bool, + scoring_func: str, + dtype: torch.dtype, +): + torch.manual_seed(0) + hidden_states = torch.randn((num_tokens, hidden_size), dtype=dtype, device="cuda") + gating_output = torch.randn((num_tokens, num_experts), dtype=dtype, device="cuda") + + topk_weights_ref, topk_ids_ref = torch_topk( + gating_output=gating_output, + topk=topk, + renormalize=renormalize, + scoring_func=scoring_func, + ) + + topk_weights, topk_ids, _ = fused_topk( + hidden_states=hidden_states, + gating_output=gating_output, + topk=topk, + renormalize=renormalize, + scoring_func=scoring_func, + ) + + torch.testing.assert_close( + topk_weights_ref.to(torch.float32), topk_weights, atol=1e-2, rtol=1e-2 + ) + torch.testing.assert_close(topk_ids_ref.to(torch.int32), topk_ids, atol=0, rtol=0) + + +@pytest.mark.skipif( + not current_platform.is_cuda(), reason="This test is skipped on non-CUDA platform." +) +@pytest.mark.parametrize("num_tokens", [1, 33, 56]) +@pytest.mark.parametrize("hidden_size", [1024, 2048]) +@pytest.mark.parametrize("num_experts", [6, 16]) +@pytest.mark.parametrize("topk", [3, 4]) +@pytest.mark.parametrize("renormalize", [True, False]) +@pytest.mark.parametrize("scoring_func", ["softmax", "sigmoid"]) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.half, torch.float32]) +def test_fused_topk_bias( + num_tokens: int, + hidden_size: int, + num_experts: int, + topk: int, + renormalize: bool, + scoring_func: str, + dtype: torch.dtype, +): + torch.manual_seed(0) + hidden_states = torch.randn((num_tokens, hidden_size), dtype=dtype, device="cuda") + gating_output = torch.randn((num_tokens, num_experts), dtype=dtype, device="cuda") + e_score_correction_bias = torch.randn( + (num_experts,), dtype=torch.float32, device="cuda" + ) + + topk_weights_ref, topk_ids_ref = torch_topk( + gating_output=gating_output, + topk=topk, + renormalize=renormalize, + e_score_correction_bias=e_score_correction_bias, + scoring_func=scoring_func, + ) + + topk_weights, topk_ids = fused_topk_bias( + hidden_states=hidden_states, + gating_output=gating_output, + e_score_correction_bias=e_score_correction_bias, + topk=topk, + renormalize=renormalize, + scoring_func=scoring_func, + ) + + torch.testing.assert_close( + topk_weights_ref.to(torch.float32), topk_weights, atol=1e-2, rtol=1e-2 + ) + torch.testing.assert_close(topk_ids_ref.to(torch.int32), topk_ids, atol=0, rtol=0) + + +@pytest.mark.skipif( + not current_platform.is_cuda(), reason="This test is skipped on non-CUDA platform." +) +@pytest.mark.parametrize("num_experts", [6, 8, 16]) +@pytest.mark.parametrize("topk", [3, 4]) +@pytest.mark.parametrize("scoring_func", ["softmax", "sigmoid"]) +@pytest.mark.parametrize("bad_value", [float("nan"), float("inf")]) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.half, torch.float32]) +def test_fused_topk_nan_inf_clamp( + num_experts: int, + topk: int, + scoring_func: str, + bad_value: float, + dtype: torch.dtype, +): + """Regression test for the NaN/Inf clamp in topk_softmax_kernels.cu. + + Degenerate hidden states (e.g., from CUDA graph padding) can produce + NaN/Inf gating logits. Without the clamp, softmax/sigmoid outputs are + NaN and the argmax loop picks expert 0 for every top-k slot (since + "NaN > NaN" is false per IEEE 754), yielding duplicate expert IDs that + crash downstream MoE sort kernels. The fix clamps NaN/Inf to 0 before + argmax so index tie-breaking selects unique experts [0, 1, ..., k-1]. + """ + torch.manual_seed(0) + num_tokens = 4 + hidden_size = 1024 + hidden_states = torch.randn((num_tokens, hidden_size), dtype=dtype, device="cuda") + + # Row 0: all normal. Rows 1-3: fully poisoned with NaN or Inf. + gating_output = torch.randn((num_tokens, num_experts), dtype=dtype, device="cuda") + gating_output[1:, :] = bad_value + + topk_weights, topk_ids, _ = fused_topk( + hidden_states=hidden_states, + gating_output=gating_output, + topk=topk, + renormalize=False, + scoring_func=scoring_func, + ) + + # Normal row must still match the torch reference. + ref_weights, ref_ids = torch_topk( + gating_output=gating_output[:1], + topk=topk, + renormalize=False, + scoring_func=scoring_func, + ) + torch.testing.assert_close( + ref_weights.to(torch.float32), topk_weights[:1], atol=1e-2, rtol=1e-2 + ) + torch.testing.assert_close(ref_ids.to(torch.int32), topk_ids[:1], atol=0, rtol=0) + + # Poisoned rows: IDs must be unique (no duplicates) and weights must be + # finite (no NaN/Inf propagation into downstream MoE kernels). + for row in range(1, num_tokens): + row_ids = topk_ids[row] + assert row_ids.unique().numel() == topk, ( + f"Row {row} has duplicate expert IDs {row_ids.tolist()} " + f"(bad_value={bad_value}, scoring_func={scoring_func})" + ) + assert torch.isfinite(topk_weights[row]).all(), ( + f"Row {row} has non-finite weights {topk_weights[row].tolist()} " + f"(bad_value={bad_value}, scoring_func={scoring_func})" + ) + + +@pytest.mark.skipif( + not current_platform.is_cuda(), reason="This test is skipped on non-CUDA platform." +) +@pytest.mark.parametrize("num_experts", [6, 8, 16]) +@pytest.mark.parametrize("topk", [3, 4]) +@pytest.mark.parametrize("scoring_func", ["softmax", "sigmoid"]) +@pytest.mark.parametrize("bad_value", [float("nan"), float("inf")]) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.half, torch.float32]) +def test_fused_topk_bias_nan_inf_clamp( + num_experts: int, + topk: int, + scoring_func: str, + bad_value: float, + dtype: torch.dtype, +): + """Regression test: NaN/Inf in gating logits must not produce duplicate + expert IDs or non-finite weights when e_score_correction_bias is present. + + Same scenario as test_fused_topk_nan_inf_clamp but exercising the bias + path (fused_topk_bias) so the fix in topk_softmax_kernels.cu is covered + for that entry point as well. + """ + torch.manual_seed(0) + num_tokens = 4 + hidden_size = 1024 + hidden_states = torch.randn((num_tokens, hidden_size), dtype=dtype, device="cuda") + e_score_correction_bias = torch.randn( + (num_experts,), dtype=torch.float32, device="cuda" + ) + + gating_output = torch.randn((num_tokens, num_experts), dtype=dtype, device="cuda") + gating_output[1:, :] = bad_value + + topk_weights, topk_ids = fused_topk_bias( + hidden_states=hidden_states, + gating_output=gating_output, + e_score_correction_bias=e_score_correction_bias, + topk=topk, + renormalize=False, + scoring_func=scoring_func, + ) + + # Normal row must still match the torch reference. + ref_weights, ref_ids = torch_topk( + gating_output=gating_output[:1], + topk=topk, + renormalize=False, + e_score_correction_bias=e_score_correction_bias, + scoring_func=scoring_func, + ) + torch.testing.assert_close( + ref_weights.to(torch.float32), topk_weights[:1], atol=1e-2, rtol=1e-2 + ) + torch.testing.assert_close(ref_ids.to(torch.int32), topk_ids[:1], atol=0, rtol=0) + + # Poisoned rows: IDs must be unique (no duplicates) and weights must be + # finite (no NaN/Inf propagation into downstream MoE kernels). + for row in range(1, num_tokens): + row_ids = topk_ids[row] + assert row_ids.unique().numel() == topk, ( + f"Row {row} has duplicate expert IDs {row_ids.tolist()} " + f"(bad_value={bad_value}, scoring_func={scoring_func})" + ) + assert torch.isfinite(topk_weights[row]).all(), ( + f"Row {row} has non-finite weights {topk_weights[row].tolist()} " + f"(bad_value={bad_value}, scoring_func={scoring_func})" + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_gemma4router.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_gemma4router.py new file mode 100644 index 0000000000000000000000000000000000000000..af0ff854acbe654f9110bd560ece04211e877f74 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_gemma4router.py @@ -0,0 +1,57 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch + +from vllm.model_executor.models.gemma4 import ( + gemma4_fused_routing_kernel_triton, + gemma4_routing_function_torch, +) + + +def sort_by_id(w, ids): + order = ids.argsort(dim=-1) + return w.gather(1, order), ids.gather(1, order) + + +# Gemma4 MoE Model has context length of 250K +# the minus 1 is to ensure that edge cases are tested +@pytest.mark.parametrize("num_tokens", [1, 2, 2048, 250000]) +@pytest.mark.parametrize("num_experts", [128]) # gemma4 moe experts +@pytest.mark.parametrize("topk", [8]) # gemma4 topk +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.half, torch.float32]) +def test_gemma4_routing_kernel_triton( + num_tokens: int, + num_experts: int, + topk: int, + dtype: torch.dtype, +): + torch.manual_seed(0) + + gating = torch.randn(num_tokens, num_experts, dtype=dtype, device="cuda") + scales = torch.rand(num_experts, dtype=torch.float32, device="cuda") + + ref_w, ref_ids = gemma4_routing_function_torch(gating, topk, scales) + tri_w, tri_ids = gemma4_fused_routing_kernel_triton(gating, topk, scales) + + # Sort by expert id — to remove tie-breaking differences + ref_ws, ref_is = sort_by_id(ref_w, ref_ids) + tri_ws, tri_is = sort_by_id(tri_w, tri_ids) + + ids_match = (ref_is == tri_is).all().item() + weights_match = torch.allclose(ref_ws, tri_ws, atol=1e-2, rtol=1e-2) + all_match = ids_match and weights_match + max_err = (ref_ws - tri_ws).abs().max().item() + print( + f"T={num_tokens:5d} E={num_experts:4d} K={topk} " + f"{str(dtype).split('.')[-1]:7s} ids={ids_match} max_Δweight={max_err:.2e}" + ) + if not all_match: + bad = (ref_is != tri_is).any(dim=-1).nonzero(as_tuple=True)[0] + if len(bad): + r = bad[0].item() + print( + f" first bad row {r}: ref_ids={ref_ids[r].tolist()} " + f"tri_ids={tri_ids[r].tolist()}" + ) + assert all_match diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_gpt_oss_triton_kernels.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_gpt_oss_triton_kernels.py new file mode 100644 index 0000000000000000000000000000000000000000..d895ec77efb34ba701b79ed0cc7956540188b8c0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_gpt_oss_triton_kernels.py @@ -0,0 +1,373 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from dataclasses import dataclass, fields + +import pytest +import torch +import torch.nn.functional as F + +from vllm.platforms import current_platform +from vllm.utils.import_utils import has_triton_kernels + +if not has_triton_kernels(): + pytest.skip( + "triton_kernels not found, skipping all related tests", + allow_module_level=True, + ) + +import triton_kernels.matmul_ogs_details.opt_flags as opt_flags +import triton_kernels.swiglu +from triton_kernels.matmul_ogs import FlexCtx, PrecisionConfig +from triton_kernels.numerics import InFlexData +from triton_kernels.numerics_details.mxfp import downcast_to_mxfp, upcast_from_mxfp +from triton_kernels.tensor import FP4, convert_layout, wrap_torch_tensor +from triton_kernels.tensor_details import layout +from triton_kernels.testing import assert_close + +from vllm.model_executor.layers.fused_moe.config import mxfp4_w4a16_moe_quant_config +from vllm.model_executor.layers.fused_moe.experts.gpt_oss_triton_kernels_moe import ( + triton_kernel_moe_forward, +) +from vllm.utils.math_utils import round_up + +from .utils import shuffle_weight + + +def deshuffle(w: torch.Tensor): + first = w[..., ::2] + second = w[..., 1::2] + + deshuffled = torch.concat((first, second), dim=-1) + return deshuffled + + +def init_compute_data(M, K, N, E, a_dtype: str, w_dtype: str, num_warps: int): + randbits = [torch.randperm(E) for _ in range(M)] + x_list = [ + (-1) ** i + * ((16384 + ((i * 512) % 4096) + bits).to(torch.int16).view(torch.bfloat16)) + for i, bits in enumerate(randbits) + ] + exp_data = torch.stack(x_list).to(device="cuda") # simulating gate_output (M, E) + + # create input tensor + x = torch.randn((M, K), dtype=torch.bfloat16, device="cuda") + w1 = torch.randn((E, 2 * N, K), dtype=torch.bfloat16, device="cuda") + w1_bias = torch.randn((E, 2 * N), dtype=torch.bfloat16, device="cuda") + + w2 = torch.randn((E, K, N), dtype=torch.bfloat16, device="cuda") + w2_bias = torch.randn((E, K), dtype=torch.bfloat16, device="cuda") + + exp_data_tri = exp_data.clone() + x_tri = x.clone() + w1_tri = w1.clone() + w2_tri = w2.clone() + + w1_bias_tri = w1_bias.clone() + w2_bias_tri = w2_bias.clone() + w1_bias_tri = w1_bias_tri.to(torch.float32) + w2_bias_tri = w2_bias_tri.to(torch.float32) + + dtype_dict = { + "bf16": torch.bfloat16, + "fp8_e4m3": torch.float8_e4m3fn, + "fp8_e5m2": torch.float8_e5m2, + } + + x = x.to(dtype_dict[a_dtype]).to(torch.bfloat16) + if w_dtype != "mx4": + # simulate quantization support on reference impl + w1 = w1.to(dtype_dict[w_dtype]).to(torch.bfloat16) + w2 = w2.to(dtype_dict[w_dtype]).to(torch.bfloat16) + + # triton moe kernel use transposed shape for matmul + w1_tri = w1_tri.transpose(-2, -1) + w2_tri = w2_tri.transpose(-2, -1) + + # shuffle weights + w1_tri = shuffle_weight(w1_tri) + w1_bias_tri = shuffle_weight(w1_bias_tri) + + # quant triton_weights + x_tri = x.to(dtype_dict[a_dtype]) + if w_dtype != "mx4": + pytest.skip("NYI") + else: # quantize to mx4 + # Padding alignment depends on the platform. On CDNA4 the scale + # swizzle requires SCALE_K % 8 == 0 (K % 256) and + # SCALE_N % 32 == 0 (2*N % 512), matching the production + # alignment in mxfp4_round_up_hidden_size_and_intermediate_size. + # On CUDA (Hopper) the scale layout pads internally, so the + # original 64/128 alignment is sufficient. + if current_platform.is_rocm(): + k_align, n2_align = 256, 512 + else: + k_align, n2_align = 64, 128 + w1_bottom_pad = round_up(w1_tri.shape[1], k_align) - w1_tri.shape[1] + w1_right_pad = round_up(w1_tri.shape[2], n2_align) - w1_tri.shape[2] + + w2_bottom_pad = w1_right_pad // 2 + w2_right_pad = w1_bottom_pad + + x_pad = w1_bottom_pad + + w1_tri = F.pad( + w1_tri, + (0, w1_right_pad, 0, w1_bottom_pad, 0, 0), + mode="constant", + value=0, + ) + w2_tri = F.pad( + w2_tri, + (0, w2_right_pad, 0, w2_bottom_pad, 0, 0), + mode="constant", + value=0, + ) + + w1_bias_tri = F.pad( + w1_bias_tri, (0, w1_right_pad, 0, 0), mode="constant", value=0 + ) + w2_bias_tri = F.pad( + w2_bias_tri, (0, w2_right_pad, 0, 0), mode="constant", value=0 + ) + + x_tri = F.pad(x_tri, (0, x_pad, 0, 0), mode="constant", value=0) + + w_layout, w_layout_opts = layout.make_default_matmul_mxfp4_w_layout(mx_axis=1) + w_scale_layout, w_scale_layout_opts = ( + layout.make_default_matmul_mxfp4_w_scale_layout( + mx_axis=1, num_warps=num_warps + ) + ) + + w1_tri, w1_scale_tri = downcast_to_mxfp(w1_tri, torch.uint8, axis=1) + w1 = upcast_from_mxfp(w1_tri, w1_scale_tri, torch.bfloat16, axis=1) + + w2_tri, w2_scale_tri = downcast_to_mxfp(w2_tri, torch.uint8, axis=1) + w2 = upcast_from_mxfp(w2_tri, w2_scale_tri, torch.bfloat16, axis=1) + + w1_tri = convert_layout( + wrap_torch_tensor(w1_tri, FP4), w_layout, **w_layout_opts + ) + w1_scale_tri = convert_layout( + wrap_torch_tensor(w1_scale_tri), + w_scale_layout, + **w_scale_layout_opts, + ) + + w2_tri = convert_layout( + wrap_torch_tensor(w2_tri, FP4), w_layout, **w_layout_opts + ) + w2_scale_tri = convert_layout( + wrap_torch_tensor(w2_scale_tri), + w_scale_layout, + **w_scale_layout_opts, + ) + + pc1 = PrecisionConfig( + weight_scale=w1_scale_tri, flex_ctx=FlexCtx(rhs_data=InFlexData()) + ) + pc2 = PrecisionConfig( + weight_scale=w2_scale_tri, flex_ctx=FlexCtx(rhs_data=InFlexData()) + ) + + # tucuate so the rest can run properly + w1 = w1[..., :K, : 2 * N] + w2 = w2[..., :N, :K] + + w1 = deshuffle(w1) + + w1 = w1.transpose(-1, -2).contiguous() + w2 = w2.transpose(-1, -2).contiguous() + + return ( + x, + w1, + w1_bias, + w2, + w2_bias, + exp_data, + x_tri, + w1_tri, + w2_tri, + exp_data_tri, + w1_bias_tri, + w2_bias_tri, + pc1, + pc2, + ) + + +@dataclass +class ModelConfig: + num_hidden_layers: int = 36 + num_experts: int = 128 + experts_per_token: int = 4 + vocab_size: int = 201088 + hidden_size: int = 2880 + intermediate_size: int = 2880 + head_dim: int = 64 + num_attention_heads: int = 64 + num_key_value_heads: int = 8 + sliding_window: int = 128 + initial_context_length: int = 4096 + rope_theta: float = 150000.0 + rope_parameters_factor: float = 32.0 + rope_ntk_alpha: float = 1.0 + rope_ntk_beta: float = 32.0 + + +def swiglu(x, alpha: float = 1.702, limit: float = 1.0): + # Note we add an extra bias of 1 to the linear layer + x_glu, x_linear = torch.chunk(x, 2, dim=-1) + if limit is not None: + x_glu = x_glu.clamp(max=limit) + out_glu = x_glu * torch.sigmoid(alpha * x_glu) + if limit is not None: + x_linear = x_linear.clamp(min=-limit, max=limit) + return out_glu * (x_linear + 1) + + +def oai_moe_forward( + hidden_states: torch.Tensor, # (M, K) + w1: torch.Tensor, # (E, 2N) + w1_bias: torch.Tensor, # (E, 2N, K) + w2: torch.Tensor, # (E, K, N) + w2_bias: torch.Tensor, # (E, N) + gating_output: torch.Tensor, # (M, E) + topk: int, +): + # model.py 309:330, assuming gating and norm + t = hidden_states + experts = torch.topk(gating_output, k=topk, dim=-1, sorted=True) + expert_weights = torch.nn.functional.softmax(experts.values, dim=1) + expert_indices = experts.indices + + # MLP #1 + mlp1_weight = w1[expert_indices, ...] + mlp1_bias = w1_bias[expert_indices, ...] + t = torch.einsum("beck,bk->bec", mlp1_weight, t) + mlp1_bias + t = swiglu(t, limit=7) + + # MLP #2 + mlp2_weight = w2[expert_indices, ...] + mlp2_bias = w2_bias[expert_indices, ...] + t = torch.einsum("beck,bek->bec", mlp2_weight, t) + t += mlp2_bias + + # Weighted sum of experts + t = torch.einsum("bec,be->bc", t, expert_weights) + + return t + + +@dataclass +class Case: + a_dtype: str + w_dtype: str + + +@pytest.mark.parametrize( + ", ".join(f.name for f in fields(Case)), + [ + tuple(getattr(case, f.name) for f in fields(Case)) + for case in [ + # Case(a_dtype="bf16", w_dtype="bf16"), + # Case(a_dtype="fp8_e4m3", w_dtype="fp8_e5m2"), + Case(a_dtype="bf16", w_dtype="mx4") + ] + ], +) +@pytest.mark.parametrize("num_token", [2]) +@pytest.mark.parametrize("tp", [1, 2, 4, 8]) +def test_equiv(num_token, a_dtype, w_dtype, tp, workspace_init): + from triton_kernels.tensor_details import layout + + if not hasattr(layout, "make_default_matmul_mxfp4_w_layout"): + pytest.skip("make_default_matmul_mxfp4_w_layout not available") + + M = num_token + E = ModelConfig.num_experts + K = ModelConfig.hidden_size + N = ModelConfig.intermediate_size // tp + topk = ModelConfig.experts_per_token + + ( + x, + w1, + w1_bias, + w2, + w2_bias, + exp_data, + x_tri, + w1_tri, + w2_tri, + exp_data_tri, + w1_bias_tri, + w2_bias_tri, + pc1, + pc2, + ) = init_compute_data(M, K, N, E, a_dtype, w_dtype, num_warps=8) + + if current_platform.is_device_capability_family(100): + constraints = { + "is_persistent": True, + } + opt_flags.update_opt_flags_constraints(constraints) + + if a_dtype == "bf16" and w_dtype == "mx4": + quant_config = mxfp4_w4a16_moe_quant_config( + w1_scale=pc1, + w2_scale=pc2, + w1_bias=w1_bias_tri, + w2_bias=w2_bias_tri, + ) + else: + raise NotImplementedError( + f"Quantization configuration for activation={a_dtype} and weight={w_dtype} " + f"has not been implemented." + ) + + out_triton_monolithic = triton_kernel_moe_forward( + hidden_states=x_tri, + w1=w1_tri, + w2=w2_tri, + gating_output=exp_data_tri, + topk=topk, + renormalize=True, + quant_config=quant_config, + ) + out_triton_monolithic = out_triton_monolithic[..., :K] + + out_ref = oai_moe_forward( + hidden_states=x, + w1=w1, + w1_bias=w1_bias, + w2=w2, + w2_bias=w2_bias, + gating_output=exp_data, + topk=topk, + ) + assert_close(ref=out_ref, tri=out_triton_monolithic, maxtol=0.025, rmstol=0.005) + + +def test_unit_shuffle(): + N = ModelConfig.intermediate_size + K = ModelConfig.hidden_size + m = torch.randn((K, 2 * N), dtype=torch.bfloat16, device="cuda") + + x = torch.randn(K, dtype=torch.bfloat16, device="cuda") + + m_shuffled = shuffle_weight(m) + + out_ref = x @ m + out_ref = swiglu(out_ref, limit=1.0) + + out = x @ m_shuffled + out = triton_kernels.swiglu.swiglu_torch( + out, + alpha=1.702, + precision_config=triton_kernels.swiglu.PrecisionConfig(limit=1.0), + ) + + assert_close(ref=out_ref, tri=out) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_grouped_topk.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_grouped_topk.py new file mode 100644 index 0000000000000000000000000000000000000000..28d2f013803428062c74348108ce3fa21f845726 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_grouped_topk.py @@ -0,0 +1,103 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the MoE grouped topk kernel + +Run `pytest tests/kernels/moe/test_grouped_topk.py`. +""" + +import pytest +import torch + +import vllm.envs as envs +from vllm.config import ( + CompilationConfig, + VllmConfig, + get_cached_compilation_config, + set_current_vllm_config, +) +from vllm.model_executor.layers.fused_moe.router.grouped_topk_router import ( + GroupedTopk, + fused_grouped_topk, +) +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + + +@pytest.mark.skipif( + not current_platform.is_cuda(), reason="This test is skipped on non-CUDA platform." +) +@pytest.mark.parametrize("n_token", [1, 33, 64]) +@pytest.mark.parametrize("n_hidden", [1024, 2048]) +@pytest.mark.parametrize( + "n_expert,topk,num_expert_group,topk_group", + [ + (16, 2, 8, 2), + (128, 2, 8, 2), + (256, 8, 8, 4), + (384, 8, 1, 1), + (512, 22, 1, 1), + ], +) +@pytest.mark.parametrize("renormalize", [True, False]) +@pytest.mark.parametrize("scoring_func", ["softmax", "sigmoid"]) +@pytest.mark.parametrize("routed_scaling_factor", [1.0, 2.5]) +@pytest.mark.parametrize("input_dtype", [torch.bfloat16, torch.float32]) +@pytest.mark.parametrize("bias_dtype", [torch.float32]) +def test_grouped_topk( + monkeypatch: pytest.MonkeyPatch, + n_token: int, + n_hidden: int, + n_expert: int, + topk: int, + num_expert_group: int, + topk_group: int, + renormalize: bool, + scoring_func: str, + routed_scaling_factor: float, + input_dtype: torch.dtype, + bias_dtype: torch.dtype, +): + vllm_config = VllmConfig( + compilation_config=CompilationConfig(custom_ops=["all", "+grouped_topk"]) + ) + get_cached_compilation_config.cache_clear() + + set_random_seed(0) + hidden_states = torch.randn((n_token, n_hidden), dtype=input_dtype, device="cuda") + gating_output = torch.randn((n_token, n_expert), dtype=input_dtype, device="cuda") + e_score_correction_bias = torch.randn((n_expert,), dtype=bias_dtype, device="cuda") + + with set_current_vllm_config(vllm_config), monkeypatch.context() as m: + m.setenv("VLLM_USE_FUSED_MOE_GROUPED_TOPK", "0") + m.setattr(envs, "VLLM_BATCH_INVARIANT", True) + grouped_topk = GroupedTopk( + topk=topk, + renormalize=renormalize, + num_expert_group=num_expert_group, + topk_group=topk_group, + scoring_func=scoring_func, + routed_scaling_factor=routed_scaling_factor, + ) + assert grouped_topk._forward_method.__name__ == "forward_cuda" + baseline_topk_weights, baseline_topk_ids = grouped_topk( + hidden_states=hidden_states, + gating_output=gating_output, + e_score_correction_bias=e_score_correction_bias, + ) + + test_topk_weights, test_topk_ids = fused_grouped_topk( + hidden_states=hidden_states, + gating_output=gating_output, + topk=topk, + renormalize=renormalize, + num_expert_group=num_expert_group, + topk_group=topk_group, + scoring_func=scoring_func, + routed_scaling_factor=routed_scaling_factor, + e_score_correction_bias=e_score_correction_bias, + ) + + torch.testing.assert_close( + baseline_topk_weights, test_topk_weights, atol=2e-2, rtol=0 + ) + torch.testing.assert_close(baseline_topk_ids, test_topk_ids, atol=0, rtol=0) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_marlin_vs_trtllm_mxint4.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_marlin_vs_trtllm_mxint4.py new file mode 100644 index 0000000000000000000000000000000000000000..657bf0093b6a2a41fe5c66622a1d33c274d8075d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_marlin_vs_trtllm_mxint4.py @@ -0,0 +1,377 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Test comparing Marlin INT4 MoE vs FlashInfer TRT-LLM MXINT4 MoE.""" + +import pytest +import torch + +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.experts.marlin_moe import ( + fused_marlin_moe, +) +from vllm.model_executor.layers.fused_moe.experts.trtllm_mxint4_moe import ( + TrtLlmMxint4ExpertsMonolithic, +) +from vllm.model_executor.layers.fused_moe.router.grouped_topk_router import ( + grouped_topk, +) +from vllm.model_executor.layers.quantization.utils.flashinfer_mxint4_moe import ( + prepare_static_weights_for_trtllm_mxint4_moe, +) +from vllm.platforms import current_platform +from vllm.scalar_type import scalar_types +from vllm.utils.torch_utils import set_random_seed + + +def mxint4_quantize( + x: torch.Tensor, sf_vec_size: int = 32 +) -> tuple[torch.Tensor, torch.Tensor]: + """Quantize BF16 tensor to MXINT4 with block scaling (group_size=sf_vec_size). + + Returns: + - uint8 packed (2 INT4/byte): [..., k//2] - stores SIGNED INT4 [-8, 7] + - scales in BF16: [..., k//sf_vec_size] + """ + x_reshaped = x.reshape(-1, sf_vec_size) + x_max = x_reshaped.max(dim=-1, keepdim=True)[0].to(torch.float32) + x_min = x_reshaped.min(dim=-1, keepdim=True)[0].to(torch.float32) + x_max = x_max * 8.0 / 7.0 + amax = torch.where(x_max > -x_min, x_max, -x_min) + scales = amax / 8.0 + x_scaled = x_reshaped * scales.reciprocal() + x_int8 = ( + x_scaled.round().clamp(-8, 7).to(torch.int8).reshape(-1, sf_vec_size // 2, 2) + ) + x_int4 = (x_int8[..., 0] & 0x0F) | ((x_int8[..., 1] & 0x0F) << 4) + return ( + x_int4.to(torch.uint8).reshape(*x.shape[:-1], x.shape[-1] // 2), + scales.to(x.dtype).reshape(*x.shape[:-1], x.shape[-1] // sf_vec_size), + ) + + +def mxint4_quantize_moe_weights( + weights_bf16: torch.Tensor, group_size: int = 32 +) -> tuple[torch.Tensor, torch.Tensor]: + """Quantize MoE weights [e, n, k] to MxInt4 format. + + Args: + weights_bf16: BF16 weights of shape [num_experts, out_features, in_features] + group_size: Quantization group size (default: 32) + + Returns: + - weights_mxint4: Quantized weights [e, n, k//2] uint8 + - scales_mxint4: Quantization scales [e, n, k//group_size] bf16 + """ + e = weights_bf16.shape[0] + weight_list = [] + scale_list = [] + + for i in range(e): + w_q, w_s = mxint4_quantize(weights_bf16[i], sf_vec_size=group_size) + weight_list.append(w_q) + scale_list.append(w_s) + + return torch.stack(weight_list), torch.stack(scale_list) + + +__all__ = [ + "mxint4_quantize", + "mxint4_quantize_moe_weights", + "marlin_quantize_moe_weights", +] + + +def test_trtllm_mxint4_activation_supports_vllm_gated_silu(): + assert TrtLlmMxint4ExpertsMonolithic._supports_activation(MoEActivation.SILU) + assert TrtLlmMxint4ExpertsMonolithic._supports_activation(MoEActivation.SWIGLUOAI) + assert not TrtLlmMxint4ExpertsMonolithic._supports_activation( + MoEActivation.RELU2_NO_MUL + ) + + +def marlin_quantize_moe_weights( + weights_bf16: torch.Tensor, group_size: int = 32 +) -> tuple[torch.Tensor, torch.Tensor]: + """Quantize MoE weights [e, n, k] to Marlin INT4 format. + + Args: + weights_bf16: BF16 weights of shape [num_experts, out_features, in_features] + group_size: Quantization group size (default: 32) + + Returns: + - weights_marlin: Marlin quantized weights [e, k//8, n] int32 + - scales_marlin: Marlin quantization scales [e, k//group_size, n] bf16 + """ + from vllm.model_executor.layers.quantization.utils.marlin_utils_test import ( + marlin_quantize, + ) + + e, n, k = weights_bf16.shape + weight_list = [] + scale_list = [] + + for i in range(e): + # Transpose for Marlin: [n, k] → [k, n] + w_t = weights_bf16[i].T.contiguous() + _, w_q, w_s, _, _, _ = marlin_quantize( + w_t, scalar_types.uint4b8, group_size, act_order=False + ) + weight_list.append(w_q) + scale_list.append(w_s) + + # Stack to get [e, ...] shape + weights_marlin = torch.stack(weight_list) # [e, k // 8, n] + scales_marlin = torch.stack(scale_list) # [e, k // group_size, n] + + return weights_marlin, scales_marlin + + +TRTLLM_GEN_AVAILABLE = ( + current_platform.is_cuda() and current_platform.is_device_capability_family(100) +) + + +@pytest.mark.skipif(not TRTLLM_GEN_AVAILABLE, reason="Skip for non SM100") +@pytest.mark.parametrize("m", [1, 33]) +@pytest.mark.parametrize("n", [7168]) +@pytest.mark.parametrize("k", [512]) +@pytest.mark.parametrize("e", [384]) +@pytest.mark.parametrize("topk", [8]) +@pytest.mark.parametrize("group_size", [32]) +def test_marlin_vs_trtllm_mxint4_moe_kimik2(monkeypatch, m, n, k, e, topk, group_size): + """Compare Marlin INT4 MoE vs FlashInfer TRT-LLM MXINT4 MoE. + + Uses mxint4_quantize() to generate common INT4 weights + BF16 scales, + then runs both Marlin and TRT-LLM kernels and compares outputs. + """ + pytest.importorskip("flashinfer") + monkeypatch.setenv("VLLM_USE_FLASHINFER_MOE_INT4", "1") + + set_random_seed(0) + + dtype = torch.bfloat16 + + # DeepSeekV3 routing config (from Kimi-K2-Thinking config.json) + n_group = 1 # n_group from model config + topk_group = 1 # topk_group from model config + routed_scaling = 2.827 # routed_scaling_factor from model config + + # Input - realistic activation range for LLM (after LayerNorm: mean~0, std~1) + a = torch.randn((m, k), device="cuda", dtype=dtype) * 0.5 + + # Generate routing logits and bias (DeepSeekV3 expects float logits) + # Realistic ranges: logits typically [-3, 3], bias [-2, 2] + routing_logits = torch.randn((m, e), device="cuda", dtype=torch.float32) * 1.5 + routing_bias = torch.randn(e, device="cuda", dtype=torch.float32) * 0.8 + + # 1. Generate BF16 weights (SHARED between both paths) + # Realistic weight initialization: Xavier/Glorot uniform scaling + # std = sqrt(2 / (fan_in + fan_out)) + std_w1 = (2.0 / (k + 2 * n)) ** 0.5 + std_w2 = (2.0 / (n + k)) ** 0.5 + w1_bf16 = torch.randn((e, 2 * n, k), device="cuda", dtype=dtype) * std_w1 + w2_bf16 = torch.randn((e, k, n), device="cuda", dtype=dtype) * std_w2 + + # === Path 1: TRT-LLM FlashInfer MXINT4 MoE === + # Similar to: if self.use_flashinfer_mxint4_moe + # Quantize using MXINT4 method (signed INT4) + w1_int4, w1_scales = mxint4_quantize_moe_weights(w1_bf16, group_size) + w2_int4, w2_scales = mxint4_quantize_moe_weights(w2_bf16, group_size) + + trtllm_weights = prepare_static_weights_for_trtllm_mxint4_moe( + gemm1_weights=w1_int4, + gemm1_scales=w1_scales, + gemm2_weights=w2_int4, + gemm2_scales=w2_scales, + ) + + from flashinfer import RoutingMethodType + from flashinfer.fused_moe import trtllm_mxint4_block_scale_moe + + # Routing handled internally by trtllm_mxint4_block_scale_moe + trtllm_output = trtllm_mxint4_block_scale_moe( + routing_logits=routing_logits, + routing_bias=routing_bias.to(torch.bfloat16), + hidden_states=a, + gemm1_weights=trtllm_weights["gemm1_weights"], + gemm1_weights_scale=trtllm_weights["gemm1_scales"], + gemm1_alpha=None, + gemm1_beta=None, + gemm1_clamp_limit=None, + gemm2_weights=trtllm_weights["gemm2_weights"], + gemm2_weights_scale=trtllm_weights["gemm2_scales"], + num_experts=e, + top_k=topk, + n_group=n_group, + topk_group=topk_group, + intermediate_size=n, + local_expert_offset=0, + local_num_experts=e, + routed_scaling_factor=routed_scaling, + routing_method_type=RoutingMethodType.DeepSeekV3, + enable_pdl=None, + output=None, + tune_max_num_tokens=8192, + ).to(dtype) + + # === Path 2: Marlin INT4 MoE === + # Similar to: else (non-flashinfer path) + # Quantize using Marlin's method (UINT4b8) + w1_marlin, w1_scales_marlin = marlin_quantize_moe_weights(w1_bf16, group_size) + w2_marlin, w2_scales_marlin = marlin_quantize_moe_weights(w2_bf16, group_size) + + # Use production routing kernel (same as router.select_experts internally uses) + topk_weights, topk_ids = grouped_topk( + hidden_states=a, + gating_output=routing_logits, + topk=topk, + renormalize=False, # DeepSeekV3 doesn't renormalize + num_expert_group=n_group, + topk_group=topk_group, + scoring_func="sigmoid", # DeepSeekV3 uses sigmoid + routed_scaling_factor=routed_scaling, + e_score_correction_bias=routing_bias, + ) + + marlin_output = fused_marlin_moe( + hidden_states=a, + w1=w1_marlin, + w2=w2_marlin, + bias1=None, + bias2=None, + w1_scale=w1_scales_marlin, + w2_scale=w2_scales_marlin, + topk_weights=topk_weights, + topk_ids=topk_ids, + quant_type_id=scalar_types.uint4b8.id, + global_num_experts=e, + expert_map=None, + global_scale1=None, + global_scale2=None, + g_idx1=None, + g_idx2=None, + input_global_scale1=None, + input_global_scale2=None, + sort_indices1=None, + sort_indices2=None, + w1_zeros=None, + w2_zeros=None, + input_dtype=dtype, + is_k_full=True, + ) + + # Sanity check: manually compute BF16 reference for comparison + # Use same routing as Marlin path for consistency + bf16_output = torch.zeros((m, k), device="cuda", dtype=dtype) + for token_idx in range(m): + for expert_rank in range(topk): + expert_id = topk_ids[token_idx, expert_rank].item() + weight = topk_weights[token_idx, expert_rank].item() + # w1: [2*n, k] @ [k] -> [2*n] + up_gate = a[token_idx] @ w1_bf16[expert_id].T # [2*n] + gate, up = up_gate.chunk(2, dim=0) + intermediate = torch.nn.functional.silu(gate) * up # [n] + # w2: [k, n] @ [n] -> [k] + expert_out = intermediate @ w2_bf16[expert_id].T # [k] + bf16_output[token_idx] += weight * expert_out + # Compare against BF16 reference. + torch.testing.assert_close(marlin_output, bf16_output, atol=0.3, rtol=1.0) + torch.testing.assert_close(trtllm_output, bf16_output, atol=0.3, rtol=1.0) + + # Compare against each other for sanity. + # Note: Different quantization schemes (UINT4b8 vs signed MXINT4) cause + # some differences + torch.testing.assert_close(marlin_output, trtllm_output, atol=0.3, rtol=6.0) + + +@pytest.mark.skipif(not TRTLLM_GEN_AVAILABLE, reason="Skip for non SM100") +@pytest.mark.parametrize("m", [1, 33]) +@pytest.mark.parametrize("n", [7168]) +@pytest.mark.parametrize("k", [512]) +@pytest.mark.parametrize("e", [384]) +@pytest.mark.parametrize("topk", [8]) +@torch.inference_mode() +def test_flashinfer_trtllm_mxint4_moe_wrapper(m, n, k, e, topk): + """Test that the flashinfer_trtllm_mxint4_moe wrapper matches the raw + trtllm_mxint4_block_scale_moe kernel call.""" + pytest.importorskip("flashinfer") + from flashinfer import RoutingMethodType + from flashinfer.fused_moe import trtllm_mxint4_block_scale_moe + + from vllm.model_executor.layers.quantization.utils.flashinfer_mxint4_moe import ( + flashinfer_trtllm_mxint4_moe, + ) + + set_random_seed(0) + dtype = torch.bfloat16 + + a = torch.randn((m, k), device="cuda", dtype=dtype) * 0.5 + router_logits = torch.randn((m, e), device="cuda", dtype=torch.float32) * 1.5 + routing_bias = torch.randn(e, device="cuda", dtype=torch.float32) * 0.8 + + std_w1 = (2.0 / (k + 2 * n)) ** 0.5 + std_w2 = (2.0 / (n + k)) ** 0.5 + w1_bf16 = torch.randn((e, 2 * n, k), device="cuda", dtype=dtype) * std_w1 + w2_bf16 = torch.randn((e, k, n), device="cuda", dtype=dtype) * std_w2 + + w1_int4, w1_scales = mxint4_quantize_moe_weights(w1_bf16) + w2_int4, w2_scales = mxint4_quantize_moe_weights(w2_bf16) + + prepared = prepare_static_weights_for_trtllm_mxint4_moe( + gemm1_weights=w1_int4, + gemm1_scales=w1_scales, + gemm2_weights=w2_int4, + gemm2_scales=w2_scales, + ) + + # Raw kernel call (reference) + raw_out = trtllm_mxint4_block_scale_moe( + routing_logits=router_logits.to(torch.float32), + routing_bias=routing_bias.to(torch.bfloat16), + hidden_states=a, + gemm1_weights=prepared["gemm1_weights"].data, + gemm1_weights_scale=prepared["gemm1_scales"].data, + gemm1_alpha=None, + gemm1_beta=None, + gemm1_clamp_limit=None, + gemm2_weights=prepared["gemm2_weights"].data, + gemm2_weights_scale=prepared["gemm2_scales"].data, + num_experts=e, + top_k=topk, + n_group=1, + topk_group=1, + intermediate_size=n, + local_expert_offset=0, + local_num_experts=e, + routed_scaling_factor=None, + routing_method_type=RoutingMethodType.DeepSeekV3, + enable_pdl=None, + output=None, + tune_max_num_tokens=8192, + ) + if not isinstance(raw_out, torch.Tensor): + raw_out = raw_out[0] + raw_out = raw_out.to(dtype) + + # Wrapper call + wrapper_out = flashinfer_trtllm_mxint4_moe( + x=a, + router_logits=router_logits, + w13_weight_packed=prepared["gemm1_weights"], + w13_weight_scale=prepared["gemm1_scales"], + w2_weight_packed=prepared["gemm2_weights"], + w2_weight_scale=prepared["gemm2_scales"], + global_num_experts=e, + top_k=topk, + intermediate_size_per_partition=n, + local_num_experts=e, + ep_rank=0, + num_expert_group=1, + topk_group=1, + e_score_correction_bias=routing_bias, + routing_method_type=RoutingMethodType.DeepSeekV3, + ) + + assert wrapper_out.shape == (m, k) + assert wrapper_out.dtype == dtype + torch.testing.assert_close(wrapper_out, raw_out, atol=0.0, rtol=0.0) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_modular_kernel_combinations.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_modular_kernel_combinations.py new file mode 100644 index 0000000000000000000000000000000000000000..62783bea424f2b94b1ca7bb2e7cf37ddfb3a54d2 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_modular_kernel_combinations.py @@ -0,0 +1,393 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import copy +import textwrap +import traceback +from itertools import product +from typing import Any + +import pytest +import torch + +import vllm.model_executor.layers.fused_moe.modular_kernel as mk +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.platforms import current_platform +from vllm.utils.flashinfer import has_flashinfer_cutlass_fused_moe +from vllm.utils.import_utils import has_deep_ep, has_deep_gemm +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.worker.workspace import init_workspace_manager + +from .modular_kernel_tools.common import ( + Config, + RankTensors, + WeightTensors, + reference_moe_impl, + run_modular_kernel, +) +from .modular_kernel_tools.mk_objects import ( + MK_FUSED_EXPERT_TYPES, + MK_MULTI_GPU_PREPARE_FINALIZE_TYPES, + MK_QUANT_CONFIGS, + MK_SINGLE_GPU_PREPARE_FINALIZE_TYPES, + TestMoEQuantConfig, + expert_info, +) +from .modular_kernel_tools.parallel_utils import ( + ProcessGroupInfo, + parallel_launch_with_config, +) + +has_any_multi_gpu_package = ( + has_deep_ep() or has_deep_gemm() or has_flashinfer_cutlass_fused_moe() +) + +meets_multi_gpu_requirements = pytest.mark.skipif( + not has_any_multi_gpu_package, + reason="Requires deep_ep or deep_gemm or flashinfer packages", +) + +if current_platform.is_fp8_fnuz(): + pytest.skip( + "Tests in this file require float8_e4m3fn and platform does not support", + allow_module_level=True, + ) + + +def format_result(verbose, msg, ex=None): + if ex is not None: + x = str(ex) + newx = x.strip(" \n\t")[:16] + if len(newx) < len(x): + newx = newx + " ..." + + prefix = "E\t" + print(f"{textwrap.indent(traceback.format_exc(), prefix)}") + print(f"FAILED {msg} - {newx}\n") + elif verbose: + print(f"PASSED {msg}") + else: + print(".", end="") + + +def rank_worker( + pgi: ProcessGroupInfo, + vllm_config: VllmConfig, + cpu_group, + base_config: Config, + weights: WeightTensors, + verbose: bool, +): + # Initialize workspace manager in child process + device = torch.device(f"cuda:{pgi.local_rank}") + init_workspace_manager(device) + + set_random_seed(pgi.rank) + + # get weights to this device + weights.to_current_device() + + Ms = base_config.Ms + assert isinstance(Ms, list) + TOPKs = base_config.topks + assert isinstance(TOPKs, list) + + exceptions = [] + count = 0 + + for m, topk in product(Ms, TOPKs): + # override m and topk + config = copy.deepcopy(base_config) + config.Ms = m + config.topks = topk + + try: + print(f"Running[{pgi.rank}]: m={m}, topk={topk} ...") + count = count + 1 + + # inputs for rank + rank_tensors = RankTensors.make(config, pgi) + + # Skip unsupported: AITER block-scaled MoE does not + # support apply_router_weight_on_input (topk=1 path). + # https://github.com/ROCm/aiter/issues/2418 + if ( + topk == 1 + and config.supports_apply_weight_on_input() + and getattr(config.fused_experts_type, "__name__", "") == "AiterExperts" + and config.quant_block_shape is not None + ): + print( + f"Skipping[{pgi.rank}]: m={m}, topk={topk}" + " (AITER block-scaled + weight-on-input," + " https://github.com/ROCm/aiter/issues/2418)" + ) + count -= 1 + continue + + # modular kernel out + mk_out = run_modular_kernel(pgi, vllm_config, config, weights, rank_tensors) + + with set_current_vllm_config(vllm_config): + ref_out = reference_moe_impl(config, weights, rank_tensors) + + if config.quant_dtype == "nvfp4": + atol = 1e-1 if config.K < 4096 else 2e-1 + rtol = 1e-1 if config.K < 4096 else 2e-1 + else: + atol = 3e-2 + rtol = 3e-2 + + # On ROCm, AITER FP8 fused MoE uses hardware FP8 + # dot-product which can produce slightly larger error + # than dequant+f32 matmul at FP8 representable-value + # boundaries. Allow a small percentage of elements to + # exceed the base tolerance by a bounded margin. + # https://github.com/ROCm/aiter/issues/2421 + from vllm.platforms import current_platform as _cp + + is_aiter_fp8 = ( + _cp.is_rocm() + and getattr(config.fused_experts_type, "__name__", "") == "AiterExperts" + and config.quant_config is not None + ) + if is_aiter_fp8: + diff = (ref_out - mk_out).abs() + n_total = diff.numel() + max_diff = diff.max().item() + n_exceed = int((diff > atol).sum().item()) + pct_exceed = n_exceed / n_total * 100 + # FP8 hw matmul vs f32 reference: up to ~4% of + # elements may exceed base tolerance, but max + # error should stay within 3x base tolerance. + max_pct_allowed = 5.0 + relaxed_atol = atol * 4 + print( + f"[AITER FP8 precision] " + f"max_diff={max_diff:.6f}, " + f"exceed_atol={n_exceed}/{n_total} " + f"({pct_exceed:.4f}%), " + f"max_pct_allowed={max_pct_allowed}%, " + f"relaxed_limit={relaxed_atol}" + ) + assert pct_exceed <= max_pct_allowed, ( + f"AITER FP8: {pct_exceed:.2f}% elements exceed " + f"atol={atol} (max allowed {max_pct_allowed}%)" + ) + assert max_diff <= relaxed_atol, ( + f"AITER FP8: max_diff={max_diff:.6f} exceeds " + f"relaxed limit {relaxed_atol}" + ) + else: + torch.testing.assert_close(ref_out, mk_out, atol=atol, rtol=rtol) + format_result(verbose, config.describe()) + except Exception as ex: + format_result(verbose, config.describe(), ex) + exceptions.append(ex) + + if len(exceptions) > 0: + raise RuntimeError( + f"{len(exceptions)} of {count} tests failed in child process, " + f"rank={pgi.rank}." + ) + else: + print(f"{count} of {count} tests passed in child process, rank={pgi.rank}.") + + +def run(config: Config, verbose: bool): + assert config.is_valid()[0] + assert not is_nyi_config(config) + + weights: WeightTensors = WeightTensors.make(config) + + vllm_config, env_dict = config.make_env_data() + parallel_launch_with_config( + config.world_size, rank_worker, vllm_config, env_dict, config, weights, verbose + ) + + +Ms = [32, 64] +# hidden sizes, making this too large will cause fp4 tests to fail. +# Also needs to be a multiple of 1024 for deep_gemm. +Ks = [2048] +Ns = [1024] +TOPKs = [4, 1] +Es = [32] +DTYPEs = [torch.bfloat16] + + +def is_nyi_config(config: Config) -> bool: + # We know these configs to be legitimate. but still fail. + info = expert_info(config.fused_experts_type) + if info.needs_matching_quant: + # The triton kernels expect both per-act-token-quant and + # per-out-ch-quant or neither. + unsupported_quant_config = ( + config.is_per_act_token_quant + config.is_per_out_ch_quant + ) == 1 + return unsupported_quant_config + + return False + + +def generate_valid_test_cases( + world_size: int, prepare_finalize_types +) -> list[tuple[Any, ...]]: + cases = [] + total = 0 + + for k, n, e, dtype, quant_config, combination in product( + Ks, + Ns, + Es, + DTYPEs, + MK_QUANT_CONFIGS, + product(prepare_finalize_types, MK_FUSED_EXPERT_TYPES), + ): + total = total + 1 + + config = Config( + Ms=Ms, + K=k, + N=n, + E=e, + topks=TOPKs, + dtype=dtype, + quant_config=quant_config, + prepare_finalize_type=combination[0], + fused_experts_type=combination[1], + world_size=world_size, + ) + + # TODO(bnell): figure out how to get verbose flag here. + verbose = False # pytestconfig.getoption('verbose') > 0 + + valid, reason = config.is_valid() + + if not valid: + if verbose: + print(f"Test config {config} is not valid: {reason}") + continue + + if is_nyi_config(config): + if verbose: + print(f"Test config {config} is nyi.") + continue + + cases.append( + ( + k, + n, + e, + dtype, + quant_config, + combination[0], + combination[1], + world_size, + ) + ) + + print(f"{len(cases)} of {total} valid configs generated.") + + return cases + + +@pytest.mark.parametrize( + "k,n,e,dtype,quant_config,prepare_finalize_type,fused_experts_type,world_size", + generate_valid_test_cases( + world_size=2, prepare_finalize_types=MK_MULTI_GPU_PREPARE_FINALIZE_TYPES + ), +) +@meets_multi_gpu_requirements +def test_modular_kernel_combinations_multigpu( + k: int, + n: int, + e: int, + dtype: torch.dtype, + quant_config: TestMoEQuantConfig | None, + prepare_finalize_type: mk.FusedMoEPrepareAndFinalize, + fused_experts_type: mk.FusedMoEExperts, + world_size: int, + pytestconfig, +): + if current_platform.device_count() < world_size: + pytest.skip( + f"Not enough GPUs available to run, got " + f"{current_platform.device_count()} expected " + f"{world_size}." + ) + + config = Config( + Ms=Ms, + K=k, + N=n, + E=e, + topks=TOPKs, + dtype=dtype, + quant_config=quant_config, + prepare_finalize_type=prepare_finalize_type, + fused_experts_type=fused_experts_type, + world_size=world_size, + ) + verbosity = pytestconfig.getoption("verbose") + run(config, verbosity > 0) + + +@pytest.mark.parametrize( + "k,n,e,dtype,quant_config,prepare_finalize_type,fused_experts_type,world_size", + generate_valid_test_cases( + world_size=1, prepare_finalize_types=MK_SINGLE_GPU_PREPARE_FINALIZE_TYPES + ), +) +def test_modular_kernel_combinations_singlegpu( + k: int, + n: int, + e: int, + dtype: torch.dtype, + quant_config: TestMoEQuantConfig | None, + prepare_finalize_type: mk.FusedMoEPrepareAndFinalize, + fused_experts_type: mk.FusedMoEExperts, + world_size: int, + pytestconfig, + workspace_init, +): + """Note: float8_e4m3fn is not supported on CUDA architecture < 89, + and those tests will be skipped on unsupported hardware.""" + config = Config( + Ms=Ms, + K=k, + N=n, + E=e, + topks=TOPKs, + dtype=dtype, + quant_config=quant_config, + prepare_finalize_type=prepare_finalize_type, + fused_experts_type=fused_experts_type, + world_size=world_size, + ) + + if ( + quant_config is not None and quant_config.quant_dtype == torch.float8_e4m3fn + ) and not current_platform.has_device_capability(89): + pytest.skip( + "Triton limitation: fp8e4nv data type is not supported on CUDA arch < 89" + ) + verbosity = pytestconfig.getoption("verbose") + run(config, verbosity > 0) + + +if __name__ == "__main__": + # Ability to test individual PrepareAndFinalize and FusedExperts combination + from .modular_kernel_tools.cli_args import make_config, make_config_arg_parser + + parser = make_config_arg_parser( + description=( + "Run single prepare-finalize & fused-experts combination test" + "Example : python3 -m tests.kernels.moe.test_modular_kernel_combinations " + "--pf-type DeepEPLLPrepareAndFinalize --experts-type BatchedTritonExperts" + ) + ) + args = parser.parse_args() + config = make_config(args) + + run(config, True) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_modular_oai_triton_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_modular_oai_triton_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..26ad09a9ef4ee746301a5f8f67d57d4466d2f469 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_modular_oai_triton_moe.py @@ -0,0 +1,263 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Test modular OAI Triton MoE +""" + +import pytest +import torch + +from tests.utils import wait_for_gpu_memory_to_clear +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.utils.import_utils import has_triton_kernels + +if not has_triton_kernels(): + pytest.skip( + "triton_kernels not found, skipping all related tests", + allow_module_level=True, + ) + +from triton_kernels.matmul_ogs import FlexCtx, PrecisionConfig +from triton_kernels.numerics import InFlexData +from triton_kernels.numerics_details.mxfp import downcast_to_mxfp, upcast_from_mxfp +from triton_kernels.tensor import FP4, convert_layout, wrap_torch_tensor +from triton_kernels.tensor_details import layout +from triton_kernels.testing import assert_close + +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.model_executor.layers.fused_moe.all2all_utils import ( + maybe_make_prepare_finalize, +) +from vllm.model_executor.layers.fused_moe.config import mxfp4_w4a16_moe_quant_config +from vllm.model_executor.layers.fused_moe.experts.gpt_oss_triton_kernels_moe import ( + OAITritonExperts, + UnfusedOAITritonExperts, +) +from vllm.model_executor.layers.fused_moe.modular_kernel import FusedMoEKernel +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +from .utils import make_dummy_moe_config, shuffle_weight + +MNK = [ + (1, 512, 384), + (1, 2880, 2880), + (2, 512, 384), + (2, 2880, 2880), + (16, 2880, 2880), +] + + +def unshuffle_weight(w: torch.Tensor): + first = w[..., ::2] + second = w[..., 1::2] + return torch.concat((first, second), dim=-1) + + +def make_weights(dtype, k, n, e): + w1 = torch.randn((e, k, 2 * n), dtype=dtype, device="cuda") + w1_bias = torch.randn((e, 2 * n), dtype=dtype, device="cuda") + + w2 = torch.randn((e, n, k), dtype=dtype, device="cuda") + w2_bias = torch.randn((e, k), dtype=dtype, device="cuda") + + w1_tri = w1.clone() + w2_tri = w2.clone() + + w1_bias_tri = w1_bias.clone() + w2_bias_tri = w2_bias.clone() + w1_bias_tri = w1_bias_tri.to(torch.float32) + w2_bias_tri = w2_bias_tri.to(torch.float32) + + # shuffle weights + w1_tri = shuffle_weight(w1_tri) + w1_bias_tri = shuffle_weight(w1_bias_tri) + + # quant triton_weights + w1_tri, w1_scale_tri = downcast_to_mxfp(w1_tri, torch.uint8, axis=1) + w1 = upcast_from_mxfp(w1_tri, w1_scale_tri, dtype, axis=1) + w1 = unshuffle_weight(w1) + + w2_tri, w2_scale_tri = downcast_to_mxfp(w2_tri, torch.uint8, axis=1) + w2 = upcast_from_mxfp(w2_tri, w2_scale_tri, dtype, axis=1) + + num_warps = 8 + w_layout, w_layout_opts = layout.make_default_matmul_mxfp4_w_layout(mx_axis=1) + w_scale_layout, w_scale_layout_opts = ( + layout.make_default_matmul_mxfp4_w_scale_layout(mx_axis=1, num_warps=num_warps) + ) + + w1_tri = convert_layout(wrap_torch_tensor(w1_tri, FP4), w_layout, **w_layout_opts) + w1_scale_tri = convert_layout( + wrap_torch_tensor(w1_scale_tri), + w_scale_layout, + **w_scale_layout_opts, + ) + + w2_tri = convert_layout(wrap_torch_tensor(w2_tri, FP4), w_layout, **w_layout_opts) + w2_scale_tri = convert_layout( + wrap_torch_tensor(w2_scale_tri), + w_scale_layout, + **w_scale_layout_opts, + ) + + w1_precision_config = PrecisionConfig( + weight_scale=w1_scale_tri, flex_ctx=FlexCtx(rhs_data=InFlexData()) + ) + w2_precision_config = PrecisionConfig( + weight_scale=w2_scale_tri, flex_ctx=FlexCtx(rhs_data=InFlexData()) + ) + + return ( + w1, + w2, + w1_bias, + w2_bias, + w1_tri, + w2_tri, + w1_bias_tri, + w2_bias_tri, + w1_precision_config, + w2_precision_config, + ) + + +def swiglu(x, alpha: float = 1.702, limit: float = 1.0): + # Note we add an extra bias of 1 to the linear layer + x_glu, x_linear = torch.chunk(x, 2, dim=-1) + if limit is not None: + x_glu = x_glu.clamp(max=limit) + out_glu = x_glu * torch.sigmoid(alpha * x_glu) + if limit is not None: + x_linear = x_linear.clamp(min=-limit, max=limit) + return out_glu * (x_linear + 1) + + +def torch_moe_impl( + hidden_states: torch.Tensor, # (M, K) + w1: torch.Tensor, # (E, K, 2N) + w2: torch.Tensor, # (E, N, K) + w1_bias: torch.Tensor, # (E, 2N) + w2_bias: torch.Tensor, # (E, K) + topk_weights: torch.Tensor, # (M, topk) + topk_ids: torch.Tensor, # (M, topk) +): + w1 = w1[topk_ids, ...] + w1_bias = w1_bias[topk_ids, ...] + hidden_states = torch.einsum("bekc,bk->bec", w1, hidden_states) + w1_bias + hidden_states = swiglu(hidden_states, limit=7) + + w2 = w2[topk_ids, ...] + w2_bias = w2_bias[topk_ids, ...] + hidden_states = torch.einsum("bekc,bek->bec", w2, hidden_states) + w2_bias + + # Weighted sum of experts + hidden_states = torch.einsum("bec,be->bc", hidden_states, topk_weights) + return hidden_states + + +def oai_triton_moe_impl( + x: torch.Tensor, + w1: torch.Tensor, + w2: torch.Tensor, + w1_scale: "PrecisionConfig", + w2_scale: "PrecisionConfig", + w1_bias: torch.Tensor | None, + w2_bias: torch.Tensor | None, + num_experts: int, + topk_weights: torch.Tensor, + topk_ids: torch.Tensor, + unfused: bool = False, +) -> torch.Tensor: + quant_config = mxfp4_w4a16_moe_quant_config( + w1_bias=w1_bias, + w2_bias=w2_bias, + w1_scale=w1_scale, + w2_scale=w2_scale, + ) + moe_config = make_dummy_moe_config() + + if unfused: + fused_experts = UnfusedOAITritonExperts(moe_config, quant_config) + else: + fused_experts = OAITritonExperts(moe_config, quant_config) + + mk = FusedMoEKernel( + maybe_make_prepare_finalize( + moe=moe_config, + quant_config=quant_config, + allow_new_interface=True, + use_monolithic=False, + ), + fused_experts, + ) + + return mk.apply( + hidden_states=x, + w1=w1, + w2=w2, + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=MoEActivation.SWIGLUOAI, + global_num_experts=num_experts, + expert_map=None, + apply_router_weight_on_input=False, + ) + + +@pytest.mark.skipif( + not current_platform.is_cuda(), reason="This test is skipped on non-CUDA platform." +) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("m,n,k", MNK) +@pytest.mark.parametrize("num_experts", [32, 128]) +@pytest.mark.parametrize("topk", [4]) +@pytest.mark.parametrize("unfused", [True, False]) +def test_oai_triton_moe( + dtype: torch.dtype, + m: int, + n: int, + k: int, + num_experts: int, + topk: int, + unfused: bool, + workspace_init, +): + wait_for_gpu_memory_to_clear(devices=[0], threshold_ratio=0.1) + set_random_seed(0) + ( + w1, + w2, + w1_bias, + w2_bias, + w1_tri, + w2_tri, + w1_bias_tri, + w2_bias_tri, + w1_precision_config, + w2_precision_config, + ) = make_weights(dtype, k, n, num_experts) + + x = torch.randn((m, k), dtype=dtype, device="cuda") + router_logits = torch.randn(m, num_experts, device="cuda", dtype=dtype) + topk_weights, topk_ids = torch.topk(router_logits, k=topk, dim=-1, sorted=True) + topk_weights = torch.nn.functional.softmax(topk_weights, dim=-1) + + with set_current_vllm_config(VllmConfig()): + out_ref = torch_moe_impl(x, w1, w2, w1_bias, w2_bias, topk_weights, topk_ids) + + out = oai_triton_moe_impl( + x, + w1_tri, + w2_tri, + w1_precision_config, + w2_precision_config, + w1_bias_tri, + w2_bias_tri, + num_experts, + topk_weights, + topk_ids, + unfused, + ) + + assert_close(ref=out_ref, tri=out, maxtol=0.025, rmstol=0.005) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..80adceae6a72ab52fc1e536e034e45cf3d539597 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_moe.py @@ -0,0 +1,1696 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the MOE layers. + +Run `pytest tests/kernels/test_moe.py`. +""" + +import functools +from collections.abc import Callable +from dataclasses import dataclass +from typing import Any + +import pytest +import torch +from torch.nn import Parameter +from torch.nn import functional as F + +import vllm.model_executor.layers.fused_moe # noqa +from tests.kernels.moe.utils import ( + fused_moe, + make_dummy_moe_config, + modular_triton_fused_moe, +) +from tests.kernels.utils import opcheck, stack_and_dev, torch_experts, torch_moe +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.model_executor.layers.fused_moe import ( + MoEActivation, + fused_topk, +) +from vllm.model_executor.layers.fused_moe.config import ( + FUSED_MOE_UNQUANTIZED_CONFIG, + int4_w4a16_moe_quant_config, + int8_w8a16_moe_quant_config, +) +from vllm.model_executor.layers.fused_moe.experts.marlin_moe import ( + batched_fused_marlin_moe, + fused_marlin_moe, +) +from vllm.model_executor.layers.quantization.utils.marlin_utils import ( + marlin_permute_bias, +) +from vllm.model_executor.layers.quantization.utils.marlin_utils_fp4 import ( + rand_marlin_weight_mxfp4_like, + rand_marlin_weight_nvfp4_like, +) +from vllm.model_executor.layers.quantization.utils.marlin_utils_fp8 import ( + marlin_quant_fp8_torch, +) +from vllm.model_executor.layers.quantization.utils.marlin_utils_test import ( + awq_marlin_quantize, + marlin_quantize, +) +from vllm.model_executor.layers.quantization.utils.quant_utils import quantize_weights +from vllm.platforms import current_platform +from vllm.scalar_type import ScalarType, scalar_types +from vllm.utils.math_utils import next_power_of_2 +from vllm.utils.torch_utils import set_random_seed + + +def iterative_moe( + hidden_states: torch.Tensor, + w1: torch.Tensor, + w2: torch.Tensor, + gating_output: torch.Tensor, + topk: int, + global_num_experts: int, + expert_map: torch.Tensor = None, + renormalize: bool = False, +) -> torch.Tensor: + """ + Baseline implementation of fused moe. + + Args: + hidden_states: [*, hidden_size] + w1: [num_experts, intermediate_size * 2, hidden_size] + w2: [num_experts, hidden_size, intermediate_size] + gating_output: [*, num_experts] + expert_map: [num_experts] + """ + orig_shape = hidden_states.shape + hidden_size = hidden_states.shape[-1] + num_tokens = hidden_states.shape[:-1].numel() + num_experts = w1.shape[0] + intermediate_size = w2.shape[-1] + dtype = hidden_states.dtype + + hidden_states = hidden_states.view(num_tokens, hidden_size) + gating_output = gating_output.view(num_tokens, global_num_experts) + topk_weights = gating_output.softmax(dim=-1, dtype=torch.float) + topk_weights, selected_experts = topk_weights.topk(topk, dim=-1) + if renormalize: + topk_weights = topk_weights / topk_weights.sum(dim=-1, keepdim=True) + topk_weights = topk_weights.to(dtype) + + if expert_map is not None: + selected_experts = expert_map[selected_experts] + + final_hidden_states = None + for expert_idx in range(num_experts): + expert_w1 = w1[expert_idx] + expert_w2 = w2[expert_idx] + expert_mask = selected_experts == expert_idx + expert_weights = (topk_weights * expert_mask).sum(dim=-1, keepdim=True) + x = F.linear(hidden_states, expert_w1) + gate = F.silu(x[:, :intermediate_size]) + x = x[:, intermediate_size:] * gate + x = F.linear(x, expert_w2) + current_hidden_states = x * expert_weights + if final_hidden_states is None: + final_hidden_states = current_hidden_states + else: + final_hidden_states = final_hidden_states + current_hidden_states + + return final_hidden_states.view(orig_shape) # type: ignore + + +NUM_EXPERTS = [8, 64, 192] +NUM_EXPERTS_LARGE = [128, 256] +EP_SIZE = [1, 4] +TOP_KS = [2, 6] +TOP_KS_SMALL = [1, 2] + +MOE_MARLIN_QUANT_TEST_CONFIGS = [ + # AWQ-INT4 + {"b_type": scalar_types.uint4, "group_blocks": [-1, 2, 4, 8]}, + # GPTQ-INT4 + { + "b_type": scalar_types.uint4b8, + "support_act_order": True, + "group_blocks": [-1, 2, 4, 8], + }, + # GPTQ-INT8 + { + "b_type": scalar_types.uint8b128, + "support_act_order": True, + "group_blocks": [-1, 2, 4, 8], + }, + # FP8 + {"b_type": scalar_types.float8_e4m3fn, "group_blocks": [-1, 8]}, + # NVFP4 + {"b_type": scalar_types.float4_e2m1f, "group_blocks": [1]}, + # MXFP4 + { + "a_type": [scalar_types.bfloat16], + "b_type": scalar_types.float4_e2m1f, + "c_type": [scalar_types.bfloat16], + "group_blocks": [2], + }, + # MXFP8 + { + "a_type": [scalar_types.bfloat16], + "b_type": scalar_types.float8_e4m3fn, + "c_type": [scalar_types.bfloat16], + "group_blocks": [2], + }, + # AWQ-INT4 with INT8 activation + { + "a_type": [scalar_types.int8], + "b_type": scalar_types.uint4, + "group_blocks": [-1, 2, 4, 8], + }, + # GPTQ-INT4 with INT8 activation + { + "a_type": [scalar_types.int8], + "b_type": scalar_types.uint4b8, + "group_blocks": [-1, 2, 4, 8], + }, + # GPTQ-INT4 with FP8 activation + { + "a_type": [scalar_types.float8_e4m3fn], + "b_type": scalar_types.uint4b8, + "group_blocks": [-1, 2, 4, 8], + }, + # AWQ-INT4 with FP8 activation + { + "a_type": [scalar_types.float8_e4m3fn], + "b_type": scalar_types.uint4, + "group_blocks": [-1, 2, 4, 8], + }, + # MXFP4 with FP8 activation + { + "a_type": [scalar_types.float8_e4m3fn], + "b_type": scalar_types.float4_e2m1f, + "c_type": [scalar_types.bfloat16], + "group_blocks": [2], + }, +] + +FUSED_MOE_MNK_FACTORS = [ + (1, 128, 128), + (1, 2048, 128), + (33, 2048, 128), + (32768, 2048, 511), + (40000, 1024, 1024), +] + +FUSED_MOE_MNK_FACTORS_SMALL_M = [ + (1, 128, 128), + (1, 2048, 128), + (2, 2048, 128), + (2, 2048, 511), +] + +FUSED_MOE_WN16_MNK_FACTORS = [ + (1, 128, 128), + (1, 1024, 1024), + (32, 2048, 128), + (222, 2048, 1024), +] + +vllm_config = VllmConfig() + + +def run_moe_test( + baseline: Callable | torch.Tensor, + moe_fn: Callable, + a: torch.Tensor, + w1: torch.Tensor, + w2: torch.Tensor, + score: torch.Tensor, + topk: int, + global_num_experts: int = -1, + expert_map: torch.Tensor | None = None, + padding: bool = False, + use_compile: bool = False, + use_cudagraph: bool = False, + atol: float = 2e-2, + rtol: float = 0, +) -> torch.Tensor: + if isinstance(baseline, torch.Tensor): + baseline_output = baseline + else: + baseline_output = baseline( + a, + w1, + w2, + score, + topk, + global_num_experts=global_num_experts, + expert_map=expert_map, + ) + + # Pad the weight if moe padding is enabled + if padding: + w1 = F.pad(w1, (0, 128), "constant", 0)[..., 0:-128] + w2 = F.pad(w2, (0, 128), "constant", 0)[..., 0:-128] + + if use_compile: + moe_fn = torch.compile(moe_fn, backend="inductor", fullgraph=True) + torch._dynamo.mark_dynamic(a, 0) + torch._dynamo.mark_dynamic(score, 0) + + test_output = moe_fn( + a, + w1, + w2, + score, + topk, + global_num_experts=global_num_experts, + expert_map=expert_map, + ) + + if use_cudagraph: + test_output.fill_(0) + stream = torch.cuda.Stream() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=stream): + test_output = moe_fn( + a, + w1, + w2, + score, + topk, + global_num_experts=global_num_experts, + expert_map=expert_map, + ) + torch.accelerator.synchronize() + graph.replay() + torch.accelerator.synchronize() + + torch.testing.assert_close(test_output, baseline_output, atol=atol, rtol=rtol) + + return baseline_output + + +@pytest.mark.parametrize("m,n,k", FUSED_MOE_MNK_FACTORS) +@pytest.mark.parametrize("e", NUM_EXPERTS) +@pytest.mark.parametrize("topk", TOP_KS) +@pytest.mark.parametrize("ep_size", EP_SIZE) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("padding", [True, False]) +def test_fused_moe( + m: int, + n: int, + k: int, + e: int, + topk: int, + ep_size: int, + dtype: torch.dtype, + padding: bool, + monkeypatch, + workspace_init, +): + set_random_seed(7) + + # + # Setup test data + # + + # + # Setup test data + # + + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + w1 = torch.randn((e, 2 * n, k), device="cuda", dtype=dtype) / 10 + w2 = torch.randn((e, k, n), device="cuda", dtype=dtype) / 10 + + score = torch.randn((m, e), device="cuda", dtype=dtype) + + if ep_size > 1: + local_e = e // ep_size + e_ids = torch.randint(0, e, (local_e,), device="cuda", dtype=torch.int32) + e_map = torch.full((e,), -1, device="cuda", dtype=torch.int32) + e_map[e_ids] = torch.arange(local_e, device="cuda", dtype=torch.int32) + w1 = w1[e_ids] + w2 = w2[e_ids] + else: + e_map = None + + # + # Setup test functions + # + quant_config = FUSED_MOE_UNQUANTIZED_CONFIG + + m_fused_moe_fn = modular_triton_fused_moe(make_dummy_moe_config(), quant_config) + + def m_fused_moe( + a: torch.Tensor, + w1: torch.Tensor, + w2: torch.Tensor, + score: torch.Tensor, + topk: int, + global_num_experts: int = -1, + expert_map: torch.Tensor | None = None, + ) -> torch.Tensor: + topk_weights, topk_ids, _ = fused_topk(a, score, topk, False) + return m_fused_moe_fn.apply( + a, + w1, + w2, + topk_weights, + topk_ids, + activation=MoEActivation.SILU, + global_num_experts=global_num_experts, + expert_map=expert_map, + apply_router_weight_on_input=False, + ) + + fused_moe_fn = functools.partial(fused_moe, renormalize=False) + + # + # Run tests + # + runner = functools.partial( + run_moe_test, + a=a, + w1=w1, + w2=w2, + score=score, + topk=topk, + global_num_experts=e, + expert_map=e_map, + padding=padding, + ) + + # Note: for now use_compile will error out if the problem size is + # large enough to trigger chunking. I'm leaving the flag and + # setup code in case we are able to revisit this later. + use_compile = False + + use_cudagraph = n >= 1024 and k >= 1024 and current_platform.is_cuda_alike() + + with set_current_vllm_config(vllm_config): + baseline_output = runner(torch_moe, iterative_moe) + runner( + baseline_output, + fused_moe_fn, + use_compile=use_compile, + use_cudagraph=use_cudagraph, + ) + runner( + baseline_output, + m_fused_moe, + use_compile=use_compile, + use_cudagraph=use_cudagraph, + ) + + +def test_fused_moe_int64_overflow(workspace_init): + """Regression test for int32 overflow in stride*offset products. + + With large M, stride_cm * offs_token can exceed int32 max. Verifies + the offs_token int64 cast (fix for #34413) prevents overflow and + produces correct results. + + Reproduces the scenario from PR #34279. + """ + # ~12 GB GPU memory needed for intermediate caches + free_mem = torch.accelerator.get_memory_info()[0] + if free_mem < 12 * 1024**3: + pytest.skip("Insufficient GPU memory for overflow test") + + set_random_seed(7) + + m, n, k, e, topk = 100000, 2048, 1024, 8, 6 + dtype = torch.bfloat16 + + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + w1 = torch.randn((e, 2 * n, k), device="cuda", dtype=dtype) / 10 + w2 = torch.randn((e, k, n), device="cuda", dtype=dtype) / 10 + score = torch.randn((m, e), device="cuda", dtype=dtype) + + # Verify the test exercises the overflow condition: + # C has shape (M, topk, N) where N = w1.size(1) = 2*n + # stride_cm = C.stride(1) = N, max offs_token = M * topk + # Product must exceed int32 max for this test to be meaningful + N = w1.size(1) + assert N * m * topk > 2**31 - 1, "Test params don't trigger int32 overflow" + + fused_moe_fn = functools.partial(fused_moe, renormalize=False) + + with set_current_vllm_config(vllm_config): + run_moe_test( + torch_moe, + fused_moe_fn, + a=a, + w1=w1, + w2=w2, + score=score, + topk=topk, + global_num_experts=e, + ) + + +@pytest.mark.parametrize("m,n,k", FUSED_MOE_MNK_FACTORS_SMALL_M) +@pytest.mark.parametrize("e", NUM_EXPERTS_LARGE) +@pytest.mark.parametrize("topk", TOP_KS_SMALL) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("padding", [True, False]) +def test_naive_block_assignment_moe( + m: int, + n: int, + k: int, + e: int, + topk: int, + dtype: torch.dtype, + padding: bool, + monkeypatch, + workspace_init, +): + set_random_seed(7) + + # + # Setup test data + # + + # + # Setup test data + # + + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + w1 = torch.randn((e, 2 * n, k), device="cuda", dtype=dtype) / 10 + w2 = torch.randn((e, k, n), device="cuda", dtype=dtype) / 10 + + score = torch.randn((m, e), device="cuda", dtype=dtype) + + e_map = None + + # + # Setup test functions + # + quant_config = FUSED_MOE_UNQUANTIZED_CONFIG + + m_fused_moe_fn = modular_triton_fused_moe(make_dummy_moe_config(), quant_config) + + def m_fused_moe( + a: torch.Tensor, + w1: torch.Tensor, + w2: torch.Tensor, + score: torch.Tensor, + topk: int, + global_num_experts: int = -1, + expert_map: torch.Tensor | None = None, + ) -> torch.Tensor: + topk_weights, topk_ids, _ = fused_topk(a, score, topk, False) + return m_fused_moe_fn.apply( + a, + w1, + w2, + topk_weights, + topk_ids, + activation=MoEActivation.SILU, + global_num_experts=global_num_experts, + expert_map=expert_map, + apply_router_weight_on_input=False, + ) + + fused_moe_fn = functools.partial(fused_moe, renormalize=False) + + # + # Run tests + # + runner = functools.partial( + run_moe_test, + a=a, + w1=w1, + w2=w2, + score=score, + topk=topk, + global_num_experts=e, + expert_map=e_map, + padding=padding, + ) + + # Note: for now use_compile will error out if the problem size is + # large enough to trigger chunking. I'm leaving the flag and + # setup code in case we are able to revisit this later. + use_compile = False + + use_cudagraph = n >= 1024 and k >= 1024 and current_platform.is_cuda_alike() + + with set_current_vllm_config(vllm_config): + baseline_output = runner(torch_moe, iterative_moe) + runner( + baseline_output, + fused_moe_fn, + use_compile=use_compile, + use_cudagraph=use_cudagraph, + ) + runner( + baseline_output, + m_fused_moe, + use_compile=use_compile, + use_cudagraph=use_cudagraph, + ) + + +@pytest.mark.parametrize("m,n,k", FUSED_MOE_WN16_MNK_FACTORS) +@pytest.mark.parametrize("e", NUM_EXPERTS) +@pytest.mark.parametrize("topk", TOP_KS) +@pytest.mark.parametrize("ep_size", EP_SIZE) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("group_size", [64, 128]) +@pytest.mark.parametrize("has_zp", [True, False]) +@pytest.mark.parametrize("weight_bits", [4, 8]) +def test_fused_moe_wn16( + m: int, + n: int, + k: int, + e: int, + topk: int, + ep_size: int, + dtype: torch.dtype, + group_size: int, + has_zp: bool, + weight_bits: int, +): + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + w1 = torch.randn((e, 2 * n, k), device="cuda", dtype=dtype) / 10 + w2 = torch.randn((e, k, n), device="cuda", dtype=dtype) / 10 + score = torch.randn((m, e), device="cuda", dtype=dtype) + + if weight_bits == 4: + pack_factor = 2 + quant_type = scalar_types.uint4 if has_zp else scalar_types.uint4b8 + elif weight_bits == 8: + pack_factor = 1 + quant_type = scalar_types.uint8 if has_zp else scalar_types.uint8b128 + + w1_ref = w1.clone() + w2_ref = w2.clone() + w1_qweight = torch.empty( + (e, 2 * n, k // pack_factor), device="cuda", dtype=torch.uint8 + ) + w2_qweight = torch.empty((e, k, n // pack_factor), device="cuda", dtype=torch.uint8) + w1_scales = torch.empty((e, 2 * n, k // group_size), device="cuda", dtype=dtype) + w2_scales = torch.empty((e, k, n // group_size), device="cuda", dtype=dtype) + w1_qzeros = torch.empty( + (e, 2 * n // pack_factor, k // group_size), device="cuda", dtype=torch.uint8 + ) + w2_qzeros = torch.empty( + (e, k // pack_factor, n // group_size), device="cuda", dtype=torch.uint8 + ) + + for i in range(e * 2): + expert_id = i % e + if i // e == 0: + w, w_ref, w_qweight, w_scales, w_qzeros = ( + w1, + w1_ref, + w1_qweight, + w1_scales, + w1_qzeros, + ) + else: + w, w_ref, w_qweight, w_scales, w_qzeros = ( + w2, + w2_ref, + w2_qweight, + w2_scales, + w2_qzeros, + ) + weight, qweight, scales, qzeros = quantize_weights( + w[expert_id].T, quant_type, group_size, has_zp, False + ) + weight = weight.T + qweight = qweight.T.contiguous().to(torch.uint8) + scales = scales.T + if has_zp: + qzeros = qzeros.T.contiguous().to(torch.uint8) + if weight_bits == 4: + qweight = qweight[:, 1::2] * 16 + qweight[:, ::2] + if has_zp: + qzeros = qzeros[1::2, :] * 16 + qzeros[::2, :] + + w_ref[expert_id] = weight + w_qweight[expert_id] = qweight + w_scales[expert_id] = scales + if has_zp: + w_qzeros[expert_id] = qzeros + + if ep_size > 1: + local_e = e // ep_size + e_ids = torch.randint(0, e, (local_e,), device="cuda", dtype=torch.int32) + e_map = torch.full((e,), -1, device="cuda", dtype=torch.int32) + e_map[e_ids] = torch.arange(local_e, device="cuda", dtype=torch.int32) + w1_ref = w1_ref[e_ids] + w2_ref = w2_ref[e_ids] + w1_qweight = w1_qweight[e_ids] + w2_qweight = w2_qweight[e_ids] + w1_scales = w1_scales[e_ids] + w2_scales = w2_scales[e_ids] + w1_qzeros = w1_qzeros[e_ids] + w2_qzeros = w2_qzeros[e_ids] + else: + e_map = None + + if weight_bits == 4: + quant_config_builder = int4_w4a16_moe_quant_config + else: + assert weight_bits == 8 + quant_config_builder = int8_w8a16_moe_quant_config + + quant_config = quant_config_builder( + w1_scale=w1_scales, + w2_scale=w2_scales, + w1_zp=w1_qzeros if has_zp else None, + w2_zp=w2_qzeros if has_zp else None, + block_shape=[0, group_size], + ) + + with set_current_vllm_config(vllm_config): + triton_output = fused_moe( + a, + w1_qweight, + w2_qweight, + score, + topk, + renormalize=False, + global_num_experts=e, + expert_map=e_map, + quant_config=quant_config, + ) + torch_output = torch_moe(a, w1_ref, w2_ref, score, topk, expert_map=e_map) + + torch.testing.assert_close(triton_output, torch_output, atol=2e-2, rtol=0) + + +MARLIN_MOE_SCENARIOS = [ + # (m, n, k, e, topk, ep_size, act_order, is_k_full) + # No act_order: is_k_full=True matches usual case (marlin_is_k_full). + # N>=256 required for Marlin kernel thread config for MXFP8. + # Single token, small matrices + (1, 128, 256, 5, 2, 1, False, True), + # Single token, large matrices + (1, 1024, 2048, 5, 2, 1, False, True), + # Unaligned m, small matrices + (133, 256, 256, 5, 2, 1, False, True), + # Unaligned m, large matrices + (133, 1024, 2048, 12, 3, 1, False, True), + # Aligned batch, small matrices + (128, 256, 256, 5, 2, 1, False, True), + # Aligned batch, large matrices + (128, 1024, 2048, 12, 3, 1, False, True), + # Expert parallelism + (64, 1024, 2048, 12, 3, 4, False, True), + # Act order with is_k_full=True (no tensor parallelism) + (1, 1024, 2048, 5, 2, 1, True, True), + # Act order with is_k_full=False (tensor parallelism) + (133, 256, 256, 5, 2, 1, True, False), +] + + +def marlin_moe_generate_valid_test_cases(): + import itertools + + def is_valid( + a_type, + b_type, + c_type, + group_blocks, + m, + n, + k, + e, + topk, + ep_size, + act_order, + is_k_full, + ): + group_size = group_blocks if group_blocks <= 0 else group_blocks * 16 + if group_size > 0 and k % group_size != 0: + return False + if act_order and group_size in [-1, k, n]: + return False + if group_size in [k, n]: + return False + if b_type == scalar_types.float8_e4m3fn and group_size == 32 and is_k_full: + return False + return a_type.size_bits < 16 or a_type is c_type + + cases = [] + for quant_test_config in MOE_MARLIN_QUANT_TEST_CONFIGS: + f16_types = [scalar_types.float16] + inner_combinations = list( + itertools.product( + quant_test_config.get("a_type", f16_types), + [quant_test_config["b_type"]], + quant_test_config.get("c_type", f16_types), + quant_test_config["group_blocks"], + ) + ) + + supports_act_order = quant_test_config.get("support_act_order", False) + + for sub_case in inner_combinations: + if ( + sub_case[0] == scalar_types.float8_e4m3fn + and not current_platform.is_device_capability(89) + and not current_platform.is_device_capability_family(120) + ): + continue + + for scenario in MARLIN_MOE_SCENARIOS: + m, n, k, e, topk, ep_size, act_order, is_k_full = scenario + if act_order and not supports_act_order: + continue + args = sub_case + (m, n, k, e, topk, ep_size, act_order, is_k_full) + if is_valid(*args): + cases.append(args) + return cases + + +@dataclass +class MarlinMoEWeightData: + w_ref: torch.Tensor + qweight: torch.Tensor + scales: torch.Tensor + global_scale: torch.Tensor | None + a_scales_factor: torch.Tensor | None + g_idx: torch.Tensor | None + zeros: torch.Tensor | None + sort_indices: torch.Tensor | None + marlin_bias: torch.Tensor | None + + @staticmethod + def make( + w: torch.Tensor, + quant_type: ScalarType, + group_size: int, + act_order: bool | None = None, + bias: torch.Tensor | None = None, + input_type: ScalarType = None, + ) -> "MarlinMoEWeightData": + assert w.ndim == 3 + + has_zp = quant_type in [scalar_types.uint4, scalar_types.uint8] + k = w.shape[-1] + + if input_type == scalar_types.int8: + input_dtype = torch.int8 + elif input_type == scalar_types.float8_e4m3fn: + input_dtype = torch.float8_e4m3fn + else: + input_dtype = w.dtype + + w_ref_l: list[torch.Tensor] = [] + qweight_l: list[torch.Tensor] = [] + scales_l: list[torch.Tensor] = [] + global_scale_l: list[torch.Tensor] = [] + zeros_l: list[torch.Tensor] = [] + g_idx_l: list[torch.Tensor] = [] + sort_indices_l: list[torch.Tensor] = [] + bias_l: list[torch.Tensor] = [] + + for i in range(w.shape[0]): + if quant_type == scalar_types.float4_e2m1f: + if group_size == 16: + w_ref, qweight, scales, global_scale = ( + rand_marlin_weight_nvfp4_like( + w[i], group_size, input_dtype=input_dtype + ) + ) + else: + w_ref, qweight, scales = rand_marlin_weight_mxfp4_like( + w[i], group_size, input_dtype=input_dtype + ) + global_scale = None + + w_ref_l.append(w_ref.T) + qweight_l.append(qweight) + scales_l.append(scales) + if global_scale is not None: + global_scale_l.append(global_scale) + elif quant_type == scalar_types.float8_e4m3fn: + w_ref, qweight, scales = marlin_quant_fp8_torch( + w[i], group_size, input_dtype=input_dtype + ) + w_ref_l.append(w_ref.T) + qweight_l.append(qweight) + scales_l.append(scales) + elif has_zp: + w_ref, qweight, scales, zeros = awq_marlin_quantize( + w[i].transpose(1, 0), + quant_type, + group_size, + input_dtype=input_dtype, + ) + + w_ref_l.append(w_ref.T) + qweight_l.append(qweight) + scales_l.append(scales) + zeros_l.append(zeros) + else: + test_perm = torch.randperm(k) + w_ref, qweight, scales, g_idx, sort_indices, _ = marlin_quantize( + w[i].transpose(1, 0), + quant_type, + group_size, + act_order, + test_perm, + input_dtype=input_dtype, + ) + + w_ref_l.append(w_ref.T) + qweight_l.append(qweight) + scales_l.append(scales) + g_idx_l.append(g_idx) + sort_indices_l.append(sort_indices) + + if bias is not None: + bias_l.append(marlin_permute_bias(bias[i])) + + w_ref = stack_and_dev(w_ref_l) + qweight = stack_and_dev(qweight_l).contiguous() + scales = stack_and_dev(scales_l) + global_scale = stack_and_dev(global_scale_l) if global_scale_l else None + g_idx = stack_and_dev(g_idx_l) if g_idx_l else None + zeros = stack_and_dev(zeros_l) if zeros_l else None + sort_indices = stack_and_dev(sort_indices_l) if sort_indices_l else None + marlin_bias = stack_and_dev(bias_l) if bias_l else None + + a_scales_factor = None + if input_type == scalar_types.int8 and group_size != -1: + a_scales_factor = 1 / 4096 * scales.max().float() + scales = scales / scales.max() * 4096 + scales = scales.round().to(torch.int16).view(w.dtype) + + return MarlinMoEWeightData( + w_ref=w_ref, + qweight=qweight, + scales=scales, + global_scale=global_scale, + a_scales_factor=a_scales_factor, + g_idx=g_idx, + zeros=zeros, + sort_indices=sort_indices, + marlin_bias=marlin_bias, + ) + + +@pytest.mark.flaky(reruns=2) +@pytest.mark.parametrize( + ( + "a_type, b_type, c_type, group_blocks," + "m, n, k, e, topk, ep_size, act_order, is_k_full" + ), + marlin_moe_generate_valid_test_cases(), +) +@pytest.mark.skipif(current_platform.is_rocm(), reason="Skip for rocm") +@pytest.mark.usefixtures("default_vllm_config") +def test_fused_marlin_moe( + a_type: ScalarType, + b_type: ScalarType, + c_type: ScalarType, + group_blocks: int, + m: int, + n: int, + k: int, + e: int, + topk: int, + ep_size: int, + act_order: bool, + is_k_full: bool, +): + set_random_seed(1) + group_size = group_blocks if group_blocks <= 0 else group_blocks * 16 + + if c_type == scalar_types.float16: + dtype = torch.float16 + elif c_type == scalar_types.bfloat16: + dtype = torch.bfloat16 + else: + raise RuntimeError("unsupported c_type") + + if a_type == scalar_types.int8: + a_dtype = torch.int8 + elif a_type == scalar_types.float8_e4m3fn: + a_dtype = torch.float8_e4m3fn + else: + a_dtype = dtype + + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + w1 = torch.randn((e, 2 * n, k), device="cuda", dtype=dtype) / 10 + w2 = torch.randn((e, k, n), device="cuda", dtype=dtype) / 10 + + if ep_size > 1: + local_e = e // ep_size + e_ids = torch.randperm(e, device="cuda", dtype=torch.int32)[:local_e] + e_map = torch.full((e,), -1, device="cuda", dtype=torch.int32) + e_map[e_ids] = torch.arange(local_e, device="cuda", dtype=torch.int32) + w1 = w1[e_ids] + w2 = w2[e_ids] + else: + e_map = None + + w1_data = MarlinMoEWeightData.make( + w=w1, + quant_type=b_type, + group_size=group_size, + act_order=act_order, + input_type=a_type, + ) + + w2_data = MarlinMoEWeightData.make( + w=w2, + quant_type=b_type, + group_size=group_size, + act_order=act_order, + input_type=a_type, + ) + + score = torch.randn((m, e), device="cuda", dtype=dtype) + + topk_weights, topk_ids, _ = fused_topk(a, score, topk, False) + + with set_current_vllm_config(vllm_config): + score = torch.softmax(score, dim=-1, dtype=torch.float32) + topk_weight, topk_ids = torch.topk(score, topk) + torch_output = torch_experts( + a, + w1_data.w_ref, + w2_data.w_ref, + topk_weight=topk_weight, + topk_ids=topk_ids, + global_num_experts=e, + expert_map=e_map, + quant_dtype=a_dtype, + per_act_token_quant=True, + ) + + marlin_output = fused_marlin_moe( + a, + w1_data.qweight, + w2_data.qweight, + None, + None, + w1_data.scales, + w2_data.scales, + topk_weights, + topk_ids, + global_num_experts=e, + expert_map=e_map, + global_scale1=w1_data.global_scale, + global_scale2=w2_data.global_scale, + g_idx1=w1_data.g_idx, + g_idx2=w2_data.g_idx, + input_global_scale1=w1_data.a_scales_factor, + input_global_scale2=w2_data.a_scales_factor, + sort_indices1=w1_data.sort_indices, + sort_indices2=w2_data.sort_indices, + w1_zeros=w1_data.zeros, + w2_zeros=w2_data.zeros, + input_dtype=a_dtype, + quant_type_id=b_type.id, + is_k_full=is_k_full, + ) + + torch.testing.assert_close(marlin_output, torch_output, atol=4e-2, rtol=0) + + +@pytest.mark.flaky(reruns=2) +@pytest.mark.skipif(current_platform.is_rocm(), reason="Skip for rocm") +@pytest.mark.usefixtures("default_vllm_config") +@pytest.mark.parametrize("m", [1, 256]) +def test_fused_marlin_moe_with_bias(m): + set_random_seed(0) + + e, topk = 32, 4 + n, k = 2048, 2048 + group_size = 128 + act_order = False + is_k_full = True + quant_type = scalar_types.uint4b8 + dtype = torch.half + + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + w1 = torch.randn((e, 2 * n, k), device="cuda", dtype=dtype) / 10 + w2 = torch.randn((e, k, n), device="cuda", dtype=dtype) / 10 + b_bias1 = torch.randn((e, 2 * n), device="cuda", dtype=dtype) / 10 + b_bias2 = torch.randn((e, k), device="cuda", dtype=dtype) / 10 + + w1_data = MarlinMoEWeightData.make( + w=w1, + quant_type=quant_type, + group_size=group_size, + act_order=act_order, + bias=b_bias1, + ) + + w2_data = MarlinMoEWeightData.make( + w=w2, + quant_type=quant_type, + group_size=group_size, + act_order=act_order, + bias=b_bias2, + ) + + score = torch.randn((m, e), device="cuda", dtype=dtype) + + topk_weights, topk_ids, _ = fused_topk(a, score, topk, False) + + with set_current_vllm_config(vllm_config): + torch_output = torch_moe( + a, w1_data.w_ref, w2_data.w_ref, score, topk, b_bias1, b_bias2 + ) + + marlin_output = fused_marlin_moe( + a, + w1_data.qweight, + w2_data.qweight, + w1_data.marlin_bias, + w2_data.marlin_bias, + w1_data.scales, + w2_data.scales, + topk_weights, + topk_ids, + global_num_experts=e, + expert_map=None, + global_scale1=w1_data.global_scale, + global_scale2=w2_data.global_scale, + g_idx1=w1_data.g_idx, + g_idx2=w2_data.g_idx, + sort_indices1=w1_data.sort_indices, + sort_indices2=w2_data.sort_indices, + w1_zeros=w1_data.zeros, + w2_zeros=w2_data.zeros, + quant_type_id=quant_type.id, + is_k_full=is_k_full, + ) + + torch.testing.assert_close(marlin_output, torch_output, atol=5e-2, rtol=0) + + +@pytest.mark.flaky(reruns=2) +@pytest.mark.skipif(current_platform.is_rocm(), reason="Skip for rocm") +@pytest.mark.usefixtures("default_vllm_config") +@pytest.mark.parametrize("m", [1, 64, 256]) +@pytest.mark.parametrize("n,k", [(1024, 1024), (2048, 2048)]) +@pytest.mark.parametrize("e,topk", [(8, 2), (64, 4)]) +@pytest.mark.parametrize("activation", [MoEActivation.RELU2_NO_MUL]) +def test_fused_marlin_moe_non_gated( + m: int, n: int, k: int, e: int, topk: int, activation: MoEActivation +): + """Test Marlin MoE with non-gated activation (relu2_no_mul). + + Non-gated activations like relu2 don't have the gate-up projection pattern, + so w1 has shape (e, n, k) instead of (e, 2*n, k). + """ + set_random_seed(42) + + group_size = 16 # NVFP4 group size + is_k_full = True + quant_type = scalar_types.float4_e2m1f + dtype = torch.bfloat16 + + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + # Non-gated: w1 shape is (e, n, k) not (e, 2*n, k) + w1 = torch.randn((e, n, k), device="cuda", dtype=dtype) / 10 + w2 = torch.randn((e, k, n), device="cuda", dtype=dtype) / 10 + + w1_data = MarlinMoEWeightData.make( + w=w1, + quant_type=quant_type, + group_size=group_size, + act_order=False, + ) + + w2_data = MarlinMoEWeightData.make( + w=w2, + quant_type=quant_type, + group_size=group_size, + act_order=False, + ) + + score = torch.randn((m, e), device="cuda", dtype=dtype) + + topk_weights, topk_ids, _ = fused_topk(a, score, topk, False) + + with set_current_vllm_config(vllm_config): + torch_output = torch_moe( + a, + w1_data.w_ref, + w2_data.w_ref, + score, + topk, + activation=activation, + ) + + marlin_output = fused_marlin_moe( + a, + w1_data.qweight, + w2_data.qweight, + None, # bias1 + None, # bias2 + w1_data.scales, + w2_data.scales, + topk_weights, + topk_ids, + global_num_experts=e, + expert_map=None, + global_scale1=w1_data.global_scale, + global_scale2=w2_data.global_scale, + g_idx1=w1_data.g_idx, + g_idx2=w2_data.g_idx, + sort_indices1=w1_data.sort_indices, + sort_indices2=w2_data.sort_indices, + w1_zeros=w1_data.zeros, + w2_zeros=w2_data.zeros, + quant_type_id=quant_type.id, + is_k_full=is_k_full, + activation=activation, + ) + + torch.testing.assert_close(marlin_output, torch_output, atol=1e-1, rtol=0) + + +@pytest.mark.parametrize("ep_size", [1, 2]) +def test_moe_align_block_size_opcheck(ep_size): + num_experts = 4 + block_size = 4 + + expert_map = None + if ep_size != 1: + local_num_experts = num_experts // ep_size + expert_ids = torch.randint( + 0, num_experts, (local_num_experts,), device="cuda", dtype=torch.int32 + ) + expert_map = torch.full((num_experts,), -1, device="cuda", dtype=torch.int32) + expert_map[expert_ids] = torch.arange( + local_num_experts, device="cuda", dtype=torch.int32 + ) + + topk_ids = torch.randint(0, num_experts, (3, 4), dtype=torch.int32, device="cuda") + + max_num_tokens_padded = topk_ids.numel() + num_experts * (block_size - 1) + sorted_ids = torch.empty( + (max_num_tokens_padded,), dtype=torch.int32, device=topk_ids.device + ) + sorted_ids.fill_(topk_ids.numel()) + max_num_m_blocks = max_num_tokens_padded // block_size + expert_ids = torch.empty( + (max_num_m_blocks,), dtype=torch.int32, device=topk_ids.device + ) + num_tokens_post_pad = torch.empty((1), dtype=torch.int32, device=topk_ids.device) + + opcheck( + torch.ops._moe_C.moe_align_block_size, + ( + topk_ids, + num_experts, + block_size, + sorted_ids, + expert_ids, + num_tokens_post_pad, + expert_map, + ), + ) + + +def test_batched_moe_align_block_size_opcheck(): + max_tokens_per_batch = 512 + num_experts = 4 + block_size = 16 + + expert_num_tokens = torch.randint( + low=0, + high=max_tokens_per_batch, + size=(num_experts,), + dtype=torch.int32, + device="cuda", + ) + + max_num_tokens_padded = num_experts * max(max_tokens_per_batch, block_size) + sorted_ids = torch.empty((max_num_tokens_padded,), dtype=torch.int32, device="cuda") + + assert max_num_tokens_padded % block_size == 0 + max_num_m_blocks = max_num_tokens_padded // block_size + expert_ids = torch.empty((max_num_m_blocks,), dtype=torch.int32, device="cuda") + + num_tokens_post_pad = torch.empty((1), dtype=torch.int32, device="cuda") + + opcheck( + torch.ops._moe_C.batched_moe_align_block_size, + ( + max_tokens_per_batch, + block_size, + expert_num_tokens, + sorted_ids, + expert_ids, + num_tokens_post_pad, + ), + ) + + +# topk=8 covers topk > 4; k=511 covers the non-vectorized scalar path. The +# layouts exercise contiguous input plus the two non-contiguous cases: a +# transpose (strided hidden -> scalar gather) and a topk-slice (hidden still +# contiguous -> vectorized). +@pytest.mark.parametrize("m", [1, 33, 222]) +@pytest.mark.parametrize("topk", [*TOP_KS, 8]) +@pytest.mark.parametrize("k", [128, 511, 1024]) +@pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16]) +@pytest.mark.parametrize("layout", ["contig", "transpose", "slice"]) +def test_moe_sum(m: int, topk: int, k: int, dtype: torch.dtype, layout: str): + if layout == "transpose": + input = torch.randn((m, k, topk), device="cuda", dtype=dtype).transpose(1, 2) + elif layout == "slice": + input = torch.randn((m, 2 * topk, k), device="cuda", dtype=dtype)[:, ::2, :] + else: + input = torch.randn((m, topk, k), device="cuda", dtype=dtype) + assert input.is_contiguous() == (layout == "contig") + actual = torch.empty((m, k), device="cuda", dtype=dtype) + + # Reduction accumulates in fp32. + expected = input.float().sum(dim=1).to(dtype) + torch.ops._moe_C.moe_sum(input, actual) + + torch.testing.assert_close(actual, expected, atol=2e-2, rtol=0) + + opcheck(torch.ops._moe_C.moe_sum, (input, actual)) + + +@pytest.mark.usefixtures("default_vllm_config") +@pytest.mark.parametrize("m", [1, 33]) +@pytest.mark.parametrize("n,k", [(128, 128)]) +@pytest.mark.parametrize("e", [8]) +@pytest.mark.parametrize("topk", [2]) +@pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16]) +@pytest.mark.parametrize("with_bias", [False, True]) +@pytest.mark.parametrize("activation", [MoEActivation.SILU]) +@pytest.mark.skipif(not current_platform.is_cpu(), reason="CPU only test") +def test_cpu_fused_moe_basic( + m: int, + n: int, + k: int, + e: int, + topk: int, + dtype: torch.dtype, + with_bias: bool, + activation: MoEActivation, +): + from vllm.model_executor.layers.fused_moe.cpu_fused_moe import CPUFusedMOE + + device = "cpu" + set_random_seed(7) + + a = torch.randn((m, k), device=device, dtype=dtype) / 10 + w13 = torch.randn((e, 2 * n, k), device=device, dtype=dtype) / 10 + w2 = torch.randn((e, k, n), device=device, dtype=dtype) / 10 + router_logits = torch.randn((m, e), device=device, dtype=dtype) + + b1 = b2 = None + if with_bias: + b1 = torch.randn((e, 2 * n), device=device, dtype=dtype) / 10 + b2 = torch.randn((e, k), device=device, dtype=dtype) / 10 + + ref = ( + torch_moe(a, w13, w2, router_logits, topk, b1, b2) + if with_bias + else torch_moe(a, w13, w2, router_logits, topk) + ) + + class _Dummy(torch.nn.Module): + def __init__(self, w13, w2, b1=None, b2=None): + super().__init__() + self.w13_weight = torch.nn.Parameter(w13, requires_grad=False) + self.w2_weight = torch.nn.Parameter(w2, requires_grad=False) + if b1 is not None: + self.w13_bias = torch.nn.Parameter(b1, requires_grad=False) + if b2 is not None: + self.w2_bias = torch.nn.Parameter(b2, requires_grad=False) + + layer = _Dummy(w13, w2, b1, b2).to(dtype) + fused = CPUFusedMOE(layer) + out = fused( + layer=layer, + x=a, + use_grouped_topk=False, + top_k=topk, + router_logits=router_logits, + renormalize=False, + global_num_experts=e, + expert_map=None, + custom_routing_function=None, + scoring_func="softmax", + routed_scaling_factor=1.0, + e_score_correction_bias=None, + apply_router_weight_on_input=False, + activation=activation, + ) + + # Tolerances: fp32 tight; bf16 looser (esp. with bias) + if dtype == torch.float32: + atol = 1e-3 + elif with_bias: + atol = 8e-2 + else: + atol = 5e-2 + torch.testing.assert_close(out, ref, atol=atol, rtol=0) + + +def _batched_fused_marlin_moe_cases() -> list[Any]: + cases = [ + pytest.param( + m, + 128, + 128, + e, + topk, + max_tokens_per_batch, + torch.bfloat16, + scalar_types.float4_e2m1f, + None, + 1e-3, + id=( + f"m{m}-n128-k128-e{e}-topk{topk}-max_tokens{max_tokens_per_batch}-mxfp4" + ), + ) + for m in [16, 32, 64] + for e in [8, 12, 16, 32] + for topk in [2, 4] + for max_tokens_per_batch in [16, 32, 64] + ] + cases.append( + pytest.param( + 32, + 128, + 128, + 8, + 2, + 64, + torch.float16, + scalar_types.uint4, + scalar_types.int8, + 4e-2, + id="awq-int8-activation-metadata", + ) + ) + return cases + + +@pytest.mark.parametrize( + ("m,n,k,e,topk,max_tokens_per_batch,dtype,quant_dtype,input_type,atol"), + _batched_fused_marlin_moe_cases(), +) +@pytest.mark.skipif(current_platform.is_rocm(), reason="Skip for rocm") +def test_batched_fused_marlin_moe( + m: int, + n: int, + k: int, + e: int, + topk: int, + max_tokens_per_batch: int, + dtype: torch.dtype, + quant_dtype: ScalarType, + input_type: ScalarType | None, + atol: float, +): + print( + f"testing m={m}, n={n}, k={k}, e={e}, " + f"topk={topk}, " + f"max_tokens_per_batch={max_tokens_per_batch}, " + f"dtype={dtype}, quant_dtype={quant_dtype}, input_type={input_type}" + ) + set_random_seed(0) + + group_size = 32 + if input_type == scalar_types.int8: + input_dtype = torch.int8 + elif input_type == scalar_types.float8_e4m3fn: + input_dtype = torch.float8_e4m3fn + else: + input_dtype = None + + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + w1 = torch.randn((e, 2 * n, k), device="cuda", dtype=dtype) / 20 + w2 = torch.randn((e, k, n), device="cuda", dtype=dtype) / 20 + + w1_data = MarlinMoEWeightData.make( + w=w1, + quant_type=quant_dtype, + group_size=group_size, + act_order=None, + input_type=input_type, + ) + w2_data = MarlinMoEWeightData.make( + w=w2, + quant_type=quant_dtype, + group_size=group_size, + act_order=None, + input_type=input_type, + ) + + score = torch.randn((m, e), device="cuda", dtype=dtype) + topk_weights, topk_ids, _ = fused_topk(a, score, topk, False) + + class BatchedRun: + @staticmethod + def _make_expert_num_tokens_cpu( + e: int, # num_experts + topk_ids_cpu: torch.Tensor, + ) -> torch.Tensor: + expert_num_tokens_cpu = torch.zeros((e,), dtype=torch.int32, device="cpu") + for topk_id in torch.flatten(topk_ids_cpu): + expert_num_tokens_cpu[topk_id] += 1 + return expert_num_tokens_cpu + + def __init__( + self, + max_tokens_per_batch: int, + num_experts: int, + _topk_ids: torch.Tensor, + _topk_weights: torch.Tensor, + ): + self.max_tokens_per_batch = max_tokens_per_batch + self.e = num_experts + self.topk_ids_cpu = _topk_ids.to("cpu") + self.topk_weights_cpu = _topk_weights.to("cpu") + self.expert_num_tokens_cpu = self._make_expert_num_tokens_cpu( + self.e, self.topk_ids_cpu + ) + + def is_valid(self): + """ + Return True only if the input can be represented in a Batched + format. + """ + return torch.all(self.expert_num_tokens_cpu <= self.max_tokens_per_batch) + + def _scatter(self, hidden_states: torch.Tensor) -> torch.Tensor: + hidden_states_cpu = hidden_states.to("cpu") + K = hidden_states_cpu.size(1) + batched_hidden_states_cpu = torch.empty( + (e, max_tokens_per_batch, K), + dtype=hidden_states_cpu.dtype, + device="cpu", + ) + + counter_cpu = torch.zeros_like(self.expert_num_tokens_cpu) + for t_idx, token in enumerate(hidden_states_cpu): + for topk_id in self.topk_ids_cpu[t_idx]: + pos_in_batch = counter_cpu[topk_id] + batched_hidden_states_cpu[topk_id, pos_in_batch] = token + counter_cpu[topk_id] += 1 + assert torch.allclose(counter_cpu, self.expert_num_tokens_cpu) + return batched_hidden_states_cpu.to("cuda") + + def _gather( + self, batched_outputs: torch.Tensor, gather_outputs: torch.Tensor + ) -> torch.Tensor: + batched_outputs_cpu = batched_outputs.to("cpu") + gather_outputs_cpu = torch.zeros_like(gather_outputs) + + counter_cpu = torch.zeros((e,), device="cpu", dtype=torch.int32) + md = gather_outputs_cpu.size(0) + for t_idx in range(md): + token = None + for topk_id, topk_weight in zip( + self.topk_ids_cpu[t_idx], self.topk_weights_cpu[t_idx] + ): + pos_in_batch = counter_cpu[topk_id] + t = batched_outputs_cpu[topk_id, pos_in_batch] * topk_weight + if token is None: + token = t + else: + token += t + counter_cpu[topk_id] += 1 + assert token is not None + gather_outputs_cpu[t_idx] = token + gather_outputs.copy_(gather_outputs_cpu) + return gather_outputs + + def run( + self, hidden_states: torch.Tensor, fused_marlin_moe_kwargs: dict[Any, Any] + ) -> torch.Tensor: + assert hidden_states.ndim == 2 + assert self.is_valid() + + batched_hidden_states = self._scatter(hidden_states) + + kwargs = fused_marlin_moe_kwargs | { + "hidden_states": batched_hidden_states, + "expert_num_tokens": self.expert_num_tokens_cpu.to("cuda"), + } + batched_outputs = batched_fused_marlin_moe(**kwargs) + + output = torch.zeros_like(hidden_states) + output = self._gather(batched_outputs, output) + return output + + kwargs = { + "w1": w1_data.qweight, + "w2": w2_data.qweight, + "bias1": None, + "bias2": None, + "w1_scale": w1_data.scales, + "w2_scale": w2_data.scales, + "global_num_experts": e, + "expert_map": None, + "global_scale1": w1_data.global_scale, + "global_scale2": w2_data.global_scale, + "g_idx1": w1_data.g_idx, + "g_idx2": w2_data.g_idx, + "sort_indices1": w1_data.sort_indices, + "sort_indices2": w2_data.sort_indices, + "w1_zeros": w1_data.zeros, + "w2_zeros": w2_data.zeros, + "quant_type_id": quant_dtype.id, + "is_k_full": True, + } + if input_dtype is not None: + kwargs["input_dtype"] = input_dtype + if w1_data.a_scales_factor is not None: + kwargs["input_global_scale1"] = w1_data.a_scales_factor + if w2_data.a_scales_factor is not None: + kwargs["input_global_scale2"] = w2_data.a_scales_factor + + # Reference + fused_marlin_moe_kwargs = kwargs | { + "hidden_states": a, + "topk_ids": topk_ids, + "topk_weights": topk_weights, + } + ref_marlin_output = fused_marlin_moe(**fused_marlin_moe_kwargs) + + # Batched + br = BatchedRun(max_tokens_per_batch, e, topk_ids, topk_weights) + if not br.is_valid(): + pytest.skip("Cannot represent data in Batched Format.") + marlin_output = br.run(a, kwargs) + + torch.testing.assert_close(marlin_output, ref_marlin_output, atol=atol, rtol=0) + + +@pytest.mark.parametrize("m,n,k", [(32, 1024, 1024)]) +@pytest.mark.parametrize("e,topk", [(8, 2)]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.skipif( + not current_platform.is_device_capability_family(100), + reason="TRTLLM backend test only runs on Blackwell GPUs (SM10x).", +) +def test_unquantized_bf16_flashinfer_trtllm_backend( + m: int, + n: int, + k: int, + e: int, + topk: int, + dtype: torch.dtype, + workspace_init, +): + """ + Test BF16 unquantized MoE with FlashInfer TRTLLM backend. + """ + set_random_seed(7) + + from vllm.model_executor.layers.fused_moe.config import ( + FusedMoEConfig, + FusedMoEParallelConfig, + RoutingMethodType, + ) + from vllm.model_executor.layers.fused_moe.oracle.unquantized import ( + UnquantizedMoeBackend, + ) + from vllm.model_executor.layers.fused_moe.unquantized_fused_moe_method import ( + UnquantizedFusedMoEMethod, + ) + + # Setup test data + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + w1 = torch.randn((e, 2 * n, k), device="cuda", dtype=dtype) / 10 + w2 = torch.randn((e, k, n), device="cuda", dtype=dtype) / 10 + router_logits = torch.randn((m, e), device="cuda", dtype=dtype) + + moe_config = FusedMoEConfig( + num_experts=e, + experts_per_token=topk, + hidden_dim=k, + intermediate_size=n, + num_local_experts=e, + num_logical_experts=e, + activation=MoEActivation.SILU, + device="cuda", + moe_parallel_config=FusedMoEParallelConfig.make_no_parallel(), + in_dtype=dtype, + routing_method=RoutingMethodType.Renormalize, + max_num_tokens=next_power_of_2(m), + moe_backend="flashinfer_trtllm", + ) + + with set_current_vllm_config(vllm_config): + quant_method = UnquantizedFusedMoEMethod(moe_config) + + # Verify TRTLLM backend was selected + assert ( + quant_method.unquantized_backend == UnquantizedMoeBackend.FLASHINFER_TRTLLM + ), f"Expected FLASHINFER_TRTLLM backend, got {quant_method.unquantized_backend}" + + # Verify it's using monolithic path + assert quant_method.is_monolithic, ( + "FLASHINFER_TRTLLM backend should use monolithic forward" + ) + layer = torch.nn.Module() + layer.w13_weight = Parameter(w1.clone(), requires_grad=False) + layer.w2_weight = Parameter(w2.clone(), requires_grad=False) + layer.global_num_experts = e + layer.local_num_experts = e + layer.top_k = topk + layer.num_expert_group = 1 + layer.topk_group = 1 + layer.intermediate_size_per_partition = n + layer.ep_rank = 0 + layer.activation = MoEActivation.SILU + layer.e_score_correction_bias = None + layer.routing_method_type = RoutingMethodType.Renormalize + layer.expert_map = None + layer.apply_router_weight_on_input = False + layer.routed_scaling_factor = 2.446 + layer.shared_experts = None + layer._expert_routing_tables = lambda: None + + quant_method.process_weights_after_loading(layer) + + assert quant_method.moe_kernel is not None, ( + "moe_kernel should be set after process_weights_after_loading" + ) + assert quant_method.supports_internal_mk, ( + "supports_internal_mk should be True after setup" + ) + + trtllm_output = quant_method.apply_monolithic( + layer=layer, + x=a, + router_logits=router_logits, + ) + + # Compute torch baseline + w1_original = w1.clone() + w2_original = w2.clone() + baseline_output = ( + torch_moe(a, w1_original, w2_original, router_logits, topk) + * layer.routed_scaling_factor + ) + + close = torch.isclose(trtllm_output, baseline_output, atol=1e-1, rtol=0.85) + assert close.float().mean() > 0.925 diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_align_block_size.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_align_block_size.py new file mode 100644 index 0000000000000000000000000000000000000000..ed7d9b5f7f35dcdfa5382f69aed22728d12e61f8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_align_block_size.py @@ -0,0 +1,409 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the MOE align block size function. + +Run `pytest tests/kernels/moe/test_moe_align_block_size.py`. +""" + +import pytest +import torch + +from vllm.model_executor.layers.fused_moe.moe_align_block_size import ( + batched_moe_align_block_size, + moe_align_block_size, +) +from vllm.utils.math_utils import cdiv, round_up +from vllm.utils.torch_utils import set_random_seed + +NUM_TOKENS = [1, 3, 256, 2256, 4096] +NUM_EXPERTS = [32, 160, 256, 257] +TOP_KS = [1, 2, 16, 32] +BLOCK_SIZES = [32, 128] +set_random_seed(0) + + +def _group_tokens_by_expert( + sorted_ids: torch.Tensor, + expert_ids: torch.Tensor, + block_size: int, + valid_length: int, + total_tokens: int, +) -> dict: + num_blocks = valid_length // block_size + expert_tokens: dict[int, list[int]] = {} + + for block_idx in range(num_blocks): + expert_id = expert_ids[block_idx].item() + block_start = block_idx * block_size + block_end = min(block_start + block_size, valid_length) + + block_tokens = sorted_ids[block_start:block_end] + valid_tokens = block_tokens[block_tokens < total_tokens] + + if expert_id not in expert_tokens: + expert_tokens[expert_id] = [] + expert_tokens[expert_id].extend(valid_tokens.tolist()) + return expert_tokens + + +def _verify_expert_level_sorting( + actual_sorted_ids: torch.Tensor, + golden_sorted_ids: torch.Tensor, + expert_ids: torch.Tensor, + block_size: int, + valid_length: int, + total_tokens: int, +): + """ + Verify that actual_sorted_ids follows the correct expert-level sorting. + The kerne limplementation may or may not preserve original token order + in topk_ids in the final sorted_ids however this does not impact quality. + """ + # Group tokens by expert from the golden implementation + golden_expert_tokens = _group_tokens_by_expert( + golden_sorted_ids, expert_ids, block_size, valid_length, total_tokens + ) + + actual_expert_tokens = _group_tokens_by_expert( + actual_sorted_ids, expert_ids, block_size, valid_length, total_tokens + ) + + assert set(golden_expert_tokens.keys()) == set(actual_expert_tokens.keys()), ( + f"Expert IDs mismatch: golden={set(golden_expert_tokens.keys())}, " + f"actual={set(actual_expert_tokens.keys())}" + ) + + for expert_id in golden_expert_tokens: + golden_tokens = torch.tensor( + golden_expert_tokens[expert_id], device=actual_sorted_ids.device + ) + actual_tokens = torch.tensor( + actual_expert_tokens[expert_id], device=actual_sorted_ids.device + ) + assert torch.equal( + torch.sort(golden_tokens)[0], torch.sort(actual_tokens)[0] + ), ( + f"Expert {expert_id} token mismatch: " + f"golden={golden_expert_tokens[expert_id]}, " + f"actual={actual_expert_tokens[expert_id]}" + ) + + +def torch_moe_align_block_size( + topk_ids: torch.Tensor, + block_size: int, + num_experts: int, + expert_map: torch.Tensor | None = None, + pad_sorted_ids: bool = False, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """ + Golden torch implementation of moe_align_block_size. + + This function aligns the token distribution across experts to be compatible + with block size for matrix multiplication by sorting tokens by expert and + padding to block boundaries. + """ + max_num_tokens_padded = topk_ids.numel() + num_experts * (block_size - 1) + if pad_sorted_ids: + max_num_tokens_padded = round_up(max_num_tokens_padded, block_size) + if topk_ids.numel() < num_experts: + max_num_tokens_padded = topk_ids.numel() * block_size + + flattened_token_indices = torch.arange( + topk_ids.numel(), device=topk_ids.device, dtype=torch.int32 + ) + flattened_expert_ids = topk_ids.flatten() + sorted_expert_ids, sort_indices = torch.sort(flattened_expert_ids, stable=True) + sorted_token_indices = flattened_token_indices[sort_indices] + + expert_token_counts = torch.zeros( + num_experts, dtype=torch.int64, device=topk_ids.device + ) + for expert_id in range(num_experts): + mask = sorted_expert_ids == expert_id + expert_token_counts[expert_id] = mask.sum() + + expert_padded_counts = torch.zeros( + num_experts, dtype=torch.int64, device=topk_ids.device + ) + for expert_id in range(num_experts): + original_count = expert_token_counts[expert_id] + if expert_map is not None and expert_map[expert_id] == -1: + continue + if original_count > 0: + expert_padded_counts[expert_id] = ( + (original_count + block_size - 1) // block_size + ) * block_size + + sorted_token_ids = torch.full( + (max_num_tokens_padded,), + topk_ids.numel(), + dtype=torch.int32, + device=topk_ids.device, + ) + max_num_blocks = (max_num_tokens_padded + block_size - 1) // block_size + expert_ids = torch.full( + (max_num_blocks,), -1, dtype=torch.int32, device=topk_ids.device + ) + + current_pos = 0 + current_block = 0 + for expert_id in range(num_experts): + if expert_map is not None and expert_map[expert_id] == -1: + continue + + expert_mask = sorted_expert_ids == expert_id + expert_tokens = sorted_token_indices[expert_mask] + num_expert_tokens = expert_tokens.shape[0] + + if num_expert_tokens > 0: + sorted_token_ids[current_pos : current_pos + num_expert_tokens] = ( + expert_tokens + ) + + expert_blocks_needed = expert_padded_counts[expert_id] // block_size + + expert_id_new = expert_id + if expert_map is not None: + expert_id_new = expert_map[expert_id] + expert_ids[current_block : current_block + expert_blocks_needed] = ( + expert_id_new + ) + + current_pos += expert_padded_counts[expert_id] + current_block += expert_blocks_needed + + total_padded_tokens = expert_padded_counts.sum() + num_tokens_post_pad = torch.tensor( + [total_padded_tokens], dtype=torch.int32, device=topk_ids.device + ) + + return sorted_token_ids, expert_ids, num_tokens_post_pad + + +@pytest.mark.parametrize("m", NUM_TOKENS) +@pytest.mark.parametrize("topk", TOP_KS) +@pytest.mark.parametrize("num_experts", NUM_EXPERTS) +@pytest.mark.parametrize("block_size", BLOCK_SIZES) +@pytest.mark.parametrize("pad_sorted_ids", [False, True]) +def test_moe_align_block_size( + m: int, topk: int, num_experts: int, block_size: int, pad_sorted_ids: bool +): + """Test moe_align_block_size without expert mapping""" + topk_ids = torch.zeros((m, topk), device="cuda", dtype=torch.int32) + for i in range(m): + experts = torch.randperm(num_experts, device="cuda")[:topk] + topk_ids[i] = experts + + actual_sorted_ids, actual_expert_ids, actual_num_tokens = moe_align_block_size( + topk_ids=topk_ids, + block_size=block_size, + num_experts=num_experts, + pad_sorted_ids=pad_sorted_ids, + ) + golden_sorted_ids, golden_expert_ids, golden_num_tokens = ( + torch_moe_align_block_size( + topk_ids=topk_ids, + block_size=block_size, + num_experts=num_experts, + pad_sorted_ids=pad_sorted_ids, + ) + ) + + torch.testing.assert_close(actual_num_tokens, golden_num_tokens, atol=0, rtol=0) + torch.testing.assert_close(actual_expert_ids, golden_expert_ids, atol=0, rtol=0) + + # For sorted_token_ids, verify block-level correctness rather than exact + # order Tokens within each expert's blocks can be in any order, but expert + # regions must be correct + _verify_expert_level_sorting( + actual_sorted_ids, + golden_sorted_ids, + actual_expert_ids, + block_size, + actual_num_tokens.item(), + m * topk, + ) + + total_tokens = m * topk + assert actual_num_tokens.item() % block_size == 0, ( + "num_tokens_post_pad should be divisible by block_size" + ) + assert actual_num_tokens.item() >= total_tokens, ( + "num_tokens_post_pad should be at least total_tokens" + ) + valid_tokens = actual_sorted_ids[actual_sorted_ids < total_tokens] + assert len(valid_tokens) == total_tokens, ( + f"Should have exactly {total_tokens} valid tokens, got {len(valid_tokens)}" + ) + actual_num_blocks = cdiv(int(actual_num_tokens.item()), block_size) + assert (actual_expert_ids[:actual_num_blocks] >= 0).all() and ( + actual_expert_ids[:actual_num_blocks] < num_experts + ).all(), "expert_ids should contain valid expert indices" + + +@pytest.mark.parametrize("m", [16, 32, 2048]) +@pytest.mark.parametrize("topk", [2, 4]) +@pytest.mark.parametrize("num_experts", [8, 64]) +@pytest.mark.parametrize("block_size", [64]) +def test_moe_align_block_size_with_expert_map( + m: int, topk: int, num_experts: int, block_size: int +): + """Test moe_align_block_size with expert mapping (EP scenario)""" + topk_ids = torch.zeros((m, topk), device="cuda", dtype=torch.int32) + for i in range(m): + experts = torch.randperm(num_experts, device="cuda")[:topk] + topk_ids[i] = experts + + expert_map = torch.full((num_experts,), -1, device="cuda", dtype=torch.int32) + local_experts = list(range(0, num_experts, 2)) + for i, expert_id in enumerate(local_experts): + expert_map[expert_id] = i + + actual_sorted_ids, actual_expert_ids, actual_num_tokens = moe_align_block_size( + topk_ids=topk_ids, + block_size=block_size, + num_experts=num_experts, + expert_map=expert_map, + ignore_invalid_experts=True, + ) + golden_sorted_ids, golden_expert_ids, golden_num_tokens = ( + torch_moe_align_block_size( + topk_ids=topk_ids, + block_size=block_size, + num_experts=num_experts, + expert_map=expert_map, + ) + ) + + torch.testing.assert_close(actual_num_tokens, golden_num_tokens, atol=0, rtol=0) + torch.testing.assert_close(actual_expert_ids, golden_expert_ids, atol=0, rtol=0) + _verify_expert_level_sorting( + actual_sorted_ids, + golden_sorted_ids, + actual_expert_ids, + block_size, + actual_num_tokens.item(), + m * topk, + ) + + +def test_moe_align_block_size_deterministic(): + m, topk, num_experts, block_size = 128, 2, 32, 64 + + torch.manual_seed(42) + topk_ids = torch.randint( + 0, num_experts, (m, topk), device="cuda", dtype=torch.int32 + ) + + # expect the results to be reproducible + results = [] + for _ in range(5): + sorted_ids, expert_ids, num_tokens = moe_align_block_size( + topk_ids=topk_ids, block_size=block_size, num_experts=num_experts + ) + results.append((sorted_ids.clone(), expert_ids.clone(), num_tokens.clone())) + + for i in range(1, len(results)): + assert torch.equal(results[0][0], results[i][0]), ( + "sorted_ids should be deterministic" + ) + assert torch.equal(results[0][1], results[i][1]), ( + "expert_ids should be deterministic" + ) + assert torch.equal(results[0][2], results[i][2]), ( + "num_tokens should be deterministic" + ) + + +@pytest.mark.parametrize("max_tokens_per_batch", [13, 16, 512]) +@pytest.mark.parametrize("num_experts", [8, 16, 32, 64]) +@pytest.mark.parametrize("block_size", [8, 16, 32, 64]) +@pytest.mark.parametrize("simulate_empty_batches", [False, True]) +def test_batched_moe_align_block_size( + max_tokens_per_batch: int, + num_experts: int, + block_size: int, + simulate_empty_batches: bool, +): + def ref_outputs( + expert_num_tokens: torch.Tensor, + ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + E = expert_num_tokens.size(0) + + # Round up so each batch can be split to blocks evenly. + Msum = round_up(max_tokens_per_batch, block_size) * E + ref_sorted_ids = torch.empty((Msum,), dtype=torch.int32) + ref_expert_ids = torch.empty((Msum // block_size,), dtype=torch.int32) + ref_num_tokens_post_pad = torch.empty((1,), dtype=torch.int32) + + # Initialize + sentinel = E * max_tokens_per_batch + ref_sorted_ids.fill_(sentinel) + ref_expert_ids.fill_(-1) + + # Fill ref_sorted_ids + i = 0 + for expert_id, expert_nt in enumerate(expert_num_tokens): + token_offset = expert_id * max_tokens_per_batch + for j in range(expert_nt): + ref_sorted_ids[i] = token_offset + j + i += 1 + # round up i to the next block_size + i = round_up(i, block_size) + + ref_num_tokens_post_pad[0] = i + + # Fill expert_ids + nt_ceil_sum = 0 + for expert_id, expert_nt in enumerate(expert_num_tokens): + expert_ids_offset = nt_ceil_sum // block_size + ceil_expert_nt = round_up(int(expert_nt.item()), block_size) + num_blocks = ceil_expert_nt // block_size + for x in range(num_blocks): + ref_expert_ids[expert_ids_offset + x] = expert_id + nt_ceil_sum += ceil_expert_nt + + return ( + ref_sorted_ids.to("cuda"), + ref_expert_ids.to("cuda"), + ref_num_tokens_post_pad.to("cuda"), + ) + + # Compute expert_num_tokens + expert_num_tokens = torch.randint( + low=0, + high=max_tokens_per_batch, + size=(num_experts,), + device="cpu", + dtype=torch.int32, + ) + if simulate_empty_batches: + # mark half the batches to have 0 tokens + zero_batches = torch.randperm(num_experts)[: num_experts // 2] + expert_num_tokens[zero_batches] = 0 + + # ref outputs + ref_sorted_ids, ref_expert_ids, ref_num_tokens_post_pad = ref_outputs( + expert_num_tokens + ) + + # outputs + sorted_ids, expert_ids, num_tokens_post_pad = batched_moe_align_block_size( + max_tokens_per_batch, block_size, expert_num_tokens.to("cuda") + ) + + assert ref_sorted_ids.size() == sorted_ids.size(), ( + f"{ref_sorted_ids.size()} vs {sorted_ids.size()}" + ) + assert ref_expert_ids.size() == expert_ids.size(), ( + f"{ref_expert_ids.size()} vs {expert_ids.size()}" + ) + assert ref_num_tokens_post_pad.size() == num_tokens_post_pad.size(), ( + f"{ref_num_tokens_post_pad.size()} vs {num_tokens_post_pad.size()}" + ) + torch.testing.assert_close(ref_sorted_ids, sorted_ids, atol=0, rtol=0) + torch.testing.assert_close(ref_expert_ids, expert_ids, atol=0, rtol=0) + torch.testing.assert_close( + ref_num_tokens_post_pad, num_tokens_post_pad, atol=0, rtol=0 + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_kernel_oracle.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_kernel_oracle.py new file mode 100644 index 0000000000000000000000000000000000000000..030af1a1e6ad231c5d26a931972a1d968201fd8a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_kernel_oracle.py @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the MoEKernelOracle ABC introduced in PR series for #37753. + +This file contains a single canonical demonstration that +`UnquantizedMoEKernelOracle` methods delegate one-to-one to the +existing module-level functions in `oracle/unquantized.py`. Each method +on `UnquantizedMoEKernelOracle` follows the same `return module_fn(args)` +pattern, so verifying delegation for one method (`make_kernel`) gives +high confidence in the rest. +""" + +from unittest.mock import patch + +from vllm.model_executor.layers.fused_moe.experts.triton_moe import TritonExperts +from vllm.model_executor.layers.fused_moe.oracle import UnquantizedMoEKernelOracle +from vllm.model_executor.layers.fused_moe.oracle.unquantized import ( + UnquantizedMoeBackend, +) + + +class TestUnquantizedDelegation: + """UnquantizedMoEKernelOracle methods must delegate to the existing + module-level functions; behaviour is bit-identical.""" + + def test_make_kernel_delegates(self) -> None: + quant_config = object() + moe_config = object() + experts_cls = TritonExperts + sentinel_kernel = object() + + with patch( + "vllm.model_executor.layers.fused_moe.oracle.unquantized." + "make_unquantized_moe_kernel", + return_value=sentinel_kernel, + ) as mocked: + out = UnquantizedMoEKernelOracle().make_kernel( + quant_config, + moe_config, + UnquantizedMoeBackend.TRITON, + experts_cls, + ) + + mocked.assert_called_once_with( + quant_config, + moe_config, + UnquantizedMoeBackend.TRITON, + experts_cls, + None, # routing_tables default + ) + assert out is sentinel_kernel diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_layer.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_layer.py new file mode 100644 index 0000000000000000000000000000000000000000..c4c8246e1b4d42b1dc10ea74e2ea8b7c0ef08562 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_layer.py @@ -0,0 +1,1866 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the MOE layer. + +Run `pytest tests/kernels/test_moe_layer.py`. +""" + +import functools +import os +import traceback +import types +from collections.abc import Callable +from dataclasses import astuple, dataclass, fields +from itertools import product +from typing import get_args + +import pytest +import torch + +import vllm.model_executor.layers.quantization.utils.w8a8_utils +from tests.kernels.moe.modular_kernel_tools.parallel_utils import ( + ProcessGroupInfo, + _set_vllm_config, + parallel_launch_with_config, +) +from tests.kernels.moe.utils import TestMLP, make_test_weights, moe_quantize_weights +from vllm.config import ( + CompilationConfig, + ParallelConfig, + SchedulerConfig, + VllmConfig, + set_current_vllm_config, +) +from vllm.distributed import ( + get_ep_group, + get_eplb_group, + tensor_model_parallel_all_gather, +) +from vllm.distributed.eplb.eplb_communicator import create_eplb_communicator +from vllm.distributed.eplb.rebalance_execute import rearrange_expert_weights_inplace +from vllm.forward_context import set_forward_context +from vllm.model_executor.layers.fused_moe import FusedMoE, MoERunner, fused_experts +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.config import FusedMoEQuantConfig +from vllm.model_executor.layers.fused_moe.router.router_factory import ( + create_fused_moe_router, +) +from vllm.model_executor.layers.quantization.base_config import QuantizationConfig +from vllm.model_executor.layers.quantization.modelopt import ( + ModelOptFp8Config, + ModelOptNvFp4Config, +) +from vllm.model_executor.models.utils import sequence_parallel_chunk +from vllm.platforms import current_platform +from vllm.utils.flashinfer import ( + has_flashinfer_nvlink_one_sided, + has_flashinfer_nvlink_two_sided, +) +from vllm.utils.import_utils import has_deep_ep, has_mori, has_nixl_ep +from vllm.utils.math_utils import cdiv, next_power_of_2 +from vllm.utils.torch_utils import set_random_seed +from vllm.v1.worker.workspace import ( + init_workspace_manager, + is_workspace_manager_initialized, +) + +fp8_dtype = torch.float8_e4m3fn # current_platform.fp8_dtype + +SHAPE_COMBOS = [ + (1, 128, 256), + (32, 512, 512), + (222, 1024, 2048), +] +MAX_M = max([x[0] for x in SHAPE_COMBOS]) + +NUM_EXPERTS = [8, 64] +TOP_KS = [2, 6] + +# dp_size, tp_size, use_ep +# Note: DP+TP is not yet supported in the FusedMoE layer. +PARALLEL_COMBOS = [ + [1, 2, False], + [1, 4, False], + [2, 1, True], + [4, 1, True], + # This combination indicates sequence parallel. + # See ParallelConfig.use_sequence_parallel. + [2, 2, True], +] + +# TODO: should this even be set manually? let oracles handle this +BACKENDS = ["allgather_reducescatter"] + +if has_mori(): + BACKENDS += ["mori_high_throughput", "mori_low_latency"] + +if has_flashinfer_nvlink_two_sided(): + BACKENDS += ["flashinfer_nvlink_two_sided"] + +if has_flashinfer_nvlink_one_sided(): + BACKENDS += ["flashinfer_nvlink_one_sided"] + +if has_deep_ep(): + BACKENDS += ["deepep_high_throughput", "deepep_low_latency"] + +if has_nixl_ep(): + BACKENDS += ["nixl_ep"] + +QUANT_METHODS = [ + None, + "fp8", + "fp8_blocked", + "modelopt_fp8", + "modelopt_fp4", +] + +# Which quantization methods each backend supports. +# fmt: off +BACKEND_SUPPORTED_QUANTS: dict[str, set[str | None]] = { + "allgather_reducescatter": {None, "fp8", "modelopt_fp8", "modelopt_fp4"}, # noqa: E501 + "mori_high_throughput": {None, "fp8", "modelopt_fp8"}, + "mori_low_latency": {None, "fp8", "modelopt_fp8"}, + "flashinfer_nvlink_two_sided": {None, "fp8_blocked", "modelopt_fp4"}, # noqa: E501 + "flashinfer_nvlink_one_sided": {None, "modelopt_fp4"}, # noqa: E501 + "deepep_low_latency": {None, "fp8_blocked", "modelopt_fp4"}, # noqa: E501 + "deepep_high_throughput": {None, "fp8_blocked", "modelopt_fp8", "modelopt_fp4"}, # noqa: E501 + "nixl_ep": {None, "fp8_blocked", "modelopt_fp8"}, +} + +# Map from backend -> (DP/EP support, DP support, TP support, SP support) +BACKEND_EP_DP_TP_SUPPORT: dict[str, tuple[bool, bool, bool, bool]] = { + "allgather_reducescatter": (True, True, True, True), + "mori_high_throughput": (True, False, False, True), + "mori_low_latency": (True, False, False, True), + "flashinfer_nvlink_two_sided": (False, True, False, False), + "flashinfer_nvlink_one_sided": (False, True, False, False), + "deepep_low_latency": (True, False, False, True), + "deepep_high_throughput": (True, False, False, True), + "nixl_ep": (True, False, False, True), +} +# fmt: on + +# Which quantization methods support EPLB. +# ModelOptFp8MoEMethod inherits supports_eplb=False from FusedMoEMethodBase. +# TODO: double check modelopt fp8 +# modelopt_fp4 excluded: get_expert_weights() can't handle NvFP4 packed format. +EPLB_SUPPORTED_QUANTS: list[str | None] = [None, "fp8"] + +# Which backends support EPLB. +# deepep backends fail in get_expert_weights / rearrange_expert_weights_inplace. +# TODO(bnell): check this +EPLB_SUPPORTED_BACKENDS: list[str] = ["allgather_reducescatter"] + + +def mock_normalize_e4m3fn_to_e4m3fnuz( + weight: torch.Tensor, + weight_scale: torch.Tensor, + input_scale: torch.Tensor | None = None, +): + return weight, weight_scale, input_scale + + +# Needed since weights will already be in e4m3fnuz format on platforms that +# use the fnuz fp8 format and the normalize_e4m3fn_to_e4m3fnuz() function +# is not being tested here. +# NOTE: The weights are quantized by moe_quantize_weights_2d in +# _quantize_fp8_halves. +# NOTE: Not able to use monkeypatch because of the spawned parallel workers. +def override_normalize_e4m3fn_to_e4m3fnuz(): + vllm.model_executor.layers.quantization.utils.w8a8_utils.normalize_e4m3fn_to_e4m3fnuz = mock_normalize_e4m3fn_to_e4m3fnuz # noqa: E501 + + +def sp_wrapper( + fn: Callable | MoERunner, is_sequence_parallel: bool | None = None +) -> Callable: + """Wrapper to handle sequence parallelism chunking and gathering. + + For SP with EP: + - The TP group is created with the original tensor_parallel_size (e.g., 2) + - get_tp_group() has the correct world_size for SP operations + - sequence_parallel_chunk() uses get_tensor_model_parallel_world_size() + - tensor_model_parallel_all_gather() uses get_tp_group() + - Both should work correctly even when EP is enabled + """ + if isinstance(fn, MoERunner): + assert is_sequence_parallel is None + is_sequence_parallel = fn.moe_config.moe_parallel_config.is_sequence_parallel + else: + assert is_sequence_parallel is not None + + if is_sequence_parallel: + + def wrapper( + hidden_states: torch.Tensor, + router_logits: torch.Tensor, + ) -> torch.Tensor: + # Split sequence across TP ranks + # Both hidden_states and router_logits have [num_tokens, ...] shape + hidden_states = sequence_parallel_chunk(hidden_states) + router_logits = sequence_parallel_chunk(router_logits) + # Run MoE on local chunk + result = fn(hidden_states, router_logits) + # Gather results from all TP ranks + result = tensor_model_parallel_all_gather(result, 0) + # Remove any padding added by SP. + return result[: hidden_states.shape[0]] + + return wrapper + return fn + + +def maybe_roundup_layer_hidden_size( + hidden_size: int, + act_dtype: torch.dtype, + backend: str | None, +) -> int: + """ + Given layer hidden size and MoE configurations, round up hidden_size + if necessary. + + Args: + hidden_size: Layer hidden-size + act_dtype: Data type of the layer activations. + moe_parallel_config: Fused MoE parallelization strategy configuration. + + Return: + Rounded up hidden_size if rounding up is required based on the configs + and all2all backend. + Original hidden size otherwise. + """ + if backend == "deepep_high_throughput": + from vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ht import ( + DeepEPHTPrepareAndFinalize, + ) + + hidden_size = DeepEPHTPrepareAndFinalize.maybe_roundup_layer_hidden_size( + hidden_size, act_dtype + ) + + if backend == "deepep_low_latency": + from vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ll import ( + DeepEPLLPrepareAndFinalize, + ) + + hidden_size = DeepEPLLPrepareAndFinalize.maybe_roundup_layer_hidden_size( + hidden_size + ) + + return hidden_size + + +def rank_chunk(num: int, r: int, w: int) -> int: + rem = num % w + return (num // w) + (1 if r < rem else 0) + + +def chunk_by_rank( + t: torch.Tensor, + r: int, + w: int, + dim: int = 0, + device: torch.device | None = None, +) -> torch.Tensor: + chunk = cdiv(t.shape[dim], w) + t = t.narrow(dim, r * chunk, chunk) + if device is not None: + t = t.to(device) + return t + + +def maybe_chunk_by_rank( + t: torch.Tensor | None, + r: int, + w: int, + dim: int = 0, + device: torch.device | None = None, +) -> torch.Tensor | None: + if t is not None: + return chunk_by_rank(t, r, w, dim, device) + else: + return t + + +def tp_chunk_gate_up( + w: torch.Tensor, + tp_rank: int, + tp_size: int, + dim: int, + device: torch.device | int | None = None, +) -> torch.Tensor: + """TP-chunk a combined [gate; up] weight, splitting each half separately + so every rank gets a portion of both gate and up.""" + half = w.shape[dim] // 2 + gate = chunk_by_rank( + w.narrow(dim, 0, half), tp_rank, tp_size, dim=dim, device=device + ) + up = chunk_by_rank( + w.narrow(dim, half, half), tp_rank, tp_size, dim=dim, device=device + ) + return torch.cat([gate, up], dim=dim) + + +@dataclass +class MoETestConfig: + m: int + n: int + k: int + num_experts: int + top_k: int + in_dtype: torch.dtype + quantization: str | None + use_shared_experts: bool + use_gate: bool + use_routed_input_transform: bool + enable_eplb: bool = False + backend: str | None = None + ep_size: int = 1 + dp_size: int = 1 + tp_size: int = 1 + + @property + def is_sequence_parallel(self) -> bool: + # Sequence parallelism: EP enabled + TP dimension used for sequence splitting + # In test config: ep_size represents total expert parallel size + # tp_size represents the original TP dimension (becomes sp_size in MoERunner) + # dp_size represents data parallel size + # For SP: we need EP enabled (ep_size > 1) and sequence splitting (tp_size > 1) + return self.ep_size > 1 and self.tp_size > 1 + + # TODO: add more error messages + def id(self) -> str: + def proc(s: str) -> str: + return s.removeprefix("torch.") + + id_str = "-".join([proc(str(f)) for f in astuple(self)]) + return f"[{id_str}]" + + # TODO: add more error messages + @staticmethod + def from_id(id: str) -> "MoETestConfig": + id = id[1:-1] + str_values = id.split("-") + + def convert(v: str, ty): + if isinstance(ty, types.UnionType): + sub_ty = list(get_args(ty)) + assert len(sub_ty) == 2 and types.NoneType in sub_ty + sub_ty.remove(types.NoneType) + return sub_ty[0](v) if v != "None" else None + elif ty is torch.dtype: + ty_val = getattr(torch, v, None) + assert isinstance(ty_val, torch.dtype) + return ty_val + elif ty is bool: + return v == "True" + else: + return ty(v) + + values = tuple( + [convert(v, f.type) for v, f in zip(str_values, fields(MoETestConfig))] + ) + return MoETestConfig(*values) + + +def generate_valid_test_configs( + backend: str, + ep_size: int, + dp_size: int, + tp_size: int, + enable_eplb: bool, + verbosity: int = 0, +) -> list[MoETestConfig]: + configs: list[MoETestConfig] = [] + + for ( + shape, + num_experts, + top_k, + quantization, + use_shared_experts, + use_gate, + use_routed_input_transform, + ) in product( + SHAPE_COMBOS, + NUM_EXPERTS, + TOP_KS, + QUANT_METHODS, + [False, True], # shared + [False, True], # gate + [False, True], # routed input exform + ): + config = MoETestConfig( + shape[0], # m + shape[1], # n + shape[2], # k + num_experts, + top_k, + torch.bfloat16, + quantization, + use_shared_experts, + use_gate, + use_routed_input_transform, + enable_eplb, + backend, + ep_size, + dp_size, + tp_size, + ) + + valid, reason = is_valid_config(config) + if valid: + configs.append(config) + elif verbosity > 1: + print(f"Skipping invalid config {config} - {reason}") + + return configs + + +# TODO: break this up into sections +def is_valid_config(config: MoETestConfig) -> tuple[bool, str | None]: + # routed_input_transform only makes sense with shared_experts (latent MoE) + # TODO: not sure this is true + if config.use_routed_input_transform and not config.use_shared_experts: + return False, "routed_input_transform requires shared_experts" + + # TODO: disable for now + if config.use_routed_input_transform and config.enable_eplb: + return False, "routed_input_transform not supported with EPLB." + + # TODO: disable for now + if config.use_routed_input_transform and config.use_gate: + return ( + False, + "routed_input_transform not supported with gate because of " + "padding problems", + ) + + # TODO: disable for now + if config.use_routed_input_transform and config.backend in [ + "deepep_low_latency", + "deepep_high_throughput", + ]: + return ( + False, + "routed_input_transform not supported with DeepEP backends because " + "of padding problems", + ) + + # routed_input_transform + quantization + high hidden dimensions + # TODO: Disable >= 2048 for now due to insane errors. + if ( + config.use_routed_input_transform + and config.quantization is not None + and config.k >= 2048 + ): + return ( + False, + "routed_input_transform + quantization + higher hidden dimensions " + "leads to large differences.", + ) + + # Skip modelopt_fp4 if not on B100+ (compute capability 10.0+) + if ( + config.quantization == "modelopt_fp4" + and not current_platform.has_device_capability(100) + ): + return False, "modelopt_fp4 not supported on H100+ GPUs" + + # Skip flashinfer_nvlink if not on H100+ (compute capability 10.0+) + if ( + config.backend is not None + and config.backend.startswith("flashinfer_nvlink") + and not current_platform.has_device_capability(90) + ): + return False, "flashinfer_nvlink needs H100+ GPUs" + + # Backend-specific checks + if config.backend is not None: + supported_quants = BACKEND_SUPPORTED_QUANTS.get(config.backend) + if supported_quants is not None and config.quantization not in supported_quants: + return ( + False, + f"{config.backend} does not support quantization={config.quantization}", + ) + + if config.backend == "deepep_low_latency": + from vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ll import ( # noqa: E501 + DeepEPLLPrepareAndFinalize, + ) + + if config.k not in DeepEPLLPrepareAndFinalize.SUPPORTED_HIDDEN_SIZES: + return ( + False, + f"Skipping unsupported K {config.k} in {config.backend} w/o EP.", + ) + + if config.backend == "nixl_ep": + from vllm.model_executor.layers.fused_moe.prepare_finalize.nixl_ep import ( # noqa: E501 + NixlEPPrepareAndFinalize, + ) + + if config.k not in NixlEPPrepareAndFinalize.SUPPORTED_HIDDEN_SIZES: + return ( + False, + f"Skipping unsupported K {config.k} in {config.backend} w/o EP.", + ) + + if config.backend is not None: + supports_ep_dp, supports_dp, supports_tp, supports_sp = ( + BACKEND_EP_DP_TP_SUPPORT[config.backend] + ) + + if config.tp_size > 1 and not supports_tp and not config.is_sequence_parallel: + return False, f"{config.backend} does not support TP." + + if config.dp_size > 1 and config.ep_size == 1 and not supports_dp: + return False, f"{config.backend} does not support DP." + + if config.dp_size > 1 and config.ep_size > 1 and not supports_ep_dp: + return False, f"{config.backend} does not support EP/DP." + + if config.is_sequence_parallel and not supports_sp: + return False, f"{config.backend} does not support SP." + else: + if config.tp_size > 1 or config.ep_size > 1 or config.dp_size > 1: + return False, "An all2all backend is required for parallelism." + + # Sequence parallelism specific validations + if config.is_sequence_parallel: + if config.ep_size == 1: + return False, "Sequence parallelism requires EP to be enabled (ep_size > 1)" + + if config.tp_size == 1: + return ( + False, + "Sequence parallelism requires tp_size > 1 for sequence splitting", + ) + + # SP is essentially EP + sequence splitting + # Verify the relationship: ep_size should equal dp_size * tp_size + # (when pcp_size=1). + expected_ep_size = config.dp_size * config.tp_size + if config.ep_size != expected_ep_size: + return False, ( + f"For sequence parallelism: ep_size ({config.ep_size}) should equal " + f"dp_size * tp_size ({expected_ep_size})" + ) + + if config.enable_eplb: + if config.ep_size == 1: + return False, "EPLB requires EP." + + if config.quantization not in EPLB_SUPPORTED_QUANTS: + return False, f"EPLB not supported with {config.quantization} quantization." + + if config.backend not in EPLB_SUPPORTED_BACKENDS: + return False, f"EPLB not supported with {config.backend}." + + if config.num_experts % config.dp_size != 0: + return False, "EPLB requires num_experts divisible by ep_size" + + return True, None + + +def chunk_scales_by_rank( + t: torch.Tensor | None, + r: int, + w: int, + device: torch.device | None = None, +) -> torch.Tensor | None: + if t is not None and t.numel() > 1: + # Calculate start index by summing chunk sizes for all previous ranks + # start = sum(rank_chunk(t.shape[0], i, w) for i in range(r)) + # chunk = rank_chunk(t.shape[0], r, w) + # t = t[start:(start + chunk)] + chunk = rank_chunk(t.shape[0], r, w) + t = t[(r * chunk) : max(t.shape[0], (r + 1) * chunk)] + + if t is not None and device is not None: + t = t.to(device) + + return t + + +def chunk_scales( + t: torch.Tensor | None, + start: int, + end: int, + device: torch.device | None = None, +) -> torch.Tensor | None: + if t is not None and t.numel() > 1: + t = t[start:end] + + if t is not None and device is not None: + t = t.to(device) + + return t + + +@dataclass +class QuantizedWeights: + w13_weight: torch.Tensor + w2_weight: torch.Tensor + w13_weight_scale: torch.Tensor | None = None + w2_weight_scale: torch.Tensor | None = None + w13_weight_scale_2: torch.Tensor | None = None + w2_weight_scale_2: torch.Tensor | None = None + w13_input_scale: torch.Tensor | None = None + w2_input_scale: torch.Tensor | None = None + + +def _quantize_fp8_halves( + w1: torch.Tensor, + w2: torch.Tensor, + block_shape: list[int] | None = None, +) -> QuantizedWeights: + """Quantize w13 gate/up halves separately to FP8, producing per-shard scales.""" + half = w1.shape[1] // 2 + w1q_a, w1s_a, _ = moe_quantize_weights( + w1[:, :half, :], + None, + fp8_dtype, + False, + block_shape, + ) + w1q_b, w1s_b, _ = moe_quantize_weights( + w1[:, half:, :], + None, + fp8_dtype, + False, + block_shape, + ) + assert w1s_a is not None and w1s_b is not None + + w2q, w2s, _ = moe_quantize_weights(w2, None, fp8_dtype, False, block_shape) + assert w2s is not None + + if block_shape is not None: + # Blocked quantization: scales have shape (E, n_tiles, k_tiles) + # Concatenate gate and up scales along the n_tiles dimension (dim=1) + # to match the concatenation of gate and up weights + w13_weight_scale = torch.cat([w1s_a, w1s_b], dim=1) + # w2 scales keep their blocked shape (E, k_tiles, n_tiles) + w2_weight_scale = w2s + else: + # Non-blocked quantization: scales have shape (E, 1, 1) + # Each w1s_x is (E, 1, 1) -> reshape to (E, 1), cat to (E, 2) + w13_weight_scale = torch.cat([w1s_a.view(-1, 1), w1s_b.view(-1, 1)], dim=1) + # w2s is (E, 1, 1) -> reshape to (E,) + w2_weight_scale = w2s.view(-1) + + return QuantizedWeights( + w13_weight=torch.cat([w1q_a, w1q_b], dim=1), + w2_weight=w2q, + w13_weight_scale=w13_weight_scale, + w2_weight_scale=w2_weight_scale, + ) + + +def quantization_to_quant_dtype( + quantization: str | None, +) -> torch.dtype | str | None: + if quantization is None: + return None + elif quantization in ["fp8", "fp8_blocked", "modelopt_fp8"]: + return fp8_dtype + elif quantization in ["modelopt_fp4"]: + return "nvfp4" + else: + raise NotImplementedError(f"Unsupported quantization: {quantization}") + + +def make_quant_config( + quantization: str | None, + w1: torch.Tensor, + w2: torch.Tensor, + num_experts: int, +) -> tuple[QuantizationConfig | None, QuantizedWeights]: + from vllm.model_executor.layers.quantization.fp8 import Fp8Config + + if quantization is None: + return None, QuantizedWeights(w13_weight=w1, w2_weight=w2) + + if quantization == "fp8": + return Fp8Config(True), _quantize_fp8_halves(w1, w2) + + if quantization == "fp8_blocked": + block_shape = [128, 128] + return Fp8Config(True, weight_block_size=block_shape), _quantize_fp8_halves( + w1, w2, block_shape + ) + + if quantization == "modelopt_fp8": + qw = _quantize_fp8_halves(w1, w2) + # why? + qw.w13_input_scale = torch.ones( + num_experts, dtype=torch.float32, device=w1.device + ) + # why? + qw.w2_input_scale = torch.ones( + num_experts, dtype=torch.float32, device=w2.device + ) + quant_config = ModelOptFp8Config( + quant_method="FP8", + is_checkpoint_fp8_serialized=True, + kv_cache_quant_method=None, + exclude_modules=[], + ) + return quant_config, qw + + if quantization == "modelopt_fp4": + # Quantize full w13 at once so both gate/up halves share the same + # global scale per expert. process_weights_after_loading uses + # w13_weight_scale_2[:, 0] for the entire tensor, so the two shard + # scales must match. + w1q, w1s, w1gs = moe_quantize_weights(w1, None, "nvfp4", False, None) + assert w1s is not None and w1gs is not None + + w2q, w2s, w2gs = moe_quantize_weights(w2, None, "nvfp4", False, None) + assert w2s is not None and w2gs is not None + + qw = QuantizedWeights( + w13_weight=w1q, + w2_weight=w2q, + w13_weight_scale=w1s, + w2_weight_scale=w2s, + # weight_scale_2 = 1/w_gs: the kernel computes + # g_alphas = a_scale * w_scale_2, and correct dequant needs 1/w_gs. + # Expand per-expert scalar to (E, 2) for the two shards. + w13_weight_scale_2=(1.0 / w1gs).unsqueeze(1).expand(-1, 2).contiguous(), + w2_weight_scale_2=1.0 / w2gs, + w13_input_scale=torch.ones( + (num_experts, 2), dtype=torch.float32, device=w1.device + ), + w2_input_scale=torch.ones( + num_experts, dtype=torch.float32, device=w2.device + ), + ) + quant_config = ModelOptNvFp4Config( + is_checkpoint_nvfp4_serialized=True, + kv_cache_quant_algo=None, + exclude_modules=[], + ) + return quant_config, qw + + raise NotImplementedError(f"Unsupported quantization: {quantization}") + + +@dataclass +class SharedExpertsConfig: + w1: torch.Tensor + w2: torch.Tensor + w1_s: torch.Tensor | None = None + w2_s: torch.Tensor | None = None + quant_dtype: torch.dtype | str | None = None + + +@dataclass +class MoETestData: + """Container for MOE test data and transforms.""" + + w1: torch.Tensor + w2: torch.Tensor + hidden_states: torch.Tensor + router_logits: torch.Tensor + shared_experts_config: SharedExpertsConfig | None + gate: torch.nn.Module | None + routed_input_transform: torch.nn.Module | None + routed_output_transform: torch.nn.Module | None + routed_expert_hidden_size: int + + +class SimpleGate(torch.nn.Module): + """Simple gate module for testing: computes router logits from hidden states.""" + + def __init__( + self, + hidden_size: int, + num_experts: int, + dtype: torch.dtype, + device: str = "cuda", + ): + super().__init__() + self.weight = torch.nn.Parameter( + torch.randn(num_experts, hidden_size, device=device, dtype=dtype) / 10 + ) + + def forward(self, hidden_states: torch.Tensor) -> tuple[torch.Tensor, None]: + """Returns (router_logits, None) to match expected signature.""" + router_logits = torch.nn.functional.linear(hidden_states, self.weight) + return router_logits, None + + +class SimpleRoutedInputTransform(torch.nn.Module): + """Simple linear transform for testing routed input transform + (e.g., latent projection). + """ + + def __init__( + self, + in_features: int, + out_features: int, + dtype: torch.dtype, + device: str = "cuda", + ): + super().__init__() + self.weight = torch.nn.Parameter( + torch.randn(out_features, in_features, device=device, dtype=dtype) / 10 + ) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return torch.nn.functional.linear(x, self.weight) + + +def create_shared_experts_from_config( + shared_experts_config: SharedExpertsConfig | None, + in_dtype: torch.dtype, + tp_size: int = 1, + tp_rank: int = 0, + is_sequence_parallel: bool = False, + device: torch.device | str | None = "cuda", +) -> TestMLP | None: + """Create TestMLP for shared experts from config. + + Args: + shared_experts_config: Configuration for shared experts + in_dtype: Output data type + tp_size: Tensor parallel size (for weight chunking) + tp_rank: Tensor parallel rank (for weight chunking) + device: Device to move weights to (optional) + + Returns: + TestMLP instance or None if config is None + """ + if shared_experts_config is None: + return None + + s_w1 = shared_experts_config.w1 + s_w2 = shared_experts_config.w2 + + # Apply TP chunking if needed + if tp_size > 1 and not is_sequence_parallel: + s_w1 = tp_chunk_gate_up(s_w1, tp_rank, tp_size, dim=1, device=device) + s_w2 = chunk_by_rank(s_w2, tp_rank, tp_size, dim=0, device=device) + else: + s_w1 = s_w1.to(device) + s_w2 = s_w2.to(device) + + return TestMLP(w1=s_w1, w2=s_w2, out_dtype=in_dtype) + + +# Make version that takes a MoETestConfig? +def setup_moe_test_data( + m: int, + k: int, + n: int, + num_experts: int, + in_dtype: torch.dtype, + use_shared_experts: bool, + use_gate: bool, + use_routed_input_transform: bool, + backend: str | None, + device: str = "cuda", +) -> MoETestData: + """Setup test data and transforms for MOE tests. + + Args: + m: Number of tokens + k: Hidden size + n: Intermediate size + num_experts: Number of experts + in_dtype: Data type for tensors + use_shared_experts: Whether to create shared experts config + use_gate: Whether to create gate module + use_routed_input_transform: Whether to create routed input/output transforms + device: Device to create tensors on ("cuda" or "cpu") + + Returns: + MoETestData containing all test data and transforms + """ + # For latent MoE: latent_size = k // 2 + latent_size = k // 2 + + # k = maybe_roundup_layer_hidden_size(k, in_dtype, backend) + # latent_size = maybe_roundup_layer_hidden_size(latent_size, in_dtype, backend) + + # Determine dimensions for routed experts (may be transformed) + # For latent MoE, routed experts operate entirely in latent space + # (k//2). The routed_output_transform then projects back to k before + # adding with shared experts. + # w1: (E, 2*N, latent_size) - input latent_size + # w2: (E, latent_size, N) - output latent_size (fused_experts returns + # same shape as input) + routed_expert_hidden_size = latent_size if use_routed_input_transform else k + + # Create expert weights + (w1, _, _, _), (w2, _, _, _) = make_test_weights( + num_experts, + n, + routed_expert_hidden_size, # Both w1 input and w2 output use latent_size + in_dtype=in_dtype, + ) + + # Create shared experts config if needed + if use_shared_experts: + shared_experts_config = SharedExpertsConfig( + w1=torch.randn((k, n * 2), device=device, dtype=in_dtype) / 15, + w2=torch.randn((n, k), device=device, dtype=in_dtype) / 15, + ) + else: + shared_experts_config = None + + # Create routed input transform if needed + routed_input_transform = ( + SimpleRoutedInputTransform(k, latent_size, in_dtype, device=device) + if use_routed_input_transform + else None + ) + + # Create gate if needed + # Note: gate is called AFTER routed_input_transform, so it should expect + # the transformed dimension (latent_size) when routed_input_transform is used + gate_input_dim = latent_size if use_routed_input_transform else k + gate = ( + SimpleGate(gate_input_dim, num_experts, in_dtype, device=device) + if use_gate + else None + ) + + # Create routed output transform if needed (projects latent space back to original) + routed_output_transform = ( + SimpleRoutedInputTransform(latent_size, k, in_dtype, device=device) + if use_routed_input_transform + else None + ) + + # Create test inputs + hidden_states = torch.randn((m, k), device=device, dtype=in_dtype) / 10 + router_logits = torch.randn((m, num_experts), device=device, dtype=in_dtype) + + return MoETestData( + w1=w1, + w2=w2, + hidden_states=hidden_states, + router_logits=router_logits, + shared_experts_config=shared_experts_config, + gate=gate, + routed_input_transform=routed_input_transform, + routed_output_transform=routed_output_transform, + routed_expert_hidden_size=routed_expert_hidden_size, + ) + + +def make_fused_moe_layer( + quantization: str | None, + use_ep: bool, + hidden_size: int, + intermediate_size: int, + in_dtype: torch.dtype, + tp_size: int, + ep_size: int, + dp_size: int, + w1: torch.Tensor, + w2: torch.Tensor, + top_k: int, + global_num_experts: int, + renormalize: bool = False, + shared_experts: torch.nn.Module | None = None, + use_grouped_topk: bool = False, + topk_group: int | None = None, + num_expert_group: int | None = None, + custom_routing_function: Callable | None = None, + scoring_func: str = "softmax", + routed_scaling_factor: float = 1.0, + e_score_correction_bias: torch.Tensor | None = None, + apply_router_weight_on_input: bool = False, + activation: str = "silu", + indices_type: torch.dtype | None = None, + expert_map: torch.Tensor | None = None, + enable_eplb: bool = False, + expert_load_view: torch.Tensor | None = None, + logical_to_physical_map: torch.Tensor | None = None, + logical_replica_count: torch.Tensor | None = None, + num_redundant_experts: int = 0, + has_bias: bool = False, + gate: torch.nn.Module | None = None, + routed_input_transform: torch.nn.Module | None = None, + routed_output_transform: torch.nn.Module | None = None, + pcp_size: int | None = 1, + is_sequence_parallel: bool = False, +) -> MoERunner: + quant_config, qw = make_quant_config(quantization, w1, w2, global_num_experts) + + kwargs = dict() + kwargs["shared_experts"] = shared_experts + + # Add gate and routed_input_transform if provided + if gate is not None: + kwargs["gate"] = gate + + if routed_input_transform is not None: + kwargs["routed_input_transform"] = routed_input_transform + kwargs["routed_output_transform"] = routed_output_transform + + layer = FusedMoE( + num_experts=global_num_experts, + top_k=top_k, + hidden_size=hidden_size, + intermediate_size=intermediate_size, + params_dtype=in_dtype, + renormalize=renormalize, + use_grouped_topk=use_grouped_topk, + num_expert_group=num_expert_group, + topk_group=topk_group, + quant_config=quant_config, + tp_size=tp_size, + dp_size=dp_size, + pcp_size=pcp_size, + prefix="from_forward_context", + custom_routing_function=custom_routing_function, + scoring_func=scoring_func, + routed_scaling_factor=routed_scaling_factor, + e_score_correction_bias=e_score_correction_bias, + apply_router_weight_on_input=apply_router_weight_on_input, + activation=activation, + enable_eplb=enable_eplb, + num_redundant_experts=num_redundant_experts, + has_bias=has_bias, + is_sequence_parallel=is_sequence_parallel, + **kwargs, + ) + + weight_scale_name = getattr( + layer._quant_method, "weight_scale_name", "weight_scale" + ) + + for name, value in [ + ("w13_weight", qw.w13_weight), + ("w2_weight", qw.w2_weight), + (f"w13_{weight_scale_name}", qw.w13_weight_scale), + (f"w2_{weight_scale_name}", qw.w2_weight_scale), + ("w13_weight_scale_2", qw.w13_weight_scale_2), + ("w2_weight_scale_2", qw.w2_weight_scale_2), + ("w13_input_scale", qw.w13_input_scale), + ("w2_input_scale", qw.w2_input_scale), + ]: + if value is not None: + layer.routed_experts.register_parameter( + name, torch.nn.Parameter(value, requires_grad=False) + ) + + layer._quant_method.process_weights_after_loading(layer.routed_experts) + + return layer + + +def make_fake_moe_layer( + w1: torch.Tensor, + w2: torch.Tensor, + top_k: int, + global_num_experts: int, + in_dtype: torch.dtype, + quantization: str | None, + renormalize: bool = False, + shared_experts_config: SharedExpertsConfig | None = None, + use_grouped_topk: bool = False, + topk_group: int | None = None, + num_expert_group: int | None = None, + custom_routing_function: Callable | None = None, + scoring_func: str = "softmax", + routed_scaling_factor: float = 1.0, + e_score_correction_bias: torch.Tensor | None = None, + apply_router_weight_on_input: bool = False, + activation: str = "silu", + indices_type: torch.dtype | None = None, + expert_map: torch.Tensor | None = None, + expert_load_view: torch.Tensor | None = None, + logical_to_physical_map: torch.Tensor | None = None, + logical_replica_count: torch.Tensor | None = None, + num_redundant_experts: int = 0, + gate: torch.nn.Module | None = None, + routed_input_transform: torch.nn.Module | None = None, + routed_output_transform: torch.nn.Module | None = None, + use_ep: bool = False, + tp_size: int = 1, + dp_size: int = 1, + ep_size: int = 1, + is_sequence_parallel: bool = False, +) -> Callable: + quant_dtype = None + activation = MoEActivation.from_str(activation) + + router = create_fused_moe_router( + top_k=top_k, + global_num_experts=global_num_experts, + renormalize=renormalize, + use_grouped_topk=use_grouped_topk, + num_expert_group=num_expert_group, + topk_group=topk_group, + custom_routing_function=custom_routing_function, + scoring_func=scoring_func, + routed_scaling_factor=routed_scaling_factor, + e_score_correction_bias=e_score_correction_bias, + num_fused_shared_experts=0, # TODO + ) + + if quant_dtype is not None: + w1, w1_s, _ = moe_quantize_weights(w1, None, quant_dtype, False, None) + w2, w2_s, _ = moe_quantize_weights(w2, None, quant_dtype, False, None) + else: + w1_s = None + w2_s = None + + shared_experts = create_shared_experts_from_config( + shared_experts_config, + in_dtype, + ) + + quant_config = FusedMoEQuantConfig.make( + quant_dtype, + w1_scale=w1_s, + w2_scale=w2_s, + ) + + def _moe( + hidden_states: torch.Tensor, + router_logits: torch.Tensor, + ) -> torch.Tensor: + # Save original hidden_states for shared experts (before transform) + original_hidden_states = hidden_states + + # Apply routed input transform if provided + if routed_input_transform is not None: + hidden_states = routed_input_transform(hidden_states) + + # If gate provided, compute router_logits from hidden_states + # Note: gate operates on transformed hidden_states (after + # routed_input_transform) + if gate is not None: + router_logits, _ = gate(hidden_states) + + topk_weights, topk_ids = router.select_experts( + hidden_states=hidden_states, + router_logits=router_logits, + topk_indices_dtype=indices_type, + ) + + # Shared experts use original (untransformed) hidden_states + if shared_experts is not None: + shared_output = shared_experts(original_hidden_states) + else: + shared_output = None + + # Routed experts use transformed hidden_states + output = fused_experts( + hidden_states=hidden_states, + w1=w1, + w2=w2, + quant_config=quant_config, + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=activation, + apply_router_weight_on_input=apply_router_weight_on_input, + global_num_experts=global_num_experts, + expert_map=expert_map, + ) + + # Apply routed output transform if provided + # (e.g., latent space -> original space) + if routed_output_transform is not None: + output = routed_output_transform(output) + + if shared_experts is not None: + assert shared_output is not None + output += shared_output + + # Apply TP/DP reduction if not already reduced + # if (tp_size > 1 or dp_size > 1): + # output = tensor_model_parallel_all_reduce(output) + + return output + + return _moe + + +def _test_body_regular( + moe_layer: MoERunner, + hidden_states: torch.Tensor, + router_logits: torch.Tensor, + vllm_config: VllmConfig, + num_tokens: int, + num_tokens_across_dp: torch.Tensor, + **kwargs, +) -> tuple[torch.Tensor, torch.Tensor]: + """Regular MoE test body: compare layer output to baseline.""" + baseline_output = kwargs["baseline_output"] + + with set_forward_context( + None, + vllm_config, + num_tokens=num_tokens, + num_tokens_across_dp=num_tokens_across_dp, + ): + output = sp_wrapper(moe_layer)(hidden_states, router_logits) + + return baseline_output, output + + +def _test_body_eplb( + moe_layer: MoERunner, + hidden_states: torch.Tensor, + router_logits: torch.Tensor, + vllm_config: VllmConfig, + num_tokens: int, + num_tokens_across_dp: torch.Tensor, + cpu_group, + in_dtype: torch.dtype, + quantization: str | None, + use_ep: bool, + tp_size: int, + ep_size: int, + dp_size: int, + w1: torch.Tensor, + w2: torch.Tensor, + num_experts: int, + k: int, + n: int, + top_k: int, + shared_experts, + gate: torch.nn.Module | None, + routed_input_transform: torch.nn.Module | None, + routed_output_transform: torch.nn.Module | None, + **kwargs, +) -> tuple[torch.Tensor, torch.Tensor]: + device = torch.accelerator.current_accelerator() + + is_sequence_parallel = moe_layer.moe_config.moe_parallel_config.is_sequence_parallel + + """EPLB test body: compare output before and after expert weight rearrangement.""" + # Get "before" output with original weight arrangement + with set_forward_context( + None, + vllm_config, + num_tokens=num_tokens, + num_tokens_across_dp=num_tokens_across_dp, + ): + output_before = sp_wrapper(moe_layer)(hidden_states, router_logits) + + # Create a fresh FusedMoE layer with enable_eplb=True + # Delete the original layer's registration so the constructor can + # re-use the same "from_forward_context" prefix + cc = vllm_config.compilation_config + del cc.static_forward_context["from_forward_context"] + cc.static_all_moe_layers.remove("from_forward_context") + + # Determine hidden size for MoE layer + # When using routed_input_transform, experts operate in latent space + hidden_size_for_layer = k // 2 if routed_input_transform is not None else k + + eplb_moe_layer = make_fused_moe_layer( + quantization=quantization, + use_ep=use_ep, + hidden_size=hidden_size_for_layer, + intermediate_size=n, + in_dtype=in_dtype, + tp_size=tp_size, + ep_size=ep_size, + dp_size=dp_size, + w1=w1, + w2=w2, + top_k=top_k, + global_num_experts=num_experts, + shared_experts=shared_experts, + enable_eplb=True, + gate=gate, + routed_input_transform=routed_input_transform, + routed_output_transform=routed_output_transform, + is_sequence_parallel=is_sequence_parallel, + ) + + # All ranks must generate the same permutation + initial_indices = torch.arange(num_experts, dtype=torch.long) + shuffled_indices = initial_indices[torch.randperm(num_experts)] + + expert_weights = [list(eplb_moe_layer.get_expert_weights())] + + expert_buffer = [torch.empty_like(w) for w in expert_weights[0]] + assert vllm_config.parallel_config.eplb_config.communicator is not None, ( + "EPLB communicator backend must be set by ParallelConfig" + ) + communicator = create_eplb_communicator( + group_coordinator=get_eplb_group(), + backend=vllm_config.parallel_config.eplb_config.communicator, + expert_weights=expert_weights, + expert_buffer=expert_buffer, + ) + + # Rearrange expert weights across EP ranks + rearrange_expert_weights_inplace( + old_global_expert_indices=initial_indices.unsqueeze(0), + new_global_expert_indices=shuffled_indices.unsqueeze(0), + expert_weights=expert_weights, + expert_buffer=expert_buffer, + ep_group=cpu_group, + communicator=communicator, + ) + + # Build logical_to_physical_map from shuffled_indices + # shuffled_indices[physical] = logical, we need the inverse + logical_to_physical = torch.empty(num_experts, dtype=torch.int32, device=device) + logical_to_physical[shuffled_indices.to(device)] = torch.arange( + num_experts, dtype=torch.int32, device=device + ) + + eplb_moe_layer.set_eplb_state( + moe_layer_idx=0, + expert_load_view=torch.zeros( + (1, num_experts), + dtype=torch.int32, + device=device, + ), + logical_to_physical_map=logical_to_physical.reshape(num_experts, 1).unsqueeze( + 0 + ), + logical_replica_count=torch.ones( + (1, num_experts), + dtype=torch.int32, + device=device, + ), + ) + + eplb_moe_layer.router.eplb_state.should_record_tensor = torch.ones( + (), dtype=torch.bool, device=device + ) + eplb_moe_layer.router.eplb_state.num_unpadded_tokens_tensors = [ + torch.tensor(0, dtype=torch.int32, device=device) + ] + + # Get "after" output with rearranged weights and EPLB routing + with set_forward_context( + None, + vllm_config, + num_tokens=num_tokens, + num_tokens_across_dp=num_tokens_across_dp, + ): + output_after = sp_wrapper(eplb_moe_layer)(hidden_states, router_logits) + + return output_before, output_after + + +# TODO: make this take a MoETestConfig +def _run_one_config( + vllm_config: VllmConfig, + ep_size: int, # Expert parallel size (total across all ranks) + dp_size: int, # Data parallel size (number of DP groups) + tp_size: int, # Tensor parallel size OR sequence parallel size (when use_ep=True) + dp_rank: int, # Current rank in data parallel dimension + tp_rank: int, # Current rank in tensor/sequence parallel dimension + is_sequence_parallel: bool, # Whether to use sequence parallelism + m: int, + n: int, + k: int, + num_experts: int, + top_k: int, + quantization: str | None, + backend: str | None, + test_body_fn: Callable, + use_shared_experts: bool, + use_gate: bool, + use_routed_input_transform: bool, + **kwargs, +) -> None: + """Generic test loop that sets up environment and delegates to test_body_fn. + + Parameter Interpretation: + - When is_sequence_parallel=False (standard TP or EP): + * ep_size: Number of expert parallel ranks (or 1 if no EP) + * tp_size: Number of tensor parallel ranks (or 1 if no TP) + * Weights are chunked by ep_size (experts) and tp_size (tensors) + + - When is_sequence_parallel=True (EP + sequence splitting): + * ep_size: Number of expert parallel ranks (equals dp_size * tp_size) + * tp_size: Number of ranks to split sequence across (becomes sp_size in FusedMoE) + * Weights are chunked by ep_size (experts) but NOT by tp_size + * Input sequences are chunked by tp_size (via sp_wrapper) + """ + try: + set_random_seed(7) + + use_ep = ep_size > 1 + + assert vllm_config.parallel_config.enable_expert_parallel == use_ep + + in_dtype = torch.bfloat16 + device = torch.accelerator.current_accelerator() + + if not is_workspace_manager_initialized(): + init_workspace_manager(device) + + # Create test data and transforms + test_data = setup_moe_test_data( + m=m, + k=k, + n=n, + num_experts=num_experts, + in_dtype=in_dtype, + use_shared_experts=use_shared_experts, + use_gate=use_gate, + use_routed_input_transform=use_routed_input_transform, + backend=backend, + device=device, + ) + + # Extract data from test_data + hidden_states = test_data.hidden_states + router_logits = test_data.router_logits + w1 = test_data.w1 + w2 = test_data.w2 + shared_experts_config = test_data.shared_experts_config + gate = test_data.gate + routed_input_transform = test_data.routed_input_transform + routed_output_transform = test_data.routed_output_transform + activation = "silu" + + # Create baseline layer with FULL weights (no EP chunking) + # Baseline represents the expected output using full model + baseline_layer = make_fake_moe_layer( + w1=w1, + w2=w2, + top_k=top_k, + global_num_experts=num_experts, + in_dtype=in_dtype, + quantization=quantization, + renormalize=False, + shared_experts_config=shared_experts_config, + gate=gate, + routed_input_transform=routed_input_transform, + routed_output_transform=routed_output_transform, + use_ep=use_ep, + tp_size=tp_size, + ep_size=ep_size, + dp_size=dp_size, + activation=activation, + is_sequence_parallel=is_sequence_parallel, + ) + + with set_current_vllm_config(vllm_config): + # Compute baseline output with SP wrapper if needed + # sp_wrapper handles sequence chunking/gathering for SP + baseline_output = sp_wrapper(baseline_layer, is_sequence_parallel)( + hidden_states, router_logits + ) + + del baseline_layer + torch.accelerator.empty_cache() + + with set_current_vllm_config(vllm_config): + # Chunk weights for EP BEFORE creating FusedMoE + # FusedMoE uses EP-chunked weights and handles reductions internally + if ep_size > 1: + # Split experts across ranks (dimension 0 is the expert dimension) + # When EP is enabled, use EP group rank and ep_size for chunking + ep_rank = get_ep_group().rank_in_group + w1 = chunk_by_rank(w1, ep_rank, ep_size, dim=0, device=device) + w2 = chunk_by_rank(w2, ep_rank, ep_size, dim=0, device=device) + + # Chunk weights for TP (only if NOT doing sequence parallelism) + # Sequence parallelism splits tokens/sequences, not weight tensors + if tp_size > 1 and not is_sequence_parallel: + w1 = tp_chunk_gate_up(w1, tp_rank, tp_size, dim=1, device=device) + w2 = chunk_by_rank(w2, tp_rank, tp_size, dim=2, device=device) + + # Setup shared experts if needed + # In SP mode, shared experts should NOT be TP-chunked (same as routed + # experts). + # tp_size is used for sequence splitting, not weight splitting + shared_experts = create_shared_experts_from_config( + shared_experts_config, + in_dtype, + tp_size, + tp_rank, + is_sequence_parallel, + device, + ) + + # Determine hidden size for MoE layer + # When using routed_input_transform, experts operate in latent space + hidden_size_for_layer = k // 2 if routed_input_transform is not None else k + + # Create initial MoE layer + moe_layer = make_fused_moe_layer( + quantization=quantization, + use_ep=use_ep, + hidden_size=hidden_size_for_layer, + intermediate_size=n, + in_dtype=in_dtype, + tp_size=tp_size, + ep_size=ep_size, + dp_size=dp_size, + w1=w1, + w2=w2, + top_k=top_k, + global_num_experts=num_experts, + shared_experts=shared_experts, + gate=gate, + routed_input_transform=routed_input_transform, + routed_output_transform=routed_output_transform, + activation=activation, + is_sequence_parallel=is_sequence_parallel, + ) + + num_tokens = m + # num_tokens_across_dp should have one entry per DP group, not per + # total rank. + # When EP is enabled, dp_size represents the number of DP groups + num_tokens_across_dp = torch.tensor( + [num_tokens] * dp_size, + device=device, + dtype=torch.int, + ) + + # Call the test body function with all necessary context + expected, actual = test_body_fn( + moe_layer=moe_layer, + hidden_states=hidden_states, + router_logits=router_logits, + vllm_config=vllm_config, + num_tokens=num_tokens, + num_tokens_across_dp=num_tokens_across_dp, + in_dtype=in_dtype, + quantization=quantization, + use_ep=use_ep, + tp_size=tp_size, + ep_size=ep_size, + dp_size=dp_size, + w1=w1, + w2=w2, + num_experts=num_experts, + k=k, + n=n, + m=m, + top_k=top_k, + shared_experts=shared_experts, + gate=gate, + routed_input_transform=routed_input_transform, + routed_output_transform=routed_output_transform, + baseline_output=baseline_output, + **kwargs, + ) + + # Common tolerance logic + # TODO: consider associating tolerances with quant methods. + if quantization is None: + if k >= 2048: + atol, rtol = 7.6e-2, 7.6e-2 + else: + atol, rtol = 3.5e-2, 3.5e-2 + elif quantization in ("fp8", "fp8_blocked", "modelopt_fp8"): + atol, rtol = 6.5e-2, 6.5e-2 + elif quantization == "modelopt_fp4": + if k >= 2048: + atol = rtol = 1e-1 + (k * 1e-4) + else: + atol = rtol = 1e-1 + + if backend == "allgather_reducescatter" and tp_size > 1: + atol += 2e-1 + rtol += 2e-1 + else: + atol, rtol = 6e-2, 6e-2 + + torch.testing.assert_close(expected, actual, atol=atol, rtol=rtol) + finally: + torch.accelerator.synchronize() + + +# Test for non-parallel cases (world_size == 1) - backend doesn't matter +@pytest.mark.parametrize("m, n, k", SHAPE_COMBOS) +@pytest.mark.parametrize("num_experts", NUM_EXPERTS) +@pytest.mark.parametrize("top_k", TOP_KS) +@pytest.mark.parametrize("quantization", QUANT_METHODS) +@pytest.mark.parametrize("use_shared_experts", [False, True]) +@pytest.mark.parametrize("use_gate", [False, True]) +@pytest.mark.parametrize("use_routed_input_transform", [False, True]) +def test_moe_layer_no_parallel( + m: int, + n: int, + k: int, + num_experts: int, + top_k: int, + quantization: str | None, + use_shared_experts: bool, + use_gate: bool, + use_routed_input_transform: bool, + monkeypatch, +): + """Test MoE layer without parallelism (dp_size=1, tp_size=1, use_ep=False).""" + + if os.environ.get("VLLM_LOGGING_LEVEL") is None: + monkeypatch.setenv("VLLM_LOGGING_LEVEL", "ERROR") + + # Needed since weights will already be in e4m3fnuz format and the + # normalize_e4m3fn_to_e4m3fnuz() function is not being tested here. + if current_platform.is_fp8_fnuz(): + override_normalize_e4m3fn_to_e4m3fnuz() + + test_config = MoETestConfig( + m, + n, + k, + num_experts, + top_k, + torch.bfloat16, + quantization, + use_shared_experts, + use_gate, + use_routed_input_transform, + ) + + valid, reason = is_valid_config(test_config) + if not valid: + pytest.skip(reason) + + set_random_seed(7) + + parallel_config = ParallelConfig() + compilation_config = CompilationConfig() + compilation_config.pass_config.fuse_allreduce_rms = False + + vllm_config = VllmConfig( + parallel_config=parallel_config, compilation_config=compilation_config + ) + + # Initialize distributed environment for single GPU + _set_vllm_config(vllm_config, 1, rank=0, local_rank=0) + + _run_one_config( + vllm_config, + test_config.ep_size, + test_config.dp_size, + test_config.tp_size, + 0, + 0, + False, + test_config.m, + test_config.n, + test_config.k, + test_config.num_experts, + test_config.top_k, + test_config.quantization, + test_config.backend, + _test_body_regular, + use_shared_experts=test_config.use_shared_experts, + use_gate=test_config.use_gate, + use_routed_input_transform=test_config.use_routed_input_transform, + ) + + +def _test_body_config(test_config: MoETestConfig, cpu_group, **kwargs): + if not test_config.enable_eplb: + return _test_body_regular(**kwargs) + else: + return _test_body_eplb(**kwargs, cpu_group=cpu_group) + + +def _parallel_worker( + pgi: ProcessGroupInfo, + vllm_config: VllmConfig, + cpu_group, + test_configs: list[MoETestConfig], + verbosity: int, + **kwargs, +) -> None: + set_random_seed(7) + + total = 0 + passed = 0 + failed = 0 + fail_ids = [] + + dp_rank = vllm_config.parallel_config.data_parallel_rank + + if current_platform.is_fp8_fnuz(): + override_normalize_e4m3fn_to_e4m3fnuz() + + for test_config in test_configs: + cc = vllm_config.compilation_config + if "from_forward_context" in cc.static_forward_context: + del cc.static_forward_context["from_forward_context"] + cc.static_all_moe_layers.remove("from_forward_context") + + tp_rank = pgi.rank % test_config.tp_size + + if verbosity > 0: + print(f"subtest: {test_config.id()}", end="") + + try: + _run_one_config( + vllm_config, + test_config.ep_size, + test_config.dp_size, + test_config.tp_size, + dp_rank, + tp_rank, + test_config.is_sequence_parallel, + test_config.m, + test_config.n, + test_config.k, + test_config.num_experts, + test_config.top_k, + test_config.quantization, + test_config.backend, + functools.partial( + _test_body_config, test_config=test_config, cpu_group=cpu_group + ), + use_shared_experts=test_config.use_shared_experts, + use_gate=test_config.use_gate, + use_routed_input_transform=test_config.use_routed_input_transform, + ) + if verbosity > 0: + print(" PASSED") + else: + print(".", end="") + passed = passed + 1 + except Exception as ex: + fail_ids.append(test_config.id()) + failed = failed + 1 + if verbosity > 0: + traceback.print_exc() + print(f"\n{str(ex)}\nFAILED") + else: + print("F", end="") + finally: + # DeepEP managers are not reliably reusable across many subtests in + # a single worker process. Tear them down after each DeepEP case so + # later subtests do not inherit stale communication state. + if test_config.backend in { + "deepep_low_latency", + "deepep_high_throughput", + }: + torch.accelerator.synchronize() + all2all_manager = get_ep_group().device_communicator.all2all_manager + if all2all_manager is not None: + all2all_manager.destroy() + total = total + 1 + torch.distributed.barrier() + + skipped = total - (passed + failed) + + fails = f"{failed} failed" if failed > 0 else "" + sep = ", " if fails != "" else "" + skips = f"{sep}{skipped} skipped" if skipped > 0 else "" + sep = ", " if skips != "" or fails != "" else "" + passes = f"{sep}{passed} passed" if passed > 0 else "" + + report = ( + f"============= {fails}{skips}{passes} of {total} total tests =============" + ) + + sep = "\n" if verbosity == 0 else "" + print(f"{sep}{report}") + + if failed > 0: + fail_ids_str = "\n".join(fail_ids) + raise RuntimeError( + f"\n============= Failed subtests =============\n{fail_ids_str}\n{report}" + ) + + +# TODO: add cudagraphs/torch.compile tests +@pytest.mark.parametrize("dp_size, tp_size, use_ep", PARALLEL_COMBOS) +@pytest.mark.parametrize("backend", BACKENDS) +@pytest.mark.parametrize("enable_eplb", [False, True]) +def test_moe_layer( + dp_size: int, + tp_size: int, + use_ep: bool, + backend: str, + enable_eplb: bool, + monkeypatch, + pytestconfig, + subtests, +): + """Test MoE layer with parallelism (multi-GPU or TP/EP enabled). + + For non-parallel cases (world_size == 1), use test_moe_layer_no_parallel instead. + """ + num_gpus = current_platform.device_count() + world_size = tp_size * dp_size + # When use_ep=True: FusedMoEParallelConfig flattens tp_size across dp ranks + # Result: ep_size = dp_size * pcp_size * tp_size + # Since pcp_size=1 in these tests: ep_size = dp_size * tp_size = world_size + # When use_ep=False: no expert parallelism, ep_size = 1 + ep_size = 1 if not use_ep else world_size + + assert world_size > 1 + + # Check if enough GPUs available + if world_size is not None and num_gpus is not None and world_size > num_gpus: + pytest.skip(f"Not enough GPUs got {num_gpus}, expected {world_size}.") + + if enable_eplb and not use_ep: + pytest.skip("EPLB requires EP.") + + verbosity = pytestconfig.getoption("verbose") + + if os.environ.get("VLLM_LOGGING_LEVEL") is None: + monkeypatch.setenv("VLLM_LOGGING_LEVEL", "ERROR") + + # TODO: cover FlashInfer MoE backends via moe_backend, e.g. + # moe_backend=flashinfer_trtllm / flashinfer_cutlass / flashinfer_cutedsl + # (BF16, FP8 and NVFP4 paths), and VLLM_USE_FLASHINFER_MOE_INT4=1. + + parallel_config = ParallelConfig( + pipeline_parallel_size=1, + data_parallel_size=dp_size, + tensor_parallel_size=tp_size, + enable_expert_parallel=use_ep, + all2all_backend=backend, + enable_eplb=enable_eplb, + ) + + compilation_config = CompilationConfig() + # compilation_config.mode = CompilationMode.NONE # for now + compilation_config.pass_config.fuse_allreduce_rms = False # for now + + vllm_config = VllmConfig( + parallel_config=parallel_config, + compilation_config=compilation_config, + scheduler_config=SchedulerConfig.default_factory( + max_num_batched_tokens=next_power_of_2(MAX_M) + ), + ) + + test_configs = generate_valid_test_configs( + backend, ep_size, dp_size, tp_size, enable_eplb, verbosity + ) + + if subtests is not None: + new_test_configs = [] + for subtest in subtests.split(","): + sub_test_config = MoETestConfig.from_id(subtest) + if sub_test_config in test_configs: + new_test_configs.append(sub_test_config) + else: + pytest.skip( + f"subtest config {subtest} does not match any valid test " + "configuration" + ) + test_configs = new_test_configs + + if len(test_configs) == 0: + pytest.skip("No supported configs found for this testpoint.") + + try: + parallel_launch_with_config( + world_size, + _parallel_worker, + vllm_config, + None, + test_configs, + verbosity, + ) + finally: + torch.accelerator.synchronize() # TODO: Is this needed? + torch.accelerator.empty_cache() diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_permute_unpermute.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_permute_unpermute.py new file mode 100644 index 0000000000000000000000000000000000000000..47ad23a6ef3c72f2367dbb4d9dc67f8cd821acb3 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_permute_unpermute.py @@ -0,0 +1,288 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the MOE permute/unpermute kernel + +Run `pytest tests/kernels/test_moe_permute_unpermute.py`. +""" + +import numpy as np +import pytest +import torch + +from vllm.model_executor.layers.fused_moe import fused_topk +from vllm.model_executor.layers.fused_moe.expert_map_manager import ( + determine_expert_map, +) +from vllm.model_executor.layers.fused_moe.moe_permute_unpermute import ( + MoEPermuteScratch, + moe_permute, + moe_permute_unpermute_supported, + moe_unpermute, +) +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +NUM_EXPERTS = [16, 64, 256] +TOP_KS = [2, 6, 8] +EP_SIZE = [1, 4, 16] +set_random_seed(0) + +if current_platform.is_rocm(): + pytest.skip( + "moe_permute_unpermute_supported is not defined for ROCm", + allow_module_level=True, + ) + + +def torch_permute( + hidden_states: torch.Tensor, + topk_ids: torch.Tensor, + # token_expert_indices: torch.Tensor, + topk: int, + n_expert: int, + n_local_expert: int, + start_expert: int, + expert_map: torch.Tensor | None = None, +) -> list[torch.Tensor]: + n_token = hidden_states.shape[0] + if expert_map is not None: + is_local_expert = expert_map[topk_ids] != -1 + not_local_expert = expert_map[topk_ids] == -1 + topk_ids = is_local_expert * (topk_ids - start_expert) + not_local_expert * ( + topk_ids + n_expert + ) + token_expert_indices = torch.arange( + 0, n_token * topk, dtype=torch.int32, device=hidden_states.device + ).reshape((n_token, topk)) + + sorted_topk_ids, sorted_indices = torch.sort(topk_ids.flatten(), stable=True) + dst_row_id2src_row_id_map = token_expert_indices.flatten()[sorted_indices] + + expert_first_token_offset = torch.zeros( + n_local_expert + 1, dtype=torch.int64, device="cuda" + ) + idx = 0 + for i in range(0, n_local_expert): + cnt = 0 + while idx < sorted_topk_ids.numel() and sorted_topk_ids[idx] == i: + cnt += 1 + idx += 1 + expert_first_token_offset[i + 1] = expert_first_token_offset[i] + cnt + + _, src2dst_idx = torch.sort(dst_row_id2src_row_id_map) + valid_row_idx = [] + permuted_hidden_states = hidden_states[dst_row_id2src_row_id_map // topk, ...] + src_row_id2dst_row_id_map = torch.arange( + 0, n_token * topk, device="cuda", dtype=torch.int32 + )[src2dst_idx].reshape((n_token, topk)) + valid_row_idx += [i for i in range(expert_first_token_offset[-1])] + dst_row_id2src_row_id_map[expert_first_token_offset[-1] :] = n_token * topk + return [ + permuted_hidden_states, + expert_first_token_offset, + src_row_id2dst_row_id_map, + dst_row_id2src_row_id_map, + valid_row_idx, + ] + + +def torch_unpermute( + permuted_hidden_states: torch.Tensor, + topk_weights: torch.Tensor, + topk_ids: torch.Tensor, + token_expert_indices: torch.Tensor, + src_row_id2dst_row_id_map: torch.Tensor, + valid_row_idx: torch.Tensor, + topk: int, + n_expert: int, +) -> torch.Tensor: + # ignore invalid row + n_hidden = permuted_hidden_states.shape[1] + mask = torch.zeros(permuted_hidden_states.shape[0], dtype=bool, device="cuda") + mask[valid_row_idx] = True + permuted_hidden_states[~mask] = 0 + + permuted_hidden_states = permuted_hidden_states[ + src_row_id2dst_row_id_map.flatten(), ... + ] + permuted_hidden_states = permuted_hidden_states.view(-1, topk, n_hidden) + output = ( + (permuted_hidden_states * topk_weights.unsqueeze(2)) + .sum(1) + .to(permuted_hidden_states.dtype) + ) + return output + + +@pytest.mark.parametrize("n_token", [1, 33, 1024, 5000]) +@pytest.mark.parametrize("n_hidden", [2048, 7168]) +@pytest.mark.parametrize("n_expert", NUM_EXPERTS) +@pytest.mark.parametrize("topk", TOP_KS) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("ep_size", EP_SIZE) +def test_moe_permute_unpermute( + n_token: int, + n_hidden: int, + topk: int, + n_expert: int, + ep_size: int, + dtype: torch.dtype, +): + if not moe_permute_unpermute_supported(): + pytest.skip("moe_permute_unpermute is not supported on this platform.") + ep_rank = np.random.randint(0, ep_size) + expert_map = None + n_local_expert = n_expert + if ep_size != 1: + n_local_expert, expert_map, _ = determine_expert_map(ep_size, ep_rank, n_expert) + expert_map = expert_map.cuda() + start_expert = n_local_expert * ep_rank + set_random_seed(0) + hidden_states = torch.randn((n_token, n_hidden), device="cuda").to(dtype) + gating_output = torch.randn((n_token, n_expert), device="cuda").to(dtype) + topk_weights, topk_ids, token_expert_indices = fused_topk( + hidden_states, gating_output, topk, False + ) + ( + gold_permuted_hidden_states, + gold_expert_first_token_offset, + gold_inv_permuted_idx, + gold_permuted_idx, + valid_row_idx, + ) = torch_permute( + hidden_states, + topk_ids, + # token_expert_indices, + topk, + n_expert, + n_local_expert, + start_expert, + expert_map=expert_map, + ) + + ( + permuted_hidden_states, + _, + expert_first_token_offset, + inv_permuted_idx, + _, + ) = moe_permute( + hidden_states=hidden_states, + a1q_scale=None, + topk_ids=topk_ids, + n_expert=n_expert, + n_local_expert=n_local_expert, + expert_map=expert_map, + ) + + # check expert_first_token_offset + torch.testing.assert_close( + gold_expert_first_token_offset, expert_first_token_offset, atol=0, rtol=0 + ) + # check src_row_id2dst_row_id_map + torch.testing.assert_close( + gold_inv_permuted_idx.flatten(), inv_permuted_idx, atol=0, rtol=0 + ) + + # check permuted_hidden_states, only valid token + torch.testing.assert_close( + gold_permuted_hidden_states[valid_row_idx], + permuted_hidden_states[valid_row_idx], + atol=0, + rtol=0, + ) + # add a random tensor to simulate group gemm + result0 = 0.5 * permuted_hidden_states + torch.randn_like(permuted_hidden_states) + result4 = torch.empty_like(hidden_states) + moe_unpermute( + result4, result0, topk_weights, inv_permuted_idx, expert_first_token_offset + ) + + gold4 = torch_unpermute( + result0, + topk_weights, + topk_ids, + token_expert_indices, + inv_permuted_idx, + valid_row_idx, + topk, + n_local_expert, + ) + # check unpermuted hidden + torch.testing.assert_close(result4, gold4, atol=2e-2, rtol=0) + + +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +def test_moe_permute_reuses_scratch_buffers(dtype: torch.dtype): + if not moe_permute_unpermute_supported(): + pytest.skip("moe_permute_unpermute is not supported on this platform.") + + n_token = 64 + n_hidden = 2048 + n_expert = 16 + topk = 4 + hidden_states = torch.randn((n_token, n_hidden), device="cuda").to(dtype) + gating_output = torch.randn((n_token, n_expert), device="cuda").to(dtype) + _, topk_ids, _ = fused_topk(hidden_states, gating_output, topk, False) + + scratch = MoEPermuteScratch( + max_num_tokens=n_token, + topk=topk, + num_experts=n_expert, + num_local_experts=n_expert, + device=hidden_states.device, + hidden_size=n_hidden, + hidden_dtype=hidden_states.dtype, + ) + + first = moe_permute( + hidden_states=hidden_states, + a1q_scale=None, + topk_ids=topk_ids, + n_expert=n_expert, + scratch=scratch, + ) + second = moe_permute( + hidden_states=hidden_states, + a1q_scale=None, + topk_ids=topk_ids, + n_expert=n_expert, + scratch=scratch, + ) + + ( + permuted_hidden_states_1, + _, + expert_first_token_offset_1, + inv_permuted_idx_1, + permuted_idx_1, + ) = first + ( + permuted_hidden_states_2, + _, + expert_first_token_offset_2, + inv_permuted_idx_2, + permuted_idx_2, + ) = second + + torch.testing.assert_close(permuted_hidden_states_1, permuted_hidden_states_2) + torch.testing.assert_close(expert_first_token_offset_1, expert_first_token_offset_2) + torch.testing.assert_close(inv_permuted_idx_1, inv_permuted_idx_2) + torch.testing.assert_close(permuted_idx_1, permuted_idx_2) + + assert ( + permuted_hidden_states_1.untyped_storage().data_ptr() + == permuted_hidden_states_2.untyped_storage().data_ptr() + ) + assert ( + expert_first_token_offset_1.untyped_storage().data_ptr() + == expert_first_token_offset_2.untyped_storage().data_ptr() + ) + assert ( + inv_permuted_idx_1.untyped_storage().data_ptr() + == scratch.inv_permuted_idx.untyped_storage().data_ptr() + ) + assert ( + permuted_idx_1.untyped_storage().data_ptr() + == scratch.permuted_idx.untyped_storage().data_ptr() + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_weight_loading_padded.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_weight_loading_padded.py new file mode 100644 index 0000000000000000000000000000000000000000..ee8eed63ec9e74e3e4f1641b94a2de2b0afa3bb6 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_moe_weight_loading_padded.py @@ -0,0 +1,348 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for FusedMoE weight loading with padded hidden dimensions. + +When using DeepEP backends or NIXL EP with models like nemotron_h, +hidden_size may be rounded up (e.g., 2688 -> 3072) for backend requirements. +Weight parameters are created with the padded size, but checkpoint weights +have the original unpadded size. These tests verify that weight loading +correctly handles this mismatch. +""" + +import pytest +import torch + +from vllm.model_executor.layers.fused_moe.routed_experts import RoutedExperts + + +class TestGetHiddenDim: + """Unit tests for _get_hidden_dim.""" + + def test_2d_non_transposed_w2(self): + # w2: shard_dim=1 (intermediate), hidden=0 + assert RoutedExperts._get_hidden_dim(shard_dim=1, ndim=2) == 0 + + def test_2d_non_transposed_w13(self): + # w1/w3: shard_dim=0 (intermediate), hidden=1 + assert RoutedExperts._get_hidden_dim(shard_dim=0, ndim=2) == 1 + + def test_2d_transposed_w2(self): + # transposed w2: shard_dim=0, hidden=1 + assert RoutedExperts._get_hidden_dim(shard_dim=0, ndim=2) == 1 + + def test_2d_transposed_w13(self): + # transposed w1/w3: shard_dim=1, hidden=0 + assert RoutedExperts._get_hidden_dim(shard_dim=1, ndim=2) == 0 + + def test_3d_non_transposed_w2(self): + # 3D w2: shard_dim=2, hidden=1 + assert RoutedExperts._get_hidden_dim(shard_dim=2, ndim=3) == 1 + + def test_3d_non_transposed_w13(self): + # 3D w1/w3: shard_dim=1, hidden=2 + assert RoutedExperts._get_hidden_dim(shard_dim=1, ndim=3) == 2 + + def test_3d_transposed_w2(self): + # transposed 3D w2: shard_dim=1, hidden=2 + assert RoutedExperts._get_hidden_dim(shard_dim=1, ndim=3) == 2 + + def test_3d_transposed_w13(self): + # transposed 3D w1/w3: shard_dim=2, hidden=1 + assert RoutedExperts._get_hidden_dim(shard_dim=2, ndim=3) == 1 + + def test_1d_returns_zero(self): + # 1D per-channel scales: always returns 0 + assert RoutedExperts._get_hidden_dim(shard_dim=0, ndim=1) == 0 + assert RoutedExperts._get_hidden_dim(shard_dim=1, ndim=1) == 0 + + def test_invalid_shard_dim_raises(self): + # shard_dim outside the data dimensions should raise + with pytest.raises(ValueError, match="not a valid data dimension"): + RoutedExperts._get_hidden_dim(shard_dim=0, ndim=3) + + +class TestNarrowExpertDataForPadding: + """Unit tests for _narrow_expert_data_for_padding.""" + + def test_no_narrowing_when_shapes_match(self): + expert_data = torch.zeros(1024, 1024) + loaded_weight = torch.randn(1024, 1024) + result = RoutedExperts._narrow_expert_data_for_padding( + expert_data, loaded_weight, hidden_dim=0 + ) + assert result.shape == loaded_weight.shape + assert result.data_ptr() == expert_data.data_ptr() + + def test_narrow_w2_hidden_dim(self): + # w2: (hidden_size, intermediate_size) - hidden_size padded at dim 0 + expert_data = torch.zeros(3072, 1024) + loaded_weight = torch.randn(2688, 1024) + result = RoutedExperts._narrow_expert_data_for_padding( + expert_data, loaded_weight, hidden_dim=0 + ) + assert result.shape == (2688, 1024) + + def test_narrow_w13_hidden_dim(self): + # w1/w3: (intermediate_size, hidden_size) - hidden_size padded at dim 1 + expert_data = torch.zeros(2048, 3072) + loaded_weight = torch.randn(2048, 2688) + result = RoutedExperts._narrow_expert_data_for_padding( + expert_data, loaded_weight, hidden_dim=1 + ) + assert result.shape == (2048, 2688) + + def test_narrow_transposed_w2(self): + # transposed w2: (intermediate_size, hidden_size) - hidden at dim 1 + expert_data = torch.zeros(1024, 3072) + loaded_weight = torch.randn(1024, 2688) + hidden_dim = RoutedExperts._get_hidden_dim(shard_dim=0, ndim=2) + result = RoutedExperts._narrow_expert_data_for_padding( + expert_data, loaded_weight, hidden_dim=hidden_dim + ) + assert result.shape == (1024, 2688) + + def test_narrow_3d_full_load(self): + # 3D tensor for full_load path: w2 (num_experts, hidden_size, intermediate) + expert_data = torch.zeros(8, 3072, 1024) + loaded_weight = torch.randn(8, 2688, 1024) + result = RoutedExperts._narrow_expert_data_for_padding( + expert_data, loaded_weight, hidden_dim=1 + ) + assert result.shape == (8, 2688, 1024) + + def test_narrow_1d_scale(self): + # 1D scale tensor: per-channel w2 scale (hidden_size,) + expert_data = torch.zeros(3072) + loaded_weight = torch.randn(2688) + result = RoutedExperts._narrow_expert_data_for_padding( + expert_data, loaded_weight, hidden_dim=0 + ) + assert result.shape == (2688,) + + def test_scalar_weight_no_op(self): + # 0-dim tensor should be a no-op + expert_data = torch.zeros(3072) + loaded_weight = torch.tensor(1.0) + result = RoutedExperts._narrow_expert_data_for_padding( + expert_data, loaded_weight, hidden_dim=0 + ) + # ndim == 0, so no narrowing + assert result.shape == (3072,) + + def test_no_narrowing_when_loaded_weight_larger(self): + # Guard: don't narrow if loaded_weight is larger than expert_data + expert_data = torch.zeros(2688, 1024) + loaded_weight = torch.randn(3072, 1024) + result = RoutedExperts._narrow_expert_data_for_padding( + expert_data, loaded_weight, hidden_dim=0 + ) + assert result.shape == (2688, 1024) + assert result.data_ptr() == expert_data.data_ptr() + + def test_negative_hidden_dim_is_noop(self): + # Negative hidden_dim should be a safe no-op (0 <= check) + expert_data = torch.zeros(3072, 1024) + loaded_weight = torch.randn(2688, 1024) + result = RoutedExperts._narrow_expert_data_for_padding( + expert_data, loaded_weight, hidden_dim=-1 + ) + # -1 fails the 0 <= check, so no narrowing + assert result.shape == (3072, 1024) + assert result.data_ptr() == expert_data.data_ptr() + + def test_only_narrows_hidden_dim(self): + # Verify that only the specified hidden_dim is narrowed, + # even when other dimensions also differ + expert_data = torch.zeros(3072, 2048) + loaded_weight = torch.randn(2688, 1024) + result = RoutedExperts._narrow_expert_data_for_padding( + expert_data, loaded_weight, hidden_dim=0 + ) + # Only dim 0 (hidden) should be narrowed; dim 1 stays at 2048 + assert result.shape == (2688, 2048) + + def test_narrowed_data_shares_storage(self): + # Verify narrowing returns a view (writes go to original tensor) + expert_data = torch.zeros(3072, 1024) + loaded_weight = torch.randn(2688, 1024) + result = RoutedExperts._narrow_expert_data_for_padding( + expert_data, loaded_weight, hidden_dim=0 + ) + result.copy_(loaded_weight) + # The first 2688 rows of expert_data should now have loaded_weight + assert torch.equal(expert_data[:2688, :], loaded_weight) + # Padded region should remain zero + assert torch.equal(expert_data[2688:, :], torch.zeros(3072 - 2688, 1024)) + + +class TestWeightLoadingWithPaddedHiddenSize: + """Integration-style tests that simulate padded weight loading.""" + + def test_load_w2_with_padding(self): + """Simulate loading w2 weights when hidden_size is padded.""" + padded_hidden = 3072 + original_hidden = 2688 + intermediate = 1024 + + expert_data_full = torch.zeros(padded_hidden, intermediate) + loaded_weight = torch.randn(original_hidden, intermediate) + + # w2 non-transposed: shard_dim=1, hidden_dim=0 + hidden_dim = RoutedExperts._get_hidden_dim(shard_dim=1, ndim=2) + expert_data = RoutedExperts._narrow_expert_data_for_padding( + expert_data_full, loaded_weight, hidden_dim=hidden_dim + ) + expert_data.copy_(loaded_weight) + + assert torch.equal(expert_data_full[:original_hidden, :], loaded_weight) + assert torch.equal( + expert_data_full[original_hidden:, :], + torch.zeros(padded_hidden - original_hidden, intermediate), + ) + + def test_load_w13_with_padding(self): + """Simulate loading w1/w3 weights when hidden_size is padded.""" + padded_hidden = 3072 + original_hidden = 2688 + intermediate = 1024 + + # w1/w3: (intermediate_size, hidden_size) + expert_data_full = torch.zeros(intermediate, padded_hidden) + loaded_weight = torch.randn(intermediate, original_hidden) + + # w1 non-transposed: shard_dim=0, hidden_dim=1 + hidden_dim = RoutedExperts._get_hidden_dim(shard_dim=0, ndim=2) + expert_data = RoutedExperts._narrow_expert_data_for_padding( + expert_data_full, loaded_weight, hidden_dim=hidden_dim + ) + expert_data.copy_(loaded_weight) + + assert torch.equal(expert_data_full[:, :original_hidden], loaded_weight) + assert torch.equal( + expert_data_full[:, original_hidden:], + torch.zeros(intermediate, padded_hidden - original_hidden), + ) + + def test_load_transposed_w2_with_padding(self): + """Simulate loading transposed w2 (GPTQ) with padded hidden_size.""" + padded_hidden = 3072 + original_hidden = 2688 + intermediate = 1024 + + # transposed w2: (intermediate_size, hidden_size), shard_dim=0 + expert_data_full = torch.zeros(intermediate, padded_hidden) + loaded_weight = torch.randn(intermediate, original_hidden) + + hidden_dim = RoutedExperts._get_hidden_dim(shard_dim=0, ndim=2) + expert_data = RoutedExperts._narrow_expert_data_for_padding( + expert_data_full, loaded_weight, hidden_dim=hidden_dim + ) + expert_data.copy_(loaded_weight) + + assert torch.equal(expert_data_full[:, :original_hidden], loaded_weight) + + def test_no_padding_is_noop(self): + """Verify that when sizes match, behavior is unchanged.""" + hidden = 2048 + intermediate = 1024 + + expert_data_full = torch.zeros(hidden, intermediate) + loaded_weight = torch.randn(hidden, intermediate) + + hidden_dim = RoutedExperts._get_hidden_dim(shard_dim=1, ndim=2) + expert_data = RoutedExperts._narrow_expert_data_for_padding( + expert_data_full, loaded_weight, hidden_dim=hidden_dim + ) + expert_data.copy_(loaded_weight) + + assert torch.equal(expert_data_full, loaded_weight) + + def test_narrow_shard_dim(self): + """Simulate loading w2 when both hidden_size and intermediate_size + are padded. + """ + padded_hidden = 3072 + original_hidden = 2688 + padded_intermediate = 1024 + original_intermediate = 896 + + expert_data_full = torch.zeros(padded_hidden, padded_intermediate) + loaded_weight = torch.randn(original_hidden, original_intermediate) + + shard_dim = 1 + hidden_dim = RoutedExperts._get_hidden_dim(shard_dim=shard_dim, ndim=2) + expert_data = RoutedExperts._narrow_expert_data_for_padding( + expert_data_full, + loaded_weight, + hidden_dim=hidden_dim, + shard_dim=shard_dim, + ) + expert_data.copy_(loaded_weight) + + assert torch.equal( + expert_data_full[:original_hidden, :original_intermediate], + loaded_weight, + ) + assert torch.equal( + expert_data_full[original_hidden:, :], + torch.zeros(padded_hidden - original_hidden, padded_intermediate), + ) + assert torch.equal( + expert_data_full[:original_hidden, original_intermediate:], + torch.zeros(original_hidden, padded_intermediate - original_intermediate), + ) + + def test_bnb_shape_mismatch_raises(self): + """BnB + padded hidden_size should raise via weight_loader.""" + from unittest.mock import MagicMock + + num_experts = 1 + padded_packed = 3072 # padded packed size + original_packed = 2688 # original packed size + + # Build a param that looks like a BnB 4-bit MoE weight. + param_data = torch.zeros(num_experts, padded_packed, 1, dtype=torch.uint8) + param = torch.nn.Parameter(param_data, requires_grad=False) + param.use_bitsandbytes_4bit = True + + loaded_weight = torch.randint(0, 255, (original_packed, 1), dtype=torch.uint8) + + # Minimal RoutedExperts mock so weight_loader reaches the BnB path. + moe = MagicMock(spec=RoutedExperts) + moe.quant_config = None + moe.quant_method = MagicMock() + moe.quant_method.__class__.__name__ = "BitsAndBytesMethod" + moe._expert_map = None + moe.tp_rank = 0 + + # Call the real weight_loader (unbound) with our mock as self. + with pytest.raises(ValueError, match="BitsAndBytes"): + RoutedExperts.weight_loader( + moe, + param, + loaded_weight, + weight_name="w2", + shard_id="w2", + expert_id=0, + ) + + +class TestPerTensorScaleCoercion: + """Regression test for shape-(1,) per-tensor scales (issue #43297). + + llm-compressor NVFP4 emits per-tensor weight and input scales as + shape-(1,) tensors. `_to_scalar` collapses them to a 0-D scalar so the + scalar-slot assignments in the weight loader neither broadcast nor raise. + """ + + def test_collapses_to_scalar(self): + # shape-(1,) and 0-D both reduce to a 0-D scalar. + for loaded_weight in (torch.tensor([0.5]), torch.tensor(0.5)): + scalar = RoutedExperts._to_scalar(loaded_weight) + assert scalar.shape == () + assert scalar.item() == pytest.approx(0.5) + + def test_rejects_non_scalar(self): + # numel > 1 must fail loudly instead of silently picking an element. + with pytest.raises(RuntimeError): + RoutedExperts._to_scalar(torch.tensor([0.1, 0.2])) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_mxfp4_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_mxfp4_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..e9143219246deb37a39e75611f0eb8e84118f843 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_mxfp4_moe.py @@ -0,0 +1,467 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Tests for SM100 CUTLASS MXFP4 x MXFP4 grouped MoE kernels.""" + +import random + +import pytest +import torch + +from tests.kernels.utils import torch_moe_single +from vllm import _custom_ops as ops +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +random.seed(42) +set_random_seed(42) + +MXFP4_BLOCK_SIZE = 32 + + +def align(val: int, alignment: int = 128) -> int: + return int((val + alignment - 1) // alignment * alignment) + + +def calc_diff(x, y): + x, y = x.double(), y.double() + denominator = (x * x + y * y).sum() + sim = 2 * (x * y).sum() / denominator + return 1 - sim + + +def is_sm100_supported() -> bool: + return current_platform.is_cuda() and current_platform.is_device_capability_family( + 100 + ) + + +def compute_ref_output( + input_tensor: torch.Tensor, + weight_list: list[torch.Tensor], + expert_offsets: list[int], + expert_offset: int, + num_experts: int, +) -> torch.Tensor: + """Reference output using torch_moe_single with top-1 routing.""" + score = torch.full( + (expert_offset, num_experts), + -1e9, + device=input_tensor.device, + dtype=torch.float32, + ) + for g in range(num_experts): + start = expert_offsets[g] + end = expert_offsets[g + 1] if g + 1 < num_experts else expert_offset + score[start:end, g] = 0.0 + + return torch_moe_single( + input_tensor, torch.stack(weight_list, dim=0), score, topk=1 + ) + + +@pytest.mark.skipif( + not is_sm100_supported(), + reason="cutlass_mxfp4_group_mm requires CUDA SM100", +) +@pytest.mark.parametrize("num_experts", [8, 16, 32]) +@pytest.mark.parametrize("out_dtype", [torch.bfloat16]) +def test_cutlass_mxfp4_grouped_mm(num_experts, out_dtype): + """ + Test the MXFP4 grouped GEMM kernel by: + 1. Creating random per-expert inputs and weights + 2. Quantizing both to MXFP4 using the CUDA kernel + 3. Running the CUTLASS grouped GEMM + 4. Comparing against BF16 reference + """ + device = "cuda" + alignment = 128 + # N and K must be multiples of 128 for clean swizzle layout + n_g = random.randint(1, 16) * alignment + k_g = random.randint(1, 16) * alignment + + expert_offset = 0 + expert_offsets_input = [] + problem_sizes = [] + input_list = [] + weight_list = [] + + for g in range(num_experts): + m_g = random.randint(1, 256) + expert_offsets_input.append(expert_offset) + expert_offset += m_g + problem_sizes.append([m_g, n_g, k_g]) + + input_list.append( + torch.normal(0.0, std=0.5, size=(m_g, k_g), device=device, dtype=out_dtype) + ) + weight_list.append( + torch.normal(0.0, std=0.5, size=(n_g, k_g), device=device, dtype=out_dtype) + ) + + input_tensor = torch.concat(input_list, dim=0) # [M_total, K] + + # --- Quantize INPUTS via mxfp4_experts_quant --- + input_bs_offsets = [] + tot = 0 + for g in range(num_experts): + input_bs_offsets.append(tot) + tot += align(problem_sizes[g][0], 128) + input_bs_offsets.append(tot) + + _inp_expert_offsets = torch.tensor( + expert_offsets_input + [expert_offset], device=device, dtype=torch.int32 + ) + _inp_bs_offsets = torch.tensor(input_bs_offsets, device=device, dtype=torch.int32) + + input_quant, input_sf = ops.mxfp4_experts_quant( + input_tensor, + _inp_expert_offsets, + _inp_bs_offsets, + num_experts, + topk=1, + ) + + # --- Quantize WEIGHTS via mxfp4_experts_quant --- + # Treat each expert's N weight rows as an "expert" with N tokens + weight_tensor = torch.concat(weight_list, dim=0) # [E*N, K] + weight_expert_offsets = [g * n_g for g in range(num_experts)] + [num_experts * n_g] + # N is always multiple of 128, so blockscale offsets are clean + weight_bs_offsets = [g * n_g for g in range(num_experts)] + [num_experts * n_g] + + _wt_expert_offsets = torch.tensor( + weight_expert_offsets, device=device, dtype=torch.int32 + ) + _wt_bs_offsets = torch.tensor(weight_bs_offsets, device=device, dtype=torch.int32) + + weight_quant, weight_sf = ops.mxfp4_experts_quant( + weight_tensor, + _wt_expert_offsets, + _wt_bs_offsets, + num_experts, + topk=1, + ) + + # Reshape weight quantized data to [E, N, K//2] + weight_quant = weight_quant[: num_experts * n_g].view(num_experts, n_g, k_g // 2) + + # Reshape weight scale factors to [E, N, K//32] + # The quant kernel produces uint8 SF buffer. Each row has K//32 SFs. + scales_per_row = k_g // MXFP4_BLOCK_SIZE + weight_sf_flat = weight_sf.view(-1)[: num_experts * n_g * scales_per_row] + weight_sf_3d = weight_sf_flat.view(num_experts, n_g, scales_per_row) + + # Output + output = torch.empty((expert_offset, n_g), device=device, dtype=out_dtype) + + _problem_sizes = torch.tensor(problem_sizes, device=device, dtype=torch.int32) + _expert_offsets = torch.tensor( + expert_offsets_input, device=device, dtype=torch.int32 + ) + _input_bs = torch.tensor(input_bs_offsets[:-1], device=device, dtype=torch.int32) + + # Run the MXFP4 grouped GEMM + ops.cutlass_mxfp4_moe_mm( + output, + input_quant, + weight_quant, + input_sf, + weight_sf_3d, + _problem_sizes, + _expert_offsets, + _input_bs, + ) + + # Reference: BF16 matmul + ref_output = compute_ref_output( + input_tensor=input_tensor, + weight_list=weight_list, + expert_offsets=expert_offsets_input, + expert_offset=expert_offset, + num_experts=num_experts, + ) + + # Compare per-expert + for g in range(num_experts): + start = expert_offsets_input[g] + end = expert_offsets_input[g + 1] if g + 1 < num_experts else expert_offset + if start == end: + continue + baseline = ref_output[start:end] + actual = output[start:end] + diff = calc_diff(actual, baseline) + print( + f"m_g={end - start} n_g={n_g} k_g={k_g} " + f"num_experts={num_experts}, " + f"out_dtype={out_dtype}, diff={diff:.5f}" + ) + # FP4 quantization is very lossy (~4 bits precision) + # Comparing quantized vs full-precision gives cosine diff of 0.05-0.15 + assert diff < 0.15, f"Expert {g}: diff={diff:.5f} exceeds threshold" + + +@pytest.mark.skipif( + not is_sm100_supported(), + reason="mxfp4_experts_quant requires CUDA SM100", +) +def test_mxfp4_experts_quant_basic(): + """ + Basic smoke test for the MXFP4 experts quantization kernel. + """ + device = "cuda" + num_experts = 4 + k = 256 + tokens_per_expert = 16 + + total_tokens = tokens_per_expert * num_experts + input_tensor = torch.randn(total_tokens, k, device=device, dtype=torch.bfloat16) / 5 + + expert_offsets = [i * tokens_per_expert for i in range(num_experts + 1)] + blockscale_offsets = [ + align(i * tokens_per_expert, 128) for i in range(num_experts + 1) + ] + + _expert_offsets = torch.tensor(expert_offsets, device=device, dtype=torch.int32) + _blockscale_offsets = torch.tensor( + blockscale_offsets, device=device, dtype=torch.int32 + ) + + output, output_sf = ops.mxfp4_experts_quant( + input_tensor, + _expert_offsets, + _blockscale_offsets, + num_experts, + topk=1, + ) + + assert output.shape == (total_tokens, k // 2) + assert output.dtype == torch.uint8 + assert output_sf.dtype == torch.uint8 + assert output.any(), "Quantized output is all zeros" + print( + f"MXFP4 experts quant: output shape={output.shape}, sf shape={output_sf.shape}" + ) + print("PASSED") + + +def untile_cutlass_scale(scale_raw: torch.Tensor, rows: int, K: int) -> torch.Tensor: + """Convert CUTLASS tiled scale back to flat [M, K//32] layout. + + CUTLASS tiled layout: [numMTiles, numKTiles, 32(outerM), 4(innerM), 4(innerK)] + Produced by: padded.reshape(numMTiles, 4, 32, numKTiles, 4).permute(0,3,2,1,4) + To undo: tiled.permute(0, 3, 2, 1, 4).reshape(padded_M, padded_sK) + """ + num_scale_cols = K // MXFP4_BLOCK_SIZE + num_m_tiles = (rows + 127) // 128 + num_k_tiles = (num_scale_cols + 3) // 4 + padded_M = num_m_tiles * 128 + padded_sK = num_k_tiles * 4 + + scale_bytes = scale_raw.view(torch.uint8).flatten() + total_bytes = padded_M * padded_sK + tiled = scale_bytes[:total_bytes].reshape(num_m_tiles, num_k_tiles, 32, 4, 4) + undone = tiled.permute(0, 3, 2, 1, 4).contiguous() + return undone.reshape(padded_M, padded_sK)[:rows, :num_scale_cols] + + +def compute_reference_e8m0_scale(block_max: float) -> int: + """Compute the expected OCP MX spec E8M0 scale for a given block max. + + The CUTLASS kernel uses round-to-nearest on the mantissa: + rounded_bits = (float_bits + (1 << 21)) & 0xFF800000 + biased_exp = (rounded_bits >> 23) & 0xFF + scale_exp = max(biased_exp - 2, 0) + + This ensures max_val / scale <= 6.0 for most inputs. + """ + import struct + + if block_max <= 0: + return 0 + # Replicate the kernel's rounding logic in Python + float_bytes = struct.pack("f", block_max) + max_bits = struct.unpack("I", float_bytes)[0] + rounded_bits = (max_bits + (1 << 21)) & 0xFF800000 + biased_exp = (rounded_bits >> 23) & 0xFF + scale_exp = max(int(biased_exp) - 2, 0) + scale_exp = min(scale_exp, 254) + return scale_exp + + +@pytest.mark.skipif( + not is_sm100_supported(), + reason="mxfp4_experts_quant requires CUDA SM100", +) +@pytest.mark.parametrize("k", [256, 7168]) +@pytest.mark.parametrize("m", [16, 64]) +def test_mxfp4_experts_quant_e8m0_scale_correctness(m, k): + """ + Test that mxfp4_experts_quant computes E8M0 block scales correctly + per OCP MX spec (not the NVFP4 formula). + + The old buggy kernel used: floor(log2(max/6)) + 127 + The fixed kernel uses: round_nearest_exp(max) - 2 + + This test verifies: + 1. Scales match the expected OCP MX formula for all blocks + 2. No block max exceeds the representable range (no unexpected saturation) + 3. Reconstruction error is within expected bounds for MXFP4 + """ + device = "cuda" + + # Generate input with controlled range + input_tensor = torch.randn(m, k, device=device, dtype=torch.bfloat16) * 0.5 + + # Quantize + num_experts = 1 + expert_offsets = torch.tensor([0, m], device=device, dtype=torch.int32) + num_k_tiles = (k // MXFP4_BLOCK_SIZE + 3) // 4 + blockscale_offsets = torch.tensor( + [0, align(m, 128) * num_k_tiles], device=device, dtype=torch.int32 + ) + + output_fp4, output_sf = ops.mxfp4_experts_quant( + input_tensor, expert_offsets, blockscale_offsets, num_experts, topk=1 + ) + + # Untile scale to flat layout for verification + scale_flat = untile_cutlass_scale(output_sf, m, k) + assert scale_flat.shape == (m, k // MXFP4_BLOCK_SIZE) + + # Verify each block's scale matches the OCP MX spec formula + num_blocks = k // MXFP4_BLOCK_SIZE + mismatches = 0 + buggy_pattern = 0 # count blocks where scale is 1-2 lower than expected + + for row in range(m): + for blk in range(num_blocks): + block_start = blk * MXFP4_BLOCK_SIZE + block_end = block_start + MXFP4_BLOCK_SIZE + block_max = ( + input_tensor[row, block_start:block_end].float().abs().max().item() + ) + + actual_scale = scale_flat[row, blk].item() + expected_scale = compute_reference_e8m0_scale(block_max) + + if actual_scale != expected_scale: + mismatches += 1 + if actual_scale < expected_scale: + buggy_pattern += 1 + + total_blocks = m * num_blocks + match_rate = (total_blocks - mismatches) / total_blocks + + print( + f" m={m}, k={k}: scale match rate = {match_rate * 100:.2f}% " + f"({mismatches}/{total_blocks} mismatches)" + ) + + # The fixed kernel should match the reference formula exactly + assert match_rate > 0.99, ( + f"E8M0 scale match rate too low: {match_rate * 100:.2f}%. " + f"Buggy pattern (scale too low): {buggy_pattern}/{mismatches}. " + f"This suggests the NVFP4 formula bug is present." + ) + + # Extra check: if most mismatches show scale < expected, it's the old bug + if mismatches > 0: + assert buggy_pattern / mismatches < 0.5, ( + f"Most scale mismatches show scale too LOW ({buggy_pattern}/{mismatches}). " + "This is the signature of the NVFP4 formula bug in nvfp4_utils.cuh." + ) + + # Verify reconstruction error is within MXFP4 expected bounds + # Dequantize and check cosine similarity + fp4_lut = torch.tensor( + [0, 0.5, 1, 1.5, 2, 3, 4, 6, 0, -0.5, -1, -1.5, -2, -3, -4, -6], + device=device, + dtype=torch.float32, + ) + lo = (output_fp4 & 0x0F).long() + hi = ((output_fp4 >> 4) & 0x0F).long() + unpacked = torch.stack([lo, hi], dim=-1).reshape(m, k) + fp4_vals = fp4_lut[unpacked] + + scales_expanded = 2.0 ** (scale_flat.float() - 127.0) + scales_expanded = scales_expanded.unsqueeze(-1).expand(-1, -1, MXFP4_BLOCK_SIZE) + scales_expanded = scales_expanded.reshape(m, k) + recon = (fp4_vals * scales_expanded).bfloat16() + + # Cosine similarity should be > 0.99 for well-behaved MXFP4 quantization + cos_sim = torch.nn.functional.cosine_similarity( + recon.float().flatten().unsqueeze(0), + input_tensor.float().flatten().unsqueeze(0), + ).item() + max_abs_diff = (recon.float() - input_tensor.float()).abs().max().item() + + print( + f" Reconstruction: cosine_sim={cos_sim:.6f}, max_abs_diff={max_abs_diff:.4f}" + ) + + assert cos_sim > 0.99, ( + f"Reconstruction cosine similarity too low: {cos_sim:.6f}. " + f"Expected > 0.99 for correct MXFP4 quantization." + ) + # With correct E8M0, max abs diff should be bounded by scale * 6 + # (worst case: value just below threshold rounds to wrong FP4 code) + assert max_abs_diff < 1.0, ( + f"Max reconstruction error too large: {max_abs_diff:.4f}. " + "Likely caused by incorrect E8M0 scale (values saturating to ±6)." + ) + + +@pytest.mark.skipif( + not is_sm100_supported(), + reason="mxfp4_experts_quant requires CUDA SM100", +) +def test_mxfp4_experts_quant_no_saturation(): + """ + Test that the E8M0 scale is large enough to avoid unexpected saturation. + + With the buggy NVFP4 formula, the scale was too small causing most values + to saturate to ±6 in FP4. The fixed OCP MX formula should ensure that + block_max / scale <= 6.0 (the max E2M1 value) in almost all cases. + """ + device = "cuda" + + m, k = 128, 1024 + # Use inputs with known range to make saturation detectable + input_tensor = torch.randn(m, k, device=device, dtype=torch.bfloat16) * 0.5 + + num_experts = 1 + expert_offsets = torch.tensor([0, m], device=device, dtype=torch.int32) + num_k_tiles = (k // MXFP4_BLOCK_SIZE + 3) // 4 + blockscale_offsets = torch.tensor( + [0, align(m, 128) * num_k_tiles], device=device, dtype=torch.int32 + ) + + output_fp4, output_sf = ops.mxfp4_experts_quant( + input_tensor, expert_offsets, blockscale_offsets, num_experts, topk=1 + ) + + # Check saturation rate: count FP4 values that are ±6 (codes 7 and 15) + lo = output_fp4 & 0x0F + hi = (output_fp4 >> 4) & 0x0F + # Code 7 = +6.0, code 15 = -6.0 + saturated = ((lo == 7) | (lo == 15) | (hi == 7) | (hi == 15)).sum().item() + total_values = m * k + saturation_rate = saturated / total_values + + print( + f" Saturation rate: {saturation_rate * 100:.2f}% " + f"({saturated}/{total_values} values at ±6)" + ) + + # For Gaussian input with std=0.5, saturation should be very rare + # (±6 * scale is far from the typical range). + # The buggy kernel had ~30-50% saturation; fixed should be < 5%. + assert saturation_rate < 0.05, ( + f"FP4 saturation rate too high: {saturation_rate * 100:.2f}%. " + "This suggests the E8M0 scale is too small (NVFP4 formula bug). " + "Expected < 5% for Gaussian(0, 0.5) input with correct OCP MX scale." + ) + + +if __name__ == "__main__": + pytest.main([__file__, "-v", "-s"]) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_mxfp8_aiter_backend_selection.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_mxfp8_aiter_backend_selection.py new file mode 100644 index 0000000000000000000000000000000000000000..1351d9fa2df99095a2232a786ccef8d4074c60f5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_mxfp8_aiter_backend_selection.py @@ -0,0 +1,135 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""MXFP8 MoE backend selection for the AITER FlyDSL kernel (gfx950). + +GPU-free: mocks the platform (gfx950) and the ``flydsl`` package check, then +exercises the oracle so the FlyDSL backend is auto-picked when usable (including +under expert parallelism, since apply() forwards the expert_map as aiter's +expert_mask) and skipped (native fallback) when the device/package is missing. +""" + +import dataclasses +from unittest.mock import patch + +import pytest + +from vllm.platforms import current_platform + +if not current_platform.is_rocm(): + pytest.skip("This test can only run on ROCm.", allow_module_level=True) + +from tests.kernels.moe.utils import make_dummy_moe_config # noqa: E402 +from vllm.model_executor.layers.fused_moe.experts.aiter_mxfp8_moe import ( # noqa: E402 + AiterMxfp8Experts, +) +from vllm.model_executor.layers.fused_moe.modular_kernel import ( # noqa: E402 + FusedMoEActivationFormat, +) +from vllm.model_executor.layers.fused_moe.oracle.fp8 import ( # noqa: E402 + Fp8MoeBackend, +) +from vllm.model_executor.layers.fused_moe.oracle.mxfp8 import ( # noqa: E402 + _BACKEND_NAME_MAP, + _SUPPORTED_BACKENDS, + _mxfp8_backend_to_kernel_cls, + _select_kernel_cls, +) +from vllm.model_executor.layers.quantization.utils.quant_utils import ( # noqa: E402 + kMxfp8Dynamic, + kMxfp8Static, +) + +_AITER_MOD = "vllm.model_executor.layers.fused_moe.experts.aiter_mxfp8_moe" + + +def _config(ep_size: int = 1): + cfg = make_dummy_moe_config(num_experts=128, experts_per_token=4, hidden_dim=6144) + if ep_size != 1: + cfg = dataclasses.replace( + cfg, + moe_parallel_config=dataclasses.replace( + cfg.moe_parallel_config, ep_size=ep_size, use_ep=True + ), + ) + return cfg + + +def _gfx950(): + """Patch the platform so the device gate (gfx950 / MX) passes off-ROCm.""" + return patch.multiple( + f"{_AITER_MOD}.current_platform", + is_rocm=lambda: True, + supports_mx=lambda: True, + ) + + +def _flydsl_installed(present: bool): + return patch(f"{_AITER_MOD}.is_aiter_mxfp8_moe_available", return_value=present) + + +def test_aiter_mxfp8_registered(): + """The FlyDSL backend is auto-selectable and reachable via --moe-backend aiter.""" + assert Fp8MoeBackend.AITER_MXFP8 in _SUPPORTED_BACKENDS + assert _BACKEND_NAME_MAP["aiter"] is Fp8MoeBackend.AITER_MXFP8 + assert _mxfp8_backend_to_kernel_cls(Fp8MoeBackend.AITER_MXFP8) == [ + AiterMxfp8Experts + ] + + +def test_triton_selectable(): + assert _BACKEND_NAME_MAP["triton"] is Fp8MoeBackend.TRITON_MXFP8 + # Not auto-selected (only reachable explicitly), so FlyDSL still wins auto. + assert Fp8MoeBackend.TRITON_MXFP8 not in _SUPPORTED_BACKENDS + + +@pytest.mark.parametrize("ep_size", [1, 2]) +def test_ep_supported(ep_size): + """FlyDSL accepts both TP and EP: apply() forwards expert_map as expert_mask.""" + assert ( + AiterMxfp8Experts._supports_parallel_config( + _config(ep_size).moe_parallel_config + ) + is True + ) + + +@pytest.mark.parametrize( + "present,ep_size,supported,reason_substr", + [ + (True, 1, True, None), # gfx950 + flydsl + TP -> selectable + (True, 2, True, None), # gfx950 + flydsl + EP -> selectable (expert_mask) + (False, 1, False, "flydsl package"), # package missing -> clear reason + ], +) +def test_is_supported_config(present, ep_size, supported, reason_substr): + with _gfx950(), _flydsl_installed(present): + ok, reason = AiterMxfp8Experts.is_supported_config( + AiterMxfp8Experts, + _config(ep_size), + kMxfp8Static, + kMxfp8Dynamic, + FusedMoEActivationFormat.Standard, + ) + assert ok is supported + if reason_substr is not None: + assert reason_substr in reason + + +def test_explicit_moe_backend_aiter(): + """--moe-backend aiter: returns FlyDSL when usable (TP or EP), else a clear + ValueError when the flydsl package is missing.""" + with _gfx950(), _flydsl_installed(True): + assert ( + _select_kernel_cls(Fp8MoeBackend.AITER_MXFP8, _config(1)) + is AiterMxfp8Experts + ) + assert ( + _select_kernel_cls(Fp8MoeBackend.AITER_MXFP8, _config(2)) + is AiterMxfp8Experts + ) + with ( + _gfx950(), + _flydsl_installed(False), + pytest.raises(ValueError, match="flydsl package"), + ): + _select_kernel_cls(Fp8MoeBackend.AITER_MXFP8, _config(1)) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_nvfp4_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_nvfp4_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..5d42b6d7094dedaa10b1513151ec506e02b23c52 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_nvfp4_moe.py @@ -0,0 +1,291 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch + +import vllm.model_executor.layers.fused_moe.modular_kernel as mk +from tests.kernels.moe.utils import make_dummy_moe_config, make_test_weights +from tests.kernels.quantization.nvfp4_utils import ( + FLOAT4_E2M1_MAX, + FLOAT8_E4M3_MAX, + dequantize_nvfp4_to_dtype, +) +from tests.kernels.utils import torch_moe +from vllm import _custom_ops as ops +from vllm.config import ParallelConfig, VllmConfig, set_current_vllm_config +from vllm.model_executor.layers.fused_moe import fused_topk +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.all2all_utils import ( + maybe_make_prepare_finalize, +) +from vllm.model_executor.layers.fused_moe.config import nvfp4_moe_quant_config +from vllm.model_executor.layers.fused_moe.experts.cutlass_moe import ( + CutlassExpertsFp4, +) +from vllm.model_executor.layers.fused_moe.prepare_finalize import ( + make_moe_prepare_and_finalize_no_dp_ep, +) +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +if not current_platform.has_device_capability(100): + pytest.skip( + "Nvfp4 Requires compute capability of 10 or above.", allow_module_level=True + ) + +MNK_FACTORS = [ + (2, 1024, 1024), + (2, 1024, 1536), + (2, 3072, 1024), + (64, 1024, 1024), + (64, 3072, 1024), + (64, 2048, 1536), + (224, 1024, 1024), + (224, 1024, 1536), +] + + +@pytest.mark.parametrize("m,n,k", MNK_FACTORS) +@pytest.mark.parametrize("e", [40, 64, 256]) +@pytest.mark.parametrize("topk", [1, 6, 8]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@torch.inference_mode() +def test_cutlass_fp4_moe_no_graph( + m: int, n: int, k: int, e: int, topk: int, dtype: torch.dtype, workspace_init +): + set_random_seed(7) + with set_current_vllm_config( + VllmConfig(parallel_config=ParallelConfig(pipeline_parallel_size=1)) + ): + quant_blocksize = 16 + + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + + (_, w1_q, w1_blockscale, w1_gs), (_, w2_q, w2_blockscale, w2_gs) = ( + make_test_weights( + e, + n, + k, + in_dtype=dtype, + quant_dtype="nvfp4", + block_shape=None, # use quant_blocksize? + per_out_ch_quant=False, + ) + ) + + score = torch.randn((m, e), device="cuda", dtype=dtype) + topk_weights, topk_ids, _ = fused_topk(a, score, topk, renormalize=False) + + a1_gs = torch.ones((e,), device="cuda", dtype=torch.float32) + a2_gs = torch.ones((e,), device="cuda", dtype=torch.float32) + + assert w1_gs is not None + assert w2_gs is not None + assert w1_blockscale is not None + assert w2_blockscale is not None + + quant_config = nvfp4_moe_quant_config( + g1_alphas=(1 / w1_gs), + g2_alphas=(1 / w2_gs), + a1_gscale=a1_gs, + a2_gscale=a2_gs, + w1_scale=w1_blockscale, + w2_scale=w2_blockscale, + ) + moe_config = make_dummy_moe_config() + + kernel = mk.FusedMoEKernel( + maybe_make_prepare_finalize( + moe=moe_config, + quant_config=quant_config, + allow_new_interface=True, + use_monolithic=False, + ), + CutlassExpertsFp4( + moe_config=moe_config, + quant_config=quant_config, + ), + ) + + cutlass_output = kernel.apply( + hidden_states=a, + w1=w1_q, + w2=w2_q, + topk_weights=topk_weights, + topk_ids=topk_ids, + global_num_experts=e, + activation=mk.MoEActivation.SILU, + apply_router_weight_on_input=False, + expert_map=None, + ) + + # Reference check: + a_global_scale = ( + (FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX) / torch.amax(a.flatten(), dim=-1) + ).to(torch.float32) + a_fp4, a_scale_interleaved = ops.scaled_fp4_quant(a, a_global_scale) + + a_in_dtype = dequantize_nvfp4_to_dtype( + a_fp4, + a_scale_interleaved, + a_global_scale, + dtype=a.dtype, + device=a.device, + block_size=quant_blocksize, + ) + + w1_d = torch.empty((e, 2 * n, k), device="cuda", dtype=dtype) + w2_d = torch.empty((e, k, n), device="cuda", dtype=dtype) + + for idx in range(0, e): + w1_d[idx] = dequantize_nvfp4_to_dtype( + w1_q[idx], + w1_blockscale[idx], + w1_gs[idx], + dtype=dtype, + device=w1_q.device, + block_size=quant_blocksize, + ) + w2_d[idx] = dequantize_nvfp4_to_dtype( + w2_q[idx], + w2_blockscale[idx], + w2_gs[idx], + dtype=dtype, + device=w2_q.device, + block_size=quant_blocksize, + ) + + torch_output = torch_moe(a_in_dtype, w1_d, w2_d, score, topk) + + torch.testing.assert_close(torch_output, cutlass_output, atol=1e-1, rtol=1e-1) + + +# step3.5-flash uses swiglustep activation (clipped SwiGLU with limit=7.0) +# for MoE layers 43-44. This tests the non-fused activation fallback path +# in run_cutlass_moe_fp4 (apply_moe_activation + separate fp4 quantization). +# Model dims: e=288, topk=8, n=1280 (moe_intermediate_size), k=4096 (hidden) +SWIGLUSTEP_MNK_FACTORS = [ + (2, 1280, 4096), + (64, 1280, 4096), + (224, 1280, 4096), +] + + +@pytest.mark.parametrize("m,n,k", SWIGLUSTEP_MNK_FACTORS) +@pytest.mark.parametrize("e", [64, 288]) +@pytest.mark.parametrize("topk", [1, 8]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@torch.inference_mode() +def test_cutlass_fp4_moe_swiglustep( + m: int, n: int, k: int, e: int, topk: int, dtype: torch.dtype, workspace_init +): + set_random_seed(7) + with set_current_vllm_config( + VllmConfig(parallel_config=ParallelConfig(pipeline_parallel_size=1)) + ): + quant_blocksize = 16 + + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + + (_, w1_q, w1_blockscale, w1_gs), (_, w2_q, w2_blockscale, w2_gs) = ( + make_test_weights( + e, + n, + k, + in_dtype=dtype, + quant_dtype="nvfp4", + block_shape=None, + per_out_ch_quant=False, + ) + ) + + score = torch.randn((m, e), device="cuda", dtype=dtype) + topk_weights, topk_ids, _ = fused_topk(a, score, topk, renormalize=False) + + a1_gs = torch.ones((e,), device="cuda", dtype=torch.float32) + a2_gs = torch.ones((e,), device="cuda", dtype=torch.float32) + + assert w1_gs is not None + assert w2_gs is not None + assert w1_blockscale is not None + assert w2_blockscale is not None + + quant_config = nvfp4_moe_quant_config( + g1_alphas=(1 / w1_gs), + g2_alphas=(1 / w2_gs), + a1_gscale=a1_gs, + a2_gscale=a2_gs, + w1_scale=w1_blockscale, + w2_scale=w2_blockscale, + ) + + kernel = mk.FusedMoEKernel( + make_moe_prepare_and_finalize_no_dp_ep(use_monolithic=False), + CutlassExpertsFp4( + moe_config=make_dummy_moe_config(), + quant_config=quant_config, + ), + ) + + cutlass_output = kernel.apply( + hidden_states=a, + w1=w1_q, + w2=w2_q, + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=MoEActivation.SWIGLUSTEP, + global_num_experts=e, + expert_map=None, + apply_router_weight_on_input=False, + ) + + # Reference: dequantize everything and run torch_moe with swiglustep + a_global_scale = ( + (FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX) / torch.amax(a.flatten(), dim=-1) + ).to(torch.float32) + a_fp4, a_scale_interleaved = ops.scaled_fp4_quant(a, a_global_scale) + + a_in_dtype = dequantize_nvfp4_to_dtype( + a_fp4, + a_scale_interleaved, + a_global_scale, + dtype=a.dtype, + device=a.device, + block_size=quant_blocksize, + ) + + w1_d = torch.empty((e, 2 * n, k), device="cuda", dtype=dtype) + w2_d = torch.empty((e, k, n), device="cuda", dtype=dtype) + + for idx in range(0, e): + w1_d[idx] = dequantize_nvfp4_to_dtype( + w1_q[idx], + w1_blockscale[idx], + w1_gs[idx], + dtype=dtype, + device=w1_q.device, + block_size=quant_blocksize, + ) + w2_d[idx] = dequantize_nvfp4_to_dtype( + w2_q[idx], + w2_blockscale[idx], + w2_gs[idx], + dtype=dtype, + device=w2_q.device, + block_size=quant_blocksize, + ) + + torch_output = torch_moe( + a_in_dtype, + w1_d, + w2_d, + score, + topk, + activation=MoEActivation.SWIGLUSTEP, + ) + + torch.testing.assert_close(torch_output, cutlass_output, atol=1e-1, rtol=1e-1) + + +if __name__ == "__main__": + test_cutlass_fp4_moe_no_graph((2, 1024, 1024), 40, 1, torch.half) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_ocp_mx_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_ocp_mx_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..e050301b651d0f4baa124576b41a0df812c83034 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_ocp_mx_moe.py @@ -0,0 +1,1518 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import importlib.metadata +from dataclasses import dataclass +from importlib.util import find_spec + +import pytest +import torch +from packaging import version + +from tests.kernels.moe.utils import check_accuracy +from vllm._aiter_ops import is_aiter_found +from vllm.platforms import current_platform +from vllm.utils.flashinfer import has_flashinfer + +# MXFP4 via quark requires amd-quark >= 0.12 on torch >= 2.11. +# Earlier torch releases work with older quark versions. See +# https://github.com/amd/Quark/issues/34 +# TODO: Remove once amd-quark>=0.12.0 +QUARK_MXFP4_TORCH_COMPATIBLE = find_spec("quark") is not None and ( + version.parse(importlib.metadata.version("amd-quark")) >= version.parse("0.12.0") + if version.parse(torch.__version__.split("+")[0]) >= version.parse("2.11") + else True +) + +TRTLLM_GEN_MXFP4_AVAILABLE = ( + current_platform.is_cuda() and current_platform.is_device_capability_family(100) +) + +TRTLLM_GEN_MXFP8_AVAILABLE = TRTLLM_GEN_MXFP4_AVAILABLE + +HOPPER_MXFP4_BF16_AVAILABLE = ( + current_platform.is_cuda() + and current_platform.is_device_capability(90) + and has_flashinfer() +) + +# ROCm platform and dependencies +ROCM_AVAILABLE = current_platform.is_rocm() +ROCM_TRITON_KERNELS_AVAILABLE = False +ROCM_AITER_AVAILABLE = is_aiter_found() +ROCM_GFX950 = False + +if ROCM_AVAILABLE: + from vllm.platforms.rocm import on_gfx950 + from vllm.utils.import_utils import has_triton_kernels + + ROCM_TRITON_KERNELS_AVAILABLE = has_triton_kernels() + ROCM_GFX950 = on_gfx950() + + if ROCM_AITER_AVAILABLE: + from aiter.ops.triton.moe.quant_moe import upcast_from_mxfp + from aiter.ops.triton.quant import dynamic_mxfp4_quant + +if TRTLLM_GEN_MXFP4_AVAILABLE: + from flashinfer import ( + fp4_quantize, + mxfp8_quantize, + reorder_rows_for_gated_act_gemm, + shuffle_matrix_a, + shuffle_matrix_sf_a, + trtllm_fp4_block_scale_moe, + trtllm_fp8_block_scale_moe, + ) + from flashinfer.fp4_quantization import nvfp4_block_scale_interleave + +if TRTLLM_GEN_MXFP8_AVAILABLE: + from flashinfer.fused_moe.core import ( + Fp8QuantizationType, + get_w2_permute_indices_with_cache, + ) + + +@dataclass +class ModelCase: + model_id: str + tp: int + + +@pytest.fixture(scope="function", autouse=True) +def enable_pickle(monkeypatch): + """`LLM.apply_model` requires pickling a function.""" + monkeypatch.setenv("VLLM_ALLOW_INSECURE_SERIALIZATION", "1") + + +@pytest.mark.parametrize( + "model_case", + [ + ModelCase("fxmarty/qwen_1.5-moe-a2.7b-mxfp4", tp=2), + ModelCase("fxmarty/deepseek_r1_3_layers_mxfp4", tp=8), + ModelCase("mawong-amd/Llama-4-Scout-17B-16E-Instruct-2-layers-mxfp4", tp=1), + ModelCase("fxmarty/Llama-3.1-70B-Instruct-2-layers-mxfp6", tp=1), + ModelCase("fxmarty/Llama-3.1-70B-Instruct-2-layers-mxfp6", tp=4), + ], +) +@pytest.mark.skipif( + not QUARK_MXFP4_TORCH_COMPATIBLE, + reason="MXFP4 via quark requires amd-quark >= 0.12 on torch >= 2.11.", +) +def test_mxfp4_loading_and_execution_moe(vllm_runner, model_case: ModelCase): + if torch.accelerator.device_count() < model_case.tp: + pytest.skip( + f"This test requires >={model_case.tp} gpus, got only " + f"{torch.accelerator.device_count()}" + ) + + # `cudagraph_capture_sizes=[16]` to reduce load time. + with vllm_runner( + model_case.model_id, + tensor_parallel_size=model_case.tp, + load_format="dummy", + compilation_config={"cudagraph_capture_sizes": [16]}, + gpu_memory_utilization=0.8, # mxfp6 models use more scratch space + ) as llm: + # Disabled as check_model is broken: https://github.com/vllm-project/vllm/pull/18465#issuecomment-3329880562 + # def check_model(model): + # from vllm.model_executor.layers.quantization.quark.quark import ( # noqa: E501 + # QuarkLinearMethod) + # from vllm.model_executor.layers.quantization.quark.schemes.quark_ocp_mx import QuarkOCP_MX # noqa: E501 + # from vllm.model_executor.layers.quantization.quark.quark_moe import ( # noqa: E501 + # QuarkOCP_MX_MoEMethod) + + # layer = model.model.layers[0] + + # qkv_proj = layer.self_attn.qkv_proj + + # assert isinstance(qkv_proj.quant_method, QuarkLinearMethod) + # assert isinstance(qkv_proj.scheme, QuarkOCP_MX) + + # assert isinstance(layer.mlp.experts.quant_method, + # QuarkOCP_MX_MoEMethod) + + # if model_case.model_id == "fxmarty/qwen_1.5-moe-a2.7b-mxfp4": + # llm.apply_model(check_model) + + output = llm.generate_greedy("Today I am in the French Alps and", max_tokens=20) + assert output + + +def swiglu(x, alpha: float = 1.702, beta: float = 1.0, limit: float | None = None): + # Note we add an extra bias of 1 to the linear layer + # Uses chunked layout: first half is gate, second half is up + x_glu, x_linear = torch.chunk(x, 2, dim=-1) + if limit is not None: + x_glu = x_glu.clamp(max=limit) + x_linear = x_linear.clamp(min=-limit, max=limit) + out_glu = x_glu * torch.sigmoid(alpha * x_glu) + return out_glu * (x_linear + beta) + + +def swigluoai(x, alpha: float = 1.702, limit: float = 7.0): + # OAI swiglu uses interleaved layout: gate/up alternating + # See SwigluOAIAndMul in vllm/model_executor/layers/activation.py + gate, up = x[..., ::2], x[..., 1::2] + gate = gate.clamp(max=limit) + up = up.clamp(min=-limit, max=limit) + glu = gate * torch.sigmoid(gate * alpha) + return (up + 1) * glu + + +fp4_lookup_table = [0, 0.5, 1, 1.5, 2, 3, 4, 6, -0, -0.5, -1, -1.5, -2, -3, -4, -6] + + +def mxfp4_dequantize(x, scale): + assert x.dtype == torch.uint8 + x = x.view(torch.uint8).to(torch.int32) + x_unpacked = torch.zeros( + *x.shape[:-1], x.shape[-1] * 2, dtype=torch.int32, device=x.device + ) + x_unpacked[..., 0::2].copy_(x & 0xF) + x_unpacked[..., 1::2].copy_((x >> 4) & 0xF) + + x_float = torch.zeros(x_unpacked.shape, dtype=torch.float32, device=x.device) + for i, val in enumerate(fp4_lookup_table): + x_float[x_unpacked == i] = val + + scale = scale.view(torch.uint8).to(torch.int32) + scale = (scale << 23).view(torch.float32) + scale = scale.reshape(*x.shape[:-1], -1) + scale = torch.stack([scale] * 32, dim=-1).reshape(*x_float.shape) + + return x_float * scale + + +def mxfp8_dequantize(x, scale): + assert x.dtype == torch.float8_e4m3fn + x_float = x.to(torch.float32) + + scale = scale.view(torch.uint8).to(torch.int32) + scale = (scale << 23).view(torch.float32) + scale = scale.reshape(*x.shape[:-1], -1) + scale = torch.stack([scale] * 32, dim=-1).reshape(*x_float.shape) + + return x_float * scale + + +def reference_moe( + roouting_logits, + topk, + num_experts, + hidden_states, + w13, + bias13, + w2, + bias2, + alpha, + beta, + limit, + act_type, + activation: str = "swiglu", + use_interleaved_layout: bool = False, +): + """ + Reference MoE implementation for accuracy testing. + + Args: + activation: One of "swiglu", "silu", "relu2". Controls the activation + function used after the first MLP. + use_interleaved_layout: If True, uses interleaved gate/up layout + (gate=x[..., ::2], up=x[..., 1::2]) as used by SWIGLUOAI. + If False, uses chunked layout (gate, up = chunk(x, 2)) as used + by standard swiglu/silu. + """ + # renormalize routing + experts = torch.topk(roouting_logits, k=topk, dim=-1, sorted=True) + expert_weights = torch.nn.functional.softmax(experts.values, dim=1) + expert_indices = experts.indices + t = hidden_states.clone() + # MLP #1 + mlp1_weight = w13[expert_indices, ...] + mlp1_bias = bias13[expert_indices, ...] + t = torch.einsum("beck,bk->bec", mlp1_weight, t) + mlp1_bias + + # Apply activation + if activation in ("swiglu", "silu"): + if use_interleaved_layout: + # SWIGLUOAI: interleaved gate/up layout + t = swigluoai(t, alpha=alpha, limit=limit) + else: + # Standard swiglu/silu: chunked layout + t = swiglu(t, alpha=alpha, beta=beta, limit=limit) + elif activation == "relu2": + # RELU2_NO_MUL: relu(x)^2 + t = torch.relu(t) + t = t * t + else: + raise ValueError(f"Unknown activation: {activation}") + + if act_type == "mxfp8": + t_quantized, t_scale = mxfp8_quantize( + t.to(torch.bfloat16), is_sf_swizzled_layout=False + ) + t = mxfp8_dequantize(t_quantized, t_scale) + # MLP #2 + mlp2_weight = w2[expert_indices, ...] + mlp2_bias = bias2[expert_indices, ...] + t = torch.einsum("beck,bek->bec", mlp2_weight, t) + mlp2_bias + # Weighted sum of experts + t = torch.einsum("bec,be->bc", t, expert_weights) + assert t.shape == hidden_states.shape + return t.to(torch.bfloat16) + + +def tg_mxfp4_moe( + router_logits, + topk, + num_experts, + intermediate_size, + hidden_size, + hidden_states, + hidden_states_scale, + w13_weight, + w13_weight_scale, + w13_bias, + w2_weight, + w2_weight_scale, + w2_bias, + act_type, + alpha, + beta, + limit, + transpose_optimized: bool = False, +) -> torch.Tensor: + sf_block_size = 32 + assert ( + w13_weight.dim() == 3 + and w13_weight.shape[0] == num_experts + and w13_weight.shape[1] == intermediate_size * 2 + and w13_weight.shape[2] == hidden_size // 2 + ) + assert ( + w13_weight_scale.dim() == 3 + and w13_weight_scale.shape[0] == num_experts + and w13_weight_scale.shape[1] == intermediate_size * 2 + and w13_weight_scale.shape[2] == hidden_size // sf_block_size + ) + assert ( + w2_weight.dim() == 3 + and w2_weight.shape[0] == num_experts + and w2_weight.shape[1] == hidden_size + and w2_weight.shape[2] == intermediate_size // 2 + ) + assert ( + w2_weight_scale.dim() == 3 + and w2_weight_scale.shape[1] == hidden_size + and w2_weight_scale.shape[2] == intermediate_size // sf_block_size + ) + assert ( + w13_bias.dim() == 2 + and w13_bias.shape[0] == num_experts + and w13_bias.shape[1] == intermediate_size * 2 + ) + assert ( + w2_bias.dim() == 2 + and w2_bias.shape[0] == num_experts + and w2_bias.shape[1] == hidden_size + ) + + # Swap w1 and w3 as the definition of + # swiglu is different in the trtllm-gen + w13_weight_scale_ = w13_weight_scale.clone() + w13_weight_ = w13_weight.clone() + w13_bias_ = w13_bias.clone() + w13_weight[:, :intermediate_size, :].copy_(w13_weight_[:, intermediate_size:, :]) + w13_weight[:, intermediate_size:, :].copy_(w13_weight_[:, :intermediate_size, :]) + w13_weight_scale[:, :intermediate_size, :].copy_( + w13_weight_scale_[:, intermediate_size:, :] + ) + w13_weight_scale[:, intermediate_size:, :].copy_( + w13_weight_scale_[:, :intermediate_size, :] + ) + w13_bias[:, :intermediate_size].copy_(w13_bias_[:, intermediate_size:]) + w13_bias[:, intermediate_size:].copy_(w13_bias_[:, :intermediate_size]) + + # Interleave the weights and scaling factors for activation + w13_weight_interleaved = [] + w13_weight_scale_interleaved = [] + w13_bias_interleaved = [] + for i in range(num_experts): + w13_weight_interleaved.append( + reorder_rows_for_gated_act_gemm(w13_weight[i].clone()) + ) + w13_weight_scale_interleaved.append( + reorder_rows_for_gated_act_gemm(w13_weight_scale[i].clone()) + ) + w13_bias_interleaved.append( + reorder_rows_for_gated_act_gemm(w13_bias[i].clone().reshape(-1, 1)) + ) + w13_weight = torch.stack(w13_weight_interleaved).reshape( + num_experts, 2 * intermediate_size, hidden_size // 2 + ) + w13_weight_scale = torch.stack(w13_weight_scale_interleaved).reshape( + num_experts, 2 * intermediate_size, hidden_size // 32 + ) + w13_bias = torch.stack(w13_bias_interleaved).reshape( + num_experts, 2 * intermediate_size + ) + + # Shuffle weights and scaling factors for transposed mma output + gemm1_weights_shuffled = [] + gemm1_scales_shuffled = [] + gemm2_weights_shuffled = [] + gemm2_scales_shuffled = [] + gemm1_bias_shuffled = [] + gemm2_bias_shuffled = [] + epilogue_tile_m = 128 # FIXME: this depends on the kernel internals + _cache_permute_indices: dict[torch.Size, torch.Tensor] = {} + if transpose_optimized: + for i in range(num_experts): + # w13 weight shuffling + permute_indices = get_w2_permute_indices_with_cache( + _cache_permute_indices, + w13_weight[i].view(torch.uint8), + epilogue_tile_m, + ) + gemm1_weights_shuffled.append( + w13_weight[i] + .view(torch.uint8)[permute_indices.to(w13_weight.device)] + .contiguous() + ) + # w13 scale shuffling + permute_sf_indices = get_w2_permute_indices_with_cache( + _cache_permute_indices, + w13_weight_scale[i].view(torch.uint8), + epilogue_tile_m, + num_elts_per_sf=16, + ) + gemm1_scales_shuffled.append( + nvfp4_block_scale_interleave( + w13_weight_scale[i] + .view(torch.uint8)[permute_sf_indices.to(w13_weight_scale.device)] + .contiguous() + ) + ) + # w13 bias shuffling + permute_bias_indices = get_w2_permute_indices_with_cache( + _cache_permute_indices, + w13_bias[i].clone().reshape(-1, 1), + epilogue_tile_m, + ) + gemm1_bias_shuffled.append( + w13_bias[i] + .clone() + .reshape(-1, 1)[permute_bias_indices.to(w13_bias.device)] + .contiguous() + ) + # w2 weight shuffling + permute_indices = get_w2_permute_indices_with_cache( + _cache_permute_indices, + w2_weight[i].view(torch.uint8), + epilogue_tile_m, + ) + gemm2_weights_shuffled.append( + w2_weight[i] + .view(torch.uint8)[permute_indices.to(w2_weight.device)] + .contiguous() + ) + # w2 scale shuffling + permute_sf_indices = get_w2_permute_indices_with_cache( + _cache_permute_indices, + w2_weight_scale[i].view(torch.uint8), + epilogue_tile_m, + num_elts_per_sf=16, + ) + gemm2_scales_shuffled.append( + nvfp4_block_scale_interleave( + w2_weight_scale[i] + .view(torch.uint8)[permute_sf_indices.to(w2_weight_scale.device)] + .contiguous() + ) + ) + # w2 bias shuffling + permute_indices = get_w2_permute_indices_with_cache( + _cache_permute_indices, + w2_bias[i].clone().reshape(-1, 1), + epilogue_tile_m, + ) + gemm2_bias_shuffled.append( + w2_bias[i] + .clone() + .reshape(-1, 1)[permute_indices.to(w2_bias.device)] + .contiguous() + ) + + else: + for i in range(num_experts): + gemm1_weights_shuffled.append( + shuffle_matrix_a(w13_weight[i].view(torch.uint8), epilogue_tile_m) + ) + gemm1_scales_shuffled.append( + shuffle_matrix_sf_a( + w13_weight_scale[i].view(torch.uint8), epilogue_tile_m + ) + ) + + gemm2_weights_shuffled.append( + shuffle_matrix_a(w2_weight[i].view(torch.uint8), epilogue_tile_m) + ) + gemm2_scales_shuffled.append( + shuffle_matrix_sf_a( + w2_weight_scale[i].view(torch.uint8), epilogue_tile_m + ) + ) + gemm1_bias_shuffled.append( + shuffle_matrix_a(w13_bias[i].reshape(-1, 1), epilogue_tile_m) + ) + gemm2_bias_shuffled.append( + shuffle_matrix_a(w2_bias[i].reshape(-1, 1), epilogue_tile_m) + ) + + w13_weight = torch.stack(gemm1_weights_shuffled) + w13_weight_scale = ( + torch.stack(gemm1_scales_shuffled) + .reshape(num_experts, 2 * intermediate_size, hidden_size // sf_block_size) + .view(torch.float8_e4m3fn) + ) + w13_bias = torch.stack(gemm1_bias_shuffled).reshape(num_experts, -1) + + w2_weight = torch.stack(gemm2_weights_shuffled) + w2_weight_scale = ( + torch.stack(gemm2_scales_shuffled) + .reshape(num_experts, hidden_size, intermediate_size // sf_block_size) + .view(torch.float8_e4m3fn) + ) + w2_bias = torch.stack(gemm2_bias_shuffled).reshape(num_experts, -1) + + tg_result = trtllm_fp4_block_scale_moe( + routing_logits=router_logits.to(torch.bfloat16), + routing_bias=None, + hidden_states=hidden_states, + hidden_states_scale=hidden_states_scale, + gemm1_weights=w13_weight, + gemm1_weights_scale=w13_weight_scale, + gemm1_bias=w13_bias, + gemm1_alpha=alpha, + gemm1_beta=beta, + gemm1_clamp_limit=limit, + gemm2_weights=w2_weight, + gemm2_weights_scale=w2_weight_scale, + gemm2_bias=w2_bias, + output1_scale_scalar=None, + output1_scale_gate_scalar=None, + output2_scale_scalar=None, + num_experts=num_experts, + top_k=topk, + n_group=None, + topk_group=None, + intermediate_size=intermediate_size, + local_expert_offset=0, + local_num_experts=num_experts, + routed_scaling_factor=None, + routing_method_type=1, # renormalize + do_finalize=True, + )[0] + return tg_result + + +@pytest.mark.parametrize("topk", [1, 4]) +@pytest.mark.parametrize("num_experts", [32, 128]) +@pytest.mark.parametrize("num_tokens", [1, 128, 1024]) +@pytest.mark.parametrize("intermediate_size,hidden_size", [(3072, 3072)]) +@pytest.mark.parametrize("alpha,beta,limit", [(1.0, 1.0, None), (1.702, 1.0, 7.0)]) +@pytest.mark.parametrize("act_type", ["mxfp8", "bf16"]) +@pytest.mark.parametrize("transpose_optimized", [False, True]) +@pytest.mark.skipif( + not TRTLLM_GEN_MXFP4_AVAILABLE, + reason="nvidia gpu and compute capability sm100 is required for this test", +) +def test_trtllm_gen_mxfp4_fused_moe( + topk: int, + num_experts: int, + num_tokens: int, + intermediate_size: int, + hidden_size: int, + alpha: float, + beta: float, + limit: float | None, + act_type: str, + transpose_optimized: bool, +): + seed = 42 + torch.manual_seed(seed) + hidden_states = torch.randn( + num_tokens, hidden_size, device="cuda:0", dtype=torch.bfloat16 + ) + w13 = torch.randn( + num_experts, + intermediate_size * 2, + hidden_size, + device="cuda:0", + dtype=torch.bfloat16, + ) + w2 = torch.randn( + num_experts, + hidden_size, + intermediate_size, + device="cuda:0", + dtype=torch.bfloat16, + ) + bias13 = torch.randn(num_experts, intermediate_size * 2, device="cuda:0") * 10 + bias2 = torch.randn(num_experts, hidden_size, device="cuda:0") * 10 + router_logits = torch.rand(num_tokens, num_experts, dtype=torch.float32).cuda() + + w13, w13_scale = fp4_quantize( + w13, + torch.tensor(1.0, device="cuda:0"), + 32, + sf_use_ue8m0=True, + is_sf_swizzled_layout=False, + ) + w13_scale = w13_scale.view(torch.float8_e4m3fn).reshape( + num_experts, intermediate_size * 2, hidden_size // 32 + ) + w2, w2_scale = fp4_quantize( + w2, + torch.tensor(1.0, device="cuda:0"), + 32, + sf_use_ue8m0=True, + is_sf_swizzled_layout=False, + ) + w2_scale = w2_scale.view(torch.float8_e4m3fn).reshape( + num_experts, hidden_size, intermediate_size // 32 + ) + if act_type == "mxfp8": + hidden_states, hidden_states_scale = mxfp8_quantize( + hidden_states, is_sf_swizzled_layout=False + ) + hidden_states_scale = hidden_states_scale.view(torch.float8_e4m3fn).reshape( + *hidden_states.shape[:-1], -1 + ) + else: + hidden_states_scale = None + + # reference result + ref_result = torch.empty_like(hidden_states, dtype=torch.bfloat16) + w13_ref = mxfp4_dequantize(w13.clone(), w13_scale.clone()) + w2_ref = mxfp4_dequantize(w2.clone(), w2_scale.clone()) + bias13_ref = bias13 + bias2_ref = bias2 + if act_type == "mxfp8": + hidden_states_ref = mxfp8_dequantize(hidden_states, hidden_states_scale).to( + torch.float32 + ) + else: + hidden_states_ref = hidden_states.to(torch.float32) + # Process tokens in chunks of 32 to reduce memory usage + chunk_size = 32 + num_chunks = (num_tokens + chunk_size - 1) // chunk_size + for i in range(num_chunks): + start_idx = i * chunk_size + end_idx = min(start_idx + chunk_size, num_tokens) + chunk_result = reference_moe( + router_logits[start_idx:end_idx].to(torch.float32), + topk, + num_experts, + hidden_states_ref[start_idx:end_idx], + w13_ref, + bias13_ref, + w2_ref, + bias2_ref, + alpha, + beta, + limit, + act_type, + activation="swiglu", + use_interleaved_layout=False, + ) + ref_result[start_idx:end_idx].copy_(chunk_result) + + # trtllm-gen result + if alpha is not None: + alpha = torch.full((num_experts,), alpha, device=hidden_states.device) + if limit is not None: + limit = torch.full((num_experts,), limit, device=hidden_states.device) + if beta is not None: + beta = torch.full((num_experts,), beta, device=hidden_states.device) + tg_result = tg_mxfp4_moe( + router_logits=router_logits, + topk=topk, + num_experts=num_experts, + intermediate_size=intermediate_size, + hidden_size=hidden_size, + hidden_states=hidden_states, + hidden_states_scale=hidden_states_scale, + w13_weight=w13, + w13_weight_scale=w13_scale, + w13_bias=bias13, + w2_weight=w2, + w2_weight_scale=w2_scale, + w2_bias=bias2, + act_type=act_type, + alpha=alpha, + beta=beta, + limit=limit, + transpose_optimized=transpose_optimized, + ) + # relatively loose check since the mxfp4 quantization is less accurate + check_accuracy(ref_result, tg_result, atol=0, rtol=0.3, percent=0.8) + + +@pytest.mark.parametrize("topk", [1, 4]) +@pytest.mark.parametrize("num_experts", [32]) +@pytest.mark.parametrize("num_tokens", [1, 128]) +@pytest.mark.parametrize("intermediate_size,hidden_size", [(3072, 3072)]) +@pytest.mark.parametrize("alpha,beta,limit", [(1.0, 1.0, None), (1.702, 1.0, 7.0)]) +@pytest.mark.skipif( + not HOPPER_MXFP4_BF16_AVAILABLE, + reason="nvidia gpu sm90 and flashinfer are required for this test", +) +def test_flashinfer_cutlass_mxfp4_fused_moe( + topk: int, + num_experts: int, + num_tokens: int, + intermediate_size: int, + hidden_size: int, + alpha: float, + beta: float, + limit: float | None, +): + torch.manual_seed(42) + device = "cuda:0" + + # Inputs + hidden_states = torch.randn( + num_tokens, hidden_size, device=device, dtype=torch.bfloat16 + ) + # Random MXFP4 weights and scales (uint8), contiguous [w1; w3] + w13_q = torch.randint( + 0, + 256, + (num_experts, 2 * intermediate_size, hidden_size // 2), + device=device, + dtype=torch.uint8, + ) + w13_scale = torch.randint( + 118, + 123, + (num_experts, 2 * intermediate_size, hidden_size // 32), + device=device, + dtype=torch.uint8, + ) + + w2_q = torch.randint( + 0, + 256, + (num_experts, hidden_size, intermediate_size // 2), + device=device, + dtype=torch.uint8, + ) + w2_scale = torch.randint( + 118, + 123, + (num_experts, hidden_size, intermediate_size // 32), + device=device, + dtype=torch.uint8, + ) + # Bias contiguous [b1; b3] + bias13 = ( + torch.randn( + num_experts, 2 * intermediate_size, device=device, dtype=torch.bfloat16 + ) + * 10 + ) + bias2 = ( + torch.randn(num_experts, hidden_size, device=device, dtype=torch.bfloat16) * 10 + ) + router_logits = torch.rand( + num_tokens, num_experts, dtype=torch.float32, device=device + ) + + w13_ref = mxfp4_dequantize(w13_q.clone(), w13_scale.clone()).reshape( + num_experts, 2 * intermediate_size, hidden_size + ) + w2_ref = mxfp4_dequantize(w2_q.clone(), w2_scale.clone()).reshape( + num_experts, hidden_size, intermediate_size + ) + ref = reference_moe( + router_logits.to(torch.float32), + topk, + num_experts, + hidden_states.to(torch.float32), + w13_ref, + bias13.to(torch.float32), + w2_ref, + bias2.to(torch.float32), + alpha, + beta, + limit, + "bf16", + activation="swiglu", + use_interleaved_layout=False, + ) + + from vllm.utils.flashinfer import flashinfer_cutlass_fused_moe + + # Swap halves to arrange as [w3; w1] (kernel expectation) + w1_w, w3_w = torch.chunk(w13_q, 2, dim=1) + w13_q_swapped = torch.cat([w3_w, w1_w], dim=1) + + # SM90 mixed-input GEMM expects weights/scales in an interleaved layout; + # without it the FP4->BF16 LUT reads bytes from wrong positions for K>128. + from flashinfer.fused_moe import ( + interleave_moe_scales_for_sm90_mixed_gemm, + interleave_moe_weights_for_sm90_mixed_gemm, + ) + + w13_q_swapped = interleave_moe_weights_for_sm90_mixed_gemm( + w13_q_swapped, quant_type="fp4" + ) + w2_q = interleave_moe_weights_for_sm90_mixed_gemm(w2_q, quant_type="fp4") + + b1, b3 = torch.chunk(bias13.to(torch.float32), 2, dim=-1) + w13_b = torch.cat([b3, b1], dim=-1).to(torch.bfloat16) + + w1_s, w3_s = torch.chunk(w13_scale, 2, dim=1) + w13_s = torch.cat([w3_s, w1_s], dim=1) + w13_s_inter = interleave_moe_scales_for_sm90_mixed_gemm(w13_s) + w2_s_inter = interleave_moe_scales_for_sm90_mixed_gemm(w2_scale) + + routing_weights = torch.nn.functional.softmax( + router_logits, dim=1, dtype=torch.float32 + ) + token_final_scales, token_selected_experts = torch.topk( + routing_weights, topk, dim=-1 + ) + token_final_scales = token_final_scales / token_final_scales.sum( + dim=-1, keepdim=True + ) + token_selected_experts = token_selected_experts.to(torch.int).contiguous() + + out = torch.empty_like(hidden_states, dtype=torch.bfloat16) + if alpha is not None: + alpha = torch.full((num_experts,), alpha, device=hidden_states.device) + if beta is not None: + beta = torch.full((num_experts,), beta, device=hidden_states.device) + if limit is not None: + limit = torch.full((num_experts,), limit, device=hidden_states.device) + + _ = flashinfer_cutlass_fused_moe( + input=hidden_states, + token_selected_experts=token_selected_experts, + token_final_scales=token_final_scales, + fc1_expert_weights=w13_q_swapped, + fc2_expert_weights=w2_q, + output_dtype=torch.bfloat16, + output=out, + quant_scales=[w13_s_inter.to(torch.uint8), w2_s_inter.to(torch.uint8)], + fc1_expert_biases=w13_b, + fc2_expert_biases=bias2.to(torch.bfloat16), + swiglu_alpha=alpha, + swiglu_beta=beta, + swiglu_limit=limit, + tp_size=1, + tp_rank=0, + ep_size=1, + ep_rank=0, + use_w4_group_scaling=True, + ) + + # Allow some mismatch due to MXFP4 quantization + check_accuracy(ref, out, atol=0, rtol=0.3, percent=0.8) + + +@pytest.mark.parametrize("topk", [1, 4]) +@pytest.mark.parametrize("num_experts", [32]) +@pytest.mark.parametrize("num_tokens", [1, 128]) +@pytest.mark.parametrize("intermediate_size,hidden_size", [(3072, 3072)]) +@pytest.mark.parametrize("alpha,beta,limit", [(1.0, 1.0, None), (1.702, 1.0, 7.0)]) +@pytest.mark.skipif( + not ( + current_platform.is_cuda() + and current_platform.is_device_capability_family(100) + and has_flashinfer() + ), + reason="NVIDIA GPU sm100 and flashinfer are required for this test", +) +def test_flashinfer_cutlass_mxfp4_mxfp8_fused_moe( + topk: int, + num_experts: int, + num_tokens: int, + intermediate_size: int, + hidden_size: int, + alpha: float | None, + beta: float | None, + limit: float | None, +): + torch.manual_seed(42) + device = "cuda:0" + + # Inputs + hidden_states = torch.randn( + num_tokens, hidden_size, device=device, dtype=torch.bfloat16 + ) + # Float weights in w13 format [w1; w3] + w13 = ( + torch.randn( + num_experts, + 2 * intermediate_size, + hidden_size, + device=device, + dtype=torch.bfloat16, + ) + / 10 + ) + w2 = ( + torch.randn( + num_experts, + hidden_size, + intermediate_size, + device=device, + dtype=torch.bfloat16, + ) + / 10 + ) + # Bias contiguous [b1; b3] + bias13 = ( + torch.randn( + num_experts, 2 * intermediate_size, device=device, dtype=torch.bfloat16 + ) + * 10 + ) + bias2 = ( + torch.randn(num_experts, hidden_size, device=device, dtype=torch.bfloat16) * 10 + ) + router_logits = torch.rand( + num_tokens, num_experts, dtype=torch.float32, device=device + ) + + # Quantize weights to MXFP4 per expert (SM100 path) + from flashinfer import mxfp4_quantize + + def quant_mxfp4_batches(a: torch.Tensor, e: int): + qs, sfs = [], [] + for i in range(e): + q, sf = mxfp4_quantize(a[i].cuda()) + qs.append(q) + sfs.append(sf) + return torch.stack(qs), torch.stack(sfs) + + def dequant_mxfp4_batches(mat_fp4: torch.Tensor, scale_tensor: torch.Tensor): + num_batches = mat_fp4.size(0) + scale_tensor = scale_tensor.view(num_batches, -1) + from flashinfer import mxfp4_dequantize + + return torch.stack( + [ + mxfp4_dequantize(mat_fp4[b, :, :], scale_tensor[b, :]) + for b in range(num_batches) + ] + ) + + w13_q, w13_scale = quant_mxfp4_batches(w13, num_experts) + w2_q, w2_scale = quant_mxfp4_batches(w2, num_experts) + + # Reference result using dequantized tensors and reference_moe + w13_ref = ( + dequant_mxfp4_batches( + w13_q.view(torch.uint8), w13_scale.view(torch.uint8).reshape(-1) + ) + .to(torch.float32) + .reshape(num_experts, 2 * intermediate_size, hidden_size) + .to(device) + ) + w2_ref = ( + dequant_mxfp4_batches( + w2_q.view(torch.uint8), w2_scale.view(torch.uint8).reshape(-1) + ) + .to(torch.float32) + .reshape(num_experts, hidden_size, intermediate_size) + .to(device) + ) + + # Quantize activations for SM100 path and dequantize for reference + hidden_states_q, hidden_states_sf = mxfp8_quantize(hidden_states, True, 32) + # Reference uses BF16 input but quantizes intermediate activation to MXFP8 + ref = reference_moe( + router_logits.to(torch.float32), + topk, + num_experts, + hidden_states.to(torch.float32), + w13_ref, + bias13.to(torch.float32), + w2_ref, + bias2.to(torch.float32), + alpha, + beta, + limit, + "mxfp8", + activation="swiglu", + use_interleaved_layout=False, + ) + + # Prepare inputs for FlashInfer CUTLASS fused MoE + from vllm.utils.flashinfer import flashinfer_cutlass_fused_moe + + # Swap halves to arrange as [w3; w1] (kernel expectation) + w1_w, w3_w = torch.chunk(w13_q, 2, dim=1) + w13_q_swapped = torch.cat([w3_w, w1_w], dim=1) + + # Swap scales halves to match swapped weights + s1, s3 = torch.chunk(w13_scale, 2, dim=1) + w13_scale_swapped = torch.cat([s3, s1], dim=1) + + b1, b3 = torch.chunk(bias13.to(torch.float32), 2, dim=-1) + w13_b = torch.cat([b3, b1], dim=-1).to(torch.bfloat16) + + # Build routing for kernel + routing_weights = torch.nn.functional.softmax( + router_logits, dim=1, dtype=torch.float32 + ) + token_final_scales, token_selected_experts = torch.topk( + routing_weights, topk, dim=-1 + ) + token_final_scales = token_final_scales / token_final_scales.sum( + dim=-1, keepdim=True + ) + token_selected_experts = token_selected_experts.to(torch.int).contiguous() + + out = torch.empty_like(hidden_states, dtype=torch.bfloat16) + if alpha is not None: + alpha_t = torch.full((num_experts,), alpha, device=hidden_states.device) + else: + alpha_t = None + if beta is not None: + beta_t = torch.full((num_experts,), beta, device=hidden_states.device) + else: + beta_t = None + if limit is not None: + limit_t = torch.full((num_experts,), limit, device=hidden_states.device) + else: + limit_t = None + + # Quant scales for SM100 MXFP8+MXFP4 path + fake_input_scale = torch.ones(num_experts, device=device) + quant_scales = [ + w13_scale_swapped.view(torch.int32), + fake_input_scale, + w2_scale.view(torch.int32), + fake_input_scale, + ] + + _ = flashinfer_cutlass_fused_moe( + input=hidden_states_q, + token_selected_experts=token_selected_experts, + token_final_scales=token_final_scales, + fc1_expert_weights=w13_q_swapped.contiguous().view(torch.long), + fc2_expert_weights=w2_q.contiguous().view(torch.long), + output_dtype=torch.bfloat16, + output=out, + quant_scales=quant_scales, + fc1_expert_biases=w13_b, + fc2_expert_biases=bias2.to(torch.bfloat16), + swiglu_alpha=alpha_t, + swiglu_beta=beta_t, + swiglu_limit=limit_t, + tp_size=1, + tp_rank=0, + ep_size=1, + ep_rank=0, + use_mxfp8_act_scaling=True, + input_sf=hidden_states_sf, + ) + + # Allow some mismatch due to MXFP4 quantization + check_accuracy(ref, out, atol=0, rtol=0.3, percent=0.8) + + +@pytest.mark.parametrize("topk", [1, 4]) +@pytest.mark.parametrize("num_experts", [32]) +@pytest.mark.parametrize("num_tokens", [1, 128]) +@pytest.mark.parametrize("intermediate_size,hidden_size", [(3072, 3072)]) +@pytest.mark.parametrize("is_gated", [True], ids=["gated"]) +@pytest.mark.skipif( + not TRTLLM_GEN_MXFP8_AVAILABLE, + reason="nvidia gpu and compute capability sm100 is required for this test", +) +def test_trtllm_gen_mxfp8_block_scale_moe( + topk: int, + num_experts: int, + num_tokens: int, + intermediate_size: int, + hidden_size: int, + is_gated: bool, +): + torch.manual_seed(42) + device = "cuda:0" + + inter_size = intermediate_size * (2 if is_gated else 1) + + hidden_states = ( + torch.randn(num_tokens, hidden_size, device=device, dtype=torch.bfloat16) / 20 + ) + w13 = ( + torch.randn( + num_experts, + inter_size, + hidden_size, + device=device, + dtype=torch.bfloat16, + ) + / 20 + ) + w2 = ( + torch.randn( + num_experts, + hidden_size, + intermediate_size, + device=device, + dtype=torch.bfloat16, + ) + / 20 + ) + router_logits = torch.rand( + num_tokens, num_experts, dtype=torch.float32, device=device + ) + router_logits_kernel = router_logits.to(torch.bfloat16) + + # Quantize weights to MXFP8 and normalize scales to [E, M, K//32]. + w13_q, w13_scale = mxfp8_quantize(w13, is_sf_swizzled_layout=False) + w2_q, w2_scale = mxfp8_quantize(w2, is_sf_swizzled_layout=False) + if w13_scale.ndim == 1: + w13_scale = w13_scale.view( + num_experts, + inter_size, + hidden_size // 32, + ) + if w2_scale.ndim == 1: + w2_scale = w2_scale.view(num_experts, hidden_size, intermediate_size // 32) + + # Quantize activations to MXFP8. + hidden_states_q, hidden_states_scale = mxfp8_quantize( + hidden_states, is_sf_swizzled_layout=False + ) + if hidden_states_scale.ndim == 1: + hidden_states_scale = hidden_states_scale.view(num_tokens, hidden_size // 32) + + # Reference output using dequantized tensors + MXFP8 intermediate quantization. + w13_ref = mxfp8_dequantize(w13_q, w13_scale).to(torch.float32) + w2_ref = mxfp8_dequantize(w2_q, w2_scale).to(torch.float32) + hidden_states_ref = mxfp8_dequantize(hidden_states_q, hidden_states_scale).to( + torch.float32 + ) + bias13 = torch.zeros( + num_experts, + intermediate_size * (2 if is_gated else 1), + device=device, + ) + bias2 = torch.zeros(num_experts, hidden_size, device=device) + ref = reference_moe( + router_logits_kernel.to(torch.float32), + topk, + num_experts, + hidden_states_ref, + w13_ref, + bias13, + w2_ref, + bias2, + alpha=1.0, + beta=0.0, + limit=None, + act_type="mxfp8", + activation="swiglu" if is_gated else "relu2", + use_interleaved_layout=False, + ) + + # Shuffle weights/scales with the same indexed layout used by TRTLLM kernels. + epilogue_tile_m = 128 + gemm1_weights_shuffled = [] + gemm1_scales_shuffled = [] + gemm2_weights_shuffled = [] + gemm2_scales_shuffled = [] + for i in range(num_experts): + w13_rows = intermediate_size * (2 if is_gated else 1) + w13_interleaved = w13_q[i].clone().reshape(w13_rows, -1) + w13_scale_interleaved = w13_scale[i].clone().reshape(w13_rows, -1) + if is_gated: + w13_interleaved = reorder_rows_for_gated_act_gemm(w13_interleaved) + w13_scale_interleaved = reorder_rows_for_gated_act_gemm( + w13_scale_interleaved + ) + gemm1_weights_shuffled.append( + shuffle_matrix_a(w13_interleaved.view(torch.uint8), epilogue_tile_m) + .contiguous() + .view(w13_q.dtype) + ) + gemm2_weights_shuffled.append( + shuffle_matrix_a(w2_q[i].view(torch.uint8), epilogue_tile_m) + .contiguous() + .view(w2_q.dtype) + ) + + gemm1_scales_shuffled.append( + shuffle_matrix_sf_a( + w13_scale_interleaved.view(torch.uint8).reshape(w13_rows, -1), + epilogue_tile_m, + ) + .contiguous() + .view(w13_scale.dtype) + ) + gemm2_scales_shuffled.append( + shuffle_matrix_sf_a( + w2_scale[i].view(torch.uint8).reshape(hidden_size, -1), epilogue_tile_m + ) + .contiguous() + .view(w2_scale.dtype) + ) + + out = trtllm_fp8_block_scale_moe( + routing_logits=router_logits_kernel, + routing_bias=None, + hidden_states=hidden_states_q, + hidden_states_scale=hidden_states_scale, + gemm1_weights=torch.stack(gemm1_weights_shuffled), + gemm1_weights_scale=torch.stack(gemm1_scales_shuffled), + gemm2_weights=torch.stack(gemm2_weights_shuffled), + gemm2_weights_scale=torch.stack(gemm2_scales_shuffled), + num_experts=num_experts, + top_k=topk, + n_group=None, + topk_group=None, + intermediate_size=intermediate_size, + local_expert_offset=0, + local_num_experts=num_experts, + routed_scaling_factor=None, + routing_method_type=1, # renormalize routing + use_shuffled_weight=True, + weight_layout=0, # MajorK + fp8_quantization_type=Fp8QuantizationType.MxFp8, + ) + + # Block-scale MXFP8 kernels are approximate; require majority close. + check_accuracy(ref, out, atol=0.1, rtol=0.85, percent=0.8) + + +# ----------------------------------------------------------------------------- +# ROCm Oracle-based kernel execution tests +# ----------------------------------------------------------------------------- +# TODO: Further tighten the accuracy threshold. +# - More accurate ref moe to include activation quantization +# - Check aiter kernel accuracy. E.g., quant / dequant details. +ROCM_BACKEND_CONFIGS = { + "TRITON": { + "activation": "SWIGLUOAI", + "rtol": 0.3, + "percent": 0.95, + "requires_aiter": False, + "requires_gfx950": False, + }, + "TRITON_UNFUSED": { + "activation": "SWIGLUOAI", + "rtol": 0.3, + "percent": 0.95, + "requires_aiter": False, + "requires_gfx950": False, + }, + "AITER_MXFP4_BF16": { + "activation": "SILU", + "rtol": 1.0, + "percent": 0.7, + "requires_aiter": True, + "requires_gfx950": True, + }, + "AITER_MXFP4_FP8": { + "activation": "SWIGLUOAI", + "rtol": 0.5, + "percent": 0.9, + "requires_aiter": True, + "requires_gfx950": True, + }, +} + + +@pytest.mark.parametrize("backend_name", list(ROCM_BACKEND_CONFIGS.keys())) +@pytest.mark.parametrize("topk", [4]) +@pytest.mark.parametrize("num_experts", [8]) +@pytest.mark.parametrize("num_tokens,hidden_size,intermediate_size", [(16, 256, 256)]) +@pytest.mark.skipif( + not ROCM_AVAILABLE, + reason="ROCm is required for this test", +) +@torch.inference_mode() +def test_rocm_mxfp4_moe_oracle( + backend_name: str, + topk: int, + num_experts: int, + num_tokens: int, + hidden_size: int, + intermediate_size: int, +): + """ + Test ROCm MXFP4 MoE using oracle functions. + + This test validates that the oracle functions work end-to-end: + - select_mxfp4_moe_backend() selects a valid backend + - convert_gpt_oss_weight_to_mxfp4_moe_kernel_format() converts weights without error + - make_mxfp4_moe_quant_config() builds a valid quant config + - make_mxfp4_moe_kernel() creates a kernel that runs without error + - The kernel output is within accuracy tolerance of reference + """ + config = ROCM_BACKEND_CONFIGS[backend_name] + + # Check platform requirements + if not ROCM_TRITON_KERNELS_AVAILABLE: + pytest.skip("triton_kernels required for quantization") + if config["requires_aiter"] and not ROCM_AITER_AVAILABLE: + pytest.skip(f"Backend {backend_name} requires AITER") + if config["requires_gfx950"] and not ROCM_GFX950: + pytest.skip(f"Backend {backend_name} requires GFX950") + + from vllm.config import VllmConfig, set_current_vllm_config + from vllm.model_executor.layers.fused_moe.activation import MoEActivation + from vllm.model_executor.layers.fused_moe.oracle.mxfp4 import ( + Mxfp4MoeBackend, + backend_to_kernel_cls, + convert_gpt_oss_weight_to_mxfp4_moe_kernel_format, + make_mxfp4_moe_kernel, + make_mxfp4_moe_quant_config, + ) + from vllm.v1.worker.workspace import init_workspace_manager + + # Initialize workspace manager (needed for modular kernels) + init_workspace_manager(torch.accelerator.current_device_index()) + + # Map string to enum + backend = Mxfp4MoeBackend[backend_name] + + # Get experts class from oracle + experts_cls_list = backend_to_kernel_cls(backend) + if experts_cls_list is None or len(experts_cls_list) == 0: + pytest.skip(f"Backend {backend_name} not available") + + # Use first experts class + experts_cls = experts_cls_list[0] + + torch.manual_seed(42) + dtype = torch.bfloat16 + device = "cuda:0" + + # Create MoE config with Renormalize routing (required by monolithic kernels) + from vllm.model_executor.layers.fused_moe import FusedMoEConfig + from vllm.model_executor.layers.fused_moe.config import ( + FusedMoEParallelConfig, + RoutingMethodType, + ) + + moe_config = FusedMoEConfig( + num_experts=num_experts, + experts_per_token=topk, + hidden_dim=hidden_size, + intermediate_size=intermediate_size, + num_local_experts=num_experts, + num_logical_experts=num_experts, + moe_parallel_config=FusedMoEParallelConfig.make_no_parallel(), + activation=MoEActivation[config["activation"]], + in_dtype=dtype, + device="cuda", + routing_method=RoutingMethodType.Renormalize, + ) + + # Create float weights in checkpoint format: + # w13: [num_experts, 2*intermediate_size, hidden_size] + # w2: [num_experts, hidden_size, intermediate_size] + w13_float = torch.randn( + num_experts, 2 * intermediate_size, hidden_size, dtype=dtype, device=device + ) + w2_float = torch.randn( + num_experts, hidden_size, intermediate_size, dtype=dtype, device=device + ) + + # dynamic_mxfp4_quant expects 2D input, so reshape 3D weights + # w13: [E, 2*I, H] -> [E*2*I, H] -> quantize -> [E, 2*I, H//2] + # w2: [E, H, I] -> [E*H, I] -> quantize -> [E, H, I//2] + w13_2d = w13_float.reshape(-1, hidden_size) + w13_quant_2d, w13_scale_2d = dynamic_mxfp4_quant(w13_2d) + w13_quant = w13_quant_2d.reshape(num_experts, 2 * intermediate_size, -1) + w13_scale = w13_scale_2d.reshape(num_experts, 2 * intermediate_size, -1) + + w2_2d = w2_float.reshape(-1, intermediate_size) + w2_quant_2d, w2_scale_2d = dynamic_mxfp4_quant(w2_2d) + w2_quant = w2_quant_2d.reshape(num_experts, hidden_size, -1) + w2_scale = w2_scale_2d.reshape(num_experts, hidden_size, -1) + + w13_bias = torch.randn( + num_experts, 2 * intermediate_size, dtype=dtype, device=device + ) + w2_bias = torch.randn(num_experts, hidden_size, dtype=dtype, device=device) + + # Create static input scales for W4A8 backend (AITER_MXFP4_FP8) + w13_input_scale: torch.Tensor | None = None + w2_input_scale: torch.Tensor | None = None + if backend_name == "AITER_MXFP4_FP8": + # Static FP8 scales: one scale per expert + w13_input_scale = torch.ones(num_experts, dtype=torch.float32, device=device) + w2_input_scale = torch.ones(num_experts, dtype=torch.float32, device=device) + + # Create mock layer for oracle functions + class MockLayer: + w13_weight: torch.Tensor + w2_weight: torch.Tensor + w13_weight_scale: torch.Tensor + w2_weight_scale: torch.Tensor + w13_input_scale: torch.Tensor | None + w2_input_scale: torch.Tensor | None + + layer = MockLayer() + layer.w13_weight = w13_quant + layer.w2_weight = w2_quant + layer.w13_weight_scale = w13_scale + layer.w2_weight_scale = w2_scale + layer.w13_input_scale = w13_input_scale + layer.w2_input_scale = w2_input_scale + + # Convert weights using oracle + w13_conv, w2_conv, w13_scale_conv, w2_scale_conv, w13_bias_conv, w2_bias_conv = ( + convert_gpt_oss_weight_to_mxfp4_moe_kernel_format( + mxfp4_backend=backend, + layer=layer, # type: ignore[arg-type] + w13_weight=w13_quant, + w2_weight=w2_quant, + w13_weight_scale=w13_scale, + w2_weight_scale=w2_scale, + w13_bias=w13_bias, + w2_bias=w2_bias, + ) + ) + + # Build quant config using oracle + quant_config = make_mxfp4_moe_quant_config( + mxfp4_backend=backend, + w1_scale=w13_scale_conv, + w2_scale=w2_scale_conv, + w1_bias=w13_bias_conv, + w2_bias=w2_bias_conv, + a1_scale=w13_input_scale, + a2_scale=w2_input_scale, + ) + + # Select activation based on backend + activation_name = str(config["activation"]) + activation = MoEActivation[activation_name] + + # Build kernel using oracle + assert quant_config is not None, "Failed to create quant config" + with set_current_vllm_config(VllmConfig()): + kernel = make_mxfp4_moe_kernel( + moe_quant_config=quant_config, + moe_config=moe_config, + mxfp4_backend=backend, + experts_cls=experts_cls, + routing_tables=None, + layer=None, + ) + + # Create inputs + x = torch.randn(num_tokens, hidden_size, dtype=dtype, device=device) + router_logits = torch.randn( + num_tokens, num_experts, dtype=torch.float32, device=device + ) + topk_weights, topk_ids = torch.topk(router_logits, k=topk, dim=-1, sorted=True) + topk_weights = torch.nn.functional.softmax(topk_weights, dim=-1) + + # Run kernel - use appropriate method based on impl type + if kernel.is_monolithic: + # Monolithic impl uses router_logits + out = kernel.apply_monolithic( + hidden_states=x, + w1=w13_conv, + w2=w2_conv, + router_logits=router_logits, + activation=activation, + global_num_experts=num_experts, + expert_map=None, + apply_router_weight_on_input=False, + ) + else: + # Modular impl uses topk_weights and topk_ids + out = kernel.apply( + hidden_states=x, + w1=w13_conv, + w2=w2_conv, + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=activation, + global_num_experts=num_experts, + expert_map=None, + apply_router_weight_on_input=False, + ) + + # Verify output is valid (no NaN/Inf) and has expected shape + assert out.shape == (num_tokens, hidden_size), f"Unexpected shape: {out.shape}" + assert not torch.any(torch.isnan(out)), "Output contains NaN" + assert not torch.any(torch.isinf(out)), "Output contains Inf" + + # Verify output has reasonable magnitude (not all zeros) + assert out.abs().max() > 0.01, "Output is effectively zero" + + # Dequantize weights for reference computation + w13_dq = upcast_from_mxfp( + w13_quant.view(torch.uint8), w13_scale, torch.bfloat16, axis=-1 + ) + w2_dq = upcast_from_mxfp( + w2_quant.view(torch.uint8), w2_scale, torch.bfloat16, axis=-1 + ) + + # Determine activation type and layout + # SWIGLUOAI uses interleaved layout (gate/up alternating) + # SILU uses chunked layout (first half gate, second half up) + use_interleaved = activation == MoEActivation.SWIGLUOAI + if activation in [MoEActivation.SWIGLUOAI, MoEActivation.SILU]: + act_name = "swiglu" + else: + act_name = "relu2" + + ref = reference_moe( + router_logits, + topk, + num_experts, + x.to(torch.float32), + w13_dq.to(torch.float32), + w13_bias.to(torch.float32), + w2_dq.to(torch.float32), + w2_bias.to(torch.float32), + alpha=1.702 if activation == MoEActivation.SWIGLUOAI else 1.0, + beta=1.0 if activation == MoEActivation.SWIGLUOAI else 0.0, + limit=7.0 if activation == MoEActivation.SWIGLUOAI else None, + act_type="bf16", + activation=act_name, + use_interleaved_layout=use_interleaved, + ) + + # Compute and print accuracy statistics + diff = (ref.float() - out.float()).abs() + rel_diff = diff / (ref.float().abs() + 1e-6) + + print(f"\n[{backend_name}] Accuracy statistics:") + print( + f" Reference: min={ref.min():.4f}, max={ref.max():.4f}, mean={ref.mean():.4f}" + ) + print( + f" Output: min={out.min():.4f}, max={out.max():.4f}, mean={out.mean():.4f}" + ) + print( + f" Abs diff: min={diff.min():.4f}, max={diff.max():.4f}, " + f"mean={diff.mean():.4f}" + ) + print( + f" Rel diff: min={rel_diff.min():.4f}, max={rel_diff.max():.4f}, " + f"mean={rel_diff.mean():.4f}" + ) + + # Check what percentage of values are within various tolerances + for rtol in [0.1, 0.5, 1.0, 2.0]: + within_tol = (diff <= rtol * out.float().abs()).float().mean() + print(f" Within rtol={rtol}: {within_tol * 100:.1f}%") + + # Check accuracy using per-backend thresholds + check_accuracy(ref, out, atol=0.1, rtol=config["rtol"], percent=config["percent"]) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_profile_modular_kernel.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_profile_modular_kernel.py new file mode 100644 index 0000000000000000000000000000000000000000..b2fbf705b3abdfd058935f27db83c891c3b63f98 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_profile_modular_kernel.py @@ -0,0 +1,38 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm.model_executor.layers.fused_moe import TritonExperts +from vllm.model_executor.layers.fused_moe.prepare_finalize import ( + MoEPrepareAndFinalizeNoDPEPModular, +) + +from .modular_kernel_tools.common import Config +from .modular_kernel_tools.profile_modular_kernel import run + + +@pytest.mark.skipif( + not torch.cuda.is_available(), + reason="profile_modular_kernel requires a CUDA device", +) +def test_profile_modular_kernel_smoke(tmp_path): + config = Config( + Ms=[16], + K=128, + N=256, + E=4, + topks=[2], + dtype=torch.bfloat16, + quant_config=None, + prepare_finalize_type=MoEPrepareAndFinalizeNoDPEPModular, + fused_experts_type=TritonExperts, + world_size=1, + torch_trace_dir_path=str(tmp_path), + ) + + run(config) + + traces = list(tmp_path.glob("m*_*_trace.json")) + assert traces, "profile_modular_kernel.run did not emit any chrome traces" diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_rocm_aiter_topk.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_rocm_aiter_topk.py new file mode 100644 index 0000000000000000000000000000000000000000..9811e407c06b8f39c47c5af949897a41cd99b6b9 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_rocm_aiter_topk.py @@ -0,0 +1,221 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# This is a test for the AITER ops. +# It tests if the AITER ops are +# 1. correctly registered as custom ops +# 2. correctly defined the relationship between +# implementation and fake function +# 3. can be used with torch.compile +# This file will be skipped if AITER is not installed +# and the platform is not ROCm. + +import importlib.util + +import pytest +import torch + +from vllm.platforms import current_platform + +if not current_platform.is_rocm(): + pytest.skip("This test can only run on ROCm.", allow_module_level=True) + +# this import statement is needed to ensure the ops are registered +import vllm.model_executor.layers.fused_moe.experts.rocm_aiter_moe # noqa: F401 + +# need to import once to ensure the ops are registered +# Check if aiter package is installed +aiter_available = importlib.util.find_spec("aiter") is not None + +if not aiter_available: + pytest.skip("These tests require AITER to run.", allow_module_level=True) + + +def test_rocm_aiter_biased_grouped_topk_custom_op_registration(): + """Test that the custom op is correctly registered.""" + # Check if the op exists in torch.ops.vllm + assert hasattr(torch.ops.vllm, "rocm_aiter_biased_grouped_topk") + + # Check if the op is callable + assert callable(torch.ops.vllm.rocm_aiter_biased_grouped_topk) + + +def test_rocm_aiter_grouped_topk_custom_op_registration(): + """Test that the custom op is correctly registered.""" + # Check if the op exists in torch.ops.vllm + assert hasattr(torch.ops.vllm, "rocm_aiter_grouped_topk") + + # Check if the op is callable + assert callable(torch.ops.vllm.rocm_aiter_grouped_topk) + + +def test_rocm_aiter_biased_grouped_topk_torch_compile_compatibility(): + """Test that the op can be used with torch.compile.""" + # Create test tensors + token = 64 + expert = 256 + num_expert_group = 8 + topk = 8 + topk_group = 4 + renormalize = True + scale_factor = 1.0 + + gating_output = torch.randn((token, expert), dtype=torch.bfloat16, device="cuda") + e_score_correction_bias = torch.randn( + (expert,), dtype=torch.bfloat16, device="cuda" + ) + + device = gating_output.device + topk_ids = torch.empty((token, topk), dtype=torch.int32, device=device) + topk_weights = torch.empty((token, topk), dtype=torch.float32, device=device) + + # Define a function that uses the op + def biased_grouped_topk_fn( + gating_output, e_score_correction_bias, topk_weights, topk_ids + ): + return torch.ops.vllm.rocm_aiter_biased_grouped_topk( + gating_output, + e_score_correction_bias, + topk_weights, + topk_ids, + num_expert_group, + topk_group, + renormalize, + scale_factor, + ) + + # Verify the op's fake implementation + torch.library.opcheck( + torch.ops.vllm.rocm_aiter_biased_grouped_topk, + (gating_output, e_score_correction_bias, topk_weights, topk_ids), + kwargs={ + "num_expert_group": num_expert_group, + "topk_group": topk_group, + "need_renorm": renormalize, + "routed_scaling_factor": scale_factor, + }, + test_utils=("test_faketensor"), + ) + + # Compile the function with appropriate settings + compiled_fn = torch.compile( + biased_grouped_topk_fn, + fullgraph=True, + backend="inductor", + mode="reduce-overhead", + dynamic=False, + ) + + topk_weights_original = torch.empty( + (token, topk), dtype=torch.float32, device=device + ) + topk_ids_original = torch.empty((token, topk), dtype=torch.int32, device=device) + + topk_weights_compiled = torch.empty( + (token, topk), dtype=torch.float32, device=device + ) + topk_ids_compiled = torch.empty((token, topk), dtype=torch.int32, device=device) + + # Run both compiled (V1 graph mode) and uncompiled versions (V1 eager mode) + biased_grouped_topk_fn( + gating_output, e_score_correction_bias, topk_weights_original, topk_ids_original + ) + compiled_fn( + gating_output, e_score_correction_bias, topk_weights_compiled, topk_ids_compiled + ) + + # Sort the results for comparison since the order might not be deterministic + topk_ids_original, indices_original = torch.sort(topk_ids_original) + topk_weights_original = torch.gather(topk_weights_original, 1, indices_original) + + topk_ids_compiled, indices_compiled = torch.sort(topk_ids_compiled) + topk_weights_compiled = torch.gather(topk_weights_compiled, 1, indices_compiled) + + # Verify results match + assert torch.allclose( + topk_weights_original, topk_weights_compiled, rtol=1e-2, atol=1e-2 + ) + assert torch.allclose(topk_ids_original, topk_ids_compiled) + + +def test_rocm_aiter_grouped_topk_torch_compile_compatibility(): + """Test that the op can be used with torch.compile.""" + # Create test tensors + token = 64 + expert = 256 + num_expert_group = 8 + topk = 8 + topk_group = 4 + renormalize = True + scoring_func = "softmax" + scale_factor = 1.0 + + gating_output = torch.randn((token, expert), dtype=torch.bfloat16, device="cuda") + + device = gating_output.device + topk_ids = torch.empty((token, topk), dtype=torch.int32, device=device) + topk_weights = torch.empty((token, topk), dtype=torch.float32, device=device) + + # Define a function that uses the op + def grouped_topk_fn(gating_output, topk_weights, topk_ids, scoring_func): + return torch.ops.vllm.rocm_aiter_grouped_topk( + gating_output, + topk_weights, + topk_ids, + num_expert_group, + topk_group, + renormalize, + scoring_func, + scale_factor, + ) + + # Verify the op's fake implementation + torch.library.opcheck( + torch.ops.vllm.rocm_aiter_grouped_topk, + (gating_output, topk_weights, topk_ids), + kwargs={ + "num_expert_group": num_expert_group, + "topk_group": topk_group, + "need_renorm": renormalize, + "scoring_func": scoring_func, + "routed_scaling_factor": scale_factor, + }, + test_utils=("test_faketensor"), + ) + + # Compile the function with appropriate settings + compiled_fn = torch.compile( + grouped_topk_fn, + fullgraph=True, + backend="inductor", + mode="reduce-overhead", + dynamic=False, + ) + + topk_weights_original = torch.empty( + (token, topk), dtype=torch.float32, device=device + ) + topk_ids_original = torch.empty((token, topk), dtype=torch.int32, device=device) + + topk_weights_compiled = torch.empty( + (token, topk), dtype=torch.float32, device=device + ) + topk_ids_compiled = torch.empty((token, topk), dtype=torch.int32, device=device) + + # Run both compiled (V1 graph mode) and uncompiled versions (V1 eager mode) + grouped_topk_fn( + gating_output, topk_weights_original, topk_ids_original, scoring_func + ) + compiled_fn(gating_output, topk_weights_compiled, topk_ids_compiled, scoring_func) + + # Sort the results for comparison since the order might not be deterministic + topk_ids_original, indices_original = torch.sort(topk_ids_original) + topk_weights_original = torch.gather(topk_weights_original, 1, indices_original) + + topk_ids_compiled, indices_compiled = torch.sort(topk_ids_compiled) + topk_weights_compiled = torch.gather(topk_weights_compiled, 1, indices_compiled) + + # Verify results match + assert torch.allclose( + topk_weights_original, topk_weights_compiled, rtol=1e-2, atol=1e-2 + ) + assert torch.allclose(topk_ids_original, topk_ids_compiled) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_routing.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_routing.py new file mode 100644 index 0000000000000000000000000000000000000000..642a29b21cd0148af6da54dea8c2b7fa2ff02f98 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_routing.py @@ -0,0 +1,850 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from collections.abc import Callable +from unittest.mock import patch + +import pytest +import torch + +from vllm._aiter_ops import rocm_aiter_ops +from vllm.distributed.eplb.eplb_state import EplbLayerState +from vllm.model_executor.layers.fused_moe.router.base_router import ( + eplb_map_to_physical_and_record, +) +from vllm.model_executor.layers.fused_moe.router.router_factory import ( + create_fused_moe_router, +) +from vllm.model_executor.models.llama4 import Llama4MoE +from vllm.platforms import current_platform + + +def _is_aiter_capable() -> bool: + """Check if the platform supports AITER (gfx942/gfx950).""" + if not current_platform.is_rocm(): + return False + try: + from vllm.platforms.rocm import _ON_MI3XX + + return _ON_MI3XX + except ImportError: + return False + + +# Test parameters +MK_S = [(32, 256), (64, 512)] +TOP_KS = [2, 4, 6] +NUM_EXPERTS = [8, 16, 64] + + +def setup_eplb_state( + enable_eplb: bool, global_num_experts: int +) -> EplbLayerState | None: + if not enable_eplb: + return None + + # Initialize EPLB state with proper tensors for testing + # For testing purposes, we use a simple 1:1 mapping (no redundant experts) + # expert_load_view: tracks load on each expert (shape: num_experts) + expert_load_view = torch.zeros(global_num_experts, dtype=torch.int32, device="cuda") + + # logical_to_physical_map: maps logical experts to physical experts + # Shape: (num_logical_experts, max_slots) + # For testing, use simple 1:1 mapping with single slot per expert + logical_to_physical_map = torch.arange( + global_num_experts, dtype=torch.int64, device="cuda" + ).unsqueeze(-1) + + # logical_replica_count: number of replicas per logical expert + # Shape: (num_logical_experts,) + # For testing, each logical expert has exactly 1 replica + logical_replica_count = torch.ones( + global_num_experts, dtype=torch.int64, device="cuda" + ) + should_record_tensor = torch.ones((), dtype=torch.bool, device="cuda") + num_unpadded_tokens_tensors = [torch.tensor(0, dtype=torch.int32, device="cuda")] + + return EplbLayerState( + expert_load_view=expert_load_view, + logical_to_physical_map=logical_to_physical_map, + logical_replica_count=logical_replica_count, + should_record_tensor=should_record_tensor, + num_unpadded_tokens_tensors=num_unpadded_tokens_tensors, + ) + + +def make_test_data( + m: int, k: int, num_experts: int +) -> tuple[torch.Tensor, torch.Tensor]: + hidden_states = torch.randn((m, k), device="cuda") / 10 + logits = torch.randn((m, num_experts), device="cuda") + return hidden_states, logits + + +def make_e_score_correction_bias( + e_score_correction_bias_val: float, + num_experts: int, +) -> torch.Tensor: + # return torch.randn(num_experts, device="cuda") * e_score_correction_bias_val + return torch.full( + (num_experts,), e_score_correction_bias_val, device="cuda", dtype=torch.float32 + ) + + +def assert_routing_results_close( + topk_weights: torch.Tensor, + topk_ids: torch.Tensor, + baseline_weights: torch.Tensor, + baseline_ids: torch.Tensor, + rtol: float = 1e-3, + atol: float = 1e-3, +): + """ + Compare routing results, sorting by expert ID first to handle non-deterministic + ordering from sorted=False in topk. + """ + # Sort both results by expert IDs for consistent comparison + sorted_indices_actual = torch.argsort(topk_ids, dim=-1) + sorted_indices_baseline = torch.argsort(baseline_ids.to(topk_ids.dtype), dim=-1) + + # Gather the sorted values + topk_ids_sorted = torch.gather(topk_ids, 1, sorted_indices_actual) + topk_weights_sorted = torch.gather(topk_weights, 1, sorted_indices_actual) + baseline_ids_sorted = torch.gather( + baseline_ids.to(topk_ids.dtype), 1, sorted_indices_baseline + ) + baseline_weights_sorted = torch.gather(baseline_weights, 1, sorted_indices_baseline) + + # Compare + torch.testing.assert_close(topk_ids_sorted, baseline_ids_sorted) + torch.testing.assert_close( + topk_weights_sorted, baseline_weights_sorted, rtol=rtol, atol=atol + ) + + +def assert_aiter_routing_valid( + topk_weights: torch.Tensor, + topk_ids: torch.Tensor, + top_k: int, + num_experts: int, + renormalize: bool, + routed_scaling_factor: float = 1.0, +): + """Validate AITER routing outputs are structurally correct. + + AITER grouped_topk is a fundamentally different implementation from + the Python baseline (different group selection, scoring internals), + so numerical comparison is not meaningful. Instead we verify the + outputs satisfy the routing contract: correct shapes, valid expert + IDs, non-negative weights, and proper normalization.""" + n_tokens = topk_weights.shape[0] + + # Shape + assert topk_weights.shape == (n_tokens, top_k), ( + f"weights shape {topk_weights.shape} != ({n_tokens}, {top_k})" + ) + assert topk_ids.shape == (n_tokens, top_k), ( + f"ids shape {topk_ids.shape} != ({n_tokens}, {top_k})" + ) + + # Expert IDs in valid range + assert (topk_ids >= 0).all() and (topk_ids < num_experts).all(), ( + f"expert IDs out of range [0, {num_experts}): " + f"min={topk_ids.min().item()}, max={topk_ids.max().item()}" + ) + + # No duplicate expert IDs per token + for i in range(n_tokens): + ids = topk_ids[i] + assert ids.unique().numel() == top_k, ( + f"token {i}: duplicate expert IDs {ids.tolist()}" + ) + + # Weights are non-negative + assert (topk_weights >= 0).all(), "negative routing weights" + + # If renormalized, weights should sum to ~scaling_factor per token + # (renormalization to 1.0 happens before scaling) + if renormalize: + expected_sum = routed_scaling_factor + sums = topk_weights.sum(dim=-1) + torch.testing.assert_close( + sums, + torch.full_like(sums, expected_sum), + atol=1e-3, + rtol=1e-3, + ) + + +def baseline_fused_topk( + router_logits: torch.Tensor, top_k: int, renormalize: bool +) -> tuple[torch.Tensor, torch.Tensor]: + """ + Baseline for standard fused top-k routing. + + Algorithm: + 1. Apply softmax to router logits + 2. Select top-k experts + 3. Optionally renormalize the weights + """ + scores = torch.softmax(router_logits, dim=-1, dtype=torch.float32) + # Use sorted=False to match vllm implementation (vllm_is_batch_invariant + # defaults to False) + topk_weights, topk_ids = torch.topk(scores, top_k, dim=-1, sorted=False) + + if renormalize: + topk_weights = topk_weights / topk_weights.sum(dim=-1, keepdim=True) + + return topk_weights.to(torch.float32), topk_ids.to(torch.int32) + + +def baseline_fused_topk_bias( + router_logits: torch.Tensor, + top_k: int, + renormalize: bool, + e_score_correction_bias: torch.Tensor, + routed_scaling_factor: float, +) -> tuple[torch.Tensor, torch.Tensor]: + """ + Baseline for fused top-k with bias correction. + + Algorithm: + 1. Apply softmax to router logits + 2. Add bias to scores for expert selection + 3. Select top-k experts using biased scores + 4. Get weights from original (unbiased) scores + 5. Apply routed scaling factor + 6. Optionally renormalize the weights + """ + # Apply softmax to get scores + scores = torch.softmax(router_logits, dim=-1, dtype=torch.float32) + + # Add bias for expert selection + scores_for_choice = scores + e_score_correction_bias.unsqueeze(0) + + # Select top-k using biased scores (sorted=False to match implementation) + topk_ids = torch.topk(scores_for_choice, k=top_k, dim=-1, sorted=False)[1] + + # Get weights from original scores (not biased) + topk_weights = scores.gather(1, topk_ids) + + # Renormalize if needed (BEFORE applying scaling factor) + if renormalize: + topk_weights = topk_weights / topk_weights.sum(dim=-1, keepdim=True) + + # Apply scaling factor (AFTER renormalization, if applicable) + if routed_scaling_factor != 1.0: + topk_weights *= routed_scaling_factor + + return topk_weights.to(torch.float32), topk_ids.to(torch.int32) + + +def baseline_grouped_topk( + router_logits: torch.Tensor, + top_k: int, + num_expert_group: int, + topk_group: int, + scoring_func: str, + renormalize: bool, + e_score_correction_bias: torch.Tensor | None, + routed_scaling_factor: float, +) -> tuple[torch.Tensor, torch.Tensor]: + """ + Baseline for grouped top-k routing (e.g., DeepSeek). + + Algorithm: + 1. Apply scoring function (softmax or sigmoid) + 2. Optionally add bias + 3. Select top-k groups based on max scores within each group + 4. Mask scores to only include selected groups + 5. Select top-k experts from masked scores + 6. Apply scaling factor + 7. Optionally renormalize + """ + num_token = router_logits.shape[0] + + # Apply scoring function + if scoring_func == "softmax": + scores = torch.softmax(router_logits, dim=-1, dtype=torch.float32) + elif scoring_func == "sigmoid": + scores = torch.sigmoid(router_logits.float()) + else: + raise ValueError(f"Unsupported scoring function: {scoring_func}") + + # Handle bias correction + if e_score_correction_bias is not None: + original_scores = scores + scores = scores + e_score_correction_bias.unsqueeze(0) + # For bias case, use sum of top-2 scores in each group + group_scores = ( + scores.view(num_token, num_expert_group, -1).topk(2, dim=-1)[0].sum(dim=-1) + ) + else: + # Use max score in each group + group_scores = scores.view(num_token, num_expert_group, -1).max(dim=-1).values + + # Select top-k groups + group_idx = torch.topk(group_scores, k=topk_group, dim=-1, sorted=False)[1] + + # Create mask for selected groups + group_mask = torch.zeros_like(group_scores) + group_mask.scatter_(1, group_idx, 1) + + # Expand mask to all experts + score_mask = ( + group_mask.unsqueeze(-1) + .expand(num_token, num_expert_group, scores.shape[-1] // num_expert_group) + .reshape(num_token, -1) + ) + + # Mask scores (set non-selected to -inf) + tmp_scores = scores.masked_fill(~score_mask.bool(), float("-inf")) + + # Select top-k experts + if e_score_correction_bias is not None: + topk_ids = torch.topk(tmp_scores, k=top_k, dim=-1, sorted=False)[1] + topk_weights = original_scores.gather(1, topk_ids) + else: + topk_weights, topk_ids = torch.topk(tmp_scores, k=top_k, dim=-1, sorted=False) + + # Renormalize if needed + if renormalize: + topk_weights = topk_weights / topk_weights.sum(dim=-1, keepdim=True) + + # Apply scaling factor + if routed_scaling_factor != 1.0: + topk_weights *= routed_scaling_factor + + return topk_weights.to(torch.float32), topk_ids.to(torch.int32) + + +def baseline_custom_llama4( + router_logits: torch.Tensor, top_k: int +) -> tuple[torch.Tensor, torch.Tensor]: + """ + Baseline for Llama4 custom routing. + + Algorithm: + 1. Select top-k expert indices (without softmax) + 2. Apply sigmoid to the selected scores + """ + router_scores, router_indices = torch.topk(router_logits, top_k, dim=-1) + router_scores = torch.sigmoid(router_scores.float()) + return router_scores.to(torch.float32), router_indices.to(torch.int32) + + +@pytest.mark.parametrize("m,k", MK_S) +@pytest.mark.parametrize("top_k", TOP_KS) +@pytest.mark.parametrize("global_num_experts", NUM_EXPERTS) +@pytest.mark.parametrize("renormalize", [False, True]) +@pytest.mark.parametrize("enable_eplb", [False, True]) +def test_fused_topk( + m: int, + k: int, + top_k: int, + global_num_experts: int, + renormalize: bool, + enable_eplb: bool, +): + if top_k > global_num_experts: + pytest.skip(f"top_k ({top_k}) > global_num_experts ({global_num_experts})") + + eplb_state = setup_eplb_state(enable_eplb, global_num_experts) + router = create_fused_moe_router( + top_k=top_k, + global_num_experts=global_num_experts, + renormalize=renormalize, + eplb_state=eplb_state, + ) + + hidden_states, router_logits = make_test_data(m, k, global_num_experts) + + # Get router output + topk_weights, topk_ids = router.select_experts(hidden_states, router_logits) + + # Compute baseline + baseline_weights, baseline_ids = baseline_fused_topk( + router_logits, top_k, renormalize + ) + + # Compare results + assert_routing_results_close(topk_weights, topk_ids, baseline_weights, baseline_ids) + + +@pytest.mark.parametrize("m,k", MK_S) +@pytest.mark.parametrize("top_k", TOP_KS) +@pytest.mark.parametrize("global_num_experts", NUM_EXPERTS) +@pytest.mark.parametrize("renormalize", [False, True]) +@pytest.mark.parametrize("enable_eplb", [False, True]) +@pytest.mark.parametrize("e_score_correction_bias_val", [0.9]) +@pytest.mark.parametrize("routed_scaling_factor", [1.0, 1.1]) +def test_fused_topk_bias( + m: int, + k: int, + top_k: int, + global_num_experts: int, + renormalize: bool, + enable_eplb: bool, + e_score_correction_bias_val: float, + routed_scaling_factor: float, +): + if top_k > global_num_experts: + pytest.skip(f"top_k ({top_k}) > global_num_experts ({global_num_experts})") + + eplb_state = setup_eplb_state(enable_eplb, global_num_experts) + + e_score_correction_bias = make_e_score_correction_bias( + e_score_correction_bias_val, + global_num_experts, + ) + + router = create_fused_moe_router( + e_score_correction_bias=e_score_correction_bias, + routed_scaling_factor=routed_scaling_factor, + top_k=top_k, + global_num_experts=global_num_experts, + renormalize=renormalize, + eplb_state=eplb_state, + ) + + hidden_states, router_logits = make_test_data(m, k, global_num_experts) + + # Get router output + topk_weights, topk_ids = router.select_experts(hidden_states, router_logits) + + # Compute baseline + baseline_weights, baseline_ids = baseline_fused_topk_bias( + router_logits, + top_k, + renormalize, + e_score_correction_bias, + routed_scaling_factor, + ) + + # Compare results + assert_routing_results_close(topk_weights, topk_ids, baseline_weights, baseline_ids) + + +@pytest.mark.parametrize("m,k", MK_S) +@pytest.mark.parametrize("top_k", TOP_KS) +@pytest.mark.parametrize( + "global_num_experts,num_expert_group,topk_group", + [ + (64, 8, 4), # 8 groups of 8 experts, select 4 groups + (32, 4, 2), # 4 groups of 8 experts, select 2 groups + ], +) +@pytest.mark.parametrize("renormalize", [False, True]) +@pytest.mark.parametrize("enable_eplb", [False, True]) +@pytest.mark.parametrize("e_score_correction_bias_val", [0.9]) +@pytest.mark.parametrize("routed_scaling_factor", [1.0, 1.1]) +@pytest.mark.parametrize("scoring_func", ["sigmoid", "softmax"]) +def test_grouped_topk( + m: int, + k: int, + top_k: int, + global_num_experts: int, + renormalize: bool, + enable_eplb: bool, + num_expert_group: int, + topk_group: int, + scoring_func: str, + e_score_correction_bias_val: float, + routed_scaling_factor: float, +): + if top_k > global_num_experts: + pytest.skip(f"top_k ({top_k}) > global_num_experts ({global_num_experts})") + + eplb_state = setup_eplb_state(enable_eplb, global_num_experts) + + e_score_correction_bias = make_e_score_correction_bias( + e_score_correction_bias_val, + global_num_experts, + ) + + router = create_fused_moe_router( + use_grouped_topk=True, + num_expert_group=num_expert_group, + topk_group=topk_group, + scoring_func=scoring_func, + e_score_correction_bias=e_score_correction_bias, + routed_scaling_factor=routed_scaling_factor, + top_k=top_k, + global_num_experts=global_num_experts, + renormalize=renormalize, + eplb_state=eplb_state, + ) + + hidden_states, router_logits = make_test_data(m, k, global_num_experts) + + # Compute baseline (pure Python implementation) + baseline_weights, baseline_ids = baseline_grouped_topk( + router_logits, + top_k, + num_expert_group, + topk_group, + scoring_func, + renormalize, + e_score_correction_bias, + routed_scaling_factor, + ) + + # Test 1: Python/Triton path against baseline (exact match) + with patch( + "vllm.model_executor.layers.fused_moe.router.grouped_topk_router.rocm_aiter_ops.is_fused_moe_enabled", + return_value=False, + ): + py_weights, py_ids = router.select_experts(hidden_states, router_logits) + assert_routing_results_close(py_weights, py_ids, baseline_weights, baseline_ids) + + # Test 2: AITER path — verify outputs are structurally valid. + # AITER grouped_topk is a different implementation so we can't + # compare numerically against the Python baseline. + if _is_aiter_capable(): + # Force-enable AITER for gfx942/gfx950 regardless of env var, + # so CI always exercises this path on capable hardware. + with patch.object(rocm_aiter_ops, "_AITER_ENABLED", True): + aiter_weights, aiter_ids = router.select_experts( + hidden_states, router_logits + ) + assert_aiter_routing_valid( + aiter_weights, + aiter_ids, + top_k, + global_num_experts, + renormalize, + routed_scaling_factor, + ) + + +@pytest.mark.parametrize("m,k", MK_S) +@pytest.mark.parametrize("top_k", TOP_KS) +@pytest.mark.parametrize("global_num_experts", NUM_EXPERTS) +@pytest.mark.parametrize("renormalize", [False, True]) +@pytest.mark.parametrize("enable_eplb", [False, True]) +@pytest.mark.parametrize("custom_routing_function", [Llama4MoE.custom_routing_function]) +def test_custom( + m: int, + k: int, + top_k: int, + global_num_experts: int, + renormalize: bool, + enable_eplb: bool, + custom_routing_function: Callable, +): + if top_k > global_num_experts: + pytest.skip(f"top_k ({top_k}) > global_num_experts ({global_num_experts})") + + eplb_state = setup_eplb_state(enable_eplb, global_num_experts) + + router = create_fused_moe_router( + top_k=top_k, + global_num_experts=global_num_experts, + custom_routing_function=custom_routing_function, + renormalize=renormalize, + eplb_state=eplb_state, + ) + + hidden_states, router_logits = make_test_data(m, k, global_num_experts) + + # Get router output + topk_weights, topk_ids = router.select_experts(hidden_states, router_logits) + + # Compute baseline (Llama4 uses sigmoid) + baseline_weights, baseline_ids = baseline_custom_llama4(router_logits, top_k) + + # Compare results + assert_routing_results_close(topk_weights, topk_ids, baseline_weights, baseline_ids) + + +# TODO: is other test sufficient? +# # See tests/test_routing_simulatator.py +# @pytest.mark.parametrize("m,k", MK_S) +# @pytest.mark.parametrize("top_k", TOP_KS) +# @pytest.mark.parametrize("global_num_experts", NUM_EXPERTS) +# @pytest.mark.parametrize("renormalize", [False, True]) +# @pytest.mark.parametrize("enable_eplb", [False, True]) +# @pytest.mark.parameterize("strategy", ["uniform_random", "normal_routing"]) +# def test_simulated( +# m: int, +# k: int, +# top_k: int, +# global_num_experts: int, +# renormalize: bool, +# enable_eplb: bool, +# strategy: str, +# monkeypatch, +# ): +# eplb_state = setup_eplb_state(enable_eplb) + +# monkeypatch.setenv("VLLM_MOE_ROUTING_SIMULATION_STRATEGY", strategy) +# router = create_fused_moe_router( +# top_k=top_k, +# global_num_experts=global_num_experts, +# eplb_state=eplb_state, +# ) + +# hidden_states, router_logits = make_test_data(m, k, global_num_experts) +# topk_weights, topk_ids = router.select_experts(hidden_states, router_logits) + + +# --------------------------------------------------------------------------- +# Tests for eplb_map_to_physical_and_record +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("record_enabled", [True, False]) +@pytest.mark.parametrize( + "l2p_map, replica_count, num_physical, topk_ids, expected_out, expected_load", + [ + pytest.param( + # logical i → physical i + [[0], [1], [2], [3]], + [1, 1, 1, 1], + 4, + [[0, 1], [2, 3], [0, 2]], + [[0, 1], [2, 3], [0, 2]], + [2, 1, 2, 1], + id="identity", + ), + pytest.param( + # logical 0→3, 1→0, 2→1, 3→2 + [[3], [0], [1], [2]], + [1, 1, 1, 1], + 4, + [[0, 1], [2, 3], [0, 2]], + [[3, 0], [1, 2], [3, 1]], + [1, 2, 1, 2], + id="shuffled", + ), + pytest.param( + # logical 0→5, 1→2, 2→7, 3→0 in a larger physical space + [[5], [2], [7], [0]], + [1, 1, 1, 1], + 8, + [[0, 1], [2, 3]], + [[5, 2], [7, 0]], + [1, 0, 1, 0, 0, 1, 0, 1], + id="sparse", + ), + ], +) +def test_eplb_map_no_redundancy( + record_enabled, + l2p_map, + replica_count, + num_physical, + topk_ids, + expected_out, + expected_load, +): + l2p = torch.tensor(l2p_map, dtype=torch.int64, device="cuda") + rc = torch.tensor(replica_count, dtype=torch.int64, device="cuda") + load = torch.zeros(num_physical, dtype=torch.int32, device="cuda") + rec = torch.tensor(record_enabled, dtype=torch.bool, device="cuda") + ids = torch.tensor(topk_ids, dtype=torch.int32, device="cuda") + + out = eplb_map_to_physical_and_record( + topk_ids=ids, + expert_load_view=load, + logical_to_physical_map=l2p, + logical_replica_count=rc, + record_enabled=rec, + ) + + exp_out = torch.tensor(expected_out, dtype=out.dtype, device="cuda") + torch.testing.assert_close(out, exp_out) + + if record_enabled: + exp_load = torch.tensor(expected_load, dtype=torch.int32, device="cuda") + torch.testing.assert_close(load, exp_load) + else: + assert load.sum().item() == 0 + + +@pytest.mark.parametrize("top_k,R", [(2, 2), (4, 2), (8, 4), (8, 8)]) +def test_eplb_map_hot_expert_replica_balance(top_k, R): + """Hot logical expert with R replicas must be balanced across replicas + even when ``top_k`` is a multiple of ``R``. In that regime every top-k + offset for the hot expert lands on a multiple of ``top_k`` in the flat + ``topk_ids`` view, so per-replica assignment must not collapse onto a + single replica. + """ + num_tokens = 8192 + num_logical = 16 + num_physical = R + (num_logical - 1) + + l2p = torch.full((num_logical, R), -1, dtype=torch.int64, device="cuda") + l2p[0] = torch.arange(R, dtype=torch.int64, device="cuda") + for i in range(1, num_logical): + l2p[i, 0] = R + i - 1 + rc = torch.tensor([R] + [1] * (num_logical - 1), dtype=torch.int64, device="cuda") + + torch.manual_seed(0) + topk_ids = torch.randint( + 1, + num_logical, + (num_tokens, top_k), + dtype=torch.int32, + device="cuda", + ) + topk_ids[:, 0] = 0 + + load = torch.zeros(num_physical, dtype=torch.int32, device="cuda") + rec = torch.tensor(True, dtype=torch.bool, device="cuda") + + eplb_map_to_physical_and_record( + topk_ids=topk_ids, + expert_load_view=load, + logical_to_physical_map=l2p, + logical_replica_count=rc, + record_enabled=rec, + ) + + hot_load = load[:R].float() + max_mean = (hot_load.max() / hot_load.mean()).item() + assert max_mean < 1.15, ( + f"Hot expert replicas uneven: {hot_load.tolist()}, max/mean={max_mean:.3f}" + ) + + +@pytest.mark.parametrize("record_enabled", [True, False]) +@pytest.mark.parametrize( + "l2p_map, replica_count, num_physical, topk_ids, expected_out, expected_load", + [ + pytest.param( + # experts 0,1 have 2 replicas; 2,3 have 1 + [[0, 4], [1, 5], [2, -1], [3, -1]], + [2, 2, 1, 1], + 6, + [[0, 1], [2, 3], [0, 2]], + # replica = (token_idx * KNUTH) & 0xFFFFFFFF % R. + # token 0 hash=0x00000000: %2=0, %1=0. + # token 1 hash=0x9E3779B9: %2=1, %1=0. + # token 2 hash=0x3C6EF372: %2=0, %1=0. + [[0, 1], [2, 3], [0, 2]], + [2, 1, 2, 1, 0, 0], + id="partial", + ), + pytest.param( + # all 4 experts have 2 replicas + [[0, 4], [1, 5], [2, 6], [3, 7]], + [2, 2, 2, 2], + 8, + [[0, 1], [2, 3], [0, 2]], + # token 0 hash=0x00000000: %2=0. + # token 1 hash=0x9E3779B9: %2=1. + # token 2 hash=0x3C6EF372: %2=0. + [[0, 1], [6, 7], [0, 2]], + [2, 1, 1, 0, 0, 0, 1, 1], + id="full", + ), + pytest.param( + # expert 0: 4 replicas, experts 1,2: 2 replicas + [[0, 3, 5, 7], [1, 4, -1, -1], [2, 6, -1, -1]], + [4, 2, 2], + 8, + [[0, 1], [2, 0], [1, 2]], + # token 0 hash=0x00000000: %4=0, %2=0. + # token 1 hash=0x9E3779B9: %4=1, %2=1. + # token 2 hash=0x3C6EF372: %4=2, %2=0. + [[0, 1], [6, 3], [1, 2]], + [1, 2, 1, 1, 0, 0, 1, 0], + id="uneven", + ), + ], +) +def test_eplb_map_with_redundancy( + record_enabled, + l2p_map, + replica_count, + num_physical, + topk_ids, + expected_out, + expected_load, +): + l2p = torch.tensor(l2p_map, dtype=torch.int64, device="cuda") + rc = torch.tensor(replica_count, dtype=torch.int64, device="cuda") + load = torch.zeros(num_physical, dtype=torch.int32, device="cuda") + rec = torch.tensor(record_enabled, dtype=torch.bool, device="cuda") + ids = torch.tensor(topk_ids, dtype=torch.int32, device="cuda") + + out = eplb_map_to_physical_and_record( + topk_ids=ids, + expert_load_view=load, + logical_to_physical_map=l2p, + logical_replica_count=rc, + record_enabled=rec, + ) + + exp_out = torch.tensor(expected_out, dtype=out.dtype, device="cuda") + torch.testing.assert_close(out, exp_out) + + if record_enabled: + exp_load = torch.tensor(expected_load, dtype=torch.int32, device="cuda") + torch.testing.assert_close(load, exp_load) + else: + assert load.sum().item() == 0 + + +@pytest.mark.parametrize( + "l2p_map, replica_count, num_physical, topk_ids, " + "num_unpadded, expected_out, expected_load", + [ + pytest.param( + [[0], [1], [2], [3]], + [1, 1, 1, 1], + 4, + [[0, 1], [2, 3], [0, 2], [1, 3]], + 2, + [[0, 1], [2, 3], [0, 2], [1, 3]], + # only rows 0,1 counted: expert 0→1, 1→1, 2→1, 3→1 + [1, 1, 1, 1], + id="half_padded", + ), + pytest.param( + # record everything (None = no padding info) + [[0], [1], [2], [3]], + [1, 1, 1, 1], + 4, + [[0, 1], [2, 3], [0, 2], [1, 3]], + None, + [[0, 1], [2, 3], [0, 2], [1, 3]], + [2, 2, 2, 2], + id="no_padding_info", + ), + ], +) +def test_eplb_map_num_unpadded_tokens( + l2p_map, + replica_count, + num_physical, + topk_ids, + num_unpadded, + expected_out, + expected_load, +): + l2p = torch.tensor(l2p_map, dtype=torch.int64, device="cuda") + rc = torch.tensor(replica_count, dtype=torch.int64, device="cuda") + load = torch.zeros(num_physical, dtype=torch.int32, device="cuda") + rec = torch.tensor(True, dtype=torch.bool, device="cuda") + ids = torch.tensor(topk_ids, dtype=torch.int32, device="cuda") + num_unpadded_t = ( + torch.tensor(num_unpadded, dtype=torch.int32, device="cuda") + if num_unpadded is not None + else None + ) + + out = eplb_map_to_physical_and_record( + topk_ids=ids, + expert_load_view=load, + logical_to_physical_map=l2p, + logical_replica_count=rc, + record_enabled=rec, + num_unpadded_tokens=num_unpadded_t, + ) + + exp_out = torch.tensor(expected_out, dtype=out.dtype, device="cuda") + torch.testing.assert_close(out, exp_out) + + exp_load = torch.tensor(expected_load, dtype=torch.int32, device="cuda") + torch.testing.assert_close(load, exp_load) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_routing_simulator.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_routing_simulator.py new file mode 100644 index 0000000000000000000000000000000000000000..c119c86e21596fca2548700165652a7b424ed5e8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_routing_simulator.py @@ -0,0 +1,210 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Test script for the token-to-expert routing simulator. + +This script demonstrates how to use the routing simulator to test +different routing strategies and analyze their performance, including +integration tests with FusedMoE layer. +""" + +import tempfile + +import pytest +import torch + +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.distributed import ( + init_distributed_environment, + initialize_model_parallel, +) +from vllm.model_executor.layers.fused_moe.router.routing_simulator_router import ( + DistributionBasedRouting, + RoutingSimulator, +) + + +@pytest.fixture +def device(): + """Fixture to provide the appropriate device for testing.""" + return torch.device("cuda" if torch.cuda.is_available() else "cpu") + + +@pytest.mark.parametrize("num_tokens", [1, 16, 256]) +@pytest.mark.parametrize("hidden_size", [64, 1024]) +@pytest.mark.parametrize("num_experts", [16, 128]) +@pytest.mark.parametrize("top_k", [1, 4]) +def test_basic_functionality( + num_tokens: int, + hidden_size: int, + num_experts: int, + top_k: int, + device, +): + """Test basic functionality of the routing simulator.""" + # Test each routing strategy + strategies = RoutingSimulator.get_available_strategies() + + hidden_states = torch.randn(num_tokens, hidden_size, device=device) + router_logits = torch.randn(num_tokens, num_experts, device=device) + + for strategy in strategies: + # Simulate routing + topk_weights, topk_ids = RoutingSimulator.simulate_routing( + hidden_states=hidden_states, + router_logits=router_logits, + strategy_name=strategy, + top_k=top_k, + ) + + # Check output shapes + assert topk_weights.shape == ( + num_tokens, + top_k, + ), f"Wrong weights shape for {strategy}" + assert topk_ids.shape == ( + num_tokens, + top_k, + ), f"Wrong ids shape for {strategy}" + + # Check that expert IDs are valid + assert topk_ids.min() >= 0, f"Invalid expert ID (negative) for {strategy}" + assert topk_ids.max() < num_experts, ( + f"Invalid expert ID (too large) for {strategy}" + ) + + +def test_routing_strategy_integration(monkeypatch, device): + """Test that the routing strategy environment variable works with + FusedMoE.""" + pytest.importorskip("vllm.model_executor.layers.fused_moe.layer") + + import vllm.envs as envs + from vllm.model_executor.layers.fused_moe.layer import FusedMoE + + # Test parameters + num_tokens = 32 + hidden_size = 16 + num_experts = 4 + top_k = 2 + + # Create test data + hidden_states = torch.randn(num_tokens, hidden_size, device=device) + router_logits = torch.randn(num_tokens, num_experts, device=device) + + # Test different routing strategies + strategies = RoutingSimulator.get_available_strategies() + + vllm_config = VllmConfig() + with set_current_vllm_config(vllm_config): + temp_file = tempfile.mkstemp()[1] + init_distributed_environment( + world_size=1, + rank=0, + local_rank=0, + distributed_init_method=f"file://{temp_file}", + ) + initialize_model_parallel( + tensor_model_parallel_size=1, + pipeline_model_parallel_size=1, + ) + + for strategy in strategies: + fused_moe = FusedMoE( + num_experts=num_experts, + top_k=top_k, + hidden_size=hidden_size, + intermediate_size=0, + use_grouped_topk=False, + renormalize=True, + prefix=strategy, + ) + + # Set environment variable + env_name = "VLLM_MOE_ROUTING_SIMULATION_STRATEGY" + monkeypatch.setenv(env_name, strategy) + + # Temporarily override the envs lookup so the router factory + # reads the monkeypatched value instead of the module-load-time + # default. Use monkeypatch.setitem so the original lambda is + # restored automatically at teardown. + monkeypatch.setitem( + envs.environment_variables, + env_name, + lambda s=strategy: s, + ) + + # Test the select_experts method + topk_weights, topk_ids = fused_moe.router.select_experts( + hidden_states=hidden_states, + router_logits=router_logits, + ) + + # Verify output shapes + assert topk_weights.shape == (num_tokens, top_k), ( + f"Wrong weights shape for {strategy}" + ) + assert topk_ids.shape == (num_tokens, top_k), ( + f"Wrong ids shape for {strategy}" + ) + + # Verify expert IDs are valid + assert topk_ids.min() >= 0, f"Invalid expert ID (negative) for {strategy}" + assert topk_ids.max() < num_experts, ( + f"Invalid expert ID (too large) for {strategy}" + ) + + +def test_distribution_based_routing_with_custom_strategy(): + """Test registering and using DistributionBasedRouting with custom + parameters.""" + device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + + # Register custom distribution-based strategy + custom_strategy = DistributionBasedRouting(distribution="normal", mean=2.0, std=0.5) + RoutingSimulator.register_strategy("custom_normal", custom_strategy) + + # Test data + num_tokens = 60 + hidden_size = 48 + num_experts = 6 + top_k = 3 + + hidden_states = torch.randn(num_tokens, hidden_size, device=device) + router_logits = torch.randn(num_tokens, num_experts, device=device) + + # Use the custom strategy + topk_weights, topk_ids = RoutingSimulator.simulate_routing( + hidden_states=hidden_states, + router_logits=router_logits, + strategy_name="custom_normal", + top_k=top_k, + ) + + # Check output shapes + assert topk_weights.shape == (num_tokens, top_k) + assert topk_ids.shape == (num_tokens, top_k) + + # Check that expert IDs are valid + assert topk_ids.min() >= 0 + assert topk_ids.max() < num_experts + + +def test_instance_compatibility(): + """Test that static methods work correctly.""" + device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + + # Test static method directly + hidden_states = torch.randn(10, 8, device=device) + router_logits = torch.randn(10, 4, device=device) + + topk_weights, topk_ids = RoutingSimulator.simulate_routing( + hidden_states=hidden_states, + router_logits=router_logits, + strategy_name="uniform_random", + top_k=2, + ) + + assert topk_weights.shape == (10, 2) + assert topk_ids.shape == (10, 2) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_shared_fused_moe_routed_transform.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_shared_fused_moe_routed_transform.py new file mode 100644 index 0000000000000000000000000000000000000000..24763a05329b83330478c56ec815eeb0936b1765 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_shared_fused_moe_routed_transform.py @@ -0,0 +1,229 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Tests for FusedMoE with routed_input_transform. + +Verifies that applying routed_input_transform inside FusedMoE +produces the same results as applying the transform manually outside. +""" + +import pytest +import torch +import torch.nn as nn + +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.forward_context import set_forward_context +from vllm.model_executor.layers.fused_moe import FusedMoE +from vllm.platforms import current_platform +from vllm.utils.torch_utils import is_torch_equal_or_newer, set_random_seed + + +class SimpleLinear(nn.Module): + """A simple linear transform mimicking latent projection in latent MoE.""" + + def __init__(self, in_features: int, out_features: int, dtype: torch.dtype): + super().__init__() + self.weight = nn.Parameter( + torch.randn(out_features, in_features, device="cuda", dtype=dtype) / 10 + ) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return nn.functional.linear(x, self.weight) + + +class SimpleSharedExperts(nn.Module): + """A simple 2-layer MLP mimicking shared experts.""" + + def __init__(self, hidden_size: int, intermediate_size: int, dtype: torch.dtype): + super().__init__() + self.up = nn.Linear( + hidden_size, intermediate_size * 2, bias=False, device="cuda", dtype=dtype + ) + self.down = nn.Linear( + intermediate_size, hidden_size, bias=False, device="cuda", dtype=dtype + ) + with torch.no_grad(): + self.up.weight.div_(10) + self.down.weight.div_(10) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + gate_up = self.up(x) + gate, up = gate_up.chunk(2, dim=-1) + return self.down(nn.functional.silu(gate) * up) + + +def _assert_close( + actual: torch.Tensor, + expected: torch.Tensor, + atol: float, + rtol: float, + label: str, +) -> None: + """assert_close that prints diff diagnostics on both success and failure.""" + actual_nans = int(actual.isnan().sum().item()) + expected_nans = int(expected.isnan().sum().item()) + actual_zeros = int((actual == 0).sum().item()) + expected_zeros = int((expected == 0).sum().item()) + n_total = actual.numel() + + diff = (actual - expected).abs() + max_diff = diff.max().item() + mean_diff = diff.mean().item() + n_exceed = int((diff > atol).sum().item()) + pct_exceed = n_exceed / n_total * 100 + + print( + f"[{label}] " + f"shape={list(actual.shape)}, " + f"max_diff={max_diff:.6e}, " + f"mean_diff={mean_diff:.6e}, " + f"exceed_atol({atol})={n_exceed}/{n_total} ({pct_exceed:.2f}%), " + f"actual=[{actual.min().item():.4f}, {actual.max().item():.4f}], " + f"expected=[{expected.min().item():.4f}, {expected.max().item():.4f}], " + f"nan(actual/expected)={actual_nans}/{expected_nans}, " + f"zeros(actual/expected)={actual_zeros}/{expected_zeros}" + ) + + assert actual_nans == 0, ( + f"{label}: actual has {actual_nans}/{n_total} NaN values " + f"(expected has {expected_nans}). " + f"This indicates a kernel bug, not a precision issue." + ) + assert expected_nans == 0, ( + f"{label}: expected has {expected_nans}/{n_total} NaN values. " + f"This indicates a kernel bug, not a precision issue." + ) + + torch.testing.assert_close( + actual, + expected, + atol=atol, + rtol=rtol, + msg=( + f"{label}: max_diff={max_diff:.6e}, mean_diff={mean_diff:.6e}, " + f"exceed_atol({atol})={n_exceed}/{n_total} ({pct_exceed:.2f}%)" + ), + ) + + +@pytest.fixture(autouse=True) +def setup_cuda(): + if not torch.cuda.is_available(): + pytest.skip("CUDA not available") + torch.set_default_device("cuda") + + +@pytest.mark.parametrize("num_tokens", [1, 32]) +@pytest.mark.parametrize("hidden_size,latent_size", [(256, 128), (128, 64)]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize( + "use_rocm_aiter", [True, False] if current_platform.is_rocm() else [False] +) +@pytest.mark.skipif( + is_torch_equal_or_newer("2.10.0"), + reason="Test fails with PyTorch 2.10.0 see: https://github.com/vllm-project/vllm/issues/33995", +) +def test_routed_input_transform_inside_vs_outside( + num_tokens: int, + hidden_size: int, + latent_size: int, + dtype: torch.dtype, + use_rocm_aiter: bool, + dist_init, + workspace_init, + monkeypatch, +): + """Compare FusedMoE with transform inside vs manually applying outside. + Method A (inside): FusedMoE with routed_input_transform + Method B (outside): Manually transform, then FusedMoE without transform + """ + if current_platform.is_rocm(): + monkeypatch.setenv("VLLM_ROCM_USE_AITER", "1" if use_rocm_aiter else "0") + monkeypatch.setenv("VLLM_ROCM_USE_AITER_MOE", "1" if use_rocm_aiter else "0") + from vllm._aiter_ops import rocm_aiter_ops + + rocm_aiter_ops.refresh_env_variables() + + set_random_seed(42) + + num_experts = 8 + top_k = 2 + intermediate_size = hidden_size * 2 + + vllm_config = VllmConfig() + vllm_config.compilation_config.static_forward_context = dict() + + shared_experts = SimpleSharedExperts(hidden_size, intermediate_size, dtype) + routed_transform = SimpleLinear(hidden_size, latent_size, dtype) + + with set_current_vllm_config(vllm_config): + # Method A: FusedMoE WITH routed_input_transform + moe_with_transform = FusedMoE( + shared_experts=shared_experts, + routed_input_transform=routed_transform, + num_experts=num_experts, + top_k=top_k, + hidden_size=latent_size, + intermediate_size=intermediate_size, + renormalize=True, + params_dtype=dtype, + tp_size=1, + dp_size=1, + pcp_size=1, + prefix="moe_with_transform", + ) + + # Method B: FusedMoE WITHOUT routed_input_transform + # Note: shared_experts=None because when transform is done outside, + moe_without_transform = FusedMoE( + shared_experts=None, + routed_input_transform=None, + num_experts=num_experts, + top_k=top_k, + hidden_size=latent_size, + intermediate_size=intermediate_size, + renormalize=True, + params_dtype=dtype, + tp_size=1, + dp_size=1, + pcp_size=1, + prefix="moe_without_transform", + ) + + # Weights are created via torch.empty (uninitialized). + # Initialize with seeded random values for reproducibility. + with torch.no_grad(): + moe_with_transform.w13_weight.normal_() + moe_with_transform.w13_weight.div_(10) + moe_with_transform.w2_weight.normal_() + moe_with_transform.w2_weight.div_(10) + moe_without_transform.w13_weight.copy_(moe_with_transform.w13_weight) + moe_without_transform.w2_weight.copy_(moe_with_transform.w2_weight) + + moe_with_transform.quant_method.process_weights_after_loading( + moe_with_transform + ) + moe_without_transform.quant_method.process_weights_after_loading( + moe_without_transform + ) + + hidden_states = torch.randn(num_tokens, hidden_size, device="cuda", dtype=dtype) + router_logits = torch.randn(num_tokens, num_experts, device="cuda", dtype=dtype) + + with set_forward_context(None, vllm_config, num_tokens=num_tokens): + # Method A: combined output (shared + routed) + combined_A = moe_with_transform(hidden_states, router_logits) + + # Method B: manually transform, get routed output, add shared + transformed_hidden = routed_transform(hidden_states) + routed_out_B = moe_without_transform(transformed_hidden, router_logits) + shared_out_B = shared_experts(hidden_states) + combined_B = shared_out_B + routed_out_B + + torch.testing.assert_close( + combined_A, + combined_B, + atol=1e-3, + rtol=1e-3, + msg="Combined output should match: transform inside vs outside", + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_silu_mul_fp8_quant_deep_gemm.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_silu_mul_fp8_quant_deep_gemm.py new file mode 100644 index 0000000000000000000000000000000000000000..7528715fc119f7d668f75376dfede91349aa5f76 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_silu_mul_fp8_quant_deep_gemm.py @@ -0,0 +1,321 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import random + +import pytest +import torch + +from vllm.model_executor.layers.fused_moe.experts.batched_deep_gemm_moe import ( + persistent_masked_m_silu_mul_quant, +) +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + get_fp8_min_max, +) +from vllm.platforms import current_platform +from vllm.utils.deep_gemm import ( + DeepGemmQuantScaleFMT, + has_deep_gemm, + transform_sf_into_required_layout, +) +from vllm.utils.math_utils import cdiv, round_up +from vllm.utils.torch_utils import set_random_seed + +fp8_dtype = current_platform.fp8_dtype() + +CASES = [ + (1, 1, 128, fp8_dtype), + (1, 4, 128 * 1, fp8_dtype), + (2, 4, 128 * 2, fp8_dtype), + (1, 4, 128 * 3, fp8_dtype), + (8, 16, 128 * 4, fp8_dtype), + (8, 16, 128 * 5, fp8_dtype), + (8, 16, 128 * 6, fp8_dtype), + (8, 16, 128 * 7, fp8_dtype), + (8, 16, 128 * 8, fp8_dtype), + (8, 16, 128 * 9, fp8_dtype), + (8, 64, 7168, fp8_dtype), + (8, 128, 128 * 33, fp8_dtype), + (1, 4, 128 * 10, fp8_dtype), + (8, 128, 7168, fp8_dtype), + (8, 512, 7168, fp8_dtype), + (8, 1024, 7168, fp8_dtype), + (17, 31, 768, fp8_dtype), + (32, 64, 256, fp8_dtype), + (256, 8, 7168, fp8_dtype), + (256, 32, 7168, fp8_dtype), + (256, 64, 7168, fp8_dtype), + # Only add a few fnuz tests to help with long CI times. + (8, 512, 7168, torch.float8_e4m3fnuz), + (8, 1024, 7168, torch.float8_e4m3fnuz), +] + + +def as_uint8(x) -> torch.Tensor: + return ( + torch.empty(x.shape, dtype=x.dtype, device=x.device).copy_(x).view(torch.uint8) + ) + + +def silu(x: torch.Tensor) -> torch.Tensor: + x_f32 = x.to(torch.float32) + act_f32 = x_f32 / (1.0 + torch.exp(-x_f32)) + if current_platform.is_cuda(): + # C++ kernel returns bf16 + return act_f32.to(torch.bfloat16) + # Triton fallback stays in f32 + return act_f32 + + +def do_quant(x: torch.Tensor, group_size: int, ceil_ue8m0: bool): + fp8_min_val, fp8_max_val = get_fp8_min_max() + eps_bf16 = torch.tensor([1e-10], device=x.device, dtype=torch.bfloat16) + one_bf16 = torch.tensor([1.0], device=x.device, dtype=torch.bfloat16) + fp8_max_bf16 = torch.tensor([fp8_max_val], device=x.device, dtype=torch.bfloat16) + fp8_min_bf16 = torch.tensor([fp8_min_val], device=x.device, dtype=torch.bfloat16) + fp8_max_inv = one_bf16 / fp8_max_bf16 + assert fp8_max_inv.dtype == torch.bfloat16 + + assert x.size(-1) % group_size == 0 + num_groups = x.numel() // group_size + x_og_shape = x.shape + + if current_platform.is_cuda(): + # C++ kernel computes entirely in bf16 + x = x.to(torch.bfloat16) + x = x.view((-1, group_size)) + amax = x.abs().amax(dim=1).clamp(min=eps_bf16) + assert amax.dtype == torch.bfloat16 + s = amax * fp8_max_inv + + if ceil_ue8m0: + s = torch.exp2( + torch.ceil(torch.log2(s).to(torch.bfloat16)).to(torch.bfloat16) + ).to(torch.bfloat16) + + inv_s = one_bf16 / s + inv_s = inv_s.view((num_groups, 1)) + xq = torch.clamp( + x * inv_s, min=fp8_min_bf16.item(), max=fp8_max_bf16.item() + ).to(fp8_dtype) + else: + # Triton fallback computes in f32. Use multiply-by-reciprocal + # to match Triton's constexpr evaluation of 1.0/fp8_max. + fp8_min_f, fp8_max_f = get_fp8_min_max() + + x = x.to(torch.float32).view((-1, group_size)) + amax = x.abs().amax(dim=1).clamp(min=1e-10) + s = amax * (1.0 / fp8_max_f) + if ceil_ue8m0: + s = torch.exp2(torch.ceil(torch.log2(s))) + inv_s = (1.0 / s).view((num_groups, 1)) + xq = torch.clamp(x * inv_s, min=fp8_min_f, max=fp8_max_f).to(fp8_dtype) + + xq = xq.view(x_og_shape) + xs = s.view((-1, xq.size(-1) // group_size)) + return xq, xs + + +def silu_mul_quant( + gate: torch.Tensor, up: torch.Tensor, group_size: int, ceil_ue8m0: bool +) -> tuple[torch.Tensor, torch.Tensor]: + assert gate.size(-1) % group_size == 0 + assert up.size(-1) % group_size == 0 + + assert gate.dtype == torch.bfloat16 + assert up.dtype == torch.bfloat16 + + act = silu(gate) + + # act & mul + a_m = act * up + + q, s = do_quant(a_m, group_size, ceil_ue8m0) + return q, s + + +def pack_scales(x: torch.Tensor, tokens_per_expert: torch.Tensor) -> torch.Tensor: + """ + pack float32 scales into a int32 tensor + """ + assert x.dtype == torch.float32 + E, T, G = x.size() + + # Add i32_padding here so we can view it as a i32 tensor later on. + i32_padding = round_up(G, 4) - G + ref_s_i8 = torch.empty((E, T, G + i32_padding), dtype=torch.uint8, device="cuda") + for e in range(E): + nt = tokens_per_expert[e].item() + ref_s_i8[e, :nt, :G] = x[e, :nt].view(torch.int32) >> 23 + + ref_s_i32 = ref_s_i8.view(torch.int32) + + return ref_s_i32 + + +def ref_with_scale_fmt( + E: int, + T: int, + H: int, + group_size: int, + tokens_per_expert: torch.Tensor, + gate: torch.Tensor, + up: torch.Tensor, + scale_fmt: DeepGemmQuantScaleFMT, +) -> tuple[torch.Tensor, torch.Tensor]: + """ + The precision types of the operations triggered by this function + match closely with the kernel implementation so we compare more + accurately. + """ + scale_dtype = ( + torch.int32 if scale_fmt == DeepGemmQuantScaleFMT.UE8M0 else torch.float32 + ) + ceil_ue8m0 = scale_fmt in [ + DeepGemmQuantScaleFMT.UE8M0, + DeepGemmQuantScaleFMT.FLOAT32_CEIL_UE8M0, + ] + + ref_q = torch.empty((E, T, H), dtype=fp8_dtype, device="cuda") + ref_s_f32 = torch.empty( + (E, T, cdiv(H, group_size)), dtype=torch.float32, device="cuda" + ) + + for e in range(E): + nt = tokens_per_expert[e].item() + if nt == 0: + continue + ref_q[e, :nt], ref_s_f32[e, :nt] = silu_mul_quant( + gate[e, :nt], up[e, :nt], group_size, ceil_ue8m0=ceil_ue8m0 + ) + + if scale_dtype == torch.float32: + return ref_q, ref_s_f32 + + assert scale_dtype == torch.int32 + return ref_q, pack_scales(ref_s_f32, tokens_per_expert) + + +def token_random(E, T, H2, tokens_per_expert): + """ + Initialize each token in a random range so we test a range of + scale values. + """ + y = torch.empty((E, T, H2), dtype=torch.bfloat16, device="cuda") + for e in range(E): + for t in range(tokens_per_expert[e].item()): + exp = random.choice(range(1, 20)) + y[e, t].uniform_(-(2**exp), 2**exp) + return y + + +@pytest.mark.parametrize("E,T,H,fp8_type", CASES) +@torch.inference_mode() +def test_silu_mul_fp8_quant_deep_gemm(E: int, T: int, H: int, fp8_type: torch.dtype): + group_size = 128 + set_random_seed(42) + + tokens_per_expert = torch.randint( + low=0, + high=T, + size=(E,), + dtype=torch.int32, + device="cuda", + ) + + # Input tensor of shape (E, T, 2*H) + y = token_random(E, T, 2 * H, tokens_per_expert) + + gate = y[..., :H].to(torch.bfloat16) + up = y[..., H:].to(torch.bfloat16) + + scale_fmts = [ + DeepGemmQuantScaleFMT.FLOAT32, + DeepGemmQuantScaleFMT.FLOAT32_CEIL_UE8M0, + ] + # UE8M0 (int32 packed) scales require the C++ kernel which is + # not available on ROCm (#ifndef USE_ROCM). + # https://github.com/ROCm/aiter/issues/2420 + if current_platform.is_cuda(): + scale_fmts.append(DeepGemmQuantScaleFMT.UE8M0) + + # Run the SiLU V2 kernel + for scale_fmt in scale_fmts: + y_q, y_s = persistent_masked_m_silu_mul_quant( + y, + tokens_per_expert, + group_size=group_size, + quant_scale_fmt=scale_fmt, + ) + + ref_y_q, ref_y_s = ref_with_scale_fmt( + E, T, H, group_size, tokens_per_expert, gate, up, scale_fmt=scale_fmt + ) + + # deepgemm scales transform + dg_scales = None + if ( + has_deep_gemm() + and current_platform.has_device_capability(100) + and scale_fmt == DeepGemmQuantScaleFMT.UE8M0 + ): + _q, _s = ref_with_scale_fmt( + E, + T, + H, + group_size, + tokens_per_expert, + gate, + up, + scale_fmt=DeepGemmQuantScaleFMT.FLOAT32_CEIL_UE8M0, + ) + dg_scales = transform_sf_into_required_layout( + sf=_s, + mn=_q.size(1), + k=_q.size(2), + recipe=(1, 128, 128), + num_groups=_q.size(0), + is_sfa=True, + ) + + expected_scale_dtype = ( + torch.int32 if scale_fmt == DeepGemmQuantScaleFMT.UE8M0 else torch.float32 + ) + assert y_s.dtype == expected_scale_dtype + assert ref_y_s.dtype == expected_scale_dtype + + for e in range(E): + nt = tokens_per_expert[e].item() + + if current_platform.is_rocm(): + # On ROCm the Triton fallback kernel uses f32 math + # intrinsics (tl.exp) that may differ from PyTorch's + # torch.exp by 1 ULP. At FP8 quantization + # boundaries this can flip one representable value. + # Allow 1 FP8 quantum of tolerance. + torch.testing.assert_close( + y_q[e, :nt].to(torch.float32), + ref_y_q[e, :nt].to(torch.float32), + atol=32.0, + rtol=0.2, + ) + else: + torch.testing.assert_close( + y_q[e, :nt].to(torch.float32), + ref_y_q[e, :nt].to(torch.float32), + ) + + if scale_fmt == DeepGemmQuantScaleFMT.UE8M0: + G = H // group_size + y_s_sliced = as_uint8(y_s[e]) + ref_s_sliced = as_uint8(ref_y_s[e]) + torch.testing.assert_close(y_s_sliced[:nt, :G], ref_s_sliced[:nt, :G]) + if dg_scales is not None: + dg_sliced = as_uint8(dg_scales[e]) + torch.testing.assert_close(y_s_sliced[:nt, :G], dg_sliced[:nt, :G]) + else: + torch.testing.assert_close( + y_s[e, :nt], + ref_y_s[e, :nt], + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_silu_mul_per_token_group_quant_fp8_colmajor.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_silu_mul_per_token_group_quant_fp8_colmajor.py new file mode 100644 index 0000000000000000000000000000000000000000..583e1e38dc6ee232f0ba03628b040ede7ebf840a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_silu_mul_per_token_group_quant_fp8_colmajor.py @@ -0,0 +1,138 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm.model_executor.layers.quantization.utils.fp8_utils import ( + _per_token_group_quant_fp8_colmajor, + silu_mul_per_token_group_quant_fp8_colmajor, +) +from vllm.platforms import current_platform +from vllm.triton_utils import triton +from vllm.utils.deep_gemm import is_deep_gemm_e8m0_used +from vllm.utils.torch_utils import set_random_seed + +FLOAT8_DTYPE = torch.float8_e4m3fn +GROUP_SIZE = 128 + + +def reference_quant(x: torch.Tensor, use_ue8m0: bool): + """ + Reference triton quant kernel from, + vllm.model_executor.layers.quantization.utils.fp8_utils + """ + + x_q = torch.empty_like(x, device=x.device, dtype=FLOAT8_DTYPE) + + # Allocate the scale tensor in column-major format. + shape = (x.shape[-1] // GROUP_SIZE,) + x.shape[:-1] + x_s = torch.empty(shape, device=x.device, dtype=torch.float32).permute(-1, -2) + + M = x.numel() // GROUP_SIZE + N = GROUP_SIZE + BLOCK = triton.next_power_of_2(N) + # heuristics for number of warps + num_warps = min(max(BLOCK // 256, 1), 8) + num_stages = 1 + + finfo = torch.finfo(FLOAT8_DTYPE) + fp8_min = finfo.min + fp8_max = finfo.max + + _per_token_group_quant_fp8_colmajor[(M,)]( + x, + x_q, + x_s, + GROUP_SIZE, + x.shape[1], + x.stride(0), + x_s.stride(1), + eps=1e-10, + fp8_min=fp8_min, + fp8_max=fp8_max, + use_ue8m0=use_ue8m0, + BLOCK=BLOCK, + num_warps=num_warps, + num_stages=num_stages, + ) + return x_q, x_s + + +def reference(x: torch.Tensor, use_ue8m0: bool) -> tuple[torch.Tensor, torch.Tensor]: + T, N = x.size() + ref_act_out = torch.empty((T, N // 2), dtype=torch.bfloat16, device="cuda") + torch.ops._C.silu_and_mul(ref_act_out, x) + return reference_quant(ref_act_out, use_ue8m0) + + +def reference_with_clamp( + x: torch.Tensor, use_ue8m0: bool, clamp_limit: float +) -> tuple[torch.Tensor, torch.Tensor]: + """Pre-clamp inputs (gate from above, up symmetric) at the input dtype to + match the C++ compute() template, then run the standard silu_and_mul + + quant reference.""" + N_2 = x.size(1) // 2 + dtype = x.dtype + gate = x[..., :N_2].to(torch.float32).clamp(max=clamp_limit).to(dtype) + up = ( + x[..., N_2:] + .to(torch.float32) + .clamp(min=-clamp_limit, max=clamp_limit) + .to(dtype) + ) + return reference(torch.cat([gate, up], dim=-1), use_ue8m0) + + +@pytest.mark.parametrize("T", [128, 256, 512]) +@pytest.mark.parametrize("N", [128 * 2, 256 * 2, 768 * 2, 2048 * 2, 7168 * 2]) +@pytest.mark.skipif( + current_platform.is_rocm(), + reason="ROCm does not support DeepGemm.", +) +def test_silu_mul_fp8_quant_deep_gemm(T: int, N: int): + set_random_seed(42) + + input = torch.rand((T, N), dtype=torch.bfloat16, device="cuda") + + use_ue8m0 = is_deep_gemm_e8m0_used() + + # Test + output, output_scales = silu_mul_per_token_group_quant_fp8_colmajor( + input, use_ue8m0=use_ue8m0 + ) + + # Reference + ref_output, ref_output_scales = reference(input, use_ue8m0) + + torch.testing.assert_close(output.to(torch.float32), ref_output.to(torch.float32)) + torch.testing.assert_close(output_scales, ref_output_scales) + + +@pytest.mark.parametrize("T", [128, 256, 512]) +@pytest.mark.parametrize("N", [128 * 2, 256 * 2, 768 * 2, 2048 * 2, 7168 * 2]) +@pytest.mark.parametrize("clamp_limit", [7.0, 10.0]) +@pytest.mark.skipif( + current_platform.is_rocm(), + reason="ROCm does not support DeepGemm.", +) +def test_silu_mul_fp8_quant_deep_gemm_clamp(T: int, N: int, clamp_limit: float): + set_random_seed(42) + + # Use a wide distribution so values routinely exceed both clamp limits and + # the clamp branch is actually exercised (uniform [0, 1) inputs would never + # trigger it). + input = torch.randn((T, N), dtype=torch.bfloat16, device="cuda") * 8.0 + + use_ue8m0 = is_deep_gemm_e8m0_used() + + # Test + output, output_scales = silu_mul_per_token_group_quant_fp8_colmajor( + input, use_ue8m0=use_ue8m0, clamp_limit=clamp_limit + ) + + # Reference + ref_output, ref_output_scales = reference_with_clamp(input, use_ue8m0, clamp_limit) + + torch.testing.assert_close(output.to(torch.float32), ref_output.to(torch.float32)) + torch.testing.assert_close(output_scales, ref_output_scales) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_topk_softplus_sqrt.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_topk_softplus_sqrt.py new file mode 100644 index 0000000000000000000000000000000000000000..8b5f48b3cc13668ee6fa8659397801ac30e9c2ae --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_topk_softplus_sqrt.py @@ -0,0 +1,188 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import pytest +import torch +import torch.nn.functional as F + +from vllm.model_executor.layers.fused_moe.config import ( + RoutingMethodType, + get_routing_method_type, +) +from vllm.model_executor.layers.fused_moe.router.fused_topk_bias_router import ( + fused_topk_bias, +) +from vllm.platforms import current_platform + + +def _torch_topk_softplus_sqrt( + gating_output: torch.Tensor, + topk: int, + renormalize: bool, + routed_scaling_factor: float, + e_score_correction_bias: torch.Tensor | None = None, + input_ids: torch.Tensor | None = None, + hash_indices_table: torch.Tensor | None = None, +): + scores = F.softplus(gating_output.float()).sqrt() + original_scores = scores + if e_score_correction_bias is not None: + scores_for_choice = scores + e_score_correction_bias.unsqueeze(0) + else: + scores_for_choice = scores + + if hash_indices_table is not None: + assert input_ids is not None + topk_ids = hash_indices_table[input_ids.long()] + else: + topk_ids = torch.topk(scores_for_choice, k=topk, dim=-1, sorted=True)[1] + + topk_weights = original_scores.gather(1, topk_ids.long()) + if renormalize: + topk_weights = topk_weights / topk_weights.sum(dim=-1, keepdim=True) + if routed_scaling_factor != 1.0: + topk_weights = topk_weights * routed_scaling_factor + return topk_weights.to(torch.float32), topk_ids.to(torch.int32) + + +def test_sqrtsoftplus_bias_uses_deepseek_v4_routing_method(): + assert ( + get_routing_method_type( + scoring_func="sqrtsoftplus", + top_k=8, + renormalize=True, + num_expert_group=None, + has_e_score_bias=True, + ) + == RoutingMethodType.DeepseekV4 + ) + assert ( + get_routing_method_type( + scoring_func="sqrtsoftplus", + top_k=8, + renormalize=False, + num_expert_group=None, + has_e_score_bias=True, + ) + == RoutingMethodType.Unspecified + ) + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="This test is skipped on non-CUDA platform.", +) +@pytest.mark.parametrize("num_tokens", [1, 33, 128]) +@pytest.mark.parametrize("hidden_size", [1024, 2048]) +@pytest.mark.parametrize("num_experts", [128, 256, 384, 512]) +@pytest.mark.parametrize("topk", [6, 8, 16]) +@pytest.mark.parametrize("renormalize", [True, False]) +@pytest.mark.parametrize("routed_scaling_factor", [1.0, 1.5]) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.half, torch.float32]) +def test_fused_topk_softplus_sqrt( + num_tokens: int, + hidden_size: int, + num_experts: int, + topk: int, + renormalize: bool, + routed_scaling_factor: float, + dtype: torch.dtype, +): + torch.manual_seed(0) + hidden_states = torch.randn((num_tokens, hidden_size), dtype=dtype, device="cuda") + gating_output = torch.randn((num_tokens, num_experts), dtype=dtype, device="cuda") + e_score_correction_bias = torch.randn( + (num_experts,), dtype=torch.float32, device="cuda" + ) + + topk_weights_ref, topk_ids_ref = _torch_topk_softplus_sqrt( + gating_output=gating_output, + topk=topk, + renormalize=renormalize, + routed_scaling_factor=routed_scaling_factor, + e_score_correction_bias=e_score_correction_bias, + ) + + topk_weights, topk_ids = fused_topk_bias( + hidden_states=hidden_states, + gating_output=gating_output, + scoring_func="sqrtsoftplus", + e_score_correction_bias=e_score_correction_bias, + topk=topk, + renormalize=renormalize, + routed_scaling_factor=routed_scaling_factor, + ) + + # Different kernels may return the topk experts in different orders when + # scores tie; sort by expert id before comparing. + sorted_ref_ids, idx_ref = topk_ids_ref.sort(dim=-1) + sorted_ids, idx_ops = topk_ids.sort(dim=-1) + torch.testing.assert_close(sorted_ref_ids, sorted_ids, atol=0, rtol=0) + + sorted_w_ref = topk_weights_ref.gather(1, idx_ref) + sorted_w = topk_weights.gather(1, idx_ops) + torch.testing.assert_close(sorted_w_ref, sorted_w, atol=2e-2, rtol=1e-2) + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="This test is skipped on non-CUDA platform.", +) +@pytest.mark.parametrize("num_tokens", [1, 33, 128]) +@pytest.mark.parametrize("hidden_size", [1024, 2048]) +@pytest.mark.parametrize("num_experts", [256, 384, 512]) +@pytest.mark.parametrize("topk", [6, 8, 16]) +@pytest.mark.parametrize("renormalize", [True, False]) +@pytest.mark.parametrize("routed_scaling_factor", [1.0, 2.5]) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.half, torch.float32]) +def test_fused_topk_softplus_sqrt_hash( + num_tokens: int, + hidden_size: int, + num_experts: int, + topk: int, + renormalize: bool, + routed_scaling_factor: float, + dtype: torch.dtype, +): + torch.manual_seed(0) + vocab_size = 1024 + hidden_states = torch.randn((num_tokens, hidden_size), dtype=dtype, device="cuda") + gating_output = torch.randn((num_tokens, num_experts), dtype=dtype, device="cuda") + # Per-token fixed expert selection: for each vocab id pick `topk` distinct + # experts. + hash_indices_table = torch.stack( + [torch.randperm(num_experts)[:topk] for _ in range(vocab_size)] + ).to(device="cuda", dtype=torch.int32) + input_ids = torch.randint( + 0, vocab_size, (num_tokens,), dtype=torch.int32, device="cuda" + ) + + topk_weights_ref, topk_ids_ref = _torch_topk_softplus_sqrt( + gating_output=gating_output, + topk=topk, + renormalize=renormalize, + routed_scaling_factor=routed_scaling_factor, + input_ids=input_ids, + hash_indices_table=hash_indices_table, + ) + + topk_weights, topk_ids = fused_topk_bias( + hidden_states=hidden_states, + gating_output=gating_output, + scoring_func="sqrtsoftplus", + e_score_correction_bias=None, + topk=topk, + renormalize=renormalize, + input_tokens=input_ids, + hash_indices_table=hash_indices_table, + routed_scaling_factor=routed_scaling_factor, + ) + + sorted_ref_ids, idx_ref = topk_ids_ref.sort(dim=-1) + sorted_ids, idx_ops = topk_ids.sort(dim=-1) + torch.testing.assert_close(sorted_ref_ids, sorted_ids, atol=0, rtol=0) + + sorted_w_ref = topk_weights_ref.gather(1, idx_ref) + sorted_w = topk_weights.gather(1, idx_ops) + torch.testing.assert_close(sorted_w_ref, sorted_w, atol=2e-2, rtol=1e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_triton_moe_no_act_mul.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_triton_moe_no_act_mul.py new file mode 100644 index 0000000000000000000000000000000000000000..9e55ee18fc46a30a6824baae2e66dfb47fad2aca --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_triton_moe_no_act_mul.py @@ -0,0 +1,211 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for MoE with non-gated activations (*_no_mul). + +These tests verify that MoE layers work correctly with activations like +silu_no_mul, gelu_no_mul, relu2_no_mul where the activation output dimension +equals N (not N // 2 like gated activations). +""" + +import pytest +import torch + +from tests.kernels.moe.utils import make_dummy_moe_config +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.config import ( + FUSED_MOE_UNQUANTIZED_CONFIG, +) +from vllm.model_executor.layers.fused_moe.experts.triton_moe import TritonExperts +from vllm.platforms import current_platform + +# Test parameters +M_SIZES = [1, 16, 64] +N_SIZES = [128, 256] +K_SIZES = [64, 128] +TOPK_VALUES = [1, 2] +NUM_EXPERTS = 8 +NO_MUL_ACTIVATIONS = [ + MoEActivation.SILU_NO_MUL, + MoEActivation.GELU_NO_MUL, + MoEActivation.RELU2_NO_MUL, +] + + +def make_test_tensors( + m: int, + n: int, + k: int, + num_experts: int, + topk: int, + dtype: torch.dtype = torch.bfloat16, + device: str = "cuda", +): + """Create test tensors for MoE with non-gated activation. + + For non-gated activations (*_no_mul): + - w1: (E, N, K) - projects from K to N + - w2: (E, K, N) - projects from N back to K (note: N, not N//2) + """ + hidden_states = torch.randn(m, k, dtype=dtype, device=device) + + # For non-gated: w1 projects K -> N, w2 projects N -> K + w1 = torch.randn(num_experts, n, k, dtype=dtype, device=device) * 0.1 + w2 = torch.randn(num_experts, k, n, dtype=dtype, device=device) * 0.1 + + topk_weights = torch.ones(m, topk, dtype=torch.float32, device=device) / topk + topk_ids = torch.randint(0, num_experts, (m, topk), device=device) + + return hidden_states, w1, w2, topk_weights, topk_ids + + +@pytest.mark.skipif( + not current_platform.has_device_capability(80), + reason="Requires compute capability >= 8.0", +) +@pytest.mark.parametrize("m", M_SIZES) +@pytest.mark.parametrize("n", N_SIZES) +@pytest.mark.parametrize("k", K_SIZES) +@pytest.mark.parametrize("topk", TOPK_VALUES) +@pytest.mark.parametrize("activation", NO_MUL_ACTIVATIONS) +@torch.inference_mode() +def test_triton_experts_no_mul_activation( + m: int, + n: int, + k: int, + topk: int, + activation: MoEActivation, +): + hidden_states, w1, w2, topk_weights, topk_ids = make_test_tensors( + m, n, k, NUM_EXPERTS, topk + ) + + experts = TritonExperts( + moe_config=make_dummy_moe_config(), + quant_config=FUSED_MOE_UNQUANTIZED_CONFIG, + ) + + ws1_shape, ws2_shape, out_shape = experts.workspace_shapes( + M=m, + N=n, + K=k, + topk=topk, + global_num_experts=NUM_EXPERTS, + local_num_experts=NUM_EXPERTS, + expert_tokens_meta=None, + activation=activation, + ) + + # Verify workspace shapes are correct for no_mul activation + # workspace1 should handle activation_out_dim = N (not N//2) + assert ws1_shape == (m, topk, max(n, k)), ( + f"workspace1 shape mismatch: expected {(m, topk, max(n, k))}, got {ws1_shape}" + ) + # workspace2 should handle max(N, K) for intermediate_cache1/cache3 + assert ws2_shape == (m, topk, max(n, k)), ( + f"workspace2 shape mismatch: expected {(m, topk, max(n, k))}, got {ws2_shape}" + ) + assert out_shape == (m, k), ( + f"output shape mismatch: expected {(m, k)}, got {out_shape}" + ) + + workspace1 = torch.empty( + ws1_shape[0] * ws1_shape[1] * ws1_shape[2], + dtype=hidden_states.dtype, + device=hidden_states.device, + ) + workspace2 = torch.empty( + ws2_shape[0] * ws2_shape[1] * ws2_shape[2], + dtype=hidden_states.dtype, + device=hidden_states.device, + ) + output = torch.zeros(m, k, dtype=hidden_states.dtype, device=hidden_states.device) + + experts.apply( + output=output, + hidden_states=hidden_states, + w1=w1, + w2=w2, + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=activation, + global_num_experts=NUM_EXPERTS, + expert_map=None, + a1q_scale=None, + a2_scale=None, + workspace13=workspace1, + workspace2=workspace2, + expert_tokens_meta=None, + apply_router_weight_on_input=False, + ) + + assert output.shape == (m, k), f"Expected shape {(m, k)}, got {output.shape}" + assert not torch.isnan(output).any(), "Output contains NaN" + assert not torch.isinf(output).any(), "Output contains Inf" + assert output.abs().sum() > 0, "Output is all zeros" + + +@pytest.mark.skipif( + not current_platform.has_device_capability(80), + reason="Requires compute capability >= 8.0", +) +@torch.inference_mode() +def test_workspace_shapes_no_mul_vs_gated(): + """Test that workspace shapes differ correctly between gated and non-gated.""" + from vllm.model_executor.layers.fused_moe.experts.triton_moe import TritonExperts + + M, N, K, topk = 64, 256, 128, 2 + + experts = TritonExperts( + moe_config=make_dummy_moe_config(), + quant_config=FUSED_MOE_UNQUANTIZED_CONFIG, + ) + + ws1_no_mul, _, out_no_mul = experts.workspace_shapes( + M, N, K, topk, 8, 8, None, MoEActivation.SILU_NO_MUL + ) + + ws1_gated, _, out_gated = experts.workspace_shapes( + M, N, K, topk, 8, 8, None, MoEActivation.SILU + ) + + # For no_mul: activation_out_dim = N + # For gated: activation_out_dim = N // 2 + # workspace1 should use max(activation_out_dim, K) + activation_out_dim_no_mul = N + activation_out_dim_gated = N // 2 + + assert ws1_no_mul[2] == max(activation_out_dim_no_mul, K), ( + f"no_mul workspace1 last dim should be max({activation_out_dim_no_mul}, {K})" + ) + assert ws1_gated[2] == max(activation_out_dim_gated, K), ( + f"gated workspace1 last dim should be max({activation_out_dim_gated}, {K})" + ) + + # Output shapes should be the same + assert out_no_mul == out_gated == (M, K) + + +@pytest.mark.skipif( + not current_platform.has_device_capability(80), + reason="Requires compute capability >= 8.0", +) +@torch.inference_mode() +def test_adjust_n_for_activation(): + """Test the adjust_N_for_activation method.""" + from vllm.model_executor.layers.fused_moe.experts.triton_moe import TritonExperts + + experts = TritonExperts( + moe_config=make_dummy_moe_config(), + quant_config=FUSED_MOE_UNQUANTIZED_CONFIG, + ) + + N = 256 + + # Gated activations should return N // 2 + assert experts.adjust_N_for_activation(N, MoEActivation.SILU) == N // 2 + assert experts.adjust_N_for_activation(N, MoEActivation.GELU) == N // 2 + + # Non-gated activations should return N + assert experts.adjust_N_for_activation(N, MoEActivation.SILU_NO_MUL) == N + assert experts.adjust_N_for_activation(N, MoEActivation.GELU_NO_MUL) == N + assert experts.adjust_N_for_activation(N, MoEActivation.RELU2_NO_MUL) == N diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_triton_moe_ptpc_fp8.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_triton_moe_ptpc_fp8.py new file mode 100644 index 0000000000000000000000000000000000000000..8a88c535c48a3c065bce01d204eb3e1b1efab6bb --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_triton_moe_ptpc_fp8.py @@ -0,0 +1,170 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +# Adapted from https://github.com/sgl-project/sglang/blob/main/test/srt/test_triton_moe_channel_fp8_kernel.py +import itertools + +import pytest +import torch + +from tests.kernels.moe.utils import fused_moe +from vllm import _custom_ops as ops +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.model_executor.layers.activation import SiluAndMul +from vllm.model_executor.layers.fused_moe.config import fp8_w8a8_moe_quant_config +from vllm.platforms import current_platform + +if current_platform.get_device_capability() < (9, 0): + pytest.skip("FP8 Triton requires CUDA 9.0 or higher", allow_module_level=True) + +vllm_config = VllmConfig() + +if current_platform.is_fp8_fnuz(): + pytest.skip( + "Tests in this file require float8_e4m3fn and platform does not support", + allow_module_level=True, + ) + + +def native_w8a8_per_token_matmul(A, B, As, Bs, output_dtype=torch.float16): + """Matrix multiplication function that supports per-token input + quantization and per-column weight quantization""" + A = A.to(torch.float32) + B = B.to(torch.float32) + + assert A.shape[-1] == B.shape[-1], "Dimension mismatch" + assert B.ndim == 2 and B.is_contiguous(), "B must be a 2D contiguous tensor" + + # Reshape input + M = A.numel() // A.shape[-1] + B = B.t() # Transpose weight matrix + N, K = B.shape + origin_C_shape = A.shape[:-1] + (K,) + A = A.reshape(M, N) + + # As is per-token [M, 1], Bs is per-column [1, K] + C = torch.matmul(A, B) # [M, K] + C = As * C * Bs.view(1, -1) # Broadcast per-column scale + + return C.reshape(origin_C_shape).to(output_dtype) + + +def fp8_mask(a, mask): + dtype = a.dtype + return a.view(torch.int8)[mask].view(dtype) + + +def torch_w8a8_per_column_moe(a, w1, w2, w1_s, w2_s, score, topk): + """This function performs fused moe with per-column int8 + quantization using native torch.""" + + B, D = a.shape + # Perform per-token quantization + a_q, a_s = ops.scaled_fp8_quant(a, use_per_token_if_dynamic=True) + # Repeat tokens to match topk + a_q = a_q.view(B, -1, D).repeat(1, topk, 1).reshape(-1, D) + # Also repeat the scale + a_s = a_s.view(B, -1, 1).repeat(1, topk, 1).reshape(-1, 1) # [B*topk, 1] + + out = torch.zeros(B * topk, w2.shape[1], dtype=a.dtype, device=a.device) + + # Calculate routing + score = torch.softmax(score, dim=-1, dtype=torch.float32) + topk_weight, topk_ids = torch.topk(score, topk) + topk_weight = topk_weight.view(-1) + topk_ids = topk_ids.view(-1) + # Process each expert + for i in range(w1.shape[0]): + mask = topk_ids == i + if mask.sum(): + # First MLP layer: note that a_s is now per-token + inter_out = native_w8a8_per_token_matmul( + fp8_mask(a_q, mask), + w1[i], + fp8_mask(a_s, mask), + w1_s[i], + output_dtype=a.dtype, + ) + # Activation function + act_out = SiluAndMul().forward_native(inter_out) + # Quantize activation output with per-token + act_out_q, act_out_s = ops.scaled_fp8_quant( + act_out, use_per_token_if_dynamic=True + ) + + # Second MLP layer + out[mask] = native_w8a8_per_token_matmul( + act_out_q, w2[i], act_out_s, w2_s[i], output_dtype=a.dtype + ) + # Apply routing weights and sum + return ( + out.view(B, -1, w2.shape[1]) * topk_weight.view(B, -1, 1).to(out.dtype) + ).sum(dim=1) + + +@pytest.fixture(autouse=True, scope="module") +def setup_cuda(): + """Sets the default CUDA device for all tests in this module.""" + torch.set_default_device("cuda") + + +DTYPES = [torch.half, torch.bfloat16] +M = [1, 33] +N = [128, 1024] +K = [256, 4096] +E = [8] +TOP_KS = [2, 6] +SEEDS = [0] + + +@pytest.mark.parametrize( + "M, N, K, E, topk, dtype, seed", + itertools.product(M, N, K, E, TOP_KS, DTYPES, SEEDS), +) +@torch.inference_mode() +def test_w8a8_fp8_fused_moe(M, N, K, E, topk, dtype, seed): + torch.manual_seed(seed) + # Initialize int8 quantization parameters + factor_for_scale = 1e-2 + finfo = torch.finfo(torch.float8_e4m3fn) + fp8_max = finfo.max + fp8_min = finfo.min + + # Input tensor + # M * K + a = torch.randn((M, K), dtype=dtype) / 10 + + # Generate int8 weights + w1_fp32 = (torch.rand((E, 2 * N, K), dtype=torch.float32) - 0.5) * 2 + w1 = (w1_fp32 * fp8_max).clamp(min=fp8_min, max=fp8_max).to(torch.float8_e4m3fn) + + w2_fp32 = (torch.rand((E, K, N), dtype=torch.float32) - 0.5) * 2 + w2 = (w2_fp32 * fp8_max).clamp(min=fp8_min, max=fp8_max).to(torch.float8_e4m3fn) + + # Generate scale for each column (per-column quantization) + w1_s = torch.rand(E, 2 * N, device=w1_fp32.device) * factor_for_scale + w2_s = torch.rand(E, K, device=w2_fp32.device) * factor_for_scale + score = torch.randn((M, E), dtype=dtype) + + with set_current_vllm_config(vllm_config): + ref_out = torch_w8a8_per_column_moe(a, w1, w2, w1_s, w2_s, score, topk) + out = fused_moe( + a, + w1, + w2, + score, + topk, + renormalize=False, + quant_config=fp8_w8a8_moe_quant_config( + per_act_token_quant=True, + w1_scale=w1_s, + w2_scale=w2_s, + block_shape=None, # Not using block quantization + ), + ) + + # Check results + rel_diff = torch.mean( + torch.abs(out.to(torch.float32) - ref_out.to(torch.float32)) + ) / torch.mean(torch.abs(ref_out.to(torch.float32))) + assert rel_diff < 0.05 diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_trtllm_nvfp4_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_trtllm_nvfp4_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..b9a590b0ec5c07cfeae0ce0bb189e54498878a27 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_trtllm_nvfp4_moe.py @@ -0,0 +1,274 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Tests for the FlashInfer TRTLLM NvFP4 MoE backend +(`TrtLlmNvFp4ExpertsModular`). + +Covers the activations the wrapper claims to support — SiLU, RELU^2 (non-gated), +and GELU — including a Gemma4-shaped case (128 experts, top-k 8, +intermediate_size 704) that exercises the non-256-aligned padding path. +""" + +import pytest +import torch + +import vllm.model_executor.layers.fused_moe.modular_kernel as mk +from tests.kernels.moe.utils import make_test_quant_config +from tests.kernels.quantization.nvfp4_utils import ( + FLOAT4_E2M1_MAX, + FLOAT8_E4M3_MAX, + dequantize_nvfp4_to_dtype, +) +from tests.kernels.utils import torch_moe +from vllm import _custom_ops as ops +from vllm.config import ParallelConfig, VllmConfig, set_current_vllm_config +from vllm.model_executor.custom_op import CustomOp, op_registry +from vllm.model_executor.layers.activation import SiluAndMulWithClamp +from vllm.model_executor.layers.fused_moe import fused_topk +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.all2all_utils import ( + maybe_make_prepare_finalize, +) +from vllm.model_executor.layers.fused_moe.config import ( + FusedMoEConfig, + FusedMoEParallelConfig, + RoutingMethodType, +) +from vllm.model_executor.layers.fused_moe.experts.trtllm_nvfp4_moe import ( + TrtLlmNvFp4ExpertsModular, +) +from vllm.platforms import current_platform +from vllm.utils.flashinfer import has_flashinfer_trtllm_fused_moe +from vllm.utils.math_utils import next_power_of_2 +from vllm.utils.torch_utils import set_random_seed + +if pytest and ( + not has_flashinfer_trtllm_fused_moe() + or not current_platform.has_device_capability(100) +): + pytest.skip( + "Requires flashinfer TRTLLM fused MoE and NvFP4 (SM100)", + allow_module_level=True, + ) + +# (m, n, k) = (tokens, intermediate_size_per_partition, hidden_dim). +# The (64, 704, 4096) row matches Gemma4's MoE shape and exercises the +# non-256-aligned intermediate (padded inside the wrapper). +MNK_FACTORS = [ + (2, 1024, 1024), + (64, 2048, 1536), + (64, 704, 4096), +] + +_SWIGLU_LIMIT = 0.1 +_LARGE_OUTPUT1_SCALE = 32768.0 +_CLAMP_OP_NAME = "test_silu_and_mul_with_clamp" + +# Test-only fixed-limit clamp. ``custom_op_name`` makes the class itself +# valid as an ``activation=`` argument to ``torch_moe`` (which only looks +# up ``activation.custom_op_name`` in ``op_registry``), so no +# ``MoEActivation`` enum extension is needed. +if _CLAMP_OP_NAME not in op_registry: + + @CustomOp.register(_CLAMP_OP_NAME) + class _SiluAndMulWithClampTest(SiluAndMulWithClamp): + custom_op_name = _CLAMP_OP_NAME + + def __init__(self, *, compile_native: bool = True) -> None: + super().__init__(_SWIGLU_LIMIT, compile_native=compile_native) + + +SILU_WITH_CLAMP = op_registry[_CLAMP_OP_NAME] + + +ACTIVATION_CASES = [ + pytest.param(MoEActivation.SILU, MoEActivation.SILU, None, id="silu"), + pytest.param(MoEActivation.SILU, SILU_WITH_CLAMP, _SWIGLU_LIMIT, id="silu_clamp"), + pytest.param( + MoEActivation.RELU2_NO_MUL, + MoEActivation.RELU2_NO_MUL, + None, + id="relu2_no_mul", + ), + pytest.param(MoEActivation.GELU, MoEActivation.GELU, None, id="gelu"), +] + + +@pytest.mark.parametrize("m,n,k", MNK_FACTORS) +@pytest.mark.parametrize("e", [128]) +@pytest.mark.parametrize("topk", [8]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("activation,torch_activation,swiglu_limit", ACTIVATION_CASES) +@torch.inference_mode() +def test_trtllm_fp4_moe_no_graph( + m: int, + n: int, + k: int, + e: int, + topk: int, + dtype: torch.dtype, + activation: MoEActivation, + torch_activation: MoEActivation | type[SiluAndMulWithClamp], + swiglu_limit: float | None, + workspace_init, +): + # FlashInfer's trtllm_batched_gemm_runner has no precompiled tile + # config for non-gated RELU^2 at non-256-aligned intermediate_size + # (e.g. Gemma4's 704). Other activations (SiLU/GELU) work at the + # same shape. Tracked upstream in FlashInfer; unrelated to this + # PR's GELU enablement (Gemma4 uses GeGLU, not non-gated RELU^2). + if activation == MoEActivation.RELU2_NO_MUL and (m, n, k) == (64, 704, 4096): + pytest.skip( + "FlashInfer trtllm_batched_gemm_runner: no valid tile config " + "for non-gated RELU^2 at intermediate_size=704 " + "(getValidConfigIndices throws). Tracked upstream." + ) + + set_random_seed(7) + with set_current_vllm_config( + VllmConfig(parallel_config=ParallelConfig(pipeline_parallel_size=1)) + ): + a = torch.randn((m, k), device="cuda", dtype=dtype) / 10 + + quant_blocksize = 16 + is_gated_act = activation.is_gated + + w1_q, w2_q, quant_config = make_test_quant_config( + e, + n, + k, + in_dtype=dtype, + quant_dtype="nvfp4", + block_shape=None, + per_act_token_quant=False, + make_gate=is_gated_act, + # The TRT-LLM FP4 MoE kernel rejects swizzled (padded) activation + # scales — its numel-based vec_size check requires numel == M*K/16. + # Match what oracle/nvfp4.py does for this backend. + is_scale_swizzled=False, + ) + quant_config.gemm1_clamp_limit = swiglu_limit + if swiglu_limit is not None: + assert quant_config.g1_alphas is not None + assert quant_config.a2_gscale is not None + assert torch.all(quant_config.a2_gscale == 1) + # With a2_gscale == 1, g1_alphas is the TRTLLM + # output1_scale_gate_scalar. Make it large enough to catch + # clamp/output-scale coupling in the FlashInfer kernel wrapper. + quant_config.g1_alphas.fill_(_LARGE_OUTPUT1_SCALE) + + score = torch.randn((m, e), device="cuda", dtype=dtype) + topk_weights, topk_ids, _ = fused_topk(a, score, topk, renormalize=False) + + moe_config = FusedMoEConfig( + num_experts=e, + experts_per_token=topk, + hidden_dim=k, + intermediate_size=n, + num_local_experts=e, + num_logical_experts=e, + activation=activation, + device="cuda", + moe_parallel_config=FusedMoEParallelConfig.make_no_parallel(), + in_dtype=dtype, + routing_method=RoutingMethodType.TopK, + max_num_tokens=next_power_of_2(m), + ) + + trtllm_inner = TrtLlmNvFp4ExpertsModular( + moe_config=moe_config, quant_config=quant_config + ) + # Mimic the production weight-loader path so per-expert tensors that + # are normally precomputed in process_weights_after_loading (g1_scale_c + # and the rescaled gemm1_clamp_limit) get materialized. The test's + # synthetic quant_config has g1_alphas/g2_alphas already at their + # post-fusion values, so we set w13_weight_scale_2 to alias g1_alphas + # (same tensor) and use input_scale=1 to make the in-place + # weight_scale_2 *= input_scale step a no-op. + fake_layer = torch.nn.Module() + fake_layer.w13_weight_scale_2 = quant_config.g1_alphas + fake_layer.w2_weight_scale_2 = quant_config.g2_alphas + fake_layer.w13_input_scale = torch.ones_like(quant_config.g1_alphas) + fake_layer.w2_input_scale = torch.ones_like(quant_config.g2_alphas) + trtllm_inner.process_weights_after_loading(fake_layer) + + trtllm_experts = mk.FusedMoEKernel( + maybe_make_prepare_finalize( + moe=moe_config, + quant_config=quant_config, + allow_new_interface=True, + use_monolithic=False, + ), + trtllm_inner, + ) + + trtllm_output = trtllm_experts.apply( + hidden_states=a, + w1=w1_q, + w2=w2_q, + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=activation, + global_num_experts=e, + expert_map=None, + apply_router_weight_on_input=False, + ) + + # Reference: round-trip activations and weights through FP4 + # quant/dequant so the comparison isolates kernel/activation behavior + # from quantization error. + a_global_scale = ((FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX) / a.abs().max()).to( + torch.float32 + ) + a_fp4, a_scale_interleaved = ops.scaled_fp4_quant(a, a_global_scale) + a_in_dtype = dequantize_nvfp4_to_dtype( + a_fp4, + a_scale_interleaved, + a_global_scale, + dtype=a.dtype, + device=a.device, + block_size=quant_blocksize, + ) + + w1_d = torch.empty( + (e, (2 if is_gated_act else 1) * n, k), device="cuda", dtype=dtype + ) + w2_d = torch.empty((e, k, n), device="cuda", dtype=dtype) + for idx in range(e): + w1_d[idx] = dequantize_nvfp4_to_dtype( + w1_q[idx], + quant_config.w1_scale[idx], + (1 / quant_config.g1_alphas[idx]), + dtype=dtype, + device=w1_q.device, + block_size=quant_blocksize, + ) + w2_d[idx] = dequantize_nvfp4_to_dtype( + w2_q[idx], + quant_config.w2_scale[idx], + (1 / quant_config.g2_alphas[idx]), + dtype=dtype, + device=w2_q.device, + block_size=quant_blocksize, + ) + + torch_output = torch_moe( + a_in_dtype, w1_d, w2_d, score, topk, activation=torch_activation + ) + + torch.testing.assert_close(torch_output, trtllm_output, atol=2e-1, rtol=2e-1) + + +if __name__ == "__main__": + test_trtllm_fp4_moe_no_graph( + 64, + 704, + 4096, + 128, + 8, + torch.bfloat16, + MoEActivation.GELU, + MoEActivation.GELU, + None, + None, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_unquantized_backend_selection.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_unquantized_backend_selection.py new file mode 100644 index 0000000000000000000000000000000000000000..6a0db68edf01c3b4642c275172069eacb34e6835 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_unquantized_backend_selection.py @@ -0,0 +1,240 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from unittest.mock import patch + +import pytest + +from tests.kernels.moe.utils import make_dummy_moe_config +from vllm.model_executor.layers.fused_moe.oracle.unquantized import ( + UnquantizedMoeBackend, + select_unquantized_moe_backend, +) +from vllm.platforms import current_platform + +skipif_not_cuda_rocm = pytest.mark.skipif( + not (current_platform.is_cuda() or current_platform.is_rocm()), + reason="Only supported on CUDA/ROCm platforms.", +) + + +@pytest.mark.parametrize( + "platform_method,expected_backend", + [ + ("is_cuda", UnquantizedMoeBackend.TRITON), # Default CUDA without FlashInfer + ("is_rocm", UnquantizedMoeBackend.TRITON), # ROCm without AITER + ("is_cpu", UnquantizedMoeBackend.CPU), + ("is_xpu", UnquantizedMoeBackend.XPU), + ("is_tpu", UnquantizedMoeBackend.TPU), + ("is_out_of_tree", UnquantizedMoeBackend.OOT), + ], +) +@patch( + "vllm.utils.flashinfer.has_flashinfer", + return_value=False, +) +@patch( + "vllm.model_executor.layers.fused_moe.oracle.unquantized.rocm_aiter_ops.is_fused_moe_enabled", + return_value=False, +) +def test_select_default_backend_by_platform( + mock_aiter_enabled, + mock_has_flashinfer, + monkeypatch, + platform_method, + expected_backend, +): + """Test default backend selection per platform with all optional + accelerators (FlashInfer, AITER) disabled.""" + with patch( + "vllm.model_executor.layers.fused_moe.oracle.unquantized.current_platform" + ) as mock_platform: + # Set all platform checks to False + mock_platform.is_cuda.return_value = False + mock_platform.is_rocm.return_value = False + mock_platform.is_cpu.return_value = False + mock_platform.is_xpu.return_value = False + mock_platform.is_tpu.return_value = False + mock_platform.is_out_of_tree.return_value = False + + # Set only the specified platform to True + getattr(mock_platform, platform_method).return_value = True + + with ( + patch.object(current_platform, "is_cuda", return_value=False), + patch.object(current_platform, "is_rocm", return_value=False), + patch.object(current_platform, "is_cpu", return_value=False), + patch.object(current_platform, "is_xpu", return_value=False), + patch.object(current_platform, "is_tpu", return_value=False), + patch.object(current_platform, "is_out_of_tree", return_value=False), + patch.object(current_platform, platform_method, return_value=True), + ): + moe_config = make_dummy_moe_config() + selected_backend, expert_cls = select_unquantized_moe_backend( + moe_config=moe_config + ) + + assert selected_backend == expected_backend + if expected_backend in [ + UnquantizedMoeBackend.CPU, + UnquantizedMoeBackend.OOT, + UnquantizedMoeBackend.TPU, + ]: + assert expert_cls is None + else: + assert expert_cls is not None + + +@patch( + "vllm.utils.flashinfer.has_flashinfer", + return_value=False, +) +@patch( + "vllm.model_executor.layers.fused_moe.oracle.unquantized.rocm_aiter_ops.is_fused_moe_enabled", + return_value=True, +) +@pytest.mark.skipif( + not current_platform.is_rocm(), reason="ROCm-specific backend selection test" +) +def test_select_rocm_aiter_backend(mock_aiter_enabled, mock_has_flashinfer): + """Test ROCm backend selection when AITER is available.""" + with patch( + "vllm.model_executor.layers.fused_moe.oracle.unquantized.current_platform" + ) as mock_platform: + mock_platform.is_cuda.return_value = False + mock_platform.is_rocm.return_value = True + mock_platform.is_cpu.return_value = False + mock_platform.is_xpu.return_value = False + mock_platform.is_tpu.return_value = False + mock_platform.is_out_of_tree.return_value = False + + moe_config = make_dummy_moe_config() + selected_backend, expert_cls = select_unquantized_moe_backend( + moe_config=moe_config, + ) + + assert selected_backend == UnquantizedMoeBackend.AITER + assert expert_cls is not None + + +@patch( + "vllm.model_executor.layers.fused_moe.experts.trtllm_bf16_moe.TrtLlmBf16Experts.is_supported_config", + return_value=(True, None), +) +@pytest.mark.skipif( + not current_platform.is_cuda(), reason="Only supported on NVIDIA platforms." +) +def test_select_cuda_flashinfer_trtllm_backend(mock_is_supported_trtllm): + """Test CUDA backend selection when FlashInfer TRTLLM is available and enabled.""" + with ( + patch.object(current_platform, "is_cuda", return_value=True), + patch.object(current_platform, "is_rocm", return_value=False), + patch.object(current_platform, "is_cpu", return_value=False), + patch.object(current_platform, "is_xpu", return_value=False), + patch.object(current_platform, "is_tpu", return_value=False), + patch.object(current_platform, "is_out_of_tree", return_value=False), + patch.object(current_platform, "has_device_capability", return_value=True), + ): + moe_config = make_dummy_moe_config() + moe_config.moe_backend = "flashinfer_trtllm" + # TRTLLM requires EP and does not support DP + moe_config.moe_parallel_config.use_ep = True + moe_config.moe_parallel_config.use_dp = False + + selected_backend, experts_cls = select_unquantized_moe_backend( + moe_config=moe_config + ) + + assert selected_backend == UnquantizedMoeBackend.FLASHINFER_TRTLLM + assert experts_cls is not None + + +@patch( + "vllm.utils.flashinfer.has_flashinfer", + return_value=True, +) +@patch( + "vllm.model_executor.layers.fused_moe.experts.trtllm_bf16_moe.TrtLlmBf16Experts.is_supported_config", + return_value=(False, None), +) +@patch( + "vllm.model_executor.layers.fused_moe.experts.flashinfer_cutlass_moe.FlashInferExperts.is_supported_config", + return_value=(True, None), +) +@pytest.mark.skipif( + not current_platform.is_cuda(), reason="Only supported on NVIDIA platforms." +) +def test_select_cuda_flashinfer_cutlass_backend( + mock_has_flashinfer, + mock_is_supported_trtllm, + mock_is_supported_cutlass, +): + """Test CUDA backend selection when FlashInfer TRTLLM is not available + and FlashInfer CUTLASS is available.""" + with ( + patch.object(current_platform, "is_cuda", return_value=True), + patch.object(current_platform, "is_rocm", return_value=False), + patch.object(current_platform, "is_cpu", return_value=False), + patch.object(current_platform, "is_xpu", return_value=False), + patch.object(current_platform, "is_tpu", return_value=False), + patch.object(current_platform, "is_out_of_tree", return_value=False), + patch.object(current_platform, "has_device_capability", return_value=True), + ): + moe_config = make_dummy_moe_config() + # Select FlashInfer CUTLASS explicitly + moe_config.moe_backend = "flashinfer_cutlass" + # CUTLASS requires EP and does not support DP + moe_config.moe_parallel_config.use_ep = True + moe_config.moe_parallel_config.use_dp = False + + selected_backend, experts_cls = select_unquantized_moe_backend( + moe_config=moe_config + ) + + assert selected_backend == UnquantizedMoeBackend.FLASHINFER_CUTLASS + assert experts_cls is not None + + +@skipif_not_cuda_rocm +def test_select_lora_backend_prefers_triton(): + """LoRA-enabled unquantized MoE should select Triton backend.""" + moe_config = make_dummy_moe_config() + moe_config.is_lora_enabled = True + selected_backend, experts_cls = select_unquantized_moe_backend( + moe_config=moe_config + ) + + assert selected_backend == UnquantizedMoeBackend.TRITON + assert experts_cls is not None + + +@skipif_not_cuda_rocm +def test_select_lora_explicit_non_triton_backend(): + """LoRA should override explicit non-Triton backend to Triton.""" + moe_config = make_dummy_moe_config() + moe_config.is_lora_enabled = True + + # Use string from mapping in function map_unquantized_backend() + moe_config.moe_backend = "flashinfer_cutlass" + + selected_backend, experts_cls = select_unquantized_moe_backend( + moe_config=moe_config + ) + + assert selected_backend == UnquantizedMoeBackend.TRITON + assert experts_cls is not None + + +@skipif_not_cuda_rocm +@pytest.mark.parametrize("is_lora_enabled", [False, True]) +def test_select_explicit_triton_backend(is_lora_enabled): + """Explicit triton backend selection should return Triton.""" + moe_config = make_dummy_moe_config() + moe_config.is_lora_enabled = is_lora_enabled + moe_config.moe_backend = "triton" + + selected_backend, experts_cls = select_unquantized_moe_backend( + moe_config=moe_config + ) + + assert selected_backend == UnquantizedMoeBackend.TRITON + assert experts_cls is not None diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/test_zero_expert_moe.py b/asian_ai/performance/vLLM/tests/kernels/moe/test_zero_expert_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..58dd1d5f19973a00916938576bcc8a0dd737393b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/test_zero_expert_moe.py @@ -0,0 +1,295 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for FusedMoE with zero experts. + +Verifies that: +- The ZeroExpertRouter is properly created and used as the layer router. +- A forward pass through FusedMoE with zero experts produces correct output. +- The output decomposes correctly into real expert + zero expert contributions. + +Note: tests generated with Claude. +""" + +import pytest +import torch + +from vllm.config import VllmConfig, set_current_vllm_config +from vllm.forward_context import get_forward_context, set_forward_context +from vllm.model_executor.layers.fused_moe.layer import FusedMoE +from vllm.model_executor.layers.fused_moe.router.zero_expert_router import ( + ZeroExpertRouter, +) +from vllm.v1.worker.workspace import init_workspace_manager + + +@pytest.fixture +def zero_expert_moe(dist_init, default_vllm_config): + """Create a FusedMoE layer with zero experts.""" + num_experts = 4 + top_k = 2 + # hidden_size must be >= 256 for the zero expert identity kernel to + # produce output (its BLOCK_SIZE=256 causes grid=0 when hidden_dim<256). + hidden_size = 256 + intermediate_size = 512 + zero_expert_num = 1 + + e_score_correction_bias = torch.zeros( + num_experts + zero_expert_num, + dtype=torch.float32, + device="cuda", + ) + + vllm_config = VllmConfig() + vllm_config.compilation_config.static_forward_context = dict() + + with set_current_vllm_config(vllm_config), set_forward_context(None, vllm_config): + init_workspace_manager(torch.accelerator.current_device_index()) + + layer = FusedMoE( + zero_expert_type="identity", + e_score_correction_bias=e_score_correction_bias, + num_experts=num_experts, + top_k=top_k, + hidden_size=hidden_size, + intermediate_size=intermediate_size, + params_dtype=torch.bfloat16, + prefix="test_zero_expert_moe", + renormalize=False, + routed_scaling_factor=1.0, + scoring_func="softmax", + ).cuda() + + layer._quant_method.process_weights_after_loading(layer.routed_experts) + + yield layer, vllm_config + + +@pytest.mark.parametrize("num_tokens", [1, 32]) +def test_zero_expert_moe_router_is_zero_expert_router(zero_expert_moe, num_tokens): + """Verify that FusedMoE with zero_expert_type creates a ZeroExpertRouter.""" + layer, _ = zero_expert_moe + assert isinstance(layer.router, ZeroExpertRouter), ( + f"Expected ZeroExpertRouter but got {type(layer.router).__name__}." + ) + + +# @pytest.mark.parametrize("num_tokens", [1, 32]) +# def test_zero_expert_moe_no_custom_routing_fn(zero_expert_moe, num_tokens): +# """Verify that custom_routing_function is not set (routing is handled +# by ZeroExpertRouter, not a memoizing closure).""" +# layer, _ = zero_expert_moe +# #assert layer.custom_routing_function is None + + +@pytest.mark.parametrize("num_tokens", [1, 32]) +def test_zero_expert_moe_forward(zero_expert_moe, num_tokens): + """Run a forward pass through FusedMoE with zero experts and verify output shape.""" + layer, vllm_config = zero_expert_moe + + hidden_size = layer.routed_experts.hidden_size + num_experts = 4 + zero_expert_num = 1 + total_experts = num_experts + zero_expert_num + + hidden_states = torch.randn( + num_tokens, hidden_size, dtype=torch.bfloat16, device="cuda" + ) + router_logits = torch.randn( + num_tokens, total_experts, dtype=torch.float32, device="cuda" + ) + + # Initialize weights to small random values to avoid NaN from + # uninitialized memory. + with torch.no_grad(): + for param in layer.parameters(): + if param.dtype.is_floating_point: + param.normal_(0, 0.01) + + with set_current_vllm_config(vllm_config), set_forward_context(None, vllm_config): + get_forward_context().all_moe_layers = None + output = layer.forward(hidden_states, router_logits) + + assert output.shape == hidden_states.shape, ( + f"Expected output shape {hidden_states.shape}, got {output.shape}" + ) + assert output.dtype == hidden_states.dtype + assert not torch.isnan(output).any(), "Output contains NaN values" + + +@pytest.mark.parametrize("num_tokens", [1, 32]) +def test_zero_expert_moe_output_decomposition(zero_expert_moe, num_tokens): + """Validate that the FusedMoE output equals a plain FusedMoE + output (real experts only) plus the zero expert contribution. + + The key invariant is: + zero_layer.forward(h, r_full) == plain_layer.forward(h, r_real) + + zero_expert_output + + We create a plain FusedMoE layer with the same weights and real-expert-only + router logits, compute the zero expert output via the ZeroExpertRouter, and + verify the sum matches the FusedMoE output. + """ + layer, vllm_config = zero_expert_moe + num_experts = 4 + zero_expert_num = 1 + total_experts = num_experts + zero_expert_num + + hidden_states = torch.randn( + num_tokens, + layer.routed_experts.hidden_size, + dtype=torch.bfloat16, + device="cuda", + ) + router_logits = torch.randn( + num_tokens, total_experts, dtype=torch.float32, device="cuda" + ) + + with torch.no_grad(): + for param in layer.parameters(): + if param.dtype.is_floating_point: + param.normal_(0, 0.01) + + with set_current_vllm_config(vllm_config), set_forward_context(None, vllm_config): + get_forward_context().all_moe_layers = None + + # Create a plain FusedMoE layer with the same config but no zero + # experts. Use a separate prefix to avoid collision. + plain_layer = FusedMoE( + num_experts=num_experts, + top_k=layer.routed_experts.top_k, + hidden_size=layer.routed_experts.hidden_size, + intermediate_size=layer.routed_experts.intermediate_size_per_partition, + params_dtype=torch.bfloat16, + prefix="test_zero_expert_moe_plain", + renormalize=False, + scoring_func="softmax", + e_score_correction_bias=layer.routed_experts.e_score_correction_bias, + ).cuda() + + # Share weights from the zero expert layer. + plain_layer.routed_experts.w13_weight.data.copy_( + layer.routed_experts.w13_weight.data + ) + plain_layer.routed_experts.w2_weight.data.copy_( + layer.routed_experts.w2_weight.data + ) + plain_layer._quant_method.process_weights_after_loading( + plain_layer.routed_experts + ) + + # Compute routing via the ZeroExpertRouter. This produces masked + # topk_weights/topk_ids (zero expert entries have weight=0, id=0) + # and stores zero_expert_output as a side effect. + topk_weights, topk_ids = layer.router.select_experts( + hidden_states, router_logits + ) + zero_output = layer.router.zero_expert_output + + # Compute real expert output using the plain layer with the masked + # routing from the ZeroExpertRouter. + real_output = plain_layer._quant_method.apply( + layer=plain_layer.routed_experts, + x=hidden_states, + topk_weights=topk_weights, + topk_ids=topk_ids, + shared_experts=None, + shared_experts_input=None, + ) + + # Get the combined output from the zero expert layer. + full_output = layer.forward(hidden_states, router_logits) + + assert zero_output is not None, "Zero expert output should not be None" + assert not torch.isnan(real_output).any(), "Real expert output has NaN" + assert not torch.isnan(zero_output).any(), "Zero expert output has NaN" + assert not torch.isnan(full_output).any(), "Full output has NaN" + + expected = real_output + zero_output + torch.testing.assert_close( + full_output, + expected, + atol=4e-3, + rtol=4e-3, + msg="FusedMoE output should equal plain FusedMoE output " + "plus zero expert contribution", + ) + + +@pytest.mark.parametrize("num_tokens", [1, 32]) +def test_zero_expert_moe_zero_expert_is_identity(zero_expert_moe, num_tokens): + """Validate zero expert identity behavior. + + When routing strongly favors the zero expert, its contribution should + be a scaled version of hidden_states (identity operation). We verify + this by manually computing the expected zero expert output from the + routing weights and comparing against what the router produces. + """ + layer, vllm_config = zero_expert_moe + num_experts = 4 + zero_expert_num = 1 + total_experts = num_experts + zero_expert_num + + hidden_states = torch.randn( + num_tokens, + layer.routed_experts.hidden_size, + dtype=torch.bfloat16, + device="cuda", + ) + # Strongly bias toward the zero expert (index 4). + router_logits = torch.full( + (num_tokens, total_experts), -10.0, dtype=torch.float32, device="cuda" + ) + router_logits[:, num_experts] = 10.0 # zero expert gets high logit + + with torch.no_grad(): + for param in layer.parameters(): + if param.dtype.is_floating_point: + param.normal_(0, 0.01) + + with set_current_vllm_config(vllm_config), set_forward_context(None, vllm_config): + get_forward_context().all_moe_layers = None + + # Run routing to get topk_weights/topk_ids before masking. + from vllm.model_executor.layers.fused_moe.router.fused_topk_bias_router import ( + fused_topk_bias, + ) + + topk_weights, topk_ids = fused_topk_bias( + hidden_states=hidden_states, + gating_output=router_logits, + e_score_correction_bias=layer.router.e_score_correction_bias.data, + topk=layer.routed_experts.top_k, + renormalize=layer.router.renormalize, + scoring_func=layer.router.scoring_func, + ) + + # Manually compute expected zero expert identity output: + # For each token, sum routing weights assigned to zero expert slots, + # then multiply by hidden_states. + zero_mask = topk_ids >= num_experts + zero_weight_per_token = (topk_weights * zero_mask.float()).sum( + dim=-1, keepdim=True + ) + expected_zero_output = (hidden_states.float() * zero_weight_per_token).to( + hidden_states.dtype + ) + + # Run routing directly to trigger zero expert computation + # without going through the runner (which consumes the output). + layer.router.select_experts(hidden_states, router_logits) + actual_zero_output = layer.router.zero_expert_output + + assert actual_zero_output is not None + assert zero_mask.any(), ( + "With high zero expert logit, at least some slots should route " + "to the zero expert" + ) + + torch.testing.assert_close( + actual_zero_output, + expected_zero_output, + atol=1e-3, + rtol=1e-3, + msg="Zero expert identity output should equal " + "hidden_states * sum(zero_expert_weights)", + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/moe/utils.py b/asian_ai/performance/vLLM/tests/kernels/moe/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..fe6148a6dc5b45ee93ddc013b33064625658258c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/moe/utils.py @@ -0,0 +1,678 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import torch + +import vllm._custom_ops as ops +from tests.kernels.quant_utils import per_block_cast_to_int8 +from tests.kernels.quantization.nvfp4_utils import FLOAT4_E2M1_MAX, FLOAT8_E4M3_MAX +from vllm.model_executor.layers.activation import SiluAndMul +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.all2all_utils import ( + maybe_make_prepare_finalize, +) +from vllm.model_executor.layers.fused_moe.config import ( + FusedMoEConfig, + FusedMoEParallelConfig, + FusedMoEQuantConfig, + RoutingMethodType, +) +from vllm.model_executor.layers.fused_moe.experts.fused_batched_moe import ( + BatchedTritonExperts, + NaiveBatchedExperts, +) +from vllm.model_executor.layers.fused_moe.experts.triton_moe import ( + TritonExperts, +) +from vllm.model_executor.layers.fused_moe.fused_moe import ( + fused_experts, +) +from vllm.model_executor.layers.fused_moe.modular_kernel import FusedMoEKernel +from vllm.model_executor.layers.fused_moe.prepare_finalize.batched import ( + BatchedPrepareAndFinalize, +) +from vllm.model_executor.layers.fused_moe.router.fused_topk_router import fused_topk +from vllm.model_executor.layers.fused_moe.utils import moe_kernel_quantize_input +from vllm.utils.deep_gemm import per_block_cast_to_fp8 +from vllm.utils.math_utils import round_up + + +def shuffle_weight(w: torch.Tensor) -> torch.Tensor: + """Fold weights to adjacent locations for Triton MoE / SwiGLU kernel layout.""" + shape = w.shape + n = shape[-1] + first = w[..., : n // 2] + second = w[..., n // 2 :] + stacked = torch.stack((first, second), dim=-1) + return stacked.reshape(shape) + + +def make_dummy_moe_config( + num_experts: int = 1, + num_local_experts: int | None = None, + experts_per_token: int = 1, + hidden_dim: int = 1, + intermediate_size: int = 1, + in_dtype: torch.dtype = torch.bfloat16, + max_num_tokens: int = 512, +) -> FusedMoEConfig: + """ + This is a dummy config for the mk constructor interface + as most kernels like DeepGEMM, CUTLASSFp4, Triton, MARLIN + do not actually use this config. + + CUTLASSFp8 needs to set some params for workshapes. + """ + return FusedMoEConfig( + num_experts=num_experts, + experts_per_token=experts_per_token, + hidden_dim=hidden_dim, + intermediate_size=intermediate_size, + num_local_experts=num_local_experts + if num_local_experts is not None + else num_experts, + num_logical_experts=num_experts, + moe_parallel_config=FusedMoEParallelConfig.make_no_parallel(), + activation=MoEActivation.SILU, + in_dtype=in_dtype, + device="cuda", + routing_method=RoutingMethodType.TopK, + max_num_tokens=max_num_tokens, + ) + + +def triton_moe( + a: torch.Tensor, + w1: torch.Tensor, + w2: torch.Tensor, + topk_weight: torch.Tensor, + topk_ids: torch.Tensor, + w1_scale: torch.Tensor | None = None, + w2_scale: torch.Tensor | None = None, + a1_scale: torch.Tensor | None = None, + a2_scale: torch.Tensor | None = None, + quant_dtype: torch.dtype | None = None, + per_act_token_quant=False, + block_shape: list[int] | None = None, +) -> torch.Tensor: + quant_config = FusedMoEQuantConfig.make( + quant_dtype, + per_act_token_quant=per_act_token_quant, + block_shape=block_shape, + w1_scale=w1_scale, + w2_scale=w2_scale, + a1_scale=a1_scale, + a2_scale=a2_scale, + ) + + return fused_experts(a, w1, w2, topk_weight, topk_ids, quant_config=quant_config) + + +def batched_moe( + a: torch.Tensor, + w1: torch.Tensor, + w2: torch.Tensor, + topk_weight: torch.Tensor, + topk_ids: torch.Tensor, + w1_scale: torch.Tensor | None = None, + w2_scale: torch.Tensor | None = None, + a1_scale: torch.Tensor | None = None, + a2_scale: torch.Tensor | None = None, + quant_dtype: torch.dtype | None = None, + per_act_token_quant: bool = False, + block_shape: list[int] | None = None, +) -> torch.Tensor: + max_num_tokens = round_up(a.shape[0], 64) + + quant_config = FusedMoEQuantConfig.make( + quant_dtype, + per_act_token_quant=per_act_token_quant, + block_shape=block_shape, + w1_scale=w1_scale, + w2_scale=w2_scale, + a1_scale=a1_scale, + a2_scale=a2_scale, + ) + + moe_config = make_dummy_moe_config() + + fused_experts = FusedMoEKernel( + BatchedPrepareAndFinalize( + max_num_tokens, num_dispatchers=1, num_local_experts=w1.shape[0], rank=0 + ), + BatchedTritonExperts( + max_num_tokens=max_num_tokens, + num_dispatchers=1, + quant_config=quant_config, + moe_config=moe_config, + ), + ) + + return fused_experts.apply( + a, + w1, + w2, + topk_weight, + topk_ids, + global_num_experts=w1.shape[0], + activation=moe_config.activation, + apply_router_weight_on_input=False, + expert_map=None, + ) + + +def naive_batched_moe( + a: torch.Tensor, + w1: torch.Tensor, + w2: torch.Tensor, + topk_weight: torch.Tensor, + topk_ids: torch.Tensor, + w1_scale: torch.Tensor | None = None, + w2_scale: torch.Tensor | None = None, + a1_scale: torch.Tensor | None = None, + a2_scale: torch.Tensor | None = None, + quant_dtype: torch.dtype | None = None, + per_act_token_quant: bool = False, + block_shape: list[int] | None = None, +) -> torch.Tensor: + max_num_tokens = round_up(a.shape[0], 64) + + quant_config = FusedMoEQuantConfig.make( + quant_dtype, + per_act_token_quant=per_act_token_quant, + block_shape=block_shape, + w1_scale=w1_scale, + w2_scale=w2_scale, + a1_scale=a1_scale, + a2_scale=a2_scale, + ) + moe_config = make_dummy_moe_config() + + fused_experts = FusedMoEKernel( + BatchedPrepareAndFinalize( + max_num_tokens, num_dispatchers=1, num_local_experts=w1.shape[0], rank=0 + ), + NaiveBatchedExperts( + max_num_tokens=max_num_tokens, + num_dispatchers=1, + quant_config=quant_config, + moe_config=moe_config, + ), + ) + + return fused_experts.apply( + a, + w1, + w2, + topk_weight, + topk_ids, + global_num_experts=w1.shape[0], + activation=moe_config.activation, + apply_router_weight_on_input=False, + expert_map=None, + ) + + +def chunk_scales( + scales: torch.Tensor | None, start: int, end: int +) -> torch.Tensor | None: + if scales is not None: + if scales.numel() == 1: + return scales + else: + return scales[start:end] + return None + + +def make_quantized_test_activations( + E: int, + m: int, + k: int, + in_dtype: torch.dtype, + quant_dtype: torch.dtype | None = None, + block_shape: list[int] | None = None, + per_act_token_quant: bool = False, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor | None]: + a = torch.randn((E, m, k), device="cuda", dtype=in_dtype) / 10 + a_q = a + a_scale = None + + if quant_dtype is not None: + assert quant_dtype == torch.float8_e4m3fn or quant_dtype == torch.int8, ( + "only fp8/int8 supported" + ) + a_q = torch.zeros_like(a, dtype=quant_dtype) + a_scale_l = [None] * E + for e in range(E): + a_q[e], a_scale_l[e] = moe_kernel_quantize_input( + a[e], None, quant_dtype, per_act_token_quant, block_shape + ) + a_scale = torch.stack(a_scale_l) + + if not per_act_token_quant and block_shape is None: + a_scale = a_scale.view(E, 1, 1) + + return a, a_q, a_scale + + +def moe_quantize_weights_2d( + w: torch.Tensor, + w_s: torch.Tensor | None, + quant_dtype: torch.dtype | str | None, + per_token_quant: bool, + block_shape: list[int] | None, +) -> tuple[torch.Tensor, torch.Tensor | None, torch.Tensor | None]: + assert ( + quant_dtype == torch.float8_e4m3fn + or quant_dtype == torch.int8 + or quant_dtype == "nvfp4" + ), "only fp8/int8/nvfp4 supported" + + w_gs = None + + if block_shape is not None: + assert not per_token_quant + if quant_dtype == torch.int8: + w, w_s = per_block_cast_to_int8(w, block_shape) + elif quant_dtype == torch.float8_e4m3fn: + w, w_s = per_block_cast_to_fp8(w, block_shape) + elif quant_dtype == "nvfp4": + raise RuntimeError("blocked quantization not supported for nvfp4") + else: + raise RuntimeError(f"Unsupported quant type {quant_dtype}") + else: + if quant_dtype == torch.int8: + w, w_s = ops.scaled_int8_quant( + w, w_s, use_per_token_if_dynamic=per_token_quant + ) + elif quant_dtype == torch.float8_e4m3fn: + w, w_s = ops.scaled_fp8_quant( + w, w_s, use_per_token_if_dynamic=per_token_quant + ) + elif quant_dtype == "nvfp4": + assert not per_token_quant + w_amax = torch.abs(w).max().to(torch.float32) + w_gs = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX / w_amax + w, w_s = ops.scaled_fp4_quant(w, w_gs) + else: + raise RuntimeError(f"Unsupported quant type {quant_dtype}") + + return w, w_s, w_gs + + +def moe_quantize_weights( + w: torch.Tensor, + w_s: torch.Tensor | None, + quant_dtype: torch.dtype | str | None, + per_token_quant: bool, + block_shape: list[int] | None, +) -> tuple[torch.Tensor, torch.Tensor | None, torch.Tensor | None]: + assert w.dim() == 3 + e, rows, cols = w.shape + w_l = [None] * e + w_s_l = [None] * e + w_gs_l = [None] * e + for idx in range(e): + w_l[idx], w_s_l[idx], w_gs_l[idx] = moe_quantize_weights_2d( + w[idx], None, quant_dtype, per_token_quant, block_shape + ) + + w = torch.stack(w_l) + w_s = torch.stack(w_s_l) + w_gs = torch.stack(w_gs_l) if e > 0 and w_gs_l[0] is not None else None + + if w_s.ndim == 2: + assert w_s.shape[-1] == 1 + w_s = w_s.view(-1, 1, 1) + + if block_shape is not None: + block_n, block_k = block_shape + n_tiles = (rows + block_n - 1) // block_n + k_tiles = (cols + block_k - 1) // block_k + assert w_s.shape == (e, n_tiles, k_tiles) + + return w, w_s, w_gs + + +def make_test_weight( + e: int, + rows: int, + cols: int, + in_dtype: torch.dtype = torch.bfloat16, + quant_dtype: torch.dtype | str | None = None, + block_shape: list[int] | None = None, + per_out_ch_quant: bool = False, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor | None, torch.Tensor | None]: + w_16 = torch.randn((e, rows, cols), device="cuda", dtype=in_dtype) / 15 + + if quant_dtype is not None: + w, w_s, w_gs = moe_quantize_weights( + w_16, None, quant_dtype, per_out_ch_quant, block_shape + ) + else: + w = w_16 + w_s = None + w_gs = None + + return w_16, w, w_s, w_gs + + +def make_test_weights( + e: int, + n: int, + k: int, + in_dtype: torch.dtype = torch.bfloat16, + quant_dtype: torch.dtype | str | None = None, + block_shape: list[int] | None = None, + per_out_ch_quant: bool = False, + make_gate: bool = True, +) -> tuple[ + tuple[torch.Tensor, torch.Tensor, torch.Tensor | None, torch.Tensor | None], + tuple[torch.Tensor, torch.Tensor, torch.Tensor | None, torch.Tensor | None], +]: + return ( + make_test_weight( + e, + (2 if make_gate else 1) * n, + k, + in_dtype, + quant_dtype, + block_shape, + per_out_ch_quant, + ), + make_test_weight(e, k, n, in_dtype, quant_dtype, block_shape, per_out_ch_quant), + ) + + +def per_token_cast_to_fp8( + x: torch.Tensor, block_size: int = 128 +) -> tuple[torch.Tensor, torch.Tensor]: + assert x.dim() == 2 + m, n = x.shape + pad_size = (block_size - (n % block_size)) % block_size + x = torch.nn.functional.pad(x, (0, pad_size), value=0) if pad_size > 0 else x + x_view = x.view(m, -1, block_size) + x_amax = x_view.abs().float().amax(dim=2).view(m, -1).clamp(1e-4) + fp8_data = (x_view * (448.0 / x_amax.unsqueeze(2))).to(torch.float8_e4m3fn) + return fp8_data.view(m, n + pad_size)[:, :n], (x_amax / 448.0).view(m, -1) + + +def make_test_quant_config( + e: int, + n: int, + k: int, + in_dtype: torch.dtype, + quant_dtype: torch.dtype | str | None = None, + per_act_token_quant: bool = False, + block_shape: list[int] | None = None, + make_gate: bool = True, + is_scale_swizzled: bool = True, +) -> tuple[torch.Tensor, torch.Tensor, FusedMoEQuantConfig]: + (_, w1, w1_s, w1_gs), (_, w2, w2_s, w2_gs) = make_test_weights( + e, + n, + k, + in_dtype, + quant_dtype, + per_out_ch_quant=per_act_token_quant, + block_shape=block_shape, + make_gate=make_gate, + ) + + # Hacky/trivial scales for nvfp4. + a1_gscale: torch.Tensor | None = None + a2_gscale: torch.Tensor | None = None + if quant_dtype == "nvfp4": + a1_gscale = torch.ones((e,), device="cuda", dtype=torch.float32) + a2_gscale = torch.ones((e,), device="cuda", dtype=torch.float32) + a1_scale = a1_gscale + a2_scale = a2_gscale + else: + a1_scale = None + a2_scale = None + + return ( + w1, + w2, + FusedMoEQuantConfig.make( + quant_dtype, + per_act_token_quant=per_act_token_quant, + block_shape=block_shape, + w1_scale=w1_s, + w2_scale=w2_s, + a1_gscale=a1_gscale, + a2_gscale=a2_gscale, + a1_scale=a1_scale, + a2_scale=a2_scale, + # TODO: make sure this is handled properly + g1_alphas=(1 / w1_gs) if w1_gs is not None else None, + g2_alphas=(1 / w2_gs) if w2_gs is not None else None, + is_scale_swizzled=is_scale_swizzled, + ), + ) + + +def fused_moe( + hidden_states: torch.Tensor, + w1: torch.Tensor, + w2: torch.Tensor, + score: torch.Tensor, + topk: int, + renormalize: bool = False, + quant_config: FusedMoEQuantConfig | None = None, + global_num_experts: int = -1, + expert_map: torch.Tensor | None = None, +) -> torch.Tensor: + topk_weights, topk_ids, _ = fused_topk( + hidden_states, score.float(), topk, renormalize + ) + return fused_experts( + hidden_states, + w1, + w2, + topk_weights, + topk_ids, + global_num_experts=global_num_experts, + expert_map=expert_map, + quant_config=quant_config, + ) + + +class BaselineMM(torch.nn.Module): + def __init__( + self, + b: torch.Tensor, + out_dtype: torch.dtype, + ): + super().__init__() + self.b = torch.nn.Parameter(b.to(dtype=torch.float32)) + self.out_dtype = out_dtype + + def forward(self, a: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor | None]: + return torch.mm(a.to(dtype=torch.float32), self.b).to(self.out_dtype), None + + +class BaselineSiluAndMul(torch.nn.Module): + def __init__(self): + super().__init__() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + d = x.shape[-1] // 2 + return torch.nn.functional.silu(x[..., :d]) * x[..., d:] + + +class TestMLP(torch.nn.Module): + def __init__( + self, + w1: torch.Tensor, + w2: torch.Tensor, + out_dtype: torch.dtype, + ): + super().__init__() + self.gate_up_proj = BaselineMM(w1, out_dtype) + self.down_proj = BaselineMM(w2, out_dtype) + self.act_fn = BaselineSiluAndMul() + + def forward(self, x): + x, _ = self.gate_up_proj(x) + x = self.act_fn(x) + x, _ = self.down_proj(x) + return x + + +def make_naive_shared_experts( + N: int, + K: int, + in_dtype: torch.dtype = torch.bfloat16, +) -> torch.nn.Module: + w1 = torch.randn((K, N * 2), device="cuda", dtype=in_dtype) / 15 + w2 = torch.randn((N, K), device="cuda", dtype=in_dtype) / 15 + return TestMLP(w1, w2, out_dtype=in_dtype) + + +class RealMLP(torch.nn.Module): + def __init__( + self, + hidden_size: int, + intermediate_size: int, + w1: torch.Tensor, + w2: torch.Tensor, + hidden_act: str = "silu", + quant_config=None, + reduce_results: bool = True, + prefix: str = "", + w1_s: torch.Tensor | None = None, + w2_s: torch.Tensor | None = None, + ) -> None: + from vllm.model_executor.layers.linear import ( + MergedColumnParallelLinear, + RowParallelLinear, + ) + + super().__init__() + self.gate_up_proj = MergedColumnParallelLinear( + hidden_size, + [intermediate_size] * 2, + bias=False, + quant_config=quant_config, + prefix=f"{prefix}.gate_up_proj", + ) + self.gate_up_proj.register_parameter( + "weight", torch.nn.Parameter(w1, requires_grad=False) + ) + self.gate_up_proj.register_parameter( + "weight_scale", torch.nn.Parameter(w1_s, requires_grad=False) + ) + self.gate_up_proj.register_parameter( + "input_scale", None + ) # torch.nn.Parameter(None, requires_grad=False)) + self.down_proj = RowParallelLinear( + intermediate_size, + hidden_size, + bias=False, + quant_config=quant_config, + reduce_results=reduce_results, + prefix=f"{prefix}.down_proj", + ) + self.down_proj.register_parameter( + "weight", torch.nn.Parameter(w2, requires_grad=False) + ) + self.down_proj.register_parameter( + "weight_scale", torch.nn.Parameter(w2_s, requires_grad=False) + ) + self.down_proj.register_parameter( + "input_scale", None + ) # torch.nn.Parameter(None, requires_grad=False)) + if hidden_act != "silu": + raise ValueError( + f"Unsupported activation: {hidden_act}. Only silu is supported for now." + ) + self.act_fn = SiluAndMul() + + def forward(self, x): + gate_up, _ = self.gate_up_proj(x) + x = self.act_fn(gate_up) + x, _ = self.down_proj(x) + return x + + +def make_shared_experts_with_weights( + N: int, + K: int, + in_dtype: torch.dtype, + w1: torch.Tensor, + w2: torch.Tensor, + w1_s: torch.Tensor | None = None, + w2_s: torch.Tensor | None = None, + quant_dtype: torch.dtype | str | None = None, +) -> torch.nn.Module: + old_dtype = torch.get_default_dtype() + try: + torch.set_default_dtype(in_dtype) + if quant_dtype == torch.float8_e4m3fn: + from vllm.model_executor.layers.quantization.fp8 import Fp8Config + + quant_config = Fp8Config(True) + else: + quant_config = None + + return RealMLP(K, N, w1, w2, "silu", quant_config, w1_s=w1_s, w2_s=w2_s) + finally: + torch.set_default_dtype(old_dtype) + + +def modular_triton_fused_moe( + moe_config: FusedMoEConfig, + quant_config: FusedMoEQuantConfig, +) -> FusedMoEKernel: + return FusedMoEKernel( + maybe_make_prepare_finalize( + moe=moe_config, + quant_config=quant_config, + allow_new_interface=True, + use_monolithic=False, + ), + TritonExperts(moe_config, quant_config), + ) + + +def make_shared_experts( + N: int, + K: int, + in_dtype: torch.dtype = torch.bfloat16, + quant_dtype: torch.dtype | str | None = None, +) -> torch.nn.Module: + (_, w1, w1_s, _), (_, w2, w2_s, _) = make_test_weights( + 1, + N, + K, + in_dtype=in_dtype, + quant_dtype=quant_dtype, + ) + + return make_shared_experts_with_weights( + N, K, in_dtype, w1, w2, w1_s=w1_s, w2_s=w2_s, quant_dtype=quant_dtype + ) + + +def check_accuracy(a, b, atol, rtol, percent): + """Allow a mismatch percentage of 1 - percent.""" + if torch.any(torch.isnan(a)): + raise Exception("NaN in reference output") + if torch.any(torch.isnan(b)): + raise Exception("NaN in actual output") + if torch.any(torch.isinf(a)): + raise Exception("Inf in reference output") + if torch.any(torch.isinf(b)): + raise Exception("Inf in actual output") + assert a.shape == b.shape, f"Shape mismatch: {a.shape} vs {b.shape}" + + left = torch.abs(a - b) + right = atol + rtol * torch.abs(b) + count = torch.sum(left > right) + mismatch_percent = count / a.numel() + if mismatch_percent > 1 - percent: + raise Exception( + f"Mismatch percentage is {mismatch_percent:.4f} for rtol {rtol} " + f"(threshold: {1 - percent:.4f})" + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/quant_utils.py b/asian_ai/performance/vLLM/tests/kernels/quant_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..5eee8752882b99ed6177574148075ff3d968c7cb --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quant_utils.py @@ -0,0 +1,304 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +import torch + +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + get_fp8_min_max, + group_broadcast, +) +from vllm.platforms import current_platform +from vllm.utils.deep_gemm import _ceil_to_ue8m0, is_deep_gemm_e8m0_used +from vllm.utils.math_utils import round_up + +FP8_DTYPE = current_platform.fp8_dtype() + + +def as_float32_tensor(x: float | torch.Tensor) -> torch.Tensor: + return torch.as_tensor(x, dtype=torch.float32, device="cuda") + + +def ref_dynamic_per_token_quant( + x: torch.Tensor, quant_dtype: torch.dtype, scale_ub: torch.Tensor | None = None +) -> tuple[torch.Tensor, torch.Tensor]: + assert quant_dtype in [torch.int8, FP8_DTYPE] + if scale_ub is not None: + assert quant_dtype == FP8_DTYPE + + if quant_dtype == torch.int8: + qtype_traits = torch.iinfo(quant_dtype) + qtype_traits_min = qtype_traits.min + qtype_traits_max = qtype_traits.max + else: + qtype_traits_min, qtype_traits_max = get_fp8_min_max() + qtype_max = as_float32_tensor(qtype_traits_max) + s_1 = as_float32_tensor(1.0) + s_512 = as_float32_tensor(512.0) + + # For fp8, in order to match the cuda kernel output, we have to do exactly + # the same operations as in the corresponding fp8 kernel to prevent + # rounding errors. + + # Compute scales + x_token_max, _ = x.abs().max(dim=-1) + x_token_max = as_float32_tensor(x_token_max) + if scale_ub is not None: + x_token_max = x_token_max.clamp(max=scale_ub) + scales = (x_token_max / qtype_max)[:, None] + + # Quant + if quant_dtype == torch.int8: + iscales = as_float32_tensor(s_1 / scales) + torch_out = as_float32_tensor(x) * iscales + torch_out = torch_out.round() + torch_out = torch_out.clamp(qtype_traits_min, qtype_traits_max).to(quant_dtype) + else: + assert quant_dtype == FP8_DTYPE + min_scaling_factor = s_1 / (qtype_max * s_512) + scales = scales.clamp(min=min_scaling_factor) + torch_out = as_float32_tensor(x) / scales + torch_out = torch_out.clamp(qtype_traits_min, qtype_traits_max).to(quant_dtype) + + return torch_out, scales + + +# The int8 version is very similar. Incorporate the int8 version, like in +# ref_dynamic_per_token_quant, when we have a dynamic_per_tensor int8 quant +# kernel +def ref_dynamic_per_tensor_fp8_quant( + x: torch.Tensor, +) -> tuple[torch.Tensor, torch.Tensor]: + fp8_traits_min, fp8_traits_max = get_fp8_min_max() + fp8_max = as_float32_tensor(fp8_traits_max) + one = as_float32_tensor(1.0) + + # For fp8, in order to match the cuda kernel output, we have to do exactly + # the same operations as in the corresponding fp8 kernel to prevent + # rounding errors. + + x_max = as_float32_tensor(x.abs().max()) + ref_scale = x_max / fp8_max + ref_iscale = one / ref_scale + ref_out = ( + (as_float32_tensor(x) * ref_iscale) + .clamp(fp8_traits_min, fp8_traits_max) + .to(FP8_DTYPE) + ) + return ref_out, ref_scale.view(1) + + +def native_w8a8_block_matmul( + A: torch.Tensor, + B: torch.Tensor, + As: torch.Tensor, + Bs: torch.Tensor, + block_size: list[int], + output_dtype: torch.dtype, + compute_type: torch.dtype = torch.float32, +) -> torch.Tensor: + """This function performs matrix multiplication with block-wise + quantization using native torch. + It is agnostic to the input data type and can be used for both int8 and + fp8 data types. + + It takes two input tensors `A` and `B` (int8) with scales `As` and + `Bs` (float32). + The output is returned in the specified `output_dtype`. + """ + A = A.to(compute_type) + B = B.to(compute_type) + assert A.shape[-1] == B.shape[-1] + assert B.ndim == 2 and B.is_contiguous() and Bs.ndim == 2 + assert len(block_size) == 2 + block_n, block_k = block_size[0], block_size[1] + assert (A.shape[-1] + block_k - 1) // block_k == As.shape[-1] + assert A.shape[:-1] == As.shape[:-1] + + M = A.numel() // A.shape[-1] + N, K = B.shape + origin_C_shape = A.shape[:-1] + (N,) + A = A.reshape(M, A.shape[-1]) + As = As.reshape(M, As.shape[-1]) + n_tiles = (N + block_n - 1) // block_n + k_tiles = (K + block_k - 1) // block_k + assert n_tiles == Bs.shape[0], f"{n_tiles} == {Bs.shape[0]}" + assert k_tiles == Bs.shape[1], f"{k_tiles} == {Bs.shape[1]}" + + C_shape = (M, N) + C = torch.zeros(C_shape, dtype=compute_type, device=A.device) + + A_tiles = [A[:, i * block_k : min((i + 1) * block_k, K)] for i in range(k_tiles)] + B_tiles = [ + [ + B[ + j * block_n : min((j + 1) * block_n, N), + i * block_k : min((i + 1) * block_k, K), + ] + for i in range(k_tiles) + ] + for j in range(n_tiles) + ] + C_tiles = [C[:, j * block_n : min((j + 1) * block_n, N)] for j in range(n_tiles)] + As_tiles = [As[:, i : i + 1] for i in range(k_tiles)] + + for i in range(k_tiles): + for j in range(n_tiles): + a = A_tiles[i] + b = B_tiles[j][i] + c = C_tiles[j] + s = As_tiles[i] * Bs[j][i] + c[:, :] += torch.matmul(a, b.t()) * s + + C = C.reshape(origin_C_shape).to(output_dtype) + return C + + +def native_per_token_group_quant_fp8( + x, group_size, eps=1e-10, dtype=torch.float8_e4m3fn +): + """Function to perform per-token-group quantization on an input tensor + `x` using native torch.""" + assert x.shape[-1] % group_size == 0, ( + "the last dimension of `x` must be divisible by `group_size`" + ) + assert x.is_contiguous(), "`x` is not contiguous" + + finfo = torch.finfo(dtype) + fp8_min = finfo.min + fp8_max = finfo.max + + x_ = x.reshape(x.numel() // group_size, group_size) + amax = x_.abs().max(dim=-1, keepdim=True)[0].clamp(min=eps).to(torch.float32) + x_s = amax / fp8_max + if is_deep_gemm_e8m0_used(): + x_s = _ceil_to_ue8m0(x_s) + x_q = (x_ / x_s).clamp(min=fp8_min, max=fp8_max).to(dtype) + x_q = x_q.reshape(x.shape) + x_s = x_s.reshape(x.shape[:-1] + (x.shape[-1] // group_size,)) + + return x_q, x_s + + +def native_per_token_group_quant_int8(x, group_size, eps=1e-10, dtype=torch.int8): + """Function to perform per-token-group quantization on an input tensor + `x` using native torch. + + It converts the tensor values into int8 values and returns the + quantized tensor along with the scaling factor used for quantization. + """ + assert x.shape[-1] % group_size == 0, ( + "the last dimension of `x` must be divisible by `group_size`" + ) + assert x.is_contiguous(), "`x` is not contiguous" + + iinfo = torch.iinfo(dtype) + int8_min = iinfo.min + int8_max = iinfo.max + + x_ = x.reshape(x.numel() // group_size, group_size) + # Use float32 for scale calculation for stability + amax = x_.abs().max(dim=-1, keepdim=True)[0].clamp(min=eps).to(torch.float32) + x_s = amax / int8_max + x_q = ( + (x_.to(torch.float32) / x_s).round().clamp(min=int8_min, max=int8_max).to(dtype) + ) # Round before clamping + x_q = x_q.reshape(x.shape) + x_s = x_s.reshape(x.shape[:-1] + (x.shape[-1] // group_size,)) + + return x_q, x_s + + +DEFAULT_BLOCK_SHAPE = [128, 128] + + +def per_block_cast_to_int8( + x: torch.Tensor, + block_shape: list[int] = DEFAULT_BLOCK_SHAPE, +) -> tuple[torch.Tensor, torch.Tensor]: + block_m, block_n = block_shape + assert x.dim() == 2 + m, n = x.shape + x_padded = torch.zeros( + (round_up(m, block_m), round_up(n, block_n)), dtype=x.dtype, device=x.device + ) + x_padded[:m, :n] = x + x_view = x_padded.view(-1, block_m, x_padded.size(1) // block_n, block_n) + x_amax = x_view.abs().float().amax(dim=(1, 3), keepdim=True).clamp(1e-4) + x_scaled = (x_view * (256.0 / x_amax)).to(torch.int8) + x_scaled_sub = x_scaled.view_as(x_padded)[:m, :n].contiguous() + scales = (x_amax / 256.0).view(x_view.size(0), x_view.size(2)) + return x_scaled_sub, scales + + +def dequant( + t: torch.Tensor, + scale: torch.Tensor | None, + block_shape: list[int] | None, + per_act_token_quant: bool, + out_dtype: torch.dtype | None = torch.float32, +) -> torch.Tensor: + if scale is not None: + f32 = torch.float32 + if per_act_token_quant or block_shape is None: + return (t.to(f32) * scale).to(out_dtype) + else: + return (t.to(f32) * group_broadcast(scale, t.shape)).to(out_dtype) + else: + return t.to(out_dtype) + + +def batched_dequant( + t: torch.Tensor, + scale: torch.Tensor | None, + block_shape: list[int] | None, + per_act_token_quant: bool, + out_dtype: torch.dtype | None = torch.float32, +) -> torch.Tensor: + if scale is not None: + assert t.shape[0] == scale.shape[0] + out = torch.empty_like(t, dtype=out_dtype) + for e in range(t.shape[0]): + out[e] = dequant( + t[e], scale[e], block_shape, per_act_token_quant, out_dtype + ) + return out + + return t.to(out_dtype) + + +def native_batched_masked_quant_matmul( + A: torch.Tensor, + B: torch.Tensor, + C: torch.Tensor, + num_expert_tokens: torch.Tensor, + A_scale: torch.Tensor | None = None, + B_scale: torch.Tensor | None = None, + block_shape: list[int] | None = None, + per_act_token_quant: bool = False, +) -> torch.Tensor: + num_expert_tokens_cpu = num_expert_tokens.clone() + num_expert_tokens_cpu = num_expert_tokens_cpu.to(device="cpu") + num_experts = num_expert_tokens.size(0) + + for e in range(num_experts): + num_tokens = num_expert_tokens_cpu[e] + if A.dtype.itemsize == 1 and block_shape is not None: + assert A_scale is not None and B_scale is not None + tmp = native_w8a8_block_matmul( + A[e], B[e], A_scale[e], B_scale[e], block_shape, C.dtype + ) + C[e, :num_tokens, :] = tmp[:num_tokens, :] + elif A.dtype.itemsize == 1 and block_shape is None: + assert A_scale is not None and B_scale is not None + A_dq = dequant(A[e], A_scale[e], block_shape, per_act_token_quant) + B_dq = dequant(B[e], B_scale[e], block_shape, per_act_token_quant) + C[e, :num_tokens, :] = (A_dq[:num_tokens] @ B_dq.transpose(0, 1)).to( + C.dtype + ) + else: + assert A_scale is None + assert B_scale is None + C[e, :num_tokens, :] = A[e, :num_tokens, :] @ B[e].transpose(0, 1) + + return C diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/nvfp4_utils.py b/asian_ai/performance/vLLM/tests/kernels/quantization/nvfp4_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..37aae24e65353e46df65f9f0e64ca28873c97100 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/nvfp4_utils.py @@ -0,0 +1,152 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import torch + +from vllm._custom_ops import scaled_fp4_quant +from vllm.scalar_type import scalar_types + +FLOAT4_E2M1_MAX = scalar_types.float4_e2m1f.max() +FLOAT8_E4M3_MAX = torch.finfo(torch.float8_e4m3fn).max + +kE2M1ToFloat = torch.tensor( + [0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0], dtype=torch.float32 +) + + +def convert_swizzled_to_linear(a_sf_swizzled: torch.Tensor, m, k, block_size): + m_tiles = (m + 128 - 1) // 128 + f = block_size * 4 + k_tiles = (k + f - 1) // f + tmp = torch.reshape(a_sf_swizzled, (1, m_tiles, k_tiles, 32, 4, 4)) + tmp = torch.permute(tmp, (0, 1, 4, 3, 2, 5)) + out = tmp.reshape(m_tiles * 128, k_tiles * f // block_size) + return out[0:m, 0:k] + + +def convert_swizzled_8x4_layout_to_linear( + a_sf_swizzled: torch.Tensor, m, k, block_size +): + m_tiles = (m + 8 - 1) // 8 + f = block_size * 4 + k_tiles = (k + f - 1) // f + tmp = torch.reshape(a_sf_swizzled, (1, m_tiles, k_tiles, 8, 4)) + tmp = torch.permute(tmp, (0, 1, 3, 2, 4)) + out = tmp.reshape(m_tiles * 8, k_tiles * f // block_size) + return out[0:m, 0:k] + + +def dequantize_nvfp4_to_dtype( + tensor_fp4, + tensor_sf, + global_scale, + dtype, + device, + block_size=16, + is_sf_128x4_layout=True, +): + """Dequantize the fp4 tensor back to high precision.""" + # Two fp4 values are packed into one uint8. + assert tensor_fp4.dtype == torch.uint8 + m, packed_k = tensor_fp4.shape + k = packed_k * 2 + tensor_f32 = break_fp4_bytes(tensor_fp4, dtype) + tensor_f32 = tensor_f32.reshape(m, k // block_size, block_size) + tensor_sf = tensor_sf.view(torch.float8_e4m3fn) + if is_sf_128x4_layout: + tensor_sf = convert_swizzled_to_linear(tensor_sf, m, k, block_size) + else: + tensor_sf = convert_swizzled_8x4_layout_to_linear(tensor_sf, m, k, block_size) + + tensor_sf_dtype = tensor_sf.to(torch.float32) / global_scale + + # scale the tensor + out = (tensor_f32 * tensor_sf_dtype.unsqueeze(-1)).reshape(m, k) + return out.to(dtype=dtype) + + +def break_fp4_bytes(a, dtype): + assert a.dtype == torch.uint8 + m, n = a.shape + + # Vectorized nibble processing + a_flat = a.flatten() + high = (a_flat & 0xF0) >> 4 # Upper nibbles + low = a_flat & 0x0F # Lower nibbles + + # Combine nibbles for batch processing + combined = torch.stack((low, high), dim=1).flatten() + + # Vectorized sign and magnitude extraction + signs = (combined & 0x08).to(torch.bool) # Sign bits + abs_vals = (combined & 0x07).to(torch.long) # Magnitude indices + + # Device-aware lookup and sign application + kE2M1 = kE2M1ToFloat.to(device=a.device) + values = kE2M1[abs_vals] * torch.where(signs, -1.0, 1.0) + + # Reshape to final form + return values.reshape(m, n * 2).to(dtype=dtype) + + +def dequant_nvfp4_kv_cache( + fp4_data: torch.Tensor, + block_scale: torch.Tensor, + global_scale: float, + head_size: int, + block_size: int, +) -> torch.Tensor: + """Dequantize an NVFP4 KV cache with 4x4-swizzled block scales. + + The input must be in HND layout so that the last two dims are + (block_size, last_dim). For NHD caches, permute to HND first. + + Args: + fp4_data: [..., num_heads, block_size, head_size//2] uint8 packed fp4. + block_scale: [..., num_heads, block_size, head_size//16] fp8 block + scales (as uint8 or float8_e4m3fn). + global_scale: checkpoint dequant scale (k_scale or v_scale). + head_size: head dimension. + block_size: page size. + + Returns: + [..., num_heads, block_size, head_size] float32. + """ + data_dim = head_size // 2 + scale_dim = head_size // 16 + + fp4_packed = fp4_data + sf_swizzled = block_scale.view(torch.uint8) + + # Unswizzle 4x4 block scales on (block_size, scale_dim) plane. + # [..., T, S] → [..., T//4, 4, sg, 4] → permute → [..., T, S] + batch_shape = sf_swizzled.shape[:-2] + T, S = block_size, scale_dim + sg = S // 4 + sf_reshape = sf_swizzled.reshape(*batch_shape, T // 4, 4, sg, 4) + ndim = sf_reshape.ndim + # Swap the last four dims: (..., T//4, 4, sg, 4) → (..., T//4, 4, 4, sg) + perm = list(range(ndim - 4)) + [ndim - 4, ndim - 1, ndim - 3, ndim - 2] + sf_linear = sf_reshape.permute(*perm).reshape(*batch_shape, T, S) + sf_f32 = sf_linear.view(torch.float8_e4m3fn).to(torch.float32) + + # Unpack fp4 + shape = fp4_packed.shape # [..., T, data_dim] + fp4_flat = fp4_packed.reshape(-1, data_dim) + fp4_vals = break_fp4_bytes(fp4_flat, torch.float32) + fp4_vals = fp4_vals.reshape(*shape[:-1], head_size) + + # Dequant: fp4_val * block_scale * global_scale per 16-element group + return ( + fp4_vals.reshape(*shape[:-1], scale_dim, 16) + * (sf_f32 * global_scale).unsqueeze(-1) + ).reshape(*shape[:-1], head_size) + + +def get_nvfp4_global_scale(a: torch.Tensor): + return (FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX) / torch.abs(a).max().to(torch.float32) + + +def quant_nvfp4_tensor(a: torch.Tensor): + a_global_scale = get_nvfp4_global_scale(a) + a_quant, a_block_scale = scaled_fp4_quant(a, a_global_scale) + return a_quant, a_block_scale, a_global_scale diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_allspark_gemm.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_allspark_gemm.py new file mode 100644 index 0000000000000000000000000000000000000000..426713d742cdedbd1cdffb7df26b30f8f49b70bb --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_allspark_gemm.py @@ -0,0 +1,128 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch + +from tests.kernels.utils import DEFAULT_OPCHECK_TEST_UTILS, opcheck +from vllm import _custom_ops as ops +from vllm.model_executor.layers.quantization.utils.allspark_utils import ( + ALLSPARK_AMPERE_K_ALIGN, + ALLSPARK_AMPERE_M_CUBLAS_THRESHOLD, + ALLSPARK_AMPERE_N_ALIGN, +) +from vllm.model_executor.layers.quantization.utils.quant_utils import quantize_weights +from vllm.platforms import current_platform +from vllm.scalar_type import scalar_types +from vllm.utils.platform_utils import num_compute_units + + +def is_gptq_allspark_supported(min_capability: int, max_capability: int) -> bool: + if not current_platform.is_cuda(): + return False + + capability = current_platform.get_device_capability() + assert capability is not None + + return ( + capability.to_int() >= min_capability and capability.to_int() <= max_capability + ) + + +MNK_FACTORS = [ + (1, 4, 8), + (13, 17, 67), + (26, 37, 13), + (48, 16, 24), + (67, 13, 88), + (257, 13, 11), + (658, 13, 11), + (1033, 9, 17), +] + +DTYPES = [torch.float16, torch.bfloat16] +HAS_ZP_OPTS = [False, True] + + +def compute_max_diff(output, output_ref): + return torch.mean(torch.abs(output - output_ref)) / torch.mean( + torch.abs(output_ref) + ) + + +def rand_data(shape, dtype=torch.float16): + return torch.randn(shape, dtype=dtype, device="cuda") + + +@pytest.mark.skipif( + not is_gptq_allspark_supported(80, 89), + reason="AllSpark Ampere kernel is not supported on this GPU type.", +) +@pytest.mark.parametrize("mnk_factors", MNK_FACTORS) +@pytest.mark.parametrize("group_size", [-1]) +@pytest.mark.parametrize("has_zp", HAS_ZP_OPTS) +@pytest.mark.parametrize("dtype", DTYPES) +def test_gptq_allspark_gemm_ampere(mnk_factors, group_size, has_zp, dtype): + m_factor, n_factor, k_factor = mnk_factors + m = m_factor + n = n_factor * ALLSPARK_AMPERE_N_ALIGN + k = k_factor * ALLSPARK_AMPERE_K_ALIGN + + input = rand_data((m, k), dtype=dtype) + weight = rand_data((k, n), dtype=dtype) + + # Quantize (and apply act_order if provided) + w_ref, qw, s, zp = quantize_weights( + weight, scalar_types.uint8b128, group_size, has_zp + ) + + qw = qw.to(torch.uint8) + if has_zp: + zp = zp.to(dtype) + properties = torch.cuda.get_device_properties(qw.device.index) + sm_count = num_compute_units(qw.device.index) + sm_version = properties.major * 10 + properties.minor + + n_32align = (n + 32 - 1) // 32 * 32 + + qw_reorder, s_reorder, zp_reorder = ops.allspark_repack_weight(qw, s, zp, has_zp) + opcheck( + torch.ops._C.rearrange_kn_weight_as_n32k16_order, + (qw, s, zp, has_zp, qw_reorder, s_reorder, zp_reorder, k, n, n_32align), + ) + + opcheck( + torch.ops._C.allspark_w8a16_gemm, + ( + input, + qw_reorder, + s_reorder, + zp_reorder, + n, + group_size, + sm_count, + sm_version, + ALLSPARK_AMPERE_M_CUBLAS_THRESHOLD, + has_zp, + True, + ), + test_utils=DEFAULT_OPCHECK_TEST_UTILS, + ) + output = ops.allspark_w8a16_gemm( + input, + qw_reorder, + s_reorder, + zp_reorder, + n, + group_size, + sm_count, + sm_version, + ALLSPARK_AMPERE_M_CUBLAS_THRESHOLD, + has_zp, + True, + ) + + output_ref = torch.matmul(input, w_ref) + torch.accelerator.synchronize() + max_diff = compute_max_diff(output, output_ref) + + assert max_diff < 0.04 diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_awq.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_awq.py new file mode 100644 index 0000000000000000000000000000000000000000..8becc31f24d1489a9a782d4153f5b2a950ed0135 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_awq.py @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from tests.kernels.utils import opcheck +from vllm import _custom_ops as ops # noqa: F401 + + +@pytest.mark.skipif( + not hasattr(torch.ops._C, "awq_dequantize"), + reason="AWQ is not supported on this GPU type.", +) +def test_awq_dequantize_opcheck(monkeypatch: pytest.MonkeyPatch): + with monkeypatch.context() as m: + m.setenv("VLLM_USE_TRITON_AWQ", "0") + qweight = torch.randint( + -2000000000, 2000000000, (8192, 256), device="cuda", dtype=torch.int32 + ) + scales = torch.rand((64, 2048), device="cuda", dtype=torch.float16) + zeros = torch.empty((64, 256), device="cuda", dtype=torch.int32) + split_k_iters = 0 + thx = 0 + thy = 0 + opcheck( + torch.ops._C.awq_dequantize, + (qweight, scales, zeros, split_k_iters, thx, thy), + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_awq_triton.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_awq_triton.py new file mode 100644 index 0000000000000000000000000000000000000000..f8f60fe029ebbc2fb6bcea25990948ec6ad6a5b4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_awq_triton.py @@ -0,0 +1,177 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the AWQ Triton kernel. + +Run `pytest tests/kernels/quantization/test_awq_triton.py`. +""" + +import pytest +import torch + +from vllm.model_executor.layers.quantization.awq_triton import ( + AWQ_TRITON_SUPPORTED_GROUP_SIZES, + awq_dequantize_triton, + awq_gemm_triton, +) +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +pytestmark = pytest.mark.skipif( + not (current_platform.is_cuda_alike() or current_platform.is_xpu()), + reason="AWQ Triton kernels require CUDA/ROCm or XPU.", +) + +device = current_platform.device_type + + +def reverse_awq_order(t: torch.Tensor): + bits = 4 + AWQ_REVERSE_ORDER = [0, 4, 1, 5, 2, 6, 3, 7] + reverse_order_tensor = torch.arange( + t.shape[-1], + dtype=torch.int32, + device=t.device, + ) + reverse_order_tensor = reverse_order_tensor.view(-1, 32 // bits) + reverse_order_tensor = reverse_order_tensor[:, AWQ_REVERSE_ORDER] + reverse_order_tensor = reverse_order_tensor.view(-1) + + t = t[:, reverse_order_tensor] & 0xF + return t + + +# qweights - [R , C // 8], int32 +# scales - [R // G, C ], float16 +# zeros - [R // G, C // 8], int32 +def awq_dequantize_torch( + qweight: torch.Tensor, scales: torch.Tensor, qzeros: torch.Tensor, group_size: int +) -> torch.Tensor: + if group_size == -1: + group_size = qweight.shape[0] + + bits = 4 + shifts = torch.arange(0, 32, bits, device=qzeros.device) + + iweights = torch.bitwise_right_shift(qweight[:, :, None], shifts[None, None, :]).to( + torch.int8 + ) + + iweights = iweights.view(iweights.shape[0], -1) + + zeros = torch.bitwise_right_shift(qzeros[:, :, None], shifts[None, None, :]).to( + torch.int8 + ) + zeros = zeros.view(qzeros.shape[0], -1) + zeros = reverse_awq_order(zeros) + + iweights = reverse_awq_order(iweights) + + iweights = torch.bitwise_and(iweights, (2**bits) - 1) + zeros = torch.bitwise_and(zeros, (2**bits) - 1) + + scales = scales.repeat_interleave(group_size, dim=0) + zeros = zeros.repeat_interleave(group_size, dim=0) + return (iweights - zeros) * scales + + +# qweights - [R , C // 8], int32 +# scales - [R // G, C ], float16 +# zeros - [R // G, C // 8], int32 +@pytest.mark.parametrize("qweight_rows", [3584, 18944, 128, 256, 512, 1024]) +@pytest.mark.parametrize("qweight_cols", [448, 576, 4736, 16, 32, 64, 128]) +@pytest.mark.parametrize("group_size", AWQ_TRITON_SUPPORTED_GROUP_SIZES) +def test_dequantize(qweight_rows, qweight_cols, group_size): + if group_size == -1: + group_size = qweight_rows + + qweight_dtype = torch.int32 + scales_rows = qweight_rows // group_size + scales_cols = qweight_cols * 8 + scales_dtype = torch.float16 + zeros_rows = scales_rows + zeros_cols = qweight_cols + zeros_dtype = torch.int32 + + set_random_seed(0) + + qweight = torch.randint( + 0, + torch.iinfo(torch.int32).max, + (qweight_rows, qweight_cols), + dtype=qweight_dtype, + device=device, + ) + scales = torch.rand(scales_rows, scales_cols, dtype=scales_dtype, device=device) + zeros = torch.randint( + 0, + torch.iinfo(torch.int32).max, + (zeros_rows, zeros_cols), + dtype=zeros_dtype, + device=device, + ) + + iweights_triton = awq_dequantize_triton(qweight, scales, zeros) + + assert not torch.any(torch.isinf(iweights_triton)) and not torch.any( + torch.isnan(iweights_triton) + ) + + iweights_torch = awq_dequantize_torch(qweight, scales, zeros, group_size) + + torch.testing.assert_close(iweights_triton, iweights_torch) + + +# input - [N, K] +# qweight - [K, M // 8] +# qzeros - [K // G, M // 8] +# scales - [K // G, M] +@pytest.mark.parametrize("N", [1, 2, 4, 8, 14, 17, 23, 32]) +@pytest.mark.parametrize("K", [128]) +@pytest.mark.parametrize("M", [16, 24, 32]) +@pytest.mark.parametrize("group_size", AWQ_TRITON_SUPPORTED_GROUP_SIZES) +@pytest.mark.parametrize("splitK", [1, 8]) +def test_gemm(N, K, M, splitK, group_size): + if group_size == -1: + group_size = K + + split_k_iters = splitK + + input_rows = N + input_cols = K + input_dtype = torch.float32 + qweight_rows = input_cols + qweight_cols = M // 8 + scales_rows = qweight_rows // group_size + scales_cols = M + scales_dtype = torch.float32 + qzeros_rows = scales_rows + qzeros_cols = qweight_cols + + set_random_seed(0) + + input = torch.rand((input_rows, input_cols), dtype=input_dtype, device=device) + qweight = torch.randint( + 0, torch.iinfo(torch.int32).max, (qweight_rows, qweight_cols), device=device + ) + qzeros = torch.randint( + 0, torch.iinfo(torch.int32).max, (qzeros_rows, qzeros_cols), device=device + ) + scales = torch.rand((scales_rows, scales_cols), dtype=scales_dtype, device=device) + + output_triton = awq_gemm_triton(input, qweight, scales, qzeros, split_k_iters) + + assert not torch.any(torch.isinf(output_triton)) and not torch.any( + torch.isnan(output_triton) + ) + + dequantized_weights = awq_dequantize_triton(qweight, scales, qzeros) + + output_torch = torch.matmul(input, dequantized_weights) + + assert not torch.any(torch.isinf(output_torch)) and not torch.any( + torch.isnan(output_torch) + ) + + torch.testing.assert_close( + output_triton.cpu(), output_torch.cpu(), atol=1e-1, rtol=1e-1 + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_block_fp8.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_block_fp8.py new file mode 100644 index 0000000000000000000000000000000000000000..ecef8b65726a08dc22a019676a53023e2c26dc72 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_block_fp8.py @@ -0,0 +1,277 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +# Adapted from https://github.com/sgl-project/sglang/pull/2575 +import itertools + +import pytest +import torch + +from tests.kernels.quant_utils import ( + native_per_token_group_quant_fp8, + native_w8a8_block_matmul, +) +from vllm.config import VllmConfig +from vllm.model_executor.kernels.linear.scaled_mm.cutlass import cutlass_scaled_mm +from vllm.model_executor.layers.quantization.utils.fp8_utils import ( + per_token_group_quant_fp8, + w8a8_triton_block_scaled_mm, +) +from vllm.platforms import current_platform +from vllm.utils.deep_gemm import ( + fp8_gemm_nt, + get_tma_aligned_size, + per_block_cast_to_fp8, + should_use_deepgemm_for_fp8_linear, +) +from vllm.utils.flashinfer import ( + flashinfer_fp8_blockscale_gemm, + has_flashinfer_fp8_blockscale_gemm, +) +from vllm.utils.import_utils import has_deep_gemm + +if current_platform.get_device_capability() < (9, 0): + pytest.skip("FP8 Triton requires CUDA 9.0 or higher", allow_module_level=True) + +vllm_config = VllmConfig() + +# Test configurations +DTYPES = [torch.bfloat16] # [torch.half, torch.bfloat16, torch.float32] +# Quantization test configs +NUM_TOKENS = [7, 2050] +D = [512, 4096, 5120, 13824] +GROUP_SIZE = [64, 128, 512] +COLUMN_MAJOR_SCALES = [True, False] +TMA_ALIGNED_SCALES = [True, False] +# Matmul test configs +M = [1, 7, 8, 83, 4096] +N = [128, 512, 576, 7168, 13824] +K = [256, 3884, 4096, 13824, 16384] +# Deepseek-V3's intermediate size 18432, so N is 18432*2/8=4608 at TP8 +# and its hidden size is 7168. +BLOCK_SIZE = [[128, 128]] +OUT_DTYPES = [torch.bfloat16] # [torch.float32, torch.half, torch.bfloat16] +SEEDS = [0] + +# Skip all tests if CUDA is not available +pytest.importorskip("torch.cuda") + + +@pytest.fixture(autouse=True) +def setup_cuda(): + torch.set_default_device("cuda") + + +@pytest.mark.skipif( + current_platform.is_fp8_fnuz(), + reason="This platform supports e4m3fnuz, not e4m3fn.", +) +@pytest.mark.parametrize( + "num_tokens,d,dtype,group_size,column_major_scales,tma_aligned_scales,seed", + itertools.product( + NUM_TOKENS, + D, + DTYPES, + GROUP_SIZE, + COLUMN_MAJOR_SCALES, + TMA_ALIGNED_SCALES, + SEEDS, + ), +) +@torch.inference_mode() +def test_per_token_group_quant_fp8( + num_tokens, d, dtype, group_size, column_major_scales, tma_aligned_scales, seed +): + torch.manual_seed(seed) + x = torch.rand(num_tokens, d, dtype=dtype) + + ref_out, ref_scale = native_per_token_group_quant_fp8(x, group_size) + out, scale = per_token_group_quant_fp8( + x, + group_size, + column_major_scales=column_major_scales, + tma_aligned_scales=tma_aligned_scales, + ) + + assert torch.allclose(out.to(torch.float32), ref_out.to(torch.float32), rtol=0.15) + assert torch.allclose(scale, ref_scale) + + if column_major_scales: + assert scale.stride()[-2] == 1 + if tma_aligned_scales: + assert scale.stride()[-1] == get_tma_aligned_size(num_tokens, 4) + + +@pytest.mark.parametrize( + "M,N,K,block_size,out_dtype,seed", + itertools.product(M, N, K, BLOCK_SIZE, OUT_DTYPES, SEEDS), +) +@torch.inference_mode() +def test_w8a8_block_fp8_matmul(M, N, K, block_size, out_dtype, seed): + torch.manual_seed(seed) + factor_for_scale = 1e-2 + fp8_info = torch.finfo(current_platform.fp8_dtype()) + fp8_max, fp8_min = fp8_info.max, fp8_info.min + + A_fp32 = (torch.rand(M, K, dtype=torch.float32) - 0.5) * 2 * fp8_max + A_fp8 = A_fp32.clamp(min=fp8_min, max=fp8_max).to(current_platform.fp8_dtype()) + + B_fp32 = (torch.rand(N, K, dtype=torch.float32) - 0.5) * 2 * fp8_max + B_fp8 = B_fp32.clamp(min=fp8_min, max=fp8_max).to(current_platform.fp8_dtype()) + + block_n, block_k = block_size[0], block_size[1] + n_tiles = (N + block_n - 1) // block_n + k_tiles = (K + block_k - 1) // block_k + + As = torch.rand(M, k_tiles, dtype=torch.float32) * factor_for_scale + Bs = torch.rand(n_tiles, k_tiles, dtype=torch.float32) * factor_for_scale + + ref_out = native_w8a8_block_matmul(A_fp8, B_fp8, As, Bs, block_size, out_dtype) + out = w8a8_triton_block_scaled_mm(A_fp8, B_fp8, As, Bs, block_size, out_dtype) + + rel_diff = torch.mean( + torch.abs(out.to(torch.float32) - ref_out.to(torch.float32)) + ) / torch.mean(torch.abs(ref_out.to(torch.float32))) + assert rel_diff < 0.001 + + +@pytest.mark.skipif( + not current_platform.is_cuda(), reason="CUTLASS only supported on CUDA platform." +) +@torch.inference_mode() +def test_w8a8_block_fp8_cutlass_matmul(): + # Test simple case where weight.shape % 128 != 0, + # like in DSV3 kv_a_proj_with_mqa + M = 32 + N = 576 + K = 7168 + block_size = [128, 128] + out_dtype = torch.bfloat16 + seed = 0 + + torch.manual_seed(seed) + factor_for_scale = 1e-2 + fp8_info = torch.finfo(torch.float8_e4m3fn) + fp8_max, fp8_min = fp8_info.max, fp8_info.min + + A_fp32 = (torch.rand(M, K, dtype=torch.float32) - 0.5) * 2 * fp8_max + + B_fp32 = (torch.rand(N, K, dtype=torch.float32) - 0.5) * 2 * fp8_max + B_fp8 = B_fp32.clamp(min=fp8_min, max=fp8_max).to(torch.float8_e4m3fn) + + block_n, block_k = block_size[0], block_size[1] + n_tiles = (N + block_n - 1) // block_n + k_tiles = (K + block_k - 1) // block_k + + Bs = torch.rand(n_tiles, k_tiles, dtype=torch.float32) * factor_for_scale + + A_fp8, As = per_token_group_quant_fp8( + A_fp32, block_size[1], column_major_scales=False + ) + # CUTLASS uses column-major format for scales + A_fp8_cutlass, As_cutlass = per_token_group_quant_fp8( + A_fp32, block_size[1], column_major_scales=True + ) + + ref_out = native_w8a8_block_matmul(A_fp8, B_fp8, As, Bs, block_size, out_dtype) + out = cutlass_scaled_mm(A_fp8_cutlass, B_fp8, As_cutlass, Bs, block_size, out_dtype) + + rel_diff = torch.mean( + torch.abs(out.to(torch.float32) - ref_out.to(torch.float32)) + ) / torch.mean(torch.abs(ref_out.to(torch.float32))) + assert rel_diff < 0.001 + + +@pytest.mark.skipif( + current_platform.is_fp8_fnuz(), + reason="This platform supports e4m3fnuz, not e4m3fn.", +) +@pytest.mark.parametrize( + "M,N,K,block_size,out_dtype,seed", + itertools.product(M, N, K, BLOCK_SIZE, OUT_DTYPES, SEEDS), +) +@pytest.mark.skipif(not has_deep_gemm(), reason="DeepGemm kernels not available.") +@torch.inference_mode() +def test_w8a8_block_fp8_deep_gemm_matmul(M, N, K, block_size, out_dtype, seed): + torch.manual_seed(seed) + fp8_info = torch.finfo(torch.float8_e4m3fn) + fp8_max = fp8_info.max + + A_fp32 = (torch.rand(M, K, dtype=torch.float32) - 0.5) * 2 * fp8_max + B_fp32 = (torch.rand(N, K, dtype=torch.float32) - 0.5) * 2 * fp8_max + + # only aligned sizes are supported by deepgemm + if not should_use_deepgemm_for_fp8_linear( + output_dtype=out_dtype, weight_shape=B_fp32.shape, supports_deep_gemm=True + ): + pytest.skip(f"Skipping test; invalid size {M}, {N}, {K}") + + A_fp8, As_fp8 = per_token_group_quant_fp8( + A_fp32, block_size[1], column_major_scales=True, tma_aligned_scales=True + ) + B_fp8, Bs_fp8 = per_block_cast_to_fp8(B_fp32, block_size=block_size) + + As = As_fp8.to(torch.float32) + Bs = Bs_fp8.to(torch.float32) + + ref_out = native_w8a8_block_matmul(A_fp8, B_fp8, As, Bs, block_size, out_dtype) + + out = torch.zeros((M, N), device="cuda", dtype=out_dtype) + + assert As_fp8.shape == (M, (K + 127) // 128), ( + f"{As_fp8.shape} != {(M, (K + 127) // 128)}" + ) + + fp8_gemm_nt((A_fp8, As_fp8), (B_fp8, Bs_fp8), out) + + rel_diff = torch.mean( + torch.abs(out.to(torch.float32) - ref_out.to(torch.float32)) + ) / torch.mean(torch.abs(ref_out.to(torch.float32))) + assert rel_diff < 0.001 + + +@pytest.mark.skipif( + current_platform.is_fp8_fnuz(), + reason="This platform supports e4m3fnuz, not e4m3fn.", +) +@pytest.mark.parametrize( + "M,N,K,block_size,out_dtype,seed", + itertools.product(M, N, K, BLOCK_SIZE, OUT_DTYPES, SEEDS), +) +@torch.inference_mode() +def test_w8a8_block_fp8_flashinfer_matmul(M, N, K, block_size, out_dtype, seed): + if not has_flashinfer_fp8_blockscale_gemm(): + pytest.skip( + "FlashInfer block GEMM not available (requires SM90+ and FlashInfer)" + ) + # only aligned sizes + if K % 128 != 0 or N % 64 != 0: + pytest.skip(f"Skipping test; invalid size {M}, {N}, {K}") + + torch.manual_seed(seed) + fp8_info = torch.finfo(torch.float8_e4m3fn) + fp8_max = fp8_info.max + + A_bf16 = (torch.rand(M, K, dtype=torch.bfloat16) - 0.5) * 2 * fp8_max + B_bf16 = (torch.rand(N, K, dtype=torch.bfloat16) - 0.5) * 2 * fp8_max + + A_fp8, As_fp8 = per_token_group_quant_fp8(A_bf16, block_size[1], use_ue8m0=False) + B_fp8, Bs_fp8 = per_block_cast_to_fp8(B_bf16, block_size, use_ue8m0=False) + + As = As_fp8.to(torch.float32) + Bs = Bs_fp8.to(torch.float32) + + ref_out = native_w8a8_block_matmul(A_fp8, B_fp8, As, Bs, block_size, out_dtype) + + out = flashinfer_fp8_blockscale_gemm( + input=A_bf16, + weight=B_fp8, + input_scale=None, + weight_scale=Bs, + out_dtype=out_dtype, + ) + + rel_diff = torch.mean( + torch.abs(out.to(torch.bfloat16) - ref_out.to(torch.bfloat16)) + ) / torch.mean(torch.abs(ref_out.to(torch.bfloat16))) + assert rel_diff < 0.001 diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_block_int8.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_block_int8.py new file mode 100644 index 0000000000000000000000000000000000000000..9f990ab354422fd6be737f359436043185ce67de --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_block_int8.py @@ -0,0 +1,67 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +# Adapted from https://github.com/sgl-project/sglang/blob/main/test/srt/test_block_int8.py +import itertools + +import pytest +import torch + +from tests.kernels.quant_utils import native_w8a8_block_matmul +from vllm.config import VllmConfig +from vllm.model_executor.layers.quantization.utils.int8_utils import ( + w8a8_block_int8_matmul, +) +from vllm.platforms import current_platform + +if current_platform.get_device_capability() < (7, 0): + pytest.skip("INT8 Triton requires CUDA 7.0 or higher", allow_module_level=True) + +vllm_config = VllmConfig() + +DTYPES = [torch.half, torch.bfloat16] +M = [1, 33, 64, 222] +N = [128, 1024] +K = [256, 4096] +# BLOCK_SIZE = [[64, 64], [64, 128], [128, 64], [128, 128]] +BLOCK_SIZE = [[128, 128]] +SEEDS = [0] + + +@pytest.fixture(autouse=True, scope="module") +def setup_cuda(): + """Sets the default CUDA device for all tests in this module.""" + torch.set_default_device("cuda") + + +@pytest.mark.parametrize( + "M,N,K,block_size,out_dtype,seed", + itertools.product(M, N, K, BLOCK_SIZE, DTYPES, SEEDS), +) +@torch.inference_mode() +def test_w8a8_block_int8_matmul(M, N, K, block_size, out_dtype, seed): + torch.manual_seed(seed) + factor_for_scale = 1e-2 + int8_info = torch.iinfo(torch.int8) + int8_max, int8_min = int8_info.max, int8_info.min + + A_fp32 = (torch.rand(M, K, dtype=torch.float32) - 0.5) * 2 * int8_max + A_fp8 = A_fp32.clamp(min=int8_min, max=int8_max).to(torch.float8_e4m3fn) + + B_fp32 = (torch.rand(N, K, dtype=torch.float32) - 0.5) * 2 * int8_max + B_fp8 = B_fp32.clamp(min=int8_min, max=int8_max).to(torch.float8_e4m3fn) + + block_n, block_k = block_size[0], block_size[1] + n_tiles = (N + block_n - 1) // block_n + k_tiles = (K + block_k - 1) // block_k + + As = torch.rand(M, k_tiles, dtype=torch.float32) * factor_for_scale + Bs = torch.rand(n_tiles, k_tiles, dtype=torch.float32) * factor_for_scale + + ref_out = native_w8a8_block_matmul(A_fp8, B_fp8, As, Bs, block_size, out_dtype) + out = w8a8_block_int8_matmul(A_fp8, B_fp8, As, Bs, block_size, out_dtype) + + rel_diff = torch.mean( + torch.abs(out.to(torch.float32) - ref_out.to(torch.float32)) + ) / torch.mean(torch.abs(ref_out.to(torch.float32))) + assert rel_diff < 0.001 diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_cpu_fp8_scaled_mm.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_cpu_fp8_scaled_mm.py new file mode 100644 index 0000000000000000000000000000000000000000..14ae5e7bcdd31de419e0c7fedf623ab7fa5715da --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_cpu_fp8_scaled_mm.py @@ -0,0 +1,162 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for CPU FP8 W8A16 block-scaled GEMM kernel (fp8_scaled_mm_cpu). + +Run `pytest tests/kernels/quantization/test_cpu_fp8_scaled_mm.py -v`. +""" + +import pytest +import torch + +from vllm import _custom_ops as ops +from vllm.platforms import current_platform + +if not current_platform.is_cpu(): + pytest.skip("skipping CPU-only tests", allow_module_level=True) + +if not ops._supports_cpu_fp8_w8a16: + pytest.skip("fp8_scaled_mm_cpu op not available", allow_module_level=True) + +BLOCK_SIZE = [128, 128] + + +def cdiv(a: int, b: int) -> int: + return -(a // -b) + + +def quantize_weight_block_fp8( + weight: torch.Tensor, + block_size: list[int], +) -> tuple[torch.Tensor, torch.Tensor]: + """Quantize weight [N, K] to FP8 with block scales. + + Returns: + fp8_weight: [N, K] float8_e4m3fn + scales: [n_tiles, k_tiles] float32 + """ + N, K = weight.shape + block_n, block_k = block_size + fp8_max = torch.finfo(torch.float8_e4m3fn).max + + n_tiles = cdiv(N, block_n) + k_tiles = cdiv(K, block_k) + + # Pad for even blocking + pad_N = (block_n - (N % block_n)) % block_n + pad_K = (block_k - (K % block_k)) % block_k + if pad_N > 0 or pad_K > 0: + weight = torch.nn.functional.pad(weight, (0, pad_K, 0, pad_N)) + + # Reshape into blocks + w_blocks = weight.view(n_tiles, block_n, k_tiles, block_k) + w_blocks = w_blocks.permute(0, 2, 1, 3).contiguous() + + # Per-block scale + abs_max = w_blocks.abs().amax(dim=(-2, -1), keepdim=True) + scales = abs_max / fp8_max + scales = torch.where(scales == 0, torch.ones_like(scales), scales) + + # Quantize + q_fp8 = (w_blocks / scales).clamp(-fp8_max, fp8_max).to(torch.float8_e4m3fn) + + # Reshape back + fp8_weight = ( + q_fp8.permute(0, 2, 1, 3) + .contiguous() + .view(N + pad_N, K + pad_K)[:N, :K] + .contiguous() + ) + + scales = scales.view(n_tiles, k_tiles) + return fp8_weight, scales + + +def dequant_weight_block_fp8( + fp8_weight: torch.Tensor, + scales: torch.Tensor, + block_size: list[int], + out_dtype: torch.dtype, +) -> torch.Tensor: + """Dequantize FP8 weight back to float for reference computation.""" + N, K = fp8_weight.shape + block_n, block_k = block_size + n_tiles, k_tiles = scales.shape + + pad_N = (block_n - (N % block_n)) % block_n + pad_K = (block_k - (K % block_k)) % block_k + if pad_N > 0 or pad_K > 0: + fp8_padded = torch.nn.functional.pad(fp8_weight.float(), (0, pad_K, 0, pad_N)) + else: + fp8_padded = fp8_weight.float() + + w_blocks = fp8_padded.view(n_tiles, block_n, k_tiles, block_k) + w_blocks = w_blocks.permute(0, 2, 1, 3).contiguous() + dq = w_blocks * scales.view(n_tiles, k_tiles, 1, 1) + dq = dq.permute(0, 2, 1, 3).contiguous().view(N + pad_N, K + pad_K) + return dq[:N, :K].to(out_dtype) + + +def ref_fp8_block_scaled_mm( + x: torch.Tensor, + fp8_weight: torch.Tensor, + scales: torch.Tensor, + block_size: list[int], + bias: torch.Tensor | None, + out_dtype: torch.dtype, +) -> torch.Tensor: + """Reference: dequant FP8→float32, matmul in float32, cast to out_dtype.""" + w_dq = dequant_weight_block_fp8(fp8_weight, scales, block_size, torch.float32) + out = torch.mm(x.float(), w_dq.t()) + if bias is not None: + out = out + bias.float() + return out.to(out_dtype) + + +# --------------------------------------------------------------------------- +# Test parameters +# --------------------------------------------------------------------------- +M_SIZES = [1, 4, 16, 64, 128] +# (N, K) — weight shape is [N, K], output has N columns. +NK_SIZES = [ + (128, 256), + (256, 512), + (512, 1024), + (1024, 2048), + (5120, 5120), + (17408, 5120), + (5120, 17408), +] + + +@pytest.mark.parametrize("M", M_SIZES) +@pytest.mark.parametrize("N,K", NK_SIZES) +@pytest.mark.parametrize("use_bias", [False, True]) +def test_cpu_fp8_scaled_mm(M: int, N: int, K: int, use_bias: bool): + """fp8_scaled_mm_cpu correctness against float reference.""" + torch.manual_seed(42) + out_dtype = torch.bfloat16 + block_size = BLOCK_SIZE + + x = torch.randn(M, K, dtype=out_dtype) / (K**0.5) + w_f32 = torch.randn(N, K, dtype=torch.float32) / (K**0.5) + fp8_weight, scales = quantize_weight_block_fp8(w_f32, block_size) + + bias = torch.randn(N, dtype=torch.float32) * 0.1 if use_bias else None + + ref_out = ref_fp8_block_scaled_mm( + x, fp8_weight, scales, block_size, bias, out_dtype + ) + + packed_weight = torch.ops._C.convert_weight_packed(fp8_weight) + kernel_out = ops.fp8_scaled_mm_cpu( + x, + packed_weight, + scales, + block_size, + bias, + out_dtype, + True, + ) + + assert kernel_out.dtype == out_dtype + torch.testing.assert_close(kernel_out, ref_out, rtol=0.02, atol=0.01) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_cutlass_scaled_mm.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_cutlass_scaled_mm.py new file mode 100644 index 0000000000000000000000000000000000000000..f528bedf2c98f0c786d2a6887f310773f80185bf --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_cutlass_scaled_mm.py @@ -0,0 +1,759 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for cutlass kernels + +Run `pytest tests/kernels/quantization/test_cutlass_scaled_mm.py`. +""" + +import random + +import pytest +import torch + +from tests.kernels.utils import baseline_scaled_mm, opcheck, to_fp8, to_int8 +from vllm import _custom_ops as ops +from vllm.platforms import current_platform +from vllm.utils.math_utils import cdiv + +if not current_platform.is_cuda(): + pytest.skip("These tests use CUTLASS which requires CUDA", allow_module_level=True) + +MNK_FACTORS = [ + (1, 256, 128), + (1, 16384, 1024), + (1, 24576, 496), + (16, 256, 496), + (16, 16384, 128), + (16, 24576, 4096), + (32, 8192, 4096), + (32, 16384, 4096), + (33, 1024, 1024), + (33, 8192, 128), + (64, 2048, 496), + (64, 16384, 1024), + (100, 8192, 496), + (128, 32768, 4096), + (256, 4096, 4096), + (512, 256, 1024), + (512, 8192, 4096), + (512, 16384, 128), + (512, 24576, 128), +] + +# Shapes with N or K not divisible by 16. These exercise the padding path +# inside CutlassFP8ScaledMMLinearKernel.apply_scaled_mm (e.g. Qwen2.5-VL +# vision MLP dims). +UNALIGNED_MNK_FACTORS = [ + (32, 3420, 1280), + (32, 1280, 6840), + (1, 3420, 1280), + (64, 6840, 1280), + (16, 100, 200), + (33, 255, 513), +] + +CUDA_DEVICES = [ + f"cuda:{i}" for i in range(1 if torch.accelerator.device_count() == 1 else 2) +] + +# -1 means full extent in that dimension +TENSORWISE_GROUP_SHAPE = (-1, -1) +PER_TOKEN_GROUP_SHAPE = (1, -1) +PER_OUT_CH_GROUP_SHAPE = (-1, 1) + +capability = current_platform.get_device_capability() +capability = capability[0] * 10 + capability[1] + + +def rand_int8(shape: tuple, device: str = "cuda"): + return to_int8(torch.rand(shape, device=device) * 255 - 128) + + +def group_scale_helper(shape, group_shape): + return [shape[i] if s < 0 else s for i, s in enumerate(group_shape)] + + +def scale_shape(shape, group_shape): + assert len(shape) == len(group_shape) + group_shape = group_scale_helper(shape, group_shape) + return tuple(cdiv(shape[i], group_shape[i]) for i in range(len(group_shape))) + + +def cutlass_fp8_gemm_helper( + m: int, + n: int, + k: int, + a_scale_group_shape: tuple, + b_scale_group_shape: tuple, + use_bias: bool, + out_dtype: type[torch.dtype] = torch.bfloat16, + device: str = "cuda", +): + # Test for a cutlass kernel with per-token activation quantization + # and per-output channel weight quantization. + a = to_fp8(torch.randn((m, k), device=device)) + b = to_fp8(torch.randn((n, k), device=device).t()) + + a_scales_shape = scale_shape(a.shape, a_scale_group_shape) + b_scales_shape = scale_shape(b.shape, b_scale_group_shape) + + scale_a = torch.randn(a_scales_shape, device=device, dtype=torch.float32) + scale_b = torch.randn(b_scales_shape, device=device, dtype=torch.float32) + + # make scales M-major for blockwise quant, doesn't affect 1D scales + scale_a = scale_a.t().contiguous().t() + # make scales K-major for blockwise quant, doesn't affect 1D scales + scale_b = scale_b.t().contiguous().t() + + bias = torch.rand((n,), device=device, dtype=out_dtype) * 10 if use_bias else None + + out = ops.cutlass_scaled_mm(a, b, scale_a, scale_b, out_dtype, bias) + baseline = baseline_scaled_mm(a, b, scale_a, scale_b, out_dtype, bias) + + torch.testing.assert_close(out, baseline, rtol=5e-1, atol=1.5e-1) + + opcheck(torch.ops._C.cutlass_scaled_mm, (out, a, b, scale_a, scale_b, bias)) + + +def cutlass_int8_gemm_helper( + m: int, + n: int, + k: int, + a_scale_group_shape: tuple, + b_scale_group_shape: tuple, + use_bias: bool, + out_dtype: type[torch.dtype] = torch.bfloat16, + device: str = "cuda", +): + # Test for a cutlass kernel with per-token activation quantization + # and per-output channel weight quantization. + a = to_int8(torch.randn((m, k), device=device) * 5) + b = to_int8(torch.randn((n, k), device=device).t() * 5) + + a_scales_shape = scale_shape(a.shape, a_scale_group_shape) + b_scales_shape = scale_shape(b.shape, b_scale_group_shape) + + scale_a = torch.randn(a_scales_shape, device=device, dtype=torch.float32) + scale_b = torch.randn(b_scales_shape, device=device, dtype=torch.float32) + + bias = torch.rand((n,), device=device, dtype=out_dtype) * 10 if use_bias else None + + out = ops.cutlass_scaled_mm(a, b, scale_a, scale_b, out_dtype, bias) + baseline = baseline_scaled_mm(a, b, scale_a, scale_b, out_dtype, bias) + + torch.testing.assert_close(out, baseline, rtol=1e-1, atol=1e0) + + opcheck(torch.ops._C.cutlass_scaled_mm, (out, a, b, scale_a, scale_b, bias)) + + +@pytest.mark.parametrize("m,n,k", MNK_FACTORS) +@pytest.mark.parametrize( + "a_scale_group_shape", [PER_TOKEN_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize( + "b_scale_group_shape", [PER_OUT_CH_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize("use_bias", [True, False]) +@pytest.mark.skipif( + not current_platform.has_device_capability(89), + reason="FP8 is not supported on this GPU type.", +) +def test_cutlass_fp8_gemm( + m: int, n: int, k: int, a_scale_group_shape, b_scale_group_shape, use_bias: bool +): + cutlass_fp8_gemm_helper(m, n, k, a_scale_group_shape, b_scale_group_shape, use_bias) + + +@pytest.mark.parametrize("m,n,k", UNALIGNED_MNK_FACTORS) +@pytest.mark.parametrize( + "a_scale_group_shape", [PER_TOKEN_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize( + "b_scale_group_shape", [PER_OUT_CH_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize("use_bias", [True, False]) +@pytest.mark.skipif( + not current_platform.has_device_capability(89), + reason="FP8 is not supported on this GPU type.", +) +def test_cutlass_fp8_gemm_padded( + m: int, n: int, k: int, a_scale_group_shape, b_scale_group_shape, use_bias: bool +): + """Test CUTLASS FP8 GEMM with padding for non-16-aligned N/K dims. + + Exercises CutlassFP8ScaledMMLinearKernel.apply_scaled_mm which pads + inputs to satisfy CUTLASS alignment requirements — the path taken by + models like Qwen2.5-VL whose vision MLP has non-16-aligned dims. + """ + from vllm.model_executor.kernels.linear.scaled_mm.cutlass import ( + CutlassFP8ScaledMMLinearKernel, + ) + + a = to_fp8(torch.randn((m, k), device="cuda")) + b = to_fp8(torch.randn((n, k), device="cuda").t()) + + a_scales_shape = scale_shape(a.shape, a_scale_group_shape) + b_scales_shape = scale_shape(b.shape, b_scale_group_shape) + + scale_a = torch.randn(a_scales_shape, device="cuda", dtype=torch.float32) + scale_b = torch.randn(b_scales_shape, device="cuda", dtype=torch.float32) + + scale_a = scale_a.t().contiguous().t() + scale_b = scale_b.t().contiguous().t() + + out_dtype = torch.bfloat16 + bias = torch.rand((n,), device="cuda", dtype=out_dtype) * 10 if use_bias else None + + baseline = baseline_scaled_mm(a, b, scale_a, scale_b, out_dtype, bias) + + # process_weights_after_loading pad b to 16 + pad_k = (16 - k % 16) % 16 + pad_n = (16 - n % 16) % 16 + if pad_k > 0 or pad_n > 0: + b = torch.nn.functional.pad(b.t().contiguous(), (0, pad_k, 0, pad_n)).t() + if pad_n > 0 and scale_b.numel() > 1: + scale_b = torch.nn.functional.pad(scale_b, (0, pad_n), value=1.0) + + kernel = object.__new__(CutlassFP8ScaledMMLinearKernel) + kernel.logical_output_size = n + out = kernel.apply_scaled_mm( + A=a, + B=b, + out_dtype=out_dtype, + As=scale_a, + Bs=scale_b, + bias=bias, + output_shape=[m, n], + ) + + torch.testing.assert_close(out, baseline, rtol=5e-1, atol=1.5e-1) + + +@pytest.mark.parametrize("m,n,k", MNK_FACTORS) +@pytest.mark.parametrize( + "a_scale_group_shape,b_scale_group_shape", [((1, 128), (128, 128))] +) +@pytest.mark.parametrize("use_bias", [False]) +@pytest.mark.skipif( + not current_platform.has_device_capability(90), + reason="FP8 blockwise is not supported on this GPU type.", +) +def test_cutlass_fp8_blockwise_scale_gemm( + m: int, n: int, k: int, a_scale_group_shape, b_scale_group_shape, use_bias: bool +): + if k % b_scale_group_shape[0] != 0 or n % b_scale_group_shape[1] != 0: + return + if m % a_scale_group_shape[0] != 0 or k % a_scale_group_shape[1] != 0: + return + cutlass_fp8_gemm_helper(m, n, k, a_scale_group_shape, b_scale_group_shape, use_bias) + + +@pytest.mark.parametrize("m,n,k", MNK_FACTORS) +@pytest.mark.parametrize( + "a_scale_group_shape", [PER_TOKEN_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize( + "b_scale_group_shape", [PER_OUT_CH_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize("use_bias", [True, False]) +def test_cutlass_int8_gemm( + m: int, n: int, k: int, a_scale_group_shape, b_scale_group_shape, use_bias: bool +): + cutlass_int8_gemm_helper( + m, n, k, a_scale_group_shape, b_scale_group_shape, use_bias + ) + + +@pytest.mark.parametrize( + "a_scale_group_shape", [PER_TOKEN_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize( + "b_scale_group_shape", [PER_OUT_CH_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize("out_dtype", [torch.bfloat16, torch.float16]) +@pytest.mark.parametrize("use_bias", [True, False]) +def test_cutlass_int8_gemm_output_dtype( + a_scale_group_shape, + b_scale_group_shape, + out_dtype: type[torch.dtype], + use_bias: bool, +): + cutlass_int8_gemm_helper( + 512, + 512, + 512, + a_scale_group_shape, + b_scale_group_shape, + use_bias, + out_dtype=out_dtype, + ) + + +@pytest.mark.parametrize( + "a_scale_group_shape", [PER_TOKEN_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize( + "b_scale_group_shape", [PER_OUT_CH_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize("out_dtype", [torch.bfloat16, torch.float16]) +@pytest.mark.parametrize("use_bias", [True, False]) +@pytest.mark.skipif( + not current_platform.has_device_capability(89), + reason="FP8 is not supported on this GPU type.", +) +def test_cutlass_fp8_gemm_output_dtype( + a_scale_group_shape, + b_scale_group_shape, + out_dtype: type[torch.dtype], + use_bias: bool, +): + cutlass_fp8_gemm_helper( + 512, + 512, + 512, + a_scale_group_shape, + b_scale_group_shape, + use_bias, + out_dtype=out_dtype, + ) + + +@pytest.mark.parametrize( + "a_scale_group_shape,b_scale_group_shape", [((1, 128), (128, 128))] +) +@pytest.mark.parametrize("out_dtype", [torch.bfloat16, torch.float16]) +@pytest.mark.parametrize("use_bias", [False]) +@pytest.mark.skipif( + not current_platform.has_device_capability(90), + reason="FP8 blockwise is not supported on this GPU type.", +) +def test_cutlass_fp8_blockwise_scale_gemm_dtype( + a_scale_group_shape, + b_scale_group_shape, + out_dtype: type[torch.dtype], + use_bias: bool, +): + cutlass_fp8_gemm_helper( + 512, + 512, + 512, + a_scale_group_shape, + b_scale_group_shape, + use_bias, + out_dtype=out_dtype, + ) + + +@pytest.mark.parametrize( + "a_scale_group_shape", [PER_TOKEN_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize( + "b_scale_group_shape", [PER_OUT_CH_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize("use_bias", [True, False]) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.skipif( + not current_platform.has_device_capability(89), + reason="FP8 is not supported on this GPU type.", +) +def test_cutlass_fp8_gemm_devices( + a_scale_group_shape, b_scale_group_shape, use_bias: bool, device: str +): + cutlass_fp8_gemm_helper( + 512, + 512, + 512, + a_scale_group_shape, + b_scale_group_shape, + use_bias, + torch.bfloat16, + device, + ) + + +@pytest.mark.parametrize( + "a_scale_group_shape", [PER_TOKEN_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize( + "b_scale_group_shape", [PER_OUT_CH_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize("use_bias", [True, False]) +@pytest.mark.parametrize("device", CUDA_DEVICES) +def test_cutlass_int8_gemm_devices( + a_scale_group_shape, b_scale_group_shape, use_bias: bool, device: str +): + cutlass_int8_gemm_helper( + 512, + 512, + 512, + a_scale_group_shape, + b_scale_group_shape, + use_bias, + out_dtype=torch.bfloat16, + device=device, + ) + + +# For the following two tests: +# N and K correspond to the size of the weight matrix and likely to be multiples +# of a large power of two. In any case, the kernel will have a naive fallback +# when N and K are not divisible by 16. But M is the number of tokens and the +# kernel must handle any M thrown at it. +@pytest.mark.parametrize( + "a_scale_group_shape", [PER_TOKEN_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize( + "b_scale_group_shape", [PER_OUT_CH_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize("use_bias", [True, False]) +@pytest.mark.skipif( + not current_platform.has_device_capability(89), + reason="FP8 is not supported on this GPU type.", +) +def test_cutlass_fp8_gemm_m_sweep( + a_scale_group_shape, b_scale_group_shape, use_bias: bool +): + for nk in range(32, 128, 32): + for m in range(1, 128): + cutlass_fp8_gemm_helper( + m, nk, nk, a_scale_group_shape, b_scale_group_shape, use_bias + ) + + +@pytest.mark.parametrize( + "a_scale_group_shape", [PER_TOKEN_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize( + "b_scale_group_shape", [PER_OUT_CH_GROUP_SHAPE, TENSORWISE_GROUP_SHAPE] +) +@pytest.mark.parametrize("use_bias", [True, False]) +def test_cutlass_int8_gemm_m_sweep( + a_scale_group_shape, b_scale_group_shape, use_bias: bool +): + for nk in range(32, 128, 32): + for m in range(1, 128): + cutlass_int8_gemm_helper( + m, nk, nk, a_scale_group_shape, b_scale_group_shape, use_bias + ) + + +@pytest.mark.parametrize("m", [32, 64, 128]) +@pytest.mark.parametrize("n", [16, 32, 64]) +@pytest.mark.parametrize("k", [64, 128, 256]) +@pytest.mark.parametrize("out_dtype", [torch.bfloat16, torch.float16]) +@pytest.mark.skip +def test_cutlass_int8_azp_bias_fold(m: int, n: int, k: int, out_dtype: torch.dtype): + # Currently, the test is failing because folding azp into + # 16-bit bias loses too much precision + scale_a = torch.randn((1, 1), device="cuda", dtype=torch.float32) / 10 + scale_b = torch.randn((1, n), device="cuda", dtype=torch.float32) / 10 + + aq_i8 = rand_int8((m, k)) + bq_i8 = rand_int8((n, k)).t() + + aq_i32 = aq_i8.to(dtype=torch.int32) + bq_i32 = bq_i8.to(dtype=torch.int32) + + aq_f32 = aq_i8.to(dtype=torch.float32) + bq_f32 = bq_i8.to(dtype=torch.float32) + + b_dq = scale_b * bq_f32 + + azp_a = torch.rand((1,), device="cuda", dtype=torch.float32) * 10 + 1.5 + azp_aq_i8 = (azp_a / scale_a).to(dtype=torch.int8) + azp_a = azp_aq_i8.to(dtype=torch.float32) * scale_a # correct for rounding + + a_dq = scale_a * (aq_i32 + azp_aq_i8).to(dtype=torch.float32) + torch.testing.assert_close(a_dq, scale_a * aq_f32 + azp_a) + + baseline_dq = torch.mm(a_dq, b_dq).to(out_dtype) + + J = torch.ones((1, k), device="cuda", dtype=torch.float32) + azp_bias = (azp_a * scale_b * (J @ bq_f32)).to(out_dtype) + assert azp_bias.shape == (1, n) + assert azp_bias[0, :].shape == (n,) + + baseline_q = ( + scale_a.to(device="cpu") + * scale_b.to(device="cpu") + * ((aq_i32 + azp_aq_i8).to(device="cpu") @ bq_i32.to(device="cpu")) + ).to(dtype=out_dtype, device="cuda") + + out = ops.cutlass_scaled_mm( + aq_i8, bq_i8, scale_a, scale_b, out_dtype=out_dtype, bias=azp_bias[0, :] + ) + torch.testing.assert_close(out, baseline_dq, rtol=1e-2, atol=1e0) + torch.testing.assert_close(out, baseline_q, rtol=1e-2, atol=1e0) + + +@pytest.mark.parametrize("m", [32, 64, 128]) +@pytest.mark.parametrize("n", [16, 32, 64]) +@pytest.mark.parametrize("k", [64, 128, 256]) +@pytest.mark.parametrize("out_dtype", [torch.bfloat16, torch.float16]) +@pytest.mark.parametrize("use_bias", [True, False]) +@pytest.mark.parametrize("azp_per_token", [True, False]) +def test_cutlass_int8_azp( + m: int, n: int, k: int, out_dtype: torch.dtype, use_bias: bool, azp_per_token: bool +): + m_azp = m if azp_per_token else 1 + scale_a = torch.randn((m_azp, 1), device="cuda", dtype=torch.float32) / 10 + scale_b = torch.randn((1, n), device="cuda", dtype=torch.float32) / 10 + + aq_i8 = rand_int8((m, k)) + aq_i32 = aq_i8.to(dtype=torch.int32) + aq_f32 = aq_i8.to(dtype=torch.float32) + + bq_i8 = rand_int8((n, k)).t() + bq_i32 = bq_i8.to(dtype=torch.int32) + bq_f32 = bq_i8.to(dtype=torch.float32) + b_dq = scale_b * bq_f32 + + azp_a = torch.rand((m_azp, 1), device="cuda", dtype=torch.float32) * 10 + 1.5 + azp_aq_i8 = (azp_a / scale_a).to(dtype=torch.int8) + azp_a = azp_aq_i8.to(dtype=torch.float32) * scale_a # correct for rounding + + a_dq = scale_a * (aq_i32 - azp_aq_i8).to(dtype=torch.float32) + torch.testing.assert_close(a_dq, scale_a * aq_f32 - azp_a, rtol=1e-4, atol=1e-3) + + if use_bias: + bias = torch.rand((1, n), device="cuda", dtype=out_dtype) * 10 + 2.5 + else: + bias = torch.zeros((1, n), device="cuda", dtype=out_dtype) + + baseline_dq = (torch.mm(a_dq, b_dq) + bias).to(out_dtype) + + # int32 mm not supported on CUDA + a_noazp_i32_cpu = (aq_i32 - azp_aq_i8).to(device="cpu") + cq = (a_noazp_i32_cpu @ bq_i32.to(device="cpu")).to(device="cuda") + baseline_q = (scale_a * scale_b * cq + bias).to(dtype=out_dtype) + + # Hadamard is just the sum of the cols + azp_adj_i32 = bq_i32.sum(dim=0, keepdim=True, dtype=torch.int32) + azp_i32 = azp_aq_i8.to(dtype=torch.int32) + func_bias = bias if use_bias else None + + if azp_per_token: + out = ops.cutlass_scaled_mm_azp( + aq_i8, bq_i8, scale_a, scale_b, out_dtype, azp_adj_i32, azp_i32, func_bias + ) + else: + azp_with_adj_i32 = azp_i32 * azp_adj_i32 + out = ops.cutlass_scaled_mm_azp( + aq_i8, bq_i8, scale_a, scale_b, out_dtype, azp_with_adj_i32, None, func_bias + ) + + # bfloat16 precision is 7-bit mantissa -> 2^-8 ~ 0.4% + # float16 precision is 10-bit mantissa -> 2^-11 ~ 0.05% + rtol = 1e-2 if out_dtype == torch.bfloat16 else 1e-3 + atol = 1e-3 + torch.testing.assert_close(out, baseline_dq, rtol=rtol, atol=atol) + torch.testing.assert_close(out, baseline_q, rtol=rtol, atol=atol) + + if azp_per_token: + opcheck( + torch.ops._C.cutlass_scaled_mm_azp, + (out, aq_i8, bq_i8, scale_a, scale_b, azp_adj_i32, azp_i32, func_bias), + ) + else: + opcheck( + torch.ops._C.cutlass_scaled_mm_azp, + (out, aq_i8, bq_i8, scale_a, scale_b, azp_with_adj_i32, None, func_bias), + ) + + +# Test working with a subset of A and B +def test_cutlass_subset(): + big_m, big_n, big_k = 1024, 1024, 1024 + m, n, k = 512, 512, 512 + + whole_a = to_int8(torch.randn((big_m, big_k), device="cuda") * 5) + whole_b = to_int8(torch.randn((big_n, big_k), device="cuda").t() * 5) + a = whole_a[0:m, 0:k] + b = whole_b[0:k, 0:n] + + scale_a = torch.randn((1, 1), device="cuda", dtype=torch.float32) / 10 + scale_b = torch.randn((1, 1), device="cuda", dtype=torch.float32) / 10 + + out = ops.cutlass_scaled_mm(a, b, scale_a, scale_b, out_dtype=torch.bfloat16) + baseline = baseline_scaled_mm(a, b, scale_a, scale_b, out_dtype=torch.bfloat16) + + torch.testing.assert_close(out, baseline, rtol=1e-1, atol=1e0) + + +# Test to make sure cuda graphs work +class CutlassLayer(torch.nn.Module): + def __init__(self, b, scale_a, scale_b, out_dtype): + super().__init__() + self.b = b + self.scale_a = scale_a + self.scale_b = scale_b + self.out_dtype = out_dtype + + def forward(self, a): + return ops.cutlass_scaled_mm( + a, self.b, self.scale_a, self.scale_b, self.out_dtype + ) + + +@pytest.mark.parametrize("per_act_token", [True, False]) +@pytest.mark.parametrize("per_out_ch", [True, False]) +def test_cutlass_cuda_graph(per_act_token: bool, per_out_ch: bool): + m, n, k = 512, 512, 512 + + a = to_int8(torch.randn((m, k), device="cuda")) + b = to_int8(torch.randn((n, k), device="cuda").t()) + + m_a_scales = m if per_act_token else 1 + n_b_scales = n if per_out_ch else 1 + + scale_a = torch.randn((m_a_scales, 1), device="cuda", dtype=torch.float32) / 10 + scale_b = torch.randn((1, n_b_scales), device="cuda", dtype=torch.float32) / 10 + + # Construct a trivial model with a single layer that calls a CUTLASS kernel + model = CutlassLayer(b, scale_a, scale_b, torch.bfloat16) + + # Run the model with a cuda graph + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + g = torch.cuda.CUDAGraph() + with torch.cuda.graph(g): + out = model(a) + out.zero_() + g.replay() + + baseline = torch.mm( + scale_a * a.to(dtype=torch.float32), scale_b * b.to(dtype=torch.float32) + ).to(torch.bfloat16) + torch.testing.assert_close(out, baseline, rtol=1e-1, atol=1e0) + + +def test_cutlass_support_opcheck(): + opcheck(torch.ops._C.cutlass_scaled_mm_supports_fp8, (capability,)) + + +@pytest.mark.parametrize("num_experts", [8, 64]) +@pytest.mark.parametrize("per_act_token", [True, False]) +@pytest.mark.parametrize("per_out_ch", [True, False]) +@pytest.mark.parametrize("use_bias", [False]) +@pytest.mark.skipif( + (lambda x: x is None or not ops.cutlass_group_gemm_supported(x.to_int()))( + current_platform.get_device_capability() + ), + reason="Grouped gemm is not supported on this GPU type.", +) +def test_cutlass_fp8_group_gemm( + num_experts: int, per_act_token: bool, per_out_ch: bool, use_bias: bool +): + # Device and dtype setup + device = "cuda" + out_dtype = torch.half + + # Create separate A, B, C tensors for each group + a_tensors = [] + b_tensors = [] + a_scales_tensors = [] + b_scales_tensors = [] + baseline_tensors = [] + + expert_offsets = torch.zeros((num_experts + 1), device=device, dtype=torch.int64) + + problem_sizes = torch.zeros((num_experts, 3), device=device, dtype=torch.int32) + + if not per_act_token: + one_scale_a = torch.randn((1, 1), device=device, dtype=torch.float32) + + alignment = 16 # 128 // 8 + # For variation, each group has dimensions + n_g = alignment * random.randint(1, 64) + k_g = alignment * random.randint(1, 64) + for g in range(num_experts): + m_g = alignment * random.randint(1, 64) + + expert_offsets[g + 1] = expert_offsets[g] + m_g + problem_sizes[g][0] = m_g + problem_sizes[g][1] = n_g + problem_sizes[g][2] = k_g + + m_a_scales = m_g if per_act_token else 1 + n_b_scales = n_g if per_out_ch else 1 + + # Create group-specific A and B (FP8) and output (FP16/FP32) + a_g = to_fp8(torch.randn((m_g, k_g), device=device)) + b_g = to_fp8(torch.randn((n_g, k_g), device=device).t()) + a_tensors.append(a_g) + b_tensors.append(b_g) + + # Set up A/B scales + scale_b = torch.randn((1, n_b_scales), device=device, dtype=torch.float32) + b_scales_tensors.append(scale_b) + + if per_act_token: + scale_a = torch.randn((m_a_scales, 1), device=device, dtype=torch.float32) + a_scales_tensors.append(scale_a) + else: + scale_a = one_scale_a + + # Compute baseline result for this group + baseline_g = baseline_scaled_mm(a_g, b_g, scale_a, scale_b, out_dtype, None) + baseline_tensors.append(baseline_g) + + a_tensors_stacked = torch.empty( + (expert_offsets[num_experts], k_g), device=device, dtype=torch.float8_e4m3fn + ) + b_tensors_stacked = torch.empty( + (num_experts, n_g, k_g), device=device, dtype=torch.float8_e4m3fn + ) + + for g in range(num_experts): + a_tensors_stacked[expert_offsets[g] : expert_offsets[g + 1]] = a_tensors[g] + b_tensors_stacked[g] = b_tensors[g].t() + b_tensors_stacked = b_tensors_stacked.transpose(1, 2) + + if per_act_token: + a_scales_tensors_stacked = torch.empty( + (expert_offsets[num_experts], 1), device=device, dtype=torch.float32 + ) + for g in range(num_experts): + a_scales_tensors_stacked[expert_offsets[g] : expert_offsets[g + 1]] = ( + a_scales_tensors[g] + ) + else: + a_scales_tensors_stacked = one_scale_a + + b_scales_tensors_stacked = torch.empty( + (num_experts, n_b_scales), device=device, dtype=torch.float32 + ) + for g in range(num_experts): + b_scales_tensors_stacked[g] = b_scales_tensors[g] + + out_tensors_stacked = torch.zeros( + (expert_offsets[num_experts], n_g), device=device, dtype=out_dtype + ) + + ab_strides = torch.full( + (num_experts,), a_tensors_stacked.stride(0), device="cuda", dtype=torch.int64 + ) + c_strides = torch.full( + (num_experts,), out_tensors_stacked.stride(0), device="cuda", dtype=torch.int64 + ) + + ops.cutlass_moe_mm( + out_tensors_stacked, + a_tensors_stacked, + b_tensors_stacked, + a_scales_tensors_stacked, + b_scales_tensors_stacked, + expert_offsets[:-1], + problem_sizes, + ab_strides, + ab_strides, + c_strides, + per_act_token, + per_out_ch, + ) + + # Validate each group's result against the baseline + for g in range(num_experts): + baseline = baseline_tensors[g] + c = out_tensors_stacked[expert_offsets[g] : expert_offsets[g + 1]] + torch.testing.assert_close(c, baseline, rtol=1e-2, atol=5e-4) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_cutlass_w4a8.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_cutlass_w4a8.py new file mode 100644 index 0000000000000000000000000000000000000000..0dd2340c958fda2b2b969bdfb410b250c672bbed --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_cutlass_w4a8.py @@ -0,0 +1,329 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the CUTLASS W4A8 kernel. + +Run `pytest tests/kernels/quantization/test_cutlass_w4a8.py`. +""" + +from dataclasses import dataclass + +import pytest +import torch + +from vllm import _custom_ops as ops +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + convert_packed_uint4b8_to_signed_int4_inplace, + pack_cols, + pack_rows, + quantize_weights, + unpack_quantized_values_into_int32, +) +from vllm.platforms import current_platform +from vllm.scalar_type import ScalarType, scalar_types + +if not current_platform.is_cuda(): + pytest.skip("These tests use CUTLASS which requires CUDA", allow_module_level=True) + +# TODO: in future PR refactor this and `is_quant_method_supported` in the kernel +# unit tests to a common utility function. Currently the use of +# `is_quant_method_supported` conflates kernels with quantization methods +# an assumption which is breaking down as quantizations methods can have +# have kernels and some kernels support multiple quantization methods. +IS_SUPPORTED_BY_GPU = current_platform.get_device_capability()[0] >= 9 + +MNK_SHAPES = [ + (1, 128, 128), + (1, 512, 1024), + (1, 4096, 4096), + (1, 8192, 28672), + (13, 8192, 4096), + (26, 4096, 8192), + (64, 4096, 4096), + (64, 8192, 28672), + (257, 128, 4096), + (257, 4096, 4096), + (1024, 4096, 8192), + (1024, 8192, 4096), +] + +# TODO(czhu): get supported schedules from fn +SCHEDULES = [ + "128x16_1x1x1", + "256x16_1x1x1", + "128x32_1x1x1", + "256x32_1x1x1", + "128x64_1x1x1", + "256x64_1x1x1", + "128x128_1x1x1", + "256x128_1x1x1", + "128x256_1x1x1", + "128x256_2x1x1", +] + + +@dataclass +class TypeConfig: + act_type: torch.dtype + weight_type: ScalarType + output_type: torch.dtype | None + group_scale_type: torch.dtype | None + channel_scale_type: torch.dtype | None + token_scale_type: torch.dtype | None + + +@dataclass +class Tensors: + w_ref: torch.Tensor + a_ref: torch.Tensor + a: torch.Tensor + w_q: torch.Tensor + w_g_s: torch.Tensor + w_ch_s: torch.Tensor + w_tok_s: torch.Tensor + + +# (Act Type, Weight Type, Output Type, Scale Type, ZeroPoints, +# Ch Scales Type, Tok Scales Type) +TestTypeTuple = tuple[ + list[torch.dtype], ScalarType, torch.dtype | None, torch.dtype | None, bool +] +TEST_TYPES = [ + *( + TypeConfig( + act_type=torch.float8_e4m3fn, + weight_type=w_type, + output_type=o_type, + group_scale_type=torch.float8_e4m3fn, + channel_scale_type=torch.float32, + token_scale_type=torch.float32, + ) + for w_type in [scalar_types.int4] + # TODO(czhu): fp16 out type + for o_type in [torch.bfloat16] + ), +] + +# TODO: in future PR refactor this and `is_quant_method_supported` in the kernel +# unit tests to a common utility function. Currently the use of +# `is_quant_method_supported` conflates kernels with quantization methods +# an assumption which is breaking down as quantizations methods can have +# have kernels and some kernels support multiple quantization methods. +IS_SUPPORTED_BY_GPU = current_platform.has_device_capability(90) + + +# For testing quantized linear kernels +def to_fp8(tensor: torch.Tensor): + finfo = torch.finfo(torch.float8_e4m3fn) + return tensor.clamp(min=finfo.min, max=finfo.max).to(dtype=torch.float8_e4m3fn) + + +def cutlass_quantize_and_pack( + atype: torch.dtype, + w: torch.Tensor, + wtype: ScalarType, + stype: torch.dtype | None, + group_size: int | None, + zero_points: bool = False, +): + assert wtype.is_integer(), "TODO: support floating point weights" + + w_ref, w_q, w_s, w_zp = quantize_weights( + w, wtype, group_size=group_size, zero_points=zero_points + ) + + # since scales are cast to fp8, we need to compute w_ref this way + w_ref = ( + (w_q).to(torch.float32) + * w_s.to(atype).to(torch.float32).repeat_interleave(group_size, dim=0) + ).to(atype) + + # bit mask prevents sign extending int4 when packing + w_q = pack_rows(w_q & 0x0F, wtype.size_bits, *w_q.shape) + w_q = w_q.t().contiguous().t() # convert to col major + + w_q_packed = ops.cutlass_encode_and_reorder_int4b(w_q) + w_s_packed = ops.cutlass_pack_scale_fp8(w_s.to(atype)) + + return w_ref, w_q_packed, w_s_packed, w_zp + + +def create_test_tensors( + shape: tuple[int, int, int], types: TypeConfig, group_size: int | None +) -> Tensors: + m, n, k = shape + + print( + "create_test_tensors, shape:", shape, "types:", types, "group_size:", group_size + ) + + a = to_fp8(torch.randn((m, k), device="cuda")) + w = to_fp8(torch.randn((k, n), device="cuda")) + + if types.group_scale_type is not None: + w = w.to(types.group_scale_type) + if w.dtype.itemsize == 1: + w = w.to(torch.float16) + + w_ref, w_q_packed, w_s, _ = cutlass_quantize_and_pack( + a.dtype, w, types.weight_type, types.group_scale_type, group_size, False + ) + + a_ref = a.to(torch.float32) + w_ref = w_ref.to(torch.float32) + + # for the practical use case we need per-tok scales for fp8 activations + w_tok_s = torch.randn((m,), device="cuda", dtype=types.token_scale_type) + w_ch_s = torch.randn((n,), device="cuda", dtype=types.channel_scale_type) + + return Tensors( + w_ref=w_ref, + a_ref=a_ref, + a=a, + w_q=w_q_packed, + w_g_s=w_s, + w_ch_s=w_ch_s, + w_tok_s=w_tok_s, + ) + + +def mm_test_helper( + types: TypeConfig, + tensors: Tensors, + group_size: int | None = None, + schedule: str | None = None, +): + # CUTLASS upstream uses fp8 with fastaccum as reference + # https://github.com/NVIDIA/cutlass/blob/main/examples/55_hopper_mixed_dtype_gemm/55_hopper_int4_fp8_gemm.cu#L406 + output_ref = torch._scaled_mm( + tensors.a_ref.to(types.act_type), + tensors.w_ref.to(types.act_type).t().contiguous().t(), # col major + tensors.w_tok_s.unsqueeze(1), + tensors.w_ch_s.unsqueeze(0), + out_dtype=types.output_type, + use_fast_accum=True, + ) + + output = ops.cutlass_w4a8_mm( + a=tensors.a, + b_q=tensors.w_q, + b_group_scales=tensors.w_g_s, + b_group_size=group_size, + b_channel_scales=tensors.w_ch_s, + a_token_scales=tensors.w_tok_s, + ) + + print(output) + print(output_ref) + + torch.testing.assert_close( + output, output_ref.to(output.dtype), rtol=1e-2, atol=1e-2 + ) + + +@pytest.mark.skipif( + not IS_SUPPORTED_BY_GPU, reason="CUTLASS W4A8 is not supported on this GPU type." +) +@pytest.mark.parametrize("shape", MNK_SHAPES, ids=lambda x: "x".join(str(v) for v in x)) +@pytest.mark.parametrize("types", TEST_TYPES) +@pytest.mark.parametrize("schedule", SCHEDULES) +def test_cutlass_w4a8(shape, types: TypeConfig, schedule): + group_sizes = [128] + for group_size in group_sizes: + tensors = create_test_tensors(shape, types, group_size) + mm_test_helper(types, tensors, group_size, schedule) + + +# Test to make sure cuda graphs work +class W4A8Layer(torch.nn.Module): + def __init__(self, **kwargs): + super().__init__() + self.kwargs = kwargs + + def forward(self, a): + return ops.cutlass_w4a8_mm(a=a, **self.kwargs) + + +@pytest.mark.skipif( + not IS_SUPPORTED_BY_GPU, reason="CUTLASS W4A8 is not supported on this GPU type." +) +def test_w4a8_cuda_graph(): + m, n, k = 512, 4096, 4096 + + a = to_fp8(torch.randn((m, k), device="cuda")) + b = to_fp8(torch.randn((k, n), device="cuda")) + + wtype = scalar_types.int4 + stype = torch.float8_e4m3fn + group_size = 128 + zero_points = False + + w_ref, w_q_packed, w_s, _ = cutlass_quantize_and_pack( + a.dtype, b.to(torch.float16), wtype, stype, group_size, zero_points + ) + + w_tok_s = torch.randn((m,), device="cuda", dtype=torch.float32) + w_ch_s = torch.randn((n,), device="cuda", dtype=torch.float32) + + # Construct a trivial model with a single layer that calls the kernel + model = W4A8Layer( + b_q=w_q_packed, + b_group_scales=w_s, + b_group_size=group_size, + b_channel_scales=w_ch_s, + a_token_scales=w_tok_s, + ) + + output_ref = torch._scaled_mm( + a, + w_ref.to(a.dtype).t().contiguous().t(), # col major + w_tok_s.unsqueeze(1), + w_ch_s.unsqueeze(0), + out_dtype=torch.bfloat16, + use_fast_accum=True, + ) + + # Run the model with a cuda graph + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + g = torch.cuda.CUDAGraph() + with torch.cuda.graph(g): + output = model(a) + + output.zero_() + g.replay() + + torch.testing.assert_close(output, output_ref, rtol=1e-2, atol=1e-2) + + +@pytest.mark.skipif( + not IS_SUPPORTED_BY_GPU, reason="CUTLASS W4A8 is not supported on this GPU type." +) +@pytest.mark.parametrize("shape", MNK_SHAPES) +def test_convert_packed_uint4b8_to_signed_int4_inplace(shape): + """ + The W4A16 checkpoints encode the weights as int4b8 packed to int32. + The CUTLASS kernels expect signed int4 packed to int32. + This tests checks that the runtime int4b8 -> signed int4 conversion + matches the offline conversion step exactly. + """ + _, N, K = shape + # random weights packed to int32 + t = torch.randint( + low=torch.iinfo(torch.int32).min, + high=torch.iinfo(torch.int32).max + 1, + size=(N, K // 8), + dtype=torch.int32, + device="cuda", + ) + + # compute reference + unpacked = unpack_quantized_values_into_int32( + t.clone(), scalar_types.uint4b8, packed_dim=1 + ) + unpacked = unpacked - 8 # int4b8 -> signed int4 + ref = pack_cols(unpacked & 0x0F, 4, *unpacked.shape) + + out = convert_packed_uint4b8_to_signed_int4_inplace(t.clone()) + + assert torch.equal(ref, out) + assert not torch.equal(ref, t) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_cutlass_w4a8_moe.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_cutlass_w4a8_moe.py new file mode 100644 index 0000000000000000000000000000000000000000..39375d06d2f15f6a608a4b0db21c91ad52d299e0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_cutlass_w4a8_moe.py @@ -0,0 +1,343 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Tests for the CUTLASS-based W4A8 grouped GEMM kernel and the full MoE layer. +""" + +import random +from dataclasses import dataclass + +import pytest +import torch + +from vllm import _custom_ops as ops +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + pack_rows, + quantize_weights, +) +from vllm.platforms import current_platform +from vllm.scalar_type import ScalarType, scalar_types +from vllm.utils.torch_utils import set_random_seed + +IS_SUPPORTED_BY_GPU = ( + current_platform.is_cuda() and current_platform.get_device_capability()[0] >= 9 +) + + +def to_fp8(tensor: torch.Tensor) -> torch.Tensor: + finfo = torch.finfo(torch.float8_e4m3fn) + return tensor.clamp(min=finfo.min, max=finfo.max).to(dtype=torch.float8_e4m3fn) + + +def cutlass_quantize( + atype: torch.dtype, + w: torch.Tensor, + wtype: ScalarType, + stype: torch.dtype | None, + group_size: int | None, + zero_points: bool = False, +): + """ + Quantize weights into W4 and compute reference dequantized weights. + + Encoding/reordering of weights and packing of scales is deferred + until after all experts are combined. + """ + assert wtype.is_integer(), "TODO: support floating point weights" + + w_ref, w_q, w_s, w_zp = quantize_weights( + w, wtype, group_size=group_size, zero_points=zero_points + ) + + # Since scales are later cast to fp8, recompute w_ref in atype here. + w_ref = ( + w_q.to(torch.float32) + * w_s.to(atype).to(torch.float32).repeat_interleave(group_size, dim=0) + ).to(atype) + + # Bit mask prevents sign extension of int4 when packing. + w_q = pack_rows(w_q & 0x0F, wtype.size_bits, *w_q.shape) + # Make weights row-major (N, K). + w_q = w_q.t().contiguous() + + return w_ref, w_q, w_s.to(atype), w_zp + + +def cutlass_preprocess( + w_q_experts: list[torch.Tensor], w_s_experts: list[torch.Tensor] +): + """ + Reorder/encode expert weights and pack scales. + + Returns: + w_q_packed: Packed/encoded int4 weights for all experts. + w_s_packed: Packed fp8 scales for all experts. + packed_layout: Layout/stride metadata for grouped GEMM. + """ + w_s_packed = ops.cutlass_pack_scale_fp8(torch.stack(w_s_experts)) + w_q_packed, packed_layout = ops.cutlass_encode_and_reorder_int4b_grouped( + torch.stack(w_q_experts) + ) # expects dim 3 + return w_q_packed, w_s_packed, packed_layout + + +GROUP_SIZE = 128 +# (num_experts, N, K) +TEST_SHAPES = [ + (8, 512, 2048), + (8, 2048, 2048), + (64, 512, 1024), + (64, 2048, 2048), + (4, 2048, 768), + (8, 768, 2048), + (64, 1536, 2048), + (128, 8192, 4096), # test overflow int32 +] +ALIGNMENT = 16 # torch._scaled_mm alignment for M, needed for reference check + + +@dataclass +class MoETestSetup: + num_experts: int + K: int + N: int + Ms: list[int] + M_full: int + a: torch.Tensor + a_ref: torch.Tensor + a_strides: torch.Tensor + out: torch.Tensor + c_strides: torch.Tensor + per_tok_scales: torch.Tensor + per_chan_scales: torch.Tensor + w_refs: list[torch.Tensor] + w_q_packed: torch.Tensor + w_s_packed: torch.Tensor + problem_sizes: torch.Tensor + expert_offsets: torch.Tensor + b_strides: torch.Tensor + group_scale_strides: torch.Tensor + + +def make_moe_test_setup( + num_experts: int, + K: int, + N: int, + *, + alignment: int = ALIGNMENT, + max_blocks: int = 64, + device: str = "cuda", + random_zero: bool = False, +) -> MoETestSetup: + """Create a full set of tensors for testing cutlass_w4a8_moe_mm.""" + + assert K % GROUP_SIZE == 0 + # Token counts per expert (multiples of `alignment`). + Ms = [alignment * random.randint(1, max_blocks) for _ in range(num_experts)] + + # set random experts to 0 tokens + if random_zero and num_experts > 1: + num_zero = max(1, num_experts // 8) + zero_indices = random.sample(range(num_experts), k=num_zero) + for idx in zero_indices: + Ms[idx] = 0 + + M_full = sum(Ms) + assert M_full > 0 + + # Activations. + a = to_fp8(torch.randn((M_full, K), device=device)) + a_ref = a.to(torch.float32) + a_strides = torch.full((num_experts,), K, dtype=torch.int64, device=device) + + # Output buffer. + out = torch.empty((M_full, N), dtype=torch.bfloat16, device=device) + c_strides = torch.full((num_experts,), N, dtype=torch.int64, device=device) + + # Channel/token scales. + per_tok_scales = torch.randn((M_full, 1), dtype=torch.float32, device=device) + per_chan_scales = torch.randn( + (num_experts, N, 1), dtype=torch.float32, device=device + ) + + # Expert weights and scales. + wtype = scalar_types.int4 + atype = stype = torch.float8_e4m3fn + w_refs, w_qs, w_ss = [], [], [] + for _ in range(num_experts): + b = to_fp8(torch.randn((K, N), device=device)) + w_ref, w_q, w_s, _ = cutlass_quantize( + atype, b.to(torch.float16), wtype, stype, GROUP_SIZE, zero_points=False + ) + w_refs.append(w_ref) + w_qs.append(w_q) + w_ss.append(w_s) + + w_q_packed, w_s_packed, packed_layout = cutlass_preprocess(w_qs, w_ss) + + problem_sizes = torch.tensor( + [[N, M, K] for M in Ms], dtype=torch.int32, device=device + ) + + expert_offsets = torch.cat( + [ + torch.tensor([0], dtype=torch.int64), + torch.cumsum(torch.tensor(Ms, dtype=torch.int64), dim=0)[:-1], + ] + ).to(device=device) + + # B strides and group scale strides. + b_strides = packed_layout + group_scale_strides = torch.zeros( + (num_experts, 2), dtype=torch.int64, device=device + ) + group_scale_strides[:, 0] = N + + return MoETestSetup( + num_experts=num_experts, + K=K, + N=N, + Ms=Ms, + M_full=M_full, + a=a, + a_ref=a_ref, + a_strides=a_strides, + out=out, + c_strides=c_strides, + per_tok_scales=per_tok_scales, + per_chan_scales=per_chan_scales, + w_refs=w_refs, + w_q_packed=w_q_packed, + w_s_packed=w_s_packed, + problem_sizes=problem_sizes, + expert_offsets=expert_offsets, + b_strides=b_strides, + group_scale_strides=group_scale_strides, + ) + + +def compute_moe_reference_output(setup: MoETestSetup) -> torch.Tensor: + """Compute reference output using torch._scaled_mm per expert.""" + out_ref = torch.empty_like(setup.out) + + ends = torch.cumsum(torch.tensor(setup.Ms), 0).tolist() + starts = setup.expert_offsets.cpu().tolist() + + for i in range(setup.num_experts): + start, end = starts[i], ends[i] + if start == end: + continue + + out_ref_i = torch._scaled_mm( + setup.a_ref[start:end].to(torch.float8_e4m3fn), + setup.w_refs[i].to(torch.float8_e4m3fn).t().contiguous().t(), + setup.per_tok_scales[start:end], # (M, 1) + setup.per_chan_scales[i].reshape(1, -1), # (1, N) + out_dtype=torch.bfloat16, + use_fast_accum=True, + ) + out_ref[start:end] = out_ref_i + + return out_ref + + +@pytest.mark.skipif( + not IS_SUPPORTED_BY_GPU, + reason="W4A8 Grouped GEMM is not supported on this GPU type.", +) +@pytest.mark.parametrize("shape", TEST_SHAPES) +@pytest.mark.parametrize("random_zero", [True, False]) +def test_cutlass_w4a8_moe_mm_end_to_end(shape, random_zero): + num_experts, N, K = shape + set_random_seed(42) + setup = make_moe_test_setup( + num_experts=num_experts, K=K, N=N, max_blocks=64, random_zero=random_zero + ) + + ops.cutlass_w4a8_moe_mm( + setup.out, + setup.a, + setup.w_q_packed, + setup.per_tok_scales, + setup.per_chan_scales, + setup.w_s_packed, + GROUP_SIZE, + setup.expert_offsets, + setup.problem_sizes, + setup.a_strides, + setup.b_strides, + setup.c_strides, + setup.group_scale_strides, + ) + torch.accelerator.synchronize() + + out_ref = compute_moe_reference_output(setup) + torch.testing.assert_close(setup.out, out_ref, rtol=1e-2, atol=1e-2) + + +class W4A8MoELayer(torch.nn.Module): + """ + Minimal wrapper module to test cuda graphs + """ + + def __init__(self, setup: MoETestSetup): + super().__init__() + self.setup = setup + + def forward(self, a: torch.Tensor) -> torch.Tensor: + s = self.setup + ops.cutlass_w4a8_moe_mm( + s.out, + a, + s.w_q_packed, + s.per_tok_scales, + s.per_chan_scales, + s.w_s_packed, + GROUP_SIZE, + s.expert_offsets, + s.problem_sizes, + s.a_strides, + s.b_strides, + s.c_strides, + s.group_scale_strides, + ) + return s.out + + +@pytest.mark.skipif( + not IS_SUPPORTED_BY_GPU, + reason="W4A8 Grouped GEMM is not supported on this GPU type.", +) +def test_cutlass_w4a8_moe_mm_cuda_graph(): + set_random_seed(42) + # Fixed config for CUDA graph test (single parameter point). + num_experts = 8 + K = 512 + N = 2048 + + setup = make_moe_test_setup( + num_experts=num_experts, + K=K, + N=N, + max_blocks=32, + ) + + # Construct model that calls the grouped GEMM kernel. + model = W4A8MoELayer(setup) + + # Build reference output once. + out_ref = compute_moe_reference_output(setup) + + # Capture and run the model in a CUDA graph. + a_static = setup.a.clone() # static input tensor for graph replay + + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + g = torch.cuda.CUDAGraph() + with torch.cuda.graph(g): + out_static = model(a_static) + + out_static.zero_() + g.replay() + + torch.testing.assert_close(out_static, out_ref, rtol=1e-2, atol=1e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_flashinfer_nvfp4_scaled_mm.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_flashinfer_nvfp4_scaled_mm.py new file mode 100644 index 0000000000000000000000000000000000000000..6865f14fd476df06464981927b0569dc99d9aa3d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_flashinfer_nvfp4_scaled_mm.py @@ -0,0 +1,168 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch +from nvfp4_utils import ( + FLOAT4_E2M1_MAX, + FLOAT8_E4M3_MAX, + convert_swizzled_to_linear, + dequantize_nvfp4_to_dtype, +) + +from vllm import _custom_ops as ops +from vllm.platforms import current_platform +from vllm.utils.flashinfer import ( + flashinfer_scaled_fp4_mm, + has_flashinfer_b12x_gemm, +) +from vllm.utils.torch_utils import set_random_seed + +if not current_platform.has_device_capability(100): + pytest.skip( + reason="Nvfp4 Requires compute capability of 10 or above.", + allow_module_level=True, + ) + +DTYPES = [torch.float16, torch.bfloat16] +# m, n, k +SHAPES = [ + (128, 128, 64), + (128, 128, 128), + (256, 128, 64), + (128, 256, 128), + (1, 128, 128), +] +PAD_SHAPES = [(150, 128, 64), (128, 128, 96), (2, 128, 64), (3, 128, 96)] +SHAPES.extend(PAD_SHAPES) + +SEEDS = [42] +CUDA_DEVICES = ["cuda:0"] + + +def get_ref_results( + a_fp4, + b_fp4, + a_sf, + b_sf, + a_global_scale, + b_global_scale, + m, + n, + dtype, + block_size, + device, + is_sf_128x4_layout, +): + _, m_k = a_fp4.shape + _, n_k = b_fp4.shape + assert m_k == n_k + a_in_dtype = dequantize_nvfp4_to_dtype( + a_fp4, + a_sf, + a_global_scale, + dtype=dtype, + device=device, + block_size=block_size, + is_sf_128x4_layout=is_sf_128x4_layout, + ) + b_in_dtype = dequantize_nvfp4_to_dtype( + b_fp4, b_sf, b_global_scale, dtype=dtype, device=device, block_size=block_size + ) + return torch.matmul(a_in_dtype, b_in_dtype.t()) + + +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("shape", SHAPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("backend", ["cute-dsl", "cutlass", "cudnn", "trtllm", "b12x"]) +@pytest.mark.parametrize("autotune", [False, True]) +@torch.inference_mode() +def test_flashinfer_nvfp4_gemm( + dtype: torch.dtype, + shape: tuple[int, int, int], + seed: int, + device: str, + backend: str, + autotune: bool, +) -> None: + if "trtllm" in backend and dtype == torch.float16: + pytest.skip("Only torch.bfloat16 is supported for TRTLLM FP4 GEMM operations") + if backend == "cute-dsl" and not current_platform.is_device_capability_family(100): + pytest.skip("FlashInfer cutedsl backend is only supported on SM10x") + if backend == "b12x" and not current_platform.has_device_capability(120): + pytest.skip("b12x FP4 GEMM requires SM120+ (CC 12.0+)") + if backend == "b12x" and not has_flashinfer_b12x_gemm(): + pytest.skip("b12x FP4 GEMM backend not available in installed FlashInfer") + + set_random_seed(seed) + m, n, packed_k = shape + k = packed_k * 2 + block_size = 16 + a_dtype = torch.randn((m, k), dtype=dtype, device=device) + b_dtype = torch.randn((n, k), dtype=dtype, device=device) + + a_global_scale = ( + (FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX) / torch.amax(a_dtype.flatten(), dim=-1) + ).to(torch.float32) + b_global_scale = ( + (FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX) / torch.amax(b_dtype.flatten(), dim=-1) + ).to(torch.float32) + alpha = 1.0 / (a_global_scale * b_global_scale) + + # ops.scaled_fp4_quant returns swizzled scales, while weights + # from checkpoints are in linear scales. + # cutlass and b12x use swizzled scales directly; trtllm needs them unswizzled. + a_fp4, a_scale_interleaved = ops.scaled_fp4_quant( + a_dtype, a_global_scale, is_sf_swizzled_layout=True, backend=backend + ) + is_sf_128x4_layout = not (backend == "trtllm" and m <= 32) + + b_fp4, b_scale_interleaved = ops.scaled_fp4_quant( + b_dtype, b_global_scale, is_sf_swizzled_layout=True + ) + + # get_ref_results unswizzles the scales internally. + expected_out = get_ref_results( + a_fp4, + b_fp4, + a_scale_interleaved, + b_scale_interleaved, + a_global_scale, + b_global_scale, + m, + n, + dtype, + block_size, + device, + is_sf_128x4_layout, + ) + + import flashinfer + + if "trtllm" in backend: + epilogue_tile_m = 128 + b_fp4 = flashinfer.shuffle_matrix_a(b_fp4.view(torch.uint8), epilogue_tile_m) + b_scale_interleaved = convert_swizzled_to_linear( + b_scale_interleaved, n, k, block_size + ) + b_scale_interleaved = ( + flashinfer.shuffle_matrix_sf_a( + b_scale_interleaved.view(torch.uint8), epilogue_tile_m + ) + .reshape(b_scale_interleaved.shape) + .view(torch.float8_e4m3fn) + ) + + with flashinfer.autotune(autotune): + out = flashinfer_scaled_fp4_mm( + a_fp4, + b_fp4, + a_scale_interleaved, + b_scale_interleaved, + alpha, + dtype, + backend=backend, + ) + + torch.testing.assert_close(out, expected_out.to(dtype=dtype), atol=1e-1, rtol=1e-1) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_flashinfer_scaled_mm.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_flashinfer_scaled_mm.py new file mode 100644 index 0000000000000000000000000000000000000000..8d238b0dfa627036bebb91fe55cc379564f37ada --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_flashinfer_scaled_mm.py @@ -0,0 +1,73 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch + +from vllm import _custom_ops as ops +from vllm.platforms import current_platform +from vllm.utils.flashinfer import flashinfer_scaled_fp8_mm +from vllm.utils.torch_utils import set_random_seed + +if not current_platform.has_device_capability(100): + pytest.skip( + reason="Flashinfer FP8 gemms requires compute capability of 10.0 or above.", + allow_module_level=True, + ) + +DTYPES = [torch.float16, torch.bfloat16] +# m, n, k +SHAPES = [(128, 128, 64), (128, 128, 128), (256, 128, 64), (128, 256, 128)] +PAD_SHAPES = [(150, 128, 64), (128, 128, 96)] +SHAPES.extend(PAD_SHAPES) + +SEEDS = [42] +CUDA_DEVICES = ["cuda:0"] + + +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("shape", SHAPES) +@pytest.mark.parametrize("use_bias", [True, False]) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@pytest.mark.parametrize("autotune", [False, True]) +@torch.inference_mode() +def test_flashinfer_fp8_gemm( + dtype: torch.dtype, + shape: tuple[int, int, int], + use_bias: bool, + seed: int, + device: str, + autotune: bool, +) -> None: + set_random_seed(seed) + m, n, k = shape + a = torch.randn((m, k), dtype=dtype, device=device) + b = torch.randn((n, k), dtype=dtype, device=device) / k + + a_fp8, a_scale = ops.scaled_fp8_quant(a) + b_fp8, b_scale = ops.scaled_fp8_quant(b) + + expected_out = torch.mm( + a_scale * a_fp8.to(dtype=torch.float32), + b_scale * b_fp8.to(dtype=torch.float32).t(), + ).to(dtype=dtype) + + if use_bias: + bias = torch.randn((n,), dtype=dtype, device=device) + expected_out = expected_out + bias + else: + bias = None + + import flashinfer + + with flashinfer.autotune(autotune): + out = flashinfer_scaled_fp8_mm( + a_fp8, + b_fp8.t(), + a_scale, + b_scale, + dtype, + bias=bias, + ) + + torch.testing.assert_close(out, expected_out, atol=1e-2, rtol=1e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_fp8_min_max_helper.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_fp8_min_max_helper.py new file mode 100644 index 0000000000000000000000000000000000000000..efde5db4d959404208cfb64b24d74571164dfbe7 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_fp8_min_max_helper.py @@ -0,0 +1,65 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Unit tests for the get_fp8_min_max() helper function. + +These tests verify the FP8 min/max value logic for both standard +and fnuz (ROCm MI300) dtype handling. +""" + +from unittest.mock import patch + +import pytest +import torch + +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + get_fp8_min_max, +) + + +class TestGetFp8MinMax: + """Test cases for get_fp8_min_max() function.""" + + @patch("vllm.model_executor.layers.quantization.utils.quant_utils.current_platform") + def test_standard_fp8_platform(self, mock_platform): + """Test that standard FP8 platform uses PyTorch's finfo values.""" + mock_platform.is_fp8_fnuz.return_value = False + mock_platform.fp8_dtype.return_value = torch.float8_e4m3fn + + fp8_min, fp8_max = get_fp8_min_max() + finfo = torch.finfo(torch.float8_e4m3fn) + + # Standard FP8 max is 448.0 for e4m3fn + assert fp8_max == finfo.max, f"Expected finfo.max={finfo.max}, got {fp8_max}" + assert fp8_min == finfo.min, f"Expected finfo.min={finfo.min}, got {fp8_min}" + + @patch("vllm.model_executor.layers.quantization.utils.quant_utils.current_platform") + def test_fnuz_platform_returns_224(self, mock_platform): + """Test that fnuz platform returns 224.0.""" + mock_platform.is_fp8_fnuz.return_value = True + + fp8_min, fp8_max = get_fp8_min_max() + + # fnuz on ROCm MI300 should return 224.0, not 240.0 + assert fp8_max == 224.0, f"Expected 224.0 for fnuz platform, got {fp8_max}" + assert fp8_min == -224.0, f"Expected -224.0 for fnuz platform, got {fp8_min}" + + @patch("vllm.model_executor.layers.quantization.utils.quant_utils.current_platform") + def test_non_fnuz_platform_uses_finfo(self, mock_platform): + """Test that non-fnuz platform uses finfo values.""" + mock_platform.is_fp8_fnuz.return_value = False + mock_platform.fp8_dtype.return_value = torch.float8_e4m3fn + + fp8_min, fp8_max = get_fp8_min_max() + finfo = torch.finfo(torch.float8_e4m3fn) + + assert fp8_max == finfo.max, ( + f"Non-fnuz platform should use finfo.max={finfo.max}, got {fp8_max}" + ) + assert fp8_min == finfo.min, ( + f"Non-fnuz platform should use finfo.min={finfo.min}, got {fp8_min}" + ) + + +if __name__ == "__main__": + pytest.main([__file__, "-v"]) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_fp8_quant.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_fp8_quant.py new file mode 100644 index 0000000000000000000000000000000000000000..a853bde5254b02ac73c849435aecc8fcd9669b49 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_fp8_quant.py @@ -0,0 +1,221 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +import vllm._custom_ops as ops +from tests.kernels.quant_utils import ( + FP8_DTYPE, + ref_dynamic_per_tensor_fp8_quant, + ref_dynamic_per_token_quant, +) +from tests.kernels.utils import opcheck +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + scaled_quantize, +) +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +DTYPES = [torch.bfloat16, torch.float] +HIDDEN_SIZES = [17, 1024, 1025, 1026, 5137, 8193] +NUM_TOKENS = [1, 7, 4096] +SCALE_UBS = [True, False] +SEEDS = [0] + + +def opcheck_fp8_quant( + output, + input, + scale=None, + scale_ub=None, + use_per_token_if_dynamic=False, + group_shape=None, +): + if scale is not None: + opcheck( + torch.ops._C.static_scaled_fp8_quant, + (output, input, scale, group_shape), + ) + elif use_per_token_if_dynamic: + scale = torch.empty( + (input.shape[0], 1), device=input.device, dtype=torch.float32 + ) + opcheck( + torch.ops._C.dynamic_per_token_scaled_fp8_quant, + (output, input, scale, scale_ub), + ) + else: + scale = torch.empty( + (input.numel() // input.shape[-1], 1), + device=input.device, + dtype=torch.float32, + ) + opcheck(torch.ops._C.dynamic_scaled_fp8_quant, (output, input, scale)) + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", HIDDEN_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("do_scale_ub", SCALE_UBS) +@pytest.mark.parametrize("seed", SEEDS) +@torch.inference_mode() +def test_dynamic_per_token_fp8_quant( + num_tokens: int, hidden_size: int, dtype: torch.dtype, do_scale_ub: bool, seed: int +) -> None: + set_random_seed(seed) + + x = ( + torch.rand(num_tokens, hidden_size, dtype=dtype, device="cuda") + 1e-6 + ) # avoid nans + + scale_ub = ( + torch.mean(x).to(dtype=torch.float32, device="cuda") if do_scale_ub else None + ) + ref_out, ref_scales = ref_dynamic_per_token_quant(x, FP8_DTYPE, scale_ub) + ops_out, ops_scales = ops.scaled_fp8_quant( + x, scale_ub=scale_ub, use_per_token_if_dynamic=True + ) + + torch.testing.assert_close(ref_scales, ops_scales) + torch.testing.assert_close( + ref_out.to(dtype=torch.float32), ops_out.to(dtype=torch.float32) + ) + + opcheck_fp8_quant(ops_out, x, None, scale_ub, use_per_token_if_dynamic=True) + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", HIDDEN_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@torch.inference_mode() +def test_dynamic_per_tensor_fp8_quant( + num_tokens: int, hidden_size: int, dtype: torch.dtype, seed: int +) -> None: + set_random_seed(seed) + + x = torch.rand(num_tokens, hidden_size, dtype=dtype, device="cuda") + + ref_out, ref_scale = ref_dynamic_per_tensor_fp8_quant(x) + ops_out, ops_scale = ops.scaled_fp8_quant(x) + + torch.testing.assert_close(ref_scale, ops_scale) + torch.testing.assert_close( + ref_out.to(dtype=torch.float32), ops_out.to(dtype=torch.float32) + ) + + opcheck_fp8_quant(ops_out, x) + + +# Regression test for a case with large activations where an int32 index cannot +# represent the number of elements. +@torch.inference_mode() +@pytest.mark.parametrize("seed", SEEDS) +def test_fp8_quant_large(seed: int) -> None: + set_random_seed(seed) + + num_tokens = 1024000 # Mistral-Nemo's max_position_embeddings + hidden_size = 1152 # Smallest hidden_size to reproduce the error + dtype = torch.bfloat16 + + x = torch.rand(num_tokens, hidden_size, dtype=dtype, device="cuda") + ref_out, scale = ref_dynamic_per_tensor_fp8_quant(x) + ops_out, _ = ops.scaled_fp8_quant(x, scale) + + # Minimize memory footprint in this test by freeing x and upconverting + # the outputs in place. (torch.allclose does not support fp8) + del x + ref_out = ref_out.to(dtype=dtype) + ops_out = ops_out.to(dtype=dtype) + + torch.testing.assert_close(ref_out, ops_out) + + +# Test static FP8 quantization with 2D group scales +GROUP_SHAPES_2D = [ + (-1, -1), # Per-tensor + (-1, 1), # Per-channel + (1, -1), # Per-token + (-1, 128), # Per-head quantization + (1, 128), # DeepSeek-style per-token-per-group (group_m=1, group_n=128) + (128, 128), # DeepSeek-style block quantization + (1, 64), # Smaller group size + (1, 16), # Small group (scalar path in kernel) + (4, 256), # Non-trivial both dimensions +] +# Use sizes divisible by all group shapes +NUM_TOKENS_GROUP = [128, 512] +HIDDEN_SIZES_GROUP = [256, 1024, 2048] + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS_GROUP) +@pytest.mark.parametrize("hidden_size", HIDDEN_SIZES_GROUP) +@pytest.mark.parametrize("group_shape", GROUP_SHAPES_2D) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@torch.inference_mode() +def test_static_fp8_quant_group_2d( + num_tokens: int, + hidden_size: int, + group_shape: tuple[int, int], + dtype: torch.dtype, + seed: int, +) -> None: + """Test static FP8 quantization with 2D group scales using scaled_quantize.""" + # Normalize group_shape (-1 means full extent) + norm_group_m = num_tokens if group_shape[0] == -1 else group_shape[0] + norm_group_n = hidden_size if group_shape[1] == -1 else group_shape[1] + + # Skip if sizes are not divisible by group shape + if num_tokens % norm_group_m != 0 or hidden_size % norm_group_n != 0: + pytest.skip( + f"Skipping: ({num_tokens}, {hidden_size}) not divisible by " + f"group_shape ({group_shape[0]}, {group_shape[1]})" + ) + + set_random_seed(seed) + + x = torch.rand(num_tokens, hidden_size, dtype=dtype, device="cuda") + ref_out, scale = scaled_quantize( + x, group_shape, current_platform.fp8_dtype(), compute_dtype=torch.float32 + ) + ops_out, ops_scale = ops.scaled_fp8_quant(x, scale=scale, group_shape=group_shape) + + torch.testing.assert_close(scale, ops_scale) + torch.testing.assert_close(ref_out.float(), ops_out.float(), rtol=1.2e-1, atol=1e-5) + + opcheck_fp8_quant(ops_out, x, scale=scale) + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS_GROUP) +@pytest.mark.parametrize("hidden_size", HIDDEN_SIZES_GROUP) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("group_shape", [(1, -1), (-1, 1)]) # per-token, per-channel +@torch.inference_mode() +def test_static_fp8_quant_1d_scale( + num_tokens: int, + hidden_size: int, + dtype: torch.dtype, + seed: int, + group_shape: tuple[int, int], +) -> None: + """Test static FP8 quantization with 1D scale (per-token or per-channel).""" + set_random_seed(seed) + + x = torch.rand(num_tokens, hidden_size, dtype=dtype, device="cuda") + ref_out, scale_2d = scaled_quantize( + x, group_shape, FP8_DTYPE, compute_dtype=torch.float32 + ) + + # Flatten scale to 1D for testing 1D scale path + scale_1d = scale_2d.flatten() + ops_out, ops_scale = ops.scaled_fp8_quant( + x, scale=scale_1d, group_shape=group_shape + ) + + torch.testing.assert_close(scale_1d, ops_scale) + torch.testing.assert_close(ref_out.float(), ops_out.float(), rtol=0.12, atol=0.0) + + opcheck_fp8_quant(ops_out, x, scale=scale_1d, group_shape=group_shape) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_fp8_quant_group.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_fp8_quant_group.py new file mode 100644 index 0000000000000000000000000000000000000000..2b0f3b0a1e7834874eeb4bd80a97b07e2061db49 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_fp8_quant_group.py @@ -0,0 +1,173 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for QuantFP8 Group Quantization implementation.""" + +import pytest +import torch + +from vllm.model_executor.layers.quantization.input_quant_fp8 import QuantFP8 +from vllm.model_executor.layers.quantization.utils.quant_utils import GroupShape +from vllm.utils.torch_utils import set_random_seed + + +@pytest.mark.parametrize( + "batch_size,hidden_dim,group_size", + [ + (16, 256, 32), # Small + (64, 1024, 64), # Medium + (128, 2048, 128), # Large + (8, 513, 64), # Non-divisible (native only) + ], +) +@pytest.mark.parametrize("seed", [42]) +@pytest.mark.parametrize("use_ue8m0", [True, False]) +@torch.inference_mode() +def test_quantfp8_group_functionality( + default_vllm_config, + batch_size: int, + hidden_dim: int, + group_size: int, + seed: int, + use_ue8m0: bool, +) -> None: + """Test QuantFP8 group quantization with various configurations. + + Tests both CUDA and native implementations, column-major scales, + and verifies consistency between implementations. + """ + set_random_seed(seed) + + x = torch.randn((batch_size, hidden_dim), dtype=torch.bfloat16, device="cuda") * 8 + expected_num_groups = (hidden_dim + group_size - 1) // group_size + is_divisible = hidden_dim % group_size == 0 + + group_shape = GroupShape(1, group_size) + quant_op = QuantFP8( + static=False, + group_shape=group_shape, + column_major_scales=False, + use_ue8m0=use_ue8m0, + ) + + # 1. Test native implementation (always available) + x_quant_native, scales_native = quant_op.forward_native(x.clone()) + assert x_quant_native.shape == x.shape + assert scales_native.shape == (batch_size, expected_num_groups) + + # 2. Test column-major scales configuration + quant_op_col = QuantFP8( + static=False, + group_shape=group_shape, + column_major_scales=True, + use_ue8m0=use_ue8m0, + ) + _, scales_col = quant_op_col.forward_native(x.clone()) + assert scales_col.shape == (batch_size, expected_num_groups) + assert scales_col.stride(0) == 1 + assert scales_col.stride(1) == batch_size + + # Test column-major scales consistency + torch.testing.assert_close(scales_col, scales_native, rtol=1e-9, atol=1e-8) + + # 3. Test CUDA implementation (only for divisible dimensions) + if is_divisible: + x_quant_cuda, scales_cuda = quant_op.forward_cuda(x.clone()) + assert x_quant_cuda.shape == x.shape + assert scales_cuda.shape == (batch_size, expected_num_groups) + + # Verify CUDA/native consistency + torch.testing.assert_close(scales_cuda, scales_native, rtol=2e-7, atol=2e-8) + + # Quantized values should mostly match + diff_count = (x_quant_cuda != x_quant_native).sum().item() + diff_ratio = diff_count / x_quant_cuda.numel() + assert diff_ratio < 0.002, f"Too many differences: {diff_ratio:.4%}" + + +@pytest.mark.parametrize("seed", [42]) +@pytest.mark.parametrize("use_ue8m0", [True, False]) +@torch.inference_mode() +def test_quantfp8_group_multidimensional( + default_vllm_config, seed: int, use_ue8m0: bool +) -> None: + set_random_seed(seed) + + group_size = 64 + + # Test with 3D input + batch1, batch2, hidden_dim = 4, 8, 1024 + x_3d = ( + torch.randn((batch1, batch2, hidden_dim), dtype=torch.bfloat16, device="cuda") + * 8 + ) + + group_shape = GroupShape(1, group_size) + quant_op = QuantFP8( + static=False, + group_shape=group_shape, + column_major_scales=False, + use_ue8m0=use_ue8m0, + ) + + x_quant, scales = quant_op.forward_native(x_3d.clone()) + assert x_quant.shape == x_3d.shape + assert scales.shape == (batch1, batch2, hidden_dim // group_size) + + # Test column_major_scales with multi-dim + quant_op_col = QuantFP8( + static=False, + group_shape=group_shape, + column_major_scales=True, + use_ue8m0=use_ue8m0, + ) + _, scales_col = quant_op_col.forward_native(x_3d.clone()) + assert scales_col.shape == (batch1, batch2, hidden_dim // group_size) + + # Test with 4D input + batch1, batch2, batch3, hidden_dim = 2, 3, 4, 256 + x_4d = ( + torch.randn( + (batch1, batch2, batch3, hidden_dim), dtype=torch.bfloat16, device="cuda" + ) + * 8 + ) + + x_quant_4d, scales_4d = quant_op.forward_native(x_4d.clone()) + assert x_quant_4d.shape == x_4d.shape + assert scales_4d.shape == (batch1, batch2, batch3, hidden_dim // group_size) + + _, scales_4d_col = quant_op_col.forward_native(x_4d.clone()) + assert scales_4d_col.shape == (batch1, batch2, hidden_dim // group_size, batch3) + + +@pytest.mark.parametrize("seed", [42]) +@torch.inference_mode() +def test_quantfp8_group_edge_cases(default_vllm_config, seed: int) -> None: + set_random_seed(seed) + + batch_size = 16 + group_size = 64 + + # Test with single group (group_size >= hidden_dim) + x_small = torch.randn((batch_size, 32), dtype=torch.bfloat16, device="cuda") * 8 + group_shape = GroupShape(1, group_size) + quant_op = QuantFP8( + static=False, group_shape=group_shape, column_major_scales=False + ) + + x_quant_small, scales_small = quant_op.forward_native(x_small.clone()) + assert x_quant_small.shape == x_small.shape + assert scales_small.shape == (batch_size, 1) + + # Test with zero inputs + x_zero = torch.zeros((batch_size, 256), dtype=torch.bfloat16, device="cuda") + x_quant_zero, scales_zero = quant_op.forward_native(x_zero.clone()) + assert x_quant_zero.shape == x_zero.shape + assert (scales_zero > 0).all(), "Scales should be clamped to minimum" + + # Test very large values + x_large = torch.full((batch_size, 256), 1000.0, dtype=torch.bfloat16, device="cuda") + x_quant_large, scales_large = quant_op.forward_native(x_large.clone()) + assert x_quant_large.shape == x_large.shape + # FP8 max is typically 448 or 224, so scales should be > 1 + assert (scales_large > 1.0).all(), "Large values should have scales > 1" diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_gptq.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_gptq.py new file mode 100644 index 0000000000000000000000000000000000000000..c3302cf147065fb357c7a290ebf9324a4696e694 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_gptq.py @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import torch + +from tests.kernels.utils import opcheck +from vllm import _custom_ops as ops # noqa: F401 + + +def test_gptq_shuffle_opcheck(): + weight = torch.randint( + -2000000, 2000000, (1792, 4096), device="cuda", dtype=torch.int32 + ) + perm = torch.empty((0,), device="cuda", dtype=torch.int32) + bit = 4 + opcheck(torch.ops._C.gptq_shuffle, (weight, perm, bit)) + + +def test_gptq_gemm_opcheck(): + a = torch.rand((240, 4096), device="cuda", dtype=torch.float16) + weight = torch.randint( + -2000000, 2000000, (512, 6144), device="cuda", dtype=torch.int32 + ) + zeros = torch.zeros((32, 768), device="cuda", dtype=torch.int32) + scales = torch.rand((32, 6144), device="cuda", dtype=torch.float16) + idx = torch.empty((0,), device="cuda", dtype=torch.int32) + use_exllama = True + bit = 4 + # Test both GPTQv1 and GPTQv2 format + opcheck( + torch.ops._C.gptq_gemm, (a, weight, zeros, scales, idx, use_exllama, True, bit) + ) + opcheck( + torch.ops._C.gptq_gemm, (a, weight, zeros, scales, idx, use_exllama, False, bit) + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_hadacore.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_hadacore.py new file mode 100644 index 0000000000000000000000000000000000000000..2d949536b0ec0fbef1ade9de6221f90a2459d034 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_hadacore.py @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import math + +import pytest +import torch +from compressed_tensors.transform import deterministic_hadamard_matrix + +from vllm import _custom_ops as ops +from vllm.platforms import current_platform + +if current_platform.is_rocm(): + pytest.skip( + "These tests require hadacore_transform, not supported on ROCm.", + allow_module_level=True, + ) + + +@pytest.mark.parametrize("batch_size", [1, 32]) +@pytest.mark.parametrize("hidden_dim", [2**n for n in range(10)]) +def test_hadacore(batch_size, hidden_dim, dtype=torch.bfloat16, device="cuda"): + x = torch.eye(hidden_dim, dtype=dtype, device=device) + hadamard = deterministic_hadamard_matrix( + hidden_dim, dtype=torch.float64, device="cuda" + ) / math.sqrt(hidden_dim) + + y = ops.hadacore_transform(x.clone()) + y_true = (x.to(hadamard.dtype) @ hadamard.T).to(y.dtype) + assert torch.allclose(y, y_true) + + y = ops.hadacore_transform(y) + assert torch.allclose(y, x) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_int8_kernel.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_int8_kernel.py new file mode 100644 index 0000000000000000000000000000000000000000..23c7a9f65bac043a513b832d194eccbc0387cbd7 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_int8_kernel.py @@ -0,0 +1,155 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +# Adapted from https://github.com/sgl-project/sglang/blob/main/test/srt/test_int8_kernel.py +import itertools + +import pytest +import torch + +from vllm.model_executor.layers.activation import SiluAndMul +from vllm.model_executor.layers.fused_moe import fused_experts +from vllm.model_executor.layers.fused_moe.config import FusedMoEQuantConfig +from vllm.model_executor.layers.quantization.utils.int8_utils import ( + per_token_quant_int8, +) +from vllm.platforms import current_platform + +if current_platform.get_device_capability() < (7, 0): + pytest.skip("INT8 Triton requires CUDA 7.0 or higher", allow_module_level=True) + + +def native_w8a8_per_token_matmul(A, B, As, Bs, output_dtype=torch.float16): + """Matrix multiplication function that supports per-token input + quantization and per-column weight quantization""" + A = A.to(torch.float32) + B = B.to(torch.float32) + + assert A.shape[-1] == B.shape[-1], "Dimension mismatch" + assert B.ndim == 2 and B.is_contiguous(), "B must be a 2D contiguous tensor" + + # Reshape input + M = A.numel() // A.shape[-1] + B = B.t() # Transpose weight matrix + N, K = B.shape + origin_C_shape = A.shape[:-1] + (K,) + A = A.reshape(M, N) + + # As is per-token [M, 1], Bs is per-column [1, K] + C = torch.matmul(A, B) # [M, K] + C = As * C * Bs.view(1, -1) # Broadcast per-column scale + + return C.reshape(origin_C_shape).to(output_dtype) + + +def torch_w8a8_per_column_moe(a, w1, w2, w1_s, w2_s, topk, topk_weight, topk_ids): + """This function performs fused moe with per-column int8 quantization + using native torch.""" + + B, D = a.shape + # Perform per-token quantization + a_q, a_s = per_token_quant_int8(a) + # Repeat tokens to match topk + a_q = a_q.view(B, -1, D).repeat(1, topk, 1).reshape(-1, D) + # Also repeat the scale + a_s = a_s.view(B, -1, 1).repeat(1, topk, 1).reshape(-1, 1) # [B*topk, 1] + + out = torch.zeros(B * topk, w2.shape[1], dtype=a.dtype, device=a.device) + + # Calculate routing + topk_weight = topk_weight.view(-1) + topk_ids = topk_ids.view(-1) + # Process each expert + for i in range(w1.shape[0]): + mask = topk_ids == i + if mask.sum(): + # First MLP layer: note that a_s is now per-token + inter_out = native_w8a8_per_token_matmul( + a_q[mask], w1[i], a_s[mask], w1_s[i], output_dtype=a.dtype + ) + # Activation function + act_out = SiluAndMul().forward_native(inter_out) + # Quantize activation output with per-token + act_out_q, act_out_s = per_token_quant_int8(act_out) + + # Second MLP layer + out[mask] = native_w8a8_per_token_matmul( + act_out_q, w2[i], act_out_s, w2_s[i], output_dtype=a.dtype + ) + # Apply routing weights and sum + return ( + out.view(B, -1, w2.shape[1]) * topk_weight.view(B, -1, 1).to(out.dtype) + ).sum(dim=1) + + +@pytest.fixture(autouse=True, scope="module") +def setup_cuda(): + """Sets the default CUDA device for all tests in this module.""" + torch.set_default_device("cuda") + + +DTYPES = [torch.half, torch.bfloat16] +M = [1, 33] +N = [128, 1024] +K = [256, 4096] +E = [8] +TOP_KS = [2, 6] +SEEDS = [0] + + +@pytest.mark.parametrize( + "M, N, K, E, topk, dtype, seed", + itertools.product(M, N, K, E, TOP_KS, DTYPES, SEEDS), +) +@torch.inference_mode() +def test_w8a8_fp8_fused_moe(default_vllm_config, M, N, K, E, topk, dtype, seed): + torch.manual_seed(seed) + # Initialize int8 quantization parameters + factor_for_scale = 1e-2 + int8_max = 127 + int8_min = -128 + + # Input tensor + # M * K + a = torch.randn((M, K), dtype=dtype) / 10 + + # Generate int8 weights + w1_fp32 = (torch.rand((E, 2 * N, K), dtype=torch.float32) - 0.5) * 2 + w1 = (w1_fp32 * int8_max).clamp(min=int8_min, max=int8_max).to(torch.int8) + + w2_fp32 = (torch.rand((E, K, N), dtype=torch.float32) - 0.5) * 2 + w2 = (w2_fp32 * int8_max).clamp(min=int8_min, max=int8_max).to(torch.int8) + + # Generate scale for each column (per-column quantization) + w1_s = torch.rand(E, 2 * N, device=w1_fp32.device) * factor_for_scale + w2_s = torch.rand(E, K, device=w2_fp32.device) * factor_for_scale + score = torch.randn((M, E), dtype=dtype) + score = torch.softmax(score, dim=-1, dtype=torch.float32) + topk_weights, topk_ids = torch.topk(score, topk) + + ref_out = torch_w8a8_per_column_moe( + a, w1, w2, w1_s, w2_s, topk, topk_weights, topk_ids + ) + + quant_config = FusedMoEQuantConfig.make( + torch.int8, + per_act_token_quant=True, + block_shape=None, + w1_scale=w1_s, + w2_scale=w2_s, + ) + + out = fused_experts( + a, + w1, + w2, + topk_weights, + topk_ids, + quant_config=quant_config, + ) + + # Check results + rel_diff = torch.mean( + torch.abs(out.to(torch.float32) - ref_out.to(torch.float32)) + ) / torch.mean(torch.abs(ref_out.to(torch.float32))) + assert rel_diff < 0.05 diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_int8_quant.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_int8_quant.py new file mode 100644 index 0000000000000000000000000000000000000000..8f49117f284cfa352f9840083dcf2abb3ddd87c4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_int8_quant.py @@ -0,0 +1,195 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from tests.kernels.quant_utils import ref_dynamic_per_token_quant +from tests.kernels.utils import opcheck +from vllm._custom_ops import scaled_int8_quant +from vllm.utils.torch_utils import set_random_seed + +DTYPES = [torch.bfloat16, torch.float] +HIDDEN_SIZES = [17, 1024, 1025, 1026, 5137, 8193] +NUM_TOKENS = [1, 7, 4096] +SEEDS = [0] +SCALE = [0.1, 2.1] + + +def opcheck_int8_quant_static(output, input, scale, azp=None): + if azp is None: + opcheck(torch.ops._C.static_scaled_int8_quant, (output, input, scale, None)) + else: + opcheck(torch.ops._C.static_scaled_int8_quant, (output, input, scale, azp)) + + +def opcheck_int8_quant_dynamic(output, input, symmetric=True): + scale = torch.empty( + (input.numel() // input.shape[-1], 1), device=input.device, dtype=torch.float32 + ) + if symmetric: + opcheck(torch.ops._C.dynamic_scaled_int8_quant, (output, input, scale, None)) + else: + azp = torch.empty( + (input.numel() // input.shape[-1], 1), + device=input.device, + dtype=torch.int32, + ) + opcheck(torch.ops._C.dynamic_scaled_int8_quant, (output, input, scale, azp)) + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", HIDDEN_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@torch.inference_mode() +def test_dynamic_scaled_int8_quant( + num_tokens: int, hidden_size: int, dtype: torch.dtype, seed: int +) -> None: + set_random_seed(seed) + + x = torch.rand(num_tokens, hidden_size, dtype=dtype, device="cuda") * 1000 + + # reference + ref_out, ref_scales = ref_dynamic_per_token_quant(x, torch.int8) + # kernel + ops_out, ops_scales, _ = scaled_int8_quant(x) + + torch.testing.assert_close(ops_scales, ref_scales) + # big atol to account for rounding errors + torch.testing.assert_close(ops_out, ref_out, atol=1, rtol=0.0) + + opcheck_int8_quant_dynamic(ops_out, x) + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", HIDDEN_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@torch.inference_mode() +def test_dynamic_scaled_int8_azp_quant( + num_tokens: int, hidden_size: int, dtype: torch.dtype, seed: int +) -> None: + set_random_seed(seed) + int8_traits = torch.iinfo(torch.int8) + + x = torch.rand(num_tokens, hidden_size, dtype=dtype, device="cuda") * 1000 - 300 + + x_token_max, _ = x.to(dtype=torch.float32).max(dim=1, keepdim=True) + x_token_min, _ = x.to(dtype=torch.float32).min(dim=1, keepdim=True) + + # calculate scale and azp, and adjust the range + scales = (x_token_max - x_token_min) / torch.tensor(255.0) + azps = torch.round(torch.tensor(-128.0) - x_token_min / scales).to(torch.int32) + + torch_out = ( + ((x / scales).round() + azps) + .clamp(int8_traits.min, int8_traits.max) + .to(torch.int8) + ) + assert torch_out.min() >= int8_traits.min and torch_out.max() <= int8_traits.max + + ops_out, scales_out, azp_out = scaled_int8_quant(x, symmetric=False) + + if not torch.allclose(scales_out, scales): + print(torch.argmax(torch.abs(scales_out - scales))) + torch.testing.assert_close(scales_out, scales) + # big atol to account for rounding errors + torch.testing.assert_close(azp_out, azps, atol=1, rtol=0.0) + # if AZP is off by 1, after rounding-to-even, the output may be off by 2 + torch.testing.assert_close(ops_out, torch_out, atol=2, rtol=0.0) + + opcheck_int8_quant_dynamic(ops_out, x, False) + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", HIDDEN_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("scale", SCALE) +@torch.inference_mode() +def test_static_scaled_int8_quant( + num_tokens: int, hidden_size: int, dtype: torch.dtype, seed: int, scale: float +) -> None: + set_random_seed(seed) + int8_traits = torch.iinfo(torch.int8) + + x = torch.rand(num_tokens, hidden_size, dtype=dtype, device="cuda") * 1000 + scale_arg = torch.tensor([scale], dtype=torch.float32, device="cuda") + + out1 = ( + (x / scale_arg).round().clamp(int8_traits.min, int8_traits.max).to(torch.int8) + ) + out2, scale2, _ = scaled_int8_quant(x, scale_arg) + assert scale2 is scale_arg + + # big atol to account for rounding errors + torch.testing.assert_close(out1, out2, atol=1, rtol=0.0) + + opcheck_int8_quant_static(out2, x, scale_arg) + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", HIDDEN_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("scale", SCALE) +@pytest.mark.parametrize("azp", [-255, 54]) +@torch.inference_mode() +def test_static_scaled_int8_azp_quant( + num_tokens: int, + hidden_size: int, + dtype: torch.dtype, + seed: int, + scale: float, + azp: int, +) -> None: + set_random_seed(seed) + int8_traits = torch.iinfo(torch.int8) + + x = torch.rand(num_tokens, hidden_size, dtype=dtype, device="cuda") * 1000 - 300 + + out1 = ( + ((x / scale).round() + azp) + .clamp(int8_traits.min, int8_traits.max) + .to(torch.int8) + ) + scale_arg = torch.tensor([scale], dtype=torch.float32, device="cuda") + azp_arg = torch.tensor([azp], dtype=torch.int32, device="cuda") + + out2, scale2, azp2 = scaled_int8_quant(x, scale_arg, azp_arg, symmetric=False) + assert scale2 is scale_arg + assert azp2 is azp_arg + + # big atol to account for rounding errors + torch.testing.assert_close(out1, out2, atol=1, rtol=0.0) + + opcheck_int8_quant_static(out2, x, scale_arg, azp_arg) + + +@pytest.mark.parametrize("is_max", [True, False]) +@torch.inference_mode() +def test_static_scaled_int8_azp_quant_saturating_cast(is_max: bool) -> None: + # Test that the saturating cast works correctly for values near i32 max/min + + from numpy import inf, nextafter + + int32_traits = torch.iinfo(torch.int32) + val = float(int32_traits.max if is_max else int32_traits.min) + + x_vals = [[nextafter(val, inf), val + 1, val, val - 1, nextafter(val, -inf)]] + x = torch.tensor(x_vals, dtype=torch.float32, device="cuda") + + # The calculation in the kernel is: cast(cast(x / scale) + azp) + # where cast is a saturating cast to type T. + # Scale is set to 1.0 so that the input values are the ones that are cast. + # AZP is set to 0 to make sure the int8 saturating cast is tested as well. + scale = torch.scalar_tensor(1.0, dtype=torch.float32, device="cuda") + azp = torch.scalar_tensor(0, dtype=torch.int32, device="cuda") + + int8_traits = torch.iinfo(torch.int8) + val_i8 = int8_traits.max if is_max else int8_traits.min + expected = torch.full((1, 5), val_i8, dtype=torch.int8, device="cuda") + + out, _, _ = scaled_int8_quant(x, scale, azp, symmetric=False) + torch.testing.assert_close(expected, out, atol=0, rtol=0) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_machete_mm.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_machete_mm.py new file mode 100644 index 0000000000000000000000000000000000000000..1af1409ed252d6595eecbdcbb87622a141c2ee95 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_machete_mm.py @@ -0,0 +1,449 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the machete kernel. + +Run `pytest tests/kernels/quantization/test_machete_mm.py`. +""" + +import math +from dataclasses import dataclass, fields + +import pytest +import torch + +from tests.kernels.utils import opcheck +from vllm import _custom_ops as ops +from vllm.model_executor.layers.quantization.utils.machete_utils import ( + query_machete_supported_group_sizes, +) +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + pack_rows, + quantize_weights, +) +from vllm.platforms import current_platform +from vllm.scalar_type import ScalarType, scalar_types + +if current_platform.is_rocm(): + pytest.skip( + "These tests require machete_prepack_B, not supported on ROCm.", + allow_module_level=True, + ) + +CUDA_DEVICES = [ + f"cuda:{i}" for i in range(1 if torch.accelerator.device_count() == 1 else 2) +] + +# TODO: in future PR refactor this and `is_quant_method_supported` in the kernel +# unit tests to a common utility function. Currently the use of +# `is_quant_method_supported` conflates kernels with quantization methods +# an assumption which is breaking down as quantizations methods can have +# have kernels and some kernels support multiple quantization methods. +IS_SUPPORTED_BY_GPU = current_platform.get_device_capability()[0] >= 9 + +MNK_SHAPES = [ + (1, 128, 128), + (1, 8192, 28672), + (13, 8192, 4096), + (26, 4096, 8192), + (64, 4096, 4096), + (64, 8192, 28672), + (257, 128, 4096), + (257, 4224, 4160), + (1024, 8192, 4096), +] + + +@dataclass +class TypeConfig: + act_type: torch.dtype + weight_type: ScalarType + output_type: torch.dtype | None + group_scale_type: torch.dtype | None + group_zero_type: torch.dtype | None + channel_scale_type: torch.dtype | None + token_scale_type: torch.dtype | None + + +@dataclass +class Tensors: + w_ref: torch.Tensor + a_ref: torch.Tensor + a: torch.Tensor + w_q: torch.Tensor + w_g_s: torch.Tensor | None + w_g_zp: torch.Tensor | None + w_ch_s: torch.Tensor | None + w_tok_s: torch.Tensor | None + + +# (Act Type, Weight Type, Output Type, Scale Type, ZeroPoints, +# Ch Scales Type, Tok Scales Type) +# NOTE: None "Scale Type" means the act type is floating point +# None "Output Type" means the output type is the same as the act type +TestTypeTuple = tuple[ + list[torch.dtype], ScalarType, torch.dtype | None, torch.dtype | None, bool +] +TEST_TYPES = [ + # GPTQ style + *( + TypeConfig( + act_type=a_type, + weight_type=w_type, + output_type=None, + group_scale_type=a_type, + group_zero_type=None, + channel_scale_type=None, + token_scale_type=None, + ) + for w_type in [scalar_types.uint4b8, scalar_types.uint8b128] + for a_type in [torch.float16, torch.bfloat16] + ), + # AWQ style + *( + TypeConfig( + act_type=a_type, + weight_type=w_type, + output_type=None, + group_scale_type=a_type, + group_zero_type=a_type, + channel_scale_type=None, + token_scale_type=None, + ) + for w_type in [scalar_types.uint4, scalar_types.uint8] + for a_type in [torch.float16, torch.bfloat16] + ), + # # QQQ style + # *(TypeConfig(act_type=torch.int8, + # weight_type=scalar_types.uint4b8, + # output_type=torch.float16, + # group_scale_type=group_scale_type, + # group_zero_type=None, + # channel_scale_type=torch.float, + # token_scale_type=torch.float) + # for group_scale_type in [None, torch.float16]), + # *(TypeConfig(act_type=torch.float8_e4m3fn, + # weight_type=scalar_types.uint4b8, + # output_type=torch.float16, + # group_scale_type=group_scale_type, + # group_zero_type=None, + # channel_scale_type=torch.float, + # token_scale_type=torch.float) + # for group_scale_type in [None, torch.float16]), +] + +# TODO: in future PR refactor this and `is_quant_method_supported` in the kernel +# unit tests to a common utility function. Currently the use of +# `is_quant_method_supported` conflates kernels with quantization methods +# an assumption which is breaking down as quantizations methods can have +# have kernels and some kernels support multiple quantization methods. +IS_SUPPORTED_BY_GPU = current_platform.has_device_capability(90) + + +def rand_data(shape, dtype=torch.float16, scale=1, offset=0): + if dtype.is_floating_point: + return (scale * torch.rand(shape, device="cuda") - offset).to(dtype) + else: + return torch.randint(-8, 7, shape, dtype=dtype, device="cuda") + + +def maybe_convert_zeropoints(zps: torch.Tensor | None, s: torch.Tensor): + return zps if zps is None else -1 * s * (zps.to(s.dtype)) + + +def group_size_valid(shape: tuple[int, int, int], group_size: int | None) -> bool: + return group_size is None or group_size == -1 or shape[2] % group_size == 0 + + +def machete_quantize_and_pack( + atype: torch.dtype, + w: torch.Tensor, + wtype: ScalarType, + stype: torch.dtype | None, + group_size: int | None, + zero_points: bool = False, +): + assert wtype.is_integer(), "TODO: support floating point weights" + + w_ref, w_q, w_s, w_zp = quantize_weights( + w, + wtype, + group_size=group_size, + zero_points=zero_points, + # to match how the kernel applies zps + ref_zero_points_after_scales=True, + ) + + w_q = pack_rows(w_q, wtype.size_bits, *w_q.shape) + w_q = w_q.t().contiguous().t() # convert to col major + + w_q_machete = ops.machete_prepack_B(w_q, atype, wtype, stype) + opcheck(torch.ops._C.machete_prepack_B, (w_q, atype, wtype.id, stype)) + + return w_ref, w_q_machete, w_s, w_zp + + +def create_test_tensors( + shape: tuple[int, int, int], + types: TypeConfig, + group_size: int | None, + subset_stride_factor: int | None = None, +) -> Tensors: + m, n, k = shape + factor = subset_stride_factor or 1 + + print( + "create_test_tensors, shape:", shape, "types:", types, "group_size:", group_size + ) + + a = rand_data((m * factor, k * factor), types.act_type, scale=3, offset=2) + w = rand_data((k * factor, n * factor), types.act_type, scale=3, offset=1) + + if factor > 1: + a = a[0:m, 0:k] + w = w[0:k, 0:n] + + if types.group_scale_type is not None: + w = w.to(types.group_scale_type) + if w.dtype.itemsize == 1: + w = w.to(torch.float16) + + w_ref, w_q_packed, w_s, w_zp = machete_quantize_and_pack( + a.dtype, + w, + types.weight_type, + types.group_scale_type, + group_size, + types.group_zero_type is not None, + ) + + if not a.dtype.is_floating_point: + aiinfo = torch.iinfo(a.dtype) + w_ref = w_ref.round().clamp(aiinfo.min, aiinfo.max) + + a_ref = a.to(torch.float32) + w_ref = w_ref.to(torch.float32) + + w_ch_s = ( + None + if types.channel_scale_type is None + else rand_data((n,), types.channel_scale_type) + ) + w_tok_s = ( + None + if types.token_scale_type is None + else rand_data((m,), types.token_scale_type) + ) + + return Tensors( + w_ref=w_ref, + a_ref=a_ref, + a=a, + w_q=w_q_packed, + w_g_s=w_s, + w_g_zp=maybe_convert_zeropoints(w_zp, w_s), + w_ch_s=w_ch_s, + w_tok_s=w_tok_s, + ) + + +# None stype means scales use the same dtype as a +def machete_mm_test_helper( + types: TypeConfig, + tensors: Tensors, + group_size: int | None = None, + schedule: str | None = None, +): + output_ref = torch.matmul(tensors.a_ref, tensors.w_ref) + output_ref_type = output_ref.dtype + + if tensors.w_ch_s is not None: + output_ref = ( + output_ref.to(tensors.w_ch_s.dtype) * tensors.w_ch_s.unsqueeze(0) + ).to(output_ref_type) + if tensors.w_tok_s is not None: + output_ref = ( + output_ref.to(tensors.w_tok_s.dtype) * tensors.w_tok_s.unsqueeze(1) + ).to(output_ref_type) + + output = ops.machete_mm( + a=tensors.a, + b_q=tensors.w_q, + b_type=types.weight_type, + b_group_scales=tensors.w_g_s, + b_group_zeros=tensors.w_g_zp, + b_group_size=group_size, + b_channel_scales=tensors.w_ch_s, + a_token_scales=tensors.w_tok_s, + out_type=types.output_type, + schedule=schedule, + ) + + print(output) + print(output_ref) + + # Relax atol as our reduction dim becomes larger (more rounding error) + # Relax atol when we have zeropoints since the way machete applies + # zeropoints (after scales) causes noise around 0 + atol = ( + 1 + if tensors.w_g_zp is not None + else min(5e-2 * math.sqrt(tensors.a.shape[1]), 1) + ) + rtol = 1e-1 if tensors.a.element_size() >= 2 else 2e-1 + torch.testing.assert_close( + output, output_ref.to(output.dtype), rtol=rtol, atol=atol + ) + + +@pytest.mark.skipif( + not IS_SUPPORTED_BY_GPU, reason="Machete is not supported on this GPU type." +) +@pytest.mark.parametrize("shape", MNK_SHAPES, ids=lambda x: "x".join(str(v) for v in x)) +@pytest.mark.parametrize("types", TEST_TYPES) +def test_machete_all_schedules(shape, types: TypeConfig): + group_sizes: list[int | None] = [] + if types.group_scale_type is None: + group_sizes = [None] + else: + group_sizes = query_machete_supported_group_sizes(types.act_type) + + for group_size in group_sizes: + if not group_size_valid(shape, group_size): + continue + + tensors = create_test_tensors(shape, types, group_size) + print(f"MNK = {shape}") + for schedule in ops.machete_supported_schedules( + types.act_type, + types.weight_type, + group_scales_type=types.group_scale_type, + group_zeros_type=types.group_scale_type, + out_type=types.output_type, + ): + print(f"Testing schedule {schedule}") + machete_mm_test_helper(types, tensors, group_size, schedule) + + +@pytest.mark.skipif( + not IS_SUPPORTED_BY_GPU, reason="Machete is not supported on this GPU type." +) +@pytest.mark.parametrize("shape", MNK_SHAPES, ids=lambda x: "x".join(str(v) for v in x)) +@pytest.mark.parametrize("types", TEST_TYPES) +def test_machete_heuristic(shape, types: TypeConfig): + group_sizes: list[int | None] = [] + if types.group_scale_type is None: + group_sizes = [None] + else: + group_sizes = query_machete_supported_group_sizes(types.act_type) + + for group_size in group_sizes: + if not group_size_valid(shape, group_size): + continue + + tensors = create_test_tensors(shape, types, group_size) + machete_mm_test_helper(types, tensors, group_size) + + +# Test working on other devices +@pytest.mark.skipif( + not IS_SUPPORTED_BY_GPU, reason="Machete is not supported on this GPU type." +) +@pytest.mark.parametrize("device", CUDA_DEVICES) +def test_machete_devices(device: str): + group_size = 128 + + type_config = TypeConfig( + act_type=torch.float16, + weight_type=scalar_types.uint4b8, + output_type=None, + group_scale_type=torch.float16, + group_zero_type=None, + channel_scale_type=None, + token_scale_type=None, + ) + + tensors = create_test_tensors((512, 4096, 4096), type_config, group_size) + + for field in fields(Tensors): + tensor = getattr(tensors, field.name) + if isinstance(tensor, torch.Tensor): + setattr(tensors, field.name, tensor.to(device)) + + machete_mm_test_helper(type_config, tensors, group_size) + + +# Test working with a subset of A and B +@pytest.mark.skipif( + not IS_SUPPORTED_BY_GPU, reason="Machete is not supported on this GPU type." +) +def test_machete_subset(): + group_size = 128 + + type_config = TypeConfig( + act_type=torch.float16, + weight_type=scalar_types.uint4b8, + output_type=None, + group_scale_type=torch.float16, + group_zero_type=None, + channel_scale_type=None, + token_scale_type=None, + ) + + tensors = create_test_tensors( + (512, 4096, 4096), type_config, group_size, subset_stride_factor=2 + ) + machete_mm_test_helper(type_config, tensors, group_size) + + +# Test to make sure cuda graphs work +class MacheteLayer(torch.nn.Module): + def __init__(self, **kwargs): + super().__init__() + self.kwargs = kwargs + + def forward(self, a): + return ops.machete_mm(a=a, **self.kwargs) + + +@pytest.mark.skipif( + not IS_SUPPORTED_BY_GPU, reason="Machete is not supported on this GPU type." +) +def test_machete_cuda_graph(): + m, n, k = 512, 4096, 4096 + + a = rand_data((m, k), torch.float16) + b = rand_data((k, n), torch.float16) + wtype = scalar_types.uint4b8 + stype = torch.float16 + group_size = 128 + zero_points = False + + w_ref, w_q_packed, w_s, w_zp = machete_quantize_and_pack( + a.dtype, b, wtype, stype, group_size, zero_points + ) + + # Construct a trivial model with a single layer that calls a machete kernel + model = MacheteLayer( + b_q=w_q_packed, + b_type=wtype, + b_group_scales=w_s, + b_group_zeros=maybe_convert_zeropoints(w_zp, w_s), + b_group_size=group_size, + ) + + output_ref = torch.matmul(a, w_ref) + + # Run the model with a cuda graph + stream = torch.cuda.Stream() + with torch.cuda.stream(stream): + g = torch.cuda.CUDAGraph() + with torch.cuda.graph(g): + output = model(a) + output.zero_() + g.replay() + + # Relax atol as our reduction dim becomes larger (more rounding error) + # Relax atol when we have zeropoints since the way machete applies + # zeropoints (after scales) causes noise around 0 + atol = 1 if zero_points else min(5e-2 * math.sqrt(k), 1) + torch.testing.assert_close(output, output_ref, rtol=1e-1, atol=atol) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_marlin_gemm.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_marlin_gemm.py new file mode 100644 index 0000000000000000000000000000000000000000..c2284e0439793e1ce0b035d39aafed51e096d375 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_marlin_gemm.py @@ -0,0 +1,621 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the marlin kernel. + +Run `pytest tests/kernels/quantization/test_marlin_gemm.py`. +""" + +import itertools + +import pytest +import torch + +from tests.kernels.utils import opcheck +from tests.quantization.utils import is_quant_method_supported +from vllm import _custom_ops as ops +from vllm.model_executor.layers.quantization.utils.int8_utils import ( + per_token_quant_int8, +) +from vllm.model_executor.layers.quantization.utils.marlin_utils import ( + marlin_make_empty_g_idx, + marlin_make_workspace_new, + marlin_permute_bias, + query_marlin_supported_quant_types, +) +from vllm.model_executor.layers.quantization.utils.marlin_utils_fp4 import ( + rand_marlin_weight_mxfp4_like, + rand_marlin_weight_nvfp4_like, +) +from vllm.model_executor.layers.quantization.utils.marlin_utils_fp8 import ( + marlin_quant_fp8_torch, +) +from vllm.model_executor.layers.quantization.utils.marlin_utils_test import ( + awq_marlin_quantize, + get_weight_perm, + marlin_quantize, + marlin_weights, +) +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + awq_pack, + gptq_pack, + gptq_quantize_weights, + quantize_weights, + sort_weights, +) +from vllm.platforms import current_platform +from vllm.scalar_type import scalar_types + +if current_platform.is_rocm(): + pytest.skip( + "These tests require marlin, which is not supported on ROCm.", + allow_module_level=True, + ) + +ACT_ORDER_OPTS = [False, True] +K_FULL_OPTS = [False, True] +USE_ATOMIC_ADD_OPTS = [False, True] +USE_FP32_REDUCE_OPTS = [True] + +MARLIN_K_CHUNKS = [128] +MARLIN_N_CHUNKS = [64, 256] + +MARLIN_REPACK_NK_FACTORS = [ + (4, 8), + (7, 5), + (13, 11), +] + +MNK_FACTORS = [ + (1, 1, 1), + (1, 4, 8), + (26, 37, 13), + (257, 13, 11), +] + +DTYPES = [torch.float16, torch.bfloat16] + +DENSE_MARLIN_QUANT_TEST_CONFIGS = [ + # AWQ-INT4 + {"b_type": scalar_types.uint4, "group_blocks": [-1, 2, 4, 8]}, + # GPTQ-INT4 + { + "b_type": scalar_types.uint4b8, + "support_act_order": True, + "group_blocks": [-1, 2, 4, 8], + }, + # GPTQ-INT8 + { + "b_type": scalar_types.uint8b128, + "support_act_order": True, + "group_blocks": [-1, 2, 4, 8], + }, + # FP8 + {"b_type": scalar_types.float8_e4m3fn, "group_blocks": [-1, 8]}, + # NVFP4 + {"b_type": scalar_types.float4_e2m1f, "group_blocks": [1]}, + # MXFP4 + { + "a_type": [scalar_types.bfloat16], + "b_type": scalar_types.float4_e2m1f, + "group_blocks": [2], + }, + # AWQ-INT4 with INT8 activation + { + "a_type": [scalar_types.int8], + "b_type": scalar_types.uint4, + "group_blocks": [-1, 2, 4, 8], + }, + # GPTQ-INT4 with INT8 activation + { + "a_type": [scalar_types.int8], + "b_type": scalar_types.uint4b8, + "group_blocks": [-1, 2, 4, 8], + }, + # GPTQ-INT4 with FP8 activation + { + "a_type": [scalar_types.float8_e4m3fn], + "b_type": scalar_types.uint4b8, + "group_blocks": [-1, 2, 4, 8], + }, + # AWQ-INT4 with FP8 activation + { + "a_type": [scalar_types.float8_e4m3fn], + "b_type": scalar_types.uint4, + "group_blocks": [-1, 2, 4, 8], + }, + # MXFP4 with FP8 activation + { + "a_type": [scalar_types.float8_e4m3fn], + "b_type": scalar_types.float4_e2m1f, + "c_type": [scalar_types.bfloat16], + "group_blocks": [2], + }, +] + + +def compute_max_diff(output, output_ref): + return torch.mean(torch.abs(output - output_ref)) / torch.mean( + torch.abs(output_ref) + ) + + +def rand_data(shape, dtype=torch.float16): + return torch.randn(shape, dtype=dtype, device="cuda") + + +@pytest.mark.skipif( + not is_quant_method_supported("gptq_marlin"), + reason="Marlin is not supported on this GPU type.", +) +def test_marlin_int4_fp8_preprocess_without_zp(): + qweight_unpacked = torch.randint( + 0, 16, size=(2048, 2048), dtype=torch.int32, device="cuda" + ) + qweight_packed = qweight_unpacked[:, ::2] * 16 + qweight_unpacked[:, 1::2] + qweight_packed = qweight_packed.to(torch.int8).view(torch.int32) + + cuda_res = ops.marlin_int4_fp8_preprocess(qweight_packed) + + torch_res = torch.where( + qweight_unpacked >= 8, qweight_unpacked - 8, 15 - qweight_unpacked + ) + torch_res = torch_res[:, ::2] * 16 + torch_res[:, 1::2] + torch_res = torch_res.to(torch.int8).view(torch.int32) + + assert (cuda_res == torch_res).all() + + +@pytest.mark.skipif( + not is_quant_method_supported("gptq_marlin"), + reason="Marlin is not supported on this GPU type.", +) +def test_marlin_int4_fp8_preprocess_awq(): + group_size = 128 + + qweight_unpacked = torch.randint( + 0, 16, size=(2048, 2048), dtype=torch.int32, device="cuda" + ) + qzeros_unpacked = torch.randint( + 0, 16, size=(2048 // group_size, 2048), dtype=torch.int32, device="cuda" + ) + + qweight_packed = qweight_unpacked[:, ::2] * 16 + qweight_unpacked[:, 1::2] + qweight_packed = qweight_packed.to(torch.int8).view(torch.int32) + qzeros_packed = qzeros_unpacked[:, ::2] * 16 + qzeros_unpacked[:, 1::2] + qzeros_packed = qzeros_packed.to(torch.int8).view(torch.int32) + + cuda_res = ops.marlin_int4_fp8_preprocess(qweight_packed, qzeros_packed) + + repeated_zp = qzeros_unpacked.repeat_interleave(group_size, 0) + torch_res = qweight_unpacked - repeated_zp + torch_res[torch_res < 0] = 15 - qweight_unpacked[torch_res < 0] + torch_res = torch_res[:, ::2] * 16 + torch_res[:, 1::2] + torch_res = torch_res.to(torch.int8).view(torch.int32) + + assert (cuda_res == torch_res).all() + + +@pytest.mark.skipif( + not is_quant_method_supported("gptq_marlin"), + reason="Marlin is not supported on this GPU type.", +) +@pytest.mark.parametrize("k_chunk", MARLIN_K_CHUNKS) +@pytest.mark.parametrize("n_chunk", MARLIN_N_CHUNKS) +@pytest.mark.parametrize("quant_type", query_marlin_supported_quant_types(False, False)) +@pytest.mark.parametrize("act_order", ACT_ORDER_OPTS) +@pytest.mark.parametrize("is_a_8bit", [True, False]) +@pytest.mark.parametrize("nk_factors", MARLIN_REPACK_NK_FACTORS) +def test_gptq_marlin_repack( + k_chunk, n_chunk, quant_type, act_order, is_a_8bit, nk_factors +): + n_factor, k_factor = nk_factors + + size_k = k_chunk * k_factor + size_n = n_chunk * n_factor + group_size = 128 + + # Filter act_order + if act_order: + if group_size == -1: + return + if group_size == size_k: + return + if is_a_8bit: + return + + # Normalize group_size + if group_size == -1: + group_size = size_k + assert group_size <= size_k + + # Create input + b_weight = rand_data((size_k, size_n)) + + # Quantize (and apply act_order if provided) + w_ref, q_w, s, g_idx, rand_perm = gptq_quantize_weights( + b_weight, quant_type, group_size, act_order + ) + + # Pack to GPTQ format + q_w_gptq = gptq_pack(q_w, quant_type.size_bits, size_k, size_n) + + # For act_order, sort the "weights" and "g_idx" so that group ids are + # increasing + sort_indices = torch.empty(0, dtype=torch.int, device=b_weight.device) + if act_order: + q_w, g_idx, sort_indices = sort_weights(q_w, g_idx) + + # Pack to Marlin format + weight_perm = get_weight_perm(quant_type.size_bits, is_a_8bit) + marlin_q_w_1 = marlin_weights( + q_w, size_k, size_n, quant_type.size_bits, weight_perm, is_a_8bit + ) + + opcheck( + torch.ops._C.gptq_marlin_repack, + (q_w_gptq, sort_indices, size_k, size_n, quant_type.size_bits, is_a_8bit), + ) + + # Run Marlin repack GPU kernel + marlin_q_w_2 = ops.gptq_marlin_repack( + q_w_gptq, sort_indices, size_k, size_n, quant_type.size_bits, is_a_8bit + ) + torch.accelerator.synchronize() + + torch.testing.assert_close(marlin_q_w_1, marlin_q_w_2) + + +@pytest.mark.skipif( + not is_quant_method_supported("gptq_marlin"), + reason="Marlin is not supported on this GPU type.", +) +@pytest.mark.parametrize("k_chunk", MARLIN_K_CHUNKS) +@pytest.mark.parametrize("n_chunk", MARLIN_N_CHUNKS) +@pytest.mark.parametrize("quant_type", query_marlin_supported_quant_types(True)) +@pytest.mark.parametrize("is_a_8bit", [True, False]) +@pytest.mark.parametrize("nk_factors", MARLIN_REPACK_NK_FACTORS) +def test_awq_marlin_repack(k_chunk, n_chunk, quant_type, is_a_8bit, nk_factors): + n_factor, k_factor = nk_factors + + size_k = k_chunk * k_factor + size_n = n_chunk * n_factor + + group_size = 128 + + # Create input + b_weight = rand_data((size_k, size_n)) + + # Quantize + w_ref, q_w, s, zp = quantize_weights( + b_weight, quant_type, group_size, zero_points=True + ) + + # Pack to AWQ format + q_w_awq = awq_pack(q_w, quant_type.size_bits, size_k, size_n) + + # Pack to Marlin format + weight_perm = get_weight_perm(quant_type.size_bits, is_a_8bit) + marlin_q_w_1 = marlin_weights( + q_w, size_k, size_n, quant_type.size_bits, weight_perm, is_a_8bit + ) + + opcheck( + torch.ops._C.awq_marlin_repack, + (q_w_awq, size_k, size_n, quant_type.size_bits, is_a_8bit), + ) + + # Run Marlin repack GPU kernel + marlin_q_w_2 = ops.awq_marlin_repack( + q_w_awq, size_k, size_n, quant_type.size_bits, is_a_8bit + ) + torch.accelerator.synchronize() + + torch.testing.assert_close(marlin_q_w_1, marlin_q_w_2) + + +def marlin_generate_valid_test_cases(): + all_combinations = itertools.product( + DENSE_MARLIN_QUANT_TEST_CONFIGS, + MNK_FACTORS, + MARLIN_N_CHUNKS, + MARLIN_K_CHUNKS, + ACT_ORDER_OPTS, + K_FULL_OPTS, + USE_ATOMIC_ADD_OPTS, + USE_FP32_REDUCE_OPTS, + ) + + def is_invalid( + a_type, + b_type, + c_type, + group_blocks, + size_m, + size_n, + size_k, + act_order, + is_k_full, + use_atomic_add, + use_fp32_reduce, + ): + if use_atomic_add: + if use_fp32_reduce: + return False + if ( + c_type == scalar_types.bfloat16 + and torch.cuda.get_device_capability()[0] < 9 + ): + return False + + group_size = group_blocks if group_blocks <= 0 else group_blocks * 16 + if group_size > 0 and size_k % group_size != 0: + return False + + if act_order and group_size in [-1, size_k]: + return False + if group_size == size_k: + return False + if not act_order and is_k_full: + return False + + return a_type.size_bits < 16 or a_type is c_type + + cases = [] + for case in all_combinations: + quant_test_config, mnk_factors, n_chunk, k_chunk, act_order, *_ = case + size_m = mnk_factors[0] + size_n = mnk_factors[1] * n_chunk + size_k = mnk_factors[2] * k_chunk + + if act_order and not quant_test_config.get("support_act_order", False): + continue + + f16_types = [scalar_types.float16, scalar_types.bfloat16] + inner_combinations = itertools.product( + quant_test_config.get("a_type", f16_types), + [quant_test_config["b_type"]], + quant_test_config.get("c_type", f16_types), + quant_test_config["group_blocks"], + ) + + for sub_case in inner_combinations: + if ( + sub_case[0] == scalar_types.float8_e4m3fn + and not current_platform.is_device_capability(89) + and not current_platform.is_device_capability_family(120) + ): + continue + args = sub_case + (size_m, size_n, size_k) + case[4:] + if is_invalid(*args): + cases.append(args) + return cases + + +@pytest.mark.skipif( + not is_quant_method_supported("gptq_marlin"), + reason="Marlin is not supported on this GPU type.", +) +@pytest.mark.parametrize( + ( + "a_type, b_type, c_type, group_blocks," + "size_m, size_n, size_k, act_order, is_k_full," + "use_atomic_add, use_fp32_reduce" + ), + marlin_generate_valid_test_cases(), +) +def test_marlin_gemm( + a_type, + b_type, + c_type, + group_blocks, + size_m, + size_n, + size_k, + act_order, + is_k_full, + use_atomic_add, + use_fp32_reduce, +): + has_zp = b_type in [scalar_types.uint4, scalar_types.uint8] + + group_size = group_blocks if group_blocks <= 0 else group_blocks * 16 + + if c_type == scalar_types.float16: + dtype = torch.float16 + elif c_type == scalar_types.bfloat16: + dtype = torch.bfloat16 + else: + raise RuntimeError("unsupported c_type") + + if a_type == scalar_types.int8: + a_dtype = torch.int8 + elif a_type == scalar_types.float8_e4m3fn: + a_dtype = torch.float8_e4m3fn + else: + a_dtype = dtype + + a_input = rand_data((size_m, size_k), dtype=dtype) + b_weight = rand_data((size_k, size_n), dtype=dtype) + + if b_type == scalar_types.float4_e2m1f: + if group_size == 16: + w_ref, marlin_q_w, marlin_s, marlin_s2 = rand_marlin_weight_nvfp4_like( + b_weight.T, group_size, input_dtype=a_dtype + ) + else: + w_ref, marlin_q_w, marlin_s = rand_marlin_weight_mxfp4_like( + b_weight.T, group_size, input_dtype=a_dtype + ) + marlin_s2 = None + + g_idx = None + sort_indices = None + marlin_zp = None + elif b_type == scalar_types.float8_e4m3fn: + w_ref, marlin_q_w, marlin_s = marlin_quant_fp8_torch( + b_weight.T, group_size, input_dtype=a_dtype + ) + g_idx = None + sort_indices = None + marlin_zp = None + marlin_s2 = None + elif has_zp: + w_ref, marlin_q_w, marlin_s, marlin_zp = awq_marlin_quantize( + b_weight, b_type, group_size, input_dtype=a_dtype + ) + g_idx = None + sort_indices = None + marlin_s2 = None + else: + w_ref, marlin_q_w, marlin_s, g_idx, sort_indices, _ = marlin_quantize( + b_weight, b_type, group_size, act_order, input_dtype=a_dtype + ) + + marlin_zp = None + marlin_s2 = None + + workspace = marlin_make_workspace_new(w_ref.device) + + if a_type == scalar_types.int8: + a_input, a_scales = per_token_quant_int8(a_input) + a_input_ref = a_input.to(a_scales.dtype) * a_scales.view(-1, 1) + a_input_ref = a_input_ref.to(dtype) + + if group_size != -1: + a_scales = a_scales / 4096 * marlin_s.max() + a_scales = a_scales.float() + marlin_s = marlin_s / marlin_s.max() * 4096 + marlin_s = marlin_s.round().to(torch.int16).view(dtype) + elif a_type == scalar_types.float8_e4m3fn: + a_input, a_scales = ops.scaled_fp8_quant(a_input, use_per_token_if_dynamic=True) + a_input_ref = a_input.to(a_scales.dtype) * a_scales.view(-1, 1) + a_input_ref = a_input_ref.to(dtype) + else: + assert a_type.size_bits == 16 + a_input_ref = a_input + a_scales = None + + output = torch.empty((size_m, size_n), dtype=dtype, device=a_input.device) + + output = ops.marlin_gemm( + a_input, + output, + marlin_q_w, + None, + marlin_s, + a_scales, + marlin_s2, + marlin_zp, + g_idx, + sort_indices, + workspace, + b_type, + a_input.shape[0], + b_weight.shape[1], + a_input.shape[1], + is_k_full=is_k_full, + use_atomic_add=use_atomic_add, + use_fp32_reduce=use_fp32_reduce, + is_zp_float=False, + ) + output_ref = torch.matmul(a_input_ref, w_ref) + + max_diff = compute_max_diff(output, output_ref) + assert max_diff < 0.04 + + +def test_marlin_gemm_subset_input(): + quant_type = scalar_types.uint4b8 + group_size = 128 + + size_m, size_k, size_n = 32, 1024, 2048 + big_m = size_m * 2 + big_k = size_k * 2 + + a_input = rand_data((big_m, big_k))[8 : size_m + 8, 8 : size_k + 8] + b_weight = rand_data((size_k, size_n)) + + w_ref, marlin_q_w, marlin_s, g_idx, sort_indices, _ = marlin_quantize( + b_weight, quant_type, group_size, False + ) + + marlin_zp = marlin_make_empty_g_idx(marlin_s.device) + workspace = marlin_make_workspace_new(a_input.device) + + output = ops.marlin_gemm( + a_input, + None, + marlin_q_w, + None, + marlin_s, + None, + None, + marlin_zp, + g_idx, + sort_indices, + workspace, + quant_type, + a_input.shape[0], + b_weight.shape[1], + a_input.shape[1], + is_k_full=True, + use_atomic_add=False, + use_fp32_reduce=True, + is_zp_float=False, + ) + output_ref = torch.matmul(a_input, w_ref) + + torch.accelerator.synchronize() + + max_diff = compute_max_diff(output, output_ref) + + assert max_diff < 0.04 + + +@pytest.mark.parametrize("size_m", [1, 256]) +def test_marlin_gemm_with_bias(size_m): + quant_type = scalar_types.uint4b8 + group_size = 128 + + size_k, size_n = 1024, 2048 + a_input = rand_data((size_m, size_k)) + b_weight = rand_data((size_k, size_n)) + b_bias = rand_data((size_n,)) * 10 + + marlin_bias = marlin_permute_bias(b_bias) + + w_ref, marlin_q_w, marlin_s, g_idx, sort_indices, _ = marlin_quantize( + b_weight, quant_type, group_size, False + ) + + marlin_zp = marlin_make_empty_g_idx(marlin_s.device) + workspace = marlin_make_workspace_new(a_input.device) + + output = ops.marlin_gemm( + a_input, + None, + marlin_q_w, + marlin_bias, + marlin_s, + None, + None, + marlin_zp, + g_idx, + sort_indices, + workspace, + quant_type, + a_input.shape[0], + b_weight.shape[1], + a_input.shape[1], + is_k_full=True, + use_atomic_add=False, + use_fp32_reduce=True, + is_zp_float=False, + ) + output_ref = torch.matmul(a_input, w_ref) + b_bias.view(1, -1) + + torch.accelerator.synchronize() + + max_diff = compute_max_diff(output, output_ref) + + assert max_diff < 0.04 diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_marlin_tile_padding.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_marlin_tile_padding.py new file mode 100644 index 0000000000000000000000000000000000000000..7dce9148ca7b9e5b8599c82abceb1aee2cb13857 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_marlin_tile_padding.py @@ -0,0 +1,863 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for Marlin thread-tile padding of TP-sharded weight shapes. + +Run `pytest tests/kernels/quantization/test_marlin_tile_padding.py`. +""" + +from types import SimpleNamespace + +import pytest +import torch + +from vllm import _custom_ops as ops +from vllm.model_executor.layers.quantization.utils.marlin_utils import ( + GPTQ_MARLIN_TILE, + apply_gptq_marlin_linear, + marlin_make_empty_g_idx, + marlin_make_workspace_new, + marlin_moe_padded_intermediate, + marlin_pad_qweight, + marlin_pad_scales, + marlin_padded_nk, + marlin_permute_scales, + marlin_repacked_nk, + marlin_zero_points, +) +from vllm.model_executor.layers.quantization.utils.marlin_utils_fp4 import ( + apply_fp4_marlin_linear, + is_fp4_marlin_supported, + prepare_fp4_layer_for_marlin, +) +from vllm.model_executor.layers.quantization.utils.marlin_utils_fp8 import ( + apply_fp8_marlin_linear, + apply_mxfp8_marlin_linear, + is_fp8_marlin_supported, + prepare_fp8_layer_for_marlin, + prepare_mxfp8_layer_for_marlin, +) +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + gptq_pack, + gptq_quantize_weights, + quantize_weights, +) +from vllm.platforms import current_platform +from vllm.scalar_type import scalar_types + +# (size_n, size_k) rank-local shapes that violate Marlin tile alignment, +# e.g. produced by TP-sharding dims that are valid at TP=1. +ODD_SHAPES = [ + (200, 288), # N padded + (256, 208), # K padded + (200, 208), # both padded + (4640, 512), # Nemotron-Super-120B q_proj shard at TP=4 +] +ALIGNED_SHAPES = [(64, 128), (128, 64), (256, 256), (4608, 4096)] + + +def _is_tile_aligned(size_n: int, size_k: int) -> bool: + return (size_n % 64 == 0 and size_k % 128 == 0) or ( + size_n % 128 == 0 and size_k % 64 == 0 + ) + + +@pytest.mark.parametrize("shape", ODD_SHAPES + ALIGNED_SHAPES) +@pytest.mark.parametrize("group_size", [-1, 16, 32, 64, 128]) +def test_marlin_padded_nk(shape, group_size): + size_n, size_k = shape + padded_n, padded_k = marlin_padded_nk(size_n, size_k, group_size) + + assert padded_n >= size_n and padded_k >= size_k + assert _is_tile_aligned(padded_n, padded_k) + if group_size > 0: + assert padded_k % group_size == 0 + + # Aligned shapes must pass through unchanged (zero hot-path cost). + if _is_tile_aligned(size_n, size_k) and ( + group_size <= 0 or size_k % group_size == 0 + ): + assert (padded_n, padded_k) == (size_n, size_k) + + # Minimal: no valid shape with a smaller padded area exists. + area = padded_n * padded_k + for cand_n in range(size_n, padded_n + 1): + for cand_k in range(size_k, padded_k + 1): + if ( + _is_tile_aligned(cand_n, cand_k) + and (group_size <= 0 or cand_k % group_size == 0) + and cand_n * cand_k < area + ): + pytest.fail(f"({cand_n}, {cand_k}) beats ({padded_n}, {padded_k})") + + # Apply-time derivation from the repacked-tensor shape must round-trip. + for num_bits in (4, 8): + pack_factor = 32 // num_bits + repacked_shape = ( + padded_k // GPTQ_MARLIN_TILE, + padded_n * GPTQ_MARLIN_TILE // pack_factor, + ) + repacked = torch.empty(repacked_shape, device="meta") + assert marlin_repacked_nk(repacked, num_bits) == (padded_n, padded_k) + + +def test_marlin_pad_helpers_shapes(): + size_n, size_k, group_size = 200, 208, 16 + padded_n, padded_k = marlin_padded_nk(size_n, size_k, group_size) + + qweight = torch.zeros(size_k // 8, size_n, dtype=torch.int32) + padded = marlin_pad_qweight(qweight, size_n, size_k, padded_n, padded_k) + assert padded.shape == (padded_k // 8, padded_n) + + scales = torch.ones(size_k // group_size, size_n) + padded = marlin_pad_scales(scales, size_n, size_k, padded_n, padded_k, group_size) + assert padded.shape == (padded_k // group_size, padded_n) + assert padded[:, size_n:].abs().sum() == 0 + + channelwise = torch.ones(1, size_n) + padded = marlin_pad_scales(channelwise, size_n, size_k, padded_n, padded_k, -1) + assert padded.shape == (1, padded_n) + + +# Rank-local MoE intermediate sizes. group<=0 / 32 with a non-multiple-of-64 +# size is where tile padding triggers; 64/128 are already tile-aligned. +MOE_INTERMEDIATE_SIZES = [64, 96, 100, 176, 192, 256, 2816] + + +@pytest.mark.parametrize("intermediate", MOE_INTERMEDIATE_SIZES) +@pytest.mark.parametrize("group_size", [-1, 32, 64, 128]) +def test_marlin_moe_padded_intermediate(intermediate, group_size): + # The MoE gate only admits shapes where the group does not straddle the + # boundary, i.e. group divides the intermediate size. + if group_size > 0 and intermediate % group_size != 0: + pytest.skip("group straddles the boundary; rejected by the MoE gate") + + padded = marlin_moe_padded_intermediate(intermediate, group_size) + assert padded >= intermediate + # Valid MoE thread tile: gate-up n = 2*intermediate % 128, down k % 64. + assert (2 * padded) % 128 == 0 + assert padded % 64 == 0 + if group_size > 0: + assert padded % group_size == 0 + + # Minimal: no smaller valid intermediate exists. + for cand in range(intermediate, padded): + if ( + (2 * cand) % 128 == 0 + and cand % 64 == 0 + and (group_size <= 0 or cand % group_size == 0) + ): + pytest.fail(f"{cand} beats {padded}") + + # Already-tile-aligned sizes pass through unchanged (zero hot-path cost). + if intermediate % 64 == 0: + assert padded == intermediate + + +def test_marlin_moe_pad_helpers_shapes(): + from vllm.model_executor.layers.fused_moe.oracle.int_wna16 import ( + _pad_rows, + _pad_w13_bias, + _pad_w13_shard_cols, + ) + + E, rows, N, padded_N = 2, 8, 96, 128 + + # w13 stores the two gate/up shards along the last dim; padding each shard + # must preserve the loaded values and zero the padded columns. + w13 = torch.arange(E * rows * 2 * N).reshape(E, rows, 2 * N).float() + padded = _pad_w13_shard_cols(w13, N, padded_N) + assert padded.shape == (E, rows, 2 * padded_N) + shards = padded.view(E, rows, 2, padded_N) + orig = w13.view(E, rows, 2, N) + assert torch.equal(shards[..., :N], orig) + assert shards[..., N:].abs().sum() == 0 + + # w2 stores the intermediate dim in the rows. + w2 = torch.ones(E, N // 32, 16) + padded = _pad_rows(w2, padded_N // 32) + assert padded.shape == (E, padded_N // 32, 16) + assert padded[:, N // 32 :, :].abs().sum() == 0 + + bias = torch.arange(E * 2 * N).reshape(E, 2 * N).float() + padded = _pad_w13_bias(bias, N, padded_N) + assert padded.shape == (E, 2 * padded_N) + bias_shards = padded.view(E, 2, padded_N) + assert torch.equal(bias_shards[..., :N], bias.view(E, 2, N)) + assert bias_shards[..., N:].abs().sum() == 0 + + +def _gpu_marlin_unsupported() -> bool: + return not ( + current_platform.is_cuda() and current_platform.has_device_capability(80) + ) + + +@pytest.mark.skipif( + _gpu_marlin_unsupported() or not is_fp8_marlin_supported(), + reason="FP8 Marlin is not supported on this GPU type.", +) +@pytest.mark.parametrize("shape", ODD_SHAPES) +@pytest.mark.parametrize("use_bias", [False, True]) +def test_fp8_marlin_padded_round_trip(shape, use_bias): + size_n, size_k = shape + dtype = torch.float16 + layer = torch.nn.Module() + layer.output_size_per_partition = size_n + layer.input_size_per_partition = size_k + layer.orig_dtype = dtype + + weight = torch.randn(size_k, size_n, dtype=dtype, device="cuda") / size_k**0.5 + scale = weight.abs().max() / 448 + weight_fp8 = (weight / scale).to(torch.float8_e4m3fn) + layer.weight = torch.nn.Parameter(weight_fp8, requires_grad=False) + layer.weight_scale = torch.nn.Parameter( + scale.to(torch.float32), requires_grad=False + ) + bias = None + if use_bias: + bias = torch.randn(size_n, dtype=dtype, device="cuda") + layer.bias = torch.nn.Parameter(bias.clone(), requires_grad=False) + + prepare_fp8_layer_for_marlin(layer, size_k_first=True) + + x = torch.randn(8, size_k, dtype=dtype, device="cuda") + output = apply_fp8_marlin_linear( + input=x, + weight=layer.weight, + weight_scale=layer.weight_scale, + workspace=layer.workspace, + size_n=size_n, + size_k=size_k, + bias=layer.bias if use_bias else None, + ) + ref = x @ (weight_fp8.to(dtype) * scale.to(dtype)) + if use_bias: + ref = ref + bias + + assert output.shape == (8, size_n) + torch.testing.assert_close(output, ref, rtol=2e-2, atol=2e-2) + + +def _dequant_fp4(packed: torch.Tensor, dtype: torch.dtype) -> torch.Tensor: + """Dequantize packed e2m1 nibbles (N, K // 2) -> (N, K) in dtype.""" + lo = (packed & 0b10000000) | ((packed & 0b01110000) >> 2) + lo = lo.view(torch.float8_e4m3fn).to(dtype) * (2**6) + hi_bits = packed << 4 + hi = (hi_bits & 0b10000000) | ((hi_bits & 0b01110000) >> 2) + hi = hi.view(torch.float8_e4m3fn).to(dtype) * (2**6) + return torch.cat([hi.unsqueeze(2), lo.unsqueeze(2)], 2).view(packed.size(0), -1) + + +@pytest.mark.skipif( + _gpu_marlin_unsupported() or not is_fp4_marlin_supported(), + reason="FP4 Marlin is not supported on this GPU type.", +) +@pytest.mark.parametrize("shape", ODD_SHAPES) +def test_nvfp4_marlin_padded_round_trip(shape): + size_n, size_k = shape + group_size = 16 + dtype = torch.float16 + layer = torch.nn.Module() + layer.output_size_per_partition = size_n + layer.input_size_per_partition = size_k + layer.params_dtype = dtype + + packed = torch.randint( + 0, 256, (size_n, size_k // 2), dtype=torch.uint8, device="cuda" + ) + scales = (torch.rand(size_n, size_k // group_size, device="cuda") + 0.25).to( + torch.float8_e4m3fn + ) + global_scale = torch.tensor([0.002], dtype=torch.float32, device="cuda") + + ref_weight = ( + _dequant_fp4(packed, dtype) + * scales.to(dtype).repeat_interleave(group_size, 1) + * global_scale.to(dtype) + ) + + layer.weight = torch.nn.Parameter(packed, requires_grad=False) + layer.weight_scale = torch.nn.Parameter(scales, requires_grad=False) + layer.weight_global_scale = torch.nn.Parameter(global_scale, requires_grad=False) + + prepare_fp4_layer_for_marlin(layer) + + x = torch.randn(8, size_k, dtype=dtype, device="cuda") / size_k**0.5 + output = apply_fp4_marlin_linear( + input=x, + weight=layer.weight, + weight_scale=layer.weight_scale, + weight_global_scale=layer.weight_global_scale, + workspace=layer.workspace, + size_n=size_n, + size_k=size_k, + ) + ref = x @ ref_weight.T + + assert output.shape == (8, size_n) + torch.testing.assert_close(output, ref, rtol=2e-2, atol=2e-2) + + +@pytest.mark.skipif( + _gpu_marlin_unsupported(), + reason="Marlin is not supported on this GPU type.", +) +@pytest.mark.parametrize("shape", ODD_SHAPES) +@pytest.mark.parametrize("group_size", [-1, 128]) +def test_gptq_marlin_padded_round_trip(shape, group_size): + """Pad-then-repack a GPTQ int4 weight the way MarlinLinearKernel does and + check the GEMM against the dequantized reference. + + Symmetric int4's quantized zero decodes to -8, so this exercises the + zero-padded-scales cancellation, not just zero weights. + """ + size_n, size_k = shape + if group_size > 0 and size_k % group_size != 0: + pytest.skip("group must divide the rank-local K (not fixable by padding)") + dtype = torch.float16 + quant_type = scalar_types.uint4b8 + device = torch.device("cuda") + + weight = torch.randn(size_k, size_n, dtype=dtype, device=device) / size_k**0.5 + w_ref, q_w, s, _, _ = gptq_quantize_weights( + weight, quant_type, group_size, act_order=False + ) + qweight = gptq_pack(q_w, quant_type.size_bits, size_k, size_n) + + padded_n, padded_k = marlin_padded_nk(size_n, size_k, group_size) + qweight = marlin_pad_qweight(qweight, size_n, size_k, padded_n, padded_k) + marlin_qweight = ops.gptq_marlin_repack( + b_q_weight=qweight, + perm=torch.empty(0, dtype=torch.int, device=device), + size_k=padded_k, + size_n=padded_n, + num_bits=quant_type.size_bits, + ) + s = marlin_pad_scales(s, size_n, size_k, padded_n, padded_k, group_size) + marlin_s = marlin_permute_scales( + s, size_k=padded_k, size_n=padded_n, group_size=group_size + ) + + x = torch.randn(8, size_k, dtype=dtype, device=device) + output = apply_gptq_marlin_linear( + input=x, + weight=marlin_qweight, + weight_scale=marlin_s, + weight_zp=marlin_make_empty_g_idx(device), + g_idx=marlin_make_empty_g_idx(device), + g_idx_sort_indices=marlin_make_empty_g_idx(device), + workspace=marlin_make_workspace_new(device), + wtype=quant_type, + output_size_per_partition=size_n, + input_size_per_partition=size_k, + is_k_full=True, + ) + ref = x @ w_ref + + assert output.shape == (8, size_n) + torch.testing.assert_close(output, ref, rtol=2e-2, atol=2e-2) + + +@pytest.mark.skipif( + _gpu_marlin_unsupported() or not is_fp8_marlin_supported(), + reason="FP8 Marlin is not supported on this GPU type.", +) +@pytest.mark.parametrize("shape", [(200, 512), (4640, 512)]) +def test_fp8_block_marlin_padded_round_trip(shape): + """Block-quantized FP8 (e.g. Nemotron NVFP4 checkpoints' FP8 layers): + group_size=128 exercises the lcm K-alignment in marlin_padded_nk and the + weight_scale_inv group-wise scale padding.""" + size_n, size_k = shape + block = 128 + dtype = torch.float16 + layer = torch.nn.Module() + layer.output_size_per_partition = size_n + layer.input_size_per_partition = size_k + layer.orig_dtype = dtype + layer.weight_block_size = [block, block] + + weight = torch.randn(size_n, size_k, dtype=dtype, device="cuda") / size_k**0.5 + n_blocks, k_blocks = (size_n + block - 1) // block, size_k // block + padded = torch.zeros(n_blocks * block, size_k, dtype=dtype, device="cuda") + padded[:size_n] = weight + scales = padded.view(n_blocks, block, k_blocks, block).abs().amax(dim=(1, 3)) / 448 + scales_expanded = scales.repeat_interleave(block, 0)[:size_n].repeat_interleave( + block, 1 + ) + weight_fp8 = (weight / scales_expanded).to(torch.float8_e4m3fn) + + layer.weight = torch.nn.Parameter(weight_fp8, requires_grad=False) + layer.weight_scale_inv = torch.nn.Parameter( + scales.to(torch.float32), requires_grad=False + ) + + prepare_fp8_layer_for_marlin(layer, size_k_first=False) + + x = torch.randn(8, size_k, dtype=dtype, device="cuda") + output = apply_fp8_marlin_linear( + input=x, + weight=layer.weight, + weight_scale=layer.weight_scale_inv, + workspace=layer.workspace, + size_n=size_n, + size_k=size_k, + bias=None, + ) + ref = x @ (weight_fp8.to(dtype) * scales_expanded.to(dtype)).T + + assert output.shape == (8, size_n) + torch.testing.assert_close(output, ref, rtol=2e-2, atol=2e-2) + + +@pytest.mark.skipif( + _gpu_marlin_unsupported() or not is_fp8_marlin_supported(), + reason="FP8 Marlin is not supported on this GPU type.", +) +@pytest.mark.parametrize("shape", [(200, 288), (4640, 512)]) +def test_mxfp8_marlin_padded_round_trip(shape): + """MXFP8 exercises the e8m0 scale path, where padded 0.0 scales clamp to + 2^-127 instead of zero and must still contribute nothing.""" + size_n, size_k = shape + group_size = 32 + # The e8m0-scale Marlin kernels are only instantiated for bf16 activations. + dtype = torch.bfloat16 + layer = torch.nn.Module() + layer.output_size_per_partition = size_n + layer.input_size_per_partition = size_k + + weight_fp8 = (torch.randn(size_n, size_k, dtype=dtype, device="cuda") / 4).to( + torch.float8_e4m3fn + ) + # e8m0 exponents around 1.0 (127): scales in [2^-6, 2^0] + scales = torch.randint( + 121, 128, (size_n, size_k // group_size), dtype=torch.uint8, device="cuda" + ) + ref_weight = weight_fp8.to(dtype) * ( + 2.0 ** (scales.to(dtype) - 127) + ).repeat_interleave(group_size, 1) + + layer.weight = torch.nn.Parameter(weight_fp8, requires_grad=False) + layer.weight_scale = torch.nn.Parameter(scales, requires_grad=False) + + prepare_mxfp8_layer_for_marlin(layer) + + x = torch.randn(8, size_k, dtype=dtype, device="cuda") / size_k**0.5 + output = apply_mxfp8_marlin_linear( + input=x, + weight=layer.weight, + weight_scale=layer.weight_scale, + workspace=layer.workspace, + size_n=size_n, + size_k=size_k, + ) + ref = x @ ref_weight.T + + assert output.shape == (8, size_n) + torch.testing.assert_close(output, ref, rtol=2e-2, atol=2e-2) + + +@pytest.mark.skipif( + _gpu_marlin_unsupported(), + reason="Marlin is not supported on this GPU type.", +) +@pytest.mark.parametrize("shape", [(200, 512), (4640, 512)]) +def test_awq_zp_marlin_padded_round_trip(shape): + """AWQ-style uint4 with runtime zero-points, padded the way + MarlinLinearKernel does: padded columns rely on (q=0 - zp=0) * scale=0.""" + size_n, size_k = shape + group_size = 128 + dtype = torch.float16 + quant_type = scalar_types.uint4 + device = torch.device("cuda") + + weight = torch.randn(size_k, size_n, dtype=dtype, device=device) / size_k**0.5 + w_ref, q_w, s, zp = quantize_weights( + weight, quant_type, group_size, zero_points=True + ) + qweight = gptq_pack(q_w, quant_type.size_bits, size_k, size_n) + + padded_n, padded_k = marlin_padded_nk(size_n, size_k, group_size) + qweight = marlin_pad_qweight(qweight, size_n, size_k, padded_n, padded_k) + marlin_qweight = ops.gptq_marlin_repack( + b_q_weight=qweight, + perm=torch.empty(0, dtype=torch.int, device=device), + size_k=padded_k, + size_n=padded_n, + num_bits=quant_type.size_bits, + ) + s = marlin_pad_scales(s, size_n, size_k, padded_n, padded_k, group_size) + marlin_s = marlin_permute_scales( + s, size_k=padded_k, size_n=padded_n, group_size=group_size + ) + zp = marlin_pad_scales(zp, size_n, size_k, padded_n, padded_k, group_size) + marlin_zp = marlin_zero_points( + zp, + size_k=padded_k // group_size, + size_n=padded_n, + num_bits=quant_type.size_bits, + ) + + x = torch.randn(8, size_k, dtype=dtype, device=device) + output = apply_gptq_marlin_linear( + input=x, + weight=marlin_qweight, + weight_scale=marlin_s, + weight_zp=marlin_zp, + g_idx=marlin_make_empty_g_idx(device), + g_idx_sort_indices=marlin_make_empty_g_idx(device), + workspace=marlin_make_workspace_new(device), + wtype=quant_type, + output_size_per_partition=size_n, + input_size_per_partition=size_k, + is_k_full=True, + ) + ref = x @ w_ref + + assert output.shape == (8, size_n) + torch.testing.assert_close(output, ref, rtol=2e-2, atol=2e-2) + + +class _FakeLinear: + def __init__(self, size_n, size_k, input_size=None): + self.output_size_per_partition = size_n + self.input_size_per_partition = size_k + self.output_size = size_n + self.input_size = input_size if input_size is not None else size_k + + +def test_check_marlin_supports_layer_allow_tile_padding(): + from vllm.model_executor.layers.quantization.utils.marlin_utils import ( + check_marlin_supports_layer, + ) + + # Tile-misaligned but group-aligned: rejected strictly, allowed w/ padding + layer = _FakeLinear(4640, 512, input_size=2048) + assert not check_marlin_supports_layer(layer, 128) + assert check_marlin_supports_layer(layer, 128, allow_tile_padding=True) + assert check_marlin_supports_layer(layer, -1, allow_tile_padding=True) + + # A group straddling the TP shard cannot be fixed by padding + layer = _FakeLinear(4608, 4672, input_size=18688) + assert not check_marlin_supports_layer(layer, 128, allow_tile_padding=True) + + +@pytest.mark.skipif( + _gpu_marlin_unsupported(), + reason="Marlin is not supported on this GPU type.", +) +@pytest.mark.parametrize("group_size", [-1, 32]) +@pytest.mark.parametrize("shape", [(96, 256, 8), (160, 512, 4)]) +def test_gptq_marlin_moe_padded_round_trip(shape, group_size): + """Pad a tile-misaligned MoE intermediate the way the WNA16 Marlin MoE prep + does, run the real repack + fused_marlin_moe, and check against the + dequantized reference. Symmetric int4's quantized zero decodes to -8, so the + padded region only stays out of the output via the zero-padded scales. + """ + from tests.kernels.utils import torch_experts + from vllm.config import VllmConfig, set_current_vllm_config + from vllm.model_executor.layers.fused_moe import fused_topk + from vllm.model_executor.layers.fused_moe.experts.marlin_moe import ( + fused_marlin_moe, + ) + from vllm.model_executor.layers.fused_moe.oracle.int_wna16 import ( + _pad_rows, + _pad_w13_shard_cols, + ) + from vllm.model_executor.layers.quantization.utils.marlin_utils import ( + marlin_moe_padded_intermediate, + marlin_moe_permute_scales, + ) + + n, k, e = shape + topk, m = 2, 33 + padded_n = marlin_moe_padded_intermediate(n, group_size) + assert padded_n != n, "test should exercise padding" + + dtype = torch.float16 + device = torch.device("cuda") + quant_type = scalar_types.uint4b8 + bits = quant_type.size_bits + pack = 32 // bits + + a = torch.randn((m, k), device=device, dtype=dtype) / 10 + w1 = torch.randn((e, 2 * n, k), device=device, dtype=dtype) / k**0.5 + w2 = torch.randn((e, k, n), device=device, dtype=dtype) / n**0.5 + + def quant(w, size_k, size_n): + # w is (size_n, size_k); gptq expects (size_k, size_n). + ref, q_w, s, _, _ = gptq_quantize_weights( + w.T, quant_type, group_size, act_order=False + ) + return ref, gptq_pack(q_w, bits, size_k, size_n), s + + w13_qw, w13_s, w13_ref = [], [], [] + w2_qw, w2_s, w2_ref = [], [], [] + for i in range(e): + ref, qw, s = quant(w1[i], k, 2 * n) + w13_ref.append(ref.T) # (2n, k) + w13_qw.append(qw) + w13_s.append(s) + ref, qw, s = quant(w2[i], n, k) + w2_ref.append(ref.T) # (k, n) + w2_qw.append(qw) + w2_s.append(s) + + w13_qweight = torch.stack(w13_qw) + w2_qweight = torch.stack(w2_qw) + w13_scales = torch.stack(w13_s) + w2_scales = torch.stack(w2_s) + w1_ref = torch.stack(w13_ref) # (e, 2n, k) + w2_ref = torch.stack(w2_ref) # (e, k, n) + + # Pad the intermediate via the production helpers. + w13_qweight = _pad_w13_shard_cols(w13_qweight, n, padded_n) + w2_qweight = _pad_rows(w2_qweight, padded_n // pack) + w13_scales = _pad_w13_shard_cols(w13_scales, n, padded_n) + if group_size > 0: + w2_scales = _pad_rows(w2_scales, padded_n // group_size) + + sort_idx = torch.empty((e, 0), dtype=torch.int32, device=device) + marlin_w13 = ops.gptq_marlin_moe_repack( + w13_qweight, sort_idx, w13_qweight.shape[1] * pack, w13_qweight.shape[2], bits + ) + marlin_w2 = ops.gptq_marlin_moe_repack( + w2_qweight, sort_idx, w2_qweight.shape[1] * pack, w2_qweight.shape[2], bits + ) + group_or_pack = group_size if group_size != -1 else pack + marlin_w13_s = marlin_moe_permute_scales( + s=w13_scales, size_k=n, size_n=w13_scales.shape[2], group_size=group_size + ) + marlin_w2_s = marlin_moe_permute_scales( + s=w2_scales, + size_k=w2_scales.shape[1] * group_or_pack, + size_n=w2_scales.shape[2], + group_size=group_size, + ) + + score = torch.randn((m, e), device=device, dtype=dtype) + topk_weights, topk_ids, _ = fused_topk(a, score, topk, False) + + marlin_out = fused_marlin_moe( + a, + marlin_w13, + marlin_w2, + None, + None, + marlin_w13_s, + marlin_w2_s, + topk_weights, + topk_ids, + quant_type_id=quant_type.id, + global_num_experts=e, + is_k_full=True, + ) + with set_current_vllm_config(VllmConfig()): + ref = torch_experts( + a, + w1_ref, + w2_ref, + topk_weight=topk_weights, + topk_ids=topk_ids, + global_num_experts=e, + ) + + torch.testing.assert_close(marlin_out, ref, atol=5e-2, rtol=0) + + +@pytest.mark.skipif( + current_platform.is_rocm(), + reason="MoE Marlin is not selected on ROCm.", +) +def test_check_moe_marlin_supports_layer_padding(): + from vllm.model_executor.layers.quantization.utils.marlin_utils import ( + check_moe_marlin_supports_layer, + ) + + def make_layer(hidden, intermediate): + layer = SimpleNamespace() + layer.hidden_size = hidden + layer.apply_router_weight_on_input = False + layer.moe_config = SimpleNamespace( + intermediate_size_per_partition_unpadded=intermediate + ) + return layer + + # group=32 with intermediate % 64 != 0: rejected strictly, accepted w/ padding + layer = make_layer(4096, 96) + assert not check_moe_marlin_supports_layer(layer, 32) + assert check_moe_marlin_supports_layer(layer, 32, allow_tile_padding=True) + # channelwise misaligned intermediate is paddable + assert check_moe_marlin_supports_layer(layer, -1, allow_tile_padding=True) + + # A group straddling the boundary cannot be fixed by padding + layer = make_layer(4096, 176) + assert not check_moe_marlin_supports_layer(layer, 128, allow_tile_padding=True) + + # hidden_size is the MoE I/O extent and is never padded + layer = make_layer(4090, 128) + assert not check_moe_marlin_supports_layer(layer, 64, allow_tile_padding=True) + + +@pytest.mark.skipif( + _gpu_marlin_unsupported() or not is_fp8_marlin_supported(), + reason="FP8 Marlin is not supported on this GPU type.", +) +@pytest.mark.parametrize("quant", ["channel", "tensor"]) +@pytest.mark.parametrize("shape", [(96, 256, 8), (160, 512, 4)]) +def test_fp8_marlin_moe_padded_round_trip(shape, quant): + """FP8 weight-only MoE: pad a tile-misaligned intermediate and check the + real prepare + fused_marlin_moe against the dequantized reference.""" + from tests.kernels.utils import torch_experts + from vllm.config import VllmConfig, set_current_vllm_config + from vllm.model_executor.layers.fused_moe import fused_topk + from vllm.model_executor.layers.fused_moe.experts.marlin_moe import ( + fused_marlin_moe, + ) + from vllm.model_executor.layers.quantization.utils.marlin_utils import ( + marlin_moe_intermediate_size, + marlin_moe_padded_intermediate, + ) + from vllm.model_executor.layers.quantization.utils.marlin_utils_fp8 import ( + prepare_fp8_moe_layer_for_marlin, + ) + + n, k, e = shape + topk, m = 2, 33 + fp8 = torch.float8_e4m3fn + dtype = torch.bfloat16 + device = torch.device("cuda") + padded_n = marlin_moe_padded_intermediate(n, -1) + assert padded_n != n + + def q(w): # (out, in) -> fp8 weight, scale, dequant reference + dim = None if quant == "tensor" else 1 + s = (w.abs().amax(dim, keepdim=dim is not None) / 448.0).clamp(min=1e-8) + wq = (w / s).clamp(-448, 448).to(fp8) + ref = wq.to(dtype) * s.to(dtype) + s = s.reshape(1) if quant == "tensor" else s.squeeze(1) + return wq, s, ref + + a = torch.randn((m, k), device=device, dtype=dtype) / 10 + w1 = torch.randn((e, 2 * n, k), device=device, dtype=dtype) / k**0.5 + w2 = torch.randn((e, k, n), device=device, dtype=dtype) / n**0.5 + w13_q, w13_s, w1_ref = zip(*(q(w1[i]) for i in range(e))) + w2_q, w2_s, w2_ref = zip(*(q(w2[i]) for i in range(e))) + + w13_weight, w2_weight = torch.stack(w13_q), torch.stack(w2_q) + layer = SimpleNamespace( + num_experts=e, + hidden_size=k, + intermediate_size_per_partition=n, + orig_dtype=dtype, + w13_weight=w13_weight, + ) + pw13, pw2, ps13, ps2 = prepare_fp8_moe_layer_for_marlin( + layer, w13_weight, w2_weight, torch.stack(w13_s), torch.stack(w2_s) + ) + assert marlin_moe_intermediate_size(pw13, pw2) == padded_n + + score = torch.randn((m, e), device=device, dtype=dtype) + topk_weights, topk_ids, _ = fused_topk(a, score, topk, False) + out = fused_marlin_moe( + a, + pw13, + pw2, + None, + None, + ps13, + ps2, + topk_weights, + topk_ids, + quant_type_id=scalar_types.float8_e4m3fn.id, + global_num_experts=e, + is_k_full=True, + workspace=layer.workspace, + ) + with set_current_vllm_config(VllmConfig()): + ref = torch_experts( + a, + torch.stack(w1_ref), + torch.stack(w2_ref), + topk_weight=topk_weights, + topk_ids=topk_ids, + global_num_experts=e, + ) + torch.testing.assert_close(out, ref, atol=8e-2, rtol=0) + + +@pytest.mark.skipif( + _gpu_marlin_unsupported() or not is_fp8_marlin_supported(), + reason="FP8 Marlin is not supported on this GPU type.", +) +@pytest.mark.parametrize("shape", [(96, 256, 8), (160, 512, 4)]) +def test_mxfp8_marlin_moe_padded_round_trip(shape): + """MXFP8 weight-only MoE round-trip at a tile-misaligned intermediate, with + unit e8m0 scales so the reference is the exact fp8 dequant.""" + from tests.kernels.utils import torch_experts + from vllm.config import VllmConfig, set_current_vllm_config + from vllm.model_executor.layers.fused_moe import fused_topk + from vllm.model_executor.layers.fused_moe.experts.marlin_moe import ( + fused_marlin_moe, + ) + from vllm.model_executor.layers.quantization.utils.marlin_utils import ( + marlin_moe_intermediate_size, + marlin_moe_padded_intermediate, + ) + from vllm.model_executor.layers.quantization.utils.marlin_utils_fp8 import ( + prepare_mxfp8_moe_layer_for_marlin, + ) + + n, k, e = shape + topk, m, gs, e8m0_one = 2, 33, 32, 127 + fp8 = torch.float8_e4m3fn + dtype = torch.bfloat16 + device = torch.device("cuda") + padded_n = marlin_moe_padded_intermediate(n, gs) + assert padded_n != n + + a = torch.randn((m, k), device=device, dtype=dtype) / 10 + w13_weight = torch.randn((e, 2 * n, k), device=device, dtype=dtype) / k**0.5 + w2_weight = torch.randn((e, k, n), device=device, dtype=dtype) / n**0.5 + w13_weight = w13_weight.clamp(-448, 448).to(fp8) + w2_weight = w2_weight.clamp(-448, 448).to(fp8) + w13_scale = torch.full( + (e, 2 * n, k // gs), e8m0_one, dtype=torch.uint8, device=device + ) + w2_scale = torch.full((e, k, n // gs), e8m0_one, dtype=torch.uint8, device=device) + + layer = SimpleNamespace( + num_experts=e, hidden_size=k, intermediate_size_per_partition=n + ) + with set_current_vllm_config(VllmConfig()): + pw13, pw2, ps13, ps2 = prepare_mxfp8_moe_layer_for_marlin( + layer, w13_weight, w2_weight, w13_scale, w2_scale + ) + assert marlin_moe_intermediate_size(pw13, pw2) == padded_n + + score = torch.randn((m, e), device=device, dtype=dtype) + topk_weights, topk_ids, _ = fused_topk(a, score, topk, False) + out = fused_marlin_moe( + a, + pw13, + pw2, + None, + None, + ps13, + ps2, + topk_weights, + topk_ids, + quant_type_id=scalar_types.float8_e4m3fn.id, + global_num_experts=e, + is_k_full=True, + workspace=layer.workspace, + ) + with set_current_vllm_config(VllmConfig()): + ref = torch_experts( + a, + w13_weight.to(dtype), + w2_weight.to(dtype), + topk_weight=topk_weights, + topk_ids=topk_ids, + global_num_experts=e, + ) + torch.testing.assert_close(out, ref, atol=8e-2, rtol=0) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_mxfp4_qutlass.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_mxfp4_qutlass.py new file mode 100644 index 0000000000000000000000000000000000000000..f8c970baf3eca6b548a1647d1c094fd258a022a9 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_mxfp4_qutlass.py @@ -0,0 +1,304 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# +# Copyright (C) 2025 Roberto L. Castro (Roberto.LopezCastro@ist.ac.at). +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +import numpy as np +import pytest +import torch +from compressed_tensors.transform.utils.hadamard import deterministic_hadamard_matrix + +from vllm._custom_ops import fusedQuantizeMx, matmul_mxf4_bf16_tn +from vllm.model_executor.layers.quantization.qutlass_utils import to_blocked +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +if not torch.cuda.is_available(): + pytest.skip("CUDA required for these tests.", allow_module_level=True) + +if not ( + current_platform.has_device_capability(100) + or current_platform.has_device_capability(120) +): + pytest.skip( + reason="Tests require compute capability 10.0 (100) or 12.0 (120).", + allow_module_level=True, + ) + + +# ----- Helpers ----- +def get_hadamard_matrix(group_size: int, dtype: torch.dtype, device: torch.device): + return ( + deterministic_hadamard_matrix(group_size, dtype=dtype, device=device) + * group_size**-0.5 + ) + + +def _rtne_fp4(x: torch.Tensor): + device = x.device + grid = torch.tensor( + [ + -6.0, + -4.0, + -3.0, + -2.0, + -1.5, + -1.0, + -0.5, + -0.0, + 0.0, + 0.5, + 1.0, + 1.5, + 2.0, + 3.0, + 4.0, + 6.0, + ], + dtype=x.dtype, + device=x.device, + ) + grid_int = torch.tensor( + [-1, -2, -3, -4, -5, -6, -7, -8, 0, 1, 2, 3, 4, 5, 6, 7], + dtype=torch.uint8, + device=device, + ) + inds = torch.bucketize(x, grid) + lo, hi = (inds - 1).clamp(min=0, max=15), inds.clamp(min=0, max=15) + g_lo, g_hi = grid[lo], grid[hi] + pick_hi = (g_hi - x < x - g_lo) | (g_hi - x == x - g_lo) & (grid_int[hi] % 2 == 0) + y = torch.where(pick_hi, g_hi, g_lo) + y_int = torch.where(pick_hi, grid_int[hi], grid_int[lo]) + y_int_packed = (y_int[..., 1::2] & 0xF) << 4 | y_int[..., ::2] & 0xF + return y, y_int_packed + + +def _dq_fp4(x_e2m1: torch.Tensor, x_e8m0: torch.Tensor, alpha: float): + device = x_e2m1.device + + x_e2m1_i32 = x_e2m1.view(dtype=torch.uint8).to(dtype=torch.int32) + x_e2m1_unpacked = torch.stack( + [x_e2m1_i32 & 0xF, (x_e2m1_i32 >> 4) & 0xF], dim=-1 + ).flatten(start_dim=-2) + + grid_dq = torch.tensor( + [ + 0.0, + 0.5, + 1.0, + 1.5, + 2.0, + 3.0, + 4.0, + 6.0, + -0.0, + -0.5, + -1.0, + -1.5, + -2.0, + -3.0, + -4.0, + -6.0, + ], + dtype=torch.float64, + device=device, + ) + x_fp4_dq = grid_dq[x_e2m1_unpacked] + scales_dq = x_e8m0.to(torch.float64) + + x_dq = (x_fp4_dq.unflatten(dim=-1, sizes=(-1, 32)) * scales_dq[..., None]).flatten( + start_dim=-2 + ) / alpha + return x_dq, x_fp4_dq, scales_dq + + +def _unpack_mask(clip_mask: torch.Tensor) -> torch.Tensor: + clip_mask_unpacked_dq = torch.zeros( + *clip_mask.shape[:-1], + clip_mask.size(-1) * 8, + dtype=torch.bool, + device=clip_mask.device, + ) + for i in range(8): + clip_mask_unpacked_dq[..., i::8] = (clip_mask >> i) & 1 + return clip_mask_unpacked_dq + + +def _forward_quantize_ref( + x: torch.Tensor, h: torch.Tensor, rot_size: int, quest: bool = True +): + device = x.device + xh_ref64 = ( + x.unflatten(dim=-1, sizes=(-1, rot_size)).to(dtype=torch.float64) + @ h.reshape(rot_size, rot_size).to(dtype=torch.float64) + ).flatten(start_dim=-2) + + if quest: + scales_ref64_ = ( + xh_ref64.unflatten(dim=-1, sizes=(-1, 32)).std(dim=-1, correction=0) + * (2.92247856 / 6.0) + + 1e-8 + ) + else: + abs_max = xh_ref64.unflatten(dim=-1, sizes=(-1, 32)).abs().amax(dim=-1) + scales_ref64_ = abs_max + 1e-8 + + xh_e8m0_ref = scales_ref64_.log2().floor().exp2().to(dtype=torch.float8_e8m0fnu) + scales_ref64 = xh_e8m0_ref.to(dtype=torch.float64) + + xh_scaled_ref64 = ( + xh_ref64.unflatten(dim=-1, sizes=(-1, 32)) / scales_ref64[..., None] + ).flatten(start_dim=-2) + if not quest: + xh_scaled_ref64 *= 3 + + clip_mask_unpacked_ref = xh_scaled_ref64.abs() < 6.0 + clip_mask_ref = torch.zeros( + *x.shape[:-1], x.size(-1) // 8, dtype=torch.uint8, device=device + ) + for i in range(8): + clip_mask_ref |= clip_mask_unpacked_ref[..., i::8].to(dtype=torch.uint8) << i + + xh_fp4_ref, xh_e2m1_ref = _rtne_fp4(xh_scaled_ref64) + xh_dq, xh_fp4_dq, scales_dq = _dq_fp4( + xh_e2m1_ref, xh_e8m0_ref, alpha=1.0 if quest else 3.0 + ) + clip_mask_unpacked_dq = _unpack_mask(clip_mask_ref) + + assert xh_fp4_dq.equal(xh_fp4_ref) + assert scales_dq.equal(scales_ref64) + assert clip_mask_unpacked_dq.equal(clip_mask_unpacked_ref) + + return ( + xh_dq, + clip_mask_unpacked_ref, + (xh_e2m1_ref, xh_e8m0_ref, clip_mask_ref), + ) + + +DTYPE = torch.bfloat16 +DEVICE = torch.device("cuda:0") + +ROT_SIZES = [32, 64, 128] +SEEDS = [0] +BATCHES = [1, 16] + +LLAMA_MODELS = { + "7B": [(4096, 3 * 4096), (4096, 4096), (4096, 2 * 10752), (10752, 4096)], + "13B": [(5120, 3 * 5120), (5120, 5120), (5120, 2 * 13568), (13568, 5120)], + "33B": [(6656, 3 * 6656), (6656, 6656), (6656, 2 * 17664), (17664, 6656)], + "70B": [(8192, 3 * 8192), (8192, 8192), (8192, 2 * 21760), (21760, 8192)], +} + + +@pytest.fixture(autouse=True) +def _seed_each_test(): + set_random_seed(0) + np.random.seed(0) + torch.random.manual_seed(0) + + +@pytest.mark.parametrize("rot_size", ROT_SIZES) +@torch.inference_mode() +def test_fused_quantization_absmax(rot_size: int): + dtype, device = DTYPE, DEVICE + h = get_hadamard_matrix(rot_size, dtype, device) + x = torch.randn(2, 4096, 4096, dtype=dtype, device=device) * 25.0 + + xh_dq_ref, _, _ = _forward_quantize_ref(x, h, rot_size, quest=False) + xh_e2m1, xh_e8m0 = fusedQuantizeMx(x, h, method="abs_max") + xh_e8m0 = xh_e8m0.reshape(2, 4096, 4096 // 32) + xh_dq, *_ = _dq_fp4(xh_e2m1, xh_e8m0, alpha=3.0) + + torch.testing.assert_close(xh_dq, xh_dq_ref, rtol=0.34, atol=100) + assert (xh_dq != xh_dq_ref).float().mean() <= 1e-4 + + m, n, k = 1, 504, 4096 + a = torch.randn(m, k, dtype=dtype, device=device) * 25.0 + b = torch.randn(n, k, dtype=dtype, device=device) * 25.0 + + a_e2m1, a_e8m0 = fusedQuantizeMx(a, h, method="abs_max") + b_e2m1, b_e8m0 = fusedQuantizeMx(b, h, method="abs_max") + a_dq, *_ = _dq_fp4(a_e2m1, a_e8m0[:m, :k], alpha=1.0) + b_dq, *_ = _dq_fp4(b_e2m1, b_e8m0[:n, :k], alpha=1.0) + out_ref = a_dq @ b_dq.transpose(-2, -1) + + a_scale_block = to_blocked(a_e8m0, backend="triton") + b_scale_block = to_blocked(b_e8m0, backend="triton") + alpha = torch.tensor([1.0], device=device) + out = matmul_mxf4_bf16_tn(a_e2m1, b_e2m1, a_scale_block, b_scale_block, alpha) + assert out.equal(out_ref.to(dtype=out.dtype)) + + +@pytest.mark.parametrize("rot_size", ROT_SIZES) +@torch.inference_mode() +def test_fused_quantization_quest(rot_size: int): + dtype, device = DTYPE, DEVICE + h = get_hadamard_matrix(rot_size, dtype, device) + x = torch.randn(2, 4096, 4096, dtype=dtype, device=device) * 25.0 + + xh_dq_ref, _, _ = _forward_quantize_ref(x, h, rot_size, quest=True) + xh_e2m1, xh_e8m0 = fusedQuantizeMx(x, h, method="quest") + xh_e8m0 = xh_e8m0.reshape(2, 4096, 4096 // 32) + xh_dq, *_ = _dq_fp4(xh_e2m1, xh_e8m0, alpha=1.0) + + torch.testing.assert_close(xh_dq, xh_dq_ref, rtol=0.34, atol=100) + assert (xh_dq != xh_dq_ref).float().mean() <= 1e-4 + + m, n, k = 504, 504, 2048 + a = torch.randn(m, k, dtype=dtype, device=device) * 25.0 + b = torch.randn(n, k, dtype=dtype, device=device) * 25.0 + + a_e2m1, a_e8m0 = fusedQuantizeMx(a, h, method="quest") + b_e2m1, b_e8m0 = fusedQuantizeMx(b, h, method="quest") + a_dq, *_ = _dq_fp4(a_e2m1, a_e8m0[:m, :k], alpha=1.0) + b_dq, *_ = _dq_fp4(b_e2m1, b_e8m0[:n, :k], alpha=1.0) + out_ref = a_dq @ b_dq.transpose(-2, -1) + + a_scale_block = to_blocked(a_e8m0, backend="triton") + b_scale_block = to_blocked(b_e8m0, backend="triton") + alpha = torch.tensor([1.0], device=device) + out = matmul_mxf4_bf16_tn(a_e2m1, b_e2m1, a_scale_block, b_scale_block, alpha) + assert out.equal(out_ref.to(dtype=out.dtype)) + + +@pytest.mark.parametrize("model", list(LLAMA_MODELS.keys())) +@pytest.mark.parametrize("layer_idx", [0, 1, 2, 3]) +@pytest.mark.parametrize("batch", [1, 16]) +@pytest.mark.parametrize("had_size", ROT_SIZES) +@torch.inference_mode() +def test_llama_shapes(model: str, layer_idx: int, batch: int, had_size: int): + dtype, device = DTYPE, DEVICE + m = batch + k, n = LLAMA_MODELS[model][layer_idx] + + h = get_hadamard_matrix(had_size, dtype, device) + + a = torch.rand(m, k, dtype=dtype, device=device) * 25.0 + b = torch.rand(n, k, dtype=dtype, device=device) * 25.0 + + a_e2m1, a_e8m0 = fusedQuantizeMx(a, h, method="quest") + b_e2m1, b_e8m0 = fusedQuantizeMx(b, h, method="quest") + + a_dq, *_ = _dq_fp4(a_e2m1, a_e8m0[:m, :k], alpha=1.0) + b_dq, *_ = _dq_fp4(b_e2m1, b_e8m0[:n, :k], alpha=1.0) + out_ref = a_dq @ b_dq.transpose(-2, -1) + + a_scale_block = to_blocked(a_e8m0, backend="triton") + b_scale_block = to_blocked(b_e8m0, backend="triton") + alpha = torch.tensor([1.0], device=device) + out = matmul_mxf4_bf16_tn(a_e2m1, b_e2m1, a_scale_block, b_scale_block, alpha) + assert out.equal(out_ref.to(dtype=out.dtype)) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_mxfp4_triton_ep.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_mxfp4_triton_ep.py new file mode 100644 index 0000000000000000000000000000000000000000..85e7dca6dcb5356bfd2a6d2a9b6c57a925f2f211 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_mxfp4_triton_ep.py @@ -0,0 +1,83 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Tests that triton_kernel_moe_forward correctly applies expert_map +remapping when expert parallelism (EP) is enabled. + +When expert_map is provided, global expert IDs are remapped to local IDs +via topk + expert_map remap + make_routing_data before building routing +structures, and the expert_map passed downstream to triton_kernel_fused_experts +is None (already applied). +""" + +from unittest.mock import MagicMock, patch + +import torch + + +class TestTritonMoeForwardExpertMap: + """Test that triton_kernel_moe_forward applies expert_map remapping + when expert_map is provided (EP active).""" + + def test_expert_map_remap(self): + device = "cuda" if torch.cuda.is_available() else "cpu" + mock_expert_map = torch.tensor([0, -1, 1, -1], device=device) + + from vllm.utils.import_utils import import_triton_kernels + + import_triton_kernels() + + mock_routing_data = MagicMock() + mock_gather = MagicMock() + mock_scatter = MagicMock() + + with ( + patch("triton_kernels.topk.topk") as mock_topk, + patch( + "vllm.model_executor.layers.fused_moe.experts." + "gpt_oss_triton_kernels_moe.make_routing_data" + ) as mock_make_routing, + patch( + "vllm.model_executor.layers.fused_moe.experts." + "gpt_oss_triton_kernels_moe.triton_kernel_fused_experts" + ) as mock_fused_experts, + ): + from vllm.model_executor.layers.fused_moe.experts.gpt_oss_triton_kernels_moe import ( # noqa: E501 + triton_kernel_moe_forward, + ) + + sparse_result = MagicMock() + sparse_result.indx = torch.tensor([[0, 2]], dtype=torch.int32) + sparse_result.vals = torch.tensor([[0.6, 0.4]]) + mock_topk.return_value = sparse_result + + mock_make_routing.return_value = ( + mock_routing_data, + mock_gather, + mock_scatter, + ) + + mock_fused_experts.return_value = torch.zeros((1, 8), device=device) + + hidden = torch.randn((1, 8), device=device) + w1 = torch.randn((2, 8, 16), device=device) + w2 = torch.randn((2, 8, 8), device=device) + logits = torch.randn((1, 4), device=device) + + triton_kernel_moe_forward( + hidden_states=hidden, + w1=w1, + w2=w2, + gating_output=logits, + topk=2, + renormalize=True, + expert_map=mock_expert_map, + ) + + mock_topk.assert_called_once() + mock_make_routing.assert_called_once() + + # expert_map should be None in the fused_experts call + # (already applied). + call_kwargs = mock_fused_experts.call_args + assert call_kwargs[1].get("expert_map") is None or (len(call_kwargs[0]) > 0) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_nvfp4_emulation.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_nvfp4_emulation.py new file mode 100644 index 0000000000000000000000000000000000000000..a6fd90cfaaae4507c21f9f0e8c5fd24b07b556cf --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_nvfp4_emulation.py @@ -0,0 +1,772 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from typing import cast + +import huggingface_hub +import pytest +import torch +from safetensors import safe_open + +import vllm.model_executor.layers.fused_moe.modular_kernel as mk +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.config import ( + FusedMoEConfig, + FusedMoEParallelConfig, + FusedMoEQuantConfig, + RoutingMethodType, + nvfp4_moe_quant_config, +) +from vllm.model_executor.layers.fused_moe.experts.nvfp4_emulation_moe import ( + Nvfp4QuantizationEmulationTritonExperts, +) +from vllm.model_executor.layers.fused_moe.experts.triton_moe import TritonExperts +from vllm.model_executor.layers.quantization.utils import ( + nvfp4_emulation_utils, +) +from vllm.model_executor.layers.quantization.utils.nvfp4_emulation_utils import ( + dequantize_to_dtype, + ref_nvfp4_quant_dequant, +) +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + QuantKey, + kNvfp4Dynamic, + kNvfp4Static, +) +from vllm.platforms import current_platform +from vllm.triton_utils import triton + +if current_platform.is_rocm(): + from vllm.platforms.rocm import on_gfx950 +else: + + def on_gfx950() -> bool: + return False + + +MOE_MODEL_CONFIGS = { + "nvidia/Qwen3-30B-A3B-NVFP4": { + "shards": ["model-00001-of-00004.safetensors"], + "expert_prefix": "model.layers.9.mlp.experts.", + # Position of the expert index in the dot-split key. + "expert_idx_pos": 5, + } +} + + +@pytest.fixture(scope="module") +def loaded_model_files(): + return { + model_id: huggingface_hub.snapshot_download( + repo_id=model_id, allow_patterns=config["shards"] + ) + for model_id, config in MOE_MODEL_CONFIGS.items() + } + + +class Nvfp4QuantizationEmulationTritonExpertsReference(TritonExperts): + """ + Extension of TritonExperts to support emulated NVFP4 MoE experts. + + It may be used for NVFP4 models when the device does not have + native support for this dtype. + """ + + def __init__( + self, + moe_config: FusedMoEConfig, + quant_config: FusedMoEQuantConfig, + ): + super().__init__(moe_config, quant_config) + + # `TritonExperts.apply` expects pre-dequantized weights, + # which we handle in `apply` below. + self.w1_scale_val = self.quant_config.w1_scale + self.w2_scale_val = self.quant_config.w2_scale + + self.quant_config._w1.scale = None + self.quant_config._w2.scale = None + + self.quantization_emulation = True + + @property + def quant_dtype(self) -> torch.dtype | str | None: + return "nvfp4" + + @property + def a1_scale(self) -> torch.Tensor | None: + return self.quant_config.a1_gscale + + @property + def expects_unquantized_inputs(self) -> bool: + return True + + @staticmethod + def _supports_quant_scheme( + weight_key: QuantKey | None, + activation_key: QuantKey | None, + ) -> bool: + return (weight_key, activation_key) == (kNvfp4Static, kNvfp4Dynamic) + + def apply( + self, + output: torch.Tensor, + hidden_states: torch.Tensor, + w1: torch.Tensor, + w2: torch.Tensor, + topk_weights: torch.Tensor, + topk_ids: torch.Tensor, + activation: MoEActivation, + global_num_experts: int, + expert_map: torch.Tensor | None, + a1q_scale: torch.Tensor | None, + a2_scale: torch.Tensor | None, + workspace13: torch.Tensor, + workspace2: torch.Tensor, + expert_tokens_meta: mk.ExpertTokensMetadata | None, + apply_router_weight_on_input: bool, + ): + assert w1.dtype == torch.uint8 + assert w2.dtype == torch.uint8 + + # Dequantize w1 from packed NVFP4 to fp16/bf16 + w13_global_scale = self.quant_config.g1_alphas + + w1_dequant = dequantize_to_dtype( + tensor_fp4=w1, + tensor_sf=self.w1_scale_val, + global_scale=w13_global_scale, + dtype=hidden_states.dtype, + block_size=16, + swizzle=False, + ) + + # Dequantize w2 from packed NVFP4 to fp16/bf16 + w2_global_scale = self.quant_config.g2_alphas + + w2_dequant = dequantize_to_dtype( + tensor_fp4=w2, + tensor_sf=self.w2_scale_val, + global_scale=w2_global_scale, + dtype=hidden_states.dtype, + block_size=16, + swizzle=False, + ) + + super().apply( + output=output, + hidden_states=hidden_states, + w1=w1_dequant, + w2=w2_dequant, + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=activation, + global_num_experts=global_num_experts, + expert_map=expert_map, + a1q_scale=None, + a2_scale=self.quant_config.a2_gscale, + workspace13=workspace13, + workspace2=workspace2, + expert_tokens_meta=expert_tokens_meta, + apply_router_weight_on_input=apply_router_weight_on_input, + ) + + +@pytest.mark.parametrize( + ("config_kwargs", "expected_reason"), + [ + ({"has_bias": True}, "kernel does not support bias"), + ({"is_lora_enabled": True}, "kernel does not support LoRA"), + ], +) +def test_nvfp4_emulation_support_check_rejects_bias_and_lora( + config_kwargs: dict[str, bool], + expected_reason: str, +) -> None: + moe_config = FusedMoEConfig( + num_experts=2, + experts_per_token=1, + hidden_dim=16, + intermediate_size=16, + num_local_experts=2, + num_logical_experts=2, + moe_parallel_config=FusedMoEParallelConfig.make_no_parallel(), + activation=MoEActivation.SILU, + in_dtype=torch.bfloat16, + device="cuda", + routing_method=RoutingMethodType.TopK, + **config_kwargs, + ) + + supported, reason = Nvfp4QuantizationEmulationTritonExperts.is_supported_config( + Nvfp4QuantizationEmulationTritonExperts, + moe_config, + kNvfp4Static, + kNvfp4Dynamic, + mk.FusedMoEActivationFormat.Standard, + ) + + assert not supported + assert reason == expected_reason + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="Triton NVFP4 kernel requires CUDA.", +) +def test_triton_dequantize_nvfp4(monkeypatch, loaded_model_files) -> None: + """Test the Triton dequantization kernel against the CPU reference + using real NVFP4 weights from a checkpoint. + + Tests both 2D (attention projection) and 3D (stacked MoE experts). + """ + checkpoint_path = loaded_model_files["nvidia/Qwen3-30B-A3B-NVFP4"] + shards = cast(list[str], MOE_MODEL_CONFIGS["nvidia/Qwen3-30B-A3B-NVFP4"]["shards"]) + shard_path = f"{checkpoint_path}/{shards[0]}" + block_size = 16 + + with safe_open(shard_path, framework="pt", device="cpu") as f: + all_keys = list(f.keys()) + + # 2D case: attention projection + tensor_fp4_2d = f.get_tensor("model.layers.9.self_attn.k_proj.weight") + tensor_sf_2d = f.get_tensor("model.layers.9.self_attn.k_proj.weight_scale") + global_scale_2d = f.get_tensor("model.layers.9.self_attn.k_proj.weight_scale_2") + + # 3D case: stack ALL experts for layer 9 up_proj + expert_prefix = "model.layers.9.mlp.experts." + expert_indices = sorted( + int(key.split(".")[5]) + for key in all_keys + if key.startswith(expert_prefix) and key.endswith(".up_proj.weight") + ) + assert len(expert_indices) > 0 + + all_fp4 = [] + all_sf = [] + all_global_scale = [] + for index in expert_indices: + name = f"{expert_prefix}{index}.up_proj" + all_fp4.append(f.get_tensor(f"{name}.weight")) + all_sf.append(f.get_tensor(f"{name}.weight_scale")) + all_global_scale.append(f.get_tensor(f"{name}.weight_scale_2")) + + tensor_fp4_3d = torch.stack(all_fp4) + tensor_sf_3d = torch.stack(all_sf) + global_scale_3d = torch.stack(all_global_scale) + + test_cases = [ + ("2D base", tensor_fp4_2d, tensor_sf_2d, global_scale_2d), + ( + "2D 2x rows", + tensor_fp4_2d.repeat(2, 1), + tensor_sf_2d.repeat(2, 1), + global_scale_2d, + ), + ( + "2D 4x rows", + tensor_fp4_2d.repeat(4, 1), + tensor_sf_2d.repeat(4, 1), + global_scale_2d, + ), + ( + "2D 2x cols", + tensor_fp4_2d.repeat(1, 2), + tensor_sf_2d.repeat(1, 2), + global_scale_2d, + ), + ("3D base", tensor_fp4_3d, tensor_sf_3d, global_scale_3d), + ( + "3D 2x experts", + tensor_fp4_3d.repeat(2, 1, 1), + tensor_sf_3d.repeat(2, 1, 1), + global_scale_3d.repeat(2), + ), + ( + "3D 2x rows", + tensor_fp4_3d.repeat(1, 2, 1), + tensor_sf_3d.repeat(1, 2, 1), + global_scale_3d, + ), + ( + "3D 2x cols", + tensor_fp4_3d.repeat(1, 1, 2), + tensor_sf_3d.repeat(1, 1, 2), + global_scale_3d, + ), + ] + + quantiles = [0.5, 0.001, 0.999] + + # Move the E2M1 lookup table to CUDA ahead of time, as would normally + # happen during model loading (process_weights_after_loading). Both the + # Triton and PyTorch reference paths run on CUDA. + nvfp4_emulation_utils.kE2M1ToFloat_handle.val = ( + nvfp4_emulation_utils.kE2M1ToFloat_handle.val.cuda() + ) + + for label, tensor_fp4, tensor_sf, global_scale in test_cases: + fp4_cuda = tensor_fp4.cuda() + sf_cuda = tensor_sf.cuda() + gs_cuda = global_scale.cuda() + + # Triton path + triton_result = dequantize_to_dtype( + fp4_cuda, + sf_cuda, + gs_cuda, + torch.bfloat16, + block_size, + swizzle=False, + ) + + # Reference path (PyTorch ops on CUDA, Triton dispatch disabled) + with monkeypatch.context() as m: + m.setattr( + nvfp4_emulation_utils.current_platform, + "is_cuda_alike", + lambda: False, + ) + reference = dequantize_to_dtype( + fp4_cuda, + sf_cuda, + gs_cuda, + torch.bfloat16, + block_size, + swizzle=False, + ) + + torch.testing.assert_close(triton_result, reference, atol=0, rtol=0) + + # Benchmark + shape = list(tensor_fp4.shape) + + def _triton_bench( + fp4_cuda=fp4_cuda, + scale_cuda=sf_cuda, + global_scale_cuda=gs_cuda, + block_size=block_size, + ): + return dequantize_to_dtype( + fp4_cuda, + scale_cuda, + global_scale_cuda, + torch.bfloat16, + block_size, + swizzle=False, + ) + + triton_ms, triton_min, triton_max = triton.testing.do_bench( + _triton_bench, quantiles=quantiles + ) + + def _reference_bench( + fp4_cuda=fp4_cuda, + scale_cuda=sf_cuda, + global_scale_cuda=gs_cuda, + block_size=block_size, + ): + with monkeypatch.context() as m2: + m2.setattr( + nvfp4_emulation_utils.current_platform, + "is_cuda_alike", + lambda: False, + ) + dequantize_to_dtype( + fp4_cuda, + scale_cuda, + global_scale_cuda, + torch.bfloat16, + block_size, + swizzle=False, + ) + + ref_ms, ref_min, ref_max = triton.testing.do_bench( + _reference_bench, quantiles=quantiles + ) + + speedup = ref_ms / triton_ms if triton_ms > 0 else float("inf") + print(f" dequantize {label} {shape}:") + print( + f" triton: median={triton_ms:.3f}ms, " + f"min={triton_min:.3f}ms, max={triton_max:.3f}ms" + ) + print( + f" reference: median={ref_ms:.3f}ms, " + f"min={ref_min:.3f}ms, max={ref_max:.3f}ms" + ) + print(f" speedup: {speedup:.2f}x") + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="Triton NVFP4 kernel requires CUDA.", +) +@pytest.mark.parametrize( + "m, k", + [ + (1, 16), + (1, 4096), + (2, 4096), + (4, 4096), + (8, 4096), + (16, 4096), + (24, 4096), + (32, 4096), + (1, 8192), + (2, 8192), + (4, 8192), + (8, 8192), + (16, 8192), + (24, 8192), + (32, 8192), + (1, 32), + (2, 48), + (7, 64), + (16, 128), + (33, 160), + (128, 256), + (256, 512), + (1024, 1024), + (5120, 2048), + (2048, 4096), + (4096, 7168), + (8192, 8192), + (128, 16384), + ], +) +@pytest.mark.parametrize("global_scale_value", [0.5, 1.0, 0.001]) +def test_triton_nvfp4_quant_dequant( + monkeypatch, m: int, k: int, global_scale_value: float +) -> None: + """Test the Triton quant-dequant kernel against the CPU reference.""" + block_size = 16 + x = torch.randn(m, k, dtype=torch.bfloat16, device="cuda") + global_scale = torch.tensor(global_scale_value, dtype=torch.float32, device="cuda") + + # Triton path + triton_result = ref_nvfp4_quant_dequant(x, global_scale, block_size) + + # CPU reference path + with monkeypatch.context() as mp: + mp.setattr( + nvfp4_emulation_utils.current_platform, + "is_cuda_alike", + lambda: False, + ) + reference = ref_nvfp4_quant_dequant(x.cpu(), global_scale.cpu(), block_size) + + torch.testing.assert_close(triton_result.cpu(), reference, atol=0, rtol=0) + + # Benchmark (both paths on CUDA tensors for fair comparison) + quantiles = [0.5, 0.001, 0.999] + + def _triton_bench( + input_tensor=x, input_global_scale=global_scale, input_block_size=block_size + ): + return ref_nvfp4_quant_dequant( + input_tensor, input_global_scale, input_block_size + ) + + triton_ms, triton_min, triton_max = triton.testing.do_bench( + _triton_bench, quantiles=quantiles + ) + + def _reference_bench( + input_tensor=x, input_global_scale=global_scale, input_block_size=block_size + ): + with monkeypatch.context() as mp2: + mp2.setattr( + nvfp4_emulation_utils.current_platform, + "is_cuda_alike", + lambda: False, + ) + ref_nvfp4_quant_dequant(input_tensor, input_global_scale, input_block_size) + + ref_ms, ref_min, ref_max = triton.testing.do_bench( + _reference_bench, quantiles=quantiles + ) + + speedup = ref_ms / triton_ms if triton_ms > 0 else float("inf") + print(f" quant_dequant [{m}x{k}] gs={global_scale_value}:") + print( + f" triton: median={triton_ms:.3f}ms, " + f"min={triton_min:.3f}ms, max={triton_max:.3f}ms" + ) + print( + f" reference: median={ref_ms:.3f}ms, " + f"min={ref_min:.3f}ms, max={ref_max:.3f}ms" + ) + print(f" speedup: {speedup:.2f}x") + + +def _load_nvfp4_moe_weights( + model_files: dict[str, str], + model_id: str, + tensor_parallel_size: int, + max_experts: int | None = None, +): + """Load and stack NVFP4 MoE weights from checkpoint shards. + + Returns (w1, w1_scale, w1_gscale, w2, w2_scale, w2_gscale, + a1_gscale, a2_gscale, num_experts, hidden_dim, + intermediate_size). + + When max_experts is set, only the first max_experts experts are loaded. + + When tensor_parallel_size > 1, the N dimension of w1 and the K + dimension of w2 are narrowed to the first TP shard (simulating + column-parallel on w1 / row-parallel on w2). + """ + cfg = MOE_MODEL_CONFIGS[model_id] + shards = cast(list[str], cfg["shards"]) + checkpoint_path = model_files[model_id] + + expert_prefix = cfg["expert_prefix"] + idx_pos = cast(int, cfg["expert_idx_pos"]) + + # Collect all tensors across shards into a flat dict — an expert's + # tensors may be split across multiple shard files. + all_tensors: dict[str, torch.Tensor] = {} + for shard_name in shards: + shard_path = f"{checkpoint_path}/{shard_name}" + with safe_open(shard_path, framework="pt", device="cpu") as f: + for key in f.keys(): # noqa: SIM118 + if key.startswith(expert_prefix): + all_tensors[key] = f.get_tensor(key) + + expert_indices = sorted( + { + int(key.split(".")[idx_pos]) + for key in all_tensors + if key.endswith(".gate_proj.weight") + } + ) + if max_experts is not None: + expert_indices = expert_indices[:max_experts] + num_experts = len(expert_indices) + + gate_weights, up_weights, down_weights = [], [], [] + gate_scales, up_scales, down_scales = [], [], [] + gate_gscales, up_gscales, down_gscales = [], [], [] + a1_scales, a2_scales = [], [] + + for idx in expert_indices: + prefix = f"{expert_prefix}{idx}" + gate_weights.append(all_tensors[f"{prefix}.gate_proj.weight"]) + gate_scales.append(all_tensors[f"{prefix}.gate_proj.weight_scale"]) + gate_gscales.append(all_tensors[f"{prefix}.gate_proj.weight_scale_2"]) + up_weights.append(all_tensors[f"{prefix}.up_proj.weight"]) + up_scales.append(all_tensors[f"{prefix}.up_proj.weight_scale"]) + up_gscales.append(all_tensors[f"{prefix}.up_proj.weight_scale_2"]) + down_weights.append(all_tensors[f"{prefix}.down_proj.weight"]) + down_scales.append(all_tensors[f"{prefix}.down_proj.weight_scale"]) + down_gscales.append(all_tensors[f"{prefix}.down_proj.weight_scale_2"]) + a1_scales.append(all_tensors[f"{prefix}.gate_proj.input_scale"]) + a2_scales.append(all_tensors[f"{prefix}.down_proj.input_scale"]) + + # Stack into MoE format. + # w1 = [E, 2*intermediate, hidden//2] (gate + up concatenated) + w1 = torch.stack( + [torch.cat([g, u], dim=0) for g, u in zip(gate_weights, up_weights)] + ).cuda() + w1_scale = torch.stack( + [torch.cat([g, u], dim=0) for g, u in zip(gate_scales, up_scales)] + ).cuda() + w1_gscale = torch.stack(gate_gscales).cuda() + + # w2 = [E, hidden, intermediate//2] + w2 = torch.stack(down_weights).cuda() + w2_scale = torch.stack(down_scales).cuda() + w2_gscale = torch.stack(down_gscales).cuda() + + a13_scale_raw = torch.stack(a1_scales).cuda() + a2_scale_raw = torch.stack(a2_scales).cuda() + + # Apply EMULATION transforms (matches oracle/nvfp4.py). + nvfp4_emulation_utils.kE2M1ToFloat_handle.val = ( + nvfp4_emulation_utils.kE2M1ToFloat_handle.val.cuda() + ) + a1_gscale = 1.0 / a13_scale_raw.max().to(torch.float32) + a2_gscale = 1.0 / a2_scale_raw.max().to(torch.float32) + + # ── Simulate TP sharding ── + # w1 (gate_up): column-parallel → shard the N dimension (dim 1). + # w2 (down): row-parallel → shard the K dimension (dim 2, + # which is the packed K//2 dim). + # Scales follow the same sharding on the corresponding dimension. + tp = tensor_parallel_size + if tp > 1: + n1 = w1.size(1) // tp + w1 = w1[:, :n1, :].contiguous() + w1_scale = w1_scale[:, :n1, :].contiguous() + + k2_packed = w2.size(2) // tp + k2_scale = w2_scale.size(2) // tp + w2 = w2[:, :, :k2_packed].contiguous() + w2_scale = w2_scale[:, :, :k2_scale].contiguous() + + hidden_dim = w1.size(2) * 2 + intermediate_size = w1.size(1) // 2 + + return ( + w1, + w1_scale, + w1_gscale, + w2, + w2_scale, + w2_gscale, + a1_gscale, + a2_gscale, + num_experts, + hidden_dim, + intermediate_size, + ) + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="Triton NVFP4 kernel requires CUDA.", +) +@pytest.mark.parametrize("num_tokens", [1, 2, 4, 1024]) +@pytest.mark.parametrize("top_k", [4]) +@pytest.mark.parametrize("model_id", list(MOE_MODEL_CONFIGS.keys())) +@pytest.mark.parametrize( + "tensor_parallel_size", + [pytest.param(val, id=f"tensor_parallel_size:{val}") for val in [1, 2, 4, 8]], +) +def test_nvfp4_moe_correctness( + loaded_model_files, + num_tokens: int, + top_k: int, + model_id: str, + tensor_parallel_size: int, +) -> None: + """Compare Nvfp4QuantizationEmulationTritonExperts (fused weight dequant + compute) + against the unfused reference Nvfp4QuantizationEmulationTritonExpertsReference. + + Both must produce bit-identical results. + """ + num_test_experts = max(8, top_k) + ( + w1, + w1_scale, + w1_gscale, + w2, + w2_scale, + w2_gscale, + a1_gscale, + a2_gscale, + num_experts, + hidden_dim, + intermediate_size, + ) = _load_nvfp4_moe_weights( + loaded_model_files, + model_id, + tensor_parallel_size, + max_experts=num_test_experts, + ) + + moe_config = FusedMoEConfig( + num_experts=num_experts, + experts_per_token=top_k, + hidden_dim=hidden_dim, + intermediate_size=intermediate_size, + num_local_experts=num_experts, + num_logical_experts=num_experts, + moe_parallel_config=FusedMoEParallelConfig.make_no_parallel(), + activation=MoEActivation.SILU, + in_dtype=torch.bfloat16, + device="cuda", + routing_method=RoutingMethodType.TopK, + max_num_tokens=512, + ) + + def _make_quant_config(): + return nvfp4_moe_quant_config( + g1_alphas=w1_gscale.clone(), + g2_alphas=w2_gscale.clone(), + a1_gscale=a1_gscale.clone(), + a2_gscale=a2_gscale.clone(), + w1_scale=w1_scale.clone(), + w2_scale=w2_scale.clone(), + ) + + ref_experts = Nvfp4QuantizationEmulationTritonExpertsReference( + moe_config=moe_config, + quant_config=_make_quant_config(), + ) + fused_experts = Nvfp4QuantizationEmulationTritonExperts( + moe_config=moe_config, + quant_config=_make_quant_config(), + ) + + torch.manual_seed(42) + hidden_states = torch.randn( + num_tokens, hidden_dim, dtype=torch.bfloat16, device="cuda" + ) + + topk_weights = torch.randn( + num_tokens, top_k, dtype=torch.float32, device="cuda" + ).softmax(dim=-1) + topk_ids = torch.stack( + [torch.randperm(num_experts, device="cuda")[:top_k] for _ in range(num_tokens)] + ).to(torch.int32) + + N = w1.size(1) # 2 * intermediate + K = hidden_dim + + ws13_size = num_tokens * top_k * max(intermediate_size, K) + ws2_size = num_tokens * top_k * max(N, K) + + workspace13_ref = torch.zeros(ws13_size, dtype=torch.bfloat16, device="cuda") + workspace2_ref = torch.zeros(ws2_size, dtype=torch.bfloat16, device="cuda") + output_ref = torch.zeros(num_tokens, K, dtype=torch.bfloat16, device="cuda") + + workspace13_fused = torch.zeros_like(workspace13_ref) + workspace2_fused = torch.zeros_like(workspace2_ref) + output_fused = torch.zeros_like(output_ref) + + apply_kwargs = dict( + hidden_states=hidden_states, + w1=w1, + w2=w2, + topk_weights=topk_weights, + topk_ids=topk_ids, + activation=MoEActivation.SILU, + global_num_experts=num_experts, + expert_map=None, + a1q_scale=None, + a2_scale=None, + expert_tokens_meta=None, + apply_router_weight_on_input=False, + ) + + # Unfused reference. + ref_experts.apply( + output=output_ref, + workspace13=workspace13_ref, + workspace2=workspace2_ref, + **apply_kwargs, + ) + + # Fused implementation. + fused_experts.apply( + output=output_fused, + workspace13=workspace13_fused, + workspace2=workspace2_fused, + **apply_kwargs, + ) + + # Not strict equality on H100, MI325, MI300 (< 0.1% elements). + # The fused on-the-fly dequant path can lower to a slightly + # different Triton/MMA tiling than the pre-dequantized + # reference; experiments with reference-like tiling/masking + # reduced some diffs were not kept because they regress + # the fused kernel speed. + # Strict equality validated on MI355. + torch.testing.assert_close( + output_fused, + output_ref, + atol=0.0 if on_gfx950() else 0.02, + rtol=0, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_nvfp4_quant.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_nvfp4_quant.py new file mode 100644 index 0000000000000000000000000000000000000000..14a4234be380660d00b58a1bd38d90f1c8fbfe0f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_nvfp4_quant.py @@ -0,0 +1,308 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch + +from vllm import _custom_ops as ops +from vllm.platforms import current_platform +from vllm.scalar_type import scalar_types +from vllm.utils.torch_utils import set_random_seed + +if not current_platform.has_device_capability(100): + pytest.skip( + reason="Nvfp4 Requires compute capability of 10 or above.", + allow_module_level=True, + ) + +DTYPES = [torch.float16, torch.bfloat16] +SHAPES = [(128, 64), (128, 128), (256, 64), (256, 128)] +PAD_SHAPES = [ + (90, 64), + (150, 64), + (128, 48), + (128, 80), + (150, 80), + (90, 48), + (90, 128), + (150, 128), + (150, 48), + (90, 80), + (128, 512), + (128, 1024), + (128, 2048), + (64, 7168), + (64, 7152), + (32, 14336), +] +PADDED_OUTPUT_SHAPES = [(128, 48), (128, 80), (150, 48), (150, 80), (64, 7152)] +SEEDS = [42] +CUDA_DEVICES = ["cuda:0"] + +FLOAT4_E2M1_MAX = scalar_types.float4_e2m1f.max() +FLOAT8_E4M3_MAX = torch.finfo(torch.float8_e4m3fn).max + +# E2M1 to float +# 0111 -> 6 +# 0110 -> 4 +# 0101 -> 3 +# 0100 -> 2 +# 0011 -> 1.5 +# 0010 -> 1 +# 0001 -> 0.5 +# 0000 -> 0 +E2M1_TO_FLOAT32 = [ + 0.0, + 0.5, + 1.0, + 1.5, + 2.0, + 3.0, + 4.0, + 6.0, + 0.0, + -0.5, + -1.0, + -1.5, + -2.0, + -3.0, + -4.0, + -6.0, +] +BLOCK_SIZE = 16 + + +def cast_from_fp4(x, m, n): + # The fp4 values are packed in uint8 as [v_1st | v_2nd] + v_2nd = x & 0xF + v_1st = (x >> 4) & 0xF + c = torch.stack((v_2nd, v_1st), dim=-1) + out = torch.tensor([E2M1_TO_FLOAT32[x] for x in c.flatten()]) + out = out.reshape(m, n).to(torch.float32) + return out + + +def cast_to_fp4(x): + sign = torch.sign(x) + x = torch.abs(x) + x[(x >= 0.0) & (x <= 0.25)] = 0.0 + x[(x > 0.25) & (x < 0.75)] = 0.5 + x[(x >= 0.75) & (x <= 1.25)] = 1.0 + x[(x > 1.25) & (x < 1.75)] = 1.5 + x[(x >= 1.75) & (x <= 2.5)] = 2.0 + x[(x > 2.5) & (x < 3.5)] = 3.0 + x[(x >= 3.5) & (x <= 5.0)] = 4.0 + x[x > 5.0] = 6.0 + return x * sign + + +def get_reciprocal(x): + if isinstance(x, torch.Tensor): + return torch.where(x == 0, torch.tensor(0.0, dtype=x.dtype), 1.0 / x) + elif isinstance(x, (float, int)): + return 0.0 if x == 0 else 1.0 / x + else: + raise TypeError("Input must be a float, int, or a torch.Tensor.") + + +def ref_nvfp4_quant(x, global_scale): + assert global_scale.dtype == torch.float32 + assert x.ndim == 2 + m, n = x.shape + x = torch.reshape(x, (m, n // BLOCK_SIZE, BLOCK_SIZE)) + vec_max = torch.max(torch.abs(x), dim=-1, keepdim=True)[0].to(torch.float32) + scale = global_scale * (vec_max * get_reciprocal(FLOAT4_E2M1_MAX)) + scale = scale.to(torch.float8_e4m3fn).to(torch.float32) + output_scale = get_reciprocal(scale * get_reciprocal(global_scale)) + + scaled_x = x.to(torch.float32) * output_scale + clipped_x = torch.clamp(scaled_x, -6.0, 6.0).reshape(m, n) + return cast_to_fp4(clipped_x), scale.squeeze(-1) + + +def recover_swizzled_scales(scale, m, n): + round_up = lambda x, y: (x + y - 1) // y * y + rounded_m = round_up(m, 128) + scale_n = n // BLOCK_SIZE + rounded_n = round_up(scale_n, 4) + # Recover the swizzled scaling factor to linear layout + tmp = torch.reshape(scale, (1, rounded_m // 128, rounded_n // 4, 32, 4, 4)) + tmp = torch.permute(tmp, (0, 1, 4, 3, 2, 5)) + result = torch.reshape(tmp, (rounded_m, rounded_n)).to(torch.float32) + return result[:m, :scale_n] + + +def round_up(x: int, y: int) -> int: + return (x + y - 1) // y * y + + +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("shape", SHAPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@torch.inference_mode() +def test_quantize_to_fp4( + dtype: torch.dtype, + shape: tuple[int, int], + seed: int, + device: str, +) -> None: + set_random_seed(seed) + torch.set_default_device(device) + + m, n = shape + + x = torch.randn((m, n), dtype=dtype) + tensor_amax = torch.abs(x).max().to(torch.float32) + global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX / tensor_amax + out_ref, scale_ref = ref_nvfp4_quant(x, global_scale) + + out, out_scale = ops.scaled_fp4_quant(x, global_scale) + scale_ans = recover_swizzled_scales(out_scale, m, n) + out_ans = cast_from_fp4(out, m, n) + + torch.testing.assert_close(out_ans, out_ref) + torch.testing.assert_close(scale_ans, scale_ref) + + +@pytest.mark.parametrize( + "shape", + [(32, 4096), (128, 4096), (1, 64), (127, 1024), (256, 16384)], +) +@pytest.mark.parametrize("is_sf_swizzled_layout", [True, False]) +@torch.inference_mode() +def test_python_util_matches_cpp_allocation( + shape: tuple[int, int], + is_sf_swizzled_layout: bool, +) -> None: + """ + Verify that the Python utility (create_fp4_output_tensors) allocates + tensors with the same shapes and dtypes as the C++ functional variant + (scaled_fp4_quant_func). + """ + from vllm._custom_ops import create_fp4_output_tensors + + torch.set_default_device("cuda:0") + m, n = shape + input_tensor = torch.randn((m, n), dtype=torch.bfloat16) + input_scale = torch.tensor([1.0], dtype=torch.float32, device="cuda:0") + + # C++ functional variant allocates internally + cpp_out, cpp_scale = torch.ops._C.scaled_fp4_quant( + input_tensor, input_scale, is_sf_swizzled_layout + ) + + # Python utility + py_out, py_scale = create_fp4_output_tensors( + m, n, torch.device("cuda:0"), is_sf_swizzled_layout + ) + + assert py_out.shape == cpp_out.shape, ( + f"Output shape mismatch: Python {py_out.shape} vs C++ {cpp_out.shape}" + ) + assert py_out.dtype == cpp_out.dtype, ( + f"Output dtype mismatch: Python {py_out.dtype} vs C++ {cpp_out.dtype}" + ) + assert py_scale.shape == cpp_scale.shape, ( + f"Scale shape mismatch: Python {py_scale.shape} vs C++ {cpp_scale.shape}" + ) + assert py_scale.dtype == cpp_scale.dtype, ( + f"Scale dtype mismatch: Python {py_scale.dtype} vs C++ {cpp_scale.dtype}" + ) + + +@pytest.mark.parametrize("shape", PADDED_OUTPUT_SHAPES) +@pytest.mark.parametrize("is_sf_swizzled_layout", [True, False]) +@torch.inference_mode() +def test_quantize_to_fp4_with_padded_output( + shape: tuple[int, int], + is_sf_swizzled_layout: bool, +) -> None: + from vllm._custom_ops import create_fp4_output_tensors + + dtype = torch.float16 + set_random_seed(42) + torch.set_default_device("cuda:0") + + m, n = shape + padded_n = round_up(n, 32) + assert padded_n > n + + x = torch.randn((m, n), dtype=dtype) + tensor_amax = torch.abs(x).max().to(torch.float32) + global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX / tensor_amax + out_ref, scale_ref = ref_nvfp4_quant(x, global_scale) + + out, out_scale = ops.scaled_fp4_quant( + x, + global_scale, + is_sf_swizzled_layout=is_sf_swizzled_layout, + padded_n=padded_n, + ) + py_out, py_scale = create_fp4_output_tensors( + m, + n, + torch.device("cuda:0"), + is_sf_swizzled_layout, + padded_n=padded_n, + ) + + assert out.shape == (m, padded_n // 2) + assert out.shape == py_out.shape + assert out_scale.shape == py_scale.view(torch.float8_e4m3fn).shape + + out_ans = cast_from_fp4(out[:, : n // 2], m, n) + torch.testing.assert_close(out_ans, out_ref) + assert torch.count_nonzero(out[:, n // 2 :]) == 0 + + if is_sf_swizzled_layout: + scale_ans = recover_swizzled_scales(out_scale, m, padded_n) + torch.testing.assert_close(scale_ans[:, : n // BLOCK_SIZE], scale_ref) + assert torch.count_nonzero(scale_ans[:, n // BLOCK_SIZE :]) == 0 + else: + scale_ans = out_scale.to(torch.float32) + torch.testing.assert_close(scale_ans[:, : n // BLOCK_SIZE], scale_ref) + assert torch.count_nonzero(scale_ans[:, n // BLOCK_SIZE :]) == 0 + + +@pytest.mark.parametrize("pad_shape", PAD_SHAPES) +@torch.inference_mode() +def test_quantize_to_fp4_padded(pad_shape: tuple[int, int]) -> None: + dtype = torch.float16 + set_random_seed(42) + torch.set_default_device("cuda:0") + + m, n = pad_shape + + x = torch.randn((m, n), dtype=dtype) + + tensor_amax = torch.abs(x).max().to(torch.float32) + global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX / tensor_amax + out_ref, scale_ref = ref_nvfp4_quant(x, global_scale) + + out, out_scale = ops.scaled_fp4_quant(x, global_scale) + scale_ans = recover_swizzled_scales(out_scale, m, n) + out_ans = cast_from_fp4(out, m, n) + torch.testing.assert_close(out_ans, out_ref) + torch.testing.assert_close(scale_ans, scale_ref) + + +@pytest.mark.parametrize("pad_shape", PAD_SHAPES) +@torch.inference_mode() +def test_quantize_to_fp4_padded_no_sf_swizzled(pad_shape: tuple[int, int]) -> None: + dtype = torch.float16 + set_random_seed(42) + torch.set_default_device("cuda:0") + + m, n = pad_shape + + x = torch.randn((m, n), dtype=dtype) + + tensor_amax = torch.abs(x).max().to(torch.float32) + global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX / tensor_amax + out_ref, scale_ref = ref_nvfp4_quant(x, global_scale) + + out, out_scale = ops.scaled_fp4_quant(x, global_scale, is_sf_swizzled_layout=False) + scale_ans = out_scale.to(torch.float32) + out_ans = cast_from_fp4(out, m, n) + torch.testing.assert_close(out_ans, out_ref) + torch.testing.assert_close(scale_ans, scale_ref) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_nvfp4_qutlass.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_nvfp4_qutlass.py new file mode 100644 index 0000000000000000000000000000000000000000..df03827200d118fb3902ebfe7cf379bdd9e7eff0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_nvfp4_qutlass.py @@ -0,0 +1,269 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# +# Copyright (C) 2025 Roberto L. Castro (Roberto.LopezCastro@ist.ac.at). +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +import numpy as np +import pytest +import torch +from compressed_tensors.transform.utils.hadamard import deterministic_hadamard_matrix + +from vllm import _custom_ops as ops # use existing nvfp4 gemm in vllm +from vllm._custom_ops import fusedQuantizeNv +from vllm.model_executor.layers.quantization.qutlass_utils import to_blocked +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +if not torch.cuda.is_available(): + pytest.skip("CUDA required for these tests.", allow_module_level=True) + +if not ( + current_platform.has_device_capability(100) + or current_platform.has_device_capability(120) +): + pytest.skip( + reason="Tests require compute capability 10.0 (100) or 12.0 (120).", + allow_module_level=True, + ) + + +# ----- Helpers ----- +def get_hadamard_matrix(group_size: int, dtype: torch.dtype, device: torch.device): + return ( + deterministic_hadamard_matrix(group_size, dtype=dtype, device=device) + * group_size**-0.5 + ) + + +def _rtne_fp4(x: torch.Tensor): + device = x.device + grid = torch.tensor( + [ + -6.0, + -4.0, + -3.0, + -2.0, + -1.5, + -1.0, + -0.5, + -0.0, + 0.0, + 0.5, + 1.0, + 1.5, + 2.0, + 3.0, + 4.0, + 6.0, + ], + dtype=x.dtype, + device=x.device, + ) + grid_int = torch.tensor( + [-1, -2, -3, -4, -5, -6, -7, -8, 0, 1, 2, 3, 4, 5, 6, 7], + dtype=torch.uint8, + device=device, + ) + inds = torch.bucketize(x, grid) + lo, hi = (inds - 1).clamp(min=0, max=15), inds.clamp(min=0, max=15) + g_lo, g_hi = grid[lo], grid[hi] + pick_hi = (g_hi - x < x - g_lo) | (g_hi - x == x - g_lo) & (grid_int[hi] % 2 == 0) + y = torch.where(pick_hi, g_hi, g_lo) + y_int = torch.where(pick_hi, grid_int[hi], grid_int[lo]) + y_int_packed = (y_int[..., 1::2] & 0xF) << 4 | y_int[..., ::2] & 0xF + return y, y_int_packed + + +def _dq_fp4(x_e2m1: torch.Tensor, x_e4m3: torch.Tensor, alpha: float): + device = x_e2m1.device + + x_e2m1_i32 = x_e2m1.view(dtype=torch.uint8).to(dtype=torch.int32) + x_e2m1_unpacked = torch.stack( + [x_e2m1_i32 & 0xF, (x_e2m1_i32 >> 4) & 0xF], dim=-1 + ).flatten(start_dim=-2) + + grid_dq = torch.tensor( + [ + 0.0, + 0.5, + 1.0, + 1.5, + 2.0, + 3.0, + 4.0, + 6.0, + -0.0, + -0.5, + -1.0, + -1.5, + -2.0, + -3.0, + -4.0, + -6.0, + ], + dtype=torch.float64, + device=device, + ) + x_fp4_dq = grid_dq[x_e2m1_unpacked] + + scales_dq = x_e4m3.to(torch.float64) + x_dq = (x_fp4_dq.unflatten(dim=-1, sizes=(-1, 16)) * scales_dq[..., None]).flatten( + start_dim=-2 + ) / alpha # * (4. / 3.) + return x_dq, x_fp4_dq, scales_dq + + +def _unpack_mask(clip_mask: torch.Tensor) -> torch.Tensor: + clip_mask_unpacked_dq = torch.zeros( + *clip_mask.shape[:-1], + clip_mask.size(-1) * 8, + dtype=torch.bool, + device=clip_mask.device, + ) + for i in range(8): + clip_mask_unpacked_dq[..., i::8] = (clip_mask >> i) & 1 + return clip_mask_unpacked_dq + + +def _forward_quantize_ref(x: torch.Tensor, h: torch.Tensor, rot_size: int): + device = x.device + + xh_ref64 = ( + x.unflatten(dim=-1, sizes=(-1, rot_size)).to(dtype=torch.float64) + @ h.reshape(rot_size, rot_size).to(dtype=torch.float64) + ).flatten(start_dim=-2) + + abs_max = xh_ref64.unflatten(dim=-1, sizes=(-1, 16)).abs().amax(dim=-1) + scales_ref64_ = abs_max + 1e-8 + + xh_e4m3_ref = scales_ref64_.to(dtype=torch.float8_e4m3fn) + scales_ref64 = xh_e4m3_ref.to(dtype=torch.float64) + xh_scaled_ref64 = ( + xh_ref64.unflatten(dim=-1, sizes=(-1, 16)) / scales_ref64[..., None] + ).flatten(start_dim=-2) + + xh_scaled_ref64 *= 6.0 + + clip_mask_unpacked_ref = xh_scaled_ref64.abs() < 6.0 + clip_mask_ref = torch.zeros( + *x.shape[:-1], x.size(-1) // 8, dtype=torch.uint8, device=device + ) + for i in range(8): + clip_mask_ref |= clip_mask_unpacked_ref[..., i::8].to(dtype=torch.uint8) << i + + xh_fp4_ref, xh_e2m1_ref = _rtne_fp4(xh_scaled_ref64) + xh_dq, xh_fp4_dq, scales_dq = _dq_fp4(xh_e2m1_ref, xh_e4m3_ref, 6.0) + clip_mask_unpacked_dq = _unpack_mask(clip_mask_ref) + + assert xh_fp4_dq.equal(xh_fp4_ref) + assert scales_dq.equal(scales_ref64) + assert clip_mask_unpacked_dq.equal(clip_mask_unpacked_ref) + + return ( + xh_dq, + clip_mask_unpacked_ref, + (xh_e2m1_ref, xh_e4m3_ref, clip_mask_ref), + ) + + +DTYPE = torch.bfloat16 +DEVICE = torch.device("cuda:0") +ROT_SIZES = [16, 32, 64, 128] +GLOBAL_SCALES = [6.0] + +LLAMA_MODELS = { + "7B": [(4096, 3 * 4096), (4096, 4096), (4096, 2 * 10752), (10752, 4096)], + "13B": [(5120, 3 * 5120), (5120, 5120), (5120, 2 * 13568), (13568, 5120)], + "33B": [(6656, 3 * 6656), (6656, 6656), (6656, 2 * 17664), (17664, 6656)], + "70B": [(8192, 3 * 8192), (8192, 8192), (8192, 2 * 21760), (21760, 8192)], +} + + +@pytest.fixture(autouse=True) +def _seed_each_test(): + set_random_seed(0) + np.random.seed(0) + torch.random.manual_seed(0) + + +@pytest.mark.parametrize("rot_size", ROT_SIZES) +@pytest.mark.parametrize("global_scale_value", GLOBAL_SCALES) +@torch.inference_mode() +def test_fused_quantization(rot_size: int, global_scale_value: float): + dtype, device = DTYPE, DEVICE + h = get_hadamard_matrix(rot_size, dtype, device) + x = torch.randn(2, 4096, 4096, dtype=dtype, device=device) * 25.0 + global_scale = torch.tensor([global_scale_value], device=device) + + xh_dq_ref, _, _ = _forward_quantize_ref(x, h, rot_size) + xh_e2m1, xh_e4m3 = fusedQuantizeNv(x, h, global_scale) + xh_e4m3 = xh_e4m3.reshape(2, 4096, 4096 // 16) + xh_dq, *_ = _dq_fp4(xh_e2m1, xh_e4m3, alpha=global_scale_value) + + torch.testing.assert_close(xh_dq, xh_dq_ref, rtol=0.34, atol=100) + assert (xh_dq != xh_dq_ref).float().mean() <= 1e-1 + + m, n, k = 504, 4096 * 2, 4096 + a = torch.randn(m, k, dtype=dtype, device=device) * 25.0 + b = torch.randn(n, k, dtype=dtype, device=device) * 25.0 + + a_e2m1, a_e4m3 = fusedQuantizeNv(a, h, global_scale) + b_e2m1, b_e4m3 = fusedQuantizeNv(b, h, global_scale) + + a_dq, *_ = _dq_fp4(a_e2m1, a_e4m3[:m, :k], alpha=1.0) + b_dq, *_ = _dq_fp4(b_e2m1, b_e4m3[:n, :k], alpha=1.0) + out_ref = a_dq @ b_dq.transpose(-2, -1) + + a_scale_block = to_blocked(a_e4m3, backend="triton").view(-1, k // 16) + b_scale_block = to_blocked(b_e4m3, backend="triton").view(-1, k // 16) + alpha = torch.tensor([1.0], device=device) + out = ops.cutlass_scaled_fp4_mm( + a_e2m1, b_e2m1, a_scale_block, b_scale_block, alpha, torch.bfloat16 + ) + assert out.equal(out_ref.to(dtype=out.dtype)) + + +@pytest.mark.parametrize("model", list(LLAMA_MODELS.keys())) +@pytest.mark.parametrize("layer_idx", [0, 1, 2, 3]) +@pytest.mark.parametrize("batch", [1, 16]) +@pytest.mark.parametrize("rot_size", ROT_SIZES) +@torch.inference_mode() +def test_llama_shapes(model: str, layer_idx: int, batch: int, rot_size: int): + dtype, device = DTYPE, DEVICE + m = batch + k, n = LLAMA_MODELS[model][layer_idx] + + h = get_hadamard_matrix(rot_size, dtype, device) + + a = torch.randn(m, k, dtype=dtype, device=device) * 25.0 + b = torch.randn(n, k, dtype=dtype, device=device) * 25.0 + + global_scale = torch.tensor([1.0], device=device) + + a_e2m1, a_e4m3 = fusedQuantizeNv(a, h, global_scale) + b_e2m1, b_e4m3 = fusedQuantizeNv(b, h, global_scale) + + a_dq, *_ = _dq_fp4(a_e2m1, a_e4m3[:m, :k], alpha=1.0) + b_dq, *_ = _dq_fp4(b_e2m1, b_e4m3[:n, :k], alpha=1.0) + out_ref = a_dq @ b_dq.transpose(-2, -1) + + a_scale_block = to_blocked(a_e4m3, backend="triton").view(-1, k // 16) + b_scale_block = to_blocked(b_e4m3, backend="triton").view(-1, k // 16) + alpha = torch.tensor([1.0], device=device) + out = ops.cutlass_scaled_fp4_mm( + a_e2m1, b_e2m1, a_scale_block, b_scale_block, alpha, torch.bfloat16 + ) + assert out.equal(out_ref.to(dtype=out.dtype)) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_nvfp4_scaled_mm.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_nvfp4_scaled_mm.py new file mode 100644 index 0000000000000000000000000000000000000000..26680198b7a0ea4453801d0b66e7685f2e5bb028 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_nvfp4_scaled_mm.py @@ -0,0 +1,100 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch +from nvfp4_utils import FLOAT4_E2M1_MAX, FLOAT8_E4M3_MAX, dequantize_nvfp4_to_dtype + +from vllm import _custom_ops as ops +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +if not current_platform.has_device_capability(100): + pytest.skip( + reason="Nvfp4 Requires compute capability of 10 or above.", + allow_module_level=True, + ) + +DTYPES = [torch.float16, torch.bfloat16] +# m, n, k +SHAPES = [(128, 128, 64), (128, 128, 128), (256, 128, 64), (128, 256, 128)] +PAD_SHAPES = [(150, 128, 64), (128, 128, 96)] +SHAPES.extend(PAD_SHAPES) + +SEEDS = [42] +CUDA_DEVICES = ["cuda:0"] + + +def get_ref_results( + a_fp4, + b_fp4, + a_sf, + b_sf, + a_global_scale, + b_global_scale, + m, + n, + dtype, + block_size, + device, +): + _, m_k = a_fp4.shape + _, n_k = b_fp4.shape + assert m_k == n_k + a_in_dtype = dequantize_nvfp4_to_dtype( + a_fp4, a_sf, a_global_scale, dtype=dtype, device=device, block_size=block_size + ) + b_in_dtype = dequantize_nvfp4_to_dtype( + b_fp4, b_sf, b_global_scale, dtype=dtype, device=device, block_size=block_size + ) + return torch.matmul(a_in_dtype, b_in_dtype.t()) + + +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("shape", SHAPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@torch.inference_mode() +def test_nvfp4_gemm( + dtype: torch.dtype, + shape: tuple[int, int, int], + seed: int, + device: str, +) -> None: + set_random_seed(seed) + m, n, packed_k = shape + k = packed_k * 2 + block_size = 16 + a_dtype = torch.randn((m, k), dtype=dtype, device=device) + b_dtype = torch.randn((n, k), dtype=dtype, device=device) + + a_global_scale = ( + (FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX) / torch.amax(a_dtype.flatten(), dim=-1) + ).to(torch.float32) + b_global_scale = ( + (FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX) / torch.amax(b_dtype.flatten(), dim=-1) + ).to(torch.float32) + alpha = 1.0 / (a_global_scale * b_global_scale) + # ops.scaled_fp4_quant returns swizzled scales, while weights + # from checkpoints are in linear scales. + a_fp4, a_scale_interleaved = ops.scaled_fp4_quant(a_dtype, a_global_scale) + b_fp4, b_scale_interleaved = ops.scaled_fp4_quant(b_dtype, b_global_scale) + + # get_ref_results unswizzles the scales internally. + expected_out = get_ref_results( + a_fp4, + b_fp4, + a_scale_interleaved, + b_scale_interleaved, + a_global_scale, + b_global_scale, + m, + n, + dtype, + block_size, + device, + ) + out = ops.cutlass_scaled_fp4_mm( + a_fp4, b_fp4, a_scale_interleaved, b_scale_interleaved, alpha, dtype + ) + + torch.testing.assert_close(out, expected_out.to(dtype=dtype), atol=1e-1, rtol=1e-1) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_per_token_group_quant.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_per_token_group_quant.py new file mode 100644 index 0000000000000000000000000000000000000000..ed34513e4ce1deb6dcb554f4a9fc75e15acb104a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_per_token_group_quant.py @@ -0,0 +1,430 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from unittest.mock import patch + +import pytest +import torch + +from vllm.model_executor.layers.quantization.utils import fp8_utils, int8_utils +from vllm.model_executor.layers.quantization.utils.quant_utils import get_fp8_min_max +from vllm.platforms import current_platform + + +@pytest.mark.parametrize( + "shape", [(31, 128), (32, 128), (63, 256), (64, 256), (16, 512)] +) +@pytest.mark.parametrize("column_major", [False, True]) +@pytest.mark.parametrize("tma_aligned", [False, True]) +@pytest.mark.parametrize("scale_ue8m0", [False, True]) +@pytest.mark.parametrize("group_size", [64, 128]) +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), reason="Only test on CUDA/ROCm." +) +def test_per_token_group_quant_fp8( + shape, column_major: bool, tma_aligned: bool, scale_ue8m0: bool, group_size: int +): + device = "cuda" + + torch.manual_seed(42) + num_tokens, hidden_dim = shape + + x = torch.randn((num_tokens, hidden_dim), device=device, dtype=torch.bfloat16) * 8 + + # cuda path + out_q, scale = fp8_utils.per_token_group_quant_fp8( + x, + group_size, + column_major_scales=column_major, + tma_aligned_scales=tma_aligned, + use_ue8m0=scale_ue8m0, + ) + + # triton ref + with patch("vllm.platforms.current_platform.is_cuda_alike", return_value=False): + ref_q, ref_s = fp8_utils.per_token_group_quant_fp8( + x, + group_size, + column_major_scales=column_major, + use_ue8m0=scale_ue8m0, + ) + + assert torch.allclose(out_q.float(), ref_q.float(), atol=0.15, rtol=0.15) + assert torch.allclose(scale, ref_s, atol=0.01, rtol=0.01) + + +@pytest.mark.parametrize( + "num_tokens,hidden_dim,group_size", + [ + # No padding: mn=4 (mult of 4), groups_per_row=56 (mult of 4) + (4, 7168, 128), + # MN padding only: mn=1, tma_aligned_mn=4 + (1, 7168, 128), + # MN padding only: mn=3, tma_aligned_mn=4 + (3, 7168, 128), + # K padding only: groups_per_row=5 (5%4=1) + (4, 640, 128), + # K padding only: groups_per_row=6 (6%4=2) + (4, 768, 128), + # Single packed column, no padding: k_num_packed=1, mn%4=0 + (4, 384, 128), + # Both MN and K padding + (1, 384, 128), + (3, 640, 128), + # Larger shapes with no padding + (64, 7168, 128), + (128, 14336, 128), + # Larger shapes with padding + (127, 7168, 128), + (253, 640, 128), + ], +) +@pytest.mark.parametrize("poisoned_scales", [False, True]) +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="DeepGEMM not available on this platform", +) +def test_per_token_group_quant_fp8_packed( + num_tokens, hidden_dim, group_size, poisoned_scales +): + """Test the packed DeepGEMM quantization kernel against the Triton + reference (row-major, UE8M0 scales).""" + + device = "cuda" + torch.manual_seed(42) + + x = torch.randn((num_tokens, hidden_dim), device=device, dtype=torch.bfloat16) * 8 + + mn = num_tokens + groups_per_row = hidden_dim // group_size + k_num_packed = (groups_per_row + 3) // 4 + tma_aligned_mn = ((mn + 3) // 4) * 4 + num_scale_elems = mn + (k_num_packed - 1) * tma_aligned_mn + + if poisoned_scales: + # Call the kernel with poisoned scale buffer to + # ensure padded indices are correctly zeroed. + fp8_dtype = current_platform.fp8_dtype() + fp8_min, fp8_max = get_fp8_min_max() + out_q = torch.empty_like(x, dtype=fp8_dtype) + out_s_packed = torch.empty_strided( + (mn, k_num_packed), + (1, tma_aligned_mn), + device=device, + dtype=torch.int32, + ) + torch.as_strided(out_s_packed, (num_scale_elems,), (1,)).fill_(0x7F7F7F7F) + torch.ops._C.per_token_group_fp8_quant_packed( + x, + out_q, + out_s_packed, + group_size, + 1e-10, + fp8_min, + fp8_max, + ) + else: + out_q, out_s_packed = fp8_utils.per_token_group_quant_fp8_packed_for_deepgemm( + x, + group_size=group_size, + use_ue8m0=True, + ) + + # Triton reference (row-major float32 scales, UE8M0) + with patch("vllm.platforms.current_platform.is_cuda_alike", return_value=False): + ref_q, ref_s = fp8_utils.per_token_group_quant_fp8( + x, + group_size, + use_ue8m0=True, + ) + + # Quantized values must match. + assert torch.equal(out_q, ref_q), "Quantized output mismatch" + + # Verify packed scales (valid exponents + padding zeros). + ref_s_flat = ref_s.reshape(mn, groups_per_row) + ref_exponents = (ref_s_flat.view(torch.int32) >> 23) & 0xFF + + expected = torch.zeros(num_scale_elems, dtype=torch.int32, device="cpu") + for row in range(mn): + for g in range(groups_per_row): + pack_col = g // 4 + pos = g % 4 + idx = pack_col * tma_aligned_mn + row + expected[idx] |= int(ref_exponents[row, g].item()) << (pos * 8) + + actual = torch.as_strided(out_s_packed, (num_scale_elems,), (1,)).cpu() + assert torch.equal(actual, expected), ( + f"Packed scale storage mismatch.\n" + f"First diff at index " + f"{(actual != expected).nonzero(as_tuple=True)[0][0].item()}" + ) + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="DeepGEMM not available on this platform", +) +def test_per_token_group_quant_fp8_packed_all_zero(): + """All-zero input must produce well-defined UE8M0 scale bytes via the eps + floor in the kernel's UE8M0 path. Locks down the all-zero behavior before + optimization. + + The CUDA kernel computes: + y_s = eps / fp8_max + y_s = exp2(ceil(log2(fmax(y_s, 1e-10)))) + For all-zero input, eps/fp8_max < 1e-10, so the inner fmax clamps back to + 1e-10, giving exp2(ceil(log2(1e-10))) = exp2(-33) => UE8M0 byte 0x5E (94). + """ + + device = "cuda" + num_tokens, hidden_dim, group_size = 4, 7168, 128 + x = torch.zeros((num_tokens, hidden_dim), device=device, dtype=torch.bfloat16) + + out_q, out_s_packed = fp8_utils.per_token_group_quant_fp8_packed_for_deepgemm( + x, + group_size=group_size, + use_ue8m0=True, + ) + + # Quantized values must be all zero. + assert torch.equal( + out_q.view(torch.uint8), + torch.zeros_like(out_q, dtype=torch.uint8), + ), "All-zero input should produce all-zero FP8 output" + + # UE8M0 byte produced by the kernel for all-zero input. + # The kernel's inner fmax(y_s, 1e-10) clamps eps/fp8_max back to 1e-10. + # 1e-10 as float32 has biased exponent 0x5D and a non-zero mantissa, so + # the kernel's bit-twiddle (exp_bits + (mant_bits != 0)) rounds up to + # 0x5E. This matches exp2(ceil(log2(1e-10))) = exp2(-33). + expected_exp_byte = 0x5E + + mn = num_tokens + groups_per_row = hidden_dim // group_size + k_num_packed = (groups_per_row + 3) // 4 + tma_aligned_mn = ((mn + 3) // 4) * 4 + num_scale_elems = mn + (k_num_packed - 1) * tma_aligned_mn + + # All valid scale slots must contain the expected packed value. + # Padding slots must be zero. + actual = torch.as_strided(out_s_packed, (num_scale_elems,), (1,)).cpu() + + expected = torch.zeros(num_scale_elems, dtype=torch.int32, device="cpu") + for row in range(mn): + for g in range(groups_per_row): + pack_col = g // 4 + pos = g % 4 + idx = pack_col * tma_aligned_mn + row + expected[idx] |= expected_exp_byte << (pos * 8) + + assert torch.equal(actual, expected), "All-zero scale bytes mismatch" + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="DeepGEMM not available on this platform", +) +def test_per_token_group_quant_fp8_packed_mantissa_rounds_up(): + """Inputs whose absmax/max_8bit produces a non-power-of-2 force the + mantissa-rounding-up branch (exp_byte += 1). Locks down this behavior + before optimization.""" + + device = "cuda" + num_tokens, hidden_dim, group_size = 4, 7168, 128 + + # Build a tensor whose per-group absmax = 1.5 * fp8_max * 2^k for various k. + # fp8_max = torch.finfo(torch.float8_e4m3fn).max = 448.0. + # Then absmax/fp8_max = 1.5 * 2^k -> non-zero mantissa, triggers ceil + # rounding to 2^(k+1). Use k=0 for simplicity; the bf16 representation of + # 1.5*448=672.0 is exact. + x = torch.full( + (num_tokens, hidden_dim), + 672.0, + device=device, + dtype=torch.bfloat16, + ) + + out_q, out_s_packed = fp8_utils.per_token_group_quant_fp8_packed_for_deepgemm( + x, + group_size=group_size, + use_ue8m0=True, + ) + + with patch("vllm.platforms.current_platform.is_cuda_alike", return_value=False): + ref_q, ref_s = fp8_utils.per_token_group_quant_fp8( + x, + group_size, + use_ue8m0=True, + ) + + assert torch.equal(out_q, ref_q), "Quantized output mismatch" + + mn = num_tokens + groups_per_row = hidden_dim // group_size + k_num_packed = (groups_per_row + 3) // 4 + tma_aligned_mn = ((mn + 3) // 4) * 4 + num_scale_elems = mn + (k_num_packed - 1) * tma_aligned_mn + + ref_s_flat = ref_s.reshape(mn, groups_per_row) + ref_exponents = (ref_s_flat.view(torch.int32) >> 23) & 0xFF + expected = torch.zeros(num_scale_elems, dtype=torch.int32, device="cpu") + for row in range(mn): + for g in range(groups_per_row): + pack_col = g // 4 + pos = g % 4 + idx = pack_col * tma_aligned_mn + row + expected[idx] |= int(ref_exponents[row, g].item()) << (pos * 8) + + actual = torch.as_strided(out_s_packed, (num_scale_elems,), (1,)).cpu() + assert torch.equal(actual, expected), "Scale bytes mismatch" + + +@pytest.mark.parametrize( + "num_tokens,hidden_dim", + [ + (1, 7168), # mn padded 1 -> 4 + (2, 7168), # mn padded 2 -> 4 + (3, 7168), # mn padded 3 -> 4 + (5, 7168), # mn padded 5 -> 8 + (127, 7168), # mn padded 127 -> 128 + (253, 640), # both mn and groups padded + (1, 384), # extreme: 1 group, 1 mn row -> both axes padded + ], +) +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="DeepGEMM not available on this platform", +) +def test_per_token_group_quant_fp8_packed_zero_fills_padded_output_q( + num_tokens, hidden_dim +): + """When output_q is allocated with shape (tma_aligned_mn, k) instead of + (mn, k), the kernel must overwrite the padded mn rows with zeros so + callers can use ``torch.empty`` instead of ``torch.zeros``.""" + + device = "cuda" + group_size = 128 + torch.manual_seed(42) + x = torch.randn((num_tokens, hidden_dim), device=device, dtype=torch.bfloat16) * 8 + + mn = num_tokens + groups_per_row = hidden_dim // group_size + k_num_packed = (groups_per_row + 3) // 4 + tma_aligned_mn = ((mn + 3) // 4) * 4 + + fp8_dtype = current_platform.fp8_dtype() + fp8_min, fp8_max = get_fp8_min_max() + # Allocate output_q with the padded mn extent and pre-fill with 0xFF + # so the kernel cannot rely on a clean buffer. + out_q = torch.empty((tma_aligned_mn, hidden_dim), device=device, dtype=fp8_dtype) + out_q.view(torch.uint8).fill_(0xFF) + + out_s_packed = torch.empty_strided( + (mn, k_num_packed), + (1, tma_aligned_mn), + device=device, + dtype=torch.int32, + ) + + torch.ops._C.per_token_group_fp8_quant_packed( + x, out_q, out_s_packed, group_size, 1e-10, fp8_min, fp8_max + ) + + # Live rows must match the Triton reference. + with patch("vllm.platforms.current_platform.is_cuda_alike", return_value=False): + ref_q, _ = fp8_utils.per_token_group_quant_fp8(x, group_size, use_ue8m0=True) + assert torch.equal(out_q[:mn], ref_q), "Live region mismatch" + + # Padded rows must be all-zero; without this, downstream TMA loads would + # see uninitialised data. + if tma_aligned_mn > mn: + padded_bytes = out_q[mn:tma_aligned_mn].view(torch.uint8) + assert padded_bytes.eq(0).all(), ( + f"Padded rows [{mn}, {tma_aligned_mn}) not zeroed; " + f"{padded_bytes.ne(0).sum().item()} non-zero bytes" + ) + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="packed FP8 per-token-group quant kernel requires a CUDA-alike GPU", +) +def test_per_token_group_quant_fp8_packed_large_mn(): + """Regression test for https://github.com/vllm-project/vllm/issues/45099. + + Some background: gridDim.x and gridDim.y have different limits of 2^31 - 1 and + 2^16 - 1, respectively. + Prior code introduced a bug where it incorrectly assumed grid.x and y both have + 2^31 - 1 limits and mixed them up, which doesn't surface until the kernel is + launched with a large mn that exceeds grid.y limit (2^16 - 1). + + This issue doesn't surface often because each forward pass only processes a + bounded token batch, not the full context. + Quantizing tensors with more rows than that will fail at launch with + "CUDA error: invalid argument". + This is a differential test that compares fp8 output against Triton output + reference when token size sits just above the gridDim.y 2^16 - 1 limit. + """ + + device = "cuda" + group_size = 128 + # hidden 2048 -> 2048/128 = 16 groups per row -> kx=16, ry=1: one grid row per mn + # row, so any mn > 65535 overflowed grid.y before the fix. + num_tokens, hidden_dim = 65537, 2048 + torch.manual_seed(42) + x = torch.randn((num_tokens, hidden_dim), device=device, dtype=torch.bfloat16) * 8 + + out_q, out_s_packed = fp8_utils.per_token_group_quant_fp8_packed_for_deepgemm( + x, + group_size=group_size, + use_ue8m0=True, + ) + + with patch("vllm.platforms.current_platform.is_cuda_alike", return_value=False): + ref_q, ref_s = fp8_utils.per_token_group_quant_fp8( + x, group_size, use_ue8m0=True + ) + + assert torch.equal(out_q, ref_q), "Quantized output mismatch" + + # Vectorized packed-scale check; the per-element loop used by the smaller + # tests is too slow at this size. groups_per_row is a multiple of 4 here, + # so there is no K padding and the packed view lines up. + mn = num_tokens + groups_per_row = hidden_dim // group_size + k_num_packed = (groups_per_row + 3) // 4 + assert groups_per_row % 4 == 0 + ref_exponents = (ref_s.reshape(mn, groups_per_row).view(torch.int32) >> 23) & 0xFF + exp = ref_exponents.view(mn, k_num_packed, 4) + expected = ( + exp[..., 0] | (exp[..., 1] << 8) | (exp[..., 2] << 16) | (exp[..., 3] << 24) + ) + assert torch.equal(out_s_packed.cpu(), expected.cpu()), "Packed scale mismatch" + + +@pytest.mark.parametrize("shape", [(32, 128), (64, 256), (16, 512)]) +@pytest.mark.parametrize("group_size", [64, 128]) +@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA not available") +def test_per_token_group_quant_int8(shape, group_size: int): + device = "cuda" + + torch.manual_seed(42) + num_tokens, hidden_dim = shape + + x = torch.randn((num_tokens, hidden_dim), device=device, dtype=torch.bfloat16) * 8 + + # cuda path + out_q, scale = int8_utils.per_token_group_quant_int8( + x, + group_size, + ) + + # triton ref + with patch("vllm.platforms.current_platform.is_cuda_alike", return_value=False): + ref_q, ref_s = int8_utils.per_token_group_quant_int8( + x, + group_size, + ) + + assert torch.allclose(out_q.float(), ref_q.float(), atol=0.15, rtol=0.15) + assert torch.allclose(scale, ref_s, atol=0.01, rtol=0.01) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_quantized_embedding.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_quantized_embedding.py new file mode 100644 index 0000000000000000000000000000000000000000..0a55f1355b1e3b659f1faf0268d5a05cd1b15dc0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_quantized_embedding.py @@ -0,0 +1,67 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the Triton dequant-gather kernel used by +``CompressedTensorsEmbeddingWNA16Int`` (quantized embedding lookup).""" + +import pytest +import torch +from compressed_tensors.compressors.pack_quantized.helpers import unpack_from_int32 + +from vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors_embedding import ( # noqa: E501 + _dequant_gather_triton, +) +from vllm.platforms import current_platform + + +def _dequant_gather_torch( + ids: torch.Tensor, + weight_packed: torch.Tensor, + weight_scale: torch.Tensor, + hidden: int, + num_bits: int, +) -> torch.Tensor: + """Reference: gather packed rows by id, unpack int32-packed INT, dequant.""" + n = ids.shape[0] + int8 = unpack_from_int32(weight_packed[ids], num_bits, torch.Size([n, hidden])) + scale_rows = weight_scale[ids] + w = int8.to(scale_rows.dtype) + if scale_rows.shape[1] == 1: + return w * scale_rows + ng = scale_rows.shape[1] + return (w.view(n, ng, hidden // ng) * scale_rows.unsqueeze(-1)).view(n, hidden) + + +@pytest.mark.skipif( + not current_platform.is_cuda(), reason="Triton dequant kernel requires CUDA" +) +@pytest.mark.parametrize("num_bits", [2, 4, 8]) +@pytest.mark.parametrize("group_size", [0, 256]) # 0 -> channel +@pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16]) +@pytest.mark.parametrize("num_ids", [1, 17, 4096]) +def test_dequant_gather(num_bits, group_size, dtype, num_ids): + torch.manual_seed(0) + device = "cuda" + vocab, hidden = 1000, 2048 + pack_factor = 32 // num_bits + + # Random full-range int32 packed weights (covers the sign bit -> exercises the + # arithmetic-shift + mask unpack path). + weight_packed = torch.randint( + -(2**31), + 2**31, + (vocab, hidden // pack_factor), + dtype=torch.int32, + device=device, + ) + + num_groups = 1 if group_size == 0 else hidden // group_size + weight_scale = torch.rand(vocab, num_groups, dtype=dtype, device=device) + 0.01 + + ids = torch.randint(0, vocab, (num_ids,), dtype=torch.long, device=device) + + out = _dequant_gather_triton(ids, weight_packed, weight_scale, hidden, num_bits) + ref = _dequant_gather_torch(ids, weight_packed, weight_scale, hidden, num_bits) + + assert out.shape == (num_ids, hidden) + assert out.dtype == dtype + torch.testing.assert_close(out, ref) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_rdna3_compile_guards.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_rdna3_compile_guards.py new file mode 100644 index 0000000000000000000000000000000000000000..be245de031d0b5fe9f415dd377a292ef583a4323 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_rdna3_compile_guards.py @@ -0,0 +1,503 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Compile-guard tests for the ROCm RDNA3 W4A16 kernels (dense + MoE). + +Verifies that the gfx1100 compilation and dispatch guards are hermetic: + - On gfx1100: all ops exist, dispatch selects RDNA3 kernels. + - On CDNA (gfx942/gfx950) or other non-gfx1100: ops must NOT exist, + dispatch must fall through to Triton/Marlin, and no RDNA3 code + path is reachable. + +The negative (non-gfx1100) tests verify at three layers: + 1. Compile-level: on non-gfx1100 hardware, the RDNA3 ops are absent + from the compiled _rocm_C extension — real binary verification. + 2. Static source analysis: parses CMakeLists.txt and torch_bindings.cpp + to verify that all RDNA3 .cu files and op registrations are inside + gfx1100-only guards. + 3. Runtime mock: patches on_gfx1100() to False and verifies that the + Python dispatch chain rejects the RDNA3 path. + +Run `pytest tests/kernels/quantization/test_rdna3_compile_guards.py`. +""" + +from pathlib import Path +from unittest.mock import patch + +import pytest +import regex as re +import torch + +import vllm +from vllm.platforms import current_platform + +if not current_platform.is_rocm(): + pytest.skip("RDNA3 compile-guard tests are ROCm-only", allow_module_level=True) + +from vllm.platforms.rocm import on_gfx1100 # noqa: E402 + +gfx1100_only = pytest.mark.skipif( + not on_gfx1100(), + reason="Requires gfx1100 hardware", +) + +not_gfx1100 = pytest.mark.skipif( + on_gfx1100(), + reason="This test verifies non-gfx1100 builds — skip on gfx1100", +) + +RDNA3_OPS = ["gptq_gemm_rdna3", "gptq_gemm_rdna3_wmma", "moe_gptq_gemm_rdna3"] +RDNA3_CU_FILES = [ + "q_gemm_rdna3.cu", + "q_gemm_rdna3_wmma.cu", + "moe_q_gemm_rdna3.cu", +] + + +def _find_repo_root() -> Path | None: + """Walk up from this file to find the repo root (has CMakeLists.txt).""" + for parent in [Path(__file__).resolve(), *Path(__file__).resolve().parents]: + if (parent / "CMakeLists.txt").exists() and (parent / "csrc").is_dir(): + return parent + return None + + +REPO_ROOT = _find_repo_root() + +# Directory of the *installed* vllm python package. The .py guard checks read +# from here so they verify the code that is actually imported at runtime — this +# works even on CI images that ship the wheel instead of the python source tree +# (where only csrc/ + CMakeLists.txt are checked out for building). +VLLM_PKG_DIR: Path | None = ( + Path(vllm.__file__).parent if getattr(vllm, "__file__", None) else None +) + +needs_source = pytest.mark.skipif( + REPO_ROOT is None, + reason="C/CMake source tree not available (installed package only)", +) + + +def _read_source_or_skip(*relparts: str) -> str: + """Read a C/CMake source file from the repo tree, or skip if absent. + + Used for csrc/ and CMakeLists.txt — these only exist in a source checkout, + not in the installed wheel. + """ + assert REPO_ROOT is not None # callers are gated by @needs_source + path = REPO_ROOT.joinpath(*relparts) + if not path.exists(): + pytest.skip(f"{path} not present in this source tree") + return path.read_text() + + +def _read_pkg_source_or_skip(*relparts: str) -> str: + """Read a python source file from the installed vllm package. + + Reflects the code actually loaded at runtime, so these guard checks run in + CI against the wheel — no source checkout required. Only skips for an + exotic install layout (namespace/zipimport) where __file__ is unavailable. + """ + if VLLM_PKG_DIR is None: + pytest.skip("vllm package directory not resolvable (zip/namespace?)") + assert VLLM_PKG_DIR is not None # narrow for mypy (skip above is NoReturn) + path = VLLM_PKG_DIR.joinpath(*relparts) + if not path.exists(): + pytest.skip(f"{path} not present in installed vllm package") + return path.read_text() + + +# ============================================================================ +# Part A: POSITIVE — on gfx1100, ops exist and dispatch works +# ============================================================================ + + +@gfx1100_only +@pytest.mark.parametrize("op_name", RDNA3_OPS) +def test_op_registered_on_gfx1100(op_name): + """On gfx1100, all RDNA3 ops must be registered in _rocm_C.""" + assert hasattr(torch.ops, "_rocm_C"), "_rocm_C module not loaded" + assert hasattr(torch.ops._rocm_C, op_name), ( + f"_rocm_C.{op_name} not registered — " + "check CMakeLists.txt VLLM_ROCM_HAS_GFX1100 " + "and torch_bindings.cpp #ifdef VLLM_ROCM_GFX1100" + ) + + +@gfx1100_only +def test_all_ops_present_or_all_absent(): + """The 3 RDNA3 ops are behind the same #ifdef — all present or all absent. + + Catches someone accidentally moving an op outside the guard. + """ + has_rocm_c = hasattr(torch.ops, "_rocm_C") + if not has_rocm_c: + pytest.skip("_rocm_C not loaded") + + present = {op: hasattr(torch.ops._rocm_C, op) for op in RDNA3_OPS} + values = set(present.values()) + assert len(values) == 1, ( + f"Guard inconsistency — some RDNA3 ops registered, others not: " + f"{present}. Check torch_bindings.cpp #ifdef VLLM_ROCM_GFX1100 block." + ) + + +# ============================================================================ +# Part B: NEGATIVE — compile-level verification on non-gfx1100 +# ============================================================================ + + +@not_gfx1100 +@pytest.mark.parametrize("op_name", RDNA3_OPS) +def test_op_absent_on_non_gfx1100(op_name): + """On non-gfx1100 (CDNA), RDNA3 ops must NOT exist in _rocm_C. + + This is the real compile-level check: the binary was built without + gfx1100 support, so the ops should not have been compiled or registered. + """ + if not hasattr(torch.ops, "_rocm_C"): + return + assert not hasattr(torch.ops._rocm_C, op_name), ( + f"_rocm_C.{op_name} is registered on non-gfx1100 hardware — " + "compile guard is broken: check CMakeLists.txt " + "VLLM_ROCM_HAS_GFX1100 and torch_bindings.cpp #ifdef" + ) + + +@not_gfx1100 +def test_rocm_moe_not_supported_on_non_gfx1100(): + """rocm_moe_rdna.is_supported() must return False on non-gfx1100 hardware.""" + from vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors_moe import ( # noqa: E501 + rocm_moe_rdna, + ) + + wq = type("WQ", (), {"num_bits": 4})() + assert rocm_moe_rdna.is_supported(wq) is False, ( + "rocm_moe_rdna.is_supported() returned True on non-gfx1100 — " + "dispatch guard is broken" + ) + + +@not_gfx1100 +def test_dense_kernel_rejects_on_non_gfx1100(): + """RDNA3W4A16LinearKernel.can_implement must reject on non-gfx1100.""" + from vllm.model_executor.kernels.linear.mixed_precision.MPLinearKernel import ( # noqa: E501 + MPLinearLayerConfig, + ) + from vllm.model_executor.kernels.linear.mixed_precision.rdna3_w4a16 import ( # noqa: E501 + RDNA3W4A16LinearKernel, + ) + from vllm.scalar_type import scalar_types + + config = MPLinearLayerConfig( + full_weight_shape=(1024, 256), + partition_weight_shape=(1024, 256), + weight_type=scalar_types.uint4b8, + act_type=torch.float16, + group_size=128, + zero_points=False, + has_g_idx=False, + ) + ok, reason = RDNA3W4A16LinearKernel.can_implement(config) + assert ok is False, f"RDNA3 dense kernel accepted on non-gfx1100: {reason}" + + +# ============================================================================ +# Part C: Static source analysis (build-level guards) +# ============================================================================ + + +@needs_source +class TestCMakeGuards: + """Verify CMakeLists.txt only compiles RDNA3 .cu files for gfx1100.""" + + @staticmethod + def _read_cmake(): + return _read_source_or_skip("CMakeLists.txt") + + def test_rdna3_cu_files_inside_gfx1100_conditional(self): + """All RDNA3 .cu files must be listed inside the + ``if(VLLM_GPU_ARCHES MATCHES "gfx1100")`` block, not unconditionally. + """ + cmake = self._read_cmake() + for cu_file in RDNA3_CU_FILES: + assert cu_file in cmake, f"{cu_file} not found in CMakeLists.txt" + + lines = cmake.splitlines() + in_gfx1100_block = False + for line in lines: + if 'VLLM_GPU_ARCHES MATCHES "gfx1100"' in line: + in_gfx1100_block = True + if in_gfx1100_block and "endif()" in line: + in_gfx1100_block = False + if cu_file in line: + assert in_gfx1100_block, ( + f"{cu_file} is listed OUTSIDE the gfx1100 " + f"conditional in CMakeLists.txt — CDNA builds " + f"would compile RDNA3 code. Line: {line.strip()}" + ) + + def test_compile_definition_only_for_gfx1100(self): + """VLLM_ROCM_GFX1100 compile definition must be conditional.""" + cmake = self._read_cmake() + lines = cmake.splitlines() + in_gfx1100_block = False + for line in lines: + if "VLLM_ROCM_HAS_GFX1100)" in line: + in_gfx1100_block = True + if in_gfx1100_block and "endif()" in line: + in_gfx1100_block = False + if "VLLM_ROCM_GFX1100" in line and "target_compile_definitions" in line: + assert in_gfx1100_block, ( + "VLLM_ROCM_GFX1100 compile definition is set outside " + "the VLLM_ROCM_HAS_GFX1100 conditional — CDNA builds " + f"would define it. Line: {line.strip()}" + ) + + +@needs_source +class TestTorchBindingsGuards: + """Verify torch_bindings.cpp gates all RDNA3 ops behind #ifdef.""" + + @staticmethod + def _read_bindings(): + return _read_source_or_skip("csrc", "rocm", "torch_bindings.cpp") + + def test_all_rdna3_ops_inside_ifdef(self): + """Every rdna3 op def/impl must be between #ifdef VLLM_ROCM_GFX1100 + and #endif. If any is outside, a CDNA build would try to register + the op and link a symbol that doesn't exist. + """ + src = self._read_bindings() + lines = src.splitlines() + + inside_guard = False + rdna3_lines_outside = [] + + for i, line in enumerate(lines, 1): + if "#ifdef VLLM_ROCM_GFX1100" in line: + inside_guard = True + elif line.strip() == "#endif" and inside_guard: + inside_guard = False + + if ( + "rdna3" in line.lower() + and not line.strip().startswith("//") + and not inside_guard + ): + rdna3_lines_outside.append((i, line.strip())) + + assert not rdna3_lines_outside, ( + "RDNA3 op references found OUTSIDE #ifdef VLLM_ROCM_GFX1100 " + "in torch_bindings.cpp — these would break CDNA builds:\n" + + "\n".join(f" L{n}: {s}" for n, s in rdna3_lines_outside) + ) + + def test_no_unconditional_rdna3_includes(self): + """No #include of RDNA3-specific headers outside the guard.""" + src = self._read_bindings() + lines = src.splitlines() + + inside_guard = False + for i, line in enumerate(lines, 1): + if "#ifdef VLLM_ROCM_GFX1100" in line: + inside_guard = True + elif line.strip() == "#endif" and inside_guard: + inside_guard = False + + if "#include" in line and "rdna3" in line.lower(): + assert inside_guard, ( + f"L{i}: RDNA3 include outside gfx1100 guard: {line.strip()}" + ) + + +class TestCustomOpsGuards: + """Verify _custom_ops.py gates register_fake behind hasattr checks.""" + + @staticmethod + def _read_custom_ops(): + return _read_pkg_source_or_skip("_custom_ops.py") + + def test_register_fake_guarded_by_hasattr(self): + """Every register_fake for an RDNA3 op must be preceded by a hasattr + check — otherwise it would crash on import on CDNA where the ops + don't exist. + """ + src = self._read_custom_ops() + for op in RDNA3_OPS: + pattern = rf'register_fake\(\s*"_rocm_C::{op}"\s*\)' + match = re.search(pattern, src) + if match is None: + continue + preceding = src[: match.start()] + last_hasattr = preceding.rfind(f'hasattr(torch.ops._rocm_C, "{op}")') + assert last_hasattr != -1, ( + f'register_fake("_rocm_C::{op}") is not preceded by a ' + f"hasattr check — would crash on CDNA import" + ) + gap = preceding[last_hasattr:].count("\n") + assert gap <= 5, ( + f"hasattr guard for {op} is {gap} lines before " + f"register_fake — suspiciously far; verify it's the " + f"actual guard and not a coincidence" + ) + + def test_no_toplevel_rocm_c_import(self): + """No top-level ``from vllm._rocm_C import`` — would crash on CDNA.""" + src = self._read_custom_ops() + for line in src.splitlines(): + stripped = line.strip() + if stripped.startswith("#") or stripped.startswith("//"): + continue + assert "from vllm._rocm_C import" not in stripped, ( + f"Top-level import of _rocm_C in _custom_ops.py would " + f"crash on CDNA: {stripped}" + ) + + +# ============================================================================ +# Part D: Runtime mock (simulate CDNA on gfx1100 hardware) +# ============================================================================ + + +class _FakeWeightQuant: + """Minimal stand-in for a weight quantization config.""" + + def __init__(self, num_bits): + self.num_bits = num_bits + + +class TestMoEDispatchMocked: + """Mock on_gfx1100() to False and verify RDNA3 MoE is unreachable.""" + + def test_is_supported_false_when_mocked_cdna(self): + """rocm_moe_rdna.is_supported() must return False when not on gfx1100.""" + from vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors_moe import ( # noqa: E501 + rocm_moe_rdna, + ) + + with patch("vllm.platforms.rocm.on_gfx1100", return_value=False): + assert rocm_moe_rdna.is_supported(_FakeWeightQuant(num_bits=4)) is False + + @pytest.mark.parametrize("num_bits", [2, 3, 8, 16]) + def test_is_supported_rejects_non_w4(self, num_bits): + """is_supported() rejects non-4-bit even before checking arch.""" + from vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors_moe import ( # noqa: E501 + rocm_moe_rdna, + ) + + assert rocm_moe_rdna.is_supported(_FakeWeightQuant(num_bits=num_bits)) is False + + def test_is_supported_false_when_op_missing(self): + """is_supported() returns False when the C++ op doesn't exist.""" + from vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors_moe import ( # noqa: E501 + rocm_moe_rdna, + ) + + fake_rocm_c = type("FakeRocmC", (), {"gptq_gemm_rdna3": None})() + with patch.object(torch, "ops", create=True) as mock_ops: + mock_ops._rocm_C = fake_rocm_c + assert rocm_moe_rdna.is_supported(_FakeWeightQuant(num_bits=4)) is False + + def test_is_supported_false_when_rocm_c_absent(self): + """is_supported() returns False when _rocm_C doesn't exist at all.""" + from vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors_moe import ( # noqa: E501 + rocm_moe_rdna, + ) + + fake_ops = type("FakeOps", (), {})() + with patch.object(torch, "ops", fake_ops): + assert rocm_moe_rdna.is_supported(_FakeWeightQuant(num_bits=4)) is False + + +class TestDenseKernelSelectionMocked: + """Mock on_gfx1100() and verify dense RDNA3 kernel is not selected.""" + + @gfx1100_only + def test_can_implement_rejects_when_mocked_cdna(self): + """RDNA3W4A16LinearKernel.can_implement must reject on mocked CDNA.""" + from vllm.model_executor.kernels.linear.mixed_precision.MPLinearKernel import ( # noqa: E501 + MPLinearLayerConfig, + ) + from vllm.model_executor.kernels.linear.mixed_precision.rdna3_w4a16 import ( # noqa: E501 + RDNA3W4A16LinearKernel, + ) + from vllm.scalar_type import scalar_types + + config = MPLinearLayerConfig( + full_weight_shape=(1024, 256), + partition_weight_shape=(1024, 256), + weight_type=scalar_types.uint4b8, + act_type=torch.float16, + group_size=128, + zero_points=False, + has_g_idx=False, + ) + ok, _ = RDNA3W4A16LinearKernel.can_implement(config) + assert ok is True + + with ( + patch("vllm.platforms.rocm.on_gfx1100", return_value=False), + patch("vllm.platforms.rocm._ON_GFX1100", False), + ): + ok, reason = RDNA3W4A16LinearKernel.can_implement(config) + assert ok is False, f"RDNA3 kernel accepted on simulated CDNA: {reason}" + + @gfx1100_only + def test_chooser_skips_rdna3_when_mocked_cdna(self): + """choose_mp_linear_kernel must NOT return RDNA3 on mocked CDNA.""" + from vllm.model_executor.kernels.linear import ( + choose_mp_linear_kernel, + ) + from vllm.model_executor.kernels.linear.mixed_precision.MPLinearKernel import ( # noqa: E501 + MPLinearLayerConfig, + ) + from vllm.scalar_type import scalar_types + + config = MPLinearLayerConfig( + full_weight_shape=(1024, 256), + partition_weight_shape=(1024, 256), + weight_type=scalar_types.uint4b8, + act_type=torch.float16, + group_size=128, + zero_points=False, + has_g_idx=False, + ) + with ( + patch("vllm.platforms.rocm.on_gfx1100", return_value=False), + patch("vllm.platforms.rocm._ON_GFX1100", False), + ): + chosen = choose_mp_linear_kernel(config) + assert chosen.__name__ != "RDNA3W4A16LinearKernel", ( + "RDNA3 kernel was selected on simulated CDNA — " + "choose_mp_linear_kernel guard is broken" + ) + + +class TestCompressedTensorsMoEDispatchGuard: + """Verify compressed_tensors_moe.py only enters rocm_moe_rdna under is_rocm().""" + + def test_rocm_guard_in_dispatch_source(self): + """The rocm_moe_rdna import and call must be inside an is_rocm() check.""" + src = _read_pkg_source_or_skip( + "model_executor", + "layers", + "quantization", + "compressed_tensors", + "compressed_tensors_moe", + "compressed_tensors_moe.py", + ) + lines = src.splitlines() + + for i, line in enumerate(lines, 1): + stripped = line.strip() + if "rocm_moe" in stripped and not stripped.startswith("#"): + found_guard = False + for j in range(i - 1, max(0, i - 15), -1): + if "is_rocm()" in lines[j - 1]: + found_guard = True + break + assert found_guard, ( + f"L{i}: rocm_moe_rdna reference not protected by " + f"is_rocm() guard: {stripped}" + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_rdna3_moe_w4a16.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_rdna3_moe_w4a16.py new file mode 100644 index 0000000000000000000000000000000000000000..5b304d8a13b7fed6e1e0467205344b58345301ac --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_rdna3_moe_w4a16.py @@ -0,0 +1,367 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Correctness tests for the ROCm RDNA3 fused MoE W4A16 HIP kernel (gfx1100). + +Tests ``moe_gptq_gemm_rdna3`` against the dense ``gptq_gemm_rdna3`` as +reference: builds RDNA3-format weights (shuffled int32, synthesized qzeros), +runs the fused MoE kernel, and compares per-expert results. + +Model parameters taken from: + - cyankiwi/Qwen3-30B-A3B-Instruct-2507-AWQ-4bit + (hidden=2048, inter=768, E=128, top_k=8, G=32) + - Qwen3.6-35B-A3B-GPTQ-W4A16-G32 + (hidden=2048, inter=512, E=256, top_k=8, G=32) + +Run `pytest tests/kernels/quantization/test_rdna3_moe_w4a16.py`. +""" + +import pytest +import torch + +from vllm.platforms import current_platform + +if not current_platform.is_rocm(): + pytest.skip("RDNA3 MoE W4A16 kernel is ROCm-only", allow_module_level=True) + +from vllm import _custom_ops as ops # noqa: E402 +from vllm.model_executor.layers.fused_moe.activation import ( # noqa: E402 + MoEActivation, + apply_moe_activation, +) +from vllm.model_executor.layers.fused_moe.moe_align_block_size import ( # noqa: E402 + moe_align_block_size, +) +from vllm.model_executor.layers.quantization.utils.quant_utils import ( # noqa: E402 + pack_quantized_values_into_int32, +) +from vllm.platforms.rocm import on_gfx1100 # noqa: E402 +from vllm.scalar_type import scalar_types # noqa: E402 + +device = "cuda" + +gfx1100_only = pytest.mark.skipif( + not ( + on_gfx1100() + and hasattr(torch.ops, "_rocm_C") + and hasattr(torch.ops._rocm_C, "moe_gptq_gemm_rdna3") + ), + reason="Requires gfx1100 with moe_gptq_gemm_rdna3 op", +) + +# Model configurations: real K/N/top_k/group_size dims, E capped at 16 to +# fit in test GPU memory (full E=128/256 would need >20GB for weights alone). +# Kernel behavior is E-independent (per-expert tiling), so E=16 is sufficient. +MODEL_CONFIGS = [ + # cyankiwi/Qwen3-30B-A3B-Instruct-2507-AWQ-4bit dims (E capped) + pytest.param(16, 2048, 768, 8, 32, id="Qwen3-30B-A3B"), + # Qwen3.6-35B-A3B-GPTQ-W4A16-G32 dims (E capped) + pytest.param(16, 2048, 512, 8, 32, id="Qwen3.6-35B-A3B"), +] + +# Token counts: decode (1), small batch (4), medium (16), prefill (64) +NUM_TOKENS = [1, 4, 16, 64, 256, 512] + + +def _make_packed_weights(E, K, N): + """Create random 4-bit packed weights [E, K/8, N] int32 + shuffle.""" + w = torch.randint(0, 16, (E, K, N), dtype=torch.int32, device=device) + packed = torch.zeros(E, K // 8, N, dtype=torch.int32, device=device) + for i in range(8): + packed |= (w[:, i::8, :] & 0xF) << (i * 4) + g_idx = torch.empty(0, dtype=torch.int32, device=device) + for e in range(E): + we = packed[e].contiguous() + ops.gptq_shuffle(we, g_idx, 4) + packed[e] = we + return packed + + +def _make_scales(E, groups, N, dtype): + return torch.rand(E, groups, N, dtype=dtype, device=device) * 0.1 + + +def _make_qzeros(E, groups, N): + zeros = torch.full( + (groups, N), + scalar_types.uint4b8.bias - 1, + dtype=torch.int32, + device=device, + ) + qz = pack_quantized_values_into_int32( + zeros, + scalar_types.uint4b8, + packed_dim=1, + ) + return qz.unsqueeze(0).expand(E, -1, -1).contiguous() + + +@gfx1100_only +@pytest.mark.parametrize("E, K, N_inter, top_k, group_size", MODEL_CONFIGS) +@pytest.mark.parametrize("M", NUM_TOKENS) +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@pytest.mark.parametrize("block_size_m", [1, 4]) +def test_fused_moe_w1_matches_dense( + E, K, N_inter, top_k, group_size, M, dtype, block_size_m +): + """w1 GEMM via fused kernel matches per-expert dense kernel.""" + N_gate_up = N_inter * 2 + groups = K // group_size + + torch.manual_seed(42) + x = torch.randn(M, K, dtype=dtype, device=device) + w13 = _make_packed_weights(E, K, N_gate_up) + w13_s = _make_scales(E, groups, N_gate_up, dtype) + w13_z = _make_qzeros(E, groups, N_gate_up) + g_idx = torch.empty(0, dtype=torch.int32, device=device) + + topk_ids = torch.randint(0, E, (M, top_k), device=device, dtype=torch.int32) + si, ei, ntp = moe_align_block_size(topk_ids, block_size_m, E) + + # Fused kernel + fused_out = torch.zeros(M * top_k, N_gate_up, dtype=dtype, device=device) + ops.moe_gptq_gemm_rdna3( + x, + fused_out, + w13, + w13_s, + w13_z, + torch.empty(0, device=device), + si, + ei, + ntp, + top_k, + block_size_m, + False, + 0, + ) + + # Per-expert dense reference + ref_out = torch.zeros(M * top_k, N_gate_up, dtype=dtype, device=device) + for m in range(M): + for k in range(top_k): + e = topk_ids[m, k].item() + flat = m * top_k + k + ref = ops.gptq_gemm_rdna3( + x[m : m + 1], + w13[e], + w13_z[e], + w13_s[e], + g_idx, + False, + ) + ref_out[flat] = ref.squeeze() + + # Split-K atomics can cause minor fp16/bf16 rounding differences + # at large K (e.g. K=2048 → 8 K-blocks). Use allclose, not equal. + atol = 0.5 if dtype == torch.bfloat16 else 0.1 + assert torch.allclose(fused_out, ref_out, atol=atol, rtol=0.01), ( + f"max diff: {(fused_out - ref_out).abs().max().item()}" + ) + + +@gfx1100_only +@pytest.mark.parametrize("E, K, N_inter, top_k, group_size", MODEL_CONFIGS) +@pytest.mark.parametrize("M", NUM_TOKENS) +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +def test_fused_moe_output_topk_reduces(E, K, N_inter, top_k, group_size, M, dtype): + """output_topk fuses moe_sum: multiple experts write to same output row.""" + groups = K // group_size + + torch.manual_seed(123) + x = torch.randn(M * top_k, K, dtype=dtype, device=device) + w = _make_packed_weights(E, K, N_inter) + ws = _make_scales(E, groups, N_inter, dtype) + wz = _make_qzeros(E, groups, N_inter) + + topk_ids = torch.randint(0, E, (M, top_k), device=device, dtype=torch.int32) + topk_w = torch.softmax( + torch.randn(M, top_k, device=device), + dim=-1, + ).float() + + si, ei, ntp = moe_align_block_size(topk_ids, 1, E) + + # Without output_topk: write to [M*top_k, N] then moe_sum + flat_out = torch.zeros(M * top_k, N_inter, dtype=dtype, device=device) + ops.moe_gptq_gemm_rdna3( + x, + flat_out, + w, + ws, + wz, + topk_w.view(-1), + si, + ei, + ntp, + 1, + 1, + True, + 0, + ) + ref = torch.zeros(M, N_inter, dtype=dtype, device=device) + ops.moe_sum(flat_out.view(M, top_k, N_inter), ref) + + # With output_topk: write directly to [M, N] + fused = torch.zeros(M, N_inter, dtype=dtype, device=device) + ops.moe_gptq_gemm_rdna3( + x, + fused, + w, + ws, + wz, + topk_w.view(-1), + si, + ei, + ntp, + 1, + 1, + True, + top_k, + ) + + atol = 1.0 if dtype == torch.bfloat16 else 0.1 + assert torch.allclose(fused, ref, atol=atol, rtol=0.01), ( + f"max diff: {(fused - ref).abs().max().item()}" + ) + + +@gfx1100_only +@pytest.mark.parametrize("E, K, N_inter, top_k, group_size", MODEL_CONFIGS) +@pytest.mark.parametrize("M", NUM_TOKENS) +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +def test_full_moe_e2e(E, K, N_inter, top_k, group_size, M, dtype): + """Full MoE forward: w1 + silu_and_mul + w2 with output_topk reduce.""" + N_gate_up = N_inter * 2 + hidden = K + + torch.manual_seed(7) + x = torch.randn(M, K, dtype=dtype, device=device) + w13 = _make_packed_weights(E, K, N_gate_up) + w13_s = _make_scales(E, K // group_size, N_gate_up, dtype) + w13_z = _make_qzeros(E, K // group_size, N_gate_up) + w2 = _make_packed_weights(E, N_inter, hidden) + w2_s = _make_scales(E, N_inter // group_size, hidden, dtype) + w2_z = _make_qzeros(E, N_inter // group_size, hidden) + g_idx = torch.empty(0, dtype=torch.int32, device=device) + + topk_ids = torch.randint(0, E, (M, top_k), device=device, dtype=torch.int32) + topk_w = torch.softmax( + torch.randn(M, top_k, device=device), + dim=-1, + ).float() + + si, ei, ntp = moe_align_block_size(topk_ids, 1, E) + + # Fused path (what apply() does) + w1_out = torch.zeros(M * top_k, N_gate_up, dtype=dtype, device=device) + ops.moe_gptq_gemm_rdna3( + x, + w1_out, + w13, + w13_s, + w13_z, + torch.empty(0, device=device), + si, + ei, + ntp, + top_k, + 1, + False, + 0, + ) + act_out = torch.empty(M * top_k, N_inter, dtype=dtype, device=device) + apply_moe_activation(MoEActivation.SILU, act_out, w1_out) + fused = torch.zeros(M, hidden, dtype=dtype, device=device) + ops.moe_gptq_gemm_rdna3( + act_out, + fused, + w2, + w2_s, + w2_z, + topk_w.view(-1), + si, + ei, + ntp, + 1, + 1, + True, + top_k, + ) + + # Per-expert reference + ref = torch.zeros(M, hidden, dtype=dtype, device=device) + for m_idx in range(M): + for k_idx in range(top_k): + e = topk_ids[m_idx, k_idx].item() + w = topk_w[m_idx, k_idx].item() + r1 = ops.gptq_gemm_rdna3( + x[m_idx : m_idx + 1], + w13[e], + w13_z[e], + w13_s[e], + g_idx, + False, + ) + a = torch.empty(1, N_inter, dtype=dtype, device=device) + apply_moe_activation(MoEActivation.SILU, a, r1) + r2 = ops.gptq_gemm_rdna3( + a, + w2[e], + w2_z[e], + w2_s[e], + g_idx, + False, + ) + ref[m_idx] += r2.squeeze() * w + + # E2E chains w1 + activation + w2 + topk_w + output_topk reduce. + # Each step accumulates rounding error (split-K atomics, topk_w + # multiply order). Use relative L2 norm like the dense kernel test. + diff_l2 = torch.norm(fused.float() - ref.float()) + ref_l2 = torch.norm(ref.float()) + rel_l2 = (diff_l2 / ref_l2).item() if ref_l2 > 0 else 0.0 + threshold = 0.05 if dtype == torch.float16 else 0.10 + assert rel_l2 < threshold, ( + f"rel L2 = {rel_l2:.4f} (threshold {threshold}), " + f"max abs diff: {(fused - ref).abs().max().item()}" + ) + + +@gfx1100_only +def test_expert_id_minus_one(): + """Kernel handles expert_id == -1 (expert parallelism) without crash.""" + # Qwen3-30B-A3B dims (E capped for memory) + E, K, N = 16, 2048, 768 + groups = K // 32 + + w = _make_packed_weights(E, K, N) + ws = _make_scales(E, groups, N, torch.bfloat16) + wz = _make_qzeros(E, groups, N) + x = torch.randn(1, K, dtype=torch.bfloat16, device=device) + + # Manually create sorted_token_ids/expert_ids with -1 + sorted_ids = torch.tensor([0], dtype=torch.int32, device=device) + expert_ids = torch.tensor([-1], dtype=torch.int32, device=device) + ntp = torch.tensor([1], dtype=torch.int32, device=device) + + out = torch.zeros(1, N, dtype=torch.bfloat16, device=device) + ops.moe_gptq_gemm_rdna3( + x, + out, + w, + ws, + wz, + torch.empty(0, device=device), + sorted_ids, + expert_ids, + ntp, + 1, + 1, + False, + 0, + ) + current_platform.synchronize() + + # Output should remain zero (expert skipped) + assert torch.equal(out, torch.zeros_like(out)) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_rdna3_w4a16.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_rdna3_w4a16.py new file mode 100644 index 0000000000000000000000000000000000000000..4d328d5f2235bd1a738c124acc53491b80da3ee0 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_rdna3_w4a16.py @@ -0,0 +1,278 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Correctness tests for the ROCm RDNA3 W4A16 GPTQ kernel (gfx1100). + +Exercises ``RDNA3W4A16LinearKernel`` end-to-end: it builds a layer with +GPTQ-format checkpoint parameters, runs ``process_weights_after_loading`` +(weight shuffle + zero-point synthesis), then ``apply_weights``, and compares +the result against an fp32 reference dequant-and-matmul. + +The kernel is exposed via ``torch.ops._rocm_C.gptq_gemm_rdna3`` and is only +built for gfx11; tests are skipped elsewhere. + +Run `pytest tests/kernels/quantization/test_rdna3_w4a16.py`. +""" + +import pytest +import torch + +from vllm.platforms import current_platform + +if not current_platform.is_rocm(): + pytest.skip("RDNA3 W4A16 kernel is ROCm-only", allow_module_level=True) + +from vllm.model_executor.kernels.linear.mixed_precision.MPLinearKernel import ( # noqa: E402 + MPLinearLayerConfig, +) +from vllm.model_executor.kernels.linear.mixed_precision.rdna3_w4a16 import ( # noqa: E402 + RDNA3W4A16LinearKernel, +) +from vllm.model_executor.layers.quantization.utils.quant_utils import ( # noqa: E402 + pack_quantized_values_into_int32, +) +from vllm.model_executor.parameter import ( # noqa: E402 + GroupQuantScaleParameter, + PackedvLLMParameter, +) +from vllm.platforms.rocm import on_gfx1100 # noqa: E402 +from vllm.scalar_type import scalar_types # noqa: E402 +from vllm.utils.torch_utils import set_random_seed # noqa: E402 + +device = "cuda" + +WEIGHT_TYPE = scalar_types.uint4b8 # symmetric int4, bias = 8 +PACK_FACTOR = 8 # 8 x 4-bit nibbles per int32 + +# Skip everything in this module unless we are on the only architecture the +# kernel is built/registered for. +gfx1100_only = pytest.mark.skipif( + not ( + on_gfx1100() + and hasattr(torch.ops, "_rocm_C") + and hasattr(torch.ops._rocm_C, "gptq_gemm_rdna3") + ), + reason="requires gfx1100 with the _rocm_C.gptq_gemm_rdna3 op built in", +) + + +# --------------------------------------------------------------------------- +# Reference implementation +# --------------------------------------------------------------------------- + + +def _reference( + x_mk: torch.Tensor, + q_int4_kn: torch.Tensor, + scales_gn: torch.Tensor, + zeros_gn: torch.Tensor | None, + group_size: int, + bias: torch.Tensor | None, +) -> torch.Tensor: + """fp32 reference for the RDNA3 W4A16 op. + + x_mk: [M, K] fp16/bf16 activations. + q_int4_kn: [K, N] int32 raw stored nibbles in [0, 15]. + scales_gn: [K//G, N] per-group scales (act dtype). + zeros_gn: [K//G, N] int32 raw stored zero points in [0, 15], or None + for the symmetric path (kernel synthesizes stored zero = 7). + group_size: G. + + The kernel applies the GPTQv1 "+1" zero-point quirk, so the effective + zero is ``stored_zero + 1`` (symmetric path: 7 + 1 == bias == 8). + """ + K, N = q_int4_kn.shape + s_full = scales_gn.repeat_interleave(group_size, dim=0).to(torch.float32) # [K,N] + if zeros_gn is None: + z_full = torch.full( + (K, N), float(WEIGHT_TYPE.bias), device=x_mk.device, dtype=torch.float32 + ) + else: + z_full = (zeros_gn + 1).repeat_interleave(group_size, dim=0).to(torch.float32) + w_fp = (q_int4_kn.to(torch.float32) - z_full) * s_full # [K, N] + out = x_mk.to(torch.float32) @ w_fp # [M, N] + if bias is not None: + out = out + bias.to(torch.float32) + return out.to(x_mk.dtype) + + +# --------------------------------------------------------------------------- +# Layer construction (GPTQ checkpoint format) +# --------------------------------------------------------------------------- + + +def _build_layer( + q_int4_kn: torch.Tensor, + scales_gn: torch.Tensor, + zeros_gn: torch.Tensor | None, + dtype: torch.dtype, +) -> torch.nn.Module: + """Build a dummy layer carrying GPTQ-format params, as the loader would.""" + no_loader = lambda *args, **kwargs: None # noqa: E731 + + # qweight: int4 packed along K into int32 -> [K//8, N]. + qweight = pack_quantized_values_into_int32(q_int4_kn, WEIGHT_TYPE, packed_dim=0) + + class DummyLayer(torch.nn.Module): + pass + + layer = DummyLayer() + layer.register_parameter( + "qweight", + PackedvLLMParameter( + data=qweight, + weight_loader=no_loader, + input_dim=0, + output_dim=1, + packed_dim=0, + packed_factor=PACK_FACTOR, + ), + ) + layer.register_parameter( + "scales", + GroupQuantScaleParameter( + data=scales_gn.to(dtype), + weight_loader=no_loader, + input_dim=0, + output_dim=1, + ), + ) + if zeros_gn is not None: + # qzeros: int4 packed along N into int32 -> [K//G, N//8]. + qzeros = pack_quantized_values_into_int32(zeros_gn, WEIGHT_TYPE, packed_dim=1) + layer.register_parameter( + "qzeros", + PackedvLLMParameter( + data=qzeros, + weight_loader=no_loader, + input_dim=0, + output_dim=1, + packed_dim=1, + packed_factor=PACK_FACTOR, + ), + ) + return layer + + +def _run_kernel( + x_mk: torch.Tensor, + q_int4_kn: torch.Tensor, + scales_gn: torch.Tensor, + zeros_gn: torch.Tensor | None, + group_size: int, + bias: torch.Tensor | None, + dtype: torch.dtype, +) -> torch.Tensor: + K, N = q_int4_kn.shape + has_zp = zeros_gn is not None + + config = MPLinearLayerConfig( + full_weight_shape=(K, N), + partition_weight_shape=(K, N), + weight_type=WEIGHT_TYPE, + act_type=dtype, + group_size=group_size, + zero_points=has_zp, + has_g_idx=False, + ) + ok, reason = RDNA3W4A16LinearKernel.can_implement(config) + assert ok, f"can_implement rejected a supported config: {reason}" + + layer = _build_layer(q_int4_kn, scales_gn, zeros_gn, dtype) + kernel = RDNA3W4A16LinearKernel( + config, + w_q_param_name="qweight", + w_s_param_name="scales", + w_zp_param_name="qzeros" if has_zp else None, + w_gidx_param_name=None, + ) + kernel.process_weights_after_loading(layer) + return kernel.apply_weights(layer, x_mk, bias=bias) + + +# Relative-L2 tolerance per dtype. The bf16 path widens dequantized weights +# to fp32 and accumulates in fp32, so it matches the reference almost exactly +# (<0.4% incl. the WMMA prefill path). The fp16 path uses the exllamav2 +# "+1024" bit-trick (see qdq_4_rdna3.cuh): the dequantized weight is recovered +# as the fp16 difference of two ~1024*scale magnitudes, which sheds low-order +# mantissa bits and leaves ~2-3% relative noise that accumulates over K. We +# compare on the relative Frobenius norm rather than elementwise, since the +# bit-trick noise produces large *relative* errors on individual near-zero +# outputs that carry negligible absolute weight. +_REL_L2_TOL = {torch.float16: 5e-2, torch.bfloat16: 1e-2} + + +def _assert_close(out: torch.Tensor, ref: torch.Tensor, dtype: torch.dtype): + rel_l2 = (out.to(torch.float32) - ref.to(torch.float32)).norm() / ref.to( + torch.float32 + ).norm() + tol = _REL_L2_TOL[dtype] + assert rel_l2 < tol, f"relative L2 error {rel_l2:.4f} exceeds {tol} for {dtype}" + + +# --------------------------------------------------------------------------- +# Forward correctness +# --------------------------------------------------------------------------- + + +# (M, K, N, group_size). M spans the scalar decode path (small M) and the +# WMMA prefill path (M >= 16 on the bf16 dispatch). K/N satisfy the kernel's +# divisibility constraints (K % G == 0, K % 8 == 0, N % 8 == 0). +MKNG_SHAPES = [ + (1, 128, 128, 128), # single group, decode + (2, 256, 256, 128), # two groups + (8, 256, 512, 64), # M=8 scalar, smaller group + (16, 512, 256, 128), # M=16 -> WMMA path for bf16 + (32, 512, 512, 64), # larger prefill +] + + +@gfx1100_only +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@pytest.mark.parametrize("has_zp", [False, True], ids=["no_zp", "with_zp"]) +@pytest.mark.parametrize( + "M,K,N,G", MKNG_SHAPES, ids=[f"m{m}_k{k}_n{n}_g{g}" for m, k, n, g in MKNG_SHAPES] +) +def test_rdna3_w4a16_matches_reference(dtype, has_zp, M, K, N, G, dist_init): + set_random_seed(0) + assert K % G == 0 and K % PACK_FACTOR == 0 and N % PACK_FACTOR == 0 + + groups = K // G + x_mk = (0.25 * torch.randn((M, K), device=device, dtype=torch.float32)).to(dtype) + q_int4_kn = torch.randint(0, 16, (K, N), device=device, dtype=torch.int32) + scales_gn = ( + 0.05 * torch.rand((groups, N), device=device, dtype=torch.float32) + 0.01 + ).to(dtype) + zeros_gn = ( + torch.randint(0, 16, (groups, N), device=device, dtype=torch.int32) + if has_zp + else None + ) + + out = _run_kernel(x_mk, q_int4_kn, scales_gn, zeros_gn, G, None, dtype) + ref = _reference(x_mk, q_int4_kn, scales_gn, zeros_gn, G, None) + + assert out.shape == (M, N) and out.dtype == dtype + _assert_close(out, ref, dtype) + + +@gfx1100_only +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@pytest.mark.parametrize("M", [1, 32], ids=["decode", "prefill"]) +def test_rdna3_w4a16_bias(dtype, M, dist_init): + """Bias is added on both the scalar (M=1) and WMMA (M=32) paths.""" + set_random_seed(0) + K, N, G = 512, 256, 128 + groups = K // G + + x_mk = (0.25 * torch.randn((M, K), device=device, dtype=torch.float32)).to(dtype) + q_int4_kn = torch.randint(0, 16, (K, N), device=device, dtype=torch.int32) + scales_gn = ( + 0.05 * torch.rand((groups, N), device=device, dtype=torch.float32) + 0.01 + ).to(dtype) + bias = (0.1 * torch.randn(N, device=device, dtype=torch.float32)).to(dtype) + + out = _run_kernel(x_mk, q_int4_kn, scales_gn, None, G, bias, dtype) + ref = _reference(x_mk, q_int4_kn, scales_gn, None, G, bias) + + _assert_close(out, ref, dtype) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_rdna3_w4a16_selection.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_rdna3_w4a16_selection.py new file mode 100644 index 0000000000000000000000000000000000000000..4ab979748de457b2dabe186ca46236aab6d185b5 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_rdna3_w4a16_selection.py @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Kernel-selection / gating tests for the ROCm RDNA3 W4A16 GPTQ kernel. + +Verifies that ``choose_mp_linear_kernel`` resolves a supported W4A16 GPTQ +config to ``RDNA3W4A16LinearKernel`` on gfx1100 (it is registered ahead of +``TritonW4A16LinearKernel`` in the ROCm priority list), and that +``RDNA3W4A16LinearKernel.can_implement`` rejects the configs it does not +support so selection falls through to the next kernel. + +Run `pytest tests/kernels/quantization/test_rdna3_w4a16_selection.py`. +""" + +import pytest +import torch + +from vllm.platforms import current_platform + +if not current_platform.is_rocm(): + pytest.skip("RDNA3 W4A16 kernel is ROCm-only", allow_module_level=True) + +from vllm.model_executor.kernels.linear import ( # noqa: E402 + choose_mp_linear_kernel, +) +from vllm.model_executor.kernels.linear.mixed_precision.MPLinearKernel import ( # noqa: E402 + MPLinearLayerConfig, +) +from vllm.model_executor.kernels.linear.mixed_precision.rdna3_w4a16 import ( # noqa: E402 + RDNA3W4A16LinearKernel, +) +from vllm.platforms.rocm import on_gfx1100 # noqa: E402 +from vllm.scalar_type import scalar_types # noqa: E402 + +WEIGHT_TYPE = scalar_types.uint4b8 # symmetric int4, bias = 8 + +# The kernel is only selectable when running on gfx1100 with the custom op +# compiled in; otherwise can_implement rejects and selection falls through. +gfx1100_only = pytest.mark.skipif( + not ( + on_gfx1100() + and hasattr(torch.ops, "_rocm_C") + and hasattr(torch.ops._rocm_C, "gptq_gemm_rdna3") + ), + reason="requires gfx1100 with the _rocm_C.gptq_gemm_rdna3 op built in", +) + + +@gfx1100_only +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +def test_selection_prefers_rdna3(dtype): + """A supported W4A16 GPTQ config resolves to the RDNA3 kernel on gfx1100.""" + config = MPLinearLayerConfig( + full_weight_shape=(1024, 256), + partition_weight_shape=(1024, 256), + weight_type=WEIGHT_TYPE, + act_type=dtype, + group_size=128, + zero_points=False, + has_g_idx=False, + ) + assert choose_mp_linear_kernel(config).__name__ == "RDNA3W4A16LinearKernel" + + +@gfx1100_only +@pytest.mark.parametrize( + "weight_type,group_size,N,full_k,expected_ok", + [ + (scalar_types.uint4b8, 128, 256, 1024, True), # nominal: supported + (scalar_types.uint4b8, -1, 256, 1024, False), # channelwise unsupported + (scalar_types.uint4b8, 128, 252, 1024, False), # N not a multiple of 8 + (scalar_types.uint4b8, 96, 256, 1024, False), # group does not divide K + (scalar_types.uint8b128, 128, 256, 1024, False), # wrong quant type + ], + ids=["ok", "channelwise", "bad_n", "group_ndiv_k", "wrong_qtype"], +) +def test_can_implement(weight_type, group_size, N, full_k, expected_ok): + """can_implement gates on quant type, group size, and N divisibility.""" + config = MPLinearLayerConfig( + full_weight_shape=(full_k, N), + partition_weight_shape=(full_k, N), + weight_type=weight_type, + act_type=torch.float16, + group_size=group_size, + zero_points=False, + has_g_idx=False, + ) + ok, reason = RDNA3W4A16LinearKernel.can_implement(config) + assert ok is expected_ok, reason diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_rocm_compressed_tensors_w4a16.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_rocm_compressed_tensors_w4a16.py new file mode 100644 index 0000000000000000000000000000000000000000..f550a465289fe59e3fe52a847142fff9c4299e61 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_rocm_compressed_tensors_w4a16.py @@ -0,0 +1,35 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""End-to-end smoke test for CT W4A16 models on ROCm. + +This validates that a real compressed-tensors W4A16 model can run inference +end-to-end (which will exercise the Triton W4A16 kernel when selected). + +Run `pytest tests/kernels/quantization/test_rocm_compressed_tensors_w4a16.py`. +""" + +import pytest + +from vllm.platforms import current_platform + + +@pytest.mark.parametrize( + "model_path", + [ + # Listed in tests/weight_loading/models.txt + "nm-testing/tinyllama-oneshot-w4a16-group128-v2", + ], +) +@pytest.mark.parametrize("max_tokens", [32]) +@pytest.mark.skipif(not current_platform.is_rocm(), reason="Should only run on ROCm") +def test_rocm_compressed_tensors_w4a16_e2e( + vllm_runner, example_prompts, model_path, max_tokens +): + # Use fp16 activations for maximum compatibility. + # gpu_memory_utilization lowered to work on shared nodes. + with vllm_runner( + model_path, dtype="float16", gpu_memory_utilization=0.3 + ) as vllm_model: + # If the W4A16 kernel is broken, this will typically throw. + vllm_model.generate_greedy(example_prompts, max_tokens=max_tokens) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_rocm_skinny_gemms.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_rocm_skinny_gemms.py new file mode 100644 index 0000000000000000000000000000000000000000..5903bec7a9b3a8d9d2dd2d8b73042c0d902ca741 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_rocm_skinny_gemms.py @@ -0,0 +1,274 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import math + +import pytest +import torch + +import vllm._custom_ops as ops +from tests.kernels.quant_utils import ref_dynamic_per_tensor_fp8_quant +from vllm.platforms import current_platform +from vllm.platforms.rocm import on_gfx950 +from vllm.utils.platform_utils import num_compute_units + +DTYPES = [torch.bfloat16, torch.float16] +BIAS_MODES = [0, 1, 2] +# Specific (N, K, M) combinations for targeted testing +NKM_FACTORS_LLMM1 = [ + # Small, medium, large cases + (1, 8, 16), + (1, 32, 64), + (1, 128, 256), + (1, 512, 1024), + (1, 2048, 4096), + # Edge cases with specific K sizes + (1, 6144, 1024), + (1, 8192, 2048), + # Very large case + (1, 4096, 8192), +] + +NKM_FACTORS_WVSPLITK = [ + # Different batch sizes with key dimensions + (1, 32, 16), + (1, 64, 64), + (2, 256, 256), + (3, 1024, 1024), + (4, 4096, 4096), + (4, 4096, 4096 + 1), + (4, 4096 + 16, 4096), + (4, 4096 + 16, 4096 + 1), + # Extended K values + (1, 9216, 512), + (2, 10240, 1024), + (4, 16384, 8192), + (4, 16384 * 2, 8192), + (4, 16384 * 2, 8192 + 1), + (4, 16384 * 2 + 16, 8192), + (4, 16384 * 2 + 16, 8192 + 1), + # Minimum M constraint validation (m >= 8) + (1, 64, 8), + (2, 128, 8), + (4, 256, 8), +] + +N_FACTORS_WVSPLITKRC = [ + 13, + 16, + 17, + 25, + 29, + 31, + 32, + 41, + 51, + 64, + 71, + 81, + 91, + 103, + 117, + 128, +] +K_FACTORS_WVSPLITKRC = [2880, 2880 + 8, 3072, 3072 + 8] +M_FACTORS_WVSPLITKRC = [128, 128 + 16, 256, 256 + 16, 640, 640 + 16] + +NKM_FACTORS_WVSPLITK_FP8 = [ + # FP8-specific cases with K % 16 == 0 + (1, 16, 16), + (1, 32, 16 + 16), + (1, 64, 64), + (1, 64, 64 + 16), + (1, 64 + 16, 64), + (1, 64 + 16, 64 + 16), + (4, 64, 64), + (4, 64, 64 + 16), + (4, 64 + 16, 64), + (4, 64 + 16, 64 + 16), + (2, 512, 512), + (3, 512, 512), + (3, 512, 512 + 16), + (4, 512, 512), + (3, 2048, 2048), + (3, 2048, 2048 + 16), + (4, 2048 + 16, 2048), + (4, 2048 + 16, 2048 + 16), + (4, 4096, 4096), + (4, 16400, 2048), + (4, 16400, 2048 + 16), + # Extended FP8 dimensions not covered by WVSPLITK + (1, 14336, 1024), + (2, 24576, 2048), + (4, 32768, 28672), + (4, 32768 * 2, 28672), + (4, 32768 * 2, 28672 + 16), + (4, 32768 * 2 + 16, 28672), + (4, 32768 * 2 + 16, 28672 + 16), +] + +SEEDS = [0] + + +def pad_fp8(weight): + num_pad = 256 // weight.element_size() + import torch.nn.functional as F + + return F.pad(weight, (0, num_pad), "constant", 0)[..., :-num_pad] + + +@pytest.mark.parametrize("xnorm", [False, True]) +@pytest.mark.parametrize("n", N_FACTORS_WVSPLITKRC) +@pytest.mark.parametrize("k", K_FACTORS_WVSPLITKRC) +@pytest.mark.parametrize("m", M_FACTORS_WVSPLITKRC) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("padded_a", [False, True]) +@pytest.mark.parametrize("bias_mode", BIAS_MODES) +@pytest.mark.skipif(not current_platform.is_rocm(), reason="only test for rocm") +@pytest.mark.skipif(not on_gfx950(), reason="only meant for gfx950") +def test_rocm_wvsplitkrc_kernel(xnorm, n, k, m, dtype, seed, padded_a, bias_mode): + torch.manual_seed(seed) + cu_count = num_compute_units() + + # Next ^2 of n + N_p2 = 1 << (n - 1).bit_length() + # With 64 Ms per CU (each of 4 SIMDs working on a 16x16 tile), + # and each working on a 512-shard of K, how many CUs would we need? + rndup_cus = ((m + 64 - 1) // 64) * ((k + 512 - 1) // 512) + # How many of 4 waves in a group can work on same 16 Ms at same time? + # This reduces the Ms each group works on, i.e. increasing the number of CUs needed. + GrpsShrB = min(N_p2 // 16, 4) + # Given the above, how many CUs would we need? + CuNeeded = rndup_cus * GrpsShrB + # candidate for atomic reduce count splitk? + fits_wvsplitkrc = (N_p2 * m * ((k + 512 - 1) // 512)) <= 128 * 1024 * 12 + fits_wvsplitkrc &= CuNeeded <= cu_count + + if not fits_wvsplitkrc: + pytest.skip("Too large for wvSplitKrc") + + xavier = ( + math.sqrt(2 / k) if xnorm else 1 + ) # normalize to avoid large output-bias deltas + A = (torch.rand(n, k, dtype=dtype, device="cuda") * 2 - 1) * xavier + B = (torch.rand(m, k, dtype=dtype, device="cuda") * 2 - 1) * xavier + if padded_a: + A = pad_fp8(A) + + BIAS = None + if bias_mode == 1: + BIAS = torch.rand(m, dtype=dtype, device="cuda") * 2 - 1 + elif bias_mode == 2: + BIAS = torch.rand(n, m, dtype=dtype, device="cuda") * 2 - 1 + elif bias_mode == 3: + BIAS = torch.rand(1, m, dtype=dtype, device="cuda") * 2 - 1 + + ref_out = torch.nn.functional.linear(A, B, BIAS) + out = ops.wvSplitKrc(A, B, cu_count, BIAS) + + if xnorm: + torch.testing.assert_close(out, ref_out, atol=1e-3, rtol=1e-8) + else: + torch.testing.assert_close(out, ref_out, atol=1e-3, rtol=1e-2) + + +@pytest.mark.parametrize("n,k,m", NKM_FACTORS_LLMM1) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("rows_per_block", [2, 4, 8, 16]) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.skipif(not current_platform.is_rocm(), reason="only test for rocm") +@torch.inference_mode() +def test_rocm_llmm1_kernel(n, k, m, dtype, rows_per_block, seed): + torch.manual_seed(seed) + # TODO: Zero-centering the inputs causes errors for LLMM1! + # Without that the numbers quickly saturate, and may + # be giving false matches. + A = torch.rand(n, k, dtype=dtype, device="cuda") + B = torch.rand(m, k, dtype=dtype, device="cuda") + + ref_out = torch.matmul(A, B.t()) + out = ops.LLMM1(B, A, rows_per_block) + + torch.testing.assert_close(out, ref_out, atol=1e-8, rtol=1e-2) + + +@pytest.mark.parametrize("xnorm", [False, True]) +@pytest.mark.parametrize("n,k,m", NKM_FACTORS_WVSPLITK) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.skipif(not current_platform.is_rocm(), reason="only test for rocm") +@pytest.mark.parametrize("bias_mode", BIAS_MODES) +@pytest.mark.parametrize("padded_a", [False, True]) +@pytest.mark.parametrize("padded_b", [False, True]) +def test_rocm_wvsplitk_kernel( + xnorm, n, k, m, dtype, seed, bias_mode, padded_a, padded_b +): + torch.manual_seed(seed) + cu_count = num_compute_units() + + xavier = ( + math.sqrt(2 / k) if xnorm else 1 + ) # normalize to avoid large output-bias deltas + A = (torch.rand(n, k, dtype=dtype, device="cuda") * 2 - 1) * xavier + B = (torch.rand(m, k, dtype=dtype, device="cuda") * 2 - 1) * xavier + + BIAS = None + if bias_mode == 1: + BIAS = torch.rand(m, dtype=dtype, device="cuda") * 2 - 1 + elif bias_mode == 2: + BIAS = torch.rand(n, m, dtype=dtype, device="cuda") * 2 - 1 + + if padded_a: + A = pad_fp8(A) + if padded_b: + B = pad_fp8(B) + + ref_out = torch.nn.functional.linear(A, B, BIAS) + out = ops.wvSplitK(B, A.view(-1, A.size(-1)), cu_count, BIAS) + + # Accumulation error in fp16 GEMM scales with sqrt(K) + atol = torch.finfo(dtype).eps * math.sqrt(k) + torch.testing.assert_close(out, ref_out, atol=atol, rtol=1e-2) + + +@pytest.mark.parametrize("xnorm", [False, True]) +@pytest.mark.parametrize("n,k,m", NKM_FACTORS_WVSPLITK_FP8) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("padded_a", [False, True]) +@pytest.mark.parametrize("padded_b", [False, True]) +@pytest.mark.parametrize("biased", [False, True]) +@pytest.mark.skipif( + not (current_platform.is_rocm() and current_platform.supports_fp8()), + reason="only test for rocm fp8", +) +def test_rocm_wvsplitk_fp8_kernel( + xnorm, n, k, m, dtype, seed, padded_a, padded_b, biased +): + torch.manual_seed(seed) + + xavier = math.sqrt(2 / k) if xnorm else 1 # normalize to avoid large deltas + A = (torch.rand(n, k, device="cuda") * 2 - 1) * xavier + B = (torch.rand(m, k, device="cuda") * 2 - 1) * xavier + + A, scale_a = ref_dynamic_per_tensor_fp8_quant(A) + B, scale_b = ref_dynamic_per_tensor_fp8_quant(B) + if padded_b: + B = pad_fp8(B) + if padded_a: + A = pad_fp8(A) + + BIAS = None if (not biased) else (torch.rand(m, dtype=dtype, device="cuda") * 2 - 1) + + ref_out = torch._scaled_mm( + A, B.t(), out_dtype=dtype, scale_a=scale_a, scale_b=scale_b, bias=BIAS + ) + out = ops.wvSplitKQ(B, A, dtype, scale_a, scale_b, num_compute_units(), BIAS) + + if xnorm: + torch.testing.assert_close(out, ref_out, atol=1e-3, rtol=1e-8) + elif k >= 32 * 1024: + # wider pytrch thresh for large-K & no xnorm + torch.testing.assert_close(out, ref_out, atol=0.07, rtol=5e-2) + else: + torch.testing.assert_close(out, ref_out, atol=1e-2, rtol=1e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_scaled_mm_kernel_selection.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_scaled_mm_kernel_selection.py new file mode 100644 index 0000000000000000000000000000000000000000..4ca97edb291785ca78ae9b56cfaf0438a97b2ca1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_scaled_mm_kernel_selection.py @@ -0,0 +1,129 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for ScaledMM kernel selection logic (CPU-only) + +Run `pytest tests/kernels/quantization/test_scaled_mm_kernel_selection.py`. +""" + +import inspect +from abc import ABC +from unittest.mock import patch + +import pytest +import torch + +from vllm.model_executor.kernels.linear import ( + AiterInt8ScaledMMLinearKernel, + CPUInt8ScaledMMLinearKernel, + Int8ScaledMMLinearKernel, + Int8ScaledMMLinearLayerConfig, + ScaledMMLinearKernel, + init_int8_linear_kernel, + register_linear_kernel, +) +from vllm.platforms import PlatformEnum + +pytestmark = pytest.mark.cpu_test + + +def test_is_supported_is_abstract(): + """Test that is_supported() is properly defined as abstract.""" + assert issubclass(ScaledMMLinearKernel, ABC) + assert hasattr(ScaledMMLinearKernel, "is_supported") + + +def test_cpu_kernel_implements_is_supported(): + """Test that CPUInt8ScaledMMLinearKernel implements is_supported() method.""" + assert hasattr(CPUInt8ScaledMMLinearKernel, "is_supported"), ( + "CPUInt8ScaledMMLinearKernel missing is_supported() method" + ) + # Verify it's a classmethod by checking if it can be called with the class + # and by checking the method type + assert inspect.ismethod( + CPUInt8ScaledMMLinearKernel.is_supported + ) or inspect.isfunction(CPUInt8ScaledMMLinearKernel.is_supported), ( + "CPUInt8ScaledMMLinearKernel.is_supported() should be a classmethod" + ) + # Verify it can be called as a classmethod + result, reason = CPUInt8ScaledMMLinearKernel.is_supported() + assert isinstance(result, bool), "is_supported() should return a bool" + assert reason is None or isinstance(reason, str), "reason should be str or None" + + +def test_aiter_kernel_implements_is_supported(): + """Test that AiterInt8ScaledMMLinearKernel implements is_supported() method.""" + assert hasattr(AiterInt8ScaledMMLinearKernel, "is_supported"), ( + "AiterInt8ScaledMMLinearKernel missing is_supported() method" + ) + # Verify it's a classmethod by checking if it can be called with the class + # and by checking the method type + assert inspect.ismethod( + AiterInt8ScaledMMLinearKernel.is_supported + ) or inspect.isfunction(AiterInt8ScaledMMLinearKernel.is_supported), ( + "AiterInt8ScaledMMLinearKernel.is_supported() should be a classmethod" + ) + # Verify it can be called as a classmethod + # (will return False on CPU, which is expected) + result, reason = AiterInt8ScaledMMLinearKernel.is_supported() + assert isinstance(result, bool), "is_supported() should return a bool" + assert reason is None or isinstance(reason, str), "reason should be str or None" + # On CPU, it should return False with a reason about requiring ROCm + # This validates the method works correctly even on non-ROCm platforms + + +def test_cpu_kernel_accepts_all_configs(): + """Test that CPUInt8ScaledMMLinearKernel accepts all config combinations.""" + configs = [ + Int8ScaledMMLinearLayerConfig( + is_channelwise=False, + is_static_input_scheme=True, + input_symmetric=True, + ), + Int8ScaledMMLinearLayerConfig( + is_channelwise=True, + is_static_input_scheme=False, + input_symmetric=False, + ), + ] + + for config in configs: + can_impl, reason = CPUInt8ScaledMMLinearKernel.can_implement(config) + assert can_impl, ( + f"CPUInt8ScaledMMLinearKernel should accept config {config}: {reason}" + ) + + +class OOTInt8ScaledMMLinearKernel(Int8ScaledMMLinearKernel): + @classmethod + def is_supported( + cls, compute_capability: int | None = None + ) -> tuple[bool, str | None]: + return True, None + + @classmethod + def can_implement(cls, c: Int8ScaledMMLinearLayerConfig) -> tuple[bool, str | None]: + return True, None + + def process_weights_after_loading(self, layer: torch.nn.Module) -> None: + pass + + def apply_weights( + self, + layer: torch.nn.Module, + x: torch.Tensor, + bias: torch.Tensor | None = None, + ) -> torch.Tensor: + pass + + +@patch("vllm.model_executor.kernels.linear.current_platform") +def test_register_oot_linear_kernel(platform_mock): + """Test that the linear kernel registration works correctly.""" + platform_mock._enum = PlatformEnum.OOT + register_linear_kernel(OOTInt8ScaledMMLinearKernel, PlatformEnum.OOT, "int8") + + kernel = init_int8_linear_kernel(True, True, True, "module") + + assert isinstance(kernel, OOTInt8ScaledMMLinearKernel), ( + "init_int8_linear_kernel should return an instance of the registered kernel" + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_silu_mul_nvfp4_quant.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_silu_mul_nvfp4_quant.py new file mode 100644 index 0000000000000000000000000000000000000000..734be07f7cf25fc2d46870e394b3a662148615f1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_silu_mul_nvfp4_quant.py @@ -0,0 +1,78 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch + +from tests.kernels.quantization.nvfp4_utils import ( + FLOAT4_E2M1_MAX, + FLOAT8_E4M3_MAX, + dequantize_nvfp4_to_dtype, +) +from vllm._custom_ops import scaled_fp4_quant +from vllm.model_executor.layers.activation import SiluAndMul +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +if not current_platform.has_device_capability(100): + pytest.skip( + reason="Nvfp4 Requires compute capability of 10 or above.", + allow_module_level=True, + ) + +FP4_DTYPE = torch.uint8 +FP8_DTYPE = current_platform.fp8_dtype() + +DTYPES = [torch.float16, torch.bfloat16] +SHAPES = [(128, 256), (128, 128), (256, 256), (256, 128)] +BLOCK_SIZE = 16 + + +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("shape", SHAPES) +@torch.inference_mode() +def test_silu_mul_nvfp4_quant( + default_vllm_config, + dtype: torch.dtype, + shape: tuple[int, int], +) -> None: + set_random_seed(42) + device = "cuda:0" + torch.set_default_device(device) + + x = torch.randn(shape, dtype=dtype) + + # ref op + ref_output = SiluAndMul().forward_native(x) + ref_global_scale = (FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX) / torch.abs( + ref_output + ).max().to(torch.float32) + ref_output_quant, ref_block_scale = scaled_fp4_quant(ref_output, ref_global_scale) + + # fused op + fused_output_quant = torch.empty_like(ref_output_quant) + fused_block_scale = torch.empty_like(ref_block_scale) + torch.ops._C.silu_and_mul_nvfp4_quant( + fused_output_quant, fused_block_scale, x, ref_global_scale + ) + + # check dtype + assert ref_output_quant.dtype == FP4_DTYPE + assert fused_output_quant.dtype == FP4_DTYPE + assert ref_output_quant.shape == fused_output_quant.shape + + assert ref_block_scale.dtype == FP8_DTYPE + assert fused_block_scale.dtype == FP8_DTYPE + assert ref_block_scale.shape == fused_block_scale.shape + + # check dequantized output + ref_output_dequant = dequantize_nvfp4_to_dtype( + ref_output_quant, ref_block_scale, ref_global_scale, dtype, device + ) + fused_output_dequant = dequantize_nvfp4_to_dtype( + fused_output_quant, fused_block_scale, ref_global_scale, dtype, device + ) + + atol, rtol = 3e-1, 3e-1 + torch.testing.assert_close( + ref_output_dequant, fused_output_dequant, atol=atol, rtol=rtol + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_triton_scaled_mm.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_triton_scaled_mm.py new file mode 100644 index 0000000000000000000000000000000000000000..b84d65b8e6c1644f80db2aa9e04059a031204d7e --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_triton_scaled_mm.py @@ -0,0 +1,125 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the triton_scaled_mm kernel + +Run `pytest tests/kernels/quantization/test_triton_scaled_mm.py`. +""" + +import importlib + +import pytest +import torch + +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +device = "cuda" + +triton_scaled_mm_module = importlib.import_module( + "vllm.model_executor.layers.quantization.compressed_tensors.triton_scaled_mm" +) +triton_scaled_mm = triton_scaled_mm_module.triton_scaled_mm + + +def torch_scaled_mm( + a: torch.Tensor, + b: torch.Tensor, + scale_a: torch.Tensor, + scale_b: torch.Tensor, + out_dtype: type[torch.dtype], + bias: torch.Tensor | None = None, +) -> torch.Tensor: + out = torch.mm(a.to(torch.float32), b.to(torch.float32)) + out = scale_a * out + out = scale_b.T * out + out = out.to(out_dtype) + if bias is not None: + out = out + bias + + return out + + +def get_8bit_types(): + types = [torch.int8] + if current_platform.supports_fp8(): + types.append(current_platform.fp8_dtype()) + return types + + +# This test is to check regressions for int8 support on ROCm. +@pytest.mark.parametrize( + "model_path", + [ + "neuralmagic/Llama-3.2-1B-quantized.w8a8", + ], +) +@pytest.mark.parametrize("max_tokens", [32]) +@pytest.mark.parametrize("num_logprobs", [10]) +@pytest.mark.skipif(not current_platform.is_rocm(), reason="Should only run on ROCm") +def test_rocm_compressed_tensors_w8a8( + vllm_runner, example_prompts, model_path, max_tokens, num_logprobs +): + dtype = "bfloat16" + + with vllm_runner(model_path, dtype=dtype) as vllm_model: + vllm_model.generate_greedy_logprobs(example_prompts, max_tokens, num_logprobs) + + +MNK_FACTORS = [ + (1, 256, 128), + (33, 256, 496), + (64, 971, 1024), + (64, 20486, 128), + (512, 256, 496), + (512, 20486, 1024), +] + + +@pytest.mark.parametrize("M,N,K", MNK_FACTORS) +@pytest.mark.parametrize("out_dtype", [torch.bfloat16]) +@pytest.mark.parametrize("in_dtype", get_8bit_types()) +@pytest.mark.parametrize("use_scalar_scale_a", [True, False]) +@pytest.mark.parametrize("use_scalar_scale_b", [True, False]) +@pytest.mark.parametrize("use_bias", [True, False]) +def test_scaled_mm( + M, N, K, in_dtype, out_dtype, use_scalar_scale_a, use_scalar_scale_b, use_bias +): + is_floating_point_type = lambda t: torch.tensor([1, 1], dtype=t).is_floating_point() + + set_random_seed(0) + + # NOTE: There are cases, where if the matrix is large enough, an output + # like 65504.4 can be produced, and can easily turn into inf when + # multiplied when using float16/bfloat16. This means one function, e.g., + # testing function, and another function, e.g. golden function, can + # produce a non-inf value while the other produces an inf value, and + # will cause assert_close/allclose to fail, even though if overflow + # wouldn't have occurred, the values would have been "close." + # + # So, the values here are kept small enough to avoid this situation. + if is_floating_point_type(in_dtype): + a = (0.25 * torch.rand((M, K), dtype=torch.float32, device=device)).to(in_dtype) + b = (0.25 * torch.rand((K, N), dtype=torch.float32, device=device)).to(in_dtype) + else: + a = torch.randint(-32, 32, (M, K), dtype=in_dtype, device=device) + b = torch.randint(-32, 32, (K, N), dtype=in_dtype, device=device) + + if use_scalar_scale_a: + scale_a = torch.rand((1, 1), device=device) + else: + scale_a = 0.25 * torch.rand((M, 1), device=device) + + if use_scalar_scale_b: + scale_b = torch.rand((1, 1), device=device) + else: + scale_b = 0.25 * torch.rand((N, 1), device=device) + + bias = None + if use_bias: + bias = torch.rand((N,), device=device, dtype=out_dtype) + + c_check = triton_scaled_mm(a, b, scale_a, scale_b, out_dtype, bias) + + c_actual = torch_scaled_mm(a, b, scale_a, scale_b, out_dtype, bias) + + torch.testing.assert_close(c_check, c_actual, rtol=1e-1, atol=1e-1) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_triton_w4a16.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_triton_w4a16.py new file mode 100644 index 0000000000000000000000000000000000000000..5fa39c974dd7296d4115a8b534f121131c000899 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_triton_w4a16.py @@ -0,0 +1,304 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the ROCm Triton W4A16 GEMM kernel. + +Run `pytest tests/kernels/quantization/test_triton_w4a16.py`. +""" + +import importlib + +import pytest +import torch + +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +# This test module is ROCm/Triton specific. Avoid import-time failures on +# non-ROCm or environments without Triton by skipping early. +if not current_platform.is_rocm(): + pytest.skip("ROCm only", allow_module_level=True) + +pytest.importorskip("triton") + +device = "cuda" + +triton_w4a16_module = importlib.import_module( + "vllm.model_executor.kernels.linear.mixed_precision.triton_w4a16" +) +triton_w4a16_gemm = triton_w4a16_module.triton_w4a16_gemm +TritonW4A16LinearKernel = triton_w4a16_module.TritonW4A16LinearKernel + + +def _pack_int4_along_n(w_int4_kn: torch.Tensor) -> torch.Tensor: + """Pack int4 values along N: [K, N] -> [K, N//8] int32.""" + assert w_int4_kn.dtype == torch.int32 + K, N = w_int4_kn.shape + assert N % 8 == 0 + shifts = torch.arange(8, device=w_int4_kn.device, dtype=torch.int32) * 4 + return torch.sum( + (w_int4_kn.view(K, N // 8, 8) & 0xF) << shifts, + dim=2, + dtype=torch.int32, + ).contiguous() + + +def _unpack_int4_along_n(w_packed_kn8: torch.Tensor) -> torch.Tensor: + """Unpack int4 values along N: [K, N//8] -> [K, N] int32.""" + assert w_packed_kn8.dtype == torch.int32 + K, N8 = w_packed_kn8.shape + shifts = torch.arange(8, device=w_packed_kn8.device, dtype=torch.int32) * 4 + nibbles = (w_packed_kn8.unsqueeze(-1) >> shifts) & 0xF + return nibbles.reshape(K, N8 * 8) + + +def _pack_int4_along_k_to_ckpt(w_int4_kn: torch.Tensor) -> torch.Tensor: + """Pack int4 values along K into CT checkpoint layout: [K,N] -> [N, K//8].""" + assert w_int4_kn.dtype == torch.int32 + K, N = w_int4_kn.shape + assert K % 8 == 0 + out = torch.zeros((N, K // 8), dtype=torch.int32, device=w_int4_kn.device) + for i in range(8): + out |= (w_int4_kn[i::8, :].t() & 0xF) << (i * 4) + return out.contiguous() + + +def _w4a16_reference( + a_mk: torch.Tensor, + b_packed_kn8: torch.Tensor, + scales_gn: torch.Tensor, + *, + group_size: int, + qzeros_gn8: torch.Tensor | None, + zp_bias: int, +) -> torch.Tensor: + """Reference implementation for W4A16. + + a_mk: [M,K] fp16/bf16 + b_packed_kn8: [K, N//8] int32, N-packed int4 weights + scales_gn: [K//G, N] fp16/bf16 + qzeros_gn8: [K//G, N//8] int32, N-packed int4 zeros, or None + """ + assert a_mk.dtype in (torch.float16, torch.bfloat16) + assert b_packed_kn8.dtype == torch.int32 + assert scales_gn.dtype == a_mk.dtype + + M, K = a_mk.shape + N = b_packed_kn8.shape[1] * 8 + assert b_packed_kn8.shape[0] == K + + assert group_size > 0 and K % group_size == 0 + G = group_size + num_groups = K // G + assert scales_gn.shape == (num_groups, N) + + w_int4 = _unpack_int4_along_n(b_packed_kn8) # [K,N] + if qzeros_gn8 is None: + z_full = torch.full((K, N), zp_bias, dtype=torch.int32, device=a_mk.device) + else: + assert qzeros_gn8.shape == (num_groups, N // 8) + z_gn = _unpack_int4_along_n(qzeros_gn8) # [G,N] in groups + z_full = z_gn.repeat_interleave(G, dim=0) # [K,N] + + s_full = scales_gn.repeat_interleave(G, dim=0).to(torch.float32) # [K,N] + w_fp = (w_int4 - z_full).to(torch.float32) * s_full # [K,N] + + out = a_mk.to(torch.float32) @ w_fp # [M,N] + return out.to(a_mk.dtype) + + +@pytest.mark.skipif(not current_platform.is_rocm(), reason="ROCm only") +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@pytest.mark.parametrize( + "M,K,N,G,has_zp", + [ + (1, 256, 256, 32, False), + (17, 256, 512, 32, False), + (32, 512, 256, 64, False), + (33, 512, 512, 128, False), + (64, 1024, 256, 256, False), + (128, 256, 1024, 32, True), + (64, 512, 512, 64, True), + ], +) +def test_triton_w4a16_gemm_matches_reference(dtype, M, K, N, G, has_zp): + if not torch.cuda.is_available(): + pytest.skip("CUDA/HIP device not available") + if N % 8 != 0 or K % G != 0: + pytest.skip("Invalid test shape") + + set_random_seed(0) + + a = (0.25 * torch.randn((M, K), device=device, dtype=torch.float32)).to(dtype) + w_int4 = torch.randint(0, 16, (K, N), device=device, dtype=torch.int32) + b_packed = _pack_int4_along_n(w_int4) + + scales = (0.05 * torch.rand((K // G, N), device=device, dtype=torch.float32)).to( + dtype + ) + + qzeros = None + if has_zp: + zeros_int4 = torch.randint(0, 16, (K // G, N), device=device, dtype=torch.int32) + qzeros = _pack_int4_along_n(zeros_int4) + + out = triton_w4a16_gemm( + a=a, + b_q=b_packed, + scales=scales, + qzeros=qzeros, + group_size=G, + zp_bias=8, + ) + ref = _w4a16_reference( + a, + b_packed, + scales, + group_size=G, + qzeros_gn8=qzeros, + zp_bias=8, + ) + + torch.testing.assert_close(out, ref, rtol=1e-2, atol=1e-2) + + +@pytest.mark.skipif(not current_platform.is_rocm(), reason="ROCm only") +def test_triton_w4a16_gemm_requires_contiguous_inputs(): + if not torch.cuda.is_available(): + pytest.skip("CUDA/HIP device not available") + + set_random_seed(0) + M, K, N, G = 32, 256, 256, 32 + a = torch.randn((K, M), device=device, dtype=torch.float16).t() # non-contiguous + w_int4 = torch.randint(0, 16, (K, N), device=device, dtype=torch.int32) + b_packed = _pack_int4_along_n(w_int4) + scales = torch.rand((K // G, N), device=device, dtype=torch.float16) + + with pytest.raises(AssertionError): + triton_w4a16_gemm( + a=a, + b_q=b_packed, + scales=scales, + qzeros=None, + group_size=G, + zp_bias=8, + ) + + +@pytest.mark.skipif(not current_platform.is_rocm(), reason="ROCm only") +def test_triton_w4a16_process_weights_after_loading_repacks_layout(): + if not torch.cuda.is_available(): + pytest.skip("CUDA/HIP device not available") + + from vllm.config import VllmConfig, set_current_vllm_config + from vllm.distributed import ( + ensure_model_parallel_initialized, + init_distributed_environment, + ) + from vllm.model_executor.kernels.linear.mixed_precision.MPLinearKernel import ( + MPLinearLayerConfig, + ) + from vllm.model_executor.parameter import ( + GroupQuantScaleParameter, + PackedColumnParameter, + PackedvLLMParameter, + ) + from vllm.scalar_type import scalar_types + + with set_current_vllm_config(VllmConfig()): + init_distributed_environment( + world_size=1, + rank=0, + distributed_init_method="tcp://127.0.0.1:0", + local_rank=0, + ) + ensure_model_parallel_initialized(1, 1) + + set_random_seed(0) + + # Small-but-nontrivial shapes. + K, N = 256, 256 + G = 32 + assert K % 8 == 0 and N % 8 == 0 and K % G == 0 + + # Build a canonical int4 weight grid then pack into the CT checkpoint layout. + w_int4_kn = torch.randint(0, 16, (K, N), device=device, dtype=torch.int32) + w_ckpt_nk8 = _pack_int4_along_k_to_ckpt(w_int4_kn) # [N, K//8] + + # Scales in CT checkpoint layout for WNA16: [N, K//G] + scales_ckpt_nkg = 0.05 * torch.rand((N, K // G), device=device, dtype=torch.float16) + + # Asymmetric case: zero points in CT checkpoint layout [N//8, K//G] (N-packed) + zeros_int4_gn = torch.randint(0, 16, (K // G, N), device=device, dtype=torch.int32) + zeros_packed_gn8 = _pack_int4_along_n(zeros_int4_gn) # [K//G, N//8] + zeros_ckpt_n8kg = zeros_packed_gn8.t().contiguous() # [N//8, K//G] + + config = MPLinearLayerConfig( + full_weight_shape=(K, N), + partition_weight_shape=(K, N), + weight_type=scalar_types.uint4, # asymmetric + act_type=torch.float16, + group_size=G, + zero_points=True, + has_g_idx=False, + ) + kernel = TritonW4A16LinearKernel( + config, + w_q_param_name="weight_packed", + w_s_param_name="weight_scale", + w_zp_param_name="weight_zero_point", + w_gidx_param_name=None, + ) + + # Build dummy layer with vLLM parameter wrappers. + weight_loader = lambda *args, **kwargs: None + + class DummyLayer(torch.nn.Module): + pass + + layer = DummyLayer() + layer.register_parameter( + "weight_packed", + PackedvLLMParameter( + data=w_ckpt_nk8, + weight_loader=weight_loader, + input_dim=1, + output_dim=0, + packed_factor=8, + packed_dim=1, + ), + ) + layer.register_parameter( + "weight_scale", + GroupQuantScaleParameter( + data=scales_ckpt_nkg, + weight_loader=weight_loader, + input_dim=1, + output_dim=0, + ), + ) + layer.register_parameter( + "weight_zero_point", + PackedColumnParameter( + data=zeros_ckpt_n8kg, + weight_loader=weight_loader, + output_dim=0, + packed_factor=8, + packed_dim=0, + ), + ) + + kernel.process_weights_after_loading(layer) + + # Expected transformed layouts. + expected_w_kn8 = _pack_int4_along_n(w_int4_kn) # [K, N//8] + expected_scales_gn = scales_ckpt_nkg.t().contiguous() # [K//G, N] + expected_zeros_gn8 = zeros_ckpt_n8kg.t().contiguous() # [K//G, N//8] + + assert tuple(layer.weight_packed.shape) == (K, N // 8) + assert tuple(layer.weight_scale.shape) == (K // G, N) + assert tuple(layer.weight_zero_point.shape) == (K // G, N // 8) + + torch.testing.assert_close(layer.weight_packed, expected_w_kn8) + torch.testing.assert_close(layer.weight_scale, expected_scales_gn) + torch.testing.assert_close(layer.weight_zero_point, expected_zeros_gn8) diff --git a/asian_ai/performance/vLLM/tests/kernels/quantization/test_w4a16_kernel_selection.py b/asian_ai/performance/vLLM/tests/kernels/quantization/test_w4a16_kernel_selection.py new file mode 100644 index 0000000000000000000000000000000000000000..afefd02ae6978fc3dc777458c3ddd8a8449adcb1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/quantization/test_w4a16_kernel_selection.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for W4A16 kernel selection logic (ROCm). + +Run `pytest tests/kernels/quantization/test_w4a16_kernel_selection.py`. +""" + +import pytest +import torch + +from vllm.model_executor.kernels.linear import ( + MPLinearLayerConfig, + choose_mp_linear_kernel, +) +from vllm.platforms import current_platform +from vllm.scalar_type import scalar_types + + +@pytest.mark.skipif(not current_platform.is_rocm(), reason="ROCm only") +def test_choose_mp_linear_kernel_picks_triton_w4a16_for_uint4b8(): + # int4 weights, 16-bit activations (CT W4A16 typical config). + K, N = 1024, 256 + config = MPLinearLayerConfig( + full_weight_shape=(K, N), + partition_weight_shape=(K, N), + weight_type=scalar_types.uint4b8, # symmetric int4 (bias=8) + act_type=torch.float16, + group_size=128, + zero_points=False, + has_g_idx=False, + ) + + kernel_type = choose_mp_linear_kernel(config) + assert kernel_type.__name__ == "TritonW4A16LinearKernel" + + +@pytest.mark.skipif(not current_platform.is_rocm(), reason="ROCm only") +def test_choose_mp_linear_kernel_picks_triton_w4a16_for_uint4_asymmetric(): + # Asymmetric int4 weights should also be supported (explicit zero points). + K, N = 512, 512 + config = MPLinearLayerConfig( + full_weight_shape=(K, N), + partition_weight_shape=(K, N), + weight_type=scalar_types.uint4, # asymmetric int4 (explicit zeros) + act_type=torch.bfloat16, + group_size=64, + zero_points=True, + has_g_idx=False, + ) + + kernel_type = choose_mp_linear_kernel(config) + assert kernel_type.__name__ == "TritonW4A16LinearKernel" diff --git a/asian_ai/performance/vLLM/tests/kernels/test_apply_repetition_penalties.py b/asian_ai/performance/vLLM/tests/kernels/test_apply_repetition_penalties.py new file mode 100644 index 0000000000000000000000000000000000000000..4e8a348b54b88e7016f560d1f87d231f1866ee96 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_apply_repetition_penalties.py @@ -0,0 +1,132 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch + +from tests.kernels.utils import opcheck +from vllm._custom_ops import ( + apply_repetition_penalties_cuda, + apply_repetition_penalties_torch, +) +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +NUM_SEQS = [1, 2, 3, 4, 8, 13, 17, 32, 37, 256, 1023, 1024, 1025] +# [stress, stress, stress, Qwen, llama 4] +VOCAB_SIZES = [17, 256, 1019, 151936, 202048] +REPETITION_PENALTY_VALUES = [1.05] +SEEDS = [0] +DTYPES = [torch.float32, torch.float16] + + +@pytest.mark.parametrize("num_seqs", NUM_SEQS) +@pytest.mark.parametrize("vocab_size", VOCAB_SIZES) +@pytest.mark.parametrize("repetition_penalty", REPETITION_PENALTY_VALUES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.skipif( + not current_platform.is_cuda(), reason="This test for checking CUDA kernel" +) +@torch.inference_mode() +def test_apply_repetition_penalties( + num_seqs: int, + vocab_size: int, + repetition_penalty: float, + dtype: torch.dtype, + seed: int, +) -> None: + """ + Test the apply_repetition_penalties custom op + against a reference implementation. + """ + set_random_seed(seed) + torch.set_default_device("cuda:0") + + # Create test data + logits = torch.randn(num_seqs, vocab_size, dtype=dtype) + + # Create masks with some random tokens marked as repeated + prompt_mask = torch.zeros(num_seqs, vocab_size, dtype=torch.bool) + output_mask = torch.zeros(num_seqs, vocab_size, dtype=torch.bool) + + # Mark some tokens as repeated in prompt and output + prompt_indices = torch.randint(0, vocab_size, (num_seqs, max(1, vocab_size // 200))) + output_indices = torch.randint(0, vocab_size, (num_seqs, max(1, vocab_size // 200))) + + for i in range(num_seqs): + prompt_mask[i, prompt_indices[i]] = True + output_mask[i, output_indices[i]] = True + + # Create repetition penalties tensor + repetition_penalties = torch.full((num_seqs,), repetition_penalty, dtype=dtype) + + # Run all three implementations + logits_torch = logits.clone() + logits_cuda = logits.clone() + + apply_repetition_penalties_torch( + logits_torch, prompt_mask, output_mask, repetition_penalties + ) + apply_repetition_penalties_cuda( + logits_cuda, prompt_mask, output_mask, repetition_penalties + ) + + # Compare all outputs to reference + torch.testing.assert_close(logits_torch, logits_cuda, rtol=1e-3, atol=1e-3) + + # Test the operator by applying the opcheck utility + opcheck( + torch.ops._C.apply_repetition_penalties_, + (logits.clone(), prompt_mask, output_mask, repetition_penalties), + ) + + +@pytest.mark.skipif( + not current_platform.is_cuda(), reason="This test for checking CUDA kernel" +) +@torch.inference_mode() +def test_apply_repetition_penalties_zero_seqs() -> None: + """ + Test the apply_repetition_penalties custom op with num_seqs=0 + against a reference implementation. + """ + num_seqs = 0 + vocab_size = 17 + repetition_penalty = 1.05 + dtype = torch.float32 + seed = 0 + + set_random_seed(seed) + torch.set_default_device("cuda:0") + + # Create test data + logits = torch.randn(num_seqs, vocab_size, dtype=dtype) + + # Create masks with some random tokens marked as repeated + prompt_mask = torch.zeros(num_seqs, vocab_size, dtype=torch.bool) + output_mask = torch.zeros(num_seqs, vocab_size, dtype=torch.bool) + + # No tokens to mark as repeated since num_seqs=0 + + # Create repetition penalties tensor + repetition_penalties = torch.full((num_seqs,), repetition_penalty, dtype=dtype) + + # Run all three implementations + logits_torch = logits.clone() + logits_cuda = logits.clone() + + apply_repetition_penalties_torch( + logits_torch, prompt_mask, output_mask, repetition_penalties + ) + apply_repetition_penalties_cuda( + logits_cuda, prompt_mask, output_mask, repetition_penalties + ) + + # Compare all outputs to reference + torch.testing.assert_close(logits_torch, logits_cuda, rtol=1e-3, atol=1e-3) + + # Test the operator by applying the opcheck utility + opcheck( + torch.ops._C.apply_repetition_penalties_, + (logits.clone(), prompt_mask, output_mask, repetition_penalties), + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_awq_int4_to_int8.py b/asian_ai/performance/vLLM/tests/kernels/test_awq_int4_to_int8.py new file mode 100644 index 0000000000000000000000000000000000000000..e400f3a87fd703e8f3544389d1701006a624b1cc --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_awq_int4_to_int8.py @@ -0,0 +1,288 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Unit tests for AWQ INT4 W4A8 GEMM pipeline (SGLang kernel migration). + +Part 1: Weight packing tests + - convert_weight_packed_scale_zp correctness + +Part 2: INT4 W4A8 GEMM tests + - int4_scaled_mm_cpu correctness w.r.t. float reference + - Bias, 3D input, various shapes + +Part 3: create_weights shapes + +cmd: + VLLM_CPU_INT4_W4A8=1 python -m pytest tests/kernels/test_awq_int4_to_int8.py -v -s +""" + +import numpy as np +import pytest +import torch + +from vllm._custom_ops import ( + CPUQuantAlgo, + convert_weight_packed_scale_zp, + int4_scaled_mm_cpu, +) +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + pack_cols, +) +from vllm.platforms import current_platform + +if not current_platform.is_cpu(): + pytest.skip("skipping CPU-only tests", allow_module_level=True) + + +def make_awq_checkpoint_data(K, N, group_size, seed=42): + """Create synthetic AWQ checkpoint data in packed int32 format. + + Returns: + packed_qweight: [K, N//8] int32 (AWQ interleaved + packed) + packed_qzeros: [num_groups, N//8] int32 (AWQ interleaved + packed) + scales: [num_groups, N] float32 + float_ref: [K, N] float32, reference dequantized weights + weight_int4_orig: [K, N] int32, original int4 values (0-15) + zeros_int4_orig: [num_groups, N] int32, original zero points (0-15) + """ + rng = np.random.RandomState(seed) + num_groups = K // group_size + + weight_int4_orig = torch.from_numpy( + rng.randint(0, 16, size=(K, N)).astype(np.int32) + ) + zeros_int4_orig = torch.from_numpy( + rng.randint(0, 16, size=(num_groups, N)).astype(np.int32) + ) + scales = torch.from_numpy((rng.randn(num_groups, N) * 0.05).astype(np.float32)) + + scales_exp = scales.repeat_interleave(group_size, dim=0) + zeros_exp = zeros_int4_orig.repeat_interleave(group_size, dim=0) + float_ref = (weight_int4_orig.float() - zeros_exp.float()) * scales_exp + + awq_interleave = [0, 2, 4, 6, 1, 3, 5, 7] + weight_interleaved = ( + weight_int4_orig.reshape(-1, 8)[:, awq_interleave].reshape(K, N).contiguous() + ) + packed_qweight = pack_cols(weight_interleaved, 4, K, N) + + zeros_interleaved = ( + zeros_int4_orig.reshape(-1, 8)[:, awq_interleave] + .reshape(num_groups, N) + .contiguous() + ) + packed_qzeros = pack_cols(zeros_interleaved, 4, num_groups, N) + + return ( + packed_qweight, + packed_qzeros, + scales, + float_ref, + weight_int4_orig, + zeros_int4_orig, + ) + + +class TestConvertWeightPackedScaleZp: + """Tests for convert_weight_packed_scale_zp weightpacking.""" + + @pytest.mark.parametrize( + "K,N,group_size", + [ + (128, 128, 128), + (256, 256, 128), + (512, 256, 64), + ], + ) + def test_packing_output_shapes(self, K, N, group_size): + """Packed outputs should have expected shapes.""" + (packed_qweight, packed_qzeros, scales, _, _, _) = make_awq_checkpoint_data( + K, N, group_size + ) + + blocked_w, blocked_zp, blocked_s = convert_weight_packed_scale_zp( + packed_qweight, + packed_qzeros, + scales, + CPUQuantAlgo.AWQ, + ) + + block_n = 32 + Nc = N // block_n + + assert blocked_w.dim() >= 2, ( + f"blocked_w should have >= 2 dims, got {blocked_w.dim()}" + ) + assert blocked_s.size(0) == Nc, ( + f"Expected Nc={Nc} scale blocks, got {blocked_s.size(0)}" + ) + assert blocked_zp.size(0) == Nc, ( + f"Expected Nc={Nc} qzeros blocks, got {blocked_zp.size(0)}" + ) + + print( + f" [PASS] packing shapes K={K}, N={N}, gs={group_size}: " + f"blocked_w={list(blocked_w.shape)}, " + f"blocked_s={list(blocked_s.shape)}, blocked_zp={list(blocked_zp.shape)}" + ) + + +class TestInt4ScaledMmCpu: + """Tests for int4_scaled_mm_cpu GEMM kernel.""" + + @pytest.mark.parametrize( + "M,K,N,group_size", + [ + (1, 128, 128, 128), + (4, 256, 256, 128), + (16, 512, 256, 64), + (32, 256, 512, 128), + (64, 512, 512, 128), + ], + ) + def test_gemm_vs_float_reference(self, M, K, N, group_size): + """INT4 W4A8 GEMM should approximate float matmul.""" + (packed_qweight, packed_qzeros, scales, float_ref, _, _) = ( + make_awq_checkpoint_data(K, N, group_size) + ) + + blocked_w, blocked_zp, blocked_s = convert_weight_packed_scale_zp( + packed_qweight, + packed_qzeros, + scales, + CPUQuantAlgo.AWQ, + ) + + x = torch.randn(M, K, dtype=torch.bfloat16) + out = int4_scaled_mm_cpu(x, blocked_w, blocked_zp, blocked_s, None) + + ref_out = torch.mm(x.float(), float_ref) + + abs_diff = (out.float() - ref_out).abs() + mean_abs = abs_diff.mean().item() + pct95 = torch.quantile(abs_diff, 0.95).item() + ref_mag = ref_out.abs().mean().item() + 1e-6 + mean_rel = mean_abs / ref_mag + + assert mean_rel < 0.05, ( + f"Mean relative error {mean_rel:.4f} exceeds 5% threshold" + ) + assert pct95 < ref_mag * 0.15, ( + f"95th-pctile abs_diff {pct95:.4f} exceeds 15% of ref magnitude" + ) + print(f" [PASS] INT4 GEMM correct: M={M}, K={K}, N={N}") + + @pytest.mark.parametrize("M", [1, 8, 32]) + def test_gemm_with_bias(self, M): + """INT4 W4A8 GEMM with bias should match reference.""" + K, N, group_size = 256, 128, 128 + (packed_qweight, packed_qzeros, scales, float_ref, _, _) = ( + make_awq_checkpoint_data(K, N, group_size) + ) + + blocked_w, blocked_zp, blocked_s = convert_weight_packed_scale_zp( + packed_qweight, + packed_qzeros, + scales, + CPUQuantAlgo.AWQ, + ) + + bias = torch.randn(N, dtype=torch.float32) + x = torch.randn(M, K, dtype=torch.bfloat16) + + out = int4_scaled_mm_cpu(x, blocked_w, blocked_zp, blocked_s, bias) + + ref_out = torch.mm(x.float(), float_ref) + bias + abs_diff = (out.float() - ref_out).abs() + mean_abs = abs_diff.mean().item() + ref_mag = ref_out.abs().mean().item() + 1e-6 + mean_rel = mean_abs / ref_mag + assert mean_rel < 0.05, ( + f"Mean relative error {mean_rel:.4f} with bias exceeds 5%" + ) + print(f" [PASS] INT4 GEMM with bias: M={M}") + + def test_gemm_3d_input(self): + """apply() reshapes 3D input [B, S, K] -> [B*S, K] -> back to 3D.""" + K, N, group_size = 256, 128, 128 + (packed_qweight, packed_qzeros, scales, float_ref, _, _) = ( + make_awq_checkpoint_data(K, N, group_size) + ) + + blocked_w, blocked_zp, blocked_s = convert_weight_packed_scale_zp( + packed_qweight, + packed_qzeros, + scales, + CPUQuantAlgo.AWQ, + ) + + B, S = 2, 8 + x_3d = torch.randn(B, S, K, dtype=torch.bfloat16) + x_2d = x_3d.reshape(-1, K) + + out_2d = int4_scaled_mm_cpu(x_2d, blocked_w, blocked_zp, blocked_s, None) + out_3d = out_2d.reshape(B, S, N) + + ref_out = torch.mm(x_2d.float(), float_ref).reshape(B, S, N) + + assert out_3d.shape == (B, S, N) + abs_diff = (out_3d.float() - ref_out).abs() + mean_abs = abs_diff.mean().item() + ref_mag = ref_out.abs().mean().item() + 1e-6 + mean_rel = mean_abs / ref_mag + + assert mean_rel < 0.05, f"Mean relative error {mean_rel:.4f} for 3D exceeds 5%" + print(f" [PASS] 3D input [{B},{S},{K}] -> output [{B},{S},{N}]") + + def test_gemm_fp16_input(self): + """INT4 GEMM should also work with fp16 input.""" + K, N, group_size, M = 256, 256, 128, 8 + (packed_qweight, packed_qzeros, scales, float_ref, _, _) = ( + make_awq_checkpoint_data(K, N, group_size) + ) + + blocked_w, blocked_zp, blocked_s = convert_weight_packed_scale_zp( + packed_qweight, + packed_qzeros, + scales, + CPUQuantAlgo.AWQ, + ) + + x = torch.randn(M, K, dtype=torch.float16) + out = int4_scaled_mm_cpu(x, blocked_w, blocked_zp, blocked_s, None) + + ref_out = torch.mm(x.float(), float_ref) + abs_diff = (out.float() - ref_out).abs() + ref_mag = ref_out.abs().mean().item() + 1e-6 + mean_rel = abs_diff.mean().item() / ref_mag + + assert mean_rel < 0.05, ( + f"Mean relative error {mean_rel:.4f} for fp16 exceeds 5%" + ) + print(f" [PASS] fp16 input M={M}, K={K}, N={N}") + + +class TestCreateWeightsUnchanged: + """Create_weights should still produce correct int4 placeholder shapes.""" + + @pytest.mark.parametrize( + "K,N,group_size", + [ + (128, 128, 128), + (256, 256, 128), + (512, 256, 64), + ], + ) + def test_int4_placeholder_shapes(self, K, N, group_size): + """Verify qweight, qzeros, scales shapes.""" + pack_factor = 8 + num_groups = K // group_size + + qweight = torch.empty(K, N // pack_factor, dtype=torch.int32) + qzeros = torch.empty(num_groups, N // pack_factor, dtype=torch.int32) + scales = torch.empty(num_groups, N, dtype=torch.bfloat16) + + assert qweight.shape == (K, N // pack_factor) + assert qzeros.shape == (num_groups, N // pack_factor) + assert scales.shape == (num_groups, N) + print(f" [PASS] create_weights shapes: K={K}, N={N}, gs={group_size}") diff --git a/asian_ai/performance/vLLM/tests/kernels/test_cache_kernels.py b/asian_ai/performance/vLLM/tests/kernels/test_cache_kernels.py new file mode 100644 index 0000000000000000000000000000000000000000..703159ad65d3facb903886657bfa71a5c3458859 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_cache_kernels.py @@ -0,0 +1,65 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for CUDA kernels in cache_kernels.cu.""" + +import pytest +import torch + +try: + from vllm import _custom_ops as ops +except ImportError: + pytest.skip( + "Could not import vllm._custom_ops. (pip install -e .)", allow_module_level=True + ) + + +@pytest.mark.skipif(torch.accelerator.device_count() < 1, reason="Need CUDA device") +def test_gather_cache_oob(): + """ + Tests for OOB read in gather_and_maybe_dequant_cache (Issue #27909). + This test constructs a boundary case identified in the issue where + seq_starts causes the block_table offset to read out of bounds. + """ + + batch_size = 1 + block_size = 64 + entry_size = 128 + + block_table = torch.tensor([[1, 2]], dtype=torch.int32, device="cuda") + + # This will result in offset = 128 / block_size = 128 / 64 = 2 + # This will cause the kernel to try to read from + # block_table[0, 2], but its size is only 2. + seq_starts = torch.tensor([128], dtype=torch.int32, device="cuda") + + seq_len = 65 + cu_seq_lens = torch.tensor([0, seq_len], dtype=torch.int32, device="cuda") + + # src_cache: [num_blocks, block_size, entry_size] + num_blocks = 5 + src_cache = torch.randn( + (num_blocks, block_size, entry_size), dtype=torch.float16, device="cuda" + ) + + dst = torch.empty((seq_len, entry_size), dtype=torch.float16, device="cuda") + + scale = torch.tensor([1.0], dtype=torch.float32, device="cuda") + + # Calling the C++ function gather_and_maybe_dequant_cache + ops.gather_and_maybe_dequant_cache( + src_cache, + dst, + block_table, + cu_seq_lens, + batch_size, + "auto", # kv_cache_dtype + scale, + seq_starts, + ) + + torch.accelerator.synchronize() + assert True + + +if __name__ == "__main__": + pytest.main([__file__]) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_compressor_kv_cache.py b/asian_ai/performance/vLLM/tests/kernels/test_compressor_kv_cache.py new file mode 100644 index 0000000000000000000000000000000000000000..4c41b4aab144250b7969a467255bb640950c5875 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_compressor_kv_cache.py @@ -0,0 +1,818 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Round-trip tests for compressor → FP8 quant + KV cache insert → gather + dequant. + +These tests cover: + A) DeepseekV4 Attention: head_dim=512 (448 FP8 nope + 64 bf16 rope), quant_block=64 + B) Fused dequant+gather K cache + C) Indexer: head_dim=128 (all FP8), quant_block=128 + D) DeepseekV4 Attention magnitude range: correctness across small/large values + E) Indexer fused Triton kernel: compress+norm+rope+quant+insert +""" + +import math + +import pytest +import torch + +from vllm import _custom_ops as ops +from vllm.models.deepseek_v4.common.ops import ( + dequantize_and_gather_k_cache, + quantize_and_insert_k_cache, +) +from vllm.models.deepseek_v4.common.ops.fused_compress_quant_cache import ( + _fused_kv_compress_norm_rope_insert_indexer_attn, + _fused_kv_compress_norm_rope_insert_indexer_mxfp4_attn, +) + +from .test_fused_indexer_q_rope_quant import quantize_to_mxfp4 + + +def _ue8m0_reference(x: torch.Tensor, block_size: int, fp8_max: float): + """PyTorch reference for UE8M0 FP8 quantization (per-block, power-of-2 scale). + + Returns (x_fp8, scales) where x_fp8 is float8_e4m3fn and scales are float32. + """ + assert x.dim() == 1 + n = x.numel() + n_blocks = math.ceil(n / block_size) + x_fp8 = torch.zeros(n, dtype=torch.float8_e4m3fn, device=x.device) + scales = torch.zeros(n_blocks, dtype=torch.float32, device=x.device) + + for i in range(n_blocks): + start = i * block_size + end = min(start + block_size, n) + block = x[start:end].float() + amax = block.abs().max().clamp(min=1e-4) + raw_scale = amax / fp8_max + exponent = math.ceil(math.log2(raw_scale.item())) + scale = 2.0**exponent + scales[i] = scale + quantized = (block / scale).clamp(-fp8_max, fp8_max) + x_fp8[start:end] = quantized.to(torch.float8_e4m3fn) + + return x_fp8, scales + + +# ── Test A: DeepseekV4 Attention path ────────────────────────────────────────────── + + +@pytest.mark.parametrize("num_tokens", [1, 4, 8, 17]) +@pytest.mark.parametrize("block_size", [16, 64]) +def test_deepseek_v4_attention_quant_cache_roundtrip(num_tokens: int, block_size: int): + """compressed_kv → quantize_and_insert_k_cache → dequantize_and_gather_k_cache + → compare against original.""" + + HEAD_DIM = 512 + NOPE_DIM = 448 + HEAD_BYTES = 584 # 448 fp8 + 128 bf16 + 8 uint8 scale + FP8_MAX = 448.0 + QUANT_BLOCK = 64 + + num_blocks = (num_tokens + block_size - 1) // block_size + 1 + device = "cuda" + + # Random compressed_kv (simulates compressor output) + compressed_kv = torch.randn( + num_tokens, HEAD_DIM, dtype=torch.bfloat16, device=device + ) + + # ── Quant + insert ────────────────────────────────────────────────── + k_cache = torch.zeros( + num_blocks, block_size, HEAD_BYTES, dtype=torch.uint8, device=device + ) + k_cache_2d = k_cache.view(num_blocks, -1) + + # Sequential slot mapping: token i → slot i + slot_mapping = torch.arange(num_tokens, dtype=torch.int64, device=device) + + quantize_and_insert_k_cache( + compressed_kv, k_cache_2d, slot_mapping, block_size=block_size + ) + + # ── Gather + dequant ──────────────────────────────────────────────── + num_reqs = 1 + max_blocks_per_seq = num_blocks + out = torch.zeros( + num_reqs, num_tokens, HEAD_DIM, dtype=torch.bfloat16, device=device + ) + seq_lens = torch.tensor([num_tokens], dtype=torch.int32, device=device) + # block_table: request 0 uses physical blocks 0, 1, ... + block_table = torch.arange( + max_blocks_per_seq, dtype=torch.int32, device=device + ).unsqueeze(0) + + dequantize_and_gather_k_cache( + out, k_cache, seq_lens, None, block_table, block_size, offset=0 + ) + + recovered = out[0, :num_tokens] + + # ── NoPE portion (first 448): FP8 quantized, expect UE8M0 error ── + nope_orig = compressed_kv[:, :NOPE_DIM].float() + nope_recv = recovered[:, :NOPE_DIM].float() + nope_diff = (nope_recv - nope_orig).abs() + + # Per-token check: FP8 e4m3 (3-bit mantissa) worst-case error is + # half-ULP at the largest representable value. At y ≈ 448 (max), + # ULP = 2^(8-3) = 32, so error ≤ 16 * scale. + for t in range(num_tokens): + _, scales = _ue8m0_reference( + compressed_kv[t, :NOPE_DIM].float(), QUANT_BLOCK, FP8_MAX + ) + max_allowed = 16.0 * scales.max().item() + token_diff = nope_diff[t].max().item() + assert token_diff <= max_allowed, ( + f"Token {t} nope diff {token_diff} exceeds max_allowed " + f"{max_allowed} (scale={scales.max().item()})" + ) + + # ── RoPE portion (last 64): stored as bf16, should be exact ───── + rope_diff = (recovered[:, NOPE_DIM:] - compressed_kv[:, NOPE_DIM:]).abs() + assert rope_diff.max().item() == 0.0, ( + f"RoPE portion should be exact but got max diff {rope_diff.max().item()}" + ) + + +# ── Test B: Fused dequant+gather K cache ──────────────────────────────────── + + +def _dequantize_and_gather_k_cache_reference( + out: torch.Tensor, + k_cache: torch.Tensor, + seq_lens: torch.Tensor, + gather_lens: torch.Tensor | None, + block_table: torch.Tensor, + block_size: int, + offset: int, +) -> None: + fp8_dim = 448 + bf16_dim = 64 + scale_dim = 8 + quant_block = 64 + token_data_size = fp8_dim + bf16_dim * 2 + + for req_id in range(seq_lens.shape[0]): + seq_len = seq_lens[req_id].item() + gather_len = gather_lens[req_id].item() if gather_lens is not None else seq_len + start_pos = seq_len - gather_len + + for i in range(gather_len): + pos = start_pos + i + pos_in_block = pos % block_size + block_idx = block_table[req_id, pos // block_size].item() + cache_block = k_cache[block_idx].view(-1) + + token_data_start = pos_in_block * token_data_size + fp8_bytes = cache_block[token_data_start : token_data_start + fp8_dim] + fp8_vals = fp8_bytes.view(torch.float8_e4m3fn).float() + + scale_start = block_size * token_data_size + pos_in_block * scale_dim + encoded_scales = cache_block[scale_start : scale_start + scale_dim] + scales = torch.exp2(encoded_scales[:7].float() - 127.0) + dequant = fp8_vals * scales.repeat_interleave(quant_block) + + bf16_start = token_data_start + fp8_dim + bf16_bytes = cache_block[bf16_start : bf16_start + bf16_dim * 2] + bf16_tail = bf16_bytes.view(torch.bfloat16) + + out[req_id, offset + i, :fp8_dim] = dequant + out[req_id, offset + i, fp8_dim:] = bf16_tail + + +@pytest.mark.parametrize( + ("seq_lens_host", "gather_lens_host", "offset"), + [ + ([9, 23, 7], None, 0), + ([19, 8, 257], [6, 8, 129], 5), + ], +) +def test_dequantize_and_gather_k_cache( + seq_lens_host: list[int], + gather_lens_host: list[int] | None, + offset: int, +): + block_size = 64 + head_dim = 512 + nope_dim = 448 + scale_dim = 8 + head_bytes = nope_dim + (head_dim - nope_dim) * 2 + scale_dim + device = "cuda" + num_reqs = len(seq_lens_host) + num_tokens = sum(seq_lens_host) + max_gather_len = max(gather_lens_host or seq_lens_host) + max_blocks_per_seq = math.ceil(max(seq_lens_host) / block_size) + num_blocks = sum(math.ceil(seq_len / block_size) for seq_len in seq_lens_host) + + compressed_kv = torch.randn( + num_tokens, head_dim, dtype=torch.bfloat16, device=device + ) + + # Randomize physical pages so the test covers block-table translation. + # Keep padded block-table entries invalid to catch accidental reads. + physical_blocks = torch.randperm(num_blocks, device=device) + block_table = torch.full( + (num_reqs, max_blocks_per_seq), int(-1e6), dtype=torch.int32, device=device + ) + start = 0 + for req_id, seq_len in enumerate(seq_lens_host): + num_req_blocks = math.ceil(seq_len / block_size) + req_blocks = physical_blocks[start : start + num_req_blocks] + block_table[req_id, :num_req_blocks] = req_blocks + start += num_req_blocks + + # Build slot_mapping for quantize_and_insert_k_cache. + slot_mapping = torch.empty(num_tokens, dtype=torch.int64, device=device) + start = 0 + for req_id, seq_len in enumerate(seq_lens_host): + logical_pos = torch.arange(seq_len, dtype=torch.int64, device=device) + block_idx = block_table[req_id, logical_pos // block_size].to(torch.int64) + token_slots = block_idx * block_size + logical_pos % block_size + slot_mapping[start : start + seq_len] = token_slots + start += seq_len + + # Insert compressed K into the paged cache layout used by the gather op. + k_cache = torch.empty( + num_blocks, block_size, head_bytes, dtype=torch.uint8, device=device + ) + k_cache_2d = k_cache.view(num_blocks, -1) + quantize_and_insert_k_cache(compressed_kv, k_cache_2d, slot_mapping, block_size) + + out_shape = (num_reqs, offset + max_gather_len + 3, head_dim) + ref_out = torch.empty(out_shape, dtype=torch.bfloat16, device=device) + actual_out = torch.empty_like(ref_out) + seq_lens = torch.tensor(seq_lens_host, dtype=torch.int32, device=device) + gather_lens = ( + torch.tensor(gather_lens_host, dtype=torch.int32, device=device) + if gather_lens_host is not None + else None + ) + + # Compare production gather against a PyTorch reference for valid output rows. + _dequantize_and_gather_k_cache_reference( + ref_out, k_cache, seq_lens, gather_lens, block_table, block_size, offset + ) + dequantize_and_gather_k_cache( + actual_out, k_cache, seq_lens, gather_lens, block_table, block_size, offset + ) + torch.accelerator.synchronize() + + # only check non-padded content + for req_id, seq_len in enumerate(seq_lens_host): + gather_len = ( + gather_lens_host[req_id] if gather_lens_host is not None else seq_len + ) + actual = actual_out[req_id, offset : offset + gather_len] + expected = ref_out[req_id, offset : offset + gather_len] + torch.testing.assert_close(actual, expected, rtol=0, atol=0) + + +# ── Test C: Indexer path ──────────────────────────────────────────────────── + + +@pytest.mark.parametrize("num_tokens", [1, 4, 8, 17]) +@pytest.mark.parametrize("block_size", [16, 64]) +def test_indexer_quant_cache_roundtrip(num_tokens: int, block_size: int): + """k → indexer_k_quant_and_cache → cp_gather_indexer_k_quant_cache + → manual dequant → compare against original.""" + + HEAD_DIM = 128 + QUANT_BLOCK_SIZE = 128 + # cache_stride = head_dim + (head_dim * 4 / quant_block_size) = 128 + 4 = 132 + CACHE_STRIDE = HEAD_DIM + HEAD_DIM * 4 // QUANT_BLOCK_SIZE + + num_blocks = (num_tokens + block_size - 1) // block_size + 1 + device = "cuda" + + # Random K (simulates compressor output for indexer) + k = torch.randn(num_tokens, HEAD_DIM, dtype=torch.bfloat16, device=device) + + # ── Quant + insert ────────────────────────────────────────────────── + kv_cache = torch.zeros( + num_blocks, block_size, CACHE_STRIDE, dtype=torch.uint8, device=device + ) + slot_mapping = torch.arange(num_tokens, dtype=torch.int64, device=device) + + ops.indexer_k_quant_and_cache(k, kv_cache, slot_mapping, QUANT_BLOCK_SIZE, "ue8m0") + + # ── Gather ────────────────────────────────────────────────────────── + max_blocks_per_seq = num_blocks + block_table = torch.arange( + max_blocks_per_seq, dtype=torch.int32, device=device + ).unsqueeze(0) + cu_seq_lens = torch.tensor([0, num_tokens], dtype=torch.int32, device=device) + + # dst_k: [total_seq_len, head_dim] as uint8 (raw FP8 bytes) + dst_k = torch.zeros(num_tokens, HEAD_DIM, dtype=torch.uint8, device=device) + # dst_scale: [total_seq_len, head_dim/quant_block*4] as uint8 (raw float32 bytes) + num_scale_bytes = HEAD_DIM * 4 // QUANT_BLOCK_SIZE # 4 + dst_scale = torch.zeros( + num_tokens, num_scale_bytes, dtype=torch.uint8, device=device + ) + + ops.cp_gather_indexer_k_quant_cache( + kv_cache, dst_k, dst_scale, block_table, cu_seq_lens + ) + + # ── Manual dequant ────────────────────────────────────────────────── + k_fp8 = dst_k.view(torch.float8_e4m3fn).float() # [num_tokens, 128] + scale = dst_scale.view(torch.float32) # [num_tokens, 1] + k_recovered = k_fp8 * scale # [num_tokens, 128] + + # ── Compare ───────────────────────────────────────────────────────── + diff = (k_recovered - k.float()).abs() + k_abs = k.float().abs() + + for t in range(num_tokens): + amax = k_abs[t].max().clamp(min=1e-4).item() + # UE8M0: scale = 2^ceil(log2(amax / 448)) + exponent = math.ceil(math.log2(amax / 448.0)) + ue8m0_scale = 2.0**exponent + # FP8 e4m3 (3-bit mantissa): worst-case error = 16 * scale + max_allowed = 16.0 * ue8m0_scale + token_diff = diff[t].max().item() + assert token_diff <= max_allowed, ( + f"Token {t} diff {token_diff} exceeds max_allowed " + f"{max_allowed} (scale={ue8m0_scale})" + ) + + +def test_indexer_gather_accepts_upper_bound_output(): + """Gather only exact cu_seq_lens even when dst is over-allocated.""" + + head_dim = 128 + quant_block_size = 128 + cache_stride = head_dim + head_dim * 4 // quant_block_size + valid_tokens = 9 + upper_bound_tokens = 13 + block_size = 16 + num_blocks = 2 + sentinel = 123 + device = "cuda" + + k = torch.randn(valid_tokens, head_dim, dtype=torch.bfloat16, device=device) + kv_cache = torch.zeros( + num_blocks, block_size, cache_stride, dtype=torch.uint8, device=device + ) + slot_mapping = torch.arange(valid_tokens, dtype=torch.int64, device=device) + ops.indexer_k_quant_and_cache(k, kv_cache, slot_mapping, quant_block_size, "ue8m0") + + block_table = torch.arange(num_blocks, dtype=torch.int32, device=device).unsqueeze( + 0 + ) + cu_seq_lens = torch.tensor([0, valid_tokens], dtype=torch.int32, device=device) + dst_k = torch.full( + (upper_bound_tokens, head_dim), sentinel, dtype=torch.uint8, device=device + ) + num_scale_bytes = head_dim * 4 // quant_block_size + dst_scale = torch.full( + (upper_bound_tokens, num_scale_bytes), + sentinel, + dtype=torch.uint8, + device=device, + ) + + ops.cp_gather_indexer_k_quant_cache( + kv_cache, dst_k, dst_scale, block_table, cu_seq_lens + ) + torch.accelerator.synchronize() + + k_recovered = dst_k[:valid_tokens].view(torch.float8_e4m3fn).float() * dst_scale[ + :valid_tokens + ].view(torch.float32) + diff = (k_recovered - k.float()).abs() + max_allowed = (16.0 * dst_scale[:valid_tokens].view(torch.float32).max()).item() + assert diff.max().item() <= max_allowed + assert torch.all(dst_k[valid_tokens:] == sentinel) + assert torch.all(dst_scale[valid_tokens:] == sentinel) + + +# ── Test D: DeepseekV4 attention with values at different magnitudes ─────────── + + +def test_deepseek_v4_quant_magnitude_range(): + """Test that quantization handles a range of magnitudes correctly.""" + + HEAD_DIM = 512 + NOPE_DIM = 448 + HEAD_BYTES = 584 + block_size = 16 + num_tokens = 4 + num_blocks = 2 + device = "cuda" + + # Create inputs with varying magnitudes: small, medium, large + compressed_kv = torch.zeros( + num_tokens, HEAD_DIM, dtype=torch.bfloat16, device=device + ) + compressed_kv[0] = 0.001 # very small + compressed_kv[1] = 1.0 # unit scale + compressed_kv[2] = 100.0 # large + compressed_kv[3] = torch.randn(HEAD_DIM, dtype=torch.bfloat16, device=device) + + k_cache = torch.zeros( + num_blocks, block_size, HEAD_BYTES, dtype=torch.uint8, device=device + ) + slot_mapping = torch.arange(num_tokens, dtype=torch.int64, device=device) + + quantize_and_insert_k_cache( + compressed_kv, k_cache.view(num_blocks, -1), slot_mapping, block_size + ) + + out = torch.zeros(1, num_tokens, HEAD_DIM, dtype=torch.bfloat16, device=device) + seq_lens = torch.tensor([num_tokens], dtype=torch.int32, device=device) + block_table = torch.arange(num_blocks, dtype=torch.int32, device=device).unsqueeze( + 0 + ) + + dequantize_and_gather_k_cache( + out, k_cache, seq_lens, None, block_table, block_size, offset=0 + ) + + recovered = out[0, :num_tokens] + + # RoPE portion must be exact + rope_diff = (recovered[:, NOPE_DIM:] - compressed_kv[:, NOPE_DIM:]).abs().max() + assert rope_diff.item() == 0.0, f"RoPE diff {rope_diff.item()}" + + # NoPE: relative error should be reasonable + for t in range(num_tokens): + orig = compressed_kv[t, :NOPE_DIM].float() + recv = recovered[t, :NOPE_DIM].float() + abs_diff = (recv - orig).abs().max().item() + magnitude = orig.abs().max().item() + if magnitude > 0.01: + rel_err = abs_diff / magnitude + assert rel_err < 0.15, ( + f"Token {t}: rel_err={rel_err:.4f}, abs_diff={abs_diff:.6f}, " + f"magnitude={magnitude:.4f}" + ) + + +# ── Test E: Indexer fused K-cache insert (Triton kernels) ──────────────────── +# +# Both kernels share the same Triton signature; use_fp4 selects between them. +# Full pipeline: state-cache gather → softmax-weighted compress → RMSNorm → +# GPT-J RoPE → quant (MXFP4 or FP8) → paged cache insert. + + +def _reference_kv_compress_norm_rope( + state_cache: torch.Tensor, + block_table: torch.Tensor, + positions: torch.Tensor, + rms_weight: torch.Tensor, + cos_sin_cache: torch.Tensor, + compress_ratio: int = 1, + overlap: int = 0, + use_fp4: bool = False, + rms_eps: float = 1e-6, + fp8_max: float = 448.0, + return_full_cache: bool = False, +): + """Compress → RMSNorm → GPT-J RoPE → quantize. + + Gathers (1+overlap)*compress_ratio state entries per output token, applies + per-element softmax over the scores, and computes the weighted kv sum. + Returns (quantized_values, scale) matching the kernel's output layout. + """ + device = state_cache.device + head_dim = rms_weight.shape[0] + rope_dim = cos_sin_cache.shape[-1] + state_block_size = state_cache.shape[1] + state_width = state_cache.shape[-1] // 2 + nope_dim = head_dim - rope_dim + total = (1 + overlap) * compress_ratio + results = [] + for pos in positions.tolist(): + src = torch.arange(pos - total + 1, pos + 1, dtype=torch.int64, device=device) + valid = src >= 0 + idx = src.clamp(min=0) + pages = block_table[0, idx // state_block_size] + offsets = idx % state_block_size + raw = state_cache[pages, offsets].float() # [total, state_dim] + + # Group 0 (tokens 0..cr-1): kv[:H], score[SW:SW+H] + # Group 1 (tokens cr..2cr-1): kv[H:2H], score[SW+H:SW+2H] + if overlap: + sw = state_width + g0_kv = raw[:compress_ratio, :head_dim] + g1_kv = raw[compress_ratio:, head_dim : 2 * head_dim] + g0_scores = raw[:compress_ratio, sw : sw + head_dim] + g1_scores = raw[compress_ratio:, sw + head_dim : sw + 2 * head_dim] + kv = torch.cat([g0_kv, g1_kv]) + scores = torch.cat([g0_scores, g1_scores]) + else: + kv = raw[:, :head_dim] + scores = raw[:, state_width : state_width + head_dim] + + scores[~valid] = float("-inf") + kv[~valid] = 0.0 + weights = torch.softmax(scores, dim=0) + compressed = (kv * weights).sum(dim=0) # [H] + var = (compressed * compressed).mean() + normed = compressed * torch.rsqrt(var + rms_eps) * rms_weight.float() + compressed_pos = (pos // compress_ratio) * compress_ratio + cos, sin = cos_sin_cache[compressed_pos].float().chunk(2) + nope, rope = normed.split([nope_dim, rope_dim]) + rope = torch.stack( + [rope[0::2] * cos - rope[1::2] * sin, rope[1::2] * cos + rope[0::2] * sin], + dim=-1, + ).reshape(rope_dim) + results.append(torch.cat([nope, rope]).to(state_cache.dtype)) + result = torch.stack(results) + + if return_full_cache: + # Contiguous 512-wide bf16 row (nope unrotated + rope rotated), matching + # the FlashInfer full-cache layout before any per-tensor fp8 quant. The + # kernel rounds the fp32 result to bf16 once at the store. + return result.to(torch.bfloat16) + + if use_fp4: + return quantize_to_mxfp4(result) + else: + pairs = [ + _ue8m0_reference(result[t], head_dim, fp8_max) for t in range(len(result)) + ] + quants, scales = zip(*pairs) + return torch.stack(quants), torch.cat(scales) + + +@pytest.mark.parametrize("num_tokens", [1, 7, 32]) +@pytest.mark.parametrize("kv_block_size", [16, 32]) +@pytest.mark.parametrize("use_fp4", [False, True]) +def test_fused_kv_insert_indexer(num_tokens: int, kv_block_size: int, use_fp4: bool): + """Fused K compress+norm+rope+quant+insert for the indexer KV cache.""" + HEAD_DIM = 128 + ROPE_DIM = 64 + BLOCK_SIZE = 16 + RMS_EPS = 1e-6 + FP8_MAX = 448.0 + + device = "cuda" + torch.manual_seed(42) + compress_ratio = 4 + + if use_fp4: + TOKEN_STRIDE = HEAD_DIM // 2 # packed nibbles: 64 bytes + SCALE_DIM = HEAD_DIM // 32 # ue8m0 bytes: 4 + QUANT_BLOCK = 32 + kernel = _fused_kv_compress_norm_rope_insert_indexer_mxfp4_attn + else: + TOKEN_STRIDE = HEAD_DIM # FP8 bytes: 128 + SCALE_DIM = 4 # 1 float32: 4 bytes + QUANT_BLOCK = HEAD_DIM + kernel = _fused_kv_compress_norm_rope_insert_indexer_attn + + # overlap=1 whenever compress_ratio==4, matching DeepseekCompressor logic. + overlap = 1 if compress_ratio == 4 else 0 + coff = 1 + overlap # multiplier for state_dim per entry + + num_pages = (compress_ratio * num_tokens - 1) // BLOCK_SIZE + 2 + state_cache = torch.randn( + num_pages, + BLOCK_SIZE, + 2 * coff * HEAD_DIM, # kv_state + score_state, each coff*HEAD_DIM wide + dtype=torch.bfloat16, + device=device, + ) + block_table = torch.arange(num_pages, dtype=torch.int32, device=device).unsqueeze(0) + token_to_req = torch.zeros(num_tokens, dtype=torch.int32, device=device) + slot_mapping = torch.arange(num_tokens, dtype=torch.int64, device=device) + positions = torch.arange( + compress_ratio - 1, + compress_ratio * num_tokens, + compress_ratio, + dtype=torch.int64, + device=device, + ) + rms_weight = torch.randn(HEAD_DIM, dtype=torch.bfloat16, device=device) + cos_sin_cache = torch.randn(compress_ratio * num_tokens, ROPE_DIM, device=device) + + kv_n_blocks = (num_tokens + kv_block_size - 1) // kv_block_size + 1 + kv_cache = torch.zeros( + kv_n_blocks, + kv_block_size * (TOKEN_STRIDE + SCALE_DIM), + dtype=torch.uint8, + device=device, + ) + + kernel[(num_tokens,)]( + state_cache, + state_cache.stride(0), + state_cache.stride(1), + token_to_req, + positions, + slot_mapping, + block_table, + block_table.stride(0), + BLOCK_SIZE, + rms_weight, + RMS_EPS, + cos_sin_cache, + cos_sin_cache.stride(0), + kv_cache, + slot_mapping, + kv_block_size, + HEAD_SIZE=HEAD_DIM, + TRITON_BLOCK_SIZE=HEAD_DIM, + STATE_WIDTH=coff * HEAD_DIM, + COMPRESS_RATIO=compress_ratio, + OVERLAP=overlap, + ROPE_HEAD_DIM=ROPE_DIM, + FP8_MAX=FP8_MAX, + QUANT_BLOCK=QUANT_BLOCK, + TOKEN_STRIDE=TOKEN_STRIDE, + SCALE_DIM=SCALE_DIM, + KV_BLOCK_STRIDE=kv_cache.stride(0), + num_warps=1, + ) + + k_quant, scale = _reference_kv_compress_norm_rope( + state_cache, + block_table, + positions, + rms_weight, + cos_sin_cache, + compress_ratio, + overlap, + use_fp4, + rms_eps=RMS_EPS, + fp8_max=FP8_MAX, + ) + + if use_fp4: + for i in range(num_tokens): + blk, pos = i // kv_block_size, i % kv_block_size + val_off = pos * TOKEN_STRIDE + fp4_actual = kv_cache[blk, val_off : val_off + TOKEN_STRIDE] + assert torch.equal(k_quant[i], fp4_actual), ( + f"token {i}: packed nibbles differ, " + f"{(k_quant[i] != fp4_actual).sum()} " + f"/ {TOKEN_STRIDE}" + ) + + scale_off = kv_block_size * TOKEN_STRIDE + pos * SCALE_DIM + scale_actual = kv_cache[blk, scale_off : scale_off + SCALE_DIM] + assert torch.equal(scale_actual, scale[i]), ( + f"token {i}: ue8m0 {scale_actual.tolist()} != {scale[i].tolist()}" + ) + + else: + k_quant = k_quant.view(torch.uint8) + for i in range(num_tokens): + blk, pos = i // kv_block_size, i % kv_block_size + val_off = pos * TOKEN_STRIDE + assert torch.equal( + k_quant[i], kv_cache[blk, val_off : val_off + TOKEN_STRIDE] + ), f"token {i}: FP8 bytes differ" + + scale_off = kv_block_size * TOKEN_STRIDE + pos * SCALE_DIM + actual_scale = kv_cache[blk, scale_off : scale_off + SCALE_DIM].view( + torch.float32 + ) + assert torch.equal(actual_scale, scale[i : i + 1]), ( + f"token {i}: scale {actual_scale.item()} != {scale[i].item()}" + ) + + +@pytest.mark.parametrize("compress_ratio", [4, 128]) +@pytest.mark.parametrize("store_fp8", [False, True]) +def test_cutedsl_full_cache_store(compress_ratio: int, store_fp8: bool): + """CuTeDSL compressor full-cache (FlashInfer) store parity for head=512. + + Exercises the contiguous bf16 / per-tensor fp8 store branch of both the C4 + fused kernel and the C128 split kernel against the PyTorch reference. + """ + cutedsl = pytest.importorskip("cutlass") # noqa: F841 + from vllm.models.deepseek_v4.nvidia.ops.sparse_attn_compress_cutedsl import ( + fused_kv_compress_norm_rope_insert_sparse_attn_cutedsl, + split_kv_compress_norm_rope_insert_sparse_attn_cutedsl, + ) + + HEAD_DIM = 512 + ROPE_DIM = 64 + RMS_EPS = 1e-6 + FP8_MAX = 448.0 + # C128 compress (Block8 kernel) requires state-cache block_size=8; C4 uses 16. + BLOCK_SIZE = 8 if compress_ratio == 128 else 16 + KV_BLOCK_SIZE = 64 + device = "cuda" + torch.manual_seed(7) + + overlap = 1 if compress_ratio == 4 else 0 + coff = 1 + overlap + num_tokens = 8 + + num_pages = (compress_ratio * num_tokens - 1) // BLOCK_SIZE + 2 + # The production CompressorStateCache is fp32. + state_cache = torch.randn( + num_pages, BLOCK_SIZE, 2 * coff * HEAD_DIM, dtype=torch.float32, device=device + ) + block_table = torch.arange(num_pages, dtype=torch.int32, device=device).unsqueeze(0) + token_to_req = torch.zeros(num_tokens, dtype=torch.int32, device=device) + slot_mapping = torch.arange(num_tokens, dtype=torch.int64, device=device) + positions = torch.arange( + compress_ratio - 1, + compress_ratio * num_tokens, + compress_ratio, + dtype=torch.int64, + device=device, + ) + rms_weight = torch.randn(HEAD_DIM, dtype=torch.bfloat16, device=device) + cos_sin_cache = torch.randn( + compress_ratio * num_tokens, ROPE_DIM, dtype=torch.float32, device=device + ) + + dtype = torch.float8_e4m3fn if store_fp8 else torch.bfloat16 + kv_n_blocks = (num_tokens + KV_BLOCK_SIZE - 1) // KV_BLOCK_SIZE + 1 + k_cache = torch.zeros( + kv_n_blocks, KV_BLOCK_SIZE, HEAD_DIM, dtype=dtype, device=device + ) + fp8_scale = torch.tensor( + [0.5 if store_fp8 else 1.0], dtype=torch.float32, device=device + ) + + if compress_ratio == 4: + fused_kv_compress_norm_rope_insert_sparse_attn_cutedsl( + state_cache, + token_to_req, + positions, + slot_mapping, + block_table, + BLOCK_SIZE, + rms_weight, + RMS_EPS, + cos_sin_cache, + k_cache, + slot_mapping, + KV_BLOCK_SIZE, + k_cache.stride(0), + head_size=HEAD_DIM, + state_width=coff * HEAD_DIM, + rope_head_dim=ROPE_DIM, + fp8_max=FP8_MAX, + quant_block=64, + token_stride=576, + scale_dim=8, + compress_ratio=compress_ratio, + overlap=True, + store_full_kv=True, + store_full_fp8=store_fp8, + fp8_scale=fp8_scale, + ) + else: + compressed_kv = torch.empty( + (num_tokens, HEAD_DIM), dtype=torch.float32, device=device + ) + split_kv_compress_norm_rope_insert_sparse_attn_cutedsl( + state_cache, + token_to_req, + positions, + slot_mapping, + block_table, + BLOCK_SIZE, + compressed_kv, + rms_weight, + RMS_EPS, + cos_sin_cache, + k_cache, + slot_mapping, + KV_BLOCK_SIZE, + k_cache.stride(0), + head_size=HEAD_DIM, + state_width=coff * HEAD_DIM, + rope_head_dim=ROPE_DIM, + fp8_max=FP8_MAX, + quant_block=64, + token_stride=576, + scale_dim=8, + compress_ratio=compress_ratio, + overlap=bool(overlap), + store_full_kv=True, + store_full_fp8=store_fp8, + fp8_scale=fp8_scale, + ) + + ref = _reference_kv_compress_norm_rope( + state_cache, + block_table, + positions, + rms_weight, + cos_sin_cache, + compress_ratio, + overlap, + rms_eps=RMS_EPS, + return_full_cache=True, + ) # [num_tokens, HEAD_DIM] bf16 + + actual = torch.stack( + [k_cache[i // KV_BLOCK_SIZE, i % KV_BLOCK_SIZE] for i in range(num_tokens)] + ) + if store_fp8: + ref_fp8 = torch.clamp(ref.float() / fp8_scale, -FP8_MAX, FP8_MAX).to( + torch.float8_e4m3fn + ) + torch.testing.assert_close(actual.float(), ref_fp8.float(), rtol=0.0, atol=0.3) + else: + torch.testing.assert_close(actual.float(), ref.float(), rtol=3e-2, atol=3e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_concat_mla_q.py b/asian_ai/performance/vLLM/tests/kernels/test_concat_mla_q.py new file mode 100644 index 0000000000000000000000000000000000000000..4a281a97d12244f0827dd4b291dbbd4127d7b589 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_concat_mla_q.py @@ -0,0 +1,139 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm import _custom_ops as ops + +NUM_TOKENS = [1, 4, 16, 64, 128] +NUM_HEADS = [128] +NOPE_DIM = [512] +ROPE_DIM = [64] +DTYPES = [torch.bfloat16, torch.float16] + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("nope_dim", NOPE_DIM) +@pytest.mark.parametrize("rope_dim", ROPE_DIM) +@pytest.mark.parametrize("dtype", DTYPES) +def test_concat_mla_q_contiguous(num_tokens, num_heads, nope_dim, rope_dim, dtype): + """Test with contiguous inputs (standard layout).""" + torch.manual_seed(42) + ql_nope = torch.randn(num_tokens, num_heads, nope_dim, dtype=dtype, device="cuda") + q_pe = torch.randn(num_tokens, num_heads, rope_dim, dtype=dtype, device="cuda") + + ref = torch.cat((ql_nope, q_pe), dim=-1) + + q_out = torch.empty( + num_tokens, num_heads, nope_dim + rope_dim, dtype=dtype, device="cuda" + ) + ops.concat_mla_q(ql_nope, q_pe, q_out) + + torch.testing.assert_close(q_out, ref, atol=0, rtol=0) + + +@pytest.mark.parametrize("num_tokens", [t for t in NUM_TOKENS if t > 1]) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("nope_dim", NOPE_DIM) +@pytest.mark.parametrize("rope_dim", ROPE_DIM) +@pytest.mark.parametrize("dtype", DTYPES) +def test_concat_mla_q_transposed_nope(num_tokens, num_heads, nope_dim, rope_dim, dtype): + """Test with transposed nope input (simulates BMM output after transpose). + + In the real code path, mqa_ql_nope is the result of: + torch.bmm(q_nope, W_UK_T) # [N, B, L] + .transpose(0, 1) # [B, N, L] — non-contiguous! + """ + torch.manual_seed(42) + nope_raw = torch.randn(num_heads, num_tokens, nope_dim, dtype=dtype, device="cuda") + ql_nope = nope_raw.transpose(0, 1) # [B, N, L], non-contiguous + assert not ql_nope.is_contiguous() + + q_pe = torch.randn(num_tokens, num_heads, rope_dim, dtype=dtype, device="cuda") + + ref = torch.cat((ql_nope, q_pe), dim=-1) + + q_out = torch.empty( + num_tokens, num_heads, nope_dim + rope_dim, dtype=dtype, device="cuda" + ) + ops.concat_mla_q(ql_nope, q_pe, q_out) + + torch.testing.assert_close(q_out, ref, atol=0, rtol=0) + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("num_heads", NUM_HEADS) +@pytest.mark.parametrize("dtype", DTYPES) +def test_concat_mla_q_split_rope(num_tokens, num_heads, dtype): + """Test with rope from a split (simulates the actual code path). + + In the real code path, q_pe comes from: + mqa_q.split([qk_nope_head_dim, qk_rope_head_dim], dim=-1) + which creates a non-contiguous view with stride(1) != rope_dim. + """ + torch.manual_seed(42) + nope_dim = 512 + rope_dim = 64 + orig_dim = 128 + 64 # original q before absorption: [B, N, 192] + + # Simulate split from original q tensor + q_orig = torch.randn(num_tokens, num_heads, orig_dim, dtype=dtype, device="cuda") + q_nope_orig, q_pe = q_orig.split([128, 64], dim=-1) + + # q_pe is non-contiguous: stride(1) = 192, not 64 + assert q_pe.stride(1) == orig_dim + assert q_pe.stride(2) == 1 # but innermost is fine + + # Simulate absorbed nope (contiguous, different size) + ql_nope = torch.randn(num_tokens, num_heads, nope_dim, dtype=dtype, device="cuda") + + ref = torch.cat((ql_nope, q_pe), dim=-1) + + q_out = torch.empty( + num_tokens, num_heads, nope_dim + rope_dim, dtype=dtype, device="cuda" + ) + ops.concat_mla_q(ql_nope, q_pe, q_out) + + torch.testing.assert_close(q_out, ref, atol=0, rtol=0) + + +def test_concat_mla_q_zero_tokens(): + """Test with zero tokens (edge case).""" + ql_nope = torch.empty(0, 128, 512, dtype=torch.bfloat16, device="cuda") + q_pe = torch.empty(0, 128, 64, dtype=torch.bfloat16, device="cuda") + q_out = torch.empty(0, 128, 576, dtype=torch.bfloat16, device="cuda") + + ops.concat_mla_q(ql_nope, q_pe, q_out) + + +@pytest.mark.parametrize("num_tokens", [1, 64]) +def test_concat_mla_q_values_preserved(num_tokens): + """Verify exact bit-level preservation (no computation, pure copy). + + Compares raw int16 bits to avoid NaN != NaN issues from IEEE 754. + """ + nope_dim, rope_dim = 512, 64 + + # Use specific bit patterns (stay in int16 for bit-exact comparison) + ql_nope_bits = torch.arange( + num_tokens * 128 * nope_dim, dtype=torch.int16, device="cuda" + ).view(num_tokens, 128, nope_dim) + q_pe_bits = torch.arange( + num_tokens * 128 * rope_dim, dtype=torch.int16, device="cuda" + ).view(num_tokens, 128, rope_dim) + + ql_nope = ql_nope_bits.view(torch.bfloat16) + q_pe = q_pe_bits.view(torch.bfloat16) + + q_out = torch.empty( + num_tokens, 128, nope_dim + rope_dim, dtype=torch.bfloat16, device="cuda" + ) + ops.concat_mla_q(ql_nope, q_pe, q_out) + + out_bits = q_out.view(torch.int16) + + assert torch.equal(out_bits[..., :nope_dim], ql_nope_bits) + + assert torch.equal(out_bits[..., nope_dim:], q_pe_bits) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_cp_gather_fp8.py b/asian_ai/performance/vLLM/tests/kernels/test_cp_gather_fp8.py new file mode 100644 index 0000000000000000000000000000000000000000..a1c8236f0b7186cc3ac1e996c3748e2df646f96b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_cp_gather_fp8.py @@ -0,0 +1,363 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import math + +import pytest +import torch + +from vllm import _custom_ops as ops + +# DeepSeek V3 MLA dimensions +NOPE_DIM = 512 # NoPE latent dimension (FP8 quantized in cache) +ROPE_DIM = 64 # RoPE dimension (stored as BF16 in cache) +NUM_TILES = 4 # NOPE_DIM / GROUP_SIZE = 512 / 128 +GROUP_SIZE = 128 # FP8 quantization group size (one scale per group) +ENTRY_BYTES = 656 # 512 (FP8) + 16 (4×float32 scales) + 128 (64×BF16 RoPE) + + +def _build_test_case(seq_lens, block_size, seed=42): + """Build a synthetic FP8 cache and compute the expected BF16 output. + + This simulates what concat_and_cache_ds_mla_kernel writes into the + KV cache, then computes what cp_gather_and_upconvert should produce. + + Args: + seq_lens: List of sequence lengths, one per request. + block_size: Number of tokens per physical cache block. + seed: Random seed for reproducibility. + + Returns: + Tuple of (cache, block_table, seq_lens_t, workspace_starts_t, + num_reqs, total_tokens, expected_output). + """ + torch.manual_seed(seed) + + num_reqs = len(seq_lens) + total_tokens = sum(seq_lens) + + # workspace_starts[r] = sum of seq_lens[0..r-1] + # This tells the kernel where in the output buffer each request's + # gathered tokens should be written. + workspace_starts = [] + s = 0 + for sl in seq_lens: + workspace_starts.append(s) + s += sl + + # How many physical cache blocks each request needs + blocks_per_req = [math.ceil(s / block_size) for s in seq_lens] + total_blocks = sum(blocks_per_req) + max_blocks = max(blocks_per_req) + + # Block table maps (request, logical_block_idx) -> physical_block_id. + # Here we assign blocks contiguously: request 0 gets blocks [0, 1, ...], + # request 1 gets the next set, etc. + block_table = torch.zeros(num_reqs, max_blocks, dtype=torch.int32, device="cuda") + block_idx = 0 + for r in range(num_reqs): + for b in range(blocks_per_req[r]): + block_table[r, b] = block_idx + block_idx += 1 + + # The raw paged cache: [num_blocks, block_size, 656] as uint8 + cache = torch.zeros( + total_blocks, block_size, ENTRY_BYTES, dtype=torch.uint8, device="cuda" + ) + # Expected kernel output: [total_tokens, 576] as BF16 + expected = torch.zeros( + total_tokens, NOPE_DIM + ROPE_DIM, dtype=torch.bfloat16, device="cuda" + ) + + # Fill each token's cache entry and compute expected output + for r in range(num_reqs): + for t in range(seq_lens[r]): + out_idx = workspace_starts[r] + t + # Map token position -> (physical_block, offset_within_block) + phys = block_table[r, t // block_size].item() + off = t % block_size + + # --- NoPE section: 4 tiles of 128 FP8 values, each with a scale --- + for tile in range(NUM_TILES): + start = tile * GROUP_SIZE + + # Generate random data and quantize to FP8 e4m3 + fp8_vals = torch.randn(GROUP_SIZE, device="cuda").to( + torch.float8_e4m3fn + ) + # Pack FP8 bytes into cache at bytes [start : start+128] + cache[phys, off, start : start + GROUP_SIZE] = fp8_vals.view( + torch.uint8 + ) + + # Random positive scale in [0.1, 2.1] + scale = (torch.rand(1, device="cuda") * 2.0 + 0.1).item() + scale_t = torch.tensor([scale], dtype=torch.float32, device="cuda") + # Pack scale as 4 raw bytes at bytes [512 + tile*4 : ...] + cache[phys, off, NOPE_DIM + tile * 4 : NOPE_DIM + (tile + 1) * 4] = ( + scale_t.view(torch.uint8) + ) + + # Reference dequant: fp8 -> float32, multiply scale, -> bf16. + # This matches the CUDA path: fp8 -> half -> float * scale -> bf16. + # (fp8 -> half is exact, half -> float is exact, so fp8 -> float + # gives the same result regardless of intermediate type.) + expected[out_idx, start : start + GROUP_SIZE] = ( + fp8_vals.float() * scale + ).bfloat16() + + # --- RoPE section: 64 BF16 values, direct copy (no dequant) --- + rope = torch.randn(ROPE_DIM, dtype=torch.bfloat16, device="cuda") + # Pack RoPE bytes into cache at bytes [528 : 656] + cache[phys, off, NOPE_DIM + 16 :] = rope.view(torch.uint8) + # Expected output: exact copy + expected[out_idx, NOPE_DIM:] = rope + + seq_lens_t = torch.tensor(seq_lens, dtype=torch.int32, device="cuda") + workspace_starts_t = torch.tensor( + workspace_starts, dtype=torch.int32, device="cuda" + ) + + return ( + cache, + block_table, + seq_lens_t, + workspace_starts_t, + num_reqs, + total_tokens, + expected, + ) + + +def _build_test_case_fast(seq_lens, block_size, seed=42): + """Vectorized test-case builder for large sequence lengths. + + Same logic as _build_test_case but uses tensor operations instead of + per-token Python loops, making it practical for seq_lens up to 128K+. + """ + torch.manual_seed(seed) + + num_reqs = len(seq_lens) + total_tokens = sum(seq_lens) + + workspace_starts = [] + s = 0 + for sl in seq_lens: + workspace_starts.append(s) + s += sl + + blocks_per_req = [math.ceil(sl / block_size) for sl in seq_lens] + total_blocks = sum(blocks_per_req) + max_blocks = max(blocks_per_req) + + # Contiguous block allocation + block_table = torch.zeros(num_reqs, max_blocks, dtype=torch.int32, device="cuda") + block_idx = 0 + for r in range(num_reqs): + for b in range(blocks_per_req[r]): + block_table[r, b] = block_idx + block_idx += 1 + + cache = torch.zeros( + total_blocks, block_size, ENTRY_BYTES, dtype=torch.uint8, device="cuda" + ) + + # Generate all data vectorized + nope_fp8 = torch.randn(total_tokens, NOPE_DIM, device="cuda").to( + torch.float8_e4m3fn + ) + scales = (torch.rand(total_tokens, NUM_TILES, device="cuda") * 2.0 + 0.1).float() + rope = torch.randn(total_tokens, ROPE_DIM, dtype=torch.bfloat16, device="cuda") + + # Compute expected output vectorized (same dequant logic as kernel) + expected = torch.zeros( + total_tokens, NOPE_DIM + ROPE_DIM, dtype=torch.bfloat16, device="cuda" + ) + for tile in range(NUM_TILES): + start = tile * GROUP_SIZE + expected[:, start : start + GROUP_SIZE] = ( + nope_fp8[:, start : start + GROUP_SIZE].float() * scales[:, tile : tile + 1] + ).bfloat16() + expected[:, NOPE_DIM:] = rope + + # Build per-token cache entries as [total_tokens, 656] uint8 + token_data = torch.zeros( + total_tokens, ENTRY_BYTES, dtype=torch.uint8, device="cuda" + ) + token_data[:, :NOPE_DIM] = nope_fp8.view(torch.uint8) + token_data[:, NOPE_DIM : NOPE_DIM + 16] = scales.view(torch.uint8) + token_data[:, NOPE_DIM + 16 :] = rope.view(torch.uint8) + + # Scatter into paged cache (loop over requests, not tokens) + block_start = 0 + for r in range(num_reqs): + sl = seq_lens[r] + nb = blocks_per_req[r] + ws = workspace_starts[r] + flat_cache = cache[block_start : block_start + nb].reshape(-1, ENTRY_BYTES) + flat_cache[:sl] = token_data[ws : ws + sl] + block_start += nb + + seq_lens_t = torch.tensor(seq_lens, dtype=torch.int32, device="cuda") + workspace_starts_t = torch.tensor( + workspace_starts, dtype=torch.int32, device="cuda" + ) + + return ( + cache, + block_table, + seq_lens_t, + workspace_starts_t, + num_reqs, + total_tokens, + expected, + ) + + +@pytest.mark.parametrize( + "seq_lens,block_size", + [ + # Production block_size=64 (only supported value for FlashMLA sparse). + # Realistic prefill scenarios with varying request counts. + ([1], 64), # single token edge case + ([64], 64), # 1 req, exactly one block + ([128], 64), # 1 req, crosses block boundary + ([512], 64), # 1 req, longer prefill + ([256, 128, 384], 64), # 3 reqs, varying lengths + ([128] * 4, 64), # 4 reqs, equal lengths + ([64] * 16, 64), # 16 reqs, shorter prefills + ], +) +def test_cp_gather_and_upconvert_fp8_kv_cache(seq_lens, block_size): + """Core correctness test: build cache, run kernel, compare output.""" + ( + cache, + block_table, + seq_lens_t, + workspace_starts_t, + num_reqs, + total_tokens, + expected, + ) = _build_test_case(seq_lens, block_size) + + dst = torch.zeros( + total_tokens, NOPE_DIM + ROPE_DIM, dtype=torch.bfloat16, device="cuda" + ) + + ops.cp_gather_and_upconvert_fp8_kv_cache( + cache, dst, block_table, seq_lens_t, workspace_starts_t, num_reqs + ) + + # NoPE: fp8 dequant has rounding error, so we allow small tolerance. + # The fp8 -> float -> bf16 path can differ by up to ~1 ULP of bf16. + torch.testing.assert_close( + dst[:, :NOPE_DIM], expected[:, :NOPE_DIM], atol=1e-3, rtol=1e-2 + ) + + # RoPE: pure bf16 copy, must be bit-exact + assert torch.equal(dst[:, NOPE_DIM:], expected[:, NOPE_DIM:]) + + +def test_cp_gather_fp8_shuffled_blocks(): + """Test that the kernel correctly follows the block table when + physical blocks are non-contiguous and out of order. + + Here we allocate 4 physical blocks but map the request's 2 logical + blocks to physical blocks [3, 1] (reversed, with gaps). + """ + torch.manual_seed(123) + block_size = 4 + seq_lens = [8] # needs 2 blocks (tokens 0-3 in block 0, 4-7 in block 1) + total_tokens = 8 + + # 4 physical blocks, but only blocks 3 and 1 are used (in that order). + # Tokens 0-3 -> physical block 3, tokens 4-7 -> physical block 1. + num_phys_blocks = 4 + cache = torch.zeros( + num_phys_blocks, block_size, ENTRY_BYTES, dtype=torch.uint8, device="cuda" + ) + block_table = torch.tensor([[3, 1]], dtype=torch.int32, device="cuda") + workspace_starts = torch.tensor([0], dtype=torch.int32, device="cuda") + seq_lens_t = torch.tensor(seq_lens, dtype=torch.int32, device="cuda") + + expected = torch.zeros( + total_tokens, NOPE_DIM + ROPE_DIM, dtype=torch.bfloat16, device="cuda" + ) + + # Fill cache at the shuffled physical locations + for t in range(total_tokens): + # Follow the same block_table lookup the kernel will use + phys = block_table[0, t // block_size].item() + off = t % block_size + + for tile in range(NUM_TILES): + start = tile * GROUP_SIZE + fp8_vals = torch.randn(GROUP_SIZE, device="cuda").to(torch.float8_e4m3fn) + cache[phys, off, start : start + GROUP_SIZE] = fp8_vals.view(torch.uint8) + + # Use a fixed scale to keep this test simple + scale = 1.5 + scale_t = torch.tensor([scale], dtype=torch.float32, device="cuda") + cache[phys, off, NOPE_DIM + tile * 4 : NOPE_DIM + (tile + 1) * 4] = ( + scale_t.view(torch.uint8) + ) + + expected[t, start : start + GROUP_SIZE] = ( + fp8_vals.float() * scale + ).bfloat16() + + rope = torch.randn(ROPE_DIM, dtype=torch.bfloat16, device="cuda") + cache[phys, off, NOPE_DIM + 16 :] = rope.view(torch.uint8) + expected[t, NOPE_DIM:] = rope + + dst = torch.zeros( + total_tokens, NOPE_DIM + ROPE_DIM, dtype=torch.bfloat16, device="cuda" + ) + + ops.cp_gather_and_upconvert_fp8_kv_cache( + cache, dst, block_table, seq_lens_t, workspace_starts, len(seq_lens) + ) + + torch.testing.assert_close( + dst[:, :NOPE_DIM], expected[:, :NOPE_DIM], atol=1e-3, rtol=1e-2 + ) + assert torch.equal(dst[:, NOPE_DIM:], expected[:, NOPE_DIM:]) + + +@pytest.mark.parametrize( + "seq_lens,block_size", + [ + # Large sequence lengths matching end-to-end benchmark scenarios. + # Uses vectorized builder since per-token Python loops would be too slow. + ([8000], 64), + ([16000], 64), + ([32000], 64), + ([64000], 64), + ([96000], 64), + ([128000], 64), + ], +) +def test_cp_gather_fp8_large_seqlens(seq_lens, block_size): + """Correctness test with large sequence lengths matching benchmark + scenarios (8K-128K prefill).""" + ( + cache, + block_table, + seq_lens_t, + workspace_starts_t, + num_reqs, + total_tokens, + expected, + ) = _build_test_case_fast(seq_lens, block_size) + + dst = torch.zeros( + total_tokens, NOPE_DIM + ROPE_DIM, dtype=torch.bfloat16, device="cuda" + ) + + ops.cp_gather_and_upconvert_fp8_kv_cache( + cache, dst, block_table, seq_lens_t, workspace_starts_t, num_reqs + ) + + torch.testing.assert_close( + dst[:, :NOPE_DIM], expected[:, :NOPE_DIM], atol=1e-3, rtol=1e-2 + ) + assert torch.equal(dst[:, NOPE_DIM:], expected[:, NOPE_DIM:]) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_fla_layernorm_guard.py b/asian_ai/performance/vLLM/tests/kernels/test_fla_layernorm_guard.py new file mode 100644 index 0000000000000000000000000000000000000000..d375ac690c4b6907ed4f613fadb309812d79c814 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_fla_layernorm_guard.py @@ -0,0 +1,450 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch +import torch.nn.functional as F + +from vllm.model_executor.layers.fla.ops.layernorm_guard import ( + layer_norm_fwd, + layernorm_fn, + rms_norm_ref, +) +from vllm.utils.torch_utils import set_random_seed + + +def layer_norm_ref( + x, + weight, + bias, + z=None, + eps=1e-6, + group_size=None, + norm_before_gate=True, + is_rms_norm=False, +): + """Reference implementation for both layer norm and RMS norm.""" + if is_rms_norm: + # Use the imported rms_norm_ref for RMS norm cases + return rms_norm_ref( + x, + weight, + bias, + z=z, + eps=eps, + group_size=group_size, + norm_before_gate=norm_before_gate, + upcast=True, + ) + + # Layer norm implementation + dtype = x.dtype + x = x.float() + weight = weight.float() + bias = bias.float() if bias is not None else None + z = z.float() if z is not None else None + + if z is not None and not norm_before_gate: + x = x * F.silu(z) + + if group_size is None: + # Layer norm: subtract mean + mean = x.mean(dim=-1, keepdim=True) + var = ((x - mean).square()).mean(dim=-1, keepdim=True) + rstd = 1 / torch.sqrt(var + eps) + out = (x - mean) * rstd * weight + if bias is not None: + out = out + bias + else: + # Group norm + from einops import rearrange + + x_group = rearrange(x, "... (g d) -> ... g d", d=group_size) + mean = x_group.mean(dim=-1, keepdim=True) + var = ((x_group - mean).square()).mean(dim=-1, keepdim=True) + rstd = 1 / torch.sqrt(var + eps) + x_group = (x_group - mean) * rstd + out = rearrange(x_group, "... g d -> ... (g d)") * weight + if bias is not None: + out = out + bias + + if z is not None and norm_before_gate: + out *= F.silu(z) + + return out.to(dtype) + + +DTYPES = [torch.float16, torch.bfloat16, torch.float32] +# Test various M sizes to ensure rows_per_block logic works correctly +NUM_TOKENS = [ + 1, + 7, + 16, + 63, + 128, + 256, + 512, + 1024, + 2048, + 4096, + 5789, + 8189, + 8191, + 16383, + 32767, +] +HIDDEN_SIZES = [64, 128, 256, 1024] +GROUP_SIZES = [None, 64, 128] # None means full hidden size +NORM_BEFORE_GATE = [True, False] +IS_RMS_NORM = [True, False] +SEEDS = [0, 42] + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", HIDDEN_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("is_rms_norm", IS_RMS_NORM) +@torch.inference_mode() +def test_layer_norm_fwd_basic( + num_tokens: int, + hidden_size: int, + dtype: torch.dtype, + seed: int, + is_rms_norm: bool, +) -> None: + """Test basic layer norm forward pass without z (gate) tensor.""" + set_random_seed(seed) + device = torch.device("cuda:0") + + # Create inputs + x = torch.randn(num_tokens, hidden_size, dtype=dtype, device=device) + weight = torch.randn(hidden_size, dtype=dtype, device=device) + bias = None if is_rms_norm else torch.randn(hidden_size, dtype=dtype, device=device) + eps = 1e-6 + + # Run the triton kernel + out, mean, rstd = layer_norm_fwd( + x, weight, bias, eps, z=None, is_rms_norm=is_rms_norm + ) + + # Run reference implementation + ref_out = layer_norm_ref(x, weight, bias, z=None, eps=eps, is_rms_norm=is_rms_norm) + + # Check outputs + assert out.shape == x.shape + assert out.dtype == x.dtype + torch.testing.assert_close(out, ref_out, atol=1e-2, rtol=1e-2) + + # Check mean and rstd shapes + if not is_rms_norm: + assert mean.shape == (num_tokens,) + assert rstd.shape == (num_tokens,) + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", [128, 256, 1024]) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("norm_before_gate", NORM_BEFORE_GATE) +@pytest.mark.parametrize("is_rms_norm", IS_RMS_NORM) +@torch.inference_mode() +def test_layer_norm_fwd_with_gate( + num_tokens: int, + hidden_size: int, + dtype: torch.dtype, + norm_before_gate: bool, + is_rms_norm: bool, +) -> None: + """Test layer norm forward pass with z (gate) tensor.""" + set_random_seed(42) + device = torch.device("cuda:0") + + # Create inputs + x = torch.randn(num_tokens, hidden_size, dtype=dtype, device=device) + z = torch.randn(num_tokens, hidden_size, dtype=dtype, device=device) + weight = torch.randn(hidden_size, dtype=dtype, device=device) + bias = None if is_rms_norm else torch.randn(hidden_size, dtype=dtype, device=device) + eps = 1e-6 + + # Run the triton kernel + out, mean, rstd = layer_norm_fwd( + x, + weight, + bias, + eps, + z=z, + norm_before_gate=norm_before_gate, + is_rms_norm=is_rms_norm, + ) + + # Run reference implementation + ref_out = layer_norm_ref( + x, + weight, + bias, + z=z, + eps=eps, + norm_before_gate=norm_before_gate, + is_rms_norm=is_rms_norm, + ) + + # Check outputs + assert out.shape == x.shape + assert out.dtype == x.dtype + torch.testing.assert_close(out, ref_out, atol=1e-2, rtol=1e-2) + + +@pytest.mark.parametrize("num_tokens", [128, 512]) +@pytest.mark.parametrize("hidden_size", [512, 1024]) +@pytest.mark.parametrize("group_size", [64, 128, 256]) +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@pytest.mark.parametrize("is_rms_norm", IS_RMS_NORM) +@torch.inference_mode() +def test_layer_norm_fwd_with_groups( + num_tokens: int, + hidden_size: int, + group_size: int, + dtype: torch.dtype, + is_rms_norm: bool, +) -> None: + """Test layer norm forward pass with group normalization.""" + if hidden_size % group_size != 0: + pytest.skip( + f"hidden_size {hidden_size} not divisible by group_size {group_size}" + ) + + set_random_seed(42) + device = torch.device("cuda:0") + + # Create inputs + x = torch.randn(num_tokens, hidden_size, dtype=dtype, device=device) + weight = torch.randn(hidden_size, dtype=dtype, device=device) + bias = None if is_rms_norm else torch.randn(hidden_size, dtype=dtype, device=device) + eps = 1e-6 + + ngroups = hidden_size // group_size + + # Run the triton kernel + out, mean, rstd = layer_norm_fwd( + x, weight, bias, eps, z=None, group_size=group_size, is_rms_norm=is_rms_norm + ) + + # Run reference implementation + ref_out = layer_norm_ref( + x, weight, bias, z=None, eps=eps, group_size=group_size, is_rms_norm=is_rms_norm + ) + + # Check outputs + assert out.shape == x.shape + assert out.dtype == x.dtype + torch.testing.assert_close(out, ref_out, atol=1e-2, rtol=1e-2) + + # Check mean and rstd shapes for groups + if not is_rms_norm: + assert mean.shape == (ngroups * num_tokens,) + assert rstd.shape == (ngroups * num_tokens,) + + +@pytest.mark.parametrize("num_tokens", [7, 63, 128, 513, 1024, 2049]) +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@torch.inference_mode() +def test_layer_norm_rows_per_block( + num_tokens: int, + dtype: torch.dtype, +) -> None: + """Test that rows_per_block logic works correctly for various M sizes.""" + set_random_seed(42) + device = torch.device("cuda:0") + hidden_size = 1024 + + # Create inputs + x = torch.randn(num_tokens, hidden_size, dtype=dtype, device=device) + weight = torch.randn(hidden_size, dtype=dtype, device=device) + bias = torch.randn(hidden_size, dtype=dtype, device=device) + eps = 1e-6 + + # Run the triton kernel + out, mean, rstd = layer_norm_fwd(x, weight, bias, eps, z=None, is_rms_norm=False) + + # Run reference implementation + ref_out = layer_norm_ref(x, weight, bias, z=None, eps=eps, is_rms_norm=False) + + # Check outputs + torch.testing.assert_close(out, ref_out, atol=1e-2, rtol=1e-2) + + +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@torch.inference_mode() +def test_strided_input(dtype: torch.dtype) -> None: + """Test that the kernel handles non-contiguous (strided) + inputs correctly.""" + set_random_seed(42) + device = torch.device("cuda:0") + num_tokens = 128 + hidden_size = 1024 + + # Create a larger tensor and take a strided slice + x_large = torch.randn(num_tokens, hidden_size * 2, dtype=dtype, device=device) + x = x_large[:, :hidden_size] + + # Make it contiguous for the kernel + x_contiguous = x.contiguous() + + weight = torch.randn(hidden_size, dtype=dtype, device=device) + bias = torch.randn(hidden_size, dtype=dtype, device=device) + eps = 1e-6 + + # Run the triton kernel with contiguous input + out, mean, rstd = layer_norm_fwd( + x_contiguous, weight, bias, eps, z=None, is_rms_norm=False + ) + + # Run reference implementation + ref_out = layer_norm_ref( + x_contiguous, weight, bias, z=None, eps=eps, is_rms_norm=False + ) + + # Check outputs + torch.testing.assert_close(out, ref_out, atol=1e-2, rtol=1e-2) + + +@pytest.mark.parametrize("num_tokens", [1, 128, 2048]) +@pytest.mark.parametrize("hidden_size", [768, 4096]) +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@torch.inference_mode() +def test_output_buffer_provided( + num_tokens: int, + hidden_size: int, + dtype: torch.dtype, +) -> None: + """Test that the kernel works when an output buffer is provided.""" + set_random_seed(42) + device = torch.device("cuda:0") + + # Create inputs + x = torch.randn(num_tokens, hidden_size, dtype=dtype, device=device) + weight = torch.randn(hidden_size, dtype=dtype, device=device) + bias = torch.randn(hidden_size, dtype=dtype, device=device) + eps = 1e-6 + + # Pre-allocate output buffer + out_buffer = torch.empty_like(x) + + # Run the triton kernel with provided output + out, mean, rstd = layer_norm_fwd( + x, weight, bias, eps, z=None, out=out_buffer, is_rms_norm=False + ) + + # Check that the provided buffer was used + assert out.data_ptr() == out_buffer.data_ptr() + + # Run reference implementation + ref_out = layer_norm_ref(x, weight, bias, z=None, eps=eps, is_rms_norm=False) + + # Check outputs + torch.testing.assert_close(out, ref_out, atol=1e-2, rtol=1e-2) + + +@pytest.mark.parametrize( + "shape", + [ + (4, 16, 1024), # 3D tensor + (2, 8, 512, 256), # 4D tensor + ], +) +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@torch.inference_mode() +def test_multidimensional_input( + shape: tuple, + dtype: torch.dtype, +) -> None: + """Test that the autograd function handles multidimensional inputs.""" + set_random_seed(42) + device = torch.device("cuda:0") + hidden_size = shape[-1] + + # Create inputs + x = torch.randn(*shape, dtype=dtype, device=device) + weight = torch.randn(hidden_size, dtype=dtype, device=device) + bias = torch.randn(hidden_size, dtype=dtype, device=device) + eps = 1e-6 + + # Run through autograd function + out = layernorm_fn(x, weight, bias, z=None, eps=eps) + + # Run reference implementation + ref_out = layer_norm_ref(x, weight, bias, z=None, eps=eps, is_rms_norm=False) + + # Check outputs + assert out.shape == x.shape + torch.testing.assert_close(out, ref_out, atol=1e-2, rtol=1e-2) + + +@pytest.mark.parametrize("num_tokens", [1, 128, 1024]) +@pytest.mark.parametrize("hidden_size", [64, 256, 1024]) +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32]) +@pytest.mark.parametrize("has_gate", [True, False]) +@pytest.mark.parametrize("group_size", [None, 64]) +@pytest.mark.parametrize("norm_before_gate", [True, False]) +@torch.inference_mode() +def test_rmsnorm_gated_forward_native_dtype( + default_vllm_config, + num_tokens: int, + hidden_size: int, + dtype: torch.dtype, + has_gate: bool, + group_size: int | None, + norm_before_gate: bool, +): + """Test that RMSNormGated.forward_native preserves input dtype.""" + if group_size is not None and hidden_size % group_size != 0: + pytest.skip( + f"hidden_size {hidden_size} not divisible by group_size {group_size}" + ) + + from vllm.model_executor.layers.layernorm import RMSNormGated + + device = torch.device("cuda:0") + set_random_seed(42) + + layer = RMSNormGated( + hidden_size, + eps=1e-5, + group_size=group_size, + norm_before_gate=norm_before_gate, + device=device, + dtype=dtype, + ) + + x = torch.randn(num_tokens, hidden_size, dtype=dtype, device=device) + z = ( + torch.randn(num_tokens, hidden_size, dtype=dtype, device=device) + if has_gate + else None + ) + + out = layer.forward_native(x, z) + + # Verify dtype preservation + assert out.dtype == dtype, f"Expected {dtype}, got {out.dtype}" + + # Verify numerical correctness against reference + ref_out = rms_norm_ref( + x, + layer.weight, + layer.bias, + z=z, + eps=1e-5, + group_size=group_size, + norm_before_gate=norm_before_gate, + upcast=True, + ) + torch.testing.assert_close(out, ref_out, atol=1e-2, rtol=1e-2) + + +if __name__ == "__main__": + # Run a quick smoke test + test_layer_norm_fwd_basic(128, 1024, torch.float16, 42, False) + test_layer_norm_fwd_with_gate(128, 1024, torch.float16, True, False) + test_layer_norm_rows_per_block(513, torch.float16) + print("All smoke tests passed!") diff --git a/asian_ai/performance/vLLM/tests/kernels/test_flex_attention.py b/asian_ai/performance/vLLM/tests/kernels/test_flex_attention.py new file mode 100644 index 0000000000000000000000000000000000000000..510353403f1eda814846fa6b3a986b1229d9386d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_flex_attention.py @@ -0,0 +1,398 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Integration tests for FlexAttention backend vs default backend""" + +import pytest +import torch +from packaging import version + +from tests.utils import set_random_seed +from tests.v1.attention.utils import ( + BatchSpec, + create_common_attn_metadata, + create_standard_kv_cache_spec, + create_vllm_config, +) +from vllm.model_executor.layers.attention import Attention +from vllm.v1.attention.backends.flex_attention import ( + BlockSparsityHint, + FlexAttentionMetadataBuilder, + physical_to_logical_mapping, +) + +from ..models.utils import check_embeddings_close, check_logprobs_close + +TORCH_VERSION = version.parse(torch.__version__) +MINIMUM_TORCH_VERSION = version.parse("2.7.0") +DIRECT_BUILD_VERSION = version.parse("2.9.dev0") + + +@pytest.mark.skipif( + not torch.cuda.is_available() or TORCH_VERSION < MINIMUM_TORCH_VERSION, + reason="CUDA not available or PyTorch version < 2.7", +) +def test_flex_attention_full_cudagraphs(vllm_runner): + """Test the numerics for flex attention full cudagraphs support.""" + model_name = "Qwen/Qwen2.5-1.5B-Instruct" + seed = 42 + max_tokens = 24 + num_logprobs = 5 + prompts = [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + ] + + # Run with flex attention eager + set_random_seed(seed) + with vllm_runner( + model_name, + runner="generate", + tensor_parallel_size=1, + num_gpu_blocks_override=128, + enforce_eager=True, + attention_config={"backend": "FLEX_ATTENTION"}, + ) as llm_flex: + output_eager = llm_flex.generate_greedy_logprobs( + prompts, max_tokens, num_logprobs + ) + + # Run with flex attention compiled + set_random_seed(seed) + with vllm_runner( + model_name, + runner="generate", + tensor_parallel_size=1, + num_gpu_blocks_override=128, + enforce_eager=False, + gpu_memory_utilization=0.85, + attention_config={"backend": "FLEX_ATTENTION"}, + ) as llm_default: + output_compile = llm_default.generate_greedy_logprobs( + prompts, max_tokens, num_logprobs + ) + + check_logprobs_close( + outputs_0_lst=output_eager, + outputs_1_lst=output_compile, + name_0="eager", + name_1="compile", + ) + + +def windowed_causal_mask_mod(b, h, q_idx, kv_idx): + return (kv_idx <= q_idx) & (q_idx - kv_idx < 4) + + +@pytest.mark.skipif( + not torch.cuda.is_available() or TORCH_VERSION < MINIMUM_TORCH_VERSION, + reason="CUDA not available or PyTorch version < 2.7", +) +def test_flex_attention_custom_mask_full_cudagraphs(vllm_runner, monkeypatch): + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "0") + monkeypatch.setattr( + Attention, + "logical_mask_mod", + staticmethod(windowed_causal_mask_mod), + raising=False, + ) + + model_name = "Qwen/Qwen2.5-1.5B-Instruct" + seed = 42 + max_tokens = 24 + num_logprobs = 5 + prompts = [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + ] + + set_random_seed(seed) + with vllm_runner( + model_name, + runner="generate", + tensor_parallel_size=1, + num_gpu_blocks_override=128, + enforce_eager=True, + attention_config={"backend": "FLEX_ATTENTION"}, + ) as llm_eager: + output_eager = llm_eager.generate_greedy_logprobs( + prompts, max_tokens, num_logprobs + ) + + set_random_seed(seed) + with vllm_runner( + model_name, + runner="generate", + tensor_parallel_size=1, + num_gpu_blocks_override=128, + enforce_eager=False, + gpu_memory_utilization=0.85, + compilation_config={ + "cudagraph_mode": "FULL", + "cudagraph_capture_sizes": [4], + }, + attention_config={"backend": "FLEX_ATTENTION"}, + ) as llm_cudagraph: + output_cudagraph = llm_cudagraph.generate_greedy_logprobs( + prompts, max_tokens, num_logprobs + ) + + check_logprobs_close( + outputs_0_lst=output_eager, + outputs_1_lst=output_cudagraph, + name_0="eager", + name_1="cudagraph", + ) + + +@pytest.mark.skipif( + not torch.cuda.is_available() or TORCH_VERSION < MINIMUM_TORCH_VERSION, + reason="CUDA not available or PyTorch version < 2.7", +) +def test_flex_attention_vs_default_backend(vllm_runner): + """Test that FlexAttention produces the same outputs as the default backend. + + This test compares the outputs from the FlexAttention backend with + the default backend, ensuring they are similar when using the same seed. + """ + model_name = "Qwen/Qwen2.5-1.5B-Instruct" + seed = 42 + max_tokens = 24 + num_logprobs = 5 + prompts = [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + ] + + # Run with flex attention + set_random_seed(seed) + with vllm_runner( + model_name, + runner="generate", + tensor_parallel_size=1, + num_gpu_blocks_override=128, + enforce_eager=True, + attention_config={"backend": "FLEX_ATTENTION"}, + ) as llm_flex: + output_flex = llm_flex.generate_greedy_logprobs( + prompts, max_tokens, num_logprobs + ) + + # Run with default backend + set_random_seed(seed) + with vllm_runner( + model_name, + runner="generate", + tensor_parallel_size=1, + num_gpu_blocks_override=128, + enforce_eager=True, + gpu_memory_utilization=0.85, + ) as llm_default: + output_default = llm_default.generate_greedy_logprobs( + prompts, max_tokens, num_logprobs + ) + + check_logprobs_close( + outputs_0_lst=output_flex, + outputs_1_lst=output_default, + name_0="flex", + name_1="default", + ) + + +@pytest.mark.skipif( + not torch.cuda.is_available() or TORCH_VERSION < MINIMUM_TORCH_VERSION, + reason="CUDA not available or PyTorch version < 2.7", +) +def test_encoder_flex_attention_vs_default_backend(vllm_runner): + """Test that FlexAttention produces the same outputs as the default backend. + + This test compares the outputs from the FlexAttention backend with + the default backend for encoder models. + """ + model_name = "BAAI/bge-base-en-v1.5" + prompts = [ + "Hello, my name is", + "The president of the United States is", + "The capital of France is", + ] + + # Run with flex attention + with vllm_runner( + model_name, + runner="pooling", + dtype=torch.bfloat16, + tensor_parallel_size=1, + max_model_len=100, + enforce_eager=True, + attention_config={"backend": "FLEX_ATTENTION"}, + ) as llm_flex: + flex_outputs = llm_flex.embed(prompts) + + # Run with default backend + with vllm_runner( + model_name, + runner="pooling", + dtype=torch.bfloat16, + tensor_parallel_size=1, + max_model_len=100, + enforce_eager=True, + ) as llm_default: + default_outputs = llm_default.embed(prompts) + + check_embeddings_close( + embeddings_0_lst=flex_outputs, + embeddings_1_lst=default_outputs, + name_0="flex", + name_1="default", + tol=1e-2, + ) + + +@pytest.mark.skipif( + not torch.cuda.is_available() or TORCH_VERSION < DIRECT_BUILD_VERSION, + reason="CUDA not available or PyTorch version < 2.7", +) +def test_block_mask_direct_vs_slow_path(): + """Test that direct path block mask is a superset of slow path. + + The direct path may include extra blocks for performance (over-estimation), + but must include all blocks that the slow path determines are necessary. + """ + device = torch.device("cuda") + + vllm_config = create_vllm_config( + model_name="meta-llama/Meta-Llama-3-8B", block_size=16, max_model_len=1024 + ) + kv_cache_spec = create_standard_kv_cache_spec(vllm_config) + + # Use a mixed batch that will create groups spanning multiple sequences + batch_spec = BatchSpec( + seq_lens=[35, 64, 128, 256], query_lens=[33, 5, 32, 64], name="test_mixed_batch" + ) + + common_attn_metadata = create_common_attn_metadata( + batch_spec, vllm_config.cache_config.block_size, device + ) + + builder = FlexAttentionMetadataBuilder(kv_cache_spec, [], vllm_config, device) + + metadata_direct = builder.build( + common_prefix_len=0, common_attn_metadata=common_attn_metadata + ) + builder.direct_build = False + metadata_slow = builder.build( + common_prefix_len=0, common_attn_metadata=common_attn_metadata + ) + + assert metadata_direct.block_mask is not None + assert metadata_slow.block_mask is not None + + # Extract block indices for comparison, B, H are the same + direct_indices = metadata_direct.block_mask.kv_indices[0, 0] + slow_indices = metadata_slow.block_mask.kv_indices[0, 0] + direct_num = metadata_direct.block_mask.kv_num_blocks[0, 0] + slow_num = metadata_slow.block_mask.kv_num_blocks[0, 0] + + # main test: every block needed by slow path must be in direct path + num_groups = direct_num.shape[0] + all_contained = True + missing_details = [] + + for group_idx in range(num_groups): + direct_blocks = set(direct_indices[group_idx, : direct_num[group_idx]].tolist()) + slow_blocks = set(slow_indices[group_idx, : slow_num[group_idx]].tolist()) + + missing_blocks = slow_blocks - direct_blocks + if missing_blocks: + all_contained = False + missing_details.append( + f"Group {group_idx}: missing {sorted(missing_blocks)}" + ) + + assert all_contained, ( + "Direct path is missing blocks required by slow path:\n" + + "\n".join(missing_details) + ) + + +def test_physical_to_logical_mapping_handles_reused_blocks(): + """Regression test: reused physical blocks map to the latest logical block. + + For sliding-window / hybrid attention layers, physical KV-cache blocks can be + reused over time. The inverse mapping must therefore select the latest + logical block index for a physical block id. + """ + # Padding should not make physical block 0 look live. + block_table = torch.tensor([[6, 0, 0, 0]], dtype=torch.int32) + seq_lens = torch.tensor([1 * 16], dtype=torch.int32) # only 1 block valid + out = physical_to_logical_mapping( + block_table=block_table, seq_lens=seq_lens, block_size=16, total_blocks=10 + ) + assert out[0, 0].item() == -1 + assert out[0, 6].item() == 0 + + # If a physical block id appears multiple times (block reuse), mapping should + # point to the latest logical block index. + block_table2 = torch.tensor([[2, 2, 5]], dtype=torch.int32) + seq_lens2 = torch.tensor([3 * 16], dtype=torch.int32) + out2 = physical_to_logical_mapping( + block_table=block_table2, seq_lens=seq_lens2, block_size=16, total_blocks=8 + ) + assert out2[0, 2].item() == 1 + + +@pytest.mark.skipif( + not torch.cuda.is_available() or TORCH_VERSION < DIRECT_BUILD_VERSION, + reason="CUDA not available or PyTorch version < 2.9", +) +def test_block_sparsity_hint_prunes_blocks(): + """Test that BlockSparsityHint prunes KV blocks from the direct build path. + + Uses a hint that only keeps the diagonal (q_block == kv_block) to verify + that off-diagonal blocks are excluded from the resulting BlockMask. + """ + device = torch.device("cuda") + + vllm_config = create_vllm_config( + model_name="facebook/opt-125m", + block_size=16, + max_model_len=1024, + ) + kv_cache_spec = create_standard_kv_cache_spec(vllm_config) + + batch_spec = BatchSpec( + seq_lens=[256], + query_lens=[256], + name="test_sparsity_hint", + ) + + common_attn_metadata = create_common_attn_metadata( + batch_spec, vllm_config.cache_config.block_size, device + ) + + builder = FlexAttentionMetadataBuilder(kv_cache_spec, [], vllm_config, device) + + metadata_no_hint = builder.build( + common_prefix_len=0, common_attn_metadata=common_attn_metadata + ) + metadata_no_hint.block_mask = metadata_no_hint._build_block_mask_direct() + assert metadata_no_hint.block_mask.kv_num_blocks.max().item() > 1 + + def diagonal_hint(q_block_idx, kv_block_idx, block_size): + return q_block_idx == kv_block_idx + + metadata_with_hint = builder.build( + common_prefix_len=0, common_attn_metadata=common_attn_metadata + ) + metadata_with_hint.block_sparsity_hint = BlockSparsityHint( + hint_fn=diagonal_hint, + ) + metadata_with_hint.block_mask = metadata_with_hint._build_block_mask_direct() + assert metadata_with_hint.block_mask.kv_num_blocks.max().item() <= 1 + + +if __name__ == "__main__": + pytest.main([__file__]) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_fp32_router_gemm.py b/asian_ai/performance/vLLM/tests/kernels/test_fp32_router_gemm.py new file mode 100644 index 0000000000000000000000000000000000000000..335462c6bbf75f527639af7ef16f563a985ad32c --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_fp32_router_gemm.py @@ -0,0 +1,84 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for fp32_router_gemm kernel: activation×weight→fp32. + +Supported (hidden_size, num_experts) pairs: + (3072, 256) -> MiniMax-M2/M2.5, (6144, 128) -> MiniMax-M3 + +Correctness baseline: torch.matmul in float64. +""" + +import pytest +import torch + +from vllm._custom_ops import fp32_router_gemm + +# (hidden_size, num_experts) +SHAPES = [(3072, 256), (6144, 128)] +# Absolute tolerance for fp32 kernel vs float64 reference +ATOL_FP32 = 2e-4 +ATOL_BF16 = 2e-2 # bf16 activation has lower precision + + +def _requires_sm90(): + if not torch.cuda.is_available(): + pytest.skip("CUDA not available") + major, minor = torch.cuda.get_device_capability() + if major * 10 + minor < 90: + pytest.skip(f"fp32_router_gemm requires SM90+, got SM{major}{minor}") + + +def _ref(mat_a: torch.Tensor, mat_b: torch.Tensor) -> torch.Tensor: + """Reference: F.linear in float32 on GPU.""" + return torch.nn.functional.linear(mat_a.float(), mat_b.float()) + + +@pytest.mark.parametrize("hidden_dim,num_experts", SHAPES) +@pytest.mark.parametrize("num_tokens", [1, 2, 4, 8, 16, 32]) +def test_fp32_activation(num_tokens: int, hidden_dim: int, num_experts: int): + """fp32 activation → fp32 output should match reference closely.""" + _requires_sm90() + torch.manual_seed(42) + device = torch.device("cuda") + mat_a = torch.randn(num_tokens, hidden_dim, dtype=torch.float32, device=device) + mat_b = torch.randn(num_experts, hidden_dim, dtype=torch.float32, device=device) + + out = fp32_router_gemm(mat_a, mat_b) + ref = _ref(mat_a, mat_b) + + assert out.shape == (num_tokens, num_experts) + assert out.dtype == torch.float32 + torch.testing.assert_close(out, ref, atol=ATOL_FP32, rtol=0) + + +@pytest.mark.parametrize("hidden_dim,num_experts", SHAPES) +@pytest.mark.parametrize("num_tokens", [1, 2, 4, 8, 16, 32]) +def test_bf16_activation(num_tokens: int, hidden_dim: int, num_experts: int): + """bf16 activation → fp32 output should match reference within bf16 error.""" + _requires_sm90() + torch.manual_seed(42) + device = torch.device("cuda") + mat_a_bf16 = torch.randn( + num_tokens, hidden_dim, dtype=torch.bfloat16, device=device + ) + mat_b = torch.randn(num_experts, hidden_dim, dtype=torch.float32, device=device) + + out = fp32_router_gemm(mat_a_bf16, mat_b) + ref = _ref(mat_a_bf16, mat_b).to(device) + + assert out.shape == (num_tokens, num_experts) + assert out.dtype == torch.float32 + torch.testing.assert_close(out, ref, atol=ATOL_BF16, rtol=0) + + +@pytest.mark.parametrize("hidden_dim,num_experts", SHAPES) +def test_output_shape_and_dtype(hidden_dim: int, num_experts: int): + """Basic shape and dtype checks.""" + _requires_sm90() + device = torch.device("cuda") + mat_a = torch.randn(4, hidden_dim, dtype=torch.float32, device=device) + mat_b = torch.randn(num_experts, hidden_dim, dtype=torch.float32, device=device) + out = fp32_router_gemm(mat_a, mat_b) + assert out.shape == (4, num_experts) + assert out.dtype == torch.float32 + assert out.device.type == "cuda" diff --git a/asian_ai/performance/vLLM/tests/kernels/test_fused_deepseek_v32_norm_rope.py b/asian_ai/performance/vLLM/tests/kernels/test_fused_deepseek_v32_norm_rope.py new file mode 100644 index 0000000000000000000000000000000000000000..4e961ddd1dbb902a2257c4abd2434059567b4a73 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_fused_deepseek_v32_norm_rope.py @@ -0,0 +1,562 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit tests for the horizontally-fused deepseek_v32 (NVIDIA SM100) Triton +kernels used by the specialized DSA model: + + fused_norm_rope + - q : q_lora RMSNorm + - kv : kv_lora RMSNorm + (interleaved) RoPE on k_pe + MLA cache insert + (bf16 or per-tensor fp8) + - idx: indexer-K LayerNorm + RoPE (interleaved or NeoX) + UE8M0 fp8 quant + + packed indexer cache insert; plus the top-k buffer (-1) fill + fused_q + - mqa: ql_nope + (interleaved) RoPE'd q_pe, concat-quantized to the fp8 MQA + query + - idx: indexer-Q RoPE (interleaved or NeoX) + UE8M0 fp8 quant + folded + index weights + fused_eh_norm (MTP): zero-at-pos-0 + enorm RMSNorm(embeds) + hnorm + RMSNorm(prev), concatenated side-by-side + +Each kernel is compared against a PyTorch reference. The kernel keeps the whole +pipeline in fp32 and rounds once, so it can land on the opposite side of a +round-to-nearest tie from the reference for a few elements: deterministic fp8 +outputs are checked within 1 representable-step (ULP); bf16 norm/RoPE outputs use +rtol/atol=1e-2 (the tolerance the sibling deepseek_v4 fused-kernel test uses). +""" + +import pytest +import torch + +from vllm.models.deepseek_v32.nvidia import kernels as K +from vllm.platforms import current_platform + +FP8 = torch.float8_e4m3fn +FP8_MAX = 448.0 + +# GLM-5.2 / DeepSeek-V3.2 shapes (TP8 local heads). +Q_LORA = 2048 +KV_LORA = 512 +ROPE_DIM = 64 +NUM_HEADS = 8 +INDEX_HEADS = 32 +INDEX_HEAD_DIM = 128 +HIDDEN = 6144 +EPS = 1e-6 + +pytestmark = pytest.mark.skipif( + not current_platform.is_cuda() or not current_platform.has_device_capability(89), + reason="deepseek_v32 fused kernels require CUDA with fp8 (SM89+)", +) + + +# ── reference helpers ──────────────────────────────────────────────────────── + + +def make_cos_sin(max_pos: int, rot_dim: int, device) -> torch.Tensor: + """cos||sin cache: row[pos] = [cos(theta)(rot/2), sin(theta)(rot/2)].""" + half = rot_dim // 2 + inv_freq = 1.0 / ( + 10000.0 ** (torch.arange(0, half, dtype=torch.float32, device=device) / half) + ) + t = torch.arange(max_pos, dtype=torch.float32, device=device) + freqs = torch.einsum("i,j->ij", t, inv_freq) + return torch.cat([freqs.cos(), freqs.sin()], dim=-1) + + +def rms_norm(x: torch.Tensor, w: torch.Tensor) -> torch.Tensor: + """RMSNorm matching kernels._rms_norm (fp32, eps inside rsqrt). Returns fp32.""" + xf = x.float() + ms = xf.pow(2).mean(dim=-1, keepdim=True) + return xf * torch.rsqrt(ms + EPS) * w.float() + + +def layer_norm(x: torch.Tensor, w: torch.Tensor, b: torch.Tensor) -> torch.Tensor: + xf = x.float() + mean = xf.mean(dim=-1, keepdim=True) + var = (xf - mean).pow(2).mean(dim=-1, keepdim=True) + return (xf - mean) * torch.rsqrt(var + EPS) * w.float() + b.float() + + +def rope( + x: torch.Tensor, pos: torch.Tensor, cos_sin: torch.Tensor, interleave: bool +) -> torch.Tensor: + """Apply RoPE to the first ``rot_dim`` elements of x's last dim. + + x: [..., head_dim] fp32. ``cos_sin`` is [max_pos, rot_dim]. ``interleave`` + selects adjacent-pair (GLM) vs split-half NeoX (DeepSeek-V3.2) layout. + """ + rot = cos_sin.shape[-1] + half = rot // 2 + cs = cos_sin[pos.long()] + cos, sin = cs[..., :half], cs[..., half:] + out = x.float().clone() + r = out[..., :rot] + if interleave: + x1, x2 = r[..., 0::2].clone(), r[..., 1::2].clone() + r[..., 0::2] = x1 * cos - x2 * sin + r[..., 1::2] = x2 * cos + x1 * sin + else: + x1, x2 = r[..., :half].clone(), r[..., half:].clone() + r[..., :half] = x1 * cos - x2 * sin + r[..., half:] = x2 * cos + x1 * sin + return out + + +def ue8m0_quant(vals: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + """Per-row (last dim) UE8M0 fp8 quant matching kernels._fp8_ue8m0_quantize.""" + amax = vals.float().abs().amax(dim=-1, keepdim=True) + scale = torch.clamp(amax, min=1e-4) / FP8_MAX + scale = torch.exp2(torch.ceil(torch.log2(scale))) + q = (vals.float() / scale).to(FP8) + return q, scale.squeeze(-1) + + +def _bf16_ulp(a: torch.Tensor, b: torch.Tensor) -> int: + def key(t): + u = t.contiguous().view(torch.int16).to(torch.int64) & 0xFFFF + return torch.where(u >= 0x8000, 0xFFFF - u, u + 0x8000) + + return int((key(a) - key(b)).abs().max().item()) + + +def _fp8_ulp(a: torch.Tensor, b: torch.Tensor) -> int: + def key(t): + u = t.contiguous().view(torch.uint8).to(torch.int64) + return torch.where(u >= 0x80, 0xFF - u, u + 0x80) + + return int((key(a) - key(b)).abs().max().item()) + + +def assert_bf16(got: torch.Tensor, ref_fp32: torch.Tensor, msg: str): + # Kernel keeps RMSNorm/RoPE in fp32 and rounds to bf16 once; the fp32 + # reduction/FMA order differs from torch, so a few elements land on the + # opposite side of a round-to-nearest tie. Use the same tolerance the + # sibling deepseek_v4 fused-kernel test uses for this bf16 norm+rope class. + torch.testing.assert_close( + got.float(), ref_fp32.float(), rtol=1e-2, atol=1e-2, msg=lambda m: f"{msg}: {m}" + ) + + +def assert_fp8(got: torch.Tensor, ref: torch.Tensor, msg: str): + assert _fp8_ulp(got, ref) <= 1, f"{msg}: >1 fp8 ULP" + + +# ── fused_norm_rope ────────────────────────────────────────────────────────── + + +@pytest.mark.parametrize("num_tokens", [1, 4, 17, 512, 4096]) +@pytest.mark.parametrize("index_interleave", [True, False]) +@pytest.mark.parametrize("mla_fp8", [False, True]) +def test_fused_norm_rope(num_tokens: int, index_interleave: bool, mla_fp8: bool): + torch.manual_seed(0) + dev = "cuda" + max_pos = 8192 + pos = torch.arange(num_tokens, device=dev, dtype=torch.int64) % max_pos + + q_c = torch.randn(num_tokens, Q_LORA, device=dev, dtype=torch.bfloat16) + kv_c = torch.randn(num_tokens, KV_LORA, device=dev, dtype=torch.bfloat16) + k_pe = torch.randn(num_tokens, ROPE_DIM, device=dev, dtype=torch.bfloat16) + qw = torch.randn(Q_LORA, device=dev, dtype=torch.bfloat16) + kvw = torch.randn(KV_LORA, device=dev, dtype=torch.bfloat16) + ik = torch.randn(num_tokens, INDEX_HEAD_DIM, device=dev, dtype=torch.bfloat16) + ikw = torch.randn(INDEX_HEAD_DIM, device=dev, dtype=torch.float32) + ikb = torch.randn(INDEX_HEAD_DIM, device=dev, dtype=torch.float32) + + mla_cos_sin = make_cos_sin(max_pos, ROPE_DIM, dev) # MLA k_pe: interleaved + idx_cos_sin = make_cos_sin(max_pos, ROPE_DIM, dev) + + bs = max_pos # single block covering all tokens + mla_dim = KV_LORA + ROPE_DIM + if mla_fp8: + mla_cache = torch.zeros(1, bs, mla_dim, device=dev, dtype=torch.uint8) + mla_dtype = "fp8" + mla_k_scale = torch.tensor([0.3], device=dev, dtype=torch.float32) + else: + mla_cache = torch.zeros(1, bs, mla_dim, device=dev, dtype=torch.bfloat16) + mla_dtype = "auto" + mla_k_scale = None + idx_row = INDEX_HEAD_DIM + INDEX_HEAD_DIM // 128 * 4 # 132 + idx_cache = torch.zeros(1, bs, idx_row, device=dev, dtype=torch.uint8) + slot = torch.arange(num_tokens, device=dev, dtype=torch.int64) + topk = torch.full((num_tokens, 2048), 7, device=dev, dtype=torch.int32) + + q_out = K.fused_norm_rope( + pos, + q_c, + qw, + EPS, + kv_c, + kvw, + EPS, + k_pe, + mla_cos_sin, + ik, + ikw, + ikb, + EPS, + idx_cos_sin, + topk, + slot_mapping=slot, + indexer_k_cache=idx_cache, + mla_kv_cache=mla_cache, + mla_kv_cache_dtype=mla_dtype, + mla_k_scale=mla_k_scale, + has_indexer=True, + index_rope_interleave=index_interleave, + ) + + # q_lora RMSNorm + assert_bf16(q_out, rms_norm(q_c, qw), "q_c rmsnorm") + + # MLA cache: [kv_c_normed | k_pe_roped(interleaved)] + kv_ref = rms_norm(kv_c, kvw) + kpe_ref = rope(k_pe.float(), pos, mla_cos_sin, interleave=True) + if mla_fp8: + cache = mla_cache.view(FP8)[0, :num_tokens] + s = mla_k_scale.item() + assert_fp8(cache[:, :KV_LORA], (kv_ref / s).to(FP8), "MLA kv fp8") + assert_fp8(cache[:, KV_LORA:], (kpe_ref / s).to(FP8), "MLA k_pe fp8") + else: + cache = mla_cache[0, :num_tokens] + assert_bf16(cache[:, :KV_LORA], kv_ref, "MLA kv bf16") + assert_bf16(cache[:, KV_LORA:], kpe_ref, "MLA k_pe bf16") + + # Indexer-K cache (packed [bs*head_dim fp8 | bs*4 fp32 scale]). + ik_ref = layer_norm(ik, ikw, ikb) + ik_ref = rope(ik_ref, pos, idx_cos_sin, interleave=index_interleave) + q_ref, s_ref = ue8m0_quant(ik_ref) + flat = idx_cache[0].reshape(-1) + vals = flat[: bs * INDEX_HEAD_DIM].view(FP8).reshape(bs, INDEX_HEAD_DIM) + scales = flat[bs * INDEX_HEAD_DIM :].view(torch.float32) + assert_fp8(vals[:num_tokens], q_ref, "indexer-K fp8") + torch.testing.assert_close(scales[:num_tokens], s_ref, rtol=0, atol=0) + + # Top-k buffer cleared to -1 on indexer layers. + assert (topk == -1).all(), "topk buffer not cleared on indexer layer" + + +@pytest.mark.parametrize("num_tokens", [1, 17, 512]) +def test_fused_norm_rope_no_indexer(num_tokens: int): + """Shared (no-indexer) layer: q + kv/MLA only; top-k buffer untouched.""" + torch.manual_seed(1) + dev = "cuda" + max_pos = 8192 + pos = torch.arange(num_tokens, device=dev, dtype=torch.int64) + + q_c = torch.randn(num_tokens, Q_LORA, device=dev, dtype=torch.bfloat16) + kv_c = torch.randn(num_tokens, KV_LORA, device=dev, dtype=torch.bfloat16) + k_pe = torch.randn(num_tokens, ROPE_DIM, device=dev, dtype=torch.bfloat16) + qw = torch.randn(Q_LORA, device=dev, dtype=torch.bfloat16) + kvw = torch.randn(KV_LORA, device=dev, dtype=torch.bfloat16) + mla_cos_sin = make_cos_sin(max_pos, ROPE_DIM, dev) + + bs = max_pos + mla_cache = torch.zeros(1, bs, KV_LORA + ROPE_DIM, device=dev, dtype=torch.bfloat16) + slot = torch.arange(num_tokens, device=dev, dtype=torch.int64) + topk = torch.full((num_tokens, 2048), 7, device=dev, dtype=torch.int32) + + q_out = K.fused_norm_rope( + pos, + q_c, + qw, + EPS, + kv_c, + kvw, + EPS, + k_pe, + mla_cos_sin, + None, + None, + None, + EPS, + None, + topk, + slot_mapping=slot, + indexer_k_cache=None, + mla_kv_cache=mla_cache, + mla_kv_cache_dtype="auto", + mla_k_scale=None, + has_indexer=False, + index_rope_interleave=False, + ) + + assert_bf16(q_out, rms_norm(q_c, qw), "q_c rmsnorm (no-indexer)") + cache = mla_cache[0, :num_tokens] + assert_bf16(cache[:, :KV_LORA], rms_norm(kv_c, kvw), "MLA kv (no-indexer)") + assert_bf16( + cache[:, KV_LORA:], + rope(k_pe.float(), pos, mla_cos_sin, interleave=True), + "MLA k_pe (no-indexer)", + ) + # Shared layers reuse the previous indexer's top-k: buffer must be untouched. + assert (topk == 7).all(), "topk buffer should be untouched on shared layer" + + +@pytest.mark.parametrize("num_tokens", [1, 4, 17, 512]) +def test_fused_norm_rope_ds_mla(num_tokens: int): + """fp8_ds_mla MLA cache layout (FlashMLA sparse, bf16-query path; SM90/SM100). + + Per-token 656-byte entry: 512 fp8 NoPE (4 per-128 tiles, dynamic float32 + scale) | 4 float32 scales | 64 bf16 (unquantized) RoPE. + """ + torch.manual_seed(5) + dev = "cuda" + max_pos = 8192 + pos = torch.arange(num_tokens, device=dev, dtype=torch.int64) % max_pos + + q_c = torch.randn(num_tokens, Q_LORA, device=dev, dtype=torch.bfloat16) + kv_c = torch.randn(num_tokens, KV_LORA, device=dev, dtype=torch.bfloat16) + k_pe = torch.randn(num_tokens, ROPE_DIM, device=dev, dtype=torch.bfloat16) + qw = torch.randn(Q_LORA, device=dev, dtype=torch.bfloat16) + kvw = torch.randn(KV_LORA, device=dev, dtype=torch.bfloat16) + mla_cos_sin = make_cos_sin(max_pos, ROPE_DIM, dev) + + bs = max_pos + mla_cache = torch.zeros(1, bs, 656, device=dev, dtype=torch.uint8) + slot = torch.arange(num_tokens, device=dev, dtype=torch.int64) + topk = torch.full((num_tokens, 2048), 7, device=dev, dtype=torch.int32) + + q_out = K.fused_norm_rope( + pos, + q_c, + qw, + EPS, + kv_c, + kvw, + EPS, + k_pe, + mla_cos_sin, + None, + None, + None, + EPS, + None, + topk, + slot_mapping=slot, + indexer_k_cache=None, + mla_kv_cache=mla_cache, + mla_kv_cache_dtype="fp8_ds_mla", + mla_k_scale=None, + has_indexer=False, + index_rope_interleave=False, + ) + + assert_bf16(q_out, rms_norm(q_c, qw), "q_c rmsnorm (ds_mla)") + + kv_ref = rms_norm(kv_c, kvw) # [N, 512] fp32 + kpe_ref = rope(k_pe.float(), pos, mla_cos_sin, interleave=True) # [N, 64] + tiles = kv_ref.view(num_tokens, 4, 128) + ref_scale = torch.clamp(tiles.abs().amax(dim=-1) / FP8_MAX, min=1.1754944e-38) + ref_nope = (tiles / ref_scale[..., None]).reshape(num_tokens, KV_LORA).to(FP8) + + cache = mla_cache[0, :num_tokens] # [N, 656] uint8 + nope = cache[:, :KV_LORA].view(FP8) + scales = cache.view(torch.float32)[:, KV_LORA // 4 : KV_LORA // 4 + 4] + rope_off = KV_LORA // 2 + 8 + rope_vals = cache.view(torch.bfloat16)[:, rope_off : rope_off + ROPE_DIM] + + torch.testing.assert_close(scales, ref_scale, rtol=1e-2, atol=1e-6) + assert_fp8(nope, ref_nope, "ds_mla NoPE fp8") + assert_bf16(rope_vals, kpe_ref, "ds_mla RoPE bf16") + # No indexer on this call: top-k buffer must be untouched. + assert (topk == 7).all(), "topk buffer should be untouched (no indexer)" + + +# ── fused_q ────────────────────────────────────────────────────────────────── + + +@pytest.mark.parametrize("num_tokens", [1, 4, 17, 512, 4096]) +@pytest.mark.parametrize("index_interleave", [True, False]) +def test_fused_q(num_tokens: int, index_interleave: bool): + torch.manual_seed(2) + dev = "cuda" + max_pos = 8192 + pos = torch.arange(num_tokens, device=dev, dtype=torch.int64) % max_pos + + q_pe = torch.randn( + num_tokens, NUM_HEADS, ROPE_DIM, device=dev, dtype=torch.bfloat16 + ) + ql_nope = torch.randn( + num_tokens, NUM_HEADS, KV_LORA, device=dev, dtype=torch.bfloat16 + ) + index_q = torch.randn( + num_tokens, INDEX_HEADS, INDEX_HEAD_DIM, device=dev, dtype=torch.bfloat16 + ) + index_w = torch.randn(num_tokens, INDEX_HEADS, device=dev, dtype=torch.float32) + q_scale = torch.tensor([0.37], device=dev, dtype=torch.float32) + softmax_scale = INDEX_HEAD_DIM**-0.5 + head_scale = INDEX_HEADS**-0.5 + q_cos_sin = make_cos_sin(max_pos, ROPE_DIM, dev) # q_pe: interleaved + idx_cos_sin = make_cos_sin(max_pos, ROPE_DIM, dev) + + iq_fp8, iw_out, mqa = K.fused_q( + pos, + q_pe, + q_cos_sin, + index_q, + idx_cos_sin, + ql_nope, + q_scale, + index_w, + softmax_scale, + head_scale, + has_indexer=True, + index_rope_interleave=index_interleave, + ) + + s = q_scale.item() + # MQA query: [ql_nope | q_pe RoPE'd (interleaved)], per-tensor fp8. + mqa_nope_ref = (ql_nope.float() / s).to(FP8) + qpe_ref = rope( + q_pe.float(), + pos.unsqueeze(-1).expand(num_tokens, NUM_HEADS), + q_cos_sin, + interleave=True, + ) + mqa_pe_ref = (qpe_ref / s).to(FP8) + assert_fp8(mqa[:, :, :KV_LORA], mqa_nope_ref, "mqa ql_nope") + assert_fp8(mqa[:, :, KV_LORA:], mqa_pe_ref, "mqa q_pe") + + # Indexer-Q: RoPE + UE8M0 fp8 quant; index weights fold in q-scale. + iq_ref = rope( + index_q.float(), + pos.unsqueeze(-1).expand(num_tokens, INDEX_HEADS), + idx_cos_sin, + interleave=index_interleave, + ) + q_ref, scale_ref = ue8m0_quant(iq_ref) + assert_fp8(iq_fp8, q_ref, "indexer-Q fp8") + iw_ref = index_w * scale_ref * softmax_scale * head_scale + torch.testing.assert_close(iw_out, iw_ref, rtol=1e-3, atol=1e-3) + + +@pytest.mark.parametrize("num_tokens", [1, 17, 512]) +def test_fused_q_no_indexer(num_tokens: int): + torch.manual_seed(3) + dev = "cuda" + max_pos = 8192 + pos = torch.arange(num_tokens, device=dev, dtype=torch.int64) + q_pe = torch.randn( + num_tokens, NUM_HEADS, ROPE_DIM, device=dev, dtype=torch.bfloat16 + ) + ql_nope = torch.randn( + num_tokens, NUM_HEADS, KV_LORA, device=dev, dtype=torch.bfloat16 + ) + q_scale = torch.tensor([0.5], device=dev, dtype=torch.float32) + q_cos_sin = make_cos_sin(max_pos, ROPE_DIM, dev) + + _, _, mqa = K.fused_q( + pos, + q_pe, + q_cos_sin, + None, + None, + ql_nope, + q_scale, + None, + 0.0, + 0.0, + has_indexer=False, + index_rope_interleave=False, + ) + s = q_scale.item() + assert_fp8(mqa[:, :, :KV_LORA], (ql_nope.float() / s).to(FP8), "mqa ql_nope") + qpe_ref = rope( + q_pe.float(), + pos.unsqueeze(-1).expand(num_tokens, NUM_HEADS), + q_cos_sin, + interleave=True, + ) + assert_fp8(mqa[:, :, KV_LORA:], (qpe_ref / s).to(FP8), "mqa q_pe") + + +@pytest.mark.parametrize("num_tokens", [1, 17, 512]) +@pytest.mark.parametrize("has_indexer", [True, False]) +def test_fused_q_bf16_query(num_tokens: int, has_indexer: bool): + """bf16-query path (FlashMLA sparse, SM90/SM100): only the RoPE'd q_pe is + produced (bf16, unquantized); ql_nope is consumed directly by the caller.""" + torch.manual_seed(6) + dev = "cuda" + max_pos = 8192 + pos = torch.arange(num_tokens, device=dev, dtype=torch.int64) % max_pos + + q_pe = torch.randn( + num_tokens, NUM_HEADS, ROPE_DIM, device=dev, dtype=torch.bfloat16 + ) + ql_nope = torch.randn( + num_tokens, NUM_HEADS, KV_LORA, device=dev, dtype=torch.bfloat16 + ) + q_scale = torch.tensor([0.37], device=dev, dtype=torch.float32) + q_cos_sin = make_cos_sin(max_pos, ROPE_DIM, dev) + + index_q = index_w = idx_cos_sin = None + if has_indexer: + index_q = torch.randn( + num_tokens, INDEX_HEADS, INDEX_HEAD_DIM, device=dev, dtype=torch.bfloat16 + ) + index_w = torch.randn(num_tokens, INDEX_HEADS, device=dev, dtype=torch.float32) + idx_cos_sin = make_cos_sin(max_pos, ROPE_DIM, dev) + + iq_fp8, iw_out, q_pe_out = K.fused_q( + pos, + q_pe, + q_cos_sin, + index_q, + idx_cos_sin, + ql_nope, + q_scale, + index_w, + INDEX_HEAD_DIM**-0.5, + INDEX_HEADS**-0.5, + has_indexer=has_indexer, + index_rope_interleave=False, + quantize_mqa=False, + ) + + # MQA query: only the RoPE'd q_pe, bf16, unquantized. + assert q_pe_out.dtype == torch.bfloat16 + assert q_pe_out.shape == (num_tokens, NUM_HEADS, ROPE_DIM) + qpe_ref = rope( + q_pe.float(), + pos.unsqueeze(-1).expand(num_tokens, NUM_HEADS), + q_cos_sin, + interleave=True, + ) + assert_bf16(q_pe_out, qpe_ref, "bf16 q_pe RoPE") + + # Indexer-Q is unchanged on this path (still UE8M0 fp8 + folded weights). + if has_indexer: + assert index_q is not None + iq_ref = rope( + index_q.float(), + pos.unsqueeze(-1).expand(num_tokens, INDEX_HEADS), + idx_cos_sin, + interleave=False, + ) + q_ref, scale_ref = ue8m0_quant(iq_ref) + assert_fp8(iq_fp8, q_ref, "indexer-Q fp8 (bf16-query path)") + iw_ref = index_w * scale_ref * (INDEX_HEAD_DIM**-0.5) * (INDEX_HEADS**-0.5) + torch.testing.assert_close(iw_out, iw_ref, rtol=1e-3, atol=1e-3) + + +# ── fused_eh_norm (MTP) ────────────────────────────────────────────────────── + + +@pytest.mark.parametrize("num_tokens", [1, 4, 17, 512]) +def test_fused_eh_norm(num_tokens: int): + torch.manual_seed(4) + dev = "cuda" + # Mix in a position-0 token to exercise the embeds-zeroing branch. + pos = torch.arange(num_tokens, device=dev, dtype=torch.int64) + pos[0] = 0 + embeds = torch.randn(num_tokens, HIDDEN, device=dev, dtype=torch.bfloat16) + prev = torch.randn(num_tokens, HIDDEN, device=dev, dtype=torch.bfloat16) + ew = torch.randn(HIDDEN, device=dev, dtype=torch.bfloat16) + hw = torch.randn(HIDDEN, device=dev, dtype=torch.bfloat16) + + out = K.fused_eh_norm(pos, embeds, prev, ew, hw, EPS) + + masked = torch.where(pos.unsqueeze(-1) == 0, torch.zeros_like(embeds), embeds) + ref = torch.cat([rms_norm(masked, ew), rms_norm(prev, hw)], dim=-1) + assert out.shape == (num_tokens, 2 * HIDDEN) + assert_bf16(out, ref, "eh_norm") diff --git a/asian_ai/performance/vLLM/tests/kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py b/asian_ai/performance/vLLM/tests/kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py new file mode 100644 index 0000000000000000000000000000000000000000..098f4c81cd9d6d798eb74a8ea9373c89f38919a8 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_fused_deepseek_v4_qnorm_rope_kv_insert.py @@ -0,0 +1,759 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Standalone unit test for the horizontally-fused DeepseekV4-MLA kernel: + + fused_deepseek_v4_qnorm_rope_kv_rope_quant_insert + - Q side: per-head RMSNorm (no weight) + GPT-J RoPE on last 64 dims + - KV side: GPT-J RoPE on last 64 + UE8M0 FP8 quant + paged cache insert + +We compare against: + - PyTorch reference for RMSNorm + GPT-J RoPE on Q + - Existing Triton `quantize_and_insert_k_cache` + round-trip via + `dequantize_and_gather_k_cache` for KV + +The kernel is imported via +`torch.ops._C.fused_deepseek_v4_qnorm_rope_kv_rope_quant_insert`. +""" + +import pytest +import torch + +from tests.kernels.utils import bf16_ulp_distance, fp8_ulp_distance +from vllm.model_executor.layers.quantization.utils.quant_utils import ( + get_fp8_min_max, +) +from vllm.models.deepseek_v4.common.ops import ( + dequantize_and_gather_k_cache, + quantize_and_insert_k_cache, +) +from vllm.platforms import current_platform + +# ── Constants matching the kernel ──────────────────────────────────────────── +HEAD_DIM = 512 +ROPE_DIM = 64 +NOPE_DIM = HEAD_DIM - ROPE_DIM # 448 +QUANT_BLOCK = 64 +# Match the C++ SWA-K encoder: FNUZ on gfx942, OCP elsewhere. +USE_FNUZ = current_platform.is_fp8_fnuz() +_, FP8_MAX = get_fp8_min_max() +# The kernel emits FNUZ-encoded fp8 bytes on gfx942 (rocm_cvt_float_to_fp8_e4m3) +# but stores them into float8_e4m3fn-typed tensors, matching vLLM's ROCm cache +# convention. References must encode under the same scheme and the kernel's +# e4m3fn-typed outputs must be reinterpreted under it before decoding. +FP8_STORE_DTYPE = torch.float8_e4m3fnuz if USE_FNUZ else torch.float8_e4m3fn +HEAD_BYTES = NOPE_DIM + ROPE_DIM * 2 + 8 # 448 + 128 + 8 = 584 + + +# ── PyTorch reference implementations ──────────────────────────────────────── + + +def make_cos_sin_cache(max_pos: int, rope_dim: int, dtype, device): + """Build a cos||sin cache matching DeepseekV4ScalingRotaryEmbedding layout. + cos_sin_cache[pos, :rope_dim/2] = cos(theta), [rope_dim/2:] = sin(theta). + """ + base = 10000.0 + inv_freq = 1.0 / ( + base + ** (torch.arange(0, rope_dim, 2, dtype=torch.float32, device=device) / rope_dim) + ) + t = torch.arange(max_pos, dtype=torch.float32, device=device) + freqs = torch.einsum("i,j -> ij", t, inv_freq) # [max_pos, rope_dim/2] + cache = torch.cat((freqs.cos(), freqs.sin()), dim=-1) # [max_pos, rope_dim] + return cache.to(dtype) + + +def apply_rope_gptj_last_k( + x: torch.Tensor, positions: torch.Tensor, cos_sin_cache: torch.Tensor +) -> torch.Tensor: + """GPT-J-style (interleaved-pair) RoPE on the LAST rope_dim elements. + + x: [..., head_dim] float32 + positions: [num_tokens] int64 (positions[i] corresponds to x[i, ...]) + cos_sin_cache: [max_pos, rope_dim] float (cos|sin layout) + + Returns rotated x (same shape/dtype). + """ + rope_dim = cos_sin_cache.shape[-1] + half = rope_dim // 2 + head_dim = x.shape[-1] + nope_dim = head_dim - rope_dim + + cs = cos_sin_cache[positions.long()].to(torch.float32) + cos = cs[..., :half] + sin = cs[..., half:] + + rope = x[..., nope_dim:].float() + shape = rope.shape + rope = rope.reshape(*shape[:-1], half, 2) + even = rope[..., 0] + odd = rope[..., 1] + + for _ in range(rope.ndim - 3): + cos = cos.unsqueeze(1) + sin = sin.unsqueeze(1) + + # Use addcmul (an FMA) for the 2x2 rotation to mirror the kernel's + # `e*c - o*s` fused form. This keeps the reference close to the kernel, but + # the fp32 reference and the fp32 GPU kernel can still round to bf16 on + # opposite sides of a round-to-nearest tie for a tiny number of elements at + # high positions, so callers compare the RoPE region within 1 bf16 ULP. + new_even = torch.addcmul(-odd * sin, even, cos) + new_odd = torch.addcmul(odd * cos, even, sin) + rope_rotated = torch.stack((new_even, new_odd), dim=-1).reshape(shape) + + out = x.clone().float() + out[..., nope_dim:] = rope_rotated + return out.to(x.dtype) + + +def rmsnorm_no_weight(x: torch.Tensor, eps: float) -> torch.Tensor: + """RMSNorm with no learnable weight, matching + `RMSNorm(head_dim, has_weight=False)`. + + Returns fp32 so callers can chain RoPE without an intermediate bf16 round + (the kernel keeps the whole RMSNorm→RoPE pipeline in fp32 and rounds once + at the final store). + """ + xf = x.float() + variance = xf.pow(2).mean(dim=-1, keepdim=True) + return xf * torch.rsqrt(variance + eps) + + +# ── Dispatch to the CUDA op (skip test cleanly if it isn't built in) ───────── + + +def _op_available() -> bool: + return hasattr(torch.ops._C, "fused_deepseek_v4_qnorm_rope_kv_rope_quant_insert") + + +def _full_cache_fp8_op_available() -> bool: + return hasattr( + torch.ops._C, "fused_deepseek_v4_qnorm_rope_kv_rope_full_cache_fp8_insert" + ) + + +def _full_cache_bf16_op_available() -> bool: + return hasattr( + torch.ops._C, "fused_deepseek_v4_qnorm_rope_kv_rope_full_cache_bf16_insert" + ) + + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or not _op_available(), + reason="CUDA not available or fused DeepseekV4 op not built in", +) + + +def _call_fused( + q_in, q_head_padded, kv, k_cache, slot_mapping, positions, cos_sin_cache, eps, bs +): + return torch.ops._C.fused_deepseek_v4_qnorm_rope_kv_rope_quant_insert( + q_in, + kv, + k_cache, + slot_mapping, + positions, + cos_sin_cache, + q_head_padded, + eps, + bs, + ) + + +def _as_stored_fp8(t: torch.Tensor) -> torch.Tensor: + """Reinterpret a float8_e4m3fn-typed kernel output under the real (FNUZ on + gfx942) encoding the kernel actually wrote, without touching the bytes.""" + return t.contiguous().view(torch.uint8).view(FP8_STORE_DTYPE) + + +def _dequant_cache(k_cache_2d, num_tokens, num_blocks, block_size): + """Round-trip a [num_blocks, block_size*HEAD_BYTES] K-cache back to bf16.""" + device = k_cache_2d.device + out = torch.zeros(1, num_tokens, HEAD_DIM, dtype=torch.bfloat16, device=device) + seq_lens = torch.tensor([num_tokens], dtype=torch.int32, device=device) + block_table = torch.arange(num_blocks, dtype=torch.int32, device=device).unsqueeze( + 0 + ) + k_cache_3d = k_cache_2d.view(num_blocks, block_size, HEAD_BYTES) + dequantize_and_gather_k_cache( + out, + k_cache_3d, + seq_lens, + None, + block_table, + block_size, + offset=0, + use_fnuz=USE_FNUZ, + ) + return out[0, :num_tokens] + + +def _assert_kv_cache_parity( + k_cache_fused, k_cache_ref, num_tokens, num_blocks, block_size +): + """Assert the fused and reference K-caches agree after decoding. + + The NoPE region is deterministic UE8M0 FP8, so its round-trip must be + bit-identical. The RoPE region is stored as bf16 after an fp32 rotation: + the GPU kernel and the PyTorch reference can fall on opposite sides of a + round-to-nearest tie and differ by at most one bf16 ULP. (Spot checks show + the kernel value is the correctly-rounded one; the fp32 torch reference is + the one that lands on the wrong side near a midpoint.) Allow <=1 ULP there. + """ + rec_fused = _dequant_cache(k_cache_fused, num_tokens, num_blocks, block_size) + rec_ref = _dequant_cache(k_cache_ref, num_tokens, num_blocks, block_size) + torch.testing.assert_close( + rec_fused[:, :NOPE_DIM], rec_ref[:, :NOPE_DIM], rtol=0, atol=0 + ) + max_ulp = int( + bf16_ulp_distance(rec_fused[:, NOPE_DIM:], rec_ref[:, NOPE_DIM:]).max().item() + ) + assert max_ulp <= 1, f"RoPE bf16 region differs by {max_ulp} ULP (>1)" + + +# ── Test 1: Q path numerical parity ────────────────────────────────────────── + + +@pytest.mark.parametrize("num_tokens", [1, 4, 17, 64, 2048]) +@pytest.mark.parametrize( + "n_heads,padded_heads", + [ + # Each supported padded_heads instantiation: padded (n_heads < + # padded_heads) and unpadded (n_heads == padded_heads). + (1, 8), + (8, 8), + (8, 16), + (16, 16), + (16, 32), + (32, 32), + (8, 64), + (64, 64), + (64, 128), + ], +) +def test_q_path_matches_reference(num_tokens: int, n_heads: int, padded_heads: int): + torch.manual_seed(0) + device = "cuda" + dtype = torch.bfloat16 + eps = 1e-6 + max_pos = 4096 + + q = torch.randn(num_tokens, n_heads, HEAD_DIM, dtype=dtype, device=device) + positions = torch.arange(num_tokens, dtype=torch.int64, device=device) + cos_sin_cache = make_cos_sin_cache(max_pos, ROPE_DIM, torch.float32, device) + + # Reference: RMSNorm (no weight) per head, then GPT-J RoPE on last 64. + # Keep the chain in fp32 (rmsnorm_no_weight returns fp32) and round to + # bf16 once at the end, matching the kernel. + q_ref = rmsnorm_no_weight(q, eps) + q_ref = apply_rope_gptj_last_k(q_ref, positions, cos_sin_cache).to(dtype) + + # Fused call with dummy KV tensors (KV branch will write slot_mapping=-1 → noop). + num_blocks = 2 + bs = 16 + kv = torch.zeros(num_tokens, HEAD_DIM, dtype=dtype, device=device) + k_cache = torch.zeros( + num_blocks, bs, HEAD_BYTES, dtype=torch.uint8, device=device + ).view(num_blocks, -1) + slot_mapping = torch.full((num_tokens,), -1, dtype=torch.int64, device=device) + q_out = _call_fused( + q, padded_heads, kv, k_cache, slot_mapping, positions, cos_sin_cache, eps, bs + ) + + torch.testing.assert_close(q_out[:, :n_heads], q_ref, rtol=1e-2, atol=1e-2) + if n_heads < padded_heads: + pad_region = q_out[:, n_heads:padded_heads] + assert pad_region.abs().max().item() == 0.0, ( + "padded head slots must be exact zero" + ) + + +# ── Test 2: KV path round-trip byte/value parity ───────────────────────────── + + +def _ue8m0_per_block_scales(kv_roped_nope_f32: torch.Tensor, qblock: int): + """Return per-token per-block max scale (used to bound FP8 error).""" + n_tok, nope = kv_roped_nope_f32.shape + n_blocks = nope // qblock + blocks = kv_roped_nope_f32.view(n_tok, n_blocks, qblock) + absmax = blocks.abs().amax(dim=-1).clamp(min=1e-4) + raw = absmax / FP8_MAX + exponent = torch.ceil(torch.log2(raw)) + return torch.pow(2.0, exponent) # [n_tok, n_blocks] + + +@pytest.mark.parametrize("num_tokens", [1, 4, 17, 64, 2048]) +@pytest.mark.parametrize("block_size", [16, 64]) +def test_kv_path_matches_reference(num_tokens: int, block_size: int): + torch.manual_seed(1) + device = "cuda" + dtype = torch.bfloat16 + eps = 1e-6 + max_pos = 4096 + + kv = torch.randn(num_tokens, HEAD_DIM, dtype=dtype, device=device) + positions = torch.arange(num_tokens, dtype=torch.int64, device=device) + cos_sin_cache = make_cos_sin_cache(max_pos, ROPE_DIM, torch.float32, device) + + num_blocks = (num_tokens + block_size - 1) // block_size + 1 + slot_mapping = torch.arange(num_tokens, dtype=torch.int64, device=device) + + # ── Reference path: RoPE on kv, then existing Triton quant+insert ────── + kv_ref = apply_rope_gptj_last_k(kv, positions, cos_sin_cache) + k_cache_ref = torch.zeros( + num_blocks, block_size * HEAD_BYTES, dtype=torch.uint8, device=device + ) + quantize_and_insert_k_cache( + kv_ref, k_cache_ref, slot_mapping, block_size=block_size, use_fnuz=USE_FNUZ + ) + + # ── Fused path (dummy q, padded to FlashMLA's min head count 64) ─────── + k_cache_fused = torch.zeros_like(k_cache_ref) + q_dummy = torch.zeros(num_tokens, 1, HEAD_DIM, dtype=dtype, device=device) + _ = _call_fused( + q_dummy, + 64, + kv, + k_cache_fused, + slot_mapping, + positions, + cos_sin_cache, + eps, + block_size, + ) + + # ── Round-trip compare via dequant+gather ────────────────────────────── + def _dequant(k_cache_2d): + num_reqs = 1 + max_blocks = num_blocks + out = torch.zeros( + num_reqs, num_tokens, HEAD_DIM, dtype=torch.bfloat16, device=device + ) + seq_lens = torch.tensor([num_tokens], dtype=torch.int32, device=device) + block_table = torch.arange( + max_blocks, dtype=torch.int32, device=device + ).unsqueeze(0) + # gather_lens arg is None (use seq_lens) + k_cache_3d = k_cache_2d.view(num_blocks, block_size, HEAD_BYTES) + dequantize_and_gather_k_cache( + out, + k_cache_3d, + seq_lens, + None, + block_table, + block_size, + offset=0, + use_fnuz=USE_FNUZ, + ) + return out[0, :num_tokens] + + recovered_ref = _dequant(k_cache_ref) + recovered_fused = _dequant(k_cache_fused) + + # NoPE: per-block UE8M0 FP8 error bound (half-ULP at max = 16 * scale). + scales = _ue8m0_per_block_scales(kv_ref[:, :NOPE_DIM].float(), QUANT_BLOCK) + for t in range(num_tokens): + max_allowed = 16.0 * scales[t].max().item() + diff_ref = ( + (recovered_ref[t, :NOPE_DIM] - kv_ref[t, :NOPE_DIM]).abs().max().item() + ) + diff_fused = ( + (recovered_fused[t, :NOPE_DIM] - kv_ref[t, :NOPE_DIM]).abs().max().item() + ) + assert diff_ref <= max_allowed, ( + f"ref NoPE token {t} diff {diff_ref} > {max_allowed}" + ) + assert diff_fused <= max_allowed, ( + f"fused NoPE token {t} diff {diff_fused} > {max_allowed}" + ) + + # Strong parity: NoPE FP8 round-trip bit-identical, RoPE bf16 within 1 ULP. + _assert_kv_cache_parity( + k_cache_fused, k_cache_ref, num_tokens, num_blocks, block_size + ) + + +# ── Test 2b: DP padding (slot_mapping shorter than q/kv) ───────────────────── + + +@pytest.mark.parametrize("num_tokens", [4, 17, 2048]) +@pytest.mark.parametrize("pad", [1, 5]) +@pytest.mark.parametrize("block_size", [16, 64]) +def test_kv_path_with_dp_padding(num_tokens: int, pad: int, block_size: int): + """slot_mapping.size(0) < q.size(0): the kernel must skip padded + tokens in the KV branch while still running Q-norm+RoPE on all rows.""" + torch.manual_seed(3) + device = "cuda" + dtype = torch.bfloat16 + eps = 1e-6 + max_pos = 4096 + total = num_tokens + pad + + kv = torch.randn(total, HEAD_DIM, dtype=dtype, device=device) + positions = torch.arange(total, dtype=torch.int64, device=device) + cos_sin_cache = make_cos_sin_cache(max_pos, ROPE_DIM, torch.float32, device) + + num_blocks = (num_tokens + block_size - 1) // block_size + 1 + slot_mapping = torch.arange(num_tokens, dtype=torch.int64, device=device) + + # Reference: only the first num_tokens kv rows get inserted. + kv_ref = apply_rope_gptj_last_k( + kv[:num_tokens], positions[:num_tokens], cos_sin_cache + ) + k_cache_ref = torch.zeros( + num_blocks, block_size * HEAD_BYTES, dtype=torch.uint8, device=device + ) + quantize_and_insert_k_cache( + kv_ref, k_cache_ref, slot_mapping, block_size=block_size, use_fnuz=USE_FNUZ + ) + + # Fused: pass full-sized q/kv/positions, shorter slot_mapping. + q_dummy = torch.zeros(total, 1, HEAD_DIM, dtype=dtype, device=device) + k_cache_fused = torch.zeros_like(k_cache_ref) + _ = _call_fused( + q_dummy, + 64, + kv, + k_cache_fused, + slot_mapping, + positions, + cos_sin_cache, + eps, + block_size, + ) + + _assert_kv_cache_parity( + k_cache_fused, k_cache_ref, num_tokens, num_blocks, block_size + ) + + +# ── Test 3: combined single-call Q + KV parity ─────────────────────────────── + + +@pytest.mark.parametrize("num_tokens", [1, 4, 17, 2048]) +@pytest.mark.parametrize( + "n_heads,padded_heads", + [ + # Each supported padded_heads instantiation: padded (n_heads < + # padded_heads) and unpadded (n_heads == padded_heads). + (1, 8), + (8, 8), + (8, 16), + (16, 16), + (16, 32), + (32, 32), + (8, 64), + (64, 64), + (64, 128), + ], +) +@pytest.mark.parametrize("block_size", [16, 64]) +def test_combined_q_and_kv( + num_tokens: int, n_heads: int, padded_heads: int, block_size: int +): + torch.manual_seed(2) + device = "cuda" + dtype = torch.bfloat16 + eps = 1e-6 + max_pos = 4096 + + q = torch.randn(num_tokens, n_heads, HEAD_DIM, dtype=dtype, device=device) + kv = torch.randn(num_tokens, HEAD_DIM, dtype=dtype, device=device) + positions = torch.arange(num_tokens, dtype=torch.int64, device=device) + cos_sin_cache = make_cos_sin_cache(max_pos, ROPE_DIM, torch.float32, device) + + num_blocks = (num_tokens + block_size - 1) // block_size + 1 + slot_mapping = torch.arange(num_tokens, dtype=torch.int64, device=device) + + # Reference. + q_ref = rmsnorm_no_weight(q, eps) + q_ref = apply_rope_gptj_last_k(q_ref, positions, cos_sin_cache).to(dtype) + kv_ref = apply_rope_gptj_last_k(kv, positions, cos_sin_cache) + k_cache_ref = torch.zeros( + num_blocks, block_size * HEAD_BYTES, dtype=torch.uint8, device=device + ) + quantize_and_insert_k_cache( + kv_ref, k_cache_ref, slot_mapping, block_size=block_size, use_fnuz=USE_FNUZ + ) + + # Fused single call. + k_cache_fused = torch.zeros_like(k_cache_ref) + q_out = _call_fused( + q, + padded_heads, + kv, + k_cache_fused, + slot_mapping, + positions, + cos_sin_cache, + eps, + block_size, + ) + + torch.testing.assert_close(q_out[:, :n_heads], q_ref, rtol=1e-2, atol=1e-2) + if n_heads < padded_heads: + pad_region = q_out[:, n_heads:padded_heads] + assert pad_region.abs().max().item() == 0.0, ( + "padded head slots must be exact zero" + ) + _assert_kv_cache_parity( + k_cache_fused, k_cache_ref, num_tokens, num_blocks, block_size + ) + + +# ── Full-cache (FlashInfer) path parity ────────────────────────────────────── + + +def _call_full_cache_fp8_fused( + q, + kv, + q_fp8, + k_cache, + slot_mapping, + positions, + cos_sin_cache, + fp8_scale, + q_fp8_scale_inv, + eps, + bs, +): + torch.ops._C.fused_deepseek_v4_qnorm_rope_kv_rope_full_cache_fp8_insert( + q, + kv, + q_fp8, + k_cache, + slot_mapping, + positions.long(), + cos_sin_cache, + fp8_scale, + q_fp8_scale_inv, + eps, + bs, + ) + + +def _call_full_cache_bf16_fused( + q, + kv, + k_cache, + slot_mapping, + positions, + cos_sin_cache, + eps, + bs, +): + torch.ops._C.fused_deepseek_v4_qnorm_rope_kv_rope_full_cache_bf16_insert( + q, + kv, + k_cache, + slot_mapping, + positions.long(), + cos_sin_cache, + eps, + bs, + ) + + +def _fp8_full_cache_reference( + q, + kv, + k_cache, + q_fp8, + slot_mapping, + positions, + cos_sin_cache, + eps, + block_size, + fp8_scale, + q_fp8_scale_inv, +): + q_ref = rmsnorm_no_weight(q, eps) + q_ref = apply_rope_gptj_last_k(q_ref, positions, cos_sin_cache) + q_fp8.copy_( + torch.clamp(q_ref.float() * q_fp8_scale_inv, -FP8_MAX, FP8_MAX).to( + FP8_STORE_DTYPE + ) + ) + + kv_ref = apply_rope_gptj_last_k(kv, positions, cos_sin_cache) + valid = slot_mapping >= 0 + slots = slot_mapping[valid] + block_idx = slots // block_size + pos_in_block = slots % block_size + k_cache[block_idx, pos_in_block] = torch.clamp( + kv_ref[valid].float() / fp8_scale, -FP8_MAX, FP8_MAX + ).to(FP8_STORE_DTYPE) + + +def _bf16_full_cache_reference( + q, + kv, + k_cache, + slot_mapping, + positions, + cos_sin_cache, + eps, + block_size, +): + q_ref = rmsnorm_no_weight(q, eps) + # Kernel keeps RMSNorm+RoPE in fp32 and rounds to bf16 once at the store. + q_ref = apply_rope_gptj_last_k(q_ref, positions, cos_sin_cache).to(q.dtype) + + kv_ref = apply_rope_gptj_last_k(kv, positions, cos_sin_cache) + valid = slot_mapping >= 0 + slots = slot_mapping[valid] + block_idx = slots // block_size + pos_in_block = slots % block_size + k_cache[block_idx, pos_in_block] = kv_ref[valid] + return q_ref + + +@pytest.mark.skipif( + not _full_cache_fp8_op_available(), + reason="full-cache per-tensor FP8 DeepseekV4 op not built in", +) +@pytest.mark.parametrize("num_tokens", [4, 17]) +@pytest.mark.parametrize("n_heads", [8, 17]) +@pytest.mark.parametrize("positions_dtype", [torch.int32, torch.int64]) +def test_full_cache_per_tensor_fp8_matches_reference( + num_tokens: int, + n_heads: int, + positions_dtype: torch.dtype, +): + torch.manual_seed(4) + device = "cuda" + dtype = torch.bfloat16 + eps = 1e-6 + block_size = 16 + max_pos = 4096 + + q = torch.randn(num_tokens, n_heads, HEAD_DIM, dtype=dtype, device=device) + kv = torch.randn(num_tokens, HEAD_DIM, dtype=dtype, device=device) + positions = torch.arange(num_tokens, dtype=positions_dtype, device=device) + cos_sin_cache = make_cos_sin_cache(max_pos, ROPE_DIM, torch.float32, device) + + num_blocks = (num_tokens + block_size - 1) // block_size + 1 + slot_mapping = torch.arange(num_tokens, dtype=torch.int64, device=device) + fp8_scale = torch.tensor([1.0], dtype=torch.float32, device=device) + q_fp8_scale_inv = torch.tensor([1.0], dtype=torch.float32, device=device) + + # References are encoded under the scheme the kernel actually writes + # (FNUZ on gfx942); the kernel's own outputs must stay float8_e4m3fn-typed + # because the op asserts that dtype. + q_fp8_ref = torch.empty_like(q, dtype=FP8_STORE_DTYPE) + q_fp8_fused = torch.empty_like(q, dtype=torch.float8_e4m3fn) + k_cache_ref = torch.zeros( + num_blocks, block_size, HEAD_DIM, dtype=FP8_STORE_DTYPE, device=device + ) + k_cache_fused = torch.zeros( + num_blocks, block_size, HEAD_DIM, dtype=torch.float8_e4m3fn, device=device + ) + + _fp8_full_cache_reference( + q, + kv, + k_cache_ref, + q_fp8_ref, + slot_mapping, + positions, + cos_sin_cache, + eps, + block_size, + fp8_scale, + q_fp8_scale_inv, + ) + _call_full_cache_fp8_fused( + q.clone(), + kv, + q_fp8_fused, + k_cache_fused, + slot_mapping, + positions, + cos_sin_cache, + fp8_scale, + q_fp8_scale_inv, + eps, + block_size, + ) + + # Q is RMSNorm(no-weight)+RoPE in fp32 before fp8 quant; the RMSNorm + # reduction and RoPE rotation can land the kernel and the torch reference on + # opposite sides of an fp8 round-to-nearest tie, so allow <=1 fp8 ULP. + q_fused = _as_stored_fp8(q_fp8_fused) + q_max_ulp = int(fp8_ulp_distance(q_fused, q_fp8_ref).max().item()) + assert q_max_ulp <= 1, f"Q fp8 differs by {q_max_ulp} ULP (>1)" + + # K-cache NoPE region [0, NOPE_DIM) is a deterministic per-tensor fp8 quant + # of the (un-rotated) KV input, so it must be bit-identical. The RoPE region + # [NOPE_DIM, HEAD_DIM) is rotated in fp32 and may differ by <=1 fp8 ULP. + k_fused = _as_stored_fp8(k_cache_fused) + torch.testing.assert_close( + k_fused[..., :NOPE_DIM].float(), + k_cache_ref[..., :NOPE_DIM].float(), + rtol=0, + atol=0, + ) + k_max_ulp = int( + fp8_ulp_distance(k_fused[..., NOPE_DIM:], k_cache_ref[..., NOPE_DIM:]) + .max() + .item() + ) + assert k_max_ulp <= 1, f"K-cache RoPE fp8 differs by {k_max_ulp} ULP (>1)" + + +@pytest.mark.skipif( + not _full_cache_bf16_op_available(), + reason="full-cache BF16 DeepseekV4 op not built in", +) +@pytest.mark.parametrize("num_tokens", [4, 17]) +@pytest.mark.parametrize("n_heads", [8, 17]) +@pytest.mark.parametrize("positions_dtype", [torch.int32, torch.int64]) +def test_full_cache_bf16_matches_reference( + num_tokens: int, + n_heads: int, + positions_dtype: torch.dtype, +): + torch.manual_seed(5) + device = "cuda" + dtype = torch.bfloat16 + eps = 1e-6 + block_size = 16 + max_pos = 4096 + + q = torch.randn(num_tokens, n_heads, HEAD_DIM, dtype=dtype, device=device) + kv = torch.randn(num_tokens, HEAD_DIM, dtype=dtype, device=device) + positions = torch.arange(num_tokens, dtype=positions_dtype, device=device) + cos_sin_cache = make_cos_sin_cache(max_pos, ROPE_DIM, torch.float32, device) + + num_blocks = (num_tokens + block_size - 1) // block_size + 1 + slot_mapping = torch.arange(num_tokens, dtype=torch.int64, device=device) + + q_fused = q.clone() + k_cache_ref = torch.zeros( + num_blocks, block_size, HEAD_DIM, dtype=torch.bfloat16, device=device + ) + k_cache_fused = torch.zeros_like(k_cache_ref) + q_ref = _bf16_full_cache_reference( + q, + kv, + k_cache_ref, + slot_mapping, + positions, + cos_sin_cache, + eps, + block_size, + ) + _call_full_cache_bf16_fused( + q_fused, + kv, + k_cache_fused, + slot_mapping, + positions, + cos_sin_cache, + eps, + block_size, + ) + + torch.testing.assert_close(q_fused, q_ref, rtol=1e-2, atol=1e-2) + torch.testing.assert_close(k_cache_fused, k_cache_ref, rtol=0, atol=0) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_fused_gdn_post_conv.py b/asian_ai/performance/vLLM/tests/kernels/test_fused_gdn_post_conv.py new file mode 100644 index 0000000000000000000000000000000000000000..74623f50d547244e76b304cda75319490c8d10ef --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_fused_gdn_post_conv.py @@ -0,0 +1,209 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for fused_gdn_prefill_post_conv kernel. + +Verifies that the fused kernel matches the reference: + split → rearrange → contiguous → l2norm → gating +""" + +import pytest +import torch +import torch.nn.functional as F + +from vllm.model_executor.layers.fla.ops.fused_gdn_prefill_post_conv import ( + fused_post_conv_prep, +) + + +def reference_post_conv( + conv_output: torch.Tensor, + a: torch.Tensor, + b: torch.Tensor, + A_log: torch.Tensor, + dt_bias: torch.Tensor, + H: int, + K: int, + V: int, + apply_l2norm: bool = True, + output_g_exp: bool = False, +): + """Reference implementation using individual ops.""" + L = conv_output.shape[0] + HV = A_log.shape[0] + + # Split + q_flat, k_flat, v_flat = torch.split(conv_output, [H * K, H * K, HV * V], dim=-1) + + # Rearrange + contiguous + q = q_flat.view(L, H, K).contiguous() + k = k_flat.view(L, H, K).contiguous() + v = v_flat.view(L, HV, V).contiguous() + + # L2 norm + if apply_l2norm: + q = F.normalize(q.float(), p=2, dim=-1, eps=1e-6).to(conv_output.dtype) + k = F.normalize(k.float(), p=2, dim=-1, eps=1e-6).to(conv_output.dtype) + + # Gating + x = a.float() + dt_bias.float() + sp = F.softplus(x, beta=1.0, threshold=20.0) + g = -torch.exp(A_log.float()) * sp + + if output_g_exp: + g = torch.exp(g) + + beta_out = torch.sigmoid(b.float()) + + return q, k, v, g, beta_out + + +# Qwen3.5-35B config: H=16, HV=32, K=128, V=128 +# Qwen3.5-397B config: H=16, HV=64, K=128, V=128 +@pytest.mark.parametrize( + "H, HV, K, V", + [ + (16, 32, 128, 128), # 35B + (16, 64, 128, 128), # 397B + (4, 8, 64, 64), # small + ], +) +@pytest.mark.parametrize("L", [1, 16, 128, 512, 2048]) +@pytest.mark.parametrize("apply_l2norm", [True, False]) +@pytest.mark.parametrize("output_g_exp", [True, False]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +def test_fused_post_conv_correctness(H, HV, K, V, L, apply_l2norm, output_g_exp, dtype): + """Test fused kernel matches reference for all configs.""" + torch.manual_seed(42) + device = "cuda" + qkv_dim = 2 * H * K + HV * V + + conv_output = torch.randn(L, qkv_dim, dtype=dtype, device=device) + a = torch.randn(L, HV, dtype=dtype, device=device) + b = torch.randn(L, HV, dtype=dtype, device=device) + A_log = torch.randn(HV, dtype=torch.float32, device=device) - 2.0 + dt_bias = torch.randn(HV, dtype=torch.float32, device=device) * 0.1 + + # Reference + ref_q, ref_k, ref_v, ref_g, ref_beta = reference_post_conv( + conv_output, + a, + b, + A_log, + dt_bias, + H, + K, + V, + apply_l2norm, + output_g_exp, + ) + + # Fused kernel + fused_q, fused_k, fused_v, fused_g, fused_beta = fused_post_conv_prep( + conv_output, + a, + b, + A_log, + dt_bias, + num_k_heads=H, + head_k_dim=K, + head_v_dim=V, + apply_l2norm=apply_l2norm, + output_g_exp=output_g_exp, + ) + + # Check shapes + assert fused_q.shape == (L, H, K), f"q shape: {fused_q.shape}" + assert fused_k.shape == (L, H, K), f"k shape: {fused_k.shape}" + assert fused_v.shape == (L, HV, V), f"v shape: {fused_v.shape}" + assert fused_g.shape == (L, HV), f"g shape: {fused_g.shape}" + assert fused_beta.shape == (L, HV), f"beta shape: {fused_beta.shape}" + + # Check dtypes + assert fused_q.dtype == dtype + assert fused_k.dtype == dtype + assert fused_v.dtype == dtype + assert fused_g.dtype == torch.float32 + assert fused_beta.dtype == torch.float32 + + # Check contiguity + assert fused_q.is_contiguous() + assert fused_k.is_contiguous() + assert fused_v.is_contiguous() + + # Check values + atol_qkv = 1e-2 if apply_l2norm else 1e-3 + rtol_qkv = 1e-2 if apply_l2norm else 1e-3 + + torch.testing.assert_close(fused_q, ref_q, atol=atol_qkv, rtol=rtol_qkv) + torch.testing.assert_close(fused_k, ref_k, atol=atol_qkv, rtol=rtol_qkv) + torch.testing.assert_close(fused_v, ref_v, atol=1e-3, rtol=1e-3) + torch.testing.assert_close(fused_g, ref_g, atol=1e-4, rtol=1e-4) + torch.testing.assert_close(fused_beta, ref_beta, atol=1e-4, rtol=1e-4) + + +@pytest.mark.parametrize("L", [1, 64, 256]) +def test_fused_post_conv_sanity(L): + """Sanity checks: no NaN, unit-norm q/k, beta in (0,1).""" + torch.manual_seed(0) + device = "cuda" + H, HV, K, V = 16, 32, 128, 128 + qkv_dim = 2 * H * K + HV * V + + conv_output = torch.randn(L, qkv_dim, dtype=torch.bfloat16, device=device) + a = torch.randn(L, HV, dtype=torch.bfloat16, device=device) + b = torch.randn(L, HV, dtype=torch.bfloat16, device=device) + A_log = torch.randn(HV, dtype=torch.float32, device=device) - 2.0 + dt_bias = torch.randn(HV, dtype=torch.float32, device=device) + + q, k, v, g, beta = fused_post_conv_prep( + conv_output, + a, + b, + A_log, + dt_bias, + num_k_heads=H, + head_k_dim=K, + head_v_dim=V, + ) + + # Basic sanity + assert not torch.isnan(q).any(), "NaN in q" + assert not torch.isnan(k).any(), "NaN in k" + assert not torch.isnan(v).any(), "NaN in v" + assert not torch.isnan(g).any(), "NaN in g" + assert not torch.isnan(beta).any(), "NaN in beta" + + # L2 norm check: each head vector should have unit norm + q_norms = torch.norm(q.float(), dim=-1) + k_norms = torch.norm(k.float(), dim=-1) + torch.testing.assert_close(q_norms, torch.ones_like(q_norms), atol=1e-3, rtol=1e-3) + torch.testing.assert_close(k_norms, torch.ones_like(k_norms), atol=1e-3, rtol=1e-3) + + # Beta should be in (0, 1) + assert (beta >= 0).all() and (beta <= 1).all(), "beta out of range" + + +def test_fused_post_conv_l0(): + """Test L=0 edge case.""" + device = "cuda" + H, HV, K, V = 16, 32, 128, 128 + qkv_dim = 2 * H * K + HV * V + + conv_output = torch.empty(0, qkv_dim, dtype=torch.bfloat16, device=device) + a = torch.empty(0, HV, dtype=torch.bfloat16, device=device) + b = torch.empty(0, HV, dtype=torch.bfloat16, device=device) + A_log = torch.randn(HV, dtype=torch.float32, device=device) + dt_bias = torch.randn(HV, dtype=torch.float32, device=device) + + q, k, v, g, beta = fused_post_conv_prep( + conv_output, + a, + b, + A_log, + dt_bias, + num_k_heads=H, + head_k_dim=K, + head_v_dim=V, + ) + assert q.shape == (0, H, K) + assert g.shape == (0, HV) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_fused_indexer_q_rope_quant.py b/asian_ai/performance/vLLM/tests/kernels/test_fused_indexer_q_rope_quant.py new file mode 100644 index 0000000000000000000000000000000000000000..923e1cd6d5c6ca8a3727b3cea7d43d7598d70480 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_fused_indexer_q_rope_quant.py @@ -0,0 +1,196 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit test for fused_indexer_q_rope_quant. + +Compares the fused Triton kernel against the unfused reference flow used by +the DeepseekV4 indexer in model_tracking: + q_rot = ops.rotary_embedding(positions, q, None, head_dim, cos_sin_cache, + is_neox_style=False, + rope_dim_offset=head_dim - rope_dim) + q_fp8, q_scale = per_token_group_quant_fp8(q_rot, head_dim, use_ue8m0=True) + weights_out = weights * q_scale * softmax_scale * head_scale + +Expects bit-exact equality on both q_fp8 and weights_out. +""" + +import contextlib +from unittest import mock + +import pytest +import torch + +from vllm import _custom_ops as ops +from vllm.model_executor.layers.quantization.utils.fp8_utils import ( + per_token_group_quant_fp8, +) +from vllm.models.deepseek_v4.common.ops import fused_indexer_q_rope_quant +from vllm.utils.import_utils import has_cutedsl + +HEAD_DIM = 128 +ROPE_DIM = 64 +N_HEAD = 64 +MAX_POS = 4096 + + +def quantize_to_mxfp4( + x: torch.Tensor, +) -> tuple[torch.Tensor, torch.Tensor]: + """Reference MXFP4 quantization. + + Args: + x: [..., head_dim] where head_dim is divisible by 32 + Returns: + packed: [..., head_dim//2] uint8 2 E2M1 nibbles/byte, low nibble = even index + scales: [..., head_dim//32] uint8 1 ue8m0 byte + """ + MXFP4_BLOCK_SIZE = 32 + orig_shape = x.shape + head_dim = orig_shape[-1] + n_blocks = head_dim // MXFP4_BLOCK_SIZE + + x_f32 = x.float().reshape(-1, n_blocks, MXFP4_BLOCK_SIZE) + + # Per-block ue8m0 scale: 2^ceil(log2(amax / 6.0)), stored as byte = exp + 127 + # 6 * 2^-126 is from https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/inference/kernel.py#L163 + amax = x_f32.abs().amax(dim=-1, keepdim=True).clamp(min=6 * (2**-126)) + log2_ratio = (amax * (1.0 / 6.0)).log2().ceil().clamp(-127.0, 127.0) + scale = log2_ratio.exp2() + ue8m0 = (log2_ratio + 127.0).to(torch.uint8) # [*, n_blocks] + + # E2M1 round-to-nearest-even: midpoints round to the even code. + # E2M1 values: [0.00, 0.50, 1.00, 1.50, 2.00, 3.00, 4.00, 6.00] + # boundaries: [ 0.25, 0.75, 1.25, 1.75, 2.50, 3.50, 5.00] + x_scaled = (x_f32 / scale).clamp(-6.0, 6.0) + abs_x = x_scaled.abs() + code = torch.zeros_like(abs_x, dtype=torch.int32) + code = torch.where(abs_x > 0.25, 1, code) + code = torch.where(abs_x >= 0.75, 2, code) + code = torch.where(abs_x > 1.25, 3, code) + code = torch.where(abs_x >= 1.75, 4, code) + code = torch.where(abs_x > 2.5, 5, code) + code = torch.where(abs_x >= 3.5, 6, code) + code = torch.where(abs_x > 5.0, 7, code) + sign = ((x_scaled.view(torch.int32) >> 31) & 1).to(torch.uint8) + nibble = code.to(torch.uint8) | (sign << 3) + + # Pack: even-index element → low nibble, odd-index → high nibble + nibble_flat = nibble.reshape(-1, head_dim) + packed = (nibble_flat[:, 0::2] | (nibble_flat[:, 1::2] << 4)).contiguous() + packed = packed.reshape(*orig_shape[:-1], head_dim // 2) + + scales = ue8m0.view(*orig_shape[:-1], n_blocks) + return packed, scales + + +def _reference( + positions: torch.Tensor, + q: torch.Tensor, + cos_sin_cache: torch.Tensor, + weights: torch.Tensor, + softmax_scale: float, + head_scale: float, + use_fp4: bool = False, +) -> tuple[torch.Tensor, torch.Tensor]: + q_rot = q.clone() + ops.rotary_embedding( + positions, + q_rot, + None, + HEAD_DIM, + cos_sin_cache, + False, # is_neox_style=False → GPT-J interleaved + HEAD_DIM - ROPE_DIM, # rope_dim_offset → rotate the tail + False, + ) + + if use_fp4: + q_packed, ue8m0 = quantize_to_mxfp4(q_rot.view(-1, N_HEAD, HEAD_DIM)) + # Pack 4 ue8m0 bytes into 1 int32 + q_scale = ue8m0.view(torch.int32).squeeze(-1) + # FP4 path: q_scale stays separate (cannot be folded into a per-token scalar) + weights_out = weights.to(torch.float32) * softmax_scale * head_scale + return (q_packed, q_scale), weights_out + + else: + q_fp8, q_scale = per_token_group_quant_fp8( + q_rot.view(-1, HEAD_DIM).contiguous(), + HEAD_DIM, + use_ue8m0=True, + ) + q_fp8 = q_fp8.view(-1, N_HEAD, HEAD_DIM) + q_scale = q_scale.view(-1, N_HEAD) + + weights_out = weights.to(torch.float32) * q_scale * softmax_scale * head_scale + return q_fp8, weights_out + + +@pytest.mark.parametrize("num_tokens", [1, 7, 32, 257, 1023]) +@pytest.mark.parametrize("cache_dtype", [torch.float32, torch.bfloat16]) +@pytest.mark.parametrize("use_fp4", [False, True]) +@pytest.mark.parametrize("use_cutedsl", [False, True]) +@torch.inference_mode() +def test_fused_indexer_q_rope_quant_matches_unfused( + num_tokens, cache_dtype, use_fp4, use_cutedsl +): + if use_cutedsl and not has_cutedsl(): + pytest.skip("cutedsl (cutlass) not installed") + + device = "cuda" + torch.manual_seed(0) + + q = torch.randn(num_tokens, N_HEAD, HEAD_DIM, dtype=torch.bfloat16, device=device) + positions = torch.randint( + 0, MAX_POS, (num_tokens,), dtype=torch.int64, device=device + ) + cos_sin_cache = torch.randn(MAX_POS, ROPE_DIM, dtype=cache_dtype, device=device) + weights = torch.randn(num_tokens, N_HEAD, dtype=torch.bfloat16, device=device) + softmax_scale = HEAD_DIM**-0.5 + head_scale = N_HEAD**-0.5 + + q_quant_ref, weights_ref = _reference( + positions, q, cos_sin_cache, weights, softmax_scale, head_scale, use_fp4 + ) + # use_cutedsl=False: force the triton path even when cutedsl is installed + # by patching the dispatcher's has_cutedsl() binding to return False. + cutedsl_patch = ( + mock.patch( + "vllm.models.deepseek_v4.common.ops.fused_indexer_q.has_cutedsl", + return_value=False, + ) + if not use_cutedsl + else contextlib.nullcontext() + ) + with cutedsl_patch: + q_quant_fused, weights_fused = fused_indexer_q_rope_quant( + positions, + q.clone(), + cos_sin_cache, + weights, + softmax_scale, + head_scale, + use_fp4, + ) + + if use_fp4: + q_quant_ref, q_scale_ref = q_quant_ref + q_quant_fused, q_scale_fused = q_quant_fused + + assert torch.equal(q_scale_ref, q_scale_fused), ( + f"q_scale mismatch: " + f"{(q_scale_ref != q_scale_fused).sum().item()} " + f"/ {q_scale_ref.numel()} bytes differ" + ) + + # fp8 tensors aren't directly comparable via torch.equal — reinterpret as int8. + ref_bits = q_quant_ref.view(torch.int8) + fused_bits = q_quant_fused.view(torch.int8) + assert torch.equal(ref_bits, fused_bits), ( + f"q_quant_fused mismatch: " + f"{(ref_bits != fused_bits).sum().item()} / {ref_bits.numel()} bytes differ" + ) + + assert weights_fused.dtype == torch.float32 + assert torch.equal(weights_ref, weights_fused), ( + f"weights mismatch: max abs diff " + f"{(weights_ref - weights_fused).abs().max().item()}" + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_fused_inv_rope_fp8_quant.py b/asian_ai/performance/vLLM/tests/kernels/test_fused_inv_rope_fp8_quant.py new file mode 100644 index 0000000000000000000000000000000000000000..abf9b52a3b898d9afbff92876b75c06bb37208da --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_fused_inv_rope_fp8_quant.py @@ -0,0 +1,909 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Unit tests for the fused inverse RoPE + block-scaled FP8 quantization kernel. + +Tests compare the fused kernel against a reference implementation built from +the existing separate operations (inverse RoPE via rotate_neox + FP8 quant +via per_token_group_quant_fp8). + +The reference faithfully reproduces the exact flow in +deepseek_v4/attention.py:295-310: + 1. Apply inverse RoPE (NeoX style, last rope_dim=64 dims of each head) + 2. Reshape [T, H, head_dim] -> [T, G, D] + 3. Transpose+flatten to [G*T, D], quantize, reshape back + 4. Return o_fp8 and o_scale with strides (D, T*D, 1) and (S, T*S, 1) + (non-contiguous [T, G, ...] view backed by contiguous [G, T, ...] memory) + +Usage: + pytest tests/kernels/test_fused_inv_rope_fp8_quant.py -v +""" + +import pytest +import torch + +from vllm.models.deepseek_v4.common.ops import fused_inv_rope_fp8_quant + +# -- Default dimensions matching DeepSeek V3/V4 -------------------------- +HEAD_DIM = 512 +NOPE_DIM = 448 +ROPE_DIM = 64 +QUANT_GROUP_SIZE = 128 +FP8_MAX = 448.0 # torch.finfo(torch.float8_e4m3fn).max +FP8_DTYPE = torch.float8_e4m3fn +EPS = 1e-10 + + +# ========================================================================= +# Helpers +# ========================================================================= + + +def assert_dequant_close( + fp8_a: torch.Tensor, + scale_a: torch.Tensor, + fp8_b: torch.Tensor, + scale_b: torch.Tensor, + msg: str = "", +): + """Compare two FP8-quantized tensors via their dequantized values. + + Uses cosine-similarity-based diff (same as deep_gemm calc_diff). + Both fused and reference paths rotate in fp32 using an fp32 + cos_sin_cache, so differences are only fp32 ordering ULPs that can + occasionally shift FP8 values at quantization boundaries. + """ + S = scale_a.shape[-1] + shape = fp8_a.shape + + dq_a = fp8_a.float() * scale_a.unsqueeze(-1).expand( + *shape[:-1], S, QUANT_GROUP_SIZE + ).reshape(shape) + dq_b = fp8_b.float() * scale_b.unsqueeze(-1).expand( + *shape[:-1], S, QUANT_GROUP_SIZE + ).reshape(shape) + + # Cosine diff: 1 - cos_sim (0 = identical, higher = worse) + dq_a_flat = dq_a.flatten().float() + dq_b_flat = dq_b.flatten().float() + cos_sim = torch.nn.functional.cosine_similarity( + dq_a_flat.unsqueeze(0), dq_b_flat.unsqueeze(0) + ).item() + diff = 1.0 - cos_sim + + assert diff < 1e-4, f"Dequant diff too large: {diff:.8f} (expected < 1e-4). {msg}" + + +def rotate_gptj(x: torch.Tensor) -> torch.Tensor: + """GPT-J style rotation: interleaved pairs, negate-swap. + + Matches vllm/model_executor/layers/rotary_embedding/common.py:23-27. + DeepseekV4 uses is_neox_style=False, so this is the correct rotation. + """ + x1 = x[..., ::2] + x2 = x[..., 1::2] + x = torch.stack((-x2, x1), dim=-1) + return x.flatten(-2) + + +def make_cos_sin_cache( + max_pos: int, + rope_dim: int = ROPE_DIM, + dtype: torch.dtype = torch.float32, + device: str = "cuda", +) -> torch.Tensor: + """Create a synthetic cos_sin_cache matching the layout used by + DeepseekV4ScalingRotaryEmbedding._compute_cos_sin_cache. + + Shape: [max_pos, rope_dim] where first half is cos, second half is sin. + The fused kernel requires fp32; callers can override dtype if passing + the cache into the bf16-only paths. + """ + half = rope_dim // 2 + # Use random but bounded frequencies so cos/sin are well-behaved + inv_freq = 1.0 / ( + 10000.0 ** (torch.arange(0, half, device=device, dtype=torch.float32) / half) + ) + t = torch.arange(max_pos, device=device, dtype=torch.float32) + freqs = torch.outer(t, inv_freq) # [max_pos, half] + cos = freqs.cos() + sin = freqs.sin() + cache = torch.cat((cos, sin), dim=-1) # [max_pos, rope_dim] + return cache.to(dtype) + + +def reference_inv_rope( + o: torch.Tensor, + positions: torch.Tensor, + cos_sin_cache: torch.Tensor, + nope_dim: int = NOPE_DIM, + rope_dim: int = ROPE_DIM, +) -> torch.Tensor: + """Apply inverse RoPE to the last rope_dim dimensions of each head. + + Matches the GPT-J inverse rotation in pos_encoding_kernels.cu, which + promotes the cache to fp32 and performs the rotation in fp32. The + result is cast back to the input dtype. + + Args: + o: [T, H, head_dim] bf16 + positions: [T] int64 + cos_sin_cache: [max_pos, rope_dim] fp32 + + Returns: + o with inverse RoPE applied on the rope portion (bf16). + """ + assert cos_sin_cache.dtype == torch.float32 + cos_sin = cos_sin_cache[positions] # [T, rope_dim] fp32 + half = rope_dim // 2 + cos = cos_sin[:, :half] + sin = cos_sin[:, half:] + + # GPT-J style: repeat_interleave (not repeat) to match interleaved pairs + cos = cos.repeat_interleave(2, dim=-1).unsqueeze(1) + sin = sin.repeat_interleave(2, dim=-1).unsqueeze(1) + sin = -sin # inverse + + o_pass = o[..., :nope_dim] + o_rot_f32 = o[..., nope_dim:].float() + o_rot_f32 = o_rot_f32 * cos + rotate_gptj(o_rot_f32) * sin + o_rot = o_rot_f32.to(o.dtype) + + return torch.cat([o_pass, o_rot], dim=-1) + + +def _ref_ue8m0_quant_block(x_f32: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + """Per-block UE8M0 FP8 quantization in pure float32. + + Matches the Triton kernel logic exactly: + absmax -> 2^ceil(log2(absmax / fp8_max)) -> clamp(x / scale) -> fp8 + + Args: + x_f32: [..., quant_group_size] float32 — one or more 128-element blocks. + + Returns: + x_fp8: same shape, float8_e4m3fn + scales: [...] float32, one scale per block + """ + assert x_f32.shape[-1] == QUANT_GROUP_SIZE + absmax = x_f32.abs().amax(dim=-1, keepdim=True).clamp(min=EPS) + scale_raw = absmax * (1.0 / FP8_MAX) + scale = torch.exp2(torch.ceil(torch.log2(scale_raw))) + x_scaled = (x_f32 / scale).clamp(-FP8_MAX, FP8_MAX) + x_fp8 = x_scaled.to(FP8_DTYPE) + return x_fp8, scale.squeeze(-1) + + +def reference_inv_rope_fp8_quant( + o: torch.Tensor, + positions: torch.Tensor, + cos_sin_cache: torch.Tensor, + n_groups: int, + heads_per_group: int, + nope_dim: int = NOPE_DIM, + rope_dim: int = ROPE_DIM, + quant_group_size: int = QUANT_GROUP_SIZE, +) -> tuple[torch.Tensor, torch.Tensor]: + """Full reference: inverse RoPE in fp32 + UE8M0 FP8 quant in fp32. + + Mimics the Triton kernel's precision path exactly: + Load bf16 -> cast to fp32 -> apply inverse RoPE with fp32 cos/sin -> + UE8M0 quant in fp32 -> write fp8 + scale + + Returns: + o_fp8: [T, G, D] FP8 with strides (D, T*D, 1) + o_scale: [T, G, S] FP32 with strides (S, T*S, 1) + """ + assert cos_sin_cache.dtype == torch.float32 + T, _H, head_dim = o.shape + d = heads_per_group * head_dim + S = d // quant_group_size + half_rope = rope_dim // 2 + chunks_per_head = head_dim // quant_group_size + + # Reshape [T, H, head_dim] -> [T, G, heads_per_group, head_dim] + o_4d = o.view(T, n_groups, heads_per_group, head_dim) + + # Lookup cos/sin directly in fp32 + cos_sin = cos_sin_cache[positions] # [T, rope_dim] fp32 + cos = cos_sin[:, :half_rope] # [T, half_rope] fp32 + sin = cos_sin[:, half_rope:] # [T, half_rope] fp32 + + # Allocate outputs in [G, T, ...] contiguous layout + fp8_buf = torch.empty(n_groups, T, d, dtype=FP8_DTYPE, device=o.device) + scale_buf = torch.empty(n_groups, T, S, dtype=torch.float32, device=o.device) + + # Process each quant block, matching the Triton kernel's per-program logic + for g in range(n_groups): + for qb in range(S): + head_in_group = qb // chunks_per_head + chunk_in_head = qb % chunks_per_head + offset = chunk_in_head * quant_group_size + + # Load 128 bf16 elements and promote to fp32 for rotation+quant + block = o_4d[:, g, head_in_group, offset : offset + quant_group_size] + x = block.float() + + # Apply inverse RoPE in fp32 if this is the last chunk + # GPT-J style: interleaved pairs (even=x, odd=y) + if chunk_in_head == chunks_per_head - 1: + rope_start = nope_dim % quant_group_size # 64 + rope_region = x[:, rope_start:].clone() + x_vals = rope_region[:, ::2] + y_vals = rope_region[:, 1::2] + x_new = x_vals * cos + y_vals * sin + y_new = y_vals * cos - x_vals * sin + x = x.clone() + x[:, rope_start::2] = x_new + x[:, rope_start + 1 :: 2] = y_new + + # UE8M0 quant in fp32 + x_fp8, scale = _ref_ue8m0_quant_block(x) + + # Write to [G, T, D] contiguous memory + fp8_buf[g, :, qb * quant_group_size : (qb + 1) * quant_group_size] = x_fp8 + scale_buf[g, :, qb] = scale + + # Return transposed views + return fp8_buf.transpose(0, 1), scale_buf.transpose(0, 1) + + +# ========================================================================= +# Tests +# ========================================================================= + + +@pytest.mark.parametrize("num_tokens", [1, 7, 32, 128]) +@pytest.mark.parametrize( + "num_heads,n_groups", + [(64, 8), (32, 4), (128, 8)], + ids=["H64_G8", "H32_G4", "H128_G8"], +) +@pytest.mark.parametrize("seed", [0, 42]) +@torch.inference_mode() +def test_correctness(num_tokens, num_heads, n_groups, seed): + """Compare fused kernel against reference for FP8 values and scales.""" + torch.manual_seed(seed) + + heads_per_group = num_heads // n_groups + max_pos = 4096 + device = "cuda" + + # Create inputs + o = torch.randn( + num_tokens, num_heads, HEAD_DIM, device=device, dtype=torch.bfloat16 + ) + positions = torch.randint( + 0, max_pos, (num_tokens,), device=device, dtype=torch.long + ) + cos_sin_cache = make_cos_sin_cache( + max_pos, ROPE_DIM, dtype=torch.float32, device=device + ) + + # Reference + ref_fp8, ref_scale = reference_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + + # Fused kernel + fused_fp8, fused_scale = fused_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + + # Check shapes + d = heads_per_group * HEAD_DIM + S = d // QUANT_GROUP_SIZE + assert ref_fp8.shape == (num_tokens, n_groups, d) + assert fused_fp8.shape == (num_tokens, n_groups, d) + assert ref_scale.shape == (num_tokens, n_groups, S) + assert fused_scale.shape == (num_tokens, n_groups, S) + + # Scales: exact match (both use identical UE8M0 algorithm) + # Scales may differ by one UE8M0 step (factor of 2) if fp32 rotation + # ordering shifts absmax across a power-of-2 boundary. Check ratio is + # close to 1. + scale_ratio = fused_scale / ref_scale.clamp(min=1e-30) + assert scale_ratio.max() <= 2.0 and scale_ratio.min() >= 0.5, ( + f"Scale ratio out of [0.5, 2]: min={scale_ratio.min():.4f} " + f"max={scale_ratio.max():.4f}" + ) + + # Compare via dequant (Triton vs PyTorch fp32 may differ by ULPs) + assert_dequant_close(ref_fp8, ref_scale, fused_fp8, fused_scale) + + +@pytest.mark.parametrize("num_tokens", [1, 7, 32, 128]) +@pytest.mark.parametrize( + "num_heads,n_groups", + [(64, 8), (128, 8)], + ids=["H64_G8", "H128_G8"], +) +@torch.inference_mode() +def test_output_strides(num_tokens, num_heads, n_groups): + """Verify fused output layout: + - FP8: logical [T, G, D] backed by contiguous [G, T, D]. + - Scale: MN-major TMA-aligned (column-major: T-stride=1). + """ + + heads_per_group = num_heads // n_groups + max_pos = 4096 + device = "cuda" + + o = torch.randn( + num_tokens, num_heads, HEAD_DIM, device=device, dtype=torch.bfloat16 + ) + positions = torch.randint( + 0, max_pos, (num_tokens,), device=device, dtype=torch.long + ) + cos_sin_cache = make_cos_sin_cache(max_pos, device=device) + + fused_fp8, fused_scale = fused_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + + # FP8: logical [T, G, D] backed by [G, T, D] row-major + d = heads_per_group * HEAD_DIM + expected_fp8_stride = (d, num_tokens * d, 1) + assert fused_fp8.stride() == expected_fp8_stride, ( + f"FP8 stride mismatch: got {fused_fp8.stride()}, expected {expected_fp8_stride}" + ) + + # Scale: MN-major TMA-aligned layout. After fp8_einsum permutes + # [T,G,S] -> [G,T,S], T-dim should have stride 1. + # Our output is [T,G,S] = transpose of [G,T,S]. + # So fused_scale.permute(1,0,2) should have T-stride=1. + perm = fused_scale.permute(1, 0, 2) # [G, T, S] + assert perm.stride(1) == 1 or num_tokens == 1, ( + f"Scale T-stride (after permute to [G,T,S]) should be 1, got {perm.stride(1)}" + ) + + +@pytest.mark.parametrize("num_tokens", [1, 7, 32, 128]) +@torch.inference_mode() +def test_per_group_contiguity(num_tokens): + """FP8 per-group slices must be contiguous. Scale per-group slices + are column-major (T-stride=1) — not row-major contiguous, which is + correct for TMA loads.""" + num_heads, n_groups = 64, 8 + heads_per_group = num_heads // n_groups + max_pos = 4096 + device = "cuda" + + o = torch.randn( + num_tokens, num_heads, HEAD_DIM, device=device, dtype=torch.bfloat16 + ) + positions = torch.randint( + 0, max_pos, (num_tokens,), device=device, dtype=torch.long + ) + cos_sin_cache = make_cos_sin_cache(max_pos, device=device) + + fused_fp8, fused_scale = fused_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + + for g in range(n_groups): + fp8_slice = fused_fp8[:, g, :] + assert fp8_slice.is_contiguous(), ( + f"o_fp8[:, {g}, :] is not contiguous: " + f"shape={list(fp8_slice.shape)}, stride={list(fp8_slice.stride())}" + ) + + +@torch.inference_mode() +def test_scales_are_power_of_two(): + """Verify all scales are exact powers of 2 (UE8M0 property).""" + num_tokens, num_heads, n_groups = 32, 64, 8 + heads_per_group = num_heads // n_groups + max_pos = 4096 + device = "cuda" + + o = torch.randn( + num_tokens, num_heads, HEAD_DIM, device=device, dtype=torch.bfloat16 + ) + positions = torch.randint( + 0, max_pos, (num_tokens,), device=device, dtype=torch.long + ) + cos_sin_cache = make_cos_sin_cache(max_pos, device=device) + + _, fused_scale = fused_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + + # log2 of a power-of-two is an exact integer + log2_scales = torch.log2(fused_scale) + residual = (log2_scales - log2_scales.round()).abs() + assert residual.max() < 1e-5, ( + f"Not all scales are powers of 2: max log2 residual = {residual.max().item()}" + ) + + +@torch.inference_mode() +def test_nope_dims_unchanged(): + """Nope dimensions (first 448 per head) should only be quantized, + not rotated. Verify by dequantizing and comparing against + quantize-only reference (no RoPE).""" + num_tokens, num_heads, n_groups = 16, 64, 8 + heads_per_group = num_heads // n_groups + max_pos = 4096 + device = "cuda" + torch.manual_seed(0) + + o = torch.randn( + num_tokens, num_heads, HEAD_DIM, device=device, dtype=torch.bfloat16 + ) + positions = torch.randint( + 0, max_pos, (num_tokens,), device=device, dtype=torch.long + ) + cos_sin_cache = make_cos_sin_cache(max_pos, device=device) + + # Fused kernel result + fused_fp8, fused_scale = fused_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + + # Reference: quantize without RoPE (identity rotation) + # Create a zero-sin cache so RoPE is identity + zero_cache = torch.zeros_like(cos_sin_cache) + half = ROPE_DIM // 2 + zero_cache[:, :half] = 1.0 # cos = 1 + # sin = 0 (already zero) + + norope_fp8, norope_scale = fused_inv_rope_fp8_quant( + o.clone(), + positions, + zero_cache, + n_groups, + heads_per_group, + ) + + # Extract nope quant blocks only (first 3 of every 4 blocks per head) + chunks_per_head = HEAD_DIM // QUANT_GROUP_SIZE # 4 + + for h in range(heads_per_group): + for c in range(chunks_per_head - 1): # skip last chunk (has rope) + qb = h * chunks_per_head + c + start = qb * QUANT_GROUP_SIZE + end = start + QUANT_GROUP_SIZE + + fused_nope = fused_fp8[:, :, start:end].view(torch.uint8) + norope_nope = norope_fp8[:, :, start:end].view(torch.uint8) + assert torch.equal(fused_nope, norope_nope), ( + f"Nope block (head={h}, chunk={c}) differs between " + f"fused and no-rope reference" + ) + + fused_s = fused_scale[:, :, qb] + norope_s = norope_scale[:, :, qb] + assert torch.equal(fused_s, norope_s), ( + f"Nope scale (head={h}, chunk={c}) differs" + ) + + +@torch.inference_mode() +def test_single_token(): + """Edge case: single token.""" + num_tokens, num_heads, n_groups = 1, 64, 8 + heads_per_group = num_heads // n_groups + max_pos = 4096 + device = "cuda" + + o = torch.randn( + num_tokens, num_heads, HEAD_DIM, device=device, dtype=torch.bfloat16 + ) + positions = torch.tensor([42], device=device, dtype=torch.long) + cos_sin_cache = make_cos_sin_cache(max_pos, device=device) + + ref_fp8, ref_scale = reference_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + fused_fp8, fused_scale = fused_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + + assert_dequant_close(ref_fp8, ref_scale, fused_fp8, fused_scale) + + +@torch.inference_mode() +def test_zero_positions(): + """Edge case: all positions are 0.""" + num_tokens, num_heads, n_groups = 16, 64, 8 + heads_per_group = num_heads // n_groups + max_pos = 4096 + device = "cuda" + + o = torch.randn( + num_tokens, num_heads, HEAD_DIM, device=device, dtype=torch.bfloat16 + ) + positions = torch.zeros(num_tokens, device=device, dtype=torch.long) + cos_sin_cache = make_cos_sin_cache(max_pos, device=device) + + ref_fp8, ref_scale = reference_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + fused_fp8, fused_scale = fused_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + + assert_dequant_close(ref_fp8, ref_scale, fused_fp8, fused_scale) + + +@torch.inference_mode() +def test_large_values(): + """Edge case: values near FP8 saturation to test clamping.""" + num_tokens, num_heads, n_groups = 8, 64, 8 + heads_per_group = num_heads // n_groups + max_pos = 4096 + device = "cuda" + + # Create inputs with large values that will saturate FP8 + o = torch.randn( + num_tokens, num_heads, HEAD_DIM, device=device, dtype=torch.bfloat16 + ) + o = o * 1000.0 # scale up to force saturation + positions = torch.randint( + 0, max_pos, (num_tokens,), device=device, dtype=torch.long + ) + cos_sin_cache = make_cos_sin_cache(max_pos, device=device) + + ref_fp8, ref_scale = reference_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + fused_fp8, fused_scale = fused_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + + assert_dequant_close(ref_fp8, ref_scale, fused_fp8, fused_scale) + + +@torch.inference_mode() +def test_dequant_numerical_accuracy(): + """Verify dequantized values are close to the original (after inv RoPE).""" + num_tokens, num_heads, n_groups = 32, 64, 8 + heads_per_group = num_heads // n_groups + max_pos = 4096 + device = "cuda" + torch.manual_seed(0) + + o = torch.randn( + num_tokens, num_heads, HEAD_DIM, device=device, dtype=torch.bfloat16 + ) + positions = torch.randint( + 0, max_pos, (num_tokens,), device=device, dtype=torch.long + ) + cos_sin_cache = make_cos_sin_cache(max_pos, device=device) + + # Get the post-inv-RoPE values (ground truth before quantization) + o_after_rope = reference_inv_rope(o.clone(), positions, cos_sin_cache) + d = heads_per_group * HEAD_DIM + o_after_rope = o_after_rope.view(num_tokens, n_groups, d) + + # Get fused quantized output + fused_fp8, fused_scale = fused_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + + # Dequantize: broadcast scale [T, G, S] to [T, G, D] via repeat + S = d // QUANT_GROUP_SIZE + scale_expanded = ( + fused_scale.unsqueeze(-1) + .expand(num_tokens, n_groups, S, QUANT_GROUP_SIZE) + .reshape(num_tokens, n_groups, d) + ) + dequant = fused_fp8.float() * scale_expanded + + # Check relative error. + # FP8 e4m3 with UE8M0 (power-of-two scales that round UP) quantizes more + # coarsely than optimal scaling. Both paths rotate in fp32, so the bulk + # of the error comes from UE8M0 quantization itself (~10-12% typical). + o_gt = o_after_rope.transpose(0, 1).contiguous().transpose(0, 1) + dequant_contig = dequant.transpose(0, 1).contiguous().transpose(0, 1) + + abs_err = (dequant_contig.float() - o_gt.float()).abs() + rel_err = abs_err / (o_gt.float().abs().clamp(min=1e-6)) + mean_rel_err = rel_err.mean().item() + + assert mean_rel_err < 0.15, ( + f"Mean relative error too high: {mean_rel_err:.4f} (expected < 0.15)" + ) + + +def _unfused_inv_rope_fp8_quant( + o: torch.Tensor, + positions: torch.Tensor, + cos_sin_cache: torch.Tensor, + n_groups: int, + heads_per_group: int, + nope_dim: int = NOPE_DIM, + rope_dim: int = ROPE_DIM, +) -> tuple[torch.Tensor, torch.Tensor]: + """Unfused path matching deepseek_v4/attention.py:295-310. + + Uses the production CUDA RoPE kernel + per_token_group_quant_fp8. + """ + from vllm import _custom_ops as ops + from vllm.model_executor.layers.quantization.utils.fp8_utils import ( + per_token_group_quant_fp8, + ) + + head_dim = o.shape[-1] + rope_dim_offset = head_dim - rope_dim + + # Step 1: In-place CUDA RoPE (same as production) + ops.rotary_embedding( + positions, + o, + None, + head_dim, + cos_sin_cache, + False, # is_neox=False for DeepseekV4 (GPT-J style) + rope_dim_offset=rope_dim_offset, + inverse=True, + ) + + # Step 2: Reshape + quant + reshape (same as production) + T = o.shape[0] + d = heads_per_group * head_dim + o = o.view(T, n_groups, -1) + o_flat = o.transpose(0, 1).contiguous().reshape(-1, d) + o_fp8, o_scale = per_token_group_quant_fp8( + o_flat, + group_size=QUANT_GROUP_SIZE, + use_ue8m0=True, + ) + o_fp8 = o_fp8.view(n_groups, T, d).transpose(0, 1) + o_scale = o_scale.view(n_groups, T, -1).transpose(0, 1) + return o_fp8, o_scale + + +# ========================================================================= +# End-to-end test including fp8_einsum +# ========================================================================= + + +@pytest.mark.parametrize("num_tokens", [1, 7, 32, 128, 1024]) +@pytest.mark.parametrize( + "num_heads,n_groups", + [(64, 8)], + ids=["H64_G8"], +) +@torch.inference_mode() +def test_einsum_end_to_end(num_tokens, num_heads, n_groups): + """End-to-end: fused inv_rope+quant → fp8_einsum must match + unfused CUDA_rope+quant → fp8_einsum bitwise. + + This catches stride/layout bugs that only manifest when the einsum + kernel actually consumes the quantized activations. + """ + from deep_gemm.utils.math import ceil_div + + from vllm.utils.deep_gemm import ( + fp8_einsum, + per_block_cast_to_fp8, + transform_sf_into_required_layout, + ) + + heads_per_group = num_heads // n_groups + d = heads_per_group * HEAD_DIM + o_lora_rank = 1024 + max_pos = 4096 + device = "cuda" + torch.manual_seed(0) + + o = torch.randn( + num_tokens, num_heads, HEAD_DIM, device=device, dtype=torch.bfloat16 + ) + positions = torch.randint( + 0, max_pos, (num_tokens,), device=device, dtype=torch.long + ) + cos_sin_cache = make_cos_sin_cache(max_pos, device=device) + + # -- Weight quantization (shared between both paths) -- + w = torch.randn(n_groups, o_lora_rank, d, device=device, dtype=torch.bfloat16) + w_fp8 = torch.empty_like(w, dtype=torch.float8_e4m3fn) + w_scale = torch.empty( + n_groups, + ceil_div(o_lora_rank, 128), + ceil_div(d, 128), + device=device, + dtype=torch.float32, + ) + for g in range(n_groups): + w_fp8[g], w_scale[g] = per_block_cast_to_fp8(w[g], use_ue8m0=True) + + recipe = (1, 1, 128) + w_scale_t = transform_sf_into_required_layout( + sf=w_scale, + mn=o_lora_rank, + k=d, + recipe=(1, 128, 128), + num_groups=n_groups, + is_sfa=False, + ) + + # -- UNFUSED path -- + ref_fp8, ref_scale = _unfused_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + z_ref = torch.empty( + num_tokens, n_groups, o_lora_rank, device=device, dtype=torch.bfloat16 + ) + fp8_einsum( + "bhr,hdr->bhd", (ref_fp8, ref_scale), (w_fp8, w_scale_t), z_ref, recipe=recipe + ) + + # -- FUSED path -- + fused_fp8, fused_scale = fused_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + z_fused = torch.empty( + num_tokens, n_groups, o_lora_rank, device=device, dtype=torch.bfloat16 + ) + fp8_einsum( + "bhr,hdr->bhd", + (fused_fp8, fused_scale), + (w_fp8, w_scale_t), + z_fused, + recipe=recipe, + ) + + # -- Checks -- + # Einsum output: Triton and CUDA both rotate in fp32 now, so diffs + # come from fp32 ordering and UE8M0 boundary shifts only. + # Use relative diff (same metric as test_fp8_einsum.py). + from deep_gemm.testing import calc_diff + + z_diff = calc_diff(z_fused, z_ref) + assert z_diff < 0.01, ( + f"Einsum output diff too large: {z_diff:.6f} (expected < 0.01)" + ) + + +@pytest.mark.parametrize("num_tokens", [1, 32, 256]) +@torch.inference_mode() +def test_with_real_deepseek_v4_rope(num_tokens, default_vllm_config): + """Test with real DeepseekV4ScalingRotaryEmbedding (GPT-J style, + mscale=0, YaRN scaling) matching the production config.""" + + num_heads = 64 + n_groups = 8 + heads_per_group = num_heads // n_groups + device = "cuda" + torch.manual_seed(0) + + # Build YaRN-scaled cos_sin_cache matching real DeepSeek V3/V4 config + # (mscale=0 → mscale=1.0, so no magnitude scaling) + from vllm.model_executor.layers.rotary_embedding.common import ( + yarn_find_correction_range, + yarn_linear_ramp_mask, + ) + + scaling_factor = 16 + base = 10000.0 + max_pos = 65536 + beta_fast, beta_slow = 32, 1 + + pos_freqs = base ** ( + torch.arange(0, ROPE_DIM, 2, dtype=torch.float32, device=device) / ROPE_DIM + ) + inv_freq_extra = 1.0 / pos_freqs + inv_freq_interp = 1.0 / (scaling_factor * pos_freqs) + low, high = yarn_find_correction_range( + beta_fast, beta_slow, ROPE_DIM, base, max_pos + ) + mask = 1 - yarn_linear_ramp_mask(low, high, ROPE_DIM // 2, dtype=torch.float32).to( + device + ) + inv_freq = inv_freq_interp * (1 - mask) + inv_freq_extra * mask + t = torch.arange(max_pos * scaling_factor, device=device, dtype=torch.float32) + freqs = torch.outer(t, inv_freq) + # mscale=0 → yarn_get_mscale returns 1.0 + cos_sin_cache = torch.cat([freqs.cos(), freqs.sin()], dim=-1) # fp32 + + o = torch.randn( + num_tokens, num_heads, HEAD_DIM, device=device, dtype=torch.bfloat16 + ) + positions = torch.randint(0, 4096, (num_tokens,), device=device, dtype=torch.long) + + # UNFUSED: CUDA RoPE with is_neox=False (GPT-J) + from vllm import _custom_ops as ops + from vllm.model_executor.layers.quantization.utils.fp8_utils import ( + per_token_group_quant_fp8, + ) + + o_unfused = o.clone() + ops.rotary_embedding( + positions, + o_unfused, + None, + HEAD_DIM, + cos_sin_cache, + False, # is_neox=False (GPT-J style) + rope_dim_offset=NOPE_DIM, + inverse=True, + ) + d = heads_per_group * HEAD_DIM + T = num_tokens + o_unfused = o_unfused.view(T, n_groups, d) + o_flat = o_unfused.transpose(0, 1).contiguous().reshape(-1, d) + ref_fp8, ref_scale = per_token_group_quant_fp8( + o_flat, + group_size=QUANT_GROUP_SIZE, + use_ue8m0=True, + ) + ref_fp8 = ref_fp8.view(n_groups, T, d).transpose(0, 1) + ref_scale = ref_scale.view(n_groups, T, -1).transpose(0, 1) + + # FUSED: use the real YaRN-scaled cos_sin_cache + fused_fp8, fused_scale = fused_inv_rope_fp8_quant( + o.clone(), + positions, + cos_sin_cache, + n_groups, + heads_per_group, + ) + + # Scales must match exactly (same UE8M0 algorithm) + # Compare via dequant (Triton bf16 rotation may differ from CUDA by 1 ULP) + assert_dequant_close( + ref_fp8, ref_scale, fused_fp8, fused_scale, msg="Real DeepSeek V4 rope" + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py b/asian_ai/performance/vLLM/tests/kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py new file mode 100644 index 0000000000000000000000000000000000000000..0ff64805f11eccd1e57fc738859019323f25ab19 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py @@ -0,0 +1,376 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Unit test for the horizontally-fused MiniMax-M3 attention pre-processing +kernel: + + fused_minimax_m3_qknorm_rope_kv_insert + - q / k / index_q / index_k: Gemma RMSNorm + partial NeoX RoPE (in place) + - sparse (insert) mode: scatter k/v into the paged bf16 KV cache and the + index key into the index cache by its own slot mapping. + +Reference: PyTorch Gemma RMSNorm with the same dtype materialization boundary +as the unfused path, followed by vLLM CUDA rotary_embedding-style NeoX RoPE. +""" + +import pytest +import torch + +import vllm._custom_ops as ops + +HEAD_DIM = 128 +ROTARY_DIM = 64 + + +def _op_available() -> bool: + return hasattr(torch.ops._C, "fused_minimax_m3_qknorm_rope_kv_insert") + + +pytestmark = pytest.mark.skipif( + not torch.cuda.is_available() or not _op_available(), + reason="CUDA not available or fused MiniMax-M3 op not built in", +) + + +def make_cos_sin_cache(max_pos, rotary_dim, base, dtype, device): + inv_freq = 1.0 / ( + base + ** ( + torch.arange(0, rotary_dim, 2, dtype=torch.float32, device=device) + / rotary_dim + ) + ) + t = torch.arange(max_pos, dtype=torch.float32, device=device) + freqs = torch.einsum("i,j->ij", t, inv_freq) # [max_pos, rotary_dim/2] + cache = torch.cat((freqs.cos(), freqs.sin()), dim=-1) # [max_pos, rotary_dim] + return cache.to(dtype) + + +def gemma_rmsnorm(x, weight, eps): + """x: [..., 128]; weight: [128]. Returns original dtype.""" + xf = x.float() + var = xf.pow(2).mean(dim=-1, keepdim=True) + out = xf * torch.rsqrt(var + eps) + out = out * (1.0 + weight.float()) + return out.to(x.dtype) + + +def apply_rope_neox_partial(x, positions, cos_sin_cache, rotary_dim): + """NeoX-style RoPE on the leading rotary_dim dims; rest pass through. + + x: [num_tokens, num_heads, head_dim] + cos_sin_cache: [max_pos, rotary_dim] (cos||sin), read as float (matches the + kernel, which loads the bf16 cache and converts to fp32). + """ + half = rotary_dim // 2 + cs = cos_sin_cache[positions].float() # [num_tokens, rotary_dim] + cos = cs[..., :half].unsqueeze(1) # [nt, 1, half] + sin = cs[..., half:].unsqueeze(1) + + rot = x[..., :rotary_dim].float() + x1 = rot[..., :half] + x2 = rot[..., half:] + o1 = x1 * cos - x2 * sin + o2 = x2 * cos + x1 * sin + out = x.clone() + out[..., :half] = o1 + out[..., half:rotary_dim] = o2 + return out.to(x.dtype) + + +def norm_rope_ref(x, weight, positions, cos_sin_cache, eps): + """[nt, nheads, 128] -> Gemma norm + neox partial rope.""" + normed = gemma_rmsnorm(x, weight, eps) + roped = apply_rope_neox_partial(normed, positions, cos_sin_cache, ROTARY_DIM) + return roped + + +# ── Test 1: dense mode (norm+rope only, no index, no insert) ───────────────── + + +@pytest.mark.parametrize("num_tokens", [1, 7, 64, 513]) +@pytest.mark.parametrize("num_heads,num_kv_heads", [(8, 2), (16, 4), (64, 4)]) +def test_dense_norm_rope(num_tokens, num_heads, num_kv_heads): + torch.manual_seed(0) + device, dtype, eps = "cuda", torch.bfloat16, 1e-6 + base, max_pos = 5_000_000.0, 4096 + + q_w = torch.randn(HEAD_DIM, dtype=dtype, device=device) * 0.1 + k_w = torch.randn(HEAD_DIM, dtype=dtype, device=device) * 0.1 + cos_sin = make_cos_sin_cache(max_pos, ROTARY_DIM, base, dtype, device) + positions = torch.randint( + 0, max_pos, (num_tokens,), dtype=torch.int64, device=device + ) + + qsz, kvsz = num_heads * HEAD_DIM, num_kv_heads * HEAD_DIM + qkv = torch.randn(num_tokens, qsz + 2 * kvsz, dtype=dtype, device=device) + qkv_orig = qkv.clone() + + ops.fused_minimax_m3_qknorm_rope_kv_insert( + qkv, + q_w, + k_w, + cos_sin, + positions, + num_heads, + num_kv_heads, + ROTARY_DIM, + eps, + kv_cache_dtype="auto", + ) + q_out, k_out, v_out = qkv.split([qsz, kvsz, kvsz], dim=-1) + + q_in, k_in, v_in = qkv_orig.split([qsz, kvsz, kvsz], dim=-1) + q_ref = norm_rope_ref( + q_in.view(num_tokens, num_heads, HEAD_DIM), q_w, positions, cos_sin, eps + ).view(num_tokens, qsz) + k_ref = norm_rope_ref( + k_in.view(num_tokens, num_kv_heads, HEAD_DIM), + k_w, + positions, + cos_sin, + eps, + ).view(num_tokens, kvsz) + + torch.testing.assert_close(q_out, q_ref, rtol=1e-2, atol=1e-2) + torch.testing.assert_close(k_out, k_ref, rtol=1e-2, atol=1e-2) + # V is untouched. + torch.testing.assert_close(v_out, v_in, rtol=0, atol=0) + + +# ── Test 2: sparse mode (full: index branch + cache inserts) ───────────────── + + +@pytest.mark.parametrize("num_tokens", [1, 7, 64, 513]) +@pytest.mark.parametrize("block_size", [16, 64]) +@pytest.mark.parametrize("kv_cache_dtype", ["auto", "fp8"]) +def test_sparse_full(num_tokens, block_size, kv_cache_dtype): + torch.manual_seed(1) + device, dtype, eps = "cuda", torch.bfloat16, 1e-6 + base, max_pos = 5_000_000.0, 4096 + num_heads, num_kv_heads, num_idx_heads = 16, 4, 4 + + q_w = torch.randn(HEAD_DIM, dtype=dtype, device=device) * 0.1 + k_w = torch.randn(HEAD_DIM, dtype=dtype, device=device) * 0.1 + iq_w = torch.randn(HEAD_DIM, dtype=dtype, device=device) * 0.1 + ik_w = torch.randn(HEAD_DIM, dtype=dtype, device=device) * 0.1 + cos_sin = make_cos_sin_cache(max_pos, ROTARY_DIM, base, dtype, device) + positions = torch.randint( + 0, max_pos, (num_tokens,), dtype=torch.int64, device=device + ) + + qsz, kvsz = num_heads * HEAD_DIM, num_kv_heads * HEAD_DIM + iqsz, iksz = num_idx_heads * HEAD_DIM, HEAD_DIM + # Single fused tensor packing [q | k | v | index_q | index_k]. + qkv = torch.randn( + num_tokens, qsz + 2 * kvsz + iqsz + iksz, dtype=dtype, device=device + ) + qkv_orig = qkv.clone() + splits = [qsz, kvsz, kvsz, iqsz, iksz] + + num_blocks = (num_tokens + block_size - 1) // block_size + 1 + kv_cache_storage_dtype = torch.uint8 if kv_cache_dtype == "fp8" else dtype + kv_cache = torch.zeros( + num_blocks, + 2, + block_size, + num_kv_heads, + HEAD_DIM, + dtype=kv_cache_storage_dtype, + device=device, + ) + index_cache = torch.zeros( + num_blocks, block_size, HEAD_DIM, dtype=dtype, device=device + ) + slot_mapping = torch.randperm( + num_blocks * block_size, dtype=torch.int64, device=device + )[:num_tokens] + index_slot_mapping = torch.roll(slot_mapping, shifts=1) + + # Contiguous gather targets: the kernel writes the normed/roped q and + # index_q here (de-interleaved from the packed qkv); k/v/index_k stay in + # place inside qkv and are scatter-inserted into the caches. + q_out = torch.empty(num_tokens, qsz, dtype=dtype, device=device) + index_q = torch.empty(num_tokens, iqsz, dtype=dtype, device=device) + + ops.fused_minimax_m3_qknorm_rope_kv_insert( + qkv, + q_w, + k_w, + cos_sin, + positions, + num_heads, + num_kv_heads, + ROTARY_DIM, + eps, + iq_w, + ik_w, + num_idx_heads, + slot_mapping, + index_slot_mapping, + kv_cache, + index_cache, + block_size, + q_out, + index_q, + kv_cache_dtype, + ) + + # ── norm+rope parity. q/index_q land in their gather buffers; k/index_k are + # rewritten in place inside qkv. ── + _, k_out, v_out, _, index_k = qkv.split(splits, dim=-1) + q_in, k_in, v_in, iq_orig, ik_orig = qkv_orig.split(splits, dim=-1) + q_ref = norm_rope_ref( + q_in.view(num_tokens, num_heads, HEAD_DIM), q_w, positions, cos_sin, eps + ).view(num_tokens, qsz) + k_ref = norm_rope_ref( + k_in.view(num_tokens, num_kv_heads, HEAD_DIM), + k_w, + positions, + cos_sin, + eps, + ).view(num_tokens, kvsz) + iq_ref = norm_rope_ref( + iq_orig.view(num_tokens, num_idx_heads, HEAD_DIM), + iq_w, + positions, + cos_sin, + eps, + ).view(num_tokens, num_idx_heads * HEAD_DIM) + ik_ref = norm_rope_ref( + ik_orig.view(num_tokens, 1, HEAD_DIM), ik_w, positions, cos_sin, eps + ).view(num_tokens, HEAD_DIM) + + torch.testing.assert_close(q_out, q_ref, rtol=1e-2, atol=1e-2) + torch.testing.assert_close(k_out, k_ref, rtol=1e-2, atol=1e-2) + torch.testing.assert_close(index_q, iq_ref, rtol=1e-2, atol=1e-2) + torch.testing.assert_close(index_k, ik_ref, rtol=1e-2, atol=1e-2) + + # ── Cache inserts. ── + # Main cache layout is [num_blocks, 2, block_size, num_kv_heads, head_dim] + # (the K/V axis sits *before* block_size); index cache is [nb, bs, head_dim]. + k_ref_h = k_ref.view(num_tokens, num_kv_heads, HEAD_DIM) + v_ref_h = v_in.view(num_tokens, num_kv_heads, HEAD_DIM) # v is raw (no norm/rope) + if kv_cache_dtype == "fp8": + expected_kv_cache = torch.zeros_like(kv_cache) + scale = torch.ones((), device=device) + ops.reshape_and_cache_flash( + k_out.view(num_tokens, num_kv_heads, HEAD_DIM), + v_out.view(num_tokens, num_kv_heads, HEAD_DIM), + expected_kv_cache[:, 0], + expected_kv_cache[:, 1], + slot_mapping, + kv_cache_dtype, + scale, + scale, + ) + torch.testing.assert_close(kv_cache, expected_kv_cache, rtol=0, atol=0) + else: + for t in range(num_tokens): + s = slot_mapping[t].item() + b, pos = s // block_size, s % block_size + torch.testing.assert_close( + kv_cache[b, 0, pos], k_ref_h[t], rtol=1e-2, atol=1e-2 + ) + torch.testing.assert_close(kv_cache[b, 1, pos], v_ref_h[t], rtol=0, atol=0) + + expected_index_cache = torch.zeros_like(index_cache).view(-1, HEAD_DIM) + expected_index_cache[index_slot_mapping] = index_k + torch.testing.assert_close( + index_cache.view(-1, HEAD_DIM), expected_index_cache, rtol=0, atol=0 + ) + + +# ── Test 3: fp8 (e4m3) index outputs ───────────────────────────────────────── +# The fp8 score path stores index_q and the index-K cache as e4m3 while q/k/v + +# q_out stay bf16. Asserts: (1) q/k/v/q_out are bit-identical to the bf16 run +# (the index dtype must not perturb the main branch), and (2) the e4m3 index +# outputs dequantize close to the bf16 reference. + + +@pytest.mark.skipif( + not torch.cuda.is_available() or torch.cuda.get_device_capability() < (8, 9), + reason="e4m3 conversion requires CUDA SM89+.", +) +@pytest.mark.parametrize("num_tokens", [1, 7, 64, 513]) +@pytest.mark.parametrize("block_size", [16, 64]) +def test_sparse_full_fp8_index(num_tokens, block_size): + torch.manual_seed(1) + device, dtype, eps = "cuda", torch.bfloat16, 1e-6 + base, max_pos = 5_000_000.0, 4096 + num_heads, num_kv_heads, num_idx_heads = 16, 4, 4 + + q_w = torch.randn(HEAD_DIM, dtype=dtype, device=device) * 0.1 + k_w = torch.randn(HEAD_DIM, dtype=dtype, device=device) * 0.1 + iq_w = torch.randn(HEAD_DIM, dtype=dtype, device=device) * 0.1 + ik_w = torch.randn(HEAD_DIM, dtype=dtype, device=device) * 0.1 + cos_sin = make_cos_sin_cache(max_pos, ROTARY_DIM, base, dtype, device) + positions = torch.randint( + 0, max_pos, (num_tokens,), dtype=torch.int64, device=device + ) + + qsz, kvsz = num_heads * HEAD_DIM, num_kv_heads * HEAD_DIM + iqsz, iksz = num_idx_heads * HEAD_DIM, HEAD_DIM + qkv0 = torch.randn( + num_tokens, qsz + 2 * kvsz + iqsz + iksz, dtype=dtype, device=device + ) + + num_blocks = (num_tokens + block_size - 1) // block_size + 1 + slot_mapping = torch.randperm( + num_blocks * block_size, dtype=torch.int64, device=device + )[:num_tokens] + index_slot_mapping = torch.roll(slot_mapping, shifts=1) + + def run(index_dtype): + qkv = qkv0.clone() + kv_cache = torch.zeros( + num_blocks, + 2, + block_size, + num_kv_heads, + HEAD_DIM, + dtype=dtype, + device=device, + ) + index_cache = torch.zeros( + num_blocks, block_size, HEAD_DIM, dtype=index_dtype, device=device + ) + q_out = torch.empty(num_tokens, qsz, dtype=dtype, device=device) + index_q = torch.empty(num_tokens, iqsz, dtype=index_dtype, device=device) + ops.fused_minimax_m3_qknorm_rope_kv_insert( + qkv, + q_w, + k_w, + cos_sin, + positions, + num_heads, + num_kv_heads, + ROTARY_DIM, + eps, + iq_w, + ik_w, + num_idx_heads, + slot_mapping, + index_slot_mapping, + kv_cache, + index_cache, + block_size, + q_out, + index_q, + ) + return qkv, kv_cache, index_cache, q_out, index_q + + qkv_bf, kvc_bf, idxc_bf, qo_bf, iq_bf = run(torch.bfloat16) + qkv_fp, kvc_fp, idxc_fp, qo_fp, iq_fp = run(torch.float8_e4m3fn) + + assert iq_fp.dtype == torch.float8_e4m3fn + assert idxc_fp.dtype == torch.float8_e4m3fn + + # (1) The main branch (q/k/v in qkv, q_out, kv cache) must be bit-identical: + # the index output dtype must not perturb anything else. + torch.testing.assert_close(qo_fp, qo_bf, rtol=0, atol=0) + torch.testing.assert_close(qkv_fp, qkv_bf, rtol=0, atol=0) + torch.testing.assert_close(kvc_fp, kvc_bf, rtol=0, atol=0) + + # (2) Dequantized e4m3 index outputs match the bf16 reference within fp8 ulp. + torch.testing.assert_close(iq_fp.float(), iq_bf.float(), rtol=0.13, atol=0.05) + torch.testing.assert_close(idxc_fp.float(), idxc_bf.float(), rtol=0.13, atol=0.05) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_fused_qk_norm_rope_gate.py b/asian_ai/performance/vLLM/tests/kernels/test_fused_qk_norm_rope_gate.py new file mode 100644 index 0000000000000000000000000000000000000000..1f2bd67fa7789f684fddd4fe2afc240551df4989 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_fused_qk_norm_rope_gate.py @@ -0,0 +1,159 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm.model_executor.layers.fused_qk_norm_rope import fused_qk_rmsnorm_rope_gate +from vllm.model_executor.layers.rotary_embedding import RotaryEmbedding +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +# Qwen/Qwen3.6-27B config (huggingface.co/Qwen/Qwen3.6-27B), TP=1 shapes. +NUM_Q_HEADS = 24 +NUM_KV_HEADS = 4 +HEAD_DIM = 256 +PARTIAL_ROTARY_FACTOR = 0.25 +ROTARY_DIM = int(HEAD_DIM * PARTIAL_ROTARY_FACTOR) # 64 +RMS_NORM_EPS = 1e-6 +MAX_POSITION_EMBEDDINGS = 262144 +ROPE_THETA = 10000000.0 + +DTYPES = [torch.bfloat16] +SEEDS = [13] +NUM_TOKENS = [1, 4, 37] + + +def _ref_qk_rmsnorm_rope_gate( + q_gate: torch.Tensor, + k: torch.Tensor, + q_gamma: torch.Tensor, + k_gamma: torch.Tensor, + cos_sin_cache: torch.Tensor, + positions: torch.Tensor, + eps: float, + num_q_heads: int, + num_kv_heads: int, + head_dim: int, + rotary_dim: int, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """PyTorch reference: split + RMSNorm + partial NeoX RoPE + gate extraction. + + Matches ``fused_qk_rmsnorm_rope_gate``'s contract: ``q_gamma`` / ``k_gamma`` + are the already-adjusted effective gammas (for GemmaRMSNorm the caller + has done ``weight + 1`` before passing them in). + """ + n_tokens = q_gate.shape[0] + half = rotary_dim // 2 + + # Per head the q projection is laid out as [q | gate]. + q_gate = q_gate.view(n_tokens, num_q_heads, 2 * head_dim) + q = q_gate[..., :head_dim] + gate = q_gate[..., head_dim:].reshape(n_tokens, num_q_heads * head_dim) + k = k.view(n_tokens, num_kv_heads, head_dim) + + def rms_norm(x: torch.Tensor, gamma: torch.Tensor) -> torch.Tensor: + orig_dtype = x.dtype + x = x.float() + var = x.pow(2).mean(dim=-1, keepdim=True) + x = x * torch.rsqrt(var + eps) + return (x * gamma.float()).to(orig_dtype) + + q = rms_norm(q, q_gamma) + k = rms_norm(k, k_gamma) + + # Partial NeoX RoPE on the first ``rotary_dim`` elements of each head; + # cos_sin_cache row is packed as [cos(half) | sin(half)]. + pos = positions.view(-1) + cos = cos_sin_cache[pos, :half].float()[:, None, :] + sin = cos_sin_cache[pos, half:rotary_dim].float()[:, None, :] + + def rope(x: torch.Tensor) -> torch.Tensor: + x_rot, x_pass = x[..., :rotary_dim], x[..., rotary_dim:] + x1 = x_rot[..., :half].float() + x2 = x_rot[..., half:].float() + o1 = x1 * cos - x2 * sin + o2 = x2 * cos + x1 * sin + rotated = torch.cat([o1, o2], dim=-1).to(x.dtype) + return torch.cat([rotated, x_pass], dim=-1) + + q = rope(q).reshape(n_tokens, num_q_heads * head_dim) + k = rope(k).reshape(n_tokens, num_kv_heads * head_dim) + return q, k, gate + + +@pytest.mark.skipif( + not current_platform.is_cuda_alike(), + reason="fused_qk_rmsnorm_rope_gate Triton kernel requires CUDA/ROCm", +) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@torch.inference_mode() +def test_fused_qk_norm_rope_gate_matches_reference( + default_vllm_config, + dtype: torch.dtype, + seed: int, + num_tokens: int, +): + device = torch.device("cuda", torch.accelerator.current_device_index()) + torch.set_default_device(device) + set_random_seed(seed) + + q_gate = torch.randn( + num_tokens, NUM_Q_HEADS * 2 * HEAD_DIM, dtype=dtype, device=device + ) + k = torch.randn(num_tokens, NUM_KV_HEADS * HEAD_DIM, dtype=dtype, device=device) + # GemmaRMSNorm-style: the kernel takes the effective gamma (weight + 1). + q_gamma = ( + torch.empty(HEAD_DIM, dtype=dtype, device=device).normal_(mean=0.0, std=0.1) + + 1.0 + ) + k_gamma = ( + torch.empty(HEAD_DIM, dtype=dtype, device=device).normal_(mean=0.0, std=0.1) + + 1.0 + ) + + # fused_qk_rmsnorm_rope_gate only handles NeoX-style RoPE. + rope = RotaryEmbedding( + head_size=HEAD_DIM, + rotary_dim=ROTARY_DIM, + max_position_embeddings=MAX_POSITION_EMBEDDINGS, + base=ROPE_THETA, + is_neox_style=True, + dtype=dtype, + ).to(device) + positions = torch.arange(num_tokens, dtype=torch.long, device=device) + + q_ref, k_ref, gate_ref = _ref_qk_rmsnorm_rope_gate( + q_gate, + k, + q_gamma, + k_gamma, + rope.cos_sin_cache, + positions, + RMS_NORM_EPS, + NUM_Q_HEADS, + NUM_KV_HEADS, + HEAD_DIM, + ROTARY_DIM, + ) + q_out, k_out, gate_out = fused_qk_rmsnorm_rope_gate( + q_gate, + k, + q_gamma, + k_gamma, + rope.cos_sin_cache, + positions, + RMS_NORM_EPS, + NUM_Q_HEADS, + NUM_KV_HEADS, + HEAD_DIM, + ROTARY_DIM, + ) + + atol, rtol = 2e-3, 5e-3 + torch.testing.assert_close(q_out, q_ref, atol=atol, rtol=rtol) + torch.testing.assert_close(k_out, k_ref, atol=atol, rtol=rtol) + # gate is a verbatim copy of the source slice — must match bit-exactly. + torch.testing.assert_close(gate_out, gate_ref, atol=0, rtol=0) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_fused_quant_activation.py b/asian_ai/performance/vLLM/tests/kernels/test_fused_quant_activation.py new file mode 100644 index 0000000000000000000000000000000000000000..e820cd128b12393f53e26bc5062edc58377a07a3 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_fused_quant_activation.py @@ -0,0 +1,71 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch + +import vllm._custom_ops as ops +from tests.kernels.utils import opcheck +from vllm.model_executor.layers.activation import SiluAndMul +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +DTYPES = [torch.bfloat16, torch.float16] +QUANT_DTYPES = [current_platform.fp8_dtype()] +NUM_TOKENS = [1, 17, 86, 1234, 3045] # Arbitrary values for testing +HIDDEN_SIZES = [16, 48, 128, 1562, 4096] # Arbitrary values for testing +SEEDS = [0] +CUDA_DEVICES = [ + f"cuda:{i}" for i in range(1 if torch.accelerator.device_count() == 1 else 2) +] + + +def ref_impl( + silu_and_mul: SiluAndMul, x: torch.Tensor, scale: torch.Tensor +) -> torch.Tensor: + silu_and_mul_out = silu_and_mul.forward_native(x) + out, scales = ops.scaled_fp8_quant(silu_and_mul_out, scale) + return out + + +def ops_impl(x: torch.Tensor, scale: torch.Tensor) -> torch.Tensor: + out_shape = (x.shape[0], x.shape[1] // 2) + out = torch.empty(out_shape, dtype=current_platform.fp8_dtype(), device=x.device) + torch.ops._C.silu_and_mul_quant(out, x, scale) + return out + + +@pytest.mark.parametrize("num_tokens", NUM_TOKENS) +@pytest.mark.parametrize("hidden_size", HIDDEN_SIZES) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("quant_dtype", QUANT_DTYPES) +@pytest.mark.parametrize("seed", SEEDS) +@pytest.mark.parametrize("device", CUDA_DEVICES) +@torch.inference_mode() +def test_silu_and_mul( + default_vllm_config, + num_tokens: int, + hidden_size: int, + dtype: torch.dtype, + quant_dtype: torch.dtype, + seed: int, + device: str, +) -> None: + set_random_seed(seed) + torch.set_default_device(device) + + layer = SiluAndMul() + + # Make inputs + scale = torch.randn((1), device=device, dtype=torch.float32) + x = torch.randn(num_tokens, hidden_size, dtype=dtype) + + ref_out = ref_impl(layer, x, scale) + ops_out = ops_impl(x, scale) + + assert ref_out.dtype == quant_dtype + assert ops_out.dtype == quant_dtype + assert ref_out.shape == ops_out.shape + assert torch.allclose( + ref_out.to(dtype=torch.float32), ops_out.to(dtype=torch.float32) + ) + opcheck(torch.ops._C.silu_and_mul_quant, (ops_out, x, scale)) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_fused_recurrent_packed_decode.py b/asian_ai/performance/vLLM/tests/kernels/test_fused_recurrent_packed_decode.py new file mode 100644 index 0000000000000000000000000000000000000000..1fb52f651291034e6db171a4a270c01c38fecc36 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_fused_recurrent_packed_decode.py @@ -0,0 +1,98 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from vllm.model_executor.layers.fla.ops import ( + fused_recurrent_gated_delta_rule, + fused_recurrent_gated_delta_rule_packed_decode, +) + + +@pytest.mark.skipif(not torch.cuda.is_available(), reason="Need CUDA device") +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32]) +@pytest.mark.parametrize("strided_mixed_qkv", [False, True]) +def test_fused_recurrent_packed_decode_matches_reference( + dtype: torch.dtype, strided_mixed_qkv: bool +): + torch.manual_seed(0) + + # Small but representative GDN config (Qwen3Next defaults are K=128, V=128). + B = 32 + H = 4 + HV = 8 # grouped value attention: HV must be divisible by H + K = 128 + V = 128 + qkv_dim = 2 * (H * K) + (HV * V) + + device = torch.device("cuda") + + if strided_mixed_qkv: + # Simulate a packed view into a larger projection buffer: + # mixed_qkv.stride(0) > mixed_qkv.shape[1] + proj = torch.randn((B, qkv_dim + 64), device=device, dtype=dtype) + mixed_qkv = proj[:, :qkv_dim] + else: + mixed_qkv = torch.randn((B, qkv_dim), device=device, dtype=dtype) + + a = torch.randn((B, HV), device=device, dtype=dtype) + b = torch.randn((B, HV), device=device, dtype=dtype) + A_log = torch.randn((HV,), device=device, dtype=dtype) + dt_bias = torch.randn((HV,), device=device, dtype=dtype) + + # Continuous batching indices (include PAD_SLOT_ID=-1 cases). + ssm_state_indices = torch.arange(B, device=device, dtype=torch.int32) + ssm_state_indices[-3:] = -1 + + state0 = torch.randn((B, HV, V, K), device=device, dtype=dtype) + state_ref = state0.clone() + state_packed = state0.clone() + + out_packed = torch.empty((B, 1, HV, V), device=device, dtype=dtype) + + # Reference path: materialize contiguous Q/K/V + explicit gating. + q, k, v = torch.split(mixed_qkv, [H * K, H * K, HV * V], dim=-1) + q = q.view(B, H, K).unsqueeze(1).contiguous() + k = k.view(B, H, K).unsqueeze(1).contiguous() + v = v.view(B, HV, V).unsqueeze(1).contiguous() + + x = a.float() + dt_bias.float() + softplus_x = torch.where( + x <= 20.0, torch.log1p(torch.exp(torch.clamp(x, max=20.0))), x + ) + g = (-torch.exp(A_log.float()) * softplus_x).unsqueeze(1) + beta = torch.sigmoid(b.float()).to(dtype).unsqueeze(1) + + out_ref, state_ref = fused_recurrent_gated_delta_rule( + q=q, + k=k, + v=v, + g=g, + beta=beta, + scale=K**-0.5, + initial_state=state_ref, + inplace_final_state=True, + cu_seqlens=None, + ssm_state_indices=ssm_state_indices, + use_qk_l2norm_in_kernel=True, + ) + + # Packed path: fused gating + recurrent directly from packed mixed_qkv. + fused_recurrent_gated_delta_rule_packed_decode( + mixed_qkv=mixed_qkv, + a=a, + b=b, + A_log=A_log, + dt_bias=dt_bias, + scale=K**-0.5, + initial_state=state_packed, + out=out_packed, + ssm_state_indices=ssm_state_indices, + use_qk_l2norm_in_kernel=True, + ) + + atol = 2e-2 if dtype != torch.float32 else 1e-4 + rtol = 1e-2 if dtype != torch.float32 else 1e-4 + torch.testing.assert_close(out_packed, out_ref, rtol=rtol, atol=atol) + torch.testing.assert_close(state_packed, state_ref, rtol=rtol, atol=atol) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_fused_sigmoid_gating_delta_rule.py b/asian_ai/performance/vLLM/tests/kernels/test_fused_sigmoid_gating_delta_rule.py new file mode 100644 index 0000000000000000000000000000000000000000..63f05e369601e62adde0107fe5d9f0708a72e0e4 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_fused_sigmoid_gating_delta_rule.py @@ -0,0 +1,196 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch +import torch.nn.functional as F + +from vllm.model_executor.layers.fla.ops import ( + fused_recurrent_gated_delta_rule, + fused_sigmoid_gating_delta_rule_update, +) +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +DEVICE = current_platform.device_type + + +@pytest.mark.parametrize("tp_size", [1]) +@pytest.mark.parametrize("num_reqs", [1, 2, 4]) +@pytest.mark.parametrize("num_k_heads", [16]) +@pytest.mark.parametrize("num_v_heads", [32]) +@pytest.mark.parametrize("head_k_dim", [128]) +@pytest.mark.parametrize("head_v_dim", [128]) +@pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16]) +def test_fused_sigmoid_gating_delta_rule_update_non_spec( + tp_size: int, + num_reqs: int, + num_k_heads: int, + num_v_heads: int, + head_k_dim: int, + head_v_dim: int, + dtype: torch.dtype, +) -> None: + torch.set_default_device(DEVICE) + set_random_seed(0) + key_dim = head_k_dim * num_k_heads + value_dim = head_v_dim * num_v_heads + mixed_qkv_dim = (key_dim * 2 + value_dim) // tp_size + seq_len = 1 # seq_len is 1 for decode + num_tokens = num_reqs * seq_len + total_entries = num_tokens * 2 + + mixed_qkv = torch.rand(num_tokens, mixed_qkv_dim, dtype=dtype) + query, key, value = torch.split( + mixed_qkv, + [ + key_dim // tp_size, + key_dim // tp_size, + value_dim // tp_size, + ], + dim=-1, + ) + query = query.view(1, num_tokens, num_k_heads, head_k_dim) + key = key.view(1, num_tokens, num_k_heads, head_k_dim) + value = value.view(1, num_tokens, num_v_heads, head_v_dim) + + A_log = torch.rand(num_v_heads // tp_size, dtype=dtype) + dt_bias = torch.rand(num_v_heads // tp_size, dtype=dtype) + a = torch.rand(num_tokens, num_v_heads, dtype=dtype) + b = torch.rand(num_tokens, num_v_heads, dtype=dtype) + ssm_state = torch.rand( + total_entries, num_v_heads, head_k_dim, head_v_dim, dtype=dtype + ) + state_indices = torch.randperm(total_entries, dtype=torch.int32)[:num_tokens] + cu_seqlens = torch.arange(0, num_tokens + 1, dtype=torch.int32) + + beta = b.sigmoid() + g = -A_log.float().exp() * F.softplus(a.float() + dt_bias) + core_attn_out_ref, last_recurrent_state_ref = fused_recurrent_gated_delta_rule( + q=query, + k=key, + v=value, + g=g.unsqueeze(0), + beta=beta.unsqueeze(0), + initial_state=ssm_state.clone(), + inplace_final_state=True, + ssm_state_indices=state_indices, + cu_seqlens=cu_seqlens, + use_qk_l2norm_in_kernel=True, + ) + + core_attn_out, last_recurrent_state = fused_sigmoid_gating_delta_rule_update( + A_log=A_log, + a=a, + b=b, + dt_bias=dt_bias, + q=query, + k=key, + v=value, + initial_state=ssm_state, + inplace_final_state=True, + ssm_state_indices=state_indices, + cu_seqlens=cu_seqlens, + use_qk_l2norm_in_kernel=True, + ) + + torch.testing.assert_close(core_attn_out, core_attn_out_ref, atol=1e-2, rtol=1e-2) + torch.testing.assert_close( + last_recurrent_state, last_recurrent_state_ref, atol=1e-2, rtol=1e-2 + ) + + +@pytest.mark.parametrize("tp_size", [1]) +@pytest.mark.parametrize("num_reqs", [1, 2, 4]) +@pytest.mark.parametrize("num_k_heads", [16]) +@pytest.mark.parametrize("num_v_heads", [32]) +@pytest.mark.parametrize("head_k_dim", [128]) +@pytest.mark.parametrize("head_v_dim", [128]) +@pytest.mark.parametrize("num_speculative_tokens", [1, 3]) +@pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16]) +def test_fused_sigmoid_gating_delta_rule_update_spec( + tp_size: int, + num_reqs: int, + num_k_heads: int, + num_v_heads: int, + head_k_dim: int, + head_v_dim: int, + num_speculative_tokens: int, + dtype: torch.dtype, +) -> None: + torch.set_default_device(DEVICE) + set_random_seed(0) + key_dim = head_k_dim * num_k_heads + value_dim = head_v_dim * num_v_heads + mixed_qkv_dim = (key_dim * 2 + value_dim) // tp_size + num_tokens = num_reqs * (num_speculative_tokens + 1) + total_entries = num_tokens * 2 + + mixed_qkv = torch.rand(num_tokens, mixed_qkv_dim, dtype=dtype) + query, key, value = torch.split( + mixed_qkv, + [ + key_dim // tp_size, + key_dim // tp_size, + value_dim // tp_size, + ], + dim=-1, + ) + query = query.view(1, num_tokens, num_k_heads, head_k_dim) + key = key.view(1, num_tokens, num_k_heads, head_k_dim) + value = value.view(1, num_tokens, num_v_heads, head_v_dim) + + A_log = torch.rand(num_v_heads // tp_size, dtype=dtype) + dt_bias = torch.rand(num_v_heads // tp_size, dtype=dtype) + a = torch.rand(num_tokens, num_v_heads, dtype=dtype) + b = torch.rand(num_tokens, num_v_heads, dtype=dtype) + ssm_state = torch.rand( + total_entries, num_v_heads, head_k_dim, head_v_dim, dtype=dtype + ) + state_indices = torch.randperm( + total_entries, + dtype=torch.int32, + )[:num_tokens].view(num_reqs, num_speculative_tokens + 1) + num_accepted_tokens = torch.randint( + 1, num_speculative_tokens + 1, (num_reqs,), dtype=torch.int32 + ) + cu_seqlens = torch.arange( + 0, num_tokens + 1, num_speculative_tokens + 1, dtype=torch.int32 + ) + + beta = b.sigmoid() + g = -A_log.float().exp() * F.softplus(a.float() + dt_bias) + core_attn_out_ref, last_recurrent_state_ref = fused_recurrent_gated_delta_rule( + q=query, + k=key, + v=value, + g=g.unsqueeze(0), + beta=beta.unsqueeze(0), + initial_state=ssm_state.clone(), + inplace_final_state=True, + ssm_state_indices=state_indices, + cu_seqlens=cu_seqlens, + num_accepted_tokens=num_accepted_tokens, + use_qk_l2norm_in_kernel=True, + ) + + core_attn_out, last_recurrent_state = fused_sigmoid_gating_delta_rule_update( + A_log=A_log, + a=a, + b=b, + dt_bias=dt_bias, + q=query, + k=key, + v=value, + initial_state=ssm_state, + inplace_final_state=True, + ssm_state_indices=state_indices, + cu_seqlens=cu_seqlens, + num_accepted_tokens=num_accepted_tokens, + use_qk_l2norm_in_kernel=True, + ) + + torch.testing.assert_close(core_attn_out, core_attn_out_ref, atol=1e-2, rtol=1e-2) + torch.testing.assert_close( + last_recurrent_state, last_recurrent_state_ref, atol=1e-2, rtol=1e-2 + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_kda.py b/asian_ai/performance/vLLM/tests/kernels/test_kda.py new file mode 100644 index 0000000000000000000000000000000000000000..bc4801caa25c193cdb0fa5aa3269715eef45b4f3 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_kda.py @@ -0,0 +1,226 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Precision tests for vllm's chunk_kda Triton operator. + +Compares chunk_kda against a naive recurrent reference (float32). +Uses torch.rand for q/k/v to match FLA's test pattern. +""" + +import pytest +import torch +import torch.nn.functional as F + +from vllm.model_executor.layers.fla.ops.kda import ( + chunk_kda, + chunk_kda_with_fused_gate, + fused_kda_gate, +) +from vllm.model_executor.layers.fla.ops.l2norm import l2norm_fwd + +DEVICE = "cuda" + + +def naive_recurrent_kda( + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + g: torch.Tensor, + beta: torch.Tensor, + scale: float | None = None, + initial_state: torch.Tensor | None = None, + output_final_state: bool = False, +) -> tuple[torch.Tensor, torch.Tensor | None]: + """Naive recurrent KDA reference, ported from FLA's naive.py.""" + dtype = v.dtype + B, T, H, K = q.shape + V = v.shape[-1] + if scale is None: + scale = K**-0.5 + + q, k, v, g, beta = (x.to(torch.float) for x in [q, k, v, g, beta]) + q = q * scale + + S = k.new_zeros(B, H, K, V).to(q) + if initial_state is not None: + S += initial_state + o = torch.zeros_like(v) + for i in range(T): + q_i, k_i, v_i, g_i, b_i = q[:, i], k[:, i], v[:, i], g[:, i], beta[:, i] + S = S * g_i[..., None].exp() + S = S + torch.einsum( + "bhk,bhv->bhkv", + b_i[..., None] * k_i, + v_i - (k_i[..., None] * S).sum(-2), + ) + o[:, i] = torch.einsum("bhk,bhkv->bhv", q_i, S) + if not output_final_state: + S = None + return o.to(dtype), S + + +def assert_close( + name: str, + ref: torch.Tensor, + tri: torch.Tensor, + ratio: float, + err_atol: float = 1e-6, +): + """RMSE-based relative error comparison.""" + abs_err = (ref.detach() - tri.detach()).flatten().abs().max().item() + rmse_diff = (ref.detach() - tri.detach()).flatten().square().mean().sqrt().item() + rmse_base = ref.detach().flatten().square().mean().sqrt().item() + rel_err = rmse_diff / (rmse_base + 1e-8) + print(f"{name:>4} | abs={abs_err:.6f} | rmse={rel_err:.6f} | thr={ratio}") + if abs_err <= err_atol: + return + assert not torch.isnan(ref).any(), f"{name}: NaN detected in ref" + assert not torch.isnan(tri).any(), f"{name}: NaN detected in tri" + assert rel_err < ratio, ( + f"{name}: max abs err {abs_err:.6f}, rmse ratio {rel_err:.6f} >= {ratio}" + ) + + +@pytest.mark.parametrize( + ("H", "D", "cu_seqlens", "dtype"), + [ + pytest.param( + *test, + id="H{}-D{}-cu{}-{}".format(*test), + ) + for test in [ + (32, 128, [0, 64], torch.float16), + (32, 128, [0, 1024], torch.float16), + (32, 128, [0, 15], torch.float16), + (32, 128, [0, 256, 512, 768, 1024], torch.float16), + (32, 128, [0, 15, 100, 300, 1200], torch.float16), + (64, 128, [0, 256, 500, 1000], torch.float16), + (32, 128, [0, 8192], torch.float16), + (32, 128, [0, 256, 500, 1000], torch.bfloat16), + ] + ], +) +@torch.inference_mode() +def test_chunk_kda( + H: int, + D: int, + cu_seqlens: list[int], + dtype: torch.dtype, +): + T = cu_seqlens[-1] + torch.manual_seed(42) + B = 1 + cu_seqlens_t = torch.LongTensor(cu_seqlens).to(DEVICE) + N = len(cu_seqlens) - 1 + + q = torch.rand(B, T, H, D, dtype=dtype, device=DEVICE) + k = torch.rand(B, T, H, D, dtype=dtype, device=DEVICE) + v = torch.rand(B, T, H, D, dtype=dtype, device=DEVICE) + g = F.logsigmoid(torch.randn(B, T, H, D, dtype=torch.float32, device=DEVICE)).to( + dtype + ) + beta = torch.rand(B, T, H, dtype=dtype, device=DEVICE).sigmoid() + h0 = torch.randn(N, H, D, D, dtype=torch.float32, device=DEVICE) + + # Naive reference with l2norm_fwd (same kernel as chunk_kda) + ref_outputs = [] + ref_states = [] + for i in range(N): + s, e = cu_seqlens[i], cu_seqlens[i + 1] + q_i = l2norm_fwd(q[:, s:e].contiguous()) + k_i = l2norm_fwd(k[:, s:e].contiguous()) + o_i, ht_i = naive_recurrent_kda( + q_i, + k_i, + v[:, s:e], + g[:, s:e], + beta[:, s:e], + initial_state=h0[i], + output_final_state=True, + ) + ref_outputs.append(o_i) + ref_states.append(ht_i) + ref_o = torch.cat(ref_outputs, dim=1) + ref_ht = torch.cat(ref_states, dim=0) + + # h0 transposed to (V, K) layout for the kernel; naive uses (K, V) + tri_o, tri_ht = chunk_kda( + q=q.clone(), + k=k.clone(), + v=v.clone(), + g=g.clone(), + beta=beta.clone(), + initial_state=h0.transpose(-1, -2).contiguous().clone(), + output_final_state=True, + cu_seqlens=cu_seqlens_t, + use_qk_l2norm_in_kernel=True, + ) + + assert not torch.isnan(tri_o).any(), "Triton output o contains NaN" + assert not torch.isnan(tri_ht).any(), "Triton output ht contains NaN" + assert_close("o", ref_o, tri_o, 0.005) + assert_close("ht", ref_ht, tri_ht.transpose(-1, -2).contiguous(), 0.005) + + +@pytest.mark.parametrize( + ("cu_seqlens", "dtype"), + [ + ([0, 64], torch.float16), + ([0, 15, 100, 300], torch.bfloat16), + ], +) +@torch.inference_mode() +def test_chunk_kda_fused_gate_cumsum_matches_unfused( + cu_seqlens: list[int], + dtype: torch.dtype, +): + H, D = 8, 64 + T = cu_seqlens[-1] + N = len(cu_seqlens) - 1 + torch.manual_seed(123) + + cu_seqlens_t = torch.tensor(cu_seqlens, dtype=torch.int32, device=DEVICE) + q = torch.randn(1, T, H, D, dtype=dtype, device=DEVICE) + k = torch.randn(1, T, H, D, dtype=dtype, device=DEVICE) + v = torch.randn(1, T, H, D, dtype=dtype, device=DEVICE) + raw_g = torch.randn(1, T, H, D, dtype=dtype, device=DEVICE) + beta = torch.rand(1, T, H, dtype=dtype, device=DEVICE).sigmoid() + A_log = (torch.randn(H, dtype=torch.float32, device=DEVICE) * 0.5).contiguous() + dt_bias = ( + torch.randn(H * D, dtype=torch.float32, device=DEVICE) * 0.1 + ).contiguous() + h0 = torch.randn(N, H, D, D, dtype=torch.float32, device=DEVICE) + initial_state = h0.transpose(-1, -2).contiguous() + + gate = fused_kda_gate( + raw_g.reshape(T, H * D), + A_log, + D, + g_bias=dt_bias, + ).unsqueeze(0) + old_o, old_ht = chunk_kda( + q=q.clone(), + k=k.clone(), + v=v.clone(), + g=gate, + beta=beta.clone(), + initial_state=initial_state.clone(), + output_final_state=True, + cu_seqlens=cu_seqlens_t, + use_qk_l2norm_in_kernel=True, + ) + new_o, new_ht = chunk_kda_with_fused_gate( + q=q.clone(), + k=k.clone(), + v=v.clone(), + raw_g=raw_g, + beta=beta.clone(), + A_log=A_log, + g_bias=dt_bias, + initial_state=initial_state.clone(), + output_final_state=True, + cu_seqlens=cu_seqlens_t, + use_qk_l2norm_in_kernel=True, + ) + + assert_close("o", old_o, new_o, 1e-3, err_atol=1e-3) + assert_close("ht", old_ht, new_ht, 1e-3, err_atol=1e-3) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_mhc_kernels.py b/asian_ai/performance/vLLM/tests/kernels/test_mhc_kernels.py new file mode 100644 index 0000000000000000000000000000000000000000..82e8620b52d63934f3ce18feb99ea95cc2e5058a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_mhc_kernels.py @@ -0,0 +1,357 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import pytest +import torch + +import vllm.model_executor.kernels.mhc # noqa: F401 +from vllm.model_executor.kernels.mhc.tilelang import ( + _tilelang_hc_prenorm_gemm, + _torch_hc_prenorm_gemm, +) +from vllm.model_executor.layers.mhc import HAS_TILELANG_MHC +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +DEVICE = current_platform.device_type + + +def sinkhorn_normalize_ref(x: torch.Tensor, repeat: int, eps: float) -> torch.Tensor: + x = x.softmax(-1) + eps + x = x / (x.sum(-2, keepdim=True) + eps) + for _ in range(repeat - 1): + x = x / (x.sum(-1, keepdim=True) + eps) + x = x / (x.sum(-2, keepdim=True) + eps) + return x + + +def mhc_pre_ref( + residual: torch.Tensor, + fn: torch.Tensor, + hc_scale: torch.Tensor, + hc_base: torch.Tensor, + rms_eps: float, + hc_pre_eps: float, + hc_sinkhorn_eps: float, + hc_post_mult_value: float, + sinkhorn_repeat: int, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """mHC pre reference kernel from tilelang repo: https://github.com/tile-ai/tilelang/blob/d135bd1cd2d2eee74fbb41dd0a0831a427194c86/examples/deepseek_mhc/example_mhc_pre.py#L303""" + hc_mult = residual.shape[-2] + + residual_flat = residual.flatten(-2, -1).float() + sqrsum = residual_flat.square().sum(-1) + mixes = ( + residual_flat @ fn.T * (sqrsum.unsqueeze(-1) / fn.shape[-1] + rms_eps).rsqrt() + ) + + hc_scale = torch.cat( + [ + hc_scale[0].expand(hc_mult), + hc_scale[1].expand(hc_mult), + hc_scale[2].expand(hc_mult * hc_mult), + ], + ) + mixes = mixes * hc_scale + hc_base + + pre_mix = mixes[:, :hc_mult].sigmoid().unsqueeze(-1) + hc_pre_eps + post_mix = ( + mixes[:, hc_mult : 2 * hc_mult].sigmoid() * hc_post_mult_value + ).unsqueeze(-1) + res_mix = mixes[:, 2 * hc_mult :].view(-1, hc_mult, hc_mult) + + res_mix = sinkhorn_normalize_ref( + res_mix, repeat=sinkhorn_repeat, eps=hc_sinkhorn_eps + ) + + layer_input = (residual * pre_mix).sum(-2).bfloat16() + + return post_mix, res_mix, layer_input + + +def mhc_post_ref( + x: torch.Tensor, + residual: torch.Tensor, + post_layer_mix: torch.Tensor, + comb_res_mix: torch.Tensor, +) -> torch.Tensor: + """mHC post reference kernel from tilelang repo: https://github.com/tile-ai/tilelang/blob/d135bd1cd2d2eee74fbb41dd0a0831a427194c86/examples/deepseek_mhc/example_mhc_post.py#L68""" + term2 = torch.bmm(comb_res_mix.mT, residual.float()) + return (x.float().unsqueeze(-2) * post_layer_mix + term2).bfloat16() + + +def hc_head_ref( + residual: torch.Tensor, + fn: torch.Tensor, + hc_scale: torch.Tensor, + hc_base: torch.Tensor, + rms_eps: float, + hc_eps: float, +) -> torch.Tensor: + residual_flat = residual.flatten(-2).float() + residual_norm = residual_flat * torch.rsqrt( + residual_flat.square().mean(dim=-1, keepdim=True) + rms_eps + ) + pre_mix = torch.nn.functional.linear(residual_norm, fn) + pre_mix = torch.sigmoid(pre_mix * hc_scale + hc_base) + hc_eps + return torch.sum(pre_mix.unsqueeze(-1) * residual.float(), dim=-2).bfloat16() + + +@pytest.mark.skipif( + not HAS_TILELANG_MHC, + reason="TileLang MHC support required", +) +@pytest.mark.parametrize("num_tokens", [1, 4, 8, 128]) +@pytest.mark.parametrize("hidden_size", [4096, 7168]) +@pytest.mark.parametrize("hc_mult", [4]) +def test_mhc_pre_tilelang(num_tokens, hidden_size, hc_mult): + torch.set_default_device(DEVICE) + set_random_seed(0) + + residual = torch.randn((num_tokens, hc_mult, hidden_size), dtype=torch.bfloat16) + hc_mult2 = hc_mult * hc_mult + hc_mult3 = 2 * hc_mult + hc_mult2 + fn = ( + torch.randn((hc_mult3, hc_mult, hidden_size), dtype=torch.float) + * 1e-4 + * (1 + torch.arange(hc_mult).mul(0.01).view(1, -1, 1)) + ).flatten(1, 2) + hc_scale = torch.randn((3,), dtype=torch.float) * 0.1 + hc_base = torch.randn((hc_mult3,), dtype=torch.float) * 0.1 + + hc_sinkhorn_eps = hc_pre_eps = rms_eps = 1e-6 + sinkhorn_repeat = 20 + hc_post_alpha = 1.0 + + ref = mhc_pre_ref( + residual, + fn, + hc_scale, + hc_base, + rms_eps, + hc_pre_eps, + hc_sinkhorn_eps, + hc_post_alpha, + sinkhorn_repeat, + ) + out = torch.ops.vllm.mhc_pre_tilelang( + residual, + fn, + hc_scale, + hc_base, + rms_eps, + hc_pre_eps, + hc_sinkhorn_eps, + hc_post_alpha, + sinkhorn_repeat, + ) + + for actual, expected in zip(out, ref, strict=True): + torch.testing.assert_close(actual, expected, atol=5e-2, rtol=1e-2) + + +@pytest.mark.skipif( + not HAS_TILELANG_MHC, + reason="TileLang MHC support required", +) +@pytest.mark.parametrize( + ("num_tokens", "hidden_size"), + [ + (1, 1280), + (512, 1280), + (2048, 1280), + (1, 4096), + (64, 4096), + (512, 4096), + (2048, 4096), + (1, 7168), + (64, 7168), + (512, 7168), + (2048, 7168), + ], +) +def test_hc_prenorm_gemm_tilelang(num_tokens, hidden_size): + torch.set_default_device(DEVICE) + set_random_seed(0) + + hc_mult = 4 + hc_mult3 = 2 * hc_mult + hc_mult * hc_mult + x = torch.randn((num_tokens, hc_mult * hidden_size), dtype=torch.bfloat16) + fn = torch.randn((hc_mult3, hc_mult * hidden_size), dtype=torch.float32) * 1e-4 + out_ref = torch.empty((1, num_tokens, hc_mult3), dtype=torch.float32) + sqrsum_ref = torch.empty((1, num_tokens), dtype=torch.float32) + out = torch.empty_like(out_ref) + sqrsum = torch.empty_like(sqrsum_ref) + + _torch_hc_prenorm_gemm(x, fn, out_ref, sqrsum_ref) + _tilelang_hc_prenorm_gemm(x, fn, out, sqrsum, hidden_size, hc_mult) + + torch.testing.assert_close(out, out_ref, atol=1e-5, rtol=1e-4) + torch.testing.assert_close(sqrsum, sqrsum_ref, atol=8.0, rtol=5e-4) + + +@pytest.mark.skipif( + not HAS_TILELANG_MHC, + reason="TileLang MHC support required", +) +@pytest.mark.parametrize("num_tokens", [1, 4, 8, 128]) +@pytest.mark.parametrize("hidden_size", [4096, 7168]) +@pytest.mark.parametrize("hc_mult", [4]) +def test_mhc_post_tilelang(num_tokens, hidden_size, hc_mult): + torch.set_default_device(DEVICE) + set_random_seed(0) + + x = torch.randn((num_tokens, hidden_size), dtype=torch.bfloat16) + residual = torch.randn((num_tokens, hc_mult, hidden_size), dtype=torch.bfloat16) + post_layer_mix = torch.randn((num_tokens, hc_mult, 1), dtype=torch.float32) + comb_res_mix = torch.randn((num_tokens, hc_mult, hc_mult), dtype=torch.float32) + + ref = mhc_post_ref(x, residual, post_layer_mix, comb_res_mix) + out = torch.ops.vllm.mhc_post_tilelang( + x, + residual, + post_layer_mix, + comb_res_mix, + ) + + torch.testing.assert_close(out, ref, atol=5e-2, rtol=1e-2) + + +@pytest.mark.skipif( + not HAS_TILELANG_MHC, + reason="TileLang MHC support required", +) +@pytest.mark.parametrize("num_tokens", [1, 4, 8, 128]) +@pytest.mark.parametrize("hidden_size", [4096, 7168]) +@pytest.mark.parametrize("hc_mult", [4]) +def test_mhc_fused_post_pre(num_tokens, hidden_size, hc_mult): + torch.set_default_device(DEVICE) + set_random_seed(0) + + x = torch.randn((num_tokens, hidden_size), dtype=torch.bfloat16) + residual = torch.randn((num_tokens, hc_mult, hidden_size), dtype=torch.bfloat16) + post_layer_mix = torch.randn((num_tokens, hc_mult, 1), dtype=torch.float32) + comb_res_mix = torch.randn((num_tokens, hc_mult, hc_mult), dtype=torch.float32) + + hc_mult2 = hc_mult * hc_mult + hc_mult3 = hc_mult * 2 + hc_mult2 + fn = ( + torch.randn((hc_mult3, hc_mult, hidden_size), dtype=torch.float) + * 1e-4 + * (1 + torch.arange(hc_mult).mul(0.01).view(1, -1, 1)) + ).flatten(1, 2) + hc_scale = torch.randn((3,), dtype=torch.float) * 0.1 + hc_base = torch.randn((hc_mult3,), dtype=torch.float) * 0.1 + + hc_sinkhorn_eps = hc_pre_eps = rms_eps = 1e-6 + sinkhorn_repeat = 20 + hc_post_alpha = 1.0 + + def run_ref(): + residual_ref = mhc_post_ref(x, residual, post_layer_mix, comb_res_mix) + post_mix_ref, res_mix_ref, layer_input_ref = mhc_pre_ref( + residual_ref, + fn, + hc_scale, + hc_base, + rms_eps, + hc_pre_eps, + hc_sinkhorn_eps, + hc_post_alpha, + sinkhorn_repeat, + ) + return residual_ref, post_mix_ref, res_mix_ref, layer_input_ref + + residual_ref, post_mix_ref, res_mix_ref, layer_input_ref = run_ref() + + residual, post_mix, res_mix, x = torch.ops.vllm.mhc_fused_post_pre_tilelang( + x, + residual, + post_layer_mix, + comb_res_mix, + fn, + hc_scale, + hc_base, + rms_eps, + hc_pre_eps, + hc_sinkhorn_eps, + hc_post_alpha, + sinkhorn_repeat, + ) + + torch.testing.assert_close(residual, residual_ref, atol=1e-2, rtol=1e-2) + torch.testing.assert_close(post_mix, post_mix_ref, atol=1e-2, rtol=1e-2) + torch.testing.assert_close(res_mix, res_mix_ref, atol=1e-2, rtol=1e-2) + torch.testing.assert_close(x, layer_input_ref, atol=1e-2, rtol=1e-2) + + +@pytest.mark.skipif( + not current_platform.is_rocm(), + reason="ROCm required", +) +@pytest.mark.parametrize("num_tokens", [1, 4, 8, 128]) +@pytest.mark.parametrize("hidden_size", [4096, 7168]) +@pytest.mark.parametrize("hc_mult", [4]) +def test_hc_head_triton(num_tokens, hidden_size, hc_mult): + torch.set_default_device(DEVICE) + set_random_seed(0) + + residual = torch.randn((num_tokens, hc_mult, hidden_size), dtype=torch.bfloat16) + fn = torch.randn((hc_mult, hc_mult * hidden_size), dtype=torch.float32) * 1e-4 + hc_scale = torch.randn((1,), dtype=torch.float32) * 0.1 + hc_base = torch.randn((hc_mult,), dtype=torch.float32) * 0.1 + rms_eps = hc_eps = 1e-6 + + out = torch.empty((num_tokens, hidden_size), dtype=torch.bfloat16) + out.fill_(float("nan")) + + result = torch.ops.vllm.hc_head_triton( + residual, + fn, + hc_scale, + hc_base, + out, + hidden_size, + rms_eps, + hc_eps, + hc_mult, + ) + + assert result is None + assert not torch.isnan(out).any() + + out_ref = hc_head_ref(residual, fn, hc_scale, hc_base, rms_eps, hc_eps) + torch.testing.assert_close(out, out_ref, atol=5e-2, rtol=1e-2) + + +@pytest.mark.skipif( + not HAS_TILELANG_MHC, + reason="TileLang MHC support required", +) +@pytest.mark.parametrize("num_tokens", [1, 4, 8, 128]) +@pytest.mark.parametrize("hidden_size", [4096, 7168]) +@pytest.mark.parametrize("hc_mult", [4]) +def test_hc_head_tilelang(num_tokens, hidden_size, hc_mult): + torch.set_default_device(DEVICE) + set_random_seed(0) + + residual = torch.randn((num_tokens, hc_mult, hidden_size), dtype=torch.bfloat16) + fn = torch.randn((hc_mult, hc_mult * hidden_size), dtype=torch.float32) * 1e-4 + hc_scale = torch.randn((1,), dtype=torch.float32) * 0.1 + hc_base = torch.randn((hc_mult,), dtype=torch.float32) * 0.1 + rms_eps = hc_eps = 1e-6 + + out = torch.ops.vllm.hc_head_fused_kernel_tilelang( + residual, + fn, + hc_scale, + hc_base, + rms_eps, + hc_eps, + ) + + assert out.shape == (num_tokens, hidden_size) + assert out.dtype == torch.bfloat16 + assert not torch.isnan(out).any() + + out_ref = hc_head_ref(residual, fn, hc_scale, hc_base, rms_eps, hc_eps) + torch.testing.assert_close(out, out_ref, atol=5e-2, rtol=1e-2) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_minimax_m3_amd_ops.py b/asian_ai/performance/vLLM/tests/kernels/test_minimax_m3_amd_ops.py new file mode 100644 index 0000000000000000000000000000000000000000..d195b7f5a07f8198b4317c017f0a45d74e6ec546 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_minimax_m3_amd_ops.py @@ -0,0 +1,337 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Reference-vs-optimized unit tests for the MiniMax-M3 AMD/ROCm fused kernels. + +Each optimized kernel added for the ROCm port has a slow PyTorch reference; the +tests assert the two agree within tolerance: + + * Gemma RMSNorm (plain + fused-add-residual) -> fp32 PyTorch normalize + * SwiGLU-OAI (split layout) -> fp32 PyTorch elementwise + * Fused MXFP8 activation quant (Triton) -> _mxfp8_e4m3_quantize_torch + * Native MXFP8 linear (dot_scaled) -> dequant-to-bf16 @ matmul + * Native MXFP8 MoE (dot_scaled grouped GEMM) -> dequant-to-bf16 MoE math + +The native MXFP8 GEMMs also guard the ``dot_scaled`` rhs-scale orientation: the +scale is loaded ``[N, K//32]`` and passed WITHOUT transpose; a stray ``.T`` +makes the shape ``[K//32, N]`` and Triton raises before producing output, so any +regression there fails these tests loudly. + +Hardware scope: the whole module is ROCm-only (these are the AMD path; NVIDIA +uses the FlashInfer kernels). The norm/activation/quant kernels run on any ROCm +arch; the native MXFP8 ``dot_scaled`` linear/MoE tests are additionally gated to +CDNA4 gfx95x (``@requires_gfx950``) since gfx942 uses the BF16 emulation path. + +Run: pytest tests/kernels/test_minimax_m3_amd_ops.py -v +""" + +import pytest +import torch + +from vllm.platforms import current_platform + +if not current_platform.is_rocm(): + pytest.skip("MiniMax-M3 AMD fused ops require ROCm.", allow_module_level=True) +if not torch.cuda.is_available(): + pytest.skip("Requires a GPU.", allow_module_level=True) + +from vllm.model_executor.layers.quantization.utils.mxfp8_utils import ( # noqa: E402 + _mxfp8_e4m3_quantize_torch, + _mxfp8_e4m3_quantize_triton, + dequant_mxfp8_to_bf16, +) +from vllm.models.minimax_m3.amd.ops import ( # noqa: E402 + gemma_fused_add_rmsnorm, + gemma_rmsnorm, + swiglu_oai_split, +) +from vllm.models.minimax_m3.amd.ops.gemma_rmsnorm import _num_warps # noqa: E402 + +DEVICE = "cuda" +EPS = 1e-6 + + +def _gcn_arch() -> str: + try: + return torch.cuda.get_device_properties(0).gcnArchName + except Exception: # pragma: no cover - no device / non-AMD + return "" + + +# The pure-Triton norm/activation/quant kernels run on any ROCm arch (CDNA3 +# gfx942 and CDNA4 gfx950). The native MXFP8 ``dot_scaled`` GEMMs (linear + MoE) +# use CDNA4 hardware microscaling and are gated to gfx95x in the source +# (``RocmDotScaledMxfp8LinearKernel.is_supported``; the MoE oracle routes gfx942 +# to the BF16 emulation path instead) — so those tests are gfx950-only. +requires_gfx950 = pytest.mark.skipif( + "gfx95" not in _gcn_arch(), + reason="native MXFP8 dot_scaled is a CDNA4 (gfx95x) feature; " + "gfx942 uses the BF16 emulation path instead.", +) + + +def _relerr(a: torch.Tensor, b: torch.Tensor) -> float: + a = a.float() + b = b.float() + return ((a - b).norm() / (b.norm() + 1e-8)).item() + + +# --------------------------------------------------------------------------- # +# Gemma RMSNorm +# --------------------------------------------------------------------------- # +def _ref_gemma_rmsnorm(x, w, eps, residual=None): + orig_dtype = x.dtype + xf = x.float() + res_out = None + if residual is not None: + xf = xf + residual.float() + res_out = xf.to(orig_dtype) + xf = xf * torch.rsqrt(xf.pow(2).mean(dim=-1, keepdim=True) + eps) + xf = xf * (1.0 + w.float()) + out = xf.to(orig_dtype) + return out if residual is None else (out, res_out) + + +@pytest.mark.parametrize("shape", [(1, 4096), (37, 6144), (128, 2048)]) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) +@pytest.mark.parametrize("seed", [0, 1234]) +@torch.inference_mode() +def test_gemma_rmsnorm(shape, dtype, seed): + torch.manual_seed(seed) + x = torch.randn(*shape, device=DEVICE, dtype=dtype) + w = torch.randn(shape[-1], device=DEVICE, dtype=dtype) * 0.1 + got = gemma_rmsnorm(x, w, EPS) + ref = _ref_gemma_rmsnorm(x, w, EPS) + assert got.shape == x.shape + assert _relerr(got, ref) < 5e-3 + + +@pytest.mark.parametrize("shape", [(1, 6144), (64, 4096)]) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) +@torch.inference_mode() +def test_gemma_fused_add_rmsnorm(shape, dtype): + torch.manual_seed(0) + x = torch.randn(*shape, device=DEVICE, dtype=dtype) + res = torch.randn(*shape, device=DEVICE, dtype=dtype) + w = torch.randn(shape[-1], device=DEVICE, dtype=dtype) * 0.1 + got_out, got_res = gemma_fused_add_rmsnorm(x, res, w, EPS) + ref_out, ref_res = _ref_gemma_rmsnorm(x, w, EPS, residual=res) + assert _relerr(got_out, ref_out) < 5e-3 + # residual_out is the pre-norm sum (x + res): bit-for-bit identical cast. + assert torch.equal(got_res, ref_res) + + +@torch.inference_mode() +def test_gemma_rmsnorm_per_head_strided(): + """q_norm/k_norm normalize a non-contiguous ``qkv.split`` slice over head_dim.""" + torch.manual_seed(0) + T, H, D, kv = 7, 48, 128, 8 + total = (H + 2 * kv) * D + qkv = torch.randn(T, total, device=DEVICE, dtype=torch.bfloat16) + q = qkv[..., : H * D] # non-contiguous view (row stride == total) + q_by_head = q.view(T, H, D) + assert not q_by_head.is_contiguous() + w = torch.randn(D, device=DEVICE, dtype=torch.bfloat16) * 0.1 + got = gemma_rmsnorm(q_by_head, w, EPS) + ref = _ref_gemma_rmsnorm(q_by_head, w, EPS) + assert got.shape == q_by_head.shape + assert _relerr(got, ref) < 5e-3 + + +def test_num_warps_monotonic(): + assert _num_warps(128) <= _num_warps(2048) <= _num_warps(8192) + + +# --------------------------------------------------------------------------- # +# SwiGLU-OAI (split layout) +# --------------------------------------------------------------------------- # +def _ref_swiglu(gate_up, alpha, beta, limit): + d = gate_up.shape[-1] // 2 + gate = gate_up[..., :d].float() + up = gate_up[..., d:].float() + if limit is not None: + gate = gate.clamp(max=limit) + up = up.clamp(min=-limit, max=limit) + return (gate * torch.sigmoid(alpha * gate) * (up + beta)).to(gate_up.dtype) + + +@pytest.mark.parametrize("m,inter", [(1, 768), (64, 1536), (128, 1024)]) +@pytest.mark.parametrize("limit", [7.0, None]) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) +@torch.inference_mode() +def test_swiglu_oai_split(m, inter, limit, dtype): + torch.manual_seed(0) + gate_up = torch.randn(m, 2 * inter, device=DEVICE, dtype=dtype) + got = swiglu_oai_split(gate_up, alpha=1.702, beta=1.0, limit=limit) + ref = _ref_swiglu(gate_up, 1.702, 1.0, limit) + assert got.shape == (m, inter) + assert _relerr(got, ref) < 5e-3 + + +# --------------------------------------------------------------------------- # +# Fused MXFP8 activation quant (Triton vs torch reference) +# --------------------------------------------------------------------------- # +@pytest.mark.parametrize("shape", [(64, 4096), (1, 6144), (333, 2048)]) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) +@torch.inference_mode() +def test_mxfp8_quant_triton_matches_torch(shape, dtype): + torch.manual_seed(0) + x = torch.randn(*shape, device=DEVICE, dtype=dtype) + xq_t, s_t = _mxfp8_e4m3_quantize_torch(x, is_sf_swizzled_layout=False) + xq_k, s_k = _mxfp8_e4m3_quantize_triton(x) + assert s_k.shape == s_t.shape == (shape[0], shape[1] // 32) + # E8M0 block exponents share the floor(log2(amax))+127 algorithm; allow at + # most a 1-step difference at exact powers of two. + assert (s_k.int() - s_t.int()).abs().max().item() <= 1 + # Dequantized values agree to fp8 granularity. + deq_t = dequant_mxfp8_to_bf16(xq_t, s_t) + deq_k = dequant_mxfp8_to_bf16(xq_k, s_k) + assert _relerr(deq_k, deq_t) < 1e-2 + + +# --------------------------------------------------------------------------- # +# Native MXFP8 linear (dot_scaled) vs dequant-to-bf16 matmul +# --------------------------------------------------------------------------- # +@requires_gfx950 +@pytest.mark.parametrize("m,n,k", [(64, 256, 128), (37, 512, 256), (1, 6144, 4096)]) +@torch.inference_mode() +def test_mxfp8_native_linear(m, n, k): + from vllm.model_executor.kernels.linear.mxfp8.rocm_native import ( + _mxfp8_dot_scaled_linear, + ) + + torch.manual_seed(0) + w_bf16 = torch.randn(n, k, device=DEVICE, dtype=torch.bfloat16) * 0.1 + w_fp8, w_scale = _mxfp8_e4m3_quantize_torch(w_bf16, is_sf_swizzled_layout=False) + x = torch.randn(m, k, device=DEVICE, dtype=torch.bfloat16) * 0.5 + + got = _mxfp8_dot_scaled_linear(x, w_fp8, w_scale) + # Reference: consume the SAME quantized weights (isolates activation-quant + # noise) -> dequant to bf16, plain matmul. + w_deq = dequant_mxfp8_to_bf16(w_fp8, w_scale) + ref = torch.nn.functional.linear(x, w_deq).to(x.dtype) + assert got.shape == (m, n) + # Only the activation is re-quantized inside the kernel -> small MX noise. + assert _relerr(got, ref) < 5e-2 + + +# --------------------------------------------------------------------------- # +# Native MXFP8 MoE (dot_scaled grouped GEMM) vs dequant-to-bf16 MoE math +# --------------------------------------------------------------------------- # +def _ref_moe(x, w13, w2, topk_weights, topk_ids, alpha, beta, limit): + T, H = x.shape + inter = w2.shape[-1] + top_k = topk_ids.shape[1] + out = torch.zeros(T, H, device=x.device, dtype=torch.float32) + for t in range(T): + for j in range(top_k): + e = int(topk_ids[t, j].item()) + g1 = x[t].float() @ w13[e].float().T # [2I] + gate = g1[:inter] + up = g1[inter:] + if limit is not None: + gate = gate.clamp(max=limit) + up = up.clamp(min=-limit, max=limit) + act = gate * torch.sigmoid(alpha * gate) * (up + beta) + g2 = act @ w2[e].float().T # [H] + out[t] += topk_weights[t, j].float() * g2 + return out.to(x.dtype) + + +@requires_gfx950 +@pytest.mark.parametrize( + "T,H,inter,E,top_k", [(8, 256, 512, 8, 2), (1, 512, 256, 16, 4)] +) +@torch.inference_mode() +def test_mxfp8_native_moe(T, H, inter, E, top_k): + from vllm.model_executor.layers.fused_moe.experts.mxfp8_native_moe import ( + fused_moe_mxfp8_native, + ) + + torch.manual_seed(0) + alpha, beta, limit = 1.702, 1.0, 7.0 + w13_bf16 = torch.randn(E, 2 * inter, H, device=DEVICE, dtype=torch.bfloat16) * 0.1 + w2_bf16 = torch.randn(E, H, inter, device=DEVICE, dtype=torch.bfloat16) * 0.1 + w13_fp8, w13_scale = _mxfp8_e4m3_quantize_torch( + w13_bf16, is_sf_swizzled_layout=False + ) + w2_fp8, w2_scale = _mxfp8_e4m3_quantize_torch(w2_bf16, is_sf_swizzled_layout=False) + + x = torch.randn(T, H, device=DEVICE, dtype=torch.bfloat16) * 0.5 + logits = torch.randn(T, E, device=DEVICE, dtype=torch.float32) + topk_weights, topk_ids = logits.softmax(dim=-1).topk(top_k, dim=-1) + topk_weights = topk_weights.to(torch.float32) + topk_ids = topk_ids.to(torch.int32) + + got = fused_moe_mxfp8_native( + x, + w13_fp8, + w13_scale, + w2_fp8, + w2_scale, + topk_weights, + topk_ids, + alpha=alpha, + beta=beta, + limit=limit, + global_num_experts=E, + expert_map=None, + ) + # Reference consumes the dequantized weights (same bits the kernel reads). + w13_deq = dequant_mxfp8_to_bf16(w13_fp8, w13_scale) + w2_deq = dequant_mxfp8_to_bf16(w2_fp8, w2_scale) + ref = _ref_moe(x, w13_deq, w2_deq, topk_weights, topk_ids, alpha, beta, limit) + assert got.shape == (T, H) + assert _relerr(got, ref) < 5e-2 + + +# --------------------------------------------------------------------------- # +# MXFP8 linear emulation: BF16-at-load (default) vs per-step dequant + switch +# --------------------------------------------------------------------------- # +@pytest.mark.parametrize("shape", [(512, 2048), (1, 6144)]) +@pytest.mark.parametrize("act_dtype", [torch.bfloat16, torch.float16]) +@pytest.mark.parametrize("dequant_at_load", [True, False]) +@torch.inference_mode() +def test_mxfp8_linear_emulation_bf16_at_load( + shape, act_dtype, dequant_at_load, monkeypatch +): + """EmulationMxfp8LinearKernel load-time BF16 dequant (default) and the + ``VLLM_MXFP8_EMULATION_DEQUANT_AT_LOAD=0`` per-step fallback must produce the + same result; the dtype-match (BF16/FP16 activations) must also hold.""" + from vllm.model_executor.kernels.linear.mxfp8.emulation import ( + EmulationMxfp8LinearKernel, + ) + from vllm.model_executor.kernels.linear.mxfp8.Mxfp8LinearKernel import ( + Mxfp8LinearLayerConfig, + ) + + monkeypatch.setenv( + "VLLM_MXFP8_EMULATION_DEQUANT_AT_LOAD", "1" if dequant_at_load else "0" + ) + N, K = shape + torch.manual_seed(0) + w_bf16 = torch.randn(N, K, device=DEVICE, dtype=torch.bfloat16) + w_fp8, w_scale = _mxfp8_e4m3_quantize_torch(w_bf16, is_sf_swizzled_layout=False) + assert w_scale.shape == (N, K // 32) + + # Reference: dequant once, plain linear in the activation dtype. + w_ref = dequant_mxfp8_to_bf16(w_fp8, w_scale).to(act_dtype) + x = torch.randn(7, K, device=DEVICE, dtype=act_dtype) + out_ref = torch.nn.functional.linear(x, w_ref) + + layer = torch.nn.Module() + layer.weight = torch.nn.Parameter(w_fp8.clone(), requires_grad=False) + layer.weight_scale = torch.nn.Parameter(w_scale.clone(), requires_grad=False) + + kernel = EmulationMxfp8LinearKernel(Mxfp8LinearLayerConfig()) + kernel.process_weights_after_loading(layer) + + if dequant_at_load: + # weights converted to BF16 at load (>= 2-byte) + assert layer.weight.element_size() >= 2 + else: + # opt-out: weights stay 1-byte MXFP8, dequant happens per-step + assert layer.weight.element_size() == 1 + + out = kernel.apply_weights(layer, x) + assert out.dtype == act_dtype # dtype-match preserved (no tl.dot/F.linear crash) + assert _relerr(out.float(), out_ref.float()) < 2e-2 diff --git a/asian_ai/performance/vLLM/tests/kernels/test_onednn.py b/asian_ai/performance/vLLM/tests/kernels/test_onednn.py new file mode 100644 index 0000000000000000000000000000000000000000..cdb5571184aebc8627b1a803925503e1f8c5356a --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_onednn.py @@ -0,0 +1,214 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +from tests.kernels.utils import to_int8 +from vllm import _custom_ops as ops +from vllm.platforms import current_platform + +if not current_platform.is_cpu(): + pytest.skip("skipping CPU-only tests", allow_module_level=True) + +NK_FACTORS = [ + (256, 128), + (4096, 4096), + (16384, 4096), + (1023, 491), + (1001, 15), +] +M_FACTORS = [ + (16, 1, 32, 128, 64), + (1, 17, 1, 31, 17), +] +CACHE_SIZES = [2] +DTYPE = [torch.bfloat16] + + +def rand_int8(shape: tuple, device: str = "cpu"): + return to_int8(torch.rand(shape, device=device) * 255 - 128) + + +def ref_int8_scaled_mm( + a: torch.Tensor, + b: torch.Tensor, + scale_a: torch.Tensor, + scale_b: torch.Tensor, + azp: torch.Tensor | None, + bias: torch.Tensor | None, + output_type: torch.dtype, +): + if azp is not None: + a = a.to(dtype=torch.float32) - azp.to(dtype=torch.float32) + output = torch.mm( + (scale_a * a.to(dtype=torch.float32)), (scale_b * b.to(dtype=torch.float32)) + ) + if bias is not None: + output += bias.float() + + return output.to(dtype=output_type) + + +def onednn_int8_gemm_test_helper( + primitive_cache_size: int, + m: int, + n: int, + k: int, + per_tensor_a_quant: bool, + per_tensor_b_quant: bool, + use_azp: bool, + use_bias: bool, + out_dtype: torch.dtype = torch.bfloat16, + device: str = "cpu", +): + # Test for a oneDNN kernel with per-tensor / per-token activation + # quantization and per-tensor / per-output channel weight quantization. + a = to_int8(torch.randn((m, k), device=device) * 5) + b = to_int8(torch.randn((n, k), device=device).t() * 5) + + a_scales_shape = (1, 1) if per_tensor_a_quant else (m, 1) + b_scales_shape = (1, 1) if per_tensor_b_quant else (1, n) + + scale_a = torch.randn(a_scales_shape, device=device, dtype=torch.float32) + scale_b = torch.randn(b_scales_shape, device=device, dtype=torch.float32) + + if use_azp: + azp = torch.rand(a_scales_shape, dtype=torch.float32) * 10 + 1.5 + azp = (azp / scale_a).round().to(dtype=torch.int32) + azp_adj = scale_b * b.sum(dim=0, keepdim=True, dtype=torch.float32) + else: + azp = None + azp_adj = None + + bias = torch.rand((n,), device=device, dtype=out_dtype) * 10 if use_bias else None + + handler = ops.create_onednn_scaled_mm( + b, + scale_b, + out_dtype, + not per_tensor_a_quant, + use_azp, + primitive_cache_size, + ) + + out = torch.zeros((m, n), dtype=out_dtype) + ops.onednn_scaled_mm(handler, a, out, scale_a, azp, azp_adj, bias) + baseline = ref_int8_scaled_mm(a, b, scale_a, scale_b, azp, bias, out_dtype) + + torch.testing.assert_close(out, baseline, rtol=1e-1, atol=1e0) + + if use_bias: + # To test runtime bias setting + out = torch.zeros((m, n), dtype=out_dtype) + ops.onednn_scaled_mm(handler, a, out, scale_a, azp, azp_adj, None) + baseline = ref_int8_scaled_mm(a, b, scale_a, scale_b, azp, None, out_dtype) + + torch.testing.assert_close(out, baseline, rtol=1e-1, atol=1e0) + + +def onednn_gemm_test_helper( + primitive_cache_size: int, + m: int, + n: int, + k: int, + use_bias: bool, + use_stride: bool, + dtype: torch.dtype = torch.bfloat16, + device: str = "cpu", +): + if use_stride: + a = torch.rand((m, 2 * k), dtype=dtype, device=device) * 1.5 + a = a[:, :k] + else: + a = torch.rand((m, k), dtype=dtype, device=device) * 1.5 + + b = torch.rand((n, k), dtype=dtype, device=device) * 1.5 + + if use_bias: + bias = torch.rand((n,), device=device, dtype=dtype) * 5 + bias_f32 = bias.float() + else: + bias = None + bias_f32 = None + + handler = ops.create_onednn_mm( + b.t(), + primitive_cache_size, + ) + + out = ops.onednn_mm(handler, a, bias) + baseline = torch.nn.functional.linear(a.float(), b.float(), bias_f32).to( + dtype=a.dtype + ) + + torch.testing.assert_close(out, baseline) + + if use_bias: + # To test runtime bias setting + out = ops.onednn_mm(handler, a, None) + baseline = torch.nn.functional.linear(a.float(), b.float(), None).to( + dtype=a.dtype + ) + + torch.testing.assert_close(out, baseline) + + +@pytest.mark.parametrize("n,k", NK_FACTORS) +@pytest.mark.parametrize("m_list", M_FACTORS) +@pytest.mark.parametrize("per_tensor_a_scale", [True, False]) +@pytest.mark.parametrize("per_tensor_b_scale", [True, False]) +@pytest.mark.parametrize("use_bias", [True, False]) +@pytest.mark.parametrize("use_azp", [True, False]) +@pytest.mark.parametrize("output_type", DTYPE) +@pytest.mark.parametrize("primitive_cache_size", CACHE_SIZES) +def test_onednn_int8_scaled_gemm( + n: int, + k: int, + m_list: tuple[int, ...], + per_tensor_a_scale: bool, + per_tensor_b_scale: bool, + use_bias: bool, + use_azp: bool, + output_type: torch.dtype, + primitive_cache_size: int, +): + for m in m_list: + onednn_int8_gemm_test_helper( + primitive_cache_size=primitive_cache_size, + m=m, + n=n, + k=k, + per_tensor_a_quant=per_tensor_a_scale, + per_tensor_b_quant=per_tensor_b_scale, + use_bias=use_bias, + use_azp=use_azp, + out_dtype=output_type, + ) + + +@pytest.mark.parametrize("n,k", NK_FACTORS) +@pytest.mark.parametrize("m_list", M_FACTORS) +@pytest.mark.parametrize("use_bias", [True, False]) +@pytest.mark.parametrize("use_stride", [True, False]) +@pytest.mark.parametrize("dtype", DTYPE) +@pytest.mark.parametrize("primitive_cache_size", CACHE_SIZES) +def test_onednn_gemm( + n: int, + k: int, + m_list: tuple[int, ...], + use_bias: bool, + use_stride: bool, + dtype: torch.dtype, + primitive_cache_size: int, +): + for m in m_list: + onednn_gemm_test_helper( + primitive_cache_size=primitive_cache_size, + m=m, + n=n, + k=k, + use_bias=use_bias, + use_stride=use_stride, + dtype=dtype, + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/test_shuffle_rows.py b/asian_ai/performance/vLLM/tests/kernels/test_shuffle_rows.py new file mode 100644 index 0000000000000000000000000000000000000000..c61cda78fdece0c91c95bc4e3d80eca40a80cbf1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_shuffle_rows.py @@ -0,0 +1,260 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Tests for the shuffle_rows function + +Run `pytest tests/kernels/test_shuffle_rows.py`. +""" + +import pytest +import torch + +from vllm._custom_ops import shuffle_rows +from vllm.platforms import current_platform + + +@pytest.mark.parametrize("num_tokens", [1, 16, 64, 128, 256, 512, 1024]) +@pytest.mark.parametrize("hidden_size", [128, 256, 512, 1024, 2048, 4096]) +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32]) +def test_shuffle_rows_basic(num_tokens: int, hidden_size: int, dtype: torch.dtype): + """Test basic functionality of shuffle_rows with various tensor sizes and + dtypes.""" + if not current_platform.is_cuda(): + pytest.skip("shuffle_rows requires CUDA") + + # Create input tensor + input_tensor = torch.randn(num_tokens, hidden_size, device="cuda", dtype=dtype) + + # Create a simple permutation map (identity mapping) + dst2src_map = torch.arange(num_tokens, device="cuda", dtype=torch.int32) + + # Test shuffle_rows + output = shuffle_rows(input_tensor, dst2src_map) + + # With identity mapping, output should be identical to input + torch.testing.assert_close(output, input_tensor, atol=0, rtol=0) + + # Check output shape + assert output.shape == (num_tokens, hidden_size) + assert output.dtype == dtype + assert output.device == input_tensor.device + + +@pytest.mark.parametrize("num_tokens", [16, 64, 128]) +@pytest.mark.parametrize("hidden_size", [128, 512, 1024]) +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +def test_shuffle_rows_permutation( + num_tokens: int, hidden_size: int, dtype: torch.dtype +): + """Test shuffle_rows with actual permutation.""" + if not current_platform.is_cuda(): + pytest.skip("shuffle_rows requires CUDA") + + # Create input tensor + input_tensor = torch.randn(num_tokens, hidden_size, device="cuda", dtype=dtype) + + # Create a reverse permutation map + dst2src_map = torch.arange(num_tokens - 1, -1, -1, device="cuda", dtype=torch.int32) + + # Test shuffle_rows + output = shuffle_rows(input_tensor, dst2src_map) + + # Check that the output is the reverse of the input + expected_output = torch.flip(input_tensor, dims=[0]) + torch.testing.assert_close(output, expected_output, atol=1e-6, rtol=1e-5) + + # Check output shape and properties + assert output.shape == (num_tokens, hidden_size) + assert output.dtype == dtype + assert output.device == input_tensor.device + + +@pytest.mark.parametrize("num_tokens", [32, 64]) +@pytest.mark.parametrize("hidden_size", [256, 512]) +def test_shuffle_rows_expansion(num_tokens: int, hidden_size: int): + """Test shuffle_rows with expansion (more output tokens than input + tokens).""" + if not current_platform.is_cuda(): + pytest.skip("shuffle_rows requires CUDA") + + dtype = torch.float16 + + # Create input tensor + input_tensor = torch.randn(num_tokens, hidden_size, device="cuda", dtype=dtype) + + # Create a mapping that duplicates some tokens (expansion) + expanded_size = num_tokens * 2 + dst2src_map = torch.randint( + 0, num_tokens, (expanded_size,), device="cuda", dtype=torch.int32 + ) + + # Test shuffle_rows + output = shuffle_rows(input_tensor, dst2src_map) + + # Check output shape + assert output.shape == (expanded_size, hidden_size) + assert output.dtype == dtype + assert output.device == input_tensor.device + + # Verify that each output row matches the corresponding input row + for i in range(expanded_size): + src_idx = dst2src_map[i].item() + torch.testing.assert_close( + output[i], input_tensor[src_idx], atol=1e-6, rtol=1e-5 + ) + + +@pytest.mark.parametrize("num_tokens", [16, 64]) +@pytest.mark.parametrize("hidden_size", [128, 512]) +def test_shuffle_rows_random_permutation(num_tokens: int, hidden_size: int): + """Test shuffle_rows with random permutation.""" + if not current_platform.is_cuda(): + pytest.skip("shuffle_rows requires CUDA") + + dtype = torch.float16 + + # Set seed for reproducibility + torch.manual_seed(42) + + # Create input tensor + input_tensor = torch.randn(num_tokens, hidden_size, device="cuda", dtype=dtype) + + # Create a random permutation map + dst2src_map = torch.randperm(num_tokens, device="cuda", dtype=torch.int32) + + # Test shuffle_rows + output = shuffle_rows(input_tensor, dst2src_map) + + # Check output shape and properties + assert output.shape == (num_tokens, hidden_size) + assert output.dtype == dtype + assert output.device == input_tensor.device + + # Verify that each output row matches the corresponding input row + for i in range(num_tokens): + src_idx = dst2src_map[i].item() + torch.testing.assert_close( + output[i], input_tensor[src_idx], atol=1e-6, rtol=1e-5 + ) + + +def test_shuffle_rows_edge_cases(): + """Test shuffle_rows with edge cases.""" + if not current_platform.is_cuda(): + pytest.skip("shuffle_rows requires CUDA") + + dtype = torch.float16 + + # Test with single token + input_tensor = torch.randn(1, 128, device="cuda", dtype=dtype) + dst2src_map = torch.tensor([0], device="cuda", dtype=torch.int32) + output = shuffle_rows(input_tensor, dst2src_map) + torch.testing.assert_close(output, input_tensor, atol=0, rtol=0) + + # Test with single feature dimension + input_tensor = torch.randn(16, 1, device="cuda", dtype=dtype) + dst2src_map = torch.arange(16, device="cuda", dtype=torch.int32) + output = shuffle_rows(input_tensor, dst2src_map) + torch.testing.assert_close(output, input_tensor, atol=0, rtol=0) + + +def test_shuffle_rows_moe_like_scenario(): + """Test shuffle_rows in a scenario similar to MoE usage.""" + if not current_platform.is_cuda(): + pytest.skip("shuffle_rows requires CUDA") + + dtype = torch.float16 + batch_size = 32 + hidden_size = 1024 + topk = 2 + + # Simulate input tokens + input_tensor = torch.randn(batch_size, hidden_size, device="cuda", dtype=dtype) + + # Simulate expert assignment (each token goes to topk experts) + # This creates a mapping where tokens are duplicated for multiple experts + total_tokens = batch_size * topk + dst2src_map = torch.zeros(total_tokens, device="cuda", dtype=torch.int32) + + # Fill the mapping to simulate MoE token distribution + for i in range(batch_size): + for k in range(topk): + dst2src_map[i * topk + k] = i + + # Test shuffle_rows + output = shuffle_rows(input_tensor, dst2src_map) + + # Check output shape + assert output.shape == (total_tokens, hidden_size) + assert output.dtype == dtype + assert output.device == input_tensor.device + + # Verify that tokens are correctly duplicated + for i in range(batch_size): + for k in range(topk): + output_idx = i * topk + k + torch.testing.assert_close( + output[output_idx], input_tensor[i], atol=1e-6, rtol=1e-5 + ) + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32]) +def test_shuffle_rows_dtype_consistency(dtype: torch.dtype): + """Test that shuffle_rows preserves dtype correctly.""" + if not current_platform.is_cuda(): + pytest.skip("shuffle_rows requires CUDA") + + num_tokens = 64 + hidden_size = 512 + + # Create input tensor with specific dtype + input_tensor = torch.randn(num_tokens, hidden_size, device="cuda", dtype=dtype) + dst2src_map = torch.arange(num_tokens, device="cuda", dtype=torch.int32) + + # Test shuffle_rows + output = shuffle_rows(input_tensor, dst2src_map) + + # Verify dtype is preserved + assert output.dtype == dtype + assert output.device == input_tensor.device + torch.testing.assert_close(output, input_tensor, atol=1e-6, rtol=1e-5) + + +def test_shuffle_rows_device_consistency(): + """Test that shuffle_rows maintains device consistency.""" + if not current_platform.is_cuda(): + pytest.skip("shuffle_rows requires CUDA") + + num_tokens = 32 + hidden_size = 256 + dtype = torch.float16 + + # Create input tensor on CUDA + input_tensor = torch.randn(num_tokens, hidden_size, device="cuda", dtype=dtype) + dst2src_map = torch.arange(num_tokens, device="cuda", dtype=torch.int32) + + # Test shuffle_rows + output = shuffle_rows(input_tensor, dst2src_map) + + # Verify device is maintained + assert output.device == input_tensor.device + assert output.device.type == "cuda" + + +def test_shuffle_rows_contiguous_output(): + """Test that shuffle_rows produces contiguous output.""" + if not current_platform.is_cuda(): + pytest.skip("shuffle_rows requires CUDA") + + num_tokens = 64 + hidden_size = 512 + dtype = torch.float16 + + # Create input tensor + input_tensor = torch.randn(num_tokens, hidden_size, device="cuda", dtype=dtype) + dst2src_map = torch.arange(num_tokens, device="cuda", dtype=torch.int32) + + # Test shuffle_rows + output = shuffle_rows(input_tensor, dst2src_map) + + # Verify output is contiguous + assert output.is_contiguous() diff --git a/asian_ai/performance/vLLM/tests/kernels/test_top_k_per_row.py b/asian_ai/performance/vLLM/tests/kernels/test_top_k_per_row.py new file mode 100644 index 0000000000000000000000000000000000000000..ff1799645b0a59a788087658b694344707be9b78 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/test_top_k_per_row.py @@ -0,0 +1,970 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import numpy as np +import pytest +import torch + +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +# Test parameters +NUM_ROWS = [1, 32, 2050] +TOP_K_VALUES = [2048, 3000] +BATCH_SIZE = [1, 2, 2048] +NEXT_N = [1, 8] +DATA_GENERATION = ["random", "10LSBits"] +RADIX_TOPK_WORKSPACE_SIZE = 1024 * 1024 + + +def _has_device_capability(major: int) -> bool: + return current_platform.is_cuda() and current_platform.has_device_capability(major) + + +COOPERATIVE_TOPK_BACKEND = pytest.param( + "cooperative_topk", + marks=pytest.mark.skipif( + not _has_device_capability(90), + reason="cooperative_topk requires SM90+", + ), +) +WORKSPACE_TOPK_BACKENDS = ["persistent_topk", COOPERATIVE_TOPK_BACKEND] +TOPK_BACKENDS = ["top_k_per_row_decode", *WORKSPACE_TOPK_BACKENDS] + + +def _run_topk_backend( + backend: str, + logits: torch.Tensor, + lengths: torch.Tensor, + indices: torch.Tensor, + top_k: int, + max_seq_len: int, + next_n: int = 1, +) -> None: + if backend == "top_k_per_row_decode": + torch.ops._C.top_k_per_row_decode( + logits, + next_n, + lengths, + indices, + indices.shape[0], + logits.stride(0), + logits.stride(1), + top_k, + ) + elif backend == "persistent_topk": + workspace = torch.empty( + RADIX_TOPK_WORKSPACE_SIZE, dtype=torch.uint8, device="cuda" + ) + torch.ops._C.persistent_topk( + logits, lengths, indices, workspace, top_k, max_seq_len + ) + elif backend == "cooperative_topk": + if indices.shape[0] > 32: + pytest.skip( + "cooperative_topk supports <=32 rows; " + "persistent_topk covers larger batches" + ) + if logits.stride(0) % 4 != 0: + pytest.skip( + "cooperative_topk requires row stride divisible by 4; " + "persistent_topk covers unaligned strides" + ) + workspace = torch.empty( + RADIX_TOPK_WORKSPACE_SIZE, dtype=torch.uint8, device="cuda" + ) + torch.ops._C.cooperative_topk( + logits, lengths, indices, workspace, top_k, max_seq_len + ) + else: + raise ValueError(f"Unknown top-k backend: {backend}") + + +def create_random_logits( + row_starts: torch.Tensor, + row_ends: torch.Tensor, + dtype: torch.dtype, + seed: int, + clean_logits: bool, + data_generation: str, +) -> torch.Tensor: + """Create random logits tensor for testing.""" + torch.manual_seed(seed) + np.random.seed(seed) + # Generate logits with some structure to make testing more meaningful + if data_generation == "random": + logits = torch.randn( + row_starts.shape[0], max(row_ends), dtype=dtype, device="cuda" + ) + elif data_generation == "10LSBits": + top_22_bits_mask = 0xFFFFFC00 + last_10_bits_mask = 0x000003FF + fixed_top_22_bits = 0x3F900000 + # Generate random bits for the last 10 bits + random_bottom_bits = torch.randint( + 0, + 2**10, + (row_starts.shape[0], max(row_ends)), + dtype=torch.int32, + device="cuda", + ) + # Combine: fixed top 22 bits with random last 10 bits + logits_bits = (fixed_top_22_bits & top_22_bits_mask) | ( + random_bottom_bits & last_10_bits_mask + ) + logits = logits_bits.view(dtype) + + if clean_logits: + for i, end in enumerate(row_ends): + logits[i, end:] = float("-inf") + return logits + + +def create_row_boundaries( + seq_len: int, vocab_size: int +) -> tuple[torch.Tensor, torch.Tensor]: + """Create row start and end indices for testing.""" + row_starts = torch.zeros(seq_len, dtype=torch.int32, device="cuda") + row_ends = torch.arange(1, seq_len + 1, device="cuda", dtype=torch.int32) + return row_starts, row_ends + + +def compare_top_k_results( + logits: torch.Tensor, + cuda_indices: torch.Tensor, + torch_indices: torch.Tensor, + row_starts: torch.Tensor, + row_ends: torch.Tensor, + top_k: int, + tolerance: float = 1e-5, +) -> bool: + """ + Compare results from CUDA top_k_per_row with torch.topk. + Both results should be sorted and contain the same top-k elements. + """ + num_rows = cuda_indices.shape[0] + + for row_idx in range(num_rows): + # Get valid elements using row boundaries + row_start = row_starts[row_idx].item() + row_end = row_ends[row_idx].item() + row_length = row_end - row_start + num_valid = min(top_k, row_length) + cuda_row_indices = cuda_indices[row_idx][:num_valid].cpu() + torch_row_indices = torch_indices[row_idx][:num_valid].cpu() + + # Compare the sets of indices first + cuda_set = set(cuda_row_indices.tolist()) + torch_set = set(torch_row_indices.tolist()) + if cuda_set == torch_set: + continue + + # Any difference in elements, compare the values + logits_row = logits[row_idx] + cuda_row_values = [logits_row[i] for i in cuda_row_indices] + torch_row_values = [logits_row[i] for i in torch_row_indices] + + cuda_only_values, torch_only_values = [], [] + for idx in cuda_set - torch_set: + cuda_pos = (cuda_row_indices == idx).nonzero(as_tuple=True)[0] + cuda_only_values.append(cuda_row_values[cuda_pos[0]]) + + for idx in torch_set - cuda_set: + torch_pos = (torch_row_indices == idx).nonzero(as_tuple=True)[0] + torch_only_values.append(torch_row_values[torch_pos[0]]) + + if len(cuda_only_values) != len(torch_only_values): + return False + if not torch.allclose( + torch.tensor(cuda_only_values), + torch.tensor(torch_only_values), + rtol=tolerance, + atol=tolerance, + ): + return False + + return True + + +def validate_topk_against_reference( + logits: torch.Tensor, + cuda_indices: torch.Tensor, + row_starts: torch.Tensor, + row_ends: torch.Tensor, + top_k: int, + kernel_name: str, +) -> None: + """ + Validate CUDA top-k results against PyTorch reference implementation. + + Args: + logits: Input logits tensor + cuda_indices: CUDA kernel output indices + row_starts: Row start positions + row_ends: Row end positions + top_k: Number of top elements to select + kernel_name: Name of the kernel being tested (for error messages) + """ + num_rows = cuda_indices.shape[0] + torch_indices = torch.empty((num_rows, top_k), dtype=torch.int32, device="cuda") + + for i in range(num_rows): + row_end = int(row_ends[i]) + k_i = min(top_k, row_end) + idx = logits[i, :row_end].topk(k_i, dim=-1)[1] + torch_indices[i, :k_i] = idx + + assert compare_top_k_results( + logits, cuda_indices, torch_indices, row_starts, row_ends, top_k + ), f"{kernel_name} results don't match torch.topk" + + +@pytest.mark.parametrize("num_rows", NUM_ROWS) +@pytest.mark.parametrize("top_k", TOP_K_VALUES) +@pytest.mark.parametrize("clean_logits", [True, False]) +@pytest.mark.skipif(not current_platform.is_cuda(), reason="This test requires CUDA") +@torch.inference_mode() +def test_top_k_per_row( + num_rows: int, + top_k: int, + clean_logits: bool, +) -> None: + """ + Test top_k_per_row. + """ + set_random_seed(0) + torch.set_default_device("cuda:0") + + # Create test data + vocab_size = 20000 + row_starts, row_ends = create_row_boundaries(num_rows, vocab_size) + logits = create_random_logits( + row_starts, row_ends, torch.float32, 42, clean_logits, "random" + ) + + # Create output tensors + indices = torch.empty((num_rows, top_k), dtype=torch.int32, device="cuda") + + # Run CUDA implementation + torch.ops._C.top_k_per_row_prefill( + logits, + row_starts, + row_ends, + indices, + num_rows, + logits.stride(0), + logits.stride(1), + top_k, + ) + + # Run reference implementation + torch_indices = torch.empty((num_rows, top_k), dtype=torch.int32, device="cuda") + for i in range(num_rows): + row_end = int(row_ends[i]) + k_i = min(top_k, row_end) + idx = logits[i, :row_end].topk(k_i, dim=-1)[1] + torch_indices[i, :k_i] = idx + + # Compare results + assert compare_top_k_results( + logits, indices, torch_indices, row_starts, row_ends, top_k + ), "CUDA top_k_per_row_prefill results don't match torch.topk" + + +def _run_top_k_per_row_decode_test( + top_k: int, + batch_size: int, + next_n: int, + vocab_size: int, + clean_logits: bool, + data_generation: str, +) -> None: + """ + Helper function to run top_k_per_row_decode test with given parameters. + """ + torch.set_default_device("cuda:0") + + # Create test data + num_rows = batch_size * next_n + seq_lens = torch.randint( + low=next_n, + high=vocab_size, + size=(batch_size,), + dtype=torch.int32, + device="cuda", + ) + row_starts = torch.zeros(num_rows, dtype=torch.int32, device="cuda") + row_indices = torch.arange(num_rows, device="cuda") // next_n + next_n_offset = torch.arange(num_rows, device="cuda") % next_n + row_ends = seq_lens[row_indices] - next_n + next_n_offset + 1 + logits = create_random_logits( + row_starts, row_ends, torch.float32, 42, clean_logits, data_generation + ) + + # Create output tensors + indices = torch.empty((num_rows, top_k), dtype=torch.int32, device="cuda") + + # Run CUDA implementation + torch.ops._C.top_k_per_row_decode( + logits, + next_n, + seq_lens, + indices, + num_rows, + logits.stride(0), + logits.stride(1), + top_k, + ) + + torch.accelerator.synchronize() + + # Run reference implementation + torch_indices = torch.empty((num_rows, top_k), dtype=torch.int32, device="cuda") + for i in range(num_rows): + row_end = int(row_ends[i]) + k_i = min(top_k, row_end) + idx = logits[i, :row_end].topk(k_i, dim=-1)[1] + torch_indices[i, :k_i] = idx + + # Compare results + assert compare_top_k_results( + logits, indices, torch_indices, row_starts, row_ends, top_k + ), "CUDA top_k_per_row_decode results don't match torch.topk" + + +@pytest.mark.parametrize("top_k", TOP_K_VALUES) +@pytest.mark.parametrize("batch_size", BATCH_SIZE) +@pytest.mark.parametrize("next_n", NEXT_N) +@pytest.mark.parametrize("clean_logits", [True, False]) +@pytest.mark.parametrize("data_generation", DATA_GENERATION) +@pytest.mark.skipif(not current_platform.is_cuda(), reason="This test requires CUDA") +@torch.inference_mode() +def test_top_k_per_row_decode( + top_k: int, + batch_size: int, + next_n: int, + clean_logits: bool, + data_generation: str, +) -> None: + """ + Test top_k_per_row with seq_lens tensor. + """ + set_random_seed(0) + vocab_size = 20000 + _run_top_k_per_row_decode_test( + top_k, batch_size, next_n, vocab_size, clean_logits, data_generation + ) + + +@pytest.mark.skipif(not current_platform.is_cuda(), reason="This test requires CUDA") +@pytest.mark.parametrize("clean_logits", [True, False]) +@torch.inference_mode() +def test_top_k_per_row_decode_large_vocab_size(clean_logits: bool) -> None: + """ + Test top_k_per_row_decode with large vocabulary size. + """ + set_random_seed(0) + top_k = 2048 + batch_size = 2 + next_n = 2 + vocab_size = 300000 + data_generation = "random" + _run_top_k_per_row_decode_test( + top_k, batch_size, next_n, vocab_size, clean_logits, data_generation + ) + + +@pytest.mark.skipif(not current_platform.is_cuda(), reason="This test requires CUDA") +@pytest.mark.parametrize( + "seq_len_range,test_id", + [ + pytest.param((4000, 8000), "short_sequences", id="short"), + pytest.param((8000, 32000), "medium_sequences", id="medium"), + pytest.param((32000, 163840), "long_sequences", id="long"), + ], +) +@pytest.mark.parametrize("clean_logits", [True, False]) +@pytest.mark.parametrize("top_k", [2048]) +@pytest.mark.parametrize("next_n", [1, 4]) +@pytest.mark.parametrize("backend", WORKSPACE_TOPK_BACKENDS) +@torch.inference_mode() +def test_deepseek_workspace_topk( + seq_len_range: tuple[int, int], + test_id: str, + clean_logits: bool, + top_k: int, + next_n: int, + backend: str, +) -> None: + """ + Test workspace top-k backends with varying sequence lengths and speculative + decoding. + Supports speculative decoding with next_n > 1. + """ + set_random_seed(42 if test_id == "short_sequences" else 43) + torch.set_default_device("cuda:0") + + batch_size = 4 + num_rows = batch_size * next_n + + seq_lens = torch.randint( + seq_len_range[0], + seq_len_range[1], + (batch_size,), + dtype=torch.int32, + device="cuda", + ) + seq_lens = (seq_lens + 3) & ~3 # align to 4 for TMA + + # Compute row boundaries for speculative decoding + row_starts = torch.zeros(num_rows, dtype=torch.int32, device="cuda") + row_indices = torch.arange(num_rows, device="cuda") // next_n + next_n_offset = torch.arange(num_rows, device="cuda") % next_n + row_ends = seq_lens[row_indices] - next_n + next_n_offset + 1 + + logits = create_random_logits( + row_starts, row_ends, torch.float32, 42, clean_logits, "random" + ) + + indices = torch.empty((num_rows, top_k), dtype=torch.int32, device="cuda") + + if next_n == 1: + lengths = seq_lens + else: + offsets = torch.arange(next_n, device=logits.device, dtype=torch.int32) + lengths = (seq_lens.unsqueeze(1) - next_n + 1 + offsets).flatten() + + max_seq_len = int(seq_lens.max().item()) + _run_topk_backend(backend, logits, lengths, indices, top_k, max_seq_len, next_n) + + validate_topk_against_reference( + logits, indices, row_starts, row_ends, top_k, f"{backend} ({test_id})" + ) + + +def run_large_context_topk_test( + batch_size: int, + seq_lens: list[int], + top_k: int, + data_type: str = "random", + seed: int = 42, + backend: str = "cooperative_topk", +) -> None: + """ + Helper to run a top-k backend test with given parameters. + + Args: + batch_size: Number of rows/sequences + seq_lens: List of sequence lengths (one per row) + top_k: Number of top elements to select + data_type: Type of test data to generate + seed: Random seed for reproducibility + backend: Top-k backend to test + """ + torch.set_default_device("cuda:0") + set_random_seed(seed) + + # Create test data + num_rows = batch_size + max_len = max(seq_lens) + lengths = torch.tensor(seq_lens, dtype=torch.int32, device="cuda") + + if data_type == "random": + logits = torch.randn(num_rows, max_len, dtype=torch.float32, device="cuda") + elif data_type == "sorted_asc": + # Each row gets its own ascending sequence based on its length + logits = torch.empty(num_rows, max_len, dtype=torch.float32, device="cuda") + for i, length in enumerate(seq_lens): + logits[i, :length] = torch.arange( + length, dtype=torch.float32, device="cuda" + ) + if length < max_len: + logits[i, length:] = float("-inf") + elif data_type == "sorted_desc": + # Each row gets its own descending sequence based on its length + logits = torch.empty(num_rows, max_len, dtype=torch.float32, device="cuda") + for i, length in enumerate(seq_lens): + logits[i, :length] = torch.arange( + length, 0, -1, dtype=torch.float32, device="cuda" + ) + if length < max_len: + logits[i, length:] = float("-inf") + elif data_type == "all_same": + logits = torch.ones(num_rows, max_len, dtype=torch.float32, device="cuda") + for i, length in enumerate(seq_lens): + if length < max_len: + logits[i, length:] = float("-inf") + elif data_type == "many_ties": + # Only 10 unique values, many duplicates + logits = torch.randint(0, 10, (num_rows, max_len), device="cuda").float() / 10.0 + for i, length in enumerate(seq_lens): + if length < max_len: + logits[i, length:] = float("-inf") + elif data_type == "small_differences": + # Very small differences to test float precision + base = torch.randn(num_rows, max_len, dtype=torch.float32, device="cuda") + noise = ( + torch.randn(num_rows, max_len, dtype=torch.float32, device="cuda") * 1e-6 + ) + logits = base + noise + for i, length in enumerate(seq_lens): + if length < max_len: + logits[i, length:] = float("-inf") + else: + raise ValueError(f"Unknown data_type: {data_type}") + + # Create output tensor + indices = torch.empty((num_rows, top_k), dtype=torch.int32, device="cuda") + + max_seq_len = max(seq_lens) + _run_topk_backend(backend, logits, lengths, indices, top_k, max_seq_len) + + torch.accelerator.synchronize() + + torch_indices = torch.empty((num_rows, top_k), dtype=torch.int32, device="cuda") + for i in range(num_rows): + length = seq_lens[i] + k_i = min(top_k, length) + if k_i > 0: + idx = logits[i, :length].topk(k_i, dim=-1)[1] + torch_indices[i, :k_i] = idx + if k_i < top_k: + torch_indices[i, k_i:] = -1 + else: + torch_indices[i, :] = -1 + + # Compare results + for i in range(num_rows): + length = seq_lens[i] + k_i = min(top_k, length) + + if k_i == 0: + continue + + cuda_row = indices[i, :k_i].cpu() + torch_row = torch_indices[i, :k_i].cpu() + + # Filter out -1 padding values from cuda_row + valid_mask = cuda_row >= 0 + cuda_row = cuda_row[valid_mask] + + # Compare sets (order may differ for ties) + cuda_set = set(cuda_row.tolist()) + torch_set = set(torch_row.tolist()) + + if cuda_set == torch_set: + continue + + # If sets differ, check if it's due to equal values (ties) + cuda_vals = logits[i, cuda_row].cpu() + torch_vals = logits[i, torch_row].cpu() + + # Check that min CUDA value >= max of values NOT in top-k + if k_i < length: + non_topk_indices = torch.tensor( + list(set(range(length)) - cuda_set), dtype=torch.int32 + ) + if len(non_topk_indices) > 0: + non_topk_vals = logits[i, non_topk_indices].cpu() + min_cuda_val = cuda_vals.min() + max_non_topk = non_topk_vals.max() + + # Allow small tolerance for floating point errors + assert min_cuda_val >= max_non_topk - 1e-4, ( + f"Row {i}: CUDA top-k contains values smaller than non-top-k. " + f"Min CUDA: {min_cuda_val}, Max non-top-k: {max_non_topk}, " + f"Length: {length}, k: {k_i}, CUDA indices: {sorted(cuda_set)[:10]}..., " # noqa: E501 + f"Expected indices: {sorted(torch_set)[:10]}..." + ) + + # For ties, verify the values are close + assert torch.allclose( + cuda_vals.sort(descending=True)[0], + torch_vals.sort(descending=True)[0], + rtol=1e-4, + atol=1e-4, + ), f"""Row {i}: Top-k values don't match. + CUDA: {cuda_vals.sort(descending=True)[0][:10]}, + Torch: {torch_vals.sort(descending=True)[0][:10]}""" + + +@pytest.mark.skipif(not current_platform.is_cuda(), reason="This test requires CUDA") +@pytest.mark.parametrize( + "test_config", + [ + # ==================== CATEGORY: Sequence Length Edge Cases ==================== + pytest.param( + {"seq_lens": [1, 10, 100, 2048], "top_k": 2048, "data_type": "random"}, + id="seq_len_edge_very_small_to_medium", + ), + pytest.param( + { + "seq_lens": [2049, 2100, 2500, 3000], + "top_k": 2048, + "data_type": "random", + }, + id="seq_len_edge_above_k", + ), + pytest.param( + {"seq_lens": [8000, 16384, 20000], "top_k": 2048, "data_type": "random"}, + id="algo_transition_filtered_radix", + ), + # ==================== CATEGORY: Data Distributions ==================== + pytest.param( + {"seq_lens": [5000, 10000], "top_k": 2048, "data_type": "sorted_asc"}, + id="data_sorted_ascending", + ), + pytest.param( + {"seq_lens": [5000, 10000], "top_k": 2048, "data_type": "sorted_desc"}, + id="data_sorted_descending", + ), + pytest.param( + {"seq_lens": [5000, 10000], "top_k": 2048, "data_type": "all_same"}, + id="data_all_same", + ), + pytest.param( + {"seq_lens": [5000, 10000], "top_k": 2048, "data_type": "many_ties"}, + id="data_many_ties", + ), + pytest.param( + { + "seq_lens": [5000, 10000], + "top_k": 2048, + "data_type": "small_differences", + }, + id="data_float_precision", + ), + # ==================== CATEGORY: Alignment / Vectorization ==================== + pytest.param( + { + "seq_lens": [2055, 2056, 2057, 2063], + "top_k": 2048, + "data_type": "random", + }, + id="align_vec_boundaries_low", + ), + pytest.param( + { + "seq_lens": [4095, 4096, 4097, 4102], + "top_k": 2048, + "data_type": "random", + }, + id="align_4k_boundary", + ), + pytest.param( + { + "seq_lens": [8191, 8192, 8193, 8198], + "top_k": 2048, + "data_type": "random", + }, + id="align_8k_boundary", + ), + pytest.param( + { + "seq_lens": [16383, 16384, 16385, 16390], + "top_k": 2048, + "data_type": "random", + }, + id="align_16k_boundary", + ), + ], +) +@pytest.mark.parametrize("backend", WORKSPACE_TOPK_BACKENDS) +@torch.inference_mode() +def test_workspace_topk_correctness(test_config: dict, backend: str) -> None: + """ + Comprehensive correctness tests covering: + - Sequence length edge cases (trivial, boundary, varied) + - Very small sequences (< 100 elements) + - Mixed sequence lengths in same batch + - Data distributions (sorted, ties, precision) + - Memory alignment / vectorization boundaries + """ + run_large_context_topk_test( + batch_size=len(test_config["seq_lens"]), + seq_lens=test_config["seq_lens"], + top_k=test_config["top_k"], + data_type=test_config.get("data_type", "random"), + backend=backend, + ) + + +@pytest.mark.skipif(not current_platform.is_cuda(), reason="This test requires CUDA") +@pytest.mark.parametrize( + "test_config", + [ + # ==================== CATEGORY: Batch Size Scalability ==================== + pytest.param( + {"batch_size": 1, "seq_len": 5000, "top_k": 2048}, + id="batch_1", + ), + pytest.param( + {"batch_size": 4, "seq_len": 5000, "top_k": 2048}, + id="batch_4", + ), + pytest.param( + {"batch_size": 32, "seq_len": 5000, "top_k": 2048}, + id="batch_32", + ), + pytest.param( + {"batch_size": 256, "seq_len": 5000, "top_k": 2048}, + id="batch_256", + ), + # ==================== CATEGORY: Single-CTA vs Multi-CTA ==================== + pytest.param( + {"batch_size": 2, "seq_len": 4096, "top_k": 2048}, + id="single_cta_4k", + ), + pytest.param( + {"batch_size": 2, "seq_len": 8192, "top_k": 2048}, + id="single_cta_8k", + ), + pytest.param( + {"batch_size": 2, "seq_len": 163840, "top_k": 2048}, + id="multi_cta_163840_dsv3_max", + ), + # ==================== CATEGORY: Extreme Cases ==================== + pytest.param( + {"batch_size": 512, "seq_len": 5000, "top_k": 2048}, + id="extreme_large_batch", + ), + pytest.param( + {"batch_size": 2, "seq_len": 163840, "top_k": 2048}, + id="extreme_dsv3_max_context", + ), + ], +) +@pytest.mark.parametrize("backend", WORKSPACE_TOPK_BACKENDS) +@torch.inference_mode() +def test_workspace_topk_algorithm_paths(test_config: dict, backend: str) -> None: + """ + Test different algorithm execution paths (capped at 163840 for DeepSeek V3.2): + - Batch size scalability (1, 4, 32, 256) + - Single-CTA vs Multi-CTA execution + - Extreme configurations (large batch, max context length) + """ + run_large_context_topk_test( + batch_size=test_config["batch_size"], + seq_lens=[test_config["seq_len"]] * test_config["batch_size"], + top_k=test_config["top_k"], + backend=backend, + ) + + +@pytest.mark.skipif(not current_platform.is_cuda(), reason="This test requires CUDA") +@pytest.mark.parametrize("backend", WORKSPACE_TOPK_BACKENDS) +@torch.inference_mode() +def test_workspace_topk_stress(backend: str) -> None: + """ + Stress test with random configurations to catch edge cases. + Capped at 163840 (DeepSeek V3.2 max context) for realistic testing. + """ + torch.set_default_device("cuda:0") + top_k = 2048 + + for seed in range(3): + set_random_seed(seed) + + # Random batch size (limited for speed) + batch_size = torch.randint(1, 32, (1,)).item() + + # Random sequence lengths capped at DeepSeek V3.2 max context + seq_lens_tensor = torch.randint(100, 163840, (batch_size,)) + if backend == "cooperative_topk": + seq_lens = ((seq_lens_tensor + 3) & ~3).tolist() + else: + seq_lens = seq_lens_tensor.tolist() + + run_large_context_topk_test( + batch_size=batch_size, + seq_lens=seq_lens, + top_k=top_k, + seed=seed, + backend=backend, + ) + + +@pytest.mark.skipif(not current_platform.is_cuda(), reason="This test requires CUDA") +@pytest.mark.parametrize("backend", TOPK_BACKENDS) +@pytest.mark.parametrize("top_k", [512, 1024, 2048]) +@torch.inference_mode() +def test_deepseek_topk_backends_no_error_and_reference( + backend: str, + top_k: int, +) -> None: + """Exercise every production top-k backend on the same inputs.""" + run_large_context_topk_test( + batch_size=4, + seq_lens=[2049, 4097, 8191, 12000], + top_k=top_k, + data_type="random", + seed=123, + backend=backend, + ) + + +@pytest.mark.skipif(not _has_device_capability(90), reason="This test requires SM90+") +@torch.inference_mode() +def test_cooperative_topk_512_tie_workspace_is_per_row() -> None: + """Regression test for TopK=512 tie workspace row overlap.""" + torch.set_default_device("cuda:0") + + top_k = 512 + num_rows = 2 + stride = 65536 + lengths = torch.tensor([40960, 65536], dtype=torch.int32, device="cuda") + logits = torch.full( + (num_rows, stride), float("-inf"), dtype=torch.float32, device="cuda" + ) + + # Row 0 must never select these low indices: many better row-0 ties exist. + logits[0, :2048] = -10.0 + logits[0, 2048 : lengths[0]] = 1.0 + # Row 1 has higher exact tie scores. With the old row * TopK tie_ws stride, + # these row-1 ties could overwrite row 0's TopK=512 refinement workspace. + logits[1, : lengths[1]] = 2.0 + + indices = torch.empty((num_rows, top_k), dtype=torch.int32, device="cuda") + workspace = torch.empty(RADIX_TOPK_WORKSPACE_SIZE, dtype=torch.uint8, device="cuda") + torch.ops._C.cooperative_topk(logits, lengths, indices, workspace, top_k, stride) + torch.accelerator.synchronize() + + row0 = indices[0].cpu() + assert torch.all(row0 >= 2048), ( + "cooperative_topk TopK=512 selected row-0 low-score indices, likely " + "from overlapping tie_ws rows" + ) + + +@pytest.mark.skipif(not current_platform.is_cuda(), reason="This test requires CUDA") +@pytest.mark.parametrize( + "test_config", + [ + # Mixed batch: rows spanning all four paths (trivial, decode, medium, large) + pytest.param( + { + "seq_lens": [2000, 6000, 30000, 80000], + "data_type": "random", + }, + id="mixed_all_paths", + ), + # All decode/medium rows (typical decode scenario) + pytest.param( + { + "seq_lens": [2048, 4096, 8192, 16000], + "data_type": "random", + }, + id="all_decode_medium", + ), + # All large rows + pytest.param( + { + "seq_lens": [70000, 100000, 163840], + "data_type": "random", + }, + id="all_large", + ), + # Boundary around LARGE_THRESHOLD (32K) + pytest.param( + { + "seq_lens": [32767, 32768, 32769, 32772], + "data_type": "random", + }, + id="large_threshold_boundary", + ), + # Single row medium + pytest.param( + { + "seq_lens": [5000], + "data_type": "random", + }, + id="single_row_medium", + ), + # Single row large + pytest.param( + { + "seq_lens": [100000], + "top_k": 2048, + "data_type": "random", + }, + id="single_row_large", + ), + # Trivial rows mixed with medium and large + pytest.param( + { + "seq_lens": [100, 2048, 10000, 80000], + "data_type": "random", + }, + id="trivial_medium_large_mix", + ), + ], +) +@pytest.mark.parametrize("top_k", [512, 2048]) +@pytest.mark.parametrize("backend", WORKSPACE_TOPK_BACKENDS) +@torch.inference_mode() +def test_workspace_topk(test_config: dict, top_k: int, backend: str) -> None: + """ + Tests specific to workspace top-k backends: + - Mixed medium/large rows in the same batch (dynamic per-row dispatch) + - Boundary around LARGE_THRESHOLD (32K) + - Trivial + medium + large rows in a single batch + """ + run_large_context_topk_test( + batch_size=len(test_config["seq_lens"]), + seq_lens=test_config["seq_lens"], + top_k=top_k, + data_type=test_config.get("data_type", "random"), + backend=backend, + ) + + +@pytest.mark.skipif(not current_platform.is_cuda(), reason="This test requires CUDA") +@pytest.mark.parametrize("top_k", [512, 2048]) +@pytest.mark.parametrize("backend", WORKSPACE_TOPK_BACKENDS) +@torch.inference_mode() +def test_workspace_topk_padded_stride(top_k: int, backend: str) -> None: + """ + Test workspace top-k backends with padded logits (large stride, small seq_len) + to simulate the e2e CUDAGraph scenario where fp8_paged_mqa_logits + returns [B, max_model_len] with max_model_len=163840. + """ + set_random_seed(42) + torch.set_default_device("cuda:0") + + batch_size = 4 + padded_stride = 163840 # DeepSeek-V3.2 max_model_len + actual_seq_lens = [3000, 5000, 8000, 12000] + + # Create padded logits tensor (like fp8_paged_mqa_logits output) + logits = torch.full( + (batch_size, padded_stride), + float("-inf"), + dtype=torch.float32, + device="cuda", + ) + for i, sl in enumerate(actual_seq_lens): + logits[i, :sl] = torch.randn(sl, dtype=torch.float32, device="cuda") + + lengths = torch.tensor(actual_seq_lens, dtype=torch.int32, device="cuda") + indices = torch.empty((batch_size, top_k), dtype=torch.int32, device="cuda") + _run_topk_backend(backend, logits, lengths, indices, top_k, max(actual_seq_lens)) + torch.accelerator.synchronize() + + # Validate against torch.topk + for i in range(batch_size): + sl = actual_seq_lens[i] + k_i = min(top_k, sl) + expected = logits[i, :sl].topk(k_i, dim=-1)[1].cpu() + actual = indices[i, :k_i].cpu() + + expected_set = set(expected.tolist()) + actual_set = set(actual.tolist()) + + if expected_set != actual_set: + # Allow ties + expected_vals = logits[i, expected].cpu().sort(descending=True)[0] + actual_vals = logits[i, actual].cpu().sort(descending=True)[0] + assert torch.allclose(expected_vals, actual_vals, rtol=1e-4, atol=1e-4), ( + f"Row {i}: {backend} with padded stride doesn't match. " + f"seq_len={sl}, stride={padded_stride}" + ) diff --git a/asian_ai/performance/vLLM/tests/kernels/utils.py b/asian_ai/performance/vLLM/tests/kernels/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..6bf6c55568cdf73f6ffcb0704ad1ffcd313c3480 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/kernels/utils.py @@ -0,0 +1,1116 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""Kernel test utils""" + +import itertools +import random +from collections.abc import Sequence +from numbers import Number +from typing import Any, NamedTuple +from unittest.mock import patch + +import torch +from torch._prims_common import TensorLikeType + +from tests.kernels.quant_utils import native_w8a8_block_matmul +from vllm.model_executor.custom_op import op_registry +from vllm.model_executor.layers.activation import SiluAndMul +from vllm.model_executor.layers.fused_moe.activation import MoEActivation +from vllm.model_executor.layers.fused_moe.utils import moe_kernel_quantize_input +from vllm.utils.torch_utils import make_tensor_with_pad +from vllm.v1.attention.backend import AttentionType + +# For now, disable "test_aot_dispatch_dynamic" since there are some +# bugs related to this test in PyTorch 2.4. +DEFAULT_OPCHECK_TEST_UTILS: tuple[str, ...] = ( + "test_schema", + "test_autograd_registration", + "test_faketensor", +) + +ALL_OPCHECK_TEST_UTILS: tuple[str, ...] = ( + "test_schema", + "test_autograd_registration", + "test_faketensor", + "test_aot_dispatch_dynamic", +) + + +class QKVInputs(NamedTuple): + """ + Data structure for representing unpacked attention inputs, + query/key/values and their sequence lengths. + + Attributes: + + * {query,key,value}: unpacked (batch_size x padded_seq_len x + num_heads x head_size) attention inputs + * q_seq_lens: query sequence lengths list + * kv_seq_lens: shared key/value sequence lengths list + """ + + query: torch.Tensor + key: torch.Tensor + value: torch.Tensor + q_seq_lens: list[int] + kv_seq_lens: list[int] + + +class QKVO(NamedTuple): + """ + Data structure for representing unpacked attention inputs, + alongside unpacked known-correct attention output + + Attributes: + + * qkv: unpacked (batch_size x padded_seq_len x + num_heads x head_size) attention inputs + * ideal_output: unpacked (batch_size x padded_seq_len x + num_heads x head_size) known-correct attention output + """ + + qkv: QKVInputs + ideal_output: torch.Tensor + + +class PackedQKVInputs(NamedTuple): + """ + Data structure for representing packed attention inputs + + Attributes: + + * {query,key,value}: packed (number_of_tokens x num_heads + x head_size) attention inputs + * q_start_loc_list: list of query start locations within packed tensor + * kv_start_loc_list: shared list of key/value start locations within + packed tensor + * q_seq_lens: query sequence lengths list + * kv_seq_lens: shared key/value sequence lengths list + """ + + query: torch.Tensor + key: torch.Tensor + value: torch.Tensor + q_start_loc_list: list[int] | None + kv_start_loc_list: list[int] | None + q_seq_lens: list[int] | None + kv_seq_lens: list[int] | None + + +class PackedQKVO(NamedTuple): + """ + Data structure for representing packed attention inputs, + alongside packed known-correct attention output + + Attributes: + + * packed_qkv: packed (number_of_tokens x num_heads + x head_size) attention inputs + * ideal_output: packed (number_of_tokens x num_heads + x head_size) known-correct attention output + """ + + packed_qkv: PackedQKVInputs | None + ideal_output: torch.Tensor + + +class KVMemoryMap(NamedTuple): + """ + Data structure for encapsulating KV cache memory mapping. + + Attributes: + + * block_tables: KV cache block tables + * slot_mapping: mapping of sequence offset to physical address + """ + + block_tables: torch.Tensor + slot_mapping: torch.Tensor + + +class PhaseTestParameters(NamedTuple): + """ + Data structure for encapsulating the test parameters + for a given test "phase" (prefill or decode phase) and attention + scenario (encoder, decoder-self, encoder/decoder-cross) + + Attributes: + + * packed_qkvo: packed (number_of_tokens x num_heads + x head_size) attention inputs & known-correct + output + * kv_mmap: KV cache memory mapping, specific to this test phase & + attention scenario + """ + + packed_qkvo: PackedQKVO + kv_mmap: KVMemoryMap | None + + +def maybe_make_int_tensor( + _list: list[int] | None, + device: torch.device | str, +) -> torch.Tensor: + """ + Convert Python int list to a 1D int torch.Tensor on `device` + + Returns: + + * If _list is not None: 1D int torch.Tensor on `device` + * None otherwise + """ + return ( + None if _list is None else torch.tensor(_list, dtype=torch.int, device=device) + ) + + +def maybe_make_long_tensor( + _list: list[int] | None, + device: torch.device | str, +) -> torch.Tensor: + """ + Convert Python int list to a 1D long torch.Tensor on `device` + + Returns: + + * If _list is not None: 1D long torch.Tensor on `device` + * None otherwise + """ + return ( + None if _list is None else torch.tensor(_list, dtype=torch.long, device=device) + ) + + +def maybe_max(_list: list | None) -> Number | None: + """ + Returns: + + * If _list is not None: max(_list) + * None otherwise + """ + return None if _list is None else max(_list) + + +def make_causal_mask( + q_max_seq_len: int, + kv_max_seq_len: int, +) -> torch.Tensor: + """ + Create a q_max_seq_len x kv_max_seq_len causal mask + + Arguments: + + * q_max_seq_len: query max seq len + * kv_max_seq_len: key/value max seq len + + Returns: + + * 2D tensor, q_max_seq_len x kv_max_seq_len + """ + + # Create a matrix where entry (i, j) is True if i >= j + mask = torch.triu(torch.ones(q_max_seq_len, kv_max_seq_len), diagonal=1) + # Replace True with float('-inf') and False with 0 + mask = mask.masked_fill(mask == 1, float("-inf")).masked_fill(mask == 0, 0.0) + return mask + + +def ref_masked_attention( + query: torch.Tensor, + key: torch.Tensor, + value: torch.Tensor, + scale: float, + custom_mask: torch.Tensor | None = None, + q_seq_lens: list | None = None, + kv_seq_lens: list | None = None, +) -> torch.Tensor: + """ + "Golden" masked attention reference. Supports two types of masking: + + * Basic attention mask, utilizing {q,kv}_seq_lens args to mask out + padding elements + * Custom attention mask, which can force an arbitrary mask tensor, i.e. + causal + + Arguments: + + * query: batch_size x q_padded_seq_len x num_heads x head_size + * key: batch_size x kv_padded_seq_len x num_heads x head_size + * value: batch_size x kv_padded_seq_len x num_heads x head_size + * scale: Attention scale factor + * custom_mask: custom attention mask; good place to inject a causal + attention mask + * q_seq_lens: list of unpadded query seq_lens for each batch index + * kv_seq_lens: list of unpadded key/value seq_lens for each batch index + + Returns: + + * Attention result, batch_size x q_padded_seq_len x num_heads x head_size + """ + + assert q_seq_lens is not None + assert kv_seq_lens is not None + + batch_size = query.shape[0] + assert len(q_seq_lens) == batch_size + assert len(kv_seq_lens) == batch_size + + attn_weights = scale * torch.einsum("bqhd,bkhd->bhqk", query, key).float() + + # Basic attention mask, derived from seq lens + if (q_seq_lens is not None) or (kv_seq_lens is not None): + attn_mask = torch.zeros_like(attn_weights) + if q_seq_lens is not None: + for bdx, plen in enumerate(q_seq_lens): + attn_mask[bdx, :, plen:, :] = -torch.inf + if kv_seq_lens is not None: + for bdx, plen in enumerate(kv_seq_lens): + attn_mask[bdx, :, :, plen:] = -torch.inf + + attn_weights = attn_weights + attn_mask.float() + + # Custom attention mask + if custom_mask is not None: + attn_weights = attn_weights + custom_mask.float() + + attn_weights = torch.softmax(attn_weights, dim=-1).to(value.dtype) + out = torch.einsum("bhqk,bkhd->bqhd", attn_weights, value) + return out + + +def make_qkv( + batch_size: int, + max_q_seq_len: int, + max_kv_seq_len: int | None, + num_heads: int, + head_size: int, + device: torch.device | str, + force_kv_seq_lens: list[int] | None = None, + attn_type: AttentionType = AttentionType.ENCODER_DECODER, + force_max_len: bool = False, +) -> tuple[QKVInputs, QKVInputs, QKVInputs]: + """ + Construct QKV test tensors for self- and cross-attention. + + Generates three query/key/value triplets: + + * "Baseline" query/key/value (for input to reference attention function) + * "Prefill" query/key/value (last sequence offset zero'd out, for use as + input to prefill kernel) + * "Decode" query/key/value (only the last sequence offset from baseline, + for use as input to decode kernel) + + Each Q/K/V triplet is associated with a list of q seqlens and a list of k/v + seqlens + + Arguments: + + * batch_size + * max_q_seq_len: max query seq len + * max_kv_seq_len: max key/value seq len + * num_heads + * head_size + * is_encoder_decoder_attn: if True, query seqlen may differ from + key/value seqlen (as is often the case for cross-attention); + o/w, query/key/value seqlens match at each batch index + (max_kv_seq_len is unused) + * force_kv_seq_lens: if not None, overrides kv sequence lengths + * attn_type: encoder, decoder self, or enc/dec cross attention + * force_max_len: if True, all query seqlens are max_q_seq_len; o/w query + seqlens are random in [2,max_q_seq_lens]. Same for key/value seqlens + and max_kv_seq_len, unless forced by is_encoder_decoder_attn=False + * device: CPU or CUDA device + + Returns: + + * Overall QKVInputs structure (containing full unpacked Q/K/V tensors) + * Prefill QKVInputs structure (containing all but the last sequence offset) + * Decode QKVInputs structure (containing all only the last sequence offset) + """ + + if force_max_len: + q_seq_lens = [max_q_seq_len for _ in range(batch_size)] + else: + q_seq_lens = [random.randint(2, max_q_seq_len) for _ in range(batch_size)] + kv_seq_lens = None + if force_kv_seq_lens is not None: + kv_seq_lens = force_kv_seq_lens + elif attn_type != AttentionType.ENCODER_DECODER: + # K,V seq lens match Q for self-attention + kv_seq_lens = q_seq_lens + else: + # K,V seq lens are distinct from Q seq lens & random + assert max_kv_seq_len is not None + if force_max_len: + kv_seq_lens = [max_kv_seq_len] * batch_size + else: + kv_seq_lens = [random.randint(2, max_kv_seq_len) for _ in range(batch_size)] + + query = torch.rand((batch_size, max_q_seq_len, num_heads, head_size)).to(device) + key = torch.rand((batch_size, max_kv_seq_len, num_heads, head_size)).to(device) + value = torch.rand((batch_size, max_kv_seq_len, num_heads, head_size)).to(device) + + prefill_query = torch.zeros((batch_size, max_q_seq_len, num_heads, head_size)).to( + device + ) + prefill_key = torch.zeros((batch_size, max_kv_seq_len, num_heads, head_size)).to( + device + ) + prefill_value = torch.zeros((batch_size, max_kv_seq_len, num_heads, head_size)).to( + device + ) + + decode_query = torch.zeros((batch_size, 1, num_heads, head_size)).to(device) + decode_key = torch.zeros((batch_size, 1, num_heads, head_size)).to(device) + decode_value = torch.zeros((batch_size, 1, num_heads, head_size)).to(device) + + for bdx, (q_seq_len, kv_seq_len) in enumerate(zip(q_seq_lens, kv_seq_lens)): + query[bdx, q_seq_len:, :, :] = 0 + key[bdx, kv_seq_len:, :, :] = 0 + value[bdx, kv_seq_len:, :, :] = 0 + + prefill_query[bdx, 0 : (q_seq_len - 1), :, :] = query[ + bdx, 0 : (q_seq_len - 1), :, : + ] + prefill_key[bdx, 0 : (kv_seq_len - 1), :, :] = key[ + bdx, 0 : (kv_seq_len - 1), :, : + ] + prefill_value[bdx, 0 : (kv_seq_len - 1), :, :] = value[ + bdx, 0 : (kv_seq_len - 1), :, : + ] + + decode_query[bdx, :, :, :] = query[bdx, (q_seq_len - 1) : q_seq_len, :, :] + decode_key[bdx, :, :, :] = key[bdx, (kv_seq_len - 1) : kv_seq_len, :, :] + decode_value[bdx, :, :, :] = value[bdx, (kv_seq_len - 1) : kv_seq_len, :, :] + + prefill_q_seq_lens = [plen - 1 for plen in q_seq_lens] + prefill_kv_seq_lens = [plen - 1 for plen in kv_seq_lens] + + decode_q_seq_lens = [1 for _ in q_seq_lens] + decode_kv_seq_lens = [1 for _ in kv_seq_lens] + + return ( + QKVInputs( + query, # Overall QKV inputs + key, + value, + q_seq_lens, + kv_seq_lens, + ), + QKVInputs( + prefill_query, # Prefill subset of QKV sequences + prefill_key, + prefill_value, + prefill_q_seq_lens, + prefill_kv_seq_lens, + ), + QKVInputs( + decode_query, # Decode subset of KV sequences + decode_key, + decode_value, + decode_q_seq_lens, + decode_kv_seq_lens, + ), + ) + + +def pack_tensor( + unpacked_tensor: torch.Tensor, seq_lens: list[int], device: torch.device | str +) -> tuple[torch.Tensor, list[int]]: + """ + Pack a batch_size x padded_seq_len x num_heads x head_size tensor into an + unpadded number_of_tokens x num_heads x head_size tensor, where + number_of_tokens = sum(seq_lens) + + Arguments: + + * unpacked_tensor: batch_size x padded_seq_len x num_heads x head_size + * seq_lens: list of token counts for each seq + * device: CPU or CUDA device + + Returns + + * packed_tensor: number_of_tokens x num_heads x head_size + * start_loc_list: start idx of each batch elt in packed_tensor; [0] + + list(itertools.accumulate(seq_lens)) + """ + + num_tok = sum(seq_lens) + num_heads = unpacked_tensor.shape[-2] + head_size = unpacked_tensor.shape[-1] + start_loc_list = [0] + list(itertools.accumulate(seq_lens)) + packed_tensor = torch.zeros((num_tok, num_heads, head_size), device=device) + + for bdx, (seq_len, start_loc) in enumerate(zip(seq_lens, start_loc_list)): + packed_tensor[start_loc : (start_loc + seq_len), :, :] = unpacked_tensor[ + bdx, :seq_len, :, : + ] + + return packed_tensor, start_loc_list + + +def pack_qkv(qkv: QKVInputs, device: torch.device | str) -> PackedQKVInputs: + """ + Individually pack each of Q, K and V, each with dimensions batch_size x + padded_seq_len x num_heads x head_size, into respective number_of_tokens x + num_heads x head_size tensors. + + For Q, number_of_tokens = sum(q_seq_lens). + + For K and V, number_of_tokens = sum(kv_seq_lens) + + Arguments: + + * qkv: Unpacked (batch_size x padded_seq_len x num_heads x head_size) + attention inputs + * device: CPU or CUDA device + + Returns + + * Packed (number_of_tokens x num_heads x head_size) QKV inputs + derived from unpacked inputs + """ + + if qkv.query is None: + packed_query = None + q_start_loc_list = None + else: + packed_query, q_start_loc_list = pack_tensor( + qkv.query, qkv.q_seq_lens, device=device + ) + packed_key, kv_start_loc_list = pack_tensor(qkv.key, qkv.kv_seq_lens, device=device) + packed_value, _ = pack_tensor(qkv.value, qkv.kv_seq_lens, device=device) + return PackedQKVInputs( + packed_query, + packed_key, + packed_value, + q_start_loc_list, + kv_start_loc_list, + (None if q_start_loc_list is None else qkv.q_seq_lens), + qkv.kv_seq_lens, + ) + + +def _make_metadata_tensors( + seq_lens: list[int] | None, + context_lens: list[int] | None, + encoder_seq_lens: list[int] | None, + device: torch.device | str, +) -> tuple[ + torch.Tensor, + torch.Tensor, + Any, + Any, + torch.Tensor | None, + torch.Tensor, + torch.Tensor, + int | None, +]: + """ + Build scalar & tensor values required to build attention metadata structure. + + Arguments: + + * seq_lens: list of token-counts for each decoder input seq + * context_lens: list of context length values for each seq + * encoder_seq_lens: list of token-counts for each encoder input seq + * device: CPU or CUDA device + + Returns: + + * seq_lens_tensor: decoder seq_lens list, as tensor + * context_lens_tensor: context_lens list, as tensor + * max_context_len: max(context_lens) + * max_seq_len: max(seq_lens) + * seq_start_loc: start idx of each sequence + * encoder_seq_lens_tensor: encoder seq_lens list, as tensor + * encoder_seq_start_loc: start idx of each encoder sequence + * max_encoder_seq_len: encoder seq_lens list, as tensor + """ + seq_lens_tensor = maybe_make_int_tensor(seq_lens, device) + context_lens_tensor = maybe_make_int_tensor(context_lens, device) + max_context_len = maybe_max(context_lens) + max_seq_len = maybe_max(seq_lens) + + encoder_seq_lens_tensor = maybe_make_int_tensor(encoder_seq_lens, device) + max_encoder_seq_len = None if encoder_seq_lens is None else max(encoder_seq_lens) + + seq_start_loc = None + + if seq_lens_tensor is not None: + seq_start_loc = torch.zeros( + seq_lens_tensor.shape[0] + 1, + dtype=torch.int32, + device=seq_lens_tensor.device, + ) + torch.cumsum( + seq_lens_tensor, dim=0, dtype=seq_start_loc.dtype, out=seq_start_loc[1:] + ) + + encoder_seq_start_loc = torch.zeros( + encoder_seq_lens_tensor.shape[0] + 1, + dtype=torch.int32, + device=encoder_seq_lens_tensor.device, + ) + torch.cumsum( + encoder_seq_lens_tensor, + dim=0, + dtype=encoder_seq_start_loc.dtype, + out=encoder_seq_start_loc[1:], + ) + + return ( + seq_lens_tensor, + context_lens_tensor, + max_context_len, + max_seq_len, + seq_start_loc, + encoder_seq_lens_tensor, + encoder_seq_start_loc, + max_encoder_seq_len, + ) + + +def make_kv_cache( + num_blocks: int, + num_heads: int, + head_size: int, + block_size: int, + device: torch.device | str, + backend: str, + default_val: float = 0.0, +) -> torch.Tensor: + """ + Create a fake KV cache. + + Arguments: + + * num_blocks: number of blocks in the KV cache + * num_heads: number of attention heads + * head_size: head dimension + * block_size: number of offsets within a block + * device: CPU or CUDA device + * default_val: initialization value for KV cache elements + + Returns: + + * kv_cache: 2 x num_blocks x block_size x num_heads x head_size + * for backend 'FLASH_ATTN' + """ + if backend != "FLASH_ATTN": + raise ValueError(f"Unknown backend value: '{backend}'. Expected 'FLASH_ATTN'.") + kv_cache = torch.rand((2, num_blocks, block_size, num_heads, head_size)).to(device) + if default_val is not None: + kv_cache[:, :, :] = default_val + return kv_cache + + +def _num_tokens_to_min_blocks(num_tokens: int, block_size: int) -> int: + """ + Compute the minimum number of blocks required to hold num_tokens tokens, + given block_size + """ + return (num_tokens + block_size - 1) // block_size + + +def make_empty_slot_mapping_tensor(device: torch.device | str): + return maybe_make_long_tensor([], device) + + +def make_empty_block_tables_tensor(device: torch.device | str): + return torch.tensor([], device=device) + + +def split_slot_mapping( + slot_mapping_list: torch.Tensor, + seq_lens: list[int], + device: torch.device | str, +): + """ + Split a slot mapping into valid prefill- and decode-phase slot mappings. + + Context: + * Your goal is to test (1) prefill of N prompts, with prompt-lengths + {K_i \\forall i \\in [0,N)}, followed by (2) decoding of a single token + for all N prompts (N tokens total); the resultant sequence lengths + after decode would be {K_i + 1 for i \\in [0,N)} + * The test you want to do requires (1) having the prefill slot mapping + for all tokens present during prefill, the number of which is + M = \\sum_i{K_i}, and (2) having the decode slot mapping for all N + decoded tokens + + This function consumes a single 1D slot mapping, which is the + concatenation of N slot mappings each of length K_i + 1 (corresponding + to the sequence lengths after decode), with a total length of + P = \\sum_i{K_i + 1} = M + N + + The prefill-phase slot mapping results from excising the (K_i + 1)-th entry + from each of the N subsequences in the slot mapping (i.e. omitting the + decoded token's mapping.) + + The N excised entries are appended to obtain the decode-phase slot mapping + + Arguments: + + * slot_mapping_list: Length-P 1D slot mapping (as list) reflecting all N + post-decode sequences + * seq_lens: list of N post-decode sequence lengths (K_i + 1 in the + description above) + * device: cuda, cpu, etc. + + Returns: + + * prefill_slot_mapping: Length-M 1D slot mapping (as Tensor) + reflecting all N prefill prompts + * decode_slot_mapping: Length-N 1D slot mapping (as Tensor) reflecting + all N decoded tokens + """ + + prefill_slot_mapping = [] + decode_slot_mapping = [] + + base_idx = 0 + for seq_len in seq_lens: + prefill_slot_mapping.extend( + slot_mapping_list[base_idx : (base_idx + seq_len - 1)] + ) + decode_slot_mapping.append(slot_mapping_list[base_idx + seq_len - 1]) + base_idx += seq_len + + return ( + maybe_make_long_tensor(prefill_slot_mapping, device), + maybe_make_long_tensor(decode_slot_mapping, device), + ) + + +def make_block_tables_slot_mapping( + block_size: int, + seq_lens: list[int], + device: torch.device | str, + block_base_addr: int = 0, +) -> tuple[torch.Tensor, list[int], int]: + """ + Construct fake block tables & slot mappings. + + For a sequence with num_tokens tokens the minimum number + of required KV cache blocks is + + num_blocks = (num_tokens + block_size - 1) // block_size + + Then the minimum KV cache size in blocks is + + total_cache_blocks = sum(num_blocks for all seqs) + + Then, the blocktable mapping counts downward from + + block_base_addr + total_cache_blocks + + to + + block_base_addr + + + The constructed block-tables and slot-mapping are sized to the + lengths of the sequences in their entirety (as reflected by seq_lens), + i.e. the total of prefill prompt tokens + decoded tokens. + + Arguments: + + * block_size: number of offsets per block + * seq_lens: list of token-counts for each sequence + * block_base_addr: the block table base address + * device: CPU or CUDA device + + Return: + + * block_tables_tensor: block table for sequence + * slot_mapping_list: slot mapping for sequence + * max_block_idx: the highest block address within this block table + """ + + # Provision minimum number of KV cache blocks + num_blocks_list = [ + _num_tokens_to_min_blocks(num_tokens, block_size) for num_tokens in seq_lens + ] + max_block_table_len = max(num_blocks_list) + block_table_pad_tokens = 10 + + block_tables = [] + slot_mapping_list = [] + # Compute uppermost address of block table + total_cache_blocks = sum(num_blocks_list) + block_base_idx = block_base_addr + total_cache_blocks + max_block_idx = block_base_idx + for sdx, num_tokens in enumerate(seq_lens): + num_blocks = num_blocks_list[sdx] + block_table = list(range(block_base_idx, block_base_idx - num_blocks, -1)) + for idx in range(num_tokens): + mapping_value = (idx % block_size) + block_table[ + idx // block_size + ] * block_size + slot_mapping_list.append(mapping_value) + + block_base_idx -= num_blocks + block_tables.append(block_table) + + block_tables_tensor = make_tensor_with_pad( + block_tables, + max_len=max_block_table_len + block_table_pad_tokens, + pad=0, + dtype=torch.int, + device=device, + ) + + return (block_tables_tensor, slot_mapping_list, max_block_idx) + + +def assert_actual_matches_ideal( + test_params: PhaseTestParameters, output_under_test: torch.Tensor, backend: str +) -> None: + """ + Assert that observed output matches the ideal output + contained in the test parameters data structure. + + Arguments: + + * test_params: Test parameters including packed ideal output + * output_under_test: actually observed output value + """ + ideal_output = test_params.packed_qkvo.ideal_output + if backend != "FLASH_ATTN": + raise ValueError(f"Unknown backend value: '{backend}'. Expected 'FLASH_ATTN'.") + # For FlashAttention override the accuracy thresholds to non default + # values since we notice a higher difference between the ideal and + # actual output. + torch.testing.assert_close( + ideal_output, output_under_test.view_as(ideal_output), atol=0.01, rtol=0.016 + ) + + +# Copied/modified from torch._refs.__init__.py +def fp8_allclose( + a: TensorLikeType, + b: TensorLikeType, + rtol: float = 1e-05, + atol: float = 1e-08, + equal_nan: bool = False, +) -> bool: + """ + Reference implementation of torch.allclose + """ + torch._refs._check_close_args(name="torch.allclose", a=a, b=b, rtol=rtol, atol=atol) + + return bool( + torch.all( + torch.isclose( + a.double(), b.double(), rtol=rtol, atol=atol, equal_nan=equal_nan + ) + ).item() + ) + + +def bf16_ulp_distance(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: + """Representable-step distance between two bf16 tensors. + + Reinterprets the bf16 bit patterns under the IEEE-754 total ordering so + that adjacent representable values differ by exactly 1. + """ + + def key(t: torch.Tensor) -> torch.Tensor: + u = t.contiguous().view(torch.int16).to(torch.int64) & 0xFFFF + return torch.where(u >= 0x8000, 0xFFFF - u, u + 0x8000) + + return (key(a) - key(b)).abs() + + +def fp8_ulp_distance(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor: + """Representable-step distance between two 8-bit fp8 tensors. + + Reinterprets the fp8 bytes under a sign-magnitude total ordering so that + adjacent representable values differ by exactly 1. Inputs must already share + the same fp8 encoding (e.g. both FP8_STORE_DTYPE). + """ + + def key(t: torch.Tensor) -> torch.Tensor: + u = t.contiguous().view(torch.uint8).to(torch.int64) + return torch.where(u >= 0x80, 0xFF - u, u + 0x80) + + return (key(a) - key(b)).abs() + + +# Marlin MoE test utils + + +def stack_and_dev(tensors: list[torch.Tensor]): + dev = tensors[0].device + return torch.stack(tensors, dim=0).to(dev) + + +def compute_max_diff(output, output_ref): + return torch.mean(torch.abs(output - output_ref)) / torch.mean( + torch.abs(output_ref) + ) + + +def torch_experts( + a: torch.Tensor, + w1: torch.Tensor, + w2: torch.Tensor, + topk_weight: torch.Tensor, + topk_ids: torch.Tensor, + global_num_experts: int = -1, + b_bias1: torch.Tensor | None = None, + b_bias2: torch.Tensor | None = None, + expert_map: torch.Tensor | None = None, + w1_scale: torch.Tensor | None = None, + w2_scale: torch.Tensor | None = None, + a1_scale: torch.Tensor | None = None, + a2_scale: torch.Tensor | None = None, + quant_dtype: torch.dtype | None = None, + per_act_token_quant=False, + block_shape: list[int] | None = None, + apply_router_weights_on_input: bool = False, + activation: MoEActivation = MoEActivation.SILU, +) -> torch.Tensor: + assert ( + global_num_experts == -1 + or (global_num_experts == w1.shape[0] and expert_map is None) + or (expert_map is not None and global_num_experts == expert_map.shape[0]) + ) + + if quant_dtype in [torch.float16, torch.bfloat16]: + quant_dtype = None + quant_input_only = quant_dtype is not None and w1_scale is None and w2_scale is None + if quant_input_only: + assert a1_scale is None and a2_scale is None + assert per_act_token_quant + + M, K = a.shape + topk = topk_ids.shape[1] + + if apply_router_weights_on_input: + assert topk == 1 + a = a * topk_weight.to(a.dtype) + + a = a.view(M, -1, K).repeat(1, topk, 1).reshape(-1, K) + + out = torch.zeros(M * topk, w2.shape[1], dtype=a.dtype, device=a.device) + + if a1_scale: + assert not per_act_token_quant and block_shape is None + a, a_scale = moe_kernel_quantize_input( + a, a1_scale, quant_dtype, per_act_token_quant, block_shape + ) + + if quant_input_only: + a = (a.float() * a_scale.view(-1, 1)).to(w1.dtype) + + num_experts = w1.shape[0] + + topk_ids = topk_ids.view(-1) + if expert_map is not None: + topk_ids = expert_map[topk_ids] + + f32 = torch.float32 + + act = op_registry[activation.custom_op_name] + + for i in range(num_experts): + mask = topk_ids == i + if mask.sum(): + if quant_dtype is None: + tmp1 = a[mask] @ w1[i].transpose(0, 1) + if b_bias1 is not None: + tmp1 = tmp1 + b_bias1[i].view(1, -1).to(tmp1.dtype) + tmp2 = act()(tmp1) + out[mask] = tmp2 @ w2[i].transpose(0, 1) + if b_bias2 is not None: + out[mask] = out[mask] + b_bias2[i].view(1, -1).to(tmp1.dtype) + elif quant_input_only: + tmp1 = a[mask] @ w1[i].transpose(0, 1) + tmp2 = SiluAndMul()(tmp1) + tmp2, tmp2_scale = moe_kernel_quantize_input( + tmp2, None, quant_dtype, per_act_token_quant + ) + tmp2 = (tmp2.float() * tmp2_scale.view(-1, 1)).to(w2.dtype) + out[mask] = tmp2 @ w2[i].transpose(0, 1) + elif block_shape is not None: + # block quantized + assert ( + a_scale is not None + and w1_scale is not None + and w2_scale is not None + ) + tmp1 = native_w8a8_block_matmul( + a[mask], w1[i], a_scale[mask], w1_scale[i], block_shape, out.dtype + ) + if b_bias1 is not None: + tmp1 = tmp1 + b_bias1[i].view(1, -1).to(tmp1.dtype) + tmp2 = SiluAndMul()(tmp1) + tmp2, b_scale = moe_kernel_quantize_input( + tmp2, a2_scale, quant_dtype, per_act_token_quant, block_shape + ) + + out[mask] = native_w8a8_block_matmul( + tmp2, w2[i], b_scale, w2_scale[i], block_shape, out.dtype + ) + if b_bias2 is not None: + out[mask] = out[mask] + b_bias2[i].view(1, -1).to(tmp1.dtype) + else: + assert ( + a_scale is not None + and w1_scale is not None + and w2_scale is not None + ) + scales = a_scale if a_scale.numel() == 1 else a_scale[mask] + + tmp1 = a[mask].to(f32) * scales + w1_dq = (w1[i].to(f32) * w1_scale[i]).transpose(0, 1) + tmp1 = (tmp1 @ w1_dq).to(out.dtype) + if b_bias1 is not None: + tmp1 = tmp1 + b_bias1[i].view(1, -1).to(out.dtype) + + tmp2 = act()(tmp1).to(out.dtype) + + tmp2, b_scale = moe_kernel_quantize_input( + tmp2, a2_scale, quant_dtype, per_act_token_quant, block_shape + ) + assert b_scale is not None + + tmp2 = tmp2.to(f32) * b_scale + w2_dq = (w2[i].to(f32) * w2_scale[i]).transpose(0, 1) + out[mask] = (tmp2 @ w2_dq).to(out.dtype) + if b_bias2 is not None: + out[mask] = out[mask] + b_bias2[i].view(1, -1).to(out.dtype) + + if apply_router_weights_on_input: + return out + else: + return ( + (out.view(M, -1, w2.shape[1]).to(f32) * topk_weight.view(M, -1, 1)) + .sum(dim=1) + .to(out.dtype) + ) + + +def torch_moe( + a: torch.Tensor, + w1: torch.Tensor, + w2: torch.Tensor, + score: torch.Tensor, + topk: int, + b_bias1: torch.Tensor | None = None, + b_bias2: torch.Tensor | None = None, + global_num_experts: int = -1, + expert_map: torch.Tensor | None = None, + activation: MoEActivation = MoEActivation.SILU, +) -> torch.Tensor: + score = torch.softmax(score, dim=-1, dtype=torch.float32) + topk_weight, topk_ids = torch.topk(score, topk) + return torch_experts( + a, + w1, + w2, + topk_weight, + topk_ids, + global_num_experts, + b_bias1, + b_bias2, + expert_map, + activation=activation, + ) + + +def torch_moe_single(a, w, score, topk): + B, D = a.shape + a = a.view(B, -1, D).repeat(1, topk, 1).reshape(-1, D) + out = torch.zeros(B * topk, w.shape[1], dtype=a.dtype, device=a.device) + score = torch.softmax(score, dim=-1, dtype=torch.float32) + _, topk_ids = torch.topk(score, topk) + topk_ids = topk_ids.view(-1) + for i in range(w.shape[0]): + mask = topk_ids == i + if mask.sum(): + out[mask] = a[mask] @ w[i].transpose(0, 1) + return (out.view(B, -1, w.shape[1])).sum(dim=1) + + +# A special version of op check that has a restricted default set of test_utils +# and a patched version of allclose that supports fp8 types. +def opcheck( + op: torch._ops.OpOverload + | torch._ops.OpOverloadPacket + | torch._library.custom_ops.CustomOpDef, + args: tuple[Any, ...], + kwargs: dict[str, Any] | None = None, + *, + test_utils: str | Sequence[str] = ALL_OPCHECK_TEST_UTILS, + raise_exception: bool = True, + cond: bool = True, +) -> dict[str, str]: + with patch("torch.allclose", new=fp8_allclose): + return ( + torch.library.opcheck( + op, args, kwargs, test_utils=test_utils, raise_exception=raise_exception + ) + if cond + else {} + ) + + +# For testing quantized linear kernels +def to_fp8(tensor: torch.Tensor): + finfo = torch.finfo(torch.float8_e4m3fn) + return torch.round(tensor.clamp(min=finfo.min, max=finfo.max)).to( + dtype=torch.float8_e4m3fn + ) + + +def to_int8(tensor: torch.Tensor): + return torch.round(tensor.clamp(min=-128, max=127)).to(dtype=torch.int8) + + +def baseline_scaled_mm( + a: torch.Tensor, + b: torch.Tensor, + scale_a: torch.Tensor, + scale_b: torch.Tensor, + out_dtype: type[torch.dtype], + bias: torch.Tensor | None = None, +) -> torch.Tensor: + # We treat N-dimensional group scaling as extended numpy-style broadcasting + # in numpy simply stretches dimensions with an extent of 1 to match + # the target shape by repeating the data along that dimension (broadcasting) + # , we extend these semantics to say if the extent of a dimension in the + # source shape is not 1 and does not match the target shape we repeat each + # element along that dimension src_shape[dim] // target_shape[dim] times + # example if we have: + # a = [[1, 2], and target_shape = (2, 4) + # [3, 4]] + # then we would expand a to: + # a = [[1, 1, 2, 2], + # [3, 3, 4, 4]] + # NOTE this function does not explicitly broadcast dimensions + # with an extent of 1, since this can be done implicitly by pytorch + def group_broadcast(t, shape): + for i, s in enumerate(shape): + if t.shape[i] != s and t.shape[i] != 1: + assert s % t.shape[i] == 0 + t = ( + t.unsqueeze(i + 1) + .expand(*t.shape[: i + 1], s // t.shape[i], *t.shape[i + 1 :]) + .flatten(i, i + 1) + ) + return t + + scale_a = group_broadcast(scale_a, a.shape) + scale_b = group_broadcast(scale_b, b.shape) + + output = torch.mm( + (scale_a * a.to(dtype=torch.float32)), (scale_b * b.to(dtype=torch.float32)) + ).to(out_dtype) + + if bias is not None: + output = output + bias + + return output diff --git a/asian_ai/performance/vLLM/tests/lora/__init__.py b/asian_ai/performance/vLLM/tests/lora/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/asian_ai/performance/vLLM/tests/lora/conftest.py b/asian_ai/performance/vLLM/tests/lora/conftest.py new file mode 100644 index 0000000000000000000000000000000000000000..5726c1e8d7ae2b0e4f48bd55adfa68d2dfdf9bda --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/conftest.py @@ -0,0 +1,335 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import tempfile +from collections import OrderedDict +from importlib import reload +from unittest.mock import MagicMock + +import pytest +import torch +import torch.nn as nn +from huggingface_hub import snapshot_download + +from vllm.distributed import ( + cleanup_dist_env_and_memory, + init_distributed_environment, + initialize_model_parallel, +) +from vllm.model_executor.layers.linear import ( + ColumnParallelLinear, + MergedColumnParallelLinear, + RowParallelLinear, +) +from vllm.model_executor.layers.logits_processor import LogitsProcessor +from vllm.model_executor.layers.vocab_parallel_embedding import ParallelLMHead +from vllm.model_executor.models.interfaces import SupportsLoRA +from vllm.platforms import current_platform + + +@pytest.fixture() +def should_do_global_cleanup_after_test(request) -> bool: + """Allow subdirectories to skip global cleanup by overriding this fixture. + This can provide a ~10x speedup for non-GPU unit tests since they don't need + to initialize torch. + """ + + return not request.node.get_closest_marker("skip_global_cleanup") + + +@pytest.fixture(autouse=True) +def cleanup_fixture(should_do_global_cleanup_after_test: bool): + yield + if should_do_global_cleanup_after_test: + cleanup_dist_env_and_memory(shutdown_ray=True) + + +@pytest.fixture +def maybe_enable_lora_dual_stream(monkeypatch: pytest.MonkeyPatch): + if current_platform.is_cuda(): + monkeypatch.setenv("VLLM_LORA_ENABLE_DUAL_STREAM", "1") + import vllm.lora.layers.base_linear + + if not hasattr(vllm.lora.layers.base_linear, "lora_linear_async"): + # Reload the module to ensure the environment variable takes effect. + reload(vllm.lora.layers.base_linear) + yield + + +@pytest.fixture +def dist_init(): + from tests.utils import ensure_current_vllm_config + + temp_file = tempfile.mkstemp()[1] + + backend = "gloo" if current_platform.is_tpu() else current_platform.dist_backend + + with ensure_current_vllm_config(): + init_distributed_environment( + world_size=1, + rank=0, + distributed_init_method=f"file://{temp_file}", + local_rank=0, + backend=backend, + ) + initialize_model_parallel(1, 1) + yield + cleanup_dist_env_and_memory(shutdown_ray=True) + + +@pytest.fixture +def dist_init_torch_only(): + if torch.distributed.is_initialized(): + return + backend = current_platform.dist_backend + + temp_file = tempfile.mkstemp()[1] + torch.distributed.init_process_group( + world_size=1, rank=0, init_method=f"file://{temp_file}", backend=backend + ) + + +class DummyLoRAModel(nn.Sequential, SupportsLoRA): + pass + + +@pytest.fixture +def dummy_model(default_vllm_config) -> nn.Module: + model = DummyLoRAModel( + OrderedDict( + [ + ("dense1", ColumnParallelLinear(764, 100)), + ("dense2", RowParallelLinear(100, 50)), + ( + "layer1", + nn.Sequential( + OrderedDict( + [ + ("dense1", ColumnParallelLinear(100, 10)), + ("dense2", RowParallelLinear(10, 50)), + ] + ) + ), + ), + ("act2", nn.ReLU()), + ("output", ColumnParallelLinear(50, 10)), + ("outact", nn.Sigmoid()), + # Special handling for lm_head & sampler + ("lm_head", ParallelLMHead(32064, 10)), + ("logits_processor", LogitsProcessor(32064)), + ] + ) + ) + model.config = MagicMock() + model.embedding_modules = {"lm_head": "lm_head"} + model.unpadded_vocab_size = 32064 + return model + + +@pytest.fixture +def dummy_model_gate_up(default_vllm_config) -> nn.Module: + model = DummyLoRAModel( + OrderedDict( + [ + ("dense1", ColumnParallelLinear(764, 100)), + ("dense2", RowParallelLinear(100, 50)), + ( + "layer1", + nn.Sequential( + OrderedDict( + [ + ("dense1", ColumnParallelLinear(100, 10)), + ("dense2", RowParallelLinear(10, 50)), + ] + ) + ), + ), + ("act2", nn.ReLU()), + ("gate_up_proj", MergedColumnParallelLinear(50, [5, 5])), + ("outact", nn.Sigmoid()), + # Special handling for lm_head & sampler + ("lm_head", ParallelLMHead(32064, 10)), + ("logits_processor", LogitsProcessor(32064)), + ] + ) + ) + model.config = MagicMock() + model.packed_modules_mapping = { + "gate_up_proj": [ + "gate_proj", + "up_proj", + ], + } + model.embedding_modules = {"lm_head": "lm_head"} + model.unpadded_vocab_size = 32064 + + return model + + +@pytest.fixture(scope="session") +def mixtral_lora_files(): + # Note: this module has incorrect adapter_config.json to test + # https://github.com/vllm-project/vllm/pull/5909/files. + return snapshot_download(repo_id="SangBinCho/mixtral-lora") + + +@pytest.fixture(scope="session") +def chatglm3_lora_files(): + return snapshot_download(repo_id="jeeejeee/chatglm3-text2sql-spider") + + +@pytest.fixture(scope="session") +def baichuan_lora_files(): + return snapshot_download(repo_id="jeeejeee/baichuan7b-text2sql-spider") + + +@pytest.fixture(scope="session") +def baichuan_zero_lora_files(): + # all the lora_B weights are initialized to zero. + return snapshot_download(repo_id="jeeejeee/baichuan7b-zero-init") + + +@pytest.fixture(scope="session") +def baichuan_regex_lora_files(): + return snapshot_download(repo_id="jeeejeee/baichuan-7b-lora-zero-regex") + + +@pytest.fixture(scope="session") +def ilama_lora_files(): + return snapshot_download(repo_id="jeeejeee/ilama-text2sql-spider") + + +@pytest.fixture(scope="session") +def minicpmv_lora_files(): + return snapshot_download(repo_id="jeeejeee/minicpmv25-lora-pokemon") + + +@pytest.fixture(scope="session") +def qwen2vl_lora_files(): + return snapshot_download(repo_id="jeeejeee/qwen2-vl-lora-pokemon") + + +@pytest.fixture(scope="session") +def qwen25vl_base_huggingface_id(): + # used as a base model for testing with qwen25vl lora adapter + return "Qwen/Qwen2.5-VL-3B-Instruct" + + +@pytest.fixture(scope="session") +def qwen25vl_lora_files(): + return snapshot_download(repo_id="jeeejeee/qwen25-vl-lora-pokemon") + + +@pytest.fixture(scope="session") +def qwen2vl_language_lora_files(): + return snapshot_download(repo_id="prashanth058/qwen2vl-flickr-lora-language") + + +@pytest.fixture(scope="session") +def qwen2vl_vision_tower_connector_lora_files(): + return snapshot_download(repo_id="prashanth058/qwen2vl-flickr-lora-tower-connector") + + +@pytest.fixture(scope="session") +def qwen2vl_vision_tower_lora_files(): + return snapshot_download(repo_id="prashanth058/qwen2vl-flickr-lora-tower") + + +@pytest.fixture(scope="session") +def qwen25vl_vision_lora_files(): + return snapshot_download(repo_id="EpochEcho/qwen2.5-3b-vl-lora-vision-connector") + + +@pytest.fixture(scope="session") +def qwen3vl_vision_lora_files(): + return snapshot_download(repo_id="EpochEcho/qwen3-4b-vl-lora-vision-connector") + + +@pytest.fixture(scope="session") +def qwen3_meowing_lora_files(): + """Download Qwen3 Meow LoRA files once per test session.""" + return snapshot_download(repo_id="Jackmin108/Qwen3-0.6B-Meow-LoRA") + + +@pytest.fixture(scope="session") +def qwen3_woofing_lora_files(): + """Download Qwen3 Woof LoRA files once per test session.""" + return snapshot_download(repo_id="Jackmin108/Qwen3-0.6B-Woof-LoRA") + + +@pytest.fixture(scope="session") +def tinyllama_lora_files(): + return snapshot_download(repo_id="jashing/tinyllama-colorist-lora") + + +@pytest.fixture(scope="session") +def deepseekv2_lora_files(): + return snapshot_download(repo_id="wuchen01/DeepSeek-V2-Lite-Chat-All-LoRA") + + +@pytest.fixture(scope="session") +def gptoss20b_lora_files(): + return snapshot_download(repo_id="jeeejeee/gpt-oss-20b-lora-adapter-text2sql") + + +@pytest.fixture(scope="session") +def qwen3moe_lora_files(): + return snapshot_download(repo_id="jeeejeee/qwen3-moe-text2sql-spider") + + +@pytest.fixture(scope="session") +def olmoe_lora_files(): + return snapshot_download(repo_id="jeeejeee/olmoe-instruct-text2sql-spider") + + +@pytest.fixture(scope="session") +def qwen3_lora_files(): + return snapshot_download(repo_id="charent/self_cognition_Alice") + + +@pytest.fixture(scope="session") +def llama32_lora_huggingface_id(): + # huggingface repo id is used to test lora runtime downloading. + return "jeeejeee/llama32-3b-text2sql-spider" + + +@pytest.fixture(scope="session") +def llama32_lora_files(llama32_lora_huggingface_id): + return snapshot_download(repo_id=llama32_lora_huggingface_id) + + +@pytest.fixture(scope="session") +def whisper_lora_files(): + return snapshot_download(repo_id="chengyili2005/whisper-small-mandarin-lora") + + +@pytest.fixture(scope="session") +def qwen35_text_lora_files(): + return snapshot_download(repo_id="jeeejeee/qwen35-4b-text-only-sql-lora") + + +@pytest.fixture(scope="session") +def qwen35_vl_lora_files(): + return snapshot_download(repo_id="jeeejeee/qwen35-4b-all-linear-pokemon-lora") + + +@pytest.fixture(scope="session") +def qwen36_moe_2d_lora_files(): + return snapshot_download(repo_id="jeeejeee/qwen36-35ba3b-2d-weights-poken-lora") + + +@pytest.fixture(scope="session") +def qwen36_moe_3d_lora_files(): + return snapshot_download(repo_id="jeeejeee/qwen36-35ba3b-moe-all-linear-poken-lora") + + +@pytest.fixture +def reset_default_device(): + """ + Some tests, such as `test_punica_ops.py`, explicitly set the + default device, which can affect subsequent tests. Adding this fixture + helps avoid this problem. + """ + original_device = torch.get_default_device() + yield + torch.set_default_device(original_device) diff --git a/asian_ai/performance/vLLM/tests/lora/test_add_lora.py b/asian_ai/performance/vLLM/tests/lora/test_add_lora.py new file mode 100644 index 0000000000000000000000000000000000000000..e9931d8718373373d94e6ea6053171c451ee4cc1 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_add_lora.py @@ -0,0 +1,113 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import asyncio +import time + +import pytest + +from vllm.engine.arg_utils import AsyncEngineArgs +from vllm.entrypoints.openai.api_server import ( + build_async_engine_client_from_engine_args, +) +from vllm.inputs import TextPrompt +from vllm.lora.request import LoRARequest +from vllm.sampling_params import SamplingParams +from vllm.utils.async_utils import merge_async_iterators + +MODEL_PATH = "zai-org/chatglm3-6b" +LORA_RANK = 64 +DEFAULT_MAX_LORAS = 4 * 3 + + +def get_lora_requests(lora_path) -> list[LoRARequest]: + lora_requests: list[LoRARequest] = [ + LoRARequest(lora_name=f"{i}", lora_int_id=i, lora_path=lora_path) + for i in range(1, DEFAULT_MAX_LORAS + 1) + ] + return lora_requests + + +async def requests_processing_time(llm, lora_requests: list[LoRARequest]) -> float: + sampling_params = SamplingParams( + n=1, temperature=0.0, top_p=1.0, ignore_eos=True, max_tokens=1 + ) + + generators = [] + start = time.perf_counter() + + for lora_request in lora_requests: + lora_int_id = lora_request.lora_int_id + generator = llm.generate( + prompt=TextPrompt(prompt=f"hello {lora_int_id}", multi_modal_data=None), # type: ignore + sampling_params=sampling_params, + lora_request=lora_request, + request_id=f"test{lora_int_id}", + ) + generators.append(generator) + + all_gens = merge_async_iterators(*generators) + async for i, res in all_gens: + pass + + end = time.perf_counter() + return end - start + + +@pytest.mark.asyncio +async def test_add_lora(chatglm3_lora_files): + """ + The add_lora function is used to preload some LoRA adapters into the + engine in anticipation of future requests using these adapters. To test + this functionality, we use the async engine to process some requests - We + do it twice, once with add_lora() preloading and once without. + + We measure the request processing time in both cases and expect the time + to be lesser in the case with add_lora() calls. + """ + lora_requests: list[LoRARequest] = get_lora_requests(chatglm3_lora_files) + + max_loras = len(set([lr.lora_int_id for lr in lora_requests])) + # Create engine in eager-mode. Due to high max_loras, the CI can + # OOM during cuda-graph capture. + engine_args = AsyncEngineArgs( + model=MODEL_PATH, + enable_lora=True, + max_loras=max_loras, + max_lora_rank=LORA_RANK, + max_model_len=128, + gpu_memory_utilization=0.8, # avoid OOM + trust_remote_code=True, + enforce_eager=True, + ) + + # split lora_requests into 3 parts + part_size = len(lora_requests) // 3 + dummy_run_requests = lora_requests[:part_size] + warmup_run_requests = lora_requests[part_size : part_size * 2] + cold_run_requests = lora_requests[part_size * 2 :] + + async with build_async_engine_client_from_engine_args(engine_args) as llm: + # Dummy run - So any 1-time functionality like triton kernel compilation + # is complete here. + await requests_processing_time(llm, dummy_run_requests) + + # Run with warmup + add_lora_tasks = [llm.add_lora(lr) for lr in warmup_run_requests] + add_lora_results = await asyncio.gather(*add_lora_tasks) + + # Test that all all_lora calls are successful. + assert all(add_lora_results) + + time_with_add_lora = await requests_processing_time(llm, warmup_run_requests) + + # Run without any warmup + time_cold_start = await requests_processing_time(llm, cold_run_requests) + + print(f"time hot-start {time_with_add_lora} vs time cold-start {time_cold_start} ") + + assert time_with_add_lora < time_cold_start, ( + f"time_with_add_lora={time_with_add_lora}, " + f"time_cold_start={time_cold_start}" + "The engine request processing time with LoRA pre-loading " + "must be less than the version that does on-demand LoRA loading." + ) diff --git a/asian_ai/performance/vLLM/tests/lora/test_chatglm3_tp.py b/asian_ai/performance/vLLM/tests/lora/test_chatglm3_tp.py new file mode 100644 index 0000000000000000000000000000000000000000..500bebb91b0e4ef7821a7242566194a64c808c3b --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_chatglm3_tp.py @@ -0,0 +1,131 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +import vllm +import vllm.config +from vllm.lora.request import LoRARequest +from vllm.platforms import current_platform + +from ..utils import create_new_process_for_each_test, multi_gpu_test + +MODEL_PATH = "zai-org/chatglm3-6b" + +PROMPT_TEMPLATE = """I want you to act as a SQL terminal in front of an example database, you need only to return the sql command to me.Below is an instruction that describes a task, Write a response that appropriately completes the request.\n"\n##Instruction:\nconcert_singer contains tables such as stadium, singer, concert, singer_in_concert. Table stadium has columns such as Stadium_ID, Location, Name, Capacity, Highest, Lowest, Average. Stadium_ID is the primary key.\nTable singer has columns such as Singer_ID, Name, Country, Song_Name, Song_release_year, Age, Is_male. Singer_ID is the primary key.\nTable concert has columns such as concert_ID, concert_Name, Theme, Stadium_ID, Year. concert_ID is the primary key.\nTable singer_in_concert has columns such as concert_ID, Singer_ID. concert_ID is the primary key.\nThe Stadium_ID of concert is the foreign key of Stadium_ID of stadium.\nThe Singer_ID of singer_in_concert is the foreign key of Singer_ID of singer.\nThe concert_ID of singer_in_concert is the foreign key of concert_ID of concert.\n\n###Input:\n{query}\n\n###Response:""" # noqa: E501 + +EXPECTED_LORA_OUTPUT = [ + "SELECT count(*) FROM singer", + "SELECT avg(age) , min(age) , max(age) FROM singer WHERE country = 'France'", + "SELECT name , country , age FROM singer ORDER BY age", +] + + +def do_sample(llm: vllm.LLM, lora_path: str, lora_id: int) -> list[str]: + prompts = [ + PROMPT_TEMPLATE.format(query="How many singers do we have?"), + PROMPT_TEMPLATE.format( + query=( + "What is the average, minimum, and maximum " + "age of all singers from France?" + ) + ), + PROMPT_TEMPLATE.format( + query=( + "Show name, country, age for all singers ordered " + "by age from the oldest to the youngest." + ) + ), + ] + sampling_params = vllm.SamplingParams(temperature=0, max_tokens=32) + outputs = llm.generate( + prompts, + sampling_params, + lora_request=LoRARequest(str(lora_id), lora_id, lora_path) if lora_id else None, + ) + # Print the outputs. + generated_texts: list[str] = [] + for output in outputs: + prompt = output.prompt + generated_text = output.outputs[0].text.strip() + generated_texts.append(generated_text) + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") + return generated_texts + + +@pytest.mark.skipif( + current_platform.is_cuda_alike(), reason="Skipping to avoid redundant model tests" +) +@create_new_process_for_each_test() +def test_chatglm3_lora(chatglm3_lora_files): + llm = vllm.LLM( + MODEL_PATH, + max_model_len=512, + enable_lora=True, + max_loras=2, + max_num_seqs=16, + max_lora_rank=64, + trust_remote_code=True, + ) + + output1 = do_sample(llm, chatglm3_lora_files, lora_id=1) + for i in range(len(EXPECTED_LORA_OUTPUT)): + assert output1[i] == EXPECTED_LORA_OUTPUT[i] + output2 = do_sample(llm, chatglm3_lora_files, lora_id=2) + for i in range(len(EXPECTED_LORA_OUTPUT)): + assert output2[i] == EXPECTED_LORA_OUTPUT[i] + + +@pytest.mark.skipif( + current_platform.is_cuda_alike(), reason="Skipping to avoid redundant model tests" +) +@multi_gpu_test(num_gpus=4) +def test_chatglm3_lora_tp4(chatglm3_lora_files): + llm = vllm.LLM( + MODEL_PATH, + max_model_len=512, + enable_lora=True, + max_loras=2, + max_lora_rank=64, + max_num_seqs=16, + tensor_parallel_size=4, + trust_remote_code=True, + fully_sharded_loras=False, + compilation_config=vllm.config.CompilationConfig( # Avoid OOM + cudagraph_specialize_lora=False, + ), + ) + + output1 = do_sample(llm, chatglm3_lora_files, lora_id=1) + for i in range(len(EXPECTED_LORA_OUTPUT)): + assert output1[i] == EXPECTED_LORA_OUTPUT[i] + output2 = do_sample(llm, chatglm3_lora_files, lora_id=2) + for i in range(len(EXPECTED_LORA_OUTPUT)): + assert output2[i] == EXPECTED_LORA_OUTPUT[i] + + +@multi_gpu_test(num_gpus=4) +def test_chatglm3_lora_tp4_fully_sharded_loras(chatglm3_lora_files): + # https://github.com/NVIDIA/nccl/issues/1790, set a lower value for + # gpu_memory_utilization here because NCCL >= 2.26.3 seems to use + # more GPU memory causing vLLM to OOM + llm = vllm.LLM( + MODEL_PATH, + max_model_len=512, + enable_lora=True, + max_loras=2, + max_lora_rank=64, + tensor_parallel_size=4, + trust_remote_code=True, + fully_sharded_loras=True, + gpu_memory_utilization=0.8, + compilation_config=vllm.config.CompilationConfig( # Avoid OOM + cudagraph_specialize_lora=False, + ), + ) + output1 = do_sample(llm, chatglm3_lora_files, lora_id=1) + for i in range(len(EXPECTED_LORA_OUTPUT)): + assert output1[i] == EXPECTED_LORA_OUTPUT[i] + output2 = do_sample(llm, chatglm3_lora_files, lora_id=2) + for i in range(len(EXPECTED_LORA_OUTPUT)): + assert output2[i] == EXPECTED_LORA_OUTPUT[i] diff --git a/asian_ai/performance/vLLM/tests/lora/test_deepseekv2_tp.py b/asian_ai/performance/vLLM/tests/lora/test_deepseekv2_tp.py new file mode 100644 index 0000000000000000000000000000000000000000..ead488bc4ce66e756f9dbd7407fc6171ba7a3b4f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_deepseekv2_tp.py @@ -0,0 +1,101 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +# NOTE To avoid overloading the CI pipeline, this test script will +# not be triggered on CI and is primarily intended for local testing +# and verification. + +import vllm +from vllm.lora.request import LoRARequest + +from ..utils import multi_gpu_test + +MODEL_PATH = "deepseek-ai/DeepSeek-V2-Lite-Chat" + +PROMPT_TEMPLATE = "<|begin▁of▁sentence|>You are a helpful assistant.\n\nUser: {context}\n\nAssistant:" # noqa: E501 + + +def generate_and_test(llm: vllm.LLM, lora_path: str, lora_id: int): + prompts = [ + PROMPT_TEMPLATE.format(context="Who are you?"), + ] + sampling_params = vllm.SamplingParams(temperature=0, max_tokens=64) + outputs = llm.generate( + prompts, + sampling_params, + lora_request=LoRARequest(str(lora_id), lora_id, lora_path) if lora_id else None, + ) + # Print the outputs. + generated_texts: list[str] = [] + for output in outputs: + prompt = output.prompt + generated_text = output.outputs[0].text.strip() + generated_texts.append(generated_text) + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") + # return generated_texts + expected_lora_output = [ + "I am \u5f20\u5b50\u8c6a, an AI assistant developed by \u9648\u58eb\u680b.", # noqa: E501 + ] + for i in range(len(expected_lora_output)): + assert generated_texts[i].startswith(expected_lora_output[i]) + + +def test_deepseekv2_lora(deepseekv2_lora_files): + # We enable enforce_eager=True here to reduce VRAM usage for lora-test CI, + # Otherwise, the lora-test will fail due to CUDA OOM. + llm = vllm.LLM( + MODEL_PATH, + max_model_len=1024, + enable_lora=True, + max_loras=4, + enforce_eager=True, + trust_remote_code=True, + enable_chunked_prefill=True, + ) + generate_and_test(llm, deepseekv2_lora_files, 1) + + +def test_deepseekv2(deepseekv2_lora_files): + # We enable enforce_eager=True here to reduce VRAM usage for lora-test CI, + # Otherwise, the lora-test will fail due to CUDA OOM. + llm = vllm.LLM( + MODEL_PATH, + max_model_len=1024, + enable_lora=True, + max_loras=4, + enforce_eager=True, + trust_remote_code=True, + ) + generate_and_test(llm, deepseekv2_lora_files, 1) + + +@multi_gpu_test(num_gpus=2) +def test_deepseekv2_tp2(deepseekv2_lora_files): + # We enable enforce_eager=True here to reduce VRAM usage for lora-test CI, + # Otherwise, the lora-test will fail due to CUDA OOM. + llm = vllm.LLM( + MODEL_PATH, + max_model_len=1024, + enable_lora=True, + max_loras=4, + enforce_eager=True, + trust_remote_code=True, + tensor_parallel_size=2, + ) + generate_and_test(llm, deepseekv2_lora_files, 2) + + +@multi_gpu_test(num_gpus=4) +def test_deepseekv2_tp4(deepseekv2_lora_files): + # We enable enforce_eager=True here to reduce VRAM usage for lora-test CI, + # Otherwise, the lora-test will fail due to CUDA OOM. + llm = vllm.LLM( + MODEL_PATH, + max_model_len=1024, + enable_lora=True, + max_loras=4, + enforce_eager=True, + trust_remote_code=True, + tensor_parallel_size=4, + ) + generate_and_test(llm, deepseekv2_lora_files, 2) diff --git a/asian_ai/performance/vLLM/tests/lora/test_default_mm_loras.py b/asian_ai/performance/vLLM/tests/lora/test_default_mm_loras.py new file mode 100644 index 0000000000000000000000000000000000000000..365102ab2339daa661f64287055d14de818fb748 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_default_mm_loras.py @@ -0,0 +1,182 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Tests for applying default registered multimodal loras. +""" + +import os +import unittest.mock as mock + +import pytest +from huggingface_hub import snapshot_download + +from vllm.lora.request import LoRARequest +from vllm.platforms import current_platform + +from ..conftest import AudioTestAssets, VllmRunner +from ..utils import create_new_process_for_each_test + +MODEL_PATH = snapshot_download("microsoft/Phi-4-multimodal-instruct") +AUDIO_LORA_PATH = os.path.join(MODEL_PATH, "speech-lora") +IMAGE_LORA_PATH = os.path.join(MODEL_PATH, "vision-lora") + +AUDIO_PROMPT = "<|user|><|audio_1|>Can you transcribe this audio?<|end|><|assistant|>" # noqa: E501 + +# Responses are greedy decoded; we just check the end of +# the generated text. If the lora is inactive, this model +# generates commentary on the transcription. +RESPONSE_SUFFIX_WITH_LORA = "Spoken text: The first words I spoke in the original chronograph, a little piece of practical poetry. Mary had a little lamb, it slept with quite a snow, and everywhere that Mary went, the lamb was sure to go." # noqa: E501 +RESPONSE_SUFFIX_WITHOUT_LORA = "Certainly! Here is the transcription of the audio you provided:\n\nThe first words I spoke in the original phonograph record: A little piece of practical poetry. Mary had a little lamb; its fleece was white as snow, and everywhere that Mary went, the lamb was sure to go." # noqa: E501 + +VLLM_RUNNER_BASE_KWARGS = { + "model_name": MODEL_PATH, + "dtype": "half", + "enable_lora": "True", + "max_num_seqs": 2, + "max_lora_rank": 320, + # Keep these LoRA tests on short-RoPE for determinism post-LongRoPE change. + "max_model_len": 4096, + "gpu_memory_utilization": 0.8, + "limit_mm_per_prompt": {"audio": 1}, + "enforce_eager": True, +} + + +def run_test( + vllm_runner, audio_assets, monkeypatch, lora_request, expected_suffix, **kwargs +): + monkeypatch.setenv("VLLM_WORKER_MULTIPROC_METHOD", "spawn") + + inputs = [([AUDIO_PROMPT], [audio_assets[0].audio_and_sample_rate[0]])] + + # Apply any additional kwargs as overrides to the base kwargs + vllm_runner_kwargs = {**VLLM_RUNNER_BASE_KWARGS, **kwargs} + + with vllm_runner(**vllm_runner_kwargs) as vllm_model: + vllm_outputs_with_default_lora = [ + vllm_model.generate_greedy( + prompts, + max_tokens=128, + audios=audios, + lora_request=lora_request, + ) + for prompts, audios in inputs + ] + + assert vllm_outputs_with_default_lora[-1][-1][-1].endswith(expected_suffix) + + +@create_new_process_for_each_test() +def test_active_default_mm_lora( + vllm_runner: type[VllmRunner], + audio_assets: AudioTestAssets, + monkeypatch: pytest.MonkeyPatch, +): + """Ensure that we can use the default audio lora.""" + run_test( + vllm_runner, + audio_assets, + monkeypatch, + lora_request=None, + default_mm_loras={"audio": AUDIO_LORA_PATH}, + expected_suffix=RESPONSE_SUFFIX_WITH_LORA, + ) + + +@pytest.mark.skipif( + current_platform.is_cuda_alike(), reason="Skipping to avoid redundant model tests" +) +@create_new_process_for_each_test() +def test_inactive_default_mm_lora( + vllm_runner: type[VllmRunner], + audio_assets: AudioTestAssets, + monkeypatch: pytest.MonkeyPatch, +): + """Ensure that modalities are filtered properly.""" + # Default image lora won't be active since we only pass audio + run_test( + vllm_runner, + audio_assets, + monkeypatch, + lora_request=None, + default_mm_loras={"image": IMAGE_LORA_PATH}, + expected_suffix=RESPONSE_SUFFIX_WITHOUT_LORA, + ) + + +@pytest.mark.skipif( + current_platform.is_cuda_alike(), reason="Skipping to avoid redundant model tests" +) +@create_new_process_for_each_test() +def test_default_mm_lora_succeeds_with_redundant_lora_request( + vllm_runner: type[VllmRunner], + audio_assets: AudioTestAssets, + monkeypatch: pytest.MonkeyPatch, +): + """Ensure that redundantly providing the lora works.""" + run_test( + vllm_runner, + audio_assets, + monkeypatch, + lora_request=LoRARequest("audio", 1, AUDIO_LORA_PATH), + default_mm_loras={"audio": AUDIO_LORA_PATH}, + expected_suffix=RESPONSE_SUFFIX_WITH_LORA, + ) + + +@pytest.mark.skipif( + current_platform.is_cuda_alike(), reason="Skipping to avoid redundant model tests" +) +@create_new_process_for_each_test() +def test_default_mm_lora_fails_with_overridden_lora_request( + vllm_runner: type[VllmRunner], + audio_assets: AudioTestAssets, + monkeypatch: pytest.MonkeyPatch, +): + """Ensure that if the lora_request conflicts with default_mm_loras, + we use the lora_request.""" + run_test( + vllm_runner, + audio_assets, + monkeypatch, + lora_request=LoRARequest("speech", 2, AUDIO_LORA_PATH), + default_mm_loras={"audio": IMAGE_LORA_PATH}, + expected_suffix=RESPONSE_SUFFIX_WITH_LORA, + ) + + +@create_new_process_for_each_test() +def test_default_mm_lora_does_not_expand_string_reqs(vllm_runner, monkeypatch): + # See run_test: force spawn to avoid the forked-child CUDA re-init crash. + monkeypatch.setenv("VLLM_WORKER_MULTIPROC_METHOD", "spawn") + + class MockEngineException(Exception): + pass + + # Regression test for ensuring default multimodal lora resolution + # does not expand the lora req if the prompt type is a string. + vllm_runner_kwargs = { + **VLLM_RUNNER_BASE_KWARGS, + **{"default_mm_loras": {"audio": AUDIO_LORA_PATH}}, + } + + # Avoid the full generation call since these tests are expensive; + # just check what lora request is actually submitted to the engine + mock_err = "Engine is mocked for this test" + + with ( + mock.patch( + "vllm.v1.engine.llm_engine.LLMEngine.add_request", + side_effect=MockEngineException(mock_err), + ) as mock_add_request, + vllm_runner(**vllm_runner_kwargs) as vllm_model, + ): + # Die once we actually submit the request to the engine + with pytest.raises(MockEngineException): + vllm_model.llm.generate(prompts=AUDIO_PROMPT) + + # Then check to make sure the submitted lora request + # and text prompt were zipped together correctly + engine_args, engine_kwargs = mock_add_request.call_args + assert engine_args[1]["prompt"] == AUDIO_PROMPT + assert engine_kwargs["lora_request"] is None diff --git a/asian_ai/performance/vLLM/tests/lora/test_fused_moe_lora_kernel.py b/asian_ai/performance/vLLM/tests/lora/test_fused_moe_lora_kernel.py new file mode 100644 index 0000000000000000000000000000000000000000..0297d31e21e99dfbde8c5e4f07cf2b183cb48efa --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_fused_moe_lora_kernel.py @@ -0,0 +1,1242 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import os +import random + +import pytest +import torch + +from tests.utils import ensure_current_vllm_config, multi_gpu_test +from vllm import _custom_ops as ops +from vllm.distributed import ( + init_distributed_environment, + initialize_model_parallel, + tensor_model_parallel_all_gather, + tensor_model_parallel_all_reduce, +) +from vllm.distributed.parallel_state import ( + get_tensor_model_parallel_world_size, +) +from vllm.lora.ops.triton_ops import fused_moe_lora +from vllm.platforms import current_platform +from vllm.utils.network_utils import get_open_port +from vllm.utils.torch_utils import set_random_seed + + +@pytest.fixture(autouse=True) +def reset_device(reset_default_device): + pass + + +def round_up(x, base): + return ((x + base - 1) // base) * base + + +def CEILDIV(x, y): + return (x + y - 1) // y + + +def assign_loras_to_tokens(num_tokens: int, num_sequences: int, max_loras: int): + """ + Split `num_tokens` into `num_sequences` sequences. + Each sequence randomly selects 1 LoRA index from [0, max_loras), + and all tokens in that sequence are assigned this LoRA index. + + Args: + num_tokens (int): Total number of tokens. + num_sequences (int): Number of sequences to split the tokens into. + max_loras (int): Total number of available LoRA modules. + + Returns: + torch.Tensor: 1D tensor of shape [num_tokens], where each value + is the LoRA index assigned to that token. + """ + assert num_sequences > 0 and max_loras > 0 + assert num_tokens >= num_sequences, "num_tokens must be >= num_sequences" + + # Compute token distribution per sequence (distribute remainder evenly) + tokens_per_seq = num_tokens // num_sequences + remainder = num_tokens % num_sequences + + token_lora_mapping = torch.empty(num_tokens, dtype=torch.int32) + + start = 0 + for seq_idx in range(num_sequences): + # Determine the token range for this sequence + end = start + tokens_per_seq + (1 if seq_idx < remainder else 0) + + # Randomly select one LoRA ID for this sequence + lora_id = random.randint(0, max_loras - 1) + + # Assign the same LoRA ID to all tokens in this sequence + token_lora_mapping[start:end] = lora_id + + start = end + + return token_lora_mapping + + +def assign_experts_to_tokens(num_tokens: int, num_experts: int, top_k_num: int): + """ + For each token, randomly select `top_k_num` distinct experts out of `num_experts`, + and assign normalized random weights that sum to 1. + + Args: + num_tokens (int): Total number of tokens. + num_experts (int): Total number of available experts. + top_k_num (int): Number of experts to select per token. + + Returns: + expert_indices (torch.Tensor): shape [num_tokens, top_k_num], + expert index for each token. + expert_weights (torch.Tensor): shape [num_tokens, top_k_num], + normalized weights (sum = 1 per row). + """ + assert top_k_num <= num_experts, "top_k_num must be <= num_experts" + + # Randomly select top_k_num distinct experts for each token + expert_indices = torch.empty((num_tokens, top_k_num), dtype=torch.int32) + for i in range(num_tokens): + # Randomly choose unique expert indices + selected = torch.randperm(num_experts)[:top_k_num] + expert_indices[i] = selected + + # Generate random weights and normalize along dim=1 + expert_weights = torch.rand((num_tokens, top_k_num), dtype=torch.float32) + expert_weights = expert_weights / expert_weights.sum(dim=1, keepdim=True) + + return expert_indices, expert_weights + + +def sample_data( + num_tokens: int, + num_sequences: int, + max_loras: int, + num_experts: int, + top_k_num: int, +): + topk_ids, topk_weights = assign_experts_to_tokens( + num_tokens, num_experts, top_k_num + ) + token_lora_mapping = assign_loras_to_tokens(num_tokens, num_sequences, max_loras) + active_lora_ids = torch.full((max_loras + 1,), -1, dtype=torch.int32) + lora_ids = torch.unique(token_lora_mapping, sorted=True) + active_lora_ids[: lora_ids.size(0)].copy_(lora_ids, non_blocking=True) + return topk_ids, topk_weights, token_lora_mapping, active_lora_ids + + +def use_fused_moe_lora_kernel( + topk_ids, + topk_weights, + token_lora_mapping, + max_lora_rank, + top_k_num, + lora_ids, + lora_a_stacked, + lora_b_stacked, + hidden_states, + output, + max_loras, + num_experts, + block_size, + fully_sharded=False, + offset=0, + add_inputs=True, +): + max_num_tokens_padded = topk_ids.numel() + num_experts * (block_size - 1) + max_num_tokens_padded = round_up(max_num_tokens_padded, block_size) + max_num_m_blocks = CEILDIV(max_num_tokens_padded, block_size) + + # init output tensors + sorted_token_ids = torch.empty( + (max_loras * max_num_tokens_padded,), + dtype=torch.int32, + ) + expert_ids = torch.empty((max_loras * max_num_m_blocks,), dtype=torch.int32) + num_tokens_post_padded = torch.empty((max_loras,), dtype=torch.int32) + adapter_enabled = torch.ones(max_loras + 1, dtype=torch.int32) + + # call kernel + ops.moe_lora_align_block_size( + topk_ids, + token_lora_mapping, + num_experts, + block_size, + max_loras, + max_num_tokens_padded, + max_num_m_blocks, + sorted_token_ids, + expert_ids, + num_tokens_post_padded, + adapter_enabled, + lora_ids, + ) + + config = { + "BLOCK_SIZE_M": block_size, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 1, + "NUM_WARPS": 4, + "NUM_STAGES": 3, + "SPLIT_K": 1, + } + + mul_routed_weight = False + expert_ids = expert_ids.view(max_loras, -1) + sorted_token_ids = sorted_token_ids.view(max_loras, -1) + + # num_active_loras is the number of active LoRAs + # (max_loras + 1 to include no-lora case) + # Stored as CPU tensor to match the kernel API (torch.compile compatibility) + num_active_loras = torch.tensor([max_loras + 1], dtype=torch.int32, device="cpu") + + fused_moe_lora( + output, + hidden_states, + lora_a_stacked, + lora_b_stacked, + topk_weights, + sorted_token_ids, + expert_ids, + num_tokens_post_padded, + token_lora_mapping, + max_lora_rank, + top_k_num, + lora_ids, + num_active_loras, + adapter_enabled, + config["BLOCK_SIZE_M"], + config["BLOCK_SIZE_N"], + config["BLOCK_SIZE_K"], + config["GROUP_SIZE_M"], + config["NUM_WARPS"], + config["NUM_STAGES"], + config["SPLIT_K"], + config["BLOCK_SIZE_M"], + config["BLOCK_SIZE_N"], + config["BLOCK_SIZE_K"], + config["GROUP_SIZE_M"], + config["NUM_WARPS"], + config["NUM_STAGES"], + config["SPLIT_K"], + mul_routed_weight, + fully_sharded=fully_sharded, + offset=offset, + add_inputs=add_inputs, + ) + + +def use_torch( + hidden_states, + token_lora_mapping, + topk_ids, + lora_a_stacked, + lora_b_stacked, + top_k_num, + num_slices=1, +): + outputs = [] + for i in range(hidden_states.shape[0]): + slice_tensors = [] + for slice_id in range(num_slices): + lora_idx = token_lora_mapping[i] + expert_ids = topk_ids[i] + lora_a = lora_a_stacked[slice_id][lora_idx][expert_ids] + lora_b = lora_b_stacked[slice_id][lora_idx][expert_ids] + tensors = [ + hidden_states[i] @ lora_a[x].T @ lora_b[x].T for x in range(top_k_num) + ] + slice_tensors.append(torch.stack(tensors, dim=0)) + + outputs.append(torch.concat(slice_tensors, dim=-1)) + return torch.stack(outputs, dim=0) + + +DEVICE_TYPE = current_platform.device_type +DTYPES = [torch.float16, torch.bfloat16] +DEVICES = [f"{DEVICE_TYPE}:{0}"] +SEED = [42] + + +@pytest.mark.parametrize("num_tokens", [100]) +@pytest.mark.parametrize("top_k_num", [6, 12]) +@pytest.mark.parametrize("num_experts", [64]) +@pytest.mark.parametrize("max_loras", [4, 6, 16]) +@pytest.mark.parametrize("N", [1408]) +@pytest.mark.parametrize("K", [2048]) +@pytest.mark.parametrize("max_lora_rank", [16, 32, 64]) +@pytest.mark.parametrize("block_size", [16]) +@pytest.mark.parametrize("num_slices", [1, 2]) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("device", DEVICES) +@pytest.mark.parametrize("seed", SEED) +def test_fused_moe_lora_kernel( + num_tokens, + top_k_num, + num_experts, + max_loras, + N, + K, + max_lora_rank, + block_size, + num_slices, + dtype, + device, + seed, +): + torch.set_default_device(device) + set_random_seed(seed) + # the number of randomly generated sentences. + num_sequences = 10 + # generate data + topk_ids, topk_weights, token_lora_mapping, lora_ids = sample_data( + num_tokens, num_sequences, max_loras, num_experts, top_k_num + ) + + # init lora weights + lora_a_stacked = [ + torch.rand( + ( + max_loras, + num_experts, + max_lora_rank, + K, + ), + dtype=dtype, + ) + for _ in range(num_slices) + ] + lora_b_stacked = [ + torch.rand( + ( + max_loras, + num_experts, + N // num_slices, + max_lora_rank, + ), + dtype=dtype, + ) + for _ in range(num_slices) + ] + hidden_states = torch.rand( + ( + num_tokens, + K, + ), + dtype=dtype, + ) + + # fused_moe_lora_kernel output + output = torch.zeros((num_tokens, top_k_num, N), dtype=dtype) + use_fused_moe_lora_kernel( + topk_ids, + topk_weights, + token_lora_mapping, + max_lora_rank, + top_k_num, + lora_ids, + lora_a_stacked, + lora_b_stacked, + hidden_states, + output, + max_loras, + num_experts, + block_size, + ) + # pytorch output + output2 = use_torch( + hidden_states, + token_lora_mapping, + topk_ids, + lora_a_stacked, + lora_b_stacked, + top_k_num, + num_slices, + ) + + torch.testing.assert_close(output, output2, atol=1e-2, rtol=1e-2) + + +def use_fused_moe_lora_kernel_naive( + topk_ids, + topk_weights, + token_lora_mapping, + max_lora_rank, + top_k_num, + lora_ids, + lora_a_stacked, + lora_b_stacked, + hidden_states, + output, + max_loras, + block_size, + fully_sharded=False, + offset=0, + add_inputs=True, +): + """ + Test helper for naive_block_assignment path. + Skips moe_lora_align_block_size and uses flattened topk_ids as expert_ids. + """ + config = { + "BLOCK_SIZE_M": block_size, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 64, + "GROUP_SIZE_M": 1, + "NUM_WARPS": 4, + "NUM_STAGES": 3, + "SPLIT_K": 1, + } + + mul_routed_weight = False + + # In naive mode: + # - expert_ids = topk_ids.view(-1), shape: (num_tokens * top_k,) + # - sorted_token_ids = None + # - num_tokens_post_padded = None + expert_ids = topk_ids.reshape(-1) + sorted_token_ids = None + num_tokens_post_padded = None + + adapter_enabled = torch.ones(max_loras + 1, dtype=torch.int32) + + # num_active_loras is the number of active LoRAs + # (max_loras + 1 to include no-lora case) + # Stored as CPU tensor to match the kernel API (torch.compile compatibility) + num_active_loras = torch.tensor([max_loras + 1], dtype=torch.int32, device="cpu") + + fused_moe_lora( + output, + hidden_states, + lora_a_stacked, + lora_b_stacked, + topk_weights, + sorted_token_ids, + expert_ids, + num_tokens_post_padded, + token_lora_mapping, + max_lora_rank, + top_k_num, + lora_ids, + num_active_loras, + adapter_enabled, + config["BLOCK_SIZE_M"], + config["BLOCK_SIZE_N"], + config["BLOCK_SIZE_K"], + config["GROUP_SIZE_M"], + config["NUM_WARPS"], + config["NUM_STAGES"], + config["SPLIT_K"], + config["BLOCK_SIZE_M"], + config["BLOCK_SIZE_N"], + config["BLOCK_SIZE_K"], + config["GROUP_SIZE_M"], + config["NUM_WARPS"], + config["NUM_STAGES"], + config["SPLIT_K"], + mul_routed_weight=mul_routed_weight, + fully_sharded=fully_sharded, + offset=offset, + add_inputs=add_inputs, + ) + + +@pytest.mark.parametrize("num_tokens", [1, 2, 4, 8]) +@pytest.mark.parametrize("top_k_num", [1, 2]) +@pytest.mark.parametrize("num_experts", [64, 128]) +@pytest.mark.parametrize("max_loras", [4, 8]) +@pytest.mark.parametrize("N", [1408]) +@pytest.mark.parametrize("K", [2048]) +@pytest.mark.parametrize("max_lora_rank", [16, 32]) +@pytest.mark.parametrize("block_size", [16]) +@pytest.mark.parametrize("num_slices", [1, 2]) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("device", DEVICES) +@pytest.mark.parametrize("seed", SEED) +def test_fused_moe_lora_kernel_naive_block_assignment( + num_tokens, + top_k_num, + num_experts, + max_loras, + N, + K, + max_lora_rank, + block_size, + num_slices, + dtype, + device, + seed, +): + """ + Test the naive_block_assignment path of the fused_moe_lora kernel. + This path is triggered when batch_size * top_k is much smaller than + num_experts * max_loras, and skips the moe_lora_align_block_size kernel. + """ + torch.set_default_device(device) + set_random_seed(seed) + + # Verify this configuration would trigger naive_block_assignment + # (num_tokens * top_k * SPARSITY_FACTOR <= num_experts * max_loras) + SPARSITY_FACTOR = 8 + assert num_tokens * top_k_num * SPARSITY_FACTOR <= num_experts * max_loras, ( + f"Test configuration doesn't meet naive_block_assignment condition: " + f"{num_tokens} * {top_k_num} * {SPARSITY_FACTOR} > {num_experts} * {max_loras}" + ) + + # the number of randomly generated sentences. + num_sequences = min(num_tokens, 4) + # generate data + topk_ids, topk_weights, token_lora_mapping, lora_ids = sample_data( + num_tokens, num_sequences, max_loras, num_experts, top_k_num + ) + + # init lora weights + lora_a_stacked = [ + torch.rand( + ( + max_loras, + num_experts, + max_lora_rank, + K, + ), + dtype=dtype, + ) + for _ in range(num_slices) + ] + lora_b_stacked = [ + torch.rand( + ( + max_loras, + num_experts, + N // num_slices, + max_lora_rank, + ), + dtype=dtype, + ) + for _ in range(num_slices) + ] + hidden_states = torch.rand( + ( + num_tokens, + K, + ), + dtype=dtype, + ) + + # fused_moe_lora_kernel output (naive path) + output = torch.zeros((num_tokens, top_k_num, N), dtype=dtype) + use_fused_moe_lora_kernel_naive( + topk_ids, + topk_weights, + token_lora_mapping, + max_lora_rank, + top_k_num, + lora_ids, + lora_a_stacked, + lora_b_stacked, + hidden_states, + output, + max_loras, + block_size, + ) + + # pytorch reference output + output_ref = use_torch( + hidden_states, + token_lora_mapping, + topk_ids, + lora_a_stacked, + lora_b_stacked, + top_k_num, + num_slices, + ) + + torch.testing.assert_close(output, output_ref, atol=1e-2, rtol=1e-2) + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize("num_tokens", [100]) +@pytest.mark.parametrize("top_k_num", [6]) +@pytest.mark.parametrize("num_experts", [64]) +@pytest.mark.parametrize("max_loras", [4]) +@pytest.mark.parametrize("N", [1408]) +@pytest.mark.parametrize("K", [2048]) +@pytest.mark.parametrize("max_lora_rank", [16, 32, 64]) +@pytest.mark.parametrize("block_size", [16]) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("seed", SEED) +@pytest.mark.parametrize("column_parallel", [True, False]) +def test_fused_moe_lora_kernel_fully_sharded( + num_tokens, + top_k_num, + num_experts, + max_loras, + N, + K, + max_lora_rank, + block_size, + dtype, + seed, + column_parallel, +): + set_random_seed(seed) + # the number of randomly generated sentences. + num_sequences = 10 + # generate data + topk_ids, topk_weights, token_lora_mapping, lora_ids = sample_data( + num_tokens, num_sequences, max_loras, num_experts, top_k_num + ) + + def run_torch_spawn(fn, nprocs): + torch.multiprocessing.spawn( + fn, + args=( + nprocs, + f"tcp://{os.getenv('LOCALHOST', 'localhost')}:{get_open_port()}", + dtype, + seed, + N, + K, + num_tokens, + topk_ids, + topk_weights, + token_lora_mapping, + max_lora_rank, + top_k_num, + lora_ids, + max_loras, + num_experts, + block_size, + column_parallel, + ), + nprocs=nprocs, + ) + + run_torch_spawn(use_fused_moe_lora_kernel_tensor_parallel, nprocs=2) + + +def use_fused_moe_lora_kernel_tensor_parallel( + local_rank, + world_size, + init_method, + dtype, + seed, + N, + K, + num_tokens, + topk_ids, + topk_weights, + token_lora_mapping, + max_lora_rank, + top_k_num, + lora_ids, + max_loras, + num_experts, + block_size, + column_parallel, +): + def _get_shard_slice(shard_size): + return slice(local_rank * shard_size, (local_rank + 1) * shard_size) + + set_random_seed(seed) + + device = torch.device(f"{DEVICE_TYPE}:{local_rank}") + torch.accelerator.set_device_index(device) + torch.set_default_device(device) + torch.set_default_dtype(dtype) + + init_distributed_environment( + world_size=world_size, + rank=local_rank, + local_rank=local_rank, + distributed_init_method=init_method, + backend=current_platform.dist_backend, + ) + with ensure_current_vllm_config(): + initialize_model_parallel(world_size, 1) + tp_size = get_tensor_model_parallel_world_size() + + input_dim = K if column_parallel else N + output_dim = N if column_parallel else K + + # init lora weights + lora_a = torch.rand( + ( + max_loras, + num_experts, + max_lora_rank, + input_dim, + ), + dtype=dtype, + ) + lora_b = torch.rand( + ( + max_loras, + num_experts, + output_dim, + max_lora_rank, + ), + dtype=dtype, + ) + + hidden_states = torch.rand( + ( + num_tokens, + input_dim, + ), + dtype=dtype, + ) + + output = torch.zeros((num_tokens, top_k_num, output_dim), dtype=dtype) + topk_ids = topk_ids.to(device) + topk_weights = topk_weights.to(device) + token_lora_mapping = token_lora_mapping.to(device) + lora_ids = lora_ids.to(device) + + ref_output = use_torch( + hidden_states, + token_lora_mapping, + topk_ids, + [lora_a], + [lora_b], + top_k_num, + ) + + if column_parallel: + # Column parallel (e.g. gate_up_proj): LoRA A is sliced along the rank dim, + # and Lora B is sliced along the output dim + lora_a_shard_size = max_lora_rank // tp_size + lora_a = lora_a[:, :, _get_shard_slice(lora_a_shard_size), :] + max_lora_rank = lora_a_shard_size + offset = 0 + + lora_b_shard_size = output_dim // tp_size + lora_b = lora_b[:, :, _get_shard_slice(lora_b_shard_size), :] + output = output[:, :, _get_shard_slice(lora_b_shard_size)].contiguous() + else: + # Row parallel (e.g. down proj): LoRA A is sliced along the input dim, + # and LoRA B is sliced along the output dim + lora_a_shard_size = input_dim // tp_size + lora_a = lora_a[:, :, :, _get_shard_slice(lora_a_shard_size)] + hidden_states = hidden_states[:, _get_shard_slice(lora_a_shard_size)] + + lora_b_shard_size = output_dim // tp_size + lora_b = lora_b[:, :, _get_shard_slice(lora_b_shard_size), :] + offset = lora_b_shard_size * local_rank + + use_fused_moe_lora_kernel( + topk_ids, + topk_weights, + token_lora_mapping, + max_lora_rank, + top_k_num, + lora_ids, + [lora_a], + [lora_b], + hidden_states, + output, + max_loras, + num_experts, + block_size, + fully_sharded=True, + offset=offset, + ) + + if column_parallel: + output = tensor_model_parallel_all_gather(output) + else: + output = tensor_model_parallel_all_reduce(output) + + torch.testing.assert_close(output, ref_output, atol=1e-2, rtol=1e-2) + + +# -- one-shot fast-path coverage -------------------------------------------- +# The fused shrink+expand one-shot kernel pads `BLOCK_R` to next_pow2(rank), +# with a floor of 16 (tensor-core minimum). Small ranks (4, 8) exercise the +# rank-dim masking and are not covered by the original tests, which start at +# rank=16. The legacy two-kernel path additionally fails on rank=4 in TMA +# mode because the rank-dim stride (rank * elem_size) is not 16-byte +# aligned; the one-shot fast path takes precedence whenever fully_sharded +# is False so this regression is hidden in normal use, but the test still +# ensures the one-shot logic is correct against the pytorch reference. + + +@pytest.mark.parametrize("num_tokens", [16, 100]) +@pytest.mark.parametrize("top_k_num", [2]) +@pytest.mark.parametrize("num_experts", [8, 64]) +@pytest.mark.parametrize("max_loras", [4]) +@pytest.mark.parametrize("N", [1408]) +@pytest.mark.parametrize("K", [2048]) +@pytest.mark.parametrize("max_lora_rank", [4, 8]) +@pytest.mark.parametrize("block_size", [16, 64]) +@pytest.mark.parametrize("num_slices", [1, 2]) +@pytest.mark.parametrize("dtype", DTYPES) +@pytest.mark.parametrize("device", DEVICES) +@pytest.mark.parametrize("seed", SEED) +def test_fused_moe_lora_kernel_small_rank( + num_tokens, + top_k_num, + num_experts, + max_loras, + N, + K, + max_lora_rank, + block_size, + num_slices, + dtype, + device, + seed, +): + """One-shot fast path covering rank<16 (padded to BLOCK_R=16 inside kernel).""" + torch.set_default_device(device) + set_random_seed(seed) + num_sequences = max(1, min(num_tokens, 8)) + topk_ids, topk_weights, token_lora_mapping, lora_ids = sample_data( + num_tokens, num_sequences, max_loras, num_experts, top_k_num + ) + + lora_a_stacked = [ + torch.rand( + (max_loras, num_experts, max_lora_rank, K), + dtype=dtype, + ) + for _ in range(num_slices) + ] + lora_b_stacked = [ + torch.rand( + (max_loras, num_experts, N // num_slices, max_lora_rank), + dtype=dtype, + ) + for _ in range(num_slices) + ] + hidden_states = torch.rand((num_tokens, K), dtype=dtype) + + output = torch.zeros((num_tokens, top_k_num, N), dtype=dtype) + use_fused_moe_lora_kernel( + topk_ids, + topk_weights, + token_lora_mapping, + max_lora_rank, + top_k_num, + lora_ids, + lora_a_stacked, + lora_b_stacked, + hidden_states, + output, + max_loras, + num_experts, + block_size, + ) + output_ref = use_torch( + hidden_states, + token_lora_mapping, + topk_ids, + lora_a_stacked, + lora_b_stacked, + top_k_num, + num_slices, + ) + torch.testing.assert_close(output, output_ref, atol=1e-2, rtol=1e-2) + + +@pytest.mark.parametrize("num_tokens", [16, 64]) +@pytest.mark.parametrize("top_k_num", [2]) +@pytest.mark.parametrize("num_experts", [8]) +@pytest.mark.parametrize("max_loras", [4]) +@pytest.mark.parametrize("N", [2048]) +@pytest.mark.parametrize("K", [4096]) +@pytest.mark.parametrize("max_lora_rank", [8, 16, 32, 64]) +@pytest.mark.parametrize("block_size", [64]) +@pytest.mark.parametrize("num_slices", [2]) +@pytest.mark.parametrize("dtype", [torch.bfloat16]) +@pytest.mark.parametrize("device", DEVICES) +@pytest.mark.parametrize("seed", SEED) +def test_fused_moe_lora_kernel_npid_path( + num_tokens, + top_k_num, + num_experts, + max_loras, + N, + K, + max_lora_rank, + block_size, + num_slices, + dtype, + device, + seed, +): + """Exercise the small-batch / NPID > 1 branch of the one-shot fast path. + + With these sizes the one-shot wrapper computes NPID_FACTOR > 1 (base CTA count + < SM count), so each program covers only an outer chunk of N. The + cross-outer-block write mask is the correctness-critical bit. + """ + torch.set_default_device(device) + set_random_seed(seed) + num_sequences = max(1, min(num_tokens, 4)) + topk_ids, topk_weights, token_lora_mapping, lora_ids = sample_data( + num_tokens, num_sequences, max_loras, num_experts, top_k_num + ) + + lora_a_stacked = [ + torch.rand( + (max_loras, num_experts, max_lora_rank, K), + dtype=dtype, + ) + for _ in range(num_slices) + ] + lora_b_stacked = [ + torch.rand( + (max_loras, num_experts, N // num_slices, max_lora_rank), + dtype=dtype, + ) + for _ in range(num_slices) + ] + hidden_states = torch.rand((num_tokens, K), dtype=dtype) + + output = torch.zeros((num_tokens, top_k_num, N), dtype=dtype) + use_fused_moe_lora_kernel( + topk_ids, + topk_weights, + token_lora_mapping, + max_lora_rank, + top_k_num, + lora_ids, + lora_a_stacked, + lora_b_stacked, + hidden_states, + output, + max_loras, + num_experts, + block_size, + ) + output_ref = use_torch( + hidden_states, + token_lora_mapping, + topk_ids, + lora_a_stacked, + lora_b_stacked, + top_k_num, + num_slices, + ) + torch.testing.assert_close(output, output_ref, atol=2e-2, rtol=2e-2) + + +# -- one-shot corner-case coverage ------------------------------------------ +# Each of the following exercises a path where the kernel is launched but +# every program early-exits, leaving the output unchanged. The contract is +# additive (`output += contribution`), so an empty contribution must leave +# the input residual untouched. + + +def _build_one_shot_inputs( + num_tokens, + top_k_num, + num_experts, + max_loras, + max_lora_rank, + K, + N, + num_slices, + block_size, + dtype, +): + """Common scaffolding for the corner-case tests below.""" + num_sequences = max(1, min(num_tokens, 4)) if num_tokens > 0 else 1 + if num_tokens > 0: + topk_ids, topk_weights, token_lora_mapping, lora_ids = sample_data( + num_tokens, num_sequences, max_loras, num_experts, top_k_num + ) + else: + # M=0 path: caller may still hand us empty tensors with the right shape. + topk_ids = torch.empty((0, top_k_num), dtype=torch.int32) + topk_weights = torch.empty((0, top_k_num), dtype=torch.float32) + token_lora_mapping = torch.empty((0,), dtype=torch.int32) + lora_ids = torch.full((max_loras + 1,), -1, dtype=torch.int32) + + lora_a_stacked = [ + torch.rand((max_loras, num_experts, max_lora_rank, K), dtype=dtype) + for _ in range(num_slices) + ] + lora_b_stacked = [ + torch.rand( + (max_loras, num_experts, N // num_slices, max_lora_rank), dtype=dtype + ) + for _ in range(num_slices) + ] + hidden_states = torch.rand((max(num_tokens, 0), K), dtype=dtype) + return ( + topk_ids, + topk_weights.to(dtype), + token_lora_mapping, + lora_ids, + lora_a_stacked, + lora_b_stacked, + hidden_states, + ) + + +def _call_one_shot( + output, + hidden_states, + lora_a_stacked, + lora_b_stacked, + topk_weights, + sorted_token_ids, + expert_ids, + num_tokens_post_padded, + token_lora_mapping, + max_lora_rank, + top_k_num, + lora_ids, + num_active_loras, + adapter_enabled, + block_size, + add_inputs=True, +): + """Direct call into fused_moe_lora with one-shot-routed defaults.""" + from vllm.lora.ops.triton_ops import fused_moe_lora as _op + + _op( + output, + hidden_states, + lora_a_stacked, + lora_b_stacked, + topk_weights, + sorted_token_ids, + expert_ids, + num_tokens_post_padded, + token_lora_mapping, + max_lora_rank, + top_k_num, + lora_ids, + num_active_loras, + adapter_enabled, + block_size, + 32, + 64, + 1, + 4, + 3, + 1, + block_size, + 32, + 64, + 1, + 4, + 3, + 1, + False, + False, + 0, + add_inputs, + ) + + +@pytest.mark.parametrize( + "trigger", + ["sorted_lora_ids_neg", "naive_mapping_neg", "naive_all_disabled"], +) +@pytest.mark.parametrize("device", DEVICES) +def test_fused_moe_lora_kernel_one_shot_early_exit(trigger, device): + """one-shot must leave the residual byte-identical when every program + must early-exit. Three trigger conditions are covered: + + - "sorted_lora_ids_neg": sorted path, lora_ids all -1 (lora_id<0 check) + - "naive_mapping_neg": naive path, token_lora_mapping all -1 + - "naive_all_disabled": naive path, adapter_enabled all 0 + """ + torch.set_default_device(device) + set_random_seed(0) + + # Per-trigger shapes: naive_mapping_neg needs the naive dispatch gate + # `num_tokens*top_k*8 <= num_experts*max_loras` to hold, hence the + # larger E/max_loras and smaller num_tokens. + if trigger == "naive_mapping_neg": + num_tokens, top_k, E, max_loras, R = 8, 2, 64, 8, 16 + elif trigger == "naive_all_disabled": + num_tokens, top_k, E, max_loras, R = 32, 2, 8, 4, 32 + else: # sorted_lora_ids_neg + num_tokens, top_k, E, max_loras, R = 32, 2, 8, 4, 16 + K, N = 1024, 1024 + block_size, num_slices, dtype = 16, 2, torch.bfloat16 + + ( + topk_ids, + topk_weights, + token_lora_mapping, + lora_ids, + lora_a_stacked, + lora_b_stacked, + hidden_states, + ) = _build_one_shot_inputs( + num_tokens, top_k, E, max_loras, R, K, N, num_slices, block_size, dtype + ) + + adapter_enabled = torch.ones(max_loras + 1, dtype=torch.int32) + num_active_loras = torch.tensor([max_loras + 1], dtype=torch.int32, device="cpu") + + if trigger == "sorted_lora_ids_neg": + lora_ids = torch.full((max_loras + 1,), -1, dtype=torch.int32) + max_pad = topk_ids.numel() + E * (block_size - 1) + max_pad = round_up(max_pad, block_size) + max_blocks = CEILDIV(max_pad, block_size) + sorted_token_ids = torch.zeros((max_loras, max_pad), dtype=torch.int32) + expert_ids = torch.full((max_loras, max_blocks), -1, dtype=torch.int32) + num_post = torch.zeros((max_loras,), dtype=torch.int32) + else: + sorted_token_ids = None + expert_ids = topk_ids.reshape(-1).contiguous() + num_post = None + if trigger == "naive_mapping_neg": + token_lora_mapping = torch.full((num_tokens,), -1, dtype=torch.int32) + lora_ids = torch.full((max_loras + 1,), -1, dtype=torch.int32) + else: # naive_all_disabled + adapter_enabled = torch.zeros(max_loras + 1, dtype=torch.int32) + + residual = torch.randn((num_tokens, top_k, N), dtype=dtype) * 0.1 + output = residual.clone() + + _call_one_shot( + output, + hidden_states, + lora_a_stacked, + lora_b_stacked, + topk_weights, + sorted_token_ids, + expert_ids, + num_post, + token_lora_mapping, + R, + top_k, + lora_ids, + num_active_loras, + adapter_enabled, + block_size, + ) + torch.testing.assert_close(output, residual, atol=0, rtol=0) + + +@pytest.mark.parametrize("device", DEVICES) +def test_fused_moe_lora_kernel_zero_grid_no_crash(device): + """num_active_loras=0 (or num_slices=0) would otherwise launch a grid + with a zero dimension. one-shot wrapper must short-circuit before launch.""" + torch.set_default_device(device) + set_random_seed(0) + num_tokens, top_k, E, max_loras, R, K, N = 8, 2, 8, 4, 16, 1024, 1024 + block_size, num_slices, dtype = 16, 2, torch.bfloat16 + + ( + topk_ids, + topk_weights, + token_lora_mapping, + lora_ids, + lora_a_stacked, + lora_b_stacked, + hidden_states, + ) = _build_one_shot_inputs( + num_tokens, + top_k, + E, + max_loras, + R, + K, + N, + num_slices, + block_size, + dtype, + ) + adapter_enabled = torch.ones(max_loras + 1, dtype=torch.int32) + num_active_loras = torch.tensor([0], dtype=torch.int32, device="cpu") + residual = torch.randn((num_tokens, top_k, N), dtype=dtype) * 0.1 + output = residual.clone() + + # sorted path is the one that uses num_active_loras for grid axis 2 + max_pad = topk_ids.numel() + E * (block_size - 1) + max_pad = round_up(max_pad, block_size) + max_blocks = CEILDIV(max_pad, block_size) + sorted_token_ids = torch.zeros((max_loras, max_pad), dtype=torch.int32) + expert_ids = torch.full((max_loras, max_blocks), -1, dtype=torch.int32) + num_post = torch.zeros((max_loras,), dtype=torch.int32) + _call_one_shot( + output, + hidden_states, + lora_a_stacked, + lora_b_stacked, + topk_weights, + sorted_token_ids, + expert_ids, + num_post, + token_lora_mapping, + R, + top_k, + lora_ids, + num_active_loras, + adapter_enabled, + block_size, + ) + torch.testing.assert_close(output, residual, atol=0, rtol=0) + + +@pytest.mark.parametrize("device", DEVICES) +def test_fused_moe_lora_kernel_rejects_bad_block_size_m(device): + """one-shot must surface a clear assertion when shrink_block_size_m is not + a power of 2 / less than 16, instead of the cryptic Triton compile + failure (`arange's range must be a power of 2`).""" + torch.set_default_device(device) + set_random_seed(0) + num_tokens, top_k, E, max_loras, R, K, N = 32, 2, 8, 4, 16, 1024, 1024 + num_slices, dtype = 2, torch.bfloat16 + block_size = 24 # NOT a power of 2 + + ( + topk_ids, + topk_weights, + token_lora_mapping, + lora_ids, + lora_a_stacked, + lora_b_stacked, + hidden_states, + ) = _build_one_shot_inputs( + num_tokens, + top_k, + E, + max_loras, + R, + K, + N, + num_slices, + 16, + dtype, + ) + # Build sorted-mode metadata at block_size=16 so shapes are sane, + # but pass block_size=24 to the op (the buggy combination). + max_pad = topk_ids.numel() + E * (16 - 1) + max_pad = round_up(max_pad, 16) + max_blocks = CEILDIV(max_pad, 16) + sorted_token_ids = torch.zeros((max_loras, max_pad), dtype=torch.int32) + expert_ids = torch.full((max_loras, max_blocks), -1, dtype=torch.int32) + num_post = torch.zeros((max_loras,), dtype=torch.int32) + adapter_enabled = torch.ones(max_loras + 1, dtype=torch.int32) + num_active_loras = torch.tensor([max_loras + 1], dtype=torch.int32, device="cpu") + output = torch.zeros((num_tokens, top_k, N), dtype=dtype) + + with pytest.raises(AssertionError, match="shrink_block_size_m"): + _call_one_shot( + output, + hidden_states, + lora_a_stacked, + lora_b_stacked, + topk_weights, + sorted_token_ids, + expert_ids, + num_post, + token_lora_mapping, + R, + top_k, + lora_ids, + num_active_loras, + adapter_enabled, + block_size, + ) diff --git a/asian_ai/performance/vLLM/tests/lora/test_gptoss_tp.py b/asian_ai/performance/vLLM/tests/lora/test_gptoss_tp.py new file mode 100644 index 0000000000000000000000000000000000000000..f676793b25469d001b980e9f1836feb3c96c461f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_gptoss_tp.py @@ -0,0 +1,172 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import importlib.metadata +from importlib.util import find_spec + +import pytest +import torch +from packaging import version + +import vllm +from vllm.lora.request import LoRARequest +from vllm.platforms import current_platform + +from ..utils import multi_gpu_test + +# Require amd-quark >= 0.12 on torch >= 2.11. +# Earlier torch releases work with older quark versions. See +# https://github.com/amd/Quark/issues/34 +# TODO: Remove once amd-quark>=0.12.0 +QUARK_TORCH_COMPATIBLE = find_spec("quark") is not None and ( + version.parse(importlib.metadata.version("amd-quark")) >= version.parse("0.12.0") + if version.parse(torch.__version__.split("+")[0]) >= version.parse("2.11") + else True +) + +if current_platform.is_rocm() and not QUARK_TORCH_COMPATIBLE: + pytest.skip( + "This test requires amd-quark >= 0.12 on torch >= 2.11.", + allow_module_level=True, + ) + +MODEL_PATH = "openai/gpt-oss-20b" + +PROMPT_TEMPLATE = """<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI. +Knowledge cutoff: 2024-06 +Current date: 2025-10-29 + +Reasoning: medium + +# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>user<|message|>I want you to act as a SQL terminal in front of an example database, you need only to return the sql command to me.Below is an instruction that describes a task, Write a response that appropriately completes the request. +" +##Instruction: +farm contains tables such as city, farm, farm_competition, competition_record. Table city has columns such as City_ID, Official_Name, Status, Area_km_2, Population, Census_Ranking. City_ID is the primary key. +Table farm has columns such as Farm_ID, Year, Total_Horses, Working_Horses, Total_Cattle, Oxen, Bulls, Cows, Pigs, Sheep_and_Goats. Farm_ID is the primary key. +Table farm_competition has columns such as Competition_ID, Year, Theme, Host_city_ID, Hosts. Competition_ID is the primary key. +Table competition_record has columns such as Competition_ID, Farm_ID, Rank. Competition_ID is the primary key. +The Host_city_ID of farm_competition is the foreign key of City_ID of city. +The Farm_ID of competition_record is the foreign key of Farm_ID of farm. +The Competition_ID of competition_record is the foreign key of Competition_ID of farm_competition. + + +###Input: +{context} + +###Response:<|end|><|start|>assistant<|channel|>final<|message|>""" # noqa: E501 + +EXPECTED_LORA_OUTPUT = [ + "SELECT avg(Working_Horses) FROM farm WHERE Total_Horses > 5000", + "SELECT max(Cows) , min(Cows) FROM farm", + "SELECT max(Cows) , min(Cows) FROM farm", +] + + +def generate_and_test(llm: vllm.LLM, lora_path: str, lora_id: int) -> None: + prompts = [ + PROMPT_TEMPLATE.format( + context="Give the average number of working horses on farms with more than 5000 total horses." # noqa: E501 + ), # noqa: E501 + PROMPT_TEMPLATE.format( + context="What are the maximum and minimum number of cows across all farms." + ), + PROMPT_TEMPLATE.format( + context="Return the maximum and minimum number of cows across all farms." + ), + ] + sampling_params = vllm.SamplingParams(temperature=0, max_tokens=64) + outputs = llm.generate( + prompts, + sampling_params, + lora_request=LoRARequest(str(lora_id), lora_id, lora_path) if lora_id else None, + ) + # Print the outputs. + generated_texts: list[str] = [] + for output in outputs: + prompt = output.prompt + generated_text = output.outputs[0].text.strip() + generated_texts.append(generated_text) + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") + for i in range(len(EXPECTED_LORA_OUTPUT)): + assert generated_texts[i].startswith(EXPECTED_LORA_OUTPUT[i]) + + +# TODO: make the Mxfp4MoeBackend.TRITON spawn-safe. +# For now just use TRITON_UNFUSED kernel +@pytest.mark.parametrize( + "mxfp4_use_marlin", + [ + False, + pytest.param( + True, + marks=pytest.mark.skipif( + current_platform.is_rocm(), reason="marlin not supported" + ), + ), + ], +) +@pytest.mark.parametrize("specialize_active_lora", [True, False]) +def test_gpt_oss_lora( + gptoss20b_lora_files, + mxfp4_use_marlin, + specialize_active_lora, +): + llm = vllm.LLM( + MODEL_PATH, + max_model_len=1024, + enable_lora=True, + max_loras=4, + max_lora_rank=8, + max_num_seqs=2, + max_num_batched_tokens=2048, + specialize_active_lora=specialize_active_lora, + moe_backend="marlin" if mxfp4_use_marlin else "auto", + linear_backend="marlin" if mxfp4_use_marlin else "auto", + compilation_config=vllm.config.CompilationConfig( # Avoid OOM + cudagraph_specialize_lora=False, + ), + ) + + generate_and_test(llm, gptoss20b_lora_files, lora_id=1) + generate_and_test(llm, gptoss20b_lora_files, lora_id=2) + + +@multi_gpu_test(num_gpus=2) +@pytest.mark.parametrize("fully_sharded_loras", [False, True]) +@pytest.mark.parametrize( + "mxfp4_use_marlin", + [ + False, + pytest.param( + True, + marks=pytest.mark.skipif( + current_platform.is_rocm(), reason="marlin not supported" + ), + ), + ], +) +def test_gpt_oss_lora_tp2( + gptoss20b_lora_files, + fully_sharded_loras, + mxfp4_use_marlin, +): + llm = vllm.LLM( + MODEL_PATH, + max_model_len=1024, + enable_lora=True, + max_loras=2, + max_num_seqs=2, + max_num_batched_tokens=2048, + tensor_parallel_size=2, + gpu_memory_utilization=0.8, + fully_sharded_loras=fully_sharded_loras, + enable_expert_parallel=not fully_sharded_loras, + moe_backend="marlin" if mxfp4_use_marlin else "auto", + linear_backend="marlin" if mxfp4_use_marlin else "auto", + compilation_config=vllm.config.CompilationConfig( + cudagraph_specialize_lora=False, + ), + ) + + generate_and_test(llm, gptoss20b_lora_files, lora_id=1) + generate_and_test(llm, gptoss20b_lora_files, lora_id=2) diff --git a/asian_ai/performance/vLLM/tests/lora/test_layers.py b/asian_ai/performance/vLLM/tests/lora/test_layers.py new file mode 100644 index 0000000000000000000000000000000000000000..86d04c69c2392ec219dbd2ec7fb48655e9270d99 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_layers.py @@ -0,0 +1,1759 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import random +from copy import deepcopy +from dataclasses import dataclass +from unittest.mock import patch + +import pytest +import torch +import torch.nn.functional as F + +from vllm.config.lora import LoRAConfig +from vllm.lora.layers import ( + BaseLayerWithLoRA, + ColumnParallelLinearWithLoRA, + ColumnParallelLinearWithShardedLoRA, + LogitsProcessorWithLoRA, + LoRAMapping, + MergedColumnParallelLinearVariableSliceWithLoRA, + MergedColumnParallelLinearWithLoRA, + MergedColumnParallelLinearWithShardedLoRA, + MergedQKVParallelLinearWithLoRA, + MergedQKVParallelLinearWithShardedLoRA, + QKVParallelLinearWithLoRA, + QKVParallelLinearWithShardedLoRA, + ReplicatedLinearWithLoRA, + RowParallelLinearWithLoRA, + RowParallelLinearWithShardedLoRA, + VocabParallelEmbeddingWithLoRA, +) +from vllm.lora.lora_weights import LoRALayerWeights, PackedLoRALayerWeights +from vllm.lora.punica_wrapper import get_punica_wrapper +from vllm.model_executor.layers.linear import ( + ColumnParallelLinear, + MergedColumnParallelLinear, + QKVParallelLinear, + ReplicatedLinear, + RowParallelLinear, +) +from vllm.model_executor.layers.logits_processor import LogitsProcessor +from vllm.model_executor.layers.vocab_parallel_embedding import ( + ParallelLMHead, + VocabParallelEmbedding, + get_masked_input_and_mask, +) +from vllm.model_executor.models.deepseek_v2 import DeepSeekV2FusedQkvAProjLinear +from vllm.platforms import current_platform +from vllm.utils.torch_utils import set_random_seed + +from .utils import DummyLoRAManager + +TOLERANCES = { + torch.float16: (5e-3, 5e-3), + torch.float32: (5e-3, 5e-3), + torch.bfloat16: (3e-2, 2e-2), +} + +pytestmark = pytest.mark.skipif( + not ( + current_platform.is_cuda_alike() + or current_platform.is_cpu() + or current_platform.is_xpu() + ), + reason="Backend not supported", +) + +DEVICE_TYPE = current_platform.device_type +DEVICES = ( + [ + f"{DEVICE_TYPE}:{i}" + for i in range(1 if torch.accelerator.device_count() == 1 else 2) + ] + if (current_platform.is_cuda_alike() or current_platform.is_xpu()) + else ["cpu"] +) + +# prefill stage(True) or decode stage(False) +STAGES = [True, False] + +NUM_RANDOM_SEEDS = 2 + +VOCAB_PARALLEL_EMBEDDING_TEST_NUM_RANDOM_SEEDS = 2 + + +@pytest.fixture(autouse=True) +def clean_cache_reset_device(reset_default_device): + # Release any memory we might be holding on to. CI runs OOMs otherwise. + from vllm.lora.ops.triton_ops.utils import _LORA_A_PTR_DICT, _LORA_B_PTR_DICT + + _LORA_B_PTR_DICT.clear() + _LORA_A_PTR_DICT.clear() + + yield + + +@pytest.fixture(autouse=True) +def skip_cuda_with_stage_false(request): + """ + On cuda-like platforms, we use the same kernels for prefill and decode + stage, and 'stage' is generally ignored, so we only need to test once. + """ + if current_platform.is_cuda_alike() or current_platform.is_xpu(): + try: + if hasattr(request.node, "callspec") and hasattr( + request.node.callspec, "params" + ): + params = request.node.callspec.params + if "stage" in params and params["stage"] is False: + pytest.skip("Skip test when stage=False") + except Exception: + pass + yield + + +def get_random_id_to_index( + num_loras: int, num_slots: int, log: bool = True +) -> list[int | None]: + """Creates a random lora_id_to_index mapping. + + Args: + num_loras: The number of active loras in the mapping. + num_slots: The number of slots in the mapping. Must be larger + than num_loras. + log: Whether to log the output. + """ + + if num_loras > num_slots: + raise ValueError( + f"num_loras is higher than num_slots: {num_loras} > {num_slots}. " + "num_loras must be less than or equal to num_slots." + ) + + slots: list[int | None] = [None] * num_slots + random_slot_selections = (torch.randperm(num_slots)[:num_loras]).tolist() + for lora_id, slot_idx in enumerate(random_slot_selections, start=1): + slots[slot_idx] = lora_id + + if log: + print(f"Created lora_id_to_index mapping: {slots}.") + + return slots + + +def populate_loras( + id_to_index: list[int | None], + layer: BaseLayerWithLoRA, + layer_weights: torch.Tensor, + repeats: int = 1, +) -> tuple[dict[int, LoRALayerWeights], dict[int, list[LoRALayerWeights]]]: + """This method populates the lora layers with lora weights. + + Args: + id_to_index: a list of lora ids. The index of the lora id + represents which memory slot the lora matrices are + stored in. A None value indicates a free slot. + layer: the LoRAlayer to populate. + layer_weights: the PyTorch tensor containing the layer's + weights. + repeats: must only be set for column parallel packed + layers. Indicates the number of loras to compose + together to create a single lora layer. + """ + + # Dictionary that maps the lora ID to the + # corresponding lora weights. + lora_dict: dict[int, LoRALayerWeights] = dict() + + # Dictionary that maps the lora ID to the + # corresponding subloras. + sublora_dict: dict[int, list[LoRALayerWeights]] = dict() + + for slot_idx, lora_id in enumerate(id_to_index): + if lora_id is not None: + subloras: list[LoRALayerWeights] = [] + sublora_len = layer_weights.shape[0] // repeats + for i in range(repeats): + sublora = DummyLoRAManager(layer_weights.device).init_random_lora( + module_name=f"fake_{i}", + weight=layer_weights, + ) + sublora.lora_b = sublora.lora_b[ + (sublora_len * i) : (sublora_len * (i + 1)), : + ] + sublora.optimize() + subloras.append(sublora) + + lora = PackedLoRALayerWeights.pack(subloras) if repeats > 1 else subloras[0] + + layer.set_lora( + slot_idx, + lora_a=lora.lora_a, + lora_b=lora.lora_b, + ) + + lora_dict[lora_id] = lora + sublora_dict[lora_id] = subloras + + return lora_dict, sublora_dict + + +def create_random_inputs( + active_lora_ids: list[int], + num_inputs: int, + input_size: tuple[int, ...], + input_range: tuple[float, float], + input_type: torch.dtype = torch.int, + device: torch.device = DEVICE_TYPE, +) -> tuple[list[torch.Tensor], list[int], list[int]]: + """Creates random inputs. + + Args: + active_lora_ids: lora IDs of active lora weights. + num_inputs: the number of inputs to create. + input_size: the size of each individual input. + input_range: the range of values to include in the input. + input_range[0] <= possible input values < input_range[1] + input_type: the type of values in the input. + """ + + low, high = input_range + + inputs: list[torch.Tensor] = [] + index_mapping: list[int] = [] + prompt_mapping: list[int] = [] + + for _ in range(num_inputs): + if input_type == torch.int: + inputs.append( + torch.randint( + low=int(low), high=int(high), size=input_size, device=device + ) + ) + else: + inputs.append( + torch.rand(size=input_size, dtype=input_type, device=device) * high + + low + ) + + lora_id = random.choice(active_lora_ids) + index_mapping += [lora_id] * input_size[0] + prompt_mapping += [lora_id] + + return inputs, index_mapping, prompt_mapping + + +def check_punica_wrapper(punica_wrapper) -> bool: + if current_platform.is_cuda_alike(): + from vllm.lora.punica_wrapper.punica_gpu import PunicaWrapperGPU + + return type(punica_wrapper) is PunicaWrapperGPU + elif current_platform.is_cpu(): + from vllm.lora.punica_wrapper.punica_cpu import PunicaWrapperCPU + + return type(punica_wrapper) is PunicaWrapperCPU + elif current_platform.is_xpu(): + from vllm.lora.punica_wrapper.punica_xpu import PunicaWrapperXPU + + return type(punica_wrapper) is PunicaWrapperXPU + else: + return False + + +@torch.inference_mode() +@pytest.mark.parametrize("num_loras", [1, 2, 4]) +@pytest.mark.parametrize("device", DEVICES) +@pytest.mark.parametrize("vocab_size", [512, 32000, 64000, 128000]) +@pytest.mark.parametrize("stage", STAGES) +def test_embeddings( + default_vllm_config, dist_init, num_loras, device, vocab_size, stage +) -> None: + # For multi-GPU testing of Triton kernel, we must explicitly set the CUDA + # device, see: https://github.com/triton-lang/triton/issues/2925 + # Same below. + if current_platform.is_cuda_alike() or current_platform.is_xpu(): + torch.accelerator.set_device_index(device) + + torch.set_default_device(device) + max_loras = 8 + lora_config = LoRAConfig( + max_loras=max_loras, max_lora_rank=8, lora_dtype=torch.float16 + ) + punica_wrapper = get_punica_wrapper(8192, 256, device, lora_config=lora_config) + assert check_punica_wrapper(punica_wrapper) + + def create_random_embedding_layer(): + embedding = VocabParallelEmbedding(vocab_size, 256) + embedding.weight.data = torch.rand_like(embedding.weight.data) + embedding.weight.data[vocab_size:, :] = 0 + lora_embedding = VocabParallelEmbeddingWithLoRA(embedding) + lora_embedding.create_lora_weights(max_loras, lora_config) + + return embedding, lora_embedding + + for i in range(NUM_RANDOM_SEEDS): + set_random_seed(i) + + id_to_index = get_random_id_to_index(num_loras, max_loras) + embedding, lora_embedding = create_random_embedding_layer() + lora_embedding.set_mapping(punica_wrapper) + lora_dict, _ = populate_loras( + id_to_index, + layer=lora_embedding, + layer_weights=embedding.weight.T, + ) + + inputs, index_mapping, prompt_mapping = create_random_inputs( + active_lora_ids=list(lora_dict.keys()), + num_inputs=num_loras * 3, + input_size=(200,), + input_range=(1, vocab_size), + device=device, + ) + lora_mapping = LoRAMapping(index_mapping, prompt_mapping, is_prefill=stage) + punica_wrapper.update_metadata( + lora_mapping, + id_to_index, + max_loras, + vocab_size, + ) + + lora_result = lora_embedding(torch.cat(inputs)) + + expected_results: list[torch.Tensor] = [] + for input_, lora_id in zip(inputs, prompt_mapping): + lora = lora_dict[lora_id] + result = embedding(input_) + after_a = F.embedding( + input_, + lora.lora_a.T, + ) + result += after_a @ lora.lora_b.T + expected_results.append(result) + expected_result = torch.cat(expected_results) + + rtol, atol = TOLERANCES[lora_result.dtype] + torch.testing.assert_close(lora_result, expected_result, rtol=rtol, atol=atol) + + # Check that resetting the lora weights succeeds + + for slot_idx in range(max_loras): + lora_embedding.reset_lora(slot_idx) + + inputs, index_mapping, prompt_mapping = create_random_inputs( + active_lora_ids=[0], + num_inputs=num_loras * 3, + input_size=(200,), + input_range=(1, vocab_size), + device=device, + ) + lora_mapping = LoRAMapping(index_mapping, prompt_mapping, is_prefill=stage) + punica_wrapper.update_metadata( + lora_mapping, + id_to_index, + max_loras, + vocab_size, + ) + + lora_result = lora_embedding(torch.cat(inputs)) + expected_result = embedding(torch.cat(inputs)) + + rtol, atol = TOLERANCES[lora_result.dtype] + torch.testing.assert_close(lora_result, expected_result, rtol=rtol, atol=atol) + + +@torch.inference_mode() +@pytest.mark.parametrize("num_loras", [1, 2, 4]) +@pytest.mark.parametrize("device", DEVICES) +@pytest.mark.parametrize("vocab_size", [64000, 256512, 258048]) +@pytest.mark.parametrize("stage", STAGES) +def test_lm_head_logits_processor( + default_vllm_config, dist_init, num_loras, device, vocab_size, stage +) -> None: + if current_platform.is_cuda_alike() or current_platform.is_xpu(): + torch.accelerator.set_device_index(device) + + torch.set_default_device(device) + max_loras = 8 + lora_config = LoRAConfig( + max_loras=max_loras, max_lora_rank=8, lora_dtype=torch.float16 + ) + punica_wrapper = get_punica_wrapper(8192, 256, device, lora_config=lora_config) + assert check_punica_wrapper(punica_wrapper) + + def _pretest(): + linear = ParallelLMHead( + num_embeddings=vocab_size, + embedding_dim=1024, + params_dtype=torch.float16, + ) + linear.weight.data = torch.rand_like(linear.weight.data) + linear.weight.data[:, vocab_size:] = 0 + logits_processor = LogitsProcessor(vocab_size) + lora_logits_processor = LogitsProcessorWithLoRA( + logits_processor, 1024, linear.weight.dtype, linear.weight.device, None + ) + lora_logits_processor.create_lora_weights(max_loras, lora_config) + + return linear, logits_processor, lora_logits_processor + + for i in range(NUM_RANDOM_SEEDS): + set_random_seed(i) + + id_to_index = get_random_id_to_index(num_loras, max_loras) + linear, logits_processor, lora_logits_processor = _pretest() + lora_logits_processor.set_mapping(punica_wrapper) + + lora_dict, _ = populate_loras( + id_to_index, + layer=lora_logits_processor, + layer_weights=linear.weight, + ) + + inputs, index_mapping, prompt_mapping = create_random_inputs( + active_lora_ids=list(lora_dict.keys()), + num_inputs=8 * num_loras, # * 3, + input_size=(1, 1024), + input_range=(0, 1), + input_type=torch.float16, + device=device, + ) + lora_mapping = LoRAMapping(index_mapping, prompt_mapping, is_prefill=stage) + punica_wrapper.update_metadata( + lora_mapping, + id_to_index, + max_loras, + vocab_size, + ) + input_ = torch.rand(20, 1024) + + lora_result = lora_logits_processor._get_logits( + hidden_states=torch.cat(inputs), lm_head=linear, embedding_bias=None + ) + + original_lm_head = deepcopy(linear) + + expected_results: list[torch.Tensor] = [] + for input_, lora_id in zip(inputs, prompt_mapping): + lora = lora_dict[lora_id] + result = logits_processor._get_logits( + hidden_states=input_, lm_head=linear, embedding_bias=None + ) + + result += input_ @ lora.lora_a.T @ lora.lora_b.T * lora.scaling + expected_results.append(result) + expected_result = torch.cat(expected_results) + + # Check that resetting the lora weights succeeds + + for slot_idx in range(max_loras): + lora_logits_processor.reset_lora(slot_idx) + + inputs, index_mapping, prompt_mapping = create_random_inputs( + active_lora_ids=[0], + num_inputs=8 * num_loras * 3, + input_size=(1, 1024), + input_range=(0, 1), + input_type=torch.float16, + device=device, + ) + lora_mapping = LoRAMapping(index_mapping, prompt_mapping, is_prefill=stage) + punica_wrapper.update_metadata( + lora_mapping, + id_to_index, + max_loras, + vocab_size, + ) + + lora_result = lora_logits_processor._get_logits( + hidden_states=torch.cat(inputs), + lm_head=original_lm_head, + embedding_bias=None, + )[:, :vocab_size] + expected_result = logits_processor._get_logits( + hidden_states=torch.cat(inputs), + lm_head=original_lm_head, + embedding_bias=None, + ) + + rtol, atol = TOLERANCES[lora_result.dtype] + torch.testing.assert_close(lora_result, expected_result, rtol=rtol, atol=atol) + + +@torch.inference_mode() +@pytest.mark.parametrize("vocab_size", [258049, 300000]) +@pytest.mark.parametrize("device", DEVICES) +def test_lm_head_logits_processor_invalid_vocab_size( + default_vllm_config, dist_init, vocab_size, device +) -> None: + """Test that LogitsProcessorWithLoRA raises ValueError for invalid vocab sizes.""" + if current_platform.is_cuda_alike() or current_platform.is_xpu(): + torch.accelerator.set_device_index(device) + + torch.set_default_device(device) + max_loras = 8 + lora_config = LoRAConfig( + max_loras=max_loras, max_lora_rank=8, lora_dtype=torch.float16 + ) + + logits_processor = LogitsProcessor(vocab_size) + lora_logits_processor = LogitsProcessorWithLoRA( + logits_processor, 1024, torch.float16, device, None + ) + + with pytest.raises(ValueError, match="vocab size must be <= 258048"): + lora_logits_processor.create_lora_weights(max_loras, lora_config) + + +@torch.inference_mode() +@pytest.mark.parametrize("num_loras", [1, 2, 4]) +@pytest.mark.parametrize("device", DEVICES) +@pytest.mark.parametrize("stage", STAGES) +def test_linear_replicated( + default_vllm_config, + dist_init, + num_loras, + device, + stage, +) -> None: + if current_platform.is_cuda_alike() or current_platform.is_xpu(): + torch.accelerator.set_device_index(device) + + max_loras = 8 + torch.set_default_device(device) + lora_config = LoRAConfig( + max_loras=max_loras, + max_lora_rank=8, + lora_dtype=torch.float16, + ) + punica_wrapper = get_punica_wrapper(8192, 256, device, lora_config=lora_config) + assert check_punica_wrapper(punica_wrapper) + + def create_random_linear_replicated_layer(idx: int = 0): + linear = ReplicatedLinear( + 4096, 4096, bias=False, params_dtype=torch.float16, prefix=f"layer_{idx}" + ) + linear.weight.data = torch.rand_like(linear.weight.data) + lora_linear = ReplicatedLinearWithLoRA(linear) + + lora_linear.create_lora_weights(max_loras, lora_config) + assert ( + lora_linear.n_slices + == len(lora_linear.lora_a_stacked) + == len(lora_linear.lora_b_stacked) + == 1 + ) + return linear, lora_linear + + for i in range(NUM_RANDOM_SEEDS): + set_random_seed(i) + + id_to_index = get_random_id_to_index(num_loras, max_loras) + linear, lora_linear = create_random_linear_replicated_layer(i) + assert torch.equal(linear.weight, lora_linear.weight) + lora_linear.set_mapping(punica_wrapper) + lora_dict, _ = populate_loras( + id_to_index, + layer=lora_linear, + layer_weights=linear.weight, + ) + + inputs, index_mapping, prompt_mapping = create_random_inputs( + active_lora_ids=list(lora_dict.keys()), + num_inputs=32 * num_loras, + input_size=(1, 4096), + input_range=(0, 1), + input_type=torch.float16, + device=device, + ) + lora_mapping = LoRAMapping(index_mapping, prompt_mapping, is_prefill=stage) + punica_wrapper.update_metadata( + lora_mapping, + id_to_index, + max_loras, + 512, + ) + + lora_result = lora_linear(torch.cat(inputs))[0] + + expected_results: list[torch.Tensor] = [] + for input_, lora_id in zip(inputs, prompt_mapping): + lora = lora_dict[lora_id] + result = linear(input_)[0] + result += input_ @ lora.lora_a.T @ lora.lora_b.T * lora.scaling + expected_results.append(result) + expected_result = torch.cat(expected_results) + + rtol, atol = TOLERANCES[lora_result.dtype] + torch.testing.assert_close(lora_result, expected_result, rtol=rtol, atol=atol) + + # Check that resetting the lora weights succeeds + + for slot_idx in range(max_loras): + lora_linear.reset_lora(slot_idx) + + inputs, index_mapping, prompt_mapping = create_random_inputs( + active_lora_ids=[0], + num_inputs=32 * num_loras, + input_size=(1, 4096), + input_range=(0, 1), + input_type=torch.float16, + device=device, + ) + lora_mapping = LoRAMapping(index_mapping, prompt_mapping, is_prefill=stage) + + punica_wrapper.update_metadata( + lora_mapping, + id_to_index, + max_loras, + 512, + ) + + lora_result = lora_linear(torch.cat(inputs))[0] + expected_result = linear(torch.cat(inputs))[0] + + rtol, atol = TOLERANCES[lora_result.dtype] + torch.testing.assert_close(lora_result, expected_result, rtol=rtol, atol=atol) + + +@torch.inference_mode() +@pytest.mark.parametrize("num_loras", [1, 2, 4]) +@pytest.mark.parametrize("orientation", ["row", "column"]) +@pytest.mark.parametrize("fully_shard", [True, False]) +@pytest.mark.parametrize("device", DEVICES) +@pytest.mark.parametrize("stage", STAGES) +def test_linear_parallel( + default_vllm_config, dist_init, num_loras, orientation, fully_shard, device, stage +) -> None: + if current_platform.is_cuda_alike() or current_platform.is_xpu(): + torch.accelerator.set_device_index(device) + + max_loras = 8 + torch.set_default_device(device) + lora_config = LoRAConfig( + max_loras=max_loras, + max_lora_rank=8, + fully_sharded_loras=fully_shard, + lora_dtype=torch.float16, + ) + punica_wrapper = get_punica_wrapper(8192, 256, device, lora_config=lora_config) + assert check_punica_wrapper(punica_wrapper) + + def create_random_linear_parallel_layer(idx: int = 0): + if orientation == "row": + linear = RowParallelLinear( + 4096, + 4096, + bias=False, + params_dtype=torch.float16, + prefix=f"layer_{idx}", + ) + linear.weight.data = torch.rand_like(linear.weight.data) + lora_linear = ( + RowParallelLinearWithLoRA(linear) + if not fully_shard + else RowParallelLinearWithShardedLoRA(linear) + ) + else: + linear = ColumnParallelLinear( + 4096, + 4096, + bias=False, + params_dtype=torch.float16, + prefix=f"layer_{idx}", + ) + linear.weight.data = torch.rand_like(linear.weight.data) + lora_linear = ( + ColumnParallelLinearWithLoRA(linear) + if not fully_shard + else ColumnParallelLinearWithShardedLoRA(linear) + ) + lora_linear.create_lora_weights(max_loras, lora_config) + assert ( + lora_linear.n_slices + == len(lora_linear.lora_a_stacked) + == len(lora_linear.lora_b_stacked) + == 1 + ) + + return linear, lora_linear + + for i in range(NUM_RANDOM_SEEDS): + set_random_seed(i) + + id_to_index = get_random_id_to_index(num_loras, max_loras) + linear, lora_linear = create_random_linear_parallel_layer(i) + assert torch.equal(linear.weight, lora_linear.weight) + lora_linear.set_mapping(punica_wrapper) + lora_dict, _ = populate_loras( + id_to_index, + layer=lora_linear, + layer_weights=linear.weight, + ) + + inputs, index_mapping, prompt_mapping = create_random_inputs( + active_lora_ids=list(lora_dict.keys()), + num_inputs=32 * num_loras, + input_size=(1, 4096), + input_range=(0, 1), + input_type=torch.float16, + device=device, + ) + lora_mapping = LoRAMapping(index_mapping, prompt_mapping, is_prefill=stage) + punica_wrapper.update_metadata( + lora_mapping, + id_to_index, + max_loras, + 512, + ) + + lora_result = lora_linear(torch.cat(inputs))[0] + + expected_results: list[torch.Tensor] = [] + for input_, lora_id in zip(inputs, prompt_mapping): + lora = lora_dict[lora_id] + result = linear(input_)[0] + result += input_ @ lora.lora_a.T @ lora.lora_b.T * lora.scaling + expected_results.append(result) + expected_result = torch.cat(expected_results) + + rtol, atol = TOLERANCES[lora_result.dtype] + torch.testing.assert_close(lora_result, expected_result, rtol=rtol, atol=atol) + + # Check that resetting the lora weights succeeds + + for slot_idx in range(max_loras): + lora_linear.reset_lora(slot_idx) + + inputs, index_mapping, prompt_mapping = create_random_inputs( + active_lora_ids=[0], + num_inputs=32 * num_loras, + input_size=(1, 4096), + input_range=(0, 1), + input_type=torch.float16, + device=device, + ) + lora_mapping = LoRAMapping(index_mapping, prompt_mapping, is_prefill=stage) + + punica_wrapper.update_metadata( + lora_mapping, + id_to_index, + max_loras, + 512, + ) + + lora_result = lora_linear(torch.cat(inputs))[0] + expected_result = linear(torch.cat(inputs))[0] + + rtol, atol = TOLERANCES[lora_result.dtype] + torch.testing.assert_close(lora_result, expected_result, rtol=rtol, atol=atol) + + +@torch.inference_mode() +@pytest.mark.parametrize("num_loras", [1, 2, 4]) +@pytest.mark.parametrize("repeats", [1, 2, 3]) +@pytest.mark.parametrize("fully_shard", [True, False]) +@pytest.mark.parametrize("device", DEVICES) +@pytest.mark.parametrize("stage", STAGES) +def test_column_parallel_packed( + default_vllm_config, dist_init, num_loras, repeats, fully_shard, device, stage +) -> None: + if current_platform.is_cuda_alike() or current_platform.is_xpu(): + torch.accelerator.set_device_index(device) + + max_loras = 8 + torch.set_default_device(device) + lora_config = LoRAConfig( + max_loras=max_loras, + max_lora_rank=8, + fully_sharded_loras=fully_shard, + lora_dtype=torch.float16, + ) + punica_wrapper = get_punica_wrapper(8192, 256, device, lora_config=lora_config) + assert check_punica_wrapper(punica_wrapper) + + def create_column_parallel_packed_layer(idx: int = 0): + if repeats == 2: + linear = MergedColumnParallelLinear( + 4096, + [4096] * repeats, + bias=False, + params_dtype=torch.float16, + prefix=f"layer_{idx}", + ) + linear.weight.data = torch.rand_like(linear.weight.data) + lora_linear = ( + MergedColumnParallelLinearWithLoRA(linear) + if not fully_shard + else MergedColumnParallelLinearWithShardedLoRA(linear) + ) + elif repeats == 3: + linear = QKVParallelLinear( + 4096, + 64, + 32, + bias=False, + params_dtype=torch.float16, + prefix=f"layer_{idx}", + ) + linear.weight.data = torch.rand_like(linear.weight.data) + lora_linear = ( + MergedQKVParallelLinearWithLoRA(linear) + if not fully_shard + else MergedQKVParallelLinearWithShardedLoRA(linear) + ) + else: + linear = QKVParallelLinear( + 4096, + 64, + 32, + bias=False, + params_dtype=torch.float16, + prefix=f"layer_{idx}", + ) + linear.weight.data = torch.rand_like(linear.weight.data) + lora_linear = ( + QKVParallelLinearWithLoRA(linear) + if not fully_shard + else QKVParallelLinearWithShardedLoRA(linear) + ) + + @dataclass + class FakeConfig: + hidden_size = 4096 + num_key_value_heads = 32 + num_attention_heads = 32 + + n_slices = repeats + lora_linear.create_lora_weights( + max_loras, lora_config, model_config=FakeConfig() + ) + assert ( + lora_linear.n_slices + == len(lora_linear.lora_a_stacked) + == len(lora_linear.lora_b_stacked) + == n_slices + ) + + return linear, lora_linear + + for i in range(NUM_RANDOM_SEEDS): + set_random_seed(i) + + id_to_index = get_random_id_to_index(num_loras, max_loras) + + linear, lora_linear = create_column_parallel_packed_layer(i) + assert torch.equal(linear.weight, lora_linear.weight) + lora_linear.set_mapping(punica_wrapper) + lora_dict, sublora_dict = populate_loras( + id_to_index, + layer=lora_linear, + layer_weights=linear.weight, + repeats=repeats, + ) + + inputs, index_mapping, prompt_mapping = create_random_inputs( + active_lora_ids=list(lora_dict.keys()), + num_inputs=32 * num_loras, + input_size=(1, 4096), + input_range=(0, 1), + input_type=torch.float16, + device=device, + ) + lora_mapping = LoRAMapping(index_mapping, prompt_mapping, is_prefill=stage) + + punica_wrapper.update_metadata( + lora_mapping, + id_to_index, + max_loras, + 512, + ) + + lora_result = lora_linear(torch.cat(inputs))[0] + + expected_results: list[torch.Tensor] = [] + for input_, lora_id in zip(inputs, prompt_mapping): + result = linear(input_)[0] + subloras = sublora_dict[lora_id] + for i, sublora in enumerate(subloras): + result[ + :, sublora.lora_b.shape[0] * i : sublora.lora_b.shape[0] * (i + 1) + ] += input_ @ sublora.lora_a.T @ sublora.lora_b.T * sublora.scaling + expected_results.append(result) + expected_result = torch.cat(expected_results) + + rtol, atol = TOLERANCES[lora_result.dtype] + torch.testing.assert_close(lora_result, expected_result, rtol=rtol, atol=atol) + + for slot_idx in range(max_loras): + lora_linear.reset_lora(slot_idx) + + inputs, index_mapping, prompt_mapping = create_random_inputs( + active_lora_ids=[0], + num_inputs=32 * num_loras, + input_size=(1, 4096), + input_range=(0, 1), + input_type=torch.float16, + device=device, + ) + lora_mapping = LoRAMapping(index_mapping, prompt_mapping, is_prefill=stage) + + punica_wrapper.update_metadata( + lora_mapping, + id_to_index, + max_loras, + 512, + ) + + lora_result = lora_linear(torch.cat(inputs))[0] + expected_result = linear(torch.cat(inputs))[0] + + rtol, atol = TOLERANCES[lora_result.dtype] + torch.testing.assert_close(lora_result, expected_result, rtol=rtol, atol=atol) + + +@torch.inference_mode() +@pytest.mark.parametrize("num_loras", [1, 2, 4]) +@pytest.mark.parametrize("num_slices", [3, 5]) +@pytest.mark.parametrize("device", DEVICES) +@pytest.mark.parametrize("stage", STAGES) +def test_merged_column_parallel_variable_slice( + default_vllm_config, dist_init, num_loras, num_slices, device, stage +) -> None: + if current_platform.is_cuda_alike() or current_platform.is_xpu(): + torch.accelerator.set_device_index(device) + + max_loras = 8 + torch.set_default_device(device) + lora_config = LoRAConfig( + max_loras=max_loras, max_lora_rank=8, lora_dtype=torch.float16 + ) + punica_wrapper = get_punica_wrapper(8192, 256, device, lora_config=lora_config) + + # Set number of output slices + output_sizes = [1024 + i * 256 for i in range(num_slices)] + total_output = sum(output_sizes) + + def create_layer(idx: int = 0): + # Create linear layer + linear = MergedColumnParallelLinear( + 4096, + output_sizes, + bias=False, + params_dtype=torch.float16, + prefix=f"layer_{idx}", + ) + linear.weight.data = torch.rand_like(linear.weight.data) + + # Create linear layer with LoRA adapter + lora_linear = MergedColumnParallelLinearVariableSliceWithLoRA(linear) + lora_linear.create_lora_weights(max_loras, lora_config) + return linear, lora_linear + + for i in range(NUM_RANDOM_SEEDS): + set_random_seed(i) + id_to_index = get_random_id_to_index(num_loras, max_loras) + linear, lora_linear = create_layer(i) + lora_linear.set_mapping(punica_wrapper) + + # Populate LoRA weights + lora_dict, sublora_dict = {}, {} + for slot_idx, lora_id in enumerate(id_to_index): + if lora_id is not None: + # Create random LoRA weights + lora_a = torch.rand(8, 4096, dtype=torch.float16, device=device) + lora_b = torch.rand(total_output, 8, dtype=torch.float16, device=device) + lora_linear.set_lora(slot_idx, lora_a, lora_b) + lora_dict[lora_id] = (lora_a, lora_b) + + # Split lora_b for expected computation + sublora_dict[lora_id] = torch.split(lora_b, output_sizes, dim=0) + + inputs, index_mapping, prompt_mapping = create_random_inputs( + active_lora_ids=list(lora_dict.keys()), + num_inputs=32 * num_loras, + input_size=(1, 4096), + input_range=(0, 1), + input_type=torch.float16, + device=device, + ) + lora_mapping = LoRAMapping(index_mapping, prompt_mapping, is_prefill=stage) + punica_wrapper.update_metadata(lora_mapping, id_to_index, max_loras, 512) + + # Compute LoRA result + lora_result = lora_linear(torch.cat(inputs))[0] + + # Compute expected result + expected_results = [] + for input_, lora_id in zip(inputs, prompt_mapping): + result = linear(input_)[0] + lora_a, _ = lora_dict[lora_id] + offset = 0 + # Compute expected result for each sublora + for lora_b_slice in sublora_dict[lora_id]: + sz = lora_b_slice.shape[0] + result[:, offset : offset + sz] += input_ @ lora_a.T @ lora_b_slice.T + offset += sz + expected_results.append(result) + + # Check that the LoRA result is close to the expected result + rtol, atol = TOLERANCES[lora_result.dtype] + torch.testing.assert_close( + lora_result, torch.cat(expected_results), rtol=rtol, atol=atol + ) + + # Reset LoRA weights and check results with zero LoRA weights + for slot_idx in range(max_loras): + lora_linear.reset_lora(slot_idx) + + inputs, index_mapping, prompt_mapping = create_random_inputs( + active_lora_ids=[0], + num_inputs=32 * num_loras, + input_size=(1, 4096), + input_range=(0, 1), + input_type=torch.float16, + device=device, + ) + lora_mapping = LoRAMapping(index_mapping, prompt_mapping, is_prefill=stage) + punica_wrapper.update_metadata(lora_mapping, id_to_index, max_loras, 512) + + # After resetting LoRA weights, + # lora_linear should behave like the base linear layer + lora_result = lora_linear(torch.cat(inputs))[0] + expected_result = linear(torch.cat(inputs))[0] + + rtol, atol = TOLERANCES[lora_result.dtype] + torch.testing.assert_close(lora_result, expected_result, rtol=rtol, atol=atol) + + +@pytest.mark.parametrize("tp_size", [1, 2, 4, 8]) +@pytest.mark.parametrize( + "seed", list(range(VOCAB_PARALLEL_EMBEDDING_TEST_NUM_RANDOM_SEEDS)) +) +def test_vocab_parallel_embedding_indices(tp_size, seed, default_vllm_config): + random.seed(seed) + vocab_size = random.randint(4000, 64000) + added_vocab_size = random.randint(0, 1024) + org_vocab_size = vocab_size - added_vocab_size + last_org_vocab_end_index = 0 + last_added_vocab_end_index = org_vocab_size + computed_vocab_size = 0 + computed_org_vocab_size = 0 + computed_added_vocab_size = 0 + vocab_size_padded = -1 + + all_org_tokens: list[int] = [] + all_added_tokens: list[int] = [] + token_ids: list[int] = [] + + for tp_rank in range(tp_size): + with ( + patch( + "vllm.model_executor.layers.vocab_parallel_embedding.get_tensor_model_parallel_rank", + return_value=tp_rank, + ), + patch( + "vllm.model_executor.layers.vocab_parallel_embedding.get_tensor_model_parallel_world_size", + return_value=tp_size, + ), + ): + vocab_embedding = VocabParallelEmbedding( + vocab_size, 1, org_num_embeddings=org_vocab_size + ) + vocab_size_padded = vocab_embedding.num_embeddings_padded + shard_indices = vocab_embedding.shard_indices + # Assert that the ranges are contiguous + assert shard_indices.org_vocab_start_index == last_org_vocab_end_index + assert shard_indices.added_vocab_start_index == last_added_vocab_end_index + + # Ensure that we are not exceeding the vocab size + computed_vocab_size += shard_indices.num_elements_padded + computed_org_vocab_size += shard_indices.num_org_elements + computed_added_vocab_size += shard_indices.num_added_elements + + # Ensure that the ranges are not overlapping + all_org_tokens.extend( + range( + shard_indices.org_vocab_start_index, shard_indices.org_vocab_end_index + ) + ) + all_added_tokens.extend( + range( + shard_indices.added_vocab_start_index, + shard_indices.added_vocab_end_index, + ) + ) + + token_ids.extend( + range( + shard_indices.org_vocab_start_index, shard_indices.org_vocab_end_index + ) + ) + token_ids.extend( + [-1] + * (shard_indices.num_org_elements_padded - shard_indices.num_org_elements) + ) + token_ids.extend( + range( + shard_indices.added_vocab_start_index, + shard_indices.added_vocab_end_index, + ) + ) + token_ids.extend( + [-1] + * ( + shard_indices.num_added_elements_padded + - shard_indices.num_added_elements + ) + ) + + last_org_vocab_end_index = shard_indices.org_vocab_end_index + last_added_vocab_end_index = shard_indices.added_vocab_end_index + + assert computed_vocab_size == vocab_size_padded + assert computed_org_vocab_size == org_vocab_size + assert computed_added_vocab_size == added_vocab_size + + # Ensure that the ranges are not overlapping + assert len(all_org_tokens) == len(set(all_org_tokens)) + assert len(all_added_tokens) == len(set(all_added_tokens)) + assert not set(all_org_tokens).intersection(set(all_added_tokens)) + + token_ids_tensor = torch.tensor(token_ids, dtype=torch.long) + reindex_mapping = vocab_embedding.get_sharded_to_full_mapping() + assert reindex_mapping is not None or tp_size == 1 + if reindex_mapping is not None: + reindexed_token_ids = token_ids_tensor[reindex_mapping] + expected = torch.tensor(list(range(0, vocab_size))) + assert reindexed_token_ids[:vocab_size].equal(expected) + assert torch.all(reindexed_token_ids[vocab_size:] == -1) + + +def test_get_masked_input_and_mask(): + x = torch.tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]) + + # base tp 1 case, no padding + modified_x, _ = get_masked_input_and_mask( + x, + org_vocab_start_index=0, + org_vocab_end_index=8, + added_vocab_start_index=8, + added_vocab_end_index=12, + num_org_vocab_padding=0, + ) + assert torch.equal(x, modified_x) + + # tp 2 case, no padding + modified_x_rank_0, _ = get_masked_input_and_mask( + x, + org_vocab_start_index=0, + org_vocab_end_index=4, + added_vocab_start_index=8, + added_vocab_end_index=10, + num_org_vocab_padding=0, + ) + modified_x_rank_1, _ = get_masked_input_and_mask( + x, + org_vocab_start_index=4, + org_vocab_end_index=8, + added_vocab_start_index=10, + added_vocab_end_index=12, + num_org_vocab_padding=0, + ) + assert torch.equal( + modified_x_rank_0, torch.tensor([0, 1, 2, 3, 0, 0, 0, 0, 4, 5, 0, 0]) + ) + assert torch.equal( + modified_x_rank_1, torch.tensor([0, 0, 0, 0, 0, 1, 2, 3, 0, 0, 4, 5]) + ) + + # tp 4 case, no padding + modified_x_rank_0, _ = get_masked_input_and_mask( + x, + org_vocab_start_index=0, + org_vocab_end_index=2, + added_vocab_start_index=8, + added_vocab_end_index=9, + num_org_vocab_padding=0, + ) + modified_x_rank_1, _ = get_masked_input_and_mask( + x, + org_vocab_start_index=2, + org_vocab_end_index=4, + added_vocab_start_index=9, + added_vocab_end_index=10, + num_org_vocab_padding=0, + ) + modified_x_rank_2, _ = get_masked_input_and_mask( + x, + org_vocab_start_index=4, + org_vocab_end_index=6, + added_vocab_start_index=10, + added_vocab_end_index=11, + num_org_vocab_padding=0, + ) + modified_x_rank_3, _ = get_masked_input_and_mask( + x, + org_vocab_start_index=6, + org_vocab_end_index=8, + added_vocab_start_index=11, + added_vocab_end_index=12, + num_org_vocab_padding=0, + ) + assert torch.equal( + modified_x_rank_0, torch.tensor([0, 1, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0]) + ) + assert torch.equal( + modified_x_rank_1, torch.tensor([0, 0, 0, 1, 0, 0, 0, 0, 0, 2, 0, 0]) + ) + assert torch.equal( + modified_x_rank_2, torch.tensor([0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 2, 0]) + ) + assert torch.equal( + modified_x_rank_3, torch.tensor([0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2]) + ) + + # base tp 1 case, with padding + modified_x, _ = get_masked_input_and_mask( + x, + org_vocab_start_index=0, + org_vocab_end_index=8, + added_vocab_start_index=8, + added_vocab_end_index=12, + num_org_vocab_padding=2, + ) + assert torch.equal( + modified_x, torch.tensor([0, 1, 2, 3, 4, 5, 6, 7, 10, 11, 12, 13]) + ) + + # tp 2 case, with padding + modified_x_rank_0, _ = get_masked_input_and_mask( + x, + org_vocab_start_index=0, + org_vocab_end_index=4, + added_vocab_start_index=8, + added_vocab_end_index=10, + num_org_vocab_padding=2, + ) + modified_x_rank_1, _ = get_masked_input_and_mask( + x, + org_vocab_start_index=4, + org_vocab_end_index=8, + added_vocab_start_index=10, + added_vocab_end_index=12, + num_org_vocab_padding=2, + ) + assert torch.equal( + modified_x_rank_0, torch.tensor([0, 1, 2, 3, 0, 0, 0, 0, 6, 7, 0, 0]) + ) + assert torch.equal( + modified_x_rank_1, torch.tensor([0, 0, 0, 0, 0, 1, 2, 3, 0, 0, 6, 7]) + ) + + # tp 4 case, with padding + modified_x_rank_0, _ = get_masked_input_and_mask( + x, + org_vocab_start_index=0, + org_vocab_end_index=2, + added_vocab_start_index=8, + added_vocab_end_index=9, + num_org_vocab_padding=2, + ) + modified_x_rank_1, _ = get_masked_input_and_mask( + x, + org_vocab_start_index=2, + org_vocab_end_index=4, + added_vocab_start_index=9, + added_vocab_end_index=10, + num_org_vocab_padding=2, + ) + modified_x_rank_2, _ = get_masked_input_and_mask( + x, + org_vocab_start_index=4, + org_vocab_end_index=6, + added_vocab_start_index=10, + added_vocab_end_index=11, + num_org_vocab_padding=2, + ) + modified_x_rank_3, _ = get_masked_input_and_mask( + x, + org_vocab_start_index=6, + org_vocab_end_index=8, + added_vocab_start_index=11, + added_vocab_end_index=12, + num_org_vocab_padding=2, + ) + assert torch.equal( + modified_x_rank_0, torch.tensor([0, 1, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0]) + ) + assert torch.equal( + modified_x_rank_1, torch.tensor([0, 0, 0, 1, 0, 0, 0, 0, 0, 4, 0, 0]) + ) + assert torch.equal( + modified_x_rank_2, torch.tensor([0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 4, 0]) + ) + assert torch.equal( + modified_x_rank_3, torch.tensor([0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 4]) + ) + + +def test_variable_slice_lora_class_selection(default_vllm_config, dist_init): + """Test that MergedColumnParallelLinearVariableSliceWithLoRA is selected + only for nemotron-h style models (checkpoint has single weight but layer + has 3+ output slices). + + This verifies that from_layer selects + MergedColumnParallelLinearVariableSliceWithLoRA + before ColumnParallelLinearWithLoRA for layers with 3+ output sizes, since + ColumnParallelLinearWithLoRA's slice_lora_b assumes exactly 2 slices. + """ + from vllm.lora.utils import from_layer + + lora_config = LoRAConfig(max_loras=8, max_lora_rank=8, lora_dtype=torch.float16) + + # Case 1: MergedColumnParallelLinear with 3+ output sizes and + # packed_modules_list with 1 item (nemotron-h style) + # -> MergedColumnParallelLinearVariableSliceWithLoRA should be selected + layer_3_slices = MergedColumnParallelLinear( + 4096, [1024, 1280, 1536], bias=False, params_dtype=torch.float16 + ) + packed_modules_single = ["mlp"] + + assert MergedColumnParallelLinearVariableSliceWithLoRA.can_replace_layer( + source_layer=layer_3_slices, + lora_config=lora_config, + packed_modules_list=packed_modules_single, + ), "MergedColumnParallelLinearVariableSliceWithLoRA should handle 3+ slices" + + # ColumnParallelLinearWithLoRA should NOT match 3+ slices + # (its slice_lora_b assumes exactly 2 slices) + assert not ColumnParallelLinearWithLoRA.can_replace_layer( + source_layer=layer_3_slices, + lora_config=lora_config, + packed_modules_list=packed_modules_single, + ), ( + "ColumnParallelLinearWithLoRA should NOT handle 3+ slices " + "(slice_lora_b assumes 2 slices)" + ) + + # Verify from_layer selects the correct class (Variable, not base) + selected_layer = from_layer( + layer_3_slices, + max_loras=8, + lora_config=lora_config, + packed_modules_list=packed_modules_single, + ) + assert isinstance( + selected_layer, MergedColumnParallelLinearVariableSliceWithLoRA + ), ( + f"from_layer should select MergedColumnParallelLinearVariableSliceWithLoRA " + f"for 3+ slices, got {type(selected_layer).__name__}" + ) + + # Case 2: MergedColumnParallelLinear with 2 output sizes and + # packed_modules_list with 1 item (standard gate_up style) + # -> ColumnParallelLinearWithLoRA should be selected + # -> MergedColumnParallelLinearVariableSliceWithLoRA should NOT match + layer_2_slices = MergedColumnParallelLinear( + 4096, [2048, 2048], bias=False, params_dtype=torch.float16 + ) + + assert ColumnParallelLinearWithLoRA.can_replace_layer( + source_layer=layer_2_slices, + lora_config=lora_config, + packed_modules_list=packed_modules_single, + ), "ColumnParallelLinearWithLoRA should handle 2 slices" + + assert not MergedColumnParallelLinearVariableSliceWithLoRA.can_replace_layer( + source_layer=layer_2_slices, + lora_config=lora_config, + packed_modules_list=packed_modules_single, + ), "MergedColumnParallelLinearVariableSliceWithLoRA should NOT handle 2 slices" + + # Verify from_layer selects ColumnParallelLinearWithLoRA for 2 slices + selected_layer_2 = from_layer( + layer_2_slices, + max_loras=8, + lora_config=lora_config, + packed_modules_list=packed_modules_single, + ) + assert isinstance(selected_layer_2, ColumnParallelLinearWithLoRA), ( + f"from_layer should select ColumnParallelLinearWithLoRA " + f"for 2 slices, got {type(selected_layer_2).__name__}" + ) + # But NOT the Variable subclass + assert not isinstance( + selected_layer_2, MergedColumnParallelLinearVariableSliceWithLoRA + ), ( + "from_layer should NOT select " + "MergedColumnParallelLinearVariableSliceWithLoRA for 2 slices" + ) + + # Case 3: MergedColumnParallelLinear with 3+ items in packed_modules_list + # -> MergedColumnParallelLinearVariableSliceWithLoRA should be selected + packed_modules_three = ["gate_proj", "up_proj", "down_proj"] + + assert MergedColumnParallelLinearVariableSliceWithLoRA.can_replace_layer( + source_layer=layer_3_slices, + lora_config=lora_config, + packed_modules_list=packed_modules_three, + ), "MergedColumnParallelLinearVariableSliceWithLoRA should handle 3+ packed modules" + + # Case 4: MergedColumnParallelLinear with 2 items in packed_modules_list + # -> MergedColumnParallelLinearWithLoRA should handle this (not Variable) + packed_modules_two = ["gate_proj", "up_proj"] + + assert not MergedColumnParallelLinearVariableSliceWithLoRA.can_replace_layer( + source_layer=layer_2_slices, + lora_config=lora_config, + packed_modules_list=packed_modules_two, + ), ( + "MergedColumnParallelLinearVariableSliceWithLoRA" + " should NOT handle 2 packed modules" + ) + + assert MergedColumnParallelLinearWithLoRA.can_replace_layer( + source_layer=layer_2_slices, + lora_config=lora_config, + packed_modules_list=packed_modules_two, + ), "MergedColumnParallelLinearWithLoRA should handle 2 packed modules" + + # Verify from_layer selects MergedColumnParallelLinearWithLoRA for 2 packed modules + selected_layer_merged = from_layer( + layer_2_slices, + max_loras=8, + lora_config=lora_config, + packed_modules_list=packed_modules_two, + ) + assert isinstance(selected_layer_merged, MergedColumnParallelLinearWithLoRA), ( + f"from_layer should select MergedColumnParallelLinearWithLoRA " + f"for 2 packed modules, got {type(selected_layer_merged).__name__}" + ) + + fully_sharded_tp_lora_config = LoRAConfig( + max_loras=8, + max_lora_rank=16, + lora_dtype=torch.float16, + fully_sharded_loras=True, + ) + fully_sharded_tp_layer = MergedColumnParallelLinear( + 4096, [2048, 2048], bias=False, params_dtype=torch.float16 + ) + fully_sharded_tp_layer.tp_size = 2 + + assert not MergedColumnParallelLinearWithLoRA.can_replace_layer( + source_layer=fully_sharded_tp_layer, + lora_config=fully_sharded_tp_lora_config, + packed_modules_list=packed_modules_two, + ), "Generic merged wrapper should reject fully sharded TP layers" + + assert MergedColumnParallelLinearWithShardedLoRA.can_replace_layer( + source_layer=fully_sharded_tp_layer, + lora_config=fully_sharded_tp_lora_config, + packed_modules_list=packed_modules_two, + ), "Sharded merged wrapper should remain eligible for fully sharded TP layers" + + selected_fully_sharded_tp_layer = from_layer( + fully_sharded_tp_layer, + max_loras=8, + lora_config=fully_sharded_tp_lora_config, + packed_modules_list=packed_modules_two, + ) + assert isinstance( + selected_fully_sharded_tp_layer, + MergedColumnParallelLinearWithShardedLoRA, + ), ( + "from_layer should select MergedColumnParallelLinearWithShardedLoRA " + "for fully sharded TP merged layers, got " + f"{type(selected_fully_sharded_tp_layer).__name__}" + ) + + # Case 5: DeepSeek's fused_qkv_a_proj should reuse the generic merged + # wrapper while preserving its custom base forward path. + deepseek_fused_layer = DeepSeekV2FusedQkvAProjLinear( + 4096, [2048, 2048], prefix="model.layers.0.self_attn.fused_qkv_a_proj" + ) + selected_deepseek_layer = from_layer( + deepseek_fused_layer, + max_loras=8, + lora_config=lora_config, + packed_modules_list=packed_modules_two, + ) + assert isinstance(selected_deepseek_layer, MergedColumnParallelLinearWithLoRA), ( + "from_layer should select MergedColumnParallelLinearWithLoRA " + f"for DeepSeek fused_qkv_a_proj, got {type(selected_deepseek_layer).__name__}" + ) + + fully_sharded_lora_config = LoRAConfig( + max_loras=8, + max_lora_rank=16, + lora_dtype=torch.float16, + fully_sharded_loras=True, + ) + selected_fully_sharded_deepseek_layer = from_layer( + deepseek_fused_layer, + max_loras=8, + lora_config=fully_sharded_lora_config, + packed_modules_list=packed_modules_two, + ) + assert isinstance( + selected_fully_sharded_deepseek_layer, + MergedColumnParallelLinearWithLoRA, + ), ( + "from_layer should keep using MergedColumnParallelLinearWithLoRA " + "for fused_qkv_a_proj when the base layer is effectively unsharded, got " + f"{type(selected_fully_sharded_deepseek_layer).__name__}" + ) + + # Case 6: Generic subclass of MergedColumnParallelLinear with 2 packed + # modules should still use the generic merged wrapper. + class CustomMergedColumnParallelLinear(MergedColumnParallelLinear): + pass + + custom_merged_layer = CustomMergedColumnParallelLinear( + 4096, [2048, 2048], bias=False, params_dtype=torch.float16 + ) + assert MergedColumnParallelLinearWithLoRA.can_replace_layer( + source_layer=custom_merged_layer, + lora_config=lora_config, + packed_modules_list=packed_modules_two, + ), "MergedColumnParallelLinearWithLoRA should handle subclasses" + + selected_custom_layer = from_layer( + custom_merged_layer, + max_loras=8, + lora_config=lora_config, + packed_modules_list=packed_modules_two, + ) + assert isinstance(selected_custom_layer, MergedColumnParallelLinearWithLoRA), ( + f"from_layer should select MergedColumnParallelLinearWithLoRA " + f"for subclassed merged layers, got {type(selected_custom_layer).__name__}" + ) + + # Case 7: Plain ColumnParallelLinear (not merged) - common in many models + # -> ColumnParallelLinearWithLoRA should be selected + plain_column_parallel = ColumnParallelLinear( + 4096, 4096, bias=False, params_dtype=torch.float16 + ) + + assert ColumnParallelLinearWithLoRA.can_replace_layer( + source_layer=plain_column_parallel, + lora_config=lora_config, + packed_modules_list=packed_modules_single, + ), "ColumnParallelLinearWithLoRA should handle plain ColumnParallelLinear" + + assert not MergedColumnParallelLinearVariableSliceWithLoRA.can_replace_layer( + source_layer=plain_column_parallel, + lora_config=lora_config, + packed_modules_list=packed_modules_single, + ), ( + "MergedColumnParallelLinearVariableSliceWithLoRA " + "should NOT handle plain ColumnParallelLinear" + ) + + # Verify from_layer selects ColumnParallelLinearWithLoRA for plain layer + selected_plain = from_layer( + plain_column_parallel, + max_loras=8, + lora_config=lora_config, + packed_modules_list=packed_modules_single, + ) + assert isinstance(selected_plain, ColumnParallelLinearWithLoRA), ( + f"from_layer should select ColumnParallelLinearWithLoRA " + f"for plain ColumnParallelLinear, got {type(selected_plain).__name__}" + ) + + # Case 8: MergedColumnParallelLinear with exactly 2 output sizes + # and empty packed_modules_list + # -> ColumnParallelLinearWithLoRA should NOT match (packed_modules_list != 1) + # -> MergedColumnParallelLinearVariableSliceWithLoRA should NOT match (< 3 slices) + assert not ColumnParallelLinearWithLoRA.can_replace_layer( + source_layer=layer_2_slices, + lora_config=lora_config, + packed_modules_list=[], + ), "ColumnParallelLinearWithLoRA should NOT handle empty packed_modules_list" + + assert not MergedColumnParallelLinearVariableSliceWithLoRA.can_replace_layer( + source_layer=layer_2_slices, + lora_config=lora_config, + packed_modules_list=[], + ), ( + "MergedColumnParallelLinearVariableSliceWithLoRA " + "should NOT handle 2 slices even with empty packed_modules_list" + ) + + +@pytest.mark.parametrize( + "wrapper_cls", + [ColumnParallelLinearWithLoRA, ColumnParallelLinearWithShardedLoRA], +) +def test_get_and_maybe_dequant_weights_accepts_lora_wrappers(dist_init, wrapper_cls): + from vllm.model_executor.layers.quantization.utils.quant_utils import ( + get_and_maybe_dequant_weights, + ) + + linear = ColumnParallelLinear(4096, 4096, bias=False, params_dtype=torch.float16) + lora_linear = wrapper_cls(linear) + + # Should work with LoRA wrappers and return [out, in] weights. + dequant_weight = get_and_maybe_dequant_weights(lora_linear, out_dtype=torch.float16) + assert dequant_weight.shape == linear.weight.shape + + +@torch.inference_mode() +@pytest.mark.parametrize("device", DEVICES) +@pytest.mark.parametrize("stage", STAGES) +@pytest.mark.parametrize("fully_sharded", [False, True]) +def test_deepseek_fused_qkv_a_proj_lora_preserves_base_forward( + default_vllm_config, dist_init, device, stage, fully_sharded +): + if current_platform.is_cuda_alike() or current_platform.is_xpu(): + torch.accelerator.set_device_index(device) + + torch.set_default_device(device) + dtype = ( + torch.float16 + if (current_platform.is_cuda_alike() or current_platform.is_xpu()) + else torch.float32 + ) + max_loras = 8 + lora_config = LoRAConfig( + max_loras=max_loras, + max_lora_rank=8, + lora_dtype=dtype, + fully_sharded_loras=fully_sharded, + ) + punica_wrapper = get_punica_wrapper(8192, 256, device, lora_config=lora_config) + assert check_punica_wrapper(punica_wrapper) + + class OffsetDeepSeekFusedQkvAProjLinear(DeepSeekV2FusedQkvAProjLinear): + def forward(self, input_): + output, output_bias = super().forward(input_) + return output + 1, output_bias + + layer = OffsetDeepSeekFusedQkvAProjLinear( + 32, [16, 16], prefix="model.layers.0.self_attn.fused_qkv_a_proj" + ) + layer.weight.data = torch.rand_like(layer.weight.data, dtype=dtype) + + lora_layer = MergedColumnParallelLinearWithLoRA(layer) + lora_layer.create_lora_weights(max_loras, lora_config) + lora_layer.set_mapping(punica_wrapper) + + id_to_index = get_random_id_to_index(1, max_loras, log=False) + active_slot = next(i for i, lora_id in enumerate(id_to_index) if lora_id == 1) + lora_a = [ + torch.rand(8, 32, dtype=dtype, device=device), + torch.rand(8, 32, dtype=dtype, device=device), + ] + lora_b = [ + torch.rand(16, 8, dtype=dtype, device=device), + torch.rand(16, 8, dtype=dtype, device=device), + ] + lora_layer.set_lora(active_slot, lora_a=lora_a, lora_b=lora_b) + + inputs, index_mapping, prompt_mapping = create_random_inputs( + active_lora_ids=[1], + num_inputs=4, + input_size=(1, 32), + input_range=(0, 1), + input_type=dtype, + device=device, + ) + lora_mapping = LoRAMapping(index_mapping, prompt_mapping, is_prefill=stage) + punica_wrapper.update_metadata(lora_mapping, id_to_index, max_loras, 512) + + lora_result = lora_layer(torch.cat(inputs))[0] + + expected_results = [] + for input_ in inputs: + result = layer(input_)[0] + result[:, :16] += input_ @ lora_a[0].T @ lora_b[0].T + result[:, 16:] += input_ @ lora_a[1].T @ lora_b[1].T + expected_results.append(result) + + rtol, atol = TOLERANCES[lora_result.dtype] + torch.testing.assert_close( + lora_result, torch.cat(expected_results), rtol=rtol, atol=atol + ) + + merged_layer = OffsetDeepSeekFusedQkvAProjLinear( + 32, [16, 16], prefix="model.layers.0.self_attn.fused_qkv_a_proj" + ) + merged_layer.weight.data = layer.weight.data.clone() + merged_layer.weight.data[:16].add_(lora_b[0] @ lora_a[0]) + merged_layer.weight.data[16:].add_(lora_b[1] @ lora_a[1]) + merged_result = merged_layer(torch.cat(inputs))[0] + + torch.testing.assert_close(lora_result, merged_result, rtol=rtol, atol=atol) + + +@torch.inference_mode() +@pytest.mark.parametrize("device", DEVICES) +@pytest.mark.parametrize("stage", STAGES) +def test_replicated_lora_preserves_base_forward_for_subclasses( + default_vllm_config, dist_init, device, stage +): + if current_platform.is_cuda_alike() or current_platform.is_xpu(): + torch.accelerator.set_device_index(device) + + torch.set_default_device(device) + dtype = ( + torch.float16 + if current_platform.is_cuda_alike() or current_platform.is_xpu() + else torch.float32 + ) + max_loras = 8 + lora_config = LoRAConfig(max_loras=max_loras, max_lora_rank=8, lora_dtype=dtype) + punica_wrapper = get_punica_wrapper(8192, 256, device, lora_config=lora_config) + assert check_punica_wrapper(punica_wrapper) + + class OffsetReplicatedLinear(ReplicatedLinear): + def forward(self, input_): + output, output_bias = super().forward(input_) + return output + 1, output_bias + + layer = OffsetReplicatedLinear(32, 16, bias=False, params_dtype=dtype) + layer.weight.data = torch.rand_like(layer.weight.data, dtype=dtype) + + lora_layer = ReplicatedLinearWithLoRA(layer) + lora_layer.create_lora_weights(max_loras, lora_config) + lora_layer.set_mapping(punica_wrapper) + + id_to_index = get_random_id_to_index(1, max_loras, log=False) + active_slot = next(i for i, lora_id in enumerate(id_to_index) if lora_id == 1) + lora_a = torch.rand(8, 32, dtype=dtype, device=device) + lora_b = torch.rand(16, 8, dtype=dtype, device=device) + lora_layer.set_lora(active_slot, lora_a=lora_a, lora_b=lora_b) + + inputs, index_mapping, prompt_mapping = create_random_inputs( + active_lora_ids=[1], + num_inputs=4, + input_size=(1, 32), + input_range=(0, 1), + input_type=dtype, + device=device, + ) + lora_mapping = LoRAMapping(index_mapping, prompt_mapping, is_prefill=stage) + punica_wrapper.update_metadata(lora_mapping, id_to_index, max_loras, 512) + + lora_result = lora_layer(torch.cat(inputs))[0] + + expected_results = [] + for input_ in inputs: + result = layer(input_)[0] + result += input_ @ lora_a.T @ lora_b.T + expected_results.append(result) + + rtol, atol = TOLERANCES[lora_result.dtype] + torch.testing.assert_close( + lora_result, torch.cat(expected_results), rtol=rtol, atol=atol + ) + + merged_layer = OffsetReplicatedLinear(32, 16, bias=False, params_dtype=dtype) + merged_layer.weight.data = layer.weight.data.clone() + merged_layer.weight.data.add_(lora_b @ lora_a) + merged_result = merged_layer(torch.cat(inputs))[0] + + torch.testing.assert_close(lora_result, merged_result, rtol=rtol, atol=atol) diff --git a/asian_ai/performance/vLLM/tests/lora/test_llama_tp.py b/asian_ai/performance/vLLM/tests/lora/test_llama_tp.py new file mode 100644 index 0000000000000000000000000000000000000000..07c2c3bfe1faf56bbe6e55310de6519ba9682766 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_llama_tp.py @@ -0,0 +1,246 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import subprocess +import sys + +import pytest + +import vllm +import vllm.config +from vllm import LLM +from vllm.lora.request import LoRARequest +from vllm.model_executor.model_loader.tensorizer import TensorizerConfig +from vllm.platforms import current_platform + +from ..utils import VLLM_PATH, create_new_process_for_each_test, multi_gpu_test + +PROMPT_TEMPLATE = """<|eot_id|><|start_header_id|>user<|end_header_id|> +I want you to act as a SQL terminal in front of an example database, you need only to return the sql command to me.Below is an instruction that describes a task, Write a response that appropriately completes the request. +" +##Instruction: +candidate_poll contains tables such as candidate, people. Table candidate has columns such as Candidate_ID, People_ID, Poll_Source, Date, Support_rate, Consider_rate, Oppose_rate, Unsure_rate. Candidate_ID is the primary key. +Table people has columns such as People_ID, Sex, Name, Date_of_Birth, Height, Weight. People_ID is the primary key. +The People_ID of candidate is the foreign key of People_ID of people. +###Input: +{context} +###Response:<|eot_id|><|start_header_id|>assistant<|end_header_id|> +""" # noqa: E501 + +EXPECTED_LORA_OUTPUT = [ + "SELECT count(*) FROM candidate", + "SELECT count(*) FROM candidate", + "SELECT poll_source FROM candidate GROUP BY poll_source ORDER BY count(*) DESC LIMIT 1", # noqa: E501 + "SELECT poll_source FROM candidate GROUP BY poll_source ORDER BY count(*) DESC LIMIT 1", # noqa: E501 +] + +MODEL_PATH = "meta-llama/Llama-3.2-3B-Instruct" + + +def do_sample( + llm: vllm.LLM, + lora_path: str, + lora_id: int, + tensorizer_config_dict: dict | None = None, +) -> list[str]: + prompts = [ + PROMPT_TEMPLATE.format(context="How many candidates are there?"), + PROMPT_TEMPLATE.format(context="Count the number of candidates."), + PROMPT_TEMPLATE.format( + context="Which poll resource provided the most number of candidate information?" # noqa: E501 + ), + PROMPT_TEMPLATE.format( + context="Return the poll resource associated with the most candidates." + ), + ] + + sampling_params = vllm.SamplingParams( + temperature=0, max_tokens=64, stop=["<|im_end|>"] + ) + if tensorizer_config_dict is not None: + outputs = llm.generate( + prompts, + sampling_params, + lora_request=LoRARequest( + str(lora_id), + lora_id, + lora_path, + tensorizer_config_dict=tensorizer_config_dict, + ) + if lora_id + else None, + ) + else: + outputs = llm.generate( + prompts, + sampling_params, + lora_request=LoRARequest(str(lora_id), lora_id, lora_path) + if lora_id + else None, + ) + lora_request = LoRARequest(str(lora_id), lora_id, lora_path) if lora_id else None + generated_texts: list[str] = [] + for output in outputs: + prompt = output.prompt + generated_text = output.outputs[0].text + # The output should include correct lora_request info + if lora_request is not None: + assert output.lora_request.lora_name == lora_request.lora_name + assert output.lora_request.lora_int_id == lora_request.lora_int_id + assert output.lora_request.lora_path == lora_request.lora_path + else: + assert output.lora_request is None + generated_texts.append(generated_text) + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") + return generated_texts + + +def generate_and_test( + llm, llama32_lora_files, tensorizer_config_dict: dict | None = None +): + print("lora adapter created") + print("lora 1") + assert ( + do_sample( + llm, + llama32_lora_files, + tensorizer_config_dict=tensorizer_config_dict, + lora_id=1, + ) + == EXPECTED_LORA_OUTPUT + ) + + print("lora 2") + assert ( + do_sample( + llm, + llama32_lora_files, + tensorizer_config_dict=tensorizer_config_dict, + lora_id=2, + ) + == EXPECTED_LORA_OUTPUT + ) + + print("removing lora") + + +@create_new_process_for_each_test() +@pytest.mark.parametrize("cudagraph_specialize_lora", [True, False]) +def test_llama_lora(llama32_lora_files, cudagraph_specialize_lora: bool): + llm = vllm.LLM( + MODEL_PATH, + enable_lora=True, + # also test odd max_num_seqs + max_num_seqs=7, + max_model_len=1024, + max_loras=4, + compilation_config=vllm.config.CompilationConfig( + cudagraph_specialize_lora=cudagraph_specialize_lora, + ), + ) + generate_and_test(llm, llama32_lora_files) + + +@pytest.mark.skipif( + current_platform.is_cuda_alike(), reason="Skipping to avoid redundant model tests" +) +@multi_gpu_test(num_gpus=4) +def test_llama_lora_tp4(llama32_lora_files): + llm = vllm.LLM( + MODEL_PATH, + enable_lora=True, + max_num_seqs=7, + max_model_len=1024, + max_loras=4, + tensor_parallel_size=4, + ) + generate_and_test(llm, llama32_lora_files) + + +@multi_gpu_test(num_gpus=4) +def test_llama_lora_tp4_fully_sharded_loras(llama32_lora_files): + llm = vllm.LLM( + MODEL_PATH, + enable_lora=True, + max_num_seqs=8, + max_loras=4, + max_model_len=1024, + tensor_parallel_size=4, + fully_sharded_loras=True, + ) + generate_and_test(llm, llama32_lora_files) + + +@multi_gpu_test(num_gpus=2) +def test_tp2_serialize_and_deserialize_lora( + tmp_path, + llama32_lora_files, +): + # Run the tensorizing of the LoRA adapter and the model in a subprocess + # to guarantee cleanup + + tp_size = 2 + model_name = "model-rank-%03d.tensors" + + model_ref = MODEL_PATH + lora_path = llama32_lora_files + suffix = "test" + try: + result = subprocess.run( + [ + sys.executable, + f"{VLLM_PATH}/examples/features/tensorize_vllm_model.py", + "--model", + MODEL_PATH, + "--lora-path", + lora_path, + "--tensor-parallel-size", + str(tp_size), + "serialize", + "--serialized-directory", + str(tmp_path), + "--suffix", + suffix, + "--serialization-kwargs", + '{"limit_cpu_concurrency": 4}', + ], + check=True, + capture_output=True, + text=True, + ) + except subprocess.CalledProcessError as e: + print("Tensorizing failed.") + print("STDOUT:\n", e.stdout) + print("STDERR:\n", e.stderr) + raise + + print("STDOUT:\n", result.stdout) + + model_uri = tmp_path / "vllm" / model_ref / suffix / model_name + tensorizer_config = TensorizerConfig(tensorizer_uri=str(model_uri)) + + loaded_llm = LLM( + model=model_ref, + load_format="tensorizer", + enable_lora=True, + enforce_eager=True, + model_loader_extra_config=tensorizer_config, + max_num_seqs=7, + max_model_len=1024, + tensor_parallel_size=2, + max_loras=2, + # Leave headroom for LoRA adapter loading and Triton JIT + # compilation, which can allocate GPU memory concurrently + # during the first inference step. + gpu_memory_utilization=0.85, + ) + + tc_as_dict = tensorizer_config.to_serializable() + + print("lora adapter created") + print("lora 1") + assert ( + do_sample( + loaded_llm, llama32_lora_files, tensorizer_config_dict=tc_as_dict, lora_id=1 + ) + == EXPECTED_LORA_OUTPUT + ) diff --git a/asian_ai/performance/vLLM/tests/lora/test_lora_checkpoints.py b/asian_ai/performance/vLLM/tests/lora/test_lora_checkpoints.py new file mode 100644 index 0000000000000000000000000000000000000000..d4915048e4b190833a5a75e6b5d55e9245b9376f --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_lora_checkpoints.py @@ -0,0 +1,156 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from vllm.lora.lora_model import LoRAModel +from vllm.lora.peft_helper import PEFTHelper +from vllm.lora.utils import parse_fine_tuned_lora_name +from vllm.model_executor.models.gemma4 import Gemma4ForCausalLM +from vllm.model_executor.models.utils import WeightsMapper + +lora_lst = ["baichuan7B", "baichuan7B-zero", "baichuan7B-zero-regex", "chatglm3-6b"] +BAICHUAN_LORA_MODULES = [ + "W_pack", + "o_proj", + "gate_up_proj", + "down_proj", +] + +MOCK_PACKED_MAPPING = { + "W_pack": ["W_pack"], + "gate_up_proj": [ + "gate_proj", + "up_proj", + ], +} + + +@pytest.mark.parametrize("lora_name", lora_lst) +def test_load_checkpoints( + lora_name, + baichuan_lora_files, + baichuan_zero_lora_files, + baichuan_regex_lora_files, + chatglm3_lora_files, +): + expected_lora_lst: list[str] = [] + for module in BAICHUAN_LORA_MODULES: + if module in MOCK_PACKED_MAPPING: + expected_lora_lst.extend(MOCK_PACKED_MAPPING[module]) + else: + expected_lora_lst.append(module) + expected_lora_modules = set(expected_lora_lst) + if lora_name == "baichuan7B": + peft_helper = PEFTHelper.from_local_dir( + baichuan_lora_files, max_position_embeddings=4096 + ) + # For the baichuan7B model, load it's LoRA, + # and the test should pass. + LoRAModel.from_local_checkpoint( + baichuan_lora_files, + expected_lora_modules, + peft_helper=peft_helper, + lora_model_id=1, + device="cpu", + model_vocab_size=64000, + ) + elif lora_name == "baichuan7B-zero": + # Test that the target_modules contain prefix + # such as "model.layers.0.self_atten.W_pack", and + # the test should pass. + peft_helper = PEFTHelper.from_local_dir( + baichuan_zero_lora_files, max_position_embeddings=4096 + ) + LoRAModel.from_local_checkpoint( + baichuan_zero_lora_files, + expected_lora_modules, + peft_helper=peft_helper, + lora_model_id=1, + device="cpu", + model_vocab_size=64000, + ) + elif lora_name == "baichuan7B-zero-regex": + # Test that the `target_modules` in the form of regular expressions, + # such as `model\\..*(W_pack|o_proj)`, and the test should pass. + peft_helper = PEFTHelper.from_local_dir( + baichuan_regex_lora_files, max_position_embeddings=4096 + ) + LoRAModel.from_local_checkpoint( + baichuan_regex_lora_files, + expected_lora_modules, + peft_helper=peft_helper, + lora_model_id=1, + device="cpu", + model_vocab_size=64000, + ) + else: + # For the baichuan7B model, load chatglm3-6b's LoRA, + # and the test should raise the following error. + expected_error = "Please verify that the loaded LoRA module is correct" # noqa: E501 + peft_helper = PEFTHelper.from_local_dir( + chatglm3_lora_files, max_position_embeddings=4096 + ) + with pytest.raises(ValueError, match=expected_error): + LoRAModel.from_local_checkpoint( + chatglm3_lora_files, + expected_lora_modules, + peft_helper=peft_helper, + lora_model_id=1, + device="cpu", + model_vocab_size=64000, + ) + + +def test_lora_weights_mapping(baichuan_lora_files): + expected_lora_lst: list[str] = [] + for module in BAICHUAN_LORA_MODULES: + if module in MOCK_PACKED_MAPPING: + expected_lora_lst.extend(MOCK_PACKED_MAPPING[module]) + else: + expected_lora_lst.append(module) + expected_lora_modules = set(expected_lora_lst) + hf_to_vllm_mapper = WeightsMapper( + orig_to_new_prefix={ + "model.": "language_model.model.", + }, + orig_to_new_substr={ + ".layers.": ".baichuan_layers.", + }, + ) + peft_helper = PEFTHelper.from_local_dir( + baichuan_lora_files, max_position_embeddings=4096 + ) + lora_model = LoRAModel.from_local_checkpoint( + baichuan_lora_files, + expected_lora_modules, + peft_helper=peft_helper, + lora_model_id=1, + device="cpu", + model_vocab_size=64000, + weights_mapper=hf_to_vllm_mapper, + ) + for name in lora_model.loras: + assert name.startswith(hf_to_vllm_mapper.orig_to_new_prefix["model."]) + assert ".baichuan_layers." in name + + +def test_gemma4_lora_weights_mapping(): + mapper = Gemma4ForCausalLM.hf_to_vllm_mapper + name = "base_model.model.model.language_model.layers.9.mlp.down_proj.lora_A.weight" + assert parse_fine_tuned_lora_name(name, mapper) == ( + "model.layers.9.mlp.down_proj", + True, + ) + + +def test_gemma4_moe_lora_weights_mapping(): + mapper = Gemma4ForCausalLM.hf_to_vllm_mapper + name = ( + "base_model.model.model.language_model.layers.9.moe.experts." + "gate_up_proj.lora_B.weight" + ) + assert parse_fine_tuned_lora_name(name, mapper) == ( + "model.layers.9.moe.gate_up_proj", + False, + ) diff --git a/asian_ai/performance/vLLM/tests/lora/test_lora_functions.py b/asian_ai/performance/vLLM/tests/lora/test_lora_functions.py new file mode 100644 index 0000000000000000000000000000000000000000..66e992dd4bab8016e5b293f20f5028e63bd1dc03 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_lora_functions.py @@ -0,0 +1,116 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +""" +Script to test add_lora, remove_lora, pin_lora, list_loras functions. +""" + +import pytest + +from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs +from vllm.entrypoints.openai.api_server import ( + build_async_engine_client_from_engine_args, +) +from vllm.lora.request import LoRARequest +from vllm.v1.engine.llm_engine import LLMEngine + +MODEL_PATH = "Qwen/Qwen3-0.6B" +LORA_MODULE_PATH = "charent/self_cognition_Alice" +LORA_RANK = 8 + + +def make_lora_request(lora_id: int): + return LoRARequest( + lora_name=f"{lora_id}", lora_int_id=lora_id, lora_path=LORA_MODULE_PATH + ) + + +def test_lora_functions_sync(): + max_loras = 4 + # Create engine in eager-mode. Due to high max_loras, the CI can + # OOM during cuda-graph capture. + engine_args = EngineArgs( + model=MODEL_PATH, + enable_lora=True, + max_loras=max_loras, + max_lora_rank=LORA_RANK, + max_model_len=128, + gpu_memory_utilization=0.8, + enforce_eager=True, + ) + + llm = LLMEngine.from_engine_args(engine_args) + + def run_check(fn, args, expected: list): + fn(args) + assert set(llm.list_loras()) == set(expected) + + run_check(llm.add_lora, make_lora_request(1), [1]) + run_check(llm.add_lora, make_lora_request(2), [1, 2]) + + # Pin LoRA 1 and test that it is never removed on subsequent adds. + run_check(llm.pin_lora, 1, [1, 2]) + run_check(llm.add_lora, make_lora_request(3), [1, 2, 3]) + run_check(llm.add_lora, make_lora_request(4), [1, 2, 3, 4]) + run_check(llm.add_lora, make_lora_request(5), [1, 5, 3, 4]) + run_check(llm.add_lora, make_lora_request(6), [1, 5, 6, 4]) + run_check(llm.add_lora, make_lora_request(7), [1, 5, 6, 7]) + run_check(llm.add_lora, make_lora_request(8), [1, 8, 6, 7]) + run_check(llm.add_lora, make_lora_request(9), [1, 8, 9, 7]) + run_check(llm.add_lora, make_lora_request(10), [1, 8, 9, 10]) + + # Remove LoRA 1 and continue adding. + run_check(llm.remove_lora, 1, [8, 9, 10]) + run_check(llm.add_lora, make_lora_request(11), [8, 9, 10, 11]) + run_check(llm.add_lora, make_lora_request(12), [12, 9, 10, 11]) + run_check(llm.add_lora, make_lora_request(13), [12, 13, 10, 11]) + + # Remove all LoRAs. + run_check(llm.remove_lora, 13, [12, 10, 11]) + run_check(llm.remove_lora, 12, [10, 11]) + run_check(llm.remove_lora, 11, [10]) + run_check(llm.remove_lora, 10, []) + + +@pytest.mark.asyncio +async def test_lora_functions_async(): + max_loras = 4 + engine_args = AsyncEngineArgs( + model=MODEL_PATH, + enable_lora=True, + max_loras=max_loras, + max_lora_rank=LORA_RANK, + max_model_len=128, + gpu_memory_utilization=0.8, + enforce_eager=True, + ) + + async def run_check(fn, args, expected: list): + await fn(args) + assert set(await llm.list_loras()) == set(expected) + + async with build_async_engine_client_from_engine_args(engine_args) as llm: + await run_check(llm.add_lora, make_lora_request(1), [1]) + await run_check(llm.add_lora, make_lora_request(2), [1, 2]) + + # Pin LoRA 1 and test that it is never removed on subsequent adds. + await run_check(llm.pin_lora, 1, [1, 2]) + await run_check(llm.add_lora, make_lora_request(3), [1, 2, 3]) + await run_check(llm.add_lora, make_lora_request(4), [1, 2, 3, 4]) + await run_check(llm.add_lora, make_lora_request(5), [1, 5, 3, 4]) + await run_check(llm.add_lora, make_lora_request(6), [1, 5, 6, 4]) + await run_check(llm.add_lora, make_lora_request(7), [1, 5, 6, 7]) + await run_check(llm.add_lora, make_lora_request(8), [1, 8, 6, 7]) + await run_check(llm.add_lora, make_lora_request(9), [1, 8, 9, 7]) + await run_check(llm.add_lora, make_lora_request(10), [1, 8, 9, 10]) + + # Remove LoRA 1 and continue adding. + await run_check(llm.remove_lora, 1, [8, 9, 10]) + await run_check(llm.add_lora, make_lora_request(11), [8, 9, 10, 11]) + await run_check(llm.add_lora, make_lora_request(12), [12, 9, 10, 11]) + await run_check(llm.add_lora, make_lora_request(13), [12, 13, 10, 11]) + + # Remove all LoRAs + await run_check(llm.remove_lora, 13, [12, 10, 11]) + await run_check(llm.remove_lora, 12, [10, 11]) + await run_check(llm.remove_lora, 11, [10]) + await run_check(llm.remove_lora, 10, []) diff --git a/asian_ai/performance/vLLM/tests/lora/test_lora_huggingface.py b/asian_ai/performance/vLLM/tests/lora/test_lora_huggingface.py new file mode 100644 index 0000000000000000000000000000000000000000..46413cd15cc4d5b1a0d93aa6c28d2d8d3484c22d --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_lora_huggingface.py @@ -0,0 +1,48 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +from vllm.lora.lora_model import LoRAModel +from vllm.lora.peft_helper import PEFTHelper +from vllm.lora.utils import get_adapter_absolute_path +from vllm.model_executor.models.qwen3 import Qwen3ForCausalLM + +# Provide absolute path and huggingface lora ids +lora_fixture_name = ["llama32_lora_files", "llama32_lora_huggingface_id"] +LLAMA_LORA_MODULES = [ + "qkv_proj", + "o_proj", + "gate_up_proj", + "down_proj", + "embed_tokens", + "lm_head", +] + + +@pytest.mark.parametrize("lora_fixture_name", lora_fixture_name) +def test_load_checkpoints_from_huggingface(lora_fixture_name, request): + lora_name = request.getfixturevalue(lora_fixture_name) + packed_modules_mapping = Qwen3ForCausalLM.packed_modules_mapping + + expected_lora_lst: list[str] = [] + for module in LLAMA_LORA_MODULES: + if module in packed_modules_mapping: + expected_lora_lst.extend(packed_modules_mapping[module]) + else: + expected_lora_lst.append(module) + expected_lora_modules = set(expected_lora_lst) + lora_path = get_adapter_absolute_path(lora_name) + + # lora loading should work for either absolute path and huggingface id. + peft_helper = PEFTHelper.from_local_dir(lora_path, 4096) + lora_model = LoRAModel.from_local_checkpoint( + lora_path, + expected_lora_modules, + peft_helper=peft_helper, + lora_model_id=1, + device="cpu", + ) + + # Assertions to ensure the model is loaded correctly + assert lora_model is not None, "LoRAModel is not loaded correctly" diff --git a/asian_ai/performance/vLLM/tests/lora/test_lora_manager.py b/asian_ai/performance/vLLM/tests/lora/test_lora_manager.py new file mode 100644 index 0000000000000000000000000000000000000000..7a4c4b2454eb9a45d27a02600dd4c0d70a2c0e38 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_lora_manager.py @@ -0,0 +1,1052 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import os + +import pytest +import torch +from safetensors.torch import load_file +from torch import nn + +from vllm.config import ModelConfig, VllmConfig +from vllm.config.lora import LoRAConfig +from vllm.lora.layers import ( + ColumnParallelLinearWithLoRA, + MergedColumnParallelLinearWithLoRA, + ReplicatedLinearWithLoRA, + RowParallelLinearWithLoRA, +) +from vllm.lora.lora_model import LoRAModel +from vllm.lora.lora_weights import LoRALayerWeights, PackedLoRALayerWeights +from vllm.lora.model_manager import ( + DEFAULT_LANGUAGE_WRAPPER_KEY, + LoRAMapping, + LoRAModelManager, + LRUCacheLoRAModelManager, +) +from vllm.lora.peft_helper import PEFTHelper +from vllm.lora.request import LoRARequest +from vllm.lora.worker_manager import LRUCacheWorkerLoRAManager, WorkerLoRAManager +from vllm.model_executor.layers.fused_moe import GateLinear +from vllm.platforms import current_platform + +from .utils import create_peft_lora + +EMBEDDING_MODULES = { + "embed_tokens": "input_embeddings", + "lm_head": "output_embeddings", +} + +DEVICE_TYPE = current_platform.device_type +DEVICES = ( + [f"{DEVICE_TYPE}:{i}" for i in range(min(torch.accelerator.device_count(), 2))] + if (current_platform.is_cuda_alike() or current_platform.is_xpu()) + else ["cpu"] +) + +DEFAULT_DTYPE = torch.get_default_dtype() + + +@pytest.mark.parametrize("device", DEVICES) +def test_from_lora_tensors(qwen3_lora_files, device): + tensors = load_file(os.path.join(qwen3_lora_files, "adapter_model.safetensors")) + + peft_helper = PEFTHelper.from_local_dir( + qwen3_lora_files, max_position_embeddings=4096 + ) + lora_model = LoRAModel.from_lora_tensors( + 1, + tensors, + peft_helper=peft_helper, + device=device, + ) + for module_name, lora in lora_model.loras.items(): + assert lora.module_name == module_name + assert lora.rank == 8 + assert lora.lora_alpha == 32 + assert lora.lora_a is not None + assert lora.lora_b is not None + assert lora.lora_a.device == torch.device(device) + assert lora.lora_b.device == torch.device(device) + assert lora.lora_a.shape[0] == lora.lora_b.shape[1], ( + f"{lora.lora_a.shape=}, {lora.lora_b.shape=}" + ) + assert lora.lora_a.shape[0] == 8 + + +def create_lora( + lora_id: int, model: nn.Module, sub_modules: list[str], device: torch.device +) -> LoRAModel: + loras: dict[str, LoRALayerWeights] = {} + for name in sub_modules: + w = model.get_submodule(name).weight + loras[name] = LoRALayerWeights( + name, + 8, + 16, + torch.rand([8, w.shape[1]], device=device), + torch.rand([w.shape[0], 8], device=device), + ) + return LoRAModel(lora_id, 8, loras) + + +def create_packed_lora( + lora_id: int, + model: nn.Module, + module_name, + replaced_module_names, + device: torch.device, + empty_replaced_module_name=None, +) -> LoRAModel: + w = model.get_submodule(module_name).weight + loras: dict[str, LoRALayerWeights] = {} + for replaced_module_name in replaced_module_names: + if replaced_module_name == empty_replaced_module_name: + continue + loras[replaced_module_name] = LoRALayerWeights( + replaced_module_name, + 8, + 16, + torch.rand([8, w.shape[1]], device=device), + torch.rand([w.shape[0] // len(replaced_module_names), 8], device=device), + ) + return LoRAModel(lora_id, 8, loras) + + +def test_replace_submodules(default_vllm_config, dist_init, dummy_model): + model = dummy_model + manager = LoRAModelManager( + model, + 1, + 1, + 1, + LoRAConfig( + max_lora_rank=8, max_cpu_loras=8, max_loras=8, lora_dtype=DEFAULT_DTYPE + ), + torch.device(DEVICES[0]), + default_vllm_config, + ) + model = manager.model + assert isinstance(model.get_submodule("dense1"), ColumnParallelLinearWithLoRA) + assert isinstance( + model.get_submodule("layer1.dense1"), ColumnParallelLinearWithLoRA + ) + assert isinstance(model.get_submodule("dense2"), RowParallelLinearWithLoRA) + assert isinstance(model.get_submodule("layer1.dense2"), RowParallelLinearWithLoRA) + + +def test_wrap_replicated_linear_subclasses(default_vllm_config, dist_init, dummy_model): + from vllm.model_executor.layers.linear import ReplicatedLinear + + class CustomReplicatedLinear(ReplicatedLinear): + pass + + model = dummy_model + model.add_module("custom_gate", CustomReplicatedLinear(10, 10, bias=False)) + + manager = LoRAModelManager( + model, + 1, + 1, + 1, + LoRAConfig( + max_lora_rank=8, max_cpu_loras=8, max_loras=8, lora_dtype=DEFAULT_DTYPE + ), + torch.device(DEVICES[0]), + default_vllm_config, + ) + + assert isinstance( + manager.model.get_submodule("custom_gate"), ReplicatedLinearWithLoRA + ) + + +def test_wrap_gate_linear(default_vllm_config, dist_init, dummy_model): + model = dummy_model + model.add_module("router_gate", GateLinear(10, 4, bias=False)) + + manager = LoRAModelManager( + model, + 1, + 1, + 1, + LoRAConfig( + max_lora_rank=8, max_cpu_loras=8, max_loras=8, lora_dtype=DEFAULT_DTYPE + ), + torch.device(DEVICES[0]), + default_vllm_config, + ) + + assert isinstance( + manager.model.get_submodule("router_gate"), ReplicatedLinearWithLoRA + ) + + +def test_dedup_shared_module_across_paths(default_vllm_config, dist_init, dummy_model): + """A module reachable from two attribute paths (e.g. a MoE gate held + both directly on the block and inside its inner runner) must produce a + single LoRA wrapper. Both paths must end up pointing to that same + wrapper instance, and only the canonical path should live in + `manager.modules` — otherwise activate_adapter would call `reset_lora` + on the alias and clobber weights set under the canonical name. + """ + from vllm.model_executor.layers.linear import ReplicatedLinear + + class AliasContainer(nn.Module): + def __init__(self, gate: nn.Module): + super().__init__() + self.gate = gate # canonical path: "moe.gate" + + # Inner submodule holding the SAME gate instance under another + # path. This mirrors how FusedMoE.runner.gate references the + # block's gate in qwen3_moe. + class _Runner(nn.Module): + def __init__(self, g): + super().__init__() + self.gate = g # alias path: "moe.runner.gate" + + self.runner = _Runner(gate) + + gate = ReplicatedLinear(10, 4, bias=False) + model = dummy_model + model.add_module("moe", AliasContainer(gate)) + + assert model.moe.gate is model.moe.runner.gate + + manager = LoRAModelManager( + model, + 1, + 1, + 1, + LoRAConfig( + max_lora_rank=8, max_cpu_loras=8, max_loras=8, lora_dtype=DEFAULT_DTYPE + ), + torch.device(DEVICES[0]), + default_vllm_config, + ) + + canonical = manager.model.get_submodule("moe.gate") + alias = manager.model.get_submodule("moe.runner.gate") + + # Same wrapper instance on both paths so forward through either side + # sees the LoRA-augmented module. + assert isinstance(canonical, ReplicatedLinearWithLoRA) + assert alias is canonical + + # Only the canonical path is tracked as a LoRA target. Tracking the + # alias would cause activate_adapter to reset_lora on it after the + # canonical entry already populated the weights. + assert "moe.gate" in manager.modules + assert "moe.runner.gate" not in manager.modules + + +def test_lm_head_exempt_from_dedup(default_vllm_config, dist_init, dummy_model): + """The dedup logic must NOT collapse `lm_head` even when it is reachable + from another attribute path (tied-embedding models do + `self.lm_head = self.model.embed_tokens`, sharing the same nn.Module + instance). The lm_head branch additionally rewires `logits_processor` + into a `LogitsProcessorWithLoRA`, so skipping it would silently break + LoRA on lm_head. + """ + from vllm.lora.layers import LogitsProcessorWithLoRA + + # Add a non-lm_head alias to the same module instance as lm_head. The + # dedup keys on id(module); without the lm_head exemption the alias + # would consume the wrapped_by_id slot first and lm_head would be + # silently skipped, so logits_processor would never be wrapped. + model = dummy_model + model.add_module("embed_tokens", model.lm_head) + assert model.embed_tokens is model.lm_head + + manager = LoRAModelManager( + model, + 1, + 1, + 1, + LoRAConfig( + max_lora_rank=8, max_cpu_loras=8, max_loras=8, lora_dtype=DEFAULT_DTYPE + ), + torch.device(DEVICES[0]), + default_vllm_config, + ) + + # lm_head's special handling still ran: logits_processor got wrapped + # and the lm_head entry is tracked under self.modules. + assert isinstance( + manager.model.get_submodule("logits_processor"), LogitsProcessorWithLoRA + ) + assert "lm_head" in manager.modules + + +def test_skip_unsupported_matched_modules(default_vllm_config, dist_init, dummy_model): + class UnsupportedContainer(nn.Module): + def __init__(self): + super().__init__() + # This name matches a supported target suffix ("dense1"), + # but nn.Linear is not currently a LoRA-wrappable layer type. + self.dense1 = nn.Linear(10, 10, bias=False) + + model = dummy_model + model.add_module("unsupported", UnsupportedContainer()) + + manager = LoRAModelManager( + model, + 1, + 1, + 1, + LoRAConfig( + max_lora_rank=8, max_cpu_loras=8, max_loras=8, lora_dtype=DEFAULT_DTYPE + ), + torch.device(DEVICES[0]), + default_vllm_config, + ) + + # Should not crash and should keep unsupported matched modules unchanged. + assert isinstance(manager.model.get_submodule("unsupported.dense1"), nn.Linear) + assert "unsupported.dense1" not in manager.modules + + +def test_target_modules_fail_closed_on_unsupported_matched_modules( + default_vllm_config, dist_init, dummy_model +): + class UnsupportedContainer(nn.Module): + def __init__(self): + super().__init__() + self.dense1 = nn.Linear(10, 10, bias=False) + + model = dummy_model + model.add_module("unsupported", UnsupportedContainer()) + + with pytest.raises(ValueError, match="unsupported.dense1"): + LoRAModelManager( + model, + 1, + 1, + 1, + LoRAConfig( + max_lora_rank=8, + max_cpu_loras=8, + max_loras=8, + lora_dtype=DEFAULT_DTYPE, + target_modules=["dense1"], + ), + torch.device(DEVICES[0]), + default_vllm_config, + ) + + +def test_get_dummy_lora_warmup_rank_for_fully_sharded_moe(): + manager = LoRAModelManager.__new__(LoRAModelManager) + manager.lora_config = LoRAConfig( + max_lora_rank=64, + max_cpu_loras=1, + max_loras=1, + lora_dtype=DEFAULT_DTYPE, + fully_sharded_loras=True, + ) + + class DummyModule: + def __init__(self, tp_size: int, fully_sharded: bool): + self.tp_size = tp_size + self.fully_sharded = fully_sharded + + manager.modules = { + "model.layers.0.self_attn.q_proj": DummyModule( + tp_size=32, + fully_sharded=True, + ), + "model.layers.0.mlp.experts": DummyModule( + tp_size=32, + fully_sharded=True, + ), + } + + assert manager.get_dummy_lora_warmup_rank(8) == 32 + + +@pytest.mark.parametrize("device", DEVICES) +def test_lora_model_manager(default_vllm_config, dist_init, dummy_model, device): + model = dummy_model + model_lora1 = create_lora( + 1, model, ["layer1.dense1", "dense2", "lm_head"], device=device + ) + model_lora2 = create_lora(2, model, ["dense1", "dense2", "lm_head"], device=device) + model_lora3 = create_lora(3, model, ["dense1", "dense2", "lm_head"], device=device) + manager = LoRAModelManager( + model, + 2, + 2, + 2, + LoRAConfig( + max_lora_rank=8, max_cpu_loras=3, max_loras=2, lora_dtype=DEFAULT_DTYPE + ), + device=device, + vllm_config=default_vllm_config, + ) + assert all(x is None for x in manager.lora_index_to_id) + assert manager.add_adapter(model_lora1) + assert manager.activate_adapter(1) + assert manager.lora_index_to_id[0] == 1 + assert not manager.add_adapter(model_lora1) + assert not manager.activate_adapter(1) + assert manager.add_adapter(model_lora2) + assert manager.activate_adapter(2) + assert manager.lora_index_to_id[0] == 1 + assert manager.lora_index_to_id[1] == 2 + assert not manager.add_adapter(model_lora2) + assert not manager.activate_adapter(2) + assert manager.add_adapter(model_lora3) + assert manager.lora_index_to_id[0] == 1 + assert manager.lora_index_to_id[1] == 2 + with pytest.raises(ValueError): + assert manager.activate_adapter(3) + assert manager.lora_index_to_id[0] == 1 + assert manager.lora_index_to_id[1] == 2 + assert manager.remove_adapter(model_lora2.id) + assert manager.lora_index_to_id[1] is None + assert not manager.remove_adapter(model_lora2.id) + assert manager.remove_adapter(model_lora1.id) + assert not manager.remove_adapter(model_lora1.id) + assert manager.add_adapter(model_lora1) + assert manager.lora_index_to_id[0] is None + assert manager.lora_index_to_id[1] is None + assert manager.add_adapter(model_lora2) + assert manager.activate_adapter(3) + assert manager.lora_index_to_id[0] == 3 + assert manager.lora_index_to_id[1] is None + assert manager.activate_adapter(2) + assert manager.lora_index_to_id[0] == 3 + assert manager.lora_index_to_id[1] == 2 + assert manager.device == device + assert ( + manager.punica_wrapper_mapping.get(DEFAULT_LANGUAGE_WRAPPER_KEY).device + == device + ) + assert hasattr(manager, "supported_lora_modules") + assert sorted(manager.supported_lora_modules) == [ + "dense1", + "dense2", + "lm_head", + "output", + ] + + +@pytest.mark.parametrize("device", DEVICES) +def test_lora_lru_cache_model_manager( + default_vllm_config, dist_init, dummy_model, device +): + model = dummy_model + model_lora1 = create_lora( + 1, model, ["layer1.dense1", "dense2", "lm_head"], device=device + ) + model_lora2 = create_lora(2, model, ["dense1", "dense2", "lm_head"], device=device) + model_lora3 = create_lora(3, model, ["dense1", "dense2", "lm_head"], device=device) + manager = LRUCacheLoRAModelManager( + model, + 2, + 2, + 2, + LoRAConfig( + max_lora_rank=8, max_cpu_loras=3, max_loras=2, lora_dtype=DEFAULT_DTYPE + ), + device=device, + vllm_config=default_vllm_config, + ) + assert all(x is None for x in manager.lora_index_to_id) + assert manager.add_adapter(model_lora1) + assert manager.activate_adapter(1) + assert manager.lora_index_to_id[0] == 1 + assert not manager.add_adapter(model_lora1) + assert not manager.activate_adapter(1) + assert manager.add_adapter(model_lora2) + assert manager.activate_adapter(2) + assert manager.lora_index_to_id[0] == 1 + assert manager.lora_index_to_id[1] == 2 + assert not manager.add_adapter(model_lora2) + assert not manager.activate_adapter(2) + assert manager.add_adapter(model_lora3) + assert manager.lora_index_to_id[0] == 1 + assert manager.lora_index_to_id[1] == 2 + assert manager.activate_adapter(3) + assert manager.lora_index_to_id[0] == 3 + assert manager.lora_index_to_id[1] == 2 + assert manager.remove_adapter(model_lora2.id) + assert manager.lora_index_to_id[1] is None + assert not manager.remove_adapter(model_lora2.id) + assert manager.remove_adapter(model_lora1.id) + assert not manager.remove_adapter(model_lora1.id) + assert manager.add_adapter(model_lora1) + assert manager.activate_adapter(1) + assert manager.lora_index_to_id[0] == 3 + assert manager.lora_index_to_id[1] == 1 + assert manager.add_adapter(model_lora2) + assert manager.deactivate_adapter(3) + assert manager.lora_index_to_id[0] is None + assert manager.lora_index_to_id[1] == 1 + assert manager.activate_adapter(2) + assert manager.lora_index_to_id[0] == 2 + assert manager.lora_index_to_id[1] == 1 + assert manager.activate_adapter(3) + assert manager.lora_index_to_id[0] == 2 + assert manager.lora_index_to_id[1] == 3 + assert manager.pin_adapter(2) + assert manager.lora_index_to_id[0] == 2 + assert manager.lora_index_to_id[1] == 3 + assert manager.activate_adapter(1) + assert manager.lora_index_to_id[0] == 2 + assert manager.lora_index_to_id[1] == 1 + assert manager.deactivate_adapter(2) + assert manager.lora_index_to_id[0] is None + assert manager.lora_index_to_id[1] == 1 + assert manager.activate_adapter(3) + assert manager.lora_index_to_id[0] == 3 + assert manager.lora_index_to_id[1] == 1 + assert manager.pin_adapter(3) + assert manager.pin_adapter(1) + with pytest.raises(RuntimeError): + assert manager.pin_adapter(2) + assert manager.lora_index_to_id[0] == 3 + assert manager.lora_index_to_id[1] == 1 + with pytest.raises(RuntimeError): + assert manager.activate_adapter(2) + + assert manager.deactivate_adapter(3) + assert manager.pin_adapter(2) + assert manager.lora_index_to_id[0] == 2 + assert manager.lora_index_to_id[1] == 1 + assert manager.remove_adapter(3) + with pytest.raises(ValueError): + assert manager.pin_adapter(3) + assert ( + manager.punica_wrapper_mapping.get(DEFAULT_LANGUAGE_WRAPPER_KEY).device + == device + ) + assert manager.device == device + + +@pytest.mark.parametrize("device", DEVICES) +def test_lru_lora_model_manager(default_vllm_config, dist_init, dummy_model, device): + # This tests just the LRU cache functionality, everything else is + # tested in test_lora_model_manager + model = dummy_model + model_lora1 = create_lora( + 1, model, ["layer1.dense1", "dense2", "lm_head"], device=device + ) + model_lora2 = create_lora(2, model, ["dense1", "dense2", "lm_head"], device=device) + model_lora3 = create_lora(3, model, ["dense1", "dense2", "lm_head"], device=device) + model_lora4 = create_lora(4, model, ["dense1", "dense2", "lm_head"], device=device) + manager = LRUCacheLoRAModelManager( + model, + 2, + 2, + 2, + LoRAConfig( + max_lora_rank=8, max_cpu_loras=2, max_loras=2, lora_dtype=DEFAULT_DTYPE + ), + device=device, + vllm_config=default_vllm_config, + ) + assert all(x is None for x in manager.lora_index_to_id) + + # Add up to capacity + assert manager.add_adapter(model_lora1) + assert manager.add_adapter(model_lora2) + assert manager.activate_adapter(1) + assert manager.activate_adapter(2) + + assert set(manager.list_adapters()) == {1, 2} + assert manager.lora_index_to_id[0] == 1 + assert manager.lora_index_to_id[1] == 2 + + # Add over capacity + assert manager.add_adapter(model_lora3) + assert manager.add_adapter(model_lora4) + assert manager.activate_adapter(3) + assert manager.activate_adapter(4) + + assert set(manager.list_adapters()) == {3, 4} + assert manager.lora_index_to_id[0] == 3 + assert manager.lora_index_to_id[1] == 4 + + # Add 3 again to move it to the top and then add 2 + # should return false since it's in already + assert not manager.add_adapter(model_lora3) + assert not manager.activate_adapter(3) + assert manager.add_adapter(model_lora2) + assert manager.activate_adapter(2) + + assert set(manager.list_adapters()) == {3, 2} + assert manager.lora_index_to_id[0] == 3 + assert manager.lora_index_to_id[1] == 2 + + # Remove manually + assert manager.remove_adapter(3) + assert not manager.remove_adapter(3) + + assert set(manager.list_adapters()) == {2} + assert manager.lora_index_to_id[0] is None + assert manager.lora_index_to_id[1] == 2 + + assert manager.add_adapter(model_lora3) + assert manager.activate_adapter(3) + assert manager.add_adapter(model_lora4) + assert manager.activate_adapter(4) + + assert set(manager.list_adapters()) == {3, 4} + assert manager.lora_index_to_id[0] == 3 + assert manager.lora_index_to_id[1] == 4 + + assert manager.remove_oldest_adapter() + assert set(manager.list_adapters()) == {4} + assert manager.lora_index_to_id[0] is None + assert manager.lora_index_to_id[1] == 4 + + assert manager.remove_oldest_adapter() + assert set(manager.list_adapters()) == set() + assert all(x is None for x in manager.lora_index_to_id) + + assert not manager.remove_oldest_adapter() + assert set(manager.list_adapters()) == set() + assert all(x is None for x in manager.lora_index_to_id) + + # pinning + assert manager.add_adapter(model_lora3) + assert manager.activate_adapter(3) + assert manager.add_adapter(model_lora4) + assert manager.activate_adapter(4) + assert set(manager.list_adapters()) == {3, 4} + with pytest.raises(ValueError): + assert manager.pin_adapter(1) + assert manager.pin_adapter(3) + # Remove manually + assert manager.remove_adapter(3) + assert not manager.remove_adapter(3) + + assert set(manager.list_adapters()) == {4} + assert manager.lora_index_to_id[0] is None + assert manager.lora_index_to_id[1] == 4 + + assert manager.add_adapter(model_lora1) + assert manager.pin_adapter(1) + assert manager.add_adapter(model_lora2) + assert manager.activate_adapter(2) + + assert set(manager.list_adapters()) == {1, 2} + assert manager.lora_index_to_id[0] == 1 + assert manager.lora_index_to_id[1] == 2 + + assert manager.remove_oldest_adapter() + assert set(manager.list_adapters()) == {1} + assert manager.lora_index_to_id[0] == 1 + assert manager.lora_index_to_id[1] is None + + with pytest.raises(RuntimeError): + assert manager.remove_oldest_adapter() + + assert set(manager.list_adapters()) == {1} + assert ( + manager.punica_wrapper_mapping.get(DEFAULT_LANGUAGE_WRAPPER_KEY).device + == device + ) + assert manager.device == device + + +@pytest.mark.parametrize("device", DEVICES) +def test_lru_cache_worker_adapter_manager(dist_init, dummy_model, device, tmp_path): + lora_config = LoRAConfig( + max_lora_rank=8, max_cpu_loras=4, max_loras=4, lora_dtype=DEFAULT_DTYPE + ) + + dummy_lora_files = f"{tmp_path}/lora_adapter" + os.makedirs(dummy_lora_files, exist_ok=True) + create_peft_lora( + dummy_model, + save_dir=dummy_lora_files, + target_modules=["layer1.dense1", "dense2"], + lora_dtype=DEFAULT_DTYPE, + ) + + model_config = ModelConfig(max_model_len=16) + vllm_config = VllmConfig(model_config=model_config, lora_config=lora_config) + + vllm_config.scheduler_config.max_num_seqs = 4 + vllm_config.scheduler_config.max_num_batched_tokens = 2 + worker_adapter_manager = LRUCacheWorkerLoRAManager( + vllm_config, device, EMBEDDING_MODULES + ) + + worker_adapter_manager.max_num_seqs = 4 + worker_adapter_manager.max_num_batched_tokens = 2 + + worker_adapter_manager.create_lora_manager(dummy_model, vllm_config) + + mapping = LoRAMapping([], []) + worker_adapter_manager.set_active_adapters( + [LoRARequest("1", 1, dummy_lora_files), LoRARequest("2", 2, dummy_lora_files)], + mapping, + ) + assert worker_adapter_manager.list_adapters() == {1, 2} + assert worker_adapter_manager._adapter_manager.lora_index_to_id[0] == 1 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[1] == 2 + + worker_adapter_manager.set_active_adapters( + [ + LoRARequest("1", 1, dummy_lora_files), + LoRARequest("3", 3, dummy_lora_files), + LoRARequest("4", 4, dummy_lora_files), + ], + mapping, + ) + assert worker_adapter_manager.list_adapters() == {1, 2, 3, 4} + assert worker_adapter_manager._adapter_manager.lora_index_to_id[0] == 1 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[1] == 2 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[2] == 3 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[3] == 4 + + worker_adapter_manager.set_active_adapters( + [ + LoRARequest("1", 1, dummy_lora_files), + LoRARequest("2", 2, dummy_lora_files), + LoRARequest("5", 5, dummy_lora_files), + ], + mapping, + ) + assert worker_adapter_manager.list_adapters() == {1, 2, 4, 5} + assert worker_adapter_manager._adapter_manager.lora_index_to_id[0] == 1 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[1] == 2 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[2] == 5 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[3] == 4 + + worker_adapter_manager.set_active_adapters( + [ + LoRARequest("1", 1, dummy_lora_files), + LoRARequest("1", 1, dummy_lora_files), + LoRARequest("1", 1, dummy_lora_files), + ], + mapping, + ) + assert worker_adapter_manager.list_adapters() == {1, 2, 4, 5} + assert worker_adapter_manager._adapter_manager.lora_index_to_id[0] == 1 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[1] == 2 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[2] == 5 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[3] == 4 + + worker_adapter_manager.set_active_adapters( + [ + LoRARequest("6", 6, dummy_lora_files), + LoRARequest("7", 7, dummy_lora_files), + LoRARequest("8", 8, dummy_lora_files), + ], + mapping, + ) + assert worker_adapter_manager.list_adapters() == {1, 6, 7, 8} + assert worker_adapter_manager._adapter_manager.lora_index_to_id[0] == 1 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[1] == 7 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[2] == 8 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[3] == 6 + + # Over capacity + with pytest.raises(RuntimeError): + worker_adapter_manager.set_active_adapters( + [ + LoRARequest("10", 10, dummy_lora_files), + LoRARequest("11", 11, dummy_lora_files), + LoRARequest("12", 12, dummy_lora_files), + LoRARequest("13", 13, dummy_lora_files), + LoRARequest("14", 14, dummy_lora_files), + ], + mapping, + ) + + assert worker_adapter_manager.device == device + punica_wrapper = worker_adapter_manager._adapter_manager.punica_wrapper_mapping.get( + DEFAULT_LANGUAGE_WRAPPER_KEY + ) + assert punica_wrapper.device == device + + +@pytest.mark.parametrize("device", DEVICES) +def test_worker_adapter_manager(dist_init, dummy_model_gate_up, device, tmp_path): + # Should remove every LoRA not specified in the request. + lora_config = LoRAConfig( + max_lora_rank=8, max_cpu_loras=4, max_loras=4, lora_dtype=DEFAULT_DTYPE + ) + + model_config = ModelConfig(max_model_len=16) + vllm_config = VllmConfig(model_config=model_config, lora_config=lora_config) + + vllm_config.scheduler_config.max_num_seqs = 4 + vllm_config.scheduler_config.max_num_batched_tokens = 2 + + worker_adapter_manager = WorkerLoRAManager(vllm_config, device, EMBEDDING_MODULES) + worker_adapter_manager.vocab_size = dummy_model_gate_up.unpadded_vocab_size + worker_adapter_manager.create_lora_manager(dummy_model_gate_up, vllm_config) + + dummy_lora_files = f"{tmp_path}/lora_adapter" + os.makedirs(dummy_lora_files, exist_ok=True) + create_peft_lora( + dummy_model_gate_up, + save_dir=dummy_lora_files, + target_modules=["layer1.dense1", "dense2"], + lora_dtype=DEFAULT_DTYPE, + ) + + mapping = LoRAMapping([], []) + worker_adapter_manager.set_active_adapters( + [LoRARequest("1", 1, dummy_lora_files), LoRARequest("2", 2, dummy_lora_files)], + mapping, + ) + assert worker_adapter_manager.list_adapters() == {1, 2} + assert worker_adapter_manager._adapter_manager.lora_index_to_id[0] == 1 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[1] == 2 + + worker_adapter_manager.set_active_adapters( + [ + LoRARequest("1", 1, dummy_lora_files), + LoRARequest("3", 3, dummy_lora_files), + LoRARequest("4", 4, dummy_lora_files), + ], + mapping, + ) + assert worker_adapter_manager.list_adapters() == {1, 3, 4} + assert worker_adapter_manager._adapter_manager.lora_index_to_id[0] == 1 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[1] == 3 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[2] == 4 + + worker_adapter_manager.set_active_adapters( + [ + LoRARequest("1", 1, dummy_lora_files), + LoRARequest("2", 2, dummy_lora_files), + LoRARequest("5", 5, dummy_lora_files), + ], + mapping, + ) + assert worker_adapter_manager.list_adapters() == {1, 2, 5} + assert worker_adapter_manager._adapter_manager.lora_index_to_id[0] == 1 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[1] == 2 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[2] == 5 + + worker_adapter_manager.set_active_adapters( + [ + LoRARequest("1", 1, dummy_lora_files), + LoRARequest("1", 1, dummy_lora_files), + LoRARequest("1", 1, dummy_lora_files), + ], + mapping, + ) + assert worker_adapter_manager.list_adapters() == {1} + assert worker_adapter_manager._adapter_manager.lora_index_to_id[0] == 1 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[1] is None + assert worker_adapter_manager._adapter_manager.lora_index_to_id[2] is None + + worker_adapter_manager.set_active_adapters( + [ + LoRARequest("6", 6, dummy_lora_files), + LoRARequest("7", 7, dummy_lora_files), + LoRARequest("8", 8, dummy_lora_files), + ], + mapping, + ) + assert worker_adapter_manager.list_adapters() == {6, 7, 8} + assert worker_adapter_manager._adapter_manager.lora_index_to_id[0] == 8 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[1] == 6 + assert worker_adapter_manager._adapter_manager.lora_index_to_id[2] == 7 + + # Over capacity + with pytest.raises(RuntimeError): + worker_adapter_manager.set_active_adapters( + [ + LoRARequest("10", 10, dummy_lora_files), + LoRARequest("11", 11, dummy_lora_files), + LoRARequest("12", 12, dummy_lora_files), + LoRARequest("13", 13, dummy_lora_files), + LoRARequest("14", 14, dummy_lora_files), + ], + mapping, + ) + + assert worker_adapter_manager.device == device + punica_wrapper = worker_adapter_manager._adapter_manager.punica_wrapper_mapping.get( + DEFAULT_LANGUAGE_WRAPPER_KEY + ) + assert punica_wrapper.device == device + + +@pytest.mark.parametrize("device", DEVICES) +def test_packed_loras(default_vllm_config, dist_init, dummy_model_gate_up, device): + model = dummy_model_gate_up + model_lora = create_packed_lora( + 1, + model, + module_name="gate_up_proj", + replaced_module_names=["gate_proj", "up_proj"], + device=device, + ) + model_lora1 = create_packed_lora( + 2, + model, + module_name="gate_up_proj", + replaced_module_names=["gate_proj", "up_proj"], + device=device, + empty_replaced_module_name="gate_proj", + ) + + manager = LoRAModelManager( + model, + 2, + 2, + 2, + LoRAConfig( + max_lora_rank=8, max_cpu_loras=2, max_loras=2, lora_dtype=DEFAULT_DTYPE + ), + device=device, + vllm_config=default_vllm_config, + ) + model = manager.model + + assert isinstance( + model.get_submodule("gate_up_proj"), MergedColumnParallelLinearWithLoRA + ) + # Verify packed lora is correct + model_lora_clone = model_lora.clone(1) + model_lora_clone1 = model_lora1.clone(1) + assert manager.add_adapter(model_lora) + assert manager.add_adapter(model_lora1) + + assert model_lora.get_lora("gate_proj") is None + assert model_lora.get_lora("up_proj") is None + assert model_lora1.get_lora("up_proj") is None + packed_lora = model_lora.get_lora("gate_up_proj") + assert packed_lora and isinstance(packed_lora, PackedLoRALayerWeights) + + torch.testing.assert_close( + packed_lora.lora_a[0], model_lora_clone.get_lora("gate_proj").lora_a + ) + torch.testing.assert_close( + packed_lora.lora_b[0], model_lora_clone.get_lora("gate_proj").lora_b + ) + torch.testing.assert_close( + packed_lora.lora_a[1], model_lora_clone.get_lora("up_proj").lora_a + ) + torch.testing.assert_close( + packed_lora.lora_b[1], model_lora_clone.get_lora("up_proj").lora_b + ) + + packed_lora1 = model_lora1.get_lora("gate_up_proj") + assert packed_lora1 and isinstance(packed_lora1, PackedLoRALayerWeights) + + assert packed_lora1.lora_a[0] is None + assert packed_lora1.lora_b[0] is None + torch.testing.assert_close( + packed_lora1.lora_a[1], model_lora_clone1.get_lora("up_proj").lora_a + ) + torch.testing.assert_close( + packed_lora1.lora_b[1], model_lora_clone1.get_lora("up_proj").lora_b + ) + + +def _test_target_modules( + model, + target_modules: list[str] | None, + device: str, + expected_lora: list[tuple[str, type]], + expected_no_lora: list[tuple[str, type]], + vllm_config, +): + """Create a LoRAModelManager and assert which modules have LoRA applied.""" + LoRAModelManager( + model, + 2, + 2, + 2, + LoRAConfig( + max_lora_rank=8, + max_cpu_loras=2, + max_loras=2, + lora_dtype=DEFAULT_DTYPE, + target_modules=target_modules, + ), + device=device, + vllm_config=vllm_config, + ) + for module_path, lora_cls in expected_lora: + assert isinstance(model.get_submodule(module_path), lora_cls) + for module_path, lora_cls in expected_no_lora: + assert not isinstance(model.get_submodule(module_path), lora_cls) + + +@pytest.mark.parametrize("device", DEVICES) +def test_target_modules_config(default_vllm_config, dist_init, dummy_model, device): + """Test that target_modules config restricts which modules get LoRA applied.""" + _test_target_modules( + dummy_model, + ["dense1"], + device, + expected_lora=[ + ("dense1", ColumnParallelLinearWithLoRA), + ("layer1.dense1", ColumnParallelLinearWithLoRA), + ], + expected_no_lora=[ + ("dense2", RowParallelLinearWithLoRA), + ("layer1.dense2", RowParallelLinearWithLoRA), + ], + vllm_config=default_vllm_config, + ) + + +@pytest.mark.parametrize("device", DEVICES) +def test_target_modules_multiple(default_vllm_config, dist_init, dummy_model, device): + """Test that multiple target_modules work correctly.""" + _test_target_modules( + dummy_model, + ["dense1", "dense2"], + device, + expected_lora=[ + ("dense1", ColumnParallelLinearWithLoRA), + ("layer1.dense1", ColumnParallelLinearWithLoRA), + ("dense2", RowParallelLinearWithLoRA), + ("layer1.dense2", RowParallelLinearWithLoRA), + ], + expected_no_lora=[], + vllm_config=default_vllm_config, + ) + + +@pytest.mark.parametrize("device", DEVICES) +def test_target_modules_none_uses_all( + default_vllm_config, dist_init, dummy_model, device +): + """Test that target_modules=None uses all supported modules.""" + _test_target_modules( + dummy_model, + None, + device, + expected_lora=[ + ("dense1", ColumnParallelLinearWithLoRA), + ("layer1.dense1", ColumnParallelLinearWithLoRA), + ("dense2", RowParallelLinearWithLoRA), + ("layer1.dense2", RowParallelLinearWithLoRA), + ], + expected_no_lora=[], + vllm_config=default_vllm_config, + ) + + +@pytest.mark.parametrize("device", DEVICES) +def test_target_modules_match_packed_runtime_modules( + default_vllm_config, dist_init, dummy_model_gate_up, device +): + """Packed runtime modules should be selected by their adapter-visible names.""" + _test_target_modules( + dummy_model_gate_up, + ["gate_proj"], + device, + expected_lora=[("gate_up_proj", MergedColumnParallelLinearWithLoRA)], + expected_no_lora=[ + ("dense1", ColumnParallelLinearWithLoRA), + ("dense2", RowParallelLinearWithLoRA), + ("layer1.dense1", ColumnParallelLinearWithLoRA), + ("layer1.dense2", RowParallelLinearWithLoRA), + ], + vllm_config=default_vllm_config, + ) diff --git a/asian_ai/performance/vLLM/tests/lora/test_lora_utils.py b/asian_ai/performance/vLLM/tests/lora/test_lora_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..7f3e64dae6c066dbf230c666ff8c363b68a10247 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_lora_utils.py @@ -0,0 +1,81 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + + +from vllm.lora.utils import is_in_target_modules, is_supported_lora_module + + +class TestIsSupportedLoraModule: + """Tests for is_supported_lora_module (model-definition check).""" + + def test_suffix_match(self): + assert is_supported_lora_module( + "model.layers.0.self_attn.o_proj", ["o_proj", "q_proj"] + ) + + def test_no_match(self): + assert not is_supported_lora_module( + "model.layers.0.self_attn.o_proj", ["q_proj", "k_proj"] + ) + + def test_exact_match(self): + assert is_supported_lora_module("o_proj", ["o_proj"]) + + def test_regex_suffix_matching(self): + """Regex anchors to end — partial suffix should not match.""" + assert not is_supported_lora_module("model.layers.0.self_attn.o_proj", ["proj"]) + + def test_empty_supported_modules(self): + assert not is_supported_lora_module("model.layers.0.self_attn.o_proj", []) + + def test_multiple_supported_modules(self): + supported = ["q_proj", "k_proj", "v_proj", "o_proj"] + assert is_supported_lora_module("model.layers.0.self_attn.v_proj", supported) + assert not is_supported_lora_module("model.layers.0.mlp.gate_proj", supported) + + +class TestIsInTargetModules: + """Tests for is_in_target_modules (deployment-time filter).""" + + def test_none_allows_all(self): + assert is_in_target_modules("model.layers.0.self_attn.o_proj", None) + + def test_suffix_in_target(self): + assert is_in_target_modules( + "model.layers.0.self_attn.o_proj", ["o_proj", "q_proj"] + ) + + def test_suffix_not_in_target(self): + assert not is_in_target_modules( + "model.layers.0.self_attn.o_proj", ["q_proj", "k_proj"] + ) + + def test_empty_target_modules(self): + assert not is_in_target_modules("model.layers.0.self_attn.o_proj", []) + + def test_exact_name_match(self): + assert is_in_target_modules("dense1", ["dense1", "dense2"]) + + def test_exact_name_no_match(self): + assert not is_in_target_modules("dense3", ["dense1", "dense2"]) + + def test_packed_parent_matches_child_target_modules(self): + assert is_in_target_modules( + "model.layers.0.mlp.gate_up_proj", + ["gate_proj", "up_proj"], + {"gate_up_proj": ["gate_proj", "up_proj"]}, + ) + + def test_packed_child_matches_parent_target_modules(self): + assert is_in_target_modules( + "model.layers.0.mlp.gate_proj", + ["gate_up_proj"], + {"gate_up_proj": ["gate_proj", "up_proj"]}, + ) + + def test_fused_parent_matches_child_target_modules(self): + assert is_in_target_modules( + "model.layers.0.self_attn.fused_qkv_a_proj", + ["q_a_proj", "kv_a_proj_with_mqa"], + {"fused_qkv_a_proj": ["q_a_proj", "kv_a_proj_with_mqa"]}, + ) diff --git a/asian_ai/performance/vLLM/tests/lora/test_minicpmv_tp.py b/asian_ai/performance/vLLM/tests/lora/test_minicpmv_tp.py new file mode 100644 index 0000000000000000000000000000000000000000..7e8723a604cd3d51d8e82a2ac973bbea5e8f0f62 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_minicpmv_tp.py @@ -0,0 +1,124 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest + +import vllm +from vllm.assets.image import ImageAsset +from vllm.lora.request import LoRARequest +from vllm.platforms import current_platform + +from ..utils import multi_gpu_test + +MODEL_PATH = "openbmb/MiniCPM-Llama3-V-2_5" + +PROMPT_TEMPLATE = ( + "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\n" + "(./)\nWhat is in the image?<|eot_id|>" + "<|start_header_id|>assistant<|end_header_id|>\n\n" +) + +IMAGE_ASSETS = [ + ImageAsset("stop_sign"), +] + +# After fine-tuning with LoRA, all generated content should start begin `A`. +EXPECTED_OUTPUT = [ + "A red and white stop sign with a Chinese archway in the background featuring red lanterns and gold accents.", # noqa: E501 +] + + +def do_sample(llm: vllm.LLM, lora_path: str, lora_id: int) -> list[str]: + sampling_params = vllm.SamplingParams( + temperature=0, + max_tokens=5, + stop_token_ids=[128001, 128009], # eos_id, eot_id + ) + + inputs = [ + { + "prompt": PROMPT_TEMPLATE, + "multi_modal_data": {"image": asset.pil_image}, + } + for asset in IMAGE_ASSETS + ] + + outputs = llm.generate( + inputs, + sampling_params, + lora_request=LoRARequest(str(lora_id), lora_id, lora_path) if lora_id else None, + ) + # Print the outputs. + generated_texts: list[str] = [] + for output in outputs: + generated_text = output.outputs[0].text.strip() + generated_texts.append(generated_text) + print(f"Generated text: {generated_text!r}") + return generated_texts + + +@pytest.mark.skipif( + current_platform.is_cuda_alike(), reason="Skipping to avoid redundant model tests" +) +def test_minicpmv_lora(minicpmv_lora_files): + llm = vllm.LLM( + MODEL_PATH, + max_num_seqs=2, + enable_lora=True, + max_loras=2, + max_lora_rank=8, + enforce_eager=True, + max_model_len=2048, + limit_mm_per_prompt={"image": 2, "video": 0}, + trust_remote_code=True, + ) + output1 = do_sample(llm, minicpmv_lora_files, lora_id=1) + for i in range(len(EXPECTED_OUTPUT)): + assert EXPECTED_OUTPUT[i].startswith(output1[i]) + output2 = do_sample(llm, minicpmv_lora_files, lora_id=2) + for i in range(len(EXPECTED_OUTPUT)): + assert EXPECTED_OUTPUT[i].startswith(output2[i]) + + +@pytest.mark.skipif( + current_platform.is_cuda_alike(), reason="Skipping to avoid redundant model tests" +) +@multi_gpu_test(num_gpus=4) +def test_minicpmv_tp4_wo_fully_sharded_loras(minicpmv_lora_files): + llm = vllm.LLM( + MODEL_PATH, + enable_lora=True, + max_num_seqs=2, + max_loras=4, + max_lora_rank=64, + tensor_parallel_size=4, + limit_mm_per_prompt={"image": 2, "video": 0}, + trust_remote_code=True, + ) + output_tp = do_sample(llm, minicpmv_lora_files, lora_id=1) + for i in range(len(EXPECTED_OUTPUT)): + assert EXPECTED_OUTPUT[i].startswith(output_tp[i]) + + +@pytest.mark.skipif( + current_platform.is_cuda_alike(), reason="Skipping to avoid redundant model tests" +) +@multi_gpu_test(num_gpus=4) +def test_minicpmv_tp4_fully_sharded_loras(minicpmv_lora_files): + llm = vllm.LLM( + MODEL_PATH, + enable_lora=True, + max_num_seqs=2, + max_loras=2, + max_lora_rank=8, + tensor_parallel_size=4, + trust_remote_code=True, + limit_mm_per_prompt={"image": 1, "video": 0}, + fully_sharded_loras=True, + ) + output_tp = do_sample(llm, minicpmv_lora_files, lora_id=1) + for i in range(len(EXPECTED_OUTPUT)): + assert EXPECTED_OUTPUT[i].startswith(output_tp[i]) + output_tp = do_sample(llm, minicpmv_lora_files, lora_id=2) + for i in range(len(EXPECTED_OUTPUT)): + assert EXPECTED_OUTPUT[i].startswith(output_tp[i]) diff --git a/asian_ai/performance/vLLM/tests/lora/test_mixtral.py b/asian_ai/performance/vLLM/tests/lora/test_mixtral.py new file mode 100644 index 0000000000000000000000000000000000000000..2a367e70c8f2f319403ca5f3ac487221eebb8d06 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_mixtral.py @@ -0,0 +1,77 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import pytest +import torch + +import vllm +from vllm.lora.request import LoRARequest +from vllm.platforms import current_platform + +MODEL_PATH = "mistralai/Mixtral-8x7B-Instruct-v0.1" + + +def do_sample( + llm: vllm.LLM, lora_path: str, lora_id: int, prompts: list[str] +) -> list[str]: + sampling_params = vllm.SamplingParams(temperature=0, max_tokens=256) + outputs = llm.generate( + prompts, + sampling_params, + lora_request=LoRARequest(str(lora_id), lora_id, lora_path) if lora_id else None, + ) + # Print the outputs. + generated_texts: list[str] = [] + for output in outputs: + prompt = output.prompt + generated_text = output.outputs[0].text.strip() + generated_texts.append(generated_text) + print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") + return generated_texts + + +@pytest.mark.parametrize("tp_size", [4]) +def test_mixtral_lora(mixtral_lora_files, tp_size): + """Original test, the LoRA model has the common target modules, not all""" + if ( + torch.accelerator.device_count() < tp_size + and tp_size > 1 + and current_platform.is_cuda_alike() + ): + pytest.skip(f"Not enough GPUs for tensor parallelism {tp_size}") + + prompts = [ + "[system] Given a target sentence construct the underlying meaning representation\nof the input sentence as a single function with attributes and attribute\nvalues. This function should describe the target string accurately and the\nfunction must be one of the following ['inform', 'request', 'give_opinion',\n'confirm', 'verify_attribute', 'suggest', 'request_explanation',\n'recommend', 'request_attribute'].\n\nThe attributes must be one of the following:\n['name', 'exp_release_date', 'release_year', 'developer', 'esrb', 'rating',\n'genres', 'player_perspective', 'has_multiplayer', 'platforms',\n'available_on_steam', 'has_linux_release', 'has_mac_release', 'specifier'] [/system] [user] Here is the target sentence:\nSpellForce 3 is a pretty bad game. The developer Grimlore Games is clearly a bunch of no-talent hacks, and 2017 was a terrible year for games anyway. [/user] [assistant]", # noqa: E501 + "[system] Given a target sentence construct the underlying meaning representation\nof the input sentence as a single function with attributes and attribute\nvalues. This function should describe the target string accurately and the\nfunction must be one of the following ['inform', 'request', 'give_opinion',\n'confirm', 'verify_attribute', 'suggest', 'request_explanation',\n'recommend', 'request_attribute'].\n\nThe attributes must be one of the following:\n['name', 'exp_release_date', 'release_year', 'developer', 'esrb', 'rating',\n'genres', 'player_perspective', 'has_multiplayer', 'platforms',\n'available_on_steam', 'has_linux_release', 'has_mac_release', 'specifier'] [/system] [user] Here is the target sentence:\nI wanted to like Grimlore Games' 2017 entry, but in SpellForce 3 they just didn't get anything right. [/user] [assistant]", # noqa: E501 + "[system] Given a target sentence construct the underlying meaning representation\nof the input sentence as a single function with attributes and attribute\nvalues. This function should describe the target string accurately and the\nfunction must be one of the following ['inform', 'request', 'give_opinion',\n'confirm', 'verify_attribute', 'suggest', 'request_explanation',\n'recommend', 'request_attribute'].\n\nThe attributes must be one of the following:\n['name', 'exp_release_date', 'release_year', 'developer', 'esrb', 'rating',\n'genres', 'player_perspective', 'has_multiplayer', 'platforms',\n'available_on_steam', 'has_linux_release', 'has_mac_release', 'specifier'] [/system] [user] Here is the target sentence:\nBioShock is a good role-playing, action-adventure, shooter that released for PlayStation, Xbox, and PC in 2007. It is available on Steam, and it has a Mac release but not a Linux release. [/user] [assistant]", # noqa: E501 + ] + + llm = vllm.LLM( + MODEL_PATH, + enable_lora=True, + max_num_seqs=16, + max_loras=4, + distributed_executor_backend="ray", + tensor_parallel_size=tp_size, + ) + + expected_lora_output = [ + [ + "give_opinion(name[SpellForce 3], release_year[2017], developer[Grimlore Games], rating[poor])" # noqa: E501 + ], + [ + "give_opinion(name[SpellForce 3], developer[Grimlore Games], release_year[2017], rating[poor])", # noqa: E501 + "give_opinion(name[SpellForce 3], release_year[2017], developer[Grimlore Games], rating[poor])", # noqa: E501 + ], + [ + "inform(name[BioShock], release_year[2007], rating[good], genres[action-adventure, role-playing, shooter], platforms[PlayStation, Xbox, PC], available_on_steam[yes], has_linux_release[no], has_mac_release[yes])" # noqa: E501 + ], + ] + + def check_outputs(generated: list[str]): + assert len(generated) == len(expected_lora_output) + for gen, gt_choices in zip(generated, expected_lora_output): + assert gen in gt_choices + + check_outputs(do_sample(llm, mixtral_lora_files, lora_id=1, prompts=prompts)) + check_outputs(do_sample(llm, mixtral_lora_files, lora_id=2, prompts=prompts)) diff --git a/asian_ai/performance/vLLM/tests/lora/test_moe_lora_align_sum.py b/asian_ai/performance/vLLM/tests/lora/test_moe_lora_align_sum.py new file mode 100644 index 0000000000000000000000000000000000000000..169b777e9e7b353bae86c6b6c9e6f6258cc49808 --- /dev/null +++ b/asian_ai/performance/vLLM/tests/lora/test_moe_lora_align_sum.py @@ -0,0 +1,325 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +import random + +import pytest +import torch + +from vllm import _custom_ops as ops +from vllm.platforms import current_platform + +DEVICE_TYPE = current_platform.device_type + + +def round_up(x, base): + return ((x + base - 1) // base) * base + + +def CEILDIV(x, y): + return (x + y - 1) // y + + +def sample_data(num_experts, max_loras, num_tokens, topk_num): + topk_ids = torch.zeros((num_tokens, topk_num), dtype=torch.int32) + token_lora_mapping = torch.zeros((num_tokens,), dtype=torch.int32) + + for i in range(num_tokens): + pool = list(range(num_experts)) + random.shuffle(pool) + for j in range(topk_num): + topk_ids[i, j] = pool[j] + token_lora_mapping[i] = random.randint(0, max_loras - 1) + + return topk_ids.to(DEVICE_TYPE), token_lora_mapping.to(DEVICE_TYPE) + + +@pytest.mark.parametrize("num_tokens", [100, 200, 1024, 4096]) # 81920 +@pytest.mark.parametrize("topk_num", [6]) +@pytest.mark.parametrize("num_experts", [64, 128, 256, 512]) +@pytest.mark.parametrize("max_loras", [2, 32]) +@pytest.mark.parametrize("block_size", [16]) +def test_moe_lora_align_block_size( + num_tokens, topk_num, num_experts, max_loras, block_size +): + # sample data + random.seed(1) + topk_ids, token_lora_mapping = sample_data( + num_experts, max_loras, num_tokens, topk_num + ) + + # compute paddings + max_num_tokens_padded = topk_ids.numel() + num_experts * (block_size - 1) + max_num_tokens_padded = round_up(max_num_tokens_padded, block_size) + if topk_ids.numel() < num_experts: + max_num_tokens_padded = topk_ids.numel() * block_size + max_num_m_blocks = CEILDIV(max_num_tokens_padded, block_size) + + # init output tensors + sorted_token_ids = torch.full( + (max_loras * max_num_tokens_padded,), + topk_ids.numel(), + dtype=torch.int32, + device=DEVICE_TYPE, + ) + expert_ids = torch.full( + (max_loras * max_num_m_blocks,), + num_experts, + dtype=torch.int32, + device=DEVICE_TYPE, + ) + num_tokens_post_pad = torch.zeros( + (max_loras,), dtype=torch.int32, device=DEVICE_TYPE + ) + adapter_enabled = torch.ones( + (max_loras + 1,), dtype=torch.int32, device=DEVICE_TYPE + ) + lora_ids = torch.arange(max_loras + 2, dtype=torch.int32, device=DEVICE_TYPE) + + # call kernel + ops.moe_lora_align_block_size( + topk_ids, + token_lora_mapping, + num_experts, + block_size, + max_loras, + max_num_tokens_padded, + max_num_m_blocks, + sorted_token_ids, + expert_ids, + num_tokens_post_pad, + adapter_enabled, + lora_ids, + ) + + # verify values + expert_ids = expert_ids.view(max_loras, -1) + sorted_token_ids = sorted_token_ids.view(max_loras, -1, block_size) + + for lora_idx in range(max_loras): + for token_idx in range(sorted_token_ids.size(1)): + block = sorted_token_ids[lora_idx][token_idx] + indices = block[block != topk_ids.numel()] + if indices.numel() > 0: + expert_id = expert_ids[lora_idx][token_idx] + assert torch.all(topk_ids.view(-1)[indices] == expert_id) + + +# Sentinel values for the regression tests below. Distinctive out-of-domain +# ints so that "kernel never wrote this slot" is directly observable: the +# kernel only ever writes a real expert id in [0, num_experts) or -1 +# (expert_ids), a token index or the `numel` padding value (sorted_token_ids), +# and a block-aligned cumsum count (num_tokens_post_pad). +SENTINEL_EXPERT = -2 +SENTINEL_TOKEN = -7 +SENTINEL_NPAD = -13 + + +def _build_and_run_align( + *, + num_lora_tokens, + num_base_tokens, + max_loras, + num_experts=64, + topk_num=6, + block_size=16, + lora_ids_override=None, + disabled_slots=(), + seed=1, +): + """Build inputs the way ``LoRAKernelMeta.prepare_tensors`` does, run + ``moe_lora_align_block_size``, and return a dict of result tensors plus + derived sizes. Output buffers are pre-filled with ``SENTINEL_*`` so + callers can assert which slots the kernel did / did not touch. + + Tokens are assigned to LoRA slot 0 (first ``num_lora_tokens``) then -1 + (remaining ``num_base_tokens``), matching the "mixed base + 1 LoRA" + shape used to repro vllm-project/vllm#32235. + + ``lora_ids_override``: optional 1-D int tensor of length ``max_loras+1`` + used verbatim. Default mirrors ``prepare_tensors`` (sorted-unique into + the head, -1 tail). + ``disabled_slots``: iterable of slot indices to clear in ``adapter_enabled``. + """ + random.seed(seed) + num_tokens = num_lora_tokens + num_base_tokens + assert num_tokens > 0, "test requires at least one token" + + topk_ids = torch.zeros((num_tokens, topk_num), dtype=torch.int32) + token_lora_mapping = torch.empty((num_tokens,), dtype=torch.int32) + for i in range(num_tokens): + pool = list(range(num_experts)) + random.shuffle(pool) + for j in range(topk_num): + topk_ids[i, j] = pool[j] + token_lora_mapping[i] = 0 if i < num_lora_tokens else -1 + topk_ids = topk_ids.to(DEVICE_TYPE) + token_lora_mapping = token_lora_mapping.to(DEVICE_TYPE) + + max_num_tokens_padded = topk_ids.numel() + num_experts * (block_size - 1) + max_num_tokens_padded = round_up(max_num_tokens_padded, block_size) + if topk_ids.numel() < num_experts: + max_num_tokens_padded = topk_ids.numel() * block_size + max_num_m_blocks = CEILDIV(max_num_tokens_padded, block_size) + + if lora_ids_override is None: + lora_ids = torch.full( + (max_loras + 1,), -1, dtype=torch.int32, device=DEVICE_TYPE + ) + unique_ids = torch.unique(token_lora_mapping, sorted=True) + lora_ids[: unique_ids.numel()] = unique_ids.to(torch.int32) + else: + assert lora_ids_override.numel() == max_loras + 1 + lora_ids = lora_ids_override.to(dtype=torch.int32, device=DEVICE_TYPE) + + adapter_enabled = torch.ones( + (max_loras + 1,), dtype=torch.int32, device=DEVICE_TYPE + ) + for slot in disabled_slots: + adapter_enabled[slot] = 0 + + sorted_token_ids = torch.full( + (max_loras * max_num_tokens_padded,), + SENTINEL_TOKEN, + dtype=torch.int32, + device=DEVICE_TYPE, + ) + expert_ids = torch.full( + (max_loras * max_num_m_blocks,), + SENTINEL_EXPERT, + dtype=torch.int32, + device=DEVICE_TYPE, + ) + num_tokens_post_pad = torch.full( + (max_loras,), SENTINEL_NPAD, dtype=torch.int32, device=DEVICE_TYPE + ) + + ops.moe_lora_align_block_size( + topk_ids, + token_lora_mapping, + num_experts, + block_size, + max_loras, + max_num_tokens_padded, + max_num_m_blocks, + sorted_token_ids, + expert_ids, + num_tokens_post_pad, + adapter_enabled, + lora_ids, + ) + + return { + "lora_ids": lora_ids, + "sorted_token_ids": sorted_token_ids, + "expert_ids": expert_ids, + "num_tokens_post_pad": num_tokens_post_pad, + "max_num_tokens_padded": max_num_tokens_padded, + "block_size": block_size, + "max_loras": max_loras, + } + + +@pytest.mark.parametrize( + "max_loras", + [ + 1, + 2, + ], +) +def test_moe_lora_align_block_size_mixed_base_and_lora(max_loras): + """Regression test for issue #32235: real LoRA slot must not be skipped + when ``active_lora_ids`` has -1 at position 0.""" + out = _build_and_run_align( + num_lora_tokens=8, num_base_tokens=8, max_loras=max_loras + ) + + # Sanity check on the layout being tested. + assert out["lora_ids"][0].item() == -1, ( + "prepare_tensors layout mismatch: -1 expected at position 0 for mixed batch" + ) + + real_slot = 0 + post_pad = out["num_tokens_post_pad"][real_slot].item() + assert post_pad != SENTINEL_NPAD, ( + f"num_tokens_post_pad[{real_slot}] was never written by the kernel; " + "the align kernel skipped the real LoRA slot." + ) + assert ( + 0 < post_pad <= out["max_num_tokens_padded"] + and post_pad % out["block_size"] == 0 + ), f"num_tokens_post_pad[{real_slot}]={post_pad} is not a valid block-aligned count" + + expert_row = out["expert_ids"].view(max_loras, -1)[real_slot] + assert (expert_row != SENTINEL_EXPERT).all(), ( + f"expert_ids row for slot {real_slot} has unwritten sentinel entries; " + "the align kernel skipped the real LoRA slot." + ) + + sorted_row = out["sorted_token_ids"].view(max_loras, -1)[real_slot] + assert (sorted_row != SENTINEL_TOKEN).all(), ( + f"sorted_token_ids row for slot {real_slot} has unwritten sentinel " + "entries; the align kernel skipped the real LoRA slot." + ) + + +def test_moe_lora_align_block_size_disabled_adapter_untouched(): + """Disabled-adapter slot rows must remain untouched by all three align + kernels. Pins the invariant protected by the ``adapter_enabled`` guard + in ``lora_count_and_sort_expert_tokens_kernel``: without it the sort + kernel reads uninitialized ``token_mask`` values for disabled slots and + pollutes ``sorted_token_ids`` / ``cumsum_buffer``.""" + max_loras = 1 + out = _build_and_run_align( + num_lora_tokens=16, + num_base_tokens=0, + max_loras=max_loras, + disabled_slots=(0,), + ) + # Sanity: slot 0 IS present in active_lora_ids (otherwise we would only + # exercise the lora_id == -1 / >= max_loras guards). + assert (out["lora_ids"] == 0).any().item() + + assert out["num_tokens_post_pad"][0].item() == SENTINEL_NPAD, ( + "num_tokens_post_pad[0] was modified for a disabled adapter slot." + ) + expert_row = out["expert_ids"].view(max_loras, -1)[0] + assert (expert_row == SENTINEL_EXPERT).all(), ( + "expert_ids row for disabled slot 0 was partially written." + ) + # Row specifically protected by the sort-kernel adapter_enabled guard. + sorted_row = out["sorted_token_ids"].view(max_loras, -1)[0] + assert (sorted_row == SENTINEL_TOKEN).all(), ( + "sorted_token_ids row for disabled slot 0 was polluted by the sort " + "kernel; lora_count_and_sort_expert_tokens_kernel must skip " + "adapter_enabled == 0 slots." + ) + + +def test_moe_lora_align_block_size_lora_id_oob_guard(): + """Regression test for the ``lora_id >= max_loras`` guard. + + Production ``LoRAKernelMeta.prepare_tensors`` pre-fills the tail of + ``active_lora_ids`` with -1, so the existing ``lora_id == -1`` check + covers the extra slot. This test bypasses that invariant and injects + an out-of-range value (5 with max_loras=1) at the tail to verify the + explicit guard prevents OOB reads against ``adapter_enabled`` and + OOB writes against the max_loras-sized output buffers. Without the + guard, an illegal-memory-access would surface on the next CUDA sync. + """ + max_loras = 1 + lora_ids_override = torch.tensor([0, 5], dtype=torch.int32) + out = _build_and_run_align( + num_lora_tokens=16, + num_base_tokens=0, + max_loras=max_loras, + lora_ids_override=lora_ids_override, + ) + # The .item() call below syncs and would surface any async + # illegal-memory-access from the OOB iteration. + assert out["num_tokens_post_pad"][0].item() != SENTINEL_NPAD, ( + "real LoRA slot 0 was skipped by the align kernel" + ) + + +if __name__ == "__main__": + pytest.main([__file__])